MOBILE DEVELOPMENT DAY TWO:
Hours' worth of hidden gotchas and forgotten details


My previous posts about making mobile apps with Phonegap/Cordova, ended with launching the simulators and running the sample app. Today's project was to get a have the page load to a simple Leaflet map centered on my location. This is about the simplest non-trivial app possible, but it took me about 4 hours because of the hidden gotchas.

So, here are a few of the snags I hit upon. If even one of them saves you even a half-hour of puzzling, you're welcome. :)


ANDROID: PHONEGAP FAILS TO CREATE ASSETS/WWW

After my first draft, I threw out my project folder and used ./create to make a new one as before.
 This new one lacked the assets/www folder and its behavior was "Hello World, YourAppNameHere!" Creating a new www folder, and copying the content of the demo app out of Phonegap's folder, didn't change anything. I even went as far as destroying the project folder, using ./create again, and getting the same result: no www, and creating one doesn't help.

The solution was to move Phonegap out of C:\Program Files where I had it. It is now C:\Users\Gregor\phonegap-2.2.0 and it creates new projects perfectly, even with their www content.

I don't know what was going wrong, as I'm an admin account, and copying files out of C:\Program Files shouldn't have been an issue.. But short version: My previous advice about using C:\Program Files may not work as well as I had thought, after a few more creations.


ANDROID: PUTTING THE APP ONTO A REAL PHONE

To deploy an app onto your phone, you will need the Google USB Driver. This allows communication between the Android SDK and the phone's innards. This is not the same as using your phone as a Disk Drive or USB Storage device.

• Go into the Start menu, type "android" You should see android.bat as an option. Click it.
• This opens the Android SDK Manager. Scroll down to Extras and make sure that Google USB Driver is listed as Installed. If it is not, check it, click Install 1 Package, and then close the SDK Manager.

The SDK Manager doesn't install the driver at all, but merely downloads the driver INF files. You still need to install them, and that process varies by your operating system. Fortunately there is a decent writeup about this already:
    http://developer.android.com/tools/extras/oem-usb.html#InstallingDriver

On my device (an HTC Incredible, which was hot stuff in 2010) the Google and OEM drivers do not work. For the HTC Incredible, I used HTC Sync from HTC's website. Despite its name, it also involves that special USB driver.

Once these two surprises were taken care of, I can now deploy my Leaflet app to my old phone, and it works A-OK.



IOS:APP SIGNING & PROVISIONING

When I tried to run my app on the borrowed iPad, it wouldn't build: The profile doesn't have a valid key for signing the app. Evidently deploying your own app onto your own iPad, you still need Apple's permission. And getting it was a half hour of looking up documentation about "provisioning profiles" so I could get permission to develop on my own iPad.

* Connect your iPad to your Mac. In XCode, look in the upper-right to find the Organizer, and look in the Devices section. You should see your iPad listed, including an Identifier, a Provisioning section, and a list of Applications which you have installed onto it.
* Do a Google search for "ios provisioning portal" and follow the link, logging in to the iOS Dev Center. Presumably you already have an account here if you signed up for a developer license.
* After you log in, follow the link for iOS Provisioning Portal, then use the Development Provisioning Assistant. This is a somewhat lengthy wizard, which will walk you through several steps.

Eventually, you will have imported your new identity into XCode and will have tagged your iPad with an identity as well, and Apple and XCode will allow you the privilege of installing programs onto your own hardware. (and folks gripe that Microsoft is invasive and controlling?)


ANDROID AND iOS CORDOVA.JS ARE DIFFERENT

This one caught me by surprise: I got my app loaded onto my iPad, but it wouldn't run. If I put alert() statements into index.js they would execute, but the deviceready event would never happen.

Turns out, cordova-2.2.0.js is not the same between platforms. The first 100 lines or so are the same (which is what threw me) but it's definitely the case that they are different later on.

In my case, I have the one codebase stored on a network drive, and I don't want to copy each file over before each editing session. My solution is as follows:

• I copied in both versions of the cordova.js, naming them IOS-cordova-2.2.0.js and ANDROID-cordova-2.2.0.js
• In index.html I have <script> tags for both versions, and one is commented out.
• When I want to compile or test on a platform, I comment and uncomment the appropriate <script> tag.

IOS: NETWORK WHITELISTING

The app was now almost running: it starts, and up comes Leaflet, but no tiles! Fortunately, XCode finally came through and gave me something to go on: "whitelist failed for http://a.tiles.mapbox.com/..."

A straightforward problem: iOS requires a whitelist if they're to make network access (Android does not). This is in the project's Cordova.plist file. Look for the ExternalHosts setting, and simple add an entry for * (yes, a single asterisk as a host name)

This will enable all hosts, and now my Leaflet app works.


It's up to you if you would rather list individual hosts. Personally I find that an unnecessary nuisance, since I'm the author of the app and I doubt that it could be tricked or hijacked to access other sites.



SUCCESS!

After learning and solving these hurdles (getting Apple's permission, finding HTC's special USB drivers, reinstalling Phonegap, and more) everything is working: One tap and I have a Leaflet map centered on my location.

Next steps: make this app something worthwhile...

Mobile Development with Phonegap / Cordova

Part 1 (again): Getting set up for iOS development on OSX

Last week I got my development environment set up for Android: Windows 7, Eclipse, and Phonegap/Cordova. I also got my iBook and set up Xcode, and got the demo app running on iOS. And here's the writeup.

Some if it may be a bit brief, as  I'm presuming you read my writeup about Android, where I describe some synonyms and terminology.




THE STACK

The way to use Cordova (aka Phonegap) to build iOS apps, is to use Xcode on a Mac. The basic stack is:

- A Mac. It must be running Mountain Lion (OSX 10.8) or later.
- An Apple ID so you can use the App Store
- Xcode 4.x
- Phonegap and its create script
- Your iOS device, or the built-in simulator

In fact, it is a violation of the iOS Terms Of Use, to develop an app for iOS not using Apple hardware and authorized tools. So this stack really is it. (comments about Microsoft and the anti-competition lawsuits of the late 1990s, tastefully deleted)

In my case, I hit up eBay and found a used iBook running Mountain Lion for $400.



GETTING AN APPLE ID AND IOS DEVELOPER LICENSE

If you want to deploy your mobile app via the App Store, you will need to extend your Apple ID with an iOS Developer License. This means visiting Apple's website (https://developer.apple.com/programs/ios/) and spending a half hour or so entering your information a few times. It also means spending $99 for a one-year registration as a Bona Fide Developer. (must fight... must not make... snarky comment)

This step is optional, required only once you're past the testing phase and want your app to be available to others.



INSTALLING XCODE

Fire up your Mac, open the App Store, so a search for Xcode. It comes right up, and a few clicks later it's downloading and installing. It's really that painless.

This is a 1.6 GB download, so on household DSL it took overnight. On a cable modem, you'll still have time to catch dinner and a couple episodes of Futurama...

After it's set up, start Xcode and then install the Xcode command-line tools: Preferences / Downloads / Components As long as I was there, I also had it grab a bunch more documentation and libraries. The whole shebang was another 1.1 GB, but again went painlessly.





INSTALLING PHONEGAP (CORDOVA)

Do a Google search for "phonegap download" and download the Cordova library. The version 2.2.0 is about 26 MB, and downloads in a few moments.

The Phonegap library doesn't really "install" but you extract it someplace. On my Mac (Mountain Lion, Safari, very default configuration) the Cordova library unzipped, leaving me a folder. I simply moved the folder into my Documents folder, and renamed it to phonegap



SET UP A PROJECT

Open up a Terminal (use the Spotlight, the magnifying glass, and type "terminal". The icon is a black square) and run these commands:

• ./Documents/phonegap/lib/ios/bin/create FOLDER_PATH PACKAGENAME PROJECTNAME
• Close the terminal

 FOLDER_PATH - This is the path to the intended folder for your new project, e.g. ./Documents/MobileApps/WalkTracker
PACKAGENAME - A dotted-format app name combined with domain name, e.g. org.greeninfo.walktracker
PROJECTNAME - The name of your app as you want it to appear, e.g. "Walk Tracker Calorie Counter"

Example (don't forget those quotes!):

./Documents/phonegap/lib/ios/bin/create./Documents/MobileApps/"Walk Tracker" org.greeninfo.walktracker "Walk Tracker Calorie Counter"

This will create a new folder, containing an Xcode project file.



RUN IT!

Open up Xcode and load that project file (or, simply open the project file).

In the upper-left is the familiar arrow button indicating Play. Pick your iOS simulator version, click the button, and there's your demo app.


CONCLUSION

Unlike Windows / Eclipse / Android setup, this one was a snap. Most of my time wasn't spent researching errors, but periodically checking a download bar. The create script ran without issues, and aside from download time I had the demo app running in 10 minutes.

The official version of this setup process is at Phonegap's own website. It's actually really accurate, and I admit that I had little to add except for narrative. (that's a compliment, Phonegap developers)

http://docs.phonegap.com/en/2.2.0/guide_getting-started_ios_index.md.html#Getting%20Started%20with%20iOS


NEXT STEPS

My goal for today (a break, on account of Veterans' Day) is to sit and play with both platforms and get a basic mobile app running. I have only vague ideas as to what it will do, but that's not the important part just yet. :)

Mobile Development with Phonegap / Cordova

Part 1: Getting set up for Android development on Windows 7

I got a new computer, which freed up my old one for dedicated use as a mobile development platform. My intended approach is to use Phonegap, so I can continue writing in HTML/CSS/JavaScript as I've been doing for 16 years. (then determine whether this method has problems such as slow speed, large app size, etc)

I run Windows and have an Android phone, so this first writeup is about those environments. A future writeup will cover Mac OSX, iOS, and xcode.

THE BASIC IDEA

The basic idea of Phonegap (sorry, I mean Cordova; they changed their name in 2012), is that you create a project folder which contains your HTML, CSS, and JavaScript assets, as well as the Phonegap library. Phonegap effectively acts as a web browser (but it's not), executing the JavaScript and rendering the HTML/CSS, and it brings some non-standard JavaScript features such as cross-domain loading, access to local files, and APIs to the phone's hardware.

Long story short: You start with a project folder, punch in HTML/CSS/JS, and export the whole shebang as an APK.

There are five basic parts to getting started with Phonegap:
* Java Development Kit (JDK), since Phonegap is Java
* Android SDK, since you're writing for Android
* Apache Ant, a build tool for Java apps
* The Cordova library itself, of course
* Eclipse SDK, the all-purpose editor to tie it all together

INSTALLING PREREQUISITES

Do a Google search for "jdk download", find Oracle, download the JDK 7 package, run the installer and click through the menus. Make a note of the directory (something like C:\Program Files\Java\jdk1.7) because you will need it later.

Do a Google search for "android sdk download", download the SDK installer, run it. I prefer to have it directly in my folder C:\Users\Gregor\AndroidSDK so it's easier to find and remember. Note that the SDK will later want to download files and create subfolders, so it should be in your documents and not someplace system-wide such as C:\Program Files. Wherever you install it, keep a note of the folder location because you will need it later.

Do a Google Search for "apache ant binary" and download a binary package. There is no installer, simply unzip it someplace. I put it under C:\Program Files\Apache Ant but had to provide admin privilege. Make a note of this directory, because you will need it later.

All three of these packages will involve some command-line usage later, so we have to set some environment variables.

• Go into your Start menu, type "environment" and select "Edit environment variables for your account"
• You'll get a panel listing a few variables such as TEMP, and some self-explanatory buttons such as New and Edit.
• Set JAVA_HOME to the installation directory for the JDK, e.g. C:\Program Files\Java\jdk1.7
• Set the PATH to include the JDK's bin directory, e.g. C:\Program Files\Java\jdk1.7\bin
• Set the PATH to include the Android SDK's tools subdirectory, e.g. C:\Users\Gregor\AndroidSDK\tools
• Set the PATH to include Apache Ant's bin subdirectory, e.g. C:\Program Files\Apache Ant\bin

Note that the syntax of a PATH is to join together multiple directory names with a semicolon ; character, like this:
C:\Program Files\Java\jdk1.7\bin;C:\Users\Gregor\AndroidSDK\tools;C:\Program Files\Apache Ant\bin
If you already had a PATH set, simply add a ; to the end of it, then add these three new ones.

Now test it by opening a command prompt (if you had one open already, close it and open a new one). Run the following commands and check the output:

# This should display the path of the JDK
echo %JAVA_HOME%

# This should run the Java compiler, which will show some usage help
"%JAVA_HOME%"\bin\javac

# This should run Ant, which will complain that build.xml doesn't exist
ant

# This should launch the Android SDK manager. just close it
android.bat

SETTING UP ECLIPSE & ANDROID

We're not finished yet!

Now Google and download Eclipse SDK (also called Eclipse Classic) and unpack it someplace. There is no installer, just unpack it someplace convenient. I like to put it in C:\Program Files\Eclipse and then drag it into my Start menu.

Now start Eclipse and install the Android Development Tools (ADT) plugin, which allows Eclipse to access the Android SDK.

• Start Eclipse
• Go into Help / Add New Software
• Add a new repository. name it "ADT Plugin" and use the URL https://dl-ssl.google.com/android/eclipse/
• It will show you a list of default packages which it will download. Click OK and let it download for a little while...
• When Eclipse restarts, you should be prompted by ADT asking where to find the Android SDK. Tell it to Use Existing SDKs, and then show it where you installed Android SDK.

I prefer to test my future apps on an Android emulator, so I can test multiple OS versions and not risk my phone while I tinker. As such, I will take an extra step here to install several more packages:

• Open Windows / Android SDK Manager, Look over the list of packages. For various versions of Android OS, the SDK platform and also the Intel x86 Atom System Image. You will need these for the versions of Android OS you want to support.
• Install the Extras / Intel x86 Emulator Accelerator (HAXM). This enables the Android Virtual Device (AVD) to run much faster than it would without.
• Click OK, Accept, and wait a while...

If you installed the HAXM, you will need to run an installer too; the SDK Manager downloaded but didn't really install it. Go into your Android SDK folder (C:\Users\Gregor\AndroidSDK\tools), go into Extras / Intel and look for the IntelHaxm.exe setup file. It's a standard wizard: double click, Next, Finish.

Now you're finally done setting up! It's time to make a project!

FINALLY, START A PROJECT!

Download the Phonegap / Cordova package from http://phonegap.com/download It is not an installer, just a ZIP file. Unpack it someplace convenient like C:\Program Files\Phonegap\Cordova 2.2.0

The Cordova distribution supports several different target platforms, all under the lib folder. Naturally, we want to use "lib/android"

To start a new project, use the command-line "create" script. This will create a folder containing various bootstrap elements for your new app: a copy of the Phonegap library, a folder for WWW assets with JS, CSS, and HTML subfolders, and so on.

• Open a command prompt
• Type: C:\Program Files\Phonegap\Cordova 2.2.0\lib\Android\bin\create FOLDER_PATH PACKAGENAME PROJECTNAME
• Exit the command prompt

 FOLDER_PATH - This is the path to the intended folder for your new project, e.g. C:\Users\Gregor\Mobile\Demo Project
PACKAGENAME - A dotted-format app name combined with domain name, e.g. org.greeninfo.demoproject1
PROJECTNAME - The name of your app as you want it to appear, e.g. "Demonstration Project"

Example (don't forget those quotes!):

C:\Program Files\Phonegap\Cordova 2.2.0\lib\Android\bin\create C:\Users\Gregor\Mobile\"Demo Project" org.greeninfo.demoproject1 "Demonstration Project"

Start up Eclipse. Go into File / New / Project, and a wizard will allow you to pick from a set of project templates. Select Android Project From Existing Code, then lead it to the project folder you just created.

You now have an Android project underway! The left-hand panel shows the files in your project (Package Explorer). The right-hand panel will show your program code.

TEST IT RIGHT AWAY

If you're like me, you'll immediately want to make test their own demo. (don't you hate it when someone's own Hello World doesn't work?)

In the Package Explorer (the left-hand side), scroll to the top and right-click on the root object, which is your project. Select Run As / Android Application.

There are three possible things that may happen here:

1. If your phone is attached and USB Debugging is enabled, or if you have previously created an Android Virtual Device (AVD) , you will be presented with a list of target devices. Pick the one you want to use.
2. If there is only one Android device (phone or AVD), it will be chosen automatically without prompting you.
3. If there are no Android devices available (no phones, no AVDs) then you will need to create one or attach one. This is a few New buttons, and selecting your choice of Android OS version (aka SDK level) and amount of virtual storage. It then lets you choose which one to use, or uses the only one.

Now the Android emulator will fire up. Like your real phone, it will take a brief slice of eternity. Eventually you'll unlock the "phone" and browse through the apps. Hey, there it is! Tap it and it works!



Note: I did get an error INSTALL_FAILED_OLDER_SDK and needed to make an adjustment to the Manifest file. See below.

SUCCESS! AND NEXT STEPS...

So far, so good. It was a long and tortuous road, but our environment is set up. For future projects, we just need to run "create" to bootstrap a folder, then open up Eclipse and Run As to test it.

The immediate next step will be development of the app, and this largely means editing HTML, CSS, and JavaScript in the assets folder. Eclipse has the build tools, but for the editing I may fall back to my trusted Komodo Edit.

A followup article will cover the development of my own Hello World app, packaging and permissions via the Manifest file, and actual deployment of a distributable APK file.

FOLLOWUP NOTES: ANDROID OS VERSION, aka SDK LEVEL

A note on terminology: Each version of the Android OS is also referred to as a SDK Level. Android 4.1 Jelly Bean is SDK Level 16. 4.0 Ice Cream Sandwich is SDK Level 15.

The Phonegap starter app probably didn't run when the emulator came up, giving an error of This means that your Manifest file (metadata and configuration about the app, device suport, etc) indicates that the app is for a different version of Android OS.

Check the AndroidManifest.xml file, down at the bottom look for android:minSdkVersion and android:targetSdkVersion In my case, the targetSdkVersion was __VERSION__ instead of a proper number. Setting this to a real number (16 for Jelly Bean) fixed it.

FOLLOWUP NOTES: TIPS AND TROUBLESHOOTING

A note about the Android SDK installation directory. I lied: I really put mine into C:\Program Files\Android SDK. This did mean some additional fussing, though: I had to provide admin privilege to install it, then I had to set the ownership of C:\Program Files\Android SDK to myself, and to grant Full Control to myself. If you have multiple accounts on your computer, this solution may not work. Keep in mind that the SDK will try to download and create files and folders here; if it fails when trying to install packages, check the ownership and permissions.

I chose to install Apache Ant and Cordova to C:\Program Files since it's a convenient place to keep them from cluttering up my documents or my desktop. I did need to provide admin privilege to put the folder there, but it works just fine after that because (unlike Android SDK) these won't make changes later and require special permission.

When running Cordova's "create", it has one very general error message: "missing one of the following". This means that one of those JAVA_HOME or PATH entries is wrong back when you set up JDK, Ant, and Android SDK. Run through those steps again, including the tests.

The Android emulator should be slow but reasonably fast, even on my two-year-old laptop. If it takes more than a few seconds to load, or is really slow, make sure that you have really installed HAXM. Remember, you must download the package via the SDK Manager and then run an installer wizard.

HAXM won't work with some older CPUs, particularly laptop CPUs before 2010. If your CPU is missing the virtualization features, either buy a newer computer or just accept that the emulator will be slow.

If you want to create more Android Virtual Devices (AVDs), e.g. for a new version of Android OS, open Eclipse, go into Window, and select the AVD Manager.

MAKING VERY LARGE REQUESTS TO WMS SERVERS

For purposes of printing, we need to fetch a giant image from a WMS. As in, 9 megapixels. Unfortunately, the WMS server isn't configured to allow requests of that size, and since we don't have an "in" with them we're not able to have them correct this.

http://gis4.oit.ohio.gov/arcgis/services/OSIP/MapServer/WMSServer?SERVICE=WMS&LAYERS=0&CRS=EPSG%3A4326&FORMAT=image%2Fpng&TRANSPARENT=FALSE&BGCOLOR=0xFFFFFF&REQUEST=GetMap&BBOX=41.383726033437%2C-81.70929151108008%2C41.39716625056923%2C-81.694329632648&STYLES=&VERSION=1.3.0&WIDTH=3236&HEIGHT=2906

Using tiles isn't an option, for reasons I won't get into here. Point is, we need a giant image from a WMS service that just won't give out images that large.

HORIZONTAL COMPOSITION PHP PROXY

My solution: I wrote a proxy server in PHP, which takes our large request, splits it into 4 logical requests, fetches the 4 images, and then composites them and returns that finished image. Ta-da, an interface that accepts any arbitrary parameters which the source WMS supports, but which also provides for very-large requests.

function ohioimagery() {
    $BASE_URL = "http://gis4.oit.ohio.gov/arcgis/services/OSIP/MapServer/WMSServer";

    // parse the bbox into 4 numbers: NOTE THIS IS NOT THE USUAL SEQUENCE
    // this is S W N E, and I have no idea how they do that. Maybe that's WMS 1.3.0 ?
    list($miny,$minx,$maxy,$maxx) = explode(",",$_GET['BBOX']);
    //list($minx,$miny,$maxx,$maxy) = explode(",",$_GET['BBOX']); // the normal way

    // calculate the W/S/E/N for the 4 quadrants, e.g. a logical "nw" box bounded by $nw_w $nw_s $nw_e $nw_n
    $halfx = ($maxx + $minx) / 2.0;
    $halfy = ($maxy + $miny) / 2.0;
    $nw_w = $minx; $nw_s = $halfy; $nw_e = $halfx; $nw_n = $maxy;
    $ne_w = $halfx; $ne_s = $halfy; $ne_e = $maxx; $ne_n = $maxy;
    $sw_w = $minx; $sw_s = $miny; $sw_e = $halfx; $sw_n = $halfy;
    $se_w = $halfx; $se_s = $miny; $se_e = $maxx; $se_n = $halfy;

    // split the width and height in half, but use subtraction so we don't fall victim to rounding
    $nw_width = $sw_width = round($_GET['WIDTH'] / 2.0);
    $ne_width = $se_width = $_GET['WIDTH'] - $nw_width;
    $nw_height = $ne_height = round($_GET['HEIGHT'] / 2.0);
    $sw_height = $se_height = $_GET['HEIGHT'] - $nw_height;

    // now generate a new set of WMS params and then URLs for each of the 4 boxes
    $nw = $_GET; $nw['WIDTH'] = $nw_width; $nw['HEIGHT'] = $nw_height; $nw['BBOX'] = "$nw_s,$nw_w,$nw_n,$nw_e";
    $ne = $_GET; $ne['WIDTH'] = $ne_width; $ne['HEIGHT'] = $ne_height; $ne['BBOX'] = "$ne_s,$ne_w,$ne_n,$ne_e";
    $sw = $_GET; $sw['WIDTH'] = $sw_width; $sw['HEIGHT'] = $sw_height; $sw['BBOX'] = "$sw_s,$sw_w,$sw_n,$sw_e";
    $se = $_GET; $se['WIDTH'] = $se_width; $se['HEIGHT'] = $se_height; $se['BBOX'] = "$se_s,$se_w,$se_n,$se_e";
    $nw_url = $BASE_URL . '?' . http_build_query($nw);
    $ne_url = $BASE_URL . '?' . http_build_query($ne);
    $sw_url = $BASE_URL . '?' . http_build_query($sw);
    $se_url = $BASE_URL . '?' . http_build_query($se);
    /*
    header("Content-type: text/plain");
    print "ORIGINAL\n"; print_r($_GET); print "\n\n";
    print "NW\n"; print_r($nw); print "\n\n";
    print "NE\n"; print_r($ne); print "\n\n";
    print "SW\n"; print_r($sw); print "\n\n";
    print "SE\n"; print_r($se); print "\n\n";
    print "NW\n$nw_url\n";
    print "NE\n$ne_url\n";
    print "SW\n$sw_url\n";
    print "SE\n$se_url\n";
    */

    // fetch the four images and composite them into a single canvas
    $canvas = imagecreatetruecolor($_GET['WIDTH'],$_GET['HEIGHT']);
    $nw_img = imagecreatefrompng($nw_url);
    $ne_img = imagecreatefrompng($ne_url);
    $sw_img = imagecreatefrompng($sw_url);
    $se_img = imagecreatefrompng($se_url);
    imagecopy($canvas, $nw_img, $sw_width-1, $ne_height-1, 0, 0, imagesx($nw_img), imagesy($nw_img) );
    imagecopy($canvas, $ne_img, $nw_width-1, 0, 0, 0, imagesx($ne_img), imagesy($ne_img) );
    imagecopy($canvas, $sw_img, 0, $nw_height-1, 0, 0, imagesx($sw_img), imagesy($sw_img) );
    imagecopy($canvas, $se_img, 0, 0, 0, 0, imagesx($se_img), imagesy($se_img) );

    // spit it back out
    header("Content-type: image/png");
    imagepng($canvas, null, 0);
}

The result, is that I can now make exactly the same WMS request, along with whatever changes in layers, styles, and SRS, and get back oversized images.

http://myserver/proxy.php?SERVICE=WMS&LAYERS=0&CRS=EPSG:4326&FORMAT=image%2Fpng&TRANSPARENT=FALSE&HEIGHT=2906&BGCOLOR=0xFFFFFF&REQUEST=GetMap&BBOX=41.370285816305135,-81.70929151108008,41.39716625056923,-81.67936775421546&WIDTH=3235&STYLES=&VERSION=1.3.0

WATCH THAT BBOX

For reasons I do not know, the remote WMS takes their BBOX parameter in an unusual order. Instead of the usual W,S,E,N they do S,W,N,E The code above has been modified to fit their bizarre case, and I have noted it.

WHY WON'T IT FIND A ROUTE? BOUNDING BOX AND TOLERANCE

Today's post is fairly brief, but is the culmination of eight hours of tediously trying routes and adjustments. If it saves you one hour, won't you consider it worthwhile? :)

So, we have a routing network of trails for a Parks N Rec district. Directions via dijkstra_sp_delta() are working A-OK and have been for some months. But, the client wrote back with some news: he's having trouble finding a longer path across this one park.

I tried it too: about half of the routes would work, and half would not. I could route the 15 miles to a location, but move the marker a few hundred feet in any direction, and it may or may not be able to route there, even though my eyes can see the trail segments laid out.

It's consistent too: the same point would always work or always fail. And I followed it back into the SQL and verified that the given pair of nodes always would or would not have a route. So it's not some sysadmin issue such as out of memory or core dumps.

TOLERANCE

The directions handed back to the browser, include WKT of the segments so they can be plotted onto the map. I noticed that the route as given would skip from one trail to an adjacent one: walking a straight line, it would suddenly be 20 feet to the left on an adjacent trail, and sometimes back to the original trail. The visual effect was rather goofy and unattractive.

And, I suspect that this was routing us onto adjacent trails which would later diverge, and coming up with false negatives.

The cause here, is a too-high tolerance (25 feet) when generating the routing network via assign_vertex_id() The data had some known "dangles" due to human error in digitizing the trails, and a higher tolerance fixed that. We reanalyzed with a tolerance of 10 feet, and this stopped happening.

This solved an annoyance, but didn't get us any closer to finding the routes. Turns out that wasn't the problem at all...

BOUNDING BOX

Ultimately I discovered it:the directions stuff uses dijkstra_sp_delta() with a tolerance of 2640 feet. That is to say, the calculation excludes any segments that are not within a 2640-foot bounding box of the start and end vertices. That excludes the 900 square miles that can't possibly be relevant to this route, and gives us plenty of wiggle room to find routes.

Wrong.

In the case of this trails data, 5280 was the magic number. The trails meander enough, that a good path really can deviate by 1 mile away from the straight-line between the two points. And the performance loss isn't that bad, certainly not as bad as complete non-performance!

Anyway, no huge breakthrough here, but a word for someone else who just isn't finding the path: check your delta bounding box, and see if you're being too strict.

My goodness, it's been some 3 weeks since my last post. Been too busy DOING instead of WRITING.  :)

SOME STEPS IN OPTIMIZING A MAP'S LOADING SPEED

A few days ago, a client asked me to look over a map app and see what I could do to speed it up. The app is ExtJS, PHP, PostGIS, and MapServer. I made some substantial improvements, and here's what we did and how well it worked.

Nothing I'll mention today is  particularly cutting edge, and some of them date back to 2007 and MapServer 5.0 But, obviously here in 2011 and 2012, some folks didn't know about it, so here goes.

THE EXTJS LIBRARY / APACHE COMPRESSION WINS THE DAY

 The first roadblock of the app loading and starting, was  the download of ext-all.js, the core of the ExtJS JavaScript framework. They were loading it from Google's CDN, and it's 700 KB. On my 1 Mbps household DSL, with other PNGs and CSSs downloading too, this was taking about 10 seconds. The theory of using a CDN is that Google's network is faster than their server's pipe, but fact is that for DSL users it's still 10 seconds.

I instead had it load a copy of ExtJS from the web server, and then enabled compression (see here). This is an old trick from my sysadmin days, and BOY DID IT WORK here. The download was now 135 KB (use Firebug, watch the Content-length header) and took about 2 seconds.

Ta-da! Eight seconds shaved off the page startup, by not blindly assuming that Google's CDN is the panacea.

MULTIPLE HOSTNAMES

An old trick, but one that was not being implemented here. The second argument to the OpenLayers.Layer.WMS constructor is the URL of the WMS, or an array of URLs, in which case it will round-robin requests to these URLs. See here.

In the case of a tile-cached layer, this is a great boon. The browser has built-in limits about how many requests it will make to a webserver, and when your tiles are served up quickly enough, more time is spent on the browser's politeness, than is spent fetching and transferring.

So I swapped this:

var countylines = new OpenLayers.Layer.WMS("Counties", "/tilecache.cgi", { blah blah });

For this:

var TILECACHE = [
    "http://maps1.example.com//tilecache.cgi",
    "http://maps2.example.com//tilecache.cgi",
    "http://maps3.example.com//tilecache.cgi",
    "http://maps4.example.com//tilecache.cgi",
    "http://maps5.example.com//tilecache.cgi",
    "http://maps6.example.com//tilecache.cgi"
];
var countylines = new OpenLayers.Layer.WMS("Counties", TILECACHE, { blah blah });

Now the web server spends as much time download as it does blocking (Firebug is great), and the difference is noticeable.

You don't want to go too far with this trick. If the layer requires calculation, or is slow at all, or isn't tile cached, this may not be a good move. If your layer is slow, hitting it 6 times simultaneously won't be any faster. Even if the layer is plenty fast, once your browser is opening up 24 requests at a time to your web server instead of 4, your webserver may start blocking clients because of the flooding. Watch your MaxClients setting and your Apache error log, and consider what will happen if this app gets a sudden surge in popularity.

CENTROIDS FOR LABELS

The map features a grid of sampling quads, and each quad has an icon in the center indicating a status. The quad rectangles and the icons in the center, are separate layers. The icons use a URL param for filtering, so one can specify a parameter and show the relevant quad icons (in this case, the ID# of an invasive species, e.g. species_id=123). This layer is not served via TileCache, due to the dynamic nature of the queries, so its performance is naturally slow. But of course, we want it faster.

To place these icons, the DATA statement was as follows:

DATA "the_centroid from (select ST_centroid(merc_geom) AS the_centroid,* from quads where
%SPECIESFILTER%) as temp using unique quad_id using SRID=900913"

 A simple optimization here was to precalculate the centroid, as it's not going to change. Second, I rewrote the mapfile syntax for readability. Third, the species_id which would be used in the %SPECIESFILTER% wasn't even indexed. I suspect that the spatial index would supersede the species_id index, but this is also used in PHP-based reports so I know that indexing it can't hurt.

# some SQL to precalculate and index
SELECT ADDGEOMETRYCOLUMN('','quads','merc_center',900913,'POINT',2);
CREATE INDEX quads_merc_center ON quads USING GIST (merc_center);
UPDATE  quads SET merc_center=ST_CENTROID(merc_geom);
CREATE INDEX quads_species_id ON quads (species_id);

# revised mapfile
DATA "merc_center from quads using unique quad_id using SRID=900913"
FILTER "%SPECIESFILTER%" 

The net result is that the queries happen in 20 milliseconds instead of 250 milliseconds. Not bad, shaving about one-quarter second off the processing time for each tile.

MAPSERVER IMAGE QUALITY: AGG/PNG vs GD/GIF

One of the avenues I investigated, was the image tiles they're downloading. The app has multiple hostnames configured,, so the browser is downloading some 8 tiles at a time. And these are pre-rendered so they should be super quick.

But they're each 50 KB to 100 KB in size. Which means that a full page load of 20 tiles (4 wide, 5 high) is about 1.5 MB. The download itself takes 15 seconds, not counting wait times and server times. Again, this is where performance testing benefits from my having crummy household DSL.

The tiles use MapServer's wonderful AGG library (the same one that powers Mapnik) and output as gorgeous PNG-24 with porcelain-smooth line edges, which is good when we're drawing rivers and county lines. But man, 100 KB average tile size?

I switched over to GD/GIF This is the older renderer, and the line quality is poor. But the rendering time is about 1/3 that of AGG, and file size is magnificent: 5 KB - 10 KB now. I presented it to the client, and they loved it: the grainy lines were a small price to pay for having a map view load up in 3-8 seconds instead of 20-60.

WHAT'S WITH THOSE BLANK LAYERS?

And lastly, I noticed that one of the layers was always blank. The roads layer was turned on at all zoom levels, but the MapServer mapfile stated that roads don't come on until we're at 1:500,000 scale (MAXSCALEDENOM). That is to say, the browser was downloading 20 tiles with each page view, when we knew for a fact that those tiles are 100% transparent. Even without the transfer time, the browser was blocking while it loaded these tiles.


The solution here was simple: specify a maxResolution or maxScale setting for the OpenLayers.Layer.WMS constructor, so the layer doesn't even try to load tiles until we get into the usable zoom levels.

TILECACHE IN MOD_PYTHON (no)

I also investigated using TileCache via mod_python. The theory here is that the Python interpreter and various modules are cached into RAM, giving a performance boost over CGI.

In our tests, we didn't go with this. The difference in start-to-finish request time, as measured via Firebug, was negligible if it was real at all. But the memory consumption is remarkable: Apache bloated up to 1 GB size within a few hours of testing.

This particular client is hosted in a VPS, and they only have 2 GB of RAM and no swap. So having 1 GB locked in memory was not welcome, especially without corresponding speed gains.

THANKS FOR READING

Well, that's it. Nothing new or awesome, but a few tricks which weren't known to these developers, and therefore which may be new and useful to you. Happy mapping!

ESTIMATING HIKING TIME ALONG A SEGMENT - TOBLER


The simplest way to calculate travel time over your route segments, is to make an assumption about walking speed, e.g. a constant 3 miles per hour. However, this client wanted to account for slope a la Tobler's hiking function, since this is substantially hilly terrain and a constant walking speed is not necessarily realistic.

This process requires 3D data, containing elevation. If your data is only 2D, get a DEM and load up ArcGIS and produce 3D data, aka LINESTRING Z. Don't worry, I'll wait...

And now it's fairly easy:

-- Supplement each trail segment with a length. The example here transforms the geometry to
-- a State Plane SRS with feet as units.
ALTER TABLE trails ADD COLUMN length_feet float;
UPDATE trails SET length_feet = ST_LENGTH(ST_TRANSFORM(the_geom,3734));

-- Now the Tobler calculation. We fetch the elevation of the first and last
-- vertices of the linestring, divide by length, and that's the slope.
-- Remember, we split our segments into 100-foot lengths so this is mostly accurate.
-- If you keep yours as mile-long trails your accuracy may vary.
-- The 0.911344 converts from Tobler's own units (kilometers per hour)
-- to our preferred units (feet per second, to match the length in feet)
ALTER TABLE routing_trails ADD COLUMN tobler_fps FLOAT;
UPDATE routing_trails SET tobler_fps =
0.911344 * 6 * EXP(
-3.5 * ABS(
    (
      ST_Z(ST_StartPoint(ST_TRANSFORM(the_geom,3734)))
      -
      ST_Z(ST_EndPoint(ST_TRANSFORM(the_geom,3734)))
    )/length )
);

-- Now that each segment has its speed in feet per second and its length in feet,
-- the duration in seconds is simple. In our case, we calculate for three different
-- travel modes and we assume that bicyclists are being safe and only doing 3X as fast as walking.
UPDATE routing_trails SET duration_hike = length_feet / tobler_fps;
UPDATE routing_trails SET duration_bike = duration_hike * 0.33;
UPDATE routing_trails SET duration_horse = duration_hike * 0.66;

Discussion

I don't really know how accurate Tobler's function is, nor the degree to which it makes a difference in the accuracy of route duration estimates. Still, our original approach is to presume 3 miles per hour (4.4 feet per second) on foot, 3X that for bicycles, and 1.5X that for equestrian. Taking the slope into account may miss a lot of factors, but it can't be any worse!