Not dead, just busy!

Happy thanksgiving holiday.
It's been some 7 weeks since my last post, simply because I've been pretty occupied with stuff at work and haven't come across anything really blog-worthy lately. But it's coming.

Some brief bits and pieces:

• Been crazy-busy at work, including a server migration in a few days over the TG weekend. Huzzah for virtualization and how it makes migrations relatively simple. Ask me about it some time.
• OSX Yosemite trashed my Mac, so I spent a day reinstalling the OS and then the whole build chain. My own blogs came in quite handy here, and I'm glad to say they're still quite applicable. The only annoyance was that virtually every package in the stack had an update: iOS 8 meant updating Xcode, meant updating to Yosemite, ... (the second upgrade from Snow Leopard to Yosemite went just fine, no sweat)
• I've begun porting over an old jQuery Mobile application from 2012, into Cordova to form a mobile app. I don't expect much in the way of surprises, but it will be the most complex phone app I've ever done. Fortunately the already-working prototype provides a clear picture of expectations and some lessons on what works well and what can be streamlined.
• Still no progress on trying to find a new HTML5 framework for building mobile apps. Just too busy on other tasks. Though the mental process is guiding me toward something very much like Ratchet but with a real page-turning behavior. (I mentioned that a while ago) I do hope to get some slack time soon to tinker around and find/invent something.

So yeah... Not a lot worth blogging, but a promise that more will come soon. :)

Raspberry Pi binary clock

And as promised, here is my exciting, awesome world-changing invention... a Raspberry Pi application which shows the time... in binary! Okay, not that killer, but a simple project demonstrating some wiring, some code, and a visual output.

You'll need:

• your Raspi and a GPIO pinout diagram 
• a breadboard
• 5 LEDs
• some jumper wires
• a Python library called RPi.GPIO

Hardware (GPIO, Wires, LEDs)

The circuitry itself is dead simple: just five LEDs in a row, with their anodes (+ side, long leg) connected to 5 GPIO pins and their cathodes (- end, short leg) connected to the GND. Just pick any five GPIO pins, connect your GND to the breadboard, and insert the LEDs.

Here it is, as both a photo and  a circuit diagram.

The choice of GPIO pins is up to you. I picked the 5 nearest the end of the bank on my B+.

Question: Why would I show only the last 5 bits of the time, instead of a nice nerdy number like 8? I didn't have 8 LEDs: I lost a bag of two LEDs and burned out another LED during an experiment, so I only had 5.

The RPi.GPIO Library

My favorite GPIO programming API, is RPi.GPIO

You can grab a copy and install it manually, here on its Python package page

Or you can let the package manager do it for you:   sudo apt-get install python-rpi.gpio

The Program (Python)

# CONSTANTS
LIGHT_PIN_0 = 21
LIGHT_PIN_1 = 20
LIGHT_PIN_2 = 16
LIGHT_PIN_3 = 12
LIGHT_PIN_4 = 26

# SETUP
import RPi.GPIO as GPIO
from time import sleep
from time import time as epoch

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

# initialization: setup the pins for output
# and set all pins HIGH for an initial blast of light, like how a car tests its lights
def init():
    GPIO.cleanup()
    GPIO.setup(LIGHT_PIN_0, GPIO.OUT)
    GPIO.setup(LIGHT_PIN_1, GPIO.OUT)
    GPIO.setup(LIGHT_PIN_2, GPIO.OUT)
    GPIO.setup(LIGHT_PIN_3, GPIO.OUT)
    GPIO.setup(LIGHT_PIN_4, GPIO.OUT)

    GPIO.output(LIGHT_PIN_0, GPIO.HIGH)
    GPIO.output(LIGHT_PIN_1, GPIO.HIGH)
    GPIO.output(LIGHT_PIN_2, GPIO.HIGH)
    GPIO.output(LIGHT_PIN_3, GPIO.HIGH)
    GPIO.output(LIGHT_PIN_4, GPIO.HIGH)

# convert an interger into binary, then pull out the 1s and 0s to light the pins
# count back from the end, so we get the last 5 digits; the first 5 digits won't change for years
def updateLightsFromInteger(value):
    bits = bin(value)
    GPIO.output(LIGHT_PIN_0, bits[-1] == '1')
    GPIO.output(LIGHT_PIN_1, bits[-2] == '1')
    GPIO.output(LIGHT_PIN_2, bits[-3] == '1')
    GPIO.output(LIGHT_PIN_3, bits[-4] == '1')
    GPIO.output(LIGHT_PIN_4, bits[-5] == '1')

# THE MAIN LOOP
# JUST SIT AND UPDATE THE BLINKY, AND WHEN IT'S TIME TO QUIT BE GRACEFUL ABOUT IT
if __name__ == '__main__':
    init()
    sleep(3)
    print("Running. Hit ctrl-c to terminate.")

    try:
        while True:
            sleep(1)
            value = round(epoch())
            updateLightsFromInteger(value)
    except KeyboardInterrupt:
        print('Shutting down')
        GPIO.cleanup()

Pretty darn simple, huh? Just keep looping forever, fetching the time and converting it into 1s and 0s for the lights. And I'm almost ashamed at how cute and entrancing it is... shiny...

For yourself, feel free to pack the LEDs more tightly and add a few more. With only 5 bits, it recycles every 32 seconds, but if it has 12 it would recycle every 4096 seconds (68 minutes).


My next project should be more interesting. As soon as I quit... watching the... blinky... lights...

Raspberry Pi and a brief primer on GPIO (still coming, a binary clock!)

I mentioned that I got my Raspberry Pi B+ last week. Sadly I've had fairly little free time to play with it. Still, a few days ago, about 24 hours after I received it, I put together a cute program I thought I'd share. It's a clock that counts out the time in binary. Nothing too earth-shaking, but a little more involved than the tutorials.


Before I get into that project, though, I want to lay out some information on how GPIO works.

My goal is to provide a tutorial and technical explanation, for those who already know what a Python comment is, but who need something more meaty than "here's a photo my my Pi, now copy-paste this Python code" The tutorials are cool, but they often lack background information and (for example, in my first project) I had to find a different site to explain how GPIO ports really work since the light wasn't blinking when it should have been.

First off, go find a copy of the Raspberry Pi GPIO diagram for your model. My personal favorite is this one for the B+. And I'll assume that you know a little bit about electricity, e.g. that GND means "ground" or "negative terminal".

GPIO ! OMG !

What's GPIO? General Purpose Input and Output. It means that there's a bank of pins on the computer, and that these pins can be connected to things that generate input (buttons, thermometers, motion sensors) and to things that generate output (LEDs, motors and relays, radio transmitters). The GPIO bank also has several pins to act as grounds (- terminals) and power supplies (+ terminals at both 3.3V and 5V).

What decides whether a GPIO pin is input or output, and when it generates voltage? Software! That's the awesome part. The same GPIO pin may attach to a button (input) in one program or to a LED to light up (output) in another program, and that choice of behaviors is controlled by your Python program.

GPIO Output

In Output mode, what defines whether a GPIO output is on or off?

Well, that's a little more involved...

When a GPIO pin is defined as Output (or GPIO.OUT in Python) the pin can be in either a High state or a Low state (GPIO.HIGH and GPIO.LOW, or True/False, or 1/0).

• When it's in a High state the pin is generating +3V DC If you connect that pin to the anode (+ side) of a LED and connect the LED's cathode (- side) to the ground, the LED would light.
• When it's in a Low state, the pin is grounded and acts as a negative terminal. In this case, you could still connect an LED but the - would go to the GPIO pin and the + would go to a power source.

So it's not quite as simple as "on and off" or "the LED lights when the pin is High" as much as "the GPIO pin will go to +3V (High) or to 0V (Low), and the behavior of your circuit depends on the structure of your circuit"

Check out the following photo and diagram.

In my initial experiments, I simply didn't quite grok this simple relationship between voltage and tyhe High/Low state, and I had plugged in an LED backwards. As a result, High meant that the LED was off and Low turned it on -- the opposite of my intent! It all makes sense now that I know these basics.

GPIO Input

When a GPIO pin is set to receive input, what is it looking for? A button being pushed, or what?

When a GPIO pin is set to GPIO.IN mode, it listens for incoming voltage, e.g. +3V being applied to the GPIO pin. When there is voltage coming in, the pin reads High. And when there is no voltage... it reads randomly!

A pin in GPIO.IN state is floating, meaning that it is not clearly energized to +3V nor is it clearly grounded out. If you have a Python program doing While True: print input(12); you'll see that the input fluctuates between 0 and 1, even if there's nothing connected to the pin at all!

This is not a signal from your button, but simply line noise. To correct for this, you must pull down the line voltage to force it to clearly be Low. This means to connect the GPIO pin to ground so it is constantly draining voltage (and constantly reads 0 or Low), until such time as you hit the switch and connect +3V to the GPIO input.

It's not as convenient as having just the button that is clearly reading High or Low. On higher end hardware they have cleaner line voltage and built-in pull down mechanisms. But not here.

That resistor is important! When you engage the switch, you're connecting +3V directly to GND, and that's a short circuit. Putting in the resistor, reduces the voltage sufficiently so as not to reboot your Raspi with a short circuit, while also "encouraging" the power to flow up the GPIO pin.

So Now What ?

Well, those are the rudiments of GPIO, that took me an evening to figure out. Hopefully it saved you some time.

Coming shortly, the walkthrough of my binary clock.

HTML5 mobile framework: part 3, Bootstrap & Ratchet

Sorry to keep splitting up threads here. A few weeks back, I posted that we're using jQuery Mobile to make our mobile apps, and that while it's good it's not awesome. So the R&D continued for a new HTML5 framework to build our mobile apps using Cordova/Phonegap.

Again, it's opinions and impressions as I try things out to see how they fit our needs. If your opinion varies please share it in the comments! I'd love your input on HTML5 and hybrid mobile apps.

Here's the next one I tried out (this was a good 3 weeks ago): Ratchet.

Bootstrap / Ratchet

Bootstrap is a rich set of CSS classes and mobile-friendly behaviors. It looks a little bit cartoonish for my tastes, and it really stands out where it's in use, but it is the current vogue and it is quite flexible as far as modifying the theme. You start with a rich set of utility classes but you're not constrained to use only what you're given. That counts for a lot!

However, Bootstrap is not a mobile development framework. There is no page swapping, no interception of # hyperlinks to swap which panel/page is currently showing, and so on. And while it is mobile-friendly and responsive, it is not designed to "look native".

Ratchet attempts to fill this gap by adding more CSS classes for listviews, thumbnail icons, badges and mini icons that float to the right-hand side, and so on. Ratchet was created for quickly prototyping mobile apps using HTML web pages: "visit this URL; after we create the app, here's what it would look like."


I was quite excited to find this, since my own idea had been to write some jQuery wrappers over Bootstrap to roll our own framework. But as they say, why reinvent the wheel? (obviously a phrase from before we had Github, where literally thousands of projects are out there reinventing wheels)

What I saw, got me excited quickly:

• It's pretty, it's pure CSS. It's tiny. It's a superset of Bootstrap.
• We're not tied down to a specific development technique. It's just CSS.
• The documentation is quite well organized, and there's not much of it.
• Separate CSS for iOS and Android, trying for a more-native look. Using Cordova's merge capability, this could turn out well.
• A nice set of glyphicons.
• Again, it's just a bunch of CSS, so could work with any programming and eventing toolkit, e.g. jQuery. (I repeat this cuz it's really core to the uber-flexible "style first then redesign repeatedly" approach we need)
• There's virtually no overhead, so rendering and response is quite swift.

But then things took a turn as I tried to use it to develop a real app:

• No events. We'd need to add jQuery to add click and tap handlers, which is fine. Then again, this could be addressed with TouchSwipe, adding swipe and pinch handlers to jQuery. Not a problem, but does illustrate that Ratchet is intended for prototyping apps, not for building real apps.
• The prefab behaviors do not include popup dialogs. Again, not killer but if we want to show a Please Wait there's no supplied way to do that.

But then the show-stopper!

• All that page-changing behavior is done with Push.js, which loads HTML pages from the server. It does not (as far as I could tell from the source and the docs) in any way support switching pages within the document. You're loading up HTML pages, not switching visibility to show and hide a specific panel.
• This means that you can't just switch over to the results page; you load results.html from the server (or from the in-app device storage)... but without any of your DOM changes. A typical use case, is a search function which populates a results list on another page, then switches over to that page. Even if you got the page-change timing right so the results content were visible at the right time... switching away from the results and then back again would effectively lose the results.

So ultimately, we had to say no here and shelve it. Push.js is cute for what it is, but isn't suitable for even a moderately simple use case of showing/hiding/adding DOM elements in another panel. (again, Ratchet doesn't claim to be this)

But it's not over...

Still, I'm reluctant to let it go entirely. My current line of thinking, is to make a fork of Ratchet minus Push.js, then replace Push.js with something that simply switches which <div class="page"> is currently visible. Given this, we would have a nice set of CSS utility classes and a homegrown page-management framework. All that's missing is support for dialogs, which doesn't sound difficult on the surface.

So this one was shelved, but remains on the radar. It's the closest thing I've found so far, to the ideally-minimal CSS toolkit we need. I'm gonna try out a few more such as Ionic and Sencha,before I go for reinventing a framework.

My first tastes of Raspberry Pi (and next, a binary clock project!)

I received my Raspberry Pi on Monday night. It's quite nifty.

What Is It ?

A Raspberry Pi is essentially a small computer, the size of a pack of playing cards. It has a microprocessor (ARM 700 MHz), 512 MB of RAM, 4 USB ports, and an Ethernet adapter... and it can run Linux. That means that any USB stuff that works on Linux should work here, e.g. wifi adapters, packet radio adapters, even a GSM adapter for cell service. Even better -- it has a bank of GPIO and I2C slots so you can plug in physical inputs and outputs such as temperature sensors, motors and relays, motion sensors, cameras, LEDs, LCD readouts, touchscreens, and serial I/O such as GPSs. And it all runs off a USB phone charger, and doesn't even have a fan to make noise.

More info here:  http://www.raspberrypi.org/products/model-b-plus/

The whole concept of the Raspberry Pi is a dirt-cheap prototyping platform. Remember those electronics kits we had as kids, where you wire up resistors to antennas and make a crystal radio? Same idea, except one of the components is a Linux computer connected to the Internet. And that computer has 26 software-controlled input and output ports.

And of course, there's a real thrill and "tee-hee" value to a working computer that's the size of a hamburger patty, which runs X Window and a SMTP server and a MySQL server, and just about anything else that runs under Debian Linux.

Anyway, it's not really map related at all but is some cool stuff. In upcoming weeks, expect a few raspi postings mingled in with my usual map postings and my (overdue) series on choosing a HTML5 framework for mobile dev.

Shellshock! A forgotten feature becomes a major security hole

My prior post, was that my next series would be about HTML5 frameworks for mobile. But I've been busy on other projects, and mobile stuff hasn't been on the high-pri list. Speaking of today's high-pri list, how about that Shellshock bash bug?

I'm gonna jump on the bandwagon a bit here,  and admit that this bug should scare the heck out of you. If you run anything that hands out shells to the unknown Internet, then anybody smart enough to make a fake User Agent can run arbitrary shell commands on your server. I tried it and verified it: it took me 30 minutes to write a working exploit, starting from when I first typed "bash bug" into Google.

In a Nutshell

What is the "bash" Unix shell? Briefly, this means "the thing that's used to type commands after you connect via SSH" but for web folks working in PHP this also means "the thing used to execute command-line scripts such as passthru() and the `` operator" If you use shell_exec() et cetera then you're using a shell, and it's probably bash.

Bash has a feature going back some 25 years, that it can auto-execute code when you start a new shell. This code will conveniently set "environment variables" which facilitate other work. The problem with this very-obscure feature, was only discovered last week: Web programs such as CGI and PHP also set environment variables, some of them read directly from the browser without any sanitization or checking. So any browser on the Internet, may be able to:

• Send an environment variable that will be passed as-is to your shell. The most common such variable would be your User Agent (your browser's name), which is loaded as-is into an environment variable called HTTP_USER_AGENT
• Which your shell would then interpret as a command to be executed before it executes your shell command.

But that means technical skill to set your browser's User Agent, right? Wrong. Browser add-ons already exist to set your User Agent string to some common values (Firefox can pretend it's GoogleBot) and it's also trivially simple to set it in your browser's config. (Firefox is described below) Again, I hadn't heard about this bug before today and it took me about 30 minutes to craft an exploit against one of our servers which was vulnerable.

The Limits

The exploit allows command execution, only if the following conditions are met:

• A remotely-executable program exists which does not require trust and authentication, e.g. a PHP script or a CGI script which executes under bash. Services which do not execute shells, or which execute them only for trusted individuals, are of significantly less concern, e.g. FTP and MySQL servers don't run subshells and don't read user-supplied data for environment variables anyway, and SSH only gives you a shell after you have logged in.
• That program executes a subshell. In PHP this is the `` or shell_exec() functions. Any program which does not use those functions is fine... until you need to use those functions.
• The system's default shell /bin/sh is in fact Bash and not an alternative such as ksh or dash. On Ubuntu 12.04 /bin/sh is in fact Dash and is not affected by this bug unless you have a CGI script which specifically uses Bash.

A Quick Test

Login to your server, and run this from your shell:

env x='() { :;}; echo vulnerable' /bin/sh -c "echo this is a test"

Hopefully you'll see an error about a function definition:
    ./bash: warning: x: ignoring function definition attempt

If you see "vulnerable" in the output, then your system shell is Bash and is a version which is potentially affected by this issue. And remember, it's not Bash here at the command-line that should worry you, but any web program which launches shells.

If you don't see vulnerable, you may not be entirely safe. Try again, this time replacing /bin/sh with /bin/bash Even if Bash isn't your system default shell, any CGI programs or launched shells whicyh specifically use Bash may still be affected.

A Quick Exploit (A)

Like I said, it's not rocket science, just setting your User Agent.

Load up Firefox.

Go into about:config Create a new value (or edit if it already exists) called

general.useragent.override

and set it to this odd-looking string:

() { :; }; touch /tmp/shellshock

Congratulations, your browser will now supply an auto-executing function in the environment... if you hit up something that spawns a subshell.

On your webserver, try creating this very small PHP program. Or, try simply hitting any program of your own which uses shell_exec() or similar functions.

<?php
# anything that uses any shell function, will trigger the bad HTTP_USER_AGENT variable
print "Tmp Folder:\n";
passthru('/bin/ls /tmp');

Now visit that PHP program in your browser, and then have your sysadmin check out the new /tmp/shellshock file. And imagine for a moment what happens if someone were to do this repeatedly, and knew their way around a shell.

P.S.: Don't forget to "reset" that User Agent override when you're done.

Another Quick Exploit (B)

For command-line wonks, here's an even simpler version of the exploit using cURL from the command line:

curl 'http://www.example.org/whatever.php' -H 'User-Agent: () { :; }; touch /tmp/shellshock'

Point it at any web service which uses a subshell (e.g. a PHP using shell_exec() ) and they'll be surprised to find a new file on their server.

Okay, I'm Scared... Now What?

The most expedient fix, is to upgrade bash. There is already a patch (4.3-25) available.

This page gives practically copy-and-paste instructions:
    https://news.ycombinator.com/item?id=8364385

Now give everything a test again, and breathe a sigh of relief!

Update: Looks like the 25 patch to Bash fixes only one of two so-far-determined methods of attack. A second version has been named CVE-2014-7169 and is not fixed by 4.3-25  The "A Quick Test" for this second vulnerability is:

env X='() { (a)=>\' sh -c "echo date"; cat echo

If the output has the date in it, then the date command was executed and you're still vulnerable.

What HTML5 mobile framework to use?

Yeah, it's been a month. Again, been working instead of talking about working. :)

Today I'll start touching on a topic as emotionally charged as religion and politics... a choice of HTML5 mobile frameworks for use with Cordova/Phonegap. That's me, always the troll, pushing the envelope of good taste and social acceptability, pushing buttons just to get a reaction. ;)

I've been using jQuery Mobile, but it has some bugs, and isn't exactly swift and fluid; on the other hand, it does have some nice button icons and default styles, and works well with jQuery. But there are other HTML5 mobile frameworks out there which may be more fluid, less buggy, have richer icons and styles.

So this next series of postings isn't really a "How I did it" sort of tech bragging, as much as current opinions on several HTML5-for-mobile setups. Some of them are subject to change as I learn something later, so if your opinion differs, this is a place where your comments would be appreciated!

Our Needs

jQuery is very visual -- you start with a HTML mockup and then you make it interactive. This is how our clients and designers think. They start with cosmetic mockups and designs, which will change often (sometimes 2-3 times in a week) as more visually-thinking people give input and do 180-degree spec changes. The "real programmers" shudder when I press the point, that the design (as implemented in HTML) defines the application, and that project specifications are typically defined after the fact based on budget leftovers and feedback. But that's the reality. We don't need a framework to build apps out of widgets, as much as a HTML/JS toolkit for making app-like behaviors starting with HTML.

• We're building our apps with Cordova / Phonegap. Virtually everything should work well with Cordova, but then again let's not presume. If the final product doesn't work well once we've plugged it into Phonegap and tried it on a few devices, it's a loss. (see Ionic in a future posting)
• The HTML/JS API must not be dogmatic, forcing us to use prefab widgets that cannot be quickly customized. The designers and the clients decide what we do; they can be quite capricious, and we never say no to any customization. If they want THIS selector to have its handle on the left while OTHERS have the handle on the right, that needs to happen and pronto. Right off the bat, this means that a lot of programming-heavy frameworks which let you "just tie widgets together" via a configuration file, are right out. (see Sencha in a future posting)
• But that said, we  do want the prefab widgets... just ones that we can ignore or modify, preferably in HTML. Utility classes are awesome, and so are glyphicons. (see jQuery Mobile and Ratchet in a future posting)
• We do most of our programming in jQuery, so anything that plays well with jQuery is a big bonus.

jQuery Mobile

Well, let's start with the framework we are already using.

- Good. It's highly visual. As I described above, you start with your HTML and then program behaviors into it. This fits perfectly into the paradigm of our project managers, designers, and clients.

- Good. A very large community and great documentation. There isn't much about jQuery Mobile that remains mysterious after five minutes in Google and StackOverflow and/or ten minutes with JQM's docs. We still need to do some tedious CSS discovery sometimes, but 90% of questions have already been answered.

- Good. Page management. Attractive cross-platform CSS. Decent widget set. Custom event handlers and DOM manipulation. It's pretty sweet.

- Bad. It's a little slow. Even with FastClick, there is a noticeable lag between tapping a button and going somewhere. I've tried changing the delay, disabling page animations, FastClick of course, ... It's not killer slow, but it's not really smooth.

- Bad. Rendering bugs. I find myself implementing corrective CSS on every new project. a) Hyperlinks that are a[data-role="button"] has a visible underlines on the text. This is easily corrected in CSS, but seems a silly thing to need to correct. b) Buttons in the header, will sometimes "stick" in their highlighted/active state long after the finger has moved on, sometimes even after switching to a different page, and then coming back minutes later. This means that the active color needs to be the same as the inactive color (CSS fixes), which sort of misses the point.

- Mediocre. The data-role paradigm is nice, but not as rich and flexible as a good set of utility classes, e.g. class="button button-panic button-wide"  We get two themes (data-theme="a" and data-theme="b"), but need to write custom HTML to simulate a third theme if we want a third button color (but need to replicate that for header buttons and so on. By comparison, Bootstrap and Ionic have several types of button.

- Dicey. Lazy rendering. One of the first executing statements in our programs is $('div[data-role="page"]').page() to effectively disable lazy rendering. Our typical use case, is that a listview needs to be populated programmatically, but we cannot guarantee that the user has previously visited the page and caused the listview to render into DOM, allowing the refresh to happen. Lazy rendering is intended to reduce memory usage, but realistically it's just a race condition waiting to happen.


So, what are other options? That's the focus of upcoming articles...

MobileMapStarter 1.1.1

Due to the surprising popularity of MobileMapStarter at FOSS4G, I pushed through a few things I'd been meaning to do, added a license, etc. I'll just quote from the release blurb:

Get a jumpstart on building your mobile mapping application!

MobileMapStarter is a ready-to-run Cordova/Phonegap application. It combined jQuery Mobile and Leaflet, as well as a host of workarounds, considerations, utility functions, etc. to get you running a mobile app in minutes.

1.1.1 improves upon 1.1.0 in the following ways:

• Includes the whole Cordova release known stable at this time (incl the File plugin specifically @1.0.1) so this thing really is ready to rock.
• Various documentation updates.
• Added a license (MIT) and clarified licenses, added required Leaflet license (etc.) to generated app
• Replaced the geocoder from Bing to Nominatim, to avoid potential TOU issues

So, go get some!

Thanks so much for the interest. It was very encouraging, and I feel jazzed. :)

FOSS4G was awesome!

Well, that was fun!

Some highlights of FOSS4G, in no specific order. There'll be some things to re-evaluate, some to prioritize and try to integrate into the next year at work, some side projects after hours, ...

FOSS4G was close by! It's only in the USA every third year, and last time (2011) it was in Denver. But this time, it was conveniently located only 2 hours away. Too far to go home every night, but close enough to stay with a friend and not need a plane ticket.

Meeting up with colleagues is one of my favorite parts. Before I get caught up in the deluge of data and techniques, I should thank again the folks I met at the conference: old colleagues like Karsten and Stephen and Matt and Bob, and new colleagues such as Tanner and Jim. It's been a pleasure meeting you; please follow my profile links and get in touch.

My presentation on the RCS Viewer was something of a dud. It seems that the 10am timeslot the morning after the party, on the last day (short day) just gets poor attendance. That's too bad: the RCS Viewer combines some of the best features for advanced GIS analysis with a simple "GIS whiteboard" as well as PDF take-aways. If you're interested, click here to see the slideshow and be sure to read the notes on each slide for some additional annotations describing the spoken part.

On the other hand, my presentation on MobileMapStarter was a real hit. I felt like a nervous, babbling idiot up there, but feedback afterward said something very different. Several folks came up afterward with compliments, good questions, intentions to use MobileMapStarter, ... and questions about licensing, because they want a clearly-stated license so they can get started using it. Huzzah! Don't worry folks, an explicit license is coming right up this weekend. If you're interested, check out the slideshow and again be sure to read the notes on each slide for the spoken material and other side notes. (oh, Chapter 1 and 2, on setting up the Cordova build environment are back up; see July 5 or so!)

MapServer 7 sounds like a small update, though in fact it seems to have involved a 25% rewrite of MapServer. Why do we use MapServer? It's lightweight and fast, configuration is dead simple (sorry, I do not consider it a painful chore to write mapfiles), you can dump temporary mapfiles to disk and use them for a session, it does variable interpolation, ... MapServer 7 brings performance improvements, then the ProTips had some useful tips, some of which I didn't already know. And lastly, MapCache is yet another tile caching system but one perhaps more flexible than TileCache and TileStache (it can handle raw WMS requests that do not fit tile boxes).

CartoDB's presentation on performance and caching was moderately enlightening. Perhaps the biggest take-home here, was the nature of their caching: it's not a tile cache holding a zillion PNGs, as much as a Varnish cache for the vector tiles that are to be rendered. Making a DB update hits a trigger that issues a varnishctl command. In conversation that day I had guessed about 2/3 of this just mentating on how I would do it... good to know that I was on the same brainwave as the smart guys. :) But it gets me thinking about whether Varnish or even CloudFront could do us some good over at GreenInfo Network, for general caching of not only map tiles but static assets such as PNGs and CSS files. (you know, network contention when downloading the 30+ assets which form a page)

RasterIO (not Raster I/O, but Rrrrrrrrasterio!) is yet another Python-NumPy-GDAL wrapper. But, it's a step beyond all the prior ones such as agoodle and even rasterstats. a) It wraps both GDAL and OGR, so you can handle both shapefiles and GeoTIFFs (et cetera). b) It uses a "more Pythonic" programming interface, e.g. using .read() and .write() c) It may consume far less memory than agoodle by virtue of being able to do a read() on a specific window of the raster, rather than loading the whole thing into memory. For one specific application where we use agoodle, this could be a game-changer.

Then some breathtaking demos of 3D data, such as Cesium. Mostly it was sit-and-spin sort of deals: load 3D data (NED), render a mesh, drape the photo (NAIP) as a texture, and there's a spinny 3D mountain demo, super cool. This is the part I find most visually intriguing, though at the same time the most baffling as to what anybody would do with it, at least as far as my customers who find a "find nearest park" application to suit their needs. I would love to find a use case for some awesome 3D stuff.

And Galileo seems like a very practical phone app that I'll probably download later this weekend. It loads OSM extracts (e.g. states in the USA) and provides a slick little pan-zoom UI. Imagine Google's "Maps" app or Apple Maps, but... without Google and Apple. If I heard correctly, it's for iOS only, which would be too bad since I'm an Android man. But I'll get on the newsletter and keep listening.

Google Closure Compiler keeps coming up. We're already in the practice of minifying our JavaScript code, but Closure goes an extra step and optimizes the code for run time performance. I've gotta give a look to this, see what differences we can achieve in file size and also runtime, compared to our existing applications.

Whew! The above are just the highest highlights that I'll be investigating at work, not even the really experimental stuff for when I get a slow day. WHAT A WHIRLWIND!

THANK YOU SO MUCH, FOSS4G!

FOSS4G here I come!

FOSS4G is in Portland, Oregon this year, a mere 2 hours away from my home. Huzzah!

I'm giving two talks:

• Wednesday afternoon (tomorrow!) I'm showing off MobileMapStarter. This is a starting template in HTML5, for a mobile app for use in Cordova/Phonegap. Copy, paste, and have a mobile app running in literally 10 minutes (5 if you use Phonegap Build).
• Friday morning, Tommy Albo will help me present the Intertwine's Regional Conservation Strategy viewer. This excellent web app, lets someone draw or upload an area and get back statistics about habitat valuation, park land, etc. The idea is that city planners and land developers can become aware that the area they're about to raze is actually valuable, and find an alternate location.

Beyond that, I'll be meeting friends, coworkers, and colleagues! Stephen from Cleveland. Karsten from Seattle. Keith from here in Corvallis. Tom from San Francisco.  I'm sure most attendees will agree, that hobnobbing with map nerds, is at least as great as the organized talks and events. :)

Mobile Development with Cordova (Phonegap): Afterword

I'd like to again thank GreenInfo Network, my generous employers, for giving me the time to learn, then giving me the okay to share the knowledge. Those 13 chapters are a distillation of real-world pitfalls and techniques, beyond the copy-and-paste world of books. I hope it helps. :)

Mobile Development with Cordova (Phonegap): Appendix C

Appendix C: Phonegap Build

Phonegap was originally an Adobe project -- and still is, Cordova is the open source version. Phonegap Build is a service provided by Adobe, to simplify the process of compiling your apps.

 

Simply put, you upload a ZIP file containing your www content (HTML, JS, CSS files, jQuery, et al) and simply upload it to the PGB website. They will then build the app for iOS, Android, and other platforms. This takes a lot of the overhead out of development, since you don't need a Mac, Xcode, Apache ant, a local copy of Cordova, etc.

But it's not a panacea:

• You do still need a Mac in order to use the Keychain Access program, to generate certificates and to convert certificates into the P12 format required by Phonegap Build.
• There is a limit of 10 MB of content in the ZIP file. This is quite generous for some apps, but our apps tend to include JSON and GeoJSON databases, pushing them over the limit or not leaving comfortable space for growth. There are techniques for downloading the database file over the network and storing on disk, so it doesn't need to be bundled with the ZIP file. But those are more complicated, and not as "zero maintenance" as simply bundling data that won't change except between versions.
• The cycle of waiting tends to be several minutes. Using local builds on the Mac, I can have turnarounds in 60 seconds on the iPad. On Android, I need to transfer and download the APK, which is ultimately about as time-consuming as the Build service.
• Android apps built in the PhoneGap Build, will not run in singleTop mode. This means that any time someone leaves the app, then resumes it by tapping the icon, it starts all over again starting with the splash screen and then the welcome page. This has been a reported bug for two years...

Because of these limitations, we don't use it anymore. But it certainly bears mention in any publication on the topic of PhoneGap and Cordova.

Mobile Development with Cordova (Phonegap): Appendix B

Appendix B: Generating a Provisioning Profile (iOS)

Your Apple developer career is managed by the Provisioning Portal. As described earlier, your key components are:

• A developer certificate
• An app ID, which is simply a logical grouping of applications
• A list of devices for beta testing
• A provisioning profile linking all of these together

Getting started:

• Log in to the Provisioning Portal
  https://developer.apple.com/
• Go into Certificates, Identifiers & Profiles
  If you don't already have a Certificate, generate one and load it onto your Mac.
  Make it a Production certificate, for "App Store and Ad Hoc"
• Defined an App ID.
  The simplest choice here is to make a wildcard matching your organization, e.g. org.greeninfo.* Using a wildcard will allow this provisioning profile to match multiple apps, as long as they start with org.greeninfo
• Enter the UDIDs for your beta tester devices
  Again, only devices listed here AND chosen for a Provisioning Profile can effectively beta test.
• Generate a Provisioning Profile
  Tie together your developer identity, your app, and those devices, into a single logical grouping.

When you're setting up your mobile app, you'll define a Team in Xcode, and Xcode will download your certificate and Provisioning Profile for you. You will find yourself doing this relatively often as project momentum builds and beta testers want to come on board. You will need to regenerate a provisioning profile and downloaded into Xcode every time. See the chapter about beta testers for more info.

Do not forget the 100 device limit. If you're doing apps for 5 or 6 companies, and each one wants 20 people to check out the app, you could run out of slots.

 

 

Mobile Development with Cordova (Phonegap): Appendix A

Appendix A: Generating a Certificate Keystore (Android)

Android apps must have a digital signature. For Android, you generate your own certificate, which is saved inside a "keystore" file. You generate this with the keytool command-line tool, which is part of the Android SDK you downloaded while setting up your build environment.

Generating a Keystore

The command below generates an Android keystore named android_certificate.keystore containing a single certificate named android, good for about 25 years (10,000 days), and with a nice large key size (8kB is current standard).
keytool -genkey -v -keyalg RSA -keysize 8192 -validity 10000 -keystore android_certificate.keystore -alias android

You will be prompted to enter a password, then for your "first and last name" which really means any identifier such as your company or department name, then other identifying info such as your location. Lastly you'll be asked for a second password to lock the keystore (as contrasted to the certificate itself) but I like to just use the same password for both.

In the end you'll have a keystore file. Back it up into a safe place along with the alias and password.

Warning: The certificate and keystore must be the same as you sign of future versions of the same app. If you lose the key store or forget the alias or the password, you will not be able to sign future versions of the app. The result is that anybody trying to upgrade, will fail: they will need to uninstall the previous version of the app before installing the upgrade.
 

Using the keystore in Cordova CLI Tools

An app compiled with cordova build android --release will not be properly signed and able to install onto phones, unless you instruct ant to use the keystore.

You can specify the path to the keystore file, as well as the alias, in the ant.properties file. This is described in the  Setting Up Certificates part of chapter 3 when setting up a new project.

Tip: I always use cordova build android --release so it prompts for a real certificate. This adds an extra step to the build/test cycle, but does ensure that the selfsame APK can be distributed to others (it's easy to forget, and send an APK that's not signed). It also ensures that I in fact have the keystore handy, as I am unlikely to lose it and then lose my backups over time, if I'm actively using it every day of development.

Mobile Development with Cordova (Phonegap): Chapter 10

Chapter 10: Distributing the App to Stores

Once you've created your mobile app and tested it thoroughly, it's time to distribute it!
There are multiple avenues for distribution: the Apple App Store for iOS, and two major app stores for Android.

Android: Google Play

This is arguably the most popular of the app stores for Android.
Sign-up is based on the same Google account that you probably already use for Gmail or Google Docs.

https://play.google.com/apps/publish/signup/

On the Dashboard, use the "Publish an Android App on Google Play"

There are rather a lot of questions if you're establishing a new app, though many can be skipped if you're simply updating a new APK. The most time-consuming part is making up the rather large set of icons and screenshots. It took me about 45 minutes my first time.

Android: Amazon App Store

This is arguably the second most popular of the app stores for Android.
Sign-up is based on the same account that you already use for shopping at Amazon.com.

https://developer.amazon.com/public/

On the Dashboard, use the "Add a New App" wizard.

There are rather a lot of questions if you're establishing a new app, though many can be skipped if you're simply updating a new APK. The most time-consuming part is making up the rather large set of icons and screenshots. It took me about 45 minutes my first time.

iOS: Apple App Store

As in most of the steps previous, submitting to Apple's App Store is more complicated than for the Android stores, but then again also more polished in some ways.

As a quick overview, there are three steps to submitting an app to the store, once you're done testing it extensively.

• Create an App Record in iTunes Connect (the web portal for managing App Store apps). This is only necessary if you're submitting a new app, not for upgrading an existing app.
• Build an Archive of it (the IPA file) and sign it with a distribution Provisioning Profile intended for Store Distribution. In that same Archive operation, proceed to upload it to the App Store.
• Wait for approval, or for an explanation of why the app was rejected.

Creating the App Record

Creating an App Record is just what it sounds like: creating the app's description, screenshots, etc.  Naturally, you only do this for a new app and not for every upgrade of the app.

This is done via the iTunes Connect website https://itunesconnect.apple.com/
Follow the Manage Your Apps option, and the Add New App button.
It's a tedious process like all the app stores, especially the need for various screenshots and splash screens.

Tip: Apple requires screenshots from a variety of devices, including an iPad, a 3.5" phone, and a 4" phone. As such, you and your beta testing team will need at least one of each of these devices. Apple is very clear that you cannot simply resize an existing image, and I have a feeling that they really are paying attention.



At this point, if you visit your app dashboard on iTunes Connect, you should see that it is waiting for upload.

Building and Uploading the Archive

Open up the Xcode project file, and make sure you have a physical device attached. The archive option will not be available, unless you have a physical device attached.

Go into Project/Archive, and follow the wizard same as you had when you are making Ad Hoc builds for your beta testers. But this time, select a Provisioning Profile intended for store distribution. Then simply walk through the wizard.

At this point, if you visit your app dashboard on iTunes Connect, you should see that it is waiting for approval.


Tip: You app record must be Waiting For Upload. If you already uploaded, but want to make a change, you must cancel your previous upload via the iTunes Connect website, which puts the app back into a Waiting For Upload state.



Approval

Apple is proud of their stringent testing process. I wasn't happy about it first, but I must admit that it's a good thing. They test on a variety of devices, and they make sure that your app isn't consuming the users' valuable iCloud space.

But keep at it, follow their instructions, and eventually you'll get through.

Mobile Development with Cordova (Phonegap): Chapter 9

Chapter 9: Debugging with weinre

Smartphones and iPads do not have Firebug, F12 Tools, etc. but there is a tool called weinre which can fill the same role. weinre is Web Inspector REmote. The author says it can be pronounced like "winery" or like "wiener" which I consider a great show of sportsmanship and humor on his part. :-)

A basic overview:

• weinre runs as a server listening on a socket
• it's written in Node, so will run on OSX, Windows, and Linux
• your index.html embeds a <script> tag which loads weinre into the running mobile application
• you point your PC's browser at weinre's console page, and get a virtual console much like Chrome's F12 tools, with the other end connected to the mobile app

So, steps:

Install NodeJS and weinre
These are well documented on their respective websites, and I don't need to repeat it.

Start up weinre and note the port number
Again, this is documented as part of weinre's own process. The important part is to take note of the TCP port number, then to verify that this port is allowed into your PCs firewall. Linux and OS X, you would adjust your firewall settings to specifically allow this port number. On Windows, you would enable that port specifically for the application node.exe

In our case, we set up weinre on our web server instead of on a local PC, so that we could use it on our various projects without dealing with the variations in our PCs' individual firewalls. This also allows us interesting opportunities, such as having me check out the console while the app runs on someone else's phone -- something you can't do if your weinre is running on localhost on your LAN.

Add the <script> tag
Insert this code into index.html:

<script src="http://192.168.100.1:8080/target/target-script-min.js#YOURAPPNAME"></script>

Replace YOURAPPNAME with some meaningful name for you app or session. It could be your app's name, your own name, anything that would keep two people from becoming confused if they were using weinre at the same time.

Of course, replace 192.168.100.1 with the hostname of your own weinre server, be that your own PC or the one you run on your website.

Open your browser
Then visit the corresponding URL where YOURAPPNAME matches whatever you used in the app:

http://192.168.100.1:8080/client/#YOURAPPNAME

This opens up the virtual console, where you can begin inspecting the HTML, inserting JavaScript via the console, and so on.




Actually having the F12 developer tools available, is a huge step forward in being able to debug the app once it's on your phone. If you go with the web server setup like we did, you can even have one of your beta testers run the app as you load the console. This is invaluable when debugging issues that only happen on their hardware and not yours.

Mobile Development with Cordova (Phonegap): Chapter 8

Chapter 8: Tips on Writing Your App

Now that you can compile and distribute your apps, the hard part is past... and now comes the lengthy part, actually writing out your application, then revising it repeatedly as specs change and budgets dwindle. That's all on you: if you're using this tutorial then you should know enough about web development to do that on your own.

But, here are a few pointers and pitfalls that may help you out.

MobileMapStarter

First, I'll get the self-promotion out of the way. I have written up a starter app, which includes jQuery mobile, leaflet for doing maps, a good starting place for a welcome page, a settings panel, and so on. You may want to check it out.

https://github.com/gregallensworth/MobileMapStarter

Choosing a HTML/CSS framework

jQuery Mobile is quite popular, and we use it quite a lot here at GreenInfo Network. The big bonus is that it is jQuery and therefore ubiquitous, with a very large community and a startling number of plug-ins, hacks, and known workarounds. It is also very much based on HTML, and therefore as the lowest barrier to entry of most mobile frameworks, by virtue of letting you use HTML, CSS, and JavaScript much as you normally would.

It is not a panacea, however. The applications tend to look very "canned" and not entirely native as much as mobile-friendly websites (which is the point). The responsiveness isn't always great either; it definitely feels slightly more lagging then a truly native app.

You can ask Google for a plethora of other frameworks and opinions. Lately, we have become interested in Ionic which is based on AngularJS. This technique involves a more programming than HTML, and requires more structure instead of chaos, but in return, the apps tend to load up very quickly and to feel very fluid and responsive during use.

Fastclick

Most HTML mobile frameworks, and a time delay on processing your taps. This is meant to detect a double-tap, but in reality it just makes your app seem lag. He as it ignores you for quarter of a second.

Check out the Fastclick library. This JavaScript file and one command, removes this time delay to make your apps seem more responsive. If you use this with jQuery Mobile, you'll probably want to use the .click() event instead of the .tap() event.

Local CSS & JS

You should really include all of your JavaScript and CSS files, as well as all other assets such as marker images, logos, etc. within the app, and not load them over the network.

If you do use remote URLs, you're actually loading over the network. This is slower than loading of them out of the apps own storage, relies upon the network being unavailable and sufficiently fast, and consumes network throughput and battery power that might better be spent elsewhere. It also slows down the startup time of your app, since the app won't really get started until it has downloaded the requisite files.

Just don't do it.

Minifying CSS and JS

Unlike a networked web browser on your PC, Cordova does not need to download the CSS and JavaScript files. And if your app is written loading CSS and JavaScript files over the network, you should definitely look at that and fix it!

As such, minifying or compressing these files does not make any sense. It may make your resulting app a few kilobytes smaller, but that really is irrelevant compared to the multi-megabyte size of even small apps. Balance that questionable gain against the overhead of compressing these files, and I find that it's totally not worthwhile.

Put <script> tags at the bottom of your HTML

With some specific recent applications that were rather intensive, I hit upon an interesting condition. When running the app on my iPad, it would fail to load - the framework would load up, HTML would display and style, ... then nothing. This would never happen. When I run it under an emulator, nor on older/slower devices, and never if I put in a blocking time delay such as a alert() call to slow it down.

Ultimately, the workaround was simply to move all of the script tags down to the bottom of my index.html file. This seems to affect the loading sequence in some way, as to avoid some sort of race condition in my iPad, but I couldn't characterize. Strange but true.

Downloaded Files and iCloud

If your app downloads and stores any files, such as map tiles for off-line use, or a JSON file database, then your app probably will not be accepted into the iOS App Store. This is because of a term, that apps may only store data specifically generated by the user, such as documents. It goes on to state that files that are automatically downloaded for off-line use, such as the map tiles and JSON database file, must be tagged, so as not to be backed up by iCloud.

Any directory or file that you create within your own code, you must use the setMetadata() call to tag it for not being backed up. This is done as follows:

file.setMetadata(null, null, { "com.apple.MobileBackup":1}); 

As you test your app, go into your iPads backup settings, and check the iCloud usage for your app. There will be a minimum of about 100 kB, for metadata files and the like, and that is acceptable. But if you see it storing more than that, look closer.

Low Ceilings

A high-end modern smart phone, can have 2 GB of RAM and two CPUs. But go back only two years, and things change somewhat. While an iPhone 5 can readily handle 150 relatively complicated trails being loaded from GeoJSON and laid out onto a Leaflet map as vectors, an iPhone 4S will crash the app trying.

It's important not to go overboard. Seemingly-small enhancements such as adding a glow effect, do cost CPU time and make rendering slower. Downloading a JSON database with thousands of records may not be a problem, but drawing all thousands of markers onto the map could be a problem same as on a PC... but perhaps with a lower ceiling that you don't notice when you do that very same data set on a PC.

The moral of the story, is simply to keep in mind that not everybody has a fancy cell phone like yours, and that you should be testing on lower-and hardware and emulators as well.

Mobile Development with Cordova (Phonegap): Chapter 7

Chapter 7: Beta Testers

Let's continue on with that default Cordova app, the one that does nothing except say Device Ready, and discuss how you'll work with beta testers. Again, your own programming style and needs will dictate how you program your app, and the real challenge is everything else: distribution, emulators, and so on.


So, beta testers. If someone wants to beta test an app, the techniques for Android and iOS are quite different.

Android

Simply get them the APK file through any convenient means. If the APK is small enough you can email it to them, but you can upload it to any convenient website, send it via Dropbox, etc.

When they open the APK, they will be prompted to install it. Easy.

Tip: Some Android phones have a default setting that they will not install any apps that do not come from the Market / Store. If the application will not install, they should check their Settings to see if this option is used.

Tip: If the digital signature has changed, and they're upgrading an existing app, they may need to uninstall the current one. This is why I always use the --release flag, it omits this accident.

iOS

An iOS device will only run an app, if the app's Provisioning Profile specifically mentions that device. Unless the specific iPhone, iPod, or iPad is already listed in the devices section, the device will refuse to run it. This means that you will need to:

• get the beta tester's device ID (UDID) via iTunes
  This tutorial is helpful:  http://whatsmyudid.com/
• enter it into the Devices list on the Apple portal
• generate a new Provisioning Profile on the Apple portal
• load the new Provisioning Profile into Xcode
  Xcode / Preferences / your identity / the reload button at the bottom
• build the IPA using the Archive tool

Tip: Remember that all beta testers must be mentioned in the Devices and the Provisioning Profile. Remember that FFFF UDIDs are not valid, and as of iOS 7.1 many "find your UDID" apps now give inaccurate answers.

 

You cannot simply email the IPA file to the beta tester. 

iOS has a special technique called Over The Air (OTA) installation. If you have already created an OTA website for this app, you should be able to replace the IPA and have your beta testers hit it. If you're uploading a second copy with a special build/feature for somebody, then create a second manifest file and create a second hyperlink in your index.html file.

But if you've not yet created an OTA setup for your app, here's how:

• Get a website that is hosted via SSL (https URLs).  Important: The URLs used in this process server MUST be via SSL (https) and the certificate must be a paid, signed certificate. Self-signed certificates will not work, and will give certificate errors. An expired certificate will also give errors.
• Upload the IPA file to the webserver. You'll want to stick it into its own subfolder so you can have multiple builds, as well as the supporting files to facilitate the downloading process. Note the URL of the IPA file, and test it in the browser using an https URL to make sure it tries to download, e.g. https://www.example.com/mobileclients/democo/storeapp.ipa
• Use this handy template and create a file named manifest.plist, uploaded into the same folder.
•  Then use this handy template to create a file named index.html, uploaded into the same folder.
• Make some edits to the manifest.plist file:
  - Line 14, the URL to the IPA file which you determined in step 1
  - Line 20, the reverse-domain name of the app, e.g. org.greeninfo.parkinfo
  - Line 22, the version number, e.g. 1.0.0
  - Line 26, the title, e.g. ParkInfo Mobile
• Make some edits to the index.html file:
  - Line 7-8, the hyperlink and optional text; fix the URL to point at the manifest.plist file as per step 3
  - Your project likely won't distinguish Alpha from Beta; you probably only want one hyperlink at the moment
  - Then any other cosmetic, wording, link names, etc. that may inspire confidence in the beta testing technique such as the title

Beta testers should now be able to visit the website you just set up, and click the link to start installing the app on their iPad.

If the app won't install, there's only the one message that it couldn't be installed, which is not helpful. THere are a few things you can do:

• Unzip the IPA file (it's actually a digitally signed ZIP file) and look through the contents.
  The embedded.mobileprovision file will have a list of certificates and devices.
  Check that the UDID really is on the list.
  Tip: It's digitally signed so has some binary crud, but your editor should still be able to open it.
• Check their UDID again, and ask if they used an app or just copied down some number from their settings page. People really do resist plugging in iTunes, and will sometimes send multiple UDIDs trying various other strategies.
• Check that the Provisioning Profile is not expired.
  They are only good for 6 months, and customers do tend to let projects slide for a while.
  If the PP is expired, generate a new one, load it into Xcode, and Archive and upload again.
• Check the Build Settings for architectures.
  iPhones are usually arm7, arm7s, or arm64 and all architectures should be selected.
• If you can get your hands onto the device for a little while, the best debugger is the Console, available via Xcode's Device Organizer when the device is connected via cable. The messages "cannot verify executable" and "MSIVerifySignatureAndCopy" indicate a signature issue.
  Unfortunately, this means that you must have physical possesssion of the phone while at your development Mac.

Mobile Development with Cordova (Phonegap): Chapters 5 and 6

Chapters 5 and 6: Running it, then Compiling APKs and IPAs

For the moment, let's forego developing the app and skip straight to running it and then compiling it. After all, what good is developing the app if you can't even run it for testing and hand it out to beta testers?


Plug in Your Hardware, Set up Emulators

iOS: Plug in that iPhone 4S and iPad you're using to test, and open the Xcode project file. In the upper-left you'll see a list of emulators and devices. When it's time to run your app, it's as simple as picking from this menu.

iOS: Look over the list of emulators, and  see if there are any missing that you would like. You may need to use Xcode/Preferences/Components to add some emulators, e.g. for testing iOS 7.0 for those folks who couldn't or didn't upgrade to 7.1

Android: Plug in your devices, then run adb devices from the command line. You should see your device listed, if USB Debugging is enabled on the device. Realistically, this almost never works and you'll spend time on Stackoverflow trying to figure out why. :)

Android: Run android from the command line to run the Android SDK manager. Install the necessary platform SDKs (e.g. Android 2.2 if you'll want to test there), then create emulators called Android Virtual Devices (ADVs).



Running it on Android


This should be as easy as:
    cordova prepare android
    cordova run android

The prepare step copies your www  into the target platform folder, where it will be bundled and run. Don't skip this step, or you'll find yourself running older code.

If you have a hardware device attached, it should be used automatically: the app will be injected onto it, and it should even launch automagically. Super convenient.

If you have no hardware attached, the platform will start up an emulator for you if you defined one. (this is mentioned above and also in Chapter 1, Android Virtual Devices or ADVs) If you already have an emulator running, that one will be reused.

To specify a device, give a --target parameter to the cordova run command, like this:
    cordova run android --target=ABC123456

Tip: To see what devices you have attached, run:
    adb devices
 If your device isn't listed, check out Google because that's a very well known and widespread phenomenon. Usually you just forgot to enable USB debugging, but some hardware just isn't gonna work. See chapter 2 where we set up your build environment and test it, for some tips.

In reality, I've had show-stopping problems with running the app on Android:

- The emulators take several minutes to load, if they happen at all. When they do, the app may or may not be there. As such, I've given up on the emulators.

- Some hardware devices are never detected via adb devices and therefore cordova run android doesn't detect them. In particular was my LG Spectrum, which would about 1 time in 50 be detected by adb devices but then disappear again. Eventually I went to Fry's and hit up Groupon, and got a Blu brand phone for $60 and a Nexus tablet for $100, and both of them worked fine. Moral of the story: the problem may be the device.

In the worst of times, my process for Android devolved back to old-school techniques: I compile the APK, then email it or upload it, so I can download it on my phone and install it. This is less than ideal, but if adb doesn't see it, not a lot I can do.


Running it on iOS

In Xcode with the project open, look in the upper-left. You'll see a list of devices, some of them your attached devices and some of them emulators. Select the one you like, then go to Project/Run.

Simple and slick.

Tip: You can terminate the app with Project/Stop.

Tip: You need a Provisioning Profile to distribute your app to other devices, but if you're using emulators and cable-attached devices, you can skip it for now.

Building an APK for Distribution (Android)

To compile an APK for Android:
    cordova build android --release

The resulting APK for Android will be in platforms/android/ant-build

You can submit this to the app store, hand it to your friends, etc. You can ship it on an SD card, email it, whatever.

Tip: Some versions of Android refuse to install an APK unless it comes from the authorized Store. If someone's trying to beta test and hits upon this, visit the Settings panel and see about enabling non-Store applications.


Tip: I prefer to always use the --release flag. It prompts me for the password (once for the keystore, then again for the specific key) but that's okay. The resulting APK will be useful to anybody who tries it. I know that if I get distracted I may generate a non-signed APK that only works for me, then hand that out to beta testers whose phones will refuse to install it.

Building an IPA for Distribution (iOS)

Load up the Xcode project. Hit up Project / Run which tries to run your app. It will run it on whichever device is displayed in the upper-left, be it an emulator or a real device.

Also in the Project menu is the Archive option. If it's greyed out, select a hardware device in the upper-left for running your app. You must have a hardware device attached and selected, for the Archive option to become enabled. Select the Archive option.

You'll be asked to choose a provisioning profile. Select whichever one you generated for Ad Hoc uses. (See Appendix D where I describe generating provisioning profiles)

Finally, the IPA file will be created for you, and is ready to hand out to your beta testers.

Mobile Development with Cordova (Phonegap): Chapter 4

Chapter 4: Stuff We Always Do

This article describes several tasks that we do on every single application, and which you will likely do as well. Consider it a continuation of Chapter 3 where we created a project.

Plugins

We tend to develop for the same platforms every time, and there's a certain subset of plugins that we use in every single project.

cordova platform add ios
cordova platform add android
cordova plugin add org.apache.cordova.file
cordova plugin add org.apache.cordova.file-transfer
cordova plugin add org.apache.cordova.geolocation
cordova plugin add org.apache.cordova.network-information
cordova plugin add org.apache.cordova.splashscreen

Tip: The console plugin allows console.log() to log to the app console, which is useful in iOS when you have the device attached via cable. I don't use it much, though, since most of my development and debugging is via weinre.

Tip: The vibration plugin breaks the app compilation on iOS. Don't bother.

Xcode Setup: iOS

Open the project's xcodeproj file, located under platforms/ios This will launch Xcode.

Select the project (left sidebar, make sure the folder icon is picked, then the project is the first item on the list) then the Info settings.

Deployment Info:

• Select yourself as the Team.
• Select appropriate entries for Device Orientation, typically all 4.
• For Status Bar Style, select Hide During Application Launch.

Info:

• Custom iOS Target Properties:
• Status Bar is initially hidden: Set to YES
• View controller-based status bar: Set to NO

Icons and Splash Screens: Android

Tip: See also http://cordova.apache.org/docs/en/edge/config_ref_images.md.html

Look in platform/android/res and you'll find drawable folders for various screen resolutions (e.g. drawables-xhdpi), plus the default with no specific resolution (drawables).

Swap in the icons with new PNGs.
These PNGs should be RGBA (24-bit plus alpha channel) and the vogue these days is rounded corners.
The sizes of these PNGs are as follows:

default -- 32 x 32
ldpi -- 32 x 32
mdpi -- 48 x 48
hdpi -- 72 x 72
xhdpi -- 96 x 96
xxhdpi -- 144 x 144

You can also place an image named splash.png into each of these subfolders too, and it will act as the splash screen as the app starts and loads. These should be high quality PNGs, of the following sizes:

ldpi -- 426 x 320
mdpi -- 470 x 320
hdpi -- 640 x 480
xhdpi -- 960 x 720
xxhdpi -- 1410 x 960

Edit config.xml and add these two lines, to show the splash screen for 3 seconds (3000 ms).

<preference name="SplashScreen" value="screen" />
<preference name="SplashScreenDelay" value="10000" />

Lastly, you may need to do a clean in order to have the new versions detected:
./platforms/android/cordova/clean

Tip: The splash screen time stated in the config.xml should be the maximum before the platform hides it, but your app can hide it whenever it's ready. Good practice is to explicitly call navigator.splashscreen.hide() in your program during the onDeviceReady event handler, so it doesn't keep showing the splash screen any longer than necessary.

Icons and Splash Screens: iOS

Look in platforms/ios/PROJECTNAME/Resources and you'll see the icons and splash folders.

Just start replacing images with your own. For the splash screens they should simply be high-quality RGB PNGs, while icons are high-quality PNGs with alpha channel (RGBA) and rounded corners.

splash/Default-568h@2x~iphone.png   640x1136
splash/Default-Landscape@2x~ipad.png   2048x1496
splash/Default-Landscape~ipad.png   1024x748
splash/Default-Portrait@2x~ipad.png   1536x2008
splash/Default-Portrait~ipad.png   768x1004
splash/Default@2x~iphone.png   640x960
splash/Default~iphone.png   320x480

icons/icon.png             57 x 57 
icons/icon@2x.png          114 x 114 
icons/icon-40.png          40 x 40 
icons/icon-40@2x.png       80 x 80 
icons/icon-50.png          50 x 50 
icons/icon-50@2x.png       100 x 100 
icons/icon-60.png          60 x 60 
icons/icon-60@2x.png       120 x 120 
icons/icon-72.png          72 x 72 
icons/icon-72@2x.png       144 x 144 
icons/icon-76.png          76 x 76 
icons/icon-76@2x.png       152 x 152 
icons/icon-small.png       29 x 29 
icons/icon-small@2x.png    58 x 58 

Lastly, you may need to do a clean in order to have the new versions detected:
./platforms/ios/cordova/clean

iOS: Disable Cloud Storage

Our apps tend to be based on maps, and on storing map tiles. Apple terms allows only user-generated documents to be backed up, and require that we disable the automated backup of map tiles (as well as downloaded database files, LocalStorage and WebSQL content, etc). However, Cordova only allows us to turn backups on and off globally; we cannot pick and choose that some files are backed up. So, disable the cloud backup as follows:

- Open up platforms/ios/APPNAME/config.xml
- Look for the key BackupWebStorage
- Change it from cloud to none

Android: Allow App to Resume

Problem: By default, Android apps will run a new instance whenever you open it. If you open the app and get it into some state (say a search has been performed, your map centered), then switch to some other app (e.g. check your text messages), then come back... you start the app over again, complete bwith splash screen, losing your previous state. That's a useless app!

Solution: Change a default setting in the Android Manifest file, stating a "single task" launchMode.

• Edit the file platforms/android/AndroidManifest.xml
• Look for the <Activity> declaration, which you'll see also defines other params about your app, e.g. android:name="YourAppName"
• Add this attribute:   android:launchMode="singleTop"

Now (after a rebuild) the Android app will behave in the proper way and will resume when you come back to it, without losing your place.  (and why this isn't the default we'll never know)

Mobile Development with Cordova (Phonegap): Chapter 3

Chapter 3: Creating Your Project

Creating the Empty Project

Open a Terminal, and "mkdir" and "cd" your way into a folder where you'll want to start placing your projects. For me, I created a "Mobile Projects" folder underneath my home folder.

This command boostraps your app, copying in some template files for a very Hello, Cordova sort of app:
    cordova create exampledir biz.example.firstapp "Example Application"

The 3 parameters are:
- the name of the folder to create, e.g. exampledir will be the name of the folder
- the application name in "reverse DNS" convention, e.g. org.greeninfo.myfirstapp
- The name of the app itself as it will be named in the store and on the device, e.g. include spaces but not version numbers, and keep it brief and snazzy

Tip: It's very tedious to change the name of an app after it has gotten underway, so you really need to get this right the first time. Yes, really. Just like in that Dilbert episode.

There's a certain list of platforms and plugins that we will always use (Android and iOS, File access, FileTransfer, Network Status, etc.) so let's just get them out of the way now:

    cd exampledir
    cordova platform add ios
    cordova platform add android
    cordova plugin add org.apache.cordova.file
    cordova plugin add org.apache.cordova.file-transfer
    cordova plugin add org.apache.cordova.geolocation
    cordova plugin add org.apache.cordova.network-information

You should get familiar with the cordova command-line utility, as well as what plugins exist. In Cordova 3, virtually everything is implemented as a plugin.

A Quick Rundown

This is a brief overview of some of the files and folders. There's a lot more that I can't cover in detail here, just some highlights:

www
This folder houses your HTML, JavaScript, and CSS. This is where you'll do 99% of your work.

config.xml
This provides build options and metadata about your app. You will definitely want to edit this, especially to set the author attribution, the app description and name, and so on.

platforms/ios/ProjectName.xcodeproj
This is the Xcode project file (actually a folder, but OS X treats it specially). Open this to set special iOS-specific settings, such as hiding the status bar. This will be covered in a different article, where we describe tips and tricks for doing the actual programming.

platforms/ios/ProjectName/Resources
For the iOS platform, this stores additional assets such as icons and splash screens.

platforms/android/ant.properties
This configuration file will be used by ant when compiling the Android version of the app. This file does not exist by default, and is covered in more detail in a different article.

platforms/android/res
For the Android platform, this stores additional assets such as icons and splash screens.

platforms/android/assets/www
platforms/ios/www
These folders hold a temporary copy of your www folder content. This is used during the build process. It is not necessary to edit these copies in addition to the main www folder; they are automated copies.

Setting Up Certificates (iOS, Xcode)

Remember, the key store and certificates and provisioning profiles you set up a long time ago? Let's go ahead and configure these utilities to use them, signing your app with the keystore (Android) or Provisioning Profile (IOS).

In Finder open up your project folder, then open up two files, one for iOS and one for Android.

Android:
Start by creating a file under your project platforms/android folder, named ant.properties
It should have these two lines, of course customized to the keystore you generated earlier:
    key.store=/users/USERNAME/android.keystore
    key.alias=keystorealias

Tip: Every time you use cordova run android --device it will prompt for the passphrase. This is a little annoying, but not very compared to some of the problems you'll have. :-) For myself, I prefer to have my APKs always signed so I don't run into problems later, when I forget to sign a build, or send someone an unsigned APK.

iOS:
Open platforms/ios/PROJECTNAME.xcodeproj This will open Xcode.

Select the project itself (probably the first item on the upper-left panel) and set some project properties, most notably the Team. When you select a Team you'll be prompted to enter an Apple identity; it will then download a certificate and provisioning profile for you, saving them into the application's Xcode settings.

Run It On Emulators

Let's start by running the Hello Cordova app on iOS and Android emulators. If you won't be using emulators, I suppose you can skip this step.

The Cordova CLI apps make it pretty easy to start up the app in an Android or iOS emulator:
    cordova emulate android
    cordova emulate ios

A poorly-documented feature, is that you can specify the device for testing: iPad, iPhone 4, etc. with the --target parameter.
    cordova emulate ios --target="ipad (retina)"

For iOS there's a specific list and you may find this set of aliases quite handy. Add these to your .bash_profile and start a new Terminal, and your emulation commands will be quite simple:
alias emipad='cordova emulate ios --target="ipad (retina)"'
alias emiphone3='cordova emulate ios --target="iphone (retina 3.5-inch)"'
alias emiphone4='cordova emulate ios --target="iphone (retina 4-inch)"'
alias emiphone='cordova emulate ios --target="iphone"'

For Android, target names are the names of AVDs (as listed in adb devices) and you can make up a similar set of "emulate android" aliases for your particular AVDs and devices.

Run It On Hardware: iOS

First, plug your iPad into your Mac. You may be prompted as to whether the iPad should trust this computer, allowing the computer to inject applications onto it, etc. Naturally, you should accept.

Refresh the iOS platform directory which will form the app:
    cordova prepare ios
If you skip this step, you'll be running an older version of your app, and that can get confusing if you're  debugging something not visually obvious.

Then in Xcode, look in the upper-left where you'll see a list of platforms, e.g. iPhone Retina 3.5" Select your hardware device from the list. Now go to Product / Run  This will compile the app and then inject it onto your iPad and then launch it. Very simple and reliable.

Tip: Every time I plug in my iPad or Android, a photo application pops up. I don't know that there is any way to stop this, and it's very annoying.

Run It On Hardware: Android (ADB)

Make sure that your Android device has "USB Debugging" enabled. This is buried in your Settings panel, in a submenu called Developer Options. Then plug your tablet into the Mac's USB port, and the Android should report "USB Debugging Connected" at the same time as it does its usual USB behavior (offering to enable SD card sharing, or to Charge Only).

On the Mac, in a Terminal, you should be able to see the device when you run adb devices
Your device has an ID that may or may not be intuitive, may or may not contain the manufacturer name. But, that's the ID# and it'll stay the same for that device.

You should now be able to cordova run android --target=ABC123456 and have the app run on that specified device. Again, see adb devices for the listing. This does allow you to run on multiple Android devices simultaneously.

If your device doesn't show in adb devices, then fuss with it a bit:
- Unplug it and plug it back in.
- While leaving it plugged in, toggle USB Debugging off and on.
- Unplug it and plug it back in. (again)
- Try another USB cable. (some low quality cables have issues)
- Try another USB port. (nobody knows why)
- Try connecting directly to the computer's USB port instead of to the USB hub. (some hubs lose enough power that USB Debugging malfunctions)
- Try another device.

Tip: On some devices such as my LG SPectrum, it's so flaky that I gave up completely on using adb. I use the other technique described next. But I went to Fry's and bought a $60 Blu brand phone, and it works perfectly with adb.

Run It On Hardware: Android (Manual Copy, Slow But Works)

Compile the Android app, forming an APK file:
    cordova build android --release

The resulting APK is in  platform/android/ant-build

Plug in your Android to the Mac, and turn on USB Storage mode. On the Mac, copy the APK file onto the Android, eject the Android device, turn off USB Storage, and install the APK by opening it.

Tip: Android 4.x has App Installer, which makes short work of finding and installing the APK. It will scan local storage, find all APK files, and present you with a list. Three taps and you're testing.

Now Get Hacking !

If everything has worked out so far, then you're ready to start actually writing your own application. Start replacing content in the www  folder, installing plug-ins, making adjustments to config.xml, and figuring out the myriad problems involved in any complicated app. :-)

Mobile Development with Cordova (Phonegap): Chapter 2

Chapter 2: Setting up Your Build Environment

Overview of Components

The Android SDK works well with the Eclipse IDE. This doesn't mean that you need to use Eclipse to type out your code, but if you run into trouble you may be glad you have installed, since other peoples' documentation and procedures will refer to Eclipse and the Android SDK.

The Xcode IDE for OSX is required to "compile" your Cordova project. You won't use the GUI much, but you will definitely use the command-line tools. You will use the GUI a little bit, though, for example making edits to the app settings to hide the status bar.

As such, we will be installing both of these tools in addition to the Cordova software itself.

Installation and upgrading of Cordova, is best done through NodeJS, so we will install that as well so that we can install the Cordova library.

Cordova is built on Java, and therefore the build process depends on having the JDK installed and not simply the JRE.

Lastly, the build process for Android requires Apache Ant, so we'll install that too.

A final note: Much of what we do here will be command-line-based within the Terminal application on your Mac. As such, you should just get used to it, and may find it useful to drag Terminal into your Dock.

Installation: Eclipse and Xcode

Eclipse can be downloaded here http://www.eclipse.org/downloads/
It simply unzips, then you run Eclipse from the resulting folder. I like to move the Eclipse folder into my Applications folder, then drag the executable into my Dock so the folder is out of the way but the application is readily accessible.

Use the App Store to install Xcode
Then, install the Xcode command-line tools:
    - open Xcode, go into Xcode / Preferences / Components
    -  Select the CLI Tools, and appropriate iOS simulators as far back as you want to go (7.1 is out, I went back to 6)

Xcode has one more step: Acknowledging the license. This is done via CLI.
    sudo xcodebuild -license

Installation: Java

In the Terminal type:
    javac

If you have the JDK installed this gives some instructions on how to use javac, and your JDK is already installed.

If not, then it will fail but OSX will recognize the command as belonging to the JDK package, and you'll be prompted to install the JDK/JRE. Neat, huh? Go for it.

After it has installed, there's still a step. The build tools will need the JAVA_HOME environment variable set.
    Edit the file ~/.bash_profile   (it may not exist yet, create it)
    Add this one line:
        export JAVA_HOME=$(/usr/libexec/java_home)

Installation: NodeJS

The npm utility is part of NodeJS, and is effectively your package manager as far as Cordova is concerned.

Visit http://nodejs.org/download/ and download the DMG file for OSX. Run it, and follow the installer. Simple.

Then let's install the Cordova library, as well as an Android debugging tool:

    sudo npm install -g cordova
    sudo npm install -g adb

Installation: Android SDK and Android Development Tools (ADT) and the Eclipse Plugin

Fire up Eclipse, and install the Android Development Tools (ADT) plugin.
Go to Help / Install New Software
Enter this for the repository:  https://dl-ssl.google.com/android/eclipse/

Eclipse will restart, and will ask for the location of the Android SDK -- but will also allow you to let it download one. (Note that older documentation has you download the SDK separately, but this is no longer necessary.) It will download the SDK tools (including the "android" configuration program which you'll use a lot) into /users/YOURNAME/android-sdks

After it comes back up, add the ADT tools to your PATH so you can use them in the terminal:
    sudo nano /etc/paths
Add these three lines:
    /users/YOURNAME/android-sdks/tools
    /users/YOURNAME/android-sdks/build-tools
    /users/YOURNAME/android-sdks/platform-tools

You should now be able to start a new Terminal window, and run the command android and get the ADT configuration tool.
You'll be seeing a lot of this later on, as you install more Android SDK components.

Your first task will be to install a bunch of SDKs for various versions of Android, as well as the HAXM.
Go ahead and run android, then install the following:
    Android SDK Tools
    Android SDK Platform Tools
    Android SDK Build Tools
    Android 4.4 SDK Platform and Intel Atom System Image
    Android 4.2 SDK Platform and Intel Atom System Image
    Android 4.1 SDK Platform and Intel Atom System Image
    Android 2.3 SDK Platform and System Image
    Extras: Intel x86 Emulator Accelerator (HAXM)

A Brief Word on Android Emulators and HAXM

If you want to test on a plethora of Android devices, you'll usually want to use the Android emulators and Android Virtual Devices (AVDs). These are exactly what they sound like: virtual machines running Android, so you can test your apps without buying the real devices.

Android emulators are painfully slow, but Intel Hardware Accelerated Execution Manager (HAXM) helps. You downloaded the HAXM earlier, but it didn't in fact install it. Installation of HAXM is done via Finder and not via CLI (it's an OSX installer, and not a command-line tool you'll use again). Use Finder and browse to your home folder, the android-sdks subfolder, then extras, and the Intel  Hardware Accelerated Execution Manager (HAXM) and open up IntelHAXM.dmg  This is a regular installer, a few click and you're done.

Tip: Some older android emulators will not run on an Intel CPU, but ARM. These will be even slower than the Intel ones because they cannot use HAXM. Even the Intel ones are painfully slow, and have trouble with actually installing the APK. For myself, I prefer to just buy the $60 tablet at Rite Aid, and a $100 Nexus tablet via Groupon, rather than wait several minutes at a time for an emulator that may or may not start successfully.

If you do choose to use emulators, they are called Android Virtual Devices (AVDs). These are managed in the android manager, under Manage AVDs. The options should be self-explanatory, such as the screen size and the storage, and there are a bunch of presets for some popular devices.

Installation: iOS simulators

Pretty simple, really:

    sudo npm install -g ios-sim
    sudo npm install -g ios-deploy

Tip: As with the Android emulators, the iOS emulators are quite slow and don't always load your package anyway. Since we need to test on real hardware anyway, I prefer to just test it there.

Ready?

By this time you should be able to execute the following commands in the Terminal, and at least have them show some help messages, or at least error messages indicating that they are installed.
    ant
    javac
    echo $JAVA_HOME
    adb
    cordova
    ios-sim

At this point, you're ready to begin creating your first project!