CodeIgniter sessions: timeouts, AJAX, and Remember Me

I love CodeIgniter. It's among the top 3 popular PHP frameworks, though the order of those 3 is a matter of opinion and taste.

Back in February I finally got to research and work around an issue that had been driving me nuts. I have an app which requires login, and uses CI sessions. Using one particular page, after a while my AJAX calls will fail because I am no longer logged in. Reload the page, yep, I'm logged out.

Turns out, this is a known issue, going way back:
CodeIgniter regenerates a new session ID periodically, to prevent session ID hijacking (session fixation attacks). At some point my session ID will change, and my session will effectively expire because the AJAX requests aren't setting the new session ID in my browser.

https://github.com/EllisLab/CodeIgniter/issues/154

This goes beyond AJAX, though. If you want the session to stay open over time, e.g. a Keep Me Logged In functionality, you will also see this problem: when you open the site tomorrow, your session ID is out of date and you're not Keeping Logged In at all.


Fortunately, the fix for this is not difficult.

Session Expiration & Session ID Regeneration

Open up config.php and look for these two session variables.
The sess_expiration setting, defines how long a session is good (idle time since last hit).
The sess_time_to_update setting, defines how often your session ID will be regenerated.

$config['sess_expiration'] = 86400 * 30; // 30 days
$config['sess_time_to_update'] = 300; // 5 minutes

Our problem is the session ID regeneration. We can effectively disable this by making sess_time_to_update very, very long:

// sessions are good for 30 days, and IDs will change every year
$config['sess_time_to_update']    = 86400 * 365;

Problem solved. I only need to login after 30 days of idle time, and even using AJAX-heavy pages without

Consequence: Session Fixation

Disabling session ID regeneration does have a consequence for security, though. Now that your session ID doesn't change, it's hypothetically possible for someone to snag your session ID and use it indefinitely (well, for a year if you set it as above).

This can be mitigated by a few other arrangements:

Shorten the session expiration and regeneration time. Do they really need a session to be good for 30 days? Would 7 days suffice? How about 3 days? The shorter it is, the less time you give a sniffer to try hijacking your session before finding it already expired.

Encrypt the cookies! Set sess_encrypt_cookie to TRUE and set a random encryption_key so thesession IDs can't be intercepted in the first place. Do a Google search for "codeigniter encryption key generator" and take your pick.


Enable user agent matching. The sess_match_useragent setting enables a check on your session, that you're using the same browser as you were when the session opened. User agent strings can be faked, but this does increase the "cost" of brute forcing or hijacking sessions, as the hacker must try the session key several times as they guess browsers.

Enable IP address matching. The sess_match_ip setting enables a check that your IP address hasn't changed since your session was last used. This one may not be appropriate if your ISP routes traffic through changing proxy servers (satellite Internet, America Online) and most of us have dynbamic IP addresses that change every few days. But, if your use case is access from a fixed IP address such as an office intranet, or you can tolerate logging back in after your reset your home router, this can provide a significant layer of security.

Use SSL. This isn't always possible, but when it is, intercepting sessions is impossible. If you use this and also cookie encryption, then intercepting sessions is doubly impossible.

Conclusion

This solution to session timeouts works fine, but opens up a security consideration. Fortunately, with cookie encryption, this security consideration doesn't need to be a security problem and your users can Keep Me Logged In without having their sessions hijacked.

MobileMapStarter update: Tile caching and Chrome compatibility

Yesterday I put the finishing touches onto a major revision of MobileMapStarter.

- The previous version used imgcache.js to passively cache map tiles as you browse. The new version uses a new system, allowing you to download a specific area for offline use. No more panning and zooming repeatedly -- just zoom out, center it, and hit Download. It even has a nice status thingy, and skips over tiles that you already have.

- I spent a lot of time getting the caching to work in Chrome too, so you should be able to prototype your app on the desktop.

- There were also some other minor things: typos, minor adjustments, and changing out the Streets basemap for the Mapbox Satellite basemap.

Go git it!
https://github.com/gregallensworth/MobileMapStarter/

Automating CartoDB and Google Fusion Spreadsheets on GoDaddy

A whole month without a posting. I've been busy on some fun stuff here. Here's a peek.

A client wanted something simple. They have a registration form which logs submissions regarding trees, to a Google Fusion Spreadsheet. They want a choropleth map of Census Block Groups (CBGs), indicating how many trees had been registered there. But it's not that simple: they don't want to pay for geospatial hosting, they want to stick with their $8 GoDaddy hosting.


So, our basic needs are:

1. Update the Google spreadsheet of trees, populating the lat and lon fields for any which don't have them.
2. Have a PostGIS table of CBGs someplace for free.
3. Perform a spatial join between the CBG polygons and the  tree points.
4. Render it as a Leaflet map, using only client-side tech, knowing that the CBGs are too much to be communicated as raw vectors and rendered as a L.Polygon layer.

Our resources are:

1. The Google Spreadsheet.
2. A website at GoDaddy which supports PHP, but not many modules.
3. Plenty of disk space.
4. A free account at CartoDB.

It all worked out beautifully, using client-side-only mapping, and PHP with no special configurations, suitable for hosting on GoDaddy's low-priced web hosting. No server-side mapping infrastructure!

An annotated version of the app's source is available here:
http:/tilelab.greeninfo.org/~gregor/cartodb_google/

Step 1: GoogleFusionTable.php

Step 1 is comparatively easy. Google Docs already has a web API for doing SQL-like queries to spreadsheets, and a PHP library already exists for this.

https://github.com/gregallensworth/CloudServices/blob/master/GoogleFusionTable.php

This was just simple: Google's geocoder service, the GFT PHP, and a few dozen lines of code. Now we just visit the URL and all non-geocoded registrations will have their Lat and Lon fields updated.

Steps 2: Uploading Google to CartoDB

We had been itching to try out CartoDB's offerings, this was a great opportunity.

CartoDB is a website, which offers an easy-to-use interface to PostGIS tables. You upload a spreadsheet or a shapefile, they grok it and load it, and you get a nice table UI and a panel for entering SQL. A step further, they have a web API for performing these SQL queries, so you can get back a JSON object full of rows, by submitting a query via GET or POST.

The heart of accessing it via PHP, boils down to this wrapper function I whipped up:
https://github.com/gregallensworth/CloudServices/blob/master/CartoDB.php

And the implementation is just a tie-together of Google's SQL API for Fusion Docs, and CartoDB's SQL API to your table:
https://github.com/gregallensworth/CloudServices/blob/master/google2cartodb.php

Remember: Each query really is a hit to CartoDB's web service. If you have 12,000 rows to insert, you really will make 12,000 hits to CartoDB. Be kind and thoughtful. Use a common key between your two tables, so you don't need to re-insert every single record if it's already there. And consider doing the initial load via CartoDB's dashboard.

Step 3: Spatial join via CartoDB's web SQL API

Pretty simple here, now that you've already seen the CartoDB web SQL API.
Just run these two queries:

DELETE FROM census_block_groups_treecounts;

INSERT INTO census_block_groups_treecounts (
    the_geom,
    treecount
)
SELECT
    blocks.the_geom,
    SUM(treeshere) AS treecount
FROM census_block_groups AS blocks
JOIN registrations AS trees
ON ST_Contains(blocks.the_geom, trees.the_geom)
GROUP BY blocks.cartodb_id;

Step 4: Client side map... CartoDB again

Further still, CartoDB created a JavaScript library which bundles Leaflet, their web SQL API, mouse behaviors, CartoCSS parsing, and a dozen other great things. So getting your data back out of CartoDB and into Leaflet requires very little code.

http://developers.cartodb.com/documentation/cartodb-js.html

http://tilelab.greeninfo.org/~gregor/cartodb_google/index.js

Not a lot to say here. The docs are less than rich in real-use-case examples, and CartoDB.js brings together a lot of moving parts. But once you figure it out, it's really something quite nice.

BONUS: CartoDB layers can have their CartoCSS changed at runtime, and have their SQL changed at runtime. This app doesn't make use of that, but in theory filtering features or changing color schemes, has never been easier.

Step 5: Nightly Automation

GoDaddy has a facility for setting up cronjobs, right there in the control panel. At 1am it runs the geocode PHP. At 2am it runs the google2cartodb PHP. At 3am it runs the spatial join. Pretty groovy.

Trouble and Surprises

Not a lot in the way of trouble and surprises.

GoDaddy's PHP has a execution time limit. This seems to be 5 minutes, and ini_set() and set_time_limit() don't seem to change it. As such, the steps really are handled as separate PHPs instead of one monolithic one to geocode, upload, and join. This is probably for the best anyway, as it's easier to debug individual parts this way.

CartoDB's free account limits you to 5 MB of storage. In the case of this client, their registrations came up to 1 MB even if we stripped down to only the Lat & Lon, and they are expecting plenty more registrations . The set of CBGs comes up to 2 MB for the SF Bay Area, and that's after I stripped out areas that were unlikely to get registrations, then simplified, and clipped to fit land. The joined layer displayed on the map, will vary in size based on the number of matching CBGs; there's a maximum of 2 MB here too, if some day every single CBG were to have at least one registered tree. As such, hitting the 5 MB limit seems very likely as they get more registrations and they may end up upgrading anyway.

CartoDB's web SQL API cannot create tables, only populate them. I had to upload a spreadsheet of the first few registrations, for example, and has to manually run the spatial join SQL in CartoDB's dashboard. After the table structure was there, a DELETE FROM followed by later INSERTs worked just perfectly.

CartoDB.js involves a lot of parts, and the docs are good but not always great. Though the final map involves fairly little code, figuring out exactly which code took some time. I had to read up on CartoCSS a bit, and had to read a few times to figure out that CartoDB's API won't return the_geom but that you must do query for "ST_ASGEOJSON(the_geom) as json_geom" to get it, e.g. if you want to highlight features.

Conclusion

A client wanted a geospatial app with some moderate back-end complexity, but without paying for a geo infrastructure. Using a Google-hosted contact "database", CartoDB's free tier plan, and a pinch of PHP to tie them together, it all came together beautifully.

In the meantime, I learned a lot about CartoDB and really admire what they've come up with. The $30 price tier is a bit steep for those intermediate use cases: this client may go up to 15 MB of usage, over the free tier but well under the need for the Magellan tier. If they had a smaller paid plan that would be really excellent.

Shrinking Well Known Text (WKT) Geometries By Rounding Decimals

It's been a good month, full of fun stuff, but nothing that REALLY seemed blogworthy. But today, someone suggested an idea.

Background: When communicating a geometry from the server to the browser, we often use Well Known Text (WKT). They're not overly verbose and they're simple and fast to generate.

Issue: WKT geometries can still be quite large. Simplifying is a great step if you can afford the introduced inaccuuracy, but still, can we make them smaller?

Solution: Round the decimals.

If you are using a SRS with meters or feet as units, then you can probably afford for your geometries to use whole numbers instead of decimals. After all, what's the rounding error? A difference of 0.999 meters isn't much, considering the accuracy of handheld or automotive GPSs.

Check out this sample WKT, the first 6 vertices of the polygon of Salinas County, California.

-- ST_ASTEXT()
 POLYGON((-13578594.747402 4380792.43128354,-13578592.2992804 4380792.29270892,-13578569.3701029 4380825.00366193,-13578580.720485 4380880.31712633,-13578605.4359713 4380926.761986

-- ST_ASTEXT_ROUNDED()
 POLYGON((-13578594 4380792,-13578592 4380792,-13578569 4380825,-13578580 4380880,-13578605 4380926,-13578607 4380932

Overall, the reduction is just under 50%, from 598 KB to 309 KB. This is cumulative with ST_SIMPLIFY() too. If you can afford to cut corners (a little geospatial pun there, ha ha), you can simplify and then round it to get 50% of even the simplified payload.

Neat. How do I do it?

I created a handy function ST_ASTEXT_ROUNDED() which simply complements ST_ASTEXT() It works for PostGIS 1 and PostGIS 2

Go git it!
https://github.com/gregallensworth/PostGIS

This is really a function wrapper around a super simple regular expression which trims off decimals. It doesn't round the numbers, but crops them. But then again, with a maximum error of 0.999 meters does it really matter?

When Not To Do This

A key point of this rounding trick, is the assumption that rounding the numbers makes no real-world difference in the accuracy of your data. For feet or meters, 4380792.43128354 and 4380792 introduces an inaccuracy less than an arm's length.

If you're using geographic coordinates (WGS84, lat & lon) then it's different. A degree is 60 miles at the equator, so rounding from -122.3 to -122 is a difference of 10-18 miles depending on your latitude. Don't do that.

Cordova File API , when getDirectory() and DirectoryReader do NOTHING!

Naturally, within a day of my first successes with PhoneGap Build, I am trying some decidedly non-trivial stuff already. I want map tiles off the Internet via HTTP, but I want them cached so the tiles will still transparently be available when I lose connectivity. My plan, is to use Cordova's File storage API.

I almost immediately hit upon an issue:

The getDirectory() method does NOTHING. Neither the success nor failure callback happen. No exception is raised. This happens in the app as it loads, and also in the weinre Console. It doesn't crash the browser, hang the iPad, create a directory, ... Nothing.

Figuring this out wasted 2 hours of my time, and in retrospect the cause seems almost trivial, almost expected, although short of obvious:

File API's getDirectory() method, cannot recursively create directories, creating required parent directories. It returns instantly, but silently fails, without a callback nor an exception.

// the set of callbacks for the two phases: requesting a handle to the filesystem,
// and requesting access to (or creation of) the specified target directory
var getFSsuccess = function(fs) {
    console.debug('Got fshandle');
    FS = fs;
    FS.root.getDirectory(tiledir, {create:true,exclusive:false}, getDIRsuccess, getDIRfail);
};
var getFSfail = function () {
    throw new Error('Could not open filesystem');
};
var getDIRsuccess = function (dir) {
    console.debug('Got dirhandle');
    cachedir = dir;
    fileurl  = fs.root.fullPath + '/' + tiledir;
};
var getDIRfail = function () {
    throw new Error('Could not open directory ' + layerinstance.options.tiledir);
};


// WORKS
// getFSsuccess will be called,
// will call FS.root.getDirectory with tiles-greeninfo.terrain
// and create the subfolder as expected
var FS, cachedir, fileurl;
var tiledir = "tiles-greeninfo.terrain";
window.requestFileSystem(LocalFileSystem.PERSISTENT, 0, getFSsuccess, getFSfail);

// FAILS
// This will not create tiles/ and then tiles/greeninfo.terrain
// as it's a subfolder and getDirectory doesn't do parent creation
// The catch is that it's SILENT failure:

// the getDIRfail callback will never happen, and no exception will be raised
var FS, cachedir, fileurl;
var tiledir = "tiles/greeninfo.terrain";

If you see this blog posting in Google, and it saves you some time, let me know. I'll be glad to know that someone benefited from my wasted evening. :)

MobileMapStarter

A trivial mobile mapping app, tying together the best of Leaflet, jQuery Mobile, and Phonegap/Cordova. This will form the framework for our future generations of mobile apps at GreenInfo Network.

https://github.com/gregallensworth/MobileMapStarter

For the narrative, I'll simply quote from the README file:

A starting framework for mobile maps using Cordova/Phonegap. A minimal but functional, standalone mobile app from which to build your own creations.
This app is designed for Phonegap/Cordova, therefore it is HTML, JavaScript, and CSS.
Components of this app:

• HTML/CSS/JS layout -- The app is ready to compile and run via Phonegap.
• config.xml -- The app is ready to upload to Phonegap Build. The included config.xml specifies permissions, icons and splash screens, and more in an easy-to-edit template.
• jQuery Mobile -- A mobile-style user interface theme. Includes jQuery which makes JavaScript useful.
• Leaflet -- Quick, pretty, easy tiled maps.
• imgcache.js -- Cache Leaflet tiles to device storage, for offline use.
   

PhoneGap Build: The Better Way

So, it's been nearly 2 months since I posted how to get set up with XCode and with Eclipse, to do Phonegap development. I've been busy since then, only about to put 10 or so hours into it. But it has not gone well.

Phonegap's .\create.bat script is messing up. It started giving cryptic "file not found" errors as it created the project. It created projects with errors, so I couldn't even run the demo app out of the box. The errors indicated something about the Android manifest, but I could never figure out what. On my HTC Incredible, I could never get a single page with a single DIV to display edge-to-edge: it would introduce some padding or margin that I didn't ask for. For my LG Spectrum, the driver packages from Google weren't applicable and the ones provided by LG still don't allow USB debugging from ADB.

XCode was a lot nicer, for a while. But on the third day, it began not updating my code changes. The new app has a black bar on top, three pages to pick through, and a Leaflet map... On the Android anyway -- on the iPad it's still Hello World on a white background, even after several reboots.

In short, it's exciting but it's difficult, and despite earlier successes, at present both of my development computers seem unable to in fact build a project.

Introduce: Phonegap Build

Adobe offers a service called Phonegap Build. It sounds great in theory: upload a ZIP file of your HTML/JS/CSS files, enter project name etc. into some control panels, and they'll compile it for you, providing you with a set of APKs and IPAs, and even packages for Blackberry and Windows Mobile and other obsolete-but-not-extinct platforms.

It sounds good in theory. But the reality... is that it works as advertised, and has additional features that are even cooler still!

So, a walkthrough:

- Sign up for an account. Their free account allows you 1 app, and for $9.99/month you can host 25 projects. Aside from these counts, you can add any number of projects from Github for free. We ourselves don't tend to host publicly-downloadable source code on Github, but if you do this is quite a deal.
- If you're developing for iOS, go to your account panel and upload your Apple Developer Key. That's a bit of a process, but is worth it.
- Upload your ZIP file of HTML/CSS/JS. A control panel allows fine-tuning such as your app's name, an icon PNG, etc.
- A bunch of little spinners start compiling your program for all these platforms. A minute later, you have 7 download links and a QR code.
- Hold up your phone and scan the QR code, download the APK or IPA, and enjoy your app.

On my HTC Incredible, the resulting packages from Phonegap Build, even work on my HTC Incredible, perfect edge-to-edge layouts. That right there exceeds my best efforts of 10 hours.

Debugging Console

So, it actually compiles and downloads, that's a huge bonus over the problems that Eclipse has been giving me. But there's more!

Upload you app with Debug mode enabled. Fire it up on your phone, then click the Debug button on the website (Chrome and Safari only). Your app was transparently rewritten so as to load its JS/CSS/etc remotely from Phonegap's servers, and the Chrome Developer Tools lookalike you're seeing in the Debug panel is tied into that.

You can now issue JavaScript commands in the Console, and examine and tweak CSS in the Elements panel. You're using your browser debugging tools, on your phone. There's a time delay of 1-2 seconds while your phone picks up the changes, which is tiny compared to repackaging, and on par with hitting Reload developing a desktop browser app. But this new ability, to examine JavaScript objects and tweak the CSS within my phone... invaluable.

The only downside at this time is that the Debug server seems to be down a lot. On December 24 it was down most of the day, after that it's up-but-slow or up-and-great or sometimes Not Responding. Hopefully they can get that solved, because it's becoming an invaluable tool worth paying for.

Conclusion

Well, it's not much of a conclusion, as much as a beginning. After weeks of arguing with Eclipse and XCode, trawling Google and StackOverflow for workarounds, fussing with driver packages, and ultimately producing nothing reliable... Phonegap Build made it possible to catch up in a single day.