Tuesday, October 13, 2015

Census blocks and attributes: An easier way

Has it really been almost 5 months since my last posting? I've been doing plenty, but not a lot of it is so inspired but also of wide enough interest that I've felt it relevant to share.

But if you work with census data, this may save your day:
https://github.com/greeninfo/CensusShapefileMaker

It does a day's worth of census downloading, joining, trimming, and calculation... in 10 to 15 minutes (most of that being the FTP download).

 

History

I created this for a client. Her specific use case, is to have census blocks for a whole state (though sometimes one county), joined to a few attribute tables so we have statistics such as number of people, number of people who are black/white/etc, number of people who are under 18, and so on. We'd then derive some statistics such as "number of people under 18" which itself means adding up 8 attributes then dropping those 8 since we really only wanted their sum. And of course, we only want a dozen or so columns but those attribute tables are huge, so she'd drop hundreds of columns with unintuitive names such as p61ia She would do this process in ArcMap and it would take an entire day or sometimes two, and was highly error-prone.

So, I sat down with MCDC's Dexter and the USCB FTP and figured things out.

 

A few tricks and paths

  • Dexter allows one to specify only specific attribute fields to be downloaded, and allows downloads of both decennial census and annual ACS. This is great, since it makes the CSV files so much simpler to parse: they have the fields you asked for.
  • I did try having ogr2ogr perform the join between the CSV and the shapefile -- after a couple of hours it was obviously not getting anywhere. So the technique is to open the shapefile via Python ogr and loop over records, assigning their attributes. For performance, it loads the whole attribute table into a dict which sounds bad, but in reality works fine since modern laptops have 8 GB of RAM.
  • Good ol' ogr2ogr is still used for stripping out attributes, e.g. the stock shapefiles have extra attributes we didn't care for. This could have been done in Python ogr as well, but was already done in ogr2ogr.
  • We wanted a YOUTH field indicating number of people under age 18, but the Age By Sex table has that broken over 8 fields. So at the merge phase, that youth sum is what it loads from the CSV in lieu of the 8 individual fields.
  • Median Household Income (MHHINC) comes from ACS and not from decennial, and comes at the tract level instead of the block level. Fortunately, the hierarchical naming convention for the GEOID makes it dead simple to figure out which tract corresponds to a given block. During the merge phase, the first 11 digits of the tract are used to look up the corresponding tract -- easy.
  • As an afterthought, the thing can also strip down to one specific county (identified by its FIPS code, in the hierarchical GEOID) if you didn't want the whole state. This is done at the very end, since frankly the processing is so fast that the seconds you'd save by pruning it first aren't worth the programming trouble.
So yeah, if you work with census data a lot you may find this one useful. Enjoy!


Wednesday, May 27, 2015

Marker "spotlight" for Leaflet and for Google Maps API

A client really liked this spotlight effect demonstrated for ModestMaps. It calls out the selected markers, without hiding the other markers or forcing the user to perform a new search. And hypothetically this would be a great feature for the legend, to mouse over the legend and have the matching markers "light up"


But it didn't exist for Google Maps API (which the client uses), nor for Leaflet (which we use on 95% of our other projects). So I spent an afternoon coding this up:
https://github.com/greeninfo/MapSpotlight

All in all it was pretty easy, and even fun. I got to read about the HTML5 Canvas which I rarely get to work with outside of some charting libraries, and got to learn about Google Maps API's OverlayView capability, and the analogous iLayer interface for Leaflet.

So yeah, go grab a copy and highlight some markers. :)

Tuesday, April 28, 2015

First experiments with Unity

I'll deviate briefly from my usual topic of maps, GIS, and web-GIS, to something different...

I started playing with the game engine Unity. I wanted to do something after work that felt creative, and completely different than anything I've done before. And video games using Unity, sure fits that: no authentication and scrubbing user input, no SQL databases, no rendering PNG tiles. Rather, a empty world with gravity and solidity, and objects that have intelligence. Neat stuff, very educational.

My first project is a spinoff of the classic Unity tutorial, Roll-a-Ball. Being myself, I couldn't just follow instructions and I had to takre it off in my own more complex direction. I added a concept of the board tilting to screw with gravity, added a behavior that when you get all targets it creates more and increases the tilting, and so on.

It's not a million-dollar idea, just experiments. So I thought I'd share with other Unity newbs.
https://github.com/gregallensworth/DrunkBubble

Sunday, April 12, 2015

New Leaflet Control: L.Control.BasemapBar

Some time back I created a basemap-selection control for use with Leaflet, which is quite nice. It can switch between standard XYZ L.TileLayer basemaps, Google basemap choices, and Bing basemap choices provided you load the APIs and layer plugins.

Check out this screenshot, then check it out on Github:



So today I polished it up a bit, removing the requirement for FontAwesome and adding some documentation, and released it on GreenInfo's Github page. We think it's a really nice tool forr switching between basemap options. It's functional, it's attractive but not gaudy, and it's flexible to handle all of our basemap needs.

Enjoy!

Monday, March 30, 2015

New Leaflet control: Credits

Lately, I've been polishing up more of these old Leaflet Controls that I've created over the last year. Here's another one.

https://github.com/gregallensworth/L.Control.Credits

This one's pretty simple: supply text and a URL and an image, and it generates a cute interactive icon in the lower-right corner. There are various ways to make credits in your Leaflet map, this is the one used by GreenInfo Network and it's quite nice.


Book Review: Phonegap for Enterprise, by Kerri Shotts

Kerri Shotts is a member of the Phonegap mailing list, who quite frequently posts helpful advice. I appreciate her contribution to Cordova/Phonegap community. She's even written a few books, such as PhoneGap 3.x Mobile Application Development Hotshot which I have. So a few weeks back, I jumped at the opportunity to get a free copy of Phonegap for Enterprise, on condition that I review it.

One of the best parts about reading someone else's perspectives on programming, is that they have different angles and use different tools. As a result, I may not use their techniques but will learn about some tool that fits a need better than my current tool. And as far as I'm concerned, every "Oh, I didn't know that!" moment is a gem. And this book had plenty of those.

Super Short Summary


A good book, including an overview of technology choices and general and specific notes about how to make your application more secure... but also more likeable. It's definitely not a cookbook nor a how-to, as much as considerations and options which may apply to your application.

It's full of outside-the-box ideas, and full of options and links to other resources so you can take those ideas and run with them in your own direction. Phonegap for Enterprise is a worthwhile read regardless of your Cordova skill level.


Openings


The book opens with the usual What is Phonegap? sort of introduction. It then moves quickly into a discussion of how to store your backend data and options for where to put your business logic, and an overview of what goes into an application. This isn't novel stuff, if you've written web services before... until she brings up some alternative angles on business logic and on database storage.

She mentions a database-in-cloud service called Parse, and considers doing business logic in the database (PL/SQL) as opposed to in external code (PHP or Node). Leveraging you're database server's PL instead of using a second-layer language? It's certainly an interesting approach, and she does describe the balance of the two approaches. Definitely food for thought here.

She wraps up this introductory chapter with discussion of planning out the application: what fields do a Task and a Person have, and what Persons can view and edit what Tasks? If you're new to web services then thinking this stuff out beforehand is something that really helps a lot, and yet it's a phase often skipped - it's good to see someone advocating for that.

Security


Chapter 3 puts the carriage before the horse a bit, to touch on security. It's rare that anyone discusses security in the opening, and I appreciate that she does.

This chapter touches on security, considering the usual concepts such as using SSL but then worse-case scenarios such as man-in-the-middle attacks, and forged copies of your app. She touches on novel ideas such as hardcoding a certificate fingerprint into the app for better security, and even really-crazy stuff like calculating HMACs for your requests so the server can ensure that they requests are likely to be from your app instead of an AJAX trawlbot or a session replay. She points out the impossibilities of protecting your source code, the possibilities of someone falsifying your app and hijacking your user's network, and more.

Individually you can take or leave each idea, and some you can't really do anything about. But as a whole they present a rich and thought-provoking set of considerations which a busy developer often skips over until the end... when they forget to do it at all!


Building Foundations


Chapters 4 and 5 proceeds to the building of your web service. Kerri describes the basics of REST and also some of the realities of how we do things instead of true REST, then gets into some code bites to get started on setting up your URL routing and authentication.

This part I found somewhat baffling and dense. It jumps around a bit, and leaves out details of authentication beyond mentioning the benefits of using a third-party service such as OAuth yet stating that we had settled on using a local password hash using PBKDF2

But getting past that, Kerri does illustrate the beauty of Passport. The details of session.createSession() are implemented inside Passport and you could just as easily be using an OAuth service to generate those tokens which you're later validating with Passport. And again with considerations as being a theme here, the benefits of a dedicated OAuth service over a local password database are discussed.

Now, her specific take on REST as following the HATEOAS concept feels a bit overwrought to me, as if she's trying to reinvent SOAP. Given that any API change would implicitly come with app changes to both the UI and the programming, it's not the case that the app could auto-configure itself if endpoints change anyway. So ultimately I'm not convinced that this is the way I would do things.

But though HATEOAS is over-the-top for this application, Kerri's coverage of it is still food for thought, if we intended to construct web services which are subject to change and intended to be consumed by something other than an app which we control. You don't even need a mobile app in the works for this to have interesting possibilities.


Cordova Considerations and UI Considerations



Chapters 5, 6, and 7 get really into the Cordova-specific considerations and techniques for your app. And this is some good stuff.

The device can lose and regain its network status. Your device could be backgrounded and then terminated before you had a chance to save state to LocalStorage. You can store stuff in SQLite and encrypt it. There are freemium services to both act as Push Notification gateways and Push Notification SDKs. And she covers these expertly: succinct and yet thorough with good advice.

Mobile UIs have different expectations than desktop UIs, and Chapter 8 really tries to cover it all.  It includes a tutorial on MVC, and observer-based data-binding frameworks. She points out UI considerations and modern HTML5 tags, and mentions a few options for charting. (she forgot to mention Highcharts, my favorite.) It's a bit scattered since a complete discussion of options could fill a lifetime, but as usual rich in provocative thoughts and in options to find your own way.

This set of chapters was my favorite of the whole book (as was Chapter 3), in terms of gems of wisdom.  The brevity of my review of this whole one third of the book, really doesn't do it justice. There's way too much for me to summarize, so buy the book and read it yourself. ;)

All In All


It's a bit dense at the start, but it's not meant to be a cookbook. It's about ideas and considerations you that may not have crossed your mind when you did your first apps. It's so rich with "Other Resources", that if you think her specific ideas aren't for you, give thought to the considerations she brings up, then read those other resources and do it your way.

Phonegap for Enterprise a worthwhile and educational read, for people of any skill level with Cordova.

Monday, March 23, 2015

More Github and leaflet awesomeness: L.TileLayer.Cordova

What a busy week! Found a cool Leaflet plugin, wrote a new Leaflet plugin, ... and finally put the polishing touches and refactoring onto a pet project from almost two years ago!

https://github.com/gregallensworth/L.TileLayer.Cordova
This Leaflet layer plugin extends L.TileLayer to add on-device caching of tiles, then the ability to switch between online mode (from web using the URL given) and offline mode (the on-device stash). Utility functions are provided to calculate tile pyramids, to accept arbitrary lists of XYZ ordinates,to view the cache usage, and to empty the cache.

If you make Leaflet maps for use with Cordova or Phonegap, you'll be interested in this!