Thursday, September 25, 2014

Shellshock! A forgotten feature becomes a major security hole

My prior post, was that my next series would be about HTML5 frameworks for mobile. But I've been busy on other projects, and mobile stuff hasn't been on the high-pri list. Speaking of today's high-pri list, how about that Shellshock bash bug?

I'm gonna jump on the bandwagon a bit here,  and admit that this bug should scare the heck out of you. If you run anything that hands out shells to the unknown Internet, then anybody smart enough to make a fake User Agent can run arbitrary shell commands on your server. I tried it and verified it: it took me 30 minutes to write a working exploit, starting from when I first typed "bash bug" into Google.

In a Nutshell


What is the "bash" Unix shell? Briefly, this means "the thing that's used to type commands after you connect via SSH" but for web folks working in PHP this also means "the thing used to execute command-line scripts such as passthru() and the `` operator" If you use shell_exec() et cetera then you're using a shell, and it's probably bash.

Bash has a feature going back some 25 years, that it can auto-execute code when you start a new shell. This code will conveniently set "environment variables" which facilitate other work. The problem with this very-obscure feature, was only discovered last week: Web programs such as CGI and PHP also set environment variables, some of them read directly from the browser without any sanitization or checking. So any browser on the Internet, may be able to:
  • Send an environment variable that will be passed as-is to your shell. The most common such variable would be your User Agent (your browser's name), which is loaded as-is into an environment variable called HTTP_USER_AGENT
  • Which your shell would then interpret as a command to be executed before it executes your shell command.
But that means technical skill to set your browser's User Agent, right? Wrong. Browser add-ons already exist to set your User Agent string to some common values (Firefox can pretend it's GoogleBot) and it's also trivially simple to set it in your browser's config. (Firefox is described below) Again, I hadn't heard about this bug before today and it took me about 30 minutes to craft an exploit against one of our servers which was vulnerable.

The Limits


The exploit allows command execution, only if the following conditions are met:
  • A remotely-executable program exists which does not require trust and authentication, e.g. a PHP script or a CGI script which executes under bash. Services which do not execute shells, or which execute them only for trusted individuals, are of significantly less concern, e.g. FTP and MySQL servers don't run subshells and don't read user-supplied data for environment variables anyway, and SSH only gives you a shell after you have logged in.
  • That program executes a subshell. In PHP this is the `` or shell_exec() functions. Any program which does not use those functions is fine... until you need to use those functions.
  • The system's default shell /bin/sh is in fact Bash and not an alternative such as ksh or dash. On Ubuntu 12.04 /bin/sh is in fact Dash and is not affected by this bug unless you have a CGI script which specifically uses Bash.

A Quick Test

Login to your server, and run this from your shell:
env x='() { :;}; echo vulnerable' /bin/sh -c "echo this is a test"
Hopefully you'll see an error about a function definition:
    ./bash: warning: x: ignoring function definition attempt

If you see "vulnerable" in the output, then your system shell is Bash and is a version which is potentially affected by this issue. And remember, it's not Bash here at the command-line that should worry you, but any web program which launches shells.

If you don't see vulnerable, you may not be entirely safe. Try again, this time replacing /bin/sh with /bin/bash Even if Bash isn't your system default shell, any CGI programs or launched shells whicyh specifically use Bash may still be affected.

A Quick Exploit (A)

Like I said, it's not rocket science, just setting your User Agent.

Load up Firefox.

Go into about:config Create a new value (or edit if it already exists) called
general.useragent.override
and set it to this odd-looking string:
() { :; }; touch /tmp/shellshock
Congratulations, your browser will now supply an auto-executing function in the environment... if you hit up something that spawns a subshell.

On your webserver, try creating this very small PHP program. Or, try simply hitting any program of your own which uses shell_exec() or similar functions.
<?php
# anything that uses any shell function, will trigger the bad HTTP_USER_AGENT variable
print "Tmp Folder:\n";
passthru('/bin/ls /tmp');

Now visit that PHP program in your browser, and then have your sysadmin check out the new /tmp/shellshock file. And imagine for a moment what happens if someone were to do this repeatedly, and knew their way around a shell.

P.S.: Don't forget to "reset" that User Agent override when you're done.


Another Quick Exploit (B)

For command-line wonks, here's an even simpler version of the exploit using cURL from the command line:
curl 'http://www.example.org/whatever.php' -H 'User-Agent: () { :; }; touch /tmp/shellshock'
Point it at any web service which uses a subshell (e.g. a PHP using shell_exec() ) and they'll be surprised to find a new file on their server.

Okay, I'm Scared... Now What?

The most expedient fix, is to upgrade bash. There is already a patch (4.3-25) available.

This page gives practically copy-and-paste instructions:
    https://news.ycombinator.com/item?id=8364385

Now give everything a test again, and breathe a sigh of relief!

Update: Looks like the 25 patch to Bash fixes only one of two so-far-determined methods of attack. A second version has been named CVE-2014-7169 and is not fixed by 4.3-25  The "A Quick Test" for this second vulnerability is:
env X='() { (a)=>\' sh -c "echo date"; cat echo
If the output has the date in it, then the date command was executed and you're still vulnerable.



Thursday, September 18, 2014

What HTML5 mobile framework to use?

Yeah, it's been a month. Again, been working instead of talking about working. :)

Today I'll start touching on a topic as emotionally charged as religion and politics... a choice of HTML5 mobile frameworks for use with Cordova/Phonegap. That's me, always the troll, pushing the envelope of good taste and social acceptability, pushing buttons just to get a reaction. ;)

I've been using jQuery Mobile, but it has some bugs, and isn't exactly swift and fluid; on the other hand, it does have some nice button icons and default styles, and works well with jQuery. But there are other HTML5 mobile frameworks out there which may be more fluid, less buggy, have richer icons and styles.

So this next series of postings isn't really a "How I did it" sort of tech bragging, as much as current opinions on several HTML5-for-mobile setups. Some of them are subject to change as I learn something later, so if your opinion differs, this is a place where your comments would be appreciated!

Our Needs

jQuery is very visual -- you start with a HTML mockup and then you make it interactive. This is how our clients and designers think. They start with cosmetic mockups and designs, which will change often (sometimes 2-3 times in a week) as more visually-thinking people give input and do 180-degree spec changes. The "real programmers" shudder when I press the point, that the design (as implemented in HTML) defines the application, and that project specifications are typically defined after the fact based on budget leftovers and feedback. But that's the reality. We don't need a framework to build apps out of widgets, as much as a HTML/JS toolkit for making app-like behaviors starting with HTML.
  • We're building our apps with Cordova / Phonegap. Virtually everything should work well with Cordova, but then again let's not presume. If the final product doesn't work well once we've plugged it into Phonegap and tried it on a few devices, it's a loss. (see Ionic in a future posting)
  • The HTML/JS API must not be dogmatic, forcing us to use prefab widgets that cannot be quickly customized. The designers and the clients decide what we do; they can be quite capricious, and we never say no to any customization. If they want THIS selector to have its handle on the left while OTHERS have the handle on the right, that needs to happen and pronto. Right off the bat, this means that a lot of programming-heavy frameworks which let you "just tie widgets together" via a configuration file, are right out. (see Sencha in a future posting)
  • But that said, we  do want the prefab widgets... just ones that we can ignore or modify, preferably in HTML. Utility classes are awesome, and so are glyphicons. (see jQuery Mobile and Ratchet in a future posting)
  • We do most of our programming in jQuery, so anything that plays well with jQuery is a big bonus.

jQuery Mobile


Well, let's start with the framework we are already using.

- Good. It's highly visual. As I described above, you start with your HTML and then program behaviors into it. This fits perfectly into the paradigm of our project managers, designers, and clients.

- Good. A very large community and great documentation. There isn't much about jQuery Mobile that remains mysterious after five minutes in Google and StackOverflow and/or ten minutes with JQM's docs. We still need to do some tedious CSS discovery sometimes, but 90% of questions have already been answered.

- Good. Page management. Attractive cross-platform CSS. Decent widget set. Custom event handlers and DOM manipulation. It's pretty sweet.

- Bad. It's a little slow. Even with FastClick, there is a noticeable lag between tapping a button and going somewhere. I've tried changing the delay, disabling page animations, FastClick of course, ... It's not killer slow, but it's not really smooth.

- Bad. Rendering bugs. I find myself implementing corrective CSS on every new project. a) Hyperlinks that are a[data-role="button"] has a visible underlines on the text. This is easily corrected in CSS, but seems a silly thing to need to correct. b) Buttons in the header, will sometimes "stick" in their highlighted/active state long after the finger has moved on, sometimes even after switching to a different page, and then coming back minutes later. This means that the active color needs to be the same as the inactive color (CSS fixes), which sort of misses the point.

- Mediocre. The data-role paradigm is nice, but not as rich and flexible as a good set of utility classes, e.g. class="button button-panic button-wide"  We get two themes (data-theme="a" and data-theme="b"), but need to write custom HTML to simulate a third theme if we want a third button color (but need to replicate that for header buttons and so on. By comparison, Bootstrap and Ionic have several types of button.

- Dicey. Lazy rendering. One of the first executing statements in our programs is $('div[data-role="page"]').page() to effectively disable lazy rendering. Our typical use case, is that a listview needs to be populated programmatically, but we cannot guarantee that the user has previously visited the page and caused the listview to render into DOM, allowing the refresh to happen. Lazy rendering is intended to reduce memory usage, but realistically it's just a race condition waiting to happen.


So, what are other options? That's the focus of upcoming articles...


Sunday, September 14, 2014

MobileMapStarter 1.1.1

Due to the surprising popularity of MobileMapStarter at FOSS4G, I pushed through a few things I'd been meaning to do, added a license, etc. I'll just quote from the release blurb:

Get a jumpstart on building your mobile mapping application!
MobileMapStarter is a ready-to-run Cordova/Phonegap application. It combined jQuery Mobile and Leaflet, as well as a host of workarounds, considerations, utility functions, etc. to get you running a mobile app in minutes.
1.1.1 improves upon 1.1.0 in the following ways:
  • Includes the whole Cordova release known stable at this time (incl the File plugin specifically @1.0.1) so this thing really is ready to rock.
  • Various documentation updates.
  • Added a license (MIT) and clarified licenses, added required Leaflet license (etc.) to generated app
  • Replaced the geocoder from Bing to Nominatim, to avoid potential TOU issues


Thanks so much for the interest. It was very encouraging, and I feel jazzed. :)



Saturday, September 13, 2014

FOSS4G was awesome!

Well, that was fun!

Some highlights of FOSS4G, in no specific order. There'll be some things to re-evaluate, some to prioritize and try to integrate into the next year at work, some side projects after hours, ...

FOSS4G was close by! It's only in the USA every third year, and last time (2011) it was in Denver. But this time, it was conveniently located only 2 hours away. Too far to go home every night, but close enough to stay with a friend and not need a plane ticket.

Meeting up with colleagues is one of my favorite parts. Before I get caught up in the deluge of data and techniques, I should thank again the folks I met at the conference: old colleagues like Karsten and Stephen and Matt and Bob, and new colleagues such as Tanner and Jim. It's been a pleasure meeting you; please follow my profile links and get in touch.

My presentation on the RCS Viewer was something of a dud. It seems that the 10am timeslot the morning after the party, on the last day (short day) just gets poor attendance. That's too bad: the RCS Viewer combines some of the best features for advanced GIS analysis with a simple "GIS whiteboard" as well as PDF take-aways. If you're interested, click here to see the slideshow and be sure to read the notes on each slide for some additional annotations describing the spoken part.

On the other hand, my presentation on MobileMapStarter was a real hit. I felt like a nervous, babbling idiot up there, but feedback afterward said something very different. Several folks came up afterward with compliments, good questions, intentions to use MobileMapStarter, ... and questions about licensing, because they want a clearly-stated license so they can get started using it. Huzzah! Don't worry folks, an explicit license is coming right up this weekend. If you're interested, check out the slideshow and again be sure to read the notes on each slide for the spoken material and other side notes. (oh, Chapter 1 and 2, on setting up the Cordova build environment are back up; see July 5 or so!)

MapServer 7 sounds like a small update, though in fact it seems to have involved a 25% rewrite of MapServer. Why do we use MapServer? It's lightweight and fast, configuration is dead simple (sorry, I do not consider it a painful chore to write mapfiles), you can dump temporary mapfiles to disk and use them for a session, it does variable interpolation, ... MapServer 7 brings performance improvements, then the ProTips had some useful tips, some of which I didn't already know. And lastly, MapCache is yet another tile caching system but one perhaps more flexible than TileCache and TileStache (it can handle raw WMS requests that do not fit tile boxes).

CartoDB's presentation on performance and caching was moderately enlightening. Perhaps the biggest take-home here, was the nature of their caching: it's not a tile cache holding a zillion PNGs, as much as a Varnish cache for the vector tiles that are to be rendered. Making a DB update hits a trigger that issues a varnishctl command. In conversation that day I had guessed about 2/3 of this just mentating on how I would do it... good to know that I was on the same brainwave as the smart guys. :) But it gets me thinking about whether Varnish or even CloudFront could do us some good over at GreenInfo Network, for general caching of not only map tiles but static assets such as PNGs and CSS files. (you know, network contention when downloading the 30+ assets which form a page)

RasterIO (not Raster I/O, but Rrrrrrrrasterio!) is yet another Python-NumPy-GDAL wrapper. But, it's a step beyond all the prior ones such as agoodle and even rasterstats. a) It wraps both GDAL and OGR, so you can handle both shapefiles and GeoTIFFs (et cetera). b) It uses a "more Pythonic" programming interface, e.g. using .read() and .write() c) It may consume far less memory than agoodle by virtue of being able to do a read() on a specific window of the raster, rather than loading the whole thing into memory. For one specific application where we use agoodle, this could be a game-changer.

Then some breathtaking demos of 3D data, such as Cesium. Mostly it was sit-and-spin sort of deals: load 3D data (NED), render a mesh, drape the photo (NAIP) as a texture, and there's a spinny 3D mountain demo, super cool. This is the part I find most visually intriguing, though at the same time the most baffling as to what anybody would do with it, at least as far as my customers who find a "find nearest park" application to suit their needs. I would love to find a use case for some awesome 3D stuff.

And Galileo seems like a very practical phone app that I'll probably download later this weekend. It loads OSM extracts (e.g. states in the USA) and provides a slick little pan-zoom UI. Imagine Google's "Maps" app or Apple Maps, but... without Google and Apple. If I heard correctly, it's for iOS only, which would be too bad since I'm an Android man. But I'll get on the newsletter and keep listening.

Google Closure Compiler keeps coming up. We're already in the practice of minifying our JavaScript code, but Closure goes an extra step and optimizes the code for run time performance. I've gotta give a look to this, see what differences we can achieve in file size and also runtime, compared to our existing applications.

Whew! The above are just the highest highlights that I'll be investigating at work, not even the really experimental stuff for when I get a slow day. WHAT A WHIRLWIND!

THANK YOU SO MUCH, FOSS4G!

Tuesday, September 9, 2014

FOSS4G here I come!

FOSS4G is in Portland, Oregon this year, a mere 2 hours away from my home. Huzzah!

I'm giving two talks:

  • Wednesday afternoon (tomorrow!) I'm showing off MobileMapStarter. This is a starting template in HTML5, for a mobile app for use in Cordova/Phonegap. Copy, paste, and have a mobile app running in literally 10 minutes (5 if you use Phonegap Build).
  • Friday morning, Tommy Albo will help me present the Intertwine's Regional Conservation Strategy viewer. This excellent web app, lets someone draw or upload an area and get back statistics about habitat valuation, park land, etc. The idea is that city planners and land developers can become aware that the area they're about to raze is actually valuable, and find an alternate location.
Beyond that, I'll be meeting friends, coworkers, and colleagues! Stephen from Cleveland. Karsten from Seattle. Keith from here in Corvallis. Tom from San Francisco.  I'm sure most attendees will agree, that hobnobbing with map nerds, is at least as great as the organized talks and events. :)