Wednesday, April 27, 2016

Heroku and Wordpress: The Migration Process

My post day-before-yesterday begun the story of my migration from a home-hosted website to Heroku. Here's part two: The Migration Process

Step: Move my Media Library content into S3


First off, the uploaded photos should not be under version control; so they shouldn't be in the repository and in the ephemeral filesystem.

This was a tedious process of a few hours since I had about 130 images.
  • I set up the S3 plugin I mentioned above, made a few tests and confirmed that it's sweet.
  • I SFTP'd into my Wordpress site's wp-content/uploads folder and grabbed everything.
  • Then deleted everything from my Media Library,
  • Then uploaded it all again and watched it load into S3 and leave my uploads folder empty.
  • Then went through every posting and replaced all of the images, which of course were now broken. Annoying, but fortunately I only had 35 pages with images and did it in about 2 hours.

Step: Init repo


I'm a fan of Gitlab. They offer unlimited private repositories for free, which is really excellent. I created the repository, then followed their simple instructions to load my Wordpress files into the repo and basically turn my site into a clone.

I also created a .gitignore file with these two entries for some folders I'll be creating in a little bit. The private is where I'll do some other work I don't want in version control, e.g. working files and database dumps that I want to keep close. The vendor would be generated by composer later on, trust me.
/private/
/vendor/

Step: Add Heroku as a Secondary Master


The trick in allowing git push to push to deploy to Heroku, is that you tell your repo clone to use your Heroku as a secondary master.

heroku git:remote -a your-server-name

As of now, when pushing you will need to distinguish between git push origin master and git push heroku master. One pushes into your git repository (Gitlab, Github) and the other would redeploy to Heroku.

Step: Wordpress Updates


I then noticed that some updates were available for some plugins and for Wordpress itself. So I ran those, and after each one noted that git status reported exactly what I would expect from each upgrade. So three commits later I had Wordpress and plugins all updated, with commit notes for each update. (I could have done this before the repo init, but why not do it under version control?)

Step: Add a Procfile and composer.json file


For Heroku compatibility, it's advisable to add a Procfile and a composer.json and composer.lock file, to indicate to heroku what PHP version to prefer, that you prefer Apache over Nginx, etc. You will want these in version control.

Procfile
web: vendor/bin/heroku-php-apache2

composer.json
{
  "require" : {
    "php": "^5.6.0"
  },
  "require-dev": {
    "heroku/heroku-buildpack-php": "*"
  }
}


composer.lock is generated from the composer.json with a command:
composer update --ignore-platform-reqs

Step: Push to Heroku


All set? Then here goes:
git push heroku master
And I visit my website. And it's a Wordpress error that it can't make the database connection. That's to be expected: my wp-config.php has the old credentials and I still need to upload the database content. But that's definitely Wordpress making the error, so a fine start.

Step: Database Configuration


First step was to scrub my database credentials from the wp-config.php file. Yeah, dummy move to forget to do that, but the credentials are wrong for Heroku so are useless, and my local MySQL is going away in an hour anyway. But yes... don't do what I did. ;)

When I scrubbed the database credentials, I replaced them with $_ENV variables like this:
define('DB_NAME', $_ENV['DATABASE_BASE']);
define('DB_USER', $_ENV['DATABASE_USER']);
define('DB_PASSWORD', $_ENV['DATABASE_PASS']);
define('DB_HOST', $_ENV['DATABASE_HOST']);
Add, commit, push. Site is still dead but it's ready for this next trick.


Run heroku config and it dumps the app's configuration variables back to me. I tease apart the database URL string, into my set of 4 environment variables for the 4 $_ENV items above.
heroku config:set DATABASE_BASE='heroku_XXX' DATABASE_USER='XXX' DATABASE_PASS='XXX' DATABASE_HOST='XXX'
My app/dyno reboots and... the site's up!

Step: Database Data


A simple mysqldump was all it took to take a backup of my database, then loading it was one more command.

mysqldump -h localhost -u olduser -p olddbname > private/dbdump.sql
mysql -h herokuhost -u herokuuser -p herokudbname < private/dbdump.sql

It doesn't get as lot easier than this.

And now the site is up! My data, on a new database on a new Heroku server. Shiny.


Step: Custom Domain


My site is www.fightingfantasyfan.info and not fightingfantasyfan.herokuapp.com Heroku calls this a custom domain. There are two steps in setting up the Heroku app to work properly with a custom domain.

  • I hit up Heroku's dashboard and the Settings for my site, and added two domains for it: www.fightingfantasyfan.info and fightingfantasyfan.info This allows it to respond to these alternate hostnames, should they point to this app.
  • I went to my domain registrar's domain control panel and set up a CNAME record, so that www.fightingfantasyfan.info is equivalent to fightingfantasyfan.herokuapp.com This took a little bit to propagate, but was done about the time I finished my cup of coffee.


And that really was it. Well, almost. More on this tomorrow.


Monday, April 25, 2016

Heroku and Wordpress

This is a bit off-topic for The Map Guy since it has nothing to do with maps, directly. But it's interesting and it's about the cloud, and about software deployment. It's about... Hosting a low-volume personal Wordpress website on Heroku for free.

Introduction


I have a couple of personal websites for hobbies. I'm almost ashamed to admit that I'm using Wordpress for them, being the hardcore, bad-ass hacker that I am... but in the evenings and with nothing at stake, the convenience of Wordpress works for me.

I had been hosting my site on a Raspberry Pi at home, but the reliability just wasn't there. The Pi would crash from time to time and require rebooting. After the second time it did this while I was away on a trip, I decided it was time for a change. Even better, this is an opportunity to say Hello To The Cloud with a real-world application with no stakes involved.

Enter Heroku, my hero.

Heroku is a service that spins up virtual machines (they call them "apps"), using as their filesystem your git repository. The basic idea is this:
  • You have a git repository which contains your website: HTML, PHP, Python, Ruby, etc.
  • You create an "app" via their dashboard, also called a "dyno", and it has an URL. You then attach "Add-Ons" such as a MySQL database, a PostgreSQL database, SMTP via SendGrid, and other such services that you'll need.
  • Using Heroku's command-line "toolbelt" you add to your clone of the repository, the fact that this clone is "connected" to that app/VPS. So your clone has two masters: git push origin master for saving your code to version control as usual, and git push heroku master to deploy it to the site.
  • When you push to heroku master, Heroku resets your VPS, loading up its filesystem from your repository content. Assuming that your code works, the site works too.
  • No permanent storage ("ephemeral filesystem"). When the VPS is load-balanced to another system, or put to rest, or you push again, it's reset from your repo. This is very important if your web app will be making filesystem modifications such as accepting uploads. More on this later.
So the effect is a build-from-components server, enabling the specific services you need.

For small personal sites, free tiers for the dyno and for the add-on services may prove sufficient to host the Wordpress code and the database entirely for free. The limitations of this free tier, as relevant to my personal Wordpress site, are:
  • The dyno (the VPS itself) will go to sleep if there's no activity, and the next incoming activity to wake it up does mean some delay. The dyno must be asleep at least 6 hours per day, so using Uptime Robot to keep it awake is a no-no. The next step up to get rid of the sleeping, is only $10 per month.
  • The ClearDB MySQL database is free only to 5 MB. This may not be enough for a long-running daily blog, but for a few postings per month maybe it's just what you need. The next step up is only $10 per month for 1 GB.

Since Heroku does support PHP, and a mysqldump of my database is only 2 MB in size, this sounds right on target for a Heroku freebie.


Ephemeral Filesystem


Let me reiterate that "permanent storage" comment above. Your Heroku filesystem is a git repository. Your dyno does have the ability to write files on disk,  e.g. an uploads folder, e.g. the Media Library component of Wordpress. But you won't like it. When your dyno resets, the filesystem is reinitialized and your modifications would be lost.

By "reset" is meant a reboot, it being load-balanced onto a new server, it going to sleep and waking back up, your next git push, or using heroku config:set to change environment variables. Bye bye, uploaded files.

The up side of this is that a hack of your website, assuming it didn't damage the database, can be solved by rebooting. The down side is that you need to find someplace else for long-term storage of any web-supplied files not in your repository (or else make a practice of manually downloading the files and adding to the repo? sounds awful)

In the case of Wordpress the easiest solution I found is the WP Offload S3 Lite plugin. When you upload to the Media Library, it goes to Amazon S3 instead, and media URLs are rewritten to point to their S3 version. Between Amazon's generous 5 GB free tier for a year and the real price being a paltry 3 pence per gigabyte per month, even an image-heavy website can get by on pocket change per month.

If you are writing your own ware, you'd want to code for your cloud storage of choice such as Amazon S3, Google Drive API, Dropbox API, etc. where you supply a file and get back a URL. I imagine you'd need to generate API keys, program the OAuth-style exchange of them, handle errors etc. and that sounds awful. For my own case here, though, the folks at Delicious Brains had generously done that heavy lifting for me.


Happy Ending


Spoiler time: My applications are all running on Heroku and the performance is quite acceptable.  And I learned a lot in the process. The rest of this story is how I got to this happy place.

So, now that I've covered the basics, tomorrow's story will be about the migration process!

Monday, February 15, 2016

Some Leaflet controls for modal dialogs

A recurring need we have, is to open a dialog or a modal from within a Leaflet map. Dialogs and modals are a great way to add Feedback links, to show the legend without taking up screen space, to add a Help or About panel, etc.

A perfectly ordinary way would be a positioned DIV anchored to the edge of the screen or to the edge of the map DIV, so it looks like a "tab". You've surely seen this before for Feedback links along the edge of the screen. But this time we wanted a button in-map, something that looked like a real Leaflet button. It wasn't tough to do, cuz Leaflet rocks.

Now... having done it, why not polish it up so that next time it's copy-paste simple? Here we go:

https://github.com/gregallensworth/L.Control.BootstrapModal
https://github.com/gregallensworth/L.Control.jQueryDialog

I even took the step of making a jQuery UI version. The project last week didn't use this, but it was a small step to adapt the button code, and it will surely serve us in the future.

Enjoy!

Friday, January 8, 2016

IonicMapStarter, the Ionic successor to MobileMapStarter

At winter solstice 2012, I took a few days of vacation time and used it to create MobileMapStarter. The intent was to boil down the essential and generally-reusable bits of a mobile mapping app, strip out the application-specific stuff, and have a starting place for our future map apps. This meant:
  • A working mobile framework, with page-view management and all
  • Working around certain bugs and issues of Leaflet when used inside page-view systems
  • Working boilerplate code for geocoding, location tracking, etc.
  • The thing being designed with configuration separate from execution so it's simple to reconfigure
  • Ability to cache tiles for offline use
And it worked out famously. I presented MobileMapStarter at FOSS4G 2014 (welcome to Portland!) and was surprised at how popular it became.

But, jQuery Mobile has problems and I have been looking for a newer framework to replace it. Some weeks back I really got to like Ionic, and winter solstice 2015 has brought you...

IonicMapStarter


It's the same concept as before: a mobile app scaffold, that's easy to reconfigure and adapt to form your own mobile app. But this time it's built with Ionic. I mentioned a few weeks back some improvements which this change brought to ParkInfo Mobile:
  • Better performance all around, from the map to general page-loading and panning behavior
  • Cleaner structure from the ground up, and a reduction in code volume
  • Improved UI for the offline tile caching, as well as improved capabilities
We still aren't releasing ParkInfo Mobile until we finish some branding
 and functional tweaks, but you can start on your next-generation mobile app tonight.


Monday, January 4, 2016

JSHint + AngularJS = Tight as a drum

So, in the development of the new version of ParkInfo Mobile, I wanted to run things through some code-quality checks as a matter of course. I'm good at what I do, but having a second pair of eyes (cybereyes!) look for missing semicolons etc. sure won't hurt.

I went with JSHint and I really like the results.
  • JSHint noticed a few stragglers such as semicolons, trivial stuff that could lead to larger goofs. Making use of JavaScript statement-chains crossing lines, means that a stray semicolon or a missing semicolon cause truly bizarre malfunctions. For this reason I prefer not to use the multi-line syntax, but with Angular it does happen a lot so this extra check is nice.
  • JSHint also complains about undeclared variables, e.g. L which is declared in leaflet.js and angular which is declared in angular.js These weren't errors at all but are easily permanently silenced... and could have been invaluable if the "undeclared global" were actually a typo.
  • JSHint also reports unused variables. In most cases this was a callback that receives an error object, and we don't use the error object. But in a few cases it was dependency injections which were no longer in use. So a second use of JSHint was to check for unused dependencies and for erroneous undeclared dependencies. Very nice.
So, I threw together a quick-n-dirty shell scrip to run everything through JSHint, and I run it when I get ready to push.

Step 1, install JSHint via npm. You're used to this if you use Cordova:

npm install -g jshint
Step 2, set up this script. Heck, add it to your source archive:
#!/bin/sh

for js in www/index.js www/controllers/*.js ; do
    echo "********** CHECKING $js"
    echo ""

    jshint $js

    echo ""
done
It's dead simple, nothing special... but it does keep things a bit tighter, and makes for easy and automated checks for some common goofs.

Now, the other tool commonly used here is JSLint. But so far I'm not impressed with it. Reporting unused variables and potential typographical errors is great, but JSLint reports truly stupid stuff such as having a space after the colon in an object literal, stuff which truly has no impact nor potential for impact. Maybe later, but not today.

Wednesday, December 23, 2015

A new ParkInfo Mobile using Ionic & AngularJS

It's been three years this week, since I wrote my first mobile app: ParkInfo Mobile. And this week, the new one is still under wraps for some branding and color choices, but is functional. ParkInfo Mobile 2.0.0 should be in the app stores in a week or two, replacing my Christmas 2012 edition.

I'm not particularly sentimental, but it's a big step forward in a few ways. Let me share a few interesting changes in the new version, and lessons learned by writing it.

What's New


First and perhaps largest: It's written in Ionic, and thus AngularJS. This is a paradigm shift from jQuery Mobile, in which the DOM and global variables rule everything. The code is broken into a bunch of bite-sized segments: each panel is one HTML page and one paragraph of code, though some of them refer to the "global" functions for some commonly-used functions and needs.

It doesn't use an onboard JSON file, but HTTP hits to a web service. The original ParkInfo Mobile had a JSON file included, which could be used for searching and estimating distance without using the data connection. But it kinda sucks. Including the polygons made the file about 100 MB and would crash any phone, so it didn't happen. Using only the centroids means that we can estimate distance and direction to the centroid; for a city park that's not too shabby, but for a 500,000-acre preserve the centroid could be miles away and be misleading. The new one uses a web service to query the database; it's swift, simple, and effective. Better, the web service can accept a hint as to your origin lat-long and return the closest point to each of your results, greatly improving the accuracy and usefulness of the results.

The performance is fabulous, and look and feel is smoother. They said that AngularJS and Ionic have decent performance, and I know that jQuery Mobile kinda sucks... but dang, it's really that much improved. Navigation is smoother, more fluid. The default styling doesn't include so many drop shadows and rounded corners, so the whole thing feels more bare... if a bit cartoonish with those colorful buttons. (Like I said, the colors and branding stuff is still pending.)

The UI for offline tile caching is improved. The underlying technology is much the same: Cordova's file API, with ngCordova wrappers. But the UI for caching is greatly improved.
  • Caching by current map view is disabled, in favor of two new options: cache around a street address, or around your current location. This is a lot more useful, generally speaking.
  • "Passive caching" will cache your map tiles to device storage while you browse the map. If a specific address or your GPS location don't fit what you need, just let the cache populate as you pan and zoom.
  • While downloading tiles, the UI doesn't block you with a modal, but starts up a progress bar and lets you go on your way. Downloading 1000 tiles can take a few minutes, but you're not waiting around.

Lessons, Hacks and Limitations

 

Some of the things that went less smoothly:

When you need two controllers to know each others' state, AngularJS is a pain in the neck. The cache panel needs a list of map layers, but it's the map controller which has the Leaflet object and the L.TileLayers. Ultimately I really did need to separate out the layer listing into a Constant where it could be seen by both. Angular's all about not believing in globals, but dogma had to take a seat while pragmatism prevailed.

Ionic's sidemenu  is cute, but not flexible. It took me only 2 hours to come up with a thing that our clients will demand that it cannot do: put an icon in the top-right corner, which varies on every page (different icon & link, or none). I tried everything from position:fixed to negative top margins, even tried a nested view in the titlebar, and it's a no -- that titlebar isn't part of the dynamic part of the view. So I went the way I knew I would eventually, started with a blank template where each page has its own titlebar so we have maximum flexibility. I knew it would happen since our capricious designers are the ones calling the shots, but it irked me that those "copy-paste my code, and you're a mobile developer" articles basically have someone copy-paste the sidemenu starter, knowing that it paints you into a corner less than 2 hours later.

I've still not wrapped my head around Services and writing my own promise chains. The web service calls would ideally be a Service instead of a function defined in $rootScope. But a service which accepts callbacks as parameters, which makes a $http call and hands off to those callbacks just eluded me. After a day, I needed to just get past it so I gave up and went with a $rootScope function that accepts a pair of callbacks, without a promise involved. Similarly, I couldn't figure out a promise chain to wrap a dynamically-generated list of potentially thousands of Cordova File API calls and perform them sequentially, so I went with my old "i+1" method. Then again, other reading indicates that a chain of a few thousand promises maybe isn't such a hot idea anyway...

Debugging is a real nuisance when there aren't any globals, and you can't see into scopes private data. Keeping everything in tiny secret boxes, doesn't do well toward being able to type MAP.getCenter() into the Firebug Console to find out something so simple -- and that makes all debugging just that much slower. I need to put in some effort, into figuring out techniques for debugging AngularJS apps aside from console.log() inside these isolated black boxes.

Angular-leaflet-directive is pretty limited, and I've had to do a bunch of work to it myself. I'll likely fork it and start adding my own patches into it in a more concerted manner. But some of the adjustments I've discovered and invented include:
  • You must use in in a ion-view with overflow-scroll="true" or else the map will cease to receive click events if you navigate away from it. It's not just the map, it's something about the underlying DIV. Those "copy-paste your first mobile app!" postings don't mention stuff like that, eh?
  • It doesn't support popups, so I added a popup bit to it.
  • It doesn't propagate bounds changes back to the scope, only center. Fixed that. 

 

The Future of MobileMapStarter


My second mobile app was actually stripped down from my first: ParkInfo Mobile's map/cache/page functionality was stripped down to form MobileMapStarter.

Last year after I presented MMS at FOSS4G it got some popularity, but I've been less than 100% satisfied with it. It's jQuery Mobile so it's slow, the cache UI isn't so hot, etc.

But here we have a new framework with a map, panels, and caching. If I strip this down a bit, it's a fresh new start for "MobileMapStarter, Ionic Edition". Schedule permitting, maybe I can have that done in a couple of weeks.



Monday, November 30, 2015

Ionic and AngularJS

In my personal time, I have one mobile app that I manage: BattleTech Calculator. It's a calculator for an old 1980s board-and-dice game, makes a lot of the tedious math move more quickly so we can focus on pretending to blow up our giant robots.

I went with Framework7 for this one, on grounds that it looked like a winner. It's HTML/CSS with some DOM manipulation capability similar to jQuery. It has basic routing, doesn't look half bad. And BTC being a non-work app, I figured there's no harm if Framework7 turned out not to work out well.

Trouble With Framework7


Ultimately, it did not work out well.

First was the freezing/crashing issues. I thought it was just my phone because the other two folks didn't have any problem... until they reported that they did. Just switching between tabs in the usual fashion, it would freeze solid and you'd have to terminate the app and start over.

Second was a reported visual rendering bug. I could never replicate it myself, but I got several reports in the Store after one update, that when they switch tabs the old tab's content would be visible on top of the new tab's content. Again, I couldn't make it happen but the reports were consistent enough that, combined with the freezing, I decided to try something new.

And that's not including some documentation inaccuracies, the $$ and $ bug, and a few other quirks and shortcomings.

Ionic Framework is based on AngularJS, and on the web side of things that's a totally new thing for me. As I described in prior blog posts searching for a HTML framework, Angular is designed around concepts like modules and components, instead of the designer-driven daily-spec-change chaos which typifies a lot of our web-map applications. But in this case, I did have a solid idea of what I wanted, and even a working prototype in my old version.

It took some getting used to... but Ionic/Angular kinda rocks.

DOM manipulation and AngularJS


DOM manipulation is out the window. Because of data-binding on the form widgets, the ng-show and ng-hide directives allow other stuff to show and hide as the form is manipulated, allow for recalculations to happen any time the form is changed, etc. The result is more intelligent form widgets and less code and code repetition.

Old, a separate piece of logic to show only the applicable table:
var which = $(this).val();
$('#tab-meleehitlocation span.meleetype').hide();
$('#tab-meleehitlocation span.meleetype-'+which).show();

New, the table is intelligent as to its visibility condition:
<div ng-show="viewdata.meleetype == 'punch'"> Lorem Ipsum </div>

Old, remove the highlight from all rows, add to this one:
$('#tab-meleehitlocation span.meleetype-punch div.row').removeClass('hit').filter('[data-roll="'+rolled+'"]').addClass('hit');

New, the row knows when it should be highlighted:
<div class="row" ng-class="{ highlighted:viewdata.outcome == 2}">
This data-binding and implicit state awareness is a key point of what makes AngularJS what it is, but felt very foreign and difficult to wrap my head around at first. But once it sunk in... I'm impressed. I think that AngularJS's "state propagation intelligence" could achieve any sort of unpredictable DOM manipulation we could need, in a more intelligible fashion than our old jQuery implementations.


Ionic Framework and Ionic View


Ionic Platform itself is a set of tools based on Cordova/Phonegap, but adding their special sauce of CSS utility classes, button colors, a set of icons, etc. And yet, I was allowed to ignore and override to suit a few specific needs, e.g. a SVG dice icon, in lieu of FontAwesome which did work just fine.

So as far as the HTML/CSS flexibility, Ionic really did work well here.

And then there's Ionic View, Ionic's test-deployment platform. It's similar in theory to the Phonegap Developer App:
  • Your testers download the Ionic View app and sign in to the Ionic View service, and see a list of app which they have been invited to test;
  • You type ionic upload to upload your content to the Ionic View servers;
  • Testers re-sync your app, and re-launch and test. No code signing, OSX builds, etc.
This significantly accelerated my testing cycle, even as one single person on a weekend project. The uploads are as fast as Xcode compiling it and far faster than dropping the APKs onto Google Drive and waiting for them to sync. And since it's over data (meaning, while I'm at home, over wifi) I can test it as I walk around, refill my coffee, etc. without ADB pitching a fit and forgetting to reconnect the device, etc.

So yeah, Ionic's non-Angular nature... also a winner.


What It Means For BattleTech Calculator

The new version of BTC isn't finished yet; I have one whole panel to go. But let's look at bits of logic and overall code volume, and see what Ionic has helped me to improve.

https://github.com/gregallensworth/BattleTechCalculator/tree/master/www
https://github.com/gregallensworth/BattleTechCalculator-Old/tree/master/www

  • The very large HTML is broken into partial views. This keeps me visually focused, which is great if you have "situational ADD" aka being interrupted all the time.
  • HTML volume is currently 609 lines compared to 1125 lines. Add another 125 for the last page, and this is still a 40% reduction.
  • JavaScript code volume is 295 lines (not counting that unfinished panel, maybe 150 more) compared to 522. This would be a mere 20% reduction... but still 100 fewer lines of code. This is largely due to the removal of DOM-fussing since AngularJS takes care of that automagically.
  • Almost every line of code is simply more readable: default settings are declared in the controller instead of as checked attributes, code is more logical since it doesn't read from fields and do the DOm shuffling.
  • The code is even easier on the eyes, since $('#') is absent.
  • The app runs very smoothly for the most part, under Ionic View, on my old Galaxy S3. Smoother than with Framework7.
  • The app hasn't crashed once yet, in a few days of development and testing. This counts for a lot!

What it Means For Work


Virtually all of our work is websites, and the designers use mobile-unfriendly layouts with floating panels and the like. So Ionic won't be doing a lot there; it can scrape by for websites, but support for IE isn't so hot. But for the few mobile apps we do each year, we have a winner.

For websites, I am starting to get into the feel of AngularJS. If we can combine this with something visual like Bootstrap, it could be very neat. The big challenge there, of course, is that our clients and designers don't have Bootstrap on the brain so we end up rewriting large parts of the visuals anyway. But for a general system of DOM management and data management, AngularJS is looking pretty sweet, and I'll be working with it some more on personal projects.