Software Walker

“About time, too. He’s been looking dreadful for days; I’ve been telling him to get a move on.“—Albus Dumbledore about his phoenix Fawkes after Fawkes burst into flame

Just as Fawkes periodically must burst into flames and rise again from the ashes, so must my longstanding website project, BirdWalker.com. To better understand why it’s time for me to “get a move on”, here’s some background.

A brief history of BirdWalker

I’ve been watching birds since the mid-1990’s and programming computers since the early 1980’s. When my wife Mary and I started birding, I kept notes on paper while out watching birds. I wanted to count how many different bird species I’d seen in a given month or year or county. I quickly realized I wanted a database to help me track these numbers, and thought it would be fun to build one from scratch. When I started photographing birds a few years later, I wanted to count how many different bird species I had photographed as well. BirdWalker 3 is the the third incarnation of that tool. Here’s a quick look at how BirdWalker evolved.

Version 1, FileMaker Pro

I got serious about birding during the height of my FileMaker Pro obsession. FileMaker Pro was a simple database engine combined with user interface tools. I made my first birding database with FileMaker Pro. I ran it locally on my own laptop and never hooked it up to the web. I used my Palm V to record sightings in the field. Although there were ways to publish FileMaker Pro databases as web sites, I never got around to using them.

Version 1.5, XSLT

In 2001, while working at Reactivity, I had the idea to export my FileMaker Pro data as XML text files while I was learning XSLT. XSLT (eXtensible Stylesheet Language Transformations) is a language for describing how to transform one file format into another. I transformed the raw data of my bird sightings into a website of different kinds of reports, all linked together. I became the XSLT expert at Reactivity (there wasn’t much competition for that title, believe me). I disliked SQL databases and hadn’t really learned about CSS or JavaScript, so I was using XSLT to do the work of a database and an application server and a presentation layer. The complexity of the resulting code, plus XSLT’s arcane syntax, lead to an idiosyncratic, unmaintainable monster.

BirdWalker, XSLT version, October 2003. This page is a species report showing all my observations related to Great Horned Owls, including when and where I observed them, plus my first photo of this species

Version 2, Linux, PHP, MySQL

In 2003, I decided to abandon XSLT, overcome my distaste for SQL databases, and begin again in PHP and MySQL. This combination of tools, also known as LAMP, was much easier to learn and manipulate than XSLT, so I was able to build things much more quickly and easily. In addition to producing summaries and reports of my birding notes, I added the ability to record new notes right into the website.

In hindsight, my biggest mistake during this PHP era was ignoring the PHP community and their wisdom about how to cope with software complexity. Lacking role models, I ended up with a codebase that mingled topics that should have been kept separate, what Brian Foote calls a Big Ball of Mud. My code didn’t cope with errors, wasn’t well tested, and couldn’t be easily extended.

Version 3, Ruby on Rails

In 2007, I saw the classic Ruby on Rails “make a blog in 15mins” video and was hooked. I created the BirdWalker 3 repository on GitHub and started on a new version using basically the same SQL schema as I used with PHP. Thanks to the Rails scaffolding generator and separation of concerns, I ended up with structured code and automated tests. I also adopted Twitter’s Bootstrap framework, so the web site would work on both my iPhone while out in the field and on my laptop back home. BirdWalker version 3 is still the code that runs when you visit birdwalker.com.

BirdWalker, Ruby on Rails version, February 2016. The homepage consists of several Rails partial templates that can be rearranged as needed.

I like it well enough, but there are some big problems:

Big ball of mud

I’ve mostly forgotten how my own Rails code works — my Rails HTML templates and CSS styles are overly complex and hard to modify. Also, there are a lot of half-finished parts, such as better ways to do editing and heuristics for adapting page templates to varying amounts of data.

Application server — slow and overpriced; necessary?

I’m hosting BirdWalker 3 on an Amazon EC2 m1.small instance running MySQL and Rails. This both costs more than I want while being slower than I’d like. Given that the data in the SQL database only changes when I add new sightings (once or twice a week), administering and paying for this whole server feels like overkill.

The ascendance of eBird

While I was building and living with BirdWalker 3, Cornell University’s eBird grew and gained prominence among the birders I know. I exported my SQL database into the eBird format, and uploaded over 20,000 sightings into my eBird account. This is my contribution to eBird’s massive citizen science project, which I love. eBird also lets me compete with my peers in logging the most eBird sightings.

But now I have two sources of truth for my birding life, BirdWalker and eBird. As eBird matures, I find that maintaining my own data entry tools for birding makes no sense. I’d much rather do all my data entry at eBird and then base my website on data exports from eBird.

Arising from the Ashes

I’ve made two experiments since 2013 in preparation for replacing BirdWalker 3. The first was about replacing Ruby on Rails with JavaScript and Angular. The second was about replacing server-side code with static hosting.

An Angular transplant

I have frequently and loudly evangelized the advantages of single page web apps as part of my work at Mozilla. Following my own advice, in August 2013 I started to rework BirdWalker using Angular. (I actually tried twice, once with a web components library named Brick and a second time with Twitter Bootstrap). I recreated almost all of the reporting and browsing parts of my site using Angular controllers and templates (see http://birdwalker.com/angular/#/home). My Rails server-side code was reduced to a modest JSON API. Unfortunately, that code was still limited by SQL performance.

Static hosting of data-driven sites

In 2015 I was inspired by talks at the first GitHub Universe conference and a publishing tool my Mozilla coworkers made called Oghliner. This led me to experiment with publishing this site on GitHub Pages, moving all the application logic into the browser. I created a new code repository called ebird-mybird. I wanted to discover whether I could build a client-side web experience for browsing the CSV text file export from eBird. I’ve used these building blocks:

• Handlebars for client-side templates
• Oghliner for offline access using Service Worker
• PapaParse for parsing CSV
• Mocha for unit tests
• Gulp for build tasks
• Lunr for full-text search

The results so far are functional, if not yet beautiful. It lets me view bird lists for individual days, whole years, specific locations, counties, and states. I’ve got a search that lets me find individual species and locations and trip names. When I make changes to the software, Oghliner automatically rewrites my service worker and updates GitHub Pages using Travis-CI. On most laptops with broadband, the front page starts drawing in less than half a second, and the whole site is ready in about 3 seconds. On my iPhone 5, the front page starts drawing in less than a second, but takes 25 seconds to load completely, way too slow. Almost all of that time is spent parsing 4MB of eBird data, which is crazy. Almost as crazy as writing a website entirely in XSLT.

Now what?

Having discussed the above with several folks, I think my next step is to parse the CSV file on the server instead of the client, and to make a simple NodeJS server with a simple REST API that talks to a client with either the Handlebars templates I built or the Angular code I abandoned. I’ll keep you posted.

[When I’m not in the office, I’m often out birding and photographing birds. To keep track of my life list and bird photos I wrote a Ruby on Rails site hosted at birdwalker.com; you can see all the source on github. This post is about some the evolution of that code]

I love maps and I love visualizing data, so when I started to accumulate location-based birding observations, I knew I wanted to visualize my data on a map. I created a Rails partial template called _google.html.erb that let me insert maps into any of my other templates. All was well.

Since the first version of that template I’ve struggled with two issues that I expect other web developers will encounter.

1. Making Google Maps responsive

When I first brought my Google Maps into a Twitter Bootstrap design, I found the Bootstrap CSS would resize the container for the map, as I hoped. Unfortunately, Bootstrap could not magically adjust the scaling or center of the map. As the map got smaller, it would show only the upper left corner of the original map area.

To fix this problem, I wrote a function to retrieve the width of the container, pass that info the Google Maps API, and trigger a resize event.

function resizeBootstrapMap() {
    var mapParentWidth = $('#mapContainer').width();
    $('#map').width(mapParentWidth);
    $('#map').height(3 * mapParentWidth / 4);
    google.maps.event.trigger($('#map'), 'resize');
    console.log(mapParentWidth);
}

I also added an event listener to invoke this function whenever the window changed size:

// resize the map whenever the window resizes
$(window).resize(resizeBootstrapMap);

2. How much interactivity should I provide?

When I finally got my maps to size up and down responsively, I was very pleased. I used Bootstrap’s grid layout, so that the multiple columns on big screens would collapse down to one column when viewed on phones, with the map taking up almost the whole width of the screen (you can use Firefox’s responsive design view to see this in action).

When you embed a Google Map, you get a lot of interactivity by default — panning, zooming, the ability to click on map pins and other objects. On laptops and desktops, I like the result. I can embed a map showing you an overview of all the places I’ve birded in my home county, where you can also let you zoom in and find the names and descriptions of each. But this interactivity can be a problem on phones. It became difficult to scroll through and beyond one of these maps. I’d be dragging my finger on the phone, and instead of scrolling the page, I’d be panning the map.

I then tested two alternatives. First, I tried using the Google Static Maps API, which has no interactivity. However, it doesn’t work with Bootstrap’s responsive layout for all the same reasons I discussed in my post about D3.js. Second, I tried using the dynamic Maps but disabling most event handlers. This also works, but eliminates the ability to browse the map and see the metadata I’ve associated with each map pin.

So, for now, I’ve restored the panning controls and draggable maps and just try to scroll carefully. Thoughts? I’d love to hear from you how you solved this problem.

[When I’m not in the office, I’m often out birding and photographing birds. To keep track of my life list and bird photos I wrote a Ruby on Rails site hosted at birdwalker.com; you can see all the source on github. This post is about some recent improvements to that code]

When I first discovered Google Image Chart API, I had written little JS and was excited about how easily I could create decent-looking bar charts without including big libraries or writing a lot of code. I added a method in my Rails ApplicationHelper class to generate a URL for the Google Image Chart API by encoding my parameters and data values into the expected format:

def counts_by_month_image_tag(totals, width=370, height=150) 
  monthly_max = 10 * (totals[1..12].max / 10.0).ceil

  stuff = {
    :chco => 555555,
    :chxt => "y",       
    :chxr => "0,0," + monthly_max.to_s,
    :cht => "bvs",
    :chd => "t:" + totals[1..12].join(","),
    :chds => "0," + monthly_max.to_s,
    :chs => width.to_s + "x" + height.to_s,
    :chl => Date::ABBR_MONTHNAMES[1..12].join("|")
  } 

  chartString = ("http://chart.googleapis.com/chart" + "?" + 
    stuff.collect { |x| x[0].to_s + "=" + x[1].to_s }.join("&")).html_safe

  ("<img src=\"" + chartString + "\" alt=\"Totals By Month\" width=\"100%\"/>").html_safe
end

I started using this helper from my various page templates and all was well. Time went by.

Eventually, I became disenchanted with this solution. I can develop and test the rest of the site locally on my laptop, but the external image URL’s like those aren’t reachable offline. The Google chart images are statically sized, so they’re ill-suited to responsive design frameworks like Twitter Bootstrap. And, rendering charts as images doesn’t allow for any interactivity. [note: Google addressed these deficiencies in a subsequent version of their Chart API, which does most of the work client-side, and adds a lot of scope for interactivity].

Meanwhile, I became more comfortable with JS and began to create more of birdwalker.com using it. After attending a conference talk about D3, I decided to copy a sample bar chart and adapt it for my purposes. I put the finished code in a Rails partial called _species_by_month.html.erb

D3.js code works by chaining many function calls together. Each call has a very specific purpose, and the calls can generally go in any order. By chaining them together, you get the overall behavior you want. For example, to create the bars in my bar graph I do this:

var bars = svg.selectAll("rect").remove().data(sightingData).enter()
  .append("rect")
  .attr("x", function(d, i) { return x(d.month); })
  .attr("y", function(d, i) { return y(d.count);})
  .attr("height", function(d, i) { return height - y(d.count); })
  .attr("width", function(d, i) { return x.rangeBand(); })
  .attr("class", "bargraph-bar"

The calls to attr() specify the location and size of an SVG rectangle and assign it a CSS style name. By calling data() you can iterate over an array of data; by calling enter() you can create new SVG nodes for each item in the array. In this case, we iterate over twelve numeric values representing birding activity during each month of the year and create twelve rectangles.

These new graphs resize nicely as elements within my Bootstrap layout. And I was able to create a tooltip when mousing over each bar like this:

svg.selectAll("rect").on("mouseover.tooltip", function(d){
    d3.select("text#" + d.month).remove();
    svg
    .append("text")
    .text(d.count)
    .attr("x", x(d.month) + 10)
    .attr("y", y(d.count) - 10)
    .attr("id", d.month);
});

And remove the tooltip on mouse exit like this:

svg.selectAll("rect").on("mouseout.tooltip", function(d){
    d3.select("text#" + d.month).remove();
});

The tooltips are styled with CSS just like the other graph elements.

So far, my experience with D3 has been really great. I now have graphs that work well in a responsive layout, don’t require external image links, and have some interactivity. Suggestions and code reviews welcome!

-Bill