Everything was fine…until I saw the localStorage entry on html5please.com.  Did you know there was such a heated debate to stop using localStorage and start using IndexDB in tutorials?  I certainly didn’t.  Suddenly, I was torn!  Do I use the simple solution with limitations OR a robust solution that’s not fully supported (I expected more from you, Safari)?

Then it hit me; this is the web.  As with any web project, let’s implement the what’s right and provide support for browsers that haven’t made it up to us yet.  Therefore, the default storage option in this tutorial will be IndexedDB with a fall back to localStorage.

(Technically, WebSQL is also an option, but it’s support/implementation issues.  In addition, it’s spec has been abandoned by the W3C so it’s off the table.)

The tutorial rolls on!

Happy Coding
-a

Before you commence wailing and gnashing your teeth, add -all_load to the “Other Linker Flags” build setting in your main project. Worked for me.

Thanks are due to Dave DeLong for answering this SO question, because otherwise I’d have probably spent the rest of the day wailing, gnashing my teeth, and ignorantly trying build options until something worked.

Note that the relevant docs make this feel kind of kludgy – but I need software that works today, you know?

An entire freaking year!? Ugh!

You may be asking this question if you’ve started down this road (part1 & part2) with me a *year ago.  Since the tutorial started, Backbone.js has changed and matured:

• The “Controller” is now called the “Router”.
• HTML5 PushState
• New Backbone.sync params
• The handy View.$el
• View.setElement() to explicitly set View.el
• Multiple **random version number updates (0.3s => 0.5s => 0.9s)

…including a bunch of features, improvements, & plugins to extend structure/framework.

I’ve also learned a lot over this past year:

• I’ve figured out better design patterns to use when creating Backbone apps.
• Tutorials stand at a specific point in time.  When things change, I can write new revised tutorials RailsCast style.
• I don’t have to cover every single part of Backbone.js in order to have a good tutorial.

With all of that said, the get ready for some action.  I’m…

1. upgrading the tutorial app to 0.9.2
2. revamping the tutorial app with the new design patterns that I’ve figured out
3. writing the post explaining the new implementation & only worrying about the important details concerning the implementation.

So for those of you who have stuck around this long, I say “Sorry for the lag & the goods are coming”

Happy Coding

* Giant Double Facepalm on how long this tutorial’s gone on.  I just realized that it’s been a year.

** I had a chance to ask Jeremy Ashkenas (@jashkenas; creator of Backbone.js and more) about this when he came to NYCBackbone’s first meetup.  He said that weren’t following any version numbering patterns for anything under 1.0

Before this point, I’ve only watched keynotes like these:

• RailsConf 2011, David Heinemeier Hansson
• RailsConf 2011, Aaron Patterson, “Double Dream Hands: So Intense!”
• The ending of this video is important for later.
• RailsConf 2011: Dr. Nic Williams, “Ruby Through the Ages”
• RailsConf 2011: Ruby Heroes Awards
• RailsConf 2011: Corey Haines
• RailsConf 2010: Yehuda Katz
• RailsConf 2009: What Killed Smalltalk Could Kill Ruby, Too

… and so on & so on.

Sentiments like “I love Ruby”, “Impossible is a relative thing”, “We’re going to make rails faster by reducing the stack”, “We’re all awesome in our own way” are all over these videos.

This year’s first 2 keynotes had a very different tone. For starters, DHH’s talk was about “the skeptical developer” & “progress” and had the following points in it:

• Don’t be afraid of change
• Don’t overreact to change
• Life will change
• Rails will change

…and this point was really hammered home when DHH everyone to read his final slide aloud, mantra-style:

“I will not fear change, I will not fight progress”

I realized the decisions like the asset pipeline & defaulting coffee script but not defaulting haml have been controversial. I can understand where he’s coming from. I also understand some of his responsibilities as the creator of Rails to ensure that people are serious about the platform and where it’s heading.

Though I was excited to see DHH, I was really excited about seeing Aaron Patterson’s keynote. I mean, did you see the “Double Dream Hands” video from last year? This year, Aaron’s talk was way more serious and ended with a slide that simply said:

“We need to be prepared”

I don’t know if you follow Aaron on twitter, @tenderlove btw, but it seemed like a large departure from his standard jovial self. I mean, this is the #fridayhug guy! In December of 2011, the Rails blog stated that the intent for Rails 4 (R4) is a smoother transition than upgrading to Rails 3 (R3). So far things like strong parameters & using patch for updates doesn’t seem that scary, but the RailsConf keynotes would have you believe that something is on the horizon in R4 that might be a litte shocking.

Even if the changes in R4 are large departures, they’ll take us to places we want to (or perhaps need to) be. In addition, the existence of R4 doesn’t undo the great framework that R3 has become. Josh Susser (@joshsusser; one of the @rubyrogues and all around great guy) has a really good & concise slide deck on prudence in which he says:

“I have a limited budget for risk in my projects [&] I hate having other people spend my risk budget for me”

I completely agree with Josh. Fortunately, even though my production budget is super tight, my “personal time” budget has wiggle room. When R4 comes out, I’m not going to flip our production servers to it on the first day. I WILL, however, start using it ASAP in personal projects. I’ll also soak up resources from RailsCasts & Destroy All Software. Basically I’ll do what I’ve always done to stay current. Thousands of other Rails developers will be doing the same. Rails is an exciting and moves fast and we’re all on board.

The second day of RailsConf had an impromptu panel which featured Aaron Patterson, Yehuda Katz(@wycats), Santiago Pastorino(@spastorino), & Jim Weirich(@jimweirich). The serious tone of the keynotes had time to marinate inside me. When Even Phoenix (@evanphx; this years MC) opened the floor for questions, I stepped up to find out what happened to Aaron & the keynote.

When I finally got my chance to ask my question, my fandom started bleeding through my words. I’m a fan of everyone sitting on the panel who make it possible for me to do what I do. I’m a fan of RailsConf & how much I knowledge I’m taking back home with me. Most importantly, I’m a fan of the rails community and how empowered I feel to be a part of it.

If I had the moment to do over again, I’d have reminded Aaron that we’re all here to work together. I’d also toss in a line like “Change isn’t bad news, it’s just new. New things can rock too”. I’ll have to hope my hug said it all.

Incremental development–grow, don’t build, software. I still remember the jolt I felt in 1958 when I first heard a friend talk about building a program, as opposed to writing one. In a flash he broadened my whole view of the software process. The metaphor shift was powerful, and accurate. Today we understand how like other building processes the construction of software is, and we freely use other elements of the metaphor, such as specifications, assembly of components, and scaffolding.

The building metaphor has outlived its usefulness. It is time to change again. If, as I believe, the conceptual structures we construct today are too complicated to be specified accurately in advance, and too complex to be built faultlessly, then we must take a radically different approach.

Let us turn nature and study complexity in living things, instead of just the dead works of man. Here we find constructs whose complexities thrill us with awe. The brain alone is intricate beyond mapping, powerful beyond imitation, rich in diversity, self-protecting, and selfrenewing. The secret is that it is grown, not built.

So it must be with our software-systems. Some years ago Harlan Mills proposed that any software system should be grown by incremental development. That is, the system should first be made to run, even if it does nothing useful except call the proper set of dummy subprograms. Then, bit by bit, it should be fleshed out, with the subprograms in turn being developed–into actions or calls to empty stubs in the level below.

I have seen most dramatic results since I began urging this technique on the project builders in my Software Engineering Laboratory class. Nothing in the past decade has so radically changed my own practice, or its effectiveness. The approach necessitates top-down design, for it is a top-down growing of the software. It allows easy backtracking. It lends itself to early prototypes. Each added function and new provision for more complex data or circumstances grows organically out of what is already there.

The morale effects are startling. Enthusiasm jumps when there is a running system, even a simple one. Efforts redouble when the first picture from a new graphics software system appears on the screen, even if it is only a rectangle. One always has, at every stage in the process, a working system. I find that teams can grow much more complex entities in four months than they can build.

(The above is, of course, excerpted from Brooks’s essay.)

Iterative development that delivers frequent releases of working software, and uses feedback to determine the direction of growth for the next increment? And a report of empirical (if decidedly anecdotal) support for the notion?

Sounds pretty Agile to me.

(The full essay is available in The Mythical Man-Month: Essays on Software Engineering, which I recommend to anyone who writes, builds, grows, or otherwise makes software.)

It’s time to build our Backbone.js app, “Comment on the Movies”.  This tutorial will, ideally, give you something interesting to build.  Take a look at the finished app (so far) on Heroku & feel free to pull it from GitHub.  In addition, I’ve updated the code to use Backbone.js 0.5.3, Underscore 1.2.2, Mustache 0.3 & jQuery 1.7.1.  Let’s do this!

*Before we get started, check out one more resource: Charlie Robbins (Co-Founder of Nodejitsu) wrote an awesome article about Scaling Isomorphic Javascript Code.  In the writeup, he describes popular methodologies in retrieving data and presenting it to the user.  It’s a great read.  The image below from that article:

Backbone.js interactions: Model-View-ViewModel

Prerequisites:  Rotten Tomatoes API Account, Ruby, Sinatra, & Haml

Rotten Tomatoes API Account

The app we’re writing is called “Comment On The Movies”.  The RottenTomatoes API is full of well formed data (which is more than I can say about the New York Times movie api calls).  Real data is always more interesting than building dummy datasets ourselves.

The Rotten Tomatoes API registration is free, quick & painless.  Get your api_key & come back to this tutorial.

We’ll also be using the R.T. assets.  R.T. makes their images available to developers to use in their apps and we’ll be taking advantage.

Ruby (Project Site)

Realistically, if you’re reading this webpage, you probably already have it installed (even if you don’t use it) .  I chose Ruby here because I 9-5 in Ruby and I really enjoy it.  You can use ANY server side scripting language here (PHP, Asp.NET, JSP), but you need to use something. You don’t want to simply expose your R.T. API App Key to the world.  By using a server side language, you can create a proxy page that protects your API App Key.  Here’s how this app will work:

1. Your Backbone app makes an AJAX call with R.T. API parameters to the proxy page
2. The proxy page appends your API_KEY & makes the call to a R.T. servers
3. The proxy page receives the response and returns it to your Backbone app
4. The Backbone app updates it’s UI.

Sinatra (Project Site)

Since we only need an index page and a proxy, Sinatra is a perfect option here.  A full Rails app would be WAY too much.

Haml (Project Site)

I enjoy haml.  It’s really easy to create markup with it.

The “Comment on the Movies” Spec

Here’s what the app needs to accomplish:
**Items in orange will be implemented in future tutorial posts**

• Users should be able to switch between the various lists of R.T. data:
• Box Office (default view on page load)
• In Theaters
• Opening
• Upcoming
• The “Movies Table” should display:
  • Title
  • Year
  • MPAA Rating (R, PG-13, etc)
  • Runtime
  • Release Date (default sort)
  • Critic’s Score
  • If it’s ‘certified fresh’ that should be indicated.
  • Number of comments
• Clicking on a table column title should sort the current set of movies in the table.
• Clicking on a movie’s row should take you to a “detailed” view of the film.  The “detailed” view should display:
  • Links [all links that lead us away from the site should open in a new window] to…
    • Showtimes
    • Awards (if any)
    • Trailers
    • Abridged Cast
    • IMDB records
• Comments
  • Add a new comment
  • Edit an existing comment
  • Delete a comment
  • Require that the review indicates if the comment is BEFORE or AFTER you’ve seen the movie
  • Comments should persists across browser restarts
• The app needs to be crawl-able by Google.
• It needs to be well tested

Make your Backbone App Google Crawl-able via hashbang URLs

Hashbanging (/#!) your URLs let’s the Google crawling bot know that there’s AJAXed content on this page.  Fortunately, it’s really easy in Backbone.js as you can see in the live demo

In The Next Post…

I’ll explain how I implemented the app thus far.  That’ll include the filling the index view & the hashbang technique mentioned above.

(Note: I actually gave a talk on this topic some time back, having implemented this same concept in Ruby using Hadoop Streaming. That implementation contained a number of flaws, some of which I’m attempting to rectify. Also, it’s not as though I’m the first guy to work this angle. I haven’t perused the latest Hadoop/GA work in the field yet, because I wanted to maximize my first-hand learning at this stage – but I’ll be looking to other sources for inspiration as I develop and generalize this code.)

This was just done for my own edification; I haven’t put enough effort into tuning this software or studying its properties for this to be considered a serious work of research (yet). I’m not a specialist or any kind of expert with genetic algorithms, beyond my recreational reading. That said, I had some success at building a gene representation and evolutionary mechanisms that drove a population of solutions for an NP-hard problem toward (mostly) monotonic improvement.

What follows is a brief description of my approach to the problem, with code attached. If you are an expert with GAs, I’d love to hear your feedback on what I could do better.

I haven’t spent any effort in this post describing the Traveling Salesman problem, genetic algorithms, or Hadoop/MapReduce. (It was plenty long already.)  If you’re interested in any of those topics but need to brush up on the basics, check out the preceding links.

More geekery below the fold…

The Map

A “city” in my representation is an (x, y) coordinate pair, with each coordinate having a floating-point value in [0.0, 1.0). The software can take any arrangement of cities, but for testing I used an arrangement with a set of trivial solutions:

Gene Representation

When choosing a non-repeating path among N distinct cities, there are N possible choices for the first city; N-1 choices for the second city, and so on; and only one choice for the final city.

In my representation, a chromosome has N – 1 integer genes. For a gene G_n, valid values are [0, N - n) – thus, the first gene has N valid values, the second gene N – 1, and so on. Each gene corresponds to the 0-based index of a city in an ordered collection of the cities not yet accounted for in the path. For example, if we start with an ordered collection of cities {C₀, C₁, C₂}, and the chromosome {0, 0}:

• The first city in this candidate solution path is at index 0 in the collection:  C₀
• The second city in the path is at index 0 in the collection of remaining cities, {C₁, C₂}: C₁
• The final city is trivially C₂.

Likewise, the chromosome {1, 1} encodes the path C₁ -> C₂ -> C₀.

Fitness Scoring

Here I’ve implemented the symmetric Traveling Salesman problem (the distance/cost from C₁ to C₂ is the same as from C₂ to C₁), and the distance between cities is just the Cartesian distance between their coordinates. The maximum possible distance between any two cities in the landscape I’ve defined is √2, so the maximum/worst-case distance possible for an N-city map is (N – 1) * √2.

A chromosome’s score is this worst-case distance minus the actual distance of the path it encodes; thus, the greater the improvement over the theoretically pessimal case, the higher the chromomsome’s score.

public class ChromosomeScorer {
    protected ArrayList cities;
    protected ArrayList citiesUsed;
    final static double SQRT2 = Math.sqrt(2.0);
    protected double maxDistance;
 
    public ChromosomeScorer(String cityString) {
        cities = getCitiesFromString(cityString);
        citiesUsed = new ArrayList(cities.size());
        maxDistance = SQRT2 * (cities.size() - 1);
        for (int i = 0; i < cities.size(); ++i) {
            citiesUsed.add(Boolean.FALSE);
        }
    }
 
    protected double score(String chromosomeString) {
        ArrayList route = getRouteFromChromosome(chromosomeString);
        double distance = 0.0;
        for (int i = 1; i < cities.size(); ++i) {
            int city1Index = route.get(i);
            int city2Index = route.get(i - 1);
            double[] city1 = cities.get(city1Index);
            double[] city2 = cities.get(city2Index);
            double city1x = city1[0]; double city1y = city1[1];
            double city2x = city2[0]; double city2y = city2[1];
 
            distance += Math.sqrt(((city1x - city2x) * (city1x - city2x)) + ((city1y - city2y) * (city1y - city2y)));
        }
 
        return maxDistance - distance;
    }
 
    // city string parsing code elided
}

The ArrayList is a list of flags indicating whether a city has been added to the route yet; it’s for implementing the chromosome parsing I described above. In my example, I described the process in terms of pulling elements from an ordered collection, and performing further operations on the reduced collection. Regenerating the ordered list of cities and manipulating it in this way every time I wanted to score a chromosome sounded computationally expensive; flipping booleans sounded cheaper. (I could probably get cheaper still using actual bit flags, but I was trying to keep the code both readable and writable at this stage. Optimizations will come later.)

Initial Population

The following code creates an initial population of chromosomes (encoded as Strings) and dumps them to a requested file:

protected static void createInitialPopulation(FSDataOutputStream populationOutfile, final int populationSize, final int numCities) throws IOException {
    for (int i = 0; i < populationSize; ++i) {
        for (int j = 0; j < (numCities - 1); ++j) {
             if (j > 0) {
                populationOutfile.writeBytes(" ");
            }
            populationOutfile.writeBytes(String.format("%d", random.nextInt(numCities - j)));
        }
        populationOutfile.writeBytes("\n");
    }
    populationOutfile.close();
}

At a later time, I’ll experiment with using Hadoop’s ArrayWritable or other serialization schemes for more efficient representation of the chromosomes; at this stage, it was important to me to keep the data human-readable.

There’s a mapper dedicated to scoring generation 0:

public class ScoringMapper extends Mapper {
    private Text outKey = new Text();
    private DoubleWritable outValue = new DoubleWritable();
 
    protected ChromosomeScorer scorer = null;
 
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String incomingValue[] = value.toString().split("\t");
        String chromosome = incomingValue[0];
        double score = 0.0;
        if (incomingValue.length > 1) {
            String scoreString = incomingValue[1];
            try {
                score = Double.parseDouble(scoreString);
            } catch(NumberFormatException nfe) {
                // weird - but let's try re-generating the score
                score = scorer.score(value.toString());
            }
        } else { // we only go through the scoring process if we don't have a map
            score = scorer.score(value.toString());
        }
 
        outValue.set(score);
	outKey.set(chromosome);
	context.write(outKey, outValue);
    }
 
    // setup code elided
}

Subsequent Generations

I managed to cram the rest of the operations for each generation into one mapper and one reducer – really, into one reducer, since the map function just randomly assigns chromosomes to groups:

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        outKey.set(random.nextInt(numBins));
        context.write(outKey, value); // shuffle
    }

So why am I shuffling the chromosomes thus? Most methods of population selection involve having some view over the data; the naïve method of truncation selection is an extreme example, where you need to sort the entire population, and keep some fraction of the highest-scoring individuals. This is hard to parallelize, and not especially scalable for large populations.

Other selection methods such as fitness proportionate selection, however, look like they’d retain their properties well if the total population was divided into large, random subpopulations. (I have yet to do a detailed mathematical treatment of this, but some combination of intuition and cocktail napkin calculation has convinced me of this well enough to forge ahead.) To this end, I’ve set up the system to randomly distribute the full population into a user-settable number of subpopulations; the size of these subpopulations can be tuned to fit within the memory allocated to a reducer instance. (Naturally, the ideal would be to have a subpopulation just large enough to be handled by one reducer instance.) Assuming that the subpopulations are sufficiently large, the random distribution should retain the aggregate characteristics (distribution of fitness scores, &c) of the total population, and give similar results under fitness proportionate selection.

After this subdivision of the population, a reducer instance receives a subpopulation, and is then responsible for selecting survivors from that population, breeding replacement members of the population from the survivors, and scoring the new population members.

    @Override
    protected void reduce(VIntWritable key, Iterable values, Context context) throws IOException, InterruptedException {
        TreeSet sortedChromosomes = getSortedChromosomeSet(values);
        normalizeScores(sortedChromosomes);
 
        int survivorsWanted = (int) ((double) sortedChromosomes.size() * survivorProportion);
        Set survivors = new HashSet(survivorsWanted);
 
        while (survivors.size() < survivorsWanted) {
            survivors.add(selectSurvivor(sortedChromosomes));
        }
 
        ArrayList parentPool = new ArrayList(survivors);
        while (survivors.size() < desiredPopulationSize) {
            survivors.add(makeOffspring(parentPool));
        }
 
        Iterator iter = survivors.iterator();
        while (iter.hasNext()) {
            ScoredChromosome sc = iter.next();
            outKey.set(sc.chromosome);
            outValue.set(sc.score);
            context.write(outKey, outValue);
        }
    }

Selection: The first step in fitness proportionate selection is to sort the new members according to their score, ascending. After that, survivors are selected from the population randomly, weighted by their scores (the higher a chromosome’s score, the more likely it is to survive).

Repopulation: Surviving chromosomes are used to breed back up to our original population numbers. I used single crossover and random mutations (maximum of one mutation per chromosome per generation):

    protected ScoredChromosome makeOffspring(ArrayList parentPool) throws InterruptedException {
        int parent1Index = random.nextInt(parentPool.size());
        int parent2Index = parent1Index;
        while (parent2Index == parent1Index) {
            parent2Index = random.nextInt(parentPool.size());
        }
 
        ScoredChromosome parent1 = parentPool.get(parent1Index);
        ScoredChromosome parent2 = parentPool.get(parent2Index);
 
        ScoredChromosome offspring = crossover(parent1, parent2);
        if (random.nextDouble() < mutationChance) {
            mutate(offspring);
        }
 
        offspring.score = scorer.score(offspring.chromosome);
 
        return offspring;
    }

Scoring: Finally, all unscored population members are scored in preparation for the next round.

This process is repeated for as many generations as is specified at the beginning of the run.  (Later, I plan to allow the number of generations to be unspecified, and have the algorithm terminate when the maximum score starts to settle/converge.)

Parameters

• Number of cities
• Population size
• Proportion of survivors in each generation
• Subpopulation size (see above for rationale for dividing up the population, and thoughts on tuning)
• Chance of random mutation
• Number of generations

Results

I’ve thrown the data for min/max/mean scores for each generation of each run into a Google spreadsheet, and made some simple graphs.

The first run was done with 10,000-member population, 30% survival chance, 500 generations. There’s an unmistakable fitness increase over time, and it gets pretty close to the theoretical maximum for my test city (~23.066).

The second run was done with a 100,000-member population and 50% survival chance, 500 generations. It gets off to a slower start, but there’s a burst of improvement late in the 500-generation run and it gets pretty close to the optimum.

The difference in early-stage performance of the second run had me kicking myself for changing two inputs. I switched back to a 30% survival chance but kept the larger population. This run landed in the same neighborhood as the others, and looked like it still had room to settle.

Interesting to note on all 3 runs is that all start with a few generations of quick improvement, then flatten out, then have an inflection point after which there’s a burst of quick improvement. (Punctuated equilibrium?) All runs seem to still be on the climb; I suspect that a moving average over a few generations and/or some statistical tests for convergence would give a clearer picture of what’s happening with the max trend; the average was still definitely on the rise at the end of all three runs.

Notes

Test coverage could be improved, especially with the use of mocks. I may move to CDH3 and try MRUnit to ease test design. As it stands I’m taking most of the logic out of the mappers and reducers and just using the overridden map() and reduce() functions to coordinate the calls that do the actual work, largely to make it easier to test the logic in isolation.

For the sake of speed, I could probably take out some of the square roots and do my scoring using the squares of the distances.

On my 13″ MacBook Pro with Core Duo processor running 2 mappers and 2 reducers, a population of 10,000 took 45-60 seconds to cull, breed, and score. The time per generation didn’t change appreciably when I went from 10,000 chromomsomes per generation to a population of 100,000 – which means most of my time is still spent in Hadoop overhead, which means that even on my little laptop, I could put up much bigger populations (or much bigger maps) and still run 500 generations in about 6 hours.

I haven’t bothered taking the simulation parameters in via the command line yet; this would be an obvious improvement, and probably the next thing I’ll do. I was a lot more focused on getting the GA code right and checking that it behaved as expected than I was with making a usable tool. I’ve gotten the itch to generalize and grow this thing, though, so I’ll need to start making it usable & configurable & scriptable.

Clearly, this is not a general toolkit for GA, or even GA applied to NP-hard problems. (I don’t like to generalize without at least two cases.) I do see plenty of places where it is generalizable, though, and will starting doing so as soon as I pick a second problem to attack. A long-term goal is to factor out a library/API for doing GA on Hadoop.

Genetic algorithms are not one single thing – there are multiple strategies for culling populations, doing crossover/breeding, etc. A longer-term design goal is to make each of those variable pieces of the algorithm into pluggable strategies selectable from the command line.

There’s optimization to do all over the place, e.g. with the multiple copies of data kept in the reducer. I chose the data structures I used for ease of implementing the algorithm – e.g., picking a Set to hold survivors so as to avoid picking up duplicate survivors – with the intent to get working software quickly, and optimize last.

The Code

The code, such as it is, is in a public repository on GitHub. More to come.

Jeff Atwood once blogged: “I don’t think it matters how you conduct the postmortem, as long as you do it“.  To that end, here’s my post mortem of the app I didn’t finish.  Ideally  other developers will learn from my successes and mistakes.  :)

The App/Idea: A web version of my iphone app ( Touch Playbook ) called Playbook Nodes.  I learned a lot creating TPB and I figured I’d learn a lot porting the code as well.  I emailed the contest organizers early to make sure I wasn’t going to violate the following rule:

“Of course, feel free to work on the concept for your application before the competition starts: pretty hand-drawn pictures on napkins, angry email threads between team members on two-space or four-space indentation, schema designs, finite state machines.  Just no digital assets including graphic design assets, code, etc.”

~Node Knockout Rules

They gave me the green light, so I starting researching the parts of the app I’d need to understand.  Looking back, here were my mistakes:

Mistake #1: Doing too much
Heading into the competition, my goal was to recreate the entire app.  That includes the user interactions (dragging players, passing the ball, shooting, etc), full replay animation logic, & state management.  On top of that, I wanted to add new features like user accounts and the ability to share pays.  Considering that it took me 5 months to build TPB (1 month to spec / 3 months to implement / 1 month of user testing ), this order was WAY too tall.  In hindsight, I should have simply picked another project.  Anything TPB inspired that’d be worthwhile would simply take too long.

Mistake #2: Assuming that things will simply fit together
I assumed that I’d simply figure out how to elegantly draw paths as the user dragged players (and animate the process later) despite the fact that I couldn’t find good examples of animating the drawing of paths and I’d never done it before using Raphael.  I also assumed Mongoose-Auth would be easily to implement and…well..would just work.  Both of these assumptions cost me precious time.

Mistake #3: Oversleeping
I was hoping to get about 5-6 hours of sleep each night.  Instead I got 7-8 each night.  I felt great, but I wish I had the extra hours to code.

Mistake #4: Failing to achieve a “thin line through” in the first 24 hours
This might be the most telling sign of failure in a 48 hour competition.  It’s really important to get a barebones version of your app working before you continue working on the features in your spec.  The 24 span INCLUDES server setup.

It wasn’t all bad though.  Here are some things I did right:

Success #1: Spec’d as much as I could BEFORE the competition
Mockingbird was super handy way to mock views & my trusty notebook saved my thoughts in the weeks leading to the competition.  Also, I was able to pull my notes from the creation of TPB which helped a bunch.

Success #2: Improved my overall JavaScript skills & got comfortable writing node JavaScript before the competition
I’m new to Node.js, but I’m not new to web development.  I have to shamefully admit that I used to think of JavaScript as a sub-language whose only use were silly DOM tricks.  Fortunately, I started following the works of Douglas Crockford and saw the light.  Not too long after that, I start hearing about NodeJS & I started soaking up examples around the first week in August.  I turned my knowledge harvesting into overdrive & gained the confidence I needed to enter the competition .  Here are some of the things that I did:

• Watched 7.5 hours of Douglas Crockford’s presentations
• Also kept a copy of “Javascript: The Good Parts” close by at all times
• Attending a free training by the co-founders of Nodejitsu, (@hij1nx & @indexzero)
• Watching as many NodeTuts as I possible could and coding the examples in the vids.
• Felix’s Node.js Guide
• Tons of links to helpful content & people to follow (twitter, blogs, etc).
• Watching videos by Ryan Dahl, the creator of Node.js

Success #3: I went with Heroku instead of Linode or Joyent
I didn’t want to lose time configuring a server.  The purpose of the competition is the app that I create, not testing my Sys-Admin skills.  Going with Heroku saved me the headache that many of my fellow participants faced.  Some teams never got their server setup, but I was deploying early and easily.

Success #4: Logging my actions
Every hour, on the hour, I wrote a summary of how I spent my time.  If I started writing a bunch of code, I’d shorten my logging to 30 minute intervals.  This was helpful because it drove me to get small victories at least once an hour.

Success #5: In Backbone.js apps, add a global logging function
I started doing this in every Backbone app that I create.  I add 3 simple methods to my controller/router definition.  This gives me a name-spaced way to log & benchmark events from my controllers, models & views.  It also gives me a clean way to disable logging when I deploy to production & an easy way to toggle the logging once the code is deployed to prod.

• debugEnv()
• The router/controller has a variable called ‘debug’ which is set to true/false.
• The first line of the method looks like this:
  return (this.debug && (typeof console !== ‘undefined’)); 
• logEvent(message)
• This method returns if the controller/router’s debug method is off
  if(! this.debugEnv() ){ return; }  
• Otherwise:
  console.log( (new Date()).toTimeString() + “: ” + message); 
• benchEvent(handle, operand_string)
• This method needs to use console.time() & console.timeEnd() so it makes sure that we can debug AND that the ‘time’ method exists (This is a problem on IE):
  if(! this.debugEnv() || typeof console.time === ‘undefined’ ){ return; }
• Otherwise, depending on the ‘operand_string’, …
  case ‘+’ : console.time(handle); break;
  case ‘-’: console.timeEnd(handle); break;

This makes it really easy to track what my app is doing and provides simple benchmarking.  I find it super helpful! 

Success #6: Entering Node Knockout 2011
The biggest success was taking part in the competition.   I learned so much more about ExpressJS, Jade, Stylus, Mongoose, MongoDB, CSS3 Gradients, and more because I challenged myself.  Was I ready for every single challenge I faced? Absolutely not!  However, failing with focused effort is better than never trying at all.

In Conclusion
I look forward to Node Knockout 2012.  I’m probably going to start spec’ing even earlier and I’ll be more realistic about what can be accomplished in 48 hours.  I also intend to complete PlaybookNodes.  While I didn’t get everything done in 48 hours, I put together a LOT of plumbing and I feel that I’m potentially only a few days away from a finished product.  I hope my experiences help you plan your next 48 coding session better than I did.

Oh, and now that I’m done with NKO2011, Part 2 of the Backbone.JS tutorial is on it’s way!

Till next time, Happy Coding!

Gotcha 1 – The .gems file:

The ‘.gems’ file is SUPER important.  Without it, Heroku won’t be able to install the gems that your app needs to function properly ( like…say…Sinatra ).  It’s mentioned as a single line in the linked page above, but it’s worth mentioning because it’s easy to miss.

Gotcha 2 – Heroku’s read-only Filesystem:

There are certain files that you shouldn’t put into source control, like an API key from the New York Times (hint, hint).  Historically, I’ve added files like these to the server directly (along with other sensitive keys/data).  While Heroku’s read-only file system doesn’t allow this, it does have a nice solution:  Define sensitive values via the Command Line Interface:

heroku config:add API_KEY=’ABC123′

… and reference them in code like this:

my_key = ENV['API_KEY']

Those are the two VERY simple gotchas I ran into which shouldn’t distract from how awesome Heroku is.  Create your free account and play with it.

In other news, Part 2 of the Backbone.js tutorial series is coming soon.  In fact, the site’s layout was changed to accomodate the code and text for the tutorial.  You can watch the unfinished product evolve on GitHub & play with it on Heroku.

Happy Coding!