I wondered how logistic regression, support vector machines or even clustering algorithms would help in classifying a given file. As I dived into the data I realized that the descriptive statistics on tokens, and even ascii faces, would be nearly as interesting as their predictive power. Thus, this post will summarize the descriptive statistics, while in my next post I’ll cover using tokens to predict a file’s programming language.

Methods

Fetching The Code

Using Github’s API I fetched the 10 most popular repositories for 10 languages (c, haskell, go, javascript, java, lua, objective-c, ruby, python and php). Those 10 were chosen for their popularity, differing paradigms (e.g. Haskell vs Java), differing syntax (Haskell vs go) and overlapping syntaxes (C, JavaScript and Java). After retrieving a list of 100 repositories I downloaded the zip ball for each repo.

Tokenizing The Code

First, a list of common programming tokens (eg: ; , . ( ), etc) was created - tokens that would be found in many of the 10 languages of interest. Using those tokens I created a tokenizer that would output an object with the keys being tokens and their values being the number of times that the token occurred in the file. Base 10 numbers, hex numbers, strings (double quotes) and ‘characters’ (single quotes) were treated as 4 different token types. This was done so the number 8 was not treated as a different token from the number 44 (they are both tokenized as “numbers”).

Each of the 100 repositories was traversed and non binary files were tokenized, with each file’s total token count stored in Redis’ sorted set data structure (sorted by number of occurrences). Using a sorted set made it trivial to retrieve the 1000 most common tokens from all 100 repositories. Each file was then re-tokenize, but only tokens that were present in the 1,000 most common token list were counted. A data set was created that included information on 65,804 files from 100 different repos. Along with the token data, the following data was also recorded: 1) the file’s extension, 2) it’s path within the repository, 3) it’s shebang, if present and 4) the token count for the first 250 most common tokens (I decided to limit my analysis on a smaller number of tokens for the first round of analysis). Finally, the counts for each token were converted to the ratio that that token occurred relative to the total occurrences. of all tokens (e.g. # of periods/total number of all tokens). The absolute number of tokens per file would be skewed by large files, which have more tokens.

Counting Smiley Faces

Two ease analysis I focused on c and JavaScript files (in c and JavaScript repositories): both languages have identical single and multi-line comment syntax. First, text from comments was separated from code. Second, the number of times an ascii face appeared in a given file’s comments was counted. The following “faces”: :( :) :-) :-D :p ;) ;-) were counted.

Statistics

All statistics were carried out using R. Welch Two Sample t-tests were used to compare groups.

Results

Tokens

While the top 1,000 tokens, by occurrence were recorded, only the top 20 are presented in Table 1 (see here for the top 1,000). Not surprisingly numbers are the most prevalent token, with commas coming a very close second. Interestingly, and what sparked my interest in ascii smiley faces, is that there are slightly more right parentheses than left. While the contents of strings were ignored, the contents of comments were not. Seeing as most (all?) of the analyzed languages require parenthesis to be balanced I presumed that the imbalance might be caused by ascii smiley faces in comments.

Interestingly, hexidecimal numbers were the 6th most common token despite rarely being used outside of c. While hex numbers are used extensively in CSS, I only classified numbers that started with ‘x’ as being hex, which precluded the hex numbers in CSS from being included.

Table 2 shows the top 20 tokens and their ratios (token/all tokens in a given file) in 16 different file types. Not surprisingly JSON files lead the pack for double quoted strings, curly brackets, colons and commas. Likewise, Clojure leads by having the highest proportion of parentheses. The right arrow -> occurred most often in Php, C and Haskell. Finally, square brackets were very prevalent in Objective-C.

ASCII Faces

To examine the discrepancy between left and right parenthesis I created a set of scripts to separate comments from code, in c and JavaScript files. Second, I then analyzed the comments and counted the number of ascii faces that appeared. I focused on 6 different types of smiley faces and I included 1 type of frown (See Table 3 for types of ascii faces and amount found).

While there were more frowns in JavaScript files, the difference wasn’t statistically significant. Furthermore there was not a statistically significant difference between the total smiley faces between ‘.c’ and ‘.js’ files. However, there were more smiley faces in files that were in “JavaScript” github repos. For instance, Node is a JavaScript repo but includes both JavaScript and c files. This makes sense that the project, with its distinct maintainers, rules and conventions is more important in determining the number of smiley faces.

Eighty percent of c files analyzed were found in the Linux repository, so it made sense to focus on Linux specifically. In Linux c comments I found 631 smiley faces and 73 frowns. In linux the most prevalent smiley faces was `:-)` followed by `:)` (See Table 3.).

Discussion

Shortcomings

Despite including 100 different repositories, Linux source files represented 30% of all files in this analysis. Ideally, the number of files from each repository, and language, would be balanced. One approach would be to randomly select a numerically identical subset of files from each language. While this approach might be valid statistically it wouldn’t produce descriptive statistics on each repository, just a subset of files within each repository. Alternatively Linux could be excluded from the analysis since the number of files it contains is an outlier, relative to the other repositories.

In all files there were 21,474 more right parenthesis than left. Given that c and JavaScript files represent nearly half of all files in this analysis, and they only had 1,901 smiley faces, it’s unlikely that the other half of the files had a nearly 20,000 smiley faces - or enough to account for the left and right parenthesis. Future analysis could attempt to locate the source for this difference (presumably within comments).

Conclusions

It is not surprising that ratios of token types can differ dramatically between different languages, however, I was surprised that several tokens (parenthesis, square & curly brackets) did not occur equally. While smiley faces can account for part of this discrepancy, they most likely do not account for all of it.

The biggest surprise was that number of smiley faces per file was not statistically different between JavaScript and C. Being low level I presumed that C code would be more serious, with fewer ascii faces. Interestingly, I was wrong and C code has a similar amount of smiley faces relative to JavaScript.

Viewing Table 2 we can start to see some patterns and differences in token ratios that might help to predict a file’s language. For instance, JSON has very different token ratios than C. In the next article I will explore the power that tokens have in predicting which programming language is being used in a given file.

If you’re interested in replicating the analysis, or obtaining the dataset, please see the links below:

Language Statistics

Language Statistics Data

Starting with AngualrJS 1.2, the ngView directive, and the $route service were moved into a separate module called ngRoute. As a result, to use routes and ngView your app must explicitly declare it as a dependency. Furthermore, the syntax for the “otherwise” statement is slightly different from older versions of AngularJS. Below is a complete example of a tiny AngularJS 1.2 app with two routes and a default:

Undocumented features of ngView

In the process of understanding the code behind ngView I came across two undocumented attributes: “onload” and “autoscroll”. The onload attribute can take any Angular expression and will execute it every time the view changes. Autoscroll uses the $autoScroll service and scrolls to a specific element based on the current value of $location.hash(). Finally, near the very end of the directive, after link(currentScope) is called the '$viewContentLoaded' is emitted within the current scope - an event which you can use within your controller. Below is a revised version of the above example that includes the onload attribute.

How ngView Works?

In order to understand ngView, I think it’s useful to create a simplified version of it. Below is ngViewLite, a version of ngView that does not include scope cleanup or animations, but is otherwise identical to ngView.

First, we bind a function update to the $routeChangeSuccess; when the route changes, update will be called. Right after attaching the function to the event we immediately call update() to load the initial contents into the view.

The update function checks if there is a defined template for the current route, if so it proceeds by calling the linker function, passing in a new scope, and a callback function. The callback function’s only parameter is the cloned element, whose html will be replaced with the route’s current template. The cloned element is then appended to div with the ng-view-lite attribute. Afterwhich we remove the previous contents from the view.

Finally, the template must be compiled ($compile(clone.contents())) and a new scope is injected into it (link(newScope)). In between those two steps we check if the current route has an associated controller: if so we instantiate the controller with the newScope and the local variables from the current route.

Making an ngMultiView

ngView works well, but what if you want multiple views to change according the url? According to the documentation ngView can only be used once within an application. To accomplish our ngMultiView we’ll slightly modify ngView and create an Angular value (MultiViewPaths) to hold the mapping between urls, views, controllers and templates.

In ngMultiView, a parameter is passed into the directive <div ng-multi-view="secondaryContent"></div>, in the directive this attribute will be called “panel”. Instead of binding to the '$routeChangeSuccess' event, we’ll bind to '$locationChangeSuccess' to make our directive completely independent of ngRoute. ngMultiView will work the following way:

1. A url change will trigger '$locationChangeSuccess', which in turn will call update()
2. Within update: grab the portion of the URL after the hash (in the code this portion is just called url).
3. Using the url variable, and the panel, we can lookup the corresponding controller and template from the MultiViewPaths value.
4. Once we have the controller and template, ngMultiView works almost identically to ngView.

Our ngMultiView is very basic, it doesn’t take into account parameters being passed through urls, nor does it deal with scope cleanup, or animations. If you need more functionality I’d recommend starting with the $routes service and modifying it to accommodate multiple views.

Conclusion

Creating custom directives can be intimidating at first. There’s a lot of jargon to overcome, and many little nuances. However, once those are overcome it becomes relatively easy to alter existing directives or create your own.

Promises can be used to unnest asynchronous functions and allows one to chain multiple functions together - increasing readability and making individual functions, within the chain, more reusable.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

function fetchData(id, cb){
  getDataFromServer(id, function(err, result){
    if(err){
      cb(err, null);
    }else{
      transformData(result, function(err, transformedResult){
        if(err){
          cb(err, null);
        }else{
          saveToIndexDB(result, function(err, savedData){
            cb(err, savedData);
          });
        }
      });
    }
  });
}

Once getDataFromServer(), transformData() and saveToIndexDB() are converted to returning promises we can refactor the above code to:

1
2
3
4
5

function fetchData(id){
  return getDataFromServer(id)
          .then(transformData)
          .then(saveToIndexDB);
}

In addition to increasing readability promises can help with error handling, progress updates, and AngularJS templates.

Handling Errors

If fetchData is called and an exception is raised in transformData() or saveToIndexDB(), it will trigger the final error callback.

1
2
3
4
5
6
7

fetchData(1)
  .then(function(result){

  }, function(error){
    // exceptions in transformData, or saveToIndexDB
    // will result in this error callback being called.
  });

Unfortunately, if an exception is raised in getDataFromServer() it will not trigger the final error callback. This happens because transformData() and saveToIndexDB() are called within the context of .then(), which uses try-catch, and automatically calls .reject() on an exception. To bring this behaviour to the first function we can introduce a try-catch block like:

1
2
3
4
5
6
7
8
9
10
11
12

function getDataFromServer(id){
  var deferred = $q.defer();

  try{
    // asynchronous function, which calls
    // deferred.resolve() on sucess
  }catch(e){
    deferred.reject(e);
  }

  return deferred.promise;
}

While adding try-catch made getDataFromServer() less elegant, it makes it more robust and easier to use as the first in a chain of promises.

Using Notify for Progress Updates

A promise can only be resolved, or rejected, once. To provide progress updates, which may happen zero or more times, a promise also includes a notify callback (introduced in AngularJS 1.2+). Notify could be used to provide incremental progress updates on a long running asynchronous task. Below is an example of a long running function, processLotsOfData(), that uses notify to provide progress updates.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

function processLotsOfData(data){
  var output = [],
      deferred = $q.defer(),
      percentComplete = 0;

  for(var i = 0; i < data.length; i++){
    output.push(processDataItem(data[i]));
    percentComplete = (i+1)/data.length * 100;
    deferred.notify(percentComplete);
  }

  deferred.resolve(output);

  return deferred.promise;
};


processLotsOfData(data)
  .then(function(result){
    // success
  }, function(error){
    // error
  }, function(percentComplete){
    $scope.progress = percentComplete;
  });

Using the notify function, we can make many updates to the $scope’s progress variable before processLotsOfData is resolved (finished), making notify ideal for progress bars.

Unfortunately, using notify in a chain or promises is cumbersome since calls to notify do not bubble up. Every function in the chain would have to manually bubble up notifications, making code a little more difficult to read.

Templates

AngularJS templates understand promises and delays their rendering until they’re resolved, or rejected. AngularJS templates no longer resolve promises - they must be resolved in the controller before they’re assigned to the scope. For instance let’s say our template looks like:

1

<p>{{bio}}</p>

We could do the following in our controller:

1
2
3
4
5
6
7
8
9

function getBio(){
  var deferred = $q.defer();
  // async call, resolved after ajax request completes
  return deferred.promise;
};

getBio().then(function(bio){
  $scope.bio = bio;
});

The view renders normally, and when the promise is resolved AngularJS automatically updates the view to include the value resolved in getBio.

Limitations of Promises in AngularJS

When a promise is resolved asynchronously, “in a future turn of the event loop”, the .resolve() function must be wrapped in a promise. In the contrived example below, a user would click a button triggering goodbye(), which should update the $scope’s greeting attribute.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

app.controller('AppCtrl',
[   '$scope',
    '$q',
    function AppCtrl($scope, $q){
      $scope.greeting = "hello";

       var updateGreeting = function(message){
          var deferred = $q.defer();

          setTimeout(function(){
              deferred.resolve(message);
          }, 5);

          return deferred.promise;
       };
      $scope.goodbye = function(){
          $scope.greeting = updateGreeting('goodbye');
      }
    }
]);

Unfortunately, it doesn’t work as expected, since the asynchronous event works outside of AngularJS’ event loop. The fix for this (besides using AngularJS’ setTimemout function), is to wrap the deferred’s resolve in $scope.$apply to trigger the digest cycle and update the $scope accordingly:

1
2
3
4
5

setTimeout(function(){
  $scope.$apply(function(){
    deferred.resolve(message);
  });
}, 5)

Jim Hoskins goes into more detail on using $apply: http://jimhoskins.com/2012/12/17/angularjs-and-apply.html

Conclusions

Using promises is an important part of writing an AngularJS app idiomatically and should help make your code more readable. Understanding their shortcomings, and their strengths make them much easier to work with.

While I enjoyed developing in JavaScript, I always felt I was missing something - that is until I started developing with AngularJS. AngularJS uses both dependency injection and promises extensively, both of which have greatly improved my code. In this article, I’ll focus on dependency injection, and discuss promises in my next article.

Dependency Injection

“Dependency injection is also a core to AngularJS. This means that any component which does not fit your needs can easily be replaced.” - angularjs.org

AngularJS doesn’t just pay lip service to dependency injection, it’s a design pattern that it uses extensively, and builders of AngularJS apps use as well. Wikipedia defines dependency injection thusly:

“Dependency injection is a software design pattern that allows the removal of hard-coded dependencies and makes it possible to change them, whether at run-time or compile-time.”

So, how has dependency injection (DI) improved my Node.js apps? Traditionally I might write a task queue like so:

1
2
3
4
5
6
7
8
9

var db = require('./database.js');

module.exports = {
  start : function(input){
    // makeThumbnail
    // save timestamp
    db.save({...});
  }
};

1
2
3
4
5
6
7
8
9

var db = require('./database.js');

module.exports = {
  start : function(input){
    // upload thumb
    // perhaps save bucket name
    db.save({...});
  }
};

Using dependency injection I’d change that to this:

1
2
3
4
5
6
7

module.exports = {
  start : function(input, db){
    // makeThumbnail
    // save timestamp
    db.save({...});
  }
};

1
2
3
4
5
6
7

module.exports = {
  start : function(input, db){
    // upload thumb
    // perhaps save bucket name
    db.save({...});
  }
};

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

var db = require('./database.js'),
    makeThumbnail = require('./tasks/makeThumbnail'),
    uploadToS3 = require('./tasks/uploadToS3'),
    taskToRun = process.argv[2],
    taskRunner;

taskRunner = function(task){
  task.start(process.argv, db);
};

if(taskToRun === 'uploadToS3'){
  taskRunner(uploadToS3);
}else{
  taskRunner(makeThumbnail);
}

While the DI example above requires more code than the original, it makes testing easier and I’d argue better. It becomes trivial to test each unit in isolation of other units of code. With the first approach, each unit test would require database calls. With the second approach, we can inject a mock database object like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

var makeThumbnail = require('makeThumbnail');

describe('Make Thumbnail', function(){
  var database = {};

  it('should make a thumbnail, and call db.save', function(done){
    var input = {
      imageId : 1
    };

    database.save = function(obj){
      assert.equal(obj.id, input.imageId);
      done();
    }

    makeThumbnail(input, database);
  });
});

This speeds testing up significantly and ensures that if the unit tests fails it’s not failing because of issues with the database code. Ultimately, this makes localizing bugs much quicker. In other words we can test just the creation of thumbnails, and not our database (which we’d do separately).

DI forces one to think more rigorously about how code will be divided into modules, and what modules will be injected into other modules. This requires more diligence upfront, but leads to greater flexibility down the line. For instance, the database object is only being required() and injected in a single spot in the code, making it much easier to swap the database from say MySQL to Postgresql.

Why not use use require?

On a post detailing the magic behind AngularJS’ DI, tjholowaychuk (of Express.js, Jade and Mocha fame) asks: “why not just require() what you want? seems odd to me”?

Despite asking 6 months ago, no one has replied, leaving readers pondering why. As the example above shows, requiring dependencies at the top of each file makes mocking more difficult. One could write a wrapper around each dependency, and serve it normally for development/production and serve the mocked version for test ENV, but at that point why not consider DI?

Conclusion

As learning new programming languages makes us better developers so does learning new frameworks. Learning a new framework helps us learn, and reinforce our knowledge of design patterns. Qes, on programmers.stackexchange.com, sums up his experiences with DI:

Extra Reading

1. Dependency Injection (Wikipedia)
2. How DI In AngularJS Works
3. AngularJS Best Practices [questions the necessity of DI]

Miško Hevery, the creator of AngularJS, gave a talk about directives and explained that compile is rarely needed for custom directives, and it is only required for directives like ng-repeat and ng-view. So the next question: how does ng-repeat work?

How does ng-repeat work?

In my quest to understand the compile function, I started examining ng-repeat. Reading the source code was difficult until I walked through an example using the Chrome debugger. After stepping through ng-repeat it became clear that most of its 150 lines of code are related to optimizing, error handling and handling objects or arrays. In order to really understand ng-repeat, and specifically compile, I set out to implement my own version of ng-repeat, which I will call lk-repeat, with just the bare minimum of code. When possible I tried to use the same variable names that ng-repeat uses, and I also used their regular expression for matching passed in attributes.

Transclusion

Before going further it’s important to review the transclude option. Transclude has two options: 1) true or 2) 'element'. First let’s examine transclude : true.

1

<div person>Ted</div>

1
2
3
4
5
6
7
8
9

app.directive('person', function(){
  return {
    transclude : true,
    template: '<h1>A Person</h1><div ng-transclude></div>',
    link : function($scope, $element, $attr){
      // some code
    }
  }
});

1

<h1>A Person</h1><div ng-transclude><span class="ng-scope">Ted</span></div>

In the above example transclude : true tells Angular to take the contents of the DOM element, using this directive, and insert them into the person’s template. To specify where in the template the HTML will be transcluded include ng-transclude in the template. The span, with class ng-scope is inserted by AngularJS.

In contrast to the above example, ng-repeat, does not have a template, and transcludes the element that calls ng-repeat. Hence, ng-repeat calls transclude : 'element', to denote that the DOM element that called ng-repeat will be used for transclusion.

lk-repeat

Below lk-repeat is used the same way ng-repeat would be used.

1
2
3

<ul>
  <li lk-repeat="name in names">{{name}}</li>
</ul>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

var app = angular.module('myApp',[]);

app.directive('lkRepeat', function(){
  return {
    transclude : 'element',
    compile : function(element, attr, linker){
      return function($scope, $element, $attr){
        var myLoop = $attr.lkRepeat,
            match = myLoop.match(/^\s*(.+)\s+in\s+(.*?)\s*(\s+track\s+by\s+(.+)\s*)?$/),
            indexString = match[1],
            collectionString = match[2],
            parent = $element.parent(),
            elements = [];

        // $watchCollection is called everytime the collection is modified
        $scope.$watchCollection(collectionString, function(collection){
          var i, block, childScope;

          // check if elements have already been rendered
          if(elements.length > 0){
            // if so remove them from DOM, and destroy their scope
            for (i = 0; i < elements.length; i++) {
              elements[i].el.remove();
              elements[i].scope.$destroy();
            };
            elements = [];
          }

          for (i = 0; i < collection.length; i++) {
            // create a new scope for every element in the collection.
            childScope = $scope.$new();
            // pass the current element of the collection into that scope
            childScope[indexString] = collection[i];

            linker(childScope, function(clone){
              // clone the transcluded element, passing in the new scope.
              parent.append(clone); // add to DOM
              block = {};
              block.el = clone;
              block.scope = childScope;
              elements.push(block);
            });
          };
        });
      }
    }
  }
});

Above you’ll note that I’m removing all the elements from the DOM, and their scope, every time the collection updates. While this makes the code easier to understand, it is extremely inefficient having to remove everything then add it again. In the real version of ng-repeat, only elements that are removed from the collection, are removed from the DOM. Furthermore, if an item moves within the collection (e.g. 2nd to 4th place) it doesn’t need a new scope, but it needs to be moved in the DOM. Reading ng-repeat’s code gives me confidence that the team behind AngularJS has created a good, well tested and efficient framework.

In part 2 I examine ngView, it’s implementation, hidden features and creating your own ngMultiView.

Between the overview below, and the code on GitHub, you should be able to follow along and, in less than 50 lines of code, add real-time updates to your Rails app.

Adding Real-Time on Github

Starting Point

In a traditional web app if a user creates a new model other users must refresh their page to see that content. Alternatively, you could poll the server every 30 second and refetch all the content. With both approaches you end up fetching all the content, and in the first case the markup as well.

In Figure 1, User 1 creates a new book, but User 2 will not see that new book unless they refresh their page.

Adding Real-Time With Redis And Socket.IO

When User 1 creates a new book, we’d like that new book to be pushed to User 2 in real-time. I’m going to cover one method that requires only a few modifications to your existing app and uses Redis, Node and Socket.IO.

How It Will Work

1. When User 1 creates a new book, an “after_create” callback publishes that new book to Redis on the “rt-change” channel.
2. On the Node server, each client subscribing to “rt-change” receives that new book.
3. The new book is pushed to the client using Socket.IO.
4. Within the browser, Socket.IO receives that new book and “publishes” that change to our Backbone.js App.
5. The Backbone.js books collection, listening for changes to books, adds the new book to itself.

The advantage of this approach is that it only requires tiny modifications to a Rails’ model, and if your Node server crashes, your application will work as it always has (without real-time). Thus, I’d consider this a real-time enhancement that gracefully degrades to a conventional Rails RESTful web app.

Socket.IO Connection

First, ensure that socket.io.js has been added to lib/assets/javascripts, and referenced in app/assets/javascripts/application.js. In the web app create a new module, called realtime, that includes the Socket.IO connection code. When the application initializes it calls app.realtime.connect() to setup the Socket.IO connection.

1
2
3
4
5
6
7
8
9
10

window.app.realtime = {
  connect : function(){
    window.app.socket = io.connect("http://0.0.0.0:5001");

    window.app.socket.on("rt-change", function(message){
      // publish the change on the client side, the channel == the resource
      window.app.trigger(message.resource, message);
    });
  }
}

Node Server & Pub/Sub

In the root of the Rails app create a new folder called ‘realtime’, where the Node server will reside. Don’t forget to create a package.json file and include socket.io, and redis in the dependencies. Finally, remember to run npm install.

1
2
3
4
5
6
7
8
9
10

var io = require('socket.io').listen(5001),
    redis = require('redis').createClient();

redis.subscribe('rt-change');

io.on('connection', function(socket){
  redis.on('message', function(channel, message){
    socket.emit('rt-change', JSON.parse(message));
  });
});

Rails Models

Assuming you have Redis installed, add redis to your Gemfile. Next, create a file called redis.rb in your initializers with the following content:

1
2

#make sure redis has been added to your Gemfile
$redis = Redis.new(:host => 'localhost', :port=> 6379)

The Rails app now has access to Redis through the $redis global variable. Below, we publish changes to Redis whenever a model is created, updated or destroyed. Changes are published to “rt-change”, which our Node.js connections are listening to (see above).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

class Book < ActiveRecord::Base
  attr_accessible :num_pages, :title
  after_create {|book| book.message 'create' }
  after_update {|book| book.message 'update' }
  after_destroy {|book| book.message 'destroy' }

  def message action
    msg = { resource: 'books',
            action: action,
            id: self.id,
            obj: self }

    $redis.publish 'rt-change', msg.to_json
  end
end

Listen For Changes in The Backbone App

In the Books Collection, we add the code to both listen for ‘books’ events and the handler to handle those events. For create, we simply add the new object (obj) to the collection. For update we update the existing model, while for destroy we remove the object from the collection.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

app.collections.Books = Backbone.Collection.extend({
  model : app.models.Book,
  url : '/books',

  initialize: function(){
    app.on('books', this.handle_change, this);
  },

  handle_change : function(message){
    var model;

    switch(message.action){
      case 'create':
        this.add(message.obj);
        break;
      case 'update':
        model = this.get(message.id);
        model.set(message.obj);
        break;
      case 'destroy':
        this.remove(message.obj);
    }
  }
});

Caveats

In production there are many edge cases to consider. For instance, if someone views your app on their mobile phone and then puts the phone in their pocket, the screen saver goes on and Socket.IO will disconnect. When the user takes the phone out of their pocket, and views the app, Socket.IO will reconnect. However, during the period of disconnection the data in the client-side app may have become out-of-date. An easy fix is just to fetch the data on reconnect. With lots of connections, or lots of data, fetching everything becomes problematic and requires a more clever method for fetching data (e.g. just fetch the new, or changed, data).

Another issue is if two people are editing the same item, and if person 1 clicks save that will replace what person 2 is editing. To solve this you can present person 2 with a message saying that the book they are editing has been updated by someone else and prevent the version of the book they are editing from being replaced. This isn’t an ideal solution, but would be fine if the chances of two people editing the same model were minimal.

In the code above there is only one channel ‘rt-change’, meaning every connected client will get every real-time change. You may want to scope your channels by user (e.g. rt-change/[USERID]). Furthermore, you’d want to create one redis client for every Socket.IO connection (currently there’s one redis client for all connections). In other words the .createClient(), and redis.subscribe('...'), would have to take place within the Socket.IO ‘connection’ callback (after line 6 above).

Alternatives To The Above

[update July 2014]: Realtime Rails Gem

Mike Atlas created a realtime gem which does all the above with minimal setup.

SockJS

Socket.IO could be swapped for SockJS, which uses a similar API to websockets. I’ve heard from several individuals that it’s significantly more stable than the current version of Socket.IO and it’s currently used by Meteor.

Engine.IO

Guillermo Rauch, the creator of Socket.IO, has publically stated that Socket.IO’s approach of starting with websockets and falling back to polling creates issues. As result, he’s been working on Engine.IO, which will power Socket.IO version 1.0, and should provide a much more stable experience. I suspect Socket.IO, v1.0, will be released in the next few months.

Rails 4.0

Rails 4.0, which is due to be released soon, will include streaming. Using a combination of Rails 4 streaming, and Puma, you could potentially remove Node and Socket.IO, and use Rails for real-time. Of course, you’d have to take care of some of what Socket.IO does such as reconnects and heart-beats.

RabbitMQ/ZeroMQ

Redis’ Pub/Sub functionality could be replaced by either RabbitMQ or ZeroMQ. I ended up using Redis, since I was using it for caching, and it has an extremely simple API for pub/sub. While RabbitMQ and ZeroMQ appear more complex, they do offer many more features for messaging.

Commercial Options

If you’re not keen on tinkering with Node, or waiting for Rails 4, there are commercial options such as Pusher and PubNub, that deal with real-time connections for you. While both options can be pricey, especially with many concurrent connections, they do save you the hassle of building the infrastructure yourself.

Conclusions

Adding real-time updates to your Ruby on Rails RESTful app has never been easier. Over the next few months Rails 4, or Socket.IO v1.0, will make the process even more painless. As Google’s services make users more accustomed to real-time updates, it becomes even more important to provide a similar experience in your webapps.

Adding Real-Time on Github

It would be a mistake to hunker down and focus singularly on hammering out new features. Now that you’ve launched you have significantly more people visiting your homepage, creating accounts and using your app. You have much more data to help you prioritize what to do next.

I’ll focus on three important metrics that are extremely important and especially so after launching. These metrics are all conversion metrics and include 1) vistors creating an account, 2) becoming an active user and 3) becoming a paying user. While there are exceptions, the order is important: a user can’t become an active user without creating an account. Likewise, paying users are most likely active users.

Create An Account

What percent of visitors create accounts? There many variables that will influence the percent of visitors that create an account and they include:

Relevant Keywords

Maybe visitors are arriving at your site searching for something else. If that’s the case you might have a high bounce rate (e.g. users coming to your site and immediately leaving). To fix this issue investigate ways to improve your SEO and ensure that your keywords are relevant to your product.

Clear Selling Point

Have you made it as clear as possible why your product is needed? Ask colleagues unfamiliar with your product to evaluate you’re site’s copy and it’s clarity.

Call-to-Action Button

Have you made it obvious how they can create an account? To test this you can use a service like Optimizely to A/B test different variants of your front page and your call-to-action button.

Pricing

Maybe your pricing page is too complicated, or your prices are too high, and as a result users don’t even bother creating an account. This is an issue that you should look into before you launch. Ideally you should speak with an appropriate number of potential customers to figure out if your price is within a reasonable range.

Become An Active User

What percent of users become active users (e.g. use your app every day/week)? Your definition of an active user is dependent on your service. If you’re creating a new email app, or a social network, an active user might be someone who uses your app multiple times per day. If you’re creating a tax app, an active user might be someone who uses it once a year for a week. In order to measure active users first you must define who they are (duh!).

Problem: users create an account and then never use your app again. It could be that you have a very interesting idea, but your implementation is off. Likewise you could be attracting a lot of curious users that have no intention of actually using your app, but just want to see what the fuss is about.

I’d recommend following-up on a user if they haven’t used your app in a week after registering. Ask them why they didn’t end up using it, and what *one* feature they’d need to use it. Your response rate will likely be between 5 - 10% but hopefully that should be enough to pick up trends. If possible I’d automate this step to ensure that emails are sent out consistently and you don’t waste your time sending out copy and pasted emails.

Problem: they use your app for a couple of weeks and then never use it again. While still bad, at least you have more data to work with. Check your database and determine what parts of your app they were using. For an email app, were they sending emails, but not creating contacts? Maybe creating contacts was too tricky and they gave up and stopped using your app.

Analyze your data carefully and see if you can pick patterns that can point you to areas that need to be improved. Alternatively there could be temporal patterns, for instance, people who only use your app once a week quickly stop using it relative to those that use it several times a week. Finding patterns of use and comparing active users to non active users can shed further light on potential problems.

Become A Paying User

What percent of your active users become paying users? Comparing patterns of use between users who’ve converted, and those who have not, in the same cohort is useful for elucidating causality in conversions.

Unlike the previous step you should have much more data to conduct your analysis. More data means that it should be easier to find statistically significant patterns, but it may be more challenging and time consuming to do analysis. I’d recommend generating fewer than five hypotheses before you start your analysis. This will both limit the complexiy of analysis and reduce the multiple comparison problem (e.g. with enough comparisons you’re bound to get significant differences that are due to chance alone).

Conclusions

Analyzing what your users are doing, why some are creating accounts, why some are becoming paid users and why others are not is extremely important. If you love coding, and adding new features, it can seem like a big time waster, however, doing the above will help you prioritize better.

Although I haven’t tried either myself both Kissmetrics or Google Analytics’s Conversion feature should help you with the above. It’s crucial to be able to quickly determine why some users never become active, or never become paying users. Use hypotheses, data and outcomes to determine how to spend your time efficiently. Time is something you have little of, use it wisely.

You’ve created a product that people want to use, and now you’re eager to launch. From my experiences launching Understoodit, in May 2012, I’ve compiled a set of steps that helped Understoodit get on several big sites including TechCrunch, Toronto Star, and BetaKit. If you have a large budget hiring a PR firm might be your best bet, otherwise the steps below will help you get started.

A Unique Angle

If you’re going to catch people’s attention in an app saturated environment, it’s important to communicate what’s unique about your app. If you’ve created a todo app, is it for Doctors, Engineers or does it something truly unique? If you can’t figure out why your product is unique it’s going to be a tough sell and it will certainly make it more difficult to get the press interested on your launch day.

The unique angle for Understoodit was focusing on its confusion feedback feature (students can click confused, and in real-time the teacher can see what percentage of students are confused). If I had said Understoodit was a “classroom response system”, I would have had a much harder time competing for attention.

Press Release

An important step in preparing for a launch is crafting a press release. If you’ve never written one, or you aren’t a strong writer, I’d recommend hiring someone to write it with you (Vicki So helped me).

My familiarity with Understoodit made it difficult to objectively write about it. When you’re focused on the technical side of your product it can be easy to loose focus on what’s important to prospective users. By asking good questions, Vicki was able to tease out of me what was important about Understoodit and why educators might be interested. She was able to turn a product launch into an interesting story about how I started Understoodit.

Reporters read many press releases each day, make sure yours is interesting, tells a story, and is well crafted.

Reporters & Bloggers

One of the most important tips Vicki gave me was: send the press release to a specific reporter, not a newspaper or website in general. A reporter who covers education is potentially more interested in Understoodit than the average person handling general enquiries. In preparation for launch I made a long list of reporters that cover education, and on launch day I emailed each of them a quick note with a press release attached. I’d also recommend adding reporters who cover small business and startups to your list, they may also be interested.

In addition to reporters, I contacted a couple local tech bloggers and asked if I could give them a face-to-face demo. This approached allowed me to pitch a reporter at BetaKit, a Toronto-based website that covers startups. They ended up doing an article about Understoodit a day after the launch.

Social Media

I’m no social media expert, but I can say it played an important role in the early success of Understoodit. Facebook and Twitter were huge sources of traffic, as were social news sites such as Hacker News. Depending on your product’s niche you might have more luck with other social networks such as Pinterest or Instagram. However, for you to have a big impact on sites like Twitter, or Pinterest, it helps to have a lot of followers. Gaining followers takes time and is something you should consider long before you launch.

Friends & Family

I owe a lot of Understoodit’s launch success to friends that not only helped me with the press release, but also tweeted, liked and voted up Understoodit on launch day. Friends and family are also critical in getting you over the trough of despair - so be kind to them!

Luck

No matter how prepared you are there is a strong component of luck to a successful launch. Was your press release the first that a reporter read, or did they read it after reading 5 others? Was there a major news event on the day of your launch? Did a writer on TechCrunch see your launch on Hacker News? All those things are mostly out of your control but can greatly affect your success. Preparation can mitigate some of those issues. For instance, it’s always a good idea to see if there might be any important news events, or tech announcements, that could overshadow your launch.

Conclusions

After following the above actions, mixed with a healthy dose of luck, I was on the front page of Hacker News, and later that day on TechCrunch. Over a 24 hour period Understoodit received hundreds of registrations. Ultimately, that initial burst of excitement made it possible to get an article in the Toronto Star and the Chronicle of Higher Education.

Next Week: I will cover 3 important metrics that you need watch after launching your product.

Unfortunately, that pretty much summed up my (immodest) perception of myself prior to launching Understoodit.com.

While the May 2012 launch of Understoodit was more successful than I anticipated, there were certainly things I could have placed more focus on prior to launching. Below I’ve informally divided the pre-launch process into 5 stages: 1) finding customers, 2) the one feature, 3) early beta testing, 4) engaged users, and 5) time to launch.

Stage 1: Finding Customers

It’s an increasingly common sentiment that you should find potential customers before you start building a product. If you have difficulty finding customers at this stage it may not get easier when you’re coding 12 hours a day. In my experience it’s easier to get help from potential customers at this stage since you aren’t necessarily selling anything yet. At this stage you’re simply doing research and building relationships with potential customers. When contacting professors for Understoodit, I found they were happy to give feedback and provide constructive criticism. If initially I had tried to sell them something, they may not have been so forthcoming with help and criticism. If all goes well, those first few relationships will turn into paying customers, so be kind and accept their feedback without becoming defensive!

Stage 2: The One Feature

Assuming you’ve mastered the previous step, getting feedback will be easy. In fact you’ll likely end up with a laundry list of features. Some features are nice to have, while others will be critical. Unfortunately teasing apart the critical features from the nice-to-haves is not always easy [1]. I’d recommend asking potential users: “If we added only one feature, what would be the most important?” This will force your users to prioritize what they think is the most important feature. A single user may not get this right, but the intuition of multiple users should converge on a single feature. That single feature will become your minimum viable product (MVP). Furthermore, if your users thought that feature was very important, it’s likely that others, in the same demographic, will also think it’s important.

Stage 3: Early Beta Testing

At this stage you’ve created an MVP, with one critical feature, and you’re eager to start early beta testing. There is a lot of good advice on user testing online, so I’ll focus on one method that I’ll dub “passive watching”. Passive watching involves sitting with potential users and passively watching them create an account, use your app, perform various actions, etc. It’s important not to help them at this point. Rather, what you want to do is see where they’re getting stuck, where they’re getting frustrated, and where things are working smoothly. Don’t be defensive if they dislike the user interface, or its flow, just listen at this stage. After testing with 10 - 20 potential users, you’ll get a very good idea of what needs to be improved, removed and what needs to be added.

Stage 4: Engaged Users?

During early beta testing you hopefully received a lot of positive feedback. (And thanks to many who gave me feedback, including readers of this blog.) But don’t conflate positive feedback with engagement. Just because a user says your app is great, it doesn’t guarantee that they will actually use it, let alone pay for it. However, if your early beta testers keep using it, and start telling their friends, then that’s a good sign. On the other hand, if they stop using it, it’s critical to find out why they’re not using it. If you can’t engage your early beta testers (that is, users who’ve invested a lot of time already), it will be difficult to engage new users after launch. This stage is important! So don’t fool yourself into thinking you’ve created a great product unless you have engaged users using your app regularly (self awareness and introspection are important attributes for entreprenuers [2]). If your users aren’t engaged you have to decide whether to go back to early beta testing, pivot, or scrap the idea entirely.

Stage 5: Time to launch

You’ve built a strong MVP that is used regularly by your beta testers. In turn those testers are telling friends and colleagues about your app. You’ve validated both your idea and your execution and it’s time to launch and grow the number of users.

Next week I’ll cover some of my experiences that helped get Understoodit featured on TechCrunch, Discovery.com News, The Chronicle of Higher Education, and the Toronto Star. Following next week’s article, on “The Launch,” I will focus on metrics that will help you decide if your startup is succeeding or floundering.

Extra Reading:

1. Tactics for Better Customer Interviews
2. Secret Ingredient for Success (The New York Times)

Running a startup can be a tough slog. You’re unsure if people like what you’re building enough for it to be successful. If you get some traction there will be people out there that will belittle your idea, or your execution. With all the unknowns, and the not-always-constructive criticism, it’s incredibly refreshing to interact with a nice customer that loves your product.

I’ve noticed that the nicer I am to customers the nicer they are back. This virtuous circle means that we get more and more people who are willing to provide thoughtful criticism and who are willing to meet with us and give us feedback. We have one customer that loves Understoodit so much that he just finished writing a blog post for us (to be posted in the next week).

After a difficult day in startup land it’s really gratifying to interact with a nice customer. It puts a face to an email address and makes the whole process of building software that much more interesting and fulfilling.

Make sure you’re nice to your customers, they may just be nice back.

I’ve heard that Stripe is significantly easier to integrate than its competitors, however, having never integrated a payment system I have no comparison. Overall I think Stripe is excellent and I’m glad we chose it, however, there are 4 areas, that if improved, would make Stripe perfect: 1) getting started, 2) taxes, 3) invoices and 4) edge cases.

Getting Started

Having never integrated a payment system into a website I wanted to be as careful as possible. I wanted to make sure that I understood as many details as possible, and had time to carefully map out all the things that needed to be done before pushing it to production. What I would have really liked was a diagram of how Stripe worked. A good diagram is much easier for me to parse, and would have helped me to understand the text documentation much quicker.

For instance, below is one possible diagram that would show the potential set of events associated with a new user signing up for a subscription plan.

Taxes

Understoodit is based in Canada, obligating us to collect taxes from Canadians, but not our international customers. The kicker is that different provinces have different tax rates. In total there are 4 different tax rates and then no taxes for international customers, for a total of 5 different tax levels. We currently have 3 monthly plans, and their 3 yearly equivalents. This means that we had to create (3 + 3) x 5 = 30 plans within Stripe.

It would have been significantly easier if Stripe had had a ‘tax’ feature, in the same way they have coupons. If such a feature existed we’d only have had to create 6 plans in Stripe and 4 tax levels. This would result in Stripe invoices including the subtotal, amount of taxes and total (one less thing for us to calculate). While Stripe does a great job of prorating payments, when a user switches from one plan to another, it becomes tricky for us when they switch province. While calculating taxes isn’t that difficult for us to do, it’s just one more thing we have to get right. It’s one more thing we have to calculate when sending an invoice to our customers.

Invoices

Every online subscription that I currently have sends me a simple invoice at the end of each billing period. Some have .pdfs attached that I can easily send to an accountant. Additionally, those services include admin panels that list all the past invoices and those can easily be downloaded as pdfs.

It would be amazing if Stripe handled 1) emailing invoices and 2) creating a .pdfs of each invoice. Developers could provide simple HTML templates for email and pdf invoices and Stripe would fill in the blanks and send the invoice at the correct time, to the correct user. I suspect people would be willing to pay extra for this feature.

Edge Cases

Like any feature, there are multiple edge cases to a payment system. For instance here are a few that we dealt with:

• Upgrade plan, and simultaneously change provinces (e.g. tax rate)
• Downgrade plan and simultaneously change provinces (e.g tax rate)
• Change credit card info, but not billing address
• Change plan, but keep credit card and billing info the same

Stripe employees can no doubt enumerate many more edge cases than someone adding Stripe for the first time. It would be helpful if they provided a checklist of edge cases for developers. The checklist could also include common security concerns that developers should understand before rolling payments out.

Conclusions

With all the hype on Hacker News, I had this vision of Stripe being magically easy to integrate. While it wasn’t rocket science it was certainly more effort than I envisioned. The whole process was made significantly more difficult by having to handle taxes.

I wonder if there would be value in Stripe conducting the following usability experiment. Get a dozen computer science and engineering undergrads and ask them each to integrate Stripe into an existing website. Stripe employees would sit with them and monitor their progress (or lack of), but provide no help. I suspect that they’d find some surprises in how novices approach Stripe integration. The information gained in such an experiment could help them create better documentation, reduce development burden and ultimately create a more perfect Stripe.

Scope: this, that and window

Like many programming languages JavaScript provides internal access to objects using the keyword this. Unfortunately, what this refers to differs depending on what called the function containing this. A common example, shown below, is what happens when setTimeout is called.

In the example below a Dog object is created, with a bark function. The Dog ralph is instantiated with the name ‘Ralph’ and when ralph.bark() is called, “Ralph” is printed to the console.

What becomes confusing is what happens when setTimeout is called with the parameters ralph.bark and 500. After 500 milliseconds ralph.bark is called, however, nothing is printed to the console.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

var Dog = function( name ){
  this.name = name;
}

Dog.prototype.bark = function(){
  console.log( this.name );
}

var ralph = new Dog('Ralph');

ralph.bark()
// Ralph is printed to the console

setTimeout( ralph.bark , 500 );
// nothing is printed to the console

Mozilla Developer Network refers to the problem above as ‘The ”this” problem’. What happens is this within bark() refers to the browser’s window variable when bark() is called from setTimeout.

Avoiding the this problem.

1
2
3
4
5
6
7
8
9
10
11

// Works in JavaScript 1.8.5
setTimeout( ralph.bark.bind(ralph), 500 );

// using jQuery
setTimeout( $.proxy( ralph.bark, ralph ), 500 );

// using undescore.js
setTimeout( _.bind( ralph.bark, ralph ), 500 );

// using an anonymous function
setTimeout( function(){ ralph.bark(); }, 500 );

In each of the above the bind, or proxy functions explicity ensure that this within ralph.bark refers to ralph and not window. In the final example an anonymous function is called and provides another way of fixing the this problem.

Callbacks in Loops

When I launched Understoodit.com in May I included a waitlist for interested users to signup. I was planning on inviting a few dozen users a day, however, due to a deluge of emails sending out invites was delayed by a week or two.

To send the invites out, I went to Understoodit’s admin panel and seleted 40 people on the waiting list and clicked invite. A few days later I noticed that only a few individuals had accepted the invite. I looked at Postmark’s logs and noticed that invites were only sent to 5 individuals. What’s more those 5 individuals had received anywhere from 5 - 15 emails each. Meanwhile, the other 35 invitees had received no emails. The bug: I had a callback in a loop that iterated over all the selected invities and 1) called databaseModule.addInvitedUser, which created an invite token and added that invite to the database and 2) sent an email with the newly created token. Below is a simplification of the code, with error handling removed.

1
2
3
4
5
6
7
8
9
10

function sendInviteEmails( emails ){
  for(var i = 0; i < emails.length; i++ ){

    databaseModule.addInvitedUser( emails[i], function( error, token ){

      emailModule.sendInvite( emails[i], token );

    });
  }
}

What went wrong was that the anonymous function “captures the variable i, not its value”. The value of i is dependent on when the anonymous function is called, which varries depending on how long it takes to add the invited user to the database.

The solution I used was to wrap the anonymous function with an immediately invoked function expression (IIFE). The IIFE “locks” in the value of i ensuring that emailModule.sendInvite() refers to a different value of i on each IIFE call. Alternatively, one could create a second function outside of the loop and then call that (see option 2), a solution that would likely be easier to read.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

// option 1
function sendInviteEmails( emails ){
  for(var i = 0; i < emails.length; i++ ){

    (function(email){
      databaseModule.addInvitedUser( email, function( error, token ){
        emailModule.sendInvite( email, token );
      });
    })( emails[i] );

  }
}

// option 2
function sendOneInviteEmail( email ){
  databaseModule.addInvitedUser( email, function( error, token ){
    emailModule.sendInvite( email, token );
  });
}

function sendManyInviteEmails( emails ){
  for(var i = 0; i < emails.length; i++ ){
    sendOneInviteEmail( emails[i] );
  }
}

The above quirk seems similar to a fairly common JavaScript interview question that takes the form of adding event listeners to an array of links.

Global Variables

The problems with global variables have been discussed many times before. Suffice it to say that you should avoid them by using var (e.g. var x = 0; vs x = 0;) when first declearing a variable. If you have a variable that has unaccounted for properties or values, there’s a chance that a global variable could be to blame.

I’d highly recommend defining all your variables at the top of the function to make it as clear as possible when a var is missing. Furthermore I’d recommend using JSHint which can warn you of global variables.

Values in HTML forms

1
2
3
4
5

<select id="order-sizes">
  <option value="1">Small</option>
  <option value="2">Medium</option>
  <option value="3">Large</option>
</select>

When it comes time to get the form’s values, and use them within an application, I usually do something like:

1
2
3
4
5

var orderSize = $('#order-sizes option:selected').val();

if( orderSize === 1 ){
  console.log('Thanks for ordering a small!');
}

Unfortunately the value that jQuery returns is a String, and comparing it to 1, a Number returns false. Here are several solutions to this:

1
2
3
4
5
6
7
8
9
10

if( Number( orderSize ) === 1 ){ ... }

if( parseInt( orderSize ) === 1 ){ ... }

if( orderSize == 1 ){ ... }

// last solution
var orderSize = Number( $('#order-sizes option:selected').val() );

if( orderSize === 1 ){ ... }

In the first two examples orderSize is explicitly converted to a Number using the Number constructor and the global parseInt function. In the third example the double equals coerces orderSize to a Number before comparing to 1 (MDN on comparison operators). However, I’d recommend going with the last approach, which allows you to use orderSize as a Number in multiple spots without having to repeatedly caste to a number. If you don’t like the last approach I’d recommend the first or second approach, since it seems to be generally preferred to use the strict equal (tripple equal signs) and not to use the double equal sign.

Conclusions

Many of the above errors can be avoided by following JavaScript style guides (Google JavaScript Style Guide, Addy Osmani on Style Guides). A sign of how tricky JavaScript is comes from Github’s JavaScript Style guide that goes as far to recommend avoiding JavaScript altogether and using CoffeeScript - a suggestion a few would disagree with!

Using the right tool for the job is a basic tenant amongst programmers. However, with all the currently available database options it’s increasingly difficult to figure out what the right tool is. Sometime it’s nice to have a very simple tool that can be used for many different tasks: Redis. Over the last 4 months I’ve been using Redis heavily and I’ve even started to use it for relational data. I’ve been curious to find out the performance differences between Redis and PostgreSQL. Below I’ll provide an example of storing a simple relational dataset in Redis, and I’ll look at the performance differences between Redis and PostgreSQL.

Why use Redis for Relational data?

I find Redis appealing because it’s the simplest database that I have ever used (relative to: MySQL, PostgreSQL, Riak & Mongo). The documentation includes the time complexity of each command, and the documentation provides an interactive console to experiment with a given command. There’s also a certain appeal to using a single database instead of 2 or 3:

1. It’s much quicker to master 1 database than 2.
2. Two different databases means twice the updates, bugs and crashes.

I’ll outline a few ways Redis can be used to store relational data and the performance differences between redis and PostgreSQL. All the examples and performance tests were done using Node.js.

Storing Relational Data in Redis

Redis values can be 1 of 5 different datatypes: strings, hashes, lists, sets and sorted sets. Each row in a relational database can be represented using a hash, and a list, set or sorted set can be used to represent a table. The datatype that’s used to represent the table is dependent on how the data needs to be retrieved.

For example, let’s say we’re storing blog posts. In Redis, each post will be stored in its own hash, with its key corresponding to the post’s url:

1
2

'a-post-about-databases' :
  { title : 'A post about databases', body : '...', createdAt : 1338751532301}

Retrieving a single post using the url becomes O(N), where N is the size of the hash (post). Since the number of keys in a post is constant, retrieving that post becomes O(1). However, if we wanted to get all the posts, or a subset of them, it becomes useful to also store the keys in a sorted set (e.g. the “table”). Using a sorted set means that posts can be stored by their createdAt date and it allows us to retrieve all the posts, or a subset of them (useful for pagination).

1
2
3
4
5
6
7
8
9

redis.zrange('posts', 0, 10, function(error, posts){
  //return the keys (urls) associated with the first 11 posts
})

var startDate = (new Date(2012, 5, 1)).getTime() ; // June 1st
var endDate = (new Date(2012, 5, 30)).getTime(); // June 30th
redis.zrangebyscore('posts', startDate, endDate, function(err, posts){
  //returns the keys (urls) associated with all the posts from June 2012
})

The above example is relatively straight forward, but what about storing the post’s comments? For every post we create a new sorted set called: ‘comments-KEYofPOST’. The comments are sorted by their creation time. To get a post, and its comments, we could do the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

var postURL = 'a-post-about-databases'

var multi = redis.multi();

// queue up the queries
multi.hgetall(postURL);
multi.zrange('comments-' + postURL, 0, -1);

// execute the queries atomically
multi.exec( function(error, results){
  /*
  results[0] will contain the post
  results[1] will contain an array with all the comments
 */
});

Redis vs. PostgreSQL Performance

In SQL you might do 1 query to get the post and another to get the comments, or use a join to get the post and the comments in one query. With the approach above, using Redis, 2 queries are atomically executed, using the multi and exec commands. Both in PostgreSQL, and Redis, a single request is sent the database to retrieve 1 post and its 10 comments.

To test the the performance I created a dataset that includes 10,000 ‘blog posts’, with each post having 10 comments (100,000 comments in total). All tests were run on a 2011 Macbook Pro (2.3 GHz i7, 8GB RAM). To test PostgreSQL, I sequentially fetched each post and used a join to retrieve its comments (10,000 separate queries). The test was repeated six times to produce an average time and was done for both PostgreSQL and Redis.

Using PostgreSQL, it took an average of 138.34 seconds to execute all 10,000 queries, or 13.8 milliseconds/query. Using the native bindings, that come with the psql node module, yielded an improvement and was associated with 12.6 milliseconds/query. When an index was added to comments (post_id), the time dropped to 2.72 seconds, or 0.27 milliseconds for a post and its 10 comments. In contrast, Redis can retrieve a post and its comments in 0.067 milliseconds. Of course the above is akin to comparing apples to oranges, but it still provides a glimpse into the performance differences between Redis and PostgreSQL.

While Redis is in memory and should be fast, PostgreSQL uses caching algorithms (LRU) to keep its contents in memory. Of course, keeping everything in memory (Redis) will most likely be faster than using LRU.

Caveats to using Redis for Relational Data

The single biggest caveat to using Redis, is that it is entirely in memory. If your relational dataset is 2.5GB (not that large), you’ll need a $160/month Linode (4GB RAM) to keep it in Redis. In contrast, a $20/month Linode (512MB RAM) has 20GB of disk space and could easily hold that same dataset using PostgreSQL. This tradeoff becomes even more of an issue as your dataset become larger than 4GB.

The above example only represents a very simple relationship between two pieces of data (posts and comments), mapping a many-to-many relationship in Redis would take a little more imagination.

Conclusions

Before storing all your app’s data in Redis it’s advisable to estimate how large your dataset will be in a year, or two, and how much much RAM will be required to use Redis. If your dataset will be greater than 4GB in a year, and money is a constraint, it probably makes sense to put all, or a portion of the data, in PostgreSQL, or use an alternative noSQL solution (e.g. Riak or Mongo).

Code on Github

Given the authentication options for Express.js I wanted to create a module that would make adding user authentication quick and easy. Moreover, I also wanted developers to be free to edit and modify the generated views. In addition to authentication I wanted the module to handle sign ups (the type you see on a just-launched startup’s page) and to handle waiting lists and invitations. Based on the module’s functionality I’ve decided to call it Drawbridge.js.

User Authentication with Drawbridge.js

Drawbridge.js uses Redis to persists its data, but it’s possible for developers to create other database adapters for Drawbridge (pull-requests accepted). I chose Redis because its ability to pipeline multiple commands reducing round trips between the server and the database. The atomic nature of pipelined commands obviates a lot of complex callbacks and makes the resulting code much easier to understand. Overall Redis is easy to use, easy to understand and fast - great features for an authentication module.

To send email, Drawbridge uses either nodemailer, or the postmark modules. I included the Postmark option because I’m currently using it and I like it. However, developers are free to add additional email adapters.

Drawbridge Screencast

I’ve created a short screencast to show how easy it is to add drawbridge to an existing Express.js application. Before you watch the screencast it’s important that I outline a couple of caveats:

1. Drawbridge is not ready for production - it’s basically a working prototype.
2. Drawbridge views and variables are inconsistently named, that will need to be fixed.
3. The code needs refactoring and more testing.
4. Drawbridge needs to be picked apart for security issues.

With those caveats out of the way here is the video:

Drawbridge.js from Liam Kaufman on Vimeo.

While I built Drawbridge.js to scratch my own itch, I hope others will find it useful as well. Once I refine it further I will most certainly start to use it in my own projects. If you’re interested in Drawbridge 1) watch the project on Github and 2) try and get it working on your toy Express apps. I welcome feedback on both the architecture of Drawbridge and its security.

Drawbridge.js on Github

Approach 1: Tables to Divs

HN’s front page is comprised of: 4 tables, 98 rows, 159 columns, 37 inline style declarations and numerous attributes that dictate style. To reduce the markup on the front page I created a new HN front page (Github link) that looks identical to the existing page but does not include tables or inline css. I also went a step further and base64 encoded both the logo and the up arrow to decrease the number of requests. The completed CSS file was run through a css minifyer to yield further reductions. With those changes only two requests are necessary, one for the HTML file and one for the CSS file. Table 1 shows that those changes yielded an overall reduction of 37%.

I also slightly modified the JavaScript responsible for sending up-votes to the server. Instead of grabbing a vote’s id from the id of the HTML node, it gets it from the ‘data-id’ attribute. Otherwise, the JavaScript remains identical. As an aside, if you have not examined the JavaScript that is responsible for sending votes to the server, I’ve included it below (the existing code). It’s a creative use of an image tag. An image node is created, but not added to the DOM. When the image node is assigned a ‘src’, which happens to include all the vote info, it then requests the ‘image’, using the constructed url. Thus the ‘image’ request becomes analogous to an AJAX GET request, but without a conventional response.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

function byId(id) {
  return document.getElementById(id);
}

function vote(node) {
  var v = node.id.split(/_/);   // {'up', '123'}
  var item = v[1];

  // hide arrows
  byId('up_'   + item).style.visibility = 'hidden';
  byId('down_' + item).style.visibility = 'hidden';

  // ping server
  var ping = new Image();
  ping.src = node.href;

  return false; // cancel browser nav
}

Approach 2: JSON API

Although approach 1 results in a 37% decrease in data transferred to the client, markup and data must be transferred to the client on every refresh. In approach two, the markup is only transferred to the client once, and then cached, while the data is sent the client via JSON. Using this approach would decrease the HTML file but no doubt increase the JavaScript file. However, both of those resources could be cached in browser, and cached on a CDN, drastically reducing the number of requests to HN’s server. Furthermore, the JSON representing the stories on the front page is 7.8KB, much smaller than the amount size of the existing solution or even approach 1.

Approach 2 is not without its drawbacks. It would require significant changes in both HN’s backend and large changes to the client-side. A JavaScript application and API would have to be created. This approach would likely be incompatible with bots that would not execute the JavaScript necessary to populate the page with stories. To get around this the agent-type could be detected and a static version could be served to bots. Alternatively, the webpage could be pre-populated with stories and subsequent requests would take advantage of AJAX get requests. This would simplify matters, but make caching more difficult, since the cache page would require updating every time the front page changes.

Conlusions

By transitioning from tables to divs, and inline css to external css, HN could dramatically reduce the bandwidth required to serve its web pages. The first approach would require minimal changes to HN’s back-end making it a good candidate for adoption. While the second approach could yield even better results, it would require drastic changes to both the server and the client, making it more suitable as a long term solution.

In addition to the two approaches above, gzip compressing both the .html and the .css would further reduce transferred data. It would also be beneficial to add the appropriate headers to enable browser caching for CSS.

While Paul Graham may have insufficient time, or interests, in implementing some of the above changes, I suspect he knows a few individuals who would be willing to help out.

Code on Github

I miss the Hacker News from four years ago. It was awesome. The discussions there are not even worth reading anymore. It’s sad.

Based on the number of retweets and favorites, I suspect that others agree. In fact the idea that Hacker News is degrading is common enough that it has been addressed on HN’s guidelines:

If your account is less than a year old, please don’t submit comments saying that HN is turning into Reddit. (It’s a common semi-noob illusion.).

However, Mr Curtis has been on HN for over five years, and is certainly not subject to the ‘common semi-noob illusion’. Is HN getting worse then?

I suspect that when HN started only those that were most passionate about hacking were familiar with HN and would take the time to comment on it. As time went on the popularity of both Y Combinator and HN may have resulted in the level of discourse regressing to the mean. That’s not to say that there aren’t still intelligent comments, in fact I’d argue that there are more intelligent comments than there were 4 or 5 years ago. However, people tend to remember the unintelligent comments more, especially when those comments contain opinions that differ from their own or originate from non-experts.

If Digg, Reddit and Hacker News are no longer the best places for discussion how can we create a place that is? While there ought to be many ways to encourage scholarly discussion, and discourage idiotic comments, I want to explore several ideas.

Exclusivity

In the early stages Digg, Reddit and Hacker News were implicitly exclusive. They didn’t discourage people from joining, but their initial lack of popularity acted as a filter to those who were technically savvy and within certain social networks. Once the exclusivity vanished the communities became diluted. Forrst is explicitly exclusive and is by invitation only. Does Forrst’s exclusivity lead to a stronger community? To reiterate, is exclusivity a necessity in keeping a social news site strong and viable?

Experts

I enjoy when an article pops up on HN about physics or biology and several graduate students in those fields provide intelligent comments. Is there a way to officially denote that someone is an expert in a field and automatically give their comments more weight? In very esoteric subjects it isn’t necessary, the complexity of the subject reduces “average” comments. However, in simpler subjects, bikeshedding becomes an issue. Could bikeshedding be prevented by weighting comments based on the user’s past comments? For instance, if an individual has been voted up when discussing physics, perhaps future comments on physics should be algorithmically voted up.

Focus

One thing I really appreciate about HN is the variety of content. However, I can’t help but wonder if a social news site restricted the content to just a specific topic if that would both decrease the probability that the content, and discussion, become watered down? Forrst focuses on design and development, has their focus helped them? Reddit has addressed this issue with subreddits, but the result seems to be many hardly used subreddits. I think focus is important, but at the same time I like being exposed to new topics, can the those two wishes be balanced?

Conclusions

There is no magic bullet for maintaining the quality of a social news site, but there are a collection of concepts that may help. It would be interesting to A/B test some of those ideas. One could imagine creating several social news websites for different topics and making some exclusive and some not, or altering other variables and see which succeed. What do you think is important in a social news site?

jQuery and jsdom

Using jsdom you can specify a local file, or url, and jsdom will return the window object for that document. Additionally, JavaScript can be inserted into the document; in our case we’re inserting the jQuery library. In the example below all the links from the Hacker News front page are logged to the console.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

var jsdom  = require('jsdom');
var fs     = require('fs');
var jquery = fs.readFileSync("./jquery-1.7.1.min.js").toString();

jsdom.env({
  html: 'http://news.ycombinator.com/',
  src: [
    jquery
  ],
  done: function(errors, window) {
    var $ = window.$;
    $('a').each(function(){
      console.log( $(this).attr('href') );
    });
  }
});

Making Scraping More Robust

Unfortunately there are few common bugs that I ran into when scraping content with jQuery and jsdom. Specifically there are two issues, that aren’t necessarily specific to jsdom, that are worth watching out for.

jQuery Return Values

The first issue are return values from jQuery function calls. Extra attention has to be paid to return values. Applying a method to undefined will crash a program, a problem that can be especially apparent in DOM parsing. Consider the example below:

1

$($('a')[7]).attr('href').split('/')

If there are 8, or more, links on a page the 8th link will be returned and its href attribute will be split into an array. However, if there are less than 8 the attr('href') will return undefined and calling split() on it will crash the program. Since HTML pages aren’t as structured as API responses, its important not to assume too much and always check return values.

Web Page Errors

It’s entirely possible that the url passed to jsdom returns an error. If the error is temporary, your scraper might miss out on important information. This issue can be mitigated by recursively retrying the url, like the example below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

var getLinks = function(retries){

  if(retries === 3){
    return;
  }else if (retries === undefined){
    retries = 0;
  }

  jsdom.env({
    html: 'http://news.ycombinator.com/',
    src: [
      jquery
    ],
    done: function(errors, window) {

      if(errors){
        return getLinks(retries + 1);
      }

      var $ = window.$;
      $('a').each(function(){
        console.log( $(this).attr('href') );
      });
    }
  });
}

With the above approach, if errors are encountered getLinks will be called recursively with a larger retries value. On the 3rd retry the function will return. If you wanted to go further you could wrap the recursive call in setTimeout to ensure that the recursive web request was not made immediately after the error was encountered.

Conclusions

Parsing web pages with jQuery on the server is a much more natural experience for developers already accustomed to using jQuery in the client. However, prior to scraping it’s worth checking if the site 1) allows scraping and 2) does not already have an API. Consuming a JSON API would be even easy than scraping and parsing!

Why is Riak a Good Fit with JavaScript and Node.js?

Riak-js makes storing JavaScript objects easy. There is no need to JSON.stringify() a JavaScript object when saving it, or applying JSON.parse() when retrieving.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

var db = require('riak-js').getClient();
var post = {id: 17,
            date: new Date(),
            title: 'a blog post',
            body: 'A blog post about Riak'};

db.save('posts', post.id, post);
db.get('posts', 17, function(err, data, meta){
    console.log(data);
});
/* prints
{ id: 17,
  date: '2012-02-29T18:26:44.400Z',
  title: 'a blog post',
  body: 'A blog post about Riak' }
*/

In the above example, a riak-js client is created and a post object is created. The post object is saved into the ‘posts’ bucket, with its id as its key. To retrieve the post, the bucket and key are referenced.

Using JavaScript in the client, and the server are nice, and being able to easily save JavaScript objects is even better. Not having to switch between languages would certainly reduce annoying syntactic problems. This would also allow you to write the entire stack in JavaScript, CoffeeScript or ClojureScript, giving you several programming paradigms to choose from.

If you decided to use Backbone.js on the server, calling toJSON() on a Backbone model would allow you to easily store the model in the Riak.

Why Riak?

At this point you might be wondering why you’d want to use Riak when CouchDB also has a JavaScript interface and can do some of the above. As Damien Katz, the creator of CouchDB has pointed out, CouchDB is slow and can’t “scale-out on it’s own”. In contrast, Riak was built for replication and scaling out. In fact, people at Basho, the company behind Riak, indicate that adding new nodes actually increases throughput.

Additional Features of Riak

Saving JavaScript objects and easy scaling are both good fits with node.js but Riak has some additional features, such as buckets and links, that make retrieval convenient. With buckets the following functions become possible:

1
2
3
4
5
6
7
8

// Get all the posts within the posts bucket
db.getAll('posts');

// Get all the posts with the title ===  'a blog post' 
db.getAlll('posts', { where: {title: 'a blog post' }});

// Get the number of posts
db.count('posts');

Another interesting property of Riak, is it’s concepts of links. A link establishes a “one-way relationships between objects in Riak”. For instance, say we wanted to link similar posts the following would do:

1
2
3
4
5
6
7
8
9
10
11
12

var aNewPost = {id: 18,
                date: new Date(),
                title: 'a second blog post',
                body: 'blog post about Riak part 2'};

// Save the second post, with a link to the first post
db.save('posts', aNewPost.id, aNewPost,
  { links: [ {bucket: 'posts', 'key': 17 } ]});

db.walk('posts', '18', [{bucket:'posts',tag:'_'}]);
// db.walk, traverses object 18's links, 
// which happens to be post 17, and returns them.

Conclusions

While the above assessment is pretty rosey, Riak shouldn’t be the only database in your toolbox. Redis’ pub/sub and sorted sets are unmatched in Riak. If you’re running map/reduce over large datasets, you’re likely better off using Hadoop. Conversely if your already well-versed in SQL, and your data is relational, using Postgresql is probably a better fit. Despite those caveats being able to easily scale your database, and save JavaScript objects, is a pretty compelling reason to use Riak with Node.js

Further Riak and Node.js Reading

• Riak with Node.js
• Riak vs Couchdb for storing 100000 Coupons
• Moving from Couchdb to Riak
• Pagination with Riak

UPDATE: see my new article on adding real-time to your Rails application.

Despite the recent distaste for Rails, I still think its a nice framework for developing websites (e.g. devise & active record). However, if you want real-time communication Socket.IO and Node.js seem to be the best options. If you already have an existing Rails application porting the entire application to Node.js is likely not on option. Fortunately, it is relatively easy to use Rails to serve your client-side Socket.IO web application, while Node.js and Socket.IO are used for real-time communication. The primary goal of this article is to show one method of integrating a real-time application, that is slightly more complex than a todo app, with Rails. Thus, I created Chatty, a simple chat room web application that allows a user to see all the messages in the chat room, or filter the messages by user. Twitter’s Bootstrap was used for the CSS and modal dialogue.

Code on Github

Rather than explain the code step-by-step, I’ll provide a high level overview of:

• File organization
• JavaScript Templates and EJS
• Application Archicture and Publish/Subscribe
• Module Architecture
• Deploying to Heroku

File Organization

The entire client-side Backbone.js application is within app/assets/javascripts. Using a JavaScript manifest file (backboneApp.js) all of the application’s JavaScript files are specified.

1
2
3
4
5
6

//= require jquery
//= require bootstrap
//= require underscore
//= require backbone
//= require socket.io
//= require app

The Backbone application is within the app folder, which also has a manifest file. The manifest files describe all the JavaScript files that comprise the application. Within the application’s html file only a single line of code is needed to include the manifest file: =javascript_include_tag "backboneApp" (haml for templating). The actual organization of the files is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

javascripts
├── app
│   ├── index.js
│   ├── launch.js.coffee
│   ├── main.js.coffee
│   ├── modules
│   │   ├── index.js
│   │   ├── loadModule.js.coffee
│   │   ├── messageModule.js.coffee
│   │   ├── socketModule.js.coffee
│   │   └── userModule.js.coffee
│   └── templates
│       ├── message.jst.ejs
│       ├── modal.jst.ejs
│       └── user.jst.ejs
├── application.js
├── backboneApp.js
└── backbone_app.js.coffee

main.js.coffee is where the app object is defined, while `launch.js.coffee` is called last, after all the files have loaded, to launch the Backbone.js application. Each module, which might contain models, collections and views, are stored within the modules folder. The module structure was modelled after Backbone Boilerplate.

JavaScript Templates and EJS

To take full advantage of the asset pipeline it seems as if Sam Stephenson’s excellent EJS Gem was the most hassle free approach for JavaScript templates. Both the ‘ejs’ and ‘jst’ extensions are require for the EJS gem to compile the template, and include it within a JavaScript file. Access to the template is done with the global JST object.

Application Architecture - Publish/Subscribe

Before creating the application I decided to forgo the use of asynchronous module definition (AMD) and use a publish/subscribe (pub/sub) architecture (see Addy Osmani’s description of Pub/Sub). Specifically, each module is wrapped in an immediately-invoked function expression, and within each module functions can attach themselves to events (subscribe), or trigger events (publish). Using this approach the applcation’s only global variable is app which contains a copy of Backbone’s event object.

To reiterate none of the modules call methods from other modules, all communication occurs with pub/sub. This design pattern was a pleasure to use; adding new functionality often required simply subscribing to events! I found that my code stayed much cleaner than previous attemps with Backbone.js.

Module Architecture

The application is comprised of two types of modules, those that contain Backbone.js code (messageModule, userModule), and one that contains the Socket.IO code (socketModule). If either the messageModule, or the userModule, require content from Socket.IO they subscribe to events that the socketModule trigger. Likewise, Socket.IO messages sent to the server are the result of the socketModule suscribing to events triggered by the messageModule and userModule.

Below is an example module that contains skeleton code for an additional Backbone.js module. The ExampleModule class is used to glue all the Backbone.js objects together. In this case their is only one, the ExampleView, in Chatty’s MessageModule there are two distinct views instantiated within its MessageModule object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

ExampleModel = Backbone.Model.extend()

ExampleCollection = Backbone.Collection.extend
  model: ExampleModel

# View for a single model
ExampleView = Backbone.View.extend
  render: () ->
    @$el.html app.template 'example', @model.toJSON()
    @$el

# View for a collection of models
ExamplesView = Backbone.View.extend
  initialize: () ->
    @collection = new ExampleCollection()
    @collection.on 'add', @addExample, @
    @eventHandlers()

  eventHandlers: () ->
    # Subscribe to the app-wide event 'new-example'. When 
    # the event is called, the call-back function is provided
    # with an example model, which is then added to the collection.
    app.events.on 'new-example', (example) =>
      @collection.add example

  addExample: (example) ->
    exampleView = new ExampleView
      model: example
    @$el.append exampleView.render()

class ExampleModule
  constructor: () ->
    @examplesView = new ExamplesView()

new ExampleView()

Deploying Node.js and Rails App to Heroku

Deploying the Node.js server

Heroku requires the following code to create the Socket.IO server and listen for connections (note that Heroku doesn’t support websockets):

1
2
3
4
5
6
7
8
9
10
11
12
13
14

var app = require('http').createServer();
var io = require('socket.io');

io = io.listen(app);
io.configure(function(){
  io.set("transports", ["xhr-polling"]);
  io.set("polling duration", 10);
  io.set("close timeout", 10);
  io.set("log level", 1);
})

io.sockets.on('connection', function (socket) {}
var port = process.env.PORT || 5001;
app.listen(port);

Unfortunately, Heroku’s documentation only contains fragments of the above code. The above code, along with deploying instructions, is posted across several pages: getting started with Node.js on Heroku/Cedar and using Socket.IO with Node.js on Heroku. The `close timeout` option was added since the default 25 seconds made the chat app seem buggy (a user would log out but other users would seem them logged in for 25 seconds).

Deploying the Rails app

Deploying a Rails application is relatively well documented, but I thought I’d provide a few additional tips.

The URL for the production and development Socket.IO server differ. To accommodate this the Backbone.js app makes an Ajax request to the Rails app and gets the URL of the Socket.IO server along with a unique id for the current user. The Rails app can serve a different Socket.IO URL depending on whether it is currently in production or development.

The other thing that might be new for nacent Rail’s developers is the inclusion of the response.headers code in the show method, this tells the browser to cache the Backbone.js app for 25,300 seconds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

class BackboneAppController < ApplicationController
  layout 'backboneApp'
  respond_to :html, :json
  def show
    response.headers['Cache-Control'] = 'public, max-age=25300' if Rails.env.production?
  end

  def user_info
    respond_with({
        'uuid' => UUIDTools::UUID.random_create.to_s,
        'socketURL' => self.get_socket_url
    })
  end

  protected
  def get_socket_url
    Rails.env.production? ? "http://chatty-server.herokuapp.com/" : "http://0.0.0.0:5001"
  end
end

In order for Heroku to manage the asset pipeline your application must be deployed to Heroku Cedar’s stack. Unfortunately the Cedar stack doesn’t include Varnish caching, requiring you to enable caching via memcache and the dalli gem. I found that deploying a new version would not necessarily clear the cache and and I had to do it manually (connect to console: heroku run console):

1
2

dc = Dalli::Client.new('localhost:11211')
dc.flush_all

Final Thoughts

Relying entirely on pub/sub to communicate between modules worked really well in this application, but I wonder if it would scale to a larger application? I’d also be curious to know how other developers are combining Backbone apps with Rails, I suspect there are a number of ways to do it.

Code on Github

While I was excited to use Backbone Boilerplate, it wasn’t immediately clear how I’d integrate it with Rails 3. After several hours of tinkering I came up with an approach that places the build product of your backbone app in the lib/assets folder. This approach means that the application can be integrated into the assets pipeline and easily deployed to heroku.

Code on Github

Creating the Rails App

1
2
3
4
5
6
7
8
9
10
11
12
13
14

// create a new rails app
rails new rails-bb
cd rails-bb/lib

//remove the index file
rm -r public/index.html

cd lib
git clone https://github.com/tbranyen/backbone-boilerplate.git
cd backbone-boilerplate

//switch to the amd branch
git checkout amd
rm -rf .git

Changing Build Settings

In biolerplate’s config.js file modify the following to:

1
2
3
4
5
6
7
8
9
10
11
12
13

concat: {
  "dist/debug/require.js": [
    "assets/js/libs/almond.js",
    "dist/debug/templates.js",
    "dist/debug/require.js"
  ],

  "../assets/javascripts/require-app.js": [
    "assets/js/libs/almond.js",
    "dist/debug/templates.js",
    "dist/debug/require.js"
  ]
},

1
2
3
4
5
6
7
8
9

mincss: {
  "dist/release/index.css": [
    "assets/css/style.css"
  ],

  "../assets/stylesheets/index.css": [
    "assets/css/style.css"
  ]
},

Creating a Rails Layout and Controller

To build the app, go to lib/backbone-boilerplate and run node build default mincss. The build script will create lib/assets/javascripts/require-app.js that includes both the backbone-boilerplate application and the template files. In next step create a rails layout file app/views/layouts/bbapp.html.erb with the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
  <meta name="viewport" content="width=device-width,initial-scale=1">

  <title>RailsBb</title>
  <%= stylesheet_link_tag    "application", :media => "all" %>
  <%= csrf_meta_tags %>
</head>
<body>

  <!-- Main container -->
  <div role="main" id="main"></div>

  <!-- Application source -->
  <%= javascript_include_tag "bbapp", "data-main" => "app/index" %>

</body>
</html>

To plug the backbone app into the asset pipeline create assets/javascript/bbapp.js with the following contents:

1
2

//
//= require require-app

Now we need to let rails know that it needs to compile bbapp.js for production. To do this open config/environments/production.rb and after config.assets.compile = false add:

1

config.assets.precompile += %w( bbapp.js )

To test that this works, we’ll create a controller that has a single action that uses the bbapp.html.erb layout. From the root of the rails-bb application:

1

rails generate controller BackBoneBoilerplate app

Go to the controller that was just created: back_bone_boilerplate_controller.rb and specify the layout:

1
2
3
4
5
6
7

class BackBoneBoilerplateController < ApplicationController

  layout 'bbapp'

  def app
  end
end

Final Steps

In config/routes.rb a root route is added to point to the controller actiona associated with the backbone app: root :to => 'back_bone_boilerplate#app' At this point we’re nearly done. Unfortunately to get the backbone.png to load I changed the src attribute, of the img tag on line 2, in lib/backbone-boilerplate/templates/example.html to src="/assets/backbone.png" and then from lib/backbone-boilerplate:

1
2
3
4
5
6
7

cp assets/img/backbone.png ../app/assets/images/backbone.png

\\recompile
node build default mincss

cd ..
rails server

If all went well, when you go to http://0.0.0.0:3000 you should see:

While the above approach works, it was laborious and requires rebuilding the application with each change. Watching for changes with node build watch would certainly help. I think the advantage of this approach is that in development you can debug the JavaScript, while in production the javascript is minified and goes through the asset pipeline like any other JavaScript file in a Rails app.