For my smaller applications, I tend to stuff a number of <script> blocks in to the HTML that the user downloads. This makes it easy for me to work with and I don’t have to worry about whether or not the template exists. But when the number of templates gets to be more than 5 or 6 small ones, this gets out of hand quickly. It makes it hard for me to manage them as I have to scroll through a lot of template code. Putting them in external files and then including them in the page with some server side technology helps the developer problem but it doesn’t solve the client problem of having to download all of these templates even if they are never used.

To deal with the issues surrounding templates in HTML files, we can split our templates in to separate files and then use asynchronous calls to our server to load them as needed.

Backbone.View Render Semantics

One of my goals, other than the asynchronous template loading, is to keep the general semantics of a Backbone View’s `render` method. It’s a short list, but it’s an important list as most of the Backbone community expects the render method to work this way.

Semantics are generally important as they give us information about how methods and objects are expected to be used, as well. This, in turn, informs the method signature and behavior. And all of this comes back to the Liskov Substitution Principle (LSP) from the SOLID principles, which tells us that we need to pay attention to semantics so that we can drop in replacements as needed.

The general semantics and method signature of a view’s render include:

• No parameters: the render method shouldn’t require any parameters. You should be able to call `view.render()` and have it work fine
• Chainable: the render method should return `this` so that it can be chained with other method and attribute calls. This is most commonly done as `view.render().el` to grab the element that was rendered to
• Populate `el`, don’t replace it: the render method should populate a view’s `el` with any contents that the view needs to display. It generally shouldn’t replace the `el` as a whole

Aside from these three items, there’s a lot of flexibility in how a view will typically be rendered. There’s also plenty of room for interpretation and divergence from this list. Many applications use a render method signature that takes parameters, or that replaces the `el` entirely. When changes like this are made, it’s a good idea to document them so that people will know why the changes are in place.

The benefit of keeping these semantics, though, is that you can swap out a synchronous, pre-loaded template rendering view with a view that uses an asynchronous template loading mechanism. Or, better yet, you can have an intelligent system that uses asynchronous calls to get the template the first time it needs it, and then uses caching to keep the template around and do synchronous rendering on subsequent requests for this view / template. If the semantics for the view are kept in place, it doesn’t matter how the view implements the rendering. The view can be dropped in or removed as needed, without having to change the surrounding code that uses it.

Simple Async Template Retrieval

We can keep this very simple, to begin with, using jQuery’s AJAX calls to load the template with a callback to do the actual rendering after the template is loaded.

Backbone.View.extend({
  template: 'my-view-template',

  render: function(){
    var that = this;
    $.get("/templates/" + this.template + ".html", function(template){
      var html = $(template).tmpl();
      that.$el.html(html);
    });
    return this;
  }

});

In this example, we’re assuming that the view has a `template` attribute. This attribute represents the file that will be loaded from the server, and that file contain the actual template to be used.

We’re also using a convention of `/templates/{name}.html` for the template location on the server. This can be implemented easily in many different web server technologies. In Sinatra, for example, you can create a “public/templates” folder and put your HTML template files directly in that folder. They will be available without having to do anything more than start the Sinatra server.

When the call to `render` is made, the code makes an AJAX call back to the server to retrieve the specified template. A callback method is provided – and at this point, it assumes a successful call to get the template. When the template is returned from the server, the callback is fired, and the standard render code (using jQuery templates in this example) is executed.

Note that the `el` for the view is populated inside of the callback, but we are still calling `return this` at the end of the function. Even if we chain access to the `el` from the `render` method and immediately add the el’s contents to the DOM, this will still work:

$(“#content”).html(view.render().el);

The reason this works is that we are only populating the `el` with contents. We are not replacing it. When the AJAX call finally returns with the template, rendering it and populating the `el` will show the contents immediately because we’re setting the `html` method of an HTML element that is already attached to the DOM. It’s as if we had called `$(“#content”).html(“<div>some html</div>”);` directly.

Caching Templates

Now that we have a template loading asynchronously, it would be nice to only load it once instead of every time it needs to be used. This will improve the overall performance of the application, from the user’s perspective.

To do this, we’ll need a little more than just some code in the render method. We want to re-use templates that we’ve already loaded, which means we can’t just store the template on the view instance. We need to store it in a place where any view instance can grab a copy of the template if it exists, or have an asynchronous call back to the server done to get the template when needed.

TemplateManager = {
  templates: {},

  get: function(id, callback){
    var template = this.templates[id];

    if (template) {
      callback(template);

    } else {

      var that = this;
      $.get("/templates/" + id + ".html", function(template)){
        var $tmpl = $(template);
        that.templates[id] = $tmpl;
        callback($tmpl);
      }

    }

  }

}

The template manager object has one primary method that we call: `get`. This method takes in a template name to be loaded and a callback method that is executed when the template is found. By using a callback method instead of returning a value directly, we can ensure both synchronous and asynchronous calls will work correctly.

When you call `get`, it will check a hash / object literal to see if the template you want is already loaded using the template name that you provide to make this check. If it exists, it executes the callback immediately and passes the template along. If it does not exist yet, an AJAX call is made with jQuery to get the template from the server. Once the template is loaded, the callback that you passed in will be executed and the template is passed to it.

We can now update our view to use the template manager:

Backbone.View.extend({
  template: 'my-view-template',

  render: function(){
    var that = this;
    TemplateManager.get(this.template, function(template){
      var html = $(template).tmpl();
      that.$el.html(html);
    });
    return this;
  }

});

This isn’t much of a change from the view’s perspective. It’s better encapsulated, though, and a little easier to read. The real work is now being done in the TemplateManager object and we can change how it behaves as needed without having to update our views.

Beyond Simple Caching

There’s one remaining problem that this code has. If you have multiple instances of a view requesting the same template at roughly the same time, multiple AJAX calls will be made – one for each instance of the view. The net result is a slowdown in the application’s performance from too many network calls, and also a visual oddity where the views will appear one at a time as the AJAX calls finish. This can be a pretty big drag on performance and UI responsiveness.

Thanks to some previous digging in to jQuery’s deferred and some help from Steve Flitcroft (@red_square), I was able to solve this problem fairly easily. I put the following code in to my BBCloneMail app, which uses my Backbone.Marionette plugin. It’s not directly implemented in Marionette’s `TemplateManager` but most of what you need is already there. There’s only a few additional lines of code that you need to make this work.

(function(){
  var promises = {};

  // Cache the returned deferred/promise by the id of the template
  // so that we can prevent multiple requests for the same template
  // from making the ajax call.
  //
  // This code is only safe to run synchronously. There exists a
  // race condition in this function, when run asynchronously,
  // which would nullify the benefit under certain circumstances.
  var loadTemplateAsync = function(tmpId){
    var promise = promises[tmpId] || $.get("/templates/" + tmpId + ".html");
    promises[tmpId] = promise;
    return promise;
  }

  // Use jQuery to asynchronously load the template. 
  Backbone.Marionette.TemplateManager.loadTemplate = function(templateId, callback){
    var tmpId = templateId.replace("#", "");
    var promise = loadTemplateAsync(tmpId);
    promise.done(function(template){
      callback.call(this, $(template));
    });
  }
})();

The basic idea is that I’m using a jQuery deferred / promise to fire the callback method from the loadTemplate parameters. To prevent multiple requests for the same template heading back to the server, I’m caching the promises by the template id. When a call to loadTemplate is made through the template manager, I check to see if I have a promise for that template already. If I do, I register the loadTemplate’s callback parameter with the promise. If I don’t, I create the promise and then store it by the template’s Id. Either way, I register the callback with the promise which guarantees it will be executed. Once the template is returned from the server and the promise is fulfilled (resolved), all of the callbacks are fired off with the template data and everything renders correctly.

Note, though, that this code is not 100% safe. If you call the template manager from code that is already asynchronous, you can end up with a race condition where multiple promises are created and multiple calls to get the template are done. You’ll still get all of your views rendered just fine, but this will eliminate the benefit of using a promise to reduce network calls. Calling this code synchronously, though, won’t cause this race condition and the templates will only load once before they are cached and re-used.

See It In Action

If you’d like to see an example of this code in action (including the deferred/promise code from above), check out my BBCloneMail application and it’s source code. You’ll see the update to the TemplateManager’s `loadTemplate` function in the code. When you view the live site, switch to the Contacts view and refresh the page. There will be a small delay in the template loading, and then all of the contacts (which all individually requested the same contact template) will all display at once. Subsequent requests to those areas of the app will be cached, of course, improving the user experience and performance even more.

I understand how daunting it can be to contribute to an unfamiliar project, especially if you recognize that the people most familiar with the codebase would take a lot less time to do so, and probably do it “right” in the first place. I also understand how frustrating it can be to have large, visible OSS project where the number of people complaining/giving feedback far dwarfs the number of people actually contributing.

So what are the rules of OSS engagement for me?

1. Un-actionable complaints will largely be ignored

Mostly because I don’t have too much time to figure out what you want. I generally will respond with prodding, but unless you respond back, I won’t have anything actionable to address. Having an example of the alternative is generally the minimal bar here.

2. Feedback is encouraged. Example client code even better

As an OSS developer maintaining libraries, I often get feedback of “Can it do XYZ?”. Sometimes the answer is No, but I would like to see what your vision of how it should behave.

What should the methods be called? What does the configuration look like? What’s an example of the behavior you’re envisioning?

I don’t even care if the code compiles – just show me an example of what you’re thinking.

3. Unit tests for bugs encouraged, but not required

I just really need code to reproduce the bug. A report of a bug with zero code is much harder to reproduce, and much less likely to get fixed.

If you want to up your odds of getting a bug fixed, the more code I get demonstrating the problem, the better.

4. The more actionable the feedback, the more likely it gets integrated.

Pull requests are at the top end of the “actionable feedback” scale. An email reporting a bug with no code as at the low end. The more actionable the feedback I receive, the more likely I will have the time/energy to respond to the feedback.

However, I won’t turn people away for not giving me a pull request. I encourage it, and often point folks to try, especially if they want to “up the odds”.

I might encourage you to send me a pull request, but I’ll never disparage you for not sending me one.

Then a few of my friends started talking about jQuery’s “deferred” objects and “promises”. I had heard of this previously but had not bothered to learn it. When I finally decided to dig in to it, this morning, I found that it was a much more robust implementation of what I had just written. I found a good article that helped me understand it better, and began working on an update for Marionette to use deferred objects instead of my own code. Addy Osmani also wrote a really good article on MSDN, which sheds even more light on how these work and how to work with them.

My New “Callbacks” Object

Here’s the end result of my digging in to this and replacing my Callbacks object with a jQuery deferred / promise implementation:

// Callbacks
// ---------

// A simple way of managing a collection of callbacks
// and executing them at a later point in time, using jQuery's
// `Deferred` object.
Marionette.Callbacks = function(){
  this.deferred = $.Deferred();
  this.promise = this.deferred.promise();
};

_.extend(Marionette.Callbacks.prototype, {
  
  // Add a callback to be executed. Callbacks added here are
  // guaranteed to execute, even if they are added after the 
  // `run` method is called.
  add: function(callback){
    this.promise.done(function(context, options){
      callback.call(context, options);
    });
  },

  // Run all registered callbacks with the context specified. 
  // Additional callbacks can be added after this has been run 
  // and they will still be executed.
  run: function(context, options){
    this.deferred.resolve(context, options);
  }
});

This is a reduction in code and complexity, by about 75% on both counts.

Things To Note

Backbone.Marionette.Callbacks is a constructor function and you are expected to instantiate it before using it:

var callbacks = new Backbone.Marionette.Callbacks();

callbacks.add(function(){ … });

I’m using a single deferred object and promise object in this code. That’s done on purpose. I am collection an arbitrary list of callbacks, at an unknown time, and want to guarantee execution of those callbacks. By limiting this code to a single deferred object and a single promise, I can do exactly that.

When you call `add`, it sets up a `done` callback using the single promise object. No matter how many times you call `add`, it uses the same promise for that instance of Callbacks.

Then when you finally call `run`, it resolves the deferred object. Calling `resolve` on the deferred object will kick off all of the callback methods that I had set up with the when/then code.

After you call `run` and resolve the deferred object, you can still add callback functions to this Callbacks instance. When you do add more, jQuery is smart enough to know that the promise has already been fulfilled and it immediately executes the callback that you registered.

Specs And Usage

If you’d like to see the specs for this to show how it works with pre-run registered and post-run registered callback methods, you can grab the source code for Backbone.Marionette and look at the Jasmine specs for Callbacks.

I’m using this Callbacks object in Marionette.Application’s “addInitializer” function, which I’m using to guarantee initializer callback execution. Once I get back to working on the client project that I wrote that original code for, I’ll also update it to use the new Callbacks implementation instead of my original implementation, too.

A lot of times the “let’s watch a couple pick a house” type shows highlight the inexperience of the buyers. Buyers tend to focus on the wrong thing, like paint colors or light fixtures, and gloss over things that are hard to change, like room layout. But the best part of the show happens right at the beginning, in a brilliant move to reset the usually unrealistic buyer expectations.

At the beginning of the show, the host asks the buyers a few key questions to identify both their budgetary constraints as well as their aspects of a home they most value. It could be location, layout, lot size, etc. But the buyers always have some sort of dollar amount they won’t go over.

The next step the Realtor does is rather brilliant – they take the first-time home buyers to a house that meets all of their aspects for an ideal home. It has the perfect layout, the perfect location, all the amenities they want, in the right condition, at the right size and so on. The buyers inevitably love this home, at which point the Realtor asks the buyers to guess how much the home costs. Also inevitably, the buyers are pretty far off the actual price, and inevitably their ideal home is typically at least double, if not triple or quadruple the buyers’ budget.

And inevitably, the buyers have sticker shock!

Talking buyers off the ledge

This fantastic approach accomplishes two important goals:

• Convey what perfection costs
• Force a prioritization

The host is never mean about showing the expensive house, but instead presents it as something to aspire towards. This also resets in the buyer’s minds that every house they see from there on out is going to have some subset of their valued aspects.

But instead of discouraging the first time home buyers, this approach tends to force them to focus on what is most important to them in a house.

Resetting expectations

We often have clients that are first-time software buyers, or first-time-with-someone-who-knows-what-they-are-actually-doing software buyers. A lot of what we do in the initial part of the project is framing the project for success. We look at everything the client wants, talk about scope and budget, then reset expectations back down to an attainable level.

What we don’t want to see happen are those buyers that look at dozens or even hundreds of homes looking for that house that checks every single checkbox and comes in at their budget. That house might exist, or it might not, but a lot of time can be wasted searching and searching.

Constraints force prioritization and hard decisions. Having an experienced guide (like that Realtor host) ensures that the buyer understands what’s feasible for their budget, as well as the guiding hand on helping to share experience on what things matter (the electric wiring needs replacing) or do not (the bedroom has shag carpet).

Delivering value is really only half the equation. It’s up to us as developers to make sure the buyer understands what they are getting for their money (or time), relative to what’s out there in the market. If you bought a house in isolation, it’s hard to know whether you’re getting taken to the cleaners. But by having frames of reference (Product XYZ is similar to yours, and took ABC man hours/dollars to build), we can center the conversation around “what’s most valuable to me” instead of “how can we squeeze everything in”.

What do I need to store?

My travel time data collection job needs the URL for the traffic data endpoint for each route that I’ll be tracking. I could have just hardcoded the URL in the script, but I knew that my co-workers would be interested in tracking their routes too, so it made sense to store the list of routes in the database.

I need to store the list of ‘trips’. I define a trip as the reported travel details for a given route and departure time (Josh’s route at 9am, Josh’s route at 9:10am, Tim’s route at 9:10am, etc.). I want to capture the date of each trip so that I can chart the trips for a given day, and compare day to day variation. Even though I really only need to the total travel time for each trip, I want to capture the entire response  from the traffic service (travel times, directions, traffic delay, etc.) so that I could add new visualizations in the future.

Setup

First, I had to install mongo on my laptop. I used the homebrew package manager, but binary releases are readily available.

brew install mongodb

I need to add the route for my commute. I fire up the mongo console by typing mongo. I’m automatically connected to the default ‘test’ database in my local mongodb server. I add my route:

> db.routes.save({
  name: 'josh',
  url: 'http://theurlformyroute...'
})

I verify the route was saved:

> db.routes.find()
{"_id" : ObjectId("4f22434d47dd721cf842bdf6"),
 "name" : "josh",
 "url" : "http://theurlformyroute..." }

It is worth noting that I haven’t skipped any steps. I fired up the mongo console, ran the save command, and now I have the route in my database. I didn’t need to create a database, since the ‘test’ database works for my needs. I didn’t need to define the routes collection – it was created as soon as I stored something in it. I didn’t need to define a schema for the data I’m storing, because there is no schema. I am now ready to run my data collection script.

Save some data

I’ll use the ruby MongoDB driver (gem install mongo) directly (you can also use something like mongoid or mongomapper for a higher-level abstraction). My update script needs to work with the URL for each route:

db = Mongo::Connection.new.db("test")
db["routes"].find({}, :fields => {"url" => 1}).each do |route|
  url = route["url"]
  # collect trip data for this route's url
end

I want to group related trips for a commute, so I create a ‘date_key’ based on the current date/time. A date_key looks like: 2012-01-25_AM, 2012-01-25_PM, or 2012-01-26_AM. Now to store the details returned from the traffic service:

trip_details = TrafficSource.get(url)
db["routes"].update({"_id" => route["_id"]}, {
  "$addToSet" => {"trip_keys" => date_key},
  "$push" => {"trips.#{date_key}" => trip_details}
})

After running for a couple days, this will result in a route document that looks something like:

{
  _id: 1234,
  name: 'josh',
  url: 'http://mytravelurl...',
  trip_keys: ['2012-01-25_AM', '2012-01-25_PM', '2012-01-26_AM',...],
  trips: {
    2012-01-25_AM: [{departure: '9:00', travelTime: 24, ...}, {departure: '9:10', travelTime: 26}, ...],
    2012-01-25_PM: [{departure: '9:00', travelTime: 28, ...}, {departure: '9:10', travelTime: 29}, ...],
    2012-01-26_AM: [{departure: '9:00', travelTime: 25, ...}, {departure: '9:10', travelTime: 25}, ...],
    ...
  }
}

That is *all* of the MongoDB-related code in the data collection script. I haven’t left out any steps – programmatic, or administratrive. None of the structue was defined ahead of time. I just ‘$push’ed some trip details into ‘trips.2012-01-25_AM’ on the route. It automatically added an object to the ‘trips’ field, with a ’2012-01-25_AM’ field, which holds an array of trip details. I also store a list of unique keys in the trip_keys field using $addToSet in the same `update` statement.

Show the data

The web page that charts the travel times makes a single call to MongoDB:

route = db["routes"].find_one(
  {:name => 'josh'},
  :fields => {"trips" => 1}
)

The entire trips field, containing all of the trips grouped by date_key, is now available in the ruby hash route. With a little help from ruby’s Enumerable#map, I transform the data into a format consumable by Highcharts JS.

Production

Just to be thorough, I’ll mention that I had to modify the script for production use. I replaced the `db` local variable with a method that uses the mongolab connection when available, or falls back to the local test connection:

def db
  @db ||=
  begin
    mongolab_uri = ENV['MONGOLAB_URI']
    return Mongo::Connection.new.db("test") unless mongolab_uri
    uri = URI.parse(mongolab_uri)
    Mongo::Connection.from_uri(mongolab_uri).db(uri.path.gsub(/^\//, ''))
  end
end

Conclusion

A couple queries, a single, powerful update statement, and no administration or schema preparation. Paired with the ruby driver‘s seemless mapping to native Hash objects, it is hard to imagine a simpler, equally powerful, persistence strategy for this type of project.

My Backbone.Marionette add-on helps to alleviate this problem by introducing the idea of application initializer callbacks. I’ve talked about these before, and I use them quite extensively in my applications. It really helps to keep my code clean and separated. And I recently realized that there’s an unintended benefit to using initializers like this: explicitly identifying the different stages of starting an application.

The 3 Stages

The three stages that I have identified in my own applications are (in order):

1. The Application’s Foundation
2. The Application’s Initialization
3. A Contextual Start (optional)

Honestly, I could break this down in to a significantly larger number of stages if I really wanted to get fine-grained and detailed. More specifically, it’s the 2nd stage that can be broken down in to pre-initialization, initialization, and post-initialization. But I think as a general place to start, identifying 3 primary stages for an application’s startup is a good idea. It gives us a better understanding of what code should go where, helps us keep our code clean, improves re-usability and flexibility, and will likely help us scale our apps as well.

Stage 1: The Application’s Foundation

Consider this to be the stage where the browser is simply grabbing all of the specified resources from the server and running through the first pass of parsing the HTML, laying out what the static HTML has provided, styling it with the basic CSS definitions, displaying any images and text from the static HTML, and parsing the JavaScript files so it can get ready to run them. I’ve never seen an HTML application that didn’t have this stage in it, because without it, your browser doesn’t have anything to do or show.

This is the one step the everyone does, and does well.

Stage 2: The Application’s Initialization

Every application has an initialization stage. Some are more elaborate than others. At times you can get away with the over-simplified “app” object with an “init” method on it, like many of the sample apps and boilerplate codes will suggest:

App = {
  init: function(){
    // app initialization and startup goes here
  }
}

App.init();

But when you get in to an application of any significant size and complexity, the single “init” method becomes a junk drawer where you tend to throw code because you haven’t yet identified where it should belong. Using a series of small initializer functions for each area of your application will help to alleviate this problem, as I’ve mentioned before. But the real question is:

What goes in the initializers?

Initializers are the bits of code, functionality, data and display that are absolutely required for your app to do it’s job, no matter what part of the application the user is trying to load up and use. They are the bits that must be initialized before the user can do anything meaningful with your application.

If your app has a menu structure generated by Backbone code, and it must always be present in the application, this should be in an initializer. If you’re building a multi-room chat application and you need to list the rooms that the user has favorited in a small block on the screen that is always visible to the user, this should be an initializer. If you’re building an image gallery and you need to load a thumb-nail list of images to show, no matter which image the user is trying to view, this is an initializer.

Other parts of your application code that the user doesn’t directly see may also be contenders for initializers, too. For example, if you have a router that needs to be up and running, the router probably needs to be instantiated inside of an initializer. But – and this is an important but – the routes on that router should not be executed during initialization. Instantiate the router and wait until after initialization has completed to call the “Backbone.history.start()” method, kicking off your route handlers.

Stage 3: A Contextual Start (optional)

I call this stage optional because not every application has a contextual kick off. Sometimes an application only needs initializers. The classic “todo” application is a great example of this. There is no contextual start up for this app. When the application is loaded, it initializes itself with the list of to do items and then it waits for the user to interact with the list.

When a Backbone application uses a router to respond to url pushState or hash fragment changes, though, it does have a contextual starting point. Each route that a user is allowed to bookmark or copy as a direct link will provide the context that our application needs to use, to get the user back to where they want to be. Even if a Backbone app has a router and contextual start, though, it may not be used. If there user hits the root of the application, there may not be any additional code to run for the empty (“”) route. When the user hits a route, though, that route must server up the context and application state that the user expects to see.

The problem that I see in most routed Backbone applications, though, is that they bundle the application initialization with the contextual start. That is, people tend to use the router and it’s callback methods as the sole place to get the application initialized and get the user back to the context of the route that they requested. For trivial applications, this may be fine. The initialization code may be so small that it doesn’t really matter if it’s crammed in to the router. But for any real application with any amount of complexity, this is a bad idea. It couples two very distinct parts of the application startup very tightly, and it can lead to bloated and unmaintainable routers with limited entry points in to the application.

So… what goes in the contextual start, other than just saying route callbacks?

Your route callbacks should be as simple as possible. They shouldn’t initialize your system as a whole. Rather, they should be used to determine the state of the application that the user wishes to see. Therefore, the code that goes in to a route callback should be the smallest amount of code that you can write, to get your application from it’s initial state (the state that it was in when the initializers completed) to the desired state.

In an image gallery application, a user may hit a bookmark that points to a specific image. For example, “#images/4″. The route callback that executes should load the requested image and display it on the screen:

Backbone.Router.extend({
  routes: {
    "image/:id": "imageById"
  },
 
  imageById: function(id){
    var image = imageCollection.get(id);
    App.showImage(image);
  }
});

In a multi-room chat application, a user may hit a bookmark that should take them directly in to a specific chat room. For example, “#backbone”. The route callback that executes should take this room name and call the code that is necessary to enter the chat room.

Chances are this is a fairly involved set of code. You’ll need to load the list of users in the chat room. You’ll need to clear the current chat windows and possibly pre-load recent messages to be displayed. You may also need to change a browser’s websocket event listeners to pick up events for this specific room instead, so that messages for this room can be displayed as they come in.

This is a lot of code to run, and it’s fare more code than should be allowed in a route callback method. It should, then, be encapsulated in an object that is responsible for registering the user as having entered that chat room. This object is then called from the router:

Backbone.Router.extend({
  routes: {
    ":roomname": "chatroom"
  },
 
  chatroom: function(roomname){
    ChatApp.enterRoom(roomname);
  }
});

Furthermore, there’s a high likelihood that the user will be able to enter chat rooms using some interaction on the web page. They may click on a chat room name in their favorite’s list, or they may type in a command like “/join #backbone” in to the chat application’s command area. For any of the the multiple ways to enter a chatroom, the code that is executed should be the same. Having the route callback be as simple and stupid as possible will promote code re-use and allow these three options to be easily implemented. If you only need to call “ChatApp.enterRoom(‘someRoom’)” to enter any room that the user specifies, providing options for how the user enters a room becomes trivial.

Putting It All Together

There are many different ways of stringing these different stages together. There are many different tools that help us to easily create the HTML that we need, include the CSS and JavaScript assets, and kick things off. Some applications are small enough that an initialization step and contextual start step can be combined. Most applications of any significant size, though, should think about separating these in to explicit steps in order to promote better architecture, clean code, and re-use of code.

For me, I’m accomplishing stages 2 and 3 with my Backbone.Marionette add-on. It provides a very clean way to create initializers:

App = new Backbone.Marionette.Application();

App.addInitializer(function(){
  // add some app initialization code, here
});

App.addInitializer(function(){
  // more initialization stuff 
  // for a different part of the app
});

// run all the initializers and start the app
App.start();

At the time of this writing, though, I don’t have an explicit “startup” callback mechanism in place. I have a “start” method on my Marionette.Application object. But right now this really only fires up the initializer callbacks. So, I’m currently using the “initialize:after” event to run my start code. I see this as a problem, though – not of a technical nature, but of a semantic nature. And yes, semantics are important.

App = new Backbone.Marionette.Application();

/* ... initializers go here ... */

// contextual startup
App.on("initialize:after", function(){
  Backbone.history.start();
});

I’m not sure what the new implementation will look like, but it may end up being a “startup” callback method, similar to the initializer methods. The startup callback (or callbacks?) would execute after all of the initialization is done. This will provide a cleaner separation between the two concerns of initialization vs startup. I’ll have to see where this leads me, though, as right now, the majority of what I do for a contextual start is simply call “Backbone.history.start()”, as I showed in the above example.

Given our history with this application and the phases we’ve gone through (initial development, maintenance / enhancements, and complete change in process) I think it’s a good case study in the feasibility of using CQRS style applications.  We’ve made some good decisions and some, err.., not so good decisions that I learned quite a bit from.

A little about the Application

The application is the standard “enterprise workflow” application, where the end users work product goes through several stages and different people do different tasks in order to complete the work.  These type of State-based, processes fit well with the CQRS model, as different events occur on the items in question, which determine what’s required and what action needs to take place.  There were very few “CRUD” type screens, but there were a few.

My definition of CQRS

Separate models for reads and writes

I think my definition heavily influenced how we built this application.  Notice there’s nothing there about messaging, MSMQ, distributed architecture, Event Sourcing, etc…  We did use some of these technologies and architecture in this app, however they are not required for CQRS. It’s simply an acknowledgement that the write model looks and behaves differently from the read model.  This simple acknowledgment can have profound impact on the way build your application and your ability to solve problems for your end users (and if you’re not doing that, well…)

Not Everything has to fit into the CQRS Mold

I think one common mistake made while building applications with CQRS is forcing every single functionality through the CQRS square hole.  There were parts of our application that were very simple and did not require the complexity of CQRS.  That’s right, CQRS is complex. I said it.  Commands must be created, validated and executed.  Events must be fired and handled.  Usually the complexity is mitigated by the simplification of the domain models.  But if you’re domain models are already simple, then a simpler approach will yield better results.

Do Not Start With a Distributed Architecture

When most people see examples or presentations of CQRS, it usually includes some sort of distributed architecture, where the commands and messages are put on some sort of queue, like MSMQ, RabbitMQ, or whatever.  And there can be some great advantages when using these techniques, mostly scalability.  Using a distributed approach increases the complexity to a whole new level.  However it is simply not necessary, at least at first.

For our application, we created a very simple in-process bus.  Meaning, execution of the command and all events were executed on the same thread.  We used an IOC container to find all of the event handlers based upon the event class type. It is a web application, so the threads were handled by the web server.  For a typical event, we had 4-6 event handlers, updating 4-5 different database tables.

There were some real advantages to starting with an in-process model.  We were able to get faster feedback when errors came up, and it was easier for us to find and fix bugs. This enabled us to build the application faster and get it into the end users quickly.

Once we were in production, we started recording metrics on the how quickly requests took and where there were performance issues.  What we discovered was the only real performance issues were when the users would perform batch operations, the same operation on many records (set based operations are nearly impossible with this architecture, or at least I haven’t figured out how to do it).  So once we determined which functions were commonly done in batch form, we moved them out of process with NServiceBus.  I think that’s one of the areas where this approach shines.  Moving these out of process, when necessary, is really simple.

Move work out of process only when necessary

Think Creatively about your Domain and View Models

One of the mistakes we made was the design of our domain models.  From a purely DDD / aggregate root / ubiquitous language, there was really only one AR.  But having just one AR means it’s REALLY big.  In retrospective, we should have been more creative with how we define the AR into something different.  Honestly, while this application has come complicated logic, I don’t think that we needed to solve it with a Domain Driven approach.  However, the library we used (per client request) required us to do so.  I don’t have a solution for this yet, when I do I’ll let you know (hint, I think the Actor/Model patter Chris Patterson has been talking about is a better fit).

Event Sourcing … Meh

Event Sourcing has been probably the most misunderstood and miscommunicated implementation strategy surrounding CQRS.  Like other pieces of the architecture, it’s an implementation choice, not a requirement for building this type of application.  Honestly, I was never sold on the concept before this project.  On this project I don’t think we got any of the benefits touted from using it.  To be clear, I think one of the problems is our implementation of the Event Sourcing persistence.  We are currently storing a binary copy of the event.  Which means that event lives in perpetuity and we can’t change it nor get rid of it. If it was serialized with something like Protocol Buffers I think we would have more flexibility, but I haven’t tested that yet so I can’t say for certain.

The biggest feature about using ES is the ability to replay the events and rebuild the state of your view models.  This sounds really cool, you get the ability to create new views in your application (like new reports)  and be fully avaialble with all data as if they were there from the beginning.  The only problem is that we’ve only done that once in the entire lifetime of the application.

But at the same time, using the event sourcing approach hasn’t really cost us anything either.  In the end, for this application, I would have been happy with just a Key-Value store of data or a document database for aggregate persistence.

Overall, I’m pleased with the application.  Like I said there was a big change in business process and we’ve been able to make those changes very fast in my opinion.  This is mainly due to the extreme decoupling you get from the way views are represented to how a domain object is changed or created.  I will likely continue to create applications of this type using this approach.  With a few changes underneath of course:)

Depending on how the application is architected, and when the persons collection is expected to be loaded, there are a few options that I can see for solving this problem.

Option: Fetch The Model From The Server

The most basic option, and probably the easiest to deal with, is just to fetch the model from the server based on the id parameter that you get from the route.

In this case you just need to create a new model instance, set the `id` attribute on the model directly and then call `.fetch` on the model. It will make the trip to the server to get the data. You can either listen to a “change” event on the model or provide a “success” callback in the fetch method, to know when the data has been returned so that you can load it in to your view and display it as needed.

personById: function(id){
  var person = new Person();
  person.id = id;
  person.fetch({
    success: function(model, response){
      App.showPerson(model);
    }
  })
}

The major problem here is that you have not loaded the persons collection at all. If the persons collection is expected to be loaded because it’s being used for something related to displaying or working with the individual person model, then this option might not work for you. You could run some additional code to load the collection separately (asynchronously, since it works that way by default). This might help get around any potential issues of needing the collection.

Option: Use The Collection’s “reset” Event

Another easy option may be to load the collection when the request for the route is made. You would set up a new collection instance, bind a callback function to the collection’s “reset” event, and then call `.fetch` on the collection. The callback method would be responsible for retrieving the specific model from the collection and then creating and displaying the view.

personById: function(id){
  var persons = new PersonCollection();

  // note that "bind" is now "on" in Backbone v0.9.x
  persons.on("reset", function(collection, response){
    var person = collection.get(id);
    App.showPerson(person);
  });

  persons.fetch();
}

There are some potential issues with this solution, though. If you already have the persons collection loaded, then you’re going to load it a second time just to get the one model from it. To mitigate this problem, you would need two different entry points in to the display of the person: one for when you hit the route directly through a bookmark, and one for when the user is already in the app and navigates to the person display through other means.

Having two different entry points in to this part of the app may not be a bad idea. This largely depends on how the application is architected. You wouldn’t want to duplicate all of the code that sets up the display of the person’s details in both of the entry points, but you wouldn’t want to have a bunch of ugly if-statements in that code to determine how to set things up, either. Some simple abstractions of the common bits would help keep this code manageable.

Option: Building An “onReset” Callback

UPDATE:

FYI – I have an updated version of this code available in my Backbone.Marionette plugin, as Backbone.Marionette.Callbacks. It reduces the code and complexity significantly, and also eliminates the race condition issues that I mention below. Be sure to use that Callbacks object instead of the code I’ve listed here.

:END UPDATE

The third option that I can think of – and the one that I implemented for this particular client project – is a variation of using the collection’s reset event. The idea is to build an “onReset” callback system that is aware of whether or not the collection has already been loaded or is still waiting to be loaded.

If you have the persons collection being loaded from some other application initialization code, then you don’t necessarily have the ability to use a simple reset event as shown above. You could try to use the reset event, but there’s a race condition that is introduced in low-latency, high speed networks (i.e. your local development machine).

If you don’t control when the `.fetch` method is called, then you may end up binding to the reset event after the collection has already been reset. In that scenario – which is very likely to happen when working in a local development environment – your view for the specific person model will never get displayed.

The solution I came up with is to have a callback mechanism built in to the collection, that pays attention to the collection’s reset event and knows to either wait for the reset event to be fired, or to fire the callbacks immediately because the collection has already been loaded. I’m calling this an “onReset” callback, for lack of a better description at this point.

The code to use the onReset callbacks would look something like this:

// App.js
// ------
// some initialization code that happens elsewhere, using Backbone.Marionette

App.addInitializer(function(){
  App.persons = new PersonCollection();
  App.persons.fetch();
});



// PersonRouter.js
// ---------------
// The router callback that needs to get the person

personById: function(id){
  App.persons.onReset(function(collection){
    var person = collection.get(id);
    App.showPerson(person);
  });
}

In this setup, adding an onReset callback guarantees the callback’s execution. If the collection has not yet been loaded, then it stores the callback and waits for the reset event to fire. If the reset event has already been fired, then it simply executes the callback immediately. Either way, your callback will be executed and you will have the collection available when it does.

Race Condition Reduced. Eliminated?

Here’s the implementation for the onReset code. It’s generally functional and I haven’t yet run in to any problems, yet.

OnResetCollection = Backbone.Collection.extend({
  constructor: function(){
    var args = slice(arguments);
    Backbone.Collection.prototype.constructor.apply(this, args);

    this.onResetCallbacks = [];
    this.on("reset", this.collectionReset, this);
  },

  onReset: function(callback){
    this.onResetCallbacks.push(callback);
    this.collectionLoaded && this.fireResetCallbacks();
  },

  collectionReset: function(){
    if (!this.collectionLoaded) {
      this.collectionLoaded = true
    }
    this.fireResetCallbacks();
  },

  fireResetCallbacks: function(){
    var callback = this.onResetCallbacks.pop();
    if (callback){
      callback(this);
      this.fireResetCallbacks();
    }
  }
});

You can then extend from OnResetCollection instead of Backbone.Collection to get this functionality.

I still worry that there’s a potential race condition in between the logic and the pop’ing of items off the array. I’ve travelled every logical path of execution for the asynchronous call and onReset call that I can think of, and I can’t find any issue. I would love to hear from someone more experienced with race conditions in JavaScript, though.

JavaScript File Organization: You’re Doing It Right

The reality of JavaScript in general web-based development (not talking about server side at all, here) is that it makes absolutely no difference how your files and folders are organized. Zero difference at all… at least as far as the JavaScript runtime is concerned.

The only thing that matters is that you include your files in your HTML with a <script> block, correctly. You also need to pay attention to which order they are included in most cases, to make sure things are defined before they are used (tools like RequireJS and other script loaders and module definition / loaders help with this).

What does this really mean? It means:

You’re Doing It RIGHT!

Yup. You’re doing it right, because it doesn’t matter how you do it.

What does matter, though, is that you and your team (if you have a team) pick an organizational convention and stick with it. It’s actually very more important for your team to have a good file and folder structure for your JavaScript. But you don’t need to worry about what that structure is. Pick a standard and use it. When it fails (and it will), re-work your structure so that it works within the newly understood constrains and move on.

Of course, that fact that your browser and it’s JavaScript runtime don’t care about your file and folder organization doesn’t I’m without my opinions on how files and folders should be organized.

For example…

M, V and C Folders

A lot of people organize JavaScript MV* application files in a folder structure like this (BackboneJS in this case):

To the best of my knowledge, this folder structure is based on the “models”, “views” and “controllers” folder structure that was popularized by Ruby on Rails. Sure others may have had it first, but it was Rails that made it popular. Other MVC framework followed suit and demanded that you put your controller objects in the controllers folder, your model objects in your models folder, etc. But unless you’re Rails (or another framework that wants to be like Rails), this folder structure is stupid.

I’m pretty sure that Rails uses this folder structure to assume the types of objects that are found within the files. And I know for sure that it uses file names to assume the class that will be defined within the file. That is, when rails sees a file called “/app/controllers/foo_controller.rb”, it expects to find a class called “FooController” and it expects that this class will inherit from some Rails controller base class. If these expectations are not met, errors are thrown to say so.

I understand why Rails does it this way: file and folder based conventions make it easy to assume what a file will contain, and that makes it easy for the runtime to optimize for performance when pre-loading and caching the code contained within the files. This makes sense to me in Rails because the convention is based on good ideas for optimizing the way Rails works and the way it looks for files and how it loads them.

But, unless you’re Rails or another framework that wants to assume certain files in certain folder contain certain code, this is a terrible way to organize files.

The Junk Drawer

There are some good examples of other standards along this line. For example, I tend to follow the convention of a “public” folder with “css”, “images”, and “javascripts” folders. But honestly, this folder structure exhibits many of the same problems of being stupid that organizing files in M, V, and C folders does.

The real problem with these types of folder structures is that they become junk drawers. Even DHH and the Rails core team recognize that this is a poor folder structure outside the confines of Rails+Ruby code. That’s one of the reasons they added the Asset Pipeline in Rails 3.1. DHH even called the “javascripts” folder a junk drawer, very directly, in a RailsConf keynote in 2011 (or was it 2010?) – complete with a slide showing a drawer full of junk.

With any application that moves beyond a trivial number of files, these content-type, mime-type, code-type and general type-based folder structures turn in to a bloated pile of junk that is very difficult to sift through. Who wants to look at a folder with 20, 50 or 100 files in it, when you only really care about 2, 5 or 10 of those files?

And what happens when you suddenly have an object type that doesn’t fit your pre-established conventions? You end with a “lib” folder, like Rails, which becomes the ultimate junk drawer. “It’s not a model? It’s not a controller? It goes in lib.” – no matter what the actual functionality contained within the file is. The “lib” folder is asking to be a junk drawer… demanding it, really. So, do you follow that same junk drawer convention for non-M, V or C type-based files in your JavaScript apps? That doesn’t any make sense to me.

How I Organize Files: By Functionality

I prefer to organize my JavaScript files the way I used to organize my C# files in .NET projects: by functional area of the application. That is, I group files together in folders based on the area of the application that they facilitate.

For example, my BBCloneMail application has the following folder structure for it’s JavaScript:

Note that I’m still using the “javascripts” parent folder, but underneath of that I’m organizing by functional area of the application. In the root “javascripts” folder, are the primary application files – the ones that define the overall application bits. In the “mail” folder are all of the files that relate to the “mail” application. And, in the “contacts” folder are all of the files that relate to the “contacts” application.

I don’t care what “type” is contained in the file. That’s a completely irrelevant way to organize files to me. It makes no sense for me to organize files this way because many of my files contain more than one “type” of object. For example, I often put very simple model, collection and view definitions that are very closely related, in the same file.

Why Type-Based Folders Might Be A Good Idea

In spite of all my over-opinionated hand-waiving above, there are some good reasons to use type based folders in JavaScript. One reason is asynchronous file loading based on conventions.

You might have a JavaScript app that makes use of a templating engine (of which there are dozens, these days). It’s not always a good idea to pre-load every possible template in to the user’s browser, for download size and performance reasons. Sometimes it makes sense to fetch the template that you want the first time it’s requested.

To do this, it might make sense to use a convention to retrieve the files. I’ve seen several Backbone apps that use a jQuery selector to load files, as one example of this. When a Backbone view specifies a template as “#my-view-template”, the application’s template manager would make a request to the server to load something along the lines of “/templates/my-view-template.html”.

If you’re trying to organize your templates in a functional area of the application, you’ll have the added overhead of inserting the functional area folder name, such as “/mail/templates/inbox-template.html” for an “Inbox” view in a “mail” app, trying to load an “#inbox-template” jQuery selector as the template to render.

So… there’s at least one possible reason to use a type-based folder. I would still stick the type-based folder name under my functional area, though. I don’t want to mix up the templates between my functional areas of the application, by accident.

A Tradeoff: Folder Names vs File Names

Here’s one potential trade-off for using functional folder names vs type based folder names: you might have to specify the type in the file name. For example, if you have a model type name “Person”, a collection type named “Persons”, and a view to represent a single person or a collection of persons, what do you call that view? If you’re organizing things by type, you can call every “Person” and “Persons”

This can be very confusing. I was recently working with a client who was using an editor that only shows the file name for the open files, and none of the folder path for the files. He ended up with 4 “person.js” files in his open file list. Which one was the Person model, view, router, or controller file? We had to open each file to find the one we needed, every time. So, we took the hit on the file names. The “person.js” file contained the person model, while “persons.js” contained the collection, “personviews.js” contained the view definitions and “personrouter.js” contained the router. Thus, we’ve moved the need for specifying the object type from the folder structure (with all it’s bad ideas for using that) to the file name.

I’ve read at least one blog post that advocated using type-based folder names specifically to avoid the “ugliness” of having type names in your files. I seriously laughed out loud when I read that. Whether the type is in the file name or folder name is a moot point. You’re likely going to end up specifying the type somewhere. I would much rather have it in my file names because it’s easier for me to see things grouped together based on functionality, than based on the type of object contained in a file.

It’s Just An Opinion, And A Loose One At That

My own use of these conventions and ideas is rather loose at this point. I don’t stick strictly to anything, and I mix and match based on the project type, number of files and other constraints that a given project presents. Because, like I said at the top of this egregiously long post, you’re doing it right.

No matter what file and folder structure you pick for you JavaScript apps (assuming you’re using a suite of libraries that doesn’t force you in to a specific folder structure), you’re doing it right. JavaScript in a browser environment really doesn’t care what the file and folder structure is. But that doesn’t mean we as developers shouldn’t care. Pick a file and folder structure that fits the constraints of your application and change the structure as your app’s constraints change.

However, the tooling just doesn’t do it justice. From the download, installation, integration and CLI experience, it always feels like (in Windows land) that you’re playing in someone else’s back yard.

Over the next few posts, I’m going to compare the experience of using Git with that of Mercurial, who has, in my opinion, lesser features, but a much MUCH better experience.

The Mercurial download experience

Let’s look at searching and downloading the Mercurial client. When I google “Mercurial” or “Mercurial Windows” or “Mercurial Windows Download” or variants, two of the top results are the official Mercurial home page, or the official Windows client, TortoiseHg.

From there, I want to download Mercurial. Both websites offer very clear ways of doing so. The Mercurial site:

And the Tortoise Hg site:

Two very large “download buttons”. These buttons are interesting in that:

• They link directly to the file to be downloaded.
• They both link to the exact same installer
• They know what OS you’re using, and display the correct installer accordingly

TortoiseHg is the official Hg client for Windows, and includes:

• The command line interface
• Windows Explorer integration
• Visual tools (Workbench etc.)
• Visual Studio integration
• Merge tools

It’s a completely out-of-the box client that includes EVERYTHING that you might need to run Mercurial, all in one package, and consistently presented to the end user.

Next, let’s look at the Git download experience.

The Git download experience

When searching for Git downloads, you’re primarily directed to one of two sites – the official Git site, or the official Git tools site for Windows, hosted on Google Code (and also GitHub, curiously enough). The Git site is clean enough:

Except I have 3 download links instead of one. Not a big deal most of the time, but already choices are presented to the end user over the Mercurial site. Clicking on the Windows link takes me to this page:

Instead of linking me directly to the installer file to download immediately, I’m directed to the downloads page of the Google Code site, where I am presented with yet even more options. There is nothing in this screen that screams “THIS IS THE INSTALLER YOU WANT IGNORE THE OTHERS”. As someone new to Git, how do I know which to choose? Probably the first one, and most people would choose the first one, but presenting choices here is pointless and confusing.

Not to mention, I’m whisked away to a site that has nothing to do with the original Git site. The official Git site didn’t mention “msysgit” but now I’m on the msysgit Google Code site. Even more confusing is that the file has the name “preview” in it, and the installer is labeled as “Beta”. So is the right one or not? I might be inclined to search for the last “good” release and not a beta/preview one.

The Git installer is also less featured than the Mercurial one. The official Git Windows tools include:

• Windows Explorer integration (very limited)
• A CLI through the Git Bash or directly in a command prompt
• Visual tools (OK tools)

However, I typically don’t point folks to the official Git client. Instead, I point them to Git Extensions, a more fully featured toolset that includes:

• Windows Explorer integration
• Visual Studio integration
• Richer visual tools
• Bundled merge tool
• Bundled Git installer

This isn’t the official Git Windows client, so you basically have to know it exists to find it. Almost none of the online tutorials recommend it, even though it matches much more closely to what Mercurial provides out of the box.

Improving the Git download experience

In two easy steps:

• Have the official Windows client be as full featured as the Hg one. Could just start with Git Extensions and go from there.
• Copy the Mercurial website’s behavior

In short, prefer Simplicity over Choices. Have defaults, and obvious ways to get to the non-defaults.

The Problem

My employer is moving to a new location in a part of the city that I’m not very familiar with. I have no idea what the traffic patterns are like, and I’m wondering when to leave for work in the morning. I tried looking at various web mapping services. Some factor in current traffic, but I couldn’t find any that could tell me historical traffic information, making it impossible to make a decision about departure time in advance.

Idea

Tracking historical travel times for the world would be a huge task, but what if I could create a small history of my specific route? The TomTom Live Traffic Route Planner site can give me directions to work, and estimate travel time based on current traffic conditions. I discovered that it returns a lot of the trip information from a single AJAX call. A quick copy/paste of the URL to curl confirmed that I could repeat the call and get the same data. I just need to hit that endpoint at various times in the morning and store the results. Later, I’ll be able to analyze the results and determine the best time to leave for work.

Keep it Minimal

Now I have an idea, but I don’t know if it is worth pursuing. I don’t know if the data I’m getting is accurate (it worked once or twice from curl now, but maybe the long URL contains some session id that will expire?). I don’t want to invest a lot of time or money in building it out. Also, it’s 6pm Tuesday night, I only have 3 more days left this week before I make the commute to the new office. I need to start collecting data as soon as possible; hopefully Wednesday morning. It’s time to write some code. Fast.

This is where the quality of available tools really make an impact. To name a few:

Ruby – low ceremony scripting with a vast ecosystem of libraries to accomplish common tasks.

HTTParty – crazy simple library to call a URL and get a Ruby hash of the response data – no parsing, no Net:HTTP.

Heroku – There is no better option for hosting applications in the early proving stage. Create and deploy a new app in seconds, for free. The free Heroku Scheduler add-on lets me run a script every 10 minutes in my hosted environment — exactly what I need for my data collection.

MongoDB – natural fit for persisting an array (the trips calculated every 10 minutes) of ruby hashes (responses from traffic service). No schema, no mapping, no fuss.
 
MongoLabs – free MongoDB hosting on Heroku. One click to add, and I have a connection string for my own 240MB in the cloud. Sweet.

By 11pm Tuesday night, my script is running in the cloud, ready to start collecting data Wednesday morning. I’m not going to spend any time on building a UI until I know if the data collection works.

Checkpoint

On Wednesday night, I use the mongo console to review the trip data that was collected in the morning. I see that the trip duration changes for each request, which gives me hope that I’ll have meaningful data to answer my question. However, I also notice that the reported “traffic delay” time is always zero. I’m a little concerned that my data source isn’t reliable. I’m glad I haven’t invested too much yet. At this point, I can just write off the time as a well-spent refresher of MongoDB.

Further exploration

I’m still curious to see a visualization of the data. I decide to spend a couple hours to see if I can build a minimal UI to chart the departure vs. duration times. Again, the available tools gave me fantastic results with minimal effort:

sinatra – an incredibly simple DSL for exposing your ruby code to the web. All I needed was a single endpoint that would pull data from mongo and dump it to the client to render a chart. Anything more than sinatra would be overkill, and anything less would be tedious.

Highcharts JS – amazing javascript library for generating slick client-side charts. A ton of options (including the very helpful datetime x-axis), well-documented, and free for non-commercial use. I didn’t have a “go-to” option for client-side charting, so I had to do a quick survey of what was available. This is the first one I tried, and it left me with absolutely no reason to look at others.

After a couple hours (mostly to learn Highcharts), I have my chart and a potential answer (leave before 7:45, or after 9):

Conclusion

I essentially spent one evening writing the data collection script, and another night building the web page that rendered the chart. I’ve proven to myself that the idea was sound, if not the data source. I will continue to poke around to see if I can find a more reliable API for travel times, but otherwise consider this project “done.” In years past, getting to this point would have meant a lot more effort on my part. It is awesome how much of the hard work was done by the people and companies that support a great developer ecosystem.

git remote | while read a; do
> git remote rm $a
> done

This took care of removing every single remote that I had in my ‘git remote’ list … including the “origin” that I actually did want, now that I think about it. :P

But my bash-scripting-fu is terrible, so I couldn’t figure out how to not destroy all of them. I tried adding an if statement in there to check if it’s not “origin”, but couldn’t get it to work. Anyone got some better bash-chops than me, and can point out how to do this?

In many of our systems, we consume flat files from third party integration partners. We take these flat files and deserialize each line of the file into a distinct message, so we first tried to do something like this:

ProcessLineInFileMessage[] messages = ConvertFileToMessages(file);

Bus.Send(messages); // Sends all logical messages in 1 transport message

The problem we hit was that the unit of work boundary in NServiceBus is around the transport message, not the logical message. In a file of a million lines, that’s a million logical messages bundled together into one single transport message, and one transaction boundary! We had assumed that the overload for sending multiple messages was simply a helper method that encapsulated a “foreach”. Well, no, it doesn’t. All the messages are wrapped in an envelope known as a “transport message”, and it’s the transport message that defines the unit of work boundary (since that’s the physical message sitting in the queue).

Needless to say, we saw database connection timeouts pretty much immediately. Instead, we modified our use of the bus to instead send one logical message per physical transport message, with our friend the “foreach”:

ProcessLineInFileMessage[] messages = ConvertFileToMessages(file);

foreach (var message in messages)
    Bus.Send(message); // Send one logical message at a time

So when would you use the overloads for sending multiple messages? I’m not sure, but I’ll update if I find out!

Speaking of posh-git, there are a bunch of new goodies in my master branch including tab expansion for remote branches, support for the Nuget console, and easier integration with posh-hg. I’ll post more details with the upcoming v0.4 release.

To “fix” the problem, I switched my code from a “change” event to a “keyup” event for the text boxes … bad idea!

Browser Auto-Fill

Most (if not all) browsers have an auto-fill feature, these days. I use it in Chrome a lot. It saves me a few seconds here and there and generally makes it easier for me to fill in the same repetitious information across websites.

But there’s a problem with auto-fill. It doesn’t fire the “keyup” event. It fires the “change” event for the things it fills in, but not “keyup”. This resulted in the data being truncated when it populated my Backbone model.

In the above screenshot, since i had typed “deri” in to the email address, the email that is stored in the Backbone model would only be “deri” – and that’s obviously not a valid email address.

Easy To Fix

There are a number of very easy ways to fix this.

• Validate the email address format
• Use a combination of “blur”, “change”, and “keyup” events
• Delay reading the data until the user clicks the “Purchase” button
• Use a KVO (key-value observer) plugin like my Backbone.ModelBinding and let it deal with that for you

For the quick-fix to get my site working properly, I went with the combination of “change” and “keyup” events. I’ll be re-writing my purchase form sometime soon, and will likely delay reading the values until they click the “Purchase” button. I’m also going to put in better email address validation to make sure it at least has an @ and a . in it, and I’ll likely use the HTML5 “email” field and see how that will help me, if at all.

Not So Easy To Automate Testing

When I originally made the change to use the “keyup” event, I did all my usual testing – which did not include using auto-fill. All of my testing passed, so I pushed the site live.

How do you test for bugs like this in an automated way? Is there even an automated way to test a browser’s auto-fill? I’d like to avoid mistakes like this in the future, and some automated testing around it would certainly be nice.

Lesson Learned: Don’t Rely Solely On “keyup”

There certainly are some great things you can do with the “keyup” event – like autocomplete, for one – but it’s not a great idea to only use this when there’s a chance that the browser’s auto-fill will be used.

And unfortunately for me, 2 of my customers ran in to this problem before I got it fixed. It makes it very hard to email someone the download instructions for their order. One of them contacted me and I corrected the order. But the other purchaser has not yet contacted me, and I don’t have enough information to figure out who they are, so I can’t track them down. I hope this person realizes that they didn’t get their order, they contact me. My contact info is all over the site… so … hopefully … the lessons learned for running an e-commerce site are sometimes painful.

This is a bit of a problem. If I send all of the JavaScript that is uses to run any part of the system to the browser, then it’s possible that a clever user could enable it and do things they aren’t supposed to do. Of course, when they send a request back to the server, the server will verify they can do what they requested and block it… but the user should not be able to even try to do things they aren’t authorized to do, in the first place.

The solution that I’ve come up with, at a very high “functional area of the application” level, is simply not to send JavaScript to the browser if the user is not allowed to use it. This keeps me from having to do extra security checks in the browser, keeps the download for the user smaller and generally makes the app appear snappier because there is less code to run. More importantly, though, it keeps the user from being able to try things they can’t do.

Modules To The Rescue

I’m facilitating this through the use of modules in my JavaScript – both “module” as in the JavaScript module pattern, and “module” as in a packaged functional area of an application.

It’s worth noting, though, that I’m not using AMD (asynchronous module definitions), RequireJS, or any other “module” framework for JavaScript. The debate about the “right way” to do modules can rage on all it wants. I’m content with simple immediate functions and logical groupings of files while the more intelligent and invested people in the JS community figure that all out.

Each of the functional areas of my application is built in it’s own set of files. Each area has multiple files that it is composed within, but the files are grouped together for easy identification of what makes up a module. I also have an HTML template for each module which provides the needed <script> tags to include the code for that module.

Only Render What Is Needed

When I need a functional area of my site to be sent down to the user, I tell my server side template language to include the correct HTML file. For example, I’m doing this in an ASP.NET MVC application:

@{var user = (CustomPrincipal)User;}

@if (user.Can("locations/manage"))
{
    @Html.LocationManagementScripts()
}

@if (user.Can("locations/search"))
{
    @Html.LocationSearchScripts()
}

In this code, I’m checking to see if the current user is allowed to manage locations. If they are, the extension method “LocationManagementScripts” is called. This in turn renders the “LocationManagementScript.html” file at this point in my HTML layout. That file contains all of the <script> tags for the location management JavaScript app. In the same way, I’m checking to see if the user can search through locations, and running the same basic process if they can.

Self-Initializing Modules

When a functional module is included after passing one of these checks, it needs a way to get itself spun up and started so that it can do it’s magic. It may need to render something on to the screen. It may need to register itself with the application’s event aggregator, or any of a number of other things. This is where my Backbone.Marionette add-on comes in to play for my Backbone apps.

Marionette has an explicit concept of an “initializer” tied to it’s Application objects. When you create an instance of an Application object, you can call “app.addInitializer” and pass a callback function. The callback function represents everything that your module needs to do, to get itself up and running. All of these initializer functions – no matter how many you add – get fired when you call “app.start()”.

myApp = new Backbone.Marionette.Application();

myApp.addInitializer(function(options){
  var myView = new MyView({
    model: options.someModel
  });
  MyApp.mainRegion.show(myView);
});

myApp.addInitializer(function(options){
  new MyRouter();
  new SomeOtherView().render();
});

myApp.start();

Each functional area of my application has it’s own initializer function. When a functional area has been included in the rendered <script> tags, the initializer gets added and when the “start” method is called, the modules for that functional area are fired up and they do there thing.

A Composite App, And Sub-Apps

One of the tricks to making all of this work, is that I need to have a primary “app” object that all of my modules know about. In the above example, the “myApp” object is this. Each of the modules for each of the functional areas has direct knowledge of this object and can call public APIs on it – including the “addInitializer” method.

A better example of what a module definition and initializer might look like, would be this:

// --------------------
// app.js

myApp = new Backbone.Marionette.Application();

myApp.addRegions({
  mainRegion: "#main"
});


// --------------------
// locationSearch.js

(function(myApp, Backbone){
  var SearchView = Backbone.View.extend({
    // ...
  });

  myApp.addRegions({
    searchRegion: "#search"
  });

  myApp.addInitializer(function(){
    var view = new SearchView();
    view.render();
    myApp.searchRegion.show(view);
  });
})(myApp, Backbone);


// --------------------
// index.html

<script language="javascript">
  myApp.start();
</script>

In this example, I’m using the simple JavaScript module pattern to encapsulate my search functionality. I’m also providing an initializer for the module that instantiates a search view and shows it to the user using a region manager.

Each of these functional areas is basically a sub-application. Many sub-applications are used to compose a larger application and overall experience for the user. The composition of a larger application through various modules that are included / excluded based on some criteria are what really make this a composite application.

I also included the final call to “myApp.start()”, showing that I do this from my main HTML page and not from my JavaScript files. This provides a single point of entry for all of the registered modules, no matter which modules are registered. The “myApp” object really doesn’t care which modules are registered, honestly. It doesn’t need to care. It only needs to execute the initializers that happen to be present. If none are present because the user didn’t have permission to do anything, then nothing happens when this method is called and the user won’t see anything.

Security: Don’t Let Them See It If They Can’t Do It

If the security check to see if the user is allowed to use the location search feature fails, the rendered HTML won’t include the <script> tags for the “locationSearch.js” file. If this file is not sent down to the browser, then it will never register itself. If a module has not registered itself for initialization, it’s views won’t show up on the screen and the user won’t be able to try and use the feature. Further, the user won’t be able to “view source” on the page and find any stray JavaScript that they shouldn’t be able to use.

It’s Not Always That Easy

Of course there are other security concerns that are not this simple. When a functional area is closed off by authorization, it’s easy to keep things clean like this. We can compose the application at run time simply by including the right files and letting the code in those files register themselves for initialization. But when we have a functional area of the system that has finer grained authorization and permissions associated with it, things get a little more tricky.

I’m still learning and exploring this space. I have some ideas and am going to be implementing some of them soon. If anyone out there has any experience in handling finer grained security needs in JavaScript apps, I’d love to hear about it. Post links to your favorite resources for this, in the comments.

Except that it’s all wrong. Every last bit of it.

… or, at least the handful of people who inevitably respond with comments like “no one does REST correctly” and “that’s not REST” would have me believe.

Your Doing It Wrong

Frankly, I’m tired of this pedantic, obnoxious response. The same people that continuously say “no one does it right” can’t seem to tell me why it’s wrong or point me to a resource that does do it right.

Here’s the deal. If everyone does it wrong, then write a blog post or a book or a screencast or something that tells the world how to do it right. Until you can point people to what REST actually is, stop telling everyone that no one does REST right.

My Journey Into Being Wrong

I have work to do. I’m obviously going to do it wrong, but frankly, I don’t care at this point. My system will work just fine, just like every other “That’s not REST” system out there.

To help me learn how to be wrong, though, here are a few resources that I’m going to be checking out:

Get Some REST: A work in progress book from Steve Klabnik. I signed up for the mailing list and am eagerly waiting. There’s rumor that Steve might actually be doing it right.

RESTFul Web Services – I bought the Kindle version last night. It’s the book DHH says everyone should read and a few others also suggested it.

REST In Practice – I’m probably going to buy this one, too. It seems to have the “30,000 foot view” that I’m also looking for, according to one reviewer on Amazon.

If anyone else has any resources for how I can learn non-REST, RESTful best practices, I’d love to know.

(end of rant)

… Updates

Thanks to everyone who is responding in the comments here and via twitter. I’m going to update this post with the resources that people are pointing me to, from Twitter.

The majority of my experience with anything REST-like comes from working with Ruby on Rails, and mimicking it’s handling of HTTP verbs with Sinatra application. I know this isn’t anything near correct, which is why I’m looking to find good resources.

I’ve been getting a lot of “you don’t need to do pure REST” responses – which I agree with. One of my problems, though, is that I’m having a hard time figuring out what “pure” REST is so that I can know the difference between that and what I’m actually doing.

Sebastian Lambla points out that it’s not the entire REST community that pulls this crap. There are plenty of people that are helping, providing good information and creating good resources to throw at people like me that are wanting to learn. My problem is not with the REST community as a whole, but with the people that behave in the way I’ve pointed out, here.

The most consistent advice I’m seeing so far, is to start with the books I’ve listed / purchases already. That will at least get me heading down a better direction.

Resources From The Twitterverse

http://www.amazon.com/RESTful-Web-Services-Cookbook-Scalability/dp/0596801688

http://code.google.com/p/implementing-rest/

http://www.amazon.com/REST-Design-Rulebook-Mark-Masse/dp/1449310508

The #rest channel on http://freenode.net/ IRC

http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

http://amundsen.com

http://www.infoq.com/search.action?queryString=stefan+tilkov+rest&searchOrder=relevance&search=stefan+tilkov+rest

http://www.sandiegodotnet.com/

I’ve been told that there is free pizza. If not, I might be able score some stale bagels from my hotel’s lobby, but no promises there.

Hope to see you there!

A Functional Calculator

To keep things simple, I created a calculator using nothing but functions. Each function takes other functions as parameters and returns a function. The only exception to this is the simple `value` function which takes a value and returns a function that itself will return the original value.

Here’s the basic code I wrote for my calculator, including the aforementioned `value` function:

// A raw value, represented as a function
var value = function(val){
  return function(){
    return val;
  }
}

// A function to add two functions together
var add = function(x, y){
  return function(){
   return x() + y();
  }
}

// A function to subtract one function from another
var subtract = function(x, y){
  return function(){
    return x() - y();
  }
}

// A function to multiply two functions
var multiply = function(x, y){
  return function(){
    return x() * y();
  }
}

I wrote all this through the use of some Jasmine tests, but as a quick example of how it can work, here’s a simple formula and result:

var one = value(1);
var two = value(2);
var three = value(3);
var four = value(4);

// (1 + 2) * 3 - 4 == 5
var resultFunc = subtract(multiply(add(one, two), three), four);

// but it's still a function right now, so evaluate it
var result = resultFunc();
consol.log(result); // => 5

My Thoughts And Reactions

These are the interesting points that stand out in my head. I’m not sure if these are “correct” or not… it’s just what pops out in my mind.

If anyone has any corrections and additional thoughts or notes, I’d love to hear them – just drop me a line in the comments, below.

Functions And Functions As State

I’ve represented everything as a function – even the simple values of 1 through 4. And every function returns a function, except for the simple values, which return a function that returns the actual value.

In the process of returning functions from functions, I’m representing the state of the application as a series of functions.

This one is a bit odd for me to wrap my head around. I’m not sure if this is really correct, or if it’s just an oversimplification of what’s really happening here. There are no data-structures, though, only functions. So it stands to reason that the state is stored as a series of functions that can be evaluated to produce a result.

Update: The Wikipedia article on functional programming says that fp “avoids state”. So… am I wrong in thinking that I’ve represented the state with functions instead of data structures?

Funcional Command Pattern

The result isn’t actually evaluated until I invoke the`resultFunc` in the above example. But I have the `resultFunc`available as a variable and I can pass it around anywhere I want. This basically turns it into a command pattern implementation, only in a functional manner instead of an object manner.

Compose-able And DSL Functions

All of these functions are side effect free. None of them actually mutates any existing application state. Of course this is a really simple implementation, but that was one of my goals. I’ll need to play with some more complex ideas in order to really see how side effect free would play out.

One of the benefits of side effect free functions is that the code in the sample is far more declarative and compose-able than most code I write. Since each function does one very small and very simple thing, taking in other functions as parameters, it’s easy to build things.

var one = value(1);
var two = value(2);
var three = value(3);
var four = value(4);

// (1 + 2) * 3 - 4 == 5
var added = add(one, two)
var multiplied = multiply(added, three)
var resultFunc = subtract(multiplied, four);

// the result is the same. 
// it's been composed the same, functionally
// it was just put together with more variables in the mix
var result = resultFunc();
consol.log(result); // => 5

This is something that I’ve played with in other languages, and I like this aspect of functional programming. It’s a great way to build DSLs for example.

Compared To Object / Prototypal

Oscar Godson posted a version of my code alongside a prototypal version for a comparison.

/**
 Prototypal
 from Oscar Godson: http://jsbin.com/atopek/edit#javascript
*/

var Calc = function(){
  this.sum = 0; //could be a param
  return this;
}

Calc.prototype.add = function(x){
  this.sum = this.sum + x;
  return this;
}

Calc.prototype.multiply = function(x){
  this.sum = this.sum * x
  return this;
}
  
Calc.prototype.subtract = function(x){
  this.sum = this.sum - x;
  return this
}

Calc.prototype.result = function(){
  return this.sum;
}

console.log(new Calc().add(1).add(2).multiply(3).subtract(4).result())

I like seeing them side by side. It’s interesting the see all of the different ways that JavaScript can do the same thing. It also shows me just how ugly I think the functional version is. When you compare the actual formula in the comments to the functional JS code, the code is inside-out-backwards. The prototypal code with it’s method chaining is closer to the formula, though it’s still a bit off.

I think it’s easier to read and understand the chaining, though. It also reminds me of LINQ and monads – pipes and filters with a single result coming from the end of it.

More Fun Functional Stuff To Try

I haven’t done anything like memoizing or currying for these simple examples, but those are ideas I want to play with, too. I’ve written a currying function in JavaScript once or twice, and there’s plenty of examples in the good JS books. Memoizing is also available in Underscore.js but should be trivial to implement a naive version of it. I’m sure there are plenty of other things that would be fun to try, as well.

Code And Continuing Thoughts (Maybe)

If you’re interested in seeing this code, it’s on Github.

I’m thinking about taking this a little further over time, just for fun. I’ve posted these comments in the read me (actually – they started there…) and I’ll continue to update the read me as I progress (assuming I actually do continue exploring this and don’t just forget about it like so many other things).

This is really pretty simple stuff… but it’s fun to write asimple exercise like this and think about what’s going on, why and see how it affects the way I think and approach software development.

My talk was on “Crafting Wicked Domain Models” that while was not recorded, all the slides and code can be found on my GitHub:

https://github.com/jbogard/presentations

Although that event wasn’t recorded, check out Claudio Lassala’s blog, where he recorded me doing the talk at the Houston Code Camp a couple months before.

A couple of folks came up to me afterwards telling me that they could never code/refactor in front of a Live Studio Audience. But, since the example was straight out of a real project I had lived through, it made going through the code a lot easier.

The questions afterwards were great, too. I always get some discussion around “it’s great and all, but now I have a bunch more classes to deal with”. It’s a fair criticism, and one to keep an eye out for. The way I see it, if the code is more understandable and more representative of real-world concepts, it’s a win. If, however, I’m having a hard time thinking of the names of classes for which I’m building responsibilities around, it’s a bit of a smell that I’m just inventing abstractions.

CodeMash really was a blast. The people were fantastic, the location was great, the food was fantastic, the beer and bacon were plentiful, and as always, the conversations were unforgettable. Hope to see everyone there next year!