For my smaller applications, I tend to stuff a number of <script> blocks in to the HTML that the user downloads. This makes it easy for me to work with and I don’t have to worry about whether or not the template exists. But when the number of templates gets to be more than 5 or 6 small ones, this gets out of hand quickly. It makes it hard for me to manage them as I have to scroll through a lot of template code. Putting them in external files and then including them in the page with some server side technology helps the developer problem but it doesn’t solve the client problem of having to download all of these templates even if they are never used.

To deal with the issues surrounding templates in HTML files, we can split our templates in to separate files and then use asynchronous calls to our server to load them as needed.

Backbone.View Render Semantics

One of my goals, other than the asynchronous template loading, is to keep the general semantics of a Backbone View’s `render` method. It’s a short list, but it’s an important list as most of the Backbone community expects the render method to work this way.

Semantics are generally important as they give us information about how methods and objects are expected to be used, as well. This, in turn, informs the method signature and behavior. And all of this comes back to the Liskov Substitution Principle (LSP) from the SOLID principles, which tells us that we need to pay attention to semantics so that we can drop in replacements as needed.

The general semantics and method signature of a view’s render include:

• No parameters: the render method shouldn’t require any parameters. You should be able to call `view.render()` and have it work fine
• Chainable: the render method should return `this` so that it can be chained with other method and attribute calls. This is most commonly done as `view.render().el` to grab the element that was rendered to
• Populate `el`, don’t replace it: the render method should populate a view’s `el` with any contents that the view needs to display. It generally shouldn’t replace the `el` as a whole

Aside from these three items, there’s a lot of flexibility in how a view will typically be rendered. There’s also plenty of room for interpretation and divergence from this list. Many applications use a render method signature that takes parameters, or that replaces the `el` entirely. When changes like this are made, it’s a good idea to document them so that people will know why the changes are in place.

The benefit of keeping these semantics, though, is that you can swap out a synchronous, pre-loaded template rendering view with a view that uses an asynchronous template loading mechanism. Or, better yet, you can have an intelligent system that uses asynchronous calls to get the template the first time it needs it, and then uses caching to keep the template around and do synchronous rendering on subsequent requests for this view / template. If the semantics for the view are kept in place, it doesn’t matter how the view implements the rendering. The view can be dropped in or removed as needed, without having to change the surrounding code that uses it.

Simple Async Template Retrieval

We can keep this very simple, to begin with, using jQuery’s AJAX calls to load the template with a callback to do the actual rendering after the template is loaded.

Backbone.View.extend({
  template: 'my-view-template',

  render: function(){
    var that = this;
    $.get("/templates/" + this.template + ".html", function(template){
      var html = $(template).tmpl();
      that.$el.html(html);
    });
    return this;
  }

});

In this example, we’re assuming that the view has a `template` attribute. This attribute represents the file that will be loaded from the server, and that file contain the actual template to be used.

We’re also using a convention of `/templates/{name}.html` for the template location on the server. This can be implemented easily in many different web server technologies. In Sinatra, for example, you can create a “public/templates” folder and put your HTML template files directly in that folder. They will be available without having to do anything more than start the Sinatra server.

When the call to `render` is made, the code makes an AJAX call back to the server to retrieve the specified template. A callback method is provided – and at this point, it assumes a successful call to get the template. When the template is returned from the server, the callback is fired, and the standard render code (using jQuery templates in this example) is executed.

Note that the `el` for the view is populated inside of the callback, but we are still calling `return this` at the end of the function. Even if we chain access to the `el` from the `render` method and immediately add the el’s contents to the DOM, this will still work:

$(“#content”).html(view.render().el);

The reason this works is that we are only populating the `el` with contents. We are not replacing it. When the AJAX call finally returns with the template, rendering it and populating the `el` will show the contents immediately because we’re setting the `html` method of an HTML element that is already attached to the DOM. It’s as if we had called `$(“#content”).html(“<div>some html</div>”);` directly.

Caching Templates

Now that we have a template loading asynchronously, it would be nice to only load it once instead of every time it needs to be used. This will improve the overall performance of the application, from the user’s perspective.

To do this, we’ll need a little more than just some code in the render method. We want to re-use templates that we’ve already loaded, which means we can’t just store the template on the view instance. We need to store it in a place where any view instance can grab a copy of the template if it exists, or have an asynchronous call back to the server done to get the template when needed.

TemplateManager = {
  templates: {},

  get: function(id, callback){
    var template = this.templates[id];

    if (template) {
      callback(template);

    } else {

      that = this;
      $.get("/templates/" + id + ".html", function(template){
        var $tmpl = $(template);
        that.templates[id] = $tmpl;
        callback($tmpl);
      }

    }

  }

}

The template manager object has one primary method that we call: `get`. This method takes in a template name to be loaded and a callback method that is executed when the template is found. By using a callback method instead of returning a value directly, we can ensure both synchronous and asynchronous calls will work correctly.

When you call `get`, it will check a hash / object literal to see if the template you want is already loaded using the template name that you provide to make this check. If it exists, it executes the callback immediately and passes the template along. If it does not exist yet, an AJAX call is made with jQuery to get the template from the server. Once the template is loaded, the callback that you passed in will be executed and the template is passed to it.

We can now update our view to use the template manager:

Backbone.View.extend({
  template: 'my-view-template',

  render: function(){
    var that = this;
    TemplateManager.get(this.template, function(template){
      var html = $(template).tmpl();
      that.$el.html(html);
    });
    return this;
  }

});

This isn’t much of a change from the view’s perspective. It’s better encapsulated, though, and a little easier to read. The real work is now being done in the TemplateManager object and we can change how it behaves as needed without having to update our views.

Beyond Simple Caching

There’s one remaining problem that this code has. If you have multiple instances of a view requesting the same template at roughly the same time, multiple AJAX calls will be made – one for each instance of the view. The net result is a slowdown in the application’s performance from too many network calls, and also a visual oddity where the views will appear one at a time as the AJAX calls finish. This can be a pretty big drag on performance and UI responsiveness.

Thanks to some previous digging in to jQuery’s deferred and some help from Steve Flitcroft (@red_square), I was able to solve this problem fairly easily. I put the following code in to my BBCloneMail app, which uses my Backbone.Marionette plugin. It’s not directly implemented in Marionette’s `TemplateManager` but most of what you need is already there. There’s only a few additional lines of code that you need to make this work.

(function(){
  var promises = {};

  // Cache the returned deferred/promise by the id of the template
  // so that we can prevent multiple requests for the same template
  // from making the ajax call.
  //
  // This code is only safe to run synchronously. There exists a
  // race condition in this function, when run asynchronously,
  // which would nullify the benefit under certain circumstances.
  var loadTemplateAsync = function(tmpId){
    var promise = promises[tmpId] || $.get("/templates/" + tmpId + ".html");
    promises[tmpId] = promise;
    return promise;
  }

  // Use jQuery to asynchronously load the template. 
  Backbone.Marionette.TemplateManager.loadTemplate = function(templateId, callback){
    var tmpId = templateId.replace("#", "");
    var promise = loadTemplateAsync(tmpId);
    promise.done(function(template){
      callback.call(this, $(template));
    });
  }
})();

The basic idea is that I’m using a jQuery deferred / promise to fire the callback method from the loadTemplate parameters. To prevent multiple requests for the same template heading back to the server, I’m caching the promises by the template id. When a call to loadTemplate is made through the template manager, I check to see if I have a promise for that template already. If I do, I register the loadTemplate’s callback parameter with the promise. If I don’t, I create the promise and then store it by the template’s Id. Either way, I register the callback with the promise which guarantees it will be executed. Once the template is returned from the server and the promise is fulfilled (resolved), all of the callbacks are fired off with the template data and everything renders correctly.

Note, though, that this code is not 100% safe. If you call the template manager from code that is already asynchronous, you can end up with a race condition where multiple promises are created and multiple calls to get the template are done. You’ll still get all of your views rendered just fine, but this will eliminate the benefit of using a promise to reduce network calls. Calling this code synchronously, though, won’t cause this race condition and the templates will only load once before they are cached and re-used.

See It In Action

If you’d like to see an example of this code in action (including the deferred/promise code from above), check out my BBCloneMail application and it’s source code. You’ll see the update to the TemplateManager’s `loadTemplate` function in the code. When you view the live site, switch to the Contacts view and refresh the page. There will be a small delay in the template loading, and then all of the contacts (which all individually requested the same contact template) will all display at once. Subsequent requests to those areas of the app will be cached, of course, improving the user experience and performance even more.

Then a few of my friends started talking about jQuery’s “deferred” objects and “promises”. I had heard of this previously but had not bothered to learn it. When I finally decided to dig in to it, this morning, I found that it was a much more robust implementation of what I had just written. I found a good article that helped me understand it better, and began working on an update for Marionette to use deferred objects instead of my own code. Addy Osmani also wrote a really good article on MSDN, which sheds even more light on how these work and how to work with them.

My New “Callbacks” Object

Here’s the end result of my digging in to this and replacing my Callbacks object with a jQuery deferred / promise implementation:

// Callbacks
// ---------

// A simple way of managing a collection of callbacks
// and executing them at a later point in time, using jQuery's
// `Deferred` object.
Marionette.Callbacks = function(){
  this.deferred = $.Deferred();
  this.promise = this.deferred.promise();
};

_.extend(Marionette.Callbacks.prototype, {
  
  // Add a callback to be executed. Callbacks added here are
  // guaranteed to execute, even if they are added after the 
  // `run` method is called.
  add: function(callback){
    this.promise.done(function(context, options){
      callback.call(context, options);
    });
  },

  // Run all registered callbacks with the context specified. 
  // Additional callbacks can be added after this has been run 
  // and they will still be executed.
  run: function(context, options){
    this.deferred.resolve(context, options);
  }
});

This is a reduction in code and complexity, by about 75% on both counts.

Things To Note

Backbone.Marionette.Callbacks is a constructor function and you are expected to instantiate it before using it:

var callbacks = new Backbone.Marionette.Callbacks();

callbacks.add(function(){ … });

I’m using a single deferred object and promise object in this code. That’s done on purpose. I am collection an arbitrary list of callbacks, at an unknown time, and want to guarantee execution of those callbacks. By limiting this code to a single deferred object and a single promise, I can do exactly that.

When you call `add`, it sets up a `done` callback using the single promise object. No matter how many times you call `add`, it uses the same promise for that instance of Callbacks.

Then when you finally call `run`, it resolves the deferred object. Calling `resolve` on the deferred object will kick off all of the callback methods that I had set up with the when/then code.

After you call `run` and resolve the deferred object, you can still add callback functions to this Callbacks instance. When you do add more, jQuery is smart enough to know that the promise has already been fulfilled and it immediately executes the callback that you registered.

Specs And Usage

If you’d like to see the specs for this to show how it works with pre-run registered and post-run registered callback methods, you can grab the source code for Backbone.Marionette and look at the Jasmine specs for Callbacks.

I’m using this Callbacks object in Marionette.Application’s “addInitializer” function, which I’m using to guarantee initializer callback execution. Once I get back to working on the client project that I wrote that original code for, I’ll also update it to use the new Callbacks implementation instead of my original implementation, too.

My Backbone.Marionette add-on helps to alleviate this problem by introducing the idea of application initializer callbacks. I’ve talked about these before, and I use them quite extensively in my applications. It really helps to keep my code clean and separated. And I recently realized that there’s an unintended benefit to using initializers like this: explicitly identifying the different stages of starting an application.

The 3 Stages

The three stages that I have identified in my own applications are (in order):

1. The Application’s Foundation
2. The Application’s Initialization
3. A Contextual Start (optional)

Honestly, I could break this down in to a significantly larger number of stages if I really wanted to get fine-grained and detailed. More specifically, it’s the 2nd stage that can be broken down in to pre-initialization, initialization, and post-initialization. But I think as a general place to start, identifying 3 primary stages for an application’s startup is a good idea. It gives us a better understanding of what code should go where, helps us keep our code clean, improves re-usability and flexibility, and will likely help us scale our apps as well.

Stage 1: The Application’s Foundation

Consider this to be the stage where the browser is simply grabbing all of the specified resources from the server and running through the first pass of parsing the HTML, laying out what the static HTML has provided, styling it with the basic CSS definitions, displaying any images and text from the static HTML, and parsing the JavaScript files so it can get ready to run them. I’ve never seen an HTML application that didn’t have this stage in it, because without it, your browser doesn’t have anything to do or show.

This is the one step the everyone does, and does well.

Stage 2: The Application’s Initialization

Every application has an initialization stage. Some are more elaborate than others. At times you can get away with the over-simplified “app” object with an “init” method on it, like many of the sample apps and boilerplate codes will suggest:

App = {
  init: function(){
    // app initialization and startup goes here
  }
}

App.init();

But when you get in to an application of any significant size and complexity, the single “init” method becomes a junk drawer where you tend to throw code because you haven’t yet identified where it should belong. Using a series of small initializer functions for each area of your application will help to alleviate this problem, as I’ve mentioned before. But the real question is:

What goes in the initializers?

Initializers are the bits of code, functionality, data and display that are absolutely required for your app to do it’s job, no matter what part of the application the user is trying to load up and use. They are the bits that must be initialized before the user can do anything meaningful with your application.

If your app has a menu structure generated by Backbone code, and it must always be present in the application, this should be in an initializer. If you’re building a multi-room chat application and you need to list the rooms that the user has favorited in a small block on the screen that is always visible to the user, this should be an initializer. If you’re building an image gallery and you need to load a thumb-nail list of images to show, no matter which image the user is trying to view, this is an initializer.

Other parts of your application code that the user doesn’t directly see may also be contenders for initializers, too. For example, if you have a router that needs to be up and running, the router probably needs to be instantiated inside of an initializer. But – and this is an important but – the routes on that router should not be executed during initialization. Instantiate the router and wait until after initialization has completed to call the “Backbone.history.start()” method, kicking off your route handlers.

Stage 3: A Contextual Start (optional)

I call this stage optional because not every application has a contextual kick off. Sometimes an application only needs initializers. The classic “todo” application is a great example of this. There is no contextual start up for this app. When the application is loaded, it initializes itself with the list of to do items and then it waits for the user to interact with the list.

When a Backbone application uses a router to respond to url pushState or hash fragment changes, though, it does have a contextual starting point. Each route that a user is allowed to bookmark or copy as a direct link will provide the context that our application needs to use, to get the user back to where they want to be. Even if a Backbone app has a router and contextual start, though, it may not be used. If there user hits the root of the application, there may not be any additional code to run for the empty (“”) route. When the user hits a route, though, that route must server up the context and application state that the user expects to see.

The problem that I see in most routed Backbone applications, though, is that they bundle the application initialization with the contextual start. That is, people tend to use the router and it’s callback methods as the sole place to get the application initialized and get the user back to the context of the route that they requested. For trivial applications, this may be fine. The initialization code may be so small that it doesn’t really matter if it’s crammed in to the router. But for any real application with any amount of complexity, this is a bad idea. It couples two very distinct parts of the application startup very tightly, and it can lead to bloated and unmaintainable routers with limited entry points in to the application.

So… what goes in the contextual start, other than just saying route callbacks?

Your route callbacks should be as simple as possible. They shouldn’t initialize your system as a whole. Rather, they should be used to determine the state of the application that the user wishes to see. Therefore, the code that goes in to a route callback should be the smallest amount of code that you can write, to get your application from it’s initial state (the state that it was in when the initializers completed) to the desired state.

In an image gallery application, a user may hit a bookmark that points to a specific image. For example, “#images/4″. The route callback that executes should load the requested image and display it on the screen:

Backbone.Router.extend({
  routes: {
    "image/:id": "imageById"
  },
 
  imageById: function(id){
    var image = imageCollection.get(id);
    App.showImage(image);
  }
});

In a multi-room chat application, a user may hit a bookmark that should take them directly in to a specific chat room. For example, “#backbone”. The route callback that executes should take this room name and call the code that is necessary to enter the chat room.

Chances are this is a fairly involved set of code. You’ll need to load the list of users in the chat room. You’ll need to clear the current chat windows and possibly pre-load recent messages to be displayed. You may also need to change a browser’s websocket event listeners to pick up events for this specific room instead, so that messages for this room can be displayed as they come in.

This is a lot of code to run, and it’s fare more code than should be allowed in a route callback method. It should, then, be encapsulated in an object that is responsible for registering the user as having entered that chat room. This object is then called from the router:

Backbone.Router.extend({
  routes: {
    ":roomname": "chatroom"
  },
 
  chatroom: function(roomname){
    ChatApp.enterRoom(roomname);
  }
});

Furthermore, there’s a high likelihood that the user will be able to enter chat rooms using some interaction on the web page. They may click on a chat room name in their favorite’s list, or they may type in a command like “/join #backbone” in to the chat application’s command area. For any of the the multiple ways to enter a chatroom, the code that is executed should be the same. Having the route callback be as simple and stupid as possible will promote code re-use and allow these three options to be easily implemented. If you only need to call “ChatApp.enterRoom(‘someRoom’)” to enter any room that the user specifies, providing options for how the user enters a room becomes trivial.

Putting It All Together

There are many different ways of stringing these different stages together. There are many different tools that help us to easily create the HTML that we need, include the CSS and JavaScript assets, and kick things off. Some applications are small enough that an initialization step and contextual start step can be combined. Most applications of any significant size, though, should think about separating these in to explicit steps in order to promote better architecture, clean code, and re-use of code.

For me, I’m accomplishing stages 2 and 3 with my Backbone.Marionette add-on. It provides a very clean way to create initializers:

App = new Backbone.Marionette.Application();

App.addInitializer(function(){
  // add some app initialization code, here
});

App.addInitializer(function(){
  // more initialization stuff 
  // for a different part of the app
});

// run all the initializers and start the app
App.start();

At the time of this writing, though, I don’t have an explicit “startup” callback mechanism in place. I have a “start” method on my Marionette.Application object. But right now this really only fires up the initializer callbacks. So, I’m currently using the “initialize:after” event to run my start code. I see this as a problem, though – not of a technical nature, but of a semantic nature. And yes, semantics are important.

App = new Backbone.Marionette.Application();

/* ... initializers go here ... */

// contextual startup
App.on("initialize:after", function(){
  Backbone.history.start();
});

I’m not sure what the new implementation will look like, but it may end up being a “startup” callback method, similar to the initializer methods. The startup callback (or callbacks?) would execute after all of the initialization is done. This will provide a cleaner separation between the two concerns of initialization vs startup. I’ll have to see where this leads me, though, as right now, the majority of what I do for a contextual start is simply call “Backbone.history.start()”, as I showed in the above example.

Depending on how the application is architected, and when the persons collection is expected to be loaded, there are a few options that I can see for solving this problem.

Option: Fetch The Model From The Server

The most basic option, and probably the easiest to deal with, is just to fetch the model from the server based on the id parameter that you get from the route.

In this case you just need to create a new model instance, set the `id` attribute on the model directly and then call `.fetch` on the model. It will make the trip to the server to get the data. You can either listen to a “change” event on the model or provide a “success” callback in the fetch method, to know when the data has been returned so that you can load it in to your view and display it as needed.

personById: function(id){
  var person = new Person();
  person.id = id;
  person.fetch({
    success: function(model, response){
      App.showPerson(model);
    }
  })
}

The major problem here is that you have not loaded the persons collection at all. If the persons collection is expected to be loaded because it’s being used for something related to displaying or working with the individual person model, then this option might not work for you. You could run some additional code to load the collection separately (asynchronously, since it works that way by default). This might help get around any potential issues of needing the collection.

Option: Use The Collection’s “reset” Event

Another easy option may be to load the collection when the request for the route is made. You would set up a new collection instance, bind a callback function to the collection’s “reset” event, and then call `.fetch` on the collection. The callback method would be responsible for retrieving the specific model from the collection and then creating and displaying the view.

personById: function(id){
  var persons = new PersonCollection();

  // note that "bind" is now "on" in Backbone v0.9.x
  persons.on("reset", function(collection, response){
    var person = collection.get(id);
    App.showPerson(person);
  });

  persons.fetch();
}

There are some potential issues with this solution, though. If you already have the persons collection loaded, then you’re going to load it a second time just to get the one model from it. To mitigate this problem, you would need two different entry points in to the display of the person: one for when you hit the route directly through a bookmark, and one for when the user is already in the app and navigates to the person display through other means.

Having two different entry points in to this part of the app may not be a bad idea. This largely depends on how the application is architected. You wouldn’t want to duplicate all of the code that sets up the display of the person’s details in both of the entry points, but you wouldn’t want to have a bunch of ugly if-statements in that code to determine how to set things up, either. Some simple abstractions of the common bits would help keep this code manageable.

Option: Building An “onReset” Callback

UPDATE:

FYI – I have an updated version of this code available in my Backbone.Marionette plugin, as Backbone.Marionette.Callbacks. It reduces the code and complexity significantly, and also eliminates the race condition issues that I mention below. Be sure to use that Callbacks object instead of the code I’ve listed here.

:END UPDATE

The third option that I can think of – and the one that I implemented for this particular client project – is a variation of using the collection’s reset event. The idea is to build an “onReset” callback system that is aware of whether or not the collection has already been loaded or is still waiting to be loaded.

If you have the persons collection being loaded from some other application initialization code, then you don’t necessarily have the ability to use a simple reset event as shown above. You could try to use the reset event, but there’s a race condition that is introduced in low-latency, high speed networks (i.e. your local development machine).

If you don’t control when the `.fetch` method is called, then you may end up binding to the reset event after the collection has already been reset. In that scenario – which is very likely to happen when working in a local development environment – your view for the specific person model will never get displayed.

The solution I came up with is to have a callback mechanism built in to the collection, that pays attention to the collection’s reset event and knows to either wait for the reset event to be fired, or to fire the callbacks immediately because the collection has already been loaded. I’m calling this an “onReset” callback, for lack of a better description at this point.

The code to use the onReset callbacks would look something like this:

// App.js
// ------
// some initialization code that happens elsewhere, using Backbone.Marionette

App.addInitializer(function(){
  App.persons = new PersonCollection();
  App.persons.fetch();
});



// PersonRouter.js
// ---------------
// The router callback that needs to get the person

personById: function(id){
  App.persons.onReset(function(collection){
    var person = collection.get(id);
    App.showPerson(person);
  });
}

In this setup, adding an onReset callback guarantees the callback’s execution. If the collection has not yet been loaded, then it stores the callback and waits for the reset event to fire. If the reset event has already been fired, then it simply executes the callback immediately. Either way, your callback will be executed and you will have the collection available when it does.

Race Condition Reduced. Eliminated?

Here’s the implementation for the onReset code. It’s generally functional and I haven’t yet run in to any problems, yet.

OnResetCollection = Backbone.Collection.extend({
  constructor: function(){
    var args = slice(arguments);
    Backbone.Collection.prototype.constructor.apply(this, args);

    this.onResetCallbacks = [];
    this.on("reset", this.collectionReset, this);
  },

  onReset: function(callback){
    this.onResetCallbacks.push(callback);
    this.collectionLoaded && this.fireResetCallbacks();
  },

  collectionReset: function(){
    if (!this.collectionLoaded) {
      this.collectionLoaded = true
    }
    this.fireResetCallbacks();
  },

  fireResetCallbacks: function(){
    var callback = this.onResetCallbacks.pop();
    if (callback){
      callback(this);
      this.fireResetCallbacks();
    }
  }
});

You can then extend from OnResetCollection instead of Backbone.Collection to get this functionality.

I still worry that there’s a potential race condition in between the logic and the pop’ing of items off the array. I’ve travelled every logical path of execution for the asynchronous call and onReset call that I can think of, and I can’t find any issue. I would love to hear from someone more experienced with race conditions in JavaScript, though.

JavaScript File Organization: You’re Doing It Right

The reality of JavaScript in general web-based development (not talking about server side at all, here) is that it makes absolutely no difference how your files and folders are organized. Zero difference at all… at least as far as the JavaScript runtime is concerned.

The only thing that matters is that you include your files in your HTML with a <script> block, correctly. You also need to pay attention to which order they are included in most cases, to make sure things are defined before they are used (tools like RequireJS and other script loaders and module definition / loaders help with this).

What does this really mean? It means:

You’re Doing It RIGHT!

Yup. You’re doing it right, because it doesn’t matter how you do it.

What does matter, though, is that you and your team (if you have a team) pick an organizational convention and stick with it. It’s actually very more important for your team to have a good file and folder structure for your JavaScript. But you don’t need to worry about what that structure is. Pick a standard and use it. When it fails (and it will), re-work your structure so that it works within the newly understood constrains and move on.

Of course, that fact that your browser and it’s JavaScript runtime don’t care about your file and folder organization doesn’t I’m without my opinions on how files and folders should be organized.

For example…

M, V and C Folders

A lot of people organize JavaScript MV* application files in a folder structure like this (BackboneJS in this case):

To the best of my knowledge, this folder structure is based on the “models”, “views” and “controllers” folder structure that was popularized by Ruby on Rails. Sure others may have had it first, but it was Rails that made it popular. Other MVC framework followed suit and demanded that you put your controller objects in the controllers folder, your model objects in your models folder, etc. But unless you’re Rails (or another framework that wants to be like Rails), this folder structure is stupid.

I’m pretty sure that Rails uses this folder structure to assume the types of objects that are found within the files. And I know for sure that it uses file names to assume the class that will be defined within the file. That is, when rails sees a file called “/app/controllers/foo_controller.rb”, it expects to find a class called “FooController” and it expects that this class will inherit from some Rails controller base class. If these expectations are not met, errors are thrown to say so.

I understand why Rails does it this way: file and folder based conventions make it easy to assume what a file will contain, and that makes it easy for the runtime to optimize for performance when pre-loading and caching the code contained within the files. This makes sense to me in Rails because the convention is based on good ideas for optimizing the way Rails works and the way it looks for files and how it loads them.

But, unless you’re Rails or another framework that wants to assume certain files in certain folder contain certain code, this is a terrible way to organize files.

The Junk Drawer

There are some good examples of other standards along this line. For example, I tend to follow the convention of a “public” folder with “css”, “images”, and “javascripts” folders. But honestly, this folder structure exhibits many of the same problems of being stupid that organizing files in M, V, and C folders does.

The real problem with these types of folder structures is that they become junk drawers. Even DHH and the Rails core team recognize that this is a poor folder structure outside the confines of Rails+Ruby code. That’s one of the reasons they added the Asset Pipeline in Rails 3.1. DHH even called the “javascripts” folder a junk drawer, very directly, in a RailsConf keynote in 2011 (or was it 2010?) – complete with a slide showing a drawer full of junk.

With any application that moves beyond a trivial number of files, these content-type, mime-type, code-type and general type-based folder structures turn in to a bloated pile of junk that is very difficult to sift through. Who wants to look at a folder with 20, 50 or 100 files in it, when you only really care about 2, 5 or 10 of those files?

And what happens when you suddenly have an object type that doesn’t fit your pre-established conventions? You end with a “lib” folder, like Rails, which becomes the ultimate junk drawer. “It’s not a model? It’s not a controller? It goes in lib.” – no matter what the actual functionality contained within the file is. The “lib” folder is asking to be a junk drawer… demanding it, really. So, do you follow that same junk drawer convention for non-M, V or C type-based files in your JavaScript apps? That doesn’t any make sense to me.

How I Organize Files: By Functionality

I prefer to organize my JavaScript files the way I used to organize my C# files in .NET projects: by functional area of the application. That is, I group files together in folders based on the area of the application that they facilitate.

For example, my BBCloneMail application has the following folder structure for it’s JavaScript:

Note that I’m still using the “javascripts” parent folder, but underneath of that I’m organizing by functional area of the application. In the root “javascripts” folder, are the primary application files – the ones that define the overall application bits. In the “mail” folder are all of the files that relate to the “mail” application. And, in the “contacts” folder are all of the files that relate to the “contacts” application.

I don’t care what “type” is contained in the file. That’s a completely irrelevant way to organize files to me. It makes no sense for me to organize files this way because many of my files contain more than one “type” of object. For example, I often put very simple model, collection and view definitions that are very closely related, in the same file.

Why Type-Based Folders Might Be A Good Idea

In spite of all my over-opinionated hand-waiving above, there are some good reasons to use type based folders in JavaScript. One reason is asynchronous file loading based on conventions.

You might have a JavaScript app that makes use of a templating engine (of which there are dozens, these days). It’s not always a good idea to pre-load every possible template in to the user’s browser, for download size and performance reasons. Sometimes it makes sense to fetch the template that you want the first time it’s requested.

To do this, it might make sense to use a convention to retrieve the files. I’ve seen several Backbone apps that use a jQuery selector to load files, as one example of this. When a Backbone view specifies a template as “#my-view-template”, the application’s template manager would make a request to the server to load something along the lines of “/templates/my-view-template.html”.

If you’re trying to organize your templates in a functional area of the application, you’ll have the added overhead of inserting the functional area folder name, such as “/mail/templates/inbox-template.html” for an “Inbox” view in a “mail” app, trying to load an “#inbox-template” jQuery selector as the template to render.

So… there’s at least one possible reason to use a type-based folder. I would still stick the type-based folder name under my functional area, though. I don’t want to mix up the templates between my functional areas of the application, by accident.

A Tradeoff: Folder Names vs File Names

Here’s one potential trade-off for using functional folder names vs type based folder names: you might have to specify the type in the file name. For example, if you have a model type name “Person”, a collection type named “Persons”, and a view to represent a single person or a collection of persons, what do you call that view? If you’re organizing things by type, you can call every “Person” and “Persons”

This can be very confusing. I was recently working with a client who was using an editor that only shows the file name for the open files, and none of the folder path for the files. He ended up with 4 “person.js” files in his open file list. Which one was the Person model, view, router, or controller file? We had to open each file to find the one we needed, every time. So, we took the hit on the file names. The “person.js” file contained the person model, while “persons.js” contained the collection, “personviews.js” contained the view definitions and “personrouter.js” contained the router. Thus, we’ve moved the need for specifying the object type from the folder structure (with all it’s bad ideas for using that) to the file name.

I’ve read at least one blog post that advocated using type-based folder names specifically to avoid the “ugliness” of having type names in your files. I seriously laughed out loud when I read that. Whether the type is in the file name or folder name is a moot point. You’re likely going to end up specifying the type somewhere. I would much rather have it in my file names because it’s easier for me to see things grouped together based on functionality, than based on the type of object contained in a file.

It’s Just An Opinion, And A Loose One At That

My own use of these conventions and ideas is rather loose at this point. I don’t stick strictly to anything, and I mix and match based on the project type, number of files and other constraints that a given project presents. Because, like I said at the top of this egregiously long post, you’re doing it right.

No matter what file and folder structure you pick for you JavaScript apps (assuming you’re using a suite of libraries that doesn’t force you in to a specific folder structure), you’re doing it right. JavaScript in a browser environment really doesn’t care what the file and folder structure is. But that doesn’t mean we as developers shouldn’t care. Pick a file and folder structure that fits the constraints of your application and change the structure as your app’s constraints change.

git remote | while read a; do
> git remote rm $a
> done

This took care of removing every single remote that I had in my ‘git remote’ list … including the “origin” that I actually did want, now that I think about it. :P

But my bash-scripting-fu is terrible, so I couldn’t figure out how to not destroy all of them. I tried adding an if statement in there to check if it’s not “origin”, but couldn’t get it to work. Anyone got some better bash-chops than me, and can point out how to do this?

To “fix” the problem, I switched my code from a “change” event to a “keyup” event for the text boxes … bad idea!

Browser Auto-Fill

Most (if not all) browsers have an auto-fill feature, these days. I use it in Chrome a lot. It saves me a few seconds here and there and generally makes it easier for me to fill in the same repetitious information across websites.

But there’s a problem with auto-fill. It doesn’t fire the “keyup” event. It fires the “change” event for the things it fills in, but not “keyup”. This resulted in the data being truncated when it populated my Backbone model.

In the above screenshot, since i had typed “deri” in to the email address, the email that is stored in the Backbone model would only be “deri” – and that’s obviously not a valid email address.

Easy To Fix

There are a number of very easy ways to fix this.

• Validate the email address format
• Use a combination of “blur”, “change”, and “keyup” events
• Delay reading the data until the user clicks the “Purchase” button
• Use a KVO (key-value observer) plugin like my Backbone.ModelBinding and let it deal with that for you

For the quick-fix to get my site working properly, I went with the combination of “change” and “keyup” events. I’ll be re-writing my purchase form sometime soon, and will likely delay reading the values until they click the “Purchase” button. I’m also going to put in better email address validation to make sure it at least has an @ and a . in it, and I’ll likely use the HTML5 “email” field and see how that will help me, if at all.

Not So Easy To Automate Testing

When I originally made the change to use the “keyup” event, I did all my usual testing – which did not include using auto-fill. All of my testing passed, so I pushed the site live.

How do you test for bugs like this in an automated way? Is there even an automated way to test a browser’s auto-fill? I’d like to avoid mistakes like this in the future, and some automated testing around it would certainly be nice.

Lesson Learned: Don’t Rely Solely On “keyup”

There certainly are some great things you can do with the “keyup” event – like autocomplete, for one – but it’s not a great idea to only use this when there’s a chance that the browser’s auto-fill will be used.

And unfortunately for me, 2 of my customers ran in to this problem before I got it fixed. It makes it very hard to email someone the download instructions for their order. One of them contacted me and I corrected the order. But the other purchaser has not yet contacted me, and I don’t have enough information to figure out who they are, so I can’t track them down. I hope this person realizes that they didn’t get their order, they contact me. My contact info is all over the site… so … hopefully … the lessons learned for running an e-commerce site are sometimes painful.

This is a bit of a problem. If I send all of the JavaScript that is uses to run any part of the system to the browser, then it’s possible that a clever user could enable it and do things they aren’t supposed to do. Of course, when they send a request back to the server, the server will verify they can do what they requested and block it… but the user should not be able to even try to do things they aren’t authorized to do, in the first place.

The solution that I’ve come up with, at a very high “functional area of the application” level, is simply not to send JavaScript to the browser if the user is not allowed to use it. This keeps me from having to do extra security checks in the browser, keeps the download for the user smaller and generally makes the app appear snappier because there is less code to run. More importantly, though, it keeps the user from being able to try things they can’t do.

Modules To The Rescue

I’m facilitating this through the use of modules in my JavaScript – both “module” as in the JavaScript module pattern, and “module” as in a packaged functional area of an application.

It’s worth noting, though, that I’m not using AMD (asynchronous module definitions), RequireJS, or any other “module” framework for JavaScript. The debate about the “right way” to do modules can rage on all it wants. I’m content with simple immediate functions and logical groupings of files while the more intelligent and invested people in the JS community figure that all out.

Each of the functional areas of my application is built in it’s own set of files. Each area has multiple files that it is composed within, but the files are grouped together for easy identification of what makes up a module. I also have an HTML template for each module which provides the needed <script> tags to include the code for that module.

Only Render What Is Needed

When I need a functional area of my site to be sent down to the user, I tell my server side template language to include the correct HTML file. For example, I’m doing this in an ASP.NET MVC application:

@{var user = (CustomPrincipal)User;}

@if (user.Can("locations/manage"))
{
    @Html.LocationManagementScripts()
}

@if (user.Can("locations/search"))
{
    @Html.LocationSearchScripts()
}

In this code, I’m checking to see if the current user is allowed to manage locations. If they are, the extension method “LocationManagementScripts” is called. This in turn renders the “LocationManagementScript.html” file at this point in my HTML layout. That file contains all of the <script> tags for the location management JavaScript app. In the same way, I’m checking to see if the user can search through locations, and running the same basic process if they can.

Self-Initializing Modules

When a functional module is included after passing one of these checks, it needs a way to get itself spun up and started so that it can do it’s magic. It may need to render something on to the screen. It may need to register itself with the application’s event aggregator, or any of a number of other things. This is where my Backbone.Marionette add-on comes in to play for my Backbone apps.

Marionette has an explicit concept of an “initializer” tied to it’s Application objects. When you create an instance of an Application object, you can call “app.addInitializer” and pass a callback function. The callback function represents everything that your module needs to do, to get itself up and running. All of these initializer functions – no matter how many you add – get fired when you call “app.start()”.

myApp = new Backbone.Marionette.Application();

myApp.addInitializer(function(options){
  var myView = new MyView({
    model: options.someModel
  });
  MyApp.mainRegion.show(myView);
});

myApp.addInitializer(function(options){
  new MyRouter();
  new SomeOtherView().render();
});

myApp.start();

Each functional area of my application has it’s own initializer function. When a functional area has been included in the rendered <script> tags, the initializer gets added and when the “start” method is called, the modules for that functional area are fired up and they do there thing.

A Composite App, And Sub-Apps

One of the tricks to making all of this work, is that I need to have a primary “app” object that all of my modules know about. In the above example, the “myApp” object is this. Each of the modules for each of the functional areas has direct knowledge of this object and can call public APIs on it – including the “addInitializer” method.

A better example of what a module definition and initializer might look like, would be this:

// --------------------
// app.js

myApp = new Backbone.Marionette.Application();

myApp.addRegions({
  mainRegion: "#main"
});


// --------------------
// locationSearch.js

(function(myApp, Backbone){
  var SearchView = Backbone.View.extend({
    // ...
  });

  myApp.addRegions({
    searchRegion: "#search"
  });

  myApp.addInitializer(function(){
    var view = new SearchView();
    view.render();
    myApp.searchRegion.show(view);
  });
})(myApp, Backbone);


// --------------------
// index.html

<script language="javascript">
  myApp.start();
</script>

In this example, I’m using the simple JavaScript module pattern to encapsulate my search functionality. I’m also providing an initializer for the module that instantiates a search view and shows it to the user using a region manager.

Each of these functional areas is basically a sub-application. Many sub-applications are used to compose a larger application and overall experience for the user. The composition of a larger application through various modules that are included / excluded based on some criteria are what really make this a composite application.

I also included the final call to “myApp.start()”, showing that I do this from my main HTML page and not from my JavaScript files. This provides a single point of entry for all of the registered modules, no matter which modules are registered. The “myApp” object really doesn’t care which modules are registered, honestly. It doesn’t need to care. It only needs to execute the initializers that happen to be present. If none are present because the user didn’t have permission to do anything, then nothing happens when this method is called and the user won’t see anything.

Security: Don’t Let Them See It If They Can’t Do It

If the security check to see if the user is allowed to use the location search feature fails, the rendered HTML won’t include the <script> tags for the “locationSearch.js” file. If this file is not sent down to the browser, then it will never register itself. If a module has not registered itself for initialization, it’s views won’t show up on the screen and the user won’t be able to try and use the feature. Further, the user won’t be able to “view source” on the page and find any stray JavaScript that they shouldn’t be able to use.

It’s Not Always That Easy

Of course there are other security concerns that are not this simple. When a functional area is closed off by authorization, it’s easy to keep things clean like this. We can compose the application at run time simply by including the right files and letting the code in those files register themselves for initialization. But when we have a functional area of the system that has finer grained authorization and permissions associated with it, things get a little more tricky.

I’m still learning and exploring this space. I have some ideas and am going to be implementing some of them soon. If anyone out there has any experience in handling finer grained security needs in JavaScript apps, I’d love to hear about it. Post links to your favorite resources for this, in the comments.

Except that it’s all wrong. Every last bit of it.

… or, at least the handful of people who inevitably respond with comments like “no one does REST correctly” and “that’s not REST” would have me believe.

Your Doing It Wrong

Frankly, I’m tired of this pedantic, obnoxious response. The same people that continuously say “no one does it right” can’t seem to tell me why it’s wrong or point me to a resource that does do it right.

Here’s the deal. If everyone does it wrong, then write a blog post or a book or a screencast or something that tells the world how to do it right. Until you can point people to what REST actually is, stop telling everyone that no one does REST right.

My Journey Into Being Wrong

I have work to do. I’m obviously going to do it wrong, but frankly, I don’t care at this point. My system will work just fine, just like every other “That’s not REST” system out there.

To help me learn how to be wrong, though, here are a few resources that I’m going to be checking out:

Get Some REST: A work in progress book from Steve Klabnik. I signed up for the mailing list and am eagerly waiting. There’s rumor that Steve might actually be doing it right.

RESTFul Web Services – I bought the Kindle version last night. It’s the book DHH says everyone should read and a few others also suggested it.

REST In Practice – I’m probably going to buy this one, too. It seems to have the “30,000 foot view” that I’m also looking for, according to one reviewer on Amazon.

If anyone else has any resources for how I can learn non-REST, RESTful best practices, I’d love to know.

(end of rant)

… Updates

Thanks to everyone who is responding in the comments here and via twitter. I’m going to update this post with the resources that people are pointing me to, from Twitter.

The majority of my experience with anything REST-like comes from working with Ruby on Rails, and mimicking it’s handling of HTTP verbs with Sinatra application. I know this isn’t anything near correct, which is why I’m looking to find good resources.

I’ve been getting a lot of “you don’t need to do pure REST” responses – which I agree with. One of my problems, though, is that I’m having a hard time figuring out what “pure” REST is so that I can know the difference between that and what I’m actually doing.

Sebastian Lambla points out that it’s not the entire REST community that pulls this crap. There are plenty of people that are helping, providing good information and creating good resources to throw at people like me that are wanting to learn. My problem is not with the REST community as a whole, but with the people that behave in the way I’ve pointed out, here.

The most consistent advice I’m seeing so far, is to start with the books I’ve listed / purchases already. That will at least get me heading down a better direction.

Resources From The Twitterverse

http://www.amazon.com/RESTful-Web-Services-Cookbook-Scalability/dp/0596801688

http://code.google.com/p/implementing-rest/

http://www.amazon.com/REST-Design-Rulebook-Mark-Masse/dp/1449310508

The #rest channel on http://freenode.net/ IRC

http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

http://amundsen.com

http://www.infoq.com/search.action?queryString=stefan+tilkov+rest&searchOrder=relevance&search=stefan+tilkov+rest

A Functional Calculator

To keep things simple, I created a calculator using nothing but functions. Each function takes other functions as parameters and returns a function. The only exception to this is the simple `value` function which takes a value and returns a function that itself will return the original value.

Here’s the basic code I wrote for my calculator, including the aforementioned `value` function:

// A raw value, represented as a function
var value = function(val){
  return function(){
    return val;
  }
}

// A function to add two functions together
var add = function(x, y){
  return function(){
   return x() + y();
  }
}

// A function to subtract one function from another
var subtract = function(x, y){
  return function(){
    return x() - y();
  }
}

// A function to multiply two functions
var multiply = function(x, y){
  return function(){
    return x() * y();
  }
}

I wrote all this through the use of some Jasmine tests, but as a quick example of how it can work, here’s a simple formula and result:

var one = value(1);
var two = value(2);
var three = value(3);
var four = value(4);

// (1 + 2) * 3 - 4 == 5
var resultFunc = subtract(multiply(add(one, two), three), four);

// but it's still a function right now, so evaluate it
var result = resultFunc();
consol.log(result); // => 5

My Thoughts And Reactions

These are the interesting points that stand out in my head. I’m not sure if these are “correct” or not… it’s just what pops out in my mind.

If anyone has any corrections and additional thoughts or notes, I’d love to hear them – just drop me a line in the comments, below.

Functions And Functions As State

I’ve represented everything as a function – even the simple values of 1 through 4. And every function returns a function, except for the simple values, which return a function that returns the actual value.

In the process of returning functions from functions, I’m representing the state of the application as a series of functions.

This one is a bit odd for me to wrap my head around. I’m not sure if this is really correct, or if it’s just an oversimplification of what’s really happening here. There are no data-structures, though, only functions. So it stands to reason that the state is stored as a series of functions that can be evaluated to produce a result.

Update: The Wikipedia article on functional programming says that fp “avoids state”. So… am I wrong in thinking that I’ve represented the state with functions instead of data structures?

Funcional Command Pattern

The result isn’t actually evaluated until I invoke the`resultFunc` in the above example. But I have the `resultFunc`available as a variable and I can pass it around anywhere I want. This basically turns it into a command pattern implementation, only in a functional manner instead of an object manner.

Compose-able And DSL Functions

All of these functions are side effect free. None of them actually mutates any existing application state. Of course this is a really simple implementation, but that was one of my goals. I’ll need to play with some more complex ideas in order to really see how side effect free would play out.

One of the benefits of side effect free functions is that the code in the sample is far more declarative and compose-able than most code I write. Since each function does one very small and very simple thing, taking in other functions as parameters, it’s easy to build things.

var one = value(1);
var two = value(2);
var three = value(3);
var four = value(4);

// (1 + 2) * 3 - 4 == 5
var added = add(one, two)
var multiplied = multiply(added, three)
var resultFunc = subtract(multiplied, four);

// the result is the same. 
// it's been composed the same, functionally
// it was just put together with more variables in the mix
var result = resultFunc();
consol.log(result); // => 5

This is something that I’ve played with in other languages, and I like this aspect of functional programming. It’s a great way to build DSLs for example.

Compared To Object / Prototypal

Oscar Godson posted a version of my code alongside a prototypal version for a comparison.

/**
 Prototypal
 from Oscar Godson: http://jsbin.com/atopek/edit#javascript
*/

var Calc = function(){
  this.sum = 0; //could be a param
  return this;
}

Calc.prototype.add = function(x){
  this.sum = this.sum + x;
  return this;
}

Calc.prototype.multiply = function(x){
  this.sum = this.sum * x
  return this;
}
  
Calc.prototype.subtract = function(x){
  this.sum = this.sum - x;
  return this
}

Calc.prototype.result = function(){
  return this.sum;
}

console.log(new Calc().add(1).add(2).multiply(3).subtract(4).result())

I like seeing them side by side. It’s interesting the see all of the different ways that JavaScript can do the same thing. It also shows me just how ugly I think the functional version is. When you compare the actual formula in the comments to the functional JS code, the code is inside-out-backwards. The prototypal code with it’s method chaining is closer to the formula, though it’s still a bit off.

I think it’s easier to read and understand the chaining, though. It also reminds me of LINQ and monads – pipes and filters with a single result coming from the end of it.

More Fun Functional Stuff To Try

I haven’t done anything like memoizing or currying for these simple examples, but those are ideas I want to play with, too. I’ve written a currying function in JavaScript once or twice, and there’s plenty of examples in the good JS books. Memoizing is also available in Underscore.js but should be trivial to implement a naive version of it. I’m sure there are plenty of other things that would be fun to try, as well.

Code And Continuing Thoughts (Maybe)

If you’re interested in seeing this code, it’s on Github.

I’m thinking about taking this a little further over time, just for fun. I’ve posted these comments in the read me (actually – they started there…) and I’ll continue to update the read me as I progress (assuming I actually do continue exploring this and don’t just forget about it like so many other things).

This is really pretty simple stuff… but it’s fun to write asimple exercise like this and think about what’s going on, why and see how it affects the way I think and approach software development.