A List Apart: The Abridged Feed

The Design of Code: Organizing JavaScript

2013-05-21T12:00:27+00:00

Great design is a product of care and attention applied to areas that matter, resulting in a useful, understandable, and hopefully beautiful user interface. But don’t be fooled into thinking that design is left only for designers.

There is a lot of design in code, and I don’t mean code that builds the user interface—I mean the design of code.

Well-designed code is much easier to maintain, optimize, and extend, making for more efficient developers. That means more focus and energy can be spent on building great things, which makes everyone happy—users, developers, and stakeholders.

There are three high-level, language-agnostic aspects to code design that are particularly important.

System architecture—The basic layout of the codebase. Rules that govern how various components, such as models, views, and controllers, interact with each other.
Maintainability—How well can the code be improved and extended?
Reusability—How reusable are the application’s components? How easily can each implementation of a component be customized?

In looser languages, specifically JavaScript, it takes a bit of discipline to write well-designed code. The JavaScript environment is so forgiving that it’s easy to throw bits and pieces everywhere and still have things work. Establishing system architecture early (and sticking to it!) provides constraints to your codebase, ensuring consistency throughout.

One approach I’m fond of consists of a tried-and-true software design pattern, the module pattern, whose extensible structure lends itself to a solid system architecture and a maintainable codebase. I like building modules within a jQuery plugin, which makes for beautiful reusability, provides robust options, and exposes a well-crafted API.

Below, I’ll walk through how to craft your code into well-organized components that can be reused in projects to come.

The module pattern

There are a lot of design patterns out there, and equally as many resources on them. Addy Osmani wrote an amazing (free!) book on design patterns in JavaScript, which I highly recommend to developers of all levels.

The module pattern is a simple structural foundation that can help keep your code clean and organized. A “module” is just a standard object literal containing methods and properties, and that simplicity is the best thing about this pattern: even someone unfamiliar with traditional software design patterns would be able to look at the code and instantly understand how it works.

In applications that use this pattern, each component gets its own distinct module. For example, to build autocomplete functionality, you’d create a module for the textfield and a module for the results list. These two modules would work together, but the textfield code wouldn’t touch the results list code, and vice versa.

That decoupling of components is why the module pattern is great for building solid system architecture. Relationships within the application are well-defined; anything related to the textfield is managed by the textfield module, not strewn throughout the codebase—resulting in clear code.

Another benefit of module-based organization is that it is inherently maintainable. Modules can be improved and optimized independently without affecting any other part of the application.

I used the module pattern for the basic structure of jPanelMenu, the jQuery plugin I built for off-canvas menu systems. I’ll use that as an example to illustrate the process of building a module.

Building a module

To begin, I define three methods and a property that are used to manage the interactions of the menu system.


var jpm = {
    animated: true,
    openMenu: function( ) {
        …
        this.setMenuStyle( );
    },
    closeMenu: function( ) {
        …
        this.setMenuStyle( );
    },
    setMenuStyle: function( ) { … }
};

The idea is to break down code into the smallest, most reusable bits possible. I could have written just one toggleMenu( ) method, but creating distinct openMenu( ) and closeMenu( ) methods provides more control and reusability within the module.

Notice that calls to module methods and properties from within the module itself (such as the calls to setMenuStyle( )) are prefixed with the this keyword—that’s how modules access their own members.

That’s the basic structure of a module. You can continue to add methods and properties as needed, but it doesn’t get any more complex than that. After the structural foundations are in place, the reusability layer—options and an exposed API—can be built on top.

jQuery plugins

The third aspect of well-designed code is probably the most crucial: reusability. This section comes with a caveat. While there are obviously ways to build and implement reusable components in raw JavaScript (we’re about 90 percent of the way there with our module above), I prefer to build jQuery plugins for more complex things, for a few reasons.

Most importantly, it’s a form of unobtrusive communication. If you used jQuery to build a component, you should make that obvious to those implementing it. Building the component as a jQuery plugin is a great way to say that jQuery is required.

In addition, the implementation code will be consistent with the rest of the jQuery-based project code. That’s good for aesthetic reasons, but it also means (to an extent) that developers can predict how to interact with the plugin without too much research. Just one more way to build a better developer interface.

Before you begin building a jQuery plugin, ensure that the plugin does not conflict with other JavaScript libraries using the $ notation. That’s a lot simpler than it sounds—just wrap your plugin code like so:


(function($) {
    // jQuery plugin code here
})(jQuery);

Next, we set up our plugin and drop our previously built module code inside. A plugin is just a method defined on the jQuery ($) object.


(function($) {
    $.jPanelMenu = function( ) {
        var jpm = {
            animated: true,
            openMenu: function( ) {
                …
                this.setMenuStyle( );
            },
            closeMenu: function( ) {
                …
                this.setMenuStyle( );
            },
            setMenuStyle: function( ) { … }
        };
    };
})(jQuery);

All it takes to use the plugin is a call to the function you just created.


var jpm = $.jPanelMenu( );

Options

Options are essential to any truly reusable plugin because they allow for customizations to each implementation. Every project brings with it a slew of design styles, interaction types, and content structures. Customizable options help ensure that you can adapt the plugin to fit within those project constraints.

It’s best practice to provide good default values for your options. The easiest way to do that is to use jQuery’s $.extend( ) method, which accepts (at least) two arguments.

As the first argument of $.extend( ), define an object with all available options and their default values. As the second argument, pass through the passed-in options. This will merge the two objects, overriding the defaults with any passed-in options.


(function($) {
    $.jPanelMenu = function(options) {
        var jpm = {
            options: $.extend({
                'animated': true,
                'duration': 500,
                'direction': 'left'
            }, options),
            openMenu: function( ) {
                …
                this.setMenuStyle( );
            },
            closeMenu: function( ) {
                …
                this.setMenuStyle( );
            },
            setMenuStyle: function( ) { … }
        };
    };
})(jQuery);

Beyond providing good defaults, options become almost self-documenting—someone can look at the code and see all of the available options immediately.

Expose as many options as is feasible. The customization will help in future implementations, and flexibility never hurts.

API

Options are terrific ways to customize how a plugin works. An API, on the other hand, enables extensions to the plugin’s functionality by exposing methods and properties for the implementation code to take advantage of.

While it’s great to expose as much as possible through an API, the outside world shouldn’t have access to all internal methods and properties. Ideally, you should expose only the elements that will be used.

In our example, the exposed API should include calls to open and close the menu, but nothing else. The internal setMenuStyle( ) method runs when the menu opens and closes, but the public doesn’t need access to it.

To expose an API, return an object with any desired methods and properties at the end of the plugin code. You can even map returned methods and properties to those within the module code—this is where the beautiful organization of the module pattern really shines.


(function($) {
    $.jPanelMenu = function(options) {
        var jpm = {
            options: $.extend({
                'animated': true,
                'duration': 500,
                'direction': 'left'
            }, options),
            openMenu: function( ) {
                …
                this.setMenuStyle( );
            },
            closeMenu: function( ) {
                …
                this.setMenuStyle( );
            },
            setMenuStyle: function( ) { … }
        };

        return {
            open: jpm.openMenu,
            close: jpm.closeMenu,
            someComplexMethod: function( ) { … }
        };
    };
})(jQuery);

API methods and properties will be available through the object returned from the plugin initialization.


var jpm = $.jPanelMenu({
    duration: 1000,
    …
});
jpm.open( );

Polishing developer interfaces

With just a few simple constructs and guidelines, we’ve built ourselves a reusable, extensible plugin that will help make our lives easier. Like any part of what we do, experiment with this structure to see if it works for you, your team, and your workflow.

Whenever I find myself building something with a potential for reuse, I break it out into a module-based jQuery plugin. The best part about this approach is that it forces you to use—and test—the code you write. By using something as you build it, you’ll quickly identify strengths, discover shortcomings, and plan changes.

This process leads to battle-tested code ready for open-source contributions, or to be sold and distributed. I’ve released my (mostly) polished plugins as open-source projects on GitHub.

Even if you aren’t building something to be released in the wild, it’s still important to think about the design of your code. Your future self will thank you.

Writing Testable JavaScript

2013-05-21T12:00:00+00:00

We’ve all been there: that bit of JavaScript functionality that started out as just a handful of lines grows to a dozen, then two dozen, then more. Along the way, a function picks up a few more arguments; a conditional picks up a few more conditions. And then one day, the bug report comes in: something’s broken, and it’s up to us to untangle the mess.

As we ask our client-side code to take on more and more responsibilities—indeed, whole applications are living largely in the browser these days—two things are becoming clear. One, we can’t just point and click our way through testing that things are working as we expect; automated tests are key to having confidence in our code. Two, we’re probably going to have to change how we write our code in order to make it possible to write tests.

Really, we need to change how we code? Yes—because even if we know that automated tests are a good thing, most of us are probably only able to write integration tests right now. Integration tests are valuable because they focus on how the pieces of an application work together, but what they don’t do is tell us whether individual units of functionality are behaving as expected.

That’s where unit testing comes in. And we’ll have a very hard time writing unit tests until we start writing testable JavaScript.

Unit vs. integration: what’s the difference?

Writing integration tests is usually fairly straightforward: we simply write code that describes how a user interacts with our app, and what the user should expect to see as she does. Selenium is a popular tool for automating browsers. Capybara for Ruby makes it easy to talk to Selenium, and there are plenty of tools for other languages, too.

Here’s an integration test for a portion of a search app:

def test_search
  fill_in('q', :with => 'cat')
  find('.btn').click
  assert( find('#results li').has_content?('cat'), 'Search results are shown' )
  assert( page.has_no_selector?('#results li.no-results'), 'No results is not shown' )
end

Whereas an integration test is interested in a user’s interaction with an app, a unit test is narrowly focused on a small piece of code:

When I call a function with a certain input, do I receive the expected output?

Apps that are written in a traditional procedural style can be very difficult to unit test—and difficult to maintain, debug, and extend, too. But if we write our code with our future unit testing needs in mind, we will not only find that writing the tests becomes more straightforward than we might have expected, but also that we’ll simply write better code, too.

To see what I’m talking about, let’s take a look at a simple search app:

When a user enters a search term, the app sends an XHR to the server for the corresponding data. When the server responds with the data, formatted as JSON, the app takes that data and displays it on the page, using client-side templating. A user can click on a search result to indicate that he “likes” it; when this happens, the name of the person he liked is added to the “Liked” list on the right-hand side.

A “traditional” JavaScript implementation of this app might look like this:

var tmplCache = {};

function loadTemplate (name) {
  if (!tmplCache[name]) {
    tmplCache[name] = $.get('/templates/' + name);
  }
  return tmplCache[name];
}

$(function () {

  var resultsList = $('#results');
  var liked = $('#liked');
  var pending = false;

  $('#searchForm').on('submit', function (e) {
    e.preventDefault();

    if (pending) { return; }

    var form = $(this);
    var query = $.trim( form.find('input[name="q"]').val() );

    if (!query) { return; }

    pending = true;

    $.ajax('/data/search.json', {
      data : { q: query },
      dataType : 'json',
      success : function (data) {
        loadTemplate('people-detailed.tmpl').then(function (t) {
          var tmpl = _.template(t);
          resultsList.html( tmpl({ people : data.results }) );
          pending = false;
        });
      }
    });

    $('<li>', {
      'class' : 'pending',
      html : 'Searching &hellip;'
    }).appendTo( resultsList.empty() );
  });

  resultsList.on('click', '.like', function (e) {
    e.preventDefault();
    var name = $(this).closest('li').find('h2').text();
    liked.find('.no-results').remove();
    $('<li>', { text: name }).appendTo(liked);
  });

});

My friend Adam Sontag calls this Choose Your Own Adventure code—on any given line, we might be dealing with presentation, or data, or user interaction, or application state. Who knows! It’s easy enough to write integration tests for this kind of code, but it’s hard to test individual units of functionality.

What makes it hard? Four things:

A general lack of structure; almost everything happens in a $(document).ready() callback, and then in anonymous functions that can’t be tested because they aren’t exposed.
Complex functions; if a function is more than 10 lines, like the submit handler, it’s highly likely that it’s doing too much.
Hidden or shared state; for example, since pending is in a closure, there’s no way to test whether the pending state is set correctly.
Tight coupling; for example, a $.ajax success handler shouldn’t need direct access to the DOM.

Organizing our code

The first step toward solving this is to take a less tangled approach to our code, breaking it up into a few different areas of responsibility:

Presentation and interaction
Data management and persistence
Overall application state
Setup and glue code to make the pieces work together

In the “traditional” implementation shown above, these four categories are intermingled—on one line we’re dealing with presentation, and two lines later we might be communicating with the server.

While we can absolutely write integration tests for this code—and we should!—writing unit tests for it is pretty difficult. In our functional tests, we can make assertions such as “when a user searches for something, she should see the appropriate results,” but we can’t get much more specific. If something goes wrong, we’ll have to track down exactly where it went wrong, and our functional tests won’t help much with that.

If we rethink how we write our code, though, we can write unit tests that will give us better insight into where things went wrong, and also help us end up with code that’s easier to reuse, maintain, and extend.

Our new code will follow a few guiding principles:

Represent each distinct piece of behavior as a separate object that falls into one of the four areas of responsibility and doesn’t need to know about other objects. This will help us avoid creating tangled code.
Support configurability, rather than hard-coding things. This will prevent us from replicating our entire HTML environment in order to write our tests.
Keep our objects’ methods simple and brief. This will help us keep our tests simple and our code easy to read.
Use constructor functions to create instances of objects. This will make it possible to create “clean” copies of each piece of code for the sake of testing.

To start with, we need to figure out how we’ll break our application into different pieces. We’ll have three pieces dedicated to presentation and interaction: the Search Form, the Search Results, and the Likes Box.

We’ll also have a piece dedicated to fetching data from the server and a piece dedicated to gluing everything together.

Let’s start by looking at one of the simplest pieces of our application: the Likes Box. In the original version of the app, this code was responsible for updating the Likes Box:

var liked = $('#liked'); 
var resultsList = $('#results'); 
 
// ...
  
resultsList.on('click', '.like', function (e) {
  e.preventDefault(); 
  var name = $(this).closest('li').find('h2').text(); 
  liked.find( '.no-results' ).remove(); 
  $('<li>', { text: name }).appendTo(liked); 
});

The Search Results piece is completely intertwined with the Likes Box piece and needs to know a lot about its markup. A much better and more testable approach would be to create a Likes Box object that’s responsible for manipulating the DOM related to the Likes Box:

var Likes = function (el) {
  this.el = $(el);
  return this;
};

Likes.prototype.add = function (name) {
  this.el.find('.no-results').remove();
  $('<li>', { text: name }).appendTo(this.el);
};

This code provides a constructor function that creates a new instance of a Likes Box. The instance that’s created has an .add() method, which we can use to add new results. We can write a couple of tests to prove that it works:

var ul;

setup(function(){
  ul = $('<ul><li class="no-results"></li></ul>');
});

test('constructor', function () {
  var l = new Likes(ul);
  assert(l);
});

test('adding a name', function () {
  var l = new Likes(ul);
  l.add('Brendan Eich');

  assert.equal(ul.find('li').length, 1);
  assert.equal(ul.find('li').first().html(), 'Brendan Eich');
  assert.equal(ul.find('li.no-results').length, 0);
});

Not so hard, is it? Here we’re using Mocha as the test framework, and Chai as the assertion library. Mocha provides the test and setup functions; Chai provides assert. There are plenty of other test frameworks and assertion libraries to choose from, but for the sake of an introduction, I find these two work well. You should find the one that works best for you and your project—aside from Mocha, QUnit is popular, and Intern is a new framework that shows a lot of promise.

Our test code starts out by creating an element that we’ll use as the container for our Likes Box. Then, it runs two tests: one is a sanity check to make sure we can make a Likes Box; the other is a test to ensure that our .add() method has the desired effect. With these tests in place, we can safely refactor the code for our Likes Box, and be confident that we’ll know if we break anything.

Our new application code can now look like this:

var liked = new Likes('#liked');
var resultsList = $('#results'); 
 
// ...
  
resultsList.on('click', '.like', function (e) {
  e.preventDefault(); 
  var name = $(this).closest('li').find('h2').text(); 
  liked.add(name);
});

The Search Results piece is more complex than the Likes Box, but let’s take a stab at refactoring that, too. Just as we created an .add() method on the Likes Box, we also want to create methods for interacting with the Search Results. We’ll want a way to add new results, as well as a way to “broadcast” to the rest of the app when things happen within the Search Results—for example, when someone likes a result.

var SearchResults = function (el) {
  this.el = $(el);
  this.el.on( 'click', '.btn.like', _.bind(this._handleClick, this) );
};

SearchResults.prototype.setResults = function (results) {
  var templateRequest = $.get('people-detailed.tmpl');
  templateRequest.then( _.bind(this._populate, this, results) );
};

SearchResults.prototype._handleClick = function (evt) {
  var name = $(evt.target).closest('li.result').attr('data-name');
  $(document).trigger('like', [ name ]);
};

SearchResults.prototype._populate = function (results, tmpl) {
  var html = _.template(tmpl, { people: results });
  this.el.html(html);
};

Now, our old app code for managing the interaction between Search Results and the Likes Box could look like this:

var liked = new Likes('#liked');
var resultsList = new SearchResults('#results'); 
 
// ...
  
$(document).on('like', function (evt, name) {
  liked.add(name);
})

It’s much simpler and less entangled, because we’re using the document as a global message bus, and passing messages through it so individual components don’t need to know about each other. (Note that in a real app, we’d use something like Backbone or the RSVP library to manage events. We’re just triggering on document to keep things simple here.) We’re also hiding all the dirty work—such as finding the name of the person who was liked—inside the Search Results object, rather than having it muddy up our application code. The best part: we can now write tests to prove that our Search Results object works as we expect:

var ul;
var data = [ /* fake data here */ ];

setup(function () {
  ul = $('<ul><li class="no-results"></li></ul>');
});

test('constructor', function () {
  var sr = new SearchResults(ul);
  assert(sr);
});

test('display received results', function () {
  var sr = new SearchResults(ul);
  sr.setResults(data);

  assert.equal(ul.find('.no-results').length, 0);
  assert.equal(ul.find('li.result').length, data.length);
  assert.equal(
    ul.find('li.result').first().attr('data-name'),
    data[0].name
  );
});

test('announce likes', function() {
  var sr = new SearchResults(ul);
  var flag;
  var spy = function () {
    flag = [].slice.call(arguments);
  };

  sr.setResults(data);
  $(document).on('like', spy);

  ul.find('li').first().find('.like.btn').click();

  assert(flag, 'event handler called');
  assert.equal(flag[1], data[0].name, 'event handler receives data' );
});

The interaction with the server is another interesting piece to consider. The original code included a direct $.ajax() request, and the callback interacted directly with the DOM:

$.ajax('/data/search.json', {
  data : { q: query },
  dataType : 'json',
  success : function( data ) {
    loadTemplate('people-detailed.tmpl').then(function(t) {
      var tmpl = _.template( t );
      resultsList.html( tmpl({ people : data.results }) );
      pending = false;
    });
  }
});

Again, this is difficult to write a unit test for, because so many different things are happening in just a few lines of code. We can restructure the data portion of our application as an object of its own:

var SearchData = function () { };

SearchData.prototype.fetch = function (query) {
  var dfd;

  if (!query) {
    dfd = $.Deferred();
    dfd.resolve([]);
    return dfd.promise();
  }

  return $.ajax( '/data/search.json', {
    data : { q: query },
    dataType : 'json'
  }).pipe(function( resp ) {
    return resp.results;
  });
};

Now, we can change our code for getting the results onto the page:

var resultsList = new SearchResults('#results'); 
var searchData = new SearchData();

// ...

searchData.fetch(query).then(resultsList.setResults);

Again, we’ve dramatically simplified our application code, and isolated the complexity within the Search Data object, rather than having it live in our main application code. We’ve also made our search interface testable, though there are a couple caveats to bear in mind when testing code that interacts with the server.

The first is that we don’t want to actually interact with the server—to do so would be to reenter the world of integration tests, and because we’re responsible developers, we already have tests that ensure the server does the right thing, right? Instead, we want to “mock” the interaction with the server, which we can do using the Sinon library. The second caveat is that we should also test non-ideal paths, such as an empty query.

test('constructor', function () {
  var sd = new SearchData();
  assert(sd);
});

suite('fetch', function () {
  var xhr, requests;

  setup(function () {
    requests = [];
    xhr = sinon.useFakeXMLHttpRequest();
    xhr.onCreate = function (req) {
      requests.push(req);
    };
  });

  teardown(function () {
    xhr.restore();
  });

  test('fetches from correct URL', function () {
    var sd = new SearchData();
    sd.fetch('cat');

    assert.equal(requests[0].url, '/data/search.json?q=cat');
  });

  test('returns a promise', function () {
    var sd = new SearchData();
    var req = sd.fetch('cat');

    assert.isFunction(req.then);
  });

  test('no request if no query', function () {
    var sd = new SearchData();
    var req = sd.fetch();
    assert.equal(requests.length, 0);
  });

  test('return a promise even if no query', function () {
    var sd = new SearchData();
    var req = sd.fetch();

    assert.isFunction( req.then );
  });

  test('no query promise resolves with empty array', function () {
    var sd = new SearchData();
    var req = sd.fetch();
    var spy = sinon.spy();

    req.then(spy);

    assert.deepEqual(spy.args[0][0], []);
  });

  test('returns contents of results property of the response', function () {
    var sd = new SearchData();
    var req = sd.fetch('cat');
    var spy = sinon.spy();

    requests[0].respond(
      200, { 'Content-type': 'text/json' },
      JSON.stringify({ results: [ 1, 2, 3 ] })
    );

    req.then(spy);

    assert.deepEqual(spy.args[0][0], [ 1, 2, 3 ]);
  });
});

For the sake of brevity, I’ve left out the refactoring of the Search Form, and also simplified some of the other refactorings and tests, but you can see a finished version of the app here if you’re interested.

When we’re done rewriting our application using testable JavaScript patterns, we end up with something much cleaner than what we started with:

$(function() {
  var pending = false;

  var searchForm = new SearchForm('#searchForm');
  var searchResults = new SearchResults('#results');
  var likes = new Likes('#liked');
  var searchData = new SearchData();

  $(document).on('search', function (event, query) {
    if (pending) { return; }

    pending = true;

    searchData.fetch(query).then(function (results) {
      searchResults.setResults(results);
      pending = false;
    });

    searchResults.pending();
  });

  $(document).on('like', function (evt, name) {
    likes.add(name);
  });
});

Even more important than our much cleaner application code, though, is the fact that we end up with a codebase that is thoroughly tested. That means we can safely refactor it and add to it without the fear of breaking things. We can even write new tests as we find new issues, and then write the code that makes those tests pass.

Testing makes life easier in the long run

It’s easy to look at all of this and say, “Wait, you want me to write more code to do the same job?”

The thing is, there are a few inescapable facts of life about Making Things On The Internet. You will spend time designing an approach to a problem. You will test your solution, whether by clicking around in a browser, writing automated tests, or—shudder—letting your users do your testing for you in production. You will make changes to your code, and other people will use your code. Finally: there will be bugs, no matter how many tests you write.

The thing about testing is that while it might require a bit more time at the outset, it really does save time in the long run. You’ll be patting yourself on the back the first time a test you wrote catches a bug before it finds its way into production. You’ll be grateful, too, when you have a system in place that can prove that your bug fix really does fix a bug that slips through.

Additional resources

This article just scratches the surface of JavaScript testing, but if you’d like to learn more, check out:

My presentation from the 2012 Full Frontal conference in Brighton, UK.
Grunt, a tool that helps automate the testing process and lots of other things.
Test-Driven JavaScript Development by Christian Johansen, the creator of the Sinon library. It is a dense but informative examination of the practice of testing JavaScript.

This week's sponsor: Igloo Software

2013-05-20T17:00:45+00:00

If SharePoint and Facebook had a baby, it would look a lot like Igloo (without the recessive balding or oversharing tendencies). Hosted, managed and with a major release every 90 days, Igloo is a whitelabel content platform built for today’s needs. We also have Sandwich Video.

Rachel Andrew on the Business of Web Dev: You Can’t Do Everything

2013-05-09T12:30:53+00:00

In any given day I can find myself reading up on a new W3C proposal, fixing an issue with our tax return, coding an add-on for our product, writing a conference presentation, building a server, creating a video tutorial, and doing front end development for one of our sites. Without clients dictating my workload I’m in the enviable position of being able to choose where to focus my efforts. However, I can’t physically do everything.

I’m one half of a two-person web development business—the team behind the little CMS, Perch. I’m also an author and speaker on subjects that range from CSS to technical support, and I enjoy all of it.

When we were a service business, what I was actually working on was largely dictated by the requirements of our clients. Whether they wanted to pay me to build servers, manage projects, or write code didn’t really matter. I was exchanging my time for money, doing a range of things I enjoyed. Now that we’re a product company, my greatest challenge is working out where I am best spending my time, while avoiding falling down a rabbit hole of interesting things I have discovered while performing some other task.

The quote that I opened this column with reflects the dilemma I seem to face daily. I can choose to place my attention anywhere. But if I dart around between tasks, none of them get my full attention. At the very least, progress on everything becomes painfuly slow as I spend an hour on one thing and two on another, inching them all forward. I can’t claim to have the perfect solution to managing this problem, but I have started to develop a process for deciding what needs to be done, and whether I am the best person to be doing it.

First and foremost you need to identify what needs doing. I am a great fan of Getting Things Done and regularly review our business and my personal goals, and the tasks that will go into meeting them. Once I have a list of tasks, I can assess them against the following criteria:

Am I the only person who can do this?
Does the business or product benefit from me in particular doing this?
Is this a task I really enjoy doing?
Will I learn anything new by doing this?
What am I not doing if I choose to do this?

Am I the only person who can do this?

Things that fall into group one, the things that only I can do, need investigating. It isn’t ideal for any business to have things that only one person can do. It might be that I need to deal with that task today, but how can I make it so that in the future someone else could? Until the middle of last year, our accounts were a case in point. Although we had an accountant do our end of year tax returns, I was the only person who fully understood the complex processes developed to deal with the many incoming small payments for Perch licenses. Taking on a bookkeeper meant I had to formalize and document all of those processes. As a result I don’t have to do the day-to-day books, but perhaps more importantly the business isn’t reliant on knowledge that is only in my head.

Does the business or product benefit from me in particular doing this?

It can make sense to keep some tasks internal. I wouldn’t completely outsource our technical support, or our social media activity, or even our marketing. The public face of our product is very much about us being a small, friendly business. Our customers get to talk to us, the product developers; we share their frustrations and they help us decide on where to put time into new features. There may well be real reasons to keep certain tasks as a role of the core person or team, even if they would seem straightforward to outsource.

Is this a task I really enjoy doing?

Running a business can involve hard work and long hours. If you feel you have to outsource bits of your job that you love doing because it makes most sense as a business, you may end up pretty miserable. For those of us running small software companies, it’s likely we have ended up here because we like to code. So it’s important to me that I spend some of my time actually writing code—even if it might be more sensible from a business perspective for me to just manage other people who are writing code.

I believe that our products and businesses are better when we love being involved with them. To have a successful business, it’s likely that you will always have important things to do that you find less enjoyable than designing or writing code, however I don’t think we should be beating ourselves over the head. Doing what we love is really what has been behind the success of our product. It is completely ok to hang onto some tasks because you simply enjoy doing them.

Will I learn anything new by doing this?

I might really enjoy a particular project, but I find a helpful way to decide if I should do something or contract it out is to see whether I will learn anything new by doing it myself. For example, I have just sent out a sizeable chunk of front-end development. It is a rebuild of an existing site, and I think there are lots of practical and performance gains to be had by rebuilding it. It would have been nice to have done that work myself, but I wouldn’t have learned anything by doing it. Therefore I made the decision that this would be a good piece of work to outsource to a contractor. I can manage that project and make sure that I’m happy with the end result, but I don’t need to actually write the code.

Our business benefits by us having knowledge and understanding. I’m currently spending quite a lot of time learning about automation (using Puppet) and modern ways of managing systems while rebuilding our infrastructure. I could have brought someone in to do this work for me, and may well do so in future. Yet by updating my systems administration skills, I’m ensuring that within the business we maintain a good level of knowledge about our infrastructure.

What am I not doing if I choose to do this?

As part of a tiny team of two, I’ll always have a number of tasks on the go. Ultimately, choosing to take on one task means not doing something else. It might be another task in the business that gets pushed back. It might be personal things like exercise, or spending time with family and friends. To be able to understand the implications of selecting one thing to work on over another, you need to have a really good overview of all the things that are trying to get your attention.

Having clear business goals and objectives in the first place can make this decision-making so much easier. When you find yourself in the position of being able to do anything, it is so easy to run around picking up tasks and trying to do everything. The trick is to take that step back; to see where you can be more strategic with which tasks you tackle and which you delegate. This approach can help you be far more productive and give you space to enjoy the work you are doing while meeting your business goals.

Karen McGrane on Content: WYSIWTF

2013-05-02T11:43:20+00:00

Arguing for “separation of content from presentation” implies a neat division between the two. The reality, of course, is that content and form, structure and style, can never be fully separated. Anyone who’s ever written a document and played around to see the impact of different fonts, heading weights, and whitespace on the way the writing flows knows this is true. Anyone who’s ever squinted at HTML code, trying to parse text from tags, knows it too.

On one hand, the division of labor between writing and presentation can be seen at every point in our history. Ancient scribes chiseling stone tablets, medieval monks copying illuminated manuscripts, printers placing movable type—we’ve never assumed that the person who produces the document and the person who comes up with the ideas must be one and the same.

And yet, we know that medium and message are intertwined so tightly, they can’t be easily split apart. Graphic designers rail against the notion that “look and feel” can be painted on at the end of the process, because design influences meaning. The more skilled we are as communicators, the more we realize that the separation of content from presentation is an industrial-age feint, an attempt to standardize and segment tasks that are deeply connected.

Today, we try to enforce the separation of content and form because it’s good for the web. It’s what makes web standards possible. It enables social sharing and flexible reuse of content. It supports accessibility. It’s what will keep us sane as we try to get content onto hundreds of new devices and form factors.

When talking about how best to separate content from presentation, designers and developers tend to focus on front-end code—which makes sense, because that’s what we have the most control over. But, as with so many challenges we have with content on the web, the real issue lies in the tools we give content creators to help them structure, manage, and publish their content. The form that content takes depends as much on CMS as it does on CSS.

How should content management tools guide content creators to focus on meaning and structure? What’s the right amount of control over presentation and styling in the CMS? And how should these tools evolve as we break out of the web page metaphor and publish content flexibly to multiple platforms? Let’s look at three tools that sit at the intersection of content and form.

Preview button

Even the most die-hard structured content editors still like seeing what their work is going to look like. Writers print out documents for editing to give them a different view from what they see on the screen. Bloggers instinctively hit the preview button to look at their work the way a user will see it.

Whoops. Decades of work refining the emulators between desktop publishing programs and laser printers means that writers can feel confident that their document will look virtually identical, regardless of where it’s printed. We’ve carried that assumption over to the web, where it’s categorically untrue. Different browsers render content in their own vexingly special way. Users can change the font size—even add their own custom style sheet. Today, the same document will render differently on desktops, tablets, and mobile devices. The preview button is a lie.

Yet we can’t just throw the baby out with the bathwater. In fact, seeing content in context becomes even more important as our content now lives across devices and platforms. Instead of throwing up our hands and saying “preview is broken,” it’s time to invent a better preview button.

One publishing company I know of has built its own custom preview rendering interface, which shows content producers an example of how each story will appear on the desktop web, the mobile web, and an app. Is it perfect? Far from it. Content will appear in many more contexts than just those three. Is it better than nothing? Absolutely.

WYSIWYG

The desktop publishing revolution ushered in by the Macintosh allowed the user to see a document on screen in a form that closely mirrored the printed version. The toolbar at the top of the screen enabled the user to add formatting—change the font, insert an image, add typographic effects like headings and bullets, and much more.

In an effort to carry over this ease of use to the web, we allow content creators to embed layout and styling information directly into their content. Unfortunately, the code added by content creators can be at odds with the style sheet, and it’s difficult for developers to parse what’s style and what’s substance. When it comes time to put that content on other platforms, we wind up with a muddled mess.

What is the right amount of formatting control to give content creators? That’s a difficult question to answer, because it pierces right to the heart of what’s stylistic and what’s semantic. Even something as simple as adding bold and italic text forces us to ask if we’re really just styling the text, or adding semantic meaning (say, a book title or a warning message.)

Better content modeling can solve some of these problems, encouraging content creators to appropriately “chunk” their text. By banishing blobs of text with formatting embedded and replacing them with chunks of clean, presentation-independent content, we’re building in the distinction between content and form right from the start.

But imagining that each “chunk” of content is a field in the database (with its own input field) rapidly devolves into the absurd. That way lies madness. The real solution isn’t necessarily to “banish blobs,” but to replace the WYSIWYG toolbar with semantic markup. Rather than entering all text into discrete fields, content authors wrap text that describes what it is. Our book title doesn’t need to be a separate field if we can wrap it in the proper tags.

Defining what goes in a field and what goes in a tag requires a tighter collaboration between content authors, CMS architects, and front-end developers. It’s time we started having these conversations.

Inline editing

We’re evolving. Not satisfied to rely just on tools that are vestiges of the desktop publishing era, we’re developing new and innovative ways to mix up content and formatting that are unique to the way the web works. There’s no better example of this than inline editing.

Inline editing allows content creators to directly manipulate content in the interface, with no separation between the editing screen and the display. Medium offers an editing interface that’s identical to the desktop display and in-place editing is being added to Drupal 8 core.

One of the questions I get asked most frequently is “how can I get my content creators to understand why it’s so important to add structure and metadata to their content?” This, I believe, is one of the fundamental challenges we’re facing on the web, particularly as we adapt to a multi-channel future. Inline editing encourages content creators to focus on the visual presentation of the desktop interface. Just at the moment when we need content creators to think about the underlying structure, we’re investing in tools that obscure the “connective tissue.”

Jeff Eaton sums up this problem nicely in a post called Inline Editing and the Cost of Leaky Abstractions:

The editing interfaces we offer to users send them important messages, whether we intend it or not. They are affordances, like knobs on doors and buttons on telephones. If the primary editing interface we present is also the visual design seen by site visitors, we are saying: “This page is what you manage! The things you see on it are the true form of your content.”

The best solution isn’t to build tools that hide that complexity from the user, that make them think that the styling they’re adding to the desktop site is the “real” version of the content. Instead, our goal should be to communicate the appropriate complexity of the interface, and help guide users to add the right structure and styling.

The era of “desktop publishing” is over. Same goes for the era where we privilege the desktop web interface above all others. The tools we create to manage our content are vestiges of the desktop publishing revolution, where we tried to enable as much direct manipulation of content as possible. In a world where we have infinite possible outputs for our content, it’s time to move beyond tools that rely on visual styling to convey semantic meaning. If we want true separation of content from form, it has to start in the CMS.

Even Better In-Browser Mockups with Node.js

2013-04-30T12:00:42+00:00

Designing in the browser has all sorts of benefits, like producing more accurate, comprehensive results and removing the extra step of converting from image file to markup and CSS. But even sites designed in a browser still require pasting in content, faking interactions with the server, and creating placeholder JavaScript that isn’t usable on the live site.

Wouldn’t it be nice if we could go from just designing layouts and interactions to designing the whole client side of the application during the same process?

This is where Node comes in.

Node.js is a server-side JavaScript platform. It isn’t a web server, but it allows you to easily create one. It also lets you create utilities that run on web servers, like setup and minification utilities and general-purpose command line tools.

Node started in 2009 and generated considerable interest, probably because it gave JavaScript developers an opportunity to write server-side code even if they lacked a server-side background. It didn’t hurt that Chrome had established a reputation for being solid and fast, and Node used its V8 engine.

Today, it’s possible to run production servers on Node, and many people are doing so successfully. Taking it that far, however, is an investment. The Node project, and all the community-created modules that make it so awesome, is still a moving target. But even if you’re not ready to write and launch entire sites with Node, it’s plenty stable enough to use as a development tool.

It’s JavaScript, so if you can wire up a jQuery event handler, you can write a web server. It’s lightweight, so you can run it on your laptop and keep streaming music in the background. It’s dead simple to download, set up, and build in, so you don’t need the esoteric knowledge of an IT support person to get going with it. And the payoff is that instead of mockups and hard-coded data, you can design a set of client-side assets, page templates, and data schemas that are ready to launch to production.

Getting started

Installing Node locally for the most common environments is a piece of cake. You can download installers that include Node as well as npm, its package manager, from the project’s site. Installing it on a remote server is not quite that easy, but good documentation is available to help you out. After running through the installation process, you should be able to go to your terminal or command line and test it out.

If you don’t tell Node to run a specific file, you get a Read-Eval-Print Loop, or REPL. If you type node in your terminal or command prompt, you can begin to execute arbitrary JavaScript. For example, after starting the REPL, type var a = 9;. The REPL will respond with undefined. Now type a * 3 (or any other number) and it will respond with the correct result. You can make things more interesting by defining a function and then calling it:

> function sayHello( name ) { return “Hello, “ + name; }
undefined
> sayHello( “A List Apart” );
‘Hello, A List Apart’

To break out of the REPL, or end any other Node execution (like a running web server), press Ctrl+C. In the case of the REPL, you’ll need to press it twice to confirm.

While it’s nice to know Node can perform basic arithmetic and string concatenation, its value to us as developers is in running programs. You can see an example of one such program, a basic web server, on the project’s homepage. They suggest creating a file called example.js with this code:

var http = require(’http’);
http.createServer(function (req, res) {
  res.writeHead(200, {’Content-Type’: ‘text/plain’});
  res.end(’Hello World\n’);
}).listen(1337, ‘127.0.0.1’);
console.log(’Server running at http://127.0.0.1:1337/’);

This makes use of only one module, the core module http. As you can probably guess, the http module contains all the basic stuff you need to serve a site over HTTP. Node contains a tightly edited collection of core modules that provide things like event handlers, file system access, and abstractions for various network protocols. But just as you probably use a JavaScript library or framework to speed up and bulletproof development on the client side, for Node development beyond a simple "Hello World" you generally add other non-core modules using npm.

The http module does contain a createServer function, though, which is all you need to create a bare-bones web server. In the code above, once the server has been created, it listens to port 1337 on your local machine. When it receives a request, it sends back a text response.

One thing to note is that the work here is done in event handlers, as are most things in Node. The callback in createServer() handles a connection event, which occurs every time a new client contacts the server. To start this server, type node example.js in your terminal, and then open a browser to http://127.0.0.1:1337. This triggers the connection event, and you should see the message in the callback.

To obtain any serious value from a Node server—even one not intended to ever go to production—it’s best to get familiar with the modules in npm. There are thousands available, but those you’d need to create a basic web application are some of the oldest and most stable, so don’t feel obligated to research them all before getting started. One that definitely comes in handy for designing an application is Express, an uncomplicated web application framework.

If you’re accustomed to installing open source projects by cloning a GitHub repository or downloading a zip file, you’ll probably enjoy npm. To install Express with npm, for example, go to your terminal or command line and type npm install express. As long as you’re online and have permission to write to your machine, this fetches all the code and assets Express needs to run, as well as any modules it has as dependencies. The first time you run npm from your working directory, all these elements will end up in a new node_modules subdirectory. Now the module is ready to be used in Node programs the same way we used http, via the require function.

Designing applications

The ideal use case for designing your application with Node is a single-page application in which the server mostly provides data, but Node is still useful for a more traditional site. Of course, you want to begin development with requirements defined as precisely as possible, but implementation tends to expose requirements you hadn’t considered, and some of those can have a considerable impact on your timeline. Even in a server-driven application where it may not be possible to reuse data structures and templates as-is, creating client-only versions helps test your assumptions about the data you need and how you’ll use it.

If you’re developing a single-page app, the justification is much easier. You’ll need to think about optimizing your communication with the server to require as few requests as possible, which means knowing how data should be packaged up by each server endpoint and how you’ll cache the data on receipt (if at all).

An advantage of having JavaScript on the server is that templates can be rendered by the same template engine on either the client or server side. This allows you to experiment on both sides and optimize for your situation. It’s also a timesaver to render the templates with JavaScript objects and consider only the way data must eventually be grouped (not how it’s stored in a database). Designing these groupings of data is the bulk of the work we can do with Node toward the end of what we traditionally consider application design.

Piecing together each page from disparate parts from all over the server is messy for an application in any language. Instead, whatever renders a page should have clear dependencies, and the result of each page or data request should combine those dependencies into a cohesive and sensibly organized unit.

If you’ve worked in a server-side framework in which a page or view is tied to a single object or model, and where additional data is imported and exposed in a different way, you probably understand how the alternative gets to be a nuisance. You’re probably also aware that the solution is a good view-model whose data is defined by each view, not the models that feed it. With this exercise, we aim to map out what goes in those view-models.

Using templates

There’s a strong likelihood that your production server does not run JavaScript, so you may end up converting templates you produce in this design phase. You could attempt to mitigate this by choosing a template engine like Mustache with existing parsers for a huge list of languages. Or you might choose one with minimal logic available (I find that the only things I want for a truly flexible template are conditionals, loops, and partials) and the option of changing the delimiters around the template tags to agree with your server template language. I’d argue that the process of getting all the data correctly placed in your HTML is a lot more difficult than doing a find and replace on the end result, so creating a template in JavaScript that you can use easily is time well spent even if it can’t be parsed by your production server.

You could choose to design the UI of your pages using hard-coded mockup data first and add the template tags afterward, or you could start with a template and some mockup data ready to go in your Node server. Even though it’s an extra step, I find the former easier, because the latter tends to require extra shifting of the mockup data. Starting with hard-coded data lets me examine the finished mockup and see what kinds of "objects" are present (e.g., a user, an item for sale, or a status update). That helps me create a flexible object hierarchy in my data more easily. But you may be naturally amazing at creating object hierarchies, so, by all means, do what you feel.

Wherever you begin, hammering out your templates should give you an indication of which parts of each page are dynamic and which data each requires. If subsections of your pages are rendered separately because they’re reused on different parent pages or because they’re also rendered by the client, converting your markup to templates also allows you to find the right balance between never repeating code and having absurdly tiny templates.

Real fake server interactions

If you’ve created high-fidelity wireframes that run in a browser, you know the annoyance of having only parts of a page be interactive, since every click means having to create a new view (even if you have a series of items that share the same behavior when clicked). You also know about copying and pasting the same data into multiple places, and updating each of them separately if the manner of presenting data should change. Designing your app with a server behind it removes those frustrations.

With the support of a server, it’s not a problem if the same data shows up in different displays all over the workflow you’re designing. Since the data lives on your Node server, you can write it once and reuse it as many ways as you like. You still have to consider how you’ll move it from server to client, though. When a user clicks on one of many items in a list, will she be taken to a new page, or will more data appear inline? Is the former the non-JavaScript fallback for the latter? Working that out for your app will tell you which endpoints the server needs, and which parameters need to be sent to it in query strings, form posts, or URLs. It’ll also help define the API for those requests, telling anyone who might work on your production server which keys you expect to correspond to which pieces of data.

Having a server to work with is especially nice if you’re in the business of making asynchronous requests. Obviously, you can get your mockup data, which is excellent, but you can also lazy-load assets like templates or stylesheets to consume that data. Testing the process of getting data and assets to the client validates your assumptions about not only the way you’re packaging them, but how you’re storing and structuring them. And, of course, it means a lot less wasted client-side JavaScript.

A mockup you can use

The end result of all this should be that you’ve moved all the mockup pieces out of your client-side JavaScript and HTML. You have a Node server that might not match your production framework, but does have clear definitions of everything the client side expects to exist—possibly even all viewable in a single file. You have templates and client-side requests that may require substitutions, but also separate your data from everything else and are at minimum easily convertible to whatever format is needed for production.

Could you do the same with any other server under the sun? Absolutely. But if you already know JavaScript and aren’t aiming to become a server-side developer, it makes sense to use the skills you already have. Node makes that pretty easy, while also letting you dig as deeply as you want into more complex servers, should your ambitions change. It’s simple to get going and flexible to extend, making Node an awesome tool for designing applications.

Ready to take your new Node skills for a spin? In “Node at Work: A Walkthrough,” I’ll take you through a live demo, and get specific about how to refine your own mockups.

Node at Work: A Walkthrough

2013-04-30T12:00:51+00:00

In my first article, “Even Better In-Browser Mockups with Node.js,” I explained why Node.js makes designing applications easier and more efficient, and how to get started. Now it’s time to see your new design process in action.

Rather than figuring out all your requirements and API schemas just to design your comps with mockup content hard-coded and server interactions faked—only to throw it all away when you go back and implement things “for real”—you can use Node.js to skip the hard-coding and produce client-side code that’s ready for beta at the end of the design stage.

The process looks a lot like good ol’ designing in the browser, but with more JavaScript and an additional layer:

Design the layout and styling
Convert the markup to a JavaScript template
Create an initialization function
Create a simple Node.js server
Add a mockup data object to the server
Add server functions to serve static pages and JSON
Request and consume the JSON on the client

Sound daunting? Don’t worry. The first step takes approximately a zillion times longer than any of the others. So if you’ve already mastered the design, you’ll find the rest of these steps more than manageable.

In this walkthrough, we’ll build a feature for a mock art store. If you want to follow along at home, you can clone my GitHub repository. (If you need help installing, see the README, or just take a peek at the live demo—I’ll cover all the steps and code below.)

Creating templates

Once you have a solid design and the markup to accompany it, converting it to a template you can use for all examples is more efficient than creating duplicate markup for each one. The hard part’s over; you already thought about where data points would be used in the design when you created it. With those choices fresh in your mind, go back and mark up your HTML with data in whatever template language you prefer.

For my example, I’m using a store selling art prints. Here’s a snippet of my initial markup:

<h2>Two Acrobats with a Dog</h2>
<h3>Pablo Picasso</h3>
<img src="img/102.jpg" alt="Two Acrobats with a Dog" class="active" />
<ul class="info">
	<li>8" x 11"</li>
	<li>acid-free paper</li>
	<li>suitable for matting</li>
</ul>
<span class="price">$49.99</span>

Think of your templates as places to define your requirements for both data and its formatting on the client side. If you can also reuse it for client-side rendering, that’s awesome—but that may not be relevant to your application. As long as you have good data, converting from one template language to another is trivial, so don’t agonize over which template engine to use.

You do need a template engine that will work in both the browser and Node.js, however. If you’re unsure, search for your template engine on GitHub and verify that there’s a guide to installing it via npm in the manual, as well as a minified script for use on the client. I prefer doT.js, so here’s that snippet again marked up to add data using doT:

<h2>{{=it.title}}</h2>
<h3>{{=it.artist.name}}</h3>
<img src="img/{{=it.id}}.jpg" alt="{{=it.title}}" class="active" />
<ul class="info">
	{{~it.info :info_item}}
	<li>{{=info_item}}</li>
	{{~}}
</ul>
<span class="price">{{=it.price}}</span>

I like to save my templates in their own directory at the same level as my JavaScript directory, so now I store that as tmpl/detail.dot.

Initializing the client

Since we want to be able to use our templates in both Node and the browser, they need to be stored outside of the HTML and loaded and compiled when we open the page. To start, save the minified version of your template engine and add a script tag to your page to include it. Once that’s done, you can fetch the template, compile it, and then continue on with any other initialization work in your main JavaScript file. I’m using jQuery in my example, so my code looks like this:

var detailTmpl;

$.when( 
	$.get( "tmpl/detail.dot", function( tmpl ) {
		detailTmpl = doT.template( tmpl );
	}, "text" ) 
).then( init );

That mysterious init function? That’s where I’ll put any interactivity I want to add to my currently static mockup. For the moment I’m only creating one interaction, so my init function is pretty simple:

function init() {
	$( "div.content" ).on( "click", "div.result", showDetail );
}

This code can be made much more elegant using Require.js with its text plugin. That’s beyond the scope of this demo, but I highly encourage it for production.

We’ll handle template rendering in showDetail(), but we have to add a server and data store before writing that function, since right now we lack any data to render.

Creating a server

If I reload my page now and open the browser console, I get a JavaScript error. That’s because I’m trying to load my template via an XMLHttpRequest (XHR) on a page being served from the file system, in violation of the same origin policy. I can’t even check that my template works until the page is served properly (i.e., from a server).

To whip up a simple Node server that allows me to run my XHRs, I do a few things:

Move all my existing assets into a new subdirectory called public
Open my terminal or command line to my working directory and run npm install express
Add a server.js file to the working directory

We could write everything from scratch, of course, but it’s more work than is necessary for a basic server. The Express framework provides a number of abstractions of server and application concepts. For the initial version of the server, the only one we’ll need is its ability to serve static resources. We can use it by adding four lines of code to server.js:

var express = require( "express" ),
	app = express();

app.use( express.static( __dirname + "/public" ) );

app.listen( 3000 );

Once you start your server by typing node server.js in your open terminal or command line, you can view your page at http://localhost:3000 (adding a filename if necessary), and the error related to loading the template ought to disappear.

Adding server-side data

While it’s certainly nice to be able to use XHRs, we’re creating the Node server to use it as a representation of the real server—and real servers store data. Though it’s not hard to create a data store that works with a Node server, it’s even less difficult to create one big object literal. For a mockup, that’s all we really need. One of the goals here is to define the data objects we need to support in our new design, so the format of this object can be determined by the template we just added. For my example, I need an object structured something like this:

var products = {
	"102": {
		id: 102,
		title: "Two Acrobats with a Dog",
		artist: {
			name: "Pablo Picasso"
		},
		price: "$49.99",
		info: [
			"8\" x 11\"",
			"acid-free paper",
			"suitable for matting"
		]
	}
};

Note that products could just as easily be an array, but I want to be able to quickly find my products—once I have more than one in my fake data store—by ID. Aside from that little twist, the data look exactly like the content hard-coded in my original HTML. If I want to add more data, including things that might break the layout in unpredictable ways, I can just copy this structure and make substitutions. Well, almost.

Returning data from the server

If you’ve dealt with other server-side frameworks, creating endpoints for XHRs might seem intimidating, but Express makes it really easy. We don’t need any special setup to define a server endpoint as a target for asynchronous requests. All we have to do is define the path on the server where we want to accept requests and a callback. The callback receives a request object (for doing things like getting passed-in data) and a response object (for defining what we return to the client). To return the data in my products object, I add a few lines of code at the bottom of server.js:

app.get( "/detail/:id", function( req, res ) {
	res.send( products[ req.params.id ] );
});

app.listen( 3000 );

See? Easy. If I restart my server and go to http://localhost:3000/detail/102, I should see my object data. To break down what’s going on with the ID in the path, we’ve named the data at that position in the path "id" with the :id bit, and it then becomes available as a property of req.params.

The names and positions of parameters are up to us, and if our path were super complex, we could also use regular expressions to split out multiple pieces of data. Express also gives us the option of accepting data from the query string or from a POST. Of all the pieces we’re creating, however, the paths are the most likely to change in production, so it’s to our advantage to keep them as readable as possible.

Besides sending pure data to the client, we also want to be able to send rendered HTML, in case a user is linked directly to a product detail or doesn’t have JavaScript available. We might also want HTML for our own consumption via XHR, if we find that client-side rendering is slowing us down. So we add a second endpoint below the one we just created to do that:

app.get( "/product/:id", function( req, res ) {
	res.render( "detail", products[ req.params.id ] );
});

For simplicity’s sake, and because the first path served JSON for an overlay while this provides a full page, I’ve used a different pathname, but kept the same pattern. This time, instead of the response’s send function, I use render(). Express provides some magic to make template rendering work out of the box, but since I’m using doT instead of Jade (the default template engine of Express), I have to do some additional setup.

First I have to go back to the terminal or command line, stop my Node server, and install my template engine using npm install doT and the consolidate module (which provides Express compatibility for a number of popular template engines) using npm install consolidate. Now I’ve got both of those in my node_modules directory and can use them in server.js.

Since doT (and probably your template engine of choice, as well) is accessed through consolidate, consolidate is the only additional module I need to require at the top of server.js:

var express = require( "express" ),
	app = express(),
	cons = require( "consolidate" );

I want to continue serving some of my other pages statically, so I add my template configuration stuff below the existing app.use line in my code:

app.use( express.static( _dirname + "/public" ) );
app.engine( "dot", cons.dot );
app.set( "view engine", "dot" );
app.set( "views", _dirname + "/public/tmpl" );

Those three new lines set doT (as exposed by consolidate) as the view engine, register files ending in .dot as templates, and tell Express to look in /public/tmpl for templates to use. So when Node sees res.render( "detail", { ... } ), it knows to expand "detail" to /public/tmpl/detail.dot and render it as a doT template. Now I can restart my server, go to http://localhost:3000/product/102, and see my template rendered statically, without creating a separate server-side file.

Fetching dynamic data

Our template now works as a static page, but there’s still one more step to get our mockup populated with the data from the server. Remember the showDetail function from our main client-side script? It’s time to flesh that out.

In my simple example, the overlay my template will populate already exists as a hidden div on the main page, and it appears when the user clicks a div containing a summary of the content. This div has a data attribute storing the ID of the product that corresponds to the key and id property in my server-side data object. Once that click event happens and showDetail() is called, I just need to do this:

function showDetail( e ) {
	var id = $( this ).data( "id" );
	$.get( "detail/" + id, function( info ) {
		$( "div.detail" ).html( detailTmpl( info ) );
		$( "div.detail" ).show();
	}
}

The path above is the same one I defined in server.js. If you chose a different name for yours, use that name here on the client. When I receive the data object from the server, I pass it to detailTmpl(), the compiled version of my template. The result of the detailTmpl function is the HTML to populate my overlay.

Onward

So there you have it! A mockup that mimics the interactions it will have with its production server precisely on the client, without the need for hard-coded data or temporary workarounds. Despite the simple exercise, the process I’ve outlined accomplishes a good deal of the setup necessary to create other workflows that require server interactions. For instance, I can fill my fake data store with more products and use that to render the initial page that triggers my overlay without having to revisit my mockup data, and my application will show the correct values in any view I add to it.

If you’d like to explore beyond just serving HTML and JSON, consider adding in Socket.io to allow real-time interaction for multiple clients or Require.js to manage your assets on the client. You could also move your CSS into templates and serve different builds of your site for different browsers or devices. Your mockup can be as sophisticated and reflect as many of its production requirements as you choose. At the end, the lion’s share of your client-side code is done and ready to use.

The W3C on Web Standards: Digital Publishing and the Web

2013-04-25T12:29:00+00:00

Electronic books are on the rise everywhere. For some this threatens centuries-old traditions; for others it opens up new possibilities in the way we think about information exchange in general, and about books in particular. Hate it or love it: electronic books are with us to stay.

A press release issued by the Pew Research Center’s Internet & American Life Project in December 2012 describes an upward trend in the consumption of electronic books. The trends are similar in the UK, China, Brazil, Japan, and other countries.

“…the number of Americans over age 16 reading eBooks rose in 2012 from 16 to 23 percent, while those reading printed books fell from 72 percent to 67. …the number of owners of either a tablet computer or e-book reading device such as a Kindle or Nook grew from 18% in late 2011 to 33% in late 2012. …in late 2012 19% of Americans ages 16 and older own e-book reading devices such as Kindles and Nooks, compared with 10% who owned such devices at the same time last year.”

What does this mean for web professionals? Electronic books represent a market that’s powered by core web technologies such as HTML, CSS, and SVG. When you use EPUB, one of the primary standards for electronic books, you are creating a packaged website or application. EPUB3 is at the bleeding edge of current web standards: it is based on HTML5, CSS2.1 with some CSS3 modules, SVG, OpenType, and WOFF. EPUB3’s embrace of scripting is sure to encourage the development of more interactivity, which is sought after in education materials and children’s books.

Recently W3C has been working more closely with digital publishers to find out what else the Open Web Platform must do to meet that industry’s needs.

One comment we’ve heard loud and clear is that people care deeply about centuries-old print traditions. For example, Japanese and Korean users have accepted that many websites display text horizontally, from left to right. While that may be ok for the web, when these users read a novel, they expect traditional layout: characters rendered vertically and from right to left. Japanese readers often find it more tiring to read a long text in any other way. To address these requirements, W3C is looking at the challenges that vertical layout poses for HTML, CSS, and other technologies; see for example CSS Writing Modes Module Level 3.

Requirement of Japanese Text Layout summarizes the typesetting traditions and resulting requirements for Japanese. These traditions should eventually be reproduced on the web as well as in electronic books. In June, W3C will hold a digital publishing workshop in Tokyo on the specific issues surrounding internationalization and electronic books.

We have also heard that the “page” paradigm—including notions of headers, footnotes, indexes, glossaries, and detailed tables of contents—is important when people read books of hundreds or thousands of pages. Web technology will need to reintegrate these UI elements smoothly; see for example the CSS Paged Media Module Level 3 (Joe Clark talked about paged media and the production of ebooks in 2010, and Nellie McKesson gave us an update in 2012). In September in Paris, W3C will hold a workshop on the creation of electronic books using web technologies. Note that both this and the Writing Modes Module are still drafts and need further work. That means now is the right time for the digital publishing community to have its voice heard!

In the realm of metadata, important to publishers, librarians, and archivists, the challenge is to agree on vocabulary (and there are many: Harvard’s reference to metadata standards is only the tip of the iceberg). Pearson Publishing recently launched the Open Linked Education Community Group to examine creating a curated subset of Wikipedia data that can be used for tagging educational content.

Here are a few other places to look for activity and convergence:

People take notes in books and highlight text. Most ebook readers these days have built-in support for these features, but they are not widely deployed on the web.
Today search engines tend to ignore electronic books; I expect that will not remain the case for long.
“Offline mode” in web technology is still difficult to use if you try to access more than a single page of a site. Since an ebook is quite often a packaged website, ebook offline mode will need to improve to support browsing.
ebooks business models are likely to drive new approaches to monetization, some of which may be found in native mobile environments but not yet on the web.

Although publishing has some specific requirements not common to the web generally, I think that the distinction between a website (or app) and an ebook will disappear with time. As I have written before, both will demand high-quality typography and layout, interactivity, linking, multimedia, offline access, annotations, metadata, and so on. Digital publishers’ interest in the Open Web Platform is a natural progression of their embrace of the early web.

David Sleight on New-School Publishing: Passing On Our Rights

2013-04-18T12:30:02+00:00

Last month, a U.S. District Court handed down a decision that’s pretty awful if you care about consumer rights and digital content.

It all started in 2011, when a company called ReDigi launched a service to let folks resell their unwanted iTunes purchases—the digital equivalent of unloading your old vinyl at a swap meet. This annoyed the legal department at Capitol Records enough that they sued ReDigi in federal court to stop it. Unfortunately for consumers, Capitol Records succeeded. This isn’t just bad news for ReDigi though. What’s really troubling is the court’s take on current copyright protections.

The ReDigi case

When it comes to the CDs, DVDs, and paper books you own, U.S. law is clear. A legal concept called the first-sale doctrine establishes your right to sell them to another person, provided you’re handing over the item you originally bought, and that you didn’t make any copies. That’s the idea behind garage sales, swap meets, and Craigslist. As repeated by the court’s own decision in the ReDigi case, first-sale doctrine states:

“The owner of a particular copy or phonorecord lawfully made…, or any person authorized by such owner, is entitled, without the authority of the copyright owner, to sell or otherwise dispose of the possession of that copy or phonorecord.”

That “owner” is you. The “copyright owner” is the record label, movie studio, or publisher. They make it. You buy it. You can sell it to somebody else if you don’t want it anymore. Simple. But according to this court, you don’t have that right if you send that movie, song, or book over an electronic network when you resell it.

Wait a minute. How’d that happen?

In the district court’s view, any transfer that takes place via the internet creates a reproduction of the work on the receiving machine, a new physical object that is “embodied” on the buyer’s hard drive. And that constitutes an illegal copy. In the judge’s own words:

[W]hen a user downloads a digital music file or “digital sequence” to his “hard disk,” the file is “reproduce[d]” on a new phonorecord within the meaning of the Copyright Act. Id.

This understanding is, of course, confirmed by the laws of physics. It is simply impossible that the same “material object” can be transferred over the Internet. Thus, logically,… the Internet transfer of a file results in a material object being “created elsewhere at its finish.”

Yeah, you read that right. According to this decision, electronic transfers generate a new “material object” on the receiving device. In legal terms, that’s a copy, and a violation of copyright law.

Even when products like ReDigi’s take reasonable measures to remove the file from the seller’s hard drive as it’s transferred, the court says it doesn’t count. Because I can’t physically hand you the original item over an electronic network, they say all bets are off.

The implications

This creates a world where we’re barred from ever transferring digital goods we own to another person if we use an electronic network. Insert internet, lose first-sale rights. The jury is still out on whether this interpretation really works from a legal standpoint. The scope of court decisions on copyright are typically papercut-narrow and excruciatingly literal. But the U.S. District Court for the Southern District of New York isn’t really the one causing all the trouble here. The main culprit is the law itself.

The copyright law at play in Capitol Records v ReDigi predates the digital world by decades, and still talks about usage rights strictly in terms of “material objects.” It doesn’t recognize a difference between photocopying a printed book and reconstructing one from a set of bit-flipping instructions. The letter of the law hasn’t caught up with the reality we live in. Right now it’s Zoolander smashing a tangerine iMac, hoping a bunch of paper files spill out.

That might not seem like a big deal now. But outdated laws aren’t just discordant with the times, they’re minefields filled with unintended consequences. Just think about your web browser. Anyone want to chat with the district court about the “material objects” a typical browser cache leaves on your hard drive? Unless you feel like upending some of the basic mechanics of the internet, I’d rather you didn’t.

And what happens as we shift to a future where the majority of our access to content is digital? Will we see the shuttering of all secondary resale markets? Who gains and who loses from that? If I can’t transfer the digital goods I own over the network, what happens when I die? Will my executor have to ship my hard drive to my next of kin because she couldn’t transmit its contents to them without risking legal action?

Fixing this mess

So far we’ve largely ducked these questions by letting content companies push us toward rental/lease models online, where we constantly pay and repay for content without enjoying full rights of ownership. That’s okay for some things. But by ignoring the bigger picture on ownership, we leave ourselves open to even further reductions in the consumer rights we’ve enjoyed for decades on movies, music, and books. We need to avoid a world where the only right we have left is the right of refusal.

Fixing the language of creaky old laws takes legislators, not courts. But we can do better than just petition Congress to drag copyright into the current century. We can set the standard ourselves. As entrepreneurs, developers, and technologists building products around digital content, we can incorporate progressive terms into our services. We can choose to grant users better rights than current law permits by default. We don’t have to sit around waiting for the next ReDigi case to tell us how to treat our customers. We can, and should, do better than that.

The first-sale doctrine needs to be defended in the digital space. We need acceptable, liveable standards for ownership of digital goods, and we can start building them now.

Cennydd Bowles on UX & Design: Hellish Other People

2013-04-11T11:30:39+00:00

Childish, inaccurate, bizarre, and condescending? Perhaps—but you can’t just ignore articles like that. Tomas Chamorro-Premuzic’s Seven Rules for Managing Creative People¹ has caused some serious ripples. The article sets lofty standards for missing the point, misrepresenting creative industries to the point of infantilization. At its nadir—“Creatives enjoy making simple things complex, rather than vice versa”—it ranks among the most baffling things ever written about creativity.

Commenters have heaped scorn on poor Chamorro-Premuzic, to the extent that I must almost apologize for adding to the criticism. But I’m intrigued by the views that prop up articles like this. Why do these misconceptions about creative work persist in an era of supposed innovative enlightenment?

The premise that underpins this and many similar articles is that creativity is a binary property: some people are blessed (or cursed) with it, others aren’t. This establishes a subtle, unwelcome construct. Creative types are “The Other”: fundamentally irregular people who don’t quite gel with the rest of society (“The Same”). Indeed, what it means to be Same is usually defined in part by not being Other.

While the language of Otherness is sometimes a deliberate tool of oppression, more often it reflects the unthinking bias of the speaker and era. Chamorro-Premuzic’s framing of creative people as The Other is no doubt unintentional. But the archetype is clear nonetheless.

Never, ever, ever let them call you a “creative”. It’s a way to be disenfranchised. You are a designer. It’s not magic, it’s a trade.
— Mike Monteiro (@Mike_FTW) April 7, 2013

On this stage, Chamorro-Premuzic plays the role of ethnographer. Having observed creative people in their natural environments, he uses the article to MBAsplain their aberrant behavior. Rather than recognizing the diverse individuality of creative workers, the author describes them as a homogenous group primarily defined by weakness—a motif that echoes throughout the history of Otherness.

Chamorro-Premuzic’s creative Other is self-centered and unable to play well with peers. He is “moody, erratic, eccentric, and arrogant.” He is unable to properly explain his process. His motivations are bizarre: money doesn’t matter to him, which at least gives you an opportunity to stiff him on pay. His neuroticism and narcissism make him a poor leader: that’s best left to the Sames.

Thankfully, the premise is flawed. Creativity is not a binary ability but a muscle that needs exercise. There is no Same and Other, no us and them; everyone has creative capacity. Personality, environment, and other pressures of life mean that some people do have less creative experience, but with simple tools—a pencil, a guitar, a hobby—it’s not hard to reverse the atrophy.

Modern creative work demands diverse perspectives. To suggest instead that it emanates from the abracadabras of an eccentric elite does a disservice to all parties. The article argues that companies should pass “trivial or meaningless work” to the clock-watchers who lack intrinsic motivation. In other words, some people are best handling the drudgery, while the unmanageables get the rewarding work to keep them quiet.

This situation, dare I say, could use some creative thinking. It can’t be healthy to encourage a cycle of disinterested employees and futile effort. Better to look closely at user needs, explore ideas that address them, and build a team that is inspired by those goals. Then no work should be meaningless.

So how can we counter the assertion that creative workers think and behave the same? By highlighting the diversity of our personalities and methods.

Both Mark Boulton and Chris Coyier have recently written about their introversion and how it affects their creative approaches. These are important, refreshing viewpoints, and no doubt many readers will identify with them. I too have noticed the subtle assumptions some people make about creative workers. Extraversion and arrogance are often presupposed.

Earlier in my career, bosses and peers encouraged me to use collaborative, extravert-friendly tools like design games. I never felt particularly comfortable with them, but persisted, believing them to be techniques that designers were meant to enjoy.

I’ve since realized these aren’t the right tools for me, at least for today. I’m more comfortable working with trusted teammates in managed environments, delving into people’s expectations, and exploring initial concepts alone. It was a relief to free myself from the process expectations of others, and I believe my results testify for my new approach.

After the lashing the article received in the comments, perhaps Chamorro-Premuzic will pause to reflect on his opinions. Perhaps the Harvard Business Review will also look more closely at its editorial policy, and consider whether it fell a little out of step with its audience. But there will be plenty more articles that treat creativity like a disorder, and more executives who brand creative workers as vain and immature.

With diversity issues rightly making headlines in our industry, we should also rejoice in the multitude of personalities and approaches that make up our disciplines. We are, of course, the best placed people of all to counter harmful stereotypes. Only variety can break us out of the Otherness box.

Growing Your Design Business

2013-04-09T12:00:26+00:00

So you’ve launched your own creative business, and you’re starting to grow. That’s great! But good growth won’t just happen. Just like a junior designer starts with small projects and slowly builds her skills, a new business needs time to mature, test new ideas, and prepare itself, too.

How did you gain the design chops that got you where you are today? With study, practice, and testing, I imagine. Business owners learn their trade the same way: by taking general business wisdom, applying it to their specific niche, and working diligently until they get it right.

If you want to grow in a sustainable, satisfying way, then you need to pay attention to how you’re growing, not just how much. After all, a bigger company isn’t necessarily a better one. Let’s look at four common pitfalls of growth in the design industry, and how to avoid them.

The wrong clients

As a business owner, you might assume you should serve everyone you can. Busy is good, right? In reality, taking on every client who’ll hire you is actually detrimental to your growth, and not strategic at all.

The more bad clients you try to serve, the less time you have to look for good clients. This is dangerous. It’s like feng shui: You have to move the bad ones out so the good ones can come in.

But what makes a good client? Your goals and expertise might give you more specific criteria, but all good clients share three traits:

They understand your value and let you remain in control of the design process as the professional.
They are enjoyable to work with.
They are profitable.

Firing clients (further, keeping the wrong clients out of your company in the first place) is one thing that sets strategic business owners apart from those who struggle through growth. This is a little scary to implement, but letting the bad clients move on to another home will be one of the most important things you can do to grow your company—and not just because they take up time.

Whether your clients are good or bad, you can bet that they will send you more clients just like themselves. If you give discounts to one client, then new clients could show up at your door looking for discounts. On the flip side, if you are treating clients well, doing good work, and getting paid well for it, then new clients will show up expecting to pay for the privilege of working with you.

Growing with the wrong clients is unintentional growth, and it actually hurts your good clients, too. When you are loaded down with clients you can never please, you can’t give enough attention to the very clients that will appreciate your excellent service. And when a good client doesn’t understand why you can’t find time to return their calls, they will leave.

Hiring for hiring’s sake

You might think that staff growth is good, but size alone is no strategy. Hire employees to serve great clients, rather than taking clients in order to pay employees.

Growing to simply reach a size often puts you in the poor position of having to farm out busywork to unchallenged employees. Busywork blunts an employee’s passion and makes her wonder if she is doing anything of value—beyond bringing another dollar into your business, that is. It’s hard to get excited about work like that.

Bad clients and busywork lead to high employee turnover—because your best employees know they don’t have to put up with your poor decisions. And employee instability can, in turn, atrophy your client list. Give the employees some control over whom your company serves. For example, if you have a method of tracking all incoming new client requests, go over these with your team in a weekly meeting to flesh out who would be good for your company. Or, share your “good client” criteria with your team, ask for their input, and discuss whether everyone currently on your client roster fits the bill.

Including your team in these intimate parts of your company will make talented employees more passionate and happier. Happy clients will be the result!

Employees long for a business owner who manages growth strategically. One way to bless your employees and feed their passion for their work is to focus on a niche. This will in turn mean you must hire the right team to work on that niche. Niche work, as opposed to serving anyone anywhere, lets you hire the best designers, who expect better work conditions, higher pay, and more creative freedoms—passionate professionals who want to work with only the best clients.

Growing too fast

Newer design companies often struggle to manage their tacit, undocumented knowledge—like where files are stored or how project handoffs happen. When your company is new, it’s challenging to maintain developed systems that can handle large amounts of growth in small periods of time. Along with nurturing great designers, take time to develop internal systems like mature project management systems and wise account managers.

Every time you onboard a new customer, you will add to your tacit knowledge about your pricing and your customer experience processes. You need to be able to apply that knowledge to the next client you bring in. This is hard to do when you are growing too fast. If you are a new owner, fight to keep your growth slow. Once you have more mature systems, you can pick up your pace.

Once you reach a certain level of size, you also need more help to grow. This can be a surprise to many business owners. More specifically, once you reach somewhere around five to ten employees, non-owner leaders need to be identified to help you continue managing the work and the growth. Now your company needs to continue its growth, often without the owner controlling the full process. Your internal processes will be tested during this phase. As an owner, you must transition out of technical work so you can be available to move into a role of leading and coaching your team as they take on new roles in helping the company grow.

Since you need to hire the best employees and can’t afford to let customer service slip while you do it, you’ll also find you need to hire new people just before you need them. But in order to hire early, you’ll need precious cash on hand to buy enough time to find the right clients to match your new hires. If you don’t have the cash reserves to get through this adjustment period, then you could be overburdened with a huge payroll and not enough of the right clients to pay it.

Low margins

High-margin work frees you from stressing over the basic needs of the business—like worrying about making payroll, or paying contractors and vendors on time. A profit margin is the total amount your client paid you, less the specific salaries or contractors needed to produce the work, less the products or additional services you had to purchase to serve the client. Profit margin is different from gross revenue. Profit margin is a minimalistic view of how profitable each job is. Gross revenue, on the other hand, is the total income your company is making, before accounting for costs.

In the profit margin calculation, ignore insurance, rent, taxes, and all of the stuff you can’t do anything about. These expenses do not factor into the calculation of a profit margin, because you can’t control them. Just focus on the few controllable costs needed to perform that specific job for that specific client. Low-margin work means you are pricing your services too close to the costs you’ll have to bear in order to serve that client.

For example, if you price your services at $100,000, and your salaries and contractors cost you $80,000 and your miscellaneous costs (fonts, hosting, and stuff like that) are $5,000, then you are dealing with a small profit margin of $15,000. This is a 15 percent profit margin. Is that enough? That is something you must decide, but I can tell you from experience that the smart design businesses my accounting firm works with are experiencing between 70 and 90 percent profit margins, and I would expect at least a 50 percent margin from our clients.

Margins matter. Low-margin work could have unintended consequences, like leaving you in debt to cover living costs and the owners’ salaries. This is very dangerous. Though it is a fearful thing to do at first, pricing your services high enough to fund profitable growth, and being committed to high-margin growth, will help your design company prosper.

Committing to only taking high-profit work also lets you offer attractive salaries, provide good workspaces and tools, and invest in employee education. In essence, you can’t build the world-class team you need without high profit margins.

Good growth can be yours

So maybe you’ve grown, but have you prospered? Not if you’ve allowed your company to be sidelined by the four growth pitfalls above. Growth doesn’t happen by default. It can only happen by design. It takes a lot of work, and demands keeping your growth patterns in check every step of the way. Be watchful for the pitfalls and run the other way. Your clients, your team, and your profitability will thank you.

Hack Your Maps

2013-04-09T12:00:10+00:00

Web maps have come a long way. Improved data, cleaner design, better performance, and more intuitive controls have made web maps a ubiquitous and critical component of many apps. They’ve also become one of the mobile space’s most successful transplants as more and more apps are powered by location-aware devices. The core web map UI paradigm itself—a continuous, pannable, zoomable surface—has even spread beyond mapping to interfaces everywhere.

Despite all this, we’ve barely begun to work web maps into our design practice. We create icon fonts, responsive grids, CSS frameworks, progressive enhancement strategies, and even new design processes. We tear down old solutions and build new ones, and even take an extra second to share battle stories in prose and in person. Yet nearly five years since Paul Smith’s article, “Take Control of Your Maps,” web maps are still a blind spot for most designers.

Have you ever taken apart a map? Worked with a map as a critical part of your design? Developed tricks, hacks, workarounds, or progressive enhancements for maps?

This article is a long overdue companion to Paul’s piece. Where he goes on a whirlwind survey of the web mapping stack at 10,000 feet, we’re going to walk through a single design process and implement a modern-day web map. By walking this path, I hope to begin making maps part of the collective conversation we have as designers.

Opinionated about open

Paul makes a strong case for why you might want to use open mapping tools instead of the established incumbent. I won’t retread his reasons here, but I would like to expand on his last: Open tools are the ones we hack best.

There is nothing mysterious about web maps. Take any spatial plane, split it up into discrete tiles, position them in the DOM, and add event handlers for panning and zooming. The basic formula can be applied to Portland, Mars, or Super Mario Land. It works for displaying large street maps, but nothing stops us from tinkering with it to explore galleries of art, create fictional game worlds, learn human anatomy, or simply navigate a web page. Open tools bare the guts of this mechanism to us, allowing us to see a wider range of possibilities.

The mechanics of web maps are not limited to street maps.

We should know the conditions under which map images are loaded and destroyed; we should argue whether map tiles are best positioned with CSS transforms or not; and we should care whether vector elements are drawn with SVG or Canvas. Open tools let us know and experiment with these working details of our maps. If you wouldn’t have it any other way with your HTML5, CSS, or JavaScript libraries, then you shouldn’t settle for less when it comes to maps.

In short, we’ll be working with a fully open mapping stack. MapBox, where I work, has pulled together several open source libraries into a single API that we publish under mapbox.js. Other open mapping libraries that are worth your time include Leaflet and D3.js.

Starting out

I’m a big fan of Sherlock Holmes. Between the recent Hollywood movies starring Robert Downey Jr. and the BBC’s contemporary series, I’m hooked. But as someone who has never been to London, I know I’m missing the richness of place and setting that Sir Arthur Conan Doyle meant to be read into his short stories.

A typical approach would be to embed a web map with pins of various locations alongside one of the Sherlock stories. With this approach the map becomes an appendix—a dispensable element that plays little part in Doyle’s storytelling. Instead, we’re going to expand the role of our map, integrating it fully into the narrative. It will set the stage, provide pace, and affect the mood of our story.

A tale of places

To establish a baseline for our tale, I restructured The Adventure of the Bruce-Partington Plans to be told around places. I picked eight key locations from the original text, pulled out the essential details of the mystery, and framed them out with HTML, CSS, and JavaScript.

A Sherlock Holmes story in text only. View Demo 1.

The story is broken up into section elements for each key location. A small amount of JavaScript implements a scrolling flow that highlights a single section at a time.
Our page is not responsive yet, but it contains scaffolding to guard against bad choices that could thwart us. The main text column is fluid at 33.33% and pins to a min-width: 320px. If our content and design flow reasonably within these constraints, we’re in good shape.

Next, we’ll get started mapping. Initially we’ll work on our map separately from our story page to focus on learning key elements of a new technology.

Maps are data

The mapping equivalent of our abridged Sherlock story is a dataset of eight geographic points. GeoJSON, a format for describing geographic data in JSON, is the perfect starting point for capturing this data:

{
    "geometry": { "type": "Point", "coordinates": [-0.15591514, 51.51830379] },
    "properties": { "title": "Baker St." }
}, {
    "geometry": { "type": "Point", "coordinates": [-0.07571203, 51.51424049] },
    "properties": { "title": "Aldgate Station" }
}, {
    "geometry": { "type": "Point", "coordinates": [-0.08533793, 51.50438536] },
    "properties": { "title": "London Bridge Station" }
}, {
    "geometry": { "type": "Point", "coordinates": [0.05991101, 51.48752939] },
    "properties": { "title": "Woolwich Arsenal" }
}, {
    "geometry": { "type": "Point", "coordinates": [-0.18335806, 51.49439521] },
    "properties": { "title": "Gloucester Station" }
}, {
    "geometry": { "type": "Point", "coordinates": [-0.19684993, 51.5033856] },
    "properties": { "title": "Caulfield Gardens" }
}, {
    "geometry": { "type": "Point", "coordinates": [-0.10669358, 51.51433123] },
    "properties": { "title": "The Daily Telegraph" }
}, {
    "geometry": { "type": "Point", "coordinates": [-0.12416858, 51.50779757] },
    "properties": { "title": "Charing Cross Station" }
}

Each object in our JSON array has a geometry—data that describe where this object is in space—and properties—freeform data of our own choosing to describe what this object is. Now that we have this data, we can create a very basic map.

The basics of web mapping. View Demo 2.

Note that the coordinates are a pair of latitude and longitude degrees. In the year 2013, it is still not possible to find a consistent order for these values across mapping APIs. Some use lat,lon to meet our expectations from grade-school geography. Others use lon,lat to match x,y coordinate order: horizontal, then vertical.
We’re using mapbox.js as our core open source mapping library. Each map is best understood as the key parameters passed into mapbox.map():
1. A DOM element container
2. One or more Photoshop-like layers that position tiles or markers
3. Event handlers that bind user input to actions, like dragging to panning
Our map has two layers. Our tile layer is made up of 256x256 square images generated from a custom map on MapBox. Our spots layer is made up of pin markers generated from the GeoJSON data above.

This is a good start for our code, but nowhere near our initial goal of using a map to tell our Sherlock Holmes story.

Beyond location

According to our first map, the eight items in our GeoJSON dataset are just places, not settings in a story full of intrigue and mystery. From a visual standpoint, pins anonymize our places and express them as nothing more than locations.

To overcome this, we can use illustrations for each location—some showing settings, others showing key plot elements. Now our audience can see right away that there is more to each location than its position in space. As a canvas for these, I’ve created another map with a custom style that blends seamlessly with the images.

Illustrations and a custom style help our map become part of the storytelling. View Demo 3, and then read the diff.

The main change here is that we define a custom factory function for our markers layer. The job of the factory function is to take each GeoJSON object and convert it to a DOM element—an a, div, img, or whatever—that the layer will then position on the map.
Here we generate divs and switch from using a title attribute in our GeoJSON to an id. This provides us with useful CSS classes for displaying illustrations with our custom markers.

Bringing it all together

Now it’s time to combine our story with our map. By using the scroll events from before, we can coordinate sections of the story with places on the map, crafting a unified experience.

As the user reads each section, the map pans to a new location. View Demo 4, then read the diff.

The bridge between the story and the map is a revamped setActive() function. Previously it only set an active class on a particular section based on scrolling position. Now it also finds the active marker, sets an active class, and eases the map to the marker’s location.
Map animation uses the easey library in the mapbox.js API, which implements animations and tweening between geographic locations. The API is dead simple—we pass it the lon,lat of the marker we want to animate to, and it handles the rest.
We disable all event handlers on our map by passing an empty array into mapbox.map(). Now the map can only be affected by the scrolling position. If users wanted to deviate from the storyline or explore London freeform, we could reintroduce event handlers—but in this case, less is more.

Displaying our fullscreen map together with text presents an interesting challenge: our map viewport should be offset to the right to account for our story on the left. The solution I’m using here is to expand our map viewport off canvas purely using CSS. You could use JavaScript, but as we’ll see later, a CSS-only approach gives us elegant ways to reapply and adjust this technique on mobile devices.

At this stage, our map and story complement each other nicely. Our map adds spatial context, visual intrigue, and an interesting temporal element as it eases between long and short distances.

Maps in responsive design

The tiled, continuous spatial plane represented by web maps is naturally well-suited to responsive design. Web maps handle different viewport sizes easily by showing a bit more or a bit less map. For our site, we adjust the layout of other elements slightly to fit smaller viewports.

Tweaking layout with web maps. View Demo 5, then read the diff.

With less screen real estate, we hide non-active text sections and pin the active text to the top of the screen.
We use the bottom half of the screen for our map and use media queries to adjust the map’s center point to be three-fourths the height of the screen, using another version of our trick from Demo 4.

With a modest amount of planning and minimal adjustments, our Sherlock story is ready to be read on the go.

Solve your own case

If you’ve been following the code between these steps, you’ve probably noticed at least one or two things I haven’t covered, like the parameters of ease.optimal(), or how tooltips picked up on the title attribute of our GeoJSON data. The devil’s in the details, so post to this GitHub repository, where you will find the code and the design.

You should also check out:

The MapBox site, which includes an overview of tiling and basic web map concepts, and MapBox.js docs and code examples.
Leaflet, another powerful open source mapping library.
D3.js, a library for powering data-driven documents that has a broad range of applications, including mapping.

This example shows just one path to integrating web maps into your designs. Don’t stick to it. Break it apart. Make it your own. Do things that might be completely genius or utterly stupid. Even if they don’t work out, you’ll be taking ownership of maps as a designer—and owning something is the only way we’ll improve on it.

Nick Sherman on Typography: Responsive Typography is a Physical Discipline, But Your Computer Doesn’t Know It (Yet)

2013-04-04T11:30:41+00:00

For ideal typography, web designers need to know as much as possible about each user’s reading environment. That may seem obvious, but the act of specifying web typography is currently like ordering slices of pizza without knowing how large the slices are or what toppings they are covered with.

If someone asked me how many slices of pizza I wanted for lunch, I would probably say it depends on how large the slices are. Then—even if they told me that each slice was one eighth of a whole pie, or that they themselves were ordering two slices, or even that the slices were coming from Joe’s Pizza—any answer I might give would still be based on relative knowledge and inexact assumptions.

Such is the current situation with the physical presentation of responsive typography on the web. The information at a designer’s disposal for responsive design is virtually nonexistent outside the realm of software. Very little knowledge about the physical presentation of content is available to inform the design. The media query features of today can only relay a very fragmented view of the content’s actual presentation, and related terms from CSS are confusing if not downright misleading.

The immeasurable pachyderm

Among all the physical qualities of web typography, the elephant in the room is the issue of size. I’m not talking about em or rem or “reference pixels”^¹ or even device pixels. I’m talking about real, actual, physical, bona fide, measurable, size!

It’s ridiculous that we can send robots to Mars yet it’s still virtually impossible to render a glyph on a web page and say with confidence: “If you measure this glyph on your screen with a ruler, it will be exactly 10 millimeters wide.” Although actual physical size isn’t always the most important factor in web design, in some cases it is critical. For example, consider content for partially-sighted or low-vision readers: the ability to tweak designs according to physical sizes would enable designers to make conscious design decisions with much more sensitivity to how the type is actually being seen. And even where physical sizing is secondary to relative sizing, why shouldn’t we nevertheless be able to factor in physical size when establishing the relationships between different elements?

Physical considerations ≠ print design

I don’t believe web typography should be a screen-based imitation of print typography. One of the greatest benefits of web typography, and web design in general, is that it is flexible, adaptable, fluidly adjustable, without being locked into any one specific configuration. However(!), that doesn’t mean web designers should be forced to design without any means to address the issues of physical presentation. On the contrary, responsive design will not reach its full potential until it allows the ability to respond to the very important physical variables of digital media.

Please pardon the cliché, but when it comes to typography, on screens or otherwise, size matters. Physical size affects optical issues that change how the eye and brain process typographic images. Not surprisingly, typographers and typeface designers have been compensating for optical size-related issues as far back as Gutenberg.

You can’t expect a paragraph of type with the same relative line-height, column width, letter-spacing, and glyph proportions to function just as well on two different displays that have the same number of pixels but completely different physical sizes. It’s great that designers can adjust proportions between typographic elements if the canvas varies in relative size, but any such compensation is still based on guesswork and assumptions about the physical size of that canvas. When people disagree about the size or spacing of type on a website, there’s a very good chance that their opinions are based on completely different physical manifestations of the same content, even if their software and settings are identical.

Resolute resolution, absolute absolution

One of the most crucial factors in the size equation is resolution. And when I say resolution, I don’t just mean “how many pixels is this?”, or even “how many device pixels is this?”, but also “how large are these pixels?”

This is very different from the W3C’s “resolution” media feature in the current draft of the Media Queries Level 4 spec. You will note that the spec refers to resolution in terms of “CSS ‘inches’”—the quotes around “inches” are theirs, implying that they are not actually inches at all.

For an example of why physical resolution matters, imagine you are rendering text on a digital billboard with a physical resolution of one pixel per inch (1 PPI). Now imagine you are rendering the same text on a 200 PPI mobile device display. Even if you knew the actual number of device pixels that would be used to render your type (which itself is difficult to do with confidence these days), you would want to treat the two compositions very differently, both in terms of the typeface as well as typographic layout. The billboard type would likely require less space between letters. The letterforms themselves would benefit from narrower proportions, and could endure a higher ratio between thick and thin strokes. The type might even require different colors to optimize contrast at that size. These are all basics of typography and typeface design.

Unfortunately, in the current landscape of media query features, there is no way to know the difference between 16 device pixels on a crude LED billboard and 16 device pixels on a high-density mobile display. Heck, there isn’t even a reliable way to know if your type is 16 device pixels at all, regardless of how large the pixels are!

Pixels still rule, for better or worse

I know what some em-based enthusiasts might be thinking: “But you shouldn’t be specifying type sizes in pixel units to start with! All type sizes should be spec’d abstractly in relation to each other or a base font size!” However, in the current world of web typography, no matter what unit of measure you use to spec your onscreen type sizes—em, rem, px, pt, in, %, vh, or whatever else—at the end of the line, your specification is being mapped to pixels. Even if you leave the base size of your document to the defaults and specify everything else with em, there is still a base size which all other sizes will ultimately refer to, and it is defined in pixels.

This is because, currently, the only unit of measure that can be rendered onscreen by any operating system with absolute confidence is the lowly pixel. Until we have media query features that allow us to spec for situations like:

@media (physical-resolution: 1device-pixels-per-physical-inch) { … }

or:

@media (device-width: 10physical-centimeters) { … }

… any compensation for physical size is based entirely on rough guesses about the devices our content will be presented on.^²

It’s a complete fallacy that the official CSS spec allows so-called “absolute” units of measure like inches, points, and centimeters to be mapped to anything but actual physical units. Ironically, previous versions of CSS treated these things as you would hope and expect, but a change was made “because too much existing content relies on the assumption of 96dpi, and breaking that assumption breaks the content.” Call me idealistic if you will, but I am more of the mind that a spec should be written based on what is best for the future, not to cater to things that were made in the past.^³

Getting physical

Any ability to leverage physical variables for web design will require a joint effort by several groups:

Device manufacturers will need to provide APIs that can inform the operating system—and, by extension, web browsers and web designers—of the actual physical properties of the hardware being used to present content to the user. Some device APIs are already beginning to show up in the world, but there is a long way to go before functionality and adoption are anywhere near dependable.
Standards organizations—the W3C in particular—will need to establish specifications for how to reference physical properties when formatting content. They will need to update (or at least augment) their existing “absolute” units of measure to be more meaningful, so they are more than just multipliers of sizeless pixels.
Software manufacturers will need to implement support for new specs relating to physical media features. Browsers are the most obvious software that will need to implement support, but the biggest challenge might be in getting native support for device APIs in operating system software.
Type manufacturers and type services will need to provide more diverse ranges of typefaces that have been optimized for a variety of physical properties. Ideally, many of the needed variations could even be provided on the fly using a broader approach to the ideas of font hinting.
Web designers and developers, last but not least, will need to build their sites to respond to physical properties, leveraging all variables to the benefit of their users.

Size and resolution are just the tip of the iceberg of physical variables that could be considered when improving web typography. Things like viewing distance, ambient light, display luminance, contrast ratio, black levels, etc., etc., could all be factored in to improve the reading experience. Even the ability to know some variables within the realm of software, like the user’s rendering engine or the presence of subpixel positioning, would go a long way toward helping web typographers design a better reading experience.

In the meantime, I’d love to see more of the players mentioned above start to at least experiment with what’s possible when physical features can be specified, detected, and factored into responsive designs in structured, meaningful, and predictable ways. Until we can do that, we’re all just ordering pizza without knowing exactly what will end up on our plate.