The timeless repository

Abstraction Creep

2012-05-25T18:19:20Z

Abstractions are very powerful when you’re writing software. You don’t want to write your web applications in assembly, or an operating system in JavaScript for that matter. Choosing the right abstraction makes you focus on the real challenges and lets you forget the insignificant details.

Not all abstractions are perfect. Some are leaky; some are just plain stupid. But wait! We can just introduce an abstraction on top of that, right? Let’s hide the stupid details and provide a new, better abstraction!

Abstraction creep is what happens when you try to save yourself from a shaky abstraction by introducing another abstraction, instead of just fixing that goddamn abstraction.

I’m going to use examples from Ruby and Ruby on Rails, but I think the examples are clear enough for any programmer to understand. Please contact me if something is unclear.

Controller tests

Controller tests in Rails are slow. Or, more correctly: Starting the environment where you can run controller tests in Rails takes time. We’re talking 5-10 seconds here. 10 seconds doesn’t sound so bad, but when you’re trying to quickly iterate over test/code, waiting 10 seconds before getting any feedback kills your flow.

Well, luckily we can introduce another abstraction!

The old code

class PicturesController < ApplicationController before_filter :required_login def create @picture = current_user.pictures.create(params[:picture]) if @picture.persisted? redirect_to @picture else render :action => :edit end end end

Let’s Objectify!

Let’s use Objectify to make this controller more testable.

First, we separate out the filter:

class RequiresLoginPolicy def allowed?(current_user) !current_user.nil? end end

Then, the actual action:

class PicturesCreateService def call(current_user, params) current_user.pictures.create params[:picture] end end

And finally, the rendering:

class PicturesCreateResponder def call(service_result, controller, renderer) if service_result.persisted? renderer.redirect_to service_result else renderer.data(service_result) renderer.render :action => :edit end end end

Now we can test each of these three classes without loading Rails. Woah, super fast tests! They’re even unit tests; only testing a single step of the flow, so your tests become simpler too.

Win/win, right?

Let’s take a step back

What have we actually done here?

We’ve created our own abstraction on top of Rails’ request/response-loop. Is it useful? Yes, we can now focus on smaller units. Does it have some problems? Yes, you can’t easily test the way the units actually work in production. Is it more complex? Hell, yes. We have more classes, more methods, more ends, more lines of code.

Is it worth it? I doubt it.

Fixing the abstraction

Okay, so what’s the real problem here?

Controller tests are slow. Why?

Because loading controllers are slow. Why?

Because controllers are too complex; they depend on too many different parts of Rails which needs to be loaded at startup.

Why are controllers complex? A controller is rather simple:

It takes a request and an empty response
It lets you define actions which change the response
It defines some convenience methods

These are the most important features. Let’s write a very simple controller:

class BasicController def initialize(request, responder) @request = request @responder = response end def params @request.params end def render(*args) @responder.render(*args) end def redirect_to(*args) @responder.redirect_to(*args) end # and so on... end # Our previous code: class PicturesController < BasicController def current_user; ... end def create @picture = current_user.pictures.create(params[:picture]) if @picture.persisted? redirect_to @picture else render :action => :edit end end end

No dependencies (i.e. no other files/classes to load), no magic. Super fast tests again:

test "uploading a new picture" do request = Request.new(:params => { :picture => { ... } }) responder = MockResponder.new picture = PicturesController.new(request, responder) picture.create assert response.redirected_to /^/pictures/(\d+)/ # or whatever we want to assert for end

Notice that I’m using the regular Request class. Creating Request objects should be just as easy as creating a new controller object. They are both very simple classes that have few dependencies.

What about filters?

You might have noticed that I silently dropped the before_filter in the last example. This was intentional because filters are not crucial to what a controller is doing. That doesn’t mean they don’t belong there, it just means you need to figure out if it’s worth the added complexity.

Does it belong another place? Maybe in the router (yes, there are frameworks that do the filtering in the router)? Somewhere else? I don’t know. Maybe filters are lightweight and useful enough to take up some controller complexity.

And this is what we should focus on: Figuring out what belongs in the most basic classes in our framework. Not creating a horrible patchwork by adding more abstractions.

Disclaimer

Just to make it clear: This is by no means an argument against Objectify itself, just the part where they introduce another layer on top of ActionController. The concept of service objects does make sense in more complex applications, but I believe it’s possible to achieve it within the current abstractions exposed by Rails.

Models

Models in Rails have the same problem as controllers: They are too complex. Too many dependencies.

class Post < ActiveRecord::Base end

It looks so simple, but in reality it’s a complicated beast:

You need a database connection to just instantiate it (!)
The table “posts” needs to exists in the database
It adds a bunch of methods

Models in Rails works great as long as you’re using them inside Rails’ request/response-loop where everything is set up for you. The moment you step out from that loop (tests, other libraries) it’s a big hassle.

It’s very tempting to add abstractions on top of ActiveRecord to make it easier to work with. Or you could fix the abstraction by using the Data Mapper pattern instead.

Strive for simple data objects

Abstraction creep often occurs when the data objects are too complex. And there’s nothing about data that makes it complex. Data is really, really simple. There’s only two things you need to do with data:

You need to be able to combine data into one structure
You need to be able to take the data apart again

If you’re not able to do those things easily, you’ve already lost the war. When the data is difficult to work with, you’re forced to add abstractions, dependency-injection, mocks, stubs, and whatnot. Instead of just working with the actual data, you’ll have to abstract away the data.

Please be aware that this is by no means “incompatible” with object-oriented programming. You can still get your precious information hiding. You can still override accessors so the interface you expose isn’t the same as the internal interface. You can still subclass your data objects. You can still add convenience methods for working with the data.

You just have to make it effortless to do the two most important things:

Construct the data object
Access the data it’s composed of

As long as you can do this (hopefully with no dependencies at all), you can actually use your data objects in other classes. In fact, it encourages you to create separate classes for separate problems. When you can easily use your data objects it’s even easier to separate functionality to other classes (as opposed to adding more functionality to the same class).

Sometimes change is imminent

Of course, using complex data objects is far from the only way to create shaky abstractions. Sometimes we don’t even realize it’s flawed until it’s too late. Or maybe the world changes in such way that it invalidates the abstraction.

Rack is a typical example of an abstraction that hasn’t handled change well. Rack tries to abstract away HTTP in a simple and clear way. When Rack was introduced in 2007, the web was pretty much stateless and every request expected an response as soon as possible. A synchronous interface makes sense:

# When the web server sees a request it runs this: env = build_env_from_request status, headers, body = App.call(env) server_response(status, headers, body)

The web has changed. Suddenly you have long-polling, where the request can wait for up to 30 seconds before the server returns a response. We have WebSocket which completely breaks the stateless request/response-cycle. The requirements have changed.

Rack wasn’t designed to handle these issues, so another abstraction was built on top:

def App.call(env) cb = env['async.callback'] # Async call wait_for_long_polling do # Now we can return the request! cb.call(200, {}, []) end # Send a dummy "response" in order to be "complient" with the Rack # abstraction. [-1, {}, []] end

This is a horrible, hacky abstraction, implemented by some Ruby web servers (outside of the Rack specification). Not only does it look weird, it also breaks the whole Rack abstraction. Any layer between the web server and the application (that follows the Rack specification) will behave incorrectly when it sees the dummy response.

You can’t predict the future, and some abstractions are perfect until the future comes and ruins everything.

Sustain the pain; go with the flow. Or: deal with the real problem.

Sometimes you’re stuck with crappy abstractions. ActiveRecord isn’t perfect. ActionController has its problems. You will feel a slight pain when you’re forced to use them in a way you don’t like. It’s very tempting to just add another abstraction. It can’t harm, right?

Personally, I try to avoid these abstractions-of-abstractions for as long as possible. I find that the complexity introduced is hardly worth the advantages. I can accept some slow tests. I can work around annoyances. All in the name of simplicity, consistency and familiarity.

Maybe you’ll reach a point of anger. You can no longer handle the pain. Or maybe the abstraction is just impossible to work around (you can’t do long-polling efficiently in Rack without changing anything). If that happens, you might be better served to rewrite the current (shaky) abstraction, instead of building another layer in an already-unstable foundation.

When you decide to abstract away the abstraction, it means you have the competence to realize that the abstraction is wrong, but not the courage to fix the actual problem. Your choice might seem sensible now (easier to integrate with the old abstraction; less changes overall), but 5 months, 5 years or even 50 years from now, people are going to say: “Why do I have to care about this stuff? Why couldn’t you just have fixed the goddamn problem?”

JSON: The JavaScript subset that isn't

2011-05-15T15:56:05Z

From Wikipedia’s article on JSON

JSON was based on a subset of the JavaScript scripting language.

All JSON-formatted text is also syntactically legal JavaScript code.

JSON is a subset of JavaScript.

All these years we’ve heard it over and over again: “JSON is a JavaScript subset”. Guess what? They’re wrong. Wrong, wrong, wrong. You see, the devil’s in the details, and there’s no way to avoid it: Not all JSON-formatted text is legal JavaScript code:

{"JSON":"ro cks!"}

Copy the exact code above and paste it into Firebug or the Web Inspector within a pair of parentheses (required to avoid an ambiguity in JavaScript’s syntax):

Wait, what? SyntaxError: Unexpected token ILLEGAL? That doesn’t make any sense! It’s just a regular object literal, how can that be a SyntaxError?

Try the same code in a proper JSON parser. No problems at all.

Of course it’s not just a regular object literal. There’s a sneaky little Unicode character in there too: Right between “ro” and “cks!” there’s a tiny U+2028. Your browser probably doesn’t display it because it’s whitespace: LINE SEPARATOR, but it’s still there. If you replace the character with a U+2029 (PARAGRAPH SEPARATOR) you would have the exact same issue.

JSON + U+2028 = ☺

According to the JSON specification, you can safely use this character in any string. It’s not a quote, not a backslash, and not a control character. It’s just a weird Unicode whitespace character:

JavaScript + U+2028 = ☹

ECMA-262 (the standard that JavaScript is based on) on the other hand defines strings a little differently: According to 7.8.4 String Literals, a string can contain anything as long as it’s not a quote, a backslash or a line terminator:

DoubleStringCharacter ::
    SourceCharacter but not double-quote " or backslash \ or LineTerminator 
    \ EscapeSequence

SingleStringCharacter ::
    SourceCharacter but not single-quote ' or backslash \ or LineTerminator 
    \ EscapeSequence

And what is a line terminator? Let’s have a look at 7.3 Line Terminators:

The following characters are consider to be line terminators:

\u000A - Line Feed

\u000D - Carriage Return

\u2028 - Line separator

\u2029 - Paragraph separator

Ouch.

No string in JavaScript can contain a literal U+2028 or a U+2029.

So what?

Because of these two invisible Unicode characters, JSON is not a subset of JavaScript. Close, but no cigar.

In most applications, you won’t notice this issue. First of all, the line separator and the paragraph separator aren’t exactly widely used. Secondly, any proper JSON parser will have no problems with parsing it.

However, when you’re dealing with JSONP there’s no way around: You’re forced to use the JavaScript parser in the browser. And if you’re sending data that other have entered, a tiny U+2028 or U+2029 might sneak in and break your pretty cross-domain API.

The solution

Luckily, the solution is simple: If we look at the JSON specification we see that the only place where a U+2028 or U+2029 can occur is in a string. Therefore we can simply replace every U+2028 with \u2028 (the escape sequence) and U+2029 with \u2029 whenever we need to send out some JSONP.

It’s already been fixed in Rack::JSONP and I encourage all frameworks or libraries that send out JSONP to do the same. It’s a one-line patch in most languages and the result is still 100% valid JSON.

Making Ruby Gems

2011-02-06T21:36:21Z

One of my favorite aspects of the Ruby community is its strong commitment to Open Source. Thousands of Rubyists have written great libraries and placed the code up on GitHub (or elsewhere on the Internet) for everyone to use. Ruby has a fantastic tool to distribute these libraries: RubyGems. One of the reasons that Open Source runs so strong through Ruby’s veins is how easy it is to share your code.

While making a gem is really easy, there are a few additional concerns that you should be aware of when you distribute your code as a gem. Here’s an example of building a simple Gem, with some notes on best practices along the way.

The structure of a gem

At its core, a gem consists of two things: some code, and a gemspec. The gemspec file defines a Gem::Specification object, which is used by rubygems to handle the management of your code. That’s it! Nothing complicated. However, there are some conventions that virtually all gems follow that will help you out.

Most gems have a directory structure that looks like this:

$ tree -d mygem
|-- lib/
|   |-- mygem.rb
|   `-- mygem/
|       |-- base.rb
|       `-- version.rb
|-- test/
|-- bin/
|-- Rakefile
`-- mygem.gemspec

All of the code for the gem goes under the lib directory. This directory is what gets added to your $LOAD_PATH, and so lib usually contains two things: mygem.rb and a directory named mygem. mygem.rb will look like this:

require 'mygem/base'

If there were more files in lib/mygem, they’d be required here, too. This is done so that you can break up your project into however many files you’d like, and name them whatever you want, and nobody will tramp on each others’ toes. Think about it: if I have two gems installed, and both of them have their lib directories included, and they both name a file json.rb, which one is going to be loaded? It causes problems. So just follow this structure, and everything will work out.

The version.rb file looks like this:

module Mygem VERSION = "0.0.1" end

This constant will be used in our gemspec. It’s nice to have it in a separate file, so that we can easily find and increment it when releasing a new version of the gem.

I’m sure that you can guess what goes in the test directory: your unit tests! You do have those, right? We’ll talk more about this later.

The bin directory holds any binaries that we want to distribute with our gem. Calling them ‘binaries’ is sort of a misnomer, though: these are almost always just Ruby scripts, starting off with a ‘shebang line’:

#!/usr/bin/env ruby begin require 'mygem' rescue LoadError require 'rubygems' require 'mygem' end #more code goes here

This is a patten you’ll often see in gems that also give you scripts to run. Why the begin/rescue/end? Well, ‘require “rubygems” is wrong’. Basically, someone may be using something other than rubygems to manage their path, and so you shouldn’t trample on their toes. But if you can’t find your gem, then giving it a second shot with Rubygems is better than crashing.

The Gemspec

Here’s a sample gemspec:

# -*- encoding: utf-8 -*- $:.push File.expand_path("../lib", __FILE__) require "mygem/version" Gem::Specification.new do |s| s.name = "mygem" s.version = Mygem::VERSION s.platform = Gem::Platform::RUBY s.authors = ["Steve Klabnik"] s.email = ["steve@steveklabnik.com"] s.homepage = "" s.summary = %q{A sample gem} s.description = %q{A sample gem. It doesn't do a whole lot, but it's still useful.} s.add_runtime_dependency "launchy" s.add_development_dependency "rspec", "~>2.5.0" s.files = `git ls-files`.split("\n") s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n") s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) } s.require_paths = ["lib"] end

As you can see, it uses the standard ‘pass a block to new’ DSL convention to build up all of the information about our Gem. Most of it is pretty self-explanatory, but there are a few interesting parts: You can see we used the Mygem::VERSION constant we defined earlier to set our version. We use git to list all of the files in our project, as well as our test files and executables. The ‘add dependency’ lines tell rubygems what other gems we’ll need to install, if the user doesn’t have them already.

Tools to build tools

Because gems follow these conventions, there are a bunch of different gems that can help you make gems, like hoe or jeweler. My favorite is a one-two punch with rvm and bundler.

If you’re not using rvm already, you should. rvm is wonderful for a few reasons, but when you’re making a gem, rvm’s gemsets feature allow you to develop your gems in a cleanroom environment. This is nice for two reasons: you can verify that you have all of your dependencies configured properly, and you won’t pollute your regular Ruby setup with your undoubtedly half-broken under-development versions of your own gems.

Bundler is great for managing dependencies of gems in your applications, but it also includes two cool tools to help you make gems: bundle gem will create a gem skeleton, and that skeleton is set up with a few rake tasks to make your development of gems nice and simple. The bundler skeleton sets up all of those directories I showed you above, as well as giving you a Gemfile, a git repository, and a .gitignore file. The three rake tasks bundler installs are ‘build,’ ‘install,’ and ‘release.’ ‘build’ builds your gem into a .gem, ‘install’ installs that gem into your current Ruby, and ‘release’ will tag your release, push it to GitHub, and then push your gem to rubygems.org. Super simple.

An example

I gave a lightning talk at pghrb recently, and actually live-coded a gem while explaining this stuff. The resulting gem, which simply opens my presentation in a web browser, is on GitHub. It’s called teachmehowtomakearubygem, and you can get it with gem install teachmehowtomakearubygem. I’m still revising the presentation to read better; I didn’t want to present a giant wall of text, but this article is much easier to read than the presentation is. Still, all of the example code is there.

Testing your Gem

If you set up your Rakefile to run your tests with rake test, you can take advantage of a really neat new project: gem-testers. The only other thing you need to do is add an empty .gemtest file to your project, and gem-testers will pick it up. Once enabled, your gem’s tests will be run on a variety of machines by a bunch of different people. This project is just getting underway, but similar efforts have provided a great benefit to people who write Perl libraries. Don’t have a Mac, but want to test your gem out on x86_64/darwin? gem-testers to the rescue!

A note on versioning

Try to follow semantic versioning when releasing your gems. This makes it much easier for people using your gem to use things like ‘~>’ when specifying the version they’d like to use, and not have to worry too much about API breakage. A little bit of work by everyone to follow conventions goes a long way.

Even further: C extensions

I don’t have much experience creating gems with C extensions, so if you have any best practices to share, please get in touch and share them.

On Camping vs Sinatra

2011-01-30T21:36:21Z

Many years ago there was a tiny little ~~ladybug~~ Ruby web framework called Camping. Compared Rails’ thousands of lines of code, camping.rb was a tempting alternative for simple applications:

Welcome to the present, where every test framework ships with its own micro framework. Or was it the other way? Anyway, my point was: With so many alternatives, why would you want to go Camping? Why not Sinatra?

Some time ago, this discussion started over at the Hackety Hack mailing list, so I decided to chime in and present SIX (UNIMPRESSIVE) REASONS CAMPING IS BETTER THAN YOU WOULD IMAGINE.

Please don’t mistake me, this is not SIX (UNIMPRESSIVE) REASONS CAMPING IS BETTER THAN SINATRA or even SIX (UNIMPRESSIVE) REASONS YOU SHOULD DROP EVERYTHING YOU HAVE IN YOUR HAND RIGHT NOW AND START USING CAMPING. All I’m saying is that Camping gets so many things right. Not necessarily in very few lines of code or very fast, but nonetheless: I look at Camping code and nod to myself: “Yeah, this is probably the correct way to do it”.

Of course, this doesn’t really matter. If we cared about correctness, we would program in Haskell, not some language where monkey patching is acceptable in production code. As long as you’re comfortable in Sinatra (or any other framework), you should continue doing that.

Without any more ado:

#!/usr/bin/ruby
SIX (UNIMPRESSIVE)                 # Markdown version:
REASONS CAMPING IS BETTER          # 1) Download http://timeless.judofyr.net/camping-vs-sinatra.rb
THAN YOU WOULD IMAGINE             # 2) ruby camping-vs-sinatra.rb

reasons.push(COMMUNITY) do %%
  Yes, Sinatra has a big community, but Camping definitely has a great
  community too. Size doesn't always matter. Because there are so few users,
  it means every single issue gets full attention.

  If you're only looking at the GitHub repo or ruby.reddit.com, Camping
  might seem a little dead, but that's because everything happens on the
  mailing lists so it's not always visible from the "outside". (And other
  times, we're simply restin', just waiting for a question, suggestion or
  comment.)

  Besides, I don't allow Camping to disappear. Not because I need it in my
  business or something like that, but because the code is so fucking great.
  I simply won't allow it to die. Therefore I will *always* do my best to
  help people who are camping (just ask Eric Mill on this mailing list).
%%%%% end

reasons.push(UNPOLLUTED) do %%
  In Sinatra it's a norm (whether you use Sinatra::Base or not), in Camping
  it's the law:

      Camping.goes :Blog
      module Blog; end

      Camping.goes :Wiki
      module Wiki; end

  Every application lives under its own namespace. Yes, it requires a few
  more characters, but when you think about it, why *should* we allow are
  apps to run directly under the global namespace? That's surely not how we
  design our other Ruby code. What makes it so different? Shouldn't you for
  instance be able to `require "app"` and `include App::Helpers` somewhere
  else?

  Think of the environment; reduce your pollution!
%%%%% end

reasons.push(RESTful) do %%
  A central idea in REST is the concept of a resource, and that you can call
  methods on the resource (in order to get a representation of it). How would
  you apply these ideas in Ruby? What about this?

      class Posts
        def get; end
        def post; end
      end

  I would say this fits the description perfectly. You can instantiate
  instances of this class (with different parameters etc.) for each request,
  and then call methods on it. Guess how it looks in Camping?

      module App::Controllers
        class Posts
          def get; end
          def post; end
        end
      end

  The best part: Camping doesn't care if you use GET, DELETE, PROPFIND or
  HELLOWORD; every method is threated equally. One of the early ideas of HTTP
  was that you could easily extend it with your own methods for your own
  needs, and Camping is a perfect match for these cases!
%%%%% end

reasons.push(RUBY) do %%
  Ruby has wonderful features such as classes, inheritance, modules and
  methods. Why should every single DSL replace these features by blocks?
  Often, all they do is to hide details, without improving anything else than
  line count. Let me show you an example:

      get '/posts' do
        # code
      end

  Now answer me:

  1. Where is this code stored?
  2. How do I override the code?
  3. What happens if I call `get '/posts'` again?

  Not quite sure? Let's have a look at Camping:

      module App::Controllers
        class Posts
          def get
            # code
          end
        end
      end

  Since this is just "plain" Ruby, it's much simpler:

  ### 1. Where is this code stored?

  The code is stored as a method, and we can easily play with it:

      Posts.instance_methods(false) # => [:get]
      Posts.instance_method(:get)   # => #
      # Given post.is_a?(Posts)
      post.methods(false)           # => [:get]
      post.method(:get)             # => #

  ### 2. How do I override the code?

  Just like you would override a method:

      class App::Controllers::Posts
        def get
          # override
        end
      end

      # or, if post.is_a?(Posts)

      def post.get
        # override
      end

  ### 3. What happens if I call `class Posts` again?

  Because Ruby has open classes, we know that it would have no effect at all.

  Another advantage of having resources as classes (and not as blocks):

      module IUseTheseMethodsALot
        def get; end
      end

      module App::Controllers
        class Posts
          include IUseTheseMethodsALot
        end

        class Users
          include IUseTheseMethodsALot
        end
      end
%%%%% end

reasons.push(NAMING) do %q%
  In Camping you'll have to give every resource a name, while in Sinatra
  they're always anonymous. By giving resources a name you have a way of
  referencing them, which can be very convenient:

      post '/issue' do
        issue = Issue.create(params[:issue])
        redirect "/issue/#{issue.id}/overview"
      end

  Since every resource is anonymous in Sinatra, you're forced to hard-code
  the path. Not very elegant, and it can be a pain to update the code if you
  for instance want to move all urls from issue/ to i/. Camping's solution:

      class Issue
        def post
          issue = Issue.create(@input.issue)
          redirect R(IssueOverview, issue)
        end
      end

  The R method in Ruby returns the URL to a resource (which takes one
  parameter). Camping automatically calls #to_param on the arguments, so you
  can safely pass in ActiveRecord objects too. If you want to change the
  route to IssueOverview, you can do this in *one* place and you're done.
%%%%% end

reasons.push(RELOADING) do %%
      $ camping app.rb
      ** Starting Mongrel on 0.0.0.0:3301

  The Camping Server provides code reloading (so you don't need to restart
  the server while you develop your app) that works out of the box on *all*
  platforms (including Windows). We actually care about our Windows users!
%%%%% end
                                                                        ''
                                                                        ''
BEGIN {def Object.const_missing(m);m.to_s end;def method_missing(*a)a[1]=
$h.pop if a[1]==$h;$h.push(a) end;$h = [];def reasons; $reas ||= {};end;''
def reasons.push(r,&b);self[r]=b.call;end;END {puts h=$h*' ','='*h.size,''
reasons.each { |name, val| puts name, '-'*name.size, val.gsub(/^  /,''),''
                                                                        ''
}}}  # Please keep all my mustaches intact.   // Magnus Holm

Haters gonna HATEOAS

2010-01-23T14:05:23Z

Every time someone mentions RESTful web services, there’s always that one person that has to chime in: “That’s not really RESTful, it’s just kinda RESTful.” I’d always filed that information away, under ‘things to learn later,’ and let it simmer in the back of my brain. I’ve finally looked into it, and they’re absolutely right: 99.99% of the RESTful APIs out there aren’t fully compliant with Roy Fielding’s conception of REST. Is that bad?

Before we answer that question, let’s back up a bit: Why aren’t these web services RESTful? Just what is REST, anyway? REST was created by Roy Fielding in his dissertation if you’d like the full lowdown, but we’re more concerned with RESTful API design than we are in the full system. A more useful framework for this discussion is the Richardson Maturity Model. Basically, it defines four possible levels of ‘REST support’:

“The Swamp of POX.” You’re using HTTP to make RPC calls. HTTP is only really used as a tunnel.
Resources. Rather than making every call to a service endpoint, you have multiple endpoints that are used to represent resources, and you’re talking to them. This is the very beginnings of supporting REST.
HTTP Verbs. This is the level that something like Rails gives you out of the box: You interact with these Resources using HTTP verbs, rather than always using POST.
Hypermedia Controls. HATEOAS. You’re 100% REST compliant.

The four levels of REST

Let’s start at the bottom and work our way up.

Now, The Swamp of POX means that you’re using HTTP. Technically, REST services can be provided over any application layer protocol as long as they conform to certain properties. In practice, basically everyone uses HTTP. And since we’re discussing the creation of an API that conforms to REST rather than a system architecture based on the principles of REST, HTTP is a solid assumption on our part.

Level one is where it starts to get interesting. REST’s ‘resources’ are the core pieces of data that your application acts on. These will often correspond to the Models in your application, if you’re following MVC. A blog, for example, would have Entry resources and Comment resources. API design at Level 1 is all about using different URLs to interact with the different resources in your application. To make a new Entry, you’d use /entries/make_new, but with comments, it’d be /comments/make_new. So far, so good. This stuff is easy.

However, there are a set of common operations that are performed on resources, and it seems kinda silly to make a new URI for every operation, especially when they’re shared. That’s where Level 2 comes in. We’re always going to need to perform CRUD operations on our resources, so why not find a way to share these operations across resources? We accomplish this using HTTP Verbs. If we want to get a list of Entries, we make a GET request to /entries, but if we want to create a new Entry, we POST rather than GET. Pretty simple.

The final level, Hypermedia Controls, is the one that everyone falls down on. There’s two parts to this: content negotiation and HATEOAS. Content negotiation is focused on different representations of a particular resource, and HATEOAS is about the discoverability of actions on a resource.

Content Negotiation

At its simplest, this is something that Rails does right, too. Check out these lines from running rails scaffold:

def index @entries = Entry.all respond_to do |format| format.html format.xml { render :xml => @entries } format.json { render :json => @entries } end end

This is content negotiation. You’re able to use MIME types to request a representation of a resource in different formats. Rails 3 has made this even better, with respond_to/respond_with:

class EntriesController < ApplicationController::Base respond_to :html, :xml, :json def index respond_with(@entries = Entry.all) end

Super simple. So why do I say people get this wrong? Well, this is a great usage of content negotiation, but there’s also one that almost everyone gets wrong. Content negotiation is the answer to the question, “How do I version my API?” The proper way to do this is with the Accepts header, and use a MIME type like application/yourcompany.v1+json. There’s a great article about this by Peter Williams here.

HATEOAS

The last constraint is incredibly simple, but nobody actually does it. It’s named Hypertext As The Engine Of Application State. I still haven’t decided how to pronounce the acronym, I always try to say “Hate ee ohs,” which sounds like a breakfast cereal. Anyway, let’s break this down. We’re using Hypertext, fine, that makes sense. But what’s it mean to be an engine? And application state?

It’s all about state transitions. It’s right there in the name: Representational State Transfer. Your application is just a big state machine.

Your APIs should do this. There should be a single endpoint for the resource, and all of the other actions you’d need to undertake should be able to be discovered by inspecting that resource. Here’s an example of what our Entry XML might look like if Rails handled HATEOAS:


  1337
  Steve Klabnik

When we GET a particular Entry, we discover where we can go next: we can make a new comment. It’s a discoverable action. The particular state we’re in shows what other states we can reach from here.

Now, when I said ‘nobody’ does this, what I meant was ‘for APIs.’ This is exactly how the Web works. Think about it. You start off on the homepage. That’s the only URL you have to know. From there, a bunch of links point you towards each state that you can reach from there. People would consider it ludicrous if they had to remember a dozen URLs to navigate a website, so why do we expect the consumers of our APIs to do so as well?

There’s another benefit here as well: we’ve decoupled the URL itself from the action we’re having it perform. Think about it like the web: If we have a link that says “click here to make a new blog entry” and next week, we change it from /entries/new to /somethingelse/whatever, users of the web site (probably) won’t notice: they’re just clicking on the link that takes them where they need to go. If you changed the text to “click here to do something else” they wouldn’t expect it to be making an Entry anymore. By the same token, we can change the URI in our tag, and a proper client will just automatically follow along. Brilliant!

Why aren’t we doing this already?

Well, for one thing, the tooling just isn’t there to do this. Think of how web development was before Rails started emphasizing REST: some people got it right, but not many people cared. I know that I had teachers telling me that a

with method="GET" was perfectly fine, and that the only real difference between GET and POST is if the parameters are in the URL… but I digress. Until we make this kind of development easy to do, people aren’t going to do it. There’s also a serious lack of education on this topic. The web development community has been steadily improving as the web grows and matures, and so I hope that eventually we’ll see more people actually adopting HATEOAS and going ‘full RESTful.’

There’s also some discussion about how useful extra constraints actually are. If this is such a big important deal, why aren’t more people doing it? I haven’t yet implemented a 100% RESTful API myself yet, so I can’t really say. I do believe that I’ll be giving it a shot in the future, and I think that as our collective understanding of Fielding’s work improves, we’ll eventually see the value in REST.

Your blog is a project

2011-01-17T20:35:03Z

Maintaining a blog is surprisingly more difficult than you would imagine. The concept is so simple that anyone instantly understands it. The software has become so easy to work with that anyone can start one in a few minutes. Starting a blog has never been easier, yet keeping it updated still remains a most demanding task.

The truth is that blogging is often more about structuring and organizing than actual writing. Well, guess what? Successful software development is also often more about structuring and organizing than actual writing, so why not apply the same tools to your blog as to any other project?

Deadlines, or: Learn to ship

Who doesn’t hate deadlines? Who doesn’t want to continually work on something until it’s absolutely perfect and then release it to the world? Deadlines are often despised because estimation is never perfect and nobody enjoys pressure, but truth is that pressure is often required to get anything done. Especially when it comes to situations where no one will complain if you don’t get anything done (a new blog, startup or open-source project).

Don’t say “I’m going to blog when I have something interesting to tell”, say “I’m going to blog something each Sunday”. The implication of this is that you’ll have to find something interesting to tell every week. Of course, this is only a soft deadline: there will be no real consequences for not holding the deadline, but it will turn your blog from a passive project into an active project which brings it into a whole new level.

Another consequence of setting deadlines (and sticking to them) is that you’ll force yourself to ship. It’s so tempting to try to turn every blog post into a masterpiece, but far to often you get a better result by simply publishing and taking feedback from there.

Force yourself out of your comfort zone and start publishing!

Move out of your comfort zone. You can only grow if you are willing to feel awkward and uncomfortable when you try something new.

— Brian Tracy

Issue tracker, or: Nobody’s perfect

Software is rarely perfect, and in order to fix bugs you must first identify and organize them. Issue trackers are an important part of any project, so why not use it for your blog too? You just got an idea for something to write about? Sounds like a feature request to me. The RSS feed isn’t working correctly? Sounds like a bug. Don’t just make mental notes about your blog, treat it right a proper project.

Here at Timeless we have an issue tracker with tons of “reported” issues. It’s a simple place where I can quickly scribble down new ideas or just organize known “bugs”. Every time I go into “blog mode” I take a look at that huge list, and I always discover something new. For instance, I had no idea I was going to write this article until I peeked at the issue list and thought: “Yep, today I want to write about how you should treat your blog.”

An issue tracker is no different than a simple to-do list, so feel free to use whatever suits you best, but the important part is to actually write things down and don’t keep everything in your mind.

But wait, there’s more!

Not really. There isn’t more in this article yet. Why? Because I’ve already passed my deadline and I really need to get this article published. Instead of listening to the perfectionist inside of me, I’m just going to ship this little piece of text and rather work on it later.

As always, please let me know what you think about this idea.

Literate Programming

2011-01-10T22:26:43Z

Literate programming is an interesting concept that Don Knuth described in 1984, as an alternate way to write software. Literate programming places emphasis on the way that people think about software, rather than the way that computers do. Rather than write code in an order that’s imposed by the compiler, the literate programmer writes in words the way he expects a program to work, and then intersperses his description with code.

Continue to full post.

Mockup-Driven Development

2011-01-02T22:26:43Z

Design in web development is usually accomplished in three main steps:

Concept
Mockup
Template

Mockup-Driven Development attempts to merge the last two in order to make it easier to build, maintain and experiment with your application. Let’s have a look at some of the real issues present in today’s workflow and how MDD tries to solve them. We’ll also check out the current implementations, what issues they have and how we can improve them.

The Steps

Step 1: Concept

I’m not going to focus on the first step at all, so let’s just define it as everything that is done before you have a mockup in HTML. That includes sketches, Photoshop mockups, and so on.

Step 2: Mockup (in HTML)

The next step is to create some HTML so you can actually see how the site would end up. It’s all static files and nothing is functional, but you can at least get a feeling about what works and what you need to improve on. More importantly, mockups are a direct step on the path to a functional application: This is actually a concrete product which will be used later on.

Step 3: Template

The last step is to integrate the mockups with the rest of the application in order to Make Things Work™. Usually this involves annotating the HTML where the dynamic data should be inserted. I like to think about it as punching out “holes” which you’ll fill out with data at every request.

The Workflows

Most web application will, at least to some extent, go through these three steps and in order to get the best result you want to have the most skilled people working at each step: A designer works on the concept, a web designer knows how to turn it into a mockup and a developer makes it work.

I’ll focus mostly on the workflow between the mockups and the templates.

Separate mockups and templates

One workflow is that the web designer only works with the mockups while the developer only works with the templates. Every time there is a change in the mockups (or the templates for that matter), you’ll have to remember to keep the two sets of files in sync.

There’s one big advantage of this workflow: The web designer can continue working on the application in the exactly same way throughout the process. The great thing about this process is that it’s only static files: You can easily view them in any browser on any computer and you don’t need to have the application running in order to view the final result.

A disadvantage with this approach is that mockups are horrible to maintain. There’s no layouts or partials, so if the web designer wants to change something that’s shared among all the pages, he’ll have to update all the pages. You might use something like Dreamweaver’s template functionality to solve this issue, but now you’ve locked everyone into using a single tool.

And of course, now that you have two sets of the design you’ll constantly need to keep them in sync. Every change in the mockup needs to be reflected in the template (and the other way around). This means that while it’s super easy to experiment at each of the steps, you’ll end up with a massive merging process at the end of each iteration.

Everyone works on the template

Because of the disadvantages of the approach above, most people have a little different workflow. This is for instance the way 37signals works:

This is how we work:

Designer cooks up static HTML template.

Programmer sprinkle it with ERB tags to make the dynamic elements dynamic.

Designer and programmer can both revisit the template to make changes.

After the initial mockup (which you turn into a template), you just forget about it and everyone rather works directly on the template. This fixes two of the issues in the previous approach: There’s only a single set of files and you can use your template engine’s features to deal with layouts, partials etc.

The main disadvantage is that you know need to have the application running to see the result; you can no longer open the files in your browser. Depending on the complexity of your application and the qualifications of your team, this may or may not be an issue for you. If all your web designers know how to code, there’s not really a problem here. If you have a very simple application, you can probably teach most people how to run it. If you on the other hand have a quite complex application with many moving parts, you’ll need to do a lot of work to make it easily runnable for everyone.

Mockup-Driven Development

The concept behind Mockup-Driven Development is essentially to combine the concept of a mockup and a template. The optimal workflow in MDD goes something like this:

Designer cooks up static HTML mockup.
Programmer annotates the mockup so the template engine understands it.
Designer can still make changes to the mockup and view it in the browser.

MDD boils down to three essential features:

The template must be a static file which can be viewed in a browser.
The template must also be renderable by the template engine.
The template should be easy to maintain (e.g. no duplication).

Let’s have a look at two template engines that’s built upon MDD to easier understand what we’re really talking about.

Implementations

Lilu and Effigy

Lilu (written by Yurii Rashkovskii) was the first mockup-based template engine I discovered. It’s not being actively developed anymore, but luckily there’s Effigy, another project which is essentially a more modern version of Lilu. The concept behind them is the same.

You start off with a plain HTML mockup:

Post Title

class="body">Content of the blog

class="comment">

Comment Title

Content of the comment

id="no-comments">There aren't any comments for this post.

Then you write some glue code in Ruby:

class PostView < Effigy::View attr_reader :post def initialize(post) @post = post end def transform text('h1', post.title) text('title', "#{post.title} - Site title") text('p.body', post.body) replace_each('.comment', post.comments) do |comment| text('h2', comment.title) text('p', comment.summary) attr('a', :href => url_for(comment)) end remove('#no-comments') if post.comments.empty? end end

This approach makes your templates very designer friendly (there’s no “weird stuff”), at the expense of a more loosely connection between the template and the data. There’s nothing in the template which tells the web designer what is dynamic data, so he might break the page if he for instance changes the comment titles to use H3 tags instead.

RuHL

RuHL is “an opinionated attribute language” written by Andrew Stone. By placing the dynamic parts into a data-ruhl attribute (as shown below), we have made it explicit which parts are dynamic. As long as the web designer preserves these annotations, he can freely move everything around without breaking the page:

data-ruhl="title"> data-ruhl="title">Post Title class="body" data-ruhl="body">Content of the blog

class="comment" data-ruhl="_use: comments"> data-ruhl="title">Comment Title data-ruhl="comment">Content of the comment

id="no-comments" data-ruhl="_unless: comments">There aren't any comments for this post.

A few issues…

Initially, the examples above may seem to follow MDD, but there’s a few issues that makes them less convincing. First of all, there are no layouts or partials. Unless you want to repeat yourself in the templates, you’ll have to handle this in code inside your application.

For instance, in Effigy we might do something like:

class Layout < Effigy::View def initialize(view, template) @view = view @template = template end def transform html('#content', @view.render_html_document(@template)) end end

And suddenly we’ve broken one rule of MDD: You should be able view the file in the browser. If you open a template file which depends on the layout, you will only see the template and not the layout (no CSS or JavaScript for you, baby).

Also, if we take another look at the Effigy mockup, we’ll notice that something isn’t quite right:

Post Title

class="body">Content of the blog

class="comment">

Comment Title

Content of the comment

id="no-comments">There aren't any comments for this post.

When the application is running, the actual comments will never be displayed together with the “No comments” message, yet that’s exactly what happens in Effigy and RuHL. In this specific case it might not be so important, but there’s a deeper failure here: The mockup doesn’t actually present the actual application. Ouch.

JavaScript to the rescue!

Let me show you two hypothetical templates:

layout.html:

yield="content">

post.html:

Post Title

class="body">Content of the blog

class="comments" mockle:scenario="comments: Many">

Comment Title

Content of the comment

id="no-comments" mockle:scenario="comments: None"> There aren't any comments for this post.

Let me introduce you to Mockle (which only exists as a concept at the moment; there’s no code available). The main idea is to exploit the fact that every browser supports JavaScript. The template engine you invoke from your application will remove the