With only a flip chart and markers I started designing from scratch a web API with heavy use of hypermedia for an imaginary “groups and events” website a la meetup.com. The following features have been covered iteratively, one by one, showing the benefits of the hypermedia approach:

• events – open and link together the core data of the site;
• personalization – user-specific resources like “my events”;
• search – allow for parameterized querying;
• groups and topics – simply more linked data mainly to illustrate addition of core features at a later stage;
• RSVP – doing updates driven by hypermedia.

Start

I asked the audience which format they prefer for the representations and they voted for JSON although I mentioned that HTML would make many further steps easier. Anyways, format is in my opinion subject to content negotiation and a new one can always be added later.

Usually the next step would be to come up with a URI tree of resources, which will form the API. This wasn’t what I did though. Instead I suggested to decide on a hypermedia-driven way the clients will discover available resources. With absence of hypermedia RESTful APIs rely on developer documentation for that purpose. A much better approach to me is a single well-known resource, serving as an entry point into the API and containing only the valid of the potential next interaction steps. The Home Document draft for example specifies how such resource might look like in JSON.

Linked Data

Having defined the entry point into the API, we can move to modeling the core domain resources. Since we solely rely on hypermedia for discovering resources starting from a single well-known entry point, it is logical to visualize the design process with a directed graph, where nodes represent resources (or states of application interaction) and edges represent hypermedia controls leading from one resource to another.

An example of such a graph is my poor flip chart drawing on the right. Here we start with the Home resource and by means of a hypermedia link known as upcoming-events navigate to a resource of type Events representing all the upcoming events on the site. Nodes on the graph really represent whole classes of resources or resource types – showing individual resources on a graph would hinder the overview.

The upcoming-events hypermedia control (a green arrow) is simply a link in a JSON Home Document, but the Events representation has a collection semantics, containing individual events, which are separate resources on their own right. To avoid reinventing the wheel, we’ll use the Collection+JSON media type to represent event collections. Collection+JSON resources can reference other resources via links, which comes in handy e.g. for pagination with link relation types next and prev.

There is a more ubiquitous representation of event collections though, which is iCalendar. iCalendar format automatically opens our data to all sort of calendar applications like Outlook, iCal and Google Calendar. It is a low hanging fruit in terms of interoperability, which is the primary aim for open data and APIs. Solutions like iCalendar or Atom feeds should probably be considered as Lo-Fi API alternatives at the early stage of API development.

In REST terms iCalendar and JSON are just different representations of the same resource and thus should share the same URI. The client then will have to ask for the desired representation with the Accept request header. This is unpractical though, since the iCalendar feed will be requested by clients like browsers and calendar applications, which won’t bother requesting the special iCalendar media type. A much more interoperable option would be to provide a link with the relation type alternate from the JSON representation (or add it directly to the Home).

Personalization

From the hypermedia perspective a resource like “my events” is just another resource, linked from somewhere and linking to somewhere else. The trick is that it can only be accessed with a valid authentication. The drawing above shows two ways of of discovering personalized resources via hypermedia: The first one is the link my-upcoming-events, which — when activated — responds with a standard HTTP authentication challenge (depicted with the red padlock). Another option is a loopback link (red arrow Auth) from Home to itself, meaning that the client receives upon authentication a personalized representation of the Home resource with a (previously not present) my-upcoming-events link.

Querying

Event search results are basically just a collection of events, for which we already have a resource type Events. We will reuse the same linking mechanisms to expose the event search results via the search-events link. Search however requires some user input to generate a representation of search results. The URI Template specifies a way of formatting parameterized URIs. There are already libraries available for expanding URI templates, so the clients shall have no problems consuming those.

More Links

We can now search for and navigate upcoming events, which are already valuable features. Even without topics and groups the API is a viable product worth exposing and consuming. Adding topics and groups to the mix at this stage is neither late nor difficult. We add all-topics link from Home to the Topics resource type. It might as well be popular-topics or any other reasonable default collection semantic, or even several of them to choose from. The Topics collection contains individual topics and may be searched or filtered using a parameterized link. A Topic resource references a collection of events and a collection of groups, both filtered by the topic. Groups and Group resources follow the same pattern of collections and links.

Collection+JSON provides collection semantics and linking capabilities. It is however not the only one option. The Hypertext Application Language (HAL) allows to embed multiple resources in a single JSON (or XML) representation and to link to other resources. The Web Linking describes a format-neutral way of putting links in a special Link HTTP response header.

Updates

Until now we’ve dealt with safe read-only HTTP GET requests, but rsvp’ing an event is an update and should be modeled with an appropriate HTTP verb.

In case of a simple yes/no answer it is sufficient to mark the respective accept or decline link (see the drawing on the left) with the "method": "POST" attribute. The client will POST the request to the specified URI and the server will redirect the client back to the Event resource. But what if we want to receive a short text note from the user along with the answer?

We could also model the RSVP resource as a Collection+JSON type with a built-in item template and defined semantics for adding new items. That way the client navigates to the RSVP collection from the corresponding event, then it finds the item template, fills the template out (adding the attendance status and the optional text note), POSTs the filled-out template to the specified URI and gets redirected to the updated RSVP collection.

Another option would be to add the notion of a POST payload template to other types of links (e.g. HAL links). The benefit of explicit accept and decline links — compared to a generic collection update — is that the server only exposes the valid options in the current context: the user can’t accept an event twice.

Bottom Line

This approach to designing Web APIs is very simple and boils down to defining resources and deciding which formats suit representations better and which hypermedia controls to use to navigate from one resource to another. In this example we used the following hypermedia controls: collections, simple links, authenticated links, query links and update links. There are already standards for implementing some these basic building blocks and more standards will emerge. Focusing on the semantics of hypermedia controls during design will help to evolve and tweak the implementation down the road.

Let’s consider the amount of out-of-band information, which client developers will have to look up in the API documentation in order to make API calls:

• The single URI of the API entry point (the Home Document);
• Link relation types used in the API (e.g. my-upcoming-events and accept) and their meaning;
• Formats of the resource representations (e.g. Events, Event, Group etc.);
• URI Template parameters and their meaning (e.g. searchTerms as the search query input);
• Parameters used in the item templates for update requests and their meaning (e.g. note for the text note of an RSVP answer).

All this information is domain-specific and is basically what differentiates a social events API from an accounting API. All the technical details are encapsulated into generic media types (e.g. Collection+JSON and HAL) and is ideally handled by generic hypermedia client libraries.

References

• Home Document draft
• Collection+JSON
• Link relation types
• iCalendar
• Atom
• URI Template
• Hypertext Application Language
• Web Linking

The most notable standardization effort in this area — Hypertext Application Language — takes in my opinion the all-or-nothing limiting and heavyweight approach: your implementation either complies with HAL or it doesn’t. So implementers who don’t want 100% HAL just roll out their own completely unique media types with no interoperability at all. Can we achieve interoperability and embrace diversity at the same time? It turns out that we can.

Evolutionary Lesson

Nature itself has solved the same problem quite elegantly: every living being carries sequence (the DNA) of standardized parts encoding a unique combination of properties the living being will have. During replication two DNAs are combined to produce a new and unique one.

What if hypermedia types would be defined as combinations of standard building blocks? These small standardized building blocks — hypermedia genes — will describe how exactly that particular hypermedia factor is supported by an implementation. API designers won’t have to decide whether the complete HAL specification meets their needs. An API designer might like HAL link objects, but dislike underscores in attribute names and not need resource embedding. No problem, just cherry-pick useful genes and replicate a new and interoperable hypermedia type.

The Hypermedia Genome

How can these hypermedia genes look like? Small hypermedia features in different flavors like a particular representation of links or ways of reporting errors can be defined, named and catalogued. Last year I started describing a set of such hypermedia controls for JSON named JSON Hypcos (hypco is short for hypermedia control). The list is by no means complete, those are just some of the hypermedia controls I use in my work.

Describing individual hypermedia controls instead of trapping them in all-or-nothing media types has many benefits. Like microformats applied to existing data, standardized hypermedia controls can describe existing API designs giving them standardized semantics.

Client developers can grok new APIs easier, if those APIs are defined out of well-known building blocks. Documenting APIs will also require much less effort, if hypermedia controls become commodity. Generic hypermedia client libraries can hide APIs’ diverse flavors behind equivalent semantic of hypermedia controls, allowing client developers to focus on application semantics instead of the wire implementation of the interface.

Discoverability

The good thing about standardized media types is that we further have standardized means of advertising and discovering client and server capabilities in terms of media types. I talk about the Accept and Content-Type HTTP headers. If individual hypermedia genes become media type description units, we also need content negotiation on this finer-grained atomic level.

First of all, each hypermedia control flavor must be uniquely addressable. URI? Probably. Consider urn:json-hypco:link-object or even shorter urn:hc:lo. Remember that a complete media type will consist of a dozen or more of these. For efficiency multiple hypermedia genes can be combined like this: urn:hc:lo+lr+sl+lh+tu (hypermedia chromosomes?).

API’s complete hypermedia DNA might be advertised by the server with an HTTP response header like Content-Controls: urn:hc:lo+lr+sl+lh+tu urn:foo:bar .... Clients might also declare their level of hypermedia support: Accept-Controls: urn:hc:lv+tu.

What Next?

This blog post is my unofficial request for comments. It is my take at bringing interoperability into the emerging hypermedia API space while still embracing its diversity and evolution. I’m more then excited to hear from anyone interested in the topic.

I was at the local agile meet-up when the same old topic of Scrum dysfunctions and a lack of process guidance arose. In my opinion the ability to act in absence of process guidance is really a sign of a competent practitioner as opposed to a “certified process follower”. Without clear prescriptions the richness of one’s own toolbox is essential. Recent study of the Lean Startup methodology and especially the read of Running Lean by Ash Maurya gave me some more ideas of what to do when the process of choice is silent.

Missing User Roles

“As a [role] I want [feature] so that [benefit].” The most important part of a user story is not the feature but rather the user role and the benefit he wants to achieve. It has been proven effective for ages in the agile world to put personas in the user stories, but customer development makes it obligatory. You have to know your customer segments and their must-have problems in order to provide benefit to the user. If your user stories don’t mention a particular customer group and a problem (which you’ve validated qualitatively), chances are you’re building some sort of waste. Note that “working software” and “definition of done” is all waste, if the problem it solves is not worth solving.

Backlog Prioritization

Juggling backlog priorities is often compared with art, politics, black magic, blind guesses, coin flipping – with anything but a scientific method. One scientific approach to prioritization would be to eliminate risks highest first and thus accelerate learning. After qualitative validation of the biggest assumptions it’s time to verify them quantitatively. So building an experiment for quantitative verification of the riskiest hypothesis is clearly the most important thing right now. And if the simplest possible way to build that experiment is by giving a working software in the users’ hands, great.

When building the solution one increment at a time, constantly trace the features back to the customer segments and their problems. Is this feature a part of the minimal viable product for that particular segment? If not, chances are you’re building something for a new customer segment which has to be validated and verified first.

The lean canvas (and its ancestor, the business model canvas, for that matter too) is a perfect tool to capture two important aspects of of a product idea: cost structure and revenue streams. A product idea can have multiple revenue streams, one for each customer segment. Analyzing which feature contributes to which revenue stream can make prioritization a lot easier.

Acceptance Criteria

This is the new definition of DONE: You’re done, when you’ve validated, verified or denied a hypothesis. The difference between validation and verification being that you validate qualitatively and verify quantitatively. Registering either a positive or a negative signal when talking to people is qualitative validation. Turning 10% newsletter recipients into paying customers is quantitative verification.

Look at the build-measure-learn principle. When defining the acceptance criteria for a user story, think about learning first. What hypothesis are you trying to verify and what measurable metric to use for it? If nothing comes to mind, chances are you’re not developing, because development implies learning.

Systems Thinking (In Closing)

It’s really about the holistic view of the value chain. You can pretend to be doing Scrum or some other agile method just in the IT, but local optimization will inevitably bite you in the ass. Customer development techniques are a powerful tool to help a product owner with his homework – backlog grooming and prioritization. Keeping a constant eye on the aspects of the lean canvas helps making uncomfortable decisions with a reason and confidence.

[Update 11th Oct 2012]: Improved wording used for the user story.

As a long-time proponent of Domain-Driven Design and a follower of its recent advancement – Command-Query Responsibility Segregation – the solution to apply CQRS to cure anemic domain model seemed quite natural to me (and it still seems so). But how exactly CQRS helps designing hypermedia APIs?

In a nutshell the approach is to model operations that change state as resources on their own, not as methods on data resources. CQRS would call such operations commands and the read-only linked data – queries. I will further call them just actions and data respectively. When represented as resources actions can be linked from associated data resources. Furthermore depending on the state of the data and availability of actions according to that state, data resources can decide to either include or exclude particular action links. Now that’s a navigable API!

Let’s see some actions in action. Consider a resource representing the last 3 calendar events of the user. Depending on whether the event isn’t yet in the past the user can either accept or decline it. An already accepted event can be declined and an already declined event can be accepted as long as they still are in the future. Here is a simplistic JSON representation of such a resource:

The structure of the event objects and the format of the links are not worth paying attention to at the moment. The point this example tries to illustrate is that the action resources are linked in the same manner as any other resources and their presence in representations depends on the state of the data resources they are linked from.

But is a label and a URL enough to describe an action? An action might accept additional information, like a textual comment in the case of a negative RSVP, or require some other data in order to be executed. Actions also tend to change the state of resources and thus should be activated via an unsafe method like POST. How to express such details in an API directly without a need of out-of-band knowledge? This will be the topic for the next installment of my hypermedia blog series which hopefully won’t take another month to write.

Let’s consider a valid case first. CouchDB, a NOSQL document-oriented database, provides an HTTP API which can be considered restful to a degree: it returns meaningful status codes, respects HTTP caching, uses meaningful HTTP methods. In CouchDB documents and document collections are all resources, they have unique identifiers – URLs, and the client interacts with resources through their representations.

Another such example is Atom Publishing Protocol – AtomPub. It has the same uniform CRUD interface over resources, which are collections and items. AtomPub has even nice hypermedia controls – links, which make it perfectly restful. One common thing about these two API examples is that none of them exposes any domain logic. At most, some CRUD operations may require authorization and basic input parameter validation, but no context-dependent logic.

For a utility API like database or publishing protocol it’s not a big deal though – database domain logic is just CRUD. For any but most simplistic API however CRUD is a symptom of an Anemic Domain Model. It is the opposite of the pit of success that any customer-focused service wants its API to be. But why exactly is that’s the case?

If resources are modeled solely after data entities the data versus operations impedance mismatch arises. On the read side everything is fine – given meaningful links between resources, the client can happily navigate all the way through the exposed API data. But as soon as anything needs to be changed the client is left alone with HTTP POST, PUT and DELETE methods, which all have very strict semantics and operate on the resource level. And no, PATCH method is not a solution here, since the problem is not how fine-grained control the client has over data.

The problem is that POST, PUT, DELETE and even PATCH all have too generic semantics to represent any valuable domain operation. POST doesn’t define how the posted resource representation should look like. PUT means representation replace at its entirety, while PATCH allows partial updates. Though neither PUT nor PATCH can communicate which parts of the representation can be updated or what the user’s intent was, making them almost unusable for modeling domain operations. And DELETE is almost never a good business operation (again, data stores and content management systems not considered), if a resource was important enough to be uniquely addressable in the first place.

Not only does complexity grow on the client, the server implementation gets complicated too. The server has to reverse-engineer the lost intent of the user from representation diffs in order to apply domain validation. Different HTTP methods are called in different contexts and require different authorization, but many web frameworks map resource actions to methods on a resource-bound class. This leads to violation of the Single Responsibility Principle and is prone to authorization and validation errors. Moreover read and write operations often require different non-functional considerations, so it makes more sense to separate reads and writes at least on the implementation level.

The described problems result in the impedance mismatch between the server and the client. If the client is not a simple forms-over-data, it “thinks” in domain operations (tasks, actions) for the write part of the domain. A solely data-centric API just doesn’t fit there.

I hope to make a case for the design smell in this blog post and plan to outline a solution in an upcoming post.

To keep updated feel free to subscribe to my feed. The feed address remained the same from my previous blog, so if you read this in your feed reader, everything’s probably fine as it is.

I intend to blog here about anything I consider valuable around software engineering and software craftsmanship, web architectures and hypermedia, and all things agile. Without further intro fasten your seatbelts and enjoy the ride. To be continued…