While writing a module to handle Google ClientLogin recently, I wanted to test error handling by simulating error responses from the server. A simple but powerful way of doing this is to use the patching ability of the mock module.

The patching ability allows you to replace objects in scope with mocks so that different side effects or return values can be defined. Note that ‘object’ is used here in the pythonic sense – referring to entities such as modules and functions as well as class instances.

This is best illustrated by a real example, so in this post we’re going to mock the requests module and generate the exceptions described in the documentation when a request is made.

Our example module sends credentials to Google’s ClientLogin service in order to receive an authentication token, required for accessing certain Google services (such as C2DM). If you are interested, you can read more about ClientLogin on the Google Code page.

So, to request an authentication token from the ClientLogin service, you POST a set of parameters including email and password to the service endpoint. Here is a cut-down version of the code that initiates the authentication request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class Authenticator(object):

    @staticmethod
    def generate_token(email, password, source,
                       account_type='GOOGLE', service='ac2dm'):

        """Returns an auth token for the supplied service."""

        base_url = 'https://www.google.com/accounts/ClientLogin'

        headers = {
            'content-type': 'application/x-www-form-urlencoded',
        }

        payload = {
            'Email': email,
            'Passwd': password,
            'source': source,
            'accountType': account_type,
            'service': service,
        }

        try:
            response = requests.post(base_url, data=payload, headers=headers)
        except(RequestException):
            raise

If the requests.post method throws an exception, we simply raise it to the caller rather than handling it immediately. In the requests module, RequestException is the base class from which others (like ConnectionError) inherit. We could improve our approach to exception handling but it is sufficient for this example.

These exceptions will be thrown from our mocked class, which is patched into the above code using a context manager:

1
2
3
4
5
6
7
8
9
10
11

import unittest2 as unittest
from requests.exceptions import RequestException, ConnectionError
from mock import patch
from clientlogin import Authenticator

class AuthenticationTests(unittest.TestCase):

    def test_connection_error(self):
        with patch.object(requests, 'post') as mock_method:
            mock_method.side_effect = ConnectionError
            Authenticator.generate_token('mail@example.com','password','tag')

Here, we have patched the post method in the requests module to throw a ConnectionError. You can think of this like code injection, where with acts as the closure.

In the real test method, we assert the exception was raised with another context manager:

1
2
3
4
5

def test_connection_error(self):
    with patch.object(requests, 'post') as mock_method:
        with self.assertRaises(RequestException):
            mock_method.side_effect = ConnectionError
            Authenticator.generate_token('mail@example.com','password','tag')

Here, we assert that the ConnectionError exception is raised to the caller, but we could easily have asserted a different condition. We could, for instance, have verified some exception handling logic.

As we’ve seen, mocking objects and methods in this manner is a simple but powerful way of running your code under different simulated conditions, allowing thorough testing of error-handling logic.

You can see the full module including tests and usage instructions at the github repository. For more information on the mock module, the full documentation is available.

Introducing Hypermedia Controls

Hypermedia as the Engine of Application State (HATEOAS) is an element of the REST architectural style, which describes how a consumer can control a given web application using hypermedia controls.

Let’s cut straight to the chase: what do we mean by hypermedia controls?

When I say hypertext, I mean the simultaneous presentation of information and controls such that the information becomes the affordance through which the user (or automaton) obtains choices and selects actions… Hypertext does not need to be HTML on a browser. Machines can follow links when they understand the data format and relationship types. –Roy Fielding

This explanation, from Roy Fielding’s widely cited blog post REST APIs must be Hypertext driven mentions the “simultaneous presentation of information and controls”. To simplify, we can imagine a hyperlink that our consumer is able to understand and manipulate.

So that’s the first word of HATEOAS. What does “Engine of Application State” mean? Quite literally, it means that the hyperlinks mentioned above can be used to control (drive) the web application. If you haven’t already drawn a state machine for your application, now’s the time!

Take, for instance, a RESTful webservice that allows virtual machines to be controlled. The Sun Cloud API is one example of such a service – the examples below are based loosely on the format it uses. If a given virtual machine is stopped, it be may started. Or, if a machine is already running, it may be stopped.

Example request for a machine representation:

1
2
3

GET /machines/1/
Host: example.com
Accept: application/xml

Example response:

1
2
3
4

HTTP/1.1 200 OK
Content-Type: application/xml
<status>stopped</status>
<link rel="start" method="post" href="machines/2?op=start" />

Note the link tag. This is a hypermedia control which tells the consumer how it could start the VM by POSTing to machines/{id}?op=start. (For simplicity I’ve omitted necessary details such as Content-Length and xml schema declaration.)

If the requested VM is running, the response would be:

1
2
3
4

HTTP/1.1 200 OK
Content-Type: application/xml
<status>running</status>
<link rel="stop" method="post" href="machines/2?op=stop" />

When our VM is running, the resource at machines/1/ contains a rel link to stop it. Both methods would probably return 201 Accepted in the case of success, as it is unlikely they would run synchronously.

In a nutshell, this is the concept of HATEOAS – these hypermedia resources allow us to control application state.

Note that the operation type in the query parameter could also have been a separate resource (such as machines/1/stop). There are arguments for either approach but it isn’t necessary to delve into this to understand the concept of HATEOAS.

Why Hypermedia Controls are Important

In the example above, the representation of a resource also gives us a way to manipulate the resource. In this way, HATEOAS allows certain aspects of the application to be self-describing, fostering loose coupling between service and consumer. If we consider that a service may evolve over time, then this loose coupling could allow the client to evolve with fewer issues than if all controls were hard-coded.

A useful resource that covers this in more detail is Wayne Lee’s presentation Why HATEOAS. Craig McClanahan also discusses the benefits of HATEOAS for service discovery and integration.

This post has necessarily glossed over many details of how HATEOAS is implemented, but has hopefully served as a useful introduction to the notion. I recommend REST in Practice as a resource for going into the subject in more detail. Additionally, the RESTful Services Cookbook contains details on implementation.

Further sources:

1. Roy Fielding: Rest APIs must be Hypermedia Driven
2. Ian Bicking on Hypertext Driven URLs
3. REST and HATEOAS
4. Interesting Stack Overflow discussion

Recently there’s been plenty of discussion about service platforms – Steve Yegge’s accidental post on Google Plus springs most prominently to mind. This article is partly motivated by these discussions, but it’s also a product of my own experience designing, implementing, and consuming services. I’ll include resources I’ve found useful when working on RESTful services.

I’m going to assume basic familiarity with REST, so you may find this introduction helpful if you need a place to start.

Using the Richardson Maturity Model as a Guide

The Richardson Maturity Model is a layered approach to REST. Aside from sounding like a psychological evaluation, it’s a helpful way of envisioning the different elements that will comprise your service, and how they are composed.

In essence, the model can be described as follows:

• The first level concerns implementing URIs for each resource (like /pizzas/1).
• The second level details how to interact with resources through HTTP methods (GET, POST, PUT, DELETE…). You can think of this like CRUD operations for resources, though we’ll discuss later how that doesn’t quite suffice.
• Finally, the slightly mysterious “hypermedia controls” level. I’ll cover this in a second article, which is why it is greyed out in the diagram.

I highly recommend you read Martin Fowler’s full article on the Richardson Maturity Model if you haven’t already. You’ll notice I’ve left out quite a bit of detail for simplicity (including the entirety of the lowest level!)

Exploring REST and CRUD Further

It’s worth looking at HTTP verbs (level 2 in our diagram), and how they map to operations on your resources, in more detail. A common approach is to map the following HTTP verbs like so:

1
2
3
4

GET    # Read a resource
POST   # Create a new resource
PUT    # Update a resource
DELETE # You guessed it...

If I want to retrieve a pizza representation, I just GET /pizzas/{id}. If I want to create a new pizza, I POST my representation to /pizzas. Change the topping on a pizza? PUT the updated representation to /pizzas/{id}.

Using HTTP verbs in this way is a good approach for many applications. But it can, in some cases, lead to lack of good separation between model and controller. Take an example: composite representations. The representation of a news story may include headlines of related stories, user comments, and author metadata. Combining data in this manner is common for e.g. mobile clients where the trade off between composing representations (and increasing response size) and the increased latency of multiple requests is deemed necessary.

If the service was designed as a CRUD layer over some database models with no additional composition ability, then implementing these representations can lead to unexpected complexity. It pays to think about resources that don’t map directly to business objects and the way in which a controller will handle this. See Subbu Allamaraju’s excellent cookbook recipes (pp. 30-37)

This is something that is worth bearing in mind when choosing a framework on which to build your RESTful service. In a number of frameworks, the tendency is to derive controllers directly from models. How would you express the above representation in such a framework? Would it save time to use something with greater flexibility?

A final point, which leads on from the notion of generating representations directly from models, is versioning. If your representations are strongly coupled to business objects, changing your database schema implies change to your service. Templating representations helps dissociate the representation from the model, forcing potentially client-breaking to be acknowledged by explicitly modifying a template.

There are a couple of different ways to ask a webservice for a specific version of a resource. Some architects prefer the version to be passed as a header, but there is also a camp that supports having the version in the URL. Ian Robinson’s presentation (slide 20) covers this well.

Conclusion

We’ve covered the lower two levels of the Richardson Maturity Model with some expansion on mapping resources to business objects. There are many other subjects to cover – caching, error-handling patterns, content negotiation, but hopefully the above has provided some useful points to consider when designing a RESTful service and choosing a framework.

In my own work, I have found REST in Practice by Jim Webber, Savas Parastatidis and Ian Robinson a good resource.

In the next article I’ll talk about the final tier of the Richardson Maturity Model: hypermedia controls.

Customers don’t know what they want. Stop expecting customers to know what they want. –Joel Spolsky

A couple of months ago I attended a presentation by the Head of EMEA Sales at Huddle, Neil Ryland. In describing the sales process, one of the most interesting points raised was how feedback from the sales force is incorporated into the development lifecycle.

Huddle’s products are aimed at the enterprise segment. As part of the sales process, there are likely to be multiple conversations between the sales representative and the customer. This direct contact with the client is so valuable that it can be used in planning the next iteration of the product. The process goes something like this:

1. Analyze the failed sales. Could the sale have been closed by offering a slightly different feature set?
2. Analyze the successful sales. Which features of the product were most influential in the customer’s decision?
3. Highlight the most compelling points to the product manager and development team at planning meetings.

Note that this is my interpretation of what was stated – I’m not acquainted with Huddle’s internal processes, so don’t assume this accurately reflects the process. Yet the important point is this: sales has a direct line to product development.

Clearly, this only one of the different factors that should be considered when a backlog is written or prioritised. But it is an important one that should not be overlooked.

Out of this process grows the phrase Sales Driven Development. I’m fairly reluctant to use it, since it implies a complete development methodology. It’s not necessarily a driver of product iteration, but behaves more akin to a regulation mechanism.

An interesting facet to this approach is how it interacts with site-specific customisation, practiced by many software firms targeting the enterprise. Of course, with a cloud-based product such as Huddle’s, site-specific customisation makes little sense. Instead of customisation, a feature may be built into the next release. And, of course, these two practices certainly aren’t mutually exclusive approaches.

Returning to the introductory quotes, what we’ve covered so far is certainly no counterpoint. I strongly agree with the notion that it is not the customer’s job to know what they want. But it is interesting to explore how an iteration cycle can incorporate what customers say they want, particularly when they have already encountered the product.

In this sense, software companies with a sales force may well be at an advantage. Why? Because of the dialogue a sales representative can engage in with a lead. In terms of customer interaction, it clearly outstrips a feedback form. But this is a fairly obvious point.

A more interesting question is this: how do we derive frank feedback, and to whom should we listen?

One method is to focus on the sales successes. With many people in an enterprise using your product, their experience of the product is likely to be wider than an individual might achieve. There may be even internal factors that drive how the product is used. Understanding the initial success and following up on how the product is used internally can provide this.

Another approach is to look at the unsuccessful cases that included a trial of the product. This provides the grounding that the potential customer was willing to invest time to assess the product and discover why it wasn’t right for them.

It’s interesting to think how this approach might factor into a different revenue model, such as freemium. But since this is essentially a separate discussion, I will attempt to cover it in a future article. In any case, we’ve covered the main point of this article: examining a case where sales feedback is given a direct line to product iteration.