Blueprint Forge.

Updates to Coder Weekly

2013-02-16T18:02:00+00:00

Coder Weekly has been running for over a year. Over that time, the subscriber base has continued to grow (now approaching 5,000), and the format and content has changed somewhat. I thought that this would be a good point to look back at what’s worked, and describe some changes that will be made to Coder Weekly in the near future.

Link submissions

I’ve had many tips submitted for content over the past year. If I recall correctly, all of them have been used because they’ve all been of high quality. I encourage people to keep submitting tips as there’s always something I will miss!

One thing to bear in mind when submitting tips is that a high proportion of readers are on mobile devices.

Job postings

This week I’ll be accepting the first job postings to Coder Weekly. This is a trial to see how things work – in particular, I want make sure it provides value both to subscribers, and those submitting job postings. Suffice it to say that there will be no agency or recruiter postings.

See here for details about submitting jobs.

Book reviews and affiliates

I’ve featured many books over the last year. There are two upcoming changes to this:

I plan to actively start reviewing books and publish the reviews either here or on the Coder Weekly website.
The links to these books (on the site and in the newsletter) may use an affiliate scheme, if one exists, for the vendor.

Aside from my time, there are material costs to running a newsletter. The affiliate links are to offset this.

Listed books will be read by me, and I will only list the best stuff. I don’t want to waste my time or anyone else’s with mediocre books.

Book tips are great too!

Let me know what you think

Comments are always welcome: you can reach me at matt@coderweekly.com, or on twitter.

Exploring SQLite's Virtual Database Engine

2012-10-16T22:12:00+01:00

I’ve written an article over at Coder Weekly on SQLite’s Virtual Database Engine. It’s part I in a series on the topic of SQLite’s inner workings.

Go to the full article

6 Weeks of Coder Weekly: the Numbers

2012-03-13T19:00:00+00:00

Coder Weekly has been running for over 6 weeks now, so it seems like a good time to share some numbers. So far, the community feedback has been really positive and we’ve reached 3,000 email subscribers, with another 800 or so on RSS.

Initial Traffic

The first peak in this graph is the submission to Hacker News, and the second is the submission to /r/programming. You can see that, overall, reddit sent a lot more visitors. Due to the ranking algorithm differences of the two sites, traffic from reddit also continued for longer.

Below, you can see how these spikes in traffic translated into new subscribers (y-axis shows new subs/day). You can’t quite attribute all signups to Hacker News and reddit since there were other traffic sources too (e.g. twitter), but it gives a reasonable indication of the conversions.

Overall, well over 2,000 subscriptions occurred from these ~19,000 unique visitors, a conversion rate over 10%. Unfortunately I haven’t run the numbers from the new version of the landing page, so I can’t on speculate whether this has improved the conversion rate.

Finally, the image below shows the number of RSS subscribers, which quickly rose past 800. It was clear from the outset that a large number of readers preferred this, so I’m glad Mailchimp made it so easy to offer.

Ongoing Growth

After each issue, there is an uptick in subscribe rate. It’s great that people are tweeting about Coder Weekly and forwarding the emails. Conversely, unsubscribe rates are extremely low – we’re looking at between 2-5 subscribers per issue.

So, although we haven’t enjoyed the same incredible growth of the first week, there is steady growth that seems to be driven primarily by current subscribers.

Click Data

There’s lots to be said about click rates for different articles. As you’d expect, there is a strong correlation between the position in the list and click rate. The article’s title is clearly an important factor too, with titles that describe an opinion or technique doing well. Below, you can see the top clicked links in Coder Weekly Issue #5.

I intend to do a longer post discussing click data in future, hopefully with some interesting split testing data.

The Future

I hope this brief overview has provided some interesting numbers for those of you who are curious. In future, there are a few ideas I’d like to work on. Providing a separate feed for individual stories is something that’s been asked for many times that I’m still working on, and it would also be illuminating to do some split testing on the landing page.

Finally, a huge thanks to all of you who’ve subscribed and/or given feedback!

Static Modification of Python with Python: the AST Module

2012-02-27T10:00:00+00:00

Source code modification can be useful in a number of testing and analysis scenarios. Here, we’ll look at how you can modify Python source code using the ast module, and some tools where this technique is used.

The CPython compilation process

To begin, let’s take a look at the CPython compilation process, as described in PEP 339.

Detailed knowledge of these steps isn’t required for reading this article, but it helps to have a rough idea of the whole process.

First, a parse tree is generated from the source code. Next, an Abstract Syntax Tree (AST) is built from the parse tree. From the AST, a control flow graph is generated, and finally, the code object is compiled from the control flow graph.

Marked in blue is the AST stage, since that’s what we’ll be focusing on today. The ast module appeared in its current form in Python 2.6, and exposes a simple method of visiting and modifying the AST.

From there, we can generate a code object from our modified AST. We can also re-generate source code from our modified AST for explanatory purposes.

Creating an AST

Let’s define a simple expression, an add function, and inspect the generated AST.

>>> import ast
>>> expr = """def add(arg1, arg2): return arg1 + arg2"""
>>> expr_ast = ast.parse(expr)
>>> expr_ast
<_ast.Module object at 0x7f8718b171d0>

Now that we have generated an ast.Module object, let’s dump the contents:

>>> ast.dump(expr_ast)
"Module(
    body=[
        FunctionDef(
            name='add', args=arguments(
                args=[
                    Name(id='arg1', ctx=Param()),
                    Name(id='arg2', ctx=Param())
                    ],
                    vararg=None,
                    kwarg=None,
                    defaults=[]),
            body=[
                Return(
                    value=BinOp(
                        left=Name(id='arg1', ctx=Load()),
                        op=Add(),
                        right=Name(id='arg2', ctx=Load())))
                ], 
             decorator_list=[])
    ])"

As we can see, Module is the parent node. The Module body contains a list with a single element: our function definition. The function definition has a name, list of arguments, and a body. The body contains a single Return node, which contains the Add operation.

Modifying the AST

How can we modify this tree and change how the code works? To illustrate, let’s do something crazy that you would never want to do in your code. We’ll traverse the tree, and replace the Add operation with a Mult operation. See, I told you it was crazy!

We’ll start by subclassing the NodeTransformer class, and defining the visit_BinOp method which is called when the transformer visits a binary operator node:

class CrazyTransformer(ast.NodeTransformer):

    def visit_BinOp(self, node):
        print node.__dict__
        node.op = ast.Mult()
        print node.__dict__
        return node

Now that we’ve defined our transformer which performs this unhealthy action, let’s see it run on the expression defined above:

>>> parser.visit(expr_ast)
{'op': <_ast.Add object at 0x226b810>, 'right': <_ast.Name object at 0x285fe10>, 'lineno': 1, 'col_offset': 28, 'left': <_ast.Name object at 0x285fdd0>}
{'op': <_ast.Mult object at 0x28eb090>, 'right': <_ast.Name object at 0x285fe10>, 'lineno': 1, 'col_offset': 28, 'left': <_ast.Name object at 0x285fdd0>}

You can see the Add node is replaced with a Mult, as shown by the modified __dict__. There are a couple of things we haven’t dealt with here, like visiting child nodes, but this is enough to illustrate the principle.

Compiling and executing the modified AST

After adding a call to our operation, i.e.:

print add(4, 5)

…to the end of our script, let’s see how the code evaluates:

>>> unmodified = ast.parse(expr)
>>> exec compile(unmodified, '<string>', 'exec')
9
>>> transformer = CrazyTransformer()
>>> modified = transformer.visit(unmodified)
>>> exec compile(modified, '<string>', 'exec')
20

As we can see, our unmodified and modified ASTs compile to code that prints 9 and 20 respectively.

Translating back to source code

Finally, we can use unparse found here to view the source code corresponding to our modified AST:

>>> unparse.Unparser(modified, sys.stdout)

def add(arg1, arg2):
    return (arg1 * arg2)
print add(4, 5)

As we can see, the * operator has replaced +. Unparse is useful for understanding how your AST transformer modifies code.

Practical applications

Clearly, our above example serves little practical purpose. However, static analysis and modification of source code can be extremely useful.

You could, for instance, inject code for testing purposes. See this Pycon talk for an understanding of how a node transformer can be used to inject instrumentation code for testing purposes.

In addition, the pythoscope project uses an AST visitor to process source files and generate tests from method signatures.

Projects such as pylint also use an AST walking method to analyze source code. In the case of pylint, Logilab have created a module which aims:

to provide a common base representation of python source code for projects such as pychecker, pyreverse, pylint…

See here for more information.

References

This Pycon talk by Matthew J Desmarais and this blog post by Eli Bendersky were invaluable in writing this post.

Websockets in Play 2.0

2012-02-13T08:10:00+00:00

Update: this post was written in early 2012 when Play 2.0 was beta software and is preserved for posterity. I make no promises about its suitability for current development with Play, and hope to do an updated post in future.

The source code for this guide can be found in the github repository.

Websockets allow full-duplex communication over a TCP socket, normally between a browser and a server-side application. In a similar vein to this post on websockets in Play 1.x, this article shows how to create a websockets-enabled application in Play 2.0.

Remember that Play 2.0 is still in beta, so the APIs shown below are subject to change. If that happens, I’ll endeavour to update this article with the changes.

Obtaining Play 2.0

I encountered an issue with the packaged beta, so I downloaded the Play source (full instructions):

git clone --depth=1 https://github.com/playframework/Play20.git
cd Play20/framework
./build

While it builds, you’ll probably want to get some coffee. Then, at the sbt prompt:

publish-local

Now, create the project:

./play new websocks2

You can run the eclipsify command (or equivalent) if you want to open the project in your IDE.

Creating the Websockets Controller

With setup done, let’s define the controller that will serve our index page and establish a websocket with the client. Open controllers/Application.java and add the following code:

package controllers;

import play.*;
import play.mvc.*;
import play.libs.F.Callback;

import views.html.*;

public class Application extends Controller {
  
  public static Result index() {
    return ok(index.render());
  }
  
  public static WebSocket<String> sockHandler() {
      return new WebSocket<String>() {
          // called when the websocket is established
          public void onReady(WebSocket.In<String> in, WebSocket.Out<String> out) {
              // register a callback for processing instream events
              in.onMessage(new Callback<String>() {
                  public void invoke(String event) {
                      Logger.info(event);
                  } 
               });
              
              // write out a greeting
              out.write("I'm contacting you regarding your recent websocket.");
          }
      };
  }
  
}

As you can see, alongside the pre-generated code, we’ve added a static method sockHandler that returns a websocket. The inner onReady method takes two arguments: the in channel and out channel.

At this point, we could pass those channels to a model which retains them. But for this example, we simply register a logger for inbound messages, and write a greeting message to the out socket.

Routing the Websocket Connection

Next, we need to add a route to our websocket connection. In conf/routes, add the following:

# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~

# Home page
GET     /                           controllers.Application.index()
GET     /greeter                    controllers.Application.sockHandler()

# Map static resources from the /public folder to the /assets URL path
GET     /assets/*file               controllers.Assets.at(path="/public", file)

We don’t actually request /greeter explicitly; instead, we’ll use a reverse generated URL in our template.

Adding the Template

Now that we have our controller and route in place, we can specify the client-side code for our websocket.

@main(null) {
    
    <div class="greeting"></div>
    <button class="send">Send</button>
    
    <script type="text/javascript" charset="utf-8">
    
        $(function() {
            var WS = window['MozWebSocket'] ? MozWebSocket : WebSocket
            var sock = new WS("@routes.Application.sockHandler().webSocketURL(request)")
            
            sock.onmessage = function(event) {
                $('.greeting').append(event.data)
            }
            
            $('button.send').click(function() {
                sock.send("I'm sending a message now.")
            });            
        })
    
    </script>

}

Here, we declare a greeting which is populated when the websocket receives a message. We also register a button action to send a message to the server, so we can demonstrate bi-directional messaging. If you’re familiar with Play 1.x, you’ll notice that the templating is a little different.

Now, when we start the app, we see the following.

Clicking the Send button generates the following log line:

[info] application - I'm sending a message now.

Conclusion

So, that’s all that needs to be done to create a websocket-enabled application in Play. We’ve both sent messages from, and to, the browser with simple code on both client and server side. In a real application, we would probably pass the in and out streams to a model, but it should be straightforward to see how this is done from the example.

For further information, you can look at the examples, particularly the websocket chat app, upon which this guide is based. In a future post I’d like to go into more detail by looking at the Akka actor model implementation, which is used in the chat application sample.

This article comes with a small disclaimer – Play is still a new framework to me, so some of the above code may not demonstrate best practice (although I did attempt to follow the sample application closely). If you notice anything that could be improved, let me know and I’ll update accordingly.

A Django Developer's First Experience of Play

2012-01-31T13:20:00+00:00

Having used Django (and other python web frameworks) on and off for three years, I came across the Play framework some time ago. However, it’s only recently that I’ve been able to use it for for a project. Note that this post is written from a Django developer’s perspective, but I’m not attempting a “Django vs Play” type of comparison. Instead, the following notes how a Django developer might approach the framework and map their own experience to it.

Play looks great for a number of reasons: it’s simple to set up, has a wide range of libraries and plugins that come out of the box, and has an active community surrounding it.

Initial Project Setup

Project setup is simple, and generates the necessary files and directory structure.

play new [app]

Replaces the familiar startproject command.

A range of libraries are already packaged with Play, including Joda Time, google-gson, Log4j, test runners, and database connectors. “Convention over configuration” is often cited as one of Play’s aims.

IDE Support

Creating an Eclipse project configuration for your project happens through the eclipsify command. Pydev, on the other hand, requires some manual setup, particularly if you are using different interpreters with vitualenv. Though simple, this can become tedious. And of course, static analysis capability is likely to differ signficantly because of the different type systems, but that’s a discussion outside the scope of this article.

Although this is not directly related to IDE support, it’s also worth mentioning that both frameworks support automatic application reloading based on file changes.

Model Declaration

Play exposes its own JPA (Java Persistence API) interface. The entity manager is already configured and can be easily accessed (e.g. for transactions).

import javax.persistence.Entity;
import play.db.jpa.Model;

@Entity
public class TestEntity extends Model {

    public String title;
    public Integer count;
}

As you can see, model declaration is concise, though the OO Architect in you might be shouting about encapsulation. Models are accessed directly through model.attribute, but getters and setters are automatically generated and used for access. These can be easily overridden. As a python developer, you might already be missing descriptors and the @property annotation!

A pre-configured entity manager is provided, but can easily be adjusted. In fact, my first application used an exisiting database, so I needed to declare models but disable DDL generation. This was easily done in application.conf.

Templating

Generally, Play’s templating engine can be described as similar to Django’s.

One nice feature is the %{ }% (script) tag that allows you to write scripts (with variable assignment, etc.) Of course, you don’t want to be doing any heavy lifting in the templates, but it can be very useful, e.g.:

%{
       nameCaps= name.toUpperCase();
}%
 
 <h1>${nameCaps}</h1>

Java object extensions (methods added to an object when used in a template) are another useful feature. To take the example from the documentation, the format method applied to Java.lang.Number gives the formatted result:

 <p>Total: ${order.total.format('## ###,00')}</p>

See the play template documentation for more details on these and more.

Test Framework Integration

Both Django and Play feature excellent test framework integration. Play has integrated the selenium test framework, and also allows tests to be run from the browser, a very handy feature.

Selenium support will ship in Django 1.4, but is not currently available.

Admin Interface

A great help in Django development is the admin interface. While it requires a little more setup, the CRUD module in Play gives a simple browser-based way of managing entities.

See more in the CRUD documentation.

Conclusion

Both frameworks have many strengths, and it is encouraging to see highly active development in both. There are a few features in Play which might make a Django developer envious, such as modules for OAuth/OAuth2 authentication and websockets built in. However, the frameworks are certainly very evenly matched.

Measuring & Optimising Django Database Performance

2012-01-24T08:47:00+00:00

Database performance is a crucial factor in web application performance, and can mean the difference between a responsive web application and a slow one. Here, we summarise methods for identifying database performance issues, and how to approach fixing them.

Benchmarking Overall Performance

First, it is worth establishing whether database queries are a performance bottleneck, or whether you should be focusing your efforts on something else. There are a number of ways to do this, and two that I’ve found simple and effective are django-snippetscream and django-debug-toolbar.

Install django-snippetscream:

pip install django-snippetscream

and add the following middleware class to settings.py:

MIDDLEWARE_CLASSES = MIDDLEWARE_CLASSES + ('snippetscream.ProfileMiddleware',)

Now, you can simply append ?prof to your application URLs to profile the code run to generate the page. This gives a quick way of telling whether any particular methods are consuming disproportionate resources. Sample output is shown below.

72063 function calls (68833 primitive calls) in 0.267 seconds

Ordered by: internal time, call count

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
8084    0.034    0.000    0.034    0.000 env/lib/python2.7/posixpath.py:60(join)
8515    0.031    0.000    0.039    0.000 env/lib/python2.7/posixpath.py:130(islink)

See the snippetscream documentation for more details.

The django-debug-toolbar is another easy-to-use profiler (and has many other functions). To install:

pip install django-debug-toolbar

And add the following to your settings.py:

MIDDLEWARE_CLASSES = MIDDLEWARE_CLASSES + ('debug_toolbar.middleware.DebugToolbarMiddleware',)
INSTALLED_APPS = INSTALLED_APPS + ('debug_toolbar',)
INTERNAL_IPS = ('127.0.0.1',)

Now, when you visit your site in a browser, the SQL query view will display queries that have been executed and the time taken.

So, you can now judge for yourself whether your application is slow executing queries or if there is some other performance bottleneck. To deal with database issues, read on.

A closer look at SQL query generation in Django

Now that you have a couple of tools by which to measure performance, let’s look at some examples of optimising database performance. We’ll use the following simple models for our discussion.

class Article(models.Model):
    title = models.CharField(max_length=255)
    owner = models.ForeignKey(User)
    tags = models.ManyToManyField('Tag')
    content = models.TextField()

class Tag(models.Model):
    name = models.CharField(max_length=255)

Fetching Multiple Models at Once

For our first example, assume we want to print a list of articles with the author’s name. We could do the following:

for article in Article.object.all():
    print article.title
    print article.owner.name

How many SQL queries would this generate? Let’s use debugsqlshell:

SELECT `testapp_article`.`id`,
       `testapp_article`.`owner_id`,
       `testapp_article`.`title`,
       `testapp_article`.`content`
FROM `testapp_article`  [0.13ms]

SELECT `auth_user`.`id`,
       `auth_user`.`username`,
       `auth_user`.`first_name`,
       `auth_user`.`last_name`,
       `auth_user`.`email`,
       `auth_user`.`password`,
       `auth_user`.`is_staff`,
       `auth_user`.`is_active`,
       `auth_user`.`is_superuser`,
       `auth_user`.`last_login`,
       `auth_user`.`date_joined`
FROM `auth_user`
WHERE `auth_user`.`id` = 1  [0.84ms]

etc...

For each article, there will be a separate query to retrieve details about the user. We can avoid this by using select_related, which tells Django’s ORM to select related models in the same query:

for article in Article.objects.all().select_related():
    print article.owner.id

SELECT `testapp_article`.`id`,
       `testapp_article`.`owner_id`,
       `testapp_article`.`title`,
       `testapp_article`.`content`,
       `auth_user`.`id`,
       `auth_user`.`username`,
       `auth_user`.`first_name`,
       `auth_user`.`last_name`,
       `auth_user`.`email`,
       `auth_user`.`password`,
       `auth_user`.`is_staff`,
       `auth_user`.`is_active`,
       `auth_user`.`is_superuser`,
       `auth_user`.`last_login`,
       `auth_user`.`date_joined`
FROM `testapp_article`
INNER JOIN `auth_user` ON (`testapp_article`.`owner_id` = `auth_user`.`id`)  [0.39ms]

Here, we see that the article’s owner is retrieved using the JOIN clause. So in this example, use of select_related has halved the number of queries executed.

One thing that should be remembered about select_related is that it does not work for many-to-many fields. In our example, that means that article tags would not be fetched in a single query. For these model attributes, there is a new method included in Django 1.4 called prefetch_related. Django 1.4 is currently an alpha release, download it here.

Let’s look at how prefetch_related reduces the number of queries with our example classes. First, observe what happens if we access article tags without prefetching:

for article in Article.objects.all().select_related():
    print article.tags.all()
   ...:
SELECT `testapp_article`.`id`,
       `testapp_article`.`owner_id`,
       `testapp_article`.`content`
FROM `testapp_article`  [0.50ms]

SELECT `testapp_tag`.`id`,
       `testapp_tag`.`name`
FROM `testapp_tag`
INNER JOIN `testapp_article_tags` ON (`testapp_tag`.`id` = `testapp_article_tags`.`tag_id`)
WHERE `testapp_article_tags`.`article_id` = 1 LIMIT 21  [1.36ms]

[<Tag: Tag object>, <Tag: Tag object>]
SELECT `testapp_tag`.`id`,
       `testapp_tag`.`name`
FROM `testapp_tag`
INNER JOIN `testapp_article_tags` ON (`testapp_tag`.`id` = `testapp_article_tags`.`tag_id`)
WHERE `testapp_article_tags`.`article_id` = 2 LIMIT 21  [0.36ms]

[<Tag: Tag object>, <Tag: Tag object>]
SELECT `testapp_tag`.`id`,
       `testapp_tag`.`name`
FROM `testapp_tag`
INNER JOIN `testapp_article_tags` ON (`testapp_tag`.`id` = `testapp_article_tags`.`tag_id`)
WHERE `testapp_article_tags`.`article_id` = 3 LIMIT 21  [0.36ms]

Separate queries are generated to retrieve the tags. And now with tag prefetching:

for article in Article.objects.all().prefetch_related('tags'):
    print article.tags.all()
   ...:
SELECT `testapp_article`.`id`,
       `testapp_article`.`owner_id`,
       `testapp_article`.`content`
FROM `testapp_article`  [0.21ms]

SELECT (`testapp_article_tags`.`article_id`) AS `_prefetch_related_val`,
       `testapp_tag`.`id`,
       `testapp_tag`.`name`
FROM `testapp_tag`
INNER JOIN `testapp_article_tags` ON (`testapp_tag`.`id` = `testapp_article_tags`.`tag_id`)
WHERE `testapp_article_tags`.`article_id` IN (1,
                                              2,
                                              3)  [121.34ms]

As we can see, all tags are fetched in a singlee query using an IN statement. So, select_related and prefetch_related are two effective ways of reducing the number of queries needed for accessing models.

Other techniques for reducing generated queries

If you find that your models are too complex to benefit from the above methods, you can always start writing your own SQL. Of course, this comes with the usual disclaimers: your code may be less portable between databases, and may be harder to maintain.

One method of doing this is to use the extra() method on a queryset like so:

for article in Article.objects.all().extra('raw SQL here')

This can be useful if you need, for example, a nested SELECT clause. The other option is to use raw queries, which are documented here.

One last point worth mentioning is to be careful with the use of iterator(), which loads only a subset of a query set into memory. If you then iterate over this queryset, this can generate lots of queries. Here’s a longer discussion.

Conclusion

Hopefully this post has provided a useful discussion of query generation in Django and how you can optimise database access in your application.

I’ll close with a couple of useful resources: some useful tips on writing performant models, and the official django optimization page.

Remote Debugging Django Applications with Pydevd

2012-01-16T13:32:00+00:00

Important disclaimer: enabling remote debugging of applications is a potential security risk. The following method should only be used in a development environment, and on a network you control. Use at your own risk.

Most django application issues can be successfully debugged using a local runserver instance on your development machine. But there are times when it is advantageous to be able to debug your application in an environment that more closely resembles production.

For instance, I recently encountered a caching issue that was caused by an incorrect Cache-Control header. The fault was at the application level, and it was useful to be able to trace the request/response behaviour in the full context of the web stack.

This post discusses debugging django, but the same technique can be used to debug any mod_wsgi application.

Environments and Remote Debugging

On my development machine, a set of VMs replicates the web stack. This includes load balancer, web cache, application server, and database machine. runserver is used to test the application locally, then I use fabric to push code to the local VMs.

To debug code when it is running on the server, you can use Aptana’s Pydev plugin for Eclipse. This allows you to run a debug session on your local machine which listens for connections. With the correct setup on your application server, you can step through your application as if it were running locally.

Setup

First, the pydev debugger needs to be accessible to your application. You could use a shared filesystem for this, or you could clone the github repository (though in theory this might introduce compatibility issues). Whichever method you choose, the pydevd module needs to be added to $PYTHONPATH, which is done in your wsgi file. See the django documentation on deployment with wsgi if you haven’t yet created this.

import sys
# append location of pydev module to $PYTHONPATH
sys.path.append('srv/site/plugins/org.python.pydev.debug_2.2.4.2011110216/pysrc/')

After that, invoking the pydevd debugger is simple:

import pydevd
pydevd.settrace(IP_ADDRESS)

Where IP_ADDRESS is the IP address of machine where you are running Eclipse. You can omit this if you are running both Eclipse and Apache on the same host.

Ensuring Eclipse has Access to Application Code

The last thing to do is to ensure that the debugger can display the source code it is stepping through. This is done by modifying pydevd_file_utils.py on your server like so:

PATHS_FROM_ECLIPSE_TO_PYTHON = [(r'/home/matt/site', r'/srv/site')]

The first element of the tuple is the path to the source code on the host where Eclipse is running. The second element gives the location on the application server. If you don’t do this, Eclipse will prompt you for the location of the source files when the debug session starts.

Once your Eclipse debug session is listening for connections, make sure you have reloaded your application:

service apache2 reload

Now, just visit the site in your browser. You should see the pydev debugger receive the connection and spring into action.

Conclusion

Remote debugging is relatively simple to set up, and can be extremely useful for debugging certain issues.

To organise the code changes shown above, it might be helpful to create a remote-debug branch containing the necessary changes to your wsgi file. Since you won’t be developing on this branch, merging master periodically should incur minimal overhead.

There are a host of other debugging methods for mod_wsgi, including interactive command-line debugger and a browser-based shell. These are described in the mod_wsgi documentation. There’s also a good discussion of how you can set up single and multi-threaded debugging on the mod_wsgi Google Group.

You may also want to view the pydev documentation on remote debugging, which was used in writing this article.

Python: Injecting Mock Objects for Powerful Testing

2012-01-08T11:10:00+00:00

The code discussed in this post can be found in the github repository.

While writing a module to handle Google ClientLogin recently, I wanted to test error handling by simulating error responses from the server. A simple but powerful way of doing this is to use the patching ability of the mock module.

The patching ability allows you to replace objects in scope with mocks so that different side effects or return values can be defined. Note that ‘object’ is used here in the pythonic sense – referring to entities such as modules and functions as well as class instances.

This is best illustrated by a real example, so in this post we’re going to mock the requests module and generate the exceptions described in the documentation when a request is made.

Our example module sends credentials to Google’s ClientLogin service in order to receive an authentication token, required for accessing certain Google services (such as C2DM). If you are interested, you can read more about ClientLogin on the Google Code page.

So, to request an authentication token from the ClientLogin service, you POST a set of parameters including email and password to the service endpoint. Here is a cut-down version of the code that initiates the authentication request:

class Authenticator(object):

    @staticmethod
    def generate_token(email, password, source,
                       account_type='GOOGLE', service='ac2dm'):

        """Returns an auth token for the supplied service."""

        base_url = 'https://www.google.com/accounts/ClientLogin'

        headers = {
            'content-type': 'application/x-www-form-urlencoded',
        }

        payload = {
            'Email': email,
            'Passwd': password,
            'source': source,
            'accountType': account_type,
            'service': service,
        }

        try:
            response = requests.post(base_url, data=payload, headers=headers)
        except(RequestException):
            raise

If the requests.post method throws an exception, we simply raise it to the caller rather than handling it immediately. In the requests module, RequestException is the base class from which others (like ConnectionError) inherit. We could improve our approach to exception handling but it is sufficient for this example.

These exceptions will be thrown from our mocked class, which is patched into the above code using a context manager:

import unittest2 as unittest
from requests.exceptions import RequestException, ConnectionError
from mock import patch
from clientlogin import Authenticator

class AuthenticationTests(unittest.TestCase):

    def test_connection_error(self):
        with patch.object(requests, 'post') as mock_method:
            mock_method.side_effect = ConnectionError
            Authenticator.generate_token('mail@example.com','password','tag')

Here, we have patched the post method in the requests module to throw a ConnectionError. You can think of this like code injection, where with acts as the closure.

In the real test method, we assert the exception was raised with another context manager:

def test_connection_error(self):
    with patch.object(requests, 'post') as mock_method:
        with self.assertRaises(RequestException):
            mock_method.side_effect = ConnectionError
            Authenticator.generate_token('mail@example.com','password','tag')

Here, we assert that the ConnectionError exception is raised to the caller, but we could easily have asserted a different condition. We could, for instance, have verified some exception handling logic.

As we’ve seen, mocking objects and methods in this manner is a simple but powerful way of running your code under different simulated conditions, allowing thorough testing of error-handling logic.

You can see the full module including tests and usage instructions at the github repository. For more information on the mock module, the full documentation is available.

A Short Explanation of Hypermedia Controls in RESTful services

2012-01-01T16:07:00+00:00

This article roughly follows on from my previous post on REST, which discusses some points relating to the first two levels of the Richardson Maturity Model. Since we’re only going to discuss the top level of the model in this post (shown below), the other levels are greyed out.

Introducing Hypermedia Controls

Hypermedia as the Engine of Application State (HATEOAS) is an element of the REST architectural style, which describes how a consumer can control a given web application using hypermedia controls.

Let’s cut straight to the chase: what do we mean by hypermedia controls?

When I say hypertext, I mean the simultaneous presentation of information and controls such that the information becomes the affordance through which the user (or automaton) obtains choices and selects actions… Hypertext does not need to be HTML on a browser. Machines can follow links when they understand the data format and relationship types. –Roy Fielding

This explanation, from Roy Fielding’s widely cited blog post REST APIs must be Hypertext driven mentions the “simultaneous presentation of information and controls”. To simplify, we can imagine a hyperlink that our consumer is able to understand and manipulate.

So that’s the first word of HATEOAS. What does “Engine of Application State” mean? Quite literally, it means that the hyperlinks mentioned above can be used to control (drive) the web application. If you haven’t already drawn a state machine for your application, now’s the time!

Take, for instance, a RESTful webservice that allows virtual machines to be controlled. The Sun Cloud API is one example of such a service – the examples below are based loosely on the format it uses. If a given virtual machine is stopped, it be may started. Or, if a machine is already running, it may be stopped.

Example request for a machine representation:

GET /machines/1/
Host: example.com
Accept: application/xml

Example response:

HTTP/1.1 200 OK
Content-Type: application/xml
<status>stopped</status>
<link rel="start" method="post" href="machines/2?op=start" />

Note the link tag. This is a hypermedia control which tells the consumer how it could start the VM by POSTing to machines/{id}?op=start. (For simplicity I’ve omitted necessary details such as Content-Length and xml schema declaration.)

If the requested VM is running, the response would be:

HTTP/1.1 200 OK
Content-Type: application/xml
<status>running</status>
<link rel="stop" method="post" href="machines/2?op=stop" />

When our VM is running, the resource at machines/1/ contains a rel link to stop it. Both methods would probably return 201 Accepted in the case of success, as it is unlikely they would run synchronously.

In a nutshell, this is the concept of HATEOAS – these hypermedia resources allow us to control application state.

Note that the operation type in the query parameter could also have been a separate resource (such as machines/1/stop). There are arguments for either approach but it isn’t necessary to delve into this to understand the concept of HATEOAS.

Why Hypermedia Controls are Important

In the example above, the representation of a resource also gives us a way to manipulate the resource. In this way, HATEOAS allows certain aspects of the application to be self-describing, fostering loose coupling between service and consumer. If we consider that a service may evolve over time, then this loose coupling could allow the client to evolve with fewer issues than if all controls were hard-coded.

A useful resource that covers this in more detail is Wayne Lee’s presentation Why HATEOAS. Craig McClanahan also discusses the benefits of HATEOAS for service discovery and integration.

This post has necessarily glossed over many details of how HATEOAS is implemented, but has hopefully served as a useful introduction to the notion. I recommend REST in Practice as a resource for going into the subject in more detail. Additionally, the RESTful Services Cookbook contains details on implementation.

Further sources:

Some Thoughts on REST

2011-12-13T12:41:00+00:00

This article is in two parts. It started with the wish to talk mostly about hypermedia controls, but I decided it would be better to do a preceding post covering some more foundational notes on building a RESTful service.

Recently there’s been plenty of discussion about service platforms – Steve Yegge’s accidental post on Google Plus springs most prominently to mind. This article is partly motivated by these discussions, but it’s also a product of my own experience designing, implementing, and consuming services. I’ll include resources I’ve found useful when working on RESTful services.

I’m going to assume basic familiarity with REST, so you may find this introduction helpful if you need a place to start.

Using the Richardson Maturity Model as a Guide

The Richardson Maturity Model is a layered approach to REST. Aside from sounding like a psychological evaluation, it’s a helpful way of envisioning the different elements that will comprise your service, and how they are composed.

In essence, the model can be described as follows:

The first level concerns implementing URIs for each resource (like /pizzas/1).
The second level details how to interact with resources through HTTP methods (GET, POST, PUT, DELETE…). You can think of this like CRUD operations for resources, though we’ll discuss later how that doesn’t quite suffice.
Finally, the slightly mysterious “hypermedia controls” level. I’ll cover this in a second article, which is why it is greyed out in the diagram.

I highly recommend you read Martin Fowler’s full article on the Richardson Maturity Model if you haven’t already. You’ll notice I’ve left out quite a bit of detail for simplicity (including the entirety of the lowest level!)

Exploring REST and CRUD Further

It’s worth looking at HTTP verbs (level 2 in our diagram), and how they map to operations on your resources, in more detail. A common approach is to map the following HTTP verbs like so:

GET    # Read a resource
POST   # Create a new resource
PUT    # Update a resource
DELETE # You guessed it...

If I want to retrieve a pizza representation, I just GET /pizzas/{id}. If I want to create a new pizza, I POST my representation to /pizzas. Change the topping on a pizza? PUT the updated representation to /pizzas/{id}.

Using HTTP verbs in this way is a good approach for many applications. But it can, in some cases, lead to lack of good separation between model and controller. Take an example: composite representations. The representation of a news story may include headlines of related stories, user comments, and author metadata. Combining data in this manner is common for e.g. mobile clients where the trade off between composing representations (and increasing response size) and the increased latency of multiple requests is deemed necessary.

If the service was designed as a CRUD layer over some database models with no additional composition ability, then implementing these representations can lead to unexpected complexity. It pays to think about resources that don’t map directly to business objects and the way in which a controller will handle this. See Subbu Allamaraju’s excellent cookbook recipes (pp. 30-37)

This is something that is worth bearing in mind when choosing a framework on which to build your RESTful service. In a number of frameworks, the tendency is to derive controllers directly from models. How would you express the above representation in such a framework? Would it save time to use something with greater flexibility?

A final point, which leads on from the notion of generating representations directly from models, is versioning. If your representations are strongly coupled to business objects, changing your database schema implies change to your service. Templating representations helps dissociate the representation from the model, forcing potentially client-breaking to be acknowledged by explicitly modifying a template.

There are a couple of different ways to ask a webservice for a specific version of a resource. Some architects prefer the version to be passed as a header, but there is also a camp that supports having the version in the URL. Ian Robinson’s presentation (slide 20) covers this well.

Conclusion

We’ve covered the lower two levels of the Richardson Maturity Model with some expansion on mapping resources to business objects. There are many other subjects to cover – caching, error-handling patterns, content negotiation, but hopefully the above has provided some useful points to consider when designing a RESTful service and choosing a framework.

In my own work, I have found REST in Practice by Jim Webber, Savas Parastatidis and Ian Robinson a good resource.

In the next article I’ll talk about the final tier of the Richardson Maturity Model: hypermedia controls.

Sales Driven Development

2011-12-06T08:57:00+00:00

Customers don’t know what they want. Stop expecting customers to know what they want. –Joel Spolsky

A couple of months ago I attended a presentation by the Head of EMEA Sales at Huddle, Neil Ryland. In describing the sales process, one of the most interesting points raised was how feedback from the sales force is incorporated into the development lifecycle.

Huddle’s products are aimed at the enterprise segment. As part of the sales process, there are likely to be multiple conversations between the sales representative and the customer. This direct contact with the client is so valuable that it can be used in planning the next iteration of the product. The process goes something like this:

Analyze the failed sales. Could the sale have been closed by offering a slightly different feature set?
Analyze the successful sales. Which features of the product were most influential in the customer’s decision?
Highlight the most compelling points to the product manager and development team at planning meetings.

Note that this is my interpretation of what was stated – I’m not acquainted with Huddle’s internal processes, so don’t assume this accurately reflects the process. Yet the important point is this: sales has a direct line to product development.

Clearly, this only one of the different factors that should be considered when a backlog is written or prioritised. But it is an important one that should not be overlooked.

Out of this process grows the phrase Sales Driven Development. I’m fairly reluctant to use it, since it implies a complete development methodology. It’s not necessarily a driver of product iteration, but behaves more akin to a regulation mechanism.

An interesting facet to this approach is how it interacts with site-specific customisation, practiced by many software firms targeting the enterprise. Of course, with a cloud-based product such as Huddle’s, site-specific customisation makes little sense. Instead of customisation, a feature may be built into the next release. And, of course, these two practices certainly aren’t mutually exclusive approaches.

Returning to the introductory quotes, what we’ve covered so far is certainly no counterpoint. I strongly agree with the notion that it is not the customer’s job to know what they want. But it is interesting to explore how an iteration cycle can incorporate what customers say they want, particularly when they have already encountered the product.

In this sense, software companies with a sales force may well be at an advantage. Why? Because of the dialogue a sales representative can engage in with a lead. In terms of customer interaction, it clearly outstrips a feedback form. But this is a fairly obvious point.

A more interesting question is this: how do we derive frank feedback, and to whom should we listen?

One method is to focus on the sales successes. With many people in an enterprise using your product, their experience of the product is likely to be wider than an individual might achieve. There may be even internal factors that drive how the product is used. Understanding the initial success and following up on how the product is used internally can provide this.

Another approach is to look at the unsuccessful cases that included a trial of the product. This provides the grounding that the potential customer was willing to invest time to assess the product and discover why it wasn’t right for them.

It’s interesting to think how this approach might factor into a different revenue model, such as freemium. But since this is essentially a separate discussion, I will attempt to cover it in a future article. In any case, we’ve covered the main point of this article: examining a case where sales feedback is given a direct line to product iteration.