Ben Godfrey

Mixcloud is a great site for sharing mixes

Tue, 13 Oct 2009 18:39:09 +0200

UK startup Mixcloud has built a great site for sharing DJ mixes. It was fun enough that I typed out the full track-listing for my now-venerable Make It Minimal mix, all 43 tracks!

Make It Minimal by Afternoon on Mixcloud

How I work

Tue, 29 Sep 2009 15:12:34 +0200

I’m an independent software engineer working on a contract basis. I have a process for projects. It’s a relatively lightweight, loosely agile methodology. It attempts to ensure projects run smoothly by building software in a series of short iterations.

Face-to-face or Skype meeting

After an introduction is made, I organise a face-to-face or Skype meeting. I get an overview of the requirements and the context in which the application will sit, the business goals. Clients often have preconceptions about what I’ll do, how long it will take, etc. I like to learn about the project so I can challenge those preconceptions if necessary. For example, if a client asks me to create a blogging tool, I’ll make sure that none of the off-the-shelf tools will do before proceeding. I’m actually pretty vigorous in recommending alternate approaches. My experience is that a lot of stress and heartache can be avoided.

Because I practise agile software development, I encourage clients to find a minimum set of features which can be implemented an iteration lasting up to 2 weeks. This is a useful process. It encourages both me and my clients to think about the business goals first. If those goals can be met without software or with off-the-shelf tools, we can make that decision now.

Specification document

After the initial meeting and a follow-up email or 2, I will produce an iteration specification document. I usually get this to the client within a few days to a week depending on how busy I am. I can rush it through if necessary.

The specification will list the set of user stories (how the app will be used, tasks it will support) and the features required (the actual stuff I’ll build). The feature set will take no more than 2 weeks to develop, test and deploy. Additional features are left for subsequent iterations.

If the client is happy, work begins. Otherwise the spec is revised. The spec is my statement of intention, this is my understanding of what the client wants me to build. I ask my clients to check this carefully. If I’ve got something wrong, it’s time-consuming to fix later. It’s often hard to make spec docs exhaustive and unambiguous and this can lead to disagreements, I try hard to avoid that.

Developing iteratively

The work begins on the first iteration. I will develop the required features and test and deploy the software within the stated time period. During this period communication slows down a bit. Coding requires a really quite extreme level of concentration so I tend to close my email, Skype, etc and put my headphones on.

Only quite minor changes to the spec can be accommodated once an iteration has begun. If changes are frequent, it’s very hard to get up to speed. A car can go much faster on a straight road. The client’s desire to provide new feedback and ideas and the developer’s desire to get the job done are often in competition. The beauty of developing in short iterations is that this competition can be resolved. Ideas can wait a week or 2, in fact they benefit from more consideration, meanwhile the develop can code away undisturbed.

Towards the end of the iteration, the new code is deployed to somewhere where it can be tested by myself, the client and other stakeholders, often a staging site. Bugs are found and fixed. Loose ends are tied up. Finally, the software is put into production. Bug testing and fixing takes from 25-50% of the iteration time. Deployment is generally straightforward, but can still take a few hours, for that reason it’s only done once per iteration.

When an iteration is complete, the process begins again. A face-to-face or Skype meeting allows the client and I to assess what’s been achieved so far. Learning from one iteration goes into the next. A new spec is produced and I get back to work.

Clojure: a stateless dynamically-typed Lisp on the JVM

Tue, 21 Jul 2009 13:15:38 +0200

I’m a big fan of programming without state in languages like Haskell and Erlang. Clojure is a modern Lisp, but follows the stateless style very closely, taking ideas from Haskell and ML. Clojure is implemented in Java and runs on the JVM, providing full access to all of Java’s libraries.

Modern

Clojure is a lot less crufty than other Lisp implementations, some of which are now pretty long in the tooth. Maps and vectors and a few other things are given syntax. Macros still work though, so I guess it’s still all s-expressions ([1 2 3] could trivially map to (vector 1 2 3)). Clojure has a terse syntax with lots of good code smells (e.g. *constant* and predicate?).

Structural sharing

Clojure benefits from state of the art data structure research. Clojure’s key data structures (lists, vectors, maps) are implemented using a technique called structural sharing. If you add an element to the front of an existing list, the new list is simply the new item plus the existing list. The existing list can be stored once. The new item simply points to the head of the existing list. This technique preserves complexity characteristics of operations on data structures (insert, remove, access, etc), minimises memory footprint and provides immutable state! Very cool stuff.

Dynamic typing

Clojure is dynamically-typed like Python or Ruby. This makes it a good entry point from these languages. For programmers looking to program without state other options are Haskell or Erlang. Erlang is awesome for building highly available, concurrent applications. It’s syntax is kind of fun, but not for everyone. Haskell is strongly typed, pure and lazy. This unique mix of features can be very powerful, but can also trip you up. Both languages have a sweet spot, Clojure feels more general purpose. For me, that means great for scripting and writing web apps :-).

I like Django’s minimal template language. There is an implementation in Erlang, but Erlang’s string processing is somewhat weak. Templating and other string processing, like parsing JSON, is a bit long-winded. Haskell doesn’t have any good template languages, the best option is Text.Xhtml, a combinator library. Not very designer-friendly. Haskell’s powerful type system isn’t very useful for implementing a text template language. It gets in the way more than it helps. It feels like an impedance mismatch. It would be an interesting project to build a template processor compatible with Django in Clojure.

Recommended viewing

For a more complete description watch Clojure creator Rich Hickey describing Clojure and walking through a concurrent application.

Software development advice for startups

Tue, 19 May 2009 20:59:59 +0200

You’ve got a great idea! You want to build a web site that saves people time and money and helps them make friends! You’ve written a rough business plan, now you want to find someone to build your site.

Software development is complicated, expensive, error-prone, regularly boring and complicated. As a programmer, here’s what I think you should know before we meet for coffee.

1. Avoid developing software

As a general rule, you should avoid doing anything you don’t need to do in a startup or new project. You have plenty of things to worry about. If you can possibly avoid writing software, do so. Need a website? Use WordPress. Intruders.tv and I Can Has Cheezburger are 2 great businesses built on WordPress. Need something like a social network? Use Ning. Prove that you can build the community, then transition to a bespoke platform to build features. Since 2004, an amazing number of high quality, free or low-cost tools have come on to the market. Tools don’t have to be perfect. Maybe half your product is already implemented by someone else’s API. Use that, build the other half. Worry about strategic risks later. You’re prototyping remember.

2. Software is complex

Software is complex and fragile. Even a simple piece of software is many times more complex than a car engine for example. Even simple web apps built with standard tools don’t tend to share that many common parts. Understanding an application later and modifying it is really a very complex task indeed.

It’s many times easier to change an idea or a document or a diagram than to change software. Try to make changes early and avoid making them late if possible. The agile methodology, building software bit by bit in short sprints, makes it more likely that you’ll change something before it’s been coded, tested and deployed.

3. Don’t make a big long list of features stretching until the end of time and space

Decide what the very core of your offering is, find out if your customers are interested and build the minimum that can solve their problem.

This approach costs less and is less risky than building your perfect dream. When you have the minimum viable product built, solicit feedback from your customers. By making the big list, you’re second guessing what they want. You’re probably wrong.

Ideas happen much more quickly than the code. An idea might take a day to think through and a week or a month to implement. If you make a list of features, those features will be irrelevant by the time the programmers get around to building them. Stay in the moment. Keep the list to what your customers are asking for today.

4. Let your developers (and designers) do their job

Cheaper development teams are great when you have a known problem and a known solution. When you’re launching a new product, you have only one or neither. Communication is critical, but it’s also important to respect the software development process. Remember that writing software is complex, boring and error-prone. This means most of the time, developers have to get their heads down and work, work, work. I find 2 weeks a good length for a block of work because it gives a good mix of regular points for communication, keeping the product connected to reality, and time to get things done.

Be flexible enough to create an environment where your team can get on with delivering a great product, and you can get on with everything else.

Bonus point: If you engage a designer, let them design. You have to be proud of the way your application or site looks, but you are not the customer, the customer is. Experienced designers are good at making things that work well and look good for the larger audience. Try not to make personal judgement, like “I don’t like that grey.” Your customers probably won’t care as long as the site or service looks professional and is easy to use. If you have A/B test data to show that the grey has lower conversion rates, that’s different (Google actually tests different shades of blue).

5. Don’t micromanage

This is kind of the same as the previous point, but it’s vitally important for software. If you’re launching a product and you start involving yourself in the minutiae of development, you will slow the process down hugely. Development takes focus, handling a micromanaging client takes time and destroys that focus, leading to mistakes and delays.

Let small decisions go. Revisit them if they really are wrong. You don’t know if your product will be a hit yet. You need to get something out there as quickly and as cheaply as possible. If it later turns out you need to change the core product (which you almost certainly will), all the time spent fine tuning before launch is time wasted.

6. Software is only part of the puzzle

Finding money, figuring out a product and having it built is not easy, but it is well understood and, given a sensible spec, can be acheived in a finite amount of time.

Most new products don’t fail because the implementation was bad, they fail because the customers never came. Make sure the customer is there before you start. Make sure you know where they will come from, why they will use your product. Make sure you know this inside out. It’s very possible that your idea, though neat, isn’t a workable business. You really want to learn that before investing time and effort in building a product. The New Business Road Test is an excellent book on evaluating opportunities.

7. Rewriting is common

Once a piece of software has been around the block once or twice, the task it’s being used for ends up pretty far from the one is was intended for. Maintaining it becomes increasingly difficult. At this point, it’s time to consider a rewrite. This is a good thing. New tools and practices will have emerged that you can take advantage of. You know more about the business you’re in and you can cut out the features you don’t need and concentrate on making the ones you do much better. Rewriting is good. Plan to rewrite. Get stuff done simply and quickly at first, revisit it later when you know more.

This is a controversial point. Trying to change software is like trying to change a car into a boat, it can be done with time and effort, but building a boat from scratch will be easier and the final boat will keep water out better, go faster, look nicer and generally be more fit for purpose. Data from NASA cited in Facts and Fallacies of Software Engineering suggest that if as little as 30% of an application needs to be changed, it is less time-consuming and expensive to start over.

And finally…

There are many great books about software engineering, but I can personally recommend none more than Robert L Glass’s Facts and Fallacies of Software Engineering. It is a mine of empirical data about what makes software projects succeed and fail. If your company’s business is making software (which it is if you’re a web business), you need to read this book.

There are also a number of great websites about startups, few are finer than Eric Ries’ Startup Lessons Learned. Read it all. Twice. Carefully.

The New Business Road Test is a great tool for stopping yourself from diving in to an ill-advised venture. Reading it may save you a lot of time and money and prevent hair-loss.

Django, Drupal, Webmachine: Different frameworks for different projects

Wed, 13 May 2009 13:26:02 +0200

Django is an awesome framework but different projects have different needs. The last 2 projects I’ve been involved with have been using Drupal. Other projects I’m planning call for very RESTful designs. Webmachine, an Erlang framework, is a great fit for these.

I do still very much love Python and Django, perhaps even more that I’m using PHP day to day. I miss the REPL. I miss first class functions. I miss Django’s very tidy organisation of code.

Drupal is something I’ve studiously avoided for a long time, thinking it to be a Zope-like mire. That’s true to an extent: there are many versions and a lot of code. Drupal apps do have good separation of concerns. The internal organisation of modules and themes is useful, although there’s a little bit too much function name magic going on (“Why isn’t my validator firing? Who knows!”). I’m interested in hooks, although I haven’t needed them yet. The same concept has served Django hackers well.

Leaky abstractions

Django and Drupal are both leaky abstractions. It’s easy to create great big joins with Django’s ORM. Drupal generates a mammoth set of CSS and JS imports per page. Both of these can be addressed with programmer discipline, but sometimes it’s nice to have a thinner level of abstraction to make you think carefully about each requirement and how best to implement it. Webmachine is such an abstraction. What it does provide, however, is system management built on top of Erlang and OTP. People doing scale (a group I’m not a member of) find that writing the initial app is easy, scaling it is hard.

Easy hacking, easy scaling

Newer frameworks seek to make initial implementation easy and scaling easy too. Webmachine is undoubtedly somewhat harder to code for than Django, there are less batteries included, but, in theory at least, scaling is easier. Only a little of that is due to implementation, most is due to architectural style. Webmachine does get in your way a bit less when hacking RESTfully. Django does things like setting cookies on every hit, making caching harder, increasing load on your app.

I think the current leader for easy hacking, easy scaling is Google App Engine. The core of Django runs happily, the system scales to Google’s infrastructure, deployment is very simple. GAE has one flaw though, porting to another platform involves work modifying code and extracting data. While the code scales effortlessly, scaling a business around that code seems harder. If your goals don’t chime with Google’s, you’re stuck. Frameworks running on open source software stacks are more trustable and it’s for this reason that EC2 is so much more popular than GAE to date.

Python syntax highlighting with Chili

Fri, 24 Apr 2009 15:54:11 +0200

I’ve enabled syntax highlighting on this site using the very tidy Chili.

The standard distribution of Chili doesn’t include a Python language definition (or Erlang, or Haskell…), so I wrote one.

Download Python recipe for Python

Use SSH public key authentication with Fabric

Fri, 24 Apr 2009 14:13:12 +0200

Fabric is a very useful Python tool for scripting administration of remote servers. Like Capistrano it allows you to define tasks as a mixture of local and remote operations and then run them for lots of hosts, different groups of hosts, etc.

Increasingly I’m using configuring sshd to allow public key authentication only. Using this method makes your server more secure against increasingly common SSH brute force attacks. You can also configure an ssh-agent app to allow password-less logins.

If you want your Fabric tasks to access machines using public key authentication, add something like to your Fabfile:

from paramiko import RSAKey

config.fab_user = "jhacker"
config.fab_pkey = RSAKey.from_private_key_file("/path/to/keyfile")

Simple, and very useful.

Working from Bali, working from London

Wed, 18 Mar 2009 19:20:17 +0200

I’m flying long haul from Bali to London. I’m returning from 3 months spent in Bali, during which I worked on and off, sometimes even for money. Long-distance telecommuting is an interesting experience. Being away from London has made me think about living and working there in a new light.

Telecommuting from a relatively well-developed place like South Bali is very easy. Free wifi is readily available and backed by “fast enough” internet connections (up to 1Mbps at time of writing). Enough for a Skype call, sometimes even video. Coffee and food are cheap enough that you can pick a café and make it your full-time office. (My thanks go to the proprietors and staff of Grocer and Grind, Sticky Fingers, Bali Buddha and Café Lazumba and Café Local.)

Contract software development lends itself well to telework. It requires some kind of rigorous technical specification to be agreed by contractor and client. The work itself involves periods of focus. This means communication is generally quite minimal. It can be concentrated into daily bursts for example, and is often better than way.

I found it difficult to work around the time difference. My UK clients would start work as my day was finishing, so the end of the day should have been set aside for communication. In reality I was often busy with other things by then, my girlfriend finished work at 5 and I got into Pilates and other pleasant evening activities. I ended up having email conversations with a 24-hour reply frequency.

I also found it difficult to sell clients my services from Bali. Work I agreed before I left went well, but people were reluctant to commit to new work without face to face meetings, though they weren’t explicit about this. After the window of remaining time shrank to less than a month, conversations degenerated to “email me when you get back” frustratingly quickly. I have a list of 14 people to email this week.

I’m looking forward to coming back to London. I’ve missed my friends, my family, the tech start-up community (and my bikes). I’m planning a busy period to make up for all the slacking I did in Bali. Amusingly enough, much of the “slacking” involved hacking away on personal stuff or projects for charity. It was great to get the mental space for things like Followize, the redesign of this site, learning more Haskell and doing some serious reading.

My busy period kicks off with a return to the tech start-up scene — attending a bunch of events to catch up with people and activity — and with lots of meetings with people I haven’t seen for a while. When I left in December, startups in London was perceived to be coping with the economicrisis better than those in some other cities. I’m intrigued to see how views have changed in the first quarter of this year, whether people are starting to feel the pinch more accutely. Many of the companies I know well are pre-revenue. Others (e.g. Huddle) offer concrete cost-savings to their customers and thus may even benefit from the downturn.

Since becoming a freelancer again I’ve been considering what I want my future role to be. I have a fair amount of web tech experience and quite a bit of startup experience (all learning from my mistakes). I like to think that my role could be as a consultant technologist attractive to some of the great companies building their products. I wonder how long I want to stick at a non-scalable profession. I took on the challenge of MyMart to chase that big pay-off that comes to the companies that win. We failed sadly, but I’m interested in buying another lottery ticket.

Client-side MVC is maturing

Wed, 18 Mar 2009 01:52:27 +0200

Rich Internet Applications (RIAs) are still increasing in popularity. We’ve reached the stage where they aren’t called RIAs any more, they’re just cool websites. (Ever heard someone call Facebook an RIA?) There is a increasing demand for tools to build great UIs easily and to build JavaScript-only applications for platforms such as OpenSocial. There are a range of toolkits for building RIAs but most of them are tied to a specific server platform.

A framework takes care of repetitive tasks. In an RIA, these include building screens, including providing rich controls, validation etc. The framework should also provide an abstraction for maintaining state and requesting data from web services via XHR. While building an OpenSocial application in December, I could have saved a lot of time developing on top of such a framework.

MVC is the dominant model for UI development in the desktop world. A modified form is very popular in web development. It makes a lot of sense to stick to MVC for RIA development. RIAs running in the browser (or Flash or AIR) bear close resemblance to desktop applications.

Of the server-and-client toolkits, Google Web Toolkit is perhaps the best known. You write Java code and it compiles it to a web application complete with HTML, JavaScript and CSS. This is an interesting idea, but the server-side component is not interchangeable and requires Java. It also leaves virtually no visibility in the compiled code.

Wouldn’t it be great if there were a client-side only JavaScript framework which allowed you to build nice UIs quickly and easily and load data from any server back-end? Given the variety of technologies for server-side web app code, you think this would be an active area. Doesn’t it slightly suck that you have an interesting web toolkit in GWT and a cool cloud hosting platform in App Engine, both from the same company, but you can’t use them together?

I was pleased to see that the cunningly-named JavaScriptMVC framework got it’s version 1.5 release. This version adds some new development tools and documentation, but it’s still back-end independent. On the other hand it does look a little heavyweight.

Topper Bowers just open-sourced his Mamoo framework. Although Mamoo doesn’t seem to be a run-of-the-mill RIA platform, it does provide some of the MVC goodness that those applications need.

I look forward to checking out both of these projects in more depth. Certainly JavaScript-only MVC frameworks are needed if smaller development operations are to produce applications with the level of quality and UI finesse seen in, for example, Facebook or GMail.

Announcing aftnn.org version 2.0

Fri, 06 Mar 2009 12:16:50 +0200

Version 2.0 of aftnn.org is here!

The new site is powered by the Byteflow with a few some custom bits. Byteflow runs on top of Django.

Some of the features I’m happiest about:

My FriendFeed collects all the stuff I’ve been doing on the web and I re-use that feed here.
The vast majority of old urls still work. My nginx config contains a rewrite line for each blog post. Not pretty, but fast.
The Content and Photos sections are dynamically generated, making them easier to maintain.
I can finally write blog posts in Markdown, not raw HTML. Byteflow actually allows me to select a text renderer per-post, so I can write one post in Markdown, the next in MediaWiki markup.
Comments are hosted locally again, not by Intense Debate, and are protected by CAPTCHA. Alternatively, you can log in with your OpenID to comment.
nginx serves up static content very quickly and gzips everything too.

There are always some down-sides though.

Blanket is gone. It was a resource hog and I didn’t want to port it to Python. Sorry.
Byteflow and Django don’t play as nicely with caches as I would like. Byteflow doesn’t provide Last-Modified, Expires or ETag headers for it’s pages. Django sets cookies immediately, instead of when a session is actually used, so it’s page are effectively uncacheable. Also, Django does not provide Content-Length headers for responses. All this degrades the performance of my site.
The Intense Debate comment import process lost the threading. I’ll try to fix the relevant posts by hand some time.

I hope you like the new site. Please post a comment if you find a bug or have any other feedback.