This is the sort of thing I muse about all the time: I keep a close eye on the web development market to ensure Eden is still in a position to add real value to potential customers. The results are important for deciding what we want Eden to look like in the future.

Ultimately, I want any decision to use Eden to be the best decision for a customer’s business. If it’s not, I prefer to point people elsewhere. So how do some of the indicators Rob mentions stack up against our business model? Are we still the best decision a customer can make?

Offshoring still a way off

The rise in offshoring development doesn’t particularly worry me. I’ve worked with offshore agencies before and there are some excellent companies out there, but the distance and the communication issues do present barriers to the addition of value. The price is lower, but the value added is commensurate.

If the spec is very (very) clear then I think that an offshoring project could work: but the risks are high and difficult to mitigate. I would hate to tie down any of our customers to a particular spec, which is why we always work in an agile a fashion as possible. It’s easier to manage risk with an agile process and to do agile well you need very clear communication pathways.

Bringing it in-house

I have also noticed a move to using permanent staff and in-house contractors to get websites built. The pool of people with Ruby on Rails skills is growing and there’s often no need to hire a contractor at a distance.

However, hiring a big experienced team takes time and the best people are difficult to find. My experience leads me to believe companies will always look to specialist development agencies to get larger jobs started quickly, and to ensure they’re done to a certain standard. We have a number of customers to whom we’ve provided development services in this fashion, often as a kick-start to an in-house team in the process of being hired.

Shifting our focus

We have had to make changes recently: I realised a few months ago we needed to shift our focus away from smaller projects, where the scope was tighter and contractors or offshore teams could price us out of the market. We grew the team to the point where we’re no longer a small group of coders with our own small independent projects. We’re now a well functioning cohesive development agency, taking on larger projects in bigger teams.

We’re looking to diversify too: we’re full of ideas for new web applications we’d just love to build, and we’ll be making some time over the summer to kickstart some prototypes. Watch this space.

Web developers: we always have to be thinking about the market we’re in, especially one as fast moving as ours, where things seem to change by the week. What are your thoughts on the ever-moving web development market?

This has been an exciting project to work on and a lot of fun. There were some pretty big technical challenges on the way, but I’m very proud of the team: they really rose to the challenge and delivered some excellent work.

Read the full case study here for more details on what it’s all been about.

This is the goal. The method is the by-product.

So after much discourse on billing methods and the complicated relations that exist between a client and supplier, the question remains: how do we do it? The following post outlines how we try to “bill with integrity”.

We do weekly billing

Virtually all our billing is now done on a weekly basis. We charge a fixed price (yes I know, bear with me), for between one to four developers for a week.

We prefer to put either two or four developers on a project for a week. This is best for most projects, so that they can pair program on the tasks. One developer is ok for R&D, prototyping work or very small amounts of work, but not recommended as you lose all the advantages of pairing.

When asked to quote work, we give an expectation of budget. We might recommend that a client book 3-5 weeks of time for 2 developers for example. It’s easy for everyone to work out how much that will cost, which promotes transparency.

We don’t charge extra for management time. All our management time with this billing method is rolled into the weekly cost. Therefore no nasty ‘management fees’ on the bill - what you’ve agreed to is what you get billed.

We build slack time into the project. All our rates are based on the fact that developers are unlikely to be able to spent more than four days of development time per week on a project. There are bug fixes, side projects, internal work, research, learning, review meetings, sick days, etc to consider.

We don’t stop at “enough”. We only guarantee to complete three days of work at the beginning of a project. That doesn’t seem like much a week, but because we don’t stop after the amount of work we’ve guaranteed, we’re not wasting the client’s time. If we have time we keep on going, all the way through to the five days if we can. Therefore we often over-deliver on a week’s booked work. We strive to ‘gently exceed expectations’ (as detailed in The Pragmatic Programmer; which is a must read, by the way). Pivotal Tracker helps track the velocity for a particular project, so we know how much we can expect to deliver on a project on any particular week.

So how is that different to fixed price?

There are elements of a fixed price methodology in this billing strategy. We do fix the price for a week’s work, for example. However, we don’t fix all the features that we can might deliver within a week, and we’re conservative with what we guarantee. This ensures that there is an appropriate amount of contingency built in, so if a project starts to get bogged down it doesn’t affect the schedule. Our estimates are done in half-day chunks using the planning poker method, which also makes them pretty accurate.

If a client is insistent that they need a fixed price, we simply say that we’ll add a certain percentage to the quoted cost for the number of iterations we’ve recommended to reflect the fact that we’re taking on more risk in the project. We’re quite honest about our reasons for doing this. If a client finds that difficult, we’re probably not the right company for them.

Aren’t you still charging a variable rate?

Well, sort of. You could say we’re charging per week as opposed to ‘per hour’, so what’s the difference? There are a number of distinct advantages that this model has over ’straight’ variable billing.

No need to clock time, for managers or developers. Variable billing requires meticulous records of time spent. We hardly ever need to do this now, saving the hassle of timesheet keeping, approval and complex invoicing.

Weeks are much easier to schedule than hours. Before switching to this model I found myself heading round the office at least four times a day to make sure everyone had enough work to do. Even when we made this more automatic, it was still a problem - it was really hard to commit to definite deadlines and everything was too fluid.

There’s much less context-switching for developers. With fully variable billing, everyone still found they were switching around between projects several times a day, which can be very confusing and inefficient. Now there’s at most 2-3 projects running at once, and there’s much less switching around - less than once a day for most people.

The team is much more focused over a whole week. Before we found we were effectively acting as several independent freelancers, all talking to individual clients and getting on with their work in isolation. This worked ok with much smaller projects, but it doesn’t scale to multiple team members on one project. ‘Silos’ of knowledge began to build up, and getting a new team member up to speed was hard. Now we’ve moved mostly away from variable billing, we’ve been able to pair program over the last few weeks and that’s really helped to break down the silos and promote communication. We’ll be posting about our experiences with pairing in a future post - it’s been so significant it’s probably worth another series :)

Some closing thoughts

It’s been great to write this series, because it’s really forced me to think about our billing strategy hard again, striving to ensure we have integrity about our billing. You can’t exactly write a series such as this and not be walking the talk, right? My thoughts have evolved as I’ve written it; there are some things about our current billing method that would be quite different had I not written this series.

A few extra thoughts as I round this up:

Billing is hard. That should be painfully obvious from some of the previous posts. We’ve learnt an enormous amount from our successes, but even more from our mistakes.

Ultimately it’s about the relationship. A great relationship with the client will help to soften any billing issue you get into. Likewise, a bad relationship will be hard to save if things go south.

And lastly, probably the most important thing:

We’re not done with improving our billing model. We’re on a journey, same as everyone. There are disadvantages to our current billing model too, which we’re looking for solutions to. Have you found a better way of working? Let us know!

Thanks for reading.

def create
  comment = Comment.new(params[:comment])
  if comment.save
    redirect_to :back
  end
end

But beware! The trouble with this is that Firefox has an option to disable referrers, so you can’t rely upon the HTTP_REFERER being set.

So you could set up a session variable when you display your comment form, to later be used in the controller. The current url is found in request.path

<% session[:return_to] = request.path -%>

def create
  comment = Comment.new(params[:comment])
  if comment.save
    redirect_to session[:return_to]
  end
end

Sure, people can turn off session cookies, but if they do that then you have all sorts of other Rails problems to do with authenticity tokens and user authentication, so I think this is a fairly safe method.

Of course, an even better option is to add the comment by AJAX and avoid the need for a page refresh at all … but that can be the topic of a different blog post!

Rails makes it easy. You probably know the standard example. You put a before_filter :authenticate in the controllers that require it, and set it up in the Application Controller.

def authenticate
  authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass'
  end
end

It’s all well and good … until you want to add a log out button. The browser stores the successful login credentials in a sort of cookie, and applies them to every page which requests basic authentication. Once you’ve logged in, it’s actually quite hard to make the browser forget you until you quit and restart the browser. It’s hard, but not impossible. If you can force basic authentication to fail, the browser will throw away the credentials.

So the solution is to add a session variable that says “NO SRSLY, LOG ME OUT PLS!” This is the logout action (a destroy method in a Sessions Controller)

def destroy
  session[:logout_requested] = true
  flash[:notice] = "You have logged out successfully"
  redirect_to(root_path)
end

Now for the tricky bit. The way this works is subtle and takes a moment to figure out each time I think about it. We change the authenticate method in the Application Controller so that as well as checking the username and password, it also ensures that this flag has not been set. Meaning we can cause basic authentication to fail when we want it to.

def authenticate
  authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass' && session[:logout_requested] != true
  end
  session[:logout_requested] = nil
end

Next time our user goes to a page which requires authentication, the browser still provides the correct username and password, but the flag causes the basic authentication to fail. Obviously we then have to clear the flag straight away, otherwise the user would not be able to get back in again even with the correct credentials. The user must type in the correct login name and password again to be able to get back in.

Perhaps we want to know whether the user is logged in or not, so that we know whether to display an edit button. We can set another session variable. Conveniently, authenticate_or_request_with_http_basic returns a boolean value.

def authenticate
  session[:logged_in] = authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass' && session[:logout_requested] != true
  end
  session[:logout_requested] = nil
end

Remember to set the flag to false when you log out. Also remember that this flag could be true, false or nil so a check in the Application Controller looks like this:

def logged_in?
  session[:logged_in] == true
end

Finally it’s worth noting that the username and password do not have to be hard-coded like this. It’s simple for an example, but don’t think that’s all there is to basic authentication. There’s nothing to stop you comparing against values in a settings table or even doing a user lookup à la RESTful authentication.

def authenticate
  session[:logged_in] = authenticate_or_request_with_http_basic do |email, password|
    user = User.authenticate(email, password)
    if user && session[:logout_requested] != true
      self.current_user = user
      true
    else
      self.current_user = nil
      false
    end
  end
  session[:logout_requested] = nil
end

Thanks to Richard and Tris for their help in figuring out the not-so-basic aspects of basic authentication! :)

Hiring in the midst of a recession is always a bit scary, but we’re finding that we’re very well placed as a company to withstand it. I’ve never been busier writing features and following up leads, and we’re rapidly becoming well-known for our team’s excellent work (I’m so proud of them :-)

So, if you’d like to join us, now is the time.

All this... for nothing?

I’ve been digging around for a decent agile PM tool for a while, and stumbled across Pivotal Tracker on Twitter a couple of days ago.

Within about thirty minutes of trying it out, I was totally hooked. We’ve rolled our own basic project management tool internally, but we’ve yet to get the time to put proper iterative development support into the app. Pivotal Tracker just works. With a clean UI and some great reporting tools, it’s all I need to run an agile project.

The one big downside is this: it’s completely free.

Wait, you say: surely that’s a good thing? Well, yes and no.

Why great software should cost money

When I pay for an app on the net, I expect a few things:

• It’s going to be up pretty much all the time.
• It’s not going to vanish tomorrow.
• It’s not going to suddenly cost a fortune.

With Pivotal Tracker, I’ve no idea when they’re going to take the software down for maintenance. I’ve seen reports of it going down without notices for 15 minutes here and there, which is fine if it’s a non-critical service. It is however most definitely not fine when my client is trying to comment on features. I also have no guarantee that it’s not going to suddenly disappear into the night, taking all my stories and possibly the project itself with it.

The other issue is that if I’m paying $20/month for it, I’ve a reasonable expectation that it’s not going to cost say $200/month next month. It’s well within Pivotal’s rights to withdraw the free service and charge what they like for the paying version, without any notice.

Are we going to use it anyway? Yes we are: it’s too good to ignore. However, we’ve put in place the following cron job to mitigate the risk:

*/20 * * * * curl -H "X-TrackerToken: <our token>" -H
 "Content-type: application/xml" -X GET
 https://www.pivotaltracker.com/services/v2/projects/39382/stories
 > stories-backup.xml

This little piece of magic copies all our stories in XML form to our backup server every 20 minutes, so we’re not screwed if the site vanishes without notice. It would only take us about an hour to import this XML into our old system should we need to. That takes care of the main risk for me.

The conclusions: Some software shouldn’t be free. If you’re using Pivotal Tracker or any other free service in anger, make sure you’ve got a backup plan.

How do you think a client would feel if he got this bill?

Given that “fixed price fixed scope” billing is flawed, a natural next step for many small businesses is to try variable billing. This time we’ll look at some of the pros and cons of such an approach.

By “variable billing” I am referring to the practice of billing per hour (or per day) for your services. It’s a pretty common way of charging out time in the web development industry. Going back to the four variables of software development from my previous post, when billing in this way you are effectively fixing quality (or you should be!), and tying time and cost together. Therefore you are allowing scope and time/cost to vary based on the client requirements.

The advantages

On the surface, variable billing appears to be the answer to all our problems. We don’t have to worry about estimates, because the client is paying per hour, so we’ll just take as long as we need to get things done. Changes in the project are handled easily: we just drop what we’re doing and change direction, so we can give the client the features they want. Goodbye to underbid projects completed for free!

So why is the client often unhappy with this sort of arrangement? Why won’t many clients even sign up to it?

The disadvantages

Many of the advantages are to do with transferring risk away from the developer and onto the client. If the developer is freed from the risk associated with getting a feature done, then that means the client is bearing the full brunt of the uncertainty. Clients want a fixed price, so they can compare you to other suppliers. Just doing variable pricing makes the proposal process very difficult.

There’s also the lack of trust. If you’re just setting up a relationship with the client, they don’t know you from Adam: they have no idea whether whether you’re as good as your word. You might say 2 days for a feature, but if the bill comes in for 4 days, they aren’t going to be happy. It only takes one large unexpected bill to wreck a client relationship.

Another main disadvantage is more subtle. With this sort of approach, a client can sometimes micromanage the project, as they’re rightly concerned about getting value for money. Therefore it’s possible to get inundated with feedback, minor tweaks, irrelevant bug reports and requests for status updates. This burns through yet more of the developer’s time, adding to a client’s bill and dragging down project efficiency. Often clients will want a very detailed breakdown of exactly what time was spent on their project, causing more management overheard.

Conclusion: Useful in some circumstances

Variable billing works well when there’s a large amount of goodwill between client and developer. It doesn’t work so well at the beginning of a client relationship. It’s worked well for us on the two extremes of projects we take on - the very short and the very long:

* Software consultancy, where we’ve been providing short term expert advice on an ad hoc basis.
* Tiny changes, where there’s only a day or two of little tweaks to make to a site.
* Open ended long term projects, where we provide development resource over months/years.

Sometimes projects degenerate into variable billing as the trust grows, a client starts believing our estimates, and everything becomes more informal.

Interestingly, on our most successful long term collaboration using this billing method we have a regular monthly budget which we can’t exceed, but we use that up in a variable ‘per hour’ manner. This is strikingly similar to our current chosen method of billing, which I’ll discuss in the next and final post in this series.

Tune in next time for an outline of our current method (you may want to subscribe for the latest updates). If you’re in a hurry for an “answer”, you could always peek at our pricing pages if you want to know how we bill :) Thanks to Kent Beck’s excellent essay on Optional Scope Contracts (PDF) for many of the ideas in this series, and the feedback I’ve received from various people.

I particularly want to write about how we fetch Twitter user icons. The icons are stored on AWS (Amazon Web Services) and cannot be deduced from the user name. You have to go through the Twitter API to find the image location.

The first thing I tried was the Twitter4R library. This seems to be a very powerful library, and if you were writing a full Twitter client in Ruby you’d definitely want to consider it. It was simple to get the user icons, but the library just felt a little over-the-top for our needs.

Twitter4R seemed to require authentication with every request, and we soon hit the API limit. I realised this is odd because you really don’t need to authenticate to look at the XML or JSON data about a Twitter user. I decided to go back to basics and do it myself. This is the code for parsing the JSON data and picking up the user icon URL:

def icon_url_for_user(username)
  require 'open-uri'
  require 'json'
  buffer = open("http://twitter.com/users/show/#{username}.json").read
  result = JSON.parse(buffer)
  result['profile_image_url']
end

See! Easy!

Most people don’t change their user icon very often, so once we know where to find a user’s icon, we don’t need to ask Twitter for it again for an arbitrary amount of time. A week seems quite sensible. To that end, we created a lookup table in our database to match Twitter user names to their user icon URL. We added an index to the user name column because it acts as the primary key lookup.

When we want to know a user’s icon, we first look up in our table. If we don’t yet have it, or if the updated_at date is more than a week ago, we check with Twitter for the image location. Otherwise we use our cached location.

Fast and easy! :)

We had a quick 15-minute meeting where Chris outlined the concept to us and drew some sketches. Tris and I got straight to work on designing the models whilst Chris put together a visual concept. We had lots of discussion about the colour scheme before agreeing on the bright pink. James and Richard were to design the front end, Tris did the backend data manipulating and URL routing. As the person with previous experience in the Twitter API, my part was to get data in from Twitter and send responses out.

By 10:20 we knew this was going to work! Having looked up search.twitter.com for a few select phrases, we discovered that people were indeed twittering about their various addictions!

Data found on Twitter!

As James and Richard were cutting up the design and turning it into HTML and CSS, Chris made a very welcome coffee round.

At about 11:15 we had a major success for the data import, upon discovering that Twitter Search can bring back up to 100 results at a time. We thought it was limited to 15. This meant that we could go back and select much more data.

Lots of data for us to use

Obviously, not all this data is particularly interesting, and some of it is just retweets of other people’s addictions. Tris helped me to tweak the regular expression to filter out unwanted results.

Meanwhile, the front end was coming along nicely:

An early version of the front end

At midday Chris bought the domain ykyat.com. We were going to use a different domain name, but after seeing that ykyat.com was available, we all agreed that it was a much better domain. I created the Twitter account @ykyat and applied the background image and colour scheme to the Twitter page.

There was a mild panic over URL routing - we wanted to give each ykyat a unique tag, but we also wanted a nice short URL to browse by topic. It was decided to use a tilde when viewing by tag. The routing is rather non-standard for Rails; I found it interesting to see how it is done. Kudos to Tris for more regular expression magic! :)

I spent my lunch break sitting outside the office wondering how best to implement the since_id so as not to bother Twitter Search by asking for the same results more than once. I had three ideas in mind, and it was useful to have a break to consider the most efficient way.

At about 14:00 things were all coming together very nicely! Tris had organised a cron job for the import, and I was very excited to see a tweet of mine imported and displayed in the gorgeous UI that James and Richard had crafted!

A tweet imported and displayed

James and Richard turned their attention to the rating system whilst Tris set up the live server on Amazon EC2. My next task was to tweet back an @ reply to the addicted person, with the URL to let them know they had been featured on #ykyat. Thanks to the simplicity of the Twitter API, this didn’t take long …

Ykyat bot responds to the addicted person

Our Git repository was looking quite exciting at about this time!

Everything coming together

I then started to look at bringing in the Twitter user icons. All was going fine until I hit the limit of API requests per hour. We realised that we couldn’t just ask Twitter for the icon each time every page was loaded; we would have to be more pragmatic about it. We put the user icons on hold. It was getting to late afternoon by now and we also decided we didn’t have time for commenting. Although Tris had written the back end to support comments, we were aware that the Twitter authentication needed to be secure and well tested so we didn’t want to rush that bit.

The last part of the afternoon was spent getting it working on the live server and making a few tweaks and polish to the user interface. We were all very excited to see what we had achieved in a day! Chris announced at 17:42 that we were done. We went home. Half an hour later Chris announced it on Twitter:

Chris announces #ykyat on Twitter

We made a few enhancements over the weekend. Chris worked on the navigator, showing popular categories and popular tweeters. I was itching to leave comments, which was very motivating for me, knowing that I had the power to enable it! I wrote the Twitter authentication and commenting system on Sunday evening and it went live on Monday!

Building #ykyat was an exciting and rewarding experience for all of us. It was good to work together as one team when so often we each work on different projects. It was fascinating to see an idea grow into a live web app in one day. I think we all knew that we could do it, but it was great to be able to prove it to the world!