:neil_middleton

Developing with an environment

neil.middleton@gmail.com (Neil Middleton) — Thu, 10 Jan 2013 00:00:00 PST

Developing with an environment

One of the best practises for developing scalable applications these days is that of seperating your configuration from your application.

To quote 12factor.net:

The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

This is all very well, but how can we carry this forward into development. Typically (and I'll use Rails as an example here), when we start a Rails process locally it'll use the environment on the machine in question. However, this isn't much fun if you're developing more than one application, or happen to have environment variables in your app that conflict with those that you want to use locally.

So, how do we tell our Rails process what environment variables we want to use? Well, the simple way is to pass them into the command in the usual way:

$ FOO=BAR ONE=TWO rails server

Running this will start the Rails process with the environment variable FOO set to BAR, and ONE set to TWO

However, this isn't overly helpful when you have a multitude of environment variables, or need to start and stop processes a lot. In these instances it would be helpful to have a file containing all the variables we need and somehow load this when we want to.

Enter Foreman

Whilst foreman is very handy for running lots of processes together from a Procfile (see here for more information on that) it is also a useful tool for running processes within a given environment.

On startup of a process, foreman will look for a .env file in the root of your project and parse that, loading the contents into the environment for the process. For instance, to duplicate the above example we could do something like this:

.env

FOO=BAR
ONE=TWO

So, now we have a process with our desired environment, and what's more, as it's in a file we can more or less forget about those variables until we need to change them.

So this is great for the processes in our Procfile, but what if we want to spin up something else not in that file on an adhoc basis, such as a rails console? Well, luckily foreman lets you do that too, including the environment as before.

For instance,

$ foreman run rails console

With this, we now have a way of running anything we want within the context of not only our application, but also our desired environment

Deploying to Heroku with Travis CI

neil.middleton@gmail.com (Neil Middleton) — Mon, 10 Dec 2012 00:00:00 PST

A while back I wrote about how you can deploy to Heroku 'continuously', i.e the practise of not having to manually deploy, but having the deployment to an environment being a automated step that occurs whenever your workflow pre-requisites are met.

The most common workflow here is that deploys occur when your test environment has run a build on a particular branch of your source repository and has passed all tests.

Back in my previous example I talked about using the CI service over at TDDium and making use of their ability to set a Git URL that TDDium pushes to when a build passes.

Enter Travis, stage left

Since I wrote that post though, the guys over at Travis CI have launched a beta of their private repos version. This version means that, for a fee, you or I can push our private code to Travis and have it be tested in exactly the same way as any other repo on Travis. What's more, as you're a paying customer you even get priority over the other builds, you don't sit in the same queue.

If you've not checked out Travis I advise you to do so. Even in it's current beta form it's a very polished service and one that we will definitely be sticking with in preference to Tddium.

Add the Heroku sauce

So, how can we get this working with Heroku for getting our deployments all automated and that? Well, it's actually not all that hard. Travis lets you define a load of steps that it should execute both before and after a deploy via the travis.yml which is how you configure Travis normally. All of these steps run in the context of your app as if you were running the commands yourself from your terminal. It's worth noting at this point that the after tasks only run if everything is tickety-boo with your test suite.

Therefore, taking this into account, you can probably start to see how this stuff is starting to take shape. We just need Travis to run the push to Heroku for you.

Whilst this might sound simple, the biggest problem we need to overcome here is that of authentication. We can't have anyone pushing to our application so we want to make sure that only Travis is able to push into our application as an authenticated user.

First of all, we want to have a user to authenticate as. You could use your own, but you need to make your API key available to Travis so I would recommend creating a user solely for deploying to this application. User accounts cost nothing, and are easy to create.

So, now we have our user we need to know our API key for using with Heroku:

$ heroku auth:token
44cd2f55f47d676b49be6d5d54365a1c5743b694

Take a note of this as you'll be needing it later on. If you're not logged into the Heroku toolbelt as you're deployer user you'll need to change this first:

$ heroku auth:logout
$ heroku auth:login

OK, so now we've got our API token and we're ready to roll

Putting it all together

So, now we just need to configure Travis to do the magic when we're done. First up, let's encrypt our API token just to add some more security:

$ gem install travis
$ travis encrypt your_github_username/your_github_repo \
  HEROKU_API_KEY=<your_heroku_key>

So, now we have our encrypted key, let's tell Travis about it by adding it to our .travis.yml:

env:
  global:
    - secure: <your_really_long_encrypted_key>

So you know, Travis takes this long key at runtime and applies the contents to the environment, hence why this works.

Doing the deploy

At this point, we have a choice as there is two ways of acheiving this next step. Which you decide to use is up to you but both have their benefits. One is the more 'standard' route of using Git to push into Heroku, the other is to upload a pre-compiled slug instead.

Pushing your application with Git

So, we've decided to use Git which means that we need to deal with SSH and some other fun. By adding the following to our .travis.yml we can acheive this:

after_script:
  - wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
  - git remote add heroku git@heroku.com:YOUR_HEROKU_APP.git
  - echo "Host heroku.com" >> ~/.ssh/config
  - echo "   StrictHostKeyChecking no" >> ~/.ssh/config
  - echo "   CheckHostIP no" >> ~/.ssh/config
  - echo "   UserKnownHostsFile=/dev/null" >> ~/.ssh/config
  - heroku keys:clear
  - yes | heroku keys:add
  - yes | git push heroku master

So, what does this do? Well, first off we're telling Travis which app we're talking about and which key we're looking to use. We're then having to tell Heroku itself about the SSH config we have locally on Travis and make sure we're OK with the SSH hosts.

We're doing this on every deploy as these things can change and it just makes things easier to debug. Lastly, we're kicking off the push into master on Heroku. It's worth making sure that you've got some sort of notifications system setup at this time so you can spot that things are working as they should. Tools such as Campfire are ideal for this sort of thing and are easily added (for Travis, and Heroku).

Pushing compiled slugs up to Heroku

This method takes care of the pre-compilation of your application on Travis rather than letting the Heroku Git system take care of it. By using this method we don't need to worry so much about SSH as we're authenticating with our API key that we set up earlier.

In order to make this possible however, we are going to use a third party library called Heroku Anvil by David Dollar, who works at Heroku. This isn't a Heroku supported library as such, but is the next best thing. What Anvil does is provide an alternate build process to the one that you get with Git, and lets you carry out a load of tasks locally as opposed to on the Heroku stack itself.

In order to get this working we need an after script like this:

after_script:
  - wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
  - heroku plugins:install https://github.com/ddollar/heroku-anvil
  - heroku build -r <YOUR_HEROKU_APP_NAME>  -b
    https://github.com/heroku/heroku-buildpack-ruby.git

Now what this code does is relatively simple. First it installs the Heroku toolbelt, installs Anvil into that as a plugin and then fires off the build process. At this point we're telling Anvil where our live application is so that the push goes to the right place.

Note we're having to declare the buildpack here. I found in testing that this was a required option. Buildpacks are Heroku's way of identifying your code for execution, so you'll need to locate the right buildpack for your project and add it to this script. All buildpacks are found in the Heroku Github account

Wrapping it up

Now, neither of these examples are exhaustive representations of what you can with these scripts. For instance, you may have multiple environments and branches in play and want to setup continuous deployment for each and every branch with a matching environment. You'll need to consider excluding CI builds carried out against Pull requests, or other Rubies for instance. Well, Travis provides a selection of environment variables that you can use to enhance the above script and make intelligent changes to things like your Heroku app name, or where your code should be pushing to:

TRAVIS_BRANCH: The name of the branch currently being built.
TRAVIS_BUILD_ID: The id of the current build that Travis uses internally.
TRAVIS_BUILD_NUMBER: The number of the current build (for example, "4").
TRAVIS_COMMIT: The commit that the current build is testing.
TRAVIS_COMMIT_RANGE: The range of commits that were included in the push or pull request.
TRAVIS_JOB_ID: The id of the current job that Travis uses internally.
TRAVIS_JOB_NUMBER: The number of the current job (for example, "4.1").
TRAVIS_PULL_REQUEST: True if the current build is for a pull request.
TRAVIS_SECURE_ENV_VARS : Whether or not secure environment vars are being used. This value is either "true" or "false".

Therefore we can now do clever things such as:

after_script:
  - gem install heroku
  - if [[ "$TRAVIS_PULL_REQUEST" == "true" ]]; then echo "This is a
      pull request. No deployment will be done."; exit 0; fi
  - if [[ "$TRAVIS_BRANCH" == "master" ]]; then git remote add heroku
      git@heroku.com:YOUR_PRODUCTION_HEROKU_APP.git; fi
  - if [[ "$TRAVIS_BRANCH" == "staging" ]]; then git remote add heroku
      git@heroku.com:YOUR_STAGING_HEROKU_APP.git; fi
  ...

and so on.

Bear in mind that you might also want to add this to this set of commands. For instance, you may want to run database migrations, or do some asset precompilation and so on. However, now you're authenticated and firing on all cylinders adding these tasks should be pretty academic a task.

So, we've got everything setup and we should have a well oiled continuous deployment machine centered around Travis CI and Heroku. What's more, now it's setup and working, we can pretty much forget about ever having to deploy manually again.

The 'Heroku' book, published by O'Reilly, is available for pre-order on Amazon now

Writing is hard, so do it.

neil.middleton@gmail.com (Neil Middleton) — Sat, 08 Dec 2012 00:00:00 PST

Over the last few months I've been working on something that I wouldn't have seen myself doing at any point in my life, and that's the task of writing a book. I'm not talking about a 'Janet & John' novel, or some sort of 'Fifty Shades of Gray' sort of affair, but a proper printed dead tree, trip over it kinda technical book.

It all started back in May/June time when I was approached by a good friend at Heroku, who had been approached in turn by O'Reilly, to produce a book about the platform and some rough ideas for what sort of book might be relevant and saleable. At the end of a fairly lengthy process it ended up that I was the author, had a contract in my hand and a schedule to try and stick to.

Shit.

Getting started

The first problem I found when starting out on this project was the getting started itself. A technical book is such a strange beast - there's no real start to a subject such as Heroku, nor any end, so how do you approach it? If I were writing some sort of grotty novel then there's a clear approach with introducing characters, developing them and then climaxing with a, er, climax - so how do I approach this with a technical book?

At this point I should probably talk about targeting. If I were writing a book about a subject such as Ruby, I would be starting in the same way as any other programming book would. A spot of "Hello World" and "Fizzbuzz", moving on in a very predictable fashion. However, these books by people who have decided 'Hey! I want to learn Ruby!'.

That's not so much the case with a book about a product, which is essentially what Heroku is. People buying this book are more likely to have used the Heroku stack before, and deployed some stuff to it, and liked it so much that they are now willing to spunk $20 on a book about it in order to learn more…

So, where to start?

Well, with this sort of book you end up with a massive list of excerpts that you think should be in. You want to talk about X, Y and Z, but with a bit of A, B and C thrown in for good measure. Only by dumping everything on paper are you able to get down to the business of making sense of it all and the writing itself. There are so may corners to a complex topic such as this that structure is the only place to start thinking about what to write and how. Only by having the skeleton of what you are writing can you move forwards - much like writing code.

Toolchains

So, what do I write with? Pen and paper? Some sort of ye olde clicky clack typewriter and a pipe, so I can discard pages like that guy at the end of the A-Team?. Or Vim, which I use each and every day to write code? Actually, I settled for IA Writer - a no-nonsense, zero fluff text editor that lets you think purely about the content at hand.

No crap, no formatting, no flashy bits - just words.

Which at this point leads me onto formatting and structure, which all books need to make any sense. In the print industry, and especially O'Reilly, books are generally written in some sort of internal toolchain supported markup. This means that a publisher can drop a load of markup into one end of a pipe and get a formatted printed book / PDF / EPub / Mobi book out the other end. This obviously saves some significant effort on one side but can introduce some pain for the author at the other end.

Therefore, you're limited to writing in a certain way. Markdown (which I prefer, and what this very article is written in) is not generally supported as it's not as feature complete as you need for most of the conventions in a full-blown book. This leaves you with a couple of common options:- Asciidoc, a sort of Markdown on steroids, and Docbook, writing in pure XML.

Obviously I went with Asciidoc.

Pushing this all into Git let's me push and publish out a new PDF to see how my text looks, and means that my editor can get down to looking at content as it arrives. Without the familiarity of Git, I'd be lost.

Changes

So, we were writing, and chapters were starting to take shape and things were going well.

Then I broke my leg. Well, someone broke it for me by knocking me off my motorcycle at 60mph, but there you go. Shit happens.

Now, with a major injury and X-Rays leaving you looking like Wolverine, it's gets kinda hard to focus on the concept of writing, and as such I had to take a three month break from writing the book.

This is where bringing in a co-author (Richard Schneeman from Heroku) is a massive boon, and frankly, a stroke of genius by whoever suggested it. You have two pairs of eyes checking over the content, two people writing the content chapter by chapter and two pairs of ears to listen to the feedback that people give you.

Having a co-author is, in my eyes, essential when writing a technical book.

But why write a book, it sounds hard?

First off, you should never write a book for the money. I've no idea how much money I might make from this work, or even how many copies I might sell, but I do know that even with a thousand copies sold the per rate hour is appalling.

For me, the reason to write is a simple one.

It's a challenge.

It's a challenge to whip up fifty thousand words about a topic that you think you know inside out, but it's more of a challenge to write a chunk of that on content that previously you never knew, nor cared that much about.

Every topic has corners that people will avoid, either because they don't need to know it, or they don't want to know it. Writing a book enforces you to not only look at these things, but also look at them enough that you're able to write well about them, and pass on your new-found knowledge. What's more, you also end up having to hunt out the bits that you haven't mentioned and make sure you've covered everything that you need to cover.

You may start a project thinking that you're proficient in any given subject, but I'll tell you now, writing a book about it will make you feel stupid in comparison with how much knowledge you'll need to cover off in order to produce a topic that people will be willing to spend real money on.

So, get writing

The act of writing itself is not hard.

Writing is easy, anyone can do it. Anyone is able to impart information to the page, but the key comes from being able to write and edit in a way that makes sense and this is where the challenge comes. You will constantly improve with practise but only by pushing yourself into areas that you feel stretched. If I were to compare my writing with that of a year ago I would say it's night and day.

Whether other people agree with me is a different matter.

So, as a developer, I will strongly recommend that writing is something that you need to do. I'm not saying go write a book, I'm saying write anything, anything at all. If you don't have a personal site, set one up, and start dumping in content that you feel you know about and others will find interesting and learn from.

Once you've done this, do it more and more.

Eventually you'll get to the point where the writing comes easy, but more importantly, you're learning as much as your readers are.

Developers need to learn constantly. If you fail to keep learning you will be left behind quicker than in any other industry. By writing you are learning, but also helping others to learn things that they don't already know.

Everyone benefits from writing, but most of all the author. So do it.

Write something - it's challenging, but rewarding.

Incidentally, if you're interested in reading the end product, you can pre-order it now on Amazon

Describing time

neil.middleton@gmail.com (Neil Middleton) — Tue, 04 Dec 2012 00:00:00 PST

One of the things that developers get all excited about when using tools such as Rails is some of the cleverer helper functions that the framework defines.

One of the ones that tickled me the most was some of the date/time functions: time_ago_in_words, distance_of_time_in_words_to_now and so on. These functions were quite literally the shit as far as I was concerned. I now had the ability to rock out a very Web 2.0-y interface that included timestamps such as '12 days ago' or 'six hours ago' without really thinking about it too much.

My interfaces became cool.

However, these days I'm older and considerably more miserable. Therefore, I don't get excited nearly as much as I used to, which is where this post now descends into a rant.

For example, today I wanted to fill out a timesheet for a few weeks back, so cracked open Pivotal Tracker* to look at the project history to see what I was working on that day. However, all the entries in this history were marked out in the 'cool' way of '18 days ago' and so on - not useful at all. I needed to know what date these things occurred on, now how long ago they were.

So, next time you're designing your application interface have a think about how people will be using it. It might be nice to not confound users with a timestamp for things that only happened a short time ago, but don't put out messages such as '246 days ago!' when a timestamp can be so much more useful.

* Note here I'm not picking on Pivotal Tracker here, it's just the first one I came across. Lots of apps do this.

Results of the Rails hosting survey 2012

neil.middleton@gmail.com (Neil Middleton) — Thu, 16 Aug 2012 00:00:00 PDT

Today, over on the Planet Argon blog the results of their 2012 Rails hosting survery were released.

Taken from a sample of 1,306 developers the survey covered items such as which tools are used for development, through to how people monitor their applications once they are in production.

There's a few interesting points which I thought were worth mentioning:

94% of developers are using Git over Subversion / Mecurial and so on.
60% of developers now say that they prefer to use PostgreSQL over MySQL ( a 43% climb! )
59% of developers are monitoring their application, mostly ( 91% ) using the excellent service over at NewRelic .
46% of developers were not being notified when they're site when down, which is verging on astonishing. There are so many simple tools out there such as Pingdom or your hosts Status site that this is a no-brainer.
Unicorn is growing big time - now bigger than Mongrel and FastCGI put together
Over a quarter of all developers prefer to host in the cloud via services such as Heroku or EngineYard . Another 21% can't be bothered with the details of application deployment and just want somewhere to stick their code.

So, the general consensus seems to be that if you're running on a stack using Git and PostgreSQL, and are making use of automation and having a library of add-ons that you can plug into your application for monitoring, then you're in the sweet spot.

Multi-provider PaaSes - what exactly do they offer?

neil.middleton@gmail.com (Neil Middleton) — Fri, 27 Jul 2012 00:00:00 PDT

Recently there has been a vast array of Platform-as-a-service providers popping up all over the place offering a vast array of feature sets, and an even wider array of pricing strategies.

One thing which has become prominent is that these new PaaS providers are using the number of different infrastructure choices that they offer as a selling point. For instance AppFog will proudly tell you that you can use them to deploy your application to HP Cloud, Rackspace, AWS US East, Ireland, Singapore, and more…. In fact, they are so proud of this fact, it's front and center on their homepage.

One provider that doesn't do this though is Heroku, effectively the pioneer of this business model. Heroku only provide sevices in the US-East region of AWS. There's no option to have it elsewhere, there's not even really any mention of this anywhere in the documentation. As far as a developer is concerned, Heroku is essentially a thing, out there somewhere, in the cloud.

So, why does Heroku not offer multiple places to deploy your application? Lots of people will argue that being able to deploy to multiple sources is a good thing and can only be good for uptime. If U-East goes down, then your app does too, right?

To start off with, generally speaking, most of the PaaS providers that let you deploy to multiple locations will only let you put your app in one particular place. I'm not aware of anyone who will let you throw up your application in more than one location / more than one platform without jumping through some hoops manually. So, given this fact, you're no better off - if your selected region goes pop, so does your application regardless of where it is.

Next, you could maybe get Heroku to potentially spread your application across multiple regions (say US-East and US-West).

Whilst this may sound nice in simple in reality, it's all too easy to forget that there is over 2,500 miles between these two locations, which means latency. This latency might only be 50ms or so, but when you're querying a dataset that happens to be based in the other region, that's going to hurt your performance significantly.

Then there's bandwidth. Whilst platforms like AWS do not charge for bandwidth within a given zone, they do charge for bandwidth cross country which is going to affect the pricing that you'd need to pay to cover costs.

So, how do you get around this problem? Well, if you're building an application that is mission-critical, and just CANNOT afford to go down at any time, even for just a few seconds, then maybe using just one provider isn't for you. Deploying your application to more than one provider on different platforms, and using one as a hot redundant instance could be a way forward (Heroku could potentially do this for you, but someone needs to bear the cost somewhere).

Alternatively, the good old fashioned do-it-yourself trick might make more sense, but even with this you need to consider deploying to more than one datacenter to mitigate localised problems such as power outages and so on, which then leads you back in the direction of a PaaS.

The key is that no two applications are the same, so the hosting can't be either. You need to think about your application, it's uptime requirements, and also how much you think you can spend to be able to achieve that.

The main thing to remember is that uptime costs. There are many ways of increasing uptime past the 'standard' that a PaaS such as Heroku might provide, but these come at a very high cost, and an even higher complexity. You will commonly have more than one application deployment to worry about, DNS failover for when things go wrong, latency issues to think about, the list goes on. 99% of people just can't justify the extra time that this work entails.

Personally, I'm happy with having my stuff only in US-East on Heroku. I know that it's going to be up far more than I could manage on my own, and that someone else is looking after it for me. What's more, I also realise that to get that additional 9 on my uptime figures will be a cost that my clients just won't enjoy bearing, and will have difficultly justifying.

Saying that though, everyone's needs are different, so your mileage may vary.

Continuous deployment with Heroku

neil.middleton@gmail.com (Neil Middleton) — Tue, 17 Apr 2012 00:00:00 PDT

One of the current trends in the agile world is that of deploy soon, deploy fast and deploy often. The idea here is that code should not be sitting still for long. Your developers should be coding a feature, and getting it out to the world as soon as possible to you're able to gain feedback on that feature and iterate as you feel the need to. There's a number of things that developers can do in order to make this process smoother, ranging from the simple things such as testing to more the more advanced such as setting up continuous integration systems and automating as much of the workflow as possible.

One of the current steps that a development team can make, that seems to generate the most debate in the office is that of continuous deployment. Continuous deployment is the practice of automating your software deployment into a particular environment, be it staging or production, and thus removing the need for someone to have to step in and carry out what should be a pretty much 100% automatable task.

Are you insane?

But what are the benefits of continous deployment? Well, for starters as a developer, you know that if the commit you are about to make is going to be live to the world in only a few short minutes that you will probably want to double check everything you have written and make sure that nothing will be broken by adding your change to the codebase. This forces you, as the developer, to sense check what they are doing and think about how features should make their way into the codebase.

Another potential benefit is that of deploying features piecemeal bit by bit as they are developed (as your coding and deploying continuously you have to develop features this way, unless you want to not commit and integrate for ages). This leads you to develop features in a more of an iterable fashion, but also allows you to get feedback on the parts you have deployed before you've finished building the rest.

Some people would argue that feature branches are the order of the day here, taking the code to one side and developing a feature whole before merging it back into master in one big chunk when complete. Whilst this is fine, you are disconnecting your feature development from everything else that is going on in your codebase and potentially introducing risk. Something you need to consider later on down the line.

So, how can we manage this with Heroku? Well, luckily there's a very simple way.

Continuous deployment with Git

In order to host your application on Heroku, Heroku require you to provide your codebase via the Git source control system. Anything that is pushed into the master branch is deployed and live, whilst all the other branches sit there minding their own business.

Now, I wouldn't advocate using Heroku as your main source store as there are much better alternatives out there such as Github, but it does demonstrate that your code can be live with a simple git push heroku master automagically fired from your workflow when appropriate.

But hang on, lets backtrack for a minute. Earlier I talked about the double checking developer committing his code into a tree that immediately goes live. Doesn't this introduce some risk? Well, yes it does. Your developer is still able to push broken code into production, and the ramifications are significantly more than just pushing it into Git only. So how to we mitigate this risk? How can we ensure that the code we are automatically deploying passes all of our carefully thought out tests without issue?

Tddium

(for information on continuous deployment with TravisCI, please see this post)

Enter the Heroku addon, Tddium stage left. Tddium is essentially the continuous integration version of Heroku's hosting platform. By simply setting up tddium, setting up your test suite and pushing the code, you have an automated way of knowing that whatever is in your chosen branch is passing tests or not, and whats more you're doing it in an environment that's detached from your local workstation, so no deep down configuration or software installation that might make your suite pass locally are around to mess things up.

Once you have this up and running you know your codebase is golden, how do we deploy? Well, there's a couple of ways. Firstly, Tddium will let you setup a git URL that will be pushed to on a successful suite pass. By entering your Heroku Git URL here Tddium will automatically run a deploy whenever tests pass, thus achieveing the goal of continuous integration with a good level of sense checking. Another way, if you're a little more adventurous is to write your own HTTP endpoint that a service like Github can hit and get Tddium to push your Git code there instead.

Smoooth

At the end of setting this up, you're now able to develop code, testing it locally, and push that code into your Git repo, knowing that there is nothing left for you to do. Your code will be on it's way to whichever environment you have configured (remember, you can start with staging only) and there's nothing more you have to do.

Whilst continuous deployment initially might sound quite scary there are numerous benefits to be had. You need to take more care with the commits you make, and you have to consider how you develop your code. You also need to consider the resilience of your deploy process and what happens should deploys not go as cleanly as desired.

But that's all good right?

Heroku / Asset Pipeline FAQ

neil.middleton@gmail.com (Neil Middleton) — Wed, 11 Apr 2012 00:00:00 PDT

From having a look around on Stackoverflow, it seems that a fair chunk of the questions on there all relate to the new Rails 3.1 Asset Pipeline, a part of Rails which has attracted a relatively large amount of comment and problems.

For starters this article won't tell you how to get the Asset pipeline working perfectly locally - there's a bunch of other documentation out there which is much better than I anything I might try to do, so if you're having problems getting the Asset pipeline working in development, take a look there first.

Additionally, Heroku also have a stack of documentation dedicated to deploying Rails 3.1 to the platform and a lot of common issues are also described there:

If these don't help, then read on.

So, what are the basic settings for deploying to Heroku?

There are numerous ways to configure the Asset pipeline for Heroku, but here's a way that I know to work just fine:

# Heroku REQUIRES this to be false
config.assets.initialize_on_precompile = false

# Include any files that you want to link to directly from your templates directly.
config.assets.precompile += ['mobile.css', 'mobile.js']

# Not required, but a good best-practice for performance.
# This setting will compress your assets as much as possible using YUI and Uglifier by default
config.assets.compress = true

# Allow fingerprinting of asset filenames - good for caching.
config.assets.digest = true

# Configure the sendfile headers for Heroku.  "X-Accel-Redirect" is also a good value for this since Heroku use Nginx.
config.action_dispatch.x_sendfile_header = nil

If you're doing anything out of the ordinary this config might need some tweaking. See the Rails guide for more information.

What do I put into source control?

Within your project you should have your raw assets (in app/assets) in source control. When you locally precompile your assets this will generate a load of new asset files, and a manifest.yml file. Do not put these into source control - Heroku does not need them.

When you deploy, Heroku will take your application and identify that there is a requirement for the asset pipeline. Heroku will then run rake assets:precompile within the context of your application and add the precompiled assets to your application slug. Once this has happened, and the other compilation steps have finished, your slug and its assets will be pushed out to the dyno grid.

If you push your own compiled assets into source control Heroku will recognise this and not try another precompile. Therefore, I generally find that letting Heroku do the compile reduces the scope for bad things to happen and saves you a step pre-deploy.

Not adding pre-compiled assets to Git is especially important if you're using something like Asset Sync.

My JavaScript / CSS isn't there

This could be caused by a couple of things. Firstly, you might be using an asset host (check your config) that doesn't resolve, or something broke during the asset precompile in deployment. To debug, manually run a

heroku run rake assets:precompile

and see if your assets appear. If so, then check your deploy output for messages about asset precompilation etc. The most common reason for failure is that your application is trying to initialise a database connection or similar. As the Heroku slug compiler isn't aware of your environment this will fail. There are a couple of ways round this. One is to fix your application so that this doesn't happen, or add the user_env_compile addon to your application which will make your application aware of its environment.

If you're still not seeing any assets, check your logs for clues on what might be going wrong.

Heroku is telling me file X isn't precompiled...

By default, Rails will not let you serve every asset as a standalone file. Therefore, if you're wanting to link to a specific JS file or similar, you need to add it to your config.assets.precompile array. See here for more information.

How can I get my assets onto a CDN such as S3 / CloudFront?

The easy answer here is asset_sync a gem available from Rumble Labs which deals with syncing your assets to S3 during the deploy process on Heroku. This simple extends the regular assets precompilation with a sync of your assets.

By using this, you can then hook up CloudFront to add a proper CDN to your S3 bucket. Either way, you'll need to tell your Rails application where the new assets location is. This is done with a simple config setting in your relevant environment:

config.action_controller.asset_host = ENV["YOUR_S3_BUCKET"]
config.action_mailer.asset_host = ENV["YOUR_S3_BUCKET"]

Note, the mailer settings are for when you are pushing out email containing images and so on. Those emails need your assets too!

Updates

This is a living article. Over time I will add further information as I see the need. In the meantime feel free to ask questions or add more information, or better still, fork this document and send me a pull request.

Creating custom image processors with Dragonfly

neil.middleton@gmail.com (Neil Middleton) — Wed, 28 Mar 2012 00:00:00 PDT

Every now and then you come across a project which does things a little different. The other day I was toying with solving one particular problem, and that is the one of heavily stylised images. In this particular case, the application I was working on needed to take a user uploaded image, and present it at a pre-defined size, but with a bottletop like shroud around it, essentially making the image round.

After some head scratching I came across the ability to add processors to Dragonfly which is the library I'd chosen to use image management. Dragonfly is a great little gem that allows you to receive uploaded images, and then present them back to the user in a variety of ways on the fly. For instance:

image_tag current_user.profile_image.thumb("75x75#").url
image_tag current_user.profile_image.thumb("100").process(:greyscale).url

and so on…

So, firstly I determined that this bottletop effect could be done by making an overlay PNG that could be applied to a resized image, and then flattened down into a new PNG. Deagonfly uses ImageMagick internally when plugged into a Rails application, so it seemed that simply creating a Dragonly processor that issued the correct command would get the job done.

Sure enough, that's all you need to do:

lib/image_processor.rb

class ImageProcessor
  def convert_to_small_bottletop(temp_object)
    tempfile = new_tempfile
    args = " -compose atop -geometry +0+0 -resize 75x73 lib/bottletop_overlay.png #{temp_object.path} #{tempfile.path}"
    full_command = "composite #{args}"
    result = `#{full_command}`
    tempfile
  end

  private

  def new_tempfile(ext=nil)
    tempfile = ext ? Tempfile.new(['dragonfly', ".#{ext}"]) : Tempfile.new('dragonfly')
    tempfile.binmode
    tempfile.close
    tempfile
  end
end

What this code does is take a given object passed in my Dragonfly, and then creates a new tempfile from it. This tempfile then has an ImageMagick composite transaction applied to it, and the file is then returned to Dragonfly to continue on with. In theory, you can apply any transformation to an image in this manner, so it makes it widely extendable.

Before this will work however, you need to tell Dragonfly about your new processor:

config/initializers/dragonfly.rb

require 'image_processor'

app = Dragonfly[:images]
app.processor.register(ImageProcessor)

Now that this is done, we can call our processor as per any other built in Dragonfly processor:

image_tag current_user.profile_image.process(:convert_to_small_bottletop).url

And there you go. Debugging these processors can be quite tricky, and Rack/Cache also does quite a good job of making your life difficult, but aside from this, you have a large amount of power at your fingertips.

Internationalisation with Rails 3

neil.middleton@gmail.com (Neil Middleton) — Fri, 23 Mar 2012 00:00:00 PDT

First of all though, what is Internationalisation (or I18n)? Well, firstly it's part of two larger topics, that and Localisation (L11n). So what's the difference? Well to quote wikipedia:

Localisation:

A localized system has been adapted or converted for use in a particular locale (other than the one it was originally developed for), including the language of the user interface (UI), input, and display, and features such as time/date display and currency.

Internationalisation:

An internationalized system is equipped for use in a range of "locales" (or by users of multiple languages), by allowing the co-existence of several languages and character sets for input, display, and UI.

From these two definitions it's easy to see that we have a variety of topics that need worrying about other than the language of the words on the page. These include:

How dates are formatted
Which direction the text is written in (whilst English is left to right, there are several right to left, and even top to bottom languages)
How numbers are formatted, including currency and so on

…and the list goes on.

Translations

So, how as a Rails developer do we tackle this? Well, firstly we're treated by the fact that Rails 3 has a very capable and mature I18n API which is pleasent to use.

For instance, in simple terms we can introduce other languages into a page, but setting a locale specific version of a particular string:

/config/locales/en.yml

en:
  hello: "Hello world"

/config/locales/es.yml

es:
  hello: "Hola, mundo"

Now, depending on the value of I18n.locale, Rails will allow us to pull the appropriate string with the simple translate helper:

I18n.locale = :en
I18n.t(:hello)
=> "Hello world"

I18n.locale = :es
I18n.t(:hello)
=> "Hola, mundo"

Awesome right?

Dates and numbers

So, what about dates, and prices etc? Well, luckily that's pretty simple too, as I18n is baked right down into Rails. For instance, all active record errors, date displays and the like are checked for translations prior to render. For instance, let's consider dates. The way that we show dates over in the UK is different to, say, the US. Where we like to do 23rd March 2012, the US likes to do March 23, 2012…

So, by default, Rails will use the US version, as that's it's spiritual home, which means we need to provide details to Rails of how to render dates over here in the UK:

/config/locales/en_GB.yml

en-GB:
  date:
    formats:
      default: ! '%d-%m-%Y'
      long: ! '%d %B, %Y'
      short: ! '%d %b'

Now, that we have this config, we'll now get the correct dates in the UK. (Note the en_GB locale, you can have more than one country which uses a language ;) ).

So, we can now localise all of our dates and numbers, but that's a fair amount of effort to do - there's hundreds… Well, luckily that's already been largely done for us via the Rails-I18n gem. This gem provides pre-baked locale files for all of the popular locales, and loads more besides. They contain translations and formats for dates, numbers, times, activerecord errors and more. Very useful stuff. Simply dropping this into your application will automatically provide you with a massive amount of localisation out of the box.

Data

So, we've now got our pages localised and we can flip between them happily serving all sorts of nationalities, but unfortunately our English data is still showing through. We've got a database full of products which are all in English being shown to a French audience - not so good to them.

Well, the solution I've found works well here is the Globalize3 gem. This gem allows you to add translations for a given ActiveRecord model in a nice simple matter. What's more you can treat them as nested attributes for simple editing and updating:

/app/controllers/products_controller.rb

def edit
    @product = Product.find_by_id(params[:id])
    Locale.all.each do |locale|
      @product.translations.build locale => locale.code
    end
end

/app/models/product.rb

has_many :translations
accepts_nested_attributes_for :translations

/app/views/products/edit.erb

<%= form_for @product do |f| %>
  <%= f.text_field :name %>
  <%= f.fields_for :translations do |t| %>
      <% if t.object.locale != :en %>
           <%= f.text_field :name %>
      <% end -%>
  <% end -%>
<% end -%>

Now, there's a few things going on here. Globalize3 is taking the translations for a model as optional, this is because it will fall back to the default locale when a translation is not available (hence the exclusion of :en in this particular example). Secondly, notice that as these are nested attributes you can stick them in the same form as the main data itself making it nice and simple to manage.

When showing data Globalize3 will respect the value of I18n.locale and render the appropriate translation for a record if available. This means that if you have translations, your front end will show them with no intervention from yourself, which is always a bonus.

So - now we have a full set of I18n complete. We can localise the dates, numbers, errors and so on that emanate from Rails, whilst also being able to provide translated data on a needs basis, and all without too much pain and suffering.

For more detailed information on the Rails I18n API see the Rails Guide. For more information on Globalize3, see the Github Page

The wonders of hStore

neil.middleton@gmail.com (Neil Middleton) — Thu, 15 Mar 2012 00:00:00 PDT

The wonders of hStore

As of yesterday, Heroku now provide support for Postgres' new fangled and very shiny hStore storage across all the Heroku Postgres 9.1 instances, which is the current default if you setup a Heroku Postgres database.

So, what the hell is hStore? Well, it's flipping awesome, that's what it is.

Let's imagine for a second or two, that we're a large electrical retailer who's having bit of an identity crisis, and that we've spent way too much money on Star Wars based advertising. We have a shed load of products that we need to store and have a PG database to put them in.

Most of the time we would reach for a products table, and then some sort of storage for the product attributes. We could do this via adding a load of attribute columns to our products table (even though we don't really know what all the attributes are), or we could create some sort of related attributes table to store stuff in and then have to muck about adding and deleting attributes to products.

…or we could just use hStore.

What is hStore?

hStore is essentially a No-SQL style storage method that sits within Postgres SQL databases. It allows you to store hash style data in a single column and then do some wonderous things with it.

For instance, consider this:

We have a products table, and it has some attributes:

CREATE EXTENSION hstore;

CREATE TABLE products (
     id serial PRIMARY KEY,
     name varchar,
     attributes hstore
   );

and we have a few products to put in there:

INSERT INTO products (name, attributes)
    VALUES (
      'Leica M9',
      'manufacturer  => Leica,
       type          => camera,
       megapixels    => 18,
       sensor        => "full-frame 35mm"'
    ),
    ( 'MacBook Air 11',
      'manufacturer  => Apple,
       type          => computer,
       ram           => 4GB,
       storage       => 256GB,
       processor     => "1.8 ghz Intel i7 dual core",
       weight        => 2.38lbs'
    ),
    ( 'Bosch WAE28166GB Washing Machine',
      'manufacturer  => Bosch,
       type          => washing_machine,
       capacity      => 6kg,
       spinspeed     => 1400rpm,
       energyrating  => A+'
    );

Now, what we have here is interesting.

We have our products table, as normal, but with this attributes column of the hStore type. This contains products as you would expect, but what we also have is three separate and very different products stored in them, with some very different attributes. The cool thing here is that this is all in one table mashed together, but is still perfectly queryable from a conventional SQL interface.

For instance, find all products that have a 6kg capacity:

SELECT name, attributes
FROM products
WHERE attributes->'capacity' = '6kg'

or find me all products that are cameras with 18 megapixels:

SELECT name, attributes
FROM products
WHERE attributes->'type' = 'camera'
AND attributes->'megapixels' = '18'

Pretty powerful, I think you'll agree. What's more, you can still index this attributes, and also join them onto other tables for more complex querying. Best of all, it's all native in the database so is nice and snappy too.

This fills an interesting gap in data modelling such as the scenario raised above. Whats more, as it's so simple to implement it much more enticing to use rather than the alternatives (such as setting up a NoSQL solution just for one model).

Using it from Rails

But what about using something like Rails to play with this stuff? Well, you're in luck. Rails 4 has support for this baked in, or you can add it to Rails 3 now by using the activerecord-postgres-hstore gem that has been made available.

You've now got the ability to do the above quite happily:

Product.create( :name => "Bosch WAE28166GB Washing Machine",
                :attributes => {'manufacturer'     => 'Bosch',
                                'type'          => washing_machine,
                                'capacity'      => '6kg',
                                'spinspeed'     => '1400rpm',
                                'energyrating'  => 'A'+
                               })

Product.where("attributes -> 'capacity' = '6kg'")

Product.where("attributes -> 'type' = 'camera'")
       .where("attributes -> 'megapixels' = '18'")

Simple.

Awesome, so how can I use it now?

OK - so lets say you love this so much you want to use this right now. Simple option is to get a Heroku Postgres instance up and running and try this stuff out there however, this can cost, so if you're on a budget create a blank Rails app, push it to Heroku, and then use the new shared PG9 add-on that has been made available. Within a couple of seconds you've got everything you need to try out all of the above and play with it yourself. With both types of database you'll be able to connect to them with tools such as PGAdmin and start poking and prodding to your hearts content.

For more information on either hStore, or Heroku Postgres, simply see the links above.

Read-only filesystems are a good thing

neil.middleton@gmail.com (Neil Middleton) — Fri, 17 Feb 2012 00:00:00 PST

Read-only filesystems are a good thing

Something that seems to pop up lot from users of Heroku are comments about the read-only filesystem and the problems it then causes them. Judging by some of the Stackoverflow questions I answer there is a generally held belief that the read-only filesystem is a really bad architectural decision.

Well, that's just not cricket.

First, some background.

In the world of one server the use case is pretty simple. An application wants to create a file, and so it generates it and writes it to the local filesystem. This file is then ready to be served up later on if so needed.

But then traffic starts to climb, and you need two servers. Your application wants to create a file and so it generates it and writes it to the local filesystem. However, the other server cannot see this file as it's not on it's own filesystem, so you need to start thinking about how to share files between servers. This leads you to some sort of network attached storage.

A few months later things are going well and you have 500 servers, all receiving files and writing them, but because you're now running against a non-local filesystem you've not got a problem.

Now think about Heroku, and think of a web process (or dyno) as a server. One dyno on its own could potentially run on the local filesystem, but once you scale you have the exact same problem that's described above. However, in the case of Heroku your attached storage is generally going to be S3, a file storage system that works better and at a much larger scale than anything Heroku could offer in the short term. Therefore S3 is the recommended place to dump files.

However, this isn't the only option. You can still use the filesystem - to a point:

On the Aspen and Bamboo stacks, you can write to ./tmp and ./log, whereas on Cedar you can write pretty much anywhere. However, it's worth remembering that should you require these files on a subsequent request, there's absolutely no guarantee they'll be there. Your dyno might have moved or been restarted, thus reverting it back to its slug state.

More information can be found here: http://devcenter.heroku.com/articles/read-only-filesystem

At the end of the day, the read-only filesystem is something thats required in todays modern application architecture. Writing to local filesystems for shared files just isn't something that fits in the modern scaling world.

As Adam from Heroku puts it:

As we continue to explore the next generation of application deployment, I think we’re going to bump into a number of ways to structure apps differently in order to make them scalable. There will be some transitionary pain with these changes, because structure implies restrictions. Many PHP developers coming to Rails have complained about not being able to access sessions from models, or write SQL in your view. MVC creates restrictions, yes, but those very restrictions are what provides the structure. Coming from an unstructured environment, those restrictions may seem cumbersome or arbitrary; but once you’re in the habit, you come to appreciate the structure they create.

Why Heroku is a game changer

neil.middleton@gmail.com (Neil Middleton) — Tue, 07 Feb 2012 00:00:00 PST

Why Heroku is a game changer

Today, I had what I intially thought to be a very simple task to carry out. The task was this:

Take an existing written and tested codebase
Deploy it to a cloud server somewhere to act as failover for a Heroku instance of the same codebase.

All in all I thought this would be pretty simple. I've setup a multitude of applications on production servers in the past, ranging from ye olde cgi-bin on UNIX boxes, ranging through various guises of FTP, up to the current automation levels of tools such as Capistrano.

As such, I'm not new to it. Whilst I may be a developer at heart, like most I have an affinity with *nix and understand how it thinks. I'm therefore not completely alien to the whole process.

So, how hard can it be?

Well, at the end of the day I was still going, and still have some more to complete tomorrow. I wrote up a todo list of all the things I had to do before I started, and these included:

Create a server somewhere
Get access to the server using SSH
Create firewall rules for the server
Ensure that the server has Ruby installed and other ancillaries
Write the capistrano recipe for the deployment itself
Create an init script for the web process so that I could control it neatly
Install and configure monit to make sure things didn't die quietly and without any help
Setup logrotate so that the disk didn't fill up with crap loads of archived logfiles
Setup syslog so that the logs were directed to our Papertrail account
Deploy the code
Test

There's a few other minor thing missing, but this list should look pretty familiar to anyone who's setup a Unix web server.

So, anyway. Due to some unforseen problems (a few compatability issues and so on) the total time required to get the server built was ranging on a few hours. OK, I'm no sys admin, but like I've said above, I've setup a few servers before and I'm a developer, not someone who worries about the plumbing.

Now, here's the kicker. All this time is billable, so this would have cost someone a few hours of time, which can easily range into a few hundred dollars without even thinking about it.

Great for the bottom line, not great for the client (or my own sanity).

Now, cast your mind back to the initial requirements. This server build was purely a redundant server in case the one that was on Heroku was unavailable for some reason, be it maintenance, connectivity issues, or whatever.

I also setup that application.

What's more I setup that application in around five minutes. In fact I can recall the entire process start to finish:

heroku create -s cedar
git push heroku master
heroku config:add ...a few config vars
heroku addons:add redistogo:nano

Job done.

The primary difference here is that this only took me a few minutes and is completely repeatable without any complex config or trying to work out the best approach. More than that, now it's done, I can pretty much forget about the details unless I need to make changes.

I'm a developer you see, not a plumber.

Now, having done both, you'll probably get to the same realisation that I had a couple of years ago. Hosting with a platform such as Heroku isn't about the cost savings over some other host. It's not about the addon library, it's not about the massively scalable architecture (which are all obviously nice to have) - it's about the fact that I, as a developer, with little in the way of real sysadmin knowledge can get applications up and running with little or no fuss and get back to doing what I do best as soon as possible.

What's more I can do it in a way that is better than if I had done it myself. My application is scalable, it's processes are isolated and configurable via an environment, and it's external resources are merely attached rather than baked in hard.

Another benefit of this approach is that I don't really need to know the details. I don't need to learn how to connect my thin process to the outside world. I don't even need to know how to install Ruby, I just push my code to Heroku and it's golden.

This really makes a difference when I'm deploying something that I might not be that familiar with. For instance, I might be deploying something a little different such as Play!. Luckily as long as Heroku knows the difference and how to run it I don't need to know anymore. Now that Heroku buildpacks are on the scene this gets even better. It only needs one person to create a buildpack and you've got another runtime that can be deployed to Heroku - no need to wait for official support from Heroku.

So therefore, if you are running your own servers, or have looked at Heroku and are put off by the cost, take a serious look at your own costs and headaches that you have. Factor in your time setting up servers and troubleshooting. Factor in those beeper calls at weekends when something in your network has gone pop. Factor in the necessary updates and patches you need to carry out on a regular basis.

Then compare them again.

Remember to timeout your tasks

neil.middleton@gmail.com (Neil Middleton) — Fri, 27 Jan 2012 00:00:00 PST

Remember to timeout your tasks

Time for a just-learned-from-experience post.

On my employers website, kyan.com, we have a number of counters which can be found in the footer of every page. These numbers include straight-forward things like the number of people that have registered on webmeetguildford.co.uk to more odd things such as the number of pixels that our designers have pushed.

Whilst you may think that these numbers are false, they are all actually based on real world counters (which may or may not be then massaged into something more entertaining). For instance, the pixels counter has a look at how many design hours have been logged in our intranet and goes from there.

But enough of this, what does this have to do with Heroku and tasks? Well, all of these stats are pulled from various internal APIs as a background rake task.

That's simple and just fine, right?

Well no.

This is normally fine, unless you couple this with a flakey internet connection (which we've been suffering from recently). It appears that if you lose your connection, the API calls in your task won't complete and the task then runs on and on, never reaching any deterministic point in which to die. Ultimately this ends up costing you money.

So, how do we get around this? Well, simple (with Ruby 1.9.2). Timeout!

require 'timeout'

Timeout::timeout(10) {
  ... thing that should not last forever
}

This will apply a 10 second time limit to the included code, and stop it with a Timeout::Error should it overrun.

By applying this to your background tasks you can be sure that you're never sitting there paying for processes which are quite happily spinning their little hearts out in silence.

The Procfile is your friend

neil.middleton@gmail.com (Neil Middleton) — Fri, 13 Jan 2012 00:00:00 PST

The Procfile is your friend

With the new Heroku Cedar stack there is a change in the way that the Heroku platform views processes. In the 'old' days of Bamboo and Aspen, you had dynos (your web processes) and workers (your background job processes). Both of these limited you in a few ways as to what you could run on the platform inside your application.

These days, the cedar stack doesn't see dyno's or workers, it just sees processes which is more inline with the Unix design philosophy that the entire stack has been designed against.

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

In order to declare these processes, and scale them individually, we need to be able to tell Heroku what these processes are.

Enter the Procfile, stage left.

The Procfile is a simple YAML-esque* file which sits in the root of your application code and is pushed to your application when you deploy. This file contains a definition of every process you require in your application, and how that process should be started. An example of this would be something like this:

web:  bundle exec rails server -p $PORT
worker: bundle exec rake jobs:work

This example is pretty typical of a Rails application. It contains the web process for the front end, and a worker for delayed_job, a background job processor.

Note the explicit mention of $PORT. This is required because the Heroku stack will explicitly define the port that it wants your process to connect to, and publish this as an environment variable. Other environment variables are also available, so something such as the following is also perfectly valid:

web:   bundle exec thin start -p $PORT -e $RACK_ENV

So, what about scaling? Well, this is simple via the Heroku command line interface:

$ heroku ps:scale web=10
$ heroku ps:scale worker+1

and so on.

By declaring each process, and giving it a name, you're able to scale them individually from each other and even turn them off altogether. You may have traditional web processes, but you may also have workers, clock processes or more. Anything goes here, it's purely up to you.

Development

In development though, your Procfile is not a wasted resource. By using the foreman gem, you're able to utilise your Procfile and use it to spin up your development environment in the exact same way. Foreman is dead simple to use too:

$ gem install foreman
$ foreman start

18:06:23 web.1     | started with pid 47219
18:06:23 worker.1  | started with pid 47220
18:06:25 worker.1  | (in /Users/foo/myapp)
18:06:27 web.1     | => Booting Mongrel
...

Now, an interesting thing to mention here is that Heroku only auto-launches the web process in your Procfile when deploying, whereas Foreman launches everything. Therefore you could use a Procfile such as this:

web: bundle exec rails server -p $PORT
worker: bundle exec rake jobs:work
redis: redis-server

On Heroku, this will only auto-launch the Rails process, and require you to scale the worker processes. However, with Foreman, it will also launch Redis for you should your application require it.

Apps without an user interface

Where it gets really interesting is when you consider that you might not even want a web process at all. For instance, you might need a Resque worker running somewhere processing incoming jobs and spitting out results to another output (file or API etc). This is a valid example of this:

worker: QUEUE=* bundle exec rake resque:work

as are these:

redis: echo "port $PORT" | bin/redis-server -

pg: bin/postmaster -p $PORT

So, as you can see, the Procfile is a very powerful thing, and something that provides you with an immense amount of control over what your applications are doing and how. It also gives you a massive amount of flexibility with how you want to scale your application and processes.

Not something to be sniffed at.

* If it had three dashes at the top it would be YAML

Sharing databases between Heroku applications

neil.middleton@gmail.com (Neil Middleton) — Thu, 05 Jan 2012 00:00:00 PST

Sharing databases between Heroku applications

When you deploy an application to Heroku, you'll probably know that all applications receive a free 5mb database on their shared PostgreSQL database service. Whilst these are on shared servers at Heroku, the performance is still pretty good, and the cost of a size upgrade is cheap too ($15/mo for 20Gb).

However, sometimes we want our applications to share our data, and by default this isn't immediately obvious as to how this works.

Well, it's possible, but let's look into the background first.

Every Heroku application is able to have the shared-database:5mb add-on, and some deployments such as Rails, get it by default. Once this add-on is added to an application two things happen. Firstly, Heroku creates the database for you, and secondly, each time you deploy your application Heroku will overwrite your database configuration where possible (so for instance, with Rails, it will overwrite `config/database.yml' with their own version).

Essentially what this does it connects your application to an environment variable driven connection. You can see this by checking your configuration:

heroku config

This will yield a whole load of settings, but in amongst them you'll see something like:

DATABASE_URL => postgres://sakldjsad:kajshdsa@ec2-107-20-155-141.compute-1.amazonaws.com/sakldjsad

This is the database connection string that your application will be using.

So, how do we connect two applications together (or even replace the Heroku database with some other connection?)

Well, there's two ways. Firstly, you could simply overwrite the DATABASE_URL config variable:

heroku config:add DATABASE_URL=your_new_connection_string

an alternative is to use pg:promote which essentially does the same thing:

heroku pg:promote your_new_connection_string

By using this technique you can string your applications together in loads of different ways, sharing free databases, or even connecting lots of applications in a single Heroku Postgres instance.

For those Rails developers interested in how Heroku overwrites your config/database.yml, you can see the code below:

<%

require 'cgi'
require 'uri'

begin
  uri = URI.parse(ENV["DATABASE_URL"])
rescue URI::InvalidURIError
  raise "Invalid DATABASE_URL"
end

raise "No RACK_ENV or RAILS_ENV found" unless ENV["RAILS_ENV"] || ENV["RACK_ENV"]

def attribute(name, value)
  value ? "#{name}: #{value}" : ""
end

adapter = uri.scheme
adapter = "postgresql" if adapter == "postgres"

database = (uri.path || "").split("/")[1]

username = uri.user
password = uri.password

host = uri.host
port = uri.port

params = CGI.parse(uri.query || "")

%>

<%= ENV["RAILS_ENV"] || ENV["RACK_ENV"] %>:
  <%= attribute "adapter",  adapter %>
  <%= attribute "database", database %>
  <%= attribute "username", username %>
  <%= attribute "password", password %>
  <%= attribute "host",     host %>
  <%= attribute "port",     port %>

<% params.each do |key, value| %>
  <%= key %>: <%= value.first %>
<% end %>

The myths of cloud hosting

neil.middleton@gmail.com (Neil Middleton) — Mon, 12 Dec 2011 00:00:00 PST

The myths of cloud hosting

Day to day I see a lot of comments from people related to hosting on a cloud hosting environment. It seems that if you're not a developer, that the idea of cloud hosting fills you with fear and uncertainty, and is something that should be trodden carefully around.

However, it's clear to me as a developer that this is the wrong way to look at it. I've been using cloud hosting (namely Heroku) for a couple of years now, as well as tried out a few others (such as EngineYard). Heroku is the one that floated my boat, so that's the one I will primarily refer to here.

So what are the problems that people have? Well, heres the most common in FAQ form:

I don't want my sites to go down...

This is by far the most common thing I've heard, that somehow once you host in the cloud and are no longer able to actually SSH into your servers you're somehow compromising on your hosting. This is simply not true.

Let me describe a couple of scenarios. One is small business, say 20 or 30 people. They manage a few servers for their clients, and the guys that look after them are typically the ones who puts themselves forward for the task, but primarily they are developers. These guys work office hours.

The second is large company that only does hosting. Their entire reputation is based on being able to provide good hosting, so their monitoring their servers 24/7 in more ways than you can think of, and employ nothing but experts to look after the servers. These guys work all the time.

Now let's describe a common scenario. It's five in the afternoon, and everything goes pop. All of your sites go down. Now, most people would feel happier starting to frantically SSH into stuff to try and see what the problem is, or randomly start rebooting servers.

With cloud hosting, it's somewhat different. You know that by the time you become aware of a problem, that it's already being looked at by a team of experts with some sort of structure (for instance, Heroku have Incident commanders, and people on call at different times) in a methodical way. You also know that the chances are they would have fixed the problem before you were even aware. A lot of people have trouble with the idea of sitting back and doing nothing when sites go down, but you're not doing it wrong, you're leaving it up to the people who know about these things the best.

Now, there is an exception to this, namely those events when your application goes down, but as that's not normally something that is affected by your hosting, I won't go into that here.

I'm concerned about the privacy of my code and my data

OK - this is a simple one. All hosting companies stake their reputation on two things. Reliability, and confidentiality. If any hosting shop is found out that they are peeking into / sharing customers code and data, they might as well pack up shop and go home. This is simply something that a company such as Heroku or others would even consider risking. What's more, most of these companies have very strict processes in place to ensure that they don't even chance privacy breaches. For instance, GMail support engineers will simply not look at any email, even if you give them explicit permission. The chances are that if you're looking at a well known cloud hosting firm, that they will take privacy /very/ seriously.

Now, think back to the shop that is generating this code / data. What are the chances that:

Everyone in the office has access to the code...
A fair few people (but you're not sure who) have access to the production servers...
A few people (but again, no idea who really) have access to your production data...

See? You're probably better off with the people who do this stuff for a living.

I'm concerned about the cost

Again, simple.

Let's say you want to host a server. You buy a new shiny Dell (let's say a middle range one, about $2,000), you then pay one of your staff to set it up which takes them five or six hours (let's call that $600), and you then buy some rackspace from somebody for it (another $500/month maybe?). Each week you need to spend a couple of hours looking at it ($500/month) and fix it when the stuff goes bang (let's average at $200/month).

So, even though these numbers are /extremely/ approximate, you're looking at $2,600 to get started, and a running cost of around $1,200/month to look after it in staffing costs. That's $3,800 for your first year. This will let you scale to, say, 5,000 active users.

Now let's put this into Heroku money. Your $3,800 will get you some dyno hours. At $0.05c/hour it will get you a LOT of hours. In fact, it will get you 76,000 dyno hours, or enough to run a single dyno for about eight years. Granted a single dyno won't give you the same grunt, so let's say you're running two concurrently - that's four years worth, and so on. What's more, you also have two or three hours a week of extra billable hours being generated because you're not having to support your own hardware.

However, there will be a tipping point. There is a point with any project where the usage will get high enough that cloud hosting costs don't make sense. If you're Facebook or Twitter you won't be looking at outsourcing your hosting, but then again, do you really think that the cost of hosting is a major concern for them? Chances are, by the time the size of your hosting becomes a problem, cost won't be.

What backups are available?

See my previous comment about reputation. What could be worse for a hosting company? Hosing your website AND all your data, only to be able to say "sorry", or hosing your website, but being able to bring it up exactly how it was with very little pain on the customers side. Whilst you may consider yourself paranoid, for the most part, your hosting company will be ten times more so.

For instance, read the Heroku Postgres docs. Can you reasonably think about implementing the following:

All Heroku Postgres databases include Continuous Protection. This feature archives the PostgreSQL Write-Ahead-Log (WAL) to a geographically-distributed, highly durable storage service. In the event of a catastrophic failure, the WAL can be “replayed” to reconstruct a database.

Continuous Protection performs the WAL archival every 60 seconds or 16 MB of new data, whichever comes first. Therefore your data is protected by this feature once it has been committed for at least one minute. During times when the write-throughput on a database is extremely high, our service may favor data transaction volume over WAL archiving for short periods of time. This may result in a slightly longer interval for data to be protected.

Most people are happy with daily backups, the paranoid go for hourly. Only the committed go further. With cloud hosting it's not really something you need to worry about.

However, with backups, you should. You should always worry about backups. If you're not protecting yourself in at least some way, you're a fool.

So, overall, I (obviously) think cloud hosting is the way forward. I genuinely believe that I, as a developer or as someone who owns a development shop, should not have to worry about servers and hosting, but should be able to just throw their code at a partner and leave them to worry about it. There are so many downsides to running your own hardware that they offset any upsides you might think of. Granted cloud hosting does not suit everyone, but you'd have to be treading outside of the ordinary to not make it worthwhile.

The dangers of A-records and Heroku

neil.middleton@gmail.com (Neil Middleton) — Wed, 07 Dec 2011 00:00:00 PST

The dangers of A-records and Heroku

Over the weekend, Heroku suffered, as all large hosting companies tend to, from a very large scale distributed denial of service (DDoS) attack. Luckily for us though, as Heroku is a fully managed platform this isn't something that us mere developers would normally need to worry about as Heroku are on it and trying to resolve the problem before you would normally even know about it. (Big thanks to Mark Imbriaco and his daughter required for this one.)

So, why the blog post?

Well, in the post event report Heroku raised an important point about DNS which I will re-raise again here in the hope that more people are aware of the problem and work around it.

Essentially the problem lies with DNS and the A-record. Typically, for a top level (or apex) domain such as neilmiddleton.com, I have to set an A-record to resolve that to an IP address. Therefore neilmiddleton.com must always resolve to an IP address. However, with subdomains such as www.neilmiddleton.com I can set this to be a CNAME (or alias) of another domain. Therefore when requesting the location of www.neilmiddleton.com, the DNS seems that this is just an alias of neilmiddleton.github.com which in itself has an IP address, so that's the one thats used.

Now, coming back to the DDoS attack over the weekend. In this situation, there's a flood of traffic coming into the Heroku IPs. Normally, one response to this would be to direct traffic via another means temporarily, but due to the fact that everyone has their A records pointing at those IP's the scaling options are more limited.

With CNAME's however, these can be moved around almost at will, meaning that companies like Heroku are able to scale horizontally, and change where their own domains are resolving to thus allowing them to move traffic around.

So, not using A records appears to be the best answer here. Always use CNAMEs where possible which means you can leave Heroku etc to worry about where your applications are sitting at any one time. However, A records don't allow you to do this, but there are a couple of ways around it. One approach is that taken by Zerigo, who let you redirect one DNS record to another (which happens purely in their system). This means you can setup neilmiddleton.com to redirect to www.neilmiddleton.com at a DNS level. This unfortunately though means that you can't host anything on the top level domain, which is where you need something like DNSimples ALIAS setup, where they allow you to CNAME an apex domain, again something that's purely handled internally to them. The net result is the outside world see a regular A record pointing at an IP, whereas internally it's a CNAME.

Both these solutions are clever, but something that isn't part of DNS itself so you're limited on where these things can be done. The core thing here is that ideally you shouldn't be pointing your DNS at ANY IP addresses when hosting on platforms such as Heroku, but instead CNAMEing everything so that Heroku can handle it when they need to.

(Note, it's worth adding that this is probably a common concept for most cloud hosting, the core differentiator being whether or not they expose your application via a CNAMEable domain)

Deploying topic branches to Heroku

neil.middleton@gmail.com (Neil Middleton) — Sat, 19 Nov 2011 00:00:00 PST

Deploying topic branches to Heroku

With Heroku it's super easy to create and deploy applications, almost to the point where hosted production applications are nothing more than a commodity.

Due to this, it's really easy to deploy your production environment, and then create sibling test environments for trying out various things such as deploys, load tests, or for deploying topic branches.

However, something worth noting is that Heroku ONLY deploy the master branch for a given git repository. So how do you manage this? Well, Git allows you to push one branch into another. For instance, lets say you want to push your staging branch onto Heroku's master branch of your staging application, which you've already setup. Simply run:

git push staging staging:master

This will push your staging changes into the Heroku master branch, and thus deploy it.

Now, if this is a common occurence you might want to shortcut this and make it happen by default. This can be done by telling Git to always push from one branch to another. For instance, for the same example as above.

git config remote.staging.push staging:master

replacing remote.staging.push with remote.your_remote.push and staging with your topic branch. Once you've done this, you can simply push to the appropriate remote:

git push staging

which will yield output confirming the branch has correctly deployed:

To git@heroku.com:myapp.git
   16c73af..b1a78b4  staging -> master

Getting more from your Heroku dynos

neil.middleton@gmail.com (Neil Middleton) — Mon, 07 Nov 2011 00:00:00 PST

Getting more from your Heroku dynos

If you're using Heroku these days, you're probably having a good look at the new Celador Cedar stack that was introduced in Beta a few months ago. This new stack gives you much more control over what processes are running and how they run.

Whilst this might not give you any immediate obvious benefits, this does give you the ability to change from the default choice of Thin as your Rack server, to something else such as Unicorn.

So, why should you bother? Well, it's all about how the Dyno's on Heroku work and how you pay for them.

In Heroku a dyno is a single process that is self contained. This dyno can do whatever you like, but has a soft limit of 512Mb of memory. When using Thin, this dyno will hammer away quite happily serving one request at a time.

However, Unicorn works slightly differently. Unicorn manages it's worker processes internally. This means that you are able to have many Unicorn workers running within a single Unicorn process, which is how the Heroku dynos would see it. Therefore, you could have four Unicorn processes running in one Heroku dyno.

In testing on a current application that we're building we're seeing around double the throughput on the same number of dynos. Note that this testing is only on a pre-production environment and not one with real users. Once we have more users we might be able to give some more accurate figures.

So, how do we use this in a Rails app for instance? Firstly, you need to add Unicorn to your Gemfile:

gem 'unicorn'

You'll also need to create a unicorn config file in your app (for instance config/unicorn.rb):

worker_processes 4
timeout 30

Note that you don't want to push the worker_processes too high or you'll start hitting the memory limits on the dyno. Only by playing with the amount and testing will you find that happy place for your application. A common symptom of too many workers is the Heroku errors R14 and R15. Generally speaking, if you're seeing R14's in your logs then dropping the number of workers by 1 will alleviate the problem. Only by repeated testing and deploying will you be able to determine the ideal number for your application.

Once you've done the above, you need to tell your Procfile how to spin up on Unicorn:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb

Commit, deploy, push, and you should start seeing Unicorn running in your logs, and a higher throughput as a result.

Testing

In order to test this I brought in the skills of Heroku and the Blitz add-on, a simple load testing platform that allows you to do free tests (as long as you don't want more than 250 concurrent users).

To start off with I created a simple Rails 3.1.3 application, which does nothing more than contain a single action. This action merely sleeps for 100ms thus simulating a typical page execution time. I then deployed this to Heroku using the defaults for everything. Against this I app I then ran a Blitz test of 15 users for 20 seconds.

Once the test was complete I then saw the following results:

(Full size)

This shows that Heroku was able to serve 159 requests within a second, and thus approximately 686,000 requests a day.

So what about Unicorn? Well, to test this I simply carried about the above instructions to change my application over to Unicorn, and then re-ran the test suite as before:

(Full size)

This resulted in an increase of 92 hits over the course of the test, or an increase of approximately 60% - not too shabby you'll have to agree.

So, from this simple test, you can see that using the Rails defaults you'll be serving 686K pages a day, whereas with Unicorn you can manage just over a million, all for the same money.

Using webfonts in the asset pipeline

neil.middleton@gmail.com (Neil Middleton) — Thu, 03 Nov 2011 00:00:00 PDT

Using webfonts in the asset pipeline

Now that we have the asset pipeline, you need to include your fonts in a slightly different way.

All assets inside /app/assets fonts, images, audios, javascripts and stylesheets are included, but to get the appropriate links in your CSS etc you need to use

asset-url("my-webfont.eot", font);

However, before this will work, you'll need sass-rails in your Gemfile:

gem 'sass-rails'

Once this is done, you should see proper url() declarations appearing in your CSS.

For more info on the asset pipeline, read the Rails Guide.

Precompiling non-default assets with Rails 3

neil.middleton@gmail.com (Neil Middleton) — Tue, 01 Nov 2011 00:00:00 PDT

Precompiling non-default assets with Rails 3

I've just been playing with an interesting problem whereby Rails 3 wasn't precompiling assets as I would expect.

I have an application which contains two main javascript 'bundles'. One for the common man, and one for the administrators only. Now there's no point in serving a whole load of admin stuff to regular users hence the seperation.

However, when precompiling my assets I noticed that Rails was only doing the default application.js, and ignoring my admin.js.

Well, it turns out you need to manually tell Rails that you've added that additional resource. Luckily this is nice and easy:

config.assets.precompile += ['admin.js']

Job jobbed.

How to do wildcard selectors with Sass

neil.middleton@gmail.com (Neil Middleton) — Tue, 18 Oct 2011 00:00:00 PDT

How to do wildcard selectors with Sass

To produce:

.blah[class*='span']  {
  text-align: center;
}

use:

.blah
  &[class*='span']
    text-align: center

Making Postgres STFU in Rails

neil.middleton@gmail.com (Neil Middleton) — Mon, 19 Sep 2011 00:00:00 PDT

Making Postgres STFU in Rails

If you’re using Heroku to host your Ruby Rack apps, then you’re probably also using Postgres, which is the default choice for the database stack.

However, one downside to this is that in development Postgres is a noisy bastard telling you about everything that you never wanted to know, plus the odd query of interest.

Well, luckily there is a couple of things you’re able to do to reduce the noise. Firstly you can tell Postgres directly that you don’t want to hear it’s warnings about table creation and the like:

development:
  adapter: postgresql
  host: localhost
  min_messages: warning
  database: foo_development

If this isn’t enough for you - you can also tell Postgres to only give you the queries that your app generates rather than everything that postgres sends over the wire. This is all handled by the silent-postgres gem.

To add, simply whack this in your gemfile.

gem "silent-postgres"

Bundle install, and you're good to go, and in peace.

Social Media Burnout

neil.middleton@gmail.com (Neil Middleton) — Tue, 23 Aug 2011 00:00:00 PDT

Social Media Burnout

This might turn into a long ranty post, so bear with me, it's been in the pipeline for a while.

These days, it appears that you cannot move for another social network popping up. We've got Facebook, Twitter, Google+, Foursquare, Gowalla, LinkedIn, and that's only to name the few that have been reasonably successful, but outside of this very small circle, there's hundreds more.

The problem we have these days is that social isn't a new thing, it's a very well established market online and the main players have already made themselves very well known. However, every day, there's some other network popping up, and it's normally a clone or vague twist on one of the established players. Sites like About.me for instance, which fulfill a small niche job, but basically do the same thing as all the other networks out there.

So why? Well, I would hazard a guess that it's to do with money. There's a billion and one startups out there who've got the next shit idea that will make them their VC billions - eBay for dentists, Facebook for pets, the list goes on, but the thing is, they're all flawed in so many ways (cats can't type). But social equals money. We've all seen how much money companies like Facebook and Twitter are allegedly worth* these days, and everyone wants a slice of that pie.

For instance, new network Zerply, a site where you can add your CV online and 'Find like minded people by tags, skills, location and more'. For starters, other than the fact that it's a bit simpler, and more colourful than LinkedIn - it's LinkedIn, just in a frock.

But what problem does this solve? It's another place to dump your information and create an online profile / CV. But would you ever rock up to an interview with a potential employer with your LinkedIn or Zerply profile, or would you spend the time to make a proper resume that you have full control over?

OK, so maybe it's the networking effect that is the key. Well, again, to this day I've never been introduced to anyone through LinkedIn for work. I've never got a job through LinkedIn, and I don't know anyone who has. As far as I am aware, every single person I know has only 'connected' with people they already know, and received a shit load of crap from recruiters.

Ah, yes. Recruiters. The underbelly of the IT industry that stalk us at our every turn, and one of the primary reasons I have a LinkedIn account no more. These buggers keyword search you and send you email for jobs that'll never interest you, and then harass you until you take some action to stop them. These are the people who like the idea of the online CV, but only because they're generally quite lazy **.

So, let's assume our new social network idea is a good one, where's the next problem? Well, the fact that everyone is using something else for starters. Take Google+, the new network from Google. Functionally it's superior to Facebook in almost every way, and looks nicer to boot, however, even though millions of people have Google accounts it's still relatively quiet. Why? Well everyone's using Facebook aren't they... Your average Joe isn't going to migrate over to Google+ until his buddies do, otherwise he's out there on his own. It takes a mass migration to get people to change their core network site of choice, and isn't something the web has seen before. When Facebook popped up there wasn't much else in the way of alternatives. Whether Facebook maintains it's position for the years to come remains to be seen.

For now though, any social network that is created has an exceptionally massive hill to climb. Not only has it got to provide something that is relatively feature complete, but it also needs to do something unique (and useful), and woo people away from the tool they know and love already - which is going to be hard.

If you believe the valuations at all and realise that they aren't just made up money.

** Note, I know plenty of GOOD recruiters, just none in IT.

Avoiding Zombie Dynos With Heroku

neil.middleton@gmail.com (Neil Middleton) — Fri, 28 Jan 2011 00:00:00 PST

Avoiding Zombie Dynos With Heroku

I’ve been looking at some long running processes on Heroku of recent and found there is a behaviour in place which is not quite what I expected.

The heroku docs state:

The Heroku routing mesh will also detect a different kind of performance problem: long-running requests. If your dyno takes more than 30 seconds to respond to a request, the mesh will serve a “Request Timeout” page to the user.

Now, initially, you’d think that this means that your dyno process is killed off after 30 seconds. What is actually happening, is literally as the docs say, the routing mesh is returning a page timeout to the user. The dyno is actually still spinning away, working it’s merry little heart out even though nobody’s listening any more. The primary problem with this is that dyno is still busy and can’t accept any more requests until it’s finished - which leads you to having a zombie’d dyno.

So, what can we do? Well, the support guys at Heroku pointed me at rack-timeout, a gem which can be used to timeout all your application requests within a set time limit (for instance, the 30 seconds that Heroku set). Gem is a dead simple install and configuration, and solves this problem neatly and elegantly.

Regurgitating Search Engine Optimisation

neil.middleton@gmail.com (Neil Middleton) — Tue, 14 Dec 2010 00:00:00 PST

Regurgitating Search Engine Optimisation

Fairly often, we at Monochrome come across ‘search engine optimisers’. These people are there to provide a service to companies that sits somewhere between the technical. the marketing, and the design.

Essentially the brief is simple. Take a website (either at concept stages, or already live) and tweak it so that when a user is looking for a relevant term on one of the major search engines, the site in question will appear near the top of the organic search results.

There’s a number of tried and tested ways of doing this, plus a wide variety of old wives tales that some people believe make a difference. However, search engine optimisation is fairly simple and straightforward, and not something that a lot of people need spend a lot of time on. Once you’ve got the basics in place you’re pretty much good to go, as long as you keep your standards up and keep doing what the search engines like you to be doing.

However, one thing that is becoming more and more apparent is the quality of these services that other people provide and the approach that is taken. From the various optimisation companies that I have come across, the vast majority follow a standard path. This will initially consist of the running of some sort of tool that spits out graphs and tables of how your site might be performing on some varying measures (which might or might not be relevant), and they then chuck all this into a standard document that get’s sent to the client.

You’ll always get the standard fare - Meta tags, page rank, canonical URLs, H1 tags etc etc etc. You’ll also find that most of the data is already available to you via tools such as Google’s Webmaster Tools

Now, this initial work could cost thousands depending on the agency and site in mind (bigger client, bigger bucks), but ultimately the work required isn’t drastically different. SEO is SEO.

What comes next is the monthly cost that the vast majority charge - for instance, charging to tend to the garden of keywords that you might be targeting, or ensuring your markup is up to par. Other than that, it’s pretty much rinse and repeat.

So, why do companies blindly pay this money out? Essentially you’re paying the vast majority of SEO’s to cut and paste into a word document, and provide you with a few snazzy graphs. You’re then paying well over the odds for a couple of hours of maintenance. If developers charged the same amount, we’d all be laughed out of the room.

The biggest problem is the charging model. Every SEO company I’ve ever met charge via the standard mechanism. However, their benefit to the site varies, so why do we not have these companies charge by results?

You engage with an SEO in order to raise the profile of your site and achieve some other objective - increase traffic, increase leads, increase conversions, raise brand awareness. You have ideas on how much you want to achieve your objective by - page views up by 25%, conversions up by 10%. Therefore, why do we pay SEO’s regardless? Surely we should only pay these people if they achieve what we wanted and to a satisfactory level? After all, you wouldn’t pay a interior decorator if he only painted half a room, would you.

So - search engine optimisers - do yourself a favour. Find out WHY the client wants to talk to you, WHAT they want to achieve and WHAT sort of budget they might have to try and do this. Only once you do this will you truly be able to add a worthwhile service to your client that they will be happy to pay for again and again.

Day Rates Suck

neil.middleton@gmail.com (Neil Middleton) — Thu, 09 Dec 2010 00:00:00 PST

Day Rates Suck

Every single day, we find ourselves as an agency, telling people what our day rate is. In most cases, for people, this is a race to the bottom, the challenge of finding the cheapest development shop out there to get a certain piece of work done.

People often believe that if someone is charging £500/day, whilst someone else is charging £600/day then the cheaper option is the best one and will save them a shed load of money.

Well, funnily enough, it’s not that simple.

For instance, let’s imagine for a minute that you don’t care what technology your idea is built in, you just send out some tenders. Two come back - a Java shop with a day rate of £400, and a Rails shop with a day rate of £750. The Java shop is an offshore setup, with unlimited developers to throw at your job, promising a quick turnaround. The other, the Rails shop, is local, and has a really nice portfolio (but still costs nearly twice the price). You’d love to employ them but you just can’t afford that rate so you plump for the cheaper option.

Six months later, you’re broke. You’ve just spent twenty grand on some Java work but it’s not finished, and you’ve got no way of moving forwards as you’ve got no more money. However, looking on the bright side, you know that you’ve not sunk nearly forty grand into the same project with the other company.

Except you’re wrong.

What you failed to realise is that everything you’ve assumed in the pricing of your job is incorrect. You’ve assumed quality is a constant, i.e. that the number of bugs is consistent between different agencies, and that performance is identical. You’ve also made a even more fundamentally bad assumption, and that is that a developer only achieves a certain amount in a certain amount of time.

They don’t.

Take the above example. Rails is a much more productive tool to use for web development than Java, and a Rails developer could reasonably expect to outperform a Java developer by as much as 100%. Add into this that Rails developers tend to unit test much more so than others it all starts to add up.

So, you go back to the Rails shop.

“Oh yes, we could have turned that project around in 22 days, which would cost you £16,500”

You slowly, and quietly, start to cry.

The moral of the story is this - day rates, for development work, are irrelevant. You, as a customer, don’t want to buy time from a development house. You want a development house to produce you a ‘thing’, whatever that ‘thing’ might be - and you want to know how much it costs. It doesn’t matter how much a day of developer might cost you. What matters is how much can be achieved and what level of quality might be achieved.

So, next time you’re out there looking for a development house for your next project, ignore day rate - it’s irrelevant.

Improvisation - Techie Style

neil.middleton@gmail.com (Neil Middleton) — Thu, 21 Oct 2010 00:00:00 PDT

Improvisation - Techie Style

Yesterday afternoon I was presenting at the Webtrends Engage conference in London on Usability, User Experience and Data Visualisation. Now, when presenting at these sort of events there’s an AV company on hand who sit at the back of the room and make sure all the sound and light are working as they should. However, Engage presented an interesting problem: my laptop, which contains all the slides, notes and presenter view, which normally sits on the podium, had to be at the back of the room with the AV guy.

Yup, my laptop, was 20 ft away. Brilliant.

So, some lateral thinking was initiated. How could I control the presentation from the front of the room and still know what the hell was coming up?

First problem was control. The IR Remote you get with a Mac only works on a clear line of site, and only over a few metres, so this was a no-go. Luckily for me I had my Magic mouse in my bag which was on bluetooth, and that progresses the slide on click so that was control sorted.

But what about my notes and timers? Well, luckily we had an iPad on hand and Apple has a Keynote Remote app on the App Store which allows an iOS device to become the remote for a keynote presentation on a network connected Mac.

Bugger….network.

In steps my Android phone running Froyo. A quick portable wi-fi network was whipped up and laptop and iPad were reunited.

So, after all that, I had my presenter view on the iPad on the podium, my mouse in hand to act as a clicker, and my laptop running the show from the back of the room - all while the AV guy had a little kip.

Macbook Pro, iPad, Magic Mouse, HTC Desire, Bluetooth and Wifi, all to produce a picture of a urinal… fun.

Case Insensitive searching with PostgreSQL

neil.middleton@gmail.com (Neil Middleton) — Tue, 08 Jun 2010 00:00:00 PDT

Case Insensitive searching with PostgreSQL

Earlier today (well, a few minutes ago), I had a requirement to make an existing LIKE search case-insensitive with PostgreSQL.

Normally this would prove to be messy, by trying dirty tricks such as downcase-ing content in the search (using UPPER and LOWER) - however, it’s pretty straightforward.

For instance, the case-sensitive code would be something like:

SELECT * FROM sometable WHERE textfield LIKE '%value%';

Whereas, to make it case-insensitive, try a bit of Borat:

SELECT * FROM sometable WHERE textfield ILIKE '%value%';

Alternatively, you can also use a tilde syntax:

SELECT * FROM sometable WHERE textfield ~ 'value'; -- case-sensitive search
SELECT * FROM sometable WHERE textfield ~* 'value'; -- case-insensitive search

Job done.

Forget Mobile The Web Is Where It's At

neil.middleton@gmail.com (Neil Middleton) — Wed, 05 May 2010 00:00:00 PDT

Forget Mobile The Web Is Where It's At

Over the last few months, it would appear that global internet thermonuclear war has set in.

This, for the most part has been surrounding the mobile platforms out there. Apple’s i(Phone/Pad/Pod Touch) has been battling Google’s Android, Apple has been battling with Adobe over Flash, and then there’s the patent wars betweenApple, Google, HTC and Microsoft.

It’s a pretty big fight.

So, what do developers do? From a skills point of view we want to be aligned with the most popular platform that will give us the most marketability - yet from a business point of view we want our applications to work on as many platforms as possible to maximise the ‘attack surface’ in the various markets out there and gain the most revenue.

Then there’s the different markets - Apple have their draconian approval process, but provide the class leading App Store, whereas Google is all about free-love and sell it how you want.

Then there’s the platforms. Objective-C, Java, Regular C, Flash etc etc etc etc. The list goes on.

So what are we developers to do - how are we supposed to navigate through this minefield of options? Well, it’s pretty simple actually:

Ignore it.

Every mobile phone manufacturer is striving to produce a browser on their phone that can view the whole web in a similar way to a desktop computer, and all are striving to support the latest and greatest. Therefore, surely the easiest way for us web developers to bring an application to a phone is to simply look at it through a browser?

Picture this, you have an application out there in the wild, you want it to work on the iPhone, so use Safari - job done. It’ll also work on Android with no changes, and is also not dependant on Apple’s approval or anything else, you can do it how you want. Additionally, with libraries like JQTouch you can access phone hardware should you want to.

But OK, this isn’t ideal for all situations. You won’t have offline access as such, you won’t be able to build games, you won’t get amazing performance - but for the most part, that’s not a big trade off for most business focused applications.

So, in short, why are we worrying about learning all these different tools to make our applications available on different mobile platforms, when we are already expert in the single tool that will make our app truly cross platform - the good old world wide web…

A List Of Ideas For The Next Killer Web Application

neil.middleton@gmail.com (Neil Middleton) — Thu, 15 Apr 2010 00:00:00 PDT

A List Of Ideas For The Next Killer Web Application

People, for some reason, always say how coming up with an idea for a web application is really hard, let alone building it.

Well, thing is, it’s not - you have a list in your head already. Just ask yourself:

What’s shit about my job?

Don't be dependant

neil.middleton@gmail.com (Neil Middleton) — Tue, 13 Apr 2010 00:00:00 PDT

Don't be dependant

Over the last couple of weeks it would appear that people are finally starting to realise that being dependant on another business is a bad idea.

For instance, last week Apple decided that it would cut off anyone not building iPhone applications using it’s own tools, cutting out Flash, Silverlight and a whole load of other people. The Flash community immediately stands up and proclaims how that’s bad and that Apple ‘aren’t allowed’ to block them out. Problem is, they are. Apple can do what the hell they like with the iPhone platform. They can block everyone out if they want, or insist that all developers paint themselves bright pink - it’s up to them, it’s their product, it’s their success.

Another thing that happened in the last week is Twitter acquired Tweetie, the iPhone based twitter client, with the aim of making this the official mobile client for Twitter on the iPhone. All the other twitter client authors out there immediately stood up and proclaimed how having an anointed client would put their own client out of business. Well, funnily enough Twitter are perfectly entitled to do this, it’s their product, it’s their success.

Overall, these simple moves have proved a simple point. Basing your business plan on anything that is dependant on another company completely is a really bad idea. If you’re in a position where you can be put out of business by another business making a single simple decision - you’re doing something wrong.

For instance, imagine that you’re a company who makes a little helper app for Windows. You see great success, and you’re dependant on the Windows product. However, Redmond wake up and see you’re selling loads of licenses, they realise that you’ve got something that is in great demand - so they build it into Windows itself as a native feature in order to reap the benefits.

WHAM!

You now have no business, no customers, and no future. You’ve been screwed in one easy move by another company and there’s nothing you can do about it.

So. Do your own thing. Make you own way. Do it the way you want to do it, and how you want to do it. And if you have to be dependant, make sure you’ve got a good Plan B.

Watch your users – they’re trying to tell you something…

neil.middleton@gmail.com (Neil Middleton) — Mon, 12 Apr 2010 00:00:00 PDT

Watch your users – they’re trying to tell you something…

Imagine this: You’re a web based company. You build a new site. You put it live and sit back and bask in glory.

You then forget about it until the next redesign.

If there’s one thing the vast majority of web companies do, it’s not keep tabs on the users of their sites, which are the whole reason the site exists in the first place.

Every single visitor to the website is giving you important information about your website. Every single user is clicking on things, moving their mouse around and searching for information. By listening to what these people are doing, you can learn a lot about where your site could be improved. For instance, that lots of people are clicking on some text that isn’t a link, or that people aren’t clicking at all on a magical image slideshow you’ve come up with.

If you’re not listening to this information, you’re losing out.

So, how do we do this? Well, first off, and easiest is to use your analytics tool as it’s meant to use. Tools such as Google Analytics don’t just tell you how much traffic your getting, but also where in the site the traffic is going and where it’s dropping out. It can also tell you about the occurrences of certain events that you can declare programatically (how many people have looked at my whizzy tooltip, how many have seen this mouseover etc).

Secondly, you can use heat mapping tools such as CrazyEgg. Apps like these show you where people are clicking, and where they’ve hovered their mouse, which gives you an insight into where on the page your users are moving – which then gives you ideas as to what parts of the page generate the most interest. By coupling this with your analytics you can work out where the weaker parts of your site/application are. You can see that people aren’t visiting a section of your site because no-one is noticing the links to it elsewhere in the site.

Overall, regardless of how you gather this information, your users are using your site and leaving behind evidence of how they are behaving. Not using this knowledge is wasting a very valuable resource that can only serve to make your site better as time goes on.

Why you shouldn't think of features to add to your application

neil.middleton@gmail.com (Neil Middleton) — Wed, 31 Mar 2010 00:00:00 PDT

Why you shouldn't think of features to add to your application

Following on from my post the other day on why your requirements are wasting your money, I thought there was one more point worth mentioning here about how people derive features in the first place.

Let’s imagine, you’ve built your application and have the bare-bones features required to make your application viable and a money-earner. What’s the first thing people tend to do? Well, they build on their basic idea. They brainstorm for all the features that users might want to make their application more useful, and more of a money spinner. They think about all the little shortcuts people might want, and all the little time-savers that are possible for the envisaged common use cases. They then package this stuff up and add it to the “requirements”.

Stop.

Do not dream up features for your app. Do not try and pre-empt what the users might want, and what features will make your app better. Just don’t.

Listen.

Your users will tell you what they want added to the app. Your users will tell you what shortcuts they want to take, and they will most probably moan unless you add stuff their way.

So don’t.

Don’t add features due to user feedback. Don’tadd features because one particular user keeps banging on about it. Take the features that you feel have bubbled to the top of the feedback pile, decide whether or not you think they make the application better, and then implement it how you want it implemented. Some users will hate you for it, some will love you for it, but ultimately you will end up with a better application as a result.

Above all though, you’d have saved yourself a bundle of cash by not implementing all the things that you thought people wanted.

Why your requirements are wasting your money

neil.middleton@gmail.com (Neil Middleton) — Mon, 29 Mar 2010 00:00:00 PDT

Why your requirements are wasting your money

Every single day, I see requirements from clients where they ream off a list of all the stuff that they would like me to build for them.

Generally speaking these lists are a vision of what the client see’s as their end product and a dream application that would satisfy their every need and then some. What every single one of these requirements sets has in common though, is that they tend to be unnecessarily bloated, and contain stuff that just isn’t that important. All they do is serve to push up costs and lengthen and complicate development.

Think about the word “requirement” for a second. There’s a reason for the word “require” to be in there. Requirements are for the stuff that has to, without fail, be in the end product. When coming up with requirements, as you write each one, ask yourself - “Does the product still function without this requirement being fulfilled?”.

If the answer is yes, take it out of your requirements and add it to the list of nice-to-haves that you might have lying around. Not only will this make your required costs much lower, but it will also serve to reduce complexity, reduce development time, and ultimately make your end product better as a result. Chances are it will also push you to drop some things that you previously thought were required, ultimately saving you your money and time.

Storing And Formatting File Sizes In Rails

neil.middleton@gmail.com (Neil Middleton) — Thu, 25 Mar 2010 00:00:00 PDT

Storing And Formatting File Sizes In Rails

I’ve just been looking at a problem relating to file sizes in a Rails application. Essentially, I needed to store a number of different file sizes in a config file, and show them to the user in as nice a way as possible. It would seem that this is something that Rails does out of the box, and here’s how. Rails provides a number of little functions in ActiveSupport::CoreExtensions::Numeric::Bytes that mean you can declare sizes really easy:

    10.megabytes
    50.gigabytes
    1.terabyte

and so on. Whilst this is a nice thing to type, Rails will then think about these items in bytes. This means that when you come to display these sizes you don’t get an exactly user friendly description:

    10485760
    53687091200
    1099511627776

Not ideal. But never fear, for ActionView::Helpers::NumberHelper is here, with number_to_human_size. Which pretty much does exactly what you want it to:

    number_to_human_size(10.megabytes) = "10 MB"
    number_to_human_size(50.gigabytes) = "50 GB "
    number_to_human_size(1.terabyte) = "1 TB "

Rails makes shit easy.

Do users get markup?

neil.middleton@gmail.com (Neil Middleton) — Thu, 25 Feb 2010 00:00:00 PST

Do users get markup?

These days, it doesn’t do to have a website that isn’t attractive, and easy to read, whilst making stuff easy to find. Yet, increasingly, websites are not looked after any more by users with any sort of technical knowledge - they’re all pounding away in the Content Management system they’ve been given. Thing is, there is a point where the users creativity will out-do the capabilities of the CMS they have (or simply FCKEditor, one of the most prolific editors). There’s a few options you can take here:

Tell them such creativity isn’t possible
Teach them HTML and let them edit manually
Teach them one of the many markup abstractions out there (Markdown, Textile etc)
Run away

But do users understand markup? For instance, I’ve worked with many clients in the past who know what they want to achieve, but unless the tools allow them to do what they want, they get frustrated - they have no way of moving around the issue. And we’re not talking complex stuff here, many systems won’t even let you do a simple float, or set a table column width, without getting into the code. So what are we, as CMS implementors to do? Do we rely on the user having some basic HTML skills? Or do we just have systems that restrict the user in what they want to do. HTML may be simple for web developers, but for normal people it’s an entirely new language.