After a long period of searching and deliberating, we have finally moved offices!

We have been looking for the right home over the last couple of years, and feel the new place is well worth the wait. We are now based on Exmouth Market, a lovely buzzing area close to Farringdon. There are plenty of reasons to swing past if you are near - not least excellent food at the stalls in the street. If you are around, feel free to pop up and say hi.

The full address is:

3rd Floor, Gensurco House, 46A Rosebery Avenue, London, EC1R 4RP (map)

There are some photos of the place, and us moving in here. Plus we’ve also make a few tweaks to our website which includes adding recent clients and some more information about how we work.

On Wednesday 22nd of April we’re going to be holding a Ruby on Rails Technology Briefing session at Claranet’s offices (address below) in London.

The session will be an informal event designed for web developers, online publishers, technology strategists and business managers to come together to learn more about Ruby on Rails as a rapidly evolving framework and explore its relevance to doing business online.

Key speeches for the evening include:

• Rails Inside Claranet: Chris Cooke - Claranet’s Rails developer & managed application hosting guru - talks about his experiences in developing Claranet’s (very large) Rails application – an internal system which is being used across all of our European offices.

• Making the Business Case for Rails: Made by Many co-founder talks about his experiences in selling Rails as a strategic asset to both internal stakeholders and clients.

• Best Practice for deploying and managing Rails: Laurie Young from New Bamboo will talk about their experiences in deploying and managing one of Channel Five’s largest websites – and recommends best practice for managing Rails in hosted environments.

…and importantly, drinks will follow the speeches in a bar around the corner.

Event Details:

Claranet Ltd, 21 Southampton Row, London WC1B 5HA Wednesday 22nd April

• 6:30 Arrive, registration & refreshments
• 7:00 Welcome & Briefing
• 8:30 Q&A
• 8:45 Drinks

Seats are limited, register here: http://www.uk.clara.net/rails-tech-briefing

To help with the upgrade path the rails command prompts you to overwrite, keep or diff with the latest files. However this isn’t always what you want - quite often you want to merge changes instead of just overwriting them.

So I thought I would experiment with a new method, using the power of git to handle the merging of the old and new. First I created a branch to do the upgrade in, so that I can commit and make sure everything is working before I merge back into master.

git branch rails_upgrade

Secondly, I create a blank branch imaginatively named fresh_rails_2.3.2. Then we remove all of the files from this branch to give us a blank slate.

git symbolic-ref HEAD refs/heads/fresh_rails_2.3.2
git rm -r --cached .
rm -rf *
rm .*

The next step is to generate a blank rails application (assuming you have the latest gems installed) and commit it to our detached branch.

rails .
git commit -m 'Generate Rails v.2.3.2 skeleton.'

Now we have the latest and greatest files we can merge back into our rails_upgrade branch.

git co rails_upgrade
git merge fresh_rails_2.3.2

Unless you have barely touched your application since it was previously generated, there should now be a number of conflicts that you need to resolve. Fix each one in turn and add it to the staging area. For example:

mate config/environment.rb
# Fix the conflicts and save
git add config/environment.rb

Note: If you have previously ignored any files from you application (config/database.yml for example) you will need to remove them from the staging area otherwise they will be added back into your project.

git reset config/database.yml

With the merge complete you can run your tests to make sure the merge was a success. If everything passes you can now merge the upgrade back into the master branch.

rake spec
rake features
git commit
git co master
git merge rails_upgrade

GitX shows my history looking like this. Overall I think the experiment worked quite well and it was defiantly useful to be able to manually pick and choose which bits of the new files I wanted.

Design becomes beautiful when there’s nothing left to take away. Google understands this; too many other web applications don’t.

Many apps grow new features simply to differentiate themselves from the competition, or to increase their target market by supporting multiple conflicting workflows. But every single thing I don’t need detracts from the usability of the things I do. In the past I’ve had to abandon apps that do more than I need, but not any more.

Now I can fix them.

Stylish is a Firefox plugin that lets you write your own css per-site, overriding the existing css. This means you can set display:none on everything unnecessary.

It’s a one-click install. You’ll also need Firebug to assist in identifying elements you want to block.

Once it’s installed, open firebug and point to the offending feature. Figure out a way to identify it with a CSS selector. In some cases this’ll be easy - it might have a classname like AnnoyingFeature. In others you’ll have to use CSS3 selectors. For example, you can do:

label[id*="requested_by_id"] {display:none}

This matches every Label element with an ‘id’ attribute containing ‘requested_by_id’. This is helpful if the site’s using ids like ‘story_34524_requested_by_id’.

But what about if you had this?

<div>
  <span class='annoying'>annoyance 1</span>
  <span>annoyance 2</span>
</div>

It’d be nice to block the parent div, thereby taking out ‘annoyance 2’, but you can’t; there's no "parent selector" even in CSS3. Greasemonkey could fix this; Stylish can’t.

While you’re there, you can also fix other usability issues like textboxes being too small - it’s as simple as ‘textarea {height:300px}’.

Stylish has a live preview, and it’s only one click to turn off custom styles, so you can afford to be both aggressive and experimental.

You can share custom styles. I’d like to suggest that styles solely designed to remove distractions, rather than ‘re-skin’ sites, include ‘Simplify’ in the name to make them easy to find. Over the next few days I’ll be redesigning some of the web tools I use frequently to make them more of a joy to work with.

Remote scrums are hard work. The first clue that something was wrong was when our sole remote coder volunteered to stop attending the scrum because he didn’t find it useful.

This is a danger sign. Further probing determined that it wasn’t that the format was wrong; it was just that he couldn’t hear anyone. We’re using Skype video calls, with the laptop sitting on a chair in its place in the circle, using the Macbook’s built-in mic and speakers. While the mic’s okay for 1:1 conferencing, it’s not sensitive enough for people standing a couple of metres away.

No problem; I bought a £15 omnidirectional mic from Amazon. But now he couldn’t hear us at all.

It turns out that Macbooks - even the new ones - don’t have a mic-in port. They have a line-in port. Microphones work at much lower voltage, and can’t be used there.

Okay. We have a Plantronics headset adapter that inputs a stereo jack and outputs USB. It doesn’t work with the new mic, though.

So we needed a preamp. I bought an iMic; £30 of glossy white plastic that also inputs a stereo jack and outputs USB. It even has a “mic mode” switch that activates the internal preamp. Should be perfect, right?

It doesn’t work. This omnidirectional conference mic is not the same as the 1:1 headsets that the iMic is designed for; it’s a ‘boundary microphone’ that provides a weak signal even for a microphone. It turns out that microphones are described in terms of impedance, measured in ohms, and divided into ‘low-impedance’ (±1000Ω) and ‘high-impedance’ (±10000Ω). A high-impedance mic, like ours, won’t work with the iMic and requires a more powerful preamp.

I’m still chasing that down. I’m not looking for production-quality here, just something good enough for intelligible speech.

In the meantime, we’re passing the mic around - it works well enough if we speak directly into it, just not in ‘omnidirectional’ mode. This actually has some benefits, since it acts as a token - only the person with the mic speaks - and makes us all feel like radio announcers. We’re hoping to get a USB speakerphone, which ought to be fully integrated, soon.

In 2009, setting up a video conference should not require an audio engineer. But it does.

Classic blogging apps were never quite up to the job in my opinion. The editing interface is just a textarea or a broken wysiwyg editor and versioning is shoehorned into a database. Since moving our wiki from an app to simply markdown files on Github we haven’t looked back, so why not the blog?

The Jekyll is a neat little static site generator which can also help with maintaining a blog. Migrating over required the use of a few scripts. Firstly to dump the old blog posts in Mephisto into markdown files, and secondly for comments. Having a statically generated site also means comments need to be hosted somewhere else. We went with Disqus which provides a pretty good js commenting service, and used a script to migrate the Mephisto comments over there.

I also thought it would be nice to have all the code snippets hosted at gists.github.com, but didn’t want to include them all using Javascript, so I wrote a little liquid filter to pull them directly into a page when the site is generated. These tweaks are all in the newbamboo jekyll branch.

Our Jekyll has the following tweaks

• Date format in urls to match Mephisto so the urls didn’t break
• Some extra lists of posts for use in the templates (our front page has 3 full text posts, followed by 5 titles and finally an archive page with the titles of all posts)
• date_to_rfc2822 for use in the rss feed

Jekyll

First step is to setup Jekyll locally, which is all covered by the readme in the repo. You’ll see that posts are simply .markdown files in the _posts directory, so if you’re migrating an old blog you’ll need to dump all your data there.

When you run the jekyll command inside your blog’s directly it’ll then convert all the markdown files in the _posts dir into html, using the little bit of yaml at the top to set the layout and provide variables for the liquid layout (e.g. we have put author in the yaml of all of our articles). If you then put a .html file in the root of your blog dir it will get run through the liquid generator after the posts have been generated, so it has access to the list of all posts etc.. This is how the index.html and archive.html pages are done.

Layouts live in the _layouts dir. Our default.html and post.html layouts were left pretty much untouched from our Mephisto theme.

Migrate articles from Mephisto

The following script is a quick hack to write all of the articles from our old MySQL Mephisto database. It’s going to need tweaking if you’re going to use it with anything else.

articles = Article.find(:all)
 
articles.each do |article|
  published_at = "#{published_at.year}-#{published_at.month}-#{published_at.day}"
  author = article.user.login
  permalink = article.permalink
  title = article.title
  excerpt = article.excerpt
  body = article.body
  fp = File.open("_posts/#{'draft-' if article.published_at.nil?}#{published_at}-#{permalink}.markdown", 'w')
  fp.write(%"---
layout: post
title: \\"#{title}\\"
author: #{author}
---
#{excerpt}
 
#{body}")
  fp.close
end

Migrate comments from Mephisto to Disqus

This script was modified a bit from the original. It’s a bit of a kludge because Disqus doesn’t allow you to delete comments via the API, and they only just added a feature in the admin to remove a site from your account. You may find yourself running this several times before you get that single clean import.

# Copyright 2009 Michael Ivey, released to public domain
# Disqus guts lifted from http://github.com/squeejee/disqus-sinatra-importer/tree/master
# I wanted it to run from MySQL and command line, instead of a Sinatra app
 
require 'rubygems'
require 'rest_client'
require 'json'
require 'sequel'
 
disqus_url = 'http://disqus.com/api'
user_api_key = ''
forum_shortname = 'myblog'
current_blog_base_url = 'http://myblog'
db = 'mephisto_production'
db_user = 'root'
 
resource = RestClient::Resource.new disqus_url
forums = JSON.parse(resource['/get_forum_list?user_api_key='+user_api_key].get)
forum_id = forums["message"].select {|forum| forum["shortname"]==forum_shortname}[0]["id"]
forum_api_key = JSON.parse(resource['/get_forum_api_key?user_api_key='+user_api_key+'&forum_id='+forum_id].get)["message"]
 
db = Sequel.mysql(db, :user => db_user, :host => 'localhost')
query = "SELECT comments.body, comments.author, comments.author_email, comments.created_at comment_created_at, articles.title, articles.permalink, articles.published_at article_published_at FROM contents AS comments LEFT JOIN contents AS articles ON comments.article_id = articles.id WHERE comments.type = 'Comment' AND comments.approved = 1 AND articles.published_at IS NOT NULL ORDER BY comments.created_at DESC"
 
db[query].each do |comment|
  article_url = "#{current_blog_base_url}/#{comment[:article_published_at].year}/#{comment[:article_published_at].month}/#{comment[:article_published_at].day}/#{comment[:permalink]}"
  
  thread = JSON.parse(resource['/get_thread_by_url?forum_api_key='+forum_api_key+'&url='+article_url].get)["message"]
  
  # If a Disqus thread is not found with the current url, create a new thread and add the url.
  if thread.nil?  
    thread = JSON.parse(resource['/thread_by_identifier'].post(:forum_api_key => forum_api_key, :identifier => comment[:title], :title => comment[:title]))["message"]["thread"]
    
    # Update the Disqus thread with the current article url
    resource['/update_thread'].post(:forum_api_key => forum_api_key, :thread_id => thread["id"], :url => article_url) 
  end
  
  # Import posts here
  if resource['/create_post'].post(:forum_api_key => forum_api_key, :thread_id => thread["id"], :message => comment[:body], :author_name => comment[:author], :author_email => comment[:author_email], :created_at => comment[:comment_created_at].strftime("%Y-%m-%dT%H:%M"))
    puts "Success: #{comment[:author]} on #{comment[:title]}"
  else
    puts "FAIL: #{comment[:author]} on #{comment[:title]}"
  end
end
 
 

Use Google for search

With no database we need to handle search elsewhere as well. Google does a pretty good job, so we changed our search form to submit to site search, and then setup an nginx redirect (see next section) so any indexed searches would end up in the right place. See the search form in our index.html for how it’s done.

Nginx redirects

Nginx is used to serve up the static site. We had to setup a several redirects to indexed pages from the Mephisto days still worked.

server {
  listen 80;
  server_name  blog.new-bamboo.co.uk;
 
  client_max_body_size 10M;
 
  root /usr/local/www/nb_blog_jekyll/_site;
 
  rewrite ^/archives/.* /archive permanent;
 
  location /search {
  if ( $args ~ "q=(.*)" ) {
        set $args "site=&hl=en&q=site%3Ablog.new-bamboo.co.uk+$1";    
      rewrite ^ http://www.google.com/search permanent;
    }
  }
 
  location / {
    if (-f $request_filename.html)       { rewrite (.*) $1.html break; }
  }
 
  default_type  text/html;
}

The search redirect is pretty interesting and took some fiddling to get right. It takes an old search url like “http://blog.new-bamboo.co.uk/search?q=rails” and redirects it to Google site search with the same query string.

We tried rephrasing the questions - “Of what you did yesterday, what gotchas do the rest of the team need to know about?”.

We tried aggressively intercepting conversations that should be taken offline, using postit notes as placeholders.

We tried ‘just trying to hurry’.

It turns out that all we needed was a ticking bomb. As the team’s scrum master, I start this mp3 playing through my phone’s speakers when each person starts speaking. It ticks - a reminder to hustle, plays a warning at 45 seconds, and sounds an alarm at 60 seconds. Being constantly aware of the time means that most people don’t hit the alarm.

It doesn’t make people speak much faster; it just makes them filter what they’re saying more effectively. More conversations are taken out of the daily scrum - sometimes even whole-team conversations, and sometimes they start immediately after the scrum - but this is preferable to extending the whole-team, mandatory standup.

So far this is working well; in the future, I hope that I can ‘forget’ the timer and find that nothing changes.

Tourdust is a website for independent tour operators to advertise their diverse and interesting experiences for free. It is centred around the kind of activities which the packaged holidays don’t offer; the real gems which lie slightly off the beaten track. Many of the people who run these tours have little access to a global audience, and building and promoting their own websites is often a problem.

I hope people will agree that the design is fantastic, it was produced by our long term Brighton pals Clearleft.

The project was run in an Agile manner, and I think that it illustrates many of the advantages of working in this way. The feature set is refreshingly minimal, and this produces a much better user experience overall. We were constantly re-evaluating the priorities throughout, and anything which wasn’t critical to the site’s function was cut out.

I think this is a win-win situation for both our client, and for us. They have got the most concentrated features to meet the requirements of their mission statement, and we have been spared the pain (which so often occurs in software development) of building things you are sure are never going to be used.

So check it out, you might even find a good reason to go off on holiday!

“When I walk in here, I don’t have to ask how it’s going”.

Our tech lead can walk in, scan the room, and see in seconds that we’re slightly ahead of schedule, that the performance data is good but needs checking, and that we won’t be deploying today until 4pm.

And because it’s easy, and it’s interesting, he does. “Write on the walls” is immune to the “ghost town” effect that afflicts most corporate intranets. Everyone likes drawing, and a bright red “WALL OF SPEED” is a much more fun way of communicating benchmark results than an Excel spreadsheet buried somewhere in source control.

It does need maintenance. If I walk past something that I know is out-of-date, I’ll tear it down. Every week we clean up the office more formally. We go through a fortune in stationery.

This only works, of course, because we’re all in the same room. For a while our team spent four days a week at the client site, and Fridays in the company office, but we’re so crippled without our War Room that we hardly move any more.

But we have to be anyway. Any web project needs constant, constant communication between graphic designers and developers, and instant messaging doesn’t cut it. There are few other areas in IT where the required specialties need such different mindsets, but are so dependent on each other. Sitting side-by-side takes all the heat out of a suggestion of web-safe fonts where an email could be taken as a challenge to the designer’s authority.

Version control is a developer skill, and with sixteen different producthomepage.psd files floating around, it was hard to keep track of the definitive version. We solved the problem by printing out the designs - all of them - and sticking them on the wall. Areas that need discussion are marked with yellow post-it notes. While I can find and open a photoshop file in thirty seconds, I can turn my chair in two. And visitors to the work room can see exactly what, and how, we’re doing.

With talk of distributed teams, drag-and-drop task management and ubiquitous video conferencing, I can’t help but feel that the industry is moving in the wrong direction. So if your team isn’t really using their internal communication tools, try turning them off. Move everyone into one room. Clear the walls. Bring in a big box of pens, paper, and bluetack. See what happens.

We tried Speed Poker without success; two developers were playing ’?’ cards or wildly varying estimates. The estimating took a half-day, using up all the time we’d saved at the workshop.

The User Stories Workshop isn’t just about writing story cards; it’s about loading a whole lot of extra data, that doesn’t make it on to the cards, into the brains of everyone there. I still think we can optimise the process, since the developer-developer interface is higher bandwidth than the client-developer interface (and much higher than the meeting-developer interface). But in cases where we do exclude some people from the workshop, we need to rigorously schedule a developer brain-dump immediately afterwards.

We can play without cards. For short sessions, or when we need to re-estimate on the fly, fingers behind the back work perfectly. Big stories need both hands. Hands behind the back means ”I’m ready to estimate”, a crooked finger means “half a point”, open palms means “I don’t know”. This works well in release planning when we’re discussing descoping a card; the client agrees to a change, and we do a new estimate on the spot.

When we use cards, we use good ones - presented to us by a client, our Planning Poker decks have been custom-printed by moo.com and feature a bonus “WTF?” card as well as the usuals.

We play fast. The ‘classic’ approach is to talk about a story, explore all the options and implications, then finally estimate. When we do this, we nearly always get matching estimates. But it’s time-consuming, and if we’re not planning to implement the story straight away (in which case we needed to have the conversation anyway), it’s wasteful.

So when there’s a big stack of cards to get through - like after a User Stories Workshop on a new project - we play Speed Poker. We select a card, read it verbatim, and estimate. If the estimates all match up, we move to the next card - we were all thinking the same thing, and we’ve saved the time needed to have the conversation. If they don’t match up, we use the classic approach for that card.

We play commitment poker. For new projects, or those without a stable velocity, we use commitment-based planning rather than velocity-based planning. That is, after each story is added to the plan, we check that each member of the team is still comfortable committing. This suffers from the same ‘peer pressure’ effect as normal estimation, though, and we solve it the same way: on three, thumbs-up means ”I’m sure we can do this”, thumbs-down means ”I’m no longer sure”.

Above a certain level in the management hierarchy, nobody’s ever really 100% present in any case. At our last workshop, both high-level stakeholders had mobiles out, dealing with incoming emails that are more polite, but not much less disruptive, than incoming calls.

So we extended the coffee break to a fifteen-minute Blackberry Break. Those with urgent issues dealt with them. Everyone else had an out-of-band chat about how the process was going, and dealt with other todo items like external dependencies and scheduling.

We returned to the meeting with everybody focused. Going forward, we might even make this explicit: “there’ll be a Blackberry Break every half hour”. For product demos, we could go as far as “this is a fifteen-minute demo. Phones off, please”.

Singletasking: it’s the future..

A few months ago we transitioned from hosting all of our apps in development on two Slicehost slices to individual Amazon EC2 instances for each. As much you want vendor frozen Rails and Merb to work (including all your app’s dependencies) it’s always plagued with issues. Having completely separate environment for each app clears up a lot of issues, including the all to often mistake of upgrading Rails only to find some old project isn’t able to co-exist with it. Our EC2 setup involved creating a base image with the basic Rails environment, and whenever a new staging server was needed we would clone the Base image and modify it specifically for the app. Being able to startup new environments with a single command is wonderfully easy, but a dozen staging servers later, the cost certainly starts to add up.

So for the cost of 1 month of our EC2 instances, we purchased a Dell box with 8Gb to run as an internal staging server and host all of our virtual servers. We chose Ubuntu as both the host and guest operating system. We started out trying KVM which is officially supported by Ubuntu. Everything went to plan and we had a great setup except for one major problem: slices would randomly lock up. After no success finding a remedy we decided to give Xen a go and with a little work got it setup and running along nicely.

Most of the instructions here have come from a combination of the Ubuntu and Virtuatopia wikis.

The first step was to install Ubuntu 8.10 (Intrepid) Server Edition (remember to select the “Server Edition” tab before downloading).

Setup Xen

Detailed instructions are on the Ubuntu wiki Xen page

sudo apt-get install ubuntu-xen-server

Install Xen kernel from Hardy

Grab the .deb from here: http://packages.ubuntu.com/hardy-updates/linux-image-2.6.24-21-xen

sudo dpkg -i linux-image-2.6.24-21-xen_2.6.24-21.43_amd64.deb

Make sure your eth0 is configured.

Reboot.

Using xen-tools to create images

We should now have an Ubuntu server running the Xen kernel with all the standard Xen tools installed.

Next we’ll create a base Xen image running Debian Etch. In an ideal world it would be running Ubuntu 8.10 (Intrepid), but there isn’t yet a xen-tools template for Ubuntu, so Debian has to do.

Configure xen-tools:

vim /etc/xen-tools/xen-tools.conf

This is our config (installing Debian Etch):

dir = ~/slices
install-method = debootstrap

size   = 4Gb      # Disk image size.
memory = 256Mb    # Memory size
swap   = 512Mb    # Swap size
fs     = ext3     # use the EXT3 filesystem for the disk image.
dist   = etch     # Default distribution to install.
image  = sparse   # Specify sparse vs. full disk images.

dhcp = 1

passwd = 1

accounts = 1

kernel      = /boot/vmlinuz-`uname -r`
initrd      = /boot/initrd.img-`uname -r`

arch = amd64

mirror = http://ftp.us.debian.org/debian/

ext3_options   = noatime,nodiratime,errors=remount-ro
ext2_options   = noatime,nodiratime,errors=remount-ro
xfs_options    = defaults
reiser_options = defaults

If you want to include anything extra in the image, you can put files in the /etc/xen-tools/skel directory. We’ve written a script to configure the slice for Rails with Apache running Phusion Passenger.

Grab the boot_xen_passenger.sh script, and copy it to /etc/xen-tools/skel if you plan on using it.

sudo vim /etc/xen-tools/skel/boot_xen_passenger.sh 
sudo chmod +x /etc/xen-tools/skel/boot_xen_passenger.sh 

Create your first slice:

mkdir ~/slices
sudo xen-create-image --hostname=xentest1

Start image:

sudo xm create /etc/xen/xentest1.cfg

View slices with:

sudo xm list

Attach to the console with:

sudo xm console xentest1

SSH in and run the setup script:

sudo ./boot_xen_passenger.sh

Networking

Hostname resloving

Dnsmasq is a nice little dns server which will allow us to reference our slices by their hostnames.

sudo apt-get install dnsmasq

Dnsmasq can also act as a dhcp server. When you slices ask for a dhcp lease they will also give dnsmasq their hostname. This way you won’t need to reserve ip addresses for slices. However we’ve had some issues with the dhcp client on Debian when used in conjunction dnsmasq. In some cases you will want to forward certain ports from the outside world to a slice, and if have a dumb router it will only be able to forward to a specific ip.

If you wish you setup the dhcp server in dnsmasq:

sudo vim /etc/dnsmasq.conf

Uncommend the line #dhcp-range=192.168.0.50,192.168.0.150,12h and edit the ip range if desired.

If you use a separate router for your internet, you’ll want to set the gateway options as well by uncommenting the following line dhcp-option=option:router,192.168.1.1 and replacing 1.2.3.4 with your router’s ip.

Now restart dnsmasq:

sudo /etc/init.d/dnsmasq restart

If you have an existing dhcp server on the network you will need to change it to forward requests to the new dnsmasq dhcp server. This is to ensure that new slices ask dnsmasq for an ip address, and when doing so it will also broadcast its hostname. On the local network this will allow you do to nice things like http://myslice and ssh myslice.

However if you’ve opted not use dnsmasq as your dhcp server, you will need to add a line to the host’s /etc/hosts file with the hostname of the slice and its ip address (every time you edit /etc/hosts you will also need to restart dnsmasq). You will also need to specify the ip address of your slices when they are created. See the Configuring Xen Guest Network Options wiki page for more information about configuring xen-tools.

On your router or for individual clients you will also need to add the ip of the server with dnsmasq setup as a DNS server.

Magic subdomain forwarding with nginx

Any slice we startup should now be accessible by its hostname. To reduce the configuration required for each slice, we setup Nginx on the host to automatically forward http requests from any subdomain under our staging domain to the slice with a hostname the same as the subdomain. So that if you visit http://myslice.mystagingdomain.com from anywhere, it will be internally redirected to http://myslice and allow all your slices to be behind one ip.

Setup nginx

Grab nginx:

sudo apt-get install nginx

Then add a new site config for the slice forwarding:

sudo vim /etc/nginx/sites-available/slices_proxy.conf

Here is the config we have:

map $host $backend {
    hostnames;
}

server {
  resolver 127.0.0.1;
  listen   80;
  server_name  _;

  access_log  /var/log/nginx/localhost.access.log;

  location / {
    if ($host ~* "^(.*)\.mystagingdomain\.co\.uk") {
      set $host_without_www $1;
      proxy_pass http://$host_without_www$request_uri;
    }
  }
}

Enable the config:

sudo ln -s /etc/nginx/sites-available/slices_proxy.conf /etc/nginx/sites-enabled/slices_proxy.conf

Best to also remove the default config:

sudo rm /etc/nginx/sites-enabled/default

Then restart nginx:

sudo /etc/init.d/nginx restart

Setup DNS

Next add an wildcard A record to your mystagingdomain.com domain to point to your host’s external ip, and all should be good to go.

Next steps

The instructions here should get you started with a local Xen setup. However there’s still a few of the finer points to perfect. The Rails setup script could be integrated with Xen better so it’s automatically run when a slice is created. The script itself is also very brittle and likely to fail on another environment. Using deprec would probably be a better solution in the long term.

You will start by storing your content in an Amazon S3 bucket and then marking the content as publicly readable. Next you’ll make a single API call to register the bucket. The call will return a domain name that you’ll use to refer to your content in your web page or application. When clients request the object via the returned domain name they’ll be routed to the nearest edge location, for high performance delivery.

Earlier in the year, we found many of our clients were looking for a solution to allow users to easily upload their own videos. We had a good look into the numerous existing services already available, but ultimately found none quite fit the bill; either being too expensive, closed source or overly complex. Panda was conceived to be a simple open source solution for handling online video. Essentially, Panda allows you seamlessly integrate video uploading, encoding and streaming into your web application, while utilizing the power of the cloud.

We took inspiration from what was already out there and distilled it down to its core. Panda runs entirely on Amazon’s Web Services utilising EC2, S3 and SimpleDB and the Merb framework. Once you’ve started up the application on an EC2 instance you can access the API and administration backend. The REST API allow painless integration with your own web application. Videos are uploaded directly to Panda and transcoded using FFmpeg to any format you desire. Video can be streamed using any Flash video player such as the JW FLV Media Player, and with the addition of h264 support in Flash 0.9.115 users can enjoy full HD quality.

So, visit the Panda site, grab the code from Github, read the Getting Started Guide and don’t forget to join the Google Group if you have any questions or success stories to talk about!

We don’t normally use the blog to announce the launch of client websites, but it might interest some people to check out some of the work we actually do. With this in mind, I’d like to introduce you to Protect The Human, a recently launched project for Amnesty International UK that we have been working on for the past few months in collaboration with our design peeps up in London Made by Many. Though it has been publicly available for a couple of weeks now, I wanted to let the dust settle a bit before posting.

So what is it? It is different things depending on who you talk to, but for me it is an assortment of different parts, playing a key role in Amnesty’s online strategy to increasing activism (catchy, eh?). It is firstly a content publishing platform; populated by fantastic Amnesty videos, galleries of images about the work they do, and masses of bookmarks to fascinating and current content all over the interweb. But this is not purely a top down publishing platform, users of the site are actively encouraged to create their own content, and rate comment and discuss any item that is on there.

This is a site about activism: the small things people can do which collectively create interest for others and effect change on important issues. Users are prompted to take various actions at every opportunity, depending on the time they have available. Examples of actions can be signing online petitions for the victims of Human rights abuses, attending protests, or simply sending great content to friends to help spread the word. This aspect of the site has been a real eye opener, I have never before seen so many opportunities to take proactive action on such important issues in such an accessible way.

There are other parts to this site, such as hubs for the various campaigns Amnesty UK are engaged in, and rather than me describe it all, the best thing to do is to have a poke around. Because we worked in an Agile way, there are still more features which are going to be rolling out soon, and we hope to be iteratively improving the site over the next few months.

We will probably be talking about how the Agile process worked in practice on this at some point, but that will have to wait a bit. For the moment, have a look, sign up, and take some actions. There really isn’t any excuse not to, the site makes it so easy to make a contribution, whatever that might be.

From my time at various web 2.0 start-ups, one of my gripes with the industry was that too many start-ups were focussing on advertising as their sole revenue stream, gunning to grow big and sell to GYM (Google Yahoo Microsoft), and hope that whoever bought them would figure out how to make them profitable. This strategy has worked for some, but I doubt that it will work for all. There are start-ups out there who may have to find other ways to make money, rather than just chase VCs and old media conglomerates, and shovel the cash on a bonfire.

My topic today is about an xml file format called APML, used to store attention profiles of users. The technology is not complex, and there are plenty of places where it could be implemented. What’s more, the interesting thing about APML is that is has the potential to turn a lot of the data that web 2.0 startups have into valuable information that can be sold as a commodity. In other words, a revenue stream that isn’t “AdBucks saves the day!”

Introducing APML

APML is an xml file format for Attention Profiles, and Attention Profiles are lists of things that you are interested in, with a value signalling how interested you are in that thing. Examples of attention profiles include:

• A list of your favourite music artists collected by Last.fm, ranked by how often you listen to them.
• A list of topics that you like to Twitter about, ranked by a count of how many times you mention each topic.
• A list of products that I looked at and purchased / bid on from Amazon or Ebay, ranked by clicks.
• A list of urls I have bookmarked via del.icio.us, ranked by the number of visits to each link.
• A list of articles I have read on BusinessWeek, ranked by the number of visits to each article.

Let’s grab a real example so you can see what I mean. Here’s my my last.fm profile. Notice the list of top artists, ranked by the number of times I’ve played songs/albums by those artists. And here is the APML file of the same data. There we go, an APML file of my favourite musicians/music categories, courtesy of SunLabs and Last.fm.

Comparing the two sources, you get a better idea of what APML does to Attention Profile data. Here’s a diagram of the schema for APML:

If you like all the juicy details, here you go. In the meantime, I’ll give you a quick runthrough.

The head element contains some basic descriptive data about the Attention Data, who it belongs to (the email), what application / site generated it, and when. Then in the Body element, we have 2 sub-elements; profile, and applications. Profile is where the real meat is, it’s essentially a grouping of the Attention Profile data, in my example Last.fm provided a list of all the music I like, and the music I have listened to in the last week.

The Profile element contains 2 sub-elements; ExplicitData, ImplicitData. Explicit and Implicit Data are similar in their schema, meaning that they store the same kind of data, but the difference is this; Explicit data is data that the user generated and gave to the application/site (in my last.fm example, that is the list of musicians I listened to), and Implicit data is data that the application generated (the list of music categories that I liked). I tell last.fm what artists I like (Explicit), and last.fm tells me what music categories I listen to (Implicit).

The Explicit/Implicit data element contains 2 sub-elements; concepts, and sources. Concepts are a collection of elements representing the thing that you are interested in, such as a music artist, and a value representing your level of interest in that item. The value is ranked from 1 to 0, with 1 being the thing which you are most interested in, and everything else being compared against that top concept. Sources are a collection of the sources of the data stored in concepts, such as blog sites, web applications, etc.

Jumping up a couple of levels, the Application element is used to store information about the application which generated the APML file, as well as any meta-data that the application carries with it. I have to put my hands up here and admit that I don’t fully understand this element, so you’re best bet is to flick through the juicy details rather than take my vague description as gospel.

Why its good

All APML does essentially is add structure to information about what we as people are interested in, which for its own sake seems a rather pointless venture. However, as Orwell once said, knowledge is power, and what APML offers is a number of opportunities to turn an excess product of web 2.0 into a commodity of value.

Finding out what stuff people are interested in and organising that information has proven to be useful information to businesses, just ask Tesco why they place beer next to nappies*. Web 2.0 start-ups are collecting a massive amount of this raw data about their users, but not necessarily putting it to full use.

But who’s buying the data? That’s where I get creative and illustrate to you a business idea that I think could work for an APML analytics service…

This diagram might be worth 1000 words, but I’ll explain it anyway (accessibility ; ) ). Various web services which collect data about their users convert that data to APML, and post it to us. We take it, store it in a big database alongside data from other providers, and then do 3 things with that data:

1 - Provide an analytics service to interested 3rd parties, advertisers and publishers to analyse a large data-set and spot interest patterns across a whole range of consumer products (nappies and beer?).

2 - Provide a recommendation engine service to interested web services looking to pool a large data-set to get highly-accuracy suggestions to users.

3 - Provide (and this is where I shoot myself in the foot) an Advertising Management service that suggests the most relevant ads based on the user’s APML data across various services.

The general idea for the revenue stream here would be to charge for these 3 individual services, and apportion a percentage of that revenue to the APML data providers, based on how much of their data was used by the customer. That’s really all there is to it.

Summary

APML helps to standardise data about what users are interested in, and in doing this, an opportunity might be open to build a web-based service that can sort all of that data and spit it out into a neat little analytics service for 3rd parties to analyse to bits, alongside the recommendation engine to serve to interested web services, as well as the (choke) Ad management service to suggest relevant ads. All in all, this could be a way to provide those web 2.0 services with a revenue stream where they provide a web service with APML, and get money back in return.

To explain the nappies and beer reference, Tesco’s loyalty card scheme provides a wealth of information about consumer purchase patterns, one of which was that men in their 30’s would be visiting their stores on a Friday night to buy nappies of all things (as my sister’s partner must be finding out). They strategically placed beer next to nappies, and boosted their sales.

Some recent picks from the archive:

• HangUp Meetings - An improvement on XP’s stand-up meeting…
• Picture Bong - Graphs for the Bong benchmarking tool…
• ManEC2 - Keep tracks of all those EC2 instances you have…

We weren’t happy with our forms and we were writing too much code. So we wrapped some goodies around the merb form helpers. We’ve got notes, inline errors, required / not required formatting, and cancel buttons. The API is so minimal you won’t be able to resist.

Here’s an example of the ruby code, the generated HTML, and what it looks like with some sexy CSS:

Example Code

Ruby

<%= field(:text, :nickname, :required => true, :note => "You'll be known by this on the site. Don't use your real name. It has to be unique, and you can't change it later.")%>
<%= field(:password, :password, :required => true)%>
<%= field(:password, :password_confirmation, :required => true, :note => "Type the same password again, just to make sure we've got it right" )%>
<%= field(:checkbox, :terms_and_conditions, :required => true, :value => true, :label=>"I accept the terms and conditions")%>
<%= form_submit "Sign Up!" %> 

Html Generated

<div class="field password required">
<label for="user_password">Password</label>
<input type="password" class="password" name="user[password]" id="user_password" />
</div>

<div class="field password required">
<label for="user_password_confirmation">Password confirmation</label>
<input type="password" class="password" name="user[password_confirmation]" id=
"user_password_confirmation" />

<p class="note">Type the same password again, just to make sure we've got it right
<abbr title="smile">: )</abbr></p>
</div>

<div class="field checkbox required">
<input type="hidden" class="hidden" name="user[terms_and_conditions]" value="0" />
<input type="checkbox" class="checkbox" name="user[terms_and_conditions]" value="1"
id="user_terms_and_conditions" />
<label for="user_terms_and_conditions">I accept the terms and conditions</label>
</div>

<div class="field controls">
<button type="submit" class="positive">Sign Up!</button>
</div>

Html with Errors

<div class="field error password required">
<label for="user_password">Password</label>
<input type="password" class="error password" name="user[password]" id="user_password" />

<p class="error">can't be blank and is too short (minimum is 4 characters)</p>
</div>

<div class="field error password required">
<label for="user_password_confirmation">Password confirmation</label>
<input type="password" class="error password" name="user[password_confirmation]" id="user_password_confirmation" />

<p class="note">Type the same password again, just to make sure we've got it right
<abbr title="smile">: )</abbr></p>

<p class="error">can't be blank</p>
</div>

<div class="field error checkbox required">
<input type="hidden" class="hidden" name="user[terms_and_conditions]" value="0" />
<input type="checkbox" class="error checkbox" name="user[terms_and_conditions]" value="1" id="user_terms_and_conditions" />
<label for="user_terms_and_conditions">I accept the terms and conditions</label>

<p class="error">must be accepted</p>
</div>

Sexy Form

You can be the judge of the sexyness of these forms, but they are much nicer than the default error message in my opinion.

Code/CSS

Code is licensed under the MIT License, and under no guarantee that it will not break if they change the external API for form helpers.

• For Merb and AR

• For Merb and DM

• The CSS

Install

Add the file in your lib folder, add the dependency in your init.rb and then include in your global helpers.

Enjoy!

APICache.get("http://twitter.com/statuses/public_timeline.rss")

You get the following functionality for free:

• New data every 10 minutes
• If the twitter API dies then keep using the last data received for a day. Then assume it’s invalid and announce that Twitter has FAILED (optional).
• Don’t hit the rate limit (70 requests per 60 minutes)

So what exactly does APICache do? Given cached data less than 10 minutes old, it returns that. Otherwise, assuming it didn’t try to request the URL within the last minute (to avoid the rate limit), it makes a get request to the Twitter API. If the Twitter API timeouts or doesn’t return a 2xx code (very likely) we’re still fine: it just returns the last data fetched (as long as it’s less than a day old). In the exceptional case that all is lost and no data can be returned, it raises an APICache::NotAvailableError exception. You’re responsible for catching this exception and complaining bitterly to the internet.

All very simple. What if you need to do something more complicated? Say you need authentication or the silly API you’re using doesn’t follow a nice convention of returning 2xx for success. Then you need a block:

APICache.get('twitter_replies', :cache => 3600) do
  Net::HTTP.start('twitter.com') do |http|
    req = Net::HTTP::Get.new('/statuses/replies.xml')
    req.basic_auth 'username', 'password'
    response = http.request(req)
    case response
    when Net::HTTPSuccess
      # 2xx response code
      response.body
    else
      raise APICache::Invalid
    end
  end
end

All the caching is still handled for you. If you supply a block then the first argument to APICache.get is assumed to be a unique key rather than a URL. Throwing APICache::Invalid signals to APICache that the request was not successful.

You can send any of the following options to APICache.get(url, options = {}, &block). These are the default values (times are all in seconds):

{
  :cache => 600,    # 10 minutes  After this time fetch new data
  :valid => 86400,  # 1 day       Maximum time to use old data
                    #             :forever is a valid option
  :period => 60,    # 1 minute    Maximum frequency to call API
  :timeout => 5     # 5 seconds   API response timeout
}

Before using the APICache you need to initialize the caches. In merb, for example, put this in your init.rb:

APICache.start

Currently there are two stores available: MemcacheStore and MemoryStore. MemcacheStore is the default but if you’d like to use MemoryStore, or another store - see AbstractStore, just supply it to the start method:

APICache.start(APICache::MemoryStore)

I suppose you’ll want to get your hands on this magic. For now just grab the source from http://github.com/mloughran/api_cache/tree/master and rake install. I’ll get a gem sorted soon.

This is the irb quickstart so that you don’t have to re-parse the above:

require 'rubygems'
require 'api_cache'
APICache.start(APICache::MemoryStore)
APICache.get("http://twitter.com/statuses/public_timeline.rss")

Please send feedback if you think of any other functionality that would be handy.

Setting up

Install edge Merb; the latest gem (0.9.2) will not work. You need merb-core, merb-more, and merb-plugins.

Merb-plugins gives you the merb_stories gem, so you don’t need to install that separately.

Add this line to your app’s config/environments/test.rb:

dependencies "merb_stories", "webrat"

(Note that merb_stories’ README file is wrong about this, for now - and the generator will create a dependency on merb-rspec, which no longer exists. My fork fixes this.)

Now generate your story:

merb-gen story mystory

Now run your story:

rake story\\[mystory\\]

Yes, you must include the square brackets, and you have to escape them.

Writing stories

Now fill out your story. There are some differences to Rails’ versions. The best places to look for help are in the Merb code itself:

• spec/public/test/controller matchersspec.rb
• lib/merb-core/test/helpers
• lib/merb-core/test/matchers

To start you off, here are the steps for a simple integration test:

steps_for(:homepage) do
  When("I visit the root") do
    @mycontroller = get("/")
  end
  Then("I should see the home page") do
    @mycontroller.should respond_successfully
    @mycontroller.body.should contain("Hello") 
  end    
end

As you write your tests, don’t trust Merb absolutely. Some things are wrong, don’t work, or aren’t meant to work [yet]. As part of debugging, look at the Merb source, and fork it and fix it if needed.

Adding Webrat

Webrat lets you write integration tests that are even closer to natural language. You can say things like:

visits '/auth/login'
fills_in 'username',:with=>"bob"
fills_in 'password',:with=>"hunter2"
clicks_button "login"
response.should be_successful
...

I’ve forked Webrat to add Merb support.

To get started, clone it, build the gem and install it.

You’ll also need to make sure you’re using the memory session store (for testing), or your sessions won’t be preserved:

Merb::Config.use do |c|
  c[:session_store] = 'memory' 
end  

Continuous integration

Autotest will run your specs continuously, but won’t run your stories. You can fix this, but stories run very slowly, since they use the full stack. At the very least, though, you want to run your stories after you deploy, in case you have “works on dev machine, dies on production” problems.

Add this to deploy.rb:

namespace :deploy do
  task :after_deploy do
    run_remote_tests
  end
  desc "Run tests on remote server"
  task :run_remote_tests do
    run "cd #{deploy_to}/current && rake spec"
    run "cd #{deploy_to}/current && rake story\[all\] MERB_ENV=test"
  end 
end

This will fail if you’re using Vendor Everything like us - it won’t be able to find merb-core. To fix it, replace stories/stories/all.rb:

env = ENV['MERB_ENV'] || 'test'
require 'rubygems'
Gem.clear_paths
Gem.path.unshift(File.join(File.dirname(__FILE__), "..","..","gems"))

require 'merb-core'
Merb.load_dependencies(:environment => env)

require 'spec'
Merb.start_environment(:testing => true, :adapter => 'runner', :environment => env)

dir = File.dirname(__FILE__)
Dir[File.expand_path("#{dir}/**/*.rb")].uniq.each do |file|
  require file
end

You’ll have the same problem with specs. Edit spec/spec_helper.rb:

Gem.clear_paths
Gem.path.unshift(File.join(File.dirname(__FILE__), "..","gems"))

Now you have natural-language integration tests that run automatically on deployment. Sweet.

Getting Started

Before we get started I’m going to assume you have the following installed:

• Ruby
• A DBMS (we’ll use MySQL)
• SVN and git (on OSX, port install git-core worked for me)

Installing Merb

If you have an older version of Merb (<0.9.2) you should remove the all the gems before continuing. Use gem list to see your installed gems.

Installing the merb gems should be as simple as:

sudo gem install merb --source http://merbivore.org 

Unfortunately we are living right on the edge of development so we’ll need to get down and dirty with building our own gems from source. Luckiliy this is much easier than it sounds…

Start by installing the gem dependancies:

sudo gem install rack mongrel json_pure erubis mime-types rspec hpricot \
    mocha rubigen haml markaby mailfactory ruby2ruby

Then download the merb source:

git clone git://github.com/wycats/merb-core.git
git clone git://github.com/wycats/merb-plugins.git
git clone git://github.com/wycats/merb-more.git

Then install the gems via rake:

cd merb-core ; rake install ; cd ..    
cd merb-more ; rake install ; cd ..
cd merb-plugins; rake install ; cd ..

The json_pure gem is needed for merb to install on JRuby (Java implementation of a Ruby Interpreter), otherwise use the json gem as it’s faster.

Merb is ORM agnostic, but as the title of this book suggests we’ll be using DataMapper. Should you want to stick with ActiveRecord or play with Sequel, check the merb documentation for install instructions.

Installing DataMapper

DataMapper is splitting into dm-core and dm-more so datamapper 0.3 will be outdated soon. If you have an older version of datamapper, data_objects, or do_mysql, merb_datamapper (< 0.9) you should remove them first. Remove the merb_datamapper gem before installing dm-merb within dm-more.

We will use MySQL in the following example, but you can use either sqlite3 or PostgreSQL, just install the appropriate gem. You will also need to ensure that MySQL is on your system path for the gem to install correctly.

(TODO) - gem instructions for DM 0.9 once they are released

To get the gems from source:

git clone git://github.com/sam/do.git

cd do
cd data_objects
rake install ; cd ..
cd do_mysql  # || do_postgres || do_sqlite3
rake install

git clone git://github.com/sam/dm-core.git
git clone git://github.com/sam/dm-more.git

cd dm-core ; rake install ; cd ..
cd dm-more
cd dm-merb ; rake install ; cd ..    
cd dm-validations ; rake install ; cd ..

To update a gem from source, run git pull and rake install again.

Install RSpec

The rspec gem was installed in the Merb section above. However, if for some reason you didn’t install it there, or want to grab the it from source, run the following commands:

gem install rspec
svn checkout http://rspec.rubyforge.org/svn/trunk rspec_trunk

Creating an App

Now that we’ve got all of that installed, time to create a test Merb application. Merb follows the same naming convention for projects as rails, so ‘my_test_app’ and ‘Test2’ are valid names but ‘T 3’ is not (they need to be valid SQL table names).

merb-gen app test

This will generate an empty Merb app, so lets go in and take a look. You’ll notice that the directory structure is similar to Rails, with a few differences.

# expected output
RubiGen::Scripts::Generate
  create  app
  create  app/controllers
  create  app/helpers
  create  app/views
  create  app/views/exceptions
  create  app/views/layout
  create  autotest
  create  config
  create  config/environments
  create  public
  create  public/images
  create  public/stylesheets
  create  spec
  create  app/controllers/application.rb
  create  app/controllers/exceptions.rb
  create  app/helpers/global_helpers.rb
  create  app/views/exceptions/internal_server_error.html.erb
  create  app/views/exceptions/not_acceptable.html.erb
  create  app/views/exceptions/not_found.html.erb
  create  app/views/layout/application.html.erb
  create  autotest/discover.rb
  create  autotest/merb.rb
  create  autotest/merb_rspec.rb
  create  config/rack.rb
  create  config/router.rb
  create  config/init.rb
  create  config/environments/development.rb
  create  config/environments/production.rb
  create  config/environments/rake.rb
  create  config/environments/test.rb
  create  public/merb.fcgi
  create  public/images/merb.jpg
  create  public/stylesheets/master.css
  create  spec/spec.opts
  create  spec/spec_helper.rb
  create  /Rakefile

Configuring Merb

Before we get the server running, you’ll need to edit the init.rb file and un-comment the following lines (this is only necessary if you need to connect to a database, which we do in our case):

config/init.rb

use_orm :dm_core

use_test :rspec

Typing merb now in your command line will try and start the server.

Started merb_init.rb ...
No database.yml file found in /Users/work/merb/example_one/config.
A sample file was created called database.sample.yml for you to copy and edit.

As you can see, we forgot to set up the database. A sample file has helpfully been generated for us. Edit this and rename it to database.yml:

# This is a sample database file for the DataMapper ORM
development:
   adapter: mysql
   database: test
   username: root
   password: 
   host: localhost
   socket: /tmp/mysql.sock

Don’t forget to specify your socket, if you do not know it’s location, you can find it by typing:

mysql_config --socket

Starting Merb again shows that everything is running okay.

The following command will give you access to the Merb console:

merb -i

You’ll notice Merb runs on port 4000, but this can be changed with flag -p [port number]. More options can be found by typing:

merb --help

You can even run Merb with any application server that supports rack (thin, evented_mongrel, fcgi, mongrel, and webrick):

merb -a thin

Communities

i’m going to become rich and famous after i invent a device that allows you to stab people in the face over the internet - <[SA]HatfulOfHollow>

The internet is a scary place, but fortunately the Ruby community is very friendly. All these open source projects rely on the contributions from the community, if something needs fixing consider helping out.

Websites

These are the first places to go for help. Check out the API documentation and see if you can find your answer there.

• http://merbivore.com/
• http://datamapper.org/
• http://rspec.info/

IRC Channels - freenode.net

If you can’t find what you were looking for in the API docs then you could join the respective IRC channel on FreeNode and ask your question in there, you may need to wait for a response.

• #merb
• #datamapper
• #rspec

Mailing Lists

The mailing lists are another good way to get help, the response time isn’t as fast as asking in an irc channel but it can be useful to do a search to see if someone else has had your problem before.

• http://groups.google.com/group/merb/
• http://groups.google.com/group/datamapper/
• http://rubyforge.org/pipermail/rspec-users/

Bug Trackers

Your problem may or may not be a known bug. Search the bug trackers and submit a ticket if its not there already (don’t forget to include a description and test cases, or better yet: a patch!). You may find the ticket is solved in the edge version.

• http://merb.lighthouseapp.com/
• http://wm.lighthouseapp.com/projects/4819-datamapper/overview
• http://rspec.lighthouseapp.com/

Wikis

• http://wiki.merbivore.com/
• http://wiki.datamapper.org/

Next we’ll cover installing the gems, and generating a bare project.

So here we go (corrections and comments welcome):

Life On The Edge With Merb, DataMapper & RSpec

What’s Merb, DataMapper & RSpec?

If you’re not living on the edge, you’re taking up too much room. - Alice Bartlett

Merb, DataMapper and RSpec are all open source projects, which can be used for building kick-ass web applications. They are all in active development and it can be hard to keep up, but we’ll try our best to keep up to date.

Merb

It’s a relatively new framework (a bit like Ruby on Rails) and was created by Ezra Zygmuntowicz. Merb stands for Mongrel + ERB although now it supports the rack webserver interface so it can user any web server that has rack support (Mongrel, Thin, ebb, etc).

If you know Ruby and have used Rails you’re likely to get the hang of Merb quite easily. Noticeable differences from Merb to Rails are, it’s stance on being less opinionated and it’s approach to modularisation.

Merb is actually made up of a number of gems: merb-core, merb-more and merb-plugins. This is so you can pick and choose the functionality you need. The merb gem installs both merb-core and merb-more so you can get started straight away. The benefit of modularity is it’s kept simple with additional functionality provided by gems.

merb-core alone can be used as an upload server, to provide an API or to create a simple web app (a la camping) where the functionality of a fully fledged framework isn’t necessary.

DataMapper

DataMapper is an Object-Relational Mapper (ORM) written in Ruby, by Sam Smoot. It’s what we’ll be using with Merb. It’s possible to use the same ORM as Rails (ActiveRecord), but as there are plenty of examples of using ActiveRecord already I’ve chosen to use DataMapper.

It has some nice features which makes it faster than ActiveRecord in some cases. What really stands out for me is the way it handles database attributes. The schema, migrations and attributes are all defined in one place: your model. So you no longer have to look around in your database or other files to see what is defined.

DataMapper has some similarities with ActiveRecord. We will be highlighting the differences as we go along.

RSpec

RSpec is a Behaviour Driven Development framework for Ruby. Merb currently supports the Test::Unit and RSpec testing frameworks. As the specs for Merb and Datamapper are written in RSpec, we will be covering some aspects of RSpec but it will not be our main focus.

What About Ruby On Rails?

Merb is Harder, Better, Faster, Stronger, to quote Daft Punk - Max Williams

So what’s the big deal? We have Ruby on Rails and that’s enough, isn’t it? There is little doubt that Ruby on Rails has rocked the web application development world. You have to give credit where credit’s due, but it can be unforgiving if you don’t want to do things ‘the Rails way’.

Where Rails is opinionated, Merb is agnostic. You can easily use your favourite ORM (ActiveRecord, DataMapper, Sequel) or none at all, Javascript Library and template language.

Merb also has super-fast routing and is thread-safe (If performant were a word, Merb would be it). The core functionality is kept separate from the other plugins and it uses less Ruby ‘magic’, which makes it easy to understand and hack.

Rails (and consequently Ruby) has received a lot of criticism for not being suitable for large scale web applications, which isn’t necessarily true and Merb has been built from the outset to prove that Ruby is a viable language for building fast and scaleable web applications.

At the end of the day it’s about choice. There are many new Ruby frameworks springing up, undoubtedly helped by the success of Rails, but in my opinion Merb shows the most promise as a Rails competitor.

If you’d like to take a look at some other frameworks these links should get you started:

• http://camping.rubyforge.org/files/README.html
• http://www.nitroproject.org/
• http://ramaze.rubyforge.org/
• http://sinatra.rubyforge.org/
• http://halcyon.rubyforge.org/
• http://wisteria.swiftcore.org/

So that’s it for now, stay tuned for how to install/set-up merb and datamapper next…

I gave a talk today on Aspect Orientated Programming in Ruby using Aquarium, it quickly covered what I mentioned in my last post on AOP, and went into more detail with actual code examples this time!

The slides are on slideshare, the demo code is currently in a git repo . The demo app is Merb/Datamapper and I’m currently trying think of more examples of cool aspects to put in there, feel free to add some.

If anyone saw my talk, I’d love to hear what you thought of it so drop me a line matt@new-bamboo.co.uk or leave a comment on this post.

| View | Upload your own

When I go to conferences, my main goal isn’t to learn lots of things it’s to be inspired and generate new ideas, consequently my “things to do in my spare time” list has grown exponentially.

The two underling themes that I detected from the conference was developers are people too and scaling is important. It was interesting to hear Java guys talk about how to make developing easier (more fun), because this is one area Ruby (in my opinion) excels over most other languages.

One of the main criticisms about the Ruby community is we don’t care about scaling, and scaling is something that the Java guys have been working on for years. So when “Ola Bini”:http://ola-bini.blogspot.com/ says that “JRuby”:http://jruby.codehaus.org/ will be the best stack/platform around, I’m starting to believe him.

Imagine having your Ruby app with objects persisting in something like “Terracotta”:http://www.terracotta.org/confluence/display/labs/Terracotta+for+JRuby?

Or being able to use existing “Java aspects”:http://www.eclipse.org/aspectj/ on Ruby code (and “visa versa”:http://blog.objectmentor.com/articles/2008/02/25/writing-java-aspects-with-jruby-and-aquarium).

There was talk about GemStone porting their “Smalltalk VM”:http://www.gemstone.com/products/smalltalk/ to Ruby so we may have an ‘enterprise’ stack.

And where the hell are the Ruby Specs? I mean the tests for the language itself. There should be one central repository where YARV, Rubinius, JRuby, GemStone and every Ruby developer out there can contribute to Spec’in out the language.

They are just a handful of ideas that popped into my head. I had a great time at the “conference”:http://flickr.com/photos/tags/qconlondon2008/ and met many interesting people, hopefully I’ll be able to do it again next year!

The slides (videos soon) from the talks are going “online”:http://jaoo.dk/london-2008/schedule/thursday.jsp so take a look. Please excuse me while I go play with Erlang.

UPDATE I’ve put some of the slides that are available on the QCon site onto slideshare tagged with “QCONLONDON2008”:http://www.slideshare.net/tag/qconlondon2008 , enjoy.

| View | Upload your own

This is what sublayout.html.erb looks like

<% throw_content :sublayout do %>
  <div class="header">
    ...
  </div>
  
  <%= catch_content :for_layout %>
  
  <div class="footer">
    ...
  </div>
<% end %>

<%= render catch_content(:sublayout), :layout => "application" %>

Your main layout contains <%= catch_content :for_layout %> as usual.

You might find it handy to use catch and throw content in a few more places. For example, in your sublayout you could <%= catch_content :sublayout_title %> and throw it in the view with

<% throw_content :sublayout_title %>
  Hippos
<% end %>

As a little bonus I wrote a helper so that you can use the same syntax as you’re used to from the rails plugin. Just add this snippet into Merb::GlobalHelpers

def inside_layout(layout, &block)
  content = render(capture(&block), :layout => layout)
  concat(content, block.binding)
end

Then your sublayout looks like

<% inside_layout :sublayout do %>
  <div class="header">
    ...
  </div>
  
  <%= catch_content :for_layout %>
  
  <div class="footer">
    ...
  </div>
<% end %>

Here’s another idea which I think would be even better: What if you could say layout ['application', 'sublayout'] in your controller and have each layout rendered inside it’s parent all up the chain? Then sublayout.html.erb wouldn’t be constrained to always render inside application.html.erb and you could use it in a much more flexible way. However Damien things it’s a mad idea. What do you think?

That’s clever because now you can use the standard tools for installing and managing the installed gems. For example, you get dependency checking and so on for free. Add -i gems to the command line options to specify the install location.

gem install merb-core -i gems

That’s the simplest case (run it from the root of you merb app). It fetches the merb-core gem from the gem servers and installs it inside a gems directory.

However, in many cases that’s not what you want. Really you want to freeze the version you’re currently developing on. Maybe one you’ve installed from the merb-core git repository. That’s really easy because you have the gem package files on your system already. For me they’re in /Library/Ruby/Gems/1.8/cache. Get your location by running gem environment gemdir and look for a cache folder. Install the package with

gem install /Library/Ruby/Gems/1.8/cache/merb-core-0.9.1.gem -i gems

You should add /gems/doc to your .gitignore since you definitely don’t want ri and rdoc files in your repository. Alternatively add --no-ri --no-rdoc to the gem install command.

The final challenge is actually getting merb started. Right now I'm relying on a [pastie](http://pastie.caboo.se/pastes/164289) that google found for me (I'm sorry I have no idea who the author is). Make that executable and use it to run merb with the usual merb command line options. That's surely a bit of a hack though and I'm hoping that someone will point out a better way :) Update: Thanks Steve - I hadn’t spotted the generator. Run merb-gen freezer frozen-merb and then run Merb with ./script/frozen-merb passing the usual merb command line options.

The trick is to use ssh agent forwarding, now supported in capistrano. It allows the server to pull the latest code from github using your local private key and ssh agent. To understand how this trickery works take a look at this illustrated guide.

set :ssh_options, { :forward_agent => true }

One of the great things about git is that it’s really fast, even if you’re cloning an entire repository. However you can go even faster by using the repository_cache option in capistrano. This essentially keeps a clone of your app on the server and then just does a git pull to fetch new changes and copies the directory across when you deploy.

set :repository_cache, "git_cache"
set :deploy_via, :remote_cache

Another recommended strategy is to specify a git branch. Otherwise you’ll default to deploying HEAD which might be some crazy experiment.

set :branch, "stable"

In case you haven’t used remote branches they’re really easy. Say you have a local stable branch. Push this to github using

git push github stable

So the complete capistrano file should look something like

set :application, "your cool app"

# Source code
set :scm, :git
set :repository, "git@github.com:githubusername/gitrepo.git"
set :branch, "stable"
set :repository_cache, "git_cache"
set :deploy_via, :remote_cache
set :ssh_options, { :forward_agent => true }

# Deployment servers
role :app, "your server"
role :web, "your server"
role :db,  "your server", :primary => true
set :deploy_to, "/var/www/#{application}"

We’ve scrapped all of these and have now migrated to a private Github repo containing markdown files managed with the help of the Textmate Wiki Bundle. The simplest solutions are the best.

Nginx Upload Progress Module Safari patch

Simplicity is great but unfortunately ImageScience has this infatuation with squares. I’ll explain.

Image science has two low level methods which allow arbitrary cropping and resizing. They look like this:

image.with_crop(left, top, right, bottom)

image.resize(width, height)

However what you really want to do is usually a combination of cropping and resizing. ImageScience helpfully provides:

# Resizes to fit within a square
image.thumbnail(size)

# Resizes and crops to be exactly a square
image.cropped_thumbnail(size)

This is great until you need an avatar that’s a fixed … rectangle. Or an image that will fit exactly inside a … rectangle.

In this case you need IrregularScience (aka irregular_science)!

You’ll get rectangular versions of thumbnail and cropped_thumbnail:

# Resizes to fit within a rectangle
image.resize_within(width, height)

# Resizes and crops to be exactly a rectangle
image.resize_exact(width, height)

I gave my methods slightly different names because I’m like that ;)

There’s also the case where you want a exact height or an exact width regardless of the other dimension (ever seen facebook?). That turns out to be a really horrible hack with RMagick involving all sorts of squashing and stretching unless you go really low level. Just follow these incantations and know that no horrible hacks are going on:

image.resize_to_width(width)

image.resize_to_height(height)

To get started download irregular_science.rb and require it. Then adapt this example:

ImageScience.with_image(upload_path) do |image|
  self.thumbnails.each do |name, size|
    image.resize_exact(*size) do |img|
      img.save(attachment_path(name))
    end
  end
end

Enjoy!

P.S. This seems to be my first blog post here - so hi!

This is a tiny one. Let's say we have a Factory that gives me Adapters#1 to a series of hosted media services (Flickr, YouTube, Vimeo, PhotoBucket, etc.).

The user of my code provides a service prefix and a URL to a picture or photo page in a given service, and the factory returns an instance from which the user can fetch different media sizes through a standardized API.

flickr_pic = ServiceParser.instance( 
  'flickr', 'http://www.flickr.com/photos/new_bamboo_london/2158168775/'
)

puts flickr_pic.thumbnail  
# => http://farm3.static.flickr.com/2147/2158168775_01ae89cbfb_s.jpg

yt_video = ServiceParser.instance( 
  'youtube', 'http://www.youtube.com/watch?v=PCunD-_mOwQ' 
)

puts yt_video.thumbnail 
# => http://i.ytimg.com/vi/PCunD-_mOwQ/default.jpg

Simple enough. But it seems a bit redundant to have the user declare both the service name and the page URL. Since URL's are unique anyway, those should be enough for our smart factory to know what adapter to hand us, kindly.

We could have a set of regular expressions in the factory, one for each URL/service.

class ServiceParser
  SERVICES = {
    FlickrAdapter   => /flickr\.com/,
    YouTubeAdapter  => /youtube\.com/,
    VimeoAdapter  => /vimeo\.com/,
    PhotoBucketAdapter  => /photobucket\.com/
  }
  # Factory method
  #
  def self.instance( url )
    subclass = SERVICES.find(nil) {|klass, exp| url =~ exp}
    raise 'not implemented' if subclass.nil?
    subclass.new url
  end
end

We could. But if we did, all hell would break loose and a mob of angry Design Patterns advocates would ram our door and... reprimand us.

While they're at it, they would tell us that one of the many golden rules of Object Oriented software design is, blockquote please:

The superclass should have no knowledge of the subclasses.

But you already know that. We need to take those URL's out of the factory class and into where they belong, in each adapter class. Also, adapters should be able to register their URL's with the factory, so it knows where to look. Ruby to the rescue.

# = The superclass / factory
class ServiceParser
  # We register adapters into this class variable
  #
  @@adapters = []
  
  # class reader
  #
  def self.adapters
    @@adapters
  end

  # Factory method here, se below...
  #
end

# = An example subclass / adapter
class FlickrAdapter < ServiceParser
  def initialize( url)
  
  end

  # API methods
  #
  def thumbnail
    # do something clever here...
  end
  
  # register ourselves with the Factory
  #
  ServiceParser.adapters << [self, /flickr\.com/]
end

The fun thing about Ruby, class definitions being executable code and all, is that you can just have subclasses add themselves (and their corresponding regex.) to the factory's list of adapters in load time.

 ServiceParser.adapters << [self, /flickr\.com/]

Then it is a matter of modifying the factory method to search for matching URL's in the dynamically populated list of subclasses / URL expressions.

# Refactored factory. No pun intended.
#
def self.instance( url )
  match = @@adapters.find(nil) {|klass, exp| url =~ exp}
  raise 'not implemented' if match.nil?
  match.first.new url
end

Finally, a little more elegance to make our adapter authors happy (and less prone to mess with the internals of our superclass!)

class FlickrAdapter < ServiceParser
  # Initializer, API and protected methods here...
  #
  # We register our URL with a neat class method on the superclass
  #
  register_url /flickr\.com/
end

class ServiceParser
  # Factory, abstract methods and cross-adapter utilities here...
  #
  # Class method for subclasses to register themselves.
  #
  @@adapters = [] #2 See note on class variables

  def self.register_url( exp )
    @@adapters << [self, exp]
  end

end

#1 Two more buzzwords and just the second paragraph! And there's more...

#2 Class @@variables. These variables are not inherited but shared by all classes in the inheritance tree. That means that, whenever a subclass adds to the superclass' version of @@adapters, it'll actually have the new value in all subclasses. This is not an issue in our simple example, though.

…we’re giving it away…

…that’s right folks, I’d like to announce the introduction of code.new-bamboo.co.uk.

We hope that over time code.new-bamboo.co.uk will grow to become a great resource for open source projects and code snippets. To kick us off we have…

• PolyPage - A jQuery plugin for adding state to wireframes.
• JSS - a jQuery plugin for adding CSS3 selector support for your stylesheets!
• The Index Inspector - A rails specific rake task for probing your database indexes.
• Dogger - A fun little log helper.

I hope you’ll agree we are off to a great start but we’d love some help so if you have any ideas or suggestions then sound off in the comments for now. hopefully there will be a more formal issue tracker setup soon.

The jQuery plugins are released under the same licensing terms as jQuery itself, snippets aren’t formally licensed but feel free to use and distribute them as you please.

Datamapper (DM) isn’t really production ready at the moment as migrations are destructive, but hopefully that’ll change in the next few months.

The first thing you’ll notice is that in DM the properties of the database are in your model, where in AR they are in a migration.

Take a look at this example DM model:

class Post < DataMapper::Base
  property :title,      :string
  property :textile,    :text
  property :created_at, :datetime
end

rake dm:db:automigrate || DataMapper::Base.auto_migrate!

With AR this would be in a migration, not to dissimilar to this:

class CreatePosts < ActiveRecord::Migration
  def self.up
  create_table :posts do |t|
    t.string :title
    t.text   :textile
    t.timestamps
    end
  end

  def self.down
    drop_table :posts
  end
end

rake db:migrate

I should point out that just like AR the created_at field is automatically filled in by DataMapper.

DataMapper has a few ActiveRecord impersonations such as the follow methods: [], all, count, create, create!, delete_all, each, find, find_by_sql, find_or_create, first, get, truncate!, destroy!, included, reload, reload!, save

Here is a list of some syntax comparisons:

Something you might want to watch out for, covered by Jon is that DM will only persist changes that it manages, so if you’re using virtual attributes you’ll need to mark the model as dirty. Also there are currently bugs in finding with conditions on associations and count.

If you haven’t already installed Merb, check out Jon’s awesome screencast, that should get you started.

Here’s my list¹:

SfsResize (sfs_resize.rb) is a super simple image resizer. Compile libgd in a few minutes (there is an OS X how-to) and then use it like this:

w = 100
h = 100
s = SfsResize.new("in.gif")
s.resize_and_write_to_file("out.gif", w, h)

Severely untested, but good fun nevertheless.

Update: Shalev NessAiver has made a pretty diagram!

The Nginx upload progress module adds track_uploads and report_uploads directives which do what they say on the tin. When the user submits a form to the location supporting track_upload, we also send an extra X-Progress-ID parameter with a unique id for that upload. Requests made to the location supporting report_uploads (typically /uploads) will return JSON with the current progress for an upload. The X-Progress-ID header is added to the ajax requests to identify which upload we're wanting to know the status of.

Having Nginx track the progress gets around an issue with using the merb upload progress method. Nginx (un)fortunately does not pass along the upload to its destination until it has fully completed uploading. So when you upload a file, your Merb application won't know about it until it has been upload to Nginx and passed along. The only gripe with this whole process is the possibility of a small delay as Merb handles the request once the upload is complete. If you're using Rails however this can be an advantage, as it won't be tied up by the upload as it happens in realtime.

To get this all going you'll first want to download the Nginx upload progress module and recompile Nginx with the following configure option.

--add-module=path/to/nginx_uploadprogress_module

Locate and open your nginx configuration file and add in the upload_progress, track_uploads and report_uploads directives.

If you're going to upload files of any considerable size you'll also want to the max_body_side to something reasonable.

http {
	upload_progress proxied 1m;

	server {
		listen 80;
		server_name _ *;
		root /usr/local/www;

		location / {
			proxy_pass http://127.0.0.1:4000;
			track_uploads proxied 30s;
			client_max_body_size 500m;
		}

		location ^~ /progress {
			report_uploads proxied;
		}
	}
}

Nginx is now ready to track uploads! Generate a Merb project and add a new upload controller.

class Uploader < Application
	def index
		render
	end

	def upload
		FileUtils.mv params[:file][:tempfile].path, \
MERB_ROOT+"/files/#{params[:file][:filename]}"
	end
end

Then add the upload view.

<div id="pandaloader">
	<div id="uploader">
		<form id="upload" enctype="multipart/form-data" \
action="/uploader/upload" method="post">
			<input name="file" type="file">
			<input type="submit" value="Upload">
		</form>
	</div>

	<div id="uploading">
		<div id="progress" class="bar">
			<div id="progressbar">&nbsp;</div>
		</div>
	</div>
</div>

In the header of your application's layout you'll also need to include the javascript and css.

Aside from jQuery, you'll notice the jquery.nginxUploadProgress.js plugin which has been included. This is a little jQuery library which will query Nginx for the upload's progress and update the length of the progress bar accordingly. Grab the file here: jquery.nginxUploadProgress.js

Here is some styling for the progress bar.

.bar {
  width: 300px;
}

#progress {
  background: #eee;
  border: 1px solid #222;
}
#progressbar {
  width: 0px;
  height: 24px;
  background: #333;
}

Put all these together and you'll have a great little progress bar for uploads!

• Use STRAIGHT_JOIN if MySQL is doing something silly
• MySQL Datetimes are not slow
• MyISAM is much, much faster than InnoDB
• Consider MySQL partitioning; it's not hard
• Watch out for background optimisation and query caching
• Log everything you do, and why

The reporting application I'm working on does a date-range select against a table with ten million rows. It took over two minutes to execute. I cut it down to around two seconds without significantly changing the application. Here's how.

Straight Joins

The original SQL statement was 12 lines, featuring three joins, a group-by and a computed column.

To start with I used EXPLAIN SELECT to look at the execution plan. MySQL had made a bad guess, and was table-scanning a four-million-row related table that could have been joined by ID instead.

I added STRAIGHT_JOIN to the SQL statement and dropped the execution time to 20 seconds.

Use STRAIGHT_JOIN if MySQL is doing something silly.

Finding the Weak Point

Progress; MySQL was now table-scanning a 10 million row 'events' table to satisfy the WHERE clause, then joining the other three tables by ID, grouping and summing. I cut the query down progressively to a simple "SELECT count(*) FROM events WHERE starttime < '2006-12-30 08:30:00' AND endtime > '2006-12-30 08:15:00'" and found that performance didn't significantly change.

This isolated the slow part of the query. My original plan had been to denormalize the table structure, but there was no point.

At this point I built a test database of two million rows on my local machine with only the one table in question, and only the three columns that were needed.

Indexes

I added indexes on starttime, endtime, and (starttime,endtime), but the MySQL optimiser chose not to use them. When I forced it to use them, using FORCE INDEX, performance deteriorated (!). OPTIMIZE TABLE made no difference.

Dead End: Int Conversion

The application needs only 15-minute resolution on the date times, so I created start_int and finish_int columns and filled them:

update events set start_int = floor(unix_timestamp(starttime)/900) 
update events set finish_int = floor(unix_timestamp(endtime)/900)

Then I created indexes on the columns and tested again. The plan here was that, by reducing the resolution, there'd be fewer unique values and the indexes would be more effective.

It didn't work. Tablescanning against datetimes was faster than indexed lookup against ints.

MySQL DateTimes are not slow.

Switching To MyISAM

I'd noticed that my test database was performing much faster than the production database, even taking into account the smaller size. Running SHOW CREATE TABLE against the two revealed that I was using MyISAM, but the production database had InnoDB. I converted my local database to InnoDB and took a speed hit on the order of 6x.

MyISAM is much faster than InnoDB. (but see notes)

Note that the conversion process is hideously slow; it's faster to create a new table and load it up with your test data yourself. If you do this, use extended inserts; they're about 25x faster than singular inserts.

You give up some goodies, including transactions, by making the switch.

Partitioning

The application never queries over a time range of greater than 15 minutes, and always aligns its requests on 15-minute boundaries. If we had to tablescan, I could use this to at least make sure we didn't have to tablescan all ten million rows.

I installed MySQL 5.1, which is the first release that supports partitioning, and created a new table using MyISAM, with a partition per 3 months, copying data from the old table in one hit:

CREATE TABLE `events2` (
 `id` int(11) NOT NULL AUTO_INCREMENT,
 `starttime` datetime DEFAULT NULL,
 `endtime` datetime DEFAULT NULL,
 KEY `id` (`id`),
 KEY `starttime` (`starttime`),
 KEY `endtime` (`endtime`),
 KEY `starttime_and_endtime` (`starttime`,`endtime`),
) ENGINE=MyISAM DEFAULT CHARSET=utf8
PARTITION BY RANGE (TO_DAYS(starttime)) (
	PARTITION p0 VALUES LESS THAN (732312),
	PARTITION p1 VALUES LESS THAN (732677),
	PARTITION p2 VALUES LESS THAN (732767),
...
	PARTITION p17 VALUES LESS THAN MAXVALUE	
) SELECT * FROM events;

I'm using TO_DAYS because partitions by range can only be described in terms of a single integer (not a datetime).

Now I could adjust my query to address a single partition only:

SELECT count(*) FROM events WHERE starttime < '2006-12-30 08:30:00' 
AND endtime > '2006-12-30 08:15:00" AND starttime > '2006-12-01 00:00:00'.

I used EXPLAIN PARTITION SELECT ... to confirm that MySQL was accessing only a single partition.

The benefit of this particular partitioning strategy is that it's transparent to the application. More advanced approaches involve splitting records across database _servers_, rather than simply changing the 'create table' statement.

Consider MySQL partitioning; it's not hard.

Results

The full query originally took around 152 seconds to run. It's now down to around two seconds, or a 75x speed improvement. This is running essentially the same query, and returning data in the same format; the only change necessary at the application level was to reword the SQL slightly.

General Tactics

Those were the specific victories. Here are the overall tactics which are more generally applicable:

I found it helpful to think not about what my application needed to know, but what it didn't need to know. For example, rules like "we never run a query that does not align on a 15-minute boundary" can simplify the problem.

Nothing substitutes for benchmarks. Things that ought to be fast are sometimes slow. Things that ought to be slow are sometimes fast.

But don't trust the benchmarks. Sometimes I'd return to my machine to find the *same query* running in less than a second instead of over a minute. I was using SQL_NO_CACHE, so I think background optimisation is responsible here. Since the database was very large, it's possible that OSX's virtual memory management was also interfering.

Watch out for background optimisation and query caching.

The only reason I caught this was that I kept notes of every single query I ran, which machine I ran it on, the time it took to execute, and anything interesting about the results. It's a simple routine to create or edit the query in Textmate, then paste it into MySQL, instead of working in the MySQL window directly.

Log everything you do, and why.

I also kept notes about theories and ideas, inline with the sql, and later marked them with PROVED / DISPROVED and the evidence so that I could be sure I'd covered all the bases.

Conclusion and Notes

Background

Last year my web development company, consisting of myself and an ever-increasing number of subcontractors, began expanding very quickly. While I stayed mostly on top of the situation, the stress was overwhelming.

I made the switch to GTD during Christmas 06.

Now that I'm working full-time as a developer, instead of as a sole trader, my task lists are much shorter. I've therefore made some simplifications. GTD can be used for large-scale project management, but that isn't covered here.

Benefits

GTD is both a workflow and an attitude. The core concept is that nothing lives in your brain, except for what you're currently working on. You must be merciless about this.

The real benefit, and the driver for me, is this: You get a stronger personal focus, as well as a stronger work focus.

When I'm at work, I'm really at work. When I'm at home, I'm really at home. This means more effective recovery, and more energy at work.

David Allen's book Getting Things Done, which is where I started with GTD, says this:

"In order to hang out with friends or take a long aimless walk and truly have nothing on your mind, you need to know what all your actionable items are, where they are, and that they can wait. And you need to be able to do that in a few seconds, not days."

Selecting Tasks

I have everything that I ever need to do in one place. The everything bit is important.

These items are described in the form of "next actions", or "the very next thing I can do to move this forward". They are very specific. For example, I don't have "deal with tax return"; I have "phone ATO re: missing BAS for FY06-07, 13 28 66, 9-6pm Adelaide time".

Because of this precision, I can allocate tasks to "Contexts", which are sets of prerequisites. For example, I can't perform the "tidy room" task unless I'm at home. I can't call the ATO unless it's Really Early (because they're in Australia).

This means that tasks are invisible to me unless I can perform them right now.

The workflow is:

1. Complete current task.
2. Select context.
3. Select a new task from tasks in this context. There's no advance scheduling and prioritising; in the midst of constantly shifting requirements, I trust myself to make the selection just-in-time.

Adding Tasks

That's how you work with tasks in the system. What about when a new task appears?

First, you stash it in one of your inboxes. You have the smallest number of these that you can get away with.

Your brain is absolutely not allowed to be an inbox. If you're filing things in your brain for even a second, you need another inbox.

I use two: my email account, and a notepad application on my phone, which is always with me.

Stashing an item in an inbox is very quick, and doesn't interrupt what you're doing. Whenever you have time - eg whenever you finish a task - "process" your inbox. For each item in it, you have four options:

Do it

If it's going to take less than two minutes, do it. Do it right now. Then forget about it.

The "two minutes" is a varying threshold. Anywhere between two and five works for me. I'll initiate system backups, answer quick emails, fill in my timesheets, and occasionally even make phone calls. If I spend ten minutes and kill off five two-minute tasks, I feel _good_.

I've taken this a step further, and will even break inbox queuing to do this: if a new task appears I'll sometimes stop what I'm doing to kill it off.

Delete it

GTD requires you to be realistic. Some things you'd like to do, you might have long-term plans to do, but realistically, you're never going to do them.

Don't feel bad about them. Don't lie to yourself, or to other people. Instead, say "I'm never going to do this", delete it from your inbox, and forget that it ever existed.

You can have a "Someday" context for items that will live in your brain if you don't put them somewhere else. I review mine periodically to weed out things that just aren't ever gonna happen.

Delegate it

If you give the job to someone else, the chances are you're still responsible for it. You need to add it to a "Waiting" context, with a description like "Waiting for ATO to send new form". Periodically scan this context, and see if any of these things:

• has happened, so you can remove it, or
• hasn't happened, and you need to follow it up.

If you don't put it in the "Waiting" context, it'll still live in your head. You don't want that.

Defer it

This is your option of last resort. It means "I can't do this now, but I do have to do it, and I can't give it to anyone else". Here's where you work out the very next action that you need to take in order to move this forward. Do take the time to calculate the next action; often, it's very small ("sort out my tax" is a huge job, but "email the ATO asking for a form 12" is not), and you will be able to do it now, after all.

Once you've got a next action, choose a context for it. Store it under that context in whatever system you're using.

You are now committed to doing this task.

Equipment

The Book tells you to buy a lot of things. I didn't have to.

You must, must have a small notebook that you carry with you all the time, particularly at first when you are dumping the contents of your brain. I use the built-in Notes application on my Symbian phone.

Until recently, I used Actiontastic, relying on the fact that I always have my laptop with me. I'm now in the process of migrating to the Notes application on my phone, because I'm using the laptop much less at home.

I didn't get hung up on syncing. I always have my phone. Why would I need to sync it to anything else?

I have all my paperwork in a single box - yes, I use shoebox filing. This works because everything I care about is also in iPhoto. I use a digital camera rather than a scanner, because it's orders of magnitude faster and the quality is "good enough". iPhoto is crap for this, but it's "good enough" - I use the description field to insert keywords.

I have a single "reference" folder in my email account. I find that gmail's search is good enough that I don't even need to use labels. Anything that I'm not sure I'll need, I delete.

The Book covers some other areas, such as the "hard landscape" - tasks that must occur at certain times - and "tickler files" (tasks you'd like to forget about for a while, but do need to revisit). I use my phone's calendar for both. I sync with my partner, friends, and co-workers using the low-tech but highly-effective "what have we got on Tuesday?" method.

These are the tools you do need:

Getting Things Done, by David Allen. It's an inspiring read, and packed full of details that are worth knowing about even if you don't use them. GTD is a style, not a religion.

Actiontastic, if you're going to keep your GTD lists on your Mac. The Omni Group is working on something called OmniFocus, but it's still in early beta.

As far as I know there are no GTD tools, or even suitable Todo-list tools, for Symbian phones. If there's enough demand, I'll write one - let me know ('gwyn' at this domain).

Making The Switch

One of the guys here at New Bamboo had previously tried to switch to GTD, but bailed out when The Book told him to take two days off to sort out his life. You don't have to do that.

But you should.

I read The Book and made the switch in the Christmas-New Year break last year. The timing worked well for me, and allowed me to go through everything, getting the full clear-head benefit. Yes, I had tasks like "charge my phone" and "tidy my desk". It's like tidying up; once you start, you go a little crazy.

Carrying a notebook is critical. I found that once I started the dump, I couldn't stop it.

Don't get sidetracked into software. Half of New Bamboo reacted to my initial presentation by planning to write software to track their tasks. This defeats the purpose, which is to increase productivity. This isn't about software, or three-colour stationery and ring binders. This is about habits.

Summary

To summarize:

• Get everything in one place.
• Think of your tasks in terms of the Next Actions that are needed to complete them.
• Classify your tasks into Contexts.

GTD takes some effort to maintain, but you'll feel it when you start to slip: the stress will come back.

Postscript: GTD at New Bamboo

The material above is drawn from an internal presentation ("GTD: Task Management For Ninjas") that I gave for New Bamboo's Talk At Two program. A month later, I did a quick survey to see if it stuck. While there's only one new full-blown implementation, nearly everybody found something useful to incorporate into their existing systems.

Three people didn't attempt GTD. Two of those weren't here for the presentation.

Damien tried it, but found that my stripped-down version didn't track larger projects. He's using the "errands" context, though, with a separate list that he looks at on his way out the door.

Pablo's using a series of paper lists, and is enjoying lower stress and the sense of achievement that comes from crossing things out. He's also been inspired to collect old tasks and to evaluate them realistically.

Ismael's using GTD's aggressive single-task focus, but finds he can store his lists in his brain without stressing himself out.

Max is running a full implementation, using Actiontastic, but he's still using his brain as an inbox for when he's not near a computer.

Matt started using GTD, then found he no longer needed it.

He stopped because he Got Everything Done.

Looking into the OpenSocial documentation you can see that it exposes 3 different types of resources: people, that person's friends, and their activities. This structure maps nicely onto Rail's concept of nested resources, where we could have a top-level users controller with friends and activities nested underneath it.

The OpenSocial API looks quite well thought out, and built upon RESTful ideas. However it doesn't fit completely with the Rails ideals. Firstly the URL's for accessing the resources aren't the standard Rails routes and secondly it doesn't include a format extension, which means we can't use a responds_to block out of the box.

However both of these problems are easily solved by adding a few mappings to the routes file and adding our own custom mimetype. The example below shows the routing for mapping the users controller to the OpenSocial people resource.

# config/environment.rb
Mime::Type.register "application/x-opensocial", :opensocial

# config/routes.rb
map.opensocial_person '/feeds/people/:id', :controller => 'users', \
                            :action => 'show', :format => 'opensocial',\
                            :conditions => {:method => :get}

map.opensocial_person '/feeds/people/:id', :controller => 'users',\
                            :action => 'update', :format => 'opensocial', \
                            :conditions => {:method => :put}

map.opensocial_person '/feeds/people/:id', :controller => 'users', \
                             :action => 'destroy', :format => 'opensocial', \
                             :conditions => {:method => :delete}

# app/controllers/users_controller.rb

respond_to do |format|
  ...
  format.opensocial
end

Now you can render show.opensocial.erb or show.opensocial.builder to create atom feeds that the OpenSocial API requires.

To avoid repeating it throughout a number of applications I wrapped it up into a little plugin that just sets the routes for those three resources, like so.

# config/routes.rb
map.resources :users do |user|
  user.resources :activities
  user.resources :relationships
end

map.opensocial_people :users do |user|
  user.opensocial_friends :relationships
  user.opensocial_activities :activities
end

A bit of history

AOP started out with AspectJ from Xerox PARC about 10 years ago, and is now an open source project managed by the Eclipse Foundation. AspectJ is an extension to Java that adds aspect-orientated capabilities, while not the only implementation of AOP it’s practically the de facto in the Java camp.

But what about Ruby? There are a few AOP Frameworks for Ruby AspectR (hasn’t been updated in a while) and Aquarium.

I use Aquarium later in the examples, you can get it with:

sudo gem install aquarium

Or the source:

svn checkout svn://rubyforge.org/var/svn/aquarium/trunk aquarium_trunk

What is AOP?

AOP is an effective technique for modularizing crosscutting concerns¹. Crosscutting concerns appear all over the place in applications such as, authentication, logging, caching etc. But why is this a problem? Current implementations of cross cutting concerns break modularization and can lead to spaghetti code, which is hard to maintain.

AOP to the rescue! With AOP concerns that crosscut many classes are modularized into an aspect. This allows you to write a logging aspect (for example), which can effect your entire application, a group of classes or just a single method.

What is an aspect?

It describes how code for a concern should be integrated into the code for the whole application, without modifying the original code

In AspectJ the java bytecode is manipulated to include the code from the aspects before, after or around the specified method calls. But in Ruby it’s done using the meta-programming mechanisms available.

Aspects comprise of²:

• Join Points: A point of execution in a program where ‘advice’ might be invoked.

• Pointcuts: A set of join points of interest, like a query over all join points in the system.
• Advice: The behavior invoked at a join point.

There are a few types of advice available. Before advice executes code before the join point, after advice does so after returning from the join point and around advise ‘wraps’ a join point.

Here are some examples of Aquarium code, but for more detailed examples see the example page

Join Point

JoinPoint.new :type => Post, :method => :new
    
JoinPoint.new :object => @post, :method => :new

Pointcut

Pointcut.new  :types => UsersController, 
  :methods => [:edit, :update, :destroy, :admin]

Before Advise

include Aquarium::Aspects::DSL::AspectDSL 

before :pointcut => your_point_cut do |jp, *args| 
  log “Entering: #{jp.inspect}” 
end 

Isn’t AOP just meta-programming?

Meta-programming can support the essence of AOP, and can also support lots of other things that fall out of the scope of AOP. Ruby can achieve the functionality that AOP provides through meta-programming, why should I use a framework?

AOP frameworks provide another level of abstraction, a nice API (Domain Specific Language) to express crosscutting concerns intuitively.

AOP vs Design Patterns

Design Patterns are templates for solving certain problems, many of which could be implemented better with AOP. As mentioned aspects modularize the concern for a problem. So you can reuse aspects in other applications unlike design patterns which have to be implemented each time.

Soon AOP will be the de-facto implementation for cross cutting concerns, just as Object-oriented programming is for inheritance, encapsulation etc.

It should be made clear that AOP isn’t a Redpill! You don’t gain extra functionality, but AOP make implementing the solutions to these problems easier, cleaner and is closer to God³.

AOP and Rails?

Rails has filters so why should I bother with AOP? Rails filters use method aliasing and Aquarium does basically the same thing, with a little more sophistication to make it more reliable to add multiple “filters” or to remove any one of them dynamically.

Also plugins that override method_missing can step on each other and behave in unexpected ways. With AOP we can overcome this problem.

There are roughly 180 cases of method aliasing in the Rails 1.2 stack, much of which would be cleaner and more robust if implemented as advice, instead.

AOP isn’t the future

It’s here NOW! The Spring Framework is built on top of AspectJ and is a popular Java/JEE application framework.

There are some AOP features penned for Ruby 2.0, such as :pre, :post, and :wrap for methods.

Where I see a possibility AOP adoption starting out is with non obtrusive (orthogonal) aspects for logging, tracing, etc. After people get comfortable with AOP, then we’ll start to see the real power of AOP being utilized within business objects.

So there you have it a quick round up of AOP, when I get some time I might have a crack at using Rails and Aquarium to make a demo app or plugin.

Interesting Links

• AOP at infoQ
• AOP google talk
• Rails’ Unusual Architecture
• Presentation on Aquarium with Exercises

More thanks goes to our fantastic team who contributed so much to an engaging stall, though the lure of the Xbox was at some points too much for some. Our thanks also goes to Gavin Strange for the all the decorative paraphenalia he produced to make our lively, if rather bemusing, display.

For those of you who managed to grab one of Gavin’s amazing panda t-shirts, we hope you enjoy them, and wear them with pride. “Even Ryan got in on the panda love”:http://www.flickr.com/photos/carsonified/1482580870/.

However, I’m sure there now exist many poor souls who could not attend, and are now feeling rather devastated at the lost opportunity. Never fear, we are gathering names and sizes of people who would like a t-shirt, and will try to produce a limited run of them soon. We are probably going to be avoiding retina-destroying green this time.

If you are interested, please drop us a line at “gimmeafreakinpandashirt@new-bamboo.co.uk”:mailto:gimmeafreakinpandashirt@new-bamboo.co.uk, and we will get everything moving once we have enough demand.

We also have a collection of “new bamboo Flickr photos”:http://www.flickr.com/photos/newbamboolondon/, and would love to see any of your own panda sightings.

So if you’re interested in meeting the whole team, maybe picking up some New Bamboo “goodies”:http://www.flickr.com/photos/newbamboolondon/1442939714/ come and say hello!

With Web 2.0 applications facing growing audiences with data hungry appetites “MOM”:http://en.wikipedia.org/wiki/Messageorientedmiddleware has suddenly had a resurgence.

Messaging systems are a great way of scaling your site by placing operations/messages on a queue and processing them asynchronously. This allows you to get quite imaginative with your architecture and distribute worker processes in geographically disparate locations if you so wished and also distribute/delay the actual processing needs as and when you can.

This can be used with great effect with services such as Amazons own simple messaging offering known as SQS and EC2. As soon as load starts peaking on one EC2 node you can have a script that calls out to a mother-ship to instantiate another EC2 instance and as soon as it’s ‘born’ starts processing messages off the SQS queue. If you wanted you could even have the original load stricken node contact the mother-ship via SQS (probably a more reliable solution). This is one of the architectures I’m currently advocating as EC2 is stateless and rather than worry about database replication across transient nodes, I try and share as much information as I can using messaging and web-services. I’m currently backing off exposing REST style interfaces for non public facing services purely for performance and scaling reasons. Don’t get me wrong, I love REST and for public a public facing API it makes life a lot easier. But as “Zed Shaw”:http://www.zedshaw.com/ says, HTTP isn’t exactly the most performant protocol.

I’ve seen people try and achieve similar architectures using a database and their own cobbled together mesh of scripts and while this may work great on small sites, you’re just pushing your scaling issues and problems onto your poor DB. And as anyone who’s had the misfortune to be stuck in a room with me would tell you, I hate relational databases on multiple fronts, they’re so 1980’s and I’ll tell you why in another blog post.

It’s seems these days that we’re currently quite spoiled with the cheap/open messaging solutions that are available. A few years ago most of what I had to play with were proprietary and expensive offerings from various large vendors. A few that I’ve played around with lately have been:

• “ActiveMQ”:http://activemq.apache.org/stomp.html - An open source solution utilising STOMP. Check out “ActiveMessaging”:http://code.google.com/p/activemessaging/wiki/ActiveMessaging for Ruby/Rails.
• “Jabber”:http://www.jabber.org - While it’s not really intended for reliable delivery of messages, it does have a lot of support and with quite a few Ruby clients available.
• “SQS”:http://aws.amazon.com/sqs - Amazons offering that is easy to plugin solution if you’re using EC2 as you don’t need to worrying about setting up the messaging infrastructure between distributed nodes that aren’t necessarily sharing a VPN. ActiveMessaging has support for SQS though I kinda feel it’s bit of a chain saw to a bunny rabbit as SQS really is a very simple service as it’s name suggests. I wrote my own lightweight SQS Rails plugin a while ago and will probably dust it off in the next week or so.
• “RabbitMQ”:http://www.rabbitmq.com - Now this is the matrons knockers when it comes to kick ass, ultra fast and scalable messaging. It simply rocks, with performance off the hook. It’s written in Erlang and supports the “AMPQ”:https://jira.amqp.org/confluence/display/AMQP/Advanced+Message+Queuing+Protocol protocol. Unfortunately I haven’t seen much in the way of Ruby support. I’ve played with “QPid’s”:http://cwiki.apache.org/qpid/ (Apaches own open source AMPQ offering) “Ruby client library”:https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/ruby/ but haven’t actually tested this against RabbitMQ, so you’re mileage may vary.

So there you have it, messaging. It rocks, it’s sexy and what’s more it’ll make you sexy… Honest guv’nor.

Find Things, Do Stuff

This is the methodology behind jQuery and it’s very sensible. The whole API is geared towards selecting elements on the page and then performing actions on that collection of elements.

Finding Things

$() is the root of everything, it allows you to search the DOM for elements and wrap them in jQuery functionality, searches can be passed in as CSS selectors, XPath selectors or native DOM nodes. The return value is always a jQuery collection, regardless of whether there is 1 match, many matches or no matches. This allows you to chain methods on without having to worry about the contents of the document.

Doing Stuff

$() isn't the only method that returns a jQuery collection, all the functions in the library work this way (with a few exceptions where it makes sense), because of this you can easily chain methods onto each other. Method chaining makes a lot of sense in jQuery as it provides a way to build up complex functionality ontop of all the matched elements. Let's look at a simple example.

$('a.button').html('hello').show()

Here we are getting all links with a class of button, then setting there html to say ‘hello’ and finally we ‘show’ the element on the page (making the assumption it was hidden to begin with).

jQuery has a rich yet simple API for ‘doing stuff’ to DOM elements, this includes:

• DOM Manipulation
• Traversal
• attribute and style getters and setters
• Events
• AJAX
• FX

Plugins

One of jQuery’s greatest strengths is it’s size. By concentrating on selecting elements and manipulating them they have managed to keep the core API lean. But when additional functionality is needed writing plugins couldn’t be easier, this is reflected in the number and quality of plugins already available on the website.

jQuery & Rails

I won’t lie to you, jQuery and rails isn’t always as natural a fit as Prototype however I think it is worth the little extra effort for the benefits of a lighter page size and truly unobtrusive JavaScript code.

If you’re a RJS fan then there is some helpful information available, and there is also some promising work going on in the jQuery community to bring Rails & jQuery closer together via a rails plugin.

Links & Downloads

• Talk@2 Presentation (pdf)
• The simple example we built (zip)
• jQuery Website

The Results Are In!

As a company we have embraced jQuery in several projects recently and the results so far have been very positive so watch out for more from us in this area.

Presenters

We have had little use for presenters, so the example is a contrived one, but I liked to include it for completeness sake. (View the slides for the full progression from bog-standard MVC to using the presenter)

# app/controllers/users_controller.rb
class UsersController
  def show
    @user = User.find(params[:id])
  end
end

# app/views/users/index.html.erb
<div>
  <% if CONFIG.date_format == :us %>
    <%= @user.created_at.strftime('%m/%d/%y') %>
  <% elsif CONFIG.date_format == :rest_of_the_world %>
    <%= @user.created_at.strftime('%d/%m/%y') %>
  <% end %>
</div>

# app/presenters/user_presenter.rb
class UserPresenter
  def initialize(user)
    @user = user
  end
  attr_reader :user

  def signup_date
    if CONFIG.date_format == :us
      self.user.created_at.strftime('%m/%d/%y')
    elsif CONFIG.date_format == :rest_of_the_world
      self.user.created_at.strftime('%d/%m/%y')
    end
  end
end

Conductors

Conductors are extremely useful, having two major uses that we have come across so far. Firstly managing an object with associations submitted from one form (the example I use in the demo application). Or secondly to manage one object that has its information gathered from multiple forms.

The demo is a simple application to manage company information and payment details. Using a conductor to manage the company and its associated credit card and avatar we greatly simplify the model and controller. I have also abstracted some repetitive stuff into a super class, which provides some DSL-ish methods for describing how the conductor works (albeit verbose). The conductor in the demo application is shown below:

class CompanyConductor < ActionConductor::Base
  conduct :company do |company|
    company.name
    company.phone
    company.website
  end
  
  conduct :credit_card do |credit_card|
    credit_card.owner_name :as => :card_holder_name
    credit_card.provider :as => :credit_card_provider
    credit_card.number :as => :credit_card_number
    credit_card.expiry_month
    credit_card.expiry_year
  end
  
  owner :company
  
  def credit_card
    @credit_card ||= (self.company.credit_card || self.company.credit_card = CreditCard.new)
  end
  
  def avatar
    @avatar ||= (self.company.avatar || Avatar.new)
  end
  
  def uploaded_data=(data)
    unless data.blank?
      self.avatar.uploaded_data = data
      @avatar_present = true
    end
  end
  
  def save
    if @avatar_present
      company.avatar = avatar if avatar.new_record?
      avatar.save
    end
    company.save && credit_card.save
  end
  
  def errors
    method_map = @reversed_method_mappings
    
    errors = ActiveRecord::Errors.new(self)
    errors.add_conductor_errors_for(company, method_map)
    errors.add_conductor_errors_for(credit_card, method_map)
    errors
  end
end

So what advantages do we get from using a conductor? Firstly our code is much more concise. Compare the create action before and after adding a conductor.

# Create action without conductor
def create
  @company = Company.new(params[:company])
  @credit_card = CreditCard.new(params[:credit_card])
  @company.credit_card = @credit_card
  
  if (@company.valid? & @credit_card.valid?) && (@company.save & @credit_card.save)
    unless params[:avatar][:uploaded_data].blank?
      @avatar = Avatar.new(params[:avatar])
      @company.avatar = @avatar
      @avatar.save
    end
    flash[:notice] = 'Company was successfully created.'
    redirect_to companies_url
  else
    render :action => 'new'
  end
end

# Create action with conductor
def create
  @company = Company.new
  @company_conductor = CompanyConductor.new(@company, params[:company_conductor])
    
  if @company_conductor.save
    flash[:notice] = 'Company was successfully created.'
    redirect_to companies_url
  else
    render :action => 'new'
  end
end

Secondly as much of that logic was duplicated between the create and update actions, we get a much DRY-er controller. Thirdly the views become more consistent, as we can now use f.text_field (and friends) instead of before where we had to use f.text_field for the company and then text_field for the associated objects.

The main thing to be said against conductors is that they add another layer of abstraction to deal with. Which increases the amount of time learning what the code does. However this can easily be solved by setting some conventions on how they are used. We haven't experimented nearly enough to come up with any good conventions that are suitable in the majority of cases. One of the possible additions we looked at is to infer the attributes to forward to the conductor from the model's attributes, to avoid the verbose conduct definitions.

The abstraction, that I whipped up for the demo, is lacking a bit of polish and is downright ugly in places. But feel free to download it, play with it, use it and improve it.

Files & Linkage

Presentation Slides (PDF)
Demo Application - The demo application I made for the demonstration. Interesting stuff is in the app/ and lib/ directories.

Jay Fields' blog post on the presenter pattern upon which this conductor is based.
Model-Conductor-Controller (MCC) - another take on the conductor pattern.

It would be great to hear your thoughts on presenters and conductors. Comment away!

ESI is very similar to SSI but has a more sophisticated method for including content that can use things like a try-catch type DSL to ask for content and fall back to alternatives if something goes wrong. To put this in context and to highlight why it is really cool, consider Rails’ whole page caching. An app can write an entire action to a static file in the public directory. Your server can pass this file to its clients without touching Rails and its routing system. This has always seemed like a fantastic concept, but one that can hardly ever be used in the wild. Necessary elements in the page such as shopping cart contents, links to your account, user’s status etc., always spoil the fun when you are trying to avoid hitting the app. This forces you to use fragment caching and model caching when caching 99% of the page at the level of Mongrel would be far more ideal.

Enter ESI (SSI will also do, but ESI has some additional nice features). Whole actions can be cached as static content above the level of your app, but embedded in them is markup that the server executes to fetch more content where necessary. A high traffic shopping site could have something along these lines, in place of some content in the page that displays a cart:

...
  <esi:include src="/cart" max-age="0"/>
...

The users’ session is passed through, and you can retrieve the items in their cart and render it as on a mini action in your Rails app. The server is responsible for inserting it into the markup and returning it to the client. The max-age of 0 means that it will ask for a fresh version from your app everytime it is run. However, setting it to a higher TTL value means that subsequent request will use a cached copy that the server stores and is responsible for. The really cool thing about this is that it puts responsibility of dealing out cached content onto your web server, rather than the actual Rails app. At the moment you need a custom version of mongrel that supports ESI to use it. Also, it is worth noting that the current version of mongrel-esi stores these cached fragments in memory, though the guys are working on a memcached version. Have a “look here”:http://revolutiononrails.blogspot.com/2007/08/fragmentfu-fun-with-fragments.html to get the install details and also a plugin that helps you integrate it into your Rails apps.

This is obviously cool stuff, but there are some other neat things you can add into the mix. The Rails default method of storing the cached pages is on the file system in your public directory, which gets a bit messy. However, in “this article”:http://blog.kovyrin.net/2007/08/05/using-nginx-ssi-and-memcache-to-make-your-web-applications-faster it describes putting them into memcache and then telling nginx to look in memcache before hitting your app to retrieve them. This is pretty fantastic as it makes managing the cache much easier.

Another option to consider for delivering the fragments would be a lovely lightweight merb application, since the src attribute of an esi:include can be another url, it doesn’t just have to be a path on the same server. I haven’t actually tried this, so it should be taken with a pinch of salt, but I think you should be able to do this:

  <esi:include src="http://super_merb.my_app.com/cart" max-age="0"/>

So things are looking pretty nifty in this area, and I am definitely going to be keeping an eye on it.

I haven’t been to Berlin in nearly 7 years when I worked at “Pixel Park”:http://www.pixelpark.com so I’ll be looking forward to seeing the sites and meeting fellow Railers! If you’re going to be at Railsconf Europe or just in the vicinity it would be great to meet up and hang out.

We have had the pleasure of working with the Seedcamp team throughout from the site to development of the application, application. Knowing several teams in London taking part, there’s going to be some stiff competition, that’s for sure.

I am not going to highlight the all of the advantages of using git here, but let me mention that Git is one of the few SCM out there that really understands merges.

As Linus, the creator of git, says “in this thread”:http://www.gelato.unsw.edu.au/archives/git/0504/2182.html bq. “The important part of a merge is not how it handles conflicts (which need to be verified by a human anyway if they are at all interesting), but that it should meld the history together right so that you have a new solid base for future merges. In other words, the important part is the trivial part: the naming of the parents, and keeping track of their relationship. Not the clashes. And it looks like 99% of SCM people seem to think that the solution to that is to be more clever about content merges. Which misses the point entirely.”

Git tracks content, not files. Git is really a filesystem, represented by tree structures. So you are really comparing trees between commits, and tracking the changes of the nodes in the tree.

There are enough tutorials out there for you to start grasping git. I can recommend these ones:

• “Beginners tutorial”:http://www.kernel.org/pub/software/scm/git/docs/tutorial.html
• “Git user manual”:http://www.kernel.org/pub/software/scm/git/docs/user-manual.html
• “Git for SVN users”:http://git.or.cz/course/svn.html

You may want to consider Git if you have ever run into a merge marathon on a big project with SVN. If you have ever tried it, it is just painful and it does not work, not the SVN way.

h3. Enter Git

Git makes branches easy to create and merge. Git branches are only a 40-digit hex hash, with the origin of the branch. Branches are cheap, and easy. You start using git, and before you realise it, you are merging and branching all the time.

So a regular working pattern would be:

------------------------------------
git branch mywork
git checkout mywork
...hack some files (file1 file2 file3)...
git add file1 file2 file3
git commit
git checkout master
git merge mywork
------------------------------------

Yes that’s good, but what happens when all your development repositories are under SVN control? Well here comes a little script that will help us integrate with our existing repositories.

h3. Enter git-svn

git-svn is a little perl script (that comes bundled with the Git distribution) that lets you track SVN repositories. You can also commit back between Git and SVN. This is very good, you can start using git without converting your working repositories yet… well until you are ready to migrate them.

h3. GIT RAILS RECIPE

mkdir beast
cd beast
git-svn init http://svn.techno-weenie.net/projects/beast/trunk  (for example) 
git-svn fetch -r2940                                            (get a particular repository revision)
git-svn fetch                                                   ( get the whole repository history)

then REPEAT{ 

git checkout -b mywork        (start my own branch)
   ....START HACKING          (fix some bugs)
git commit -a                 (commit the changes to git)
git checkout master           (switch to master branch) 
git-svn rebase                (update master with the upstream svn repository)
git checkout -b mymerge       (create a branch mymerge and checkout from master)
git merge --squash mywork     (merge into current branch, mywork branch) 
git commit                    (commit the changes to git)
git-svn dcommit               (commit the changes to svn repository....if you have enough permissions)
git checkout master           (switch to master branch) 
git-svn rebase                (update master with the upstream svn repository again)
git branch -D mywork mymerge  (get rid of branch mywork and mymerge)

}

See http://cheat.errtheblog.com/s/gitsvn/ for a cheat sheet on this.

Here James Bowes tells us why rebase is important: http://jbowes.dangerouslyinc.com/2007/01/26/git-rebase-keeping-your-branches-current/

There is one problem with git-svn. It can not handle svn:externals yet.

But there is a workaround to it. Once you fetch your RAILS svn project, you may want Rails edge in your vendor directory, or some handy plugins as well.

Get RAILS EDGE into a separate git clone as well as your handy plugins. Then do a symlink into the vendor directory.

mkdir rails_edge
cd rails_edge
git-svn init http://svn.rubyonrails.org/rails/trunk  (for example)
git-svn fetch -r7300  (get just the latest revision you know about) 

git-svn fetch   (perform a fetch once in a while to keep it up to date)

then cd myproj/vendor
ln -s ../../rails_edge rails

Perform the same task for every plugin you use. You may also track a stable version of rails like http://svn.rubyonrails.org/rails/branches/1-2-stable

There are pros and cons of this approach. You can decide if this is enough for your current needs or wait until git supports subprojects. Keeping Rails stable and edge in different repositories, and tracking their changes independently seems to be a very clean way to work.

I hope that gives you a warm welcome to the world of git, and that you start enjoying Rails development along with Git.

Source: Mike Riley

From the start, I have to warn you that this melodramatic title may not have been terribly well considered. When Shelley asked me what my talk at 2 was going to be, I responded in rather a flustered manner without thinking through the ramifications. I think a sense of rather foolhardy bravado may have seeped into the name of the talk, and I am not ashamed to admit that.

Not wanting to disappoint, I wracked my brain for Actionscript 3 scenarios which could relate to the theme of Dragons. No luck. Cursing the English Literature degree that had seemingly created this intractable situation, I got to work on the talk.

The thing I wanted to convey most is that making things with AS3 is not really hard or indeed Evil. Many people have misconceptions of Flash, and most Rails developers I know don’t really like the idea of playing around with it at all. In my experience, Flash is a really great medium for doing loads of different things, though definitely not a replacement for HTML/CSS/JS in most cases. I really enjoy creating stuff in Flash, and intended to get some people to listen to me.

Also worth noting, this talk was not about Flex. I have experimented with Flex, and kind of like the idea behind it for RIAs. Sometimes though you just want to work with the actual language behind the interface, rather rely on a whole bunch of classes that get mysteriously instantiated from your MXML.

I wanted to educate people about the fundamentals of how these interfaces are build from the primary elements. We cover creating sprites on the stage, importing other classes, attaching behaviour and some of the basics of drawing. Sticking with the mythological theme (though not of a dragonesque nature), the examples are set in the depths of time in Ancient Greece. Gods chill out on Mount Olympus, and everything is hunky dory. With that in mind, here is the content of the talk. You can get the files here:

Zeus

The first example we look at is as simple as you can get. We put this in the file ‘Main.as’ in the example called 01_zeus. When this is compiled, we include the Sprite class and make a class called Main that is inherited from it. The constructor function is named the same as the class, so when this is compiled and run, the Main function is called when an object of class Main is initialised and added to the stage.

In this Main function is where all the magic happens, we create a new Sprite (not a pun) called zeus, and start drawing his likeness. Now whatever people may claim, no-one has ever seen a greek god, and I don’t believe the likenesses of them on pots are right. In my opinion, Greek gods are all square in shape, and covered in colourful hair. This has nothing to do with it being easier to draw sprites like that, so don’t even suggest it. We specify that zeus will be black, and 50x50 pixels. We do this by calling methods on the Sprite’s built in graphics object. once we have draw him, we add him to the stage for all to admire. For fun, we nudge him into the middle a bit more to demonstrate how Sprites have x and y coords for positioning them.

  package {
	import flash.display.Sprite;
	
	public class Main extends Sprite {
		
		public function Main():void {
			var zeus:Sprite = new Sprite();
			zeus.graphics.beginFill(0x000000);
			zeus.graphics.drawRect(0, 0, 50, 50);
			this.addChild(zeus)
			zeus.x = 30
			zeus.y = 30
		}
	}
  } 

“Wow”, I hear thee cry, “surely the talk was over at this point and roses were being cast at your feet?”. This was not the case Gentle Reader, greater marvels were to appear shortly.

Athena

Now Zeus is a bit old and cannot move very well. To demonstrate a bit more action we pan to Athena. In this project (02_athena) we have a new directory called “gods”. This will contain all our other classes we are going to use. In our Main.as we import “gods.Athena;” which is the class in gods/Athena.as, and add it to the stage like we did with Zeus. Athena has all her internals nicely encapsulated inside her, unlike slovenly Zeus. She can do this because she is created from a class that extends Sprite and can add to it as we want. We have to import all the classses we are interested in inside the package that contains Athena. In this case we want the sprite class, and all the events “flash.events.*;” which we use a wildcard for.

When Athena is instantiated, we draw her as a square (totally correct), but her class has a public variable called hair color:

public var hairColor:Number = 0xe4a406;

Olympus

Again, a miraculous thing, but the fun doesn’t stop there. We now decide to make a home for all these gods, and feel that Olympus is a fiting name. In 03_olympus, we make an olympus object and add it to the stage. Like we did with the gods, we represent Olympus as a square, it is big and grey. Olympus has an event handler on it which takes mouse clicks and creates new instances of RandomGod on the x and y of the event. Clicking on the stage creates a multitude of gods. Unfortunately there is a bit of a bug here which was noticed afterwards and that I haven’t fixed, if you click on a random god, the x and y are taken from it rather than Olympus. Not the end of the world, but there ya go.

Naughty Gods

In the final example a bit more goes on. We create olympus as before, and also create zeus with his own special class. The random gods are populated automatically this time, and Olympus takes care of creating them when it is instantiated. Unfortunately these gods have been drinking some rather bad wine. This has the unintended consequence of turning their hair green, and making them rather unruly. Intolerable.

Zeus, not gifted with ample amounts of patience for his immortal responsibilities responds to this chaos by delivering a thunderbolt into the rabble, killing them dead.

Let’s analyse what is involved in this catastrophe.

When Zeus was created, he was not created equal. The Zeus class has the following:

		public function Zeus(posX:Number, posY:Number):void {
			...
			this.addEventListener(MouseEvent.MOUSE_DOWN, mouseIsDown);
		}

		public function mouseIsDown(event:MouseEvent):void {
			strikeDownNaughtyGods();
		}
		
		public function strikeDownNaughtyGods():void{
			var lightning:LightningBolt = new gods.LightningBolt(this, olympus.naughtyGods);
			olympus.addChild(lightning)
		}

When clicked, Zeus calls the function strikeDownNaughtyGods(). This makes a new ferocious lightningbolt taking Zeus (this), as the thrower, and olympus.naughtyGods as the targets. The variable olympus is just a reference to olympus we set on zeus at the beginning. When our lightnning bolt is made, the fun starts. The graphics object is called, and we say we are going to start drawing with the thickness of boltWidth and a nice blueish colour. the drawing pointer is moved to the center of the srcGod, and we send the bolt off.

 package gods{
  import flash.display.Sprite;
  import flash.events.\*;
  import flash.utils.Timer;
  import flash.events.TimerEvent;
  import flash.display.Sprite;
  
  public class LightningBolt extends Sprite {
    
    ...
    
    public function sendBolt():void{
      killCount = 0;
      timer = new Timer(500);
      timer.addEventListener("timer", timerHandler);
      timer.start();
    }
    
     public function timerHandler(event:TimerEvent):void {
        if (killCount == targetGods.length){
          timer.stop();
          this.graphics.clear();
        }else{
          this.graphics.lineStyle(boltWidth*=0.8, 0xb2ebf4);
          this.graphics.lineTo(targetGods[killCount].heartX(), targetGods[killCount].heartY());
          targetGods[killCount].kill();
          killCount++;
        }
     }
  }
 }

A Timer object is created, and every 500 miliseconds it will call timerHandler. When we haven’t killed all the gods yet, the bolt width diminishes to reflect the principle of Conservation of Energy, and then we draw a line to the next god in the array. This new god has the method kill() called on him, which in itself is scary. This method makes him change colour, shake from side to side and finally expire, turning red in the process. When the naughty gods are dealt with, the timer stops, and Zeus can read a book in peace.

So there you have it, some basic principles explained. I am by no means an expert, I just play around every now and again for fun. Hopefully a few brave souls will have a go and come up with some other neat stuff. I have found that graphing in particular is an area where Flash shines, especially where it is being fed xml from a beautiful Rails app. Maybe I will blog about my recreation of a zoomable graphing for showing tons of data at some point if anyone is interested (a bit like Google Finance, but made by me).

With this in mind, we have been trying out a new scheme in New Bamboo Towers to encourage sharing of all those crazy gems of knowledge we discover in our own time. Thus the “Talk at 2” came into being (Damien has suggested “talk@2” but this seems a little 2002 to me). On Friday afternoon, one member of the team presents something that they find particularly interesting, and that other people might not be familiar with. It can be related to anything really, but usually has an element of programming in it.

So far Jonathan has done a talk on rSpec, and on Friday I presented an unbelievably useful and emotional piece about ActionScript 3 that moved many of the guys to tears. We will try to get these written up after the event so everyone can share the fun.

At this point I have to give a disclaimer that this might not happen every week, sometimes there just isn’t enough time, but we will certainly try to be diligent with it for as long as we can.

I also feel it needs a theme tune and maybe a logo, but don’t feel capable of creating either. In my head, the theme tune should be a mixture of the Countdown theme tune, and the Darth Vader music from Star Wars. If anyone can come up with something that encapsulates this, I’d be very interested in hearing from you.

With BDD you’re interested in the behavior of the object you’re testing.. Simple duh! OK so I’ve just given you the Zen master says little but explains a lot explanation, not very helpful I now. Anyway, objects in a system never work in solitary at least in any sort of semi-complex system Of course you could cheat and do what one supposed “Sun”:http://www.sun.com certified engineer I interviewed once did and write a whole forum in a monolith of a class, but that’s another story and it’s wrong for me to mock the man like this (did you see the pun! Mock! Ha! Ah, stop throwing the tomatoes!).

Now that I’ve managed to totally side track this conversation I’ll get us back onto topic. The other entities that an objects interacts with are called collaborators and with BDD we’re interested in how our objects interact with each other via their interfaces and verifying these interactions. The way we do this in BDD is by the use of mock objects. With a mock object you can not only stub out methods but also attach assertions to verify behavior of the object with it’s collaborator. And this is what we’re interested in, behaviour not state.

We’re not interested in verifying the return value i.e the collaborators state. This is actually something that shocks quite a few Rails developers who are entrenched in their world of fixtures and testing concrete instances. Sure, I’m not saying that I never instantiate a concrete collaborator and use it in memory but I’m very rarely interested in state especially in my unit tests and the Rails ‘functional’ tests.

The point at which I get interested in state usually occurs around the point of my integration tests and end to end tests. Being interested in behavior rather than state can be quite a big mind shift for a lot of developers but if you give it a try I promise that you’ll find your path to testing will be a more enjoyable one.

One solution to this is to remove the directories manually, and optionally save the project for the future. But that is boring, and can lead to mistakes. Enter neato trick in the form of folder references. If you go to Preferences > Advanced > Folder References, you can see the regular expression in the second box for folders that TextMate will ignore.

If you change it to this, your log directory will be ignored automatically:

!.*/(\.[^/]*|log|CVS|_darcs|_MTN|\{arch\}|blib|.*~\.nib|.*\.(framework|app|pbproj|pbxproj|xcode(proj)?|bundle))$

If you are feeling Spartan, you could also add sessions, cache, rails and anything else you don’t search often which that slows search down.

I hope this is helpful to someone, it seemed rather obvious but overlooked feature to me.

We’re hiring across the board from juniors just out of univeristy to seniors with years in the web industry.

For all applicants we’d expect the following:

• Experience developing web applications
• Scripting/programmer experience of some kind
• You like using Macs
• You’re comfortable in a Unix environment
• Good OO knowledge
• At least 6 months experience with Ruby and/or Ruby on Rails
• An eye for detail

For senior level developers we expect:

• TDD experience
• Ruby on Rails experience
• Scripting experience in another language apart from Ruby
• CSS and XHTML/HTML knowledge
• Good knowledge of JavaScript
• Understanding of common design patterns

For junior developers you’ll be given mentoring/guidance and if you’ve contributed to any open source projects then we’d consider this as extra brownie points.

Salaries are dependant on experience and start from £23K+ for juniors and even more for seniors.

Please send your CV and salary expectations to iwantajob@new-bamboo.co.uk

• Install “MacPorts”:http://www.macports.org/
• Start up the Io VM (Assuming you have /opt/local/bin in your path)

  osxvm

Enjoy

Subjects of tonights talk will be:

*  A brief overview of Ruby on Rails
* REST in Rails
* Javascript-less Javascript with RJS
* Beautiful testing with Rails (This will actually be about RSpec)

If you’re a member of the BCS then please drop by. Also if I have time I’ll try and make it down to the Oxford geek night tonight.

A big thanks to everyone who joined us on the night and those who have supported us along the way. We had a really awesome evening and hope you enjoyed it as much as we did!

We are going to try to get some photos up on Flickr in the next few days, but many of the ones we have collected seem to have been taken by persons in an advanced state of inebriation… mysterious.

Now we just need to think of a new excuse to celebrate so we can keep the parties going…

We’ve all heard about unobtrusive Javascript and hopefully you’ve had a look at “Dan’s”:http://www.danwebb.net and “Lukes”:http://www.lukeredpath.co.uk “UJS4Rails”:http://www.ujs4rails.com plugin. Which allows you to still use the Rails helpers in an unobtrusive way but more importantly declare behaviors in an easy and declarative manner. It’s also a great springboard into the world of unobstrusive Javascript for those people who’s Javascript-fu isn’t yet strong enough to roll your own.

With the release of “Prototype 1.5.1”:http://prototypejs.org/assets/2007/3/12/prototype.js rolling your own just got a bit easier. Let’s take your standard rails form (ignore the loader gif, I’m getting a bit ahead of myself).

<% form_for(:section,@section, :url => sections_path, :html => 
  {:id => "new_section_form"}) do |f|%>
	<%= f.text_field :name %>
	<%= submit_tag 'Add Section'%>
        <%= image_tag('spinner.gif', :id => 'spinner_create', 
                                              :style => 'display:none;') %>
<% end %>

It’s a pretty standard form that creates a section given a name, nothing exciting but it works and does this in a non Ajax way. Now this is the bit of Javascript that I would probably use before in order to Ajaxify this form:

$('new_section_form).onsubmit = function(){	
	  if(!$(this.name).present()){alert('Please enter a name'); 
                                                return false;}

		new Ajax.Request('<%= sections_path) %>', {
			asynchronous: true, 
			evalScripts: true,
			method: 'post',
			parameters: this.serialize(), 
			onLoading: function(request){
			    Element.toggle('spinner_create');
			  },
			onFailure: function(request){
			    alert('Sorry, there were problems processing your request.\n 
                                   Please try again.');
				},
			onComplete: function(request){
				Element.toggle('spinner_create');
			  }
		});
		return false;
	};

Nothing exciting here really, the code would normally be called in a dom ready block in case you were wondering. Apart from looking a bit verbose you’ve probably noticed that I’m not being very DRY in that I’ve duplicated information that could and should be retrieved from the actual form. Things such as the path to post the details to and the method in which to do this (in this case ‘post’).

Yeah I know I could just use a bit more javascript-fu to get those details (which I normally would do especially if I wanted to separate this code into a another JS file), but again this is going to get even more verbose. No wonder people like to use the Rails Javascript abstractions.

Let’s tidy this up a bit. In Prototype 1.5.1 the guys have given you the request() method which does pretty much the same as the Ajax.Request method. In fact it even accepts the same optional parameters. The main difference though between request() and Ajax.Request() is that the former gets all those little details it needs for the form itself so the above now can be shortened to this:

$('new_section_form').onsubmit = function(){
		if(!this.section_name.present()){
			alert('Section name cannot be blank');
			return false;
		}
		this.request({
                   onLoading: function(){
                                     Element.toggle('spinner_create')
                  ;}, 
                   onComplete: function(){
                                    Element.toggle('spinner_create');
                  }
                 });
		return false;
	}

I could of probably extracted the spinner out to a separate variable and called toggle on it to make it even DRY’er, but at least you get the idea. There are lots of other little improvements and goodies to be found in the new release of Prototype and I seriously encourage you to play around and have some fun.

Tagged with “Prototype”:http://technorati.com/tag/Prototype, “Javascript”:http://technorati.com/tag/Javascript, “Unobtrusive Javascript”:http://technorati.com/tag/Unobtrusive%20Javascript

But seriously things are pretty busy and we’re on the look out for someone with crazy Ruby skills. If you’re interested get in touch. Contact details are to be found on our main company website.

• “S3DFS for EC2”:http://www.openfount.com/blog/s3dfs-for-ec2 - Now we can really scale with EC2, no more worries about where the database is.

• “8 OpenID Resources for Rails Developers”:http://www.rubyinside.com/7-openid-resources-for-rails-developers-418.html - Everyone’s at it these days.

• “Sequel”:http://code.google.com/p/ruby-sequel/ is an ORM framework for Ruby. Sequel provides thread safety, connection pooling, and a DSL for constructing queries and table schemas - Have to check this out!

!http://blog.nbwd.co.uk/assets/2007/3/2/squawk.png!

“Download the Squawk ruby script”:http://pastie.caboo.se/44304

Now when I first started doing TDD over six years ago the major thing that struck me was how well designed the interactions between associated objects were compared to applications I’d written in the past without TDD. This was all because by writing test first I was forcing myself to think about how my objects were going to interact with my tests and thus with other objects within the system. Another thing I discovered was also how using mocks even more so allowed me to concentrate even more on the interfaces of the object while developing my models. OK so thats a bit of a lie, back then when I was doing TDD I was doing J2EE development with EJB 2.0 and as we all know this led to anaemic models because all our business logic had to be pulled in via service objects. So no, back then I wasn’t thinking too much about the interfaces of my “models”, more on my service objects;).

I’m going to run through a fairly trivial example just to give you a little taste of what I’m talking about. I’m going to be using “RSpec”:http://rspec.rubyforge.org with its built in mocking support in my examples but I could of easily used Test::Unit with “Flexmock”:http://onestepback.org/software/flexmock/ or “Mocha/Stubba”:http://mocha.rubyforge.org/.

Step one - Make a cup of tea

Step two - Install the RSpec gem

$ sudo gem install rspec

Step three - Create a Rails application, in this case we’re going to work on a small part of an auction application.

$ rails rspec_auction
$ cd rspec_auction

Setup both your development and test database by editing the database.yml. We won’t actually be hitting the database in the tests but unfortunately you’ll need to set it up.

Step four - Install the RSpec on Rails plugin and generate the helpers and directories that the plugin needs.

$ ruby script/plugin install \ 
svn://rubyforge.org/var/svn/rspec \
/tags/REL_0_7_5_1/rspec_on_rails/vendor/plugins/rspec_on_rails

$ script/generate rspec

Step five - Make another cup of tea.

Step six - Lets create an Auction model. Now the RSpec plugin for Rails provides you with several generators which will be familiar to you if you’ve used the Rails generators before.

$ script/generate rspec_model auction

Step seven - Hopefully you’ve already got the project open in a text editor or your favourite IDE. We need to make a small alteration the the file spec_helper.rb otherwise we’re going to have problems with Activerecords type checking. Yes, you heard me correctly, I said type checking;). Anyways at the bottom of the spec_helper.rb file you’ll need to add the following lines.

module ActiveRecord
  module Associations
    class AssociationProxy
      def raise_on_type_mismatch(record)
        # do nothing
      end
    end
  end
end

Oh and you’ll also need a basic migration file for your auction model so in the file 001_create_auctions.rb that’s already been generated for you add the following:

class CreateAuctions < ActiveRecord::Migration
  def self.up
    create_table :auctions do |t|
      t.column :title, :string
      t.column :description, :text
      t.column :updated_at, :datetime
      t.column :created_at, :datetime
      t.column :expires_at, :datetime
      t.column :starting_bid, :float
      t.column :active, :boolean
    end
  end

  def self.down
    drop_table :auctions
  end
end

Step eight - OK, now we’re all set to do some coding. Open up the auction_spec.rb file and the first thing you’ll see is something called a context with a description and specifications. When testing our model we may have multiple contexts all depicting the model in different states such as a new auction, an auction about to expire or an auction that has been canceled for example.

In this case the default context generated for us is “Given a generated auction_spec.rb with fixtures loaded”… Not exactly what we’re looking for. So delete this entire context including the specification and replace it with this:

require File.dirname(__FILE__) + '/../spec_helper'

module AuctionSpecHelper
  #Valid attributes for an auction
  def self.valid_auction_attributes
    {
     :title => "Big bamboo",
     :description => "One caring owner, still in good condition",
     :expires_at => 7.days.from_now,
     :starting_bid => 0.20,
     :active => true
    }
  end
  
  def self.invalid_auction_attributes
    {
      :title => nil,
      :description => nil,
      :expires_at => 7.days.ago,
      :starting_bid => "moo",
      :active => true
    }
  end
  
end

context "A general auction" do
  
  setup do
    @auction = Auction.new
  end
  
  specify "should have errors given invalid values" do
    @auction.attributes = AuctionSpecHelper.invalid_auction_attributes
    @auction.valid?.should_equal false
  end
  
  specify "should have no errors given valid values" do
    @auction.attributes = AuctionSpecHelper.valid_auction_attributes
    @auction.valid?.should_equal true
  end
  
end

The module helper at the top of the source should be self explanatory. I’m using it to store hashes of invalid and valid attributes for an auction.

Run the model spec’s by typing the following at the command line:

rake spec:models

You’ve probably noticed that I’ve shot ahead and added not one but two specifications. There wouldn’t be much point in the “should have no errors” specification as I haven’t actually written any constraints in the auction model yet. Really my tests or specifications in this case should always fail first and then it’s up to me to do what ever is required to make them spec pass…. Even if involves secret underground Artic bases… Eerrmm OK, maybe not. Anyway, as you can tell I’ve added the first specification in at the top to intentionally fail. Let’s make this pumpkin pass. Make your auction.rb look like this:

class Auction < ActiveRecord::Base
  
  validates_presence_of :title
  validates_presence_of :description
  validates_numericality_of :starting_bid
  
end

If you run the specifications now you’ll see that they all now pass.

Step nine - I’m going to digress for a minute and talk about stubs and mocks. If you don’t already know what a stub or a mock is then “here”:http://www.martinfowler.com/bliki/TestDouble.html is a good description of the various terms. Now you’ve probably been told to use stubs and/or mocks when working with systems external to your application, 3rd party libraries or dependencies that aren’t solely related to what you’re currently testing. In this case the database is a dependency so lets stub out any database calls. The only problem is that we’re currently not hitting the database, so let write a specification that would normally hit the database, but errrm doesn’t. Sorry if I sound like a fortune cookie. Add this context and specifcation to the auction_spec.rb file.

context "An auction that we want to remove" do
  
  setup do
    @auction = Auction.new
    @auction.attributes = AuctionSpecHelper.valid_auction_attributes
    @auction.stub!(:save!).and_return(:true)
  end
    
  specify "should be able to deactivate auction" do
    @auction.deactivate!
    @auction.active?.should_equal false
  end
  
end

The first thing you’ll see is that I’ve actually stubbed out the default Activerecord save! method of the auction model so that it doesn’t hit the database. You currently don’t see me calling save! but thats a coming implementation detail which will become obvious in a few minutes. Of course if you try running the above you’ll get a few errors, the first being that there isn’t a deactivate! method on auction yet. Let’s fix that now, inside auction.rb add:

def deactivate!
    self.active = false
    #Aha! The fabled save! method!!!
    self.save!
 end

OK, the spec should now pass, but you know what? Call me picky but I’d like to hide the active attribute of the model, I don’t want any Tom, Dick or Tony Blair playing with the auction innards. Let’s make the following modifications:

In auction.rb add:

def activated?
    self.active?
end

And then change the previous spec to:

specify "should be able to deactivate auction" do
    @auction.deactivate!
    @auction.activated?.should_equal false
 end

The specifications should now still pass.

Step ten - Let’s now interact with another model, in this case it’s going to be a user model as our auction belongsto one owner who created the auction. Things are going to start getting interesting. Add the following context and specification to your auctionspec.rb file:

context "An auction with an owner" do
  
  setup do
    @auction = Auction.new
    @auction.attributes = AuctionSpecHelper.valid_auction_attributes
    @owner = mock('user')
    @owner.stub!(:new_record?).and_return(false)
    #make sure you put in the Activerecord type checking hack in 
    #the spec_helper.rb file or this bit will go BOOM BOOM
    @auction.owner = @owner
  end
  
  specify "should have an associated owner" do
    @auction.owner.should_not_equal nil
  end
  
end

OK, so this isn’t a very helpful test/spec in itself. However did you notice that we didn’t drop to the command line, run the RSpec model generator, create a User model migration, decide what fields to put into our migration, run the migration and then add various methods to our User class that our Auction model might need to interact with. Instead we can concentrate purely on our Auction model’s business logic and it’s interfaces/interactions with other models. You’ll find that by using stubs and mocks it allows you to experiment a little more easily without detracting from your train of thought.

I think here’s a good place to stop for now. Next time in part two I’ll introduce you lovely people to more advanced examples of mocking, but first some sleep. Good night chaps (and chapettes).

Tagged with “Mocks”:http://technorati.com/tag/mocks, “Ruby”:http://technorati.com/tag/ruby, “Mocking”:http://technorati.com/tag/mocking, “RSpec”:http://technorati.com/tag/rspec

Tagged with “Eye of sauron”:http://technorati.com/tag/eye%20of%20sauron

Unfortunately I only had the camera on my Sony mobile phone to talk pictures with, which is why the quality of the photos isn’t terribly sharp, but all the same they’re up on our Flickerstream to be seen.

A big thank you to Ian Forrester, all the Barcamp organisors and the sponsors for making this possible!

Tagged with “Barcamplondon2”:http://technorati.com/tag/barcamplondon2

Now I’m a happy Textmate user and have been using it for quite a while now, but I have to admit there are various things that have become annoying such as slow search, no inbuilt refactoring support, intellisense and just “understanding” my code within the context of the project. Now I don’t need these things but it would sure make working with large Ruby on Rails projects a heck of a lot easier.

If you want to check out some of the smarter intellisense features for Ruby that are to be found in Netbeans then you’re going to currently have to do a little bit of work. There’s currently a wiki page up detailing everything you need to do in order to build the latest Ruby modules for the IDE. A link to precompiled binaries that can be added to an existing install of Netbeans can also be found on the page. A word of caution though that it’s all very work in progress and bad things may happen so I wouldn’t go dumping your copy of Textmate just yet.

Tagged with “Netbeans”:http://technorati.com/tag/netbeans, “Ruby editing”:http://technorati.com/tag/ruby%20editing

Talks of note for me were Chad Fowlers talk on Quick and clean Rails, Tom Locke's Hobo demonstration and Ben Griffith's talk on testing.

Of particular note was Chad Fowler describing how Ruby is designed/best used by smart programmers who know what they’re doing. He continued to explain that if put into the wrong hands Ruby gives a bad programmer enough rope to hang himself with. And this is something I have to totally agree with, as over the past two and a half years I’ve either audited or rewritten some very crazily written Ruby on Rails projects. These were coded by either people trying to write large commercial applications after reading the Agile Web development with Rails book, or written by people who didn’t have a good grounding in Ruby and/or web application architecture. There’s nothing wrong with this if it’s a simple site with a small number of users, but not when it’s a high usage site that people expect to be available around the clock.

After everyones talks were over Skills Matter were nice enough to take everyone down to Fluid for some drinks, with the first one hundred being on the house. Overall a very nice end to an interesting day.

But seriously, the reason that Agile Evolved joined forces with New Bamboo was to combine all of our talents to form the UK’s premier Ruby on Rails consultancy. With us all coming together you’ll have:

• Max and Damien’s impeccable experience with working with cutting edge start ups
• My experience of applying web 2.0 technologies/principles in the enterprise
• Dr Fishers knowledge of AI, search algorithms et al
• And finally Marek’s deep understanding of all things Ruby

The first thing we did when the merger occurred was to make a cup of Early Grey. The second thing we did was to finally kit the office out with some furniture. Previously I’d gone for the minimalist approach of a few desks and some Aerons, yep I never said I was the artistic one in the company. Pictures of our new refurbished office can be found in our “Flickr account”:http://www.flickr.com/photos/newbamboolondon.

If you’re ever around the King’s Cross area feel more than free to pop in for a chat. However don’t expect to escape without a game of Wii tennis!

Tagged with “New Bamboo”:http://technorati.com/tag/new%20bamboo, “Agile Evolved”:http://technorati.com/tag/Agile%20Evolved, “Company Merger”:http://technorati.com/tag/company%20merger