Luckily, this helpful flow-chart helps you decide what to do next.

One of the major trends in software development at that time (at least for desktop applications) was to offer options – make it configurable and you can please everyone. And you can then sell “consultancy days” as it takes two weeks post-installation before the thing is usable, as you tick all the right boxes and choose all the right options. It also means you can say “hmm, well, there are eight different ways to do that, each with different pros and cons” (as opposed to saying “we can do exactly what you want” or “no, sorry, we don’t do that”).

So this was a big complicated software project; large enough that no one person knew it all the way through, with a variety of different architectures in-built (COM objects is how you do this stuff; oh no, now it’s COM+, now it’s ADO-disconnected and so on).

But we were only a small company. At its largest there were probably ten people working on the product at any one time. However, if you read the company communications you could never tell – it was all the bland, marketing-droid, use three-hundred-words-when-one-would-do style. Our email signatures had the sixteen paragraph disclaimer – “this is only intended for the intended recipient and if you are not its intended then wipe your memory immediately and burn your computer to prevent accidental ingestion”. This worked so well, some of our customers were amazed when they found out how many people we actually had – they thought we were a multinational with hundreds of developers across thousands of countries.

Over the years, we would consistently come across a single competitor – let’s call them Crump Builder. We poached our sales director from them, so we had a pretty good idea of what they were capable of and how they worked. We were small fry, they had funding and a team several times bigger than ours. They had also been going longer than us, so they had an even bigger set of preferences, options and modules to choose from. Most of the time, the sales process would come down to a choice between Roloduck and Crump Builder, and in the vast majority of cases, the prospect would choose Crump Builder.

At the time, I thought this was because we were behind them. They had the more polished, more finished product that did more and went further. But, since the demise of Roloduck (the company went bust recently) I’ve had to revise that view. You see, one of Roloduck’s old customers has chosen Crump Builder as the replacement for their now defunct Roloduck system. And she said that Crump Builder simply does not do a lot of the stuff that they need, whereas Roloduck does.

This news amazed me. Why did we consistently lose out to Crump Builder in those countless sales pitches?

Having thought about this, the only reason I can come up with is that we were not playing to our strengths. We were chasing their tails all the time. “We know they do X and we don’t yet; but we will soon”. “Yes, their version is pretty slick and ours is a bit clunky”. We were trying to present ourselves as their equals, equivalent in size and structure, in fact bigger than them in size and structure. But, because we were smaller, we were being found out. Not once, in our competitive analysis did we say “they are strong there, but we are strong here, so we should push this instead”. We should have pitched it as “we are the small guys – meaning you will get personalised service”, “our bit isn’t finished yet, so we can tailor it exactly to how you need it”

This is exactly what we have planned for our new venture. There are only two of us, so you will have the focussed attention of at least 50% of the company. We can’t afford lawyers or committees, so all the personality won’t be mangled out of our communications (bad jokes and all). And as we’re only small we can’t afford any waste, in either resources or time. So we will do what you need in the quickest, most efficient way possible. It will not work for everyone, but it will work fantastically for some. Those are our strengths and we will play to them.

Gemcutter’s the new kid on the block – it works as a set of plugins to the gem command that mean you can simply go `gem push my.gem` and it becomes available to the world. Nice as this is, gemcutter’s still pretty young and is currently hosted off one guy’s S3 account.

So we went back to good old-fashioned Rubyforge. Except, believe it or not, I’d never published to Rubyforge before. And it wasn’t as straightforward as it could have been.

I was using the echoe library to build my gem – which gave me a simple `rake manifest`, `rake release` workflow. All good. When you `rake release` it asks if you want to publish to rubyforge; you say “yes” and go to bed happy.

Or not in my case.

Firstly, you need to actually configure rubyforge. Install the rubyforge gem, then use `rubyforge setup` to put in your username and password. Then use `rubyforge config` to make them stick.

That done, I `rake release`d. And got a “no group_id configured for mygem” error. Eh? Googling seemed no help whatsoever – it was mentioned in a few places but people seemed to have resolved it without actually posting their solutions (or I was being blind).

What you need to do is tell Rubyforge where to put your gem. In our case we already had a project on Rubyforge that the gem could be attached to – so `rubyforge create_package project_name gem_name` did the trick. Otherwise you need to get a project set up to house your gem.

But it still didn’t work. This time complaining about a processor id. I had no idea what I was looking for this time, until I discovered this pastie by Dr Nic. The autogenerated config file (normally in ~/.rubyforge/auto-config.yml) is missing some vital information – namely codes for different processor architectures. I pasted the list into your config file and then tried the `rake release` again. And this time it worked!

processor_ids:
  IA64: 6000
  AMD-64: 1500
  Any: 8000
  Sparc: 4000
  PPC: 2000
  Other: 9999
  Alpha: 7000
  i386: 1000
  UltraSparc: 5000
  MIPS: 3000

So for the sake of posterity – and anyone else who gets “no group_id configured for mygem” or “no processor_id configured for mygem” messages, I thought I should let you know how I got there.

But the weirdest thing was we couldn’t figure out the cause. Well I could see why the code was failing (after adding some extra log messages). But ‘git blame’ said those lines of code were unchanged in twelve months. Why hadn’t people complained before? Why hadn’t we noticed it?

After much hunting through log files we found the point when the feature last worked. It coincided with a deployment. That deployment was our Rails 2.3.4 forms vulnerability fix. And the bug was in a form – a missing form parameter that earlier versions of Rails ignored, the newer Rails was choking on.

But why didn’t the tests catch it?

After more hunting I saw that the Cucumber test that exercised the form didn’t have a “When I press the Update button” step. And the subsequent tests were passing, even though the update button hadn’t been pressed.

So I added the step in and made the feature pass. Then deployed it as an emergency fix.

However, what are the lessons to learn here (as there are always some)?

• Firstly, testing cannot catch everything.
• Secondly, the cracks in your tests are where the bugs are.
• Thirdly, we probably need some sort of peer review for tests. I feel that this is more important than for code, because once the tests are right you can refactor the code without worry.
• Fourthly, you really need to log everything. Absolutely everything. Don’t worry about your huge log files – that’s what `logrotate` is for. Get it written down so that one time when you have an obscure bug, you’ll be able to find it easily.

But it wasn’t shit. It wasn’t good. It was absolutely great.

Things started on the Friday with a free seminar by Tim Nash – all about user profiling. I missed the first half hour (as my bus had to stop and give another bus a jump start – it’s true, honest), so was a bit befuddled as we had to decide on keywords for the other attendees (based upon the introductions they had given whilst I was on the other side of town). But I came away from the session with a list of improvements to make to one of my blogs so it wasn’t a wasted trip.

This was swiftly followed by several pints of Leeds Brewery Pale Ale on an empty stomach, resulting in me failing to buy Al Carlton a drink, but being bought one by Chris Garrett (thank you). I sloped off home before I could get caught up in one of Al’s legendary nights out.

On to Saturday and the conference proper. Despite living in Leeds for years, I’d never really been to Clarence Dock, home to the Alea Casino. I was actually quite impressed – in a “let’s try and look like London” kind of way. Slide into the venue just before ten, pick up my lanyard and creep into the introductory session.

Joost de Valk told us all about optimising Wordpress. Speed is key – cache (using Memcached if possible) and distribute your assets using a Content Delivery Network to reduce those page load times.

Next, Julian Sambles told us about how the Telegraph changed its entire culture to adapt to the digital world. It’s certainly a success story (look at the number of times Gruber links to Telegraph stories; a couple of years ago if he wanted a UK angle it would have been the Guardian) – and Julian emphasised that it was mainly a cultural change, not a technical one. Another interesting fact – their digital division not only generates revenue but actually makes a profit (unlike, if you believe the rumours, the print division).

And on to my highlight. I had the pleasure of meeting Judith Lewis in the pub the night before and she was absolutely charming. But her talk was even better. She told us about how Google’s embedding search results for other media types within the standard search results page has changed the game (images, videos and news results pushing other, textual results, further down the list and often off the first page completely). Meaning you should work at getting into the video results, the image results and the news results. But the best bits were when she stepped away from the mic and told us about the “dark arts” of search. Which, of course, I could never repeat here. She then ended her talk with a lolcat.

Leading us to the lunchtime panel. There were a few complaints about the food. Personally I wasn’t bothered. I was very hungry, grabbed a few ropey sandwiches and had a couple of lovely cookies. No complaints from me; especially for less than £100. But, by the sounds of it, some people missed out because of the queues, which would be annoying. While we were eating, there was the lunchtime panel – Al Carlton, Chris Garrett, Kieron Donoghue and Patrick Altoft Dominic Hodgson taking questions via Twitter. There was also a side-competition – how many times will Kieron mention Share My Playlists? Of course, yours truly got a big laugh and round of applause for my question.

On to the afternoon sessions – where the venue was split into three. First up was Paul Robinson, who I vaguely knew from Geekups and my day job. He was explaining how the likes of Amazon and Last.fm’s recommendation engines work – and how by matching similarities between different user data and behaviour, you can add intelligence to your site. Interesting stuff.

Next up was Chris Clarkson telling us the story behind Holiday Watchdog – an affiliate site he and a friend founded in 2003 and sold in 2007. The main thing to take away from his talk was that being a limited company may protect you from company liabilities – but it doesn’t save you from being sued for defamation.

Then was Zoe Piper talking about Google Adwords. There was a lot of interesting stuff here, but unfortunately, I didn’t hear parts of it. Because the main room had been split into three, the speakers could not use the microphones – and Zoe had a quiet voice, meaning I missed big chunks of her talk. This is my only complaint of the day – it’s not Zoe’s fault – just an artefact of the room arrangement.

And finally Artur Ortega gave a talk on how disabilities have fueled innovation in the technology industry. Basically every piece of technology, that we take for granted, in our offices has a history of helping overcome disabilities; from keyboards to scanners (and mostly designed by Bell Labs, as Alexander Graham Bell’s wife was deaf and so he was motivated to solve the problems that triggered).

As I said up-front, this was a great day and a half. I learnt a lot, got to meet some great people and am once again reminded how much there is in the “web industry” that I know nothing about. It has certainly changed the way I approach my next venture.

The next Think Visibility will be in March. You better make sure that you attend.

I have no idea who he is or why he’s gone. But _why meant a lot to Ruby developers, myself included. He represented style and grace and art and weirdness – that Ruby was about more than just getting the job done in the most efficient way possible.

He had a class, descending from ActiveRecord::Base, with a custom constructor (initialize method). To debug it, he had the constructor raise an exception. In the console, Thingy.new(params) raised the exception as expected. But wotsit.thingies.find_by_field(value) did not. Even though it was instantiating an instance of Thingy and returning it.

“It must not be calling the constructor” he said.
“Rubbish” said I, “it’s a constructor. Constructors are always called. That’s the point of them”.

But as he dug deeper it certainly looked like the constructor wasn’t being called.

And then he found an article explaining that you should never rely on things being set up on in an Active Record constructor. Mainly because Active Record uses allocate to instantiate associated objects. And what is this mysterious allocate? Why explains it all.

To be honest, I’ve got mixed feelings about this. I can see the use of “allocate” – why’s example of marshalling an object makes sense (I’m slightly less sure about the way that Active Record uses it to load associations). But, to my mind, the definition of a constructor is “the code that is always called when an object is created”. So maybe I should just stop thinking of initialize as a constructor and more as an initialiser.

Suppose you’ve been working in branch X and you’re about to merge those changes into branch Y …

• work in branch X and make your commits as needed
• switch to branch Y git checkout Y
• compare the differences between X and Y git log Y..X – the order here is important; it is merge-target..merge-source
• merge the changes from X (merge-source) into Y (merge-target) git merge X

Even better, before doing the merge, while you are still in branch X, you can squash multiple commits into a single one. This way, branch Y, when you examine the log, has a single entry “implemented feature X”, instead of thirty-five entries all related to feature X in some way.

• examine the log git log or even better git log --pretty=oneline
• choose the commit that immediately precedes the one that starts your piece of work (probably the last one before you branched to start work on feature X)
• start an interactive rebase session – this is git-fancy-talk for picking some commits and typing a message – git rebase -i THE-COMMIT-ID-OF-THE-COMMIT-YOU-SELECTED-ABOVE
• Your favourite editor (or nano) will open listing the commits for feature X – edit the word “pick” to “squash” for all except the top entry, save and exit
• Your favourite editor will open again – again showing your commit messages – add a line at the top stating something like “implementation of feature X” and leave the list of individual commits on separate lines below it (adding in a Fixes #123 or whatever your issue tracker demands as the last line), save and exit
• Now if you do a git log you will see a single commit with a nice “implementation of feature X” log message
• And you’re all set to merge to your master branch without cluttering its history with a long string of commit messages

All the tests passed. Poking it on the staging server worked well. On to production and all was good.

Time passed.

Then, the app needed to be moved to a new server. Actually to a cluster. Slightly more nerve-wracking, but some playing around with the ghost gem and DNS settings and the deployment completed.

More time passed.

And then disaster struck. Well, not disaster, but it wasn’t good. Not at all.

The app had a rake task that was cronned to run every night. And it failed silently. We found out the day after and I scanned the production log file looking for evidence of what went wrong. Nowt. It didn’t even look like the rake task had run – but it had, as I had the data in front of me.

And the day after, it failed again. Another scan of the production log revealed nothing. Things were getting serious.

I suspected the cluster deployment. Maybe cron didn’t have permission to write to the log file, even though Passenger did. But everything looked good on that front. Maybe there was something else weird about the cluster setup – all the cron jobs were set to run on one of the app-servers; maybe moving it would make a difference?

To test this, I opened the Rails console and started manually recreating what the rake task was doing, on a different app-server. Which is when I noticed that script/console wasn’t writing to the production log either. What?

After a couple of hours of head scratching, I finally found out that the cluster deployment was not at fault. In fact, it seems to be the move to Rails 2.2 that had happened many weeks before – in particular it appears that the logger object no longer auto-flushes itself after writing.

We added a quick Rails.logger.auto_flushing = 1 to an initialiser and both script/console and the rake tasks wrote to the log as expected. Phew!

I seem to have nailed it down to two problems.

Firstly, monit only seems to like monitoring ferret server under certain circumstances.

Secondly, the ferret indexes don’t seem to like being rebuilt.

The first one is weird … my monit config looks like this:

check process ferret with pidfile /home/rails/myapp/current/log/ferret.pid
  start program = "/home/rails/myapp/current/script/ferret_server -e production start"
  as uid rails and gid rails
  stop program  = "/home/rails/myapp/current/script/ferret_server -e production stop"
  as uid rails and gid rails
  group ferret
  if failed port 9010 type TCP then restart
  if 5 restarts within 5 cycles then timeout

But monit often says “not monitored”. And completely fails to restart the server if it does go down.

No solution to that as yet.

The second problem is even weirder.

If I stop the server, then rebuild the indexes (I have a custom rake task that rebuilds all the indexes for my app) and restart, every search crashes ferret.

If I rebuild the indexes while the server is running, it appears to run fine. Why? I have absolutely no idea.