My hovercraft is full of eels

Problem Solving

2009-06-25T16:03:00.008+10:00

It occurred to me recently that I have this notion of programming as a process that involves breaking a problem down into a sets of smaller and smaller problems until I have something I know how to solve. (I mentioned this to Steve yesterday which reminded him of a joke about an engineer and a mathematician.)

I have previously just assumed that I therefore follow this process when I'm actually problem solving however, on reflection, I'n not so sure. More specifically, I'm either not doing it at all or, at the very least, I'm doing it intuitively.

I wonder how many people do (or have done) this as an explicit part of their own problem solving and if so, what effects they've noticed as a consequence.

Collaboration

2009-06-25T12:00:00.013+10:00

I've heard the phrase "Excuse my poor code" (or words to that effect) a number of times recently. I've said it to myself about my own code (if not to others), I've heard two of my work colleagues say it, and I've had a customer's sole developer apologise ad nauseum about his code.

I'm not here to make any determination as to the veracity of these claims but the thing that I find interesting is that in all cases, the majority of the code in question was written by a single person.

Now this may be difficult for some to believe but I inherently don't trust my own opinion. I do tend to confidently put ideas forward as bold assertions to be shot down when I'm wrong. The confidence comes from knowing I'll defend my ideas to the hilt and not take it personally when my argument is proven flawed.

I hate working alone because I don't trust decisions that weren't arrived at through furious debate. I don't like developers working alone no matter how good they are (or think they are) because I don't trust they can be objective enough by themselves. In fact I don't care if they're working in pairs of developers or not, I just want their decisions to be scrutinised by smart people as early as possible.

The idea that the majority of a developer's work involves going off to think on their own is totally nonsensical to me. It feels like a lot of hocus pocus going on behind closed doors and then *poof* a few days later something is divined. Not only is the process opaque, but the possibility for smart people to scrutinise is left as late as possible. Sure, I like to sit and ponder without the hullabaloo of every man and his dog trying to give me their 2c worth as much as the next but the lack of transparency and scrutiny in this as a process is something I find very difficult to accept.

Smart people working alone is even worse. They're often implicitly promoted to the position of grand wizard, the seer and knower of all things. They go away, think about a problem and come back with much fanfare (trumpets playing, drums beating) to bestow upon the people their creation who will wonder in amazement at the design, so simple and yet, so complex, that only they can truly grasp the significance of what they have achieved.

When I write code on my own, I feel personally, individually, responsible. When I write code on my own, the pressures I feel are, mostly, self imposed. They make me lie awake at night. They make me code into the wee hours of the morning, on the train home. I worry that I've missed something. I start to believe my own hype. That I'm good enough to do this on my own. That because I did most of it on my own, I should fix it on my own lest someone else realise how crap my code is. In a vacuum my ideas have no predators. As importantly (if not more so), my priorities have no predators. I race towards a goal without stopping to re-think that goal because I'm stuck in a vicious cycle of self doubt and self confidence. Moreover, when I work alone, the production of code becomes the focus and not, as I believe it should be, thinking!

When we write code together we have shared code ownership. I feel like someone has my back. That even if we got the decision "wrong", we decided together and if two smart people can't get it right then maybe it's good enough, maybe there are diminishing returns for adding more people to solving the problem. When I go home at night, sure I might tinker but I know there's not much point to spending a lot of effort because I'll just be re-doing it tomorrow anyway. And if there is something we missed, someone has my back if need be.

I speak as one who has been, done and experienced others doing all of the above and decided once and for all that it's just not worth working alone. Having more than one person working on something WILL cost more but I assert that the result will be better and will be achieved more quickly and with less churn. Anyone who thinks they're smart enough to do otherwise needs to learn to keep it in their pants.

I have this mental model of software development that is akin to the way I've heard Abraham Lincoln's cabinet described. Apparently he got a whole bunch of really smart people together, even those from opposite sides of politics, and let them argue it out. His (Lincoln's) role was to decide when to stop the debate and which idea(s) to act on.

I think that's what collaborative development should be.

YMMV :)

No, Sleep, Till Bedtime

2009-06-14T09:10:00.009+10:00

Or at least until all those Twitter client developers have fixed their Twitpocalypse bugs. In case you didn't know, a few days ago the ID range used for Twitter's messages exceeded 2^31 (approximately 2 billion) causing any apps that stored them as 32-bit integers to think they were really small negative numbers.

It's usual--and I say usual because I don't always adhere to it--policy for storing external identifiers is to treat them as text, even when I know they are numbers. Why? Essentially because I consider it a coincidence that they're numbers. That identifier, number though it may be, has no special significance to me over and above being an opaque handle to some entity in another system. As such, I like to treat them as text.

Discussing this with a good friend and colleague of mine, the question of column width came up. Ie. if you're going to make it text, how long should the column be? If you're lucky enough to be using a database such as PostgreSQL, then the answer is: it doesn't matter--there's no performance benefit to artificially limiting the size of the column. For other databases, the common practice is to use something like VARCHAR(255). Think about it, even if it is a number that's 10^255!

Twitter claims that its API is RESTful. And if to you, REST means nice, predictable URLs with some semantic path possibly followed by a numeric id and returning numeric ids in search results, then yes, it's RESTful. Want to see the most recent messages for a user? There's a simple HTTP request you can make to a nice, semantic (if you speak English) URL that returns a list of them and their identifiers. And, as expected, our Twitter clients have been dutifully squirrelling these ids away in integer fields (probably because that's the default) and all has well until 2 days ago.

Now, without going into too much of an ideological rant, I happen subscribe to the principle that RESTful URLs should be opaque. That is, a URL is a URL is a URL. No slicing, no dicing, no assembling, no joining. If I have a URL to a resource then that's what I use. Period. End of story. (You can find plenty of discussion on this by Roy Fielding using Google.)

So, back to our column widths. Assuming we have a text field in our database large enough to accommodate a URL, we could go one step further. Rather than treat the identifier as text and storing that, why not go the whole hog and store the URL instead?

As far as I can tell, the only reason is that Twitter's API, RESTful though they may claim, sends back numeric identifiers rather than URLs which in turn leads developers to incorrectly assume that they should be storing them as numbers.

On their own, identifiers are meaningless and in fact, useless. To utilise an identifier requires us to know the system in which it is stored and the collection in which it belongs. If instead each piece of information was identified by a URL we get all that context for free and the power to share information grows phenomenally.

To me, the beauty and power of the internet is the ability to link together disparate systems in ways no one had previously imagined. More specifically, in ways the publishers of the information never considered.

Opaque URLs combined with idiomatic use of HTTP verbs can help reduce the coupling between producers and consumers by giving back control to producers in how and where they store information and at the same time increasing the freedom for others to share and use that information.

(That last paragraph reads like an Amnesty International commercial!)

Shameless Self Promotion

2009-05-15T19:26:00.007+10:00

So the past couple of months, I've finally had the luxury of starting to realise my (and Cogent's) dream of doing product development.

We just recently launched what we hope is a very simple, easy to use and somewhat opinionated web application for Getting Things Done™ (GTD). It's a crowded market to be sure but we really believe we understand GTD well enough to deliver a system that is more than just a to-do list with GTD inspired keywords.

Runway is still in the early stages of feature development. For those that know anything about GTD, you'll be happy to hear that we're working on delivering Projects, Artifacts, Agendas and of course an Inbox, to name but a few, in the very near future.

What you see now is, and will always be, free. at some point we'll be adding pay-for features but we'll also be doing the right thing by all our early adopters. So, if you have 5 or so minutes, we'd love for you to sign up, have a play, and of course, tell us what you think.

Web standards and all I got was this lousy website

2009-04-17T11:43:00.008+10:00

Over the Easter long weekend, I had a great break from work and a great opportunity to think about and reflect on my career, my job, and my profession as a whole. It's safe to say I become a bit disheartened and disillusioned. The one striking conclusion I kept arriving at is that we are so technology focused that we spend too much time, money and effort building things that the customer is "happy" with but not blown away by. That we artificially constrain the end user experience based on our notions of "correctness". In particular, web application development is largely a bunch of dick-pulling technical masturbation, forever re-inventing the wheel at a ridiculously low level of
abstraction shoving our technological solutions down user's throats in the name of "software engineering". What's worse is that I've been complicit. Not only by buying the hype but often by trying to do "the right thing" even when I felt as though I was bashing my head against a brick wall.

Rewind the clock to somewhere between 1996 and 1999. During those years I, along with a good friend and colleague built a desktop application that was delivered to thousands of users across Australia using nothing more than good old-fashioned client-server SQL written in, of all things, PowerBuilder--kinda like VisualBasic. More than 10 years ago, the user experience was compelling and sophisticated, it performed exceptionally well over 2400 baud dialup modems, and we built the initial release with only 2 people over 3 months from scratch. As shameful as it is, especially coming from one so vocal about automated testing as I, we had nothing but manual testing but we also had few bugs and when users did find a problem, we fixed and redeployed within 24 hours--mostly because we didn't want to interrupt users as they worked and so waited until after hours. Over the next 12 months, we were able to adapt to the user's needs immediately. Rarely did we add the features as request but we always managed to produce a solution they actually needed. Fast forward a decade and I feel like I suck because I honestly don't believe I could do the same thing again today. In fact, I challenge any of us to build the same user experience with our existing technology stack.

To those that know me well, I will no doubt sound like a broken record but I can't help feel we've been trying to coerce HTML & CSS into something they just aren't and doing so for a decade now.

Think about it, HTML: HyperText Markup Language. Does that sound like it has anything to do with layout and design? In fact do you know any designers, even those that call themselves web designers, that do any of their design work in HTML/CSS? No--well none that I've ever heard of. The closest I can think of is a colleague who does his wireframes in OmniGraffle and then generates HTML/CSS. Why? I put it to you it's because we don't think in HTML/CSS. You CAN'T effectively think in HTML/CSS and if a guy who's expertise lies in designing user interfaces can't think in terms of HTML/CSS why the hell do we think we should?

HTML was designed for linking documents with a modicum of layout and has served that purpose admirably. As a result, the web browser largely won the battle for desktop supremacy and almost everyone has a web browser and regularly uses a number of web sites. Similarly, pretty much everyone has a computer running an 80x86 based CPU and run dozens of applications built specifically for it. HTML/CSS are the machine language of the web.

For those of us lucky enough to have done any assembler programming, we've also been lucky enough not to have had to do any for a very long time. Instead, we chose to move away from assembler to other languages. C, C++, Java, Smalltalk, Python, Perl, Ruby, literally dozens of other programming languages that have systematically improved the level of abstraction. Many of these languages now run on top of the JVM, LLVM, CLR, etc. themselves abstractions on top of the underlying CPU.

Did we move because the runtime was faster? Hardly. In fact in almost all cases outrageous claims were made early on that poor performance would be the undoing of these languages and in almost all cases these claims ultimately proved unfounded. No, we moved to these languages because we hoped they would give us a better level of abstraction. That we could code more closely to the way we think. That we would one day realise the dream of literally thinking in code.

Even within languages we constantly strive to improve the level of abstraction. In many cases we've created Domain-Specific-Languages in order that we are better able to think IN the language most appropriate to the task at hand rather than needing to perform some contorted mapping process. This is the reason the Ruby community has slowly moved from Test::Unit to RSpec/Shoulda: Test::Unit does the job just fine but it's verbose and "too close to the metal". Just like assembler. When I'm the most productive I'm literally thinking in code.

We've largely sorted the back-end problems: Database access layers, routing, data format conversion, validation, you name it it's all been largely worked out in whatever framework and language combination you can imagine. The same cannot be said of the front-end WHERE IT ACTUALLY MATTERS.

Granted, HTML/CSS has undergone change but to what extent and to what end? We have JSP, ASP, ERB, HAML, SASS, Liquid, blueprint, jQuery, Prototype, MooTools, Dojo, YUI, etc. but none of them appreciably raises the level of abstraction. Most advances in the world of HTML/CSS are lipstick. They're all constrained by the fallacy that HTML/CSS is the holy grail of web design. No, the whole problem with web development is that we haven't abstracted away the underlying technology, instead we've been conned by a bunch of HTML/CSS gurus and boffins who think that designing the perfect machine code is all the world needs. There is nothing more primitive than HTML+CSS when it comes to the web.

HTML & CSS try to be all things to all people and by doing so, much like J2EE, we ended up with a set of primitive tools that are repetitive, verbose, hard to test, maintain and refactor and ultimately provide a user experience that can best be described as a tarted up, 24-bit 3270 terminal. Don't believe me? Point me at a website where the user experience feels liquid and natural. Where it literally gets out of your way so that you never even realise you're using it? For the most part you can't. The poster children of the Rails world provide at best a rudimentary user experience. I suspect people use them because there is no alternative, not because it's actually a great UX. Why? IMHO because the technology choices are just plain awful. If you can find a website with a rich user experience that just melts away, you'll likely find a bunch of developers who either had nervous breakdowns or spent many years building some superduper framework, or both!

To be fair I'm no doubt coming across as though HTML/CSS is to blame for all the world's problems. Not at all. We suffer from similar problems across the board in software development. It just so happens that I've been in the world of web development for a long time now and feeling the effects.

I'm not advocating the use of any particular technology--that would kinda defeat the purpose of my argument. What I am saying is that I believe we're stuck in a mindset that only allows us to think inside the incredibly narrow bounds of something we're used to, IMHO, only because it's all we're used to.

Rather than embracing the "web paradigm" how about we embrace the user and their experience and decide what technology would best enable us to deliver that.

A Title Case Gem for Ruby

2009-02-02T09:39:00.005+11:00

A project I'm working on called for some "smart" capitalisation of page titles. Essentially I wanted to take a URL slug and generate a page title.

Rails comes with a built-in String#titleize method that capitalises every word but that looked a little odd when the title was something like: "My Hovercraft Is Full Of Eels". So I went on a hunt for something "smarter".

After a little search I stumbled upon Marshall Elfstrand's JavaScript, Ruby, and Objective-C ports of John Gruber's "Title Case" algorithm and decided to turn it into a Gem that adds String#titleize and String#titleize! (aliased as #titlecase, and #titlecase! respectively). When used in a Rails environment, this effectively replaces the Rails versions.

Now my page titles look a little more human-like: "My Hovercraft is Full of Eels".

Plugins move

2009-01-18T08:46:00.005+11:00

Following hot on the heels of my blog move, I've finally moved all my rails plugins off the venerable RubyForge and onto GitHub.

Since I started working at CogentConsulting--no we're not "The Company of ex-ThoughtWorkers" unless you count all 3 of us as somehow statistically significant--I've had less and less time and less and less inclination to spend any appreciable effort on RedHill related stuff to the point where the company really exists just to support and market Simian.

As a consequence, I've also dropped the RedhillOnRails moniker in favour of publishing the plugins under my personal account.

Blog move

2009-01-17T22:43:00.002+11:00

If you're reading this then the move of my blog was successful and thank-you for putting up with a screwy RSS feed during the transition. No doubt you received double or possibly even triple posts.

Why the move? Well, even though GeekISP have been a fantastic hosting provider over the years and MovableType has been pretty reliable as a blogging platform, in my never ending quest to Do Less Stuff, I figured it was time to move the pain somewhere else.

From a technical perspective, the move was fairly easy though not without some pain. There is no direct way to import from MT to Blogger however I did find a tool that helped convert the MT export file into something Blogger could import.

I also wrote a quick script to replace all internal references with new links as well as generating a new .htaccess file for any links from the outside world. This step was pretty easy although it took some trial and error to work out what how Blogger converts titles into URLs--as near as I can tell it truncates to a maximum of 40 characters with a bias towards word boundaries. The duplicate posts appearing in the RSS feed were as a direct result of me re-creating the entire blog several times fixing little things here and there.

And so it is that my blog comes to be here on Blogger. The next step is to move all my domain hosting to Google Sites but that's for another day. Hopefully this will be the last move for some time and, with someone else maintaining my blogging software, hopefully less stuffing around on my part.

Rails, meet Drupal.

2009-01-12T20:26:00.001+11:00

If you've been considering integrating (or replacing) your Drupal application with a Rails application, then Drupal Fu may come in handy.

It's pretty rough-and-ready--I essentially just ripped the code out of an existing application and cobbled it together--with, as yet, no plugin infrastructure, Rakefile, or anything else that might give you a degree of confidence in the quality of the code :)

That said, the code has been working in a production application for a while and we figured it might help out some others going through the same pain.

Acts As Teapot

2009-01-12T07:32:00.001+11:00

No, it's not April Fools yet but I thought I'd get in early this year. Acts As Teapot is a Ruby on Rails plugin that ensures your Ruby on Rails applications conform to RFC2324. My assumption here is that your application is not a coffee pot and therefore does not understand the Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0). Thus, if ever a BREW request or any other request with the Content-Type set to "application/coffee-pot-command" is received, the server will respond with 418 I’m a teapot.

TimeMachine FTW!

2008-07-25T22:41:00.001+10:00

Not withstanding the fact that I needed to restore my operating system in the first place--due to an inexplicable and catastrophic failure of the Java installation resulting in segfaults--I was able to restore my entire 100GB system in around 4 hours. For posterity:

Boot off the OS X System Install DVD--hold down option while the system starts

Connect the external drive with the TimeMachine backup--in my case a TimeCapsule attached via ethernet

Select "Restore from TimeMachine backup" in the Utilities menu

Select the specific backup (by timestamp) from which to restore

And away you go!

The disk is then automatically erased and a fully bootable system is restored sans temp directories and cache files. It even managed to restore my PostgreSQL databases that were running at the time--which probably says more about PostgreSQL than anything.

The one grumble I do have is that the timestamps in the name of the backups were some non-obvious period relative to the actual date the backup was made. The difference wouldn't have been much of an issue had I simply needed to restore the most recent backup but as it turned out I needed to go back a couple of days in order to get a clean system. Thankfully I got lucky on the second attempt :)

Once I had restored the system I took a look at the backup folders and sure enough there are two timestamps: the one in the folder name, and the created date. The created timestamp was spot on but the one in the folder name--the one presented to you when restoring--was whacky. I honestly didn't spend long enough to calculate if the difference was consistent.

What is really interesting is that I had SuperDuper! on my list of software to start using but it would appear there is little need--at least in my case.

Prevent accidental deployment with a prompt

2008-06-07T21:41:00.001+10:00

This morning I went to push out a new version of an application to our staging environment on an Engine Yard slice. I knew I had done exactly that last night so I navigated through my bash history and hit enter. Two minutes later the new version had been deployed and I was about to walk out the door to do some chores before coming back to start playing with the app. Thankfully, Steve came online and informed me that production was broken.

A quick look through my bash history and it seemed I'd used the deploy to production rather than deploy to staging but, being in a hurry, hadn't looked carefully enough. Of course some might argue that I should have looked more carefully. That I shouldn't have deployed before heading out. All valid points but I very rarely have full control over what's going on around me. So, while it's all very well and good to hope that I will be more careful next time, that's a bit like hoping global warming isn't a reality: we all hope it's not but maybe we should do something about it just in case?

And so I added the following at the start of the :production task in my deploy.rb file:

  unless Capistrano::CLI.ui.agree("Are you sure you want to deploy to production? (yes/no): ")
puts "Phew! That was a close call."
exit
end

For all other environments, the deployment goes through without question. Attempt to deploy to production however, and I'm now forced to be explicit about my intentions.

Generating lots of little test cases

2008-06-07T01:51:00.000+10:00

When writing code that is largely algorithmic, I find I end up writing specs that sit in a loop repeating the same operations over a set of data.

This works well enough but it has the downside that the tests abort as soon as a single failing case is detected which can lead to vicious cycles of fixing one case only to find you've broken another.

The solution is of course to break out each expectation--inputs and expected outputs--so they are all run and reported individually. Doing so by hand however, is tedious to say the least so why not generate them on the fly instead:

describe Fibonacci do
[[0, 0], [1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 8]].each do |input, output|
it "should generate #{output} for #{input}" do
Fibonacci.calculate(input).should == output
end
end
end

This is such a elegant solution I'm not sure why it only just occurred to me.

Quickly Migrate all database times to UTC

2008-06-03T04:22:00.000+10:00

If you're thinking about updating to Rails 2.1 to get the timezone support, you'll need to update all database records to UTC. Here's a quick migration script to do just that:

class ConvertTimestampsToUtc < ActiveRecord::Migration
# Assume all times were in UTC+10:00
OFFSET = "interval '10 hours'"

# Adjust any date/time column
COLUMN_TYPES = [:datetime, :timestamp]

def self.up
adjust("-")
end

def self.down
adjust("+")
end

private

def self.adjust(direction)
connection = ActiveRecord::Base.connection
connection.tables.each do |table|
columns = connection.columns(table).select { |column| COLUMN_TYPES.include?(column.type) }
updates = columns.map { |column| "#{column.name} = #{column.name} #{direction} #{OFFSET}"}.join(", ")
execute("UPDATE #{table} SET #{updates}") unless updates.blank?
end
end
end

As you can see, I've assumed that the dates were previously stored as AEST (UTC+10:00) so you'll likely need to adjustthat and I'm also assuming PostgreSQL for date manipulation though it should be pretty simple to convert to run under MySQL. It may even work asis.

Deploying branches with Capistrano

2008-05-30T19:07:00.000+10:00

This morning I had occasion to deploy a branch of a git repository to a staging server but hadn't the foggiest idea how. A quick search through the capistrano source code revealed that I could use set :branch "branch_name" in my deploy script. I tried it and it worked. I then figured I would need to make a similar change across all my branches. Of course, I'm a lazy sod and wondered if there wasn't a better way.

If you're not familiar with git, the output of the git branch command is a list of branches with an asterisk marking the one currently checked out on your local machine. For example:

$ git branch
* drupal_authentication
fragment_caching
master

So, I figured, what if I just parsed the output and searched for the branch marked as current:

set :branch, $1 if `git branch` =~ /\* (\S+)\s/m

Now I'm able to deploy whatever branch is current on my local machine from a single, shared, deploy script.

JavaScript Date Helpers

2008-05-13T22:27:00.000+10:00

It's all the rage these days to have timestamps displayed in words to indicate how long ago some event occurred. You know something like "less than a minute ago" or "about 2 months ago", etc. You'll see plenty of examples on news sites, blog entries, and bug tracking tickets to name but a few.

If you've ever had to build this kind of thing in Rails you'll be familiar with all the Date Helper methods that make the task pretty trivial. The problem is that the result is fixed to whatever the date was when the page was rendered and as a result these timestamps go stale very quickly.

Save refreshing the page every minute--or hour or whatever--just to update the times, I figured what was needed was a little but of client-side action. Rather than send the text in the HTML, I decided to instead send the raw timestamps and have the browser periodically generate the textual representation.

To this end, I blatantly copied two methods from the afore-mentioned Rails helper-- distance_of_time_in_words(from, to), and time_ago_in_words(from)--and, taking some liberties along the way, converted them to JavaScript:

  distance_in_milliseconds = to - this;
distance_in_minutes = Math.abs(distance_in_milliseconds / 60000).round();

if (distance_in_minutes == 0) {
words = "less than a minute";
} else if (distance_in_minutes == 1) {
words = "1 minute";
} else if (distance_in_minutes < 45) {
words = distance_in_minutes + " minutes";
} else if (distance_in_minutes < 90) {
words = "about 1 hour";
} else if (distance_in_minutes < 1440) {
words = "about " + (distance_in_minutes / 60).round() + " hours";
} else if (distance_in_minutes < 2160) {
words = "about 1 day";
} else if (distance_in_minutes < 43200) {
words = (distance_in_minutes / 1440).round() + " days";
} else if (distance_in_minutes < 86400) {
words = "about 1 month";
} else if (distance_in_minutes < 525600) {
words = (distance_in_minutes / 43200).round() + " months";
} else if (distance_in_minutes < 1051200) {
words = "about 1 year";
} else {
words = "over " + (distance_in_minutes / 525600).round() + " years";
}

return words;
};

Date.prototype.time_ago_in_words = function() {
return this.distance_of_time_in_words(new Date());
};

Now all I do is periodically invoke a function that calls one or other of these methods and updates the text of whatever display element is appropriate. Even better, because the raw timestamps have timezone information in them, the display doesn't suffer from, in my case here in Australia, always being 10 hours out because the server is sitting in the US with a US date/time.

Giving the Anchor tag some Ajax Lov'n

2008-05-13T21:58:00.000+10:00

It seems I'm forever needing to submit links using an XMLHttpRequest rather than the default full-page refresh. One approach commonly used in the Rails community is to render each link with the JavaScript already in place. My preferred approach however is to keep the HTML as free from JavaScript as possible and unobtrusively add behaviour using LowPro.

LowPro already comes with a built-in behaviour for links but sometimes I need something little more complex than simply submitting the request and so I usually end up doing the following:

anchor = ...;
new Ajax.Request(anchor.href, { method: "get", parameters: ... });

Granted that's not a lot of effort but it still felt as though I were repeating myself and that the overall intention of my code was largely obscured by the infrastructure. It then struck me that submitting a form using the Prototype JavaScript framework is almost trivial:

form = ...;
form.request({ parameters: ... });

So I cooked up a version for anchors as well:

Element.addMethods("A", {
request: function(anchor, options) {
new Ajax.Request(anchor.href, Object.extend({ method : "get" }, options || {}));
}
});

Now I can submit links in pretty much the same was as I do forms:

anchor = ...;
anchor.request({ parameters: ... });

I'm wondering what other possibilities might occur were I to add a serialize() method to extract the request parameters.

HTTP Ranges for Pagination?

2008-04-22T05:46:00.000+10:00

Would it be a gross perversion to use HTTP ranges for pagination?:

Client asks the server what range types it accepts for people:

HEAD /people HTTP/1.1

Server responds:

Status: 200
Accept-Ranges: pages; records

Client requests the first page of people:

GET /people HTTP/1.1
Range: pages=1-1

Server Responds:

Status: 206
Content-Range: pages 1-1/13

Finding the index of an item using a block

2008-04-18T01:01:00.000+10:00

Update: Ruby 1.8.7 also has this.

Ruby 1.9 has it but if you're not that bleeding edge, you can have it now:

class Array
def index_with_block(*args)
return index_without_block(*args) unless block_given?
each_with_index do |entry, index|
return index if yield(entry)
end
nil
end
alias_method :index_without_block, :index
alias_method :index, :index_with_block

def rindex_with_block(*args)
return rindex_without_block(*args) unless block_given?

index = size
reverse_each do |entry|
index -= 1
return index if yield(entry)
end
nil
end
alias_method :rindex_without_block, :rindex
alias_method :rindex, :rindex_with_block
end

If you're using Rails you can substitute the two calls each to alias_method with a single call to alias_method_chain.

It's OK for GET Requests to Update the Database

2008-04-16T04:59:00.000+10:00

We've all been indoctrinated into associating the HTTP request methods POST, GET, PUT and DELETE with the standard database (aka CRUD) operations Create (INSERT), Read (SELECT), Update and Delete respectively. For the most part the analogue holds. When we make a GET request, our intention is to read whatever the server hands back. When we POST some data, our intention is to update something.

The general view however, seems to be that the HTTP methods relate directly to database operations. In fact many developers seem to think that they are in fact one in the same thing: POST is for INSERTing data, GET for SELECTing, etc. The popularity of which seems to have strengthened with the growing interest in REST and the wide-spread adoption of Rails 2¹. In fact read of the HTTP/1.1 Method Definitions specification explicitly states that so-called "Safe Methods" such as GET:

... SHOULD NOT have the significance of taking an action other than retrieval.

But what happens when we have a site that tracks where you have visited, updates the "last read date" on each record retrieved, or remembers the last search criteria you used? Each of these features requires recording information into some kind of database--be it relational or simply a log file.

What people seem to overlook is the paragraph that follows in the same document:

Naturally, it is not possible to ensure that the server does not generate side-effects ... The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.

The HTTP methods should be used to indicate the user's intention without regard to the underlying implementation. The web application is an abstraction so we need to model the interaction on that abstraction. If the user's intention is to make a change to something then go ahead and use a PUT but if they're only reading some data use a GET even if you know it involves some database writes.

It may seem somewhat esoteric but spending a bit of time thinking about what the user's intention is exactly has helped me better flesh out an application's API.

¹ I don't mean to imply that Rails is the culprit here. Nothing in Rails explicitly makes these assertions. However the fact that the idiom is explicitly referred to as CRUD resources certainly doesn't help.

Getting Too Fancy with HTTP Response Codes

2008-04-15T00:07:00.000+10:00

As part of my adoption of REST and all its goodness, I've started using HTTP response codes more, responsibly ;-) So, for example, instead of always returning 200 (OK) for just about everything, I'm using 201 (Created) with a Location header set to the new URL after a POST. For PUT, I send back 204 (No Content), a 404 (Not Found) after a GET for a resource that no longer exists, and a good old 200 (OK) after a successful DELETE or GET.

Interestingly, in the system I'm developing at present, an update (PUT) might actually cause a resource to move due to the application of server-side business rules. In this case, the 204 response also sets the Location header so that the client knows where it can be found.

All this was working beautifully on my local machine using both Safari and Firefox so once I was happy with the result I deployed it into the remote test site and started playing in FireFox. So far so good. Everything checks out. Next let's try Safari...not so great.

Some bits of the application worked just fine but others seemed to have no effect. Then mysteriously things would start working again. Even stranger was the fact that hitting the browser's refresh button had no effect either.

At first I suspected that nesting Ajax calls might be to blame but as everything seemed to work perfectly in FireFox and a Google search turned up nothing, I decided to do some more investigation.

I logged in to the server box and tailed the logs for signs of life. Everything looked normal. All the expected requests and responses were there but still nothing client side. Using Safari's new Network Timeline I could see what the browser thought was going on. All the requests and responses were there but something was odd. In all but a few cases, the response code was 204 (No Content). I double checked the server logs but no, the server was definitely sending back the correct responses; a mixture of 204, 200 and 404 as appropriate.

On a hunch I went back and re-read the HTTP Status Codes document and in particular the definition for 204:

The server has fulfilled the request but does not need to return an entity-body ... the client SHOULD NOT change its document view from that which caused the request to be sent ...

That might actually explain it. If Safari received a 204 and interpreted that to mean "Don't change anything" then hitting refresh would indeed have no effect even if my code subsequently went on to perform more asynchronous requests as a result.

So, I dutifully changed all the 204s to 200s and voila! Safari started to behave just as expected and FireFox continued to work as had previously.

I've also noticed a difference in the way both browsers handle 303 (Redirect) from within an XML HTTP Request: Safari performs the redirect and keeps all the headers as per the original, whereas FireFox seems to essentially construct an entirely new request. The upshot is that you can't actually detect (server-side) an XML HTTP Request from FireFox if it is the result of a redirect.

I'm really not sure why the two browsers have such differing opinions of what the appropriate behaviour should be in either case but I hope this helps some other poor sod keep from pulling their hair out.

Drag & Drop Prioritizable Lists

2008-04-12T08:36:00.000+10:00

Yes, it's true, Scriptaculous already provides a Sortable that makes it almost trivial to enable drag'n'drop sorting of your HTML lists. Whenever an item is moved an onUpdate() event is called (if provided) allowing you to inspect the new order and presumably perform an AJAX request to record the change. In principle, this sounds great but I've never really liked it for a couple of reasons.

For a start, if you have any appreciable number of items updating each in the database just to re-order one seems somewhat unnecessary. Not withstanding the fact that we need to send all those ids to the server in the first place.

Secondly, if you're doing any kind of filtering, it's difficult at best to take the newly constructed ordering and apply that at the back-end; what happens to all the items that may be lurking in between that aren't presently displayed?

Enter Prioritizable (itself built on top of Sortable).

You use it in much the same way as Sortable with the major difference being that the onUpdate() event is called with three
arguments: the item that was moved, the sibling relative to which it was moved, and the relative position ("higher" or "lower"). And, if like me, you're feeling a bit RESTful, it's pretty easy to turn these arguments into a nice semantic URL and parameters as shown:

Prioritizable.create($("chores"), {
onUpdate: function(item, position, sibling) {
id = item.substring(6);                           // "chore_17" => "17"
sibling_id = to.substring(6);                     // "chore_2" => "2"
url = "/chores/" + sibling_id + "/" + position;   // "/chores/2/higher"

new Ajax.Request(url, {
method: "post",
parameters: { id: id }
});
}
});

When the onUpdate() event is called we POST the id of the item to be moved to a path constructed from the id of the sibling and the relative position. Assuming the the user moves chore_17 just above chore_2 we would POST "id=17" to /chores/2/higher.

In practice, I combine this client-side behaviour with some server-side code that provides move_higher_than() and move_lower_than() methods that efficiently handle all the necessary database updates.

All the pieces mentioned will eventually be available alongside Cogent's other Rails plugins but until then, here's enough of the Javascript side of things to get you going.

var Prioritizable = {
create: function(element) {
options = Object.extend(arguments[1] || {}, {
onChange: Prioritizable.onChange,
onUpdate: Prioritizable.onUpdate
});

Sortable.create(element, options);
},

destroy: function(element) {
Sortable.destroy(element);
},

onChange: function(item) {
Sortable.options(item)._item = item;
},

onUpdate: function(element) {
options = Sortable.options(element);
item = options._item;
options._item = null;

sibling = item.previous();
if (other) {
position = "higher";
} else {
sibling = item.next();
position = "lower"
}

options.onUpdate(item, position, sibling);
}
};

Enjoy!

Getting Chronic to Parse Non-U.S. Dates

2008-04-10T04:47:00.000+10:00

In an attempt to push yet more behaviour from my Rails controllers into model classes, I was extracting some code into a yet-to-be-published plugin that allows date and time columns to be set using more human-readable values. For examples:

>> task.completed_at = "now"
>> task.completed_at
=> Wed Apr 09 14:39:12 +1000 2008

Although my actual requirement was to support "now" and "today" I figured it would be rather cool if I could support anything that Chronic does. (If you haven't used Chronic before, it's a natural language date/time parser written in pure Ruby that understands a vast array of expressions including ranges.)

Naturally (no pun intended), Chronic also supports explicit dates such as "1/2/08". As part of my testing however, I discovered that the date parsing is decidedly US-centric presuming, of course, that dates are specified as "month-day-year". So for anyone living in say, Australia, it can be pretty frustrating to have Chronic.parse("1/2/08") return "Wed Jan 02 12:00:00 +1100 2008" rather than the expected "Fri Feb 01 12:00:00 +1100 2008".

The good news is, there is a solution. The bad news is, the solution is far from elegant. But first some context.

Chronic is actually written very nicely and the code is fairly easy to follow. In essence it works as follows: the input string is tokenized; each token is inspected to see if it's a keyword such as a month name, day name, or a number, etc.; and finally tries to matches the sequence of tokens against a pattern such as "a day number followed by a month name and then a year." The problem arises because the pattern for matching "month-day-year" comes before the one for "day-month-year" meaning that unless the first number is greater than 12, Chronic will always consider it to be a month.

The less than elegant solution is almost trivial and involves switching the order in which the patterns are matched. Doing so returns the desired result:

>> Chronic.parse("1/2/08")
=> "Fri Feb 01 12:00:00 +1100 2008"

Which is all very well and good but now we have the reverse problem. What would be better is if we had a more general solution, one that allows us to specify the desired precedence when parsing:

>> Chronic.parse("1/2/08")                                    # Default to U.S. date formats
=> Wed Jan 02 12:00:00 +1100 2008

>> Chronic.parse("1/2/08", :explicit_date_format => :non_us)  # Prefer Non-U.S. formats
=> Fri Feb 01 12:00:00 +1100 2008

>> Chronic.parse("1/2/08", :explicit_date_format => :us)      # Prefer U.S. formats
=> Wed Jan 02 12:00:00 +1100 2008

Among other things, this allows us to have users in Australia enter dates with one format and users in the U.S. another.

For anyone interested, I've pasted a diff for each approach below.

A less than elegant solution:

--- a/chronic-0.2.3/lib/chronic/handlers.rb
+++ b/chronic-0.2.3/lib/chronic/handlers.rb
@@ -13,8 +13,8 @@ module Chronic
Handler.new([:repeater_month_name, :ordinal_day, :separator_at?, 'time?'], :handle_rmn_od),
Handler.new([:repeater_month_name, :scalar_year], :handle_rmn_sy),
Handler.new([:scalar_day, :repeater_month_name, :scalar_year, :separator_at?, 'time?'], :handle_sd_rmn_sy),
-                 Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_day, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sm_sd_sy),
Handler.new([:scalar_day, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sd_sm_sy),
+                 Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_day, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sm_sd_sy),
Handler.new([:scalar_year, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_day, :separator_at?, 'time?'], :handle_sy_sm_sd),
Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_year], :handle_sm_sy)],

A more general solution:

--- a/chronic-0.2.3/lib/chronic/chronic.rb
+++ b/chronic-0.2.3/lib/chronic/chronic.rb
@@ -43,7 +43,8 @@ module Chronic
default_options = {:context => :future,
-                         :ambiguous_time_range => 6}
+                         :ambiguous_time_range => 6,
+                         :explicit_date_format => :us}
options = default_options.merge specified_options

# ensure the specified options are valid
@@ -51,6 +52,7 @@ module Chronic
default_options.keys.include?(key) || raise(InvalidArgumentException, "#{key} is not a valid option key.")
end
[:past, :future, :none].include?(options[:context]) || raise(InvalidArgumentException, "Invalid value ':#{options[:context]}' for :context specified. Valid values are :past and :future.")
+      [:us, :non_us].include?(options[:explicit_date_format]) || raise(InvalidArgumentException, "Invalid value ':#{options[:explicit_date_format]}' for :explicit_date_format specified. Valid values are :us and :non_us.")

# store now for later =)
@now = options[:now]
--- a/chronic-0.2.3/lib/chronic/handlers.rb
+++ b/chronic-0.2.3/lib/chronic/handlers.rb
@@ -3,41 +3,50 @@ module Chronic
class << self

def definitions #:nodoc:
-	    @definitions ||=
-      {:time => [Handler.new([:repeater_time, :repeater_day_portion?], nil)],
+	    if @definitions.nil?
+        us_date = [Handler.new([:repeater_day_name, :repeater_month_name, :scalar_day, :repeater_time, :time_zone, :scalar_year], :handle_rdn_rmn_sd_t_tz_sy),
+                   Handler.new([:repeater_month_name, :scalar_day, :scalar_year], :handle_rmn_sd_sy),
+                   Handler.new([:repeater_month_name, :scalar_day, :scalar_year, :separator_at?, 'time?'], :handle_rmn_sd_sy),
+                   Handler.new([:repeater_month_name, :scalar_day, :separator_at?, 'time?'], :handle_rmn_sd),
+                   Handler.new([:repeater_month_name, :ordinal_day, :separator_at?, 'time?'], :handle_rmn_od),
+                   Handler.new([:repeater_month_name, :scalar_year], :handle_rmn_sy),
+                   Handler.new([:scalar_day, :repeater_month_name, :scalar_year, :separator_at?, 'time?'], :handle_sd_rmn_sy),
+                   Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_day, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sm_sd_sy),
+                   Handler.new([:scalar_day, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sd_sm_sy),
+                   Handler.new([:scalar_year, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_day, :separator_at?, 'time?'], :handle_sy_sm_sd),
+                   Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_year], :handle_sm_sy)]
+
+        non_us_date = us_date.dup
+        non_us_date[7] = us_date[8]
+        non_us_date[8] = us_date[7]
+
+	      @definitions =
+        {:time => [Handler.new([:repeater_time, :repeater_day_portion?], nil)],

-       :date => [Handler.new([:repeater_day_name, :repeater_month_name, :scalar_day, :repeater_time, :time_zone, :scalar_year], :handle_rdn_rmn_sd_t_tz_sy),
-                 Handler.new([:repeater_month_name, :scalar_day, :scalar_year], :handle_rmn_sd_sy),
-                 Handler.new([:repeater_month_name, :scalar_day, :scalar_year, :separator_at?, 'time?'], :handle_rmn_sd_sy),
-                 Handler.new([:repeater_month_name, :scalar_day, :separator_at?, 'time?'], :handle_rmn_sd),
-                 Handler.new([:repeater_month_name, :ordinal_day, :separator_at?, 'time?'], :handle_rmn_od),
-                 Handler.new([:repeater_month_name, :scalar_year], :handle_rmn_sy),
-                 Handler.new([:scalar_day, :repeater_month_name, :scalar_year, :separator_at?, 'time?'], :handle_sd_rmn_sy),
-                 Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_day, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sm_sd_sy),
-                 Handler.new([:scalar_day, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sd_sm_sy),
-                 Handler.new([:scalar_year, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_day, :separator_at?, 'time?'], :handle_sy_sm_sd),
-                 Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_year], :handle_sm_sy)],
+         :date => {:us => us_date, :non_us => non_us_date},

-       # tonight at 7pm
-       :anchor => [Handler.new([:grabber?, :repeater, :separator_at?, :repeater?, :repeater?], :handle_r),
-                   Handler.new([:grabber?, :repeater, :repeater, :separator_at?, :repeater?, :repeater?], :handle_r),
-                   Handler.new([:repeater, :grabber, :repeater], :handle_r_g_r)],
+         # tonight at 7pm
+         :anchor => [Handler.new([:grabber?, :repeater, :separator_at?, :repeater?, :repeater?], :handle_r),
+                     Handler.new([:grabber?, :repeater, :repeater, :separator_at?, :repeater?, :repeater?], :handle_r),
+                     Handler.new([:repeater, :grabber, :repeater], :handle_r_g_r)],

-       # 3 weeks from now, in 2 months
-       :arrow => [Handler.new([:scalar, :repeater, :pointer], :handle_s_r_p),
-                  Handler.new([:pointer, :scalar, :repeater], :handle_p_s_r),
-                  Handler.new([:scalar, :repeater, :pointer, 'anchor'], :handle_s_r_p_a)],
+         # 3 weeks from now, in 2 months
+         :arrow => [Handler.new([:scalar, :repeater, :pointer], :handle_s_r_p),
+                    Handler.new([:pointer, :scalar, :repeater], :handle_p_s_r),
+                    Handler.new([:scalar, :repeater, :pointer, 'anchor'], :handle_s_r_p_a)],

-       # 3rd week in march
-       :narrow => [Handler.new([:ordinal, :repeater, :separator_in, :repeater], :handle_o_r_s_r),
-                   Handler.new([:ordinal, :repeater, :grabber, :repeater], :handle_o_r_g_r)]
-      }
+         # 3rd week in march
+         :narrow => [Handler.new([:ordinal, :repeater, :separator_in, :repeater], :handle_o_r_s_r),
+                     Handler.new([:ordinal, :repeater, :grabber, :repeater], :handle_o_r_g_r)]
+        }
+      end
+      @definitions
end

def tokens_to_span(tokens, options) #:nodoc:
# maybe it's a specific date

-      self.definitions[:date].each do |handler|
+      self.definitions[:date][options[:explicit_date_format]].each do |handler|
if handler.match(tokens, self.definitions)
puts "-date" if Chronic.debug
good_tokens = tokens.select { |o| !o.get_tag Separator }

Fixing lowpro form submission failures

2008-04-03T00:13:00.000+11:00

I finally worked out why my forms weren't submitting when the user hits the ENTER key. lowpro serializes the button that was clicked along with any other parmeters when submitting a form via AJAX. However, when the user hits enter under FireFox, there is no button and consequently the browser barfs. Safari on the other hand tries to be too helpful and triggers an onclick event for the first submit button (which is why I never noticed it).

So anyway, rather than try to be too clever myself, I simply changed the parameter serialization in Remote.Form.onsubmit to look like:

parameters : this.element.serialize({ submit: this._submitButton ? this._submitButton.name : null })

Problem solved.

Culturally Sensitive JavaScript

2008-03-20T05:43:00.000+11:00

JavaScript is a fantastic little language and with the likes of Prototype, Scriptaculous, and my newest favourite, lowpro, you can build some quite frankly, remarkable web applications.

One area where most web browsers fall down however is in their error-reporting, or lack thereof. A fact that has caused me to waste seemingly countless hours trying to find the source of some problem or other only to realise that a typo that had been staring me in the face the entire time was to blame!

Now, like just about any programming library I use these days, most JavaScript libraries use American english. initialize, capitalize, you know what I'm talking about.

For the most part the use of 'z' instead of 's' isn't too much of a problem but just recently I consistently tried to use lowpro's addBehaviour method, only there isn't one. It's called addBehavior (sans 'u').

So today after about 20 minutes cursing and swearing at the spelling Steve asked "is there anyway you could create an alias?" Being JavaScript the answer is of course "abso-bloody-lutely!":

Event.addBehaviour = Event.addBehavior

You can alias just about anything this way.

No more will my code silently fail due to differences in spelling :)