Ruby Pond

JRuby, and why bouncers are assholes

Thu, 30 Jul 2009 12:58:30 GMT

I’m normally not one to use this site as a soap box or an outlet for opinion pieces. But I figured it’s easier than being continually baited into replying on all the various blogs/forums/groups out there. There’s a whole heap of talk in the ruby community at the moment on JRuby and it’s place in the ecosystem. Most of it has been kicked off in response to Charles Nutter’s post on JRuby’s importance to Ruby. I was fortunate enough to see Charles’ presentation at Rails Underground last week and have a chat to him, and he had some great insights and examples of why JRuby actually is pretty damn awesome.

The sky is falling

The only real problem I have with Charles’ post is the inference that all is lost for the Ruby community if we don’t embrace JRuby quickly. It’s not, it won’t suddenly disappear. What we will possibly lose is the migration of great java developers like Jim Weirich and Charles. I’m willing to excuse a bit of sensationalism in his post though, we’re effectively talking about his livelihood here. He needs to get people to pay attention, he needs people to use JRuby, he needs more of us to embrace it.

Whatever his motivations, he’s right.

Java sucks

Yes, we know. But you’ve missed the point. This isn’t about giving us* java, it’s about giving *java ruby. Or better yet, giving ruby the JVM. Just try and think through the argument logically; why on earth would a java developer who is unhappy with java for whatever reason move to a new language, that gave you all the same obstacles? They wouldn’t (no matter what your opinion of the average java dev is). It’s about combining the flexibility of ruby with the good bits of java, the bits that are good enough to make people want to stick with java.

Java/The JVM is slow

“Good morning Mr. Kettle, I’m the pot… and you my friend look black”. Really? People that are happy enough with the performance of ruby 1.8 are pointing the “unperformant” stick at other interpreters? Give me a break. The fact is that the JRuby implementation (in most cases) is as good as 1.8, and in many is as good as 1.9. The fact that such a minor section of the community seems to be aggressively migrating to 1.9 highlights the fact that performance really isn’t such a high concern for you in real life work, it’s just a great scape goat for debates. Oh, and worried about that additional few secs startup overhead from starting the JVM for little scripts? Then don’t start it up

Ruby/rails is a competitive advantage

Bull. Shit. Your competitive advantage is your expertise and the ability to meet your client requirements, as efficiently as possible. Period. Ruby and Rails have helped you have an edge for the past few years, but there’s many alternatives that are closing the gap. Most of my clients don’t care what technology I use so long as I do the best job for them and make the most efficient use of their money. And why should they care? The few clients I have worked for that would care, would probably still prefer I use java over ruby.

The EngineYard deal is acknowledgement from Sun that even they think JRuby is dead

Wrong. JRuby is never going to be a core part of Sun’s business. There are so many people that are happy with the status quo that they will continue coding away in java until such a day it might be declared deceased. Why would they have any desire to invest both time and money to stop such a minor outflow from their business? They wouldn’t, they’ve got bigger problems to deal with. On the other hand, EngineYard is all about infrastructure… and infrastructure for hosting ruby apps specifically. Higher performance ruby implementations means they need to invest less in hardware for the same return. It has a direct and appreciable impact on their bottom line. Sun was a great fit initially to allow the guys access to the java team so we could get the most out of the JVM. EngineYard have more to gain out of a successful JRuby at this point than Sun do, so they’re a much better fit.

We don’t want more java developers or the java community

This is the one that irks me the most. I can only recall two occasions in life where one group of people proclaims they are too cool to associate with another group of people. Bouncers at nightclubs, and “that” group in high school. Bouncers are assholes, and those kids from high school probably grew up to become bouncers.

If you’ve got a problem with someone else, that’s your own insecurities and your problem to deal with, don’t infect the rest of us with it. What’s the issue here? You don’t like their code? Then fork it, fix it, or do it yourself and the problem magically goes away. If you don’t want to do that, then shut your pie hole and pay some respect to those that actually do the work. You don’t like the competition? Go read a book or five, find some better developers to pair with, stop trying to hide.

If we’re using a technology that really is as great as we make out, we should see it as inevitable that the community is going to grow significantly in size at some point. I think Malcolm Gladwell refers to it as the tipping point. The technical ability of the average developer may drop a little. Commercial interests may infest us and move conferences to Vegas. But for every 10 hacks you still end up with a Jim Weirich, Giles Bowkett, David A. Black, etc. And for every large conference you end up with a Rails Underground, Scotland on Rails, or Ruby Hoedown.

Looking at the world through ruby tinted glasses

The fact is that for a long time the performance of ruby has sucked. We’ve worked around it and justified it for a long time, and for the vast majority of people it didn’t suck bad enough to cause problems. Heck, we were all happy using webrick until Zed Shaw came along and showed us a better way. But 3rd party library support has also sucked. Trying to create PDFs was a nightmare for ages. Trying to work with Microsoft document formats still is. I can think of at least 5 occasions where I’ve solved a simple problem in another language because doing it in ruby would have been far too difficult.

For all the talent the community has, there are so many areas where good support is still lacking that it’s unrealistic to think there are enough talented people who care and have the time to do anything about it. Combine that with the fact java has at least (commercially) a 10 year head start and the gap between the two languages is huge. Java has plenty to offer us.

Why can’t it all just be peace, love, rainbows, and freakin flying ponies?

All this arguing seems to do is instill a whole “us and them” mentality. There’s no need for it to be that way. Embrace JRuby, take advantage of the performance of the JVM, get access to a whole swathe of new libraries that go far beyond what your current list of gems currently offers you… and get to keep using the language you love.

One of the things I admire most about the ruby community is the ability it has to support innovations and allow the proverbial cream to rise to the top. WebBrick was replaced by Mongrel, then we had Thin, and now we’ve got Passenger. SOAP became REST. AttachmentFu became Paperclipped, etc. I’d hate if in this instance it wasn’t allowed to happen because of a whole heap of misconceptions about what JRuby was and what it had to offer.

Or maybe we just need to get a mascot like Django and chill the fuck out.

Rails Underground - Day 2

Sat, 25 Jul 2009 08:59:19 GMT

update: videos are now online

I was a little late to the kick-off of the keynote on day 2 with Yehuda. For the most part if is another discussion on what’s happening in Rails 3 so I don’t think I’ve missed anything I hadn’t heard before. He then started to go into more detail on how things have changed than he has previously, probably due to the progress since I last saw him in March…

Yehuda Katz - Keynote

Yehuda went through the upcoming changes in approach and philosophy on rails Dev. Having well defined contracts, consistent conventions, and allowing things be be changeable.

One of the changes of not is splitting off ideas and interfaces into modules which can then be imported into a class at runtime. He used the analogy that your parents didn’t define everything you could do when you were born, you can learn new things. So to when you define ta class you don’t have to define the complete interface up front, nor if you want to make changes to you need to re-implement the entire class. Instead you can have sub-sections of interfaces included or changed at runtime where appropriate.

In Rails 3 this translates into better separation of the components and clearly defined interfaces. No longer are you forced to use ActionView, you just need to use something that is “ActionView compliant”. The same goes for ActiveRecord, your models are just “ActionModel compliant”.

An ActionView compliant interface just needs to implement the following 3 methods:

for_controller(paths, assigns, controller)
render_template(opts)
render_partial(opts)

An ActiveModel compliant interface needs the following 4 methods:

new_record?
valid?
errors
model_name

That last model_name method needs to return plural, partial name, and a few other things rails expects internally. It’ll only be 10-20 lines of code to implement manually, or there is a module to include if you just want the default behaviour.

Next up is a cleaner separation of components, one which we got a demo of was splitting out the ActiveModel::Validations. Now you can create your own ActiveModel compliant persistance/ORM layer, and pull in the existing validations for free regardless of where it’s stored (YAML, CouchDB, etc.).

By implementing the above you get a system that fully works with ActionPack, so all the usual form and error helpers will just work.

The other benefit is that pulling in support for these doesn’t need you to load the entire Rails stack and rob your machine of RAM, it’s only a 1.5mb hit.

Next was the new ActionController::Metal demo. At first I wondered why I’d bother, it was slightly more vebose than just using Rackable or rolling it manually, and seemed to carry more overhead than Sinatra. But it gives you the ability to opt into additional features easily. If you wall before/after filters you can pull in the callbacks module. If you want better rendering options (various formats, etc) pull in the rendering module, if you want different layouts pull in that module. ActionController::Base is basically just ActionController::Metal with all of the modules pulled in, so you’ve now got the way to strip back to metal, or anything in between.

The final piece is ActiveSupport. If for some reason you want to pull in the entire ActiveSupport behemoth you just call:

require 'active_support/all'

Otherwise you can call in the various newly separated modules, one of note was

require 'active_support/ruby/shim'

Which backports a bunch of useful ruby1.9 features on datatypes that could be easily implemented in pure ruby. So you can use them now, and when you upgrade to ruby1.9 you get the performance improvement of having them implemented in C.

The end result is a new level of granularity. Just because you want to use DataMapper doesn’t mean you have to say goodbye to all of ActiveRecord. So you can opt-out of just the tiny pieces you don’t want. Conversely, you can start completely bare and opt-in on various features to build your own framework on top of rails. You could now build Sinatra on top of rails, and when the complexity of your app increases you can much more easily pull in the additional features you need. All in all, it sounds really cool. Cool enough that I might even try using it on my next app.

Dr. Nic Williams - Dead Simple JavaScript Unit Tests in Rails with Blue Ridge and Screw.Unit

Dr. Nic has dropped a very important bombshell right at the start of his talk. The while Australia may indeed be losing the current 5 game series, we are still holders of The Ashes. Now on with the less important aspects of the presentation.

Nic has a bunch of open source plugins and projects which he could have come and given a presentation on, but he’s so impressed with BlueRidge that he feels it’s the most important thing to happen to the Rails ecosystem since Cucumber. At a high level it allows you to do both in browser and headless testing, it fits in with the existing rake tasks, and means that javascript testing should just become part of the usual TDD/BDD development approach and not be an afterthought in Cucumber.

Generating a javascript test with BlueRidge creates a spec file, and a fixture file which will give you some test HTML. The idea being that most of your javascript ultimately want to modify the DOM, so run the app and copy out the rendered HTML you need to use as your stub.

An example of a test is as follows:

require("spec_helper");
require("../../public/javascripts/accounts.js");

Screw.Unit(function() {
  describe('accounts/new initially has fields', function() {
    it("should have name field", function() {
      expect($('#account_name').size()).to(equal, 1);
    });
  });

  describe('accounts/new to fail if missing name', function() {
    it("should have erroneous name field", function() {
      $('#account_submit').click();
      expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
    });
  });
});

The first test will check that an account_name field exists, and in TDD it shouldn’t. So next you should go insert the HTML you need into your stub HTML file. (note: you’ll also need to ensure that the form doesn’t actually submit on the test, so you need to add a function to return false when the button is clicked).

When you run the tests you should find that you get nicely green/red tests in browser for any passing/failing tests.

require("spec_helper");
require("../../public/javascripts/accounts.js");

Screw.Unit(function() {  
  describe('accounts/new to fail if missing name', function() {
    it("should have erroneous name field", function() {
      $('#account_submit').click();
      expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
    });
  });
});

Next it is checking that when you submit the form, it wraps the empty form input with the standard rails error div. Run the test, get a failure. Implement the code and the tests go green. You’ll probably notice that the test passes but you don’t get the red box, so you need to include the CSS in the test (just below the require) to actually see what the user would:

css("../../public/stylesheets/scaffold.js");

Next Dr. Nic went through a big gotcha, in that that tests don’t run in transactions. Which means the DOM isn’t reset back to it’s an initial state, and he highlighted it by adding an additional test to check that when you input a name that you don’t get the error div. Unfortunately the test fails, as the div had been inserted by previous test.

The way to deal with it was using a before block in your test:

require("spec_helper");
require("../../public/javascripts/accounts.js");
css("../../public/stylesheets/scaffold.js");

Screw.Unit(function() {  
 before(function() {
   $('#account_name').val('');
   $('div.fieldWithErrors').remove();
 });

 describe('accounts/new to fail if missing name', function() {
   it("should have erroneous name field", function() {
     $('#account_submit').click();
     expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
   });
 });
});

If someone can come up with an easy way to snapshot the DOM/body.innerHTML and all attached events and substitute it back in after each test that would be awesome.

Defining your own matches is also pretty simple. You simply add an object with match (to do the comparison) and failure_message to display the feedback and push it onto the matchers hash/dictionary.

If anybody is wondering if they should test more, you need to watch the last 30secs of the presentation once the videos are available. Great job Nic :)

Paolo Negri - Divide and conquer riding rabbits and trading gems

Paolo had a problem where he had 1,000,000 search phrases and wanted to compare the results between Google and Bing (warning, don’t try this at home… it’s a breach of the terms of service). Trying to fetch 2m pages is quite a time consuming task, enter a distributed approach.

How many nodes will you need? How many workers? What mechanism should you use to distribute the work? RabbitMQ is an open source implementation of AMQP, written in Erlang. The fact it’s written in Erlang is relevant is it inherently supports concurrency, fault tolerance, and distribution of tasks.

To get started you need to install the rabbitmq package and a gem:

sudo apt-get install rabbitmq
sudo gem install tmm1-amqp

There’s a link to the code at the bottom so rather than transcribe it all I’ll just talk through it.

First you create a worker which does the setup and then subscribe to the queue. The queue is where RabbitMQ will store all outstanding work, and the workers will request new work whenever they’re free for more processing. At the other end of the process what happens is that your app pushes a message to an exchange, the exchange pops it on a queue, and then workers come take it off the queue. You can have multiple exchanges, multiple queues, and multiple workers. Queues and messages are resident in RAM, however you can persist them to disk for further fault tolerance (for a performance hit).

There are problems though, if a worker has a problem you could lose some messages. To get around it you can send an ACK message from the worker to prefetch a message. You then process the message, and acknowledge completion. Dead client connections go unacknowledged and the work is handed off elsewhere.

Another problem is there is no easy way to control the workers. The solution to this is to great a system queue. You give each worker a unique ID and explicitly define which exchange you want to use, and make it use the system queue. You then subscribe the worker to that queue. If you want to do something to the worker you simply post the appropriate command onto the system queue, and the worker will then pick up that command as soon as it can. You can also multicast messages to affect all workers.

Next up was drilling into some of the detail of fetching the pages Google/Bing. For anyone who’s not done something like this, you really need to use EventMachine and EventMachine::Deferrables. It means you’re not blocked waiting for socket responses, but can fire callbacks when you get a response back and you can do other tasks (fetch other pages) concurrently in the one process.

Problem #3 was working out how many workers you had, and keeping track of what they were doing. The solution was somewhat similar to the system approach, setup a heartbeat queue and have the workers post to the queue what they’re doing. Then create a different worker which subscribes to that queue, takes all the messages off, and posts them somewhere for you to interrogate or view later (like a web page?).

For doing TDD with RabbitMQ there is moqueue which gives you easy testing of workers. There is also a web front-end for monitoring your RabbitMQ clients and make sure the system is alive called alice. For testing with EventMachine you’ve got em-spec.

Slides are available at SlideShare. Code is on Github

Pat Allan - “Sphinx - Beyond the basics”

Sphinx is a full-text search engine, and Thinking Sphinx is an ActiveRecord plugin which will use Sphinx. Once you’ve set it up if you want to search you just do:

Article.search "my name"

You’ll also need to use a few rake tasks to create/update the indexes. Now the basics are out of the way.

Facets

Pat’s written some ruby code to provide a means of offering facets (search summaries) like other engines like Solr offer. To use you do

indexed field, :facet => true
has_attribute , :fa

rebuild your indexes, then you’re good to search.

@facets = Article.facets 'pancakes'
@facets == { :author => { "Pat Allan" => 12,
                         "Glenn Gillen" => 1}.
            :tags => { "breakfast" => 2,
                       "brunch" => 3}}

You can then drill into the results to return just the Articles for a specific author:

@facet.for(:author => "Pat Allan")

Excerpts

It got a bit of a cheer, apparently some people have been hanging out for them. They’re basically the bold bits on a Google page of results, where the actual search terms are highlighted within the results. The syntax is:

@articles = Article.search 'pancakes'
@article.excerpts.body

Geo-searching

Pat thinks this could be the killer feature. The only gotcha is that the datatypes need to be in radians as floats. To define lats and longs on your models just do:

has latitude
has longitude

If you’ve already got lat and long data as degrees and decimals you can do:

has 'RADIANS(latitude)',
    :as => :latitude,
    :type => :float
has 'RADIANS(latitude)',
    :as => :latitude,
    :type => :float

Once that’s done, you can find all pubs that serve pancakes near a given point:

@pubs = Pub.search 'pancakes'
  :geo => [@lat, @lng],
  :order => '@geodist ASC'

Once you’ve got your pubs you probably want

@pubs.each_with_geodist do |pub, distance|
end

International Sphinx

Using the character set table means you can now handle international characters easier, so an “e” with an accent can be treated as a plain old english “e”. Same with other similar characters.

With word stemming you can now define a custom stemmer to help you handle stemming in various languages without too much trouble.

Different Delta Approaches

In most rails apps you want information to be available as quickly as possible, so when users update a field you don’t want to have to rebuild the whole index. To get around that you can do the following:

define_index do
  set_property :delta => true
end

The downside is you end up with a new column on the table, and a little additional HTTP overhead.

Instead you can skip the extra column and use the updated_at field to manage deltas:

define_index do
  set_property :delta => :datetime, :threshold => 2.hours
end

The downside is they’re not updated immediately, they’re updated every 2 hours. Another alternative is to use the delayed option which pushes the request out of process using DelayedJob.

define_index do
  set_property :delta => :delayed
end

Removes indexing from HTTP requests, and the results are available soon(ish). The downside is it needs a constantly running rake task (using delayedjob) and a long queue of updates could take a while to get through.

Multi-server deployment

Basically, set the remote_spinx value to true, use the DelayedJob approach, and make sure Spinx and the database are on the same server. You’ll probably want the DelayedJob and ThinkingSphinx on that server too. Pat’s not actively using multi-server setups so is looking for feedback on this.

Sphinx Scopes

People have been using named_scopes with Sphinx and it’s caused some problems because of the difference between SQL and Sphinx. The new sphinx scopes are an implementation that works just like named scopes, except they chain together properly and work with pagination.

Brendan Lim - Mobilize Your Rails Application

There are an estimated 4 billing mobile users, which means there are more phones than there are personal computers. Approximately half of those users have mobile web access. With that much reach you need to make your apps accessible to as many of these users as possible.

THe problem with trying to do just one webpage for all devices is problematic because of resolution differences, javascript, flash, bandwidth, etc.

Enter MobileFu. It’ll detect if a user is mobile, can add custom styling based on user agent and takes you one step closer to one webpage. You use it like:

class MyController < ActionController::Base
  has_mobile_fu

  def index
    respond_to do |format
      format.html
      format.mobile
    end
  end
end

Which will output the page in a mobile friendly format. Some of the magic happens in the stylessheet_include_tag:

stylessheet_include_tag "foo"

will look for foo_iphone.css, foo_android.css, etc. so you can have custom styles for each device. You’ve also got various helper methods to take actions within you view:

is_mobile_device?
is_device?('blackberry')
in_mobile_view?

Rails iUI is a wrapper for the iUI user interface framework. It takes care of detecing rotating an iPhone and various other helpers. To use it you do the following in your controller:

class MyController < ActionController::Base
  acts_as_iphone_controller

  def index
    respond_to do |format|
      format.html
      format.iphone
    end
  end
end

And then within your view:

<%=include_iui_files%>

To display iPhone styled default interface objects you’ve the following methods:

iui_toolbar( initial_caption, search_url)
button_link_to()
iui_list(items, opts)
iui_grouped_list(items, opts, &group_by_block)

Again you need add some stuff into your views to make it work:

<body <%=register_orientation_change%>>
observe_orientation_change(url_for_options)

you’ll receive params[:position] in the request to the url you put in observe_orientation_change, 0 is upright. 90 and -90 is the two various landscape modes (depending on rotation).

Next up is SMS. You can use Clickatell’s API. Once you connect to the API you just do:

api.send_message(phone_number, message)

Brendan also wrote something called SMSFu. The problems are that you need to know the recipients carrier and it doesn’t support as many carriers. But because it sends SMS via email it’s free to send (to supported carriers). To send you just do:

deliver_sms(phone_number, carrier, message)

Now how do you receive SMS? Well you can get a shortcode (an inbound SMS number like 80800), but they’re quite expensive to setup with monthly costs associated. Alternatively there are providers (like TextMarks) where you’re given a keyword that users need to prepend on the front of their message, and you share a shortcode. TextMarks then send the next set of words (up to 9) as an array to you to do what you want with.

If you’re wanting to receive MMS, then MMS2R is the gem to use. It strips out any carrier advertising and other stuff that may be added to the core attachment that you actually want. You’ll get access to the message subject, body, and default_media (the attachment).

Jim Weirich, DHH, Obie Fernandez, Geoffrey Grosenbach, Jonathan Siegel - Q&A Session

Bah, I can’t keep pace with questions and responses in a fashion that does them justice. Best wait for the videos.

Eleanor McHugh - The Ruby Guide to *nix Plumbing

Unix isn’t just a server based operating system, it’s a philosophy. It consists of a whole heap of little tools that all receive a little piece of input and do just a little piece of work. And they can all work together to achieve much bigger tasks.

Eleanor went through a bit of an explanation of how she felt the fascination on better threading in ruby is a bit stupid, and that the introduction of the threading approach we have in 1.8 was a backward step from 1.6. Most problems you can get around by forking the process, and threading is very rarely the appropriate solution to the problem.

The shell system is simply a process which provides you an interactive environment, job/process management, and a scripting language. However, you can get around using the standard shell scripting languages by using the under-documented shell.rb.

A little known fact is that the IO/File new methods don’t just accept a string, but will also accept a file descriptor (which is just a number). So if you open 0, 1, or 2 you’ll end up writing directly to STDIN, STDOUT, and STDERR respectively.

Taking the approach further means that you can effectively communicate with any process in a similar fashion by sending signals to the appropriate process descriptors.

When working with sockets in ruby, Eleanor said you should turn off reverse DNS lookups as they will greatly slow down network connection and tie up your processes needlessly.

There’s also a whole host of examples of using low level system calls and examples in the slides which I’ve no chance of transcribing quickly enough with the right context.

Lindsay Holmwood - Behaviour driven monitorins with cucumber-nagios

Everyone is probably familiar these days with what Cucumber is and the descriptive syntax it gives you for defining tests. You can combine it with webrat and mechanize to give you a way of monitoring external websites. To get started you need to install the gem and setup a new project:

sudo gem install auxesis-cucumber-nagios
cucumber-nagios-gen project mysites
cd mysites
rake deps

You then create a new feature, for the sake of example create one to test the navigation:

cucumber-nagios-gen feature rails-underground.com navigation

which will create the file features/rails-underground.com/navigation.feature. From there you can use all the usual cucumber/webrat commands to go to a page, click links, test for the result text. To run the test you:

cucumber-nagios features/rails-underground.com/navigation.feature

cucumber-nagios will then take care of taking the cucumber results and outputting them in a nagios plugin format.

The caveats are that you’re going to run problems if you site is javascript dependent. But you’re embracing progressive enhancement and your site works fine without javascript, right? You could also run into some problems if you’re trying to run multiple scenarios within the same feature file.

The big benefits of this approach is that you end up making your system monitoring actually test what you care about. Traditionally these tests are just a ping or socket connection. While if you’ve got a server down your tests will definitely fail, but there is a whole host of other cases where your site might be responsive but not usable by a real person (DB down?). Lindsay argues that this approach is really (and should be) continuous integration for systems monitoring.

Some cool examples of people extending it include telephony-systems-test which allows you to test an Asterisk/Adhearsion telephony dial plan. cucumber+dash which pull metrics out of the hosted Dash metrics system to alert you of application performance problems.

Lindsay went through what seemed like an awesome replacement for nagios that he’s been working on, which is handling upto 6,000 tests per second. It’s super top secret at the moment and didn’t want specifics to go outside the room. He’s auxesis on github, you join the dots.

I’ve also got summaries for Rails Underground - Day 1

Rails Underground - Day 1

Fri, 24 Jul 2009 08:54:50 GMT

update I’ve also got summaries for Rails Underground - Day 2, plus videos are now online

Today kicks off the first day of talks for Rails Underground in London. Thankfully they’ve got usable wifi, and two days of what sound like mostly excellent talks. So over the next couple of days I’ll hopefully be able to post summaries of the various talks I’m able to squeeze into.

So first up…

Fred George - Rails is a Hammer (Keynote)

Taking a queue from Martin Fowler’s “Patterns of Enterprise Application Architectures” Fred goes in to discuss the various trade offs that happen in different application styles. Essentially, approaches that get you going quickly usually become exponentially more difficult as the complexity of your problem increases. Rails with it’s “convention over configuration” offers a great way to prototype very quickly, but by Fowler’s assumption this should get slower and slower to extend as your app complexity grows. So Fred decided to start from scratch, take his learnings from smalltalk and see how things would turn out on a new project.

Models were just pure ruby, storing to a YAML persistance layer. Sinatra took control of the controllers, with HAML/SASS for the view layer. I turns out that with very little code, YAML provided an easy way to persist the objects quickly to disk. Incrementing filenames on each save also meant that object state was versioned, and actions were simple to undo/redo (and note we’re talking full objects being serialised to disk with fill state, not just instance variables/attributes).

Then to link the model with the view comes in Sinatra. It’s so light and easy, does nothing more than it needs to, it’s essentially just a set of regexps that push a request to the appropriate code. And how much more do you really need to do in most controllers?

Next was the view, and for anyone using HAML there is nothing groundbreaking to learn here. It’s awesome, heaps better than the prescribed ERB, and you should be using it (my conclusion on Fred’s discussion and my own experience, don’t flame him directly). It’s not just less typing, the prescriptive structure makes the code more readable and being able to use variables in CSS is incredibly liberating and DRY.

Some good debate post keynote on whether it was really a comparison of rails vs alternatives, or just a test project using stuff rails already offers you without using rails.

Gwyn Morfey - Refactoring with Fire

A high paced and amusing presentation about trying to refactor old apps. Why would you need to consider a rewrite? “Maybe it was PHP on Rails, or some guys first Rails project”. The options are burn it to the ground and start again, or try and refactor it. The default option should always be to try and refactor. It’s usually tested, and it works, and if you re-write it you’re basically shutting down and delivering absolutely nothing for an extended period town. At which point you eventually release, with the same feature set. And even if you’re re-writing in an agile fashion, your first releases still have only a minor percentage of the features expected by the users and will be considered a fail.

You refactor unless:

You have a very, very, very good reason to rewrite

You rewrite if:

You can drastically cut scope
You won’t stall the business
You have access to the original coder
The existing business logic is fundamentally wrong (the requirements have changed?)

When going through the estimation phase, it’s often not accurate to just take a cursory glance at the code and see how easy it is to understand. So much can be hidden under the surface that it’s usually best to try a spike at implementing something new and see how easy it is.

If you’re considering a re-write, you need to approach it as a complete outsider and say “I know nothing, what does this thing need to do”. Otherwise you open yourself up to the stealth feature that was hidden deep in the code that you lost in your re-write. Boot it up, use the application… and watch the client use the application.

To make it work and keep the client happy, make sure that any re-write is actually adding value to the existing system. Deliver early, deliver often.

That’s it on re-writing, next is how to make refactoring manageable.

Gwyn also added in a shameless plug for his “noisy partials” plugin which will insert HTML comments in the rendered views to help you identify where exactly things are being rendered. The “partial dependencies” plugin will draw a nice pretty dependency graph showing the links between all the partials in a project.

Split the long methods into smaller workable and meaningful chunks. You can then work on refactoring these bite sized pieces while you’re working through the codebase. If there’s code you don’t understand, delete it and see what breaks. If nothing breaks, kiss it goodbye… “Deleted code is debugged code”. Make sure that you’re only working on the small bites, and get it back to a working state before you continue. Otherwise you’ll forget what it was you were originally working on, and before you know it you’ll have upgraded your entire rails stack and plugins and nothing works. And don’t refactor code just because you hate it, much sure you hate it and it’s in your way.

Check out Sequel as the database adapter if you’re trying to move legacy data between SQL database schemas.

Desi McAdam - Working with Legacy Rails Apps - Technical Debt Hell and how to work your way out of it

Another talk on dealing with old rails projects and code, am I detecting a general theme here? So what constitutes legacy apps here?

Old Rails versions
No tests
Complexity in architecture and design
Violates best practices (fat controllers, poor separation, etc.)

Essentially, code that is difficult to change. It all boils down to technical debt.

At Hash Rocket they have rescue missions which they do for clients they inherit with these problems. First off they’ll spend a couple of days doing a code audit. As Gwyn pointed out though, much of the detail isn’t uncovered until you try and implement something.

So what to do and how to make it workable?

Pair all the f*ing time. You need someone to check you and stop you from being sucked in to writing the same style of bad code. If you look at it long enough you’ll start to find it normal.
Be fearless. Take the risks and delete the code.
Write tests. Use cucumber to wrap the main functionality to help give yourself a sense of stability.
Stay focussed. At the point of change wrap tests around the current functionality, BDD the changes. And don’t get distracted (at which point it an amp starts squealing in the background and disrupts the talk… brilliant timing)
Refactor. Always refactor with tests, and do it as you go. It helps estimate velocity, removes technical debt, and alleviates the broken windows problem.
Use Git (or anything that lets you branch easily)
Use your customer. Deliver early and often, and get them to help you test. They’ve got the best idea of what it’s meant to do.

The specific things hash Rocket do in their approach is:

Update the app to the latest rails version
Update the plugins and external dependencies (or remove/un-fork where possible).
Fix the easy stuff. Clean the formatting, remove comment code as you’re unlikely to get much value from is.
Use the free stuff. Replace custom code with built-in rails functionality, use new features (like named scopes) to decrease complexity.

Obie Fernandez - Blood, Sweat, and Rails

Obie got introduced to ruby via Aslak of Cucumber fame back in 2004, with rails in 2005 for a client in London.

Sales & Marketing

First piece of advice, controversy can be useful. One of the first things Obie did was writing the top 10 reasons why Java sucks. Just make sure you can take the abuse, especially if you deserve it… he’s willing to admit his initial ideas for Rails Maturity Model sucked. Confidence always win. On the flip side of that, don’t ever seem desperate. The moment you go in and convey, even subconsciously, that you need the work you make the client wonder why you need the work. They’d much rather work with someone who is in demand. Referrals will be the lifeblood of your business, so go ahead and videotape your clients. Get them used to it, video meetings, testimonials, and help document the project. It’s a tactical nuclear weapon.

Make it personal, pick up the phone and call a potential client. It’s way too easy to just fire off an email or IM.

Legal Matters

You need to write bulletproof contracts. Keeping them simple is not ideal, too many potential problems. Master services agreements, warranties, digital signing/acceptance, etc. all need to be included. Don’t negotiate under pressure. You’ll end up agreeing to things that you shouldn’t to and it will cost you more in the long run. And when things go wrong, lawyers cost a lot of money.

Finances

Don’t undercapitalise. Don’t do fixed bid contracts. None of Hash Rocket’s contracts state a deliverable, you’re buying time and expertise. There is no definition of scope. By documenting scope you’re breaking the philosophy of agile and you lose the ability to adjust scope. You’ll probably need to gather some momentum before these are applicable. Make sure you charge what you are worth (a slide showing that Obie billed out at $250/hr, other devs at $190/hr for short term work $150/hr for anything greater than 6 months). Allow some budget for non-paying clients. When you’re extending credit, some times they’re just not going to pay (either because they’re bastards, they ran out of funding, they’ve drained the kids college fund, etc.). Allow some budget for product development, everyone wants to be 37signals eventually. Don’t invoice manually, Harvest is a good example of how to handle it easily.

Agile Practices

You shouldn’t have to defend agile. Just start from the assumption that “this is how we work” and that it’s that the normal case, then you’re not having to justify a change in approach. Always do agile by the book first, don’t try and tweak it and then wonder why it doesn’t work. Storycard the work, either on index cards or in something like Pivotal Tracker.

Client Relationships

Execution is absolutely critical. If you’re not consistently striving for excellence then it’s all worthless. Going hand-in-hand with that is perception is reality. If you can’t relay to the client that you’re doing an excellent job then it’s worthless also. Bend the rules, the client just wants a result and doesn’t really care how you go about it. In some cases you should even break the rules. Another simple one is to establish contact, especially with big value contracts, call them every day. Mind their budget, especially if you’ve made the jump to time based contracts. The responsibility falls back on to you to make sure you’re not burning through all their cash so have transparency on who’s working and what the cash burn rate is. Do what you can to win their people over. Fire clients if they deserve to be fired. Hire people on a contract to perm basis (bah! I say they should all be contractors if they’re any good). Keep your employees constantly learning. And make them pair all the time. Empower your employees for change, be open for them to challenge the generally accepted practices (like pairing all the time). You need to actively work to make that possible. Make the work environment appealing and always have fun. Keep everyone in the loop all the time.

Charles Nutter - The Present and Future of JRuby (and the Future of Rails as it relates to JRuby)

The current version of JRuby is 1.3.1, it’s ruby 1.8.6 compatible (give or take, it can’t do continuations and some other things). Has some ruby 1.9 support (somewhere 75%-90% done).

It’s roughly equivalent in performance terms to ruby 1.9, with real native threads and runs rails fine. It takes at least 0.5secs to start up, but can take several seconds so it’s not great. Once up, it should be faster than 1.9 in almost all cases. Most ruby application bottlenecks are in the core classes and not ruby itself (string manipulations, working with hashes, etc.) and the JRuby performance in these classes is mixed which makes providing meaningful benchmarks almost impossible. Some cases are great, others are bad. Also because of the iterative optimisations that the JVM does internally a single run of an application doesn’t show real world performance. To demonstrate Charles ran a fractal generation program 5 times and you could see how much quicker the latter runs were.

It’s the only ruby implementation with true native threads. Again another demonstration showing how with JRuby you can effectively max out two cores, yet with regular ruby, even 1.9, you only ever really get the equivalent of 1 core working (spread across the two cores). Next was a really cool demo that I’ll do no justice here. Basically using FFI to call C functions and have them then execute ruby callbacks.

Next was a demo of starting an app (Typo) using Glassfish. Just download the gem, go into the app and:

"glassfish -e production"

If you’re running threadsafe rails it will start just one instance of the JVM and keep memory usage low. In any event, it takes care of concurrency and all the usual problems for you.

Ruby 1.9 support

It’s at about maybe 80% of 1.9.1. The 1.9.2 release adds a bunch more, but they’d like some help (so get in touch if you’re interested/able). He also gave a cool example of a change to regexps in 1.9 that I’d not seen. If you want to extract a grouped match from a regexp pattern you’d normally do something like:

str = "Welcome to Rails Underground"
matched = str.match(/Welcome to Rails (.*)/)
matched[1] == "Underground"

The problem is if you change the regexp to:

matched = str.match(/(Welcome) to Rails (.*)/)

matched[1] now becomes “Welcome” rather than underground, and your code brakes. In ruby 1.9 you can name grouped matches:

str = "Welcome to Rails Underground"
matched = str.match(/Welcome to Rails (?<conf>.*)/)
matched[:conf] == "Underground"

What’s missing?

One thing that isn’t missing is 1.8 compatibility, it’s a mostly solved problem. There are some edge cases that wont work, but it’s got the best coverage of all the alternative estimates. On the todo list is better performance, better java integration. To highlight some of the upcoming performance improvements Charles went through the intermediate code the new compiler generates which was quite fascinating to see. Makes me glad I don’t have to build compilers for a living though. As a result there is also a “ruby2java” executable which generates java classes from ruby code now. And to further help with the java and ruby integration there is a “become_java” method you can call on a ruby class to turn it into an object that can be used natively within java.

The other tasks are rubifying the java libraries. Hibernate, Ant, Maven, etc. It’s basically an attempt to keep all of the rich functionality and performance that java offers, without the inherent java ugliness. The big news is that Hibernate is basically done now and just needs to be wrapped up in a nice ruby DSL. But you can use Hibernate as a persistance layer in JRuby now.

How can you help

Use JRuby every chance you get
Help improve JRuby (especially 1.9)
Start evangelising ruby at java conferences
Study Groovy, Scala, Clojure, etc
Study java libraries and help rubify them

Gwyn Morfey (filling in for Laurie Young) - Agile Deployment with Ruby

You’re going to need a lot of servers, so you may as well make the job as easy as possible. You’ve got lost of serving options, you should be using passenger though. There’s now Ruby Intelligent Packaging called RIP which is great and you should use it, except it doesn’t work… so just keep an eye on it. You also need to have continuous integration setup and working. CruiseControl.rb isn’t great but it works.

As far as actually deploying, capistrano is there. But use webistrano, it’s a thin layer over capistrano. The main benefit is you get an audit history so you can see who deployed what, where, and when. Next is puppet to manage your server config. It’s not as quick is making disk images, but it’s more flexible and much easier to maintain (Personally, I’d advocate Chef instead of puppet).

You need multiple servers (either actual dedicated iron for each, or virtualised environments) to deploy to. A test server for external testers and/or clients to run through the completed development prior to a production deployment. A pre-production server that closely mirrors the production server. It’s there as a final sanity check prior to going live, and there to check there isn’t some configuration difference on the production box that will trip you up. Make sure you test both the up and down migrations, it should work, but it often doesn’t. And there is nothing worth than discovering the rollback won’t work once you get to production.

On production, make sure you’ve got notifications setup. “Exception Notifier”, “Get Exceptional”, “HopToad”, “New Relic” test them all and make a call, just use one. Service monitoring, make sure things stay up with monit or god. To help with peak loads consider a Content Delivery Network like Limelight Networks.

You’ll probably need a debug server. If you push things all the way to production and you get errors, it’s quite possibly data related. You need to copy the logs, assets, and all other data over to try and work through. In practice, New Bamboo don’t run a separate debug server but just re-provision pre-production very quickly (it’s there, it’s got the production code already, it’s almost ready to go).

Martin Kleppmann - Sales & payments in your app

The Ruby Invoicing Framework tries to give a solid foundation where you can build a solid web application. It offers a few classes:

Invoice - you owe us money
Credit note - oops you billed to much
Payment -thanks for the cash

They all inherit from LedgerItem which throws is a naming throw back to accounting terms, but it essentially offers you access to discrete billing line items and two companies, one on each side of the transaction. What it gives you for free is an automatically rendered invoice (an actual invoice you can post/email to the client) with all the legally required fields included.

Accountant need to know the exact dates of transactions, that bank statements reconciling, and VAT/sales or other applicable taxes.

insert various slides about accounting practices and how ledger items are calculated. I’d expect anybody who runs their own business already knows all this stuff

I’ve had to roll most of this stuff myself over the past 12-18 months, and for the most part our schemas and approach is almost identical. Just to save myself the maintenance headache, I’ll probably look at porting over to this invoicing gem. I also need to look at the existing open standards (UBL, XBRL-GL, OAccounts, and OASIS) and see how they fit in.

George Palmer - CouchDB and Ruby

I’ve covered the background on what CouchDB is in previous posts, or it’s otherwise readily available with a quick search so I’ll leave you to find the really high level stuff out yourself. Documents are stored as JSON, you get subsets of documents via views. Done.

Unlike relational databases which you pay a performance hit on indexes when you insert data, CouchDB makes you pay the penalty on the first read of a view. That means if you path insert a bunch of data, the next request to see that information is going to be slow. In a typical read heavy web application, this isn’t that noticeable. If you’re inserting data often, then there are ways to mitigate the problem.

So when should you use CouchDB? If you want a schema-less database. FriendFeed was used as an example, given the amount of data they store adding a new column in MySQL could take several hours. Often it also relates more directly to your real-world models. George went through an example from constructing his “5ft Shelf” site and the complexities associated with books (numerous ISBNs, different titles in different countries, difference retail prices in different locales, different editions, etc.) and how difficult it is to map all these permutations in a relational system. The final scenario is when you know you’re going to need replication or sharding, either for offline capability or scaling.

You shouldn’t use it is when your problem domain is very fixed. Finance is a good example, real estate transactions are another.

Enter George’s couch_foo plugin which is a way of interfacing with CouchDB in an ActiveRecord fashion. Because everything is ultimately stored as JSON, there’s no real concept of datatypes. So long as the attributes can be translated to JSON and reconstructed from JSON you’re good to use any datatype/object you type. But the plugin does natively support validations, associations, callbacks, and pretty much everything you’d expect from AR. One gotcha is that if you try and order by an attribute that isn’t exposed by the key, it will return all the results and then order them in ruby. It’s a double sting if you then try and limit, as you’ve pulled back a heap of records you never needed.

Performance wise, George said there’s a whole heap of naive benchmarks claiming CouchDB is faster than this or that. It’s a different approach, some things are going to be quicker, some are going to be slower. The latest CouchDB release (0.9) does offer some speed improvements over previous versions though.

Scotland on Rails 09 - Videos are up

Thu, 21 May 2009 10:41:37 GMT

Not much more to say really, the title sums it up. For those that weren’t there or those that were that just want a recap go and check out the Scotland on rails videos now.

And I will eventually get around to doing that round-up of the post/out-of-conference activities to do should you be looking to attend the conference yourself next year.

Belated Scotland on Rails round up

Fri, 08 May 2009 13:15:08 GMT

Okay, so I’m almost a month and a half late in finishing this. And the content might already seem a little dated given RailsConf is currently on in Vegas and EuRuKo kicks off this weekend in Barcelona. But better late than never.

The Summary

So in summary, how was it? Pretty good and great value. It wasn’t blow my mind amazing, but there was enough good stuff going on to justify the trip. Plus Scotland is a beautiful country, and Edinburgh in-particular a great city, so any means of justifying a business expense to get there is worthwhile ;)

So onto the sessions I saw or chatted in depth to people about.

Sessions

Keynote - Marcel Molina Jr

I’d heard of Marcel before, I’d even seen his name grace the cover of some books I own, but I didn’t know much about him beyond that. For those like me, he was one of the original core team at 37signals that worked on Rails. And his keynote was quite a personal look at the beginnings of the framework. Digging up old usenet posts from DHH showing how he’s trying to learn ruby, to slowly building the hype around rails, to a public release. For a presentation that was essentially reading old emails and newsgroup postings I found it thoroughly enjoyable. It was particularly interesting to see just how quickly DHH went from new to ruby to releasing probably one of the most widely used tools written in the language.

Implementing Geo-awareness - Jim Remsik

I’d had big hopes for this presentation. I’ve dabbled with geo-awareness stuff before, heck I’ve even written my own rails mapping plugin because I thought YM4R and the alternatives weren’t very ruby (btw. it’s in use for the collaborative mapping/direction site I quickly rustled up for my girlfriend Map.ly). I was really looking at some idea re how to make an existing site more “geo aware”, and this presentation was little more than “how to put a google map on your site 101”. For all my complaints about the existing plugins and their approach, they’re still quite simple to use. With any level of ruby/rails knowledge, this talk could have probably been reduced to 5-10mins.

Getting Git: Getting effective project management - Scott Chacon

Scott is part of the team at Logical Awesome working on Github. He’s also done the peepcode on git, and maintains the community book. He really knows his git, and by all accounts I’m disappointed I missed this one for the geo-awareness session. Thankfully the guys I pair with most days, Graham Ashton and Jon Ramsey, went and filled me in somewhat on it. The first half was a bit of an introduction on what it is, why it’s different, how to use it, etc. that could mostly be ignored if you’ve used it at any length. Apparently the second half of the talk (which unfortunately he tried to cram into 10mins) was amazing. Showing how to use bisect to identify which commit introduced a bug as one of the highlights. The slides are available at github and hopefully the videos will be up soon.

Ruby on the couch eating potatoes - Alexander Lang

A well times presentation for my perspective. I’ve been looking at a number of relational database alternatives of late for one of my side projects, and CouchDB had come up as one of the most likely contenders. So I’d done a fair degree of research on CouchDB itself, but not much on the most suitable adapter for using it with rails.

Alex did a good job going through all the various know adapters for talking to CouchDB (including his own, CouchPotato) and pointed out the pros and cons of each. Amazingly for a presentation like this he admitted that you should avoid his. For those interested, the winner was Paul Carey’s RelaxDB. A point of contention which spilled onto the twittersphere was Alex’s recommendation to avoid George Palmer’s CouchFoo (George was in the audience). The rationale being that CouchFoo, much like CouchPotato, is too “ActiveRecord” and that document-based databases are different to relational databases so this means the approach must be inherently wrong.

Personally I think no, the rationale is wrong. Sure, you don’t want to treat CouchDB like a relational database because you’re missing the benefits gained in the document approach and kind of missing the whole point. But that doesn’t mean ActiveRecord for CouchDB is wrong. ActiveRecord does not have to mean Object Relational Mapper, it’s just up until this point it’s all people have used it for. In fact, there is is nothing in the name ActiveRecord that inherently infers it is only for relational schemas. I think if people had grown up being accustomed to AR being an Object Document Mapper instead of an ORM then this argument would be moot. In rails you interface with an object, the adapter takes care of how to store it be-it in a document store or a relational store. We’ve got a pretty well documented API for AR, with lots of validation rules and other plugins that are useful. If you then decide to go build a relational schema in your document database that isn’t the fault of CouchFoo, just like AR can’t be blamed if you decide to try and keep everything in a single big flat table in MySql.

My own conclusion after some research was the two best adapters out there were RelaxDB and CouchFoo. RelaxDB seems the best (I’ll go into details in a future post) but CouchFoo is ideal if you want to transition quickly or need access to some of the AR helpers/mixins. So all in all, a good presentation and made me go back and re-assess my own research on the topic and we ultimately shared the same conclusion albeit with different reasoning.

Merb and Rails 3.0 - Yehuda Katz

Probably not a lot to go through here which wont be out of date due to RailsConf. An interesting and high-paced presentation from Yehuda on what’s coming in rails 3.0, some of the design decisions, etc. Probably not a huge amount of new information for anybody that had been using merb or following the public updates.

In Praise of non-fixtured data - Kevin Barnes

I often wonder how I’d go on stage in front of a bunch of peers, but I’m thankful that many of the presenters at these conferences looks as nervous up there as I’d be. Kevin made me rethink if I should ever bother. A really excellent and clear presenter that was a joy to listen to.

That being said, I didn’t take from this as much as I’d hoped. It’s my own fault as the talk was exactly what was presented on the label, mostly about why you shouldn’t use fixtured data and then a look at what the alternatives are. I was looking for a bit more of a detailed comparison of the pros/cons of each of the object factory approaches (much like Alex had done in the CouchDB talk) but it was really just a cursory mention of them all and then a look at how to use the one Kevin had developed, ObjectDaddy.

Confessions of a PackRat - Scott Raymond

I only caught the tail-end of this talk, but from those I spoke to it was quite insightful. Scott went into detail about how the developed a Facebook app and then needing to scale quickly to deal with the load. Everything from scaling into EC2, switching between technologies, and having to handle people trying to game the system. The 10 mins I saw were thoroughly enjoyable.

Advanced Deployment - Johnathon Weiss

This is one that I felt wasn’t at all what was described on the label. I’d normally consider deployment something more akin to “getting your stuff onto servers”. Instead this covered lots of high-level architectural options you have available to you, the alternatives for each, but without any real explanation on why you should choose one over another. An interesting topic, but pitched to the wrong crowd with the wrong name I feel. Much better for manager types with little-to-no hands on rails experience. I think anybody who’s actually deployed an application would have known the answers to all the theoretical questions raise.

The Ruby Object Model - Dave Thomas

The guy that brought is the Pragmatic Programmers and a number of ruby books and videos from that stable gave a great presentation on the object model with ruby. The shoeless maestro strutted around the stage and made what could be a very dull and overly technical topic seem approachable and simplistic. When you call a method where does it look to first to run it? Where does it bubble up to? In what order? And how do you access an anonymous method? All this arnd more was answered! Not sure if I picked up anything new, but it was a great cementing of my understanding in a highly graphical fashion. 6 weeks on when I’m debugging the internals of something I’m still visualising Dave’s charts in my head to help me. Much of the content you can go through at leisure and in more detail in the $5 videos from pragprog.

R-house - Fernand Galiana

Fernand took the noble risk of trying to do a live demonstration of the possibilities of using home automation with ruby. Thankfully for the most part it worked, and when it didn’t he was entertaining enough to cover it and continue. And I have to appreciate the level of work that has gone into R-house and the libraries it depends on to make this happen, with such a range of different home automation equipment out there and each with there own slight nuances on how to interface it can’t have been a simple job. A great presentation, not sure how soon I’ll be able to use any of it though.

How to makes a successful Rails app - Steven A Bristol

A presentation with no slides?! The horror!! Steven was another excellent presenter, and with a great presentation to boot. He went into what it takes to actually make a successful Rails app (hint: actually releasing it is a big part), and the advice was sound. The other big one which I think many developers ignore, I know I’ve been guilty of it, is trying to do the design yourself. Pay for a few days of their time and make it look professional. He was awfully opinionated on a number of topics without any real basis. That was fine when I agreed with him, I thought he was full of shit when he didn’t justify it ;) One point that niggled at me throughout most of the presentation was that the reason I know of Steven and his company was through the the open-source Lovd by Less social networking framework they’d built in rails. I’d needed something like it once upon a time and looked at Lovd, and the code was horrible. So here I was being lectured by someone on how to make a Rails app who’d produced some of the ugliest code I’d tried to use.

Then the bombshell hit (paraphrasing), “I don’t care what the code looks like if it works, just get it out the door and see if people like it. Anybody who’s seen some of our code will know it’s horrible”. And he was right, on all counts. But… I still had hesitations reservations. This was an open source framework, so the consumers in this instance are developers. So for the same reasons that you should get a designer to make your site look good for regular punters, I think you should take the time to make your code elegant and usable for developers. He’s got a great looking product which does what it needs to, it’s a shame that it’s such a pain to use and made me question the validity of what he was saying. To that end, and rather than continuing to pay other developers to extend my own social networking engine I’ve taken it upon myself to fix everything I think is broken with Lovd. There’s still a long way to go and I’ll make a more formal announcement once it’s nearer completion but it’s already come a long way IMHO. Get it from github.

Who needs Photoshop? Image manipulation in Ruby - Cory Forsyth

Reading through this initially I thought it was going was a bit of a sensational headline for what was going to be an intermediate level look at rmagick. And given my own experience with rmagick, I couldn’t possibly stomach the thought of it ever being considered a viable photoshop replacement. I was wrong.

This was another thoroughly entertaining and informative presentation. Cory went through in detail the logic behind the one feature that has been making me want to upgrade to the latest version of photoshop, content aware scaling, complete with code samples. Next up was facial recognition which could be applied to suggest smart cropping, or apparently put party hats on cats. It’s most likely made my CS4 upgrade redundant, which paid for the trip numerous times over.

Edgecase dialog: Ruby code review - Joe O’Brian and Jim Weirich

It sounded like a boring topic, it should have been a boring topic, and a very odd note on which to end a conference. It turns out Joe and Jim are brilliant entertainers taking the audience through a hypothetical role play of a consultant coming in to refactor an existing rails code base. While it’s unlikely they’ll be invited back to do their return for the Edinburgh Fringe Festival in August, they thoroughly deserved their rapturous applause at the end. I’ve not got any notes unfortunately, I’m not sure if that’s because I was laughing too hard or most of it was common sense. In any event, it was a great reminder that you should step back and look at you code to make sure you aren’t trying to be clever for the sake of it. Simple solutions are often the best.

Seeya next year

It was well worth the trip. The conference was super cheap so it’s really not hard to justify the expense and Edinburgh is a great place to visit. I’ve been there as both a tourist and lived there for 12 months, so I’ll do a subsequent post for those looking to attend next year with my own off-conference highlights.

Capturing a form cancel

Fri, 06 Feb 2009 22:40:02 GMT

I’ve been meaning to write this one up for months now, ever since I originally posted the sexy forms in rails. The form builder I talk about in that post makes it easy to put a submit and cancel button on a the form, and a few people have asked how to handle the cancel action. Some have suggested using javascript to redirect back using an onclick, ick! What happens if the user doesn’t have javascript support? Sure you might try and justify it by saying, “Well, how many users don’t have javascript these days?” but there is absolutely no reason why this has to use javascript, so lets make it work for everybody.

It’s actually much easier than you’d think, within application.rb I’ve got the following:

class ApplicationController < ActionController::Base
  before_filter :catch_cancel, :update => [:create, :update, :destroy]
  after_filter :set_referrer, :only => [:index, :show]

  private
    def set_referrer
      session[:referrer] = url_for(params)
    end

    def catch_cancel
      redirect_to session[:referrer] if params[:commit] == "Cancel"
    end
end

The logic being, that if we’ve got a fully RESTful architecture the only pages we’ll ever want to go “back” will be the index or show actions on each controller. So we cheekily call urlfor in an after filter with the current params to get the path of the page we’ve just display the user, and stash it in the session. Theoretically it may be possible to use ENV[“HTTPREFERER”] instead but it’s not guaranteed, this approach however ensures we keep track of the page to go back to explicitly ourselves.

Then, to piece it all together there’s a before filter checking if the user has clicked the submit button labeled with “Cancel”. You’ll need to change that equality test dependent on what text you put on your button. You may also need to move the before filter into the controllers that require them, depending on the order it needs to run with other filters you have.

Hope that helps.

Ruby Manor Videos Available

Fri, 16 Jan 2009 14:38:44 GMT

For anybody that was unable to attend (like me), or for those that just want a recap then a selection of the Ruby Manor videos are now available for download. Hopefully some more will be posted soon.

For any of the bits that are missing you can read through Graham’s review of the various sessions. Ruby Manor part 1 and Ruby Manor part 2

scRUBYt! Gets Plugins!

Fri, 16 Jan 2009 01:18:45 GMT

Yes! You heard right! As you may have gathered, it’s been a rather frantic month of development for scRUBYt! and currently this is the addition I’m most proud of. One of the most common requests used to be along the lines of “when do you plan to support xxx format output?”. Now, scRUBYt! is oblivious to output formats. That’s right, it natively supports nothing, nada, zilch. But to make it useful, we’ve written a Hash output plugin which we’re shipping it with.

How to use a scRUBYt! output plugin

Firstly, you need to make sure you have the plugin you require installed. At the time of writing there will only be the two I’ve written, Hash and XmlFile. Then in your ruby file require the plugin. As the current edge release isn’t yet packaged as a gem you’ll need to test this with the github checkout and reference the output plugin explicitly:

require "plugins/scrubyt_xml_file_output/scrubyt_xml_file_output"

If you’ve been following the tutorials talking about web scraping with the new version for the past few weeks you’ll have seen how to direct output to a plugin. To request Hash output it’s:

@extractor = Scrubyt::Extractor.new :output => :hash do
    fetch "http://www.google.com/search?&q=ruby"
    result "//html/body/div[5]/div/div/h2/a"
end

and for XmlFile it is:

@file = File.open("results.xml", "w")
@extractor = Scrubyt::Extractor.new :output => :xml_file, :file => @file do
    fetch "http://www.google.com/search?&q=ruby"
    result "//html/body/div[5]/div/div/h2/a"
end

The XmlFile output takes an additional parameter which is the file to stream the results out to.

Creating your own plugin

That’s great for those of you that are happy with XML or Hash output, but what about if you want some other custom format? Well it’s time to create your own. I’ll show you the actual code that implements the XmlFile output to show you how simple it is:

require 'rexml/document'
require "#{File.dirname(__FILE__)}/inflector"
require "#{File.dirname(__FILE__)}/inflections"

class Scrubyt::Output::XmlFile < Scrubyt::Output::Plugin  
  @subscribers = {}
  on_initialize :setup_file
  before_extractor :open_root_node
  after_extractor :close_root_node
  on_save_result :save_xml


  def setup_file(args = {})
    @file = args[:file]
  end

  def open_root_node(*args)
    @file.write("<root>")
  end

  def save_xml(name, results)
    if results.is_a?(::Hash)
      @file.write results.to_xml
    else
      results.each do |result|
        @file.write result.to_xml(name)
      end
    end
  end

  def close_root_node(*args)
    @file.write("</root>")
  end
end

The require lines at the top are only needed for this output format. REXML to construct the XML tags for me, and some inflections I’ve put together to turn the Hash and Array objects into XML. Now into analysing the class proper.

class Scrubyt::Output::XmlFile < Scrubyt::Output::Plugin  
  @subscribers = {}

At the moment, you’ll need to inintialize this instance variable to be an empty Hash for the events to get attached correctly. I’m looking for a way to remove it, stay tuned. But for now you’ll need to put it in.

on_initialize :setup_file
before_extractor :open_root_node
after_extractor :close_root_node
on_save_result :save_xml

Here we’ve got four events to listen for, the concept should be familiar if you’re coming from Rails. Essentially all we are doing is saying “When we initialize run the setupfile method. Before the extractor actually starts, run the method called openrootnode. Whenever we get a result to save, call savexml. And finally, after the extractor run the method called closerootnode.”

def setup_file(args = {})
  @file = args[:file]
end

This is fairly straightforward. If you’ve got any custom logic that needs to happen when the output plugin is initialized you can place it in here. Any parameter that is passed in to Extractor.new() is passed through for you to access here.

def open_root_node(*args)
  @file.write("<root>")
end

Now just to open the XML file, and keep it somewhat consistent with the old scRUBYt! XML output we open a node within the file.

def save_xml(name, results)
  if results.is_a?(::Hash)
    @file.write results.to_xml
  else
    results.each do |result|
      @file.write result.to_xml(name)
    end
  end
end

Here is where the majority of the magic happens. The save_xml method will be passed the desired name for the result, and a hash of the results. This is essentially the same format you’d get if you used the Hash output format, except for each individual detail block rather than then entire extractor.

The reason for the if/else scenario is for when results are not part of a detail block. If you’re just returning results straight (like the Google example at the top of this post) then “results” in this context will be a list/Array of all the matching results rather than a Hash.

Passing results back to the extractor

Not everyone is going to want to stream results out to a file though, so to deal with this you can make a results method available on the instance of your plugin. As I said earlier, even Hash operates as a plugin now so we can see an example of how this work in the Hash output plugin:

class Scrubyt::Output::Hash < Scrubyt::Output::Plugin
  @subscribers = {}
  on_initialize :setup_results
  on_save_result :store_hash

  def setup_results(args = {})
    @results = []
  end

  def results
    @results
  end

  def store_hash(name, passed_results)
    @results << passed_results
  end
end

Here we setup a @results instance oninitialize, and then onsaveresults simply pushes the passedresults into @results. Confused yet? Hopefully the code is clear enough to make sense.

All that happens then is that back in your extractor definition the call to @extractor.results is passed through to the first output plugin it can find.

Naming Conventions and Namespacing

The only additional requirement for a plugin to work in scRUBYt! is that it is correctly named and namespaced. As you may have noticed the ones I’ve provided are called Scrubyt::Output::Hash and Scrubyt::Output::XmlFile, that means they can be targeted using :output => :hash and :output => :xmlfile respectively. If you wanted to call your output GlennsBadExample it would be namespaced as Scrubyt::Output::GlennsBadExample and you’d then just need to require the appropriate file and use :output => :glennsbad_example

Oh the possibilities! So what’s next?

We’re only just starting to see the possibilities that this will offer our extractors. It opens up the possibility of pushing results not only to a different format, but possibly a completely different service. It’s now trivial to create an output format that streams results directly into backgroundRB, a nanite worker, or a web service for further processing and data warehousing. By the time you read this, you’ll also be able to pass in an array of outputs like :output => [:hash, :xml_file] and have both plugins generate the appropriate format(s). For the scraper I’m currently working on where I have two different companies wanting the same data, this could be just the ticket for interfacing directly to their API as I scrape.

I’d love to hear what ideas people might have for this, or how you think it could be improved. We’re really hopeful that this is the kind of thing that makes developing and extending scRUBYt! really easy for those with more complicated needs.

Web Scraping - The Amazon Example Finale

Thu, 15 Jan 2009 20:52:12 GMT

So yesterday I showed you a fairly contrived example of how to build a web scraper using scRUBYt! to get data from Amazon (they’ve got an API that would be much easier and more robust if you need access to this info, but that’s not the point at the moment). But if you look at the results, they’re not the greatest. There is too much noise in some fields, and we probably want to share there data with another system so a ruby based Hash object isn’t going to work.

Removing empty results with scRUBYt!

For various reasons, sometimes you may not get all the data you want back for every record. It’s usually related to your result definition being too restrictive, or a change in format on a specific page. Maybe the price information is in a different DIV if it is on sale. In any event, you need to make a decision on what to do. In the new release of scRUBYt! there are three immediate options that come to mind, but we may well build more in if required. First, and what we did yesterday in the example, is to do nothing. You’ll get the nil/empty result returned back to you to handle as you see fit. Second, you can simply drop any fields that are nil (I’ve just displayed the first few results):

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']", :remove_blank => true
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"}, 
             {:price=>"$26.39"}, 
             {:saving=>"$13.60\n      (34%)\n    "}, 
             {:isbn=>"ISBN-10: 0596516177"}]}
    {:book=>[{:title=>"The Ruby Programming Language (Paperback)"}, 
             {:saving=>nil}, 
             {:isbn=>"ISBN-10: 020171096X"}]}, 
    {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"}, 
             {:price=>"$26.39"}, 
             {:saving=>"$13.60\n      (34%)\n    "}, 
             {:isbn=>"ISBN-10: 1590597664"}]}
    ...]

We’ve set :remove_blank to true on the price field, and as a result you’ll see that the 2nd result contains no price element. Alternatively, you could drop any detail block (in this example, any single book) which is missing the field:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']", :required => true
    price "//b[@class='priceLarge']", :required => true
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 0596516177"}]},
    {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 1590597664"}]},
    {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"},
             {:price=>"$29.67"},
             {:saving=>"$15.28\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 0974514055"}]},
    ...]

This time the “The Ruby Programming Language (Paperback)” book isn’t included in the results at all. For our purposes though, I’ve decided that I only want to know about books that I have all the details for. Instead of setting :required on every result, I can specify it on the book_detail definition which will give the same output:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end

Sanitizing scRUBYt! output

The output still isn’t quite what we want. We’ve dumped the empty results, but we’ve still got that ugly “ISBN-10: ” in front of the ISBN. We could clean it up later, but it’s creating additional work for ourselves. And if we want this thing to scale (I’ve got scrapers which scrape thousands of pages off a single site) trying to keep all that data hanging around in memory isn’t going to work. So let’s do as much of possible within the scRUBYt! definition as we’re collecting the data:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ", "")}
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"0596516177"}]},
    ... ]

And now you’ll see that we’re getting a much cleaner ISBN result. Just create a Proc, the result will be passed into in and then do as you see fit. An if statement to check it contains something you expect, a regexp, the possibilities are endless. You can also combine this with the other options like :require and :remove_blank. So let’s really jazz this thing up. We’ll clean up the saving, and pull in the description and that’s all the data we need:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all  do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']", :script => Proc.new{|saving| saving.match(/(\$[\d\.]*)/)[1]}
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ","")}
    description "//div[@id='productDescription']//div[@class='content']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60"},
             {:isbn=>"1590597664"}, 
             {:description=>"Product Description\n  Ruby is perhaps best known as the engine powering the..."}]}, 
    {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"}, 
             {:price=>"$29.67"}, 
             {:saving=>"$15.28"}, 
             {:isbn=>"0974514055"}, 
             {:description=>"Product Description\n  Ruby is an increasingly popular, fully object-oriented dynamic..."}]},
    ...]

Outputting results to XML

Inevitably there comes a time where you want to consume this data in something other than your ruby application. At that point, passing around a Hash is probably not the best idea. Alternatively, you might have a scraper that has to scrape hundreds to thousands of pages. Storing all the results in a Hash as you go will bring your machine to it’s knees. So here comes one of the largest changes to the way the new release of scRUBYt! works.

Previously, you always had results returned as a Hash and/or XML depending on your need. Everything was held in memory until you destroyed your extractor. Now, the standard XML option is to stream the results out to a file as they are processed and remove them from memory. There is no way to retrieve the results as XML within your program, they have to be streamed out to a file (and really, why would you want XML within your app when you can have native ruby structures instead?). So to save our scraper above out to an XML file you just pass a new output format and an instance of a File to the extractor:

@file = File.new("amazon_results.xml", "w")
@extractor = Scrubyt::Extractor.new :output => :xml_file, :file => @file do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all  do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']", :script => Proc.new{|saving| saving.match(/(\$[\d\.]*)/)[1]}
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ","")}
    description "//div[@id='productDescription']//div[@class='content']"
  end
end

Now a call to @extractor.results at the end of the scrape will return no results. I hope that wasn’t too much and it’s given you a good view into how to create your very own web scraper. If you have any questions, head on over to the scRUBYt! forums or post them in the comments.

More web scrapers with the upcoming scRUBYt!

Tue, 13 Jan 2009 01:20:13 GMT

In the previous articles I gave a brief glimpse at the upcoming scRUBYt! release. We learned how to do some basic html scraping, and then followed it up by scraping multiple pages. As promised, in this installment I’ll go over how to get more detailed logging output to help you debug your scraper while in development and how to submit and navigate forms.

Logging Scraper Output

There’s been quite a fundamental change to the way scRUBYt! works internally of late. It doesn’t manifest itself visually in the way you interface to it, but it does mean logging the output is now much easier and cleaner. Using our most basic example from the first tutorial, you just need to pass the :log_level into the extractor:

@extractor = Scrubyt::Extractor.new(:log_level => :verbose) do
  fetch "http://www.google.com/ncr"
end

By default, the output is directed to stdout so you’d see the following on your screen:

start
fetch: http://www.google.com/ncr
end

If you’re after a more complex example, here is the scraper definition from the second part of the tutorial series and the corresponding log output:

@extractor = Scrubyt::Extractor.new(:log_level => :verbose) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    title "//title"
    summary "//p", :script => Proc.new{|result| result if result.match(%r{(\w+\W+){25}})}
  end
  next_page "//a[text()*='Next']", :limit => 2
end

start
fetch: http://www.google.com/ncr
textfield: 'q' = 'ruby'
with options ''
submit
next detail: 'page' = 'http://ruby-lang.org/'
with args: ''
next detail: 'page' = 'http://en.wikipedia.org/wiki/Ruby_(programming_language)'
with args: ''

etc...

next page: /search?hl=en&ie=UTF-8&q=ruby&start=10&sa=N
fetch: http://www.google.com/search?hl=en&ie=UTF-8&q=ruby&start=10&sa=N
next detail: 'page' = 'http://www.rubycentral.com/book/'
with args: ''

etc...

end

At present the valid :log_level values are :none, :critical, :error, :warn, :info, :debug, and :verbose (in increasing order of noise). If you want to direct the log output to something other than stdout the only way at the moment is to override the Scrubyt::Logger#log method. I’m looking at ways to make it easier to substitute in a file based or other logging approach.

Making Your Scraper Navigate Forms

So now that we know how to log the output, let’s do something more useful with our scraper. We can’t really take our Google example from previous posts any further given how simple their interface is, so let us move over to Amazon. Say I wanted to grab a list the books on ruby that are for sale. Sure, I could probably get this information via an Amazon API… but that’s not really the point now is it ;)

@extractor = Scrubyt::Extractor.new(:log => :debug) do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end

So let’s run through what we’ve got here, hopefully some of it looks familiar from the previous examples we’ve gone through. First we fetch the page to start with, then in the select field named “url” we choose the options that says “Books”, and we then submit the form. scRUBYt! will keep track of the last form you input any data to, so if there are multiple forms on the page then you just need to target the appropriate input fields. From there, the submit action will work out what it needs to do.

Next is to define a detail block, so we point out the XPath to the heading/link for each book on the page and say we want to navigate to that page and extract the title, price, saving, etc. I’ve been a little cheeky with the isbn definition, saying just find me any LI tag that contains the string “ISBN-10:”. If we were to look at the results generated you’d see:

puts @extractor.results.inspect
[{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"}, {:price=>"$26.39"}, 
          {:saving=>"$13.60\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0596516177"}]}, 
 {:book=>[{:title=>"The Ruby Programming Language (Paperback)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>"ISBN-10: 020171096X"}]}, 
 {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"}, 
          {:price=>"$26.39"}, 
          {:saving=>"$13.60\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 1590597664"}]}, 
 {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Kindle Edition)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>nil}]}, 
 {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"}, 
          {:price=>"$29.67"}, 
          {:saving=>"$15.28\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0974514055"}]}, 
 {:book=>[{:title=>"The Ruby Way, Second Edition: Solutions and Techniques in Ruby Programming (2nd Edition) (Addison-Wesley Professional Ruby Series) (Paperback)"}, 
          {:price=>"$29.69"}, 
          {:saving=>"$15.30\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0672328844"}]}, 
 {:book=>[{:title=>"Ruby Way, The: Solutions and Techniques in Ruby Programming (Kindle Edition)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>nil}]}, 
 {:book=>[{:title=>"Learning Ruby [ILLUSTRATED]  (Paperback)"}, 
          {:price=>"$23.09"}, 
          {:saving=>"$11.90\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0596529864"}]}
]

And we have a reasonable snapshot of the data. You’ll see there though, that it’s not perfect. Firstly, we are missing information for some results. We’ve got spaces and carriage returns in the saving data, and we probably don’t need the “ISBN-10:” string at the front of the ISBN result. And what if we wanted to link to the actual result so someone could actually buy the book on Amazon?

All good questions, and all easily solvable. I’ll follow it up with a post in the next day or so and highlight some of the new ways of specifying constraints on your data in scRUBYt!. And special thanks have to go to Homeflow who have been funding at least a day of time for Peter and I each week lately, hence the increased level of development in scRUBYt!. It’s nice when you have clients who want to actively give back.