Ruby Pond

Scotland on Rails 09 - Videos are up

Thu, 21 May 2009 10:41:37 GMT

Not much more to say really, the title sums it up. For those that weren’t there or those that were that just want a recap go and check out the Scotland on rails videos now.

And I will eventually get around to doing that round-up of the post/out-of-conference activities to do should you be looking to attend the conference yourself next year.

Belated Scotland on Rails round up

Fri, 08 May 2009 13:15:08 GMT

Okay, so I’m almost a month and a half late in finishing this. And the content might already seem a little dated given RailsConf is currently on in Vegas and EuRuKo kicks off this weekend in Barcelona. But better late than never.

The Summary

So in summary, how was it? Pretty good and great value. It wasn’t blow my mind amazing, but there was enough good stuff going on to justify the trip. Plus Scotland is a beautiful country, and Edinburgh in-particular a great city, so any means of justifying a business expense to get there is worthwhile ;)

So onto the sessions I saw or chatted in depth to people about.

Sessions

Keynote - Marcel Molina Jr

I’d heard of Marcel before, I’d even seen his name grace the cover of some books I own, but I didn’t know much about him beyond that. For those like me, he was one of the original core team at 37signals that worked on Rails. And his keynote was quite a personal look at the beginnings of the framework. Digging up old usenet posts from DHH showing how he’s trying to learn ruby, to slowly building the hype around rails, to a public release. For a presentation that was essentially reading old emails and newsgroup postings I found it thoroughly enjoyable. It was particularly interesting to see just how quickly DHH went from new to ruby to releasing probably one of the most widely used tools written in the language.

Implementing Geo-awareness - Jim Remsik

I’d had big hopes for this presentation. I’ve dabbled with geo-awareness stuff before, heck I’ve even written my own rails mapping plugin because I thought YM4R and the alternatives weren’t very ruby (btw. it’s in use for the collaborative mapping/direction site I quickly rustled up for my girlfriend Map.ly). I was really looking at some idea re how to make an existing site more “geo aware”, and this presentation was little more than “how to put a google map on your site 101”. For all my complaints about the existing plugins and their approach, they’re still quite simple to use. With any level of ruby/rails knowledge, this talk could have probably been reduced to 5-10mins.

Getting Git: Getting effective project management - Scott Chacon

Scott is part of the team at Logical Awesome working on Github. He’s also done the peepcode on git, and maintains the community book. He really knows his git, and by all accounts I’m disappointed I missed this one for the geo-awareness session. Thankfully the guys I pair with most days, Graham Ashton and Jon Ramsey, went and filled me in somewhat on it. The first half was a bit of an introduction on what it is, why it’s different, how to use it, etc. that could mostly be ignored if you’ve used it at any length. Apparently the second half of the talk (which unfortunately he tried to cram into 10mins) was amazing. Showing how to use bisect to identify which commit introduced a bug as one of the highlights. The slides are available at github and hopefully the videos will be up soon.

Ruby on the couch eating potatoes - Alexander Lang

A well times presentation for my perspective. I’ve been looking at a number of relational database alternatives of late for one of my side projects, and CouchDB had come up as one of the most likely contenders. So I’d done a fair degree of research on CouchDB itself, but not much on the most suitable adapter for using it with rails.

Alex did a good job going through all the various know adapters for talking to CouchDB (including his own, CouchPotato) and pointed out the pros and cons of each. Amazingly for a presentation like this he admitted that you should avoid his. For those interested, the winner was Paul Carey’s RelaxDB. A point of contention which spilled onto the twittersphere was Alex’s recommendation to avoid George Palmer’s CouchFoo (George was in the audience). The rationale being that CouchFoo, much like CouchPotato, is too “ActiveRecord” and that document-based databases are different to relational databases so this means the approach must be inherently wrong.

Personally I think no, the rationale is wrong. Sure, you don’t want to treat CouchDB like a relational database because you’re missing the benefits gained in the document approach and kind of missing the whole point. But that doesn’t mean ActiveRecord for CouchDB is wrong. ActiveRecord does not have to mean Object Relational Mapper, it’s just up until this point it’s all people have used it for. In fact, there is is nothing in the name ActiveRecord that inherently infers it is only for relational schemas. I think if people had grown up being accustomed to AR being an Object Document Mapper instead of an ORM then this argument would be moot. In rails you interface with an object, the adapter takes care of how to store it be-it in a document store or a relational store. We’ve got a pretty well documented API for AR, with lots of validation rules and other plugins that are useful. If you then decide to go build a relational schema in your document database that isn’t the fault of CouchFoo, just like AR can’t be blamed if you decide to try and keep everything in a single big flat table in MySql.

My own conclusion after some research was the two best adapters out there were RelaxDB and CouchFoo. RelaxDB seems the best (I’ll go into details in a future post) but CouchFoo is ideal if you want to transition quickly or need access to some of the AR helpers/mixins. So all in all, a good presentation and made me go back and re-assess my own research on the topic and we ultimately shared the same conclusion albeit with different reasoning.

Merb and Rails 3.0 - Yehuda Katz

Probably not a lot to go through here which wont be out of date due to RailsConf. An interesting and high-paced presentation from Yehuda on what’s coming in rails 3.0, some of the design decisions, etc. Probably not a huge amount of new information for anybody that had been using merb or following the public updates.

In Praise of non-fixtured data - Kevin Barnes

I often wonder how I’d go on stage in front of a bunch of peers, but I’m thankful that many of the presenters at these conferences looks as nervous up there as I’d be. Kevin made me rethink if I should ever bother. A really excellent and clear presenter that was a joy to listen to.

That being said, I didn’t take from this as much as I’d hoped. It’s my own fault as the talk was exactly what was presented on the label, mostly about why you shouldn’t use fixtured data and then a look at what the alternatives are. I was looking for a bit more of a detailed comparison of the pros/cons of each of the object factory approaches (much like Alex had done in the CouchDB talk) but it was really just a cursory mention of them all and then a look at how to use the one Kevin had developed, ObjectDaddy.

Confessions of a PackRat - Scott Raymond

I only caught the tail-end of this talk, but from those I spoke to it was quite insightful. Scott went into detail about how the developed a Facebook app and then needing to scale quickly to deal with the load. Everything from scaling into EC2, switching between technologies, and having to handle people trying to game the system. The 10 mins I saw were thoroughly enjoyable.

Advanced Deployment - Johnathon Weiss

This is one that I felt wasn’t at all what was described on the label. I’d normally consider deployment something more akin to “getting your stuff onto servers”. Instead this covered lots of high-level architectural options you have available to you, the alternatives for each, but without any real explanation on why you should choose one over another. An interesting topic, but pitched to the wrong crowd with the wrong name I feel. Much better for manager types with little-to-no hands on rails experience. I think anybody who’s actually deployed an application would have known the answers to all the theoretical questions raise.

The Ruby Object Model - Dave Thomas

The guy that brought is the Pragmatic Programmers and a number of ruby books and videos from that stable gave a great presentation on the object model with ruby. The shoeless maestro strutted around the stage and made what could be a very dull and overly technical topic seem approachable and simplistic. When you call a method where does it look to first to run it? Where does it bubble up to? In what order? And how do you access an anonymous method? All this arnd more was answered! Not sure if I picked up anything new, but it was a great cementing of my understanding in a highly graphical fashion. 6 weeks on when I’m debugging the internals of something I’m still visualising Dave’s charts in my head to help me. Much of the content you can go through at leisure and in more detail in the $5 videos from pragprog.

R-house - Fernand Galiana

Fernand took the noble risk of trying to do a live demonstration of the possibilities of using home automation with ruby. Thankfully for the most part it worked, and when it didn’t he was entertaining enough to cover it and continue. And I have to appreciate the level of work that has gone into R-house and the libraries it depends on to make this happen, with such a range of different home automation equipment out there and each with there own slight nuances on how to interface it can’t have been a simple job. A great presentation, not sure how soon I’ll be able to use any of it though.

How to makes a successful Rails app - Steven A Bristol

A presentation with no slides?! The horror!! Steven was another excellent presenter, and with a great presentation to boot. He went into what it takes to actually make a successful Rails app (hint: actually releasing it is a big part), and the advice was sound. The other big one which I think many developers ignore, I know I’ve been guilty of it, is trying to do the design yourself. Pay for a few days of their time and make it look professional. He was awfully opinionated on a number of topics without any real basis. That was fine when I agreed with him, I thought he was full of shit when he didn’t justify it ;) One point that niggled at me throughout most of the presentation was that the reason I know of Steven and his company was through the the open-source Lovd by Less social networking framework they’d built in rails. I’d needed something like it once upon a time and looked at Lovd, and the code was horrible. So here I was being lectured by someone on how to make a Rails app who’d produced some of the ugliest code I’d tried to use.

Then the bombshell hit (paraphrasing), “I don’t care what the code looks like if it works, just get it out the door and see if people like it. Anybody who’s seen some of our code will know it’s horrible”. And he was right, on all counts. But… I still had hesitations reservations. This was an open source framework, so the consumers in this instance are developers. So for the same reasons that you should get a designer to make your site look good for regular punters, I think you should take the time to make your code elegant and usable for developers. He’s got a great looking product which does what it needs to, it’s a shame that it’s such a pain to use and made me question the validity of what he was saying. To that end, and rather than continuing to pay other developers to extend my own social networking engine I’ve taken it upon myself to fix everything I think is broken with Lovd. There’s still a long way to go and I’ll make a more formal announcement once it’s nearer completion but it’s already come a long way IMHO. Get it from github.

Who needs Photoshop? Image manipulation in Ruby - Cory Forsyth

Reading through this initially I thought it was going was a bit of a sensational headline for what was going to be an intermediate level look at rmagick. And given my own experience with rmagick, I couldn’t possibly stomach the thought of it ever being considered a viable photoshop replacement. I was wrong.

This was another thoroughly entertaining and informative presentation. Cory went through in detail the logic behind the one feature that has been making me want to upgrade to the latest version of photoshop, content aware scaling, complete with code samples. Next up was facial recognition which could be applied to suggest smart cropping, or apparently put party hats on cats. It’s most likely made my CS4 upgrade redundant, which paid for the trip numerous times over.

Edgecase dialog: Ruby code review - Joe O’Brian and Jim Weirich

It sounded like a boring topic, it should have been a boring topic, and a very odd note on which to end a conference. It turns out Joe and Jim are brilliant entertainers taking the audience through a hypothetical role play of a consultant coming in to refactor an existing rails code base. While it’s unlikely they’ll be invited back to do their return for the Edinburgh Fringe Festival in August, they thoroughly deserved their rapturous applause at the end. I’ve not got any notes unfortunately, I’m not sure if that’s because I was laughing too hard or most of it was common sense. In any event, it was a great reminder that you should step back and look at you code to make sure you aren’t trying to be clever for the sake of it. Simple solutions are often the best.

Seeya next year

It was well worth the trip. The conference was super cheap so it’s really not hard to justify the expense and Edinburgh is a great place to visit. I’ve been there as both a tourist and lived there for 12 months, so I’ll do a subsequent post for those looking to attend next year with my own off-conference highlights.

Capturing a form cancel

Fri, 06 Feb 2009 22:40:02 GMT

I’ve been meaning to write this one up for months now, ever since I originally posted the sexy forms in rails. The form builder I talk about in that post makes it easy to put a submit and cancel button on a the form, and a few people have asked how to handle the cancel action. Some have suggested using javascript to redirect back using an onclick, ick! What happens if the user doesn’t have javascript support? Sure you might try and justify it by saying, “Well, how many users don’t have javascript these days?” but there is absolutely no reason why this has to use javascript, so lets make it work for everybody.

It’s actually much easier than you’d think, within application.rb I’ve got the following:

class ApplicationController < ActionController::Base
  before_filter :catch_cancel, :update => [:create, :update, :destroy]
  after_filter :set_referrer, :only => [:index, :show]

  private
    def set_referrer
      session[:referrer] = url_for(params)
    end

    def catch_cancel
      redirect_to session[:referrer] if params[:commit] == "Cancel"
    end
end

The logic being, that if we’ve got a fully RESTful architecture the only pages we’ll ever want to go “back” will be the index or show actions on each controller. So we cheekily call urlfor in an after filter with the current params to get the path of the page we’ve just display the user, and stash it in the session. Theoretically it may be possible to use ENV[“HTTPREFERER”] instead but it’s not guaranteed, this approach however ensures we keep track of the page to go back to explicitly ourselves.

Then, to piece it all together there’s a before filter checking if the user has clicked the submit button labeled with “Cancel”. You’ll need to change that equality test dependent on what text you put on your button. You may also need to move the before filter into the controllers that require them, depending on the order it needs to run with other filters you have.

Hope that helps.

Ruby Manor Videos Available

Fri, 16 Jan 2009 14:38:44 GMT

For anybody that was unable to attend (like me), or for those that just want a recap then a selection of the Ruby Manor videos are now available for download. Hopefully some more will be posted soon.

For any of the bits that are missing you can read through Graham’s review of the various sessions. Ruby Manor part 1 and Ruby Manor part 2

scRUBYt! Gets Plugins!

Fri, 16 Jan 2009 01:18:45 GMT

Yes! You heard right! As you may have gathered, it’s been a rather frantic month of development for scRUBYt! and currently this is the addition I’m most proud of. One of the most common requests used to be along the lines of “when do you plan to support xxx format output?”. Now, scRUBYt! is oblivious to output formats. That’s right, it natively supports nothing, nada, zilch. But to make it useful, we’ve written a Hash output plugin which we’re shipping it with.

How to use a scRUBYt! output plugin

Firstly, you need to make sure you have the plugin you require installed. At the time of writing there will only be the two I’ve written, Hash and XmlFile. Then in your ruby file require the plugin. As the current edge release isn’t yet packaged as a gem you’ll need to test this with the github checkout and reference the output plugin explicitly:

require "plugins/scrubyt_xml_file_output/scrubyt_xml_file_output"

If you’ve been following the tutorials talking about web scraping with the new version for the past few weeks you’ll have seen how to direct output to a plugin. To request Hash output it’s:

@extractor = Scrubyt::Extractor.new :output => :hash do
    fetch "http://www.google.com/search?&q=ruby"
    result "//html/body/div[5]/div/div/h2/a"
end

and for XmlFile it is:

@file = File.open("results.xml", "w")
@extractor = Scrubyt::Extractor.new :output => :xml_file, :file => @file do
    fetch "http://www.google.com/search?&q=ruby"
    result "//html/body/div[5]/div/div/h2/a"
end

The XmlFile output takes an additional parameter which is the file to stream the results out to.

Creating your own plugin

That’s great for those of you that are happy with XML or Hash output, but what about if you want some other custom format? Well it’s time to create your own. I’ll show you the actual code that implements the XmlFile output to show you how simple it is:

require 'rexml/document'
require "#{File.dirname(__FILE__)}/inflector"
require "#{File.dirname(__FILE__)}/inflections"

class Scrubyt::Output::XmlFile < Scrubyt::Output::Plugin  
  @subscribers = {}
  on_initialize :setup_file
  before_extractor :open_root_node
  after_extractor :close_root_node
  on_save_result :save_xml


  def setup_file(args = {})
    @file = args[:file]
  end

  def open_root_node(*args)
    @file.write("<root>")
  end

  def save_xml(name, results)
    if results.is_a?(::Hash)
      @file.write results.to_xml
    else
      results.each do |result|
        @file.write result.to_xml(name)
      end
    end
  end

  def close_root_node(*args)
    @file.write("</root>")
  end
end

The require lines at the top are only needed for this output format. REXML to construct the XML tags for me, and some inflections I’ve put together to turn the Hash and Array objects into XML. Now into analysing the class proper.

class Scrubyt::Output::XmlFile < Scrubyt::Output::Plugin  
  @subscribers = {}

At the moment, you’ll need to inintialize this instance variable to be an empty Hash for the events to get attached correctly. I’m looking for a way to remove it, stay tuned. But for now you’ll need to put it in.

on_initialize :setup_file
before_extractor :open_root_node
after_extractor :close_root_node
on_save_result :save_xml

Here we’ve got four events to listen for, the concept should be familiar if you’re coming from Rails. Essentially all we are doing is saying “When we initialize run the setupfile method. Before the extractor actually starts, run the method called openrootnode. Whenever we get a result to save, call savexml. And finally, after the extractor run the method called closerootnode.”

def setup_file(args = {})
  @file = args[:file]
end

This is fairly straightforward. If you’ve got any custom logic that needs to happen when the output plugin is initialized you can place it in here. Any parameter that is passed in to Extractor.new() is passed through for you to access here.

def open_root_node(*args)
  @file.write("<root>")
end

Now just to open the XML file, and keep it somewhat consistent with the old scRUBYt! XML output we open a node within the file.

def save_xml(name, results)
  if results.is_a?(::Hash)
    @file.write results.to_xml
  else
    results.each do |result|
      @file.write result.to_xml(name)
    end
  end
end

Here is where the majority of the magic happens. The save_xml method will be passed the desired name for the result, and a hash of the results. This is essentially the same format you’d get if you used the Hash output format, except for each individual detail block rather than then entire extractor.

The reason for the if/else scenario is for when results are not part of a detail block. If you’re just returning results straight (like the Google example at the top of this post) then “results” in this context will be a list/Array of all the matching results rather than a Hash.

Passing results back to the extractor

Not everyone is going to want to stream results out to a file though, so to deal with this you can make a results method available on the instance of your plugin. As I said earlier, even Hash operates as a plugin now so we can see an example of how this work in the Hash output plugin:

class Scrubyt::Output::Hash < Scrubyt::Output::Plugin
  @subscribers = {}
  on_initialize :setup_results
  on_save_result :store_hash

  def setup_results(args = {})
    @results = []
  end

  def results
    @results
  end

  def store_hash(name, passed_results)
    @results << passed_results
  end
end

Here we setup a @results instance oninitialize, and then onsaveresults simply pushes the passedresults into @results. Confused yet? Hopefully the code is clear enough to make sense.

All that happens then is that back in your extractor definition the call to @extractor.results is passed through to the first output plugin it can find.

Naming Conventions and Namespacing

The only additional requirement for a plugin to work in scRUBYt! is that it is correctly named and namespaced. As you may have noticed the ones I’ve provided are called Scrubyt::Output::Hash and Scrubyt::Output::XmlFile, that means they can be targeted using :output => :hash and :output => :xmlfile respectively. If you wanted to call your output GlennsBadExample it would be namespaced as Scrubyt::Output::GlennsBadExample and you’d then just need to require the appropriate file and use :output => :glennsbad_example

Oh the possibilities! So what’s next?

We’re only just starting to see the possibilities that this will offer our extractors. It opens up the possibility of pushing results not only to a different format, but possibly a completely different service. It’s now trivial to create an output format that streams results directly into backgroundRB, a nanite worker, or a web service for further processing and data warehousing. By the time you read this, you’ll also be able to pass in an array of outputs like :output => [:hash, :xml_file] and have both plugins generate the appropriate format(s). For the scraper I’m currently working on where I have two different companies wanting the same data, this could be just the ticket for interfacing directly to their API as I scrape.

I’d love to hear what ideas people might have for this, or how you think it could be improved. We’re really hopeful that this is the kind of thing that makes developing and extending scRUBYt! really easy for those with more complicated needs.

Web Scraping - The Amazon Example Finale

Thu, 15 Jan 2009 20:52:12 GMT

So yesterday I showed you a fairly contrived example of how to build a web scraper using scRUBYt! to get data from Amazon (they’ve got an API that would be much easier and more robust if you need access to this info, but that’s not the point at the moment). But if you look at the results, they’re not the greatest. There is too much noise in some fields, and we probably want to share there data with another system so a ruby based Hash object isn’t going to work.

Removing empty results with scRUBYt!

For various reasons, sometimes you may not get all the data you want back for every record. It’s usually related to your result definition being too restrictive, or a change in format on a specific page. Maybe the price information is in a different DIV if it is on sale. In any event, you need to make a decision on what to do. In the new release of scRUBYt! there are three immediate options that come to mind, but we may well build more in if required. First, and what we did yesterday in the example, is to do nothing. You’ll get the nil/empty result returned back to you to handle as you see fit. Second, you can simply drop any fields that are nil (I’ve just displayed the first few results):

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']", :remove_blank => true
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"}, 
             {:price=>"$26.39"}, 
             {:saving=>"$13.60\n      (34%)\n    "}, 
             {:isbn=>"ISBN-10: 0596516177"}]}
    {:book=>[{:title=>"The Ruby Programming Language (Paperback)"}, 
             {:saving=>nil}, 
             {:isbn=>"ISBN-10: 020171096X"}]}, 
    {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"}, 
             {:price=>"$26.39"}, 
             {:saving=>"$13.60\n      (34%)\n    "}, 
             {:isbn=>"ISBN-10: 1590597664"}]}
    ...]

We’ve set :remove_blank to true on the price field, and as a result you’ll see that the 2nd result contains no price element. Alternatively, you could drop any detail block (in this example, any single book) which is missing the field:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']", :required => true
    price "//b[@class='priceLarge']", :required => true
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 0596516177"}]},
    {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 1590597664"}]},
    {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"},
             {:price=>"$29.67"},
             {:saving=>"$15.28\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 0974514055"}]},
    ...]

This time the “The Ruby Programming Language (Paperback)” book isn’t included in the results at all. For our purposes though, I’ve decided that I only want to know about books that I have all the details for. Instead of setting :required on every result, I can specify it on the book_detail definition which will give the same output:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end

Sanitizing scRUBYt! output

The output still isn’t quite what we want. We’ve dumped the empty results, but we’ve still got that ugly “ISBN-10: ” in front of the ISBN. We could clean it up later, but it’s creating additional work for ourselves. And if we want this thing to scale (I’ve got scrapers which scrape thousands of pages off a single site) trying to keep all that data hanging around in memory isn’t going to work. So let’s do as much of possible within the scRUBYt! definition as we’re collecting the data:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ", "")}
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"0596516177"}]},
    ... ]

And now you’ll see that we’re getting a much cleaner ISBN result. Just create a Proc, the result will be passed into in and then do as you see fit. An if statement to check it contains something you expect, a regexp, the possibilities are endless. You can also combine this with the other options like :require and :remove_blank. So let’s really jazz this thing up. We’ll clean up the saving, and pull in the description and that’s all the data we need:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all  do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']", :script => Proc.new{|saving| saving.match(/(\$[\d\.]*)/)[1]}
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ","")}
    description "//div[@id='productDescription']//div[@class='content']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60"},
             {:isbn=>"1590597664"}, 
             {:description=>"Product Description\n  Ruby is perhaps best known as the engine powering the..."}]}, 
    {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"}, 
             {:price=>"$29.67"}, 
             {:saving=>"$15.28"}, 
             {:isbn=>"0974514055"}, 
             {:description=>"Product Description\n  Ruby is an increasingly popular, fully object-oriented dynamic..."}]},
    ...]

Outputting results to XML

Inevitably there comes a time where you want to consume this data in something other than your ruby application. At that point, passing around a Hash is probably not the best idea. Alternatively, you might have a scraper that has to scrape hundreds to thousands of pages. Storing all the results in a Hash as you go will bring your machine to it’s knees. So here comes one of the largest changes to the way the new release of scRUBYt! works.

Previously, you always had results returned as a Hash and/or XML depending on your need. Everything was held in memory until you destroyed your extractor. Now, the standard XML option is to stream the results out to a file as they are processed and remove them from memory. There is no way to retrieve the results as XML within your program, they have to be streamed out to a file (and really, why would you want XML within your app when you can have native ruby structures instead?). So to save our scraper above out to an XML file you just pass a new output format and an instance of a File to the extractor:

@file = File.new("amazon_results.xml", "w")
@extractor = Scrubyt::Extractor.new :output => :xml_file, :file => @file do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all  do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']", :script => Proc.new{|saving| saving.match(/(\$[\d\.]*)/)[1]}
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ","")}
    description "//div[@id='productDescription']//div[@class='content']"
  end
end

Now a call to @extractor.results at the end of the scrape will return no results. I hope that wasn’t too much and it’s given you a good view into how to create your very own web scraper. If you have any questions, head on over to the scRUBYt! forums or post them in the comments.

More web scrapers with the upcoming scRUBYt!

Tue, 13 Jan 2009 01:20:13 GMT

In the previous articles I gave a brief glimpse at the upcoming scRUBYt! release. We learned how to do some basic html scraping, and then followed it up by scraping multiple pages. As promised, in this installment I’ll go over how to get more detailed logging output to help you debug your scraper while in development and how to submit and navigate forms.

Logging Scraper Output

There’s been quite a fundamental change to the way scRUBYt! works internally of late. It doesn’t manifest itself visually in the way you interface to it, but it does mean logging the output is now much easier and cleaner. Using our most basic example from the first tutorial, you just need to pass the :log_level into the extractor:

@extractor = Scrubyt::Extractor.new(:log_level => :verbose) do
  fetch "http://www.google.com/ncr"
end

By default, the output is directed to stdout so you’d see the following on your screen:

start
fetch: http://www.google.com/ncr
end

If you’re after a more complex example, here is the scraper definition from the second part of the tutorial series and the corresponding log output:

@extractor = Scrubyt::Extractor.new(:log_level => :verbose) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    title "//title"
    summary "//p", :script => Proc.new{|result| result if result.match(%r{(\w+\W+){25}})}
  end
  next_page "//a[text()*='Next']", :limit => 2
end

start
fetch: http://www.google.com/ncr
textfield: 'q' = 'ruby'
with options ''
submit
next detail: 'page' = 'http://ruby-lang.org/'
with args: ''
next detail: 'page' = 'http://en.wikipedia.org/wiki/Ruby_(programming_language)'
with args: ''

etc...

next page: /search?hl=en&ie=UTF-8&q=ruby&start=10&sa=N
fetch: http://www.google.com/search?hl=en&ie=UTF-8&q=ruby&start=10&sa=N
next detail: 'page' = 'http://www.rubycentral.com/book/'
with args: ''

etc...

end

At present the valid :log_level values are :none, :critical, :error, :warn, :info, :debug, and :verbose (in increasing order of noise). If you want to direct the log output to something other than stdout the only way at the moment is to override the Scrubyt::Logger#log method. I’m looking at ways to make it easier to substitute in a file based or other logging approach.

Making Your Scraper Navigate Forms

So now that we know how to log the output, let’s do something more useful with our scraper. We can’t really take our Google example from previous posts any further given how simple their interface is, so let us move over to Amazon. Say I wanted to grab a list the books on ruby that are for sale. Sure, I could probably get this information via an Amazon API… but that’s not really the point now is it ;)

@extractor = Scrubyt::Extractor.new(:log => :debug) do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end

So let’s run through what we’ve got here, hopefully some of it looks familiar from the previous examples we’ve gone through. First we fetch the page to start with, then in the select field named “url” we choose the options that says “Books”, and we then submit the form. scRUBYt! will keep track of the last form you input any data to, so if there are multiple forms on the page then you just need to target the appropriate input fields. From there, the submit action will work out what it needs to do.

Next is to define a detail block, so we point out the XPath to the heading/link for each book on the page and say we want to navigate to that page and extract the title, price, saving, etc. I’ve been a little cheeky with the isbn definition, saying just find me any LI tag that contains the string “ISBN-10:”. If we were to look at the results generated you’d see:

puts @extractor.results.inspect
[{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"}, {:price=>"$26.39"}, 
          {:saving=>"$13.60\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0596516177"}]}, 
 {:book=>[{:title=>"The Ruby Programming Language (Paperback)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>"ISBN-10: 020171096X"}]}, 
 {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"}, 
          {:price=>"$26.39"}, 
          {:saving=>"$13.60\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 1590597664"}]}, 
 {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Kindle Edition)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>nil}]}, 
 {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"}, 
          {:price=>"$29.67"}, 
          {:saving=>"$15.28\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0974514055"}]}, 
 {:book=>[{:title=>"The Ruby Way, Second Edition: Solutions and Techniques in Ruby Programming (2nd Edition) (Addison-Wesley Professional Ruby Series) (Paperback)"}, 
          {:price=>"$29.69"}, 
          {:saving=>"$15.30\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0672328844"}]}, 
 {:book=>[{:title=>"Ruby Way, The: Solutions and Techniques in Ruby Programming (Kindle Edition)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>nil}]}, 
 {:book=>[{:title=>"Learning Ruby [ILLUSTRATED]  (Paperback)"}, 
          {:price=>"$23.09"}, 
          {:saving=>"$11.90\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0596529864"}]}
]

And we have a reasonable snapshot of the data. You’ll see there though, that it’s not perfect. Firstly, we are missing information for some results. We’ve got spaces and carriage returns in the saving data, and we probably don’t need the “ISBN-10:” string at the front of the ISBN result. And what if we wanted to link to the actual result so someone could actually buy the book on Amazon?

All good questions, and all easily solvable. I’ll follow it up with a post in the next day or so and highlight some of the new ways of specifying constraints on your data in scRUBYt!. And special thanks have to go to Homeflow who have been funding at least a day of time for Peter and I each week lately, hence the increased level of development in scRUBYt!. It’s nice when you have clients who want to actively give back.

Rails Hosting - Updated

Mon, 05 Jan 2009 23:10:53 GMT

This is just a quick note so that those subscribing to my feed or checking the site for updates are aware. I’ve made a series of minor changes to the rails hosting post from last month based on the feedback provided by you guys. If you’ve got any additional experiences with different rails hosts that you’d like to share, then please send them through.

Ruby on Rails Tutorials

Mon, 05 Jan 2009 02:34:09 GMT

There has been a few occasions over the previous months when various people I meet have asked me to point them in the right direction to get started with ruby on rails. I remember when I started, it was a pretty easy question to answer… everyone just got pointed in the general direction of why_’s poignant guide, the pickaxe book, and agile web development with rails. But things have come a long way since then. There is not only a veritable cornucopia of options now out there, but for somebody who isn’t already engrossed in the community it’s almost impossible to tell which ones are still relevant and useful and which ones are so dated that they’ll make your attempts to learning markedly more difficult.

So with that in mind, I’ve attempted to compile a list of resources which are still (at least for the most part) still relevant at the time of posting. Some will have some minor inconsistencies either in implementation or current convention but I’ve made a genuine effort to review and ensure they are not way out of line. If you find in errors on that front, please let me know in the comments.

So without any further ado, here are the ruby on rails tutorials I’ve found broken into various categories (a couple are listed in more than one):

Ruby on Rails Tutorials

Getting Started and Example Applications
Models/Migrations
Controllers
Views
Routing
Mail
Testing
Performance, Administration, and Debugging
Miscellaneous
Extra Special Mention

Getting Started and Example Applications

Getting started with Rails A great guide on building your first rails app and understanding the framework.

Ruby On Rails Tutorial : The Basics A two-part tutorial to getting up and running on rails by Huw Collingbourne.

Building a blog About.com have have this guide for Rails 2.1 which runs you through the classic “15-minute blog” project.

Starting a rails project using Komodo For those of you using Komodo this is a great guide to show you how to develop within Komodo and run the tests within your IDE.

Building a bookmark manager A quick introductory guide to rails and an example app to manage your bookmarks.

Building a contact manager A ruby on rails tutorial on how to get acquainted with the framework by building a contact list manager.

A simple product catalog A tutorial from Oracle on how to create a comics catalog on top of an Oracle back-end.

The official 15-min blog screencast A rather high paced run-through of how to get a blog up and running in 15mins.

The ultimate guide to ruby & rails Quite a detailed and useful tutorial from Patrick Lenz which not only goes through rails but is a pretty decent intro to the ruby you should get your head around first too.

What I wish I knew A bit of a retrospective from someone after they got their first few rails applications under their belt.

Sending Instant Messages in Rails How to use XMPP and BackgroundRB to send messages to GMail and other Jabber based clients.

Programming Conventions A discussion on what exactly DRY means, why you should stick to conventions, and when you should look at optimization.

Models/Migrations

Bulk Insert into ActiveRecord with Rails How to quickly insert multiple records at once with ActiveRecord.

Using Oracle A tutorial from Oracle on how to create a comics catalog on top of an Oracle back-end.

Rails email validation Check that a provided email address really is legit, by doing a DNS/MX lookup on the domain.

Rail Migrations for beginners This tutorial is a little dated, but thankfully the examples all still work in Rails 2.x. What I particularly like, is that they actually cover how to add an index to a column which most don’t cover and thus many don’t bother to do.

Rails Guide to Migrations A very detailed guide on what you can and can’t do in migrations. Even I learnt a few new tricks from reading through it.

ActiveRecord Validations and Callbacks All the various validation options you have, and what method callbacks you can hook into when modifying an instance of a model.

ActiveRecord Associations How to link up the associations between the models in your applications.

Finding records in your database A tutorial on all the various ways you can find either a single record, or a collection of records, using ActiveRecord. The built in finders, and how to pass your own custom conditions in are all handled.

Controllers

Simplify Ruby on Rails controllers with make_resourceful Take advantage of the DRY approach by quickly implementing common controller actions in just a few lines of code.

Controlling Information Access A quick explanation of the why/when you should make methods in your controllers private or protected.

Displaying Useful Flash Messages How to provide more useful feedback on errors or actions to your users within your application.

Consistent and sexy rails forms How to both DRY up your code and ensure your users get a consistent experience by using a Rails Custom Form Builder.

ActionController Basics How to use filters, disabling the session, cookies, authentication, and protecting against forgery. It’s called basics, but it covers quite a lot.

Catching rails error messages An easy example of how to use the 2.x approach to catching errors in your rails applications.

Views

Introduction to form helpers A quick tutorial on when you should use the object/model based form helpers in rails, and when you should use the standard ones.

Rendering layouts and views How to render a response to the web browser within your rails app. Also goes into providing only a HEAD response, nesting layouts, using partials, redirects, and using yield.

Implementing unobtrusive javascript in rails How to stop using the built-in javascript helpers and instead use lowpro to keep your views and code clean.

Routing

Nested Resources in Rails 2.x A guide to bring you up to speed on the changes between the way resources work in 1.x and the 2.x releases.

Rails Routing from the Outside-In A very comprehensive guide on how to configure and use the routing options in rails.

Mail

Stop Rails from sending email A useful way to check that mail is getting sent, without actually sending it.

Testing

Test Rails XML Output with rspec How to easily test XML output using rspec.

Testing webservices using rspec A guide on testing the old style SOAP/XML webservices with rspec.

Using rspec have_tag How to test anything you need to in a view with rspec have_tag

Performance, Administration, and Debugging

Improve Rails Performance with Thin A tutorial on replacing mongrel or webrick with the event driven thin web server.

Let nginx automatically become aware of your Rails apps How to configure nginx to automatically become aware of any new apps you deploy and start serving them without any additional configuration.

Monitoring your Rails app with god How to use the god monitoring tool to manage, start, stop, and restart your rails app.

Monitoring BackgroundRB with god Another tutorial on using god, this one for monitoring BackgroundRB.

Improving Bulk Data Insertion with ActiveRecord Reduce the number of round-trips to the database by batch inserting records into ActiveRecord.

Background Processing with Starling How to off-load long running tasks to a queue so that you don’t tie up your webserver processes requests needlessly.

Ruby on Rails Caching Tutorial Another two-part tutorial. This one, by Gregg from RailsEnvy, runs through the various ruby on rails caching options and how to implement them to improve the performance of your site.

Ruby on Rails Rake Tutorial Gregg from RailsEnvy again running through how to make the most of rake. If you’re not already writing your own custom tasks in rake you should definitely take a look because there is probably a handful of things your could (and should) be doing in it.

Debug your app with ruby-debug I don’t know how I ever diagnosed problems in development before I learnt how to use ruby-debug. If you’ve got applications even just staring to come together, you really need to get your head around this tutorial.

19 Rails Tricks Most Rails Coders Don’t Know Not a tutorial per se, but a good collection of handy tips on how to benchmark performance, test various aspects of your app, improve it’s performance, and more.

Reading a stack trace You’re inevitably going to see a few when your developing, you better learn how to make sense of them.

Ruby on Rails Security Guide Storing and protecting the session, user management, projecting from injection, and more.

Debugging Rails Applications How to use views to help you debug your app, detecting memory leaks, and using the logger.

Benchmarking and Profiling Rails The why and how of benchmarking and profiling. Still a work in progress.

Miscellaneous

Define your own base rails framework with templates Stop manually importing and installing all the same plugins in all of your applications, define a template in just a few lines and let rails know what it should start with.

Integrate an IM Bot into your Rails App How to make an Instant Messaging BOT and use it in your rails application.

Creating Plugins How to create a plugin for use in your rails app.

Working with Rails Timezones How to use the 2.x enhancements to easily switch between timezones or display dates/times in a users local zone.

Super extra special mention

Ryan Bates is simply tireless in providing useful screencasts showing various bits of functionality and tips within rails. If I was to list each of the ones I thought would be useful here, then the list would probably be 3 times longer than it already is. Instead, make sure you check out Railscasts and scan through the archive.

Web Spider Creation with scRUBYt! - Part II

Wed, 17 Dec 2008 10:00:35 GMT

Continuing on from the previous post, Web Spidering and Data Extraction with scRUBYt!, this article will help you delve a little deeper with the scRUBYt! scraping framework both in terms of your understanding of how to use it… and actually delving deeper in your crawl to more pages.

A quick recap on the last web spider

So as far as we got last week was to go off to Google, put in a search for the word “ruby”, and then list the link text for each of the results. Here’s the code we ended up with to get that far:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_title "//h3[@class='r']"
end

How to scrape deeper pages

But this is a fairly contrived example, and it’s not exactly going to save you a huge amount of time over a quick manual copy-and-paste job from the results page. But what if you didn’t want to just have a list of the links, you wanted some kind of summary or additional detail on each of the links? Lets actually go to each website, and see what kind of content they’ve got and grab something useful from it:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    page_body "//body"
  end
end

If you offer up the XPath to a link element with a result name ending in _detail, and then pass in a block, scRUBYt! will follow the link before trying to process the block. That means we can use this technique for following each of the results Google gives us, and at each we return all of the text contain on the page between the <body> tags.

Now you could take the hash returned, and do some post processing on it to get something meaningful from the text extracted. But for the sake of example, I’ll make some big assumptions and assume that every page is going to have at least a <title> tag, and a <p> tag with more than just a few words in it.

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    title "//title"
    summary "//p", :script => Proc.new{|result| result if result.match(%r{(\w+\W+){25}})}
  end
end

As you may have noticed, you can pass a proc in as a parameter to your result definition. The output of the XPath match will be passed in to the proc, and the result ultimately returned as the final result for that definition? Make sense? If not, what I’ve done above is look for all <p> tags on the page and pass them in to my proc definition. The proc then runs a regexp against it to check that at least 25 words exist within the <p>, if there is then all the <p> content is returned otherwise nil is returned.

What if the results are paginated?

You could potentially create a highly recursive extractor to handle this, but it’s such a common case that we’ve included a method to do it for you:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    title "//title"
    summary "//p", :script => Proc.new{|result| result if result.match(%r{(\w+\W+){25}})}
  end
  next_page "//a[text()*='Next']", :limit => 2
end

I’ve used the XPath text() function here to highlight its usefulness. I use it quite a lot as a shortcut to get things working and test, and it’s been a lifesaver in many scenarios where the markup is inconsistent or you want to keep the scraper definition generic. And thankfully, it works for this scenario. However, be wary of using it as is in production as it may have some unexpected side effects. If one of the results that came back had the word “Next” in the title, then scRUBYt! would diligently follow that link and you’d end up on the wrong page.

What’s next?

In the next installment I’ll briefly cover how to handle logging of the scrape to help you diagnose any problems, and how to handle more complex form completion.