spatten design - blog

Searching Beast and WordPress from a Rails app

2008-05-17T14:35:00Z

I did some work for ecolect recently, integrating search across their main site, a Beast forum and a WordPress blog. It was pretty straightforward once I had it figured out, but I couldn’t find a walkthrough on the net.

So, I decided to write one.

The Search Engine

Ecolect was already using acts_as_ferret for their main site search, so it was a no-brainer to keep using it. For a fun and thorough introduction to acts_as_ferret, see the Rails Envy Tutorial

Searching the Beast Forum

(Note: the beast Forum search isn’t live at the moment as the Forums are not fully functioning yet.)

Beast setup

In Beast, each Forum has many Topics (saved in the topics table), and each Topic has many Posts (saved in the posts table).

The Beast forum was set up by Shanti Braford as described in option #3 of this great article on integrating a beast forum into a Rails app.

In this setup, the Beast tables are added to the main site’s database. So, the posts and topics tables that we want to search are already in the main site’s database. This made things pretty easy: you can search the Beast forum just like you would any database table.

Searching a non-integrated Beast forum

If you don’t have the Beast tables integrated in to your main site’s database, you can still search them. You just need to point the Topic and Post models to the correct database. This is a two step process.

First, set up a database entry for your Beast forum in config/database.yml. Something like this:

beast:
  adapter: mysql
  database: beast_forum
  username: app
  password: your_password
  host: localhost

Then, in the bottom of config/environment.rb, add the following lines:

12	Post.establish_connection "beast_forum"Topic.establish_connection "beast_forum"

I haven’t actually tried this, so let me know if you get it working or if you needed to make any changes to what I’ve written here.

The Topic Model

Even with the integrated setup there were a few wrinkles. First, although the Beast tables are in the database, there are no models associated with them. I wanted to search post body and topic titles, so I created Topic and Post models.

Here’s the Topic model. It’s only here so that the Post model can search a posts’s titles, so there’s not much to it.

class Topic < ActiveRecord::Base
  
  has_many :posts
    
end

The Post Model

The Post model is a bit more complicated.

class Post < ActiveRecord::Base
  
  belongs_to :topic
  
  SEARCH_FIELDS = [:scrubbed_body, :topic_title]
  
  acts_as_ferret :fields => { :scrubbed_body => {:boost => 0, :store => :yes}, 
                              :topic_title => {:boost => 3, :store => :yes}}  
  
  extend FullTextSearch
  
  # remove the html tags from body
  def scrubbed_body
    body_html.gsub(/<\/?[^>]*>/, "")
  end
  
  def topic_title
    topic.title
  end
      
  # Construct the url to the post    
  def url
    File.join("http://forums.ecolect.net", "forums", topic.forum_id.to_s, 
              "topics", topic.id.to_s + "#post_#{id}")
  end
  
end

acts_as_ferret

The acts_as_ferret declaration makes the Post model searchable. Notice that the actual fields being searched are not taken directly from the database; they are both manipulated in some way. acts_as_ferret doesn’t really care if the stuff it is indexing is coming directly from the database or from methods you have added to your model.

scrubbing the html tags

The post body is stored with HTML tags in them, so I wanted to search and show the posts with tags scrubbed out of them. This is done using the Post#scrubbed_body method, which is just an ugly regexp that takes out anything between < and > signs.

the url method

I also wanted to link to the posts, so I created a Post#url method which is used in the view.

Finally, the actual search is done using the FullTextSearch mixin, which adds a class method Post::full_text_search to Post. The FullTextSearch mixin is described in more detail below.

Searching the WordPress Blog

The only table from the WordPress Blog that you really care about is the wp_post table. To get access to it in your Rails app, make a WpPost model and point it at your WordPress db.

First, create a database entry in config/database.yml that looks like this:

wordpress:
  adapter: mysql
  database: wordpress
  username: app
  password: your_password
  host: localhost

Then, add the following line to the bottom of config/environment.rb

WpPost.establish_connection "wordpress"

Here’s the model:

class WpPost < ActiveRecord::Base
  
  primary_key = "ID"
  
  SEARCH_FIELDS = [:scrubbed_title, :scrubbed_content]
                   
  acts_as_ferret :fields => { :scrubbed_title => {:boost => 3, :store => :yes},
                              :scrubbed_content => {:boost => 0, :store => :yes}}
  
  extend FullTextSearch
  
  def id
    read_attribute(:ID)
  end
  
  # title and content need to have the html tags removed from them, and should
  # only be searchable if the post has been assigned a url (the guid) and the post 
  # is actually a post and not an asset.
  def scrubbed_title
   post_title.gsub(/<\/?[^>]*>/, "") if post_type == "post" and !guid.empty?
  end
  
  def scrubbed_content
    post_content.gsub(/<\/?[^>]*>/, "") if post_type == "post" and !guid.empty?
  end
end

There are a few things to note here:

WordPress uses ID, rather than id, as its primary key. The line primary_key = "ID" lets Rails know about that. You also need to add an id method that returns ID to get ferret indexing things properly.

You will need to scrub the html tags from the content and title; that’s what the scrubbed_title and scrubbed_content methods do.

Finally, you don’t want search results to index assets (which are stored in the wp_post model as well) or any un-published posts.

Real posts will have a post_type of 'post'.
An unpublished post won’t have its guid set.

This is taken care of by only returning titles or content if post_type == "post" and !guid.empty?.

The FullTextSearch mixin

This is based on code by Roman Mackovcak’s article on full text search in Rails. All I did was extract the method he provides out in to a mixin so I could use it in multiple models.

To use the mixin in a model, the model needs to define SEARCH_FIELDS and have an acts_as_ferret declaration. SEARCH_FIELDS is an array of symbols giving the model fields to be searched.

module FullTextSearch
  
  ##
  # FERRET SEARCH METHOD
  # This method requires that you set the following in the model:
  # SEARCH_FIELDS: 
  # a list of symbols giving the fields to be searched.
  # E.g., SEARCH_FIELDS = [:post_title, :post_content]
  #
  # The acts_as_ferret declaration.  
  # Use :store => :yes for each field if you want to use highlighting for that field.
  # E.g., 
  # acts_as_ferret :fields => { :post_title => {:boost => 3, :store => :yes},
  #                             :post_content => {:boost => 0, :store => :yes}}
  ##
  DEFAULT_PER_PAGE = 10
  
  def full_text_search(q, options = {})
     return nil if q.nil? || q == ""
     
     default_options = {:limit => FullTextSearch::DEFAULT_PER_PAGE, 
                        :page => 1, 
                        :lazy => self.const_get(:SEARCH_FIELDS)
                       }
     options = default_options.merge options

     # Get the offset based on what page we're on
     options[:offset] = options[:limit] * (options.delete(:page).to_i-1)  
     # Now do the query with our options
     results = self.find_by_contents(q, options)

     return [results.total_hits, results]
  end  
  
  
end

You use it like this:

12	WpPost.full_text_search('test')WpPost.full_text_search('test', :limit => 30, :page => 2)

The full_text_search method returns an array of length two. The first value in the array is the number of search results, and the second value the actual search results.

Re-indexing the non-local models

As Ruben pointed out in the comments, I forgot to mention how I deal with re-indexing the WordPress and Beast database tables. This is necessary as these tables have data that is modified by another application, so your Rails app doesn’t know that changes have been made to them.

To deal with this, I wrote a simple Rake task that reindexes the WpPost, WpArticle and Post models, and then added a line to the crontab to run it hourly. Here’s the rake file, which I put in RAILS_ROOT/lib/tasks/ferret.rake:

namespace :ferret do
  
  desc "rebuild the non-local Ferret indices, as these need to be done manually."
  task :rebuild_nonlocal_indices => :environment do
    [WpPost, WpArticle, Post].each do |model|
      puts "rebuilding #{model} index"
      model.rebuild_index
    end    
  end
  
end

and, for posterity, here’s the crontab line:

20 * * * *  cd <rails_root> && /usr/local/bin/rake ferret:rebuild_nonlocal_indices >> <log_directory>/ferret_reindex.log  2>&1

Rubyize This: Live in Vancouver. Refactoring #3

2008-01-27T05:02:00Z

Here’s the final refactoring from the Rubyize This workshop. See the first refactoring for an explanation of what’s going on and why this code is so darn ugly! The second refactoring is worth checking out as well.

#!/usr/bin/env ruby

def delete_large_files(directory, max_size)
  
  # Make sure directory ends in a slash
  if directory !~ /\/$/
    directory += '/'
  end
  
  # Find all of the files in the directory 
  files = Dir.glob(directory + '*')
  
  # Delete all files with size larger than max_size
  for file in files
    size = File.size(file)
    if size >= max_size
      File.delete(file)
    end
  end
  
end

max_size = 1024 * 50 # 50 kb
directory = './files_to_delete'
delete_large_files(directory, max_size)

This was the final refactoring, and the end of the conference, so we sort of ran out of time.

I didn’t get a change to show off my solution. Here it is.

def delete_large_files(dir, max_size)

  files_to_delete = Dir.glob(File.join(dir, '*')).select do |file|
    File.size(file) > max_size
  end

  File.delete(*files_to_delete)

end

What’s going on with that File.delete call?

First, File.delete takes multiple arguments and deletes all of them.

Second, I used the asterisk operator. From here:

“The asterisk operator may also precede an Array argument in a method call. In this case the Array will be expanded and the values passed in as if they were separated by commas.”

I don’t think I’ve every actually used the asterisk operator in production code, but it sure came in handy here.

Rubyize This: Live in Vancouver. Refactoring #2

2008-01-27T04:54:00Z

Here’s the second refactoring from the Rubyize This workshop. See the first refactoring for an explanation of what’s going on and why this code is so darn ugly! Don’t forget to check out the third and final refactoring

This script loads the file full of random numbers from the first refactoring and makes a beautiful ascii-art histogram from it.

#!/usr/bin/env ruby

require 'pp'

BIG_NUMBER = 32768

# Get data from a file, turn it in to a float, and find the max
data = File.readlines('random.txt')
max = -BIG_NUMBER
for n in (0 ... data.length)
  data[n] = data[n].chomp.to_f
  if data[n] > max
    max = data[n].to_i + 1 # max is ceil(max(data[n]))
  end
end

# Create the empty histogram
histogram = []
for n in (0 .. max)
  histogram.push(0)
end

# Fill the histogram
for n in (0 ... data.length)
  histogram[data[n].to_i] += 1
end

# Print the histogram
pp histogram
puts
for n in (0 .. max)
  puts "*" * histogram[n]
end

Here’s a very concise one-liner from the crowd:

0.upto((data=File.readlines('random.txt').collect {|e| e.chomp!.to_f}).max.to_i) {|i| puts i.to_s + " " + data.select{|a| a==i}.size.to_s}

Here’s Owen’s refactoring:

require 'pp' 
histogram = [] 
File.readlines('random.txt').each do |value| 
  i = value.chomp.to_i 
  histogram[i] ||= 0 
  histogram[i] += 1 
end 

# Print the histogram 
pp histogram 
puts 
histogram.each { | v | puts "*" * v }

Here’s Sam Livingstone Gray’s refactoring

#!/usr/bin/env ruby

require 'pp'

# Get data from a file, turn it in to a float, and find the max

module Enumerable
  def value_counts
    h = Hash.new(0)
    each { |e| h[e] += 1 } 
    h
  end
end

lines = File.readlines('random.txt')
histogram = lines.map { |line| line.chomp.to_i }.value_counts

# Print the histogram
pp histogram
puts
histogram.keys.sort.each { |n| puts '*' * histogram[n] }

I really like this one. Sam created a simple extension to Enumerable that I can see using over and over again.

Here’s what I came up with, with some debugging help from the group

#!/usr/bin/env ruby

require 'pp'

data = File.readlines('random.txt').collect {|datum| datum.chomp.to_f}
max = data.max

histogram = data.inject([0] * (max + 1)) do |histogram, datum|
  histogram[datum.to_i] += 1
  histogram
end

pp histogram
histogram.each do |bucket|
  puts "*" * bucket
end

Rubyize this: Live in Vancouver. Refactoring #1.

2008-01-27T00:11:00Z

At rubycamp today, I ran a little Rubyize This workshop. Rubyize This is a game invented by François Lamontagne. The idea is that someone puts up a chunk of code that is written in Ruby, but in a not very Rubyish way. Then, everyone in the audience gets to Rubyize it!

We tried to use the most excellent RefactorMyCode.com, but the WiFi was too saturated so we had to resort to posting the code on my blog and people plugging their laptops in to the projector to present their results.

So, this post and the following two will show the original code. Keep in mind that this is intentionally ugly!

We had some great refactorings. If you were at the workshop, please post your refactorings in the comments. I’ll make the pretty and put them in the posting.

The other two refactorings are at Rubyize this #2 and Rubyize this #3

Without further ado, here’s the first refactoring. The code is supposed to create a file filled with random numbers, one number per line.

#!/usr/bin/env ruby

# create num random numbers between min and max 
# and print them to a file called file_name, one per line
def create_random_numbers(file_name, min, max, num)
  file = File.open(file_name, 'w')
  n = 0
  while n < num
    r = rand(max - min) + min
    file.puts(r)
    n += 1
  end
  file.close   
end

create_random_numbers('random.txt', 0, 10, 1000)

Here’s the first refactoring:

def create_random_numbers(file_name, min, max, num) 
  File.open(file_name, 'w') do |f| 
    1.upto(num) {|i| f.puts rand(max-min)+min }
  end 
end

Here’s one from Sam Livingston-Gray

#!/usr/bin/env ruby

# create num random numbers between min and max 
# and print them to a file called file_name, one per line

class Range
  def random_member
    offset = rand(max - min)
    min + offset
  end
end

def create_random_numbers(file_name, range, num)
  File.open(file_name, 'w') do |file|
    num.times { file.puts(range.random_member) }
  end
end

create_random_numbers('random.txt', (0..10), 1000)

Here’s what I came up with:

def create_random_numbers(file_name, min, max, num)
  File.open(file_name, 'w') do |file|
    num.times {file.puts(rand(max - min) + min)}
  end
end

create_random_numbers('random.txt', 0, 10, 1000)

Adding json responses to all of your controller actions

2008-01-14T18:39:00Z

Gerald has decided to use JSON responses for the mobile phone app we’ll be building at the intermediate Rails workshop. I’m a lazy guy by nature, so there was no way I was going to paste in

format.json  { render :json => @bookmark }

in to very controller method.

Now, the easiest way to do this would be to add JSON response to the scaffolding. The templates are in RAILS_ROOT/vendor/rails/lib/rails_generator/generators/components/scaffold/. Open up the controller.rb template and add in the JSON response shown above to the index, show, update and destroy methods and you’re done.

Unfortunately, I had already mostly developed the API when I realized that I had to add in the JSON responses (It’s not Gerald’s fault, he told me long ago). So, I decided to write a quick rake task to do the job for me. To use it, create a file called add_json.rake in RAILS_ROOT/lib/tasks and copy the following code in to it:

namespace :rails do
  
  desc "Adds a 'format.json ...' line for each 'format.xml ...' line in every controller"
  task :add_json => :get_controller_list do
    @controllers.each do |controller|
      new_lines = File.open(controller, 'r').readlines.collect do |line|
        if line =~ /^\s+format\.xml/
          [line, line.gsub(/xml/, 'json')]
        else
          line
        end
      end.flatten
        
      file = File.open(controller, 'w') do |file|
        file.puts new_lines
      end
    end
  end
  
  task :get_controller_list => :environment do
    @controllers = Dir.glob(File.join(RAILS_ROOT, 'app', 'controllers', '*_controller.rb'))
  end
end

Call the task from RAILS_ROOT like this: rake rails:add_json

The task opens up every controller, and every time it finds a line that starts with format.xml, it makes a copy of that line just below with every instance of ‘xml’ replaced with ‘json’. It’s ~~pretty~~ very brain-dead: it doesn’t check if the json response is already there, so don’t run it twice on the same project.

Fun with Single Table Inheritance

2007-12-31T22:14:00Z

I’m working on the sample application for the Intermediate Ruby on Rails Workshop that Gerald Bauer and I are putting on in January. I had to remind myself of some of the foibles of Single Table Inheritance (STI). I thought others might find this useful, so here’s what I found.

The Set-up

I have two types of posts in the application I’m building: posts by people with an empty room looking for a roomie (RoomiePosts), and posts by people looking for a room (RoomPosts). They both share a lot of the same attributes, so it’s not very DRY to create two separate models.

This is where Single Table Inheritance comes in. STI is basically a way of subclassing a single model, creating subclasses that all use the same database table.

What I did is create a Post model and two subclasses: RoomiePost and RoomPost. Here’s how you do it.

Creating the Database Table

Here’s the migration for the posts database table. Note that this uses the new migrations syntax in Rails 2.0.

class CreatePosts < ActiveRecord::Migration
  def self.up
    create_table :posts do |t|
      t.integer :rent
      t.date :start_date, :expiry_date
      t.boolean :includes_utilities
      t.string :text, :type
      
      t.timestamps
    end
  end

  def self.down
    drop_table :posts
  end
end

Most of the columns are pretty straightforward. text will contain the text of the post. start_date and expiry_date will determine when the post will start and stop being shown on the site. rent and includes_utilities will only be used by the RoomiePost subclass, but still need to be created for all subclasses of the Post model.

The interesting column is type. This allows Rails to determine which subclass the row should be loaded in to. If you set type = "RoomiePost", then it’s a RoomiePost. If you set type = "RoomPost", then it’s a RoomPost.

Creating the Models

You need to create a Post, RoomiePost and RoomPost model. They look like this:

In /app/models/post.rb

123	class Post < ActiveRecord::Base end

In /app/models/roomie_post.rb

123	class RoomiePost < Post end

In /app/models/room_post.rb

123	class RoomPost < Post end

Note that RoomiePost and RoomPost are both subclasses of the Post model. Here’s where you run in to a “gotcha”.

You have to create the models in separate files. If you don’t, Rails ignores the subclassed models.

For more information, and another workaround, see this blog post. The comment by Chris at the bottom gives the ‘create the models in separate files’ solution.

Okay, so now you have the models all set up. You can do things like

>> p = RoomiePost.create(:rent => 800, :includes_utilities => true, :text => "Great room in a shared house in the Commercial Drive area")

And it will create a row in the posts table with type = "RoomiePost". If you have 5 RoomiePosts and 2 RoomPosts, then Posts.count will return 7.

>> RoomiePost.count           
=> 5
>> RoomPost.count
=> 2
>> Post.count
=> 7

Adding behaviour to STI models

I want to be able to determine if a post is active by looking at the start and expiry dates, and to get a list of all active posts, roomie_posts and room_posts. I also want to have a different icon for each post type.

The Post.active_post? and Post::active_posts methods are straight-forward:

class Post < ActiveRecord::Base
  
  def is_active?
    (start_date .. expiry_date) === DateTime.now
  end 
  
  def self.active_posts
    self.find(:all).select {|p| p.is_active? }
  end  
  
end

The only tricky part is that I used self.find(:all) instead of Post.find(:all) in the Post::active_posts method. This makes sure that RoomiePost::active_posts only returns RoomiePosts and RoomPost::active_posts only returns RoomPosts.

Getting the icon to work took a little more figuring out. If I wanted to do this in straight-up Ruby, I’d do something like

#!/usr/bin/env ruby

class Post
  ICON_PATH = "icons"
  
  def initialize
    @icon_name = 'base.gif'
  end
  
  def icon
    File.join(ICON_PATH, @icon_name)
  end
  
end

class RoomiePost < Post
  
  def initialize
    @icon_name = 'roomie.gif'
  end
  
end

class RoomPost < Post
  
  def initialize
    @icon_name = 'room.gif'
  end
  
end

Then I could do this

ruby $>irb -r subclass_test
irb(main):001:0> Post.new.icon
=> "icons/base.gif"
irb(main):002:0> RoomiePost.new.icon
=> "icons/roomie.gif"
irb(main):003:0> RoomPost.new.icon
=> "icons/room.gif"

This doesn’t quite work in the Rails version. Not to worry, you only have to make one simple change:

replace the initialize method with an after_initialize callback.

The final post.rb, roomie_post.rb and room_post.rb files look like this:

post.rb:

class Post < ActiveRecord::Base

  ICON_PATH = "/images/icons"
  
  def after_initialize
    @icon_name = 'base.gif'
  end
  
  def icon
    File.join(ICON_PATH, @icon_name)
  end  
  
  def is_active?
    (start_date .. expiry_date) === DateTime.now
  end 
  
  def self.active_posts
    self.find(:all).select {|p| p.is_active? }
  end  
  
end

roomie_post.rb:

class RoomiePost < Post
  
  def after_initialize
    @icon_name = 'roomie.gif'
  end
  
end

room_post.rb:

class RoomPost < Post
  
  def after_initialize
    @icon_name = 'room.gif'
  end
  
end

One final note: because of the way the after_initialize callback works, you have to actually define an after_initialize method. You can’t do something like this:

class RoomiePost < Post
  after_initialize :set_icon_name

  private

  def set_icon_name
    @icon_name = 'roomie.gif'
  end
  
end

If you want to learn more about STI, and a whole slew of other info on Rails, sign up for the Intermediate Rails Workshop on January 25th, 2008 in Vancouver.

Early Bird rates are available until January 9th!

RubyCamp Vancouver

2007-12-19T23:39:00Z

RubyCamp is a free one-day gathering for Rubyists and Railsers.

When and Where

WorkSpace in downtown Vancouver, B.C., Canada
January 26th, 2008 from 9:00 to 5:00

Who should come

Anyone who’s interested in Ruby and Rails, whether you’re just interested in learning what this Ruby thing is all about or you know Ruby inside out.

The Conference Track

A conference-style track with “classic” talks on Ruby or Rails topics. We’re looking for a few more speakers, so see here if you’re interested in giving a talk.

The Hackathon Track

An informal un-conference track focusing on hacking some Ruby code, showing off a cool feature you just added to your Rails application or demonstrating a new addition to Rails 2.0.

If you’re interested in working on some code or showing off something, get started by promoting your ideas and getting some buzz going .

A weekend of Ruby

There are two more reasons to come to Vancouver for those of you who are out of town.

On the Friday before RubyCamp, Rails Advance is giving a one day intermediate Ruby and Rails workshop. (Rails Advance is Gerald Bauer and I, by the way)

On the Sunday after RubyCamp, Peter Armstrong is giving a one day workshop on Flex and Rails.

For more information, see the RubyCamp web site

See you all at RubyCamp in Vancouver!

Amazon S3, Ruby and Rails slides

2007-12-05T20:19:00Z

The slides from the talk are here. (Yes, they’re hosted on S3).

There are two points in the presentation where I switched to a different window.

At the ‘S3SH DEMO’ slide, I did some live coding showing how you can work with S3 using s3sh. It basically followed the script shown in ‘s3sh demo script’ below, so read that part when you see the ‘S3SH DEMO’ slide.

At the ‘Example: S3Syncer’ slide, I switched over to textmate and showed the code for a simple script to synchronize a single directory to S3. I then demoed the script to show it working. So, when you see the ‘Example; S3Syncer’ slide, read the s3syncer code and s3syncer demo sections below.

s3sh demo script

Start up s3sh

$> s3sh

Create a bucket. Show that you can create a bucket multiple times if you own it, but trying to create a bucket that somebody else owns raises an error.

>> Bucket.create('spatten_s3demo')
=> true
>> Bucket.create('spatten_s3demo')
=> true
>> Bucket.create('test')
AWS::S3::BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/error.rb:38:in `raise'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/base.rb:72:in `request'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/base.rb:83:in `put'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/bucket.rb:79:in `create'
        from (irb):3

You can save a bucket in a variable using Bucket.find

>> b = Bucket.find('spatten_s3demo')
=> #<AWS::S3::Bucket:0x14ae7b8 @attributes={"prefix"=>nil, "name"=>"spatten_s3demo", "marker"=>nil, "max_keys"=>1000, "is_truncated"=>false, "xmlns"=>"http://s3.amazonaws.com/doc/2006-03-01/"}, @object_cache=[]>

Create a text object

>> S3Object.store('test.txt', 'This is a test', 'spatten_s3demo')
=> #<AWS::S3::S3Object::Response:0x10830590 200 OK>
>> b.objects
=> [#<AWS::S3::S3Object:0x10804170 '/spatten_s3demo/test.txt'>]
>> pp b.objects[0].about
{"last-modified"=>"Wed, 05 Dec 2007 19:56:49 GMT",
 "x-amz-id-2"=>
  "JACm9T+m9CgZhmj4q6q00OSGHgSyBVAbQ1cgRWGydYZLTKdhLc/IUZ+K7b/1snOc",
 "content-type"=>"text/plain",
 "etag"=>"\"ce114e4501d2f4e2dcea3e17b546f339\"",
 "date"=>"Wed, 05 Dec 2007 19:57:03 GMT",
 "x-amz-request-id"=>"CA170D2AA5DEB0C9",
 "server"=>"AmazonS3",
 "content-length"=>"14"}
=> nil
>> b.objects[0].key
=> "test.txt" 
>> b.objects[0].value
=> "This is a test"

Create a binary object and show it in a browser

>> S3Object.store('vampire.jpg', File.open('vampire.jpg'), 'spatten_s3demo')
=> #<AWS::S3::S3Object::Response:0x10764700 200 OK>

Show the photo in browser

This doesn’t work, as the file is only readable by me. Make it public readable and do it again.

>> S3Object.store('vampire.jpg', File.open('vampire.jpg'), 'spatten_s3demo', 
     :access => :public_read)
=> #<AWS::S3::S3Object::Response:0x10747950 200 OK>

Show it in a browser again. It works this time.

Look at bucket.objects. We have to reload the bucket to show the new object.

>> b.objects
=> [#<AWS::S3::S3Object:0x10804170 '/spatten_s3demo/test.txt'>]
>> b.objects(:reload)
=> [#<AWS::S3::S3Object:0x10708080 '/spatten_s3demo/test.txt'>, #<AWS::S3::S3Object:0x10708070 '/spatten_s3demo/vampire.jpg'>]

Hash access to bucket objects

>> b['vampire.jpg']
=> #<AWS::S3::S3Object:0x10708070 '/spatten_s3demo/vampire.jpg'>
>> vamp = b['vampire.jpg']
=> #<AWS::S3::S3Object:0x10708070 '/spatten_s3demo/vampire.jpg'>

A look at metadata

>> vamp.content_type
=> "image/jpeg" 
>> vamp.size
=> 10817
>> vamp.metadata
=> {}
>> vamp.metadata['subject'] = 'Claire'
=> "Claire" 
>> vamp.metadata['photographer'] = 'Nadine Inkster'
=> "Nadine Inkster" 
>> vamp.store
=> true

Storing the picture data in a variable

>> picdata = vamp.value
=> "\377\330\377\340\000\020JFIF\000\001\002\000.......

Downloading a picture by streaming it to an IO object.

>> File.open('vampire_downloaded.jpg', 'w') {|file| file.write(vamp.value)}
=> 10817
>> exit
s3demo $>ls
flowers.jpg             vampire.jpg
test.txt                vampire_downloaded.jpg
s3demo $>open vampire_downloaded.jpg 
s3demo $>

S3Syncer Code

Please note that this code is really only useful as an example of how to synchronize with S3.

It won’t recurse directories and it dies a horrible death if there are any symlinked files in a directory.

If you are looking for something to synchronize directories, check out s3sync.rb.

#!/usr/bin/env ruby

require 'digest/md5'
require 'aws/s3'
include AWS::S3

class S3Syncer
  attr_reader :local_files, :files_to_upload
  
  def initialize(directory, bucket_name)
    @directory = directory
    @bucket_name = bucket_name
  end
  
  def S3Syncer.sync(directory, bucket)
    syncer = S3Syncer.new(directory, bucket)
    syncer.get_local_files
    syncer.connect_to_s3
    syncer.get_bucket
    syncer.select_files_to_upload
    syncer.sync
  end
  
  # This does not recurse directories.
  def get_local_files
    @local_files = Dir.entries(@directory)
  end
  
  def connect_to_s3
    Base.establish_connection!(
        :access_key_id     => ENV['AMAZON_ACCESS_KEY_ID'],
        :secret_access_key => ENV['AMAZON_SECRET_ACCESS_KEY']
      )
  
    raise "\nERROR: Connection not made or bad access key " +
          "or bad secret access key.  Exiting" unless AWS::S3::Base.connected? 
  end  
  
  def get_bucket
    Bucket.create(@bucket_name)
    @bucket = Bucket.find(@bucket_name) 
  end
  
  # Files should be uploaded if 
  #   The file doesn't exist in the bucket
  #      OR
  #   The MD5 hashes don't match
  def select_files_to_upload
    @files_to_upload = @local_files.select do |file|                 
      case
      when File.directory?(local_name(file))
         false # Don't upload directories
      when !@bucket[file]
         true  # Upload if file does not exist on S3
      when @bucket[file].etag != Digest::MD5.hexdigest(File.read(local_name(file)))
         true  # Upload if MD5 sums don't match
      else
        false  # the MD5 matches and it exists already, so don't upload it
      end
    end
  end
  
  # This will choke on symlinked files
  def sync
    (puts "Directories are in sync"; return) if @files_to_upload.empty?

    @files_to_upload.each do |file|
      puts "#{file} ===> #{@bucket.name}:#{file}"
      S3Object.store(file, File.open(local_name(file), 'r'), @bucket_name)      
    end
  end
  
  private 
  
  def local_name(file)
    File.join(@directory, file)
  end
  
end

if __FILE__ == $0
  S3Syncer.sync('/Users/Scott/versioned/spattendesign/presentations/s3-on-rails/s3demo', 
                'spatten_syncdemo')
end

S3Syncer demo

Start with spatten_syncdemo bucket empty, and four files in the local directory.

Run the script

s3demo $>ls
flowers.jpg             vampire.jpg
test.txt                vampire_downloaded.jpg
s3demo $>s3syncer
flowers.jpg ===> spatten_syncdemo:flowers.jpg
test.txt ===> spatten_syncdemo:test.txt
vampire.jpg ===> spatten_syncdemo:vampire.jpg
vampire_downloaded.jpg ===> spatten_syncdemo:vampire_downloaded.jpg

Run it again, it says there’s no need to do anything

s3demo $>s3syncer
Directories are in sync

Change a file locally and sync again

s3demo $> vi test.txt
Make some changes using vi
s3demo $>s3syncer
test.txt ===> spatten_syncdemo:test.txt

Delete flower.jpg using the Firefox S3 Organizer and then sync again.

s3demo $>s3syncer
flowers.jpg ===> spatten_syncdemo:flowers.jpg

So there you go, a quick intro to the wonders of Amazon S3.

synch s3 host plugin update

2007-12-04T00:32:00Z

I’ve re-written the plugin to use s3sync.rb rather than trying to figure out which files have changed in svn. It’s a much better solution, so hopefully those of you who were having trouble will find it works for you now. Thanks, especially, to Chris in the comments for helping me figure out what was going wrong.

The plug-in also now has a permanent home in my new Projects section, here.

Amazon S3, Ruby and Rails talk

2007-11-28T21:17:00Z

I’m going to be giving a talk on Amazon S3, Ruby and Rails at the next Vancouver Ruby and Rails meetup. It’s this Monday, December 3rd.

Here’s the talk teaser:

Amazon’s Simple Storage Service (Amazon S3) is an online storage system. It can be used for backup, serving assets for your Rails application, storing and streaming large media files or storing customer generated assets.

The AWS/S3 gem, written by Marcel Molina, provides an elegant interface to Amazon S3. I’ll be talking about how S3 works, how to use it in your Ruby scripts and Rails applications and showing off some example code.

Hope to see you there!

Synching Your Amazon S3 Asset Host using Capistrano

2007-11-06T23:16:00Z

Note: This article is out of date. The latest version of this article is on the new permanent page in the projects section.

So you’ve got multiple asset hosts running in your Rails application, and you’re using Amazon’s S3 to host your assets. Now you want to make sure that your assets are kept up to date. This plugin is a Capistrano recipe that keeps the asset hosts synchronized with the public directory in your subversion repository.

Usage

After you get everything setup and do your first deploy, just run cap deploy as normal and all changed files in RAILS_ROOT/public will be uploaded to all of your asset host buckets before the final deploy:symlink task.

The following tasks are also available:

cap s3_asset_host:get_s3_revision
cap s3_asset_host:find_changed
cap s3_asset_host:list_changed
cap s3_asset_host:find_all
cap s3_asset_host:upload_changed
cap s3_asset_host:upload_all
cap s3_asset_host:upload
cap s3_asset_host:reset_and_upload
cap s3_asset_host:setup
cap s3_asset_host:create_buckets
cap s3_asset_host:delete_all
cap s3_asset_host:connect

You can get documentation on these tasks by running cap -T

Requirements

This plug-in is a Capistrano extension. It requires Capistrano 2.0.0 or greater.

You will also require the aws-s3 gem

So far, this plug-in:

assumes that you are using the ‘checkout’ method of deployment.
only works with svn.

If you are using another version control system, I think all you’ll have to change is the two methods in lib/scm.rb. If you do get something other than svn working, please let me know.

If you want to use more than one asset host, then you have to either install the multiple asset hosts plugin or upgrade to Rails 2.0 (see setting up multiple asset hosts in Rails)

Setup

To set-up, you need to do the following

Install the plug-in
Install the AWS-S3 gem.
Set up your Rails application to use asset hosts.
Set up your asset hosts.
Configure Capistrano.

Installing the plug-in

From RAILS_ROOT, run:

script/plugin install svn://svn.spattendesign.com/svn/plugins/synch_s3_asset_host

Installing the AWS-S3 gem

You need to do this on both your local computer and the computer that is defined as the asset_host_syncher (see Capistrano Configuration, below).

$> sudo gem install aws-s3

Setting up your Rails app to use asset hosts

Single asset host

For a single asset host, simply add the following line to RAILS_ROOT/config/environments/production.rb:

config.action_controller.asset_host = "http://assets.example.com"

Multiple asset hosts

Follow the instructions in setting up multiple asset hosts in Rails

Setting up your asset hosts

Set up a CNAME entry for each asset host pointing to s3.amazonaws.com. How you do this depends on your domain host. Here’s what it looks like on easydns

You may need to wait up to 24 hours for the DNS entries for these new hosts to propagate.

Configuring Capistrano

Capistrano installation

This plugin requires Capistrano 2.0.0 or greater.

To upgrade to the latest version (currently 2.1.0):

$> gem install capistrano

Once the plug-in is installed, make sure that the recipes are seen by Capistrano

$> cap -T | grep s3_asset_host

should return a bunch of tasks. If you don’t see anything listed, then you need to update your Capfile by doing the following (this is from Jamis Buck):

In Capistrano 2.1.0 or above:

$> cd RAILS_ROOT
$> rm Capfile
$> capify .

If you do not want to delete your Capify file, or if you are using Capistrano 2.0.0, add the following line to your Capify file:
Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) }

Capistrano configuration

Create a new file in RAILS_ROOT/config called synch_s3_asset_host.rb. Add the following lines to it, and edit to suit:

# =============================================================================
# S3 ASSET HOST OPTIONS
# =============================================================================
set :asset_host_name, "assets%d.example.com" 
set :aws_access_key, "your Amazon AWS access key" # You can also set this in your environment as AMAZON_ACCESS_KEY_ID
set :amazon_secret_access_key, "your Amazon AWS secret" # You can also set this in your environment as AMAZON_SECRET_ACCESS_KEY
# set :dry_run, false# Set to true if you want to test the asset_host uploading without doing anything on Amazon S3
before "deploy:symlink", "s3_asset_host:upload_changed"

You have to do one more thing: in RAILS_ROOT/config/deploy.rb. Specify one of your web hosts as an “asset_host_syncher”, like this:

role :web, webserver1, :asset_host_syncher => true

The first deploy

Commit all changes to your rails application and do the initial bucket setup:

$> cap s3_asset_host:setup
$> svn commit -m "Adding synch_s3_asset_host plugin" 
$> cap deploy

This will do the following:

Create your Amazon S3 AWS buckets
upload everything in RAILS_ROOT/public (in your svn repository) to each bucket
Set the revision in each bucket to the latest revision in your repository.

This could take a while if you have lots of images or other big files.

You’re done!

That should do it. Now, every time you run cap deploy, your asset hosts should be updated with any changes to files in RAILS_ROOT/public.

Let me know if you have any problems, suggestions or comments.