Pluckety pluck pluck

First up, Rails 3.2 has added a pluck method, which functions a lot like Valium, minus the ability to re-use already-loaded attributes and select multiple columns. I made an attempt to get the Valium implementation in to replace pluck, but it was rejected. Valium is currently working just fine for all 3.x versions of Rails, so given the missing functionality (and unfortunate naming) of pluck, I’ll continue to maintain it. It’s a small enough library that the maintenance overhead is minimal.

The attribute store

Rails 3.2 has also added something called the Active Record Store. At first glance, it looks pretty similar to attr_bucket, a library I cooked up last year to satisfy a need in one of my own apps. It allows for a simple database-backed key-value store that piggybacks on the AR serialized attribute support.

Based on a quick review of the Active Record Store code, the implementation in core is, similarly to pluck, pretty simplistic. At this point, I plan to continue to maintain attr_bucket as well. It’s way overdue for some love to allow the use of alternate coders and such in 3.1+.

Conclusion

In the end, the increased visibility for this kind of functionality in core should only be good for Valium and attr_bucket, as people who may not have been aware that such functionality had already been developed will now have reason to go looking for gems when they exhaust the core implementations’ capabilities. As is so often the case in open source, everyone wins. :)

Verify it’s enabled

TM2 Terminal Preferences

Just verify that the “Accept rmate connections” checkbox is checked, “Access for” is set to “local clients” and note the port number (52698 be default) or set it to one of your choosing.

Once you’ve done so, click on the “rmate” link in the preference’s descriptive text to open the rmate script in a TextMate window. Copy the script’s contents.

The Tunnel o’ (TextMate 2) Love

If you’re not familiar with SSH tunneling, all you need to know for the purposes of setting this up is that SSH allows you to send your traffic destined for certain ports over the SSH connection to your local machine (possibly to an alternate port). SSH to a server you’d like to edit some files on, using the following command:

ssh -R 52698:localhost:52698 <user>@<server>

Be sure to replace 52698 with the port you chose, if you altered it.

This command says that for the duration of your SSH session, any connections to port 52698 on the remote machine will have their traffic forwarded on to your local machine’s port 52698, where your running TextMate 2 instance will happily accept it.

Copy the rmate script (first time only)

We need to get the script installed on the server. Since you’re already SSHed onto the server, the easiest way to do that is probably just to paste the contents of the script into a new file. Substitute your own editor of (second) choice as appropriate:

# If installing for all users...
sudo vi /usr/local/bin/rmate
# If installing for yourself (make sure it's in your PATH!)... 
vi ~/bin/rmate
# Make it executable
chmod +x <previously-used-directory>/rmate

Now, if all went well, you should be able to run:

rmate rmate_rocks.txt

A TextMate window should pop up with a blank file. Type whatever you like into the file (might I suggest “rmate rocks!”) and save it. On the server, your rmate_rocks.txt file will have been saved!

Troubleshooting

/usr/bin/env: ruby: No such file or directory

Your server doesn’t have Ruby installed. The rmate script is written in Ruby, so you’ll need to install a Ruby interpreter.

Warning: remote port forwarding failed for listen port 52698

Someone else is already using port 52698 on the server for something. No big deal. Disconnect from the server, and choose another port:

ssh -R <my-port-number>:localhost:52698 <user>@<server>

Now, when you run rmate, you’ll just need to specify the alternate port, like so:

rmate -p <my-port-number> rmate_rocks.txt

Since the SSH tunneling takes care of forwarding the rmate traffic to 52698 locally, you don’t have to change any setting in the TextMate 2 Terminal preferences!

That’s it!

I’m really excited about TextMate 2, and haven’t gotten to play around with it much yet, but I hope this helps you get started with what I think is one of the coolest new features. Thanks for reading!

[Update 12/14/11: If you want to further simplify your rmate setup, check out Jon Simpson's writeup on setting up RemoteForward in your SSH configuration file]

I’d been holding off on making this announcement on my weblog until we had something on our company blog, but things are moving quickly, and since it happens that we are looking for another Machinist, I decided that I’d like to announce that here. So, if you love Web Operations, beer, Rails, ping pong, espresso, and working with awesome customers, then we should talk.

Scopes Fall Short

We can all generally agree on the principle of DRY, or “Don’t Repeat Yourself” when it comes to writing software. In the realm of ActiveRecord, we have a really useful tool in our arsenal to enhance readability and enable easy reuse of query logic. That tool is scope, and it needs no introduction.

In the case of adding conditions to a WHERE clause, scopes work great, so long as you want to add those conditions against the base model being used in your query. We can just write something like:

class User < ActiveRecord::Base
  scope :name_starts_or_ends_with, 
        lambda {|str| where{(name =~ "#{str}%") | (name =~ "%#{str}")}}
end
 
User.name_starts_or_ends_with('bob')
# => SELECT "users".* FROM "users"  
     WHERE (("users"."name" LIKE 'bob%' OR "users"."name" LIKE '%bob'))

This works great, for that limited case. But what if we would really like to use that condition against users, but users are part of a join? Scopes fail us, then.

Sure, we could write a scope on, say, an Article class…

class Article < ActiveRecord::Base
  scope :authored_by_users_with_name_starting_or_ending_with,
        lambda {|str| joins(:user).
                      where{user => ((name =~ "#{str}%") | (name =~ "%#{str}"))}
end

…but that’s a bad idea for a few reasons, the most problematic being that now we’re making assumptions about a User’s column names from the article class. There’s also the problem of needing a similar method on any other class we might like to filter based on user names.

It would be great if we had a way to:

1. Write a reusable bundle of conditions against a model’s attributes in the code of the model it pertains to
2. Use these condition bundles even through an association
3. Abstract away the implementation details of the condition

Sifters Fill the Gap

That’s exactly what sifters are designed to solve. A sifter is defined similarly to a scope:

# Define a sifter via a class macro...
sifter :name_starts_or_ends_with do |str|
  (name =~ "#{str}%") | (name =~ "%#{str}")
end
# ...or a class method...
def self.name_starts_or_ends_with(str)
  squeel{(name =~ "#{str}%") | (name =~ "%#{str}")}
end

You can then use the sifter by calling sift within the Squeel DSL block, with the sifter name and its parameters (this example used only one, but you can write sifters that take as many as you like):

User.where{sift :name_starts_or_ends_with, 'bob'}
# => SELECT "users".* FROM "users"  
     WHERE (("users"."name" LIKE 'bob%' OR "users"."name" LIKE '%bob'))
 
Article.joins(:user).where{user.sift :name_starts_or_ends_with, 'bob'}
# => SELECT "articles".* FROM "articles" 
     INNER JOIN "users" ON "users"."id" = "articles"."person_id" 
     WHERE (("users"."name" LIKE 'bob%' OR "users"."name" LIKE '%bob'))

That’s about it. I hope you find sifters useful!

The Code

Just add this to your app’s application_helper:

  def link_to(*args, &block)
    unless block_given?
      html_options = args[2] || {}
 
      unless html_options.delete(:follow)
        if html_options[:rel]
          html_options[:rel] += ' nofollow'
        else
          html_options[:rel] = 'nofollow'
        end
      end
 
      args[2] = html_options
    end
 
    super(*args)
  end

Use WillPaginate? You may also want to add the following (adapted from this StackOverflow post):

  def will_paginate(*args)
    options = args.extract_options!
    options[:renderer] = PaginationNoFollow unless options[:renderer] || options.delete(:follow)
    super(args.first, options)
  end

Set PaginationNoFollow up with an initializer:

require 'will_paginate/view_helpers/link_renderer'
 
class PaginationNoFollow < WillPaginate::ViewHelpers::LinkRenderer
  def rel_value(page)
    case page
    when @collection.previous_page; 'prev nofollow' + (page == 1 ? ' start nofollow' : '')
    when @collection.next_page; 'next nofollow'
    when 1; 'start nofollow'
    else
      'nofollow'
    end
  end
end

Now, all of your app’s links will default to rel="nofollow", but obviously you will want some of your links to be crawled. Just add :follow => true to the options hash of either link_to or will_paginate to opt-in specific links for search engine crawling.

The solution

I figure this is going to be the tip of the iceberg in terms of potential gotchas, so before we get anywhere near to 1.0, I’m pulling out the [] syntax, but adding a new alias: values_of. This is just to help code read better — the functionality is exactly the same. So now, instead of:

# Don't do this. It won't work anymore. :(
User.where(:name => 'ernie')[:id]
User.where(:name => 'ernie')[:id, :email]

You will write:

# Do this:
User.where(:name => 'ernie').value_of :id
User.where(:name => 'ernie').values_of :id, :email

Sorry for the inconvenience. Of course, you’re always free to alias these to whatever you want, in your own code. Just don’t blame me if things randomly blow up.

And Pratik, if you’re reading this: Yes, you told me so.

MetaWhere -> Squeel

Some time ago, I blogged about a MetaWhere 2.0 rewrite. This rewrite made changes that were extensive enough that I decided to forgo a new version number and jump straight to releasing an entirely new gem, Squeel.

Squeel was written in tandem with the Rails 3.1 beta development process, so it’s ready to go on day one. It’s also been backported to 3.0.x since then, and while doing so has required backporting and monkey-patching certain ARel functionality as well, I’m pretty happy with the results — all specs are green against Rails 3.0.10.

This means that there is a clear migration path ahead for your app, if you’re using MetaWhere. You can migrate your MetaWhere queries to the Squeel DSL, first (I’d encourage you to bypass the Symbol extensions, if you can), then migrate to Rails 3.1 once all your specs are passing.

Why the break with the old? Read the original rewrite announcement for the details. In short, I made some bad architectural decisions in MetaWhere. While a lot of people really liked the gem, under the hood, things were a mess. I could have hobbled along with the existing codebase for a while longer, but it was becoming increasingly painful to do so. Besides, I’d had some new ideas about the way I’d approach a query DSL. Squeel is what came of those new ideas.

MetaSearch -> Ransack (eventually)

MetaSearch is a bit of a different animal. First, its hooks into the ActiveRecord internals are much more shallow than MetaWhere’s, so it was simpler to port over to 3.1. Second, while Squeel is mostly a superset of MetaWhere, in terms of functionality, Ransack provides fundamentally different types of search functionality, and hasn’t been completely bulletproofed. I’m using it in a few projects, myself, but for those who are looking to do a drop-in replacement for MetaSearch, it’s not quite there, yet.

As such, I’ve released a v1.1.0 of MetaSearch, which works with Rails 3.1. I’d still encourage those of you who are starting a new greenfield application on 3.1 to start with Ransack, as this is the long-term path forward.

Many of the same decoupling steps taken in the MetaWhere -> Squeel transition were also taken with Ransack, so it should provide a solid foundation to build on, going forward.

Conclusion

Thanks again for using my gems! The steady stream of appreciation and kind words have made working on them a real pleasure (on most days!). Still, I’m only one guy, and I have clients to serve, and a family I need to love and spend time with. As such, I just can’t justify spending tons of time maintaining MetaSearch and MetaWhere in addition to their replacements.

I hope you’ll understand, and I hope that you will give Squeel and Ransack a try!

Simple Schema

So, consider the case of… Oh, pretty much any Rails application, anywhere. For purposes of this discussion, let’s consider a simple blog, with articles and comments. Here’s the schema:

ActiveRecord::Schema.define do
 
  create_table :articles, :force => true do |t|
    t.string   :title
    t.text     :body
    t.boolean  :published, :null => false, :default => false
    t.timestamps
  end
 
  create_table :comments, :force => true do |t|
    t.belongs_to :article
    t.string     :email
    t.text       :body
    t.timestamps
  end
 
  add_index :comments, :article_id
 
end

No rocket science here. Article has_many :comments and Comment belongs_to :article. We put an index on the article_id column in the comments table. There’s a good chance we’ll end up doing something like chaining from an association scope, which will result in a query for comments by article_id.

Simple Select

Now, let’s say we want to do something really simple, like post a list of the 3 most recent comments on our blog’s homepage. But we only want to include comments against published articles, just in case we make a mistake and need to “un-publish” one later — we don’t want comments against articles a person can’t read showing up on our homepage!

Comment.joins(:article).
        where(:articles => {:published => true}).
        order('comments.id desc').limit(3)

This results in a query that looks like:

SELECT  `comments`.* FROM `comments` 
INNER JOIN `articles` ON `articles`.`id` = `comments`.`article_id` 
WHERE `articles`.`published` = 1 
ORDER BY comments.id DESC 
LIMIT 3

Surprising Sluggishness!

All goes well for a few months. You post a bunch of articles, some of which end up with huge comment threads, because you’re a master of your craft (or a master troll). Next thing you know, you’ve got 100 articles, around 20,000 comments, and a big performance problem. That simple query you wrote above is taking over half a second to execute, every single time. All of that time, just to retrieve 3 records? Something has got to be wrong with that. You fire up an EXPLAIN, and you see this in your query plan:

Using where; Using temporary; Using filesort

A temporary table and a filesort? Really?? Then you notice that the query planner is starting off with the articles table and using the comments table’s index on article_id for the join. You wonder for a moment if there is any likely scenario in which this is the most performant optimization plan for this query, given the ORDER and LIMIT you supplied.

Subselect or Supplementation

The fix is easy enough. You can split things into two queries, either using something like Valium to get the valid article_ids first, then eliminating the need for the join, or (preferably) using a subquery in the DB:

# Rails 3.1 only (Or with Squeel)
Comment.where(:article_id => Article.where(:published => true)).
        order('comments.id desc').
        limit(3)

SELECT  `comments`.* 
FROM `comments`  
WHERE `comments`.`article_id` 
  IN (
    SELECT `articles`.`id` 
    FROM `articles`  
    WHERE `articles`.`published` = 1
  ) 
ORDER BY comments.id DESC 
LIMIT 3

An even easier and more performant way, in this case, is to just give MySQL a nudge in the right direction:

Comment.from('comments use index (PRIMARY)').
        joins(:article).
        where(:articles => {:published => true}).
        order('comments.id desc').limit(3)

SELECT  `comments`.* 
FROM comments USE INDEX (PRIMARY)
INNER JOIN `articles` ON `articles`.`id` = `comments`.`article_id` 
WHERE `articles`.`published` = 1 
ORDER BY comments.id DESC 
LIMIT 3

Rage Rising

It’s great that we have these workarounds available, but I find it more than a little disappointing that the results of this benchmark script comparing MySQL, PostgreSQL, and SQLite (yes, SQLite!) look like this:

                user     system      total        real
mysql       0.020000   0.010000   0.030000 (  0.617876)
sqlite      0.000000   0.000000   0.000000 (  0.001234)
postgresql  0.000000   0.000000   0.000000 (  0.002423)

If the query being described were something abnormal, I’d understand. I expect to need hints and goofy hacks when optimizing an edge case. Not when doing something as simple as outlined above.

Anyway, I hope the above explanation helps you implement workarounds in your own queries, or maybe gives you the last nudge you need to finally get around to checking out PostgreSQL. :)