Strictly Untyped

Foreigner comes full circle

2009-09-02T22:46:00.000-07:00

After actively developing multiple Rails projects over the last three years, I've learned when to not fight the framework, and I've also learned when it's appropriate to roll up your sleeves and write some Ruby code. I've stopped using composite primary keys, I've stopped attempting to namespace models, I always use the default "id" for a primary key, the list goes on.

One thing my mind has not changed on is using foreign keys. I use them, and I've finally completed Foreigner, a foreign key migration library that meets my initial requirements:

Do not change existing behavior of the application
Perform a no-op when the adapter does not support foreign keys
Follow similar API as 'add_index'
Dump foreign keys into schema.rb

Do not change existing behavior of the application

I found numerous existing foreign key plugins that attempted to automagically add foreign keys. For example, automatically adding foreign keys based on column names, or automatically adding associations to active record models base on foreign keys in the database. I believe both of these are mistakes. Automatically adding foreign keys changes existing migrations, and always seems to break things. Automatically creating active record associations puts too much guesswork into play. There's a difference between using foreign keys to enforce integrity (good), and using foreign keys to change the behavior of the application (bad).

Lastly, I thought long and hard about whether calls to t.references should automatically add a foreign key. I finally resolved on not doing this, and compromised by adding a new :foreign_key option. For example:

change_table :comments do |t|
  t.references :author, :foreign_key => true
end

Perform a no-op when the adapter does not support foreign keys

While this does not help me, it is required to support adapters such as sqlite3. The AbstractAdapter implements all necessary methods with a no-op. With Foreigner installed, you can develop with sqlite3 and run production with MySql.

Follow similar API as 'add_index'

Adding foreign keys isn't much different than adding indexes, and everyone is familiar with using add_index. You add_foreign_key, remove_foreign_key and can even use change_table.

Dump foreign keys into schema.rb

The final feature I added to Foreigner is dumping foreign keys to schema.rb, so that you can avoid using the SQL structure. Similar to how indexes are added to schema.rb, Foreigner inspects the database and determines the foreign keys.

In conclusion...

There's been a lot of discussion in the past about whether or not you should be using foreign keys. This shows to me is that there is a large group of people who probably want this available out of the box with Rails. The library was designed to not affect those who don't want them; the primary purpose in doing so was to show that everyone can have it their own way. My next step is turning this into a gem.

Get the source: http://github.com/matthuhiggins/foreigner/tree/master.

Scopes for conditions

2009-05-14T10:31:00.001-07:00

It is common for your named and anonymous scopes to only specify the :conditions option. In RailsCast #112, Ryan Bates suggests adding a conditions named_scope:

This allows you to replace:

active_people = People.scoped(:conditions => {:active => true})

with:

active_people = People.conditions(:active => true)

A step further

This can be taken a step further by adding a class method, named_conditions, to ActiveRecord::Base:

Now you can replace:

class Post
  named_scope :recent, lambda { {:conditions => ['created_at > ?',  1.day.ago] } }
end

with:

class Post
  named_conditions :recent, lambda { ['created_at > ?',  1.day.ago] }
end

Not very exciting, but it's the little things that count. You can download the module at http://gist.github.com/111783.

A Foreign Key migration plugin for Rails

2009-04-16T14:26:00.000-07:00

I use foreign keys in my Rails projects, and they work great. There was a point in time where Rails tests did not work well when using foreign keys, but those days are long gone. I've been hand rolling foreign key migrations for quite some time, and decided to write a plugin that covers most of my needs. This resulted in Foreigner, which is now available at http://github.com/matthuhiggins/foreigner/tree/master.

One of my goals was to follow the API for add_index and remove_index. Another goal was to make sure migrations did not break when run against adapters that do not support foreign keys. The API simply no-ops by default, and only supported adapters perform the operation. (MySql is currently the only implemented adapter.)

You should be able to add most of your foreign keys with following (See the Readme for more details.)

add_foreign_key(:comments, :posts)

If you drank the Rails lemonade and don't think foreign keys are necessary, you're correct. They're not necessary, they are a nice to have.

A Survey of URL Sharing Debris

2009-03-21T15:42:00.000-07:00

As a loyal follower of Tufte, I believe in removing all unnecessary junk and debris from the visual design of a website. (The layout for this blog is a preselected template from Blogger; don't hold it against me). I'm going to point out a trend I'm noticing everywhere, and you will either start being irritated by it, or disagree with me. I call it URL Sharing Debris, and it is best described with examples. I surfed to random websites to collect a few samples.

huffingtonpost.com

Here's your classic example. Since it's too hard to copy and paste a URL, the user is provided with approximately 8-10 icons to share the URL.

indecisionforever.com

At least they realize the icons detract from the content and gray them out.

www.dailymail.co.uk

Since most users won't know what the icons mean, this example adds labels.

youtube.com

More space is wasted on the sharing section than on the video control. Am I the only one who knows how to copy and paste the URL displayed in my browser?

failblog.org

Finally, a site that makes fun of the sharing links.

scm.jadeferret.com

In my quest to find the worst offender, I came across the above winner.

Final words

I think a lot of site developers responsible for this are simply copying popular sites who started this trend. Putting a Digg button on your site does not make it popular. It needs to be popular in the first place.

It is difficult to fit content on a page. It is difficult to design clean aesthetics. I believe this fad will eventually die, and join the graveyard with webrings.

5 new CSS features you can use if you drop IE6

2009-03-18T19:54:00.001-07:00

It is time for most sites to drop IE6, especially for projects that are just getting started. If you do make this choice, it is important to realize that there are some new CSS features that you can begin using, and some old habits worth dropping. While there are many sites that describe CSS bugs in IE6, I found no websites that describe working examples to improve your CSS and HTML if you simply drop support for it.

Here are 5 examples from my own project, Haiku Village.

1. Attribute Selectors

You can use attribute selectors for any arbitrary attribute, but I have used it exclusively for form elements.

<style>
input[type="submit"] {
  margin-top: 5px;
}
</style>

<input type="submit" value="Save" />

This will help you delete those arbitrary class names you made up to select certain types of inputs.

2. Multiple Classes on a Single Element

This is best described with an example:

<style>
.icon {
  background: transparent url(/images/icons.png) no-repeat scroll 0 0;
  padding: 2px 0 0 20px;
}

.icon.trash {
  background-position: 0 -80px;    
}
</style>

<span class="icon trash">Delete</span>

With IE6, you either needed to make a class name of "icon-delete", or nest one element inside of the other.

3. Sibling Selectors

There are many ways to use sibling selectors. My most recent use of them is to add borders between the list items in a sub menu:

<style>
ul.menu li + li {
  border-left: 1px dotted #DADADA;
  padding-left: 8px;
}
</style>

Normally, you would need to introduce a "first" or "last" class name to make the borders only appear between items, and not on the outside.

3. Child Selectors

If you know that a certain style should only apply immediately within an element, then simply insert a ">" into your stylesheets:

<style>
ul.menu > li {
    float: left;
    margin-right: 7px;
    padding: 3px 0 1px;
  }
</style>

This will ensure that other list items within this list item do not receive the style. It also prevents the necessity to define a new class name, such as "menu-item"

4. Use :hover on any element

I started web development in 1995, and until recently, I never considered using the :hover meta selector for anything besides the "a" tag. Remove this from your mind, because it is no longer the case. I use :hover to expose some icons when you mouse over a haiku:

<style>
.haiku .actions {
  visibility: hidden;
}

.haiku:hover .actions
  :visibility visible
}
</style>

Most of us have probably worked around this by either introducing an arbitrary "a" element with an "href='#'", or by adding a class name during a 'mouseover' event.

5. New properties available

There are plenty of new properties that you can start using. The most significant are:

min-width/min-height
max-width/max-height
background-attachment - You can use a value of 'fixed'

There are hacks to make all of these work in IE6, but that's the point: You should stop the hacking. Also, did I mention transparent PNGs?

A matter of style...

2009-01-18T22:40:00.000-08:00

I've seen plenty of code examples that show how to set the current user or subdomain to a variable. For example:

class ProfileController < ActionController::Base
  before_filter :set_current_user
  
  private
    def set_current_user
      @current_user = User.find_by_username(session[:username]) if session[:username]
    end
end

However, I prefer this way:

class ProfileController < ActionController::Base
  private
    def current_user
      @current_user ||= User.find_by_username(session[:username]) if session[:username]
    end
    helper_method :current_user
end

With this technique, I always use the current_user method to determine the current user, and never reference the @current_user variable. If an action doesn't need to know the current_user, no time is wasted to set it.

I killed ApplicationController

2008-11-26T16:16:00.001-08:00

For the longest time, developers have asked for application.rb to be renamed to application_controller.rb. This is finally fixed in RailsEdge, and there's even a Death of Application.rb blog posting about it.

That's great, but none of this affects me. Why? Because ApplicationController always annoyed me, so I killed it.

Life without ApplicationController

You might ask, "how do you survive without this loving, parental figure; the ApplicationController?" Because my parent is ActionController::Base, in the same way that most of your models inherit from ActiveRecord::Base. Can you tell me why your models don't inherit from an arbitrary ApplicationRecord class?

The primary reason why a developer puts code into ApplicationController is for code reuse and changing default behavior. The ultimate result is that this class becomes a dumping ground for any common methods required across two or more controllers. Ruby has a better pattern for reusing methods and changing behavior across classes: Modules.

I create my own controller modules in the lib folder, and include them into ActionController::Base. If I want only certain controllers to be affected, the module defines a class level method, similar to the before_filter and has_many macros. For example, some controllers require a login. Rather than adding a filter to ApplicationController, I would make a 'require_login' module. My controllers could optionally use it like this:

123	class PostController < ActionController::Base require_loginend

Why do I go through all this trouble? Because it (1) cuts one layer out of the inheritance tree, and (2) forces me to think about packaging functionality into small, testable modules. I encourage you to look at the code in your ApplicationController, and question whether any or all of your controllers should be inheriting from it.

I am not dead

2008-11-21T21:25:00.000-08:00

To the dedicated numbers subscribing to my blog: I'd like to notify everyone that I still exist. I just awakened from a temporary, semi-comatose state. Thankfully, I have numerous blog posts queued in my non-volatile persistence mechanism.

Please stand by for tips in perfectionism with HTTP, SQL and Ruby on Rails.

Ruby 1.8.7's Enumerator Class

2008-09-15T22:38:00.000-07:00

Early in my Ruby career, I found myself needing access to the index while using Enumerable methods such as map and select. I quickly discovered the obscurely documented enum_for method:

%w(fun with enumerator).enum_for(:each_with_index).map { |value, i| "#{i+1}. #{value}" }

Since MacPorts is now on Ruby 1.8.7, I read the 1.8.7 summary log and noticed a bunch of existing methods that now "Return an enumerator if no block is given." I soon realized that the above can now be written as:

%w(fun with enumerator).each_with_index.map { |value, i| "#{i+1}. #{value}" }

It's a seemingly simple and minor improvement, but I think this change will encourage developers to use Enumerator more often with Ruby. For example, the enum_for method is never used in Rails 2.1. Contrarily, I bet that the new "Return an enumerator if no block is given" change will be used in Rails (eventually).

Arbitrary Uses

Since Integer#downto returns an Enumerator, let's write factorial:

n.downto(1).inject(:*)

Why not sort the letters in a word? Let's combine each_char with sort:

"rails".each_char.sort

Since Enumerator is just an object, we can pass it around, and customize how other methods behave. For example, here I determine whether the items are processed in ascending or descending order:

def do_stuff(enumerator)
  # stuff
end
do_stuff([1,3,5].reverse_each)
do_stuff([1,3,5].each)

All of this fun can be added to prior versions of Ruby by opening classes. I think this might work...

class Array
  [:find, :find_all, :partition, :reject, :select, :sort_by].each do |method|
    class_eval <<-EOV
      def #{method}_with_enumerator(*args)
        if block_given?
          #{method}_without_enumerator(*args)
        else
          Enumerable::Enumerator.new(self, :enumerator, *args)
        end
      end
    EOV
    alias_method_chain method
  end
end

Please comment and let me know if there are better uses for this change!

Obsessing over URLs

2008-08-29T18:47:00.000-07:00

One of the most important components of a website is URL design. I have obsessed over URLs since the beginning of time, and REST caused me to focus on them even more. Here's a classic example of a clean URL:

www.strictlyuntyped.com/articles/1/comments/1

This seems quite ordinary by today's standards. But admit it, a few years ago us web developers were getting really excited about this stuff.

What was the original problem?

Let's start from the beginning. The 'dirty' version of the above URL:

www.strictlyuntyped.com/read?article_id=1&comment_id=1

One of the most notable problems with this type of URL is that the user sees it. The URL is an important piece of the user experience, since (1) it is displayed at the top of the browser window, (2) users determine where they are on a site by the URL, and (3) they type in and share URLs, so they must be 'readable'. Essentially, no information is provided to the user about the structure and/or hierarchy of your site.

The second problem with dirty URLs are developer facing. The application code pretty much consists of a function that takes in a hash. A Ruby analogy would be the following method:

def read(options={})

The method has zero self-documentation, and a comment would be required to determine what options it expects. Our world here is essentially a one dimensional array of methods that take a hash.

As a comparison, here is what I believe the Rails analogy of the aforementioned clean URL would be:

Article.find(article_id).comments.find(comment_id)

The idea of introducing structure into a line of characters is a simple, yet profound, concept. It seems to have become a requirement for any 'Web 2.0' site.

Don't go overboard with clean RESTful URLs

Clean URLs and REST works incredibly well for content managing systems such as blogs. Without coincidence, Rails works very well with these types of systems. In other cases, I see myself, and others, forcing a square through a round hole when it comes to URL design. Here are two mistakes that myself, and probably others, have made.

Mistake #1: Overcomplicated Hierarchies

There exists an implicit hierarchy to clean URL design; an article having comments is a good example. On the other hand, there is no reason to create deeply nested hierarchies for the sake of representing the real world. the { buckblogs :here } says not to nest resources more than one level deep from a programmatic point of view. I completely agree, and amend his reasons from an information design perspective. If you were to make a website about animals, would the urls represent biological classifications? Here is what the URL for a lion will look like:

/kingdom/animalia/phylum/chordata/class/mammalia/order/carnivora/family/felidae/genus/panthera/species/lion

The point is, flatten your hierarchy and make Tufte proud!

Mistake #2: Unordered Parameters

Oftentimes, the input required for a URL does not fit in a hierarchy. A great analogy is a Ruby method. Think of how many methods in Ruby on Rails use the following format:

def some_method(foo, bar, options = {})

This method makes the options optional, and they can be passed in without any specific order. Why not leverage these same benefits with a URL? With that in mind, there is nothing wrong with the following URL:

www.strictlyuntyped.com/articles/1/comments?order=date&rating=4

How else would Google Maps make locations sharable?

To wrap up my ramblings, don't get stuck with your project obsessing over how perfect your URLs are. Keep the hierarchy simple, and use a hash of parameters when you need it.

ActiveRecord: Can haz namespaces?

2008-08-14T09:40:00.001-07:00

There are plenty of blog postings from has_many :through, err the blog and Pratik's blog describing how to completely avoid namespaced models. I unfortunately discovered these sites only after struggling with namespaces myself.

If namespaces were completely unsupported by ActiveRecord, I would leave it at that. Contrarily, namespaces are moderately supported, but lead first timers to the spooky edges of the Rails framework.

After giving it some thought, I determined that the best way to describe the state of namespacing in ActiveRecord is to act as a first time novice, pretending to use them for the first time!

The Requirements

I am an intern working for a company that has built their entire website with Ruby on Rails. My boss wants me to add statistics tracking similar to Google Analytics, and he wants it all to go into the Statistics:: namespace. The initial requirements include tracking the milliseconds that each HTTP request took to complete, how long users remained logged into their session, and which session each request belonged to.

This sounds like a really easy project, and I want to impress my boss by using all the best practices and conventions of Rails.

Table Name Surprise

I will start with a model that stores the length of each request. It makes sense to use a Rails script to generate Statistics::Request:

ruby script/generate model Statistics::Request milliseconds:integer
    create  app/models/statistics
    create  test/unit/statistics
    create  test/fixtures/statistics
    create  app/models/statistics/request.rb
    create  test/unit/statistics/request_test.rb
    create  test/fixtures/statistics_requests.yml
    create  db/migrate
    create  db/migrate/20080814034925_create_statistics_requests.rb

The generator conveniently created a statistics folder inside both app/models and test/unit. I am a little bit confused as to why the folder test/fixtures/statistics was created, when the actual fixture was placed in test/fixtures/statistics_requests.yml, but I will figure that out later.

The table name in the migration is statistics_requests, so I immediately assume that table names include the namespace. After migrating the database, I fire up the console to try out the model:

12	>> Statistics::Request.create(:milliseconds => 10)ActiveRecord::StatementInvalid: Could not find table 'requests'

How strange, I thought the table's name is statistics_requests. Just to make sure:

12	>> Statistics::Request.table_name=> "requests"

Is this a bug in Rails? Apparently not. I guess it's time to do some dirty monkey patching:

ActiveRecord::Base.class_eval do
  def self.table_name
    name.split("::").map { |package| package.underscore.pluralize }.join("_")
  end
end

Good, now the model can find its table. Time to write some fixtures and tests.

Tests are looking good

Feeling confident after getting my namespaced model to work, it would be nice to make sure the the new test runs:

123	>> rake test:units... "test/unit/statistics/request_test.rb"1 tests, 1 assertions, 0 failures, 0 errors

Sweet - It looks like tests are discovered recursively in the test/unit folder.

Associations

I need to add the Statistics::Session model, and also update Statistics::Request to belong to a session.

The migration:

create_table :statistics_sessions do |t|
  t.datetime :start, :end
end

change_table :statistics_requests do |t|
  t.integer :session_id
end

The model:

123	class Statistics::Request < ActiveRecord::Base belongs_to :sessionend

I use the console to test out the association:

>> session = Statistics::Session.create(:start => 4.hours.ago, :end => Time.now)
=> #<Statistics::Session id: 1, start: "2008-08-14 00:42:00", end: "2008-08-14 04:42:00">
>> request = Statistics::Request.create(:milliseconds => 10, :session => session)
ArgumentError: Statistics is not missing constant Session!

This is a bizarre error. It turns out that Rails can't find Statistics::Request from Statistics::Session. I finally figured out that I need to do this:

belongs_to :session, :class_name => "Statistics::Session"

It would make more sense if I only had to specify :class_name if it existed in a different namespace.

Namespaced Fixtures = WTF?

Development and testing will be easier if I generate some fixture data. Remember when the model generator script created an unused folder?

123	create test/fixtures/statistics...create test/fixtures/statistics_requests.yml

Based on the other conventions, it seems like this file belongs in test/fixtures/statistics/requests.yml. I'll move it there, and load up some fake data:

rake db:fixtures:load

It will be fun to play around with all that fixture data:

12	>> Statistics::Request.count=> 0

How bizarre, the data did not even get loaded. I soon find out that unlike unit tests, fixture files are not recursively loaded. The file can't be moved after all.

I need to get the request -> session association working in the fixtures too. Let me edit statistics_sessions.yml and statistics_requests.yml:

long_session:
  start: 2008-08-13 21:32:32
  end: 2008-08-13 21:32:32

short_request:
  milliseconds: 1
  session: long_session

Unfortunately, this does not work:

123	rake db:fixtures:loadrake aborted!SQLite3::SQLException: table statistics_requests has no column named session...

For some reason, the label referencing is not working with my namespaced models. I guess I need to hand hold Rails on this one:

123	short_request: milliseconds: 1 session: <%= Fixtures.identify(:long_session) %>

Observers - OMG they work!

Feeling a little down after my battle with fixtures, I decide to make a request observer that logs whenever we get a request:

ruby script/generate observer Statistics::Request
  exists  app/models/statistics
  exists  test/unit/statistics
  create  app/models/statistics/request_observer.rb
  create  test/unit/statistics/request_observer_test.rb

Looking good so far. Let me implement the class:

class Statistics::RequestObserver < ActiveRecord::Observer
  def after_create(request)
    puts "We got a hit!!!"
  end
end

I also won't forget to add the following line to environment.rb:

config.active_record.observers = 'statistics/request_observer'

And viola, it all works:

12	>> Statistics::Request.create(:milliseconds => 10)We got a hit!!!

In Summary

Ok, I'm done playing stupid. My final word is that namespaced models can be made to work, but there are some rough edges. Some final tips for you:

Don't have conflicting namespaces and class names. For example, I should not introduce a class named Statistics in the above example. Otherwise, I will end up seeing the following message: "warning: toplevel constant Statistics referenced by Statistics::Statistics".
Even when referencing other classes in the same namespace, it is best to always include the namespace prefix. Otherwise, you will run into issues with conflicting class names across different namespaces and Rails' automagical class loading.

Some things that Rails should do:

All APIs in Rails that support defining classes by Symbols should also support Strings. I was fortunate that I could pass in 'statistics/request_observer' to active_record.observers. However, other methods such as ActiveRecord::Observer.observe do not support strings.
I believe that there is enough reason to support prefixing database table names with the namespace. This is as simple as introducing a new option to ActiveRecord called 'include_namespace_in_table_name'.
Fix the fixtures.

Googtaculous, now with SSL support

2008-08-09T19:55:00.000-07:00

This is just a short note. Google recently added SSL support to their AJAX Libraries API. Googtaculous will now automatically use https if the incoming request comes in over SSL. (http://github.com/matthuhiggins/googtaculous/tree/master).

Browsers show warnings if a website includes both http and https requests. A few readers asked for this addition.

In place Array methods for Prototype

2008-08-05T10:04:00.000-07:00

Unlike Ruby, Prototype does not offer in place iterators for Array methods such as compact, map and reject. I started down the road of implementing these methods, and discovered significant improvements with large arrays. Since Javascript does not support the "!" character in function names, I decided to name the in place versions with underscore prefixes.

map

_map is simple to implement. We replace the value in each index with the result of the iterator function:

Array.prototype._map = function(iterator, context) {
  iterator = iterator ? iterator.bind(context) : Prototype.K;
  for(var index = 0, length = this.length; index < length; index++) {
    this[index] = iterator(this[index], index);
  }
  return this;
};

In this case, and the remaining ones, this is returned so that the methods can be chained.

reject

For in place _reject, I scan the array once from left to right. Elements that are not rejected are placed in the array from left to right, and finally the length is truncated. I believe it is important to maintain the original order of the elements:

Array.prototype._reject = function(iterator, context) {
  iterator = iterator ? iterator.bind(context) : Prototype.K;
  var position = 0;
  for (var index = 0, length = this.length; index < length; index++) {
    if (!iterator(this[index], index))
      this[position++] = this[index];
  }
  this.length = position;
  return this;
};

compact

_compact is very similar to reject:

Array.prototype._compact = function(iterator) {
  var position = 0;
  for (var index = 0, length = this.length; index < length; index++) {
    if (this[index] != null)
      this[position++] = this[index];
  }
  this.length = position;
  return this;
};

The above implementation can be shortened by using _reject, but it significantly reduces performance. For completeness, that might be written as:

return this._reject(function(value) { return value == null; });

Results

These are results in Firefox 3, using an array of 100,000 numbers and null values. Changing the size did not seem to affect the relative performance.

Method	Prototype Time	In Place Time
map	159ms	42ms
reject	158ms	45ms
compact	161ms	13ms

One should probably do a more comprehensive test, but these results are enough to show that performance gains can be made with my faster versions. In addition, less memory is being used.

Time for protodate

2008-07-29T01:51:00.000-07:00

I decided to stay up late and implement Ruby's Time.strftime in Javascript. Prototype's Template class ended up making this easy to implement. To clarify, here is how you might use this function.

1 new Date().format("Printed on %m/%d/%Y at %I:%M%p")
2 => "Printed on 07/29/2008 at 12:31AM"

I checked my code into github, so take at your own peril. (http://github.com/matthuhiggins/protodate/tree/master)

And yes, I know that this has already been blogged about at alternateidea. However, that particular solution converts all values to UTC, whereas I almost always want to use the browser's local time zone when prettying up dates.

MySql Lovin' Part 3: Summing up with group_concat

2008-07-22T09:35:00.000-07:00

Ruby on Rails provides some handy methods for using SQL aggregate functions:

count
average
minimum
maximum
sum

While this is a good start, MySql has additional functions that are not included in this list. I will show an example using group_concat.

group_concat

If you recall from my post about on duplicate key update, we had a Post model that included a virtual attribute called tag_list:

1 class Post < ActiveRecord::Base
2   ...
3   def tag_list
4     self.tags.map(&:name).join(', ')
5   end
6 end

This code retrieves all tags for the post, creates Tag objects, maps that to an array of strings, and joins them together. While this code looks nifty, it is performing a hefty amount of work for the task.

Thankfully, MySql has a group_concat function, and Rails exposes a calculate method to perform arbitrary aggregate functions by name. Without further ado, here is how we can use group_concat:

1 def tag_list
2   self.tags.calculate(:group_concat, :name)
3 end

The separator defaults to ','. If you want to change it, sneak in a separator command:

1 self.tags.calculate(:group_concat, "name separator ' - '")

group_concat will be significantly faster than the original method.

Another group of underused aggregate functions are bit_and, bit_or, and bit_xor. If you have a many-many association between a model and an enumerated set of 64 or less items, this information can be stored in a single BIGINT field on the model.

This wraps up my unglamorous MySql series. If I saved one clock cycle on one database server, then I claim success.

MySql Lovin' Part 2: Add New Data Types To Migrations

2008-07-12T11:17:00.000-07:00

This is a follow up to last week's MySql Lovin Part 1.

You will eventually want to use MySql specific data types. While arbitrary SQL statements can be executed in migrations, an alternative is to extend the MySql adapter to support new types of columns.

Two examples of columns I use are:

BIGINT. Rails 2.1 attempts to use the :limit option to predict you want a BIGINT, but there seems to be confusion in Rails source about what the N means when you write BIGINT(N). Nevertheless, BIGINT always has the same maximum and minimum limit, and if I want to use one, I like to be explicit.

Primary Key without auto increment. Sometimes I have enums defined in Ruby code, and want to explicitly define their value when inserting into the database, rather than having their id value automatically incremented by the database.

Let's add support for these columns, so that they can be used with create_table. First, have a look at native_database_types, defined in ActiveRecord::ConnectionAdapters::MysqlAdapter. Rails provides no facility to register more column types with an adapter, but that does not prevent merging them into the column type hash:

 1 module ActiveRecord
 2   module ConnectionAdapters
 3     class MysqlAdapter < AbstractAdapter
 4       alias :original_native_database_types :native_database_types
 5 
 6       def native_database_types
 7         original_native_database_types.update(
 8             :big_integer  => { :name => "bigint", :limit => 21 },
 9             :manual_pk    => "int(11) NOT NULL PRIMARY KEY")
10       end
11     end
12   end
13 end

With the new types added, they can be used in migrations:

1 create_table :foo, :id => false do |t|
2   t.column :id, :manual_pk
3   t.column :bytes, :big_integer
4 end

Shorthand Columns
To make these migrations a bit more appealing, extend ActiveRecord::ConnectionAdapters::TableDefinition to support shorthanded column names:

 1 module ActiveRecord
 2   module ConnectionAdapters
 3     class TableDefinition
 4       %w(big_integer manual_pk).each do |column_type|
 5         class_eval <<-EOV
 6           def #{column_type}(*args)
 7             options = args.extract_options!
 8             column_names = args
 9 
10             column_names.each { |name| column(name, '#{column_type}', options) }
11           end
12         EOV
13       end
14     end
15   end
16 end

Refactor the migration:

1 create_table :foo, :id => false do |t|
2   t.manual_pk :id
3   t.big_integer :bytes
4 end

Adding Native Support to Rails
It would be nifty if Rails allowed you to perform the above in an initializer (very similar to how you register mime_types). For example:

1 ActiveRecord::ConnectionAdapters::MysqlAdapter.register :big_integer, { :name => "bigint", :limit => 21 }
2 ActiveRecord::ConnectionAdapters::MysqlAdapter.register :manual_pk, "int(11) NOT NULL PRIMARY KEY"

I might follow this up with a Rails patch, but need to think about it more. Also, the weather is too nice for me to continue blogging.

MySql Lovin' Part 1: On Duplicate Key Update

2008-07-04T15:06:00.000-07:00

By coding through the ActiveRecord abstraction, we lose close ties to SQL. This is a major selling point for Rails, but it should not be why you are sold on Rails.

Avoiding vendor specific database features is silly, and doing so does not make you a better Rails developer. Database neutrality has never helped me because I have yet to swap database vendors. If I did need to swap, it would be due to a vendor specific feature. ActiveRecord is great for the majority of operations, but you should not live exclusively within its constraints. And so I begin the MySql Lovin' series...

Perfect Rails Code
Start with a simple data model for a blog. A post has text, and can have zero or more tags. Here is what the migration looks like:

 1 create_table :posts do |t|
 2   t.text :text
 3   t.timestamps
 4 end
 5  
 6 create_table :post_tags do |t|
 7   t.integer :post_id, :tag_id, :null => false
 8 end
 9 
10 create_table :tags do |t|
11   t.string :name
12   t.string :post_tags_count
13   t.timestamps
14 end
15 
16 add_index :tags, :name, :unique => true

The tags table also keeps track of the total posts per tag.

The models for the blog might look like this:

 1 class Post < ActiveRecord::Base
 2   has_many :post_tags
 3   has_many :tags, :through => :post_tags
 4   
 5   def tag_list=(tag_string)
 6     self.tag = tag_string.split(',').map { |tag_name| Tag.find_or_create_by_name(tag_name.squish!) }
 7   end
 8   
 9   def tag_list
10     self.tags.map(&:name).join(', ')
11   end
12 end
13 
14 class PostTag < ActiveRecord::Base
15   belongs_to :tag, :counter_cache => true
16   belongs_to :post
17 end
18 
19 class Tag < ActiveRecord::Base
20 end

Notice that the Post model has a virtual tag_list attribute. This automatically splits a comma delimited list of tags, creates new tags if necessary, and generates the associations between posts and tags.

Too Many Queries
Fill out the form:

Now check the query log to see what queries hit the database:

 1 SELECT * FROM tags WHERE (tags.name = 'latin') LIMIT 1
 2 INSERT INTO tags (name, post_tags_count, created_at, updated_at) VALUES('latin', NULL, '2008-07-04 22:41:28', '2008-07-04 22:41:28')
 3 SELECT * FROM tags WHERE (tags.name = ' illegible') LIMIT 1
 4 INSERT INTO tags (name, post_tags_count, created_at, updated_at) VALUES(' illegible', NULL, '2008-07-04 22:41:28', '2008-07-04 22:41:28')
 5 SELECT * FROM tags WHERE (tags.name = ' rails') LIMIT 1
 6 INSERT INTO tags (name, post_tags_count, created_at, updated_at) VALUES(' rails', NULL, '2008-07-04 22:41:28', '2008-07-04 22:41:28')
 7 INSERT INTO posts (created_at, updated_at, text) VALUES('2008-07-04 22:41:28', '2008-07-04 22:41:28', 'Lorem ipsum dolor sit amet...')
 8 INSERT INTO post_tags (tag_id, post_id) VALUES(1, 1)
 9 SELECT * FROM tags WHERE (tags.id = 1) 
10 UPDATE tags SET post_tags_count = post_tags_count + 1 WHERE (id = 1) 
11 INSERT INTO post_tags (tag_id, post_id) VALUES(2, 1)
12 SELECT * FROM tags WHERE (tags.id = 2) 
13 UPDATE tags SET post_tags_count = post_tags_count + 1 WHERE (id = 2) 
14 INSERT INTO post_tags (tag_id, post_id) VALUES(3, 1)
15 SELECT * FROM tags WHERE (tags.id = 3) 
16 UPDATE tags SET post_tags_count = post_tags_count + 1 WHERE (id = 3)

16 queries. Gross, let's try to do better.

On Duplicate Key Update
MySql's INSERT ... ON DUPLICATE KEY UPDATE is a perfect solution to reduce the number of queries. It can be used to:

Insert non-existing tags
Update the counter cache for posts per tag.
Remove the need to instantiate a Tag object for each tag.

Let's give it a try:

 1 class Post < ActiveRecord::Base
 2   has_many :post_tags
 3   has_many :tags, :through => :post_tags
 4   
 5   def tag_list=(tag_string)
 6     self.tag_ids = tag_string.split(',').map { |tag_name| create_or_update_tag(tag_name.squish!) }
 7   end
 8   
 9   def tag_list
10     self.tags.map(&:name).join(', ')
11   end
12   
13   private
14     def create_or_update_tag(name)
15       statement = %{
16         INSERT INTO tags(name, post_tags_count, created_at, updated_at)
17         VALUES (#{connection.quote(name)}, utc_timestamp, utc_timestamp, 1)
18         ON DUPLICATE KEY UPDATE
19           updated_at = utc_timestamp,
20           post_tags_count = post_tags_count + 1,
21           id=LAST_INSERT_ID(id)
22       }.squish!
23       
24       connection.insert_sql(statement)
25     end
26 end
27 
28 class PostTag < ActiveRecord::Base
29   belongs_to :tag
30   belongs_to :post
31 end
32 
33 class Tag < ActiveRecord::Base
34 end

That SQL code sure looks ugly. It probably belongs in the Tag class, and can be generated automatically from the model's attributes. What if I made a plugin that added create_or_update to ActiveRecord::Base?

More importantly, the number of queries have decreased:

1 INSERT INTO tags(name, post_tags_count, created_at, updated_at) VALUES ('latin', utc_timestamp, utc_timestamp, 1) ON DUPLICATE KEY UPDATE updated_at = utc_timestamp, post_tags_count = post_tags_count + 1, id=LAST_INSERT_ID(id)
2 INSERT INTO tags(name, post_tags_count, created_at, updated_at) VALUES ('illegible', utc_timestamp, utc_timestamp, 1) ON DUPLICATE KEY UPDATE updated_at = utc_timestamp, post_tags_count = post_tags_count + 1, id=LAST_INSERT_ID(id)
3 INSERT INTO tags(name, post_tags_count, created_at, updated_at) VALUES ('rails', utc_timestamp, utc_timestamp, 1) ON DUPLICATE KEY UPDATE updated_at = utc_timestamp, post_tags_count = post_tags_count + 1, id=LAST_INSERT_ID(id)
4 SELECT * FROM `tags` WHERE (`tags`.`id` IN (1,2,3)
5 INSERT INTO `posts` (`created_at`, `updated_at`, `text`) VALUES('2008-07-04 23:17:20', '2008-07-04 23:17:20', 'Lorem ipsum dolor sit amet...')
6 INSERT INTO `post_tags` (`tag_id`, `post_id`) VALUES(1, 1)
7 INSERT INTO `post_tags` (`tag_id`, `post_id`) VALUES(2, 1)
8 INSERT INTO `post_tags` (`tag_id`, `post_id`) VALUES(3, 1)

This is a good start. The number of queries has decreased by a 2x factor, and Tag objects are no longer being created.

By alas, what is that query on line 4 for? It looks like Rails runs this query after assigning to tag_ids. This can be avoided by directly creating the entries for post_tags, but I can accept this annoyance for today.

Refactoring block helpers with more block helpers

2008-06-27T17:04:00.000-07:00

This post shows how to build helpers around create_table and form_for, which are my two most commonly used block helpers in Rails. This stuff is obvious to a Rubyist, but not everyone is a Rubyist.

create_table

If you reuse the same columns between tables, your migrations can end up containing a lot of duplicate code. For example, let's say you are making a content management system for your office supply company. Since each product is sufficiently different, you want to make a different ActiveRecord model for each product.

As a result, your migration to add products will look like this:

 1 create_table :tapes do |t|
 2   t.string :product_name
 3   t.integer :category
 4   t.integer :length, :width
 5   t.float :price
 6   t.integer :total_sold, :total_in_stock, :default => 0
 7   t.timestamps
 8 end
 9 
10 create_table :staplers do |t|
11   t.string :product_name
12   t.integer :capacity, :type
13   t.string :color
14   t.float :price
15   t.integer :total_sold, :total_in_stock, :default => 0
16   t.timestamps
17 end

A polymorphic model can be used in this scenario. But what if you have 15 very unique products, and each needs their own indexing strategy? Nonetheless, that is not the point. The problem here is that duplicate columns exist between each table, and this can be refactored.

Simply create a new helper method create_product_table that wraps create_table:

 1 def self.create_product_table(*args)
 2   create_table(*args) do |t|
 3     t.string :product_name
 4     t.float :price
 5     t.integer :total_sold, :total_in_stock, :default => 0
 6     t.timestamps
 7     yield(t) if block_given?
 8   end
 9 end

Then use the helper to refactor your existing migration. It removes the duplication, and makes the code more readable:

 1 create_product_table :tapes do |t|
 2   t.integer :category
 3   t.integer :length, :width
 4 end
 5 
 6 create_product_table :staplers do |t|
 7   t.integer :capacity, :type
 8   t.string :color
 9 end

form_for

The solution for form_for is very similar to create_table.

In this arbitrary but typical example, your awesome web 2.0 website supports tagging of just about anything. Comments, Posts, Images, Movies, all taggable. That means all your forms need a "Tags:" input box.

Rather than duplicating the same code in each form for the tagging input, make a taggable_form_for helper:

1 def taggable_form_for(*args)
2   form_for(*args) do |f|
3     yield(f)
4     concat(f.label :tags, 'Tags')
5     concat(f.text_field :tags)
6   end
7 end

The forms are a bit tricky because you must call concat so that the html is emitted in the correct location.

Broken 'pre' tags with Blogger

2008-06-27T02:17:00.001-07:00

Sometime in the last week, a significant change was introduced into Blogger. Previously, any content inside of a <pre> section was left alone. Now, a <br/> tag is insterted for every line break within a <pre> tag, which results in a double line break. As a result, all of my code snippets have an extra line break.

I will write a more relevant post tomorrow. I just wanted to explain why my existing snippets look wrong.

Googtaculous

2008-06-24T08:57:00.000-07:00

To follow up on my post about Google's Ajax library API, I made a plugin that you can install to make your life easier.

1 script/plugin install git://github.com/matthuhiggins/googtaculous.git

The name is pretty clever, eh?

Clearing memcache without restart

2008-06-20T14:16:00.000-07:00

Up until now, I always restarted memcache to clear it. However, you can make a rake task to do so:

1 namespace :cache do
2   desc 'Clear memcache'
3   task :clear => :environment do
4     ActionController::Base.cache_store.clear
5   end
6 end

Drop this into /lib/tasks/cache.rake. Now just run the task:

1 rake cache:clear

This only works if you have configured action_controller.cache_store to be memcache.

Have a good weekend!

table_for @appetizers

2008-06-16T21:53:00.001-07:00

My First Plugin
I woke this morning realizing that I suck with git and have never made a public plugin. Finding this unacceptable, I set out today to make those truisms become falsehoods. The result is table_for, an ActionView plugin hosted on github.

The Distasteful Tables
You come up with a brilliant recipe site that focuses on appetizers. The idea seems perfect, but where will you put the appetizers? A table, of course!

Let's make an AppetizerController to retrieve those tasty morsels:

1 class AppetizerController < ApplicationController
2   def index
3     @appetizers = Appetizer.all
4   end
5 end

Then create a table in the view:

 1 <table>
 2   <tr>
 3     <th>Name</th>
 4     <th>Price</th>
 5     <th>Tastiness</th>
 6     <th>Chef</th>
 7     <th>Description</th>
 8   </tr>
 9
10 <% for appetizer in @appetizers %>
11   <tr>
12     <td><%=h appetizer.name %></td>
13     <td><%=h appetizer.price %></td>
14     <td><%=h appetizer.tastiness %></td>
15     <td><%=h appetizer.chef %></td>
16     <td><%=h appetizer.description %></td>
17   </tr>
18 <% end %>
19 </table>

Mmm, delicious. But wait, I see some duplication.

Can we get a table_for those appetizers?
Let's use table_for to trim down the code. First git it:

1 ruby script/plugin install git://github.com/matthuhiggins/table_for.git

Refactor the table:

1 <% table_for @appetizers do |t| %>
2   <% t.name %>
3   <% t.price %>
4   <% t.tastiness %>
5   <% t.chef %>
6   <% t.description %>
7 <% end %>

There we go. Much better. A week later, you find that your prices should be labelled with '$', descriptions are way too long, and the tastiness field needs to be right aligned. Not to mention, there are multiple tables on the page requiring different class names. Easy:

1 <% table_for @appetizers, :html => {:class => 'appetizers'} do |t| %>
2   <% t.name %>
3   <% t.price, :helper_method => :number_to_currency %>
4   <% t.tastiness, :html => {:class => 'tastiness'} %>
5   <% t.chef %>
6   <% t.description, :helper_method => [:truncate, 100] %>
7 <% end %>

Other options exist, which can be found in the RDoc. I think something similar to table_for should go into ActionView, and with a few rounds of feedback, maybe I will suggest a patch.

Meanwhile, Gregg from RailsEnvy asked me to change my google library post into a plugin. With my newfound plugin skillz, expect that soon.

Rails: Where to put the 'other' files

2008-06-14T09:05:00.000-07:00

Introduction
When starting a Rails project, four golden folders are predefined: Models, Views, Controllers, Helpers. Could we possibly need anything more? In my experience, the answer is yes. This leads to the question of, where do these extra files go? Compared to models, views, controllers and helpers, Rails provides little guidance about where to put this other stuff. Fortunately, Rails comes packaged with two folders to put additional Ruby code: lib and config/initializers. Each have their unique properties, and they can be used together to neatly package your Ruby code.

The weakling lib folder
In Agile Web Development on Rails, the authors describe how to use a naming convention, such that your lib file names match the classes they contain. For example, a class of FooBar must be in the file foo_bar.rb. Classes can even be namespaced. For example, Foo::Bar will be loaded if placed inside of foo/bar.rb.

Files in lib are not loaded when Rails starts. Rails has overridden both Class.const_missing and Module.const_missing to dynamically load the file based on the class name. In fact, this is exactly how Rails loads your models and controllers.

While the functionality of the lib folder is nice, it is extremely limited. It prevents writing modules to extend or override the functionality of another class, because they will never get loaded.

The bloated initializers folder
Rails first loads the framework, then plugins, and finally, it loads the files inside of config/initializers (in alphabetic order, no less). Previous to 2.0, naughty developers pasted code at the bottom of environment.rb, and the initializer folder was a welcome convention to help organize this madness.

I find myself using the initializer folder to configure plugins such as HAML, or loading the action_mailer configuration out of a YML file.

Unfortunately, I see many code snippets showing how to extend the functionality of Rails simply by creating a new file in initializers, and pasting in the code. In a matter of fact, my last post suggested doing this. This is OK in light doses, but after a few months of this practice, you will find yourself with an initializers folder with many files and little structure, not to mention that they get loaded in alphabetic order.

Lib folder, meet initializers
I think I hurt lib and initializer's feelings. I feel bad about this, so maybe it is time to introduce them to each other. A simple example is in order:

We want to add a new 'user_logger' mixin to ActionController::Base. Any controller can choose to enable 'user_logger', and it will put the current user's name in the log file for each request they make. This won't be the last time we extend ActionController's functionality in this project, so let's do it the Right Way.

We first write the module. Start by creating a new folder inside of /lib called rails_extensions. Now paste this code inside the file user_logger.rb:

 1 module ActionControllerExtra
 2   module UserLogger
 3     def self.included(base)
 4       base.extend(ClassMethods)
 5     end
 6
 7     module ClassMethods
 8       # Logs that a user requested the current action:
 9       #
10       #   MyController < ApplicationController
11       #     user_logger
12       #   end
13       #
14       # The same options for ActionController::Filters are available. For example,
15       # to only log the user in the create and destroy actions:
16       #
17       #   MyController < ApplicationController
18       #     user_logger :only => [:create, :destroy]
19       #   end
20       #
21       def user_logger(options = {})
22         include InstanceMethods
23         before_filter :log_user, options
24       end
25     end
26
27     module InstanceMethods
28       def log_user
29         logger.info("#{session[:username]} requested #{controller_name}/#{action_name}")
30       end
31     end
32   end
33 end

Next, create the file rails_extensions.rb in /config/initializers. Paste this code inside:

1 require 'rails_extensions/user_logger'
2
3 ActionController::Base.class_eval do
4   include ActionController::UserLogger
5 end

You may want to create an even deeper directory structure inside of rails_extensions. In my larger projects, I keep a folder for action_controller, active_record, etc. Nonetheless, using initializers and lib together can help you avoid a bloated initializer folder and weak set of functionality in the lib folder.

Using Google Ajax Libraries API with Ruby on Rails

2008-06-10T16:33:00.000-07:00

Intro
The new Google Ajax Libraries API can help you offload bandwidth from your site. Their servers are configured with gzip, expires headers and e-tags, which leads to good scores from YSlow. I will show two ways to use Google's new API to offload Ruby on Rails javascript (prototype and scriptaculous). In both solutions, your existing calls to javascript_include_tag can be left the same, because all of the magic is handled beneath the covers.

Implementation #1: Create a module
This first example is a module that chains the expand_javascript_sources method found in ActionView::Helpers::AssetTagHelper. If prototype or a scriptaculous file is loaded via javascript_include_tag, we will replace the normal functionality with a path to the file on Google's servers. Start by creating the file /config/initializers/google_asset_tag_helper.rb. Paste in the following:

 1 module ActionView
 2   module Helpers
 3     module GoogleAssetTagHelper
 4       GOOGLE_PATHS = {
 5         'prototype' => 'http://ajax.googleapis.com/ajax/libs/prototype/1.6.0.2/prototype',
 6         'controls'  => 'http://ajax.googleapis.com/ajax/libs/scriptaculous/1.8.1/controls',
 7         'dragdrop'  => 'http://ajax.googleapis.com/ajax/libs/scriptaculous/1.8.1/dragdrop',
 8         'effects'   => 'http://ajax.googleapis.com/ajax/libs/scriptaculous/1.8.1/effects'
 9       }
10
11       def self.included(base)
12         base.send :alias_method_chain, :expand_javascript_sources, :google unless ActionController::Base.consider_all_requests_local
13       end
14
15       def expand_javascript_sources_with_google(sources)
16         google_sources, normal_sources = sources.partition { |source| GOOGLE_PATHS.include? source }
17         GOOGLE_PATHS.values_at(*google_sources) + expand_javascript_sources_without_google(normal_sources)
18       end
19     end
20   end
21 end
22
23 ActionView::Base.class_eval do
24   include ActionView::Helpers::GoogleAssetTagHelper
25 end

This defines GoogleAssetTagHelper, which chains expand_javascript_sources. The module is included into ActionView::Base. Notice that on line 12, the original method is only aliased if ActionController::Base.consider_all_requests_local is false. Since this is set to true in development mode, it is a sufficient way to determine whether or not Google AJAX Library should be used. If using the remote libraries in development is desirable, the condition can be safely removed.

Implementation #2: Assign lambda to ActionController::Base.asset_host
An alternative solution uses ActionController::Base.asset_host to determine the root path to the javascript files. In /config/environments/production.rb, you find:

1 # config.action_controller.asset_host = "http://some.asset.host/"

As of Rails 2.0.2, asset_host can a string or Proc. Since we want to conditionally change the asset_host based on what file is being include, a Proc is needed:

 1 google_paths = {
 2   'prototype' => 'http://ajax.googleapis.com/ajax/libs/prototype/1.6.0.2/',
 3   'controls'  => 'http://ajax.googleapis.com/ajax/libs/scriptaculous/1.8.1/',
 4   'dragdrop'  => 'http://ajax.googleapis.com/ajax/libs/scriptaculous/1.8.1/',
 5   'effects'   => 'http://ajax.googleapis.com/ajax/libs/scriptaculous/1.8.1/'
 6 }
 7
 8 config.action_controller.asset_host = Proc.new do |source, request|
 9   google_asset = google_paths.keys.detect { |asset| source.starts_with?("/javascripts/#{asset}") }
10   google_asset ? google_paths[google_asset] : "#{request.protocol}#{request.host_with_port}"
11 end

In this example, the code assumes that you do not already have an asset_host defined. If you do, simply replace the false expression on line 10 with your original asset_host string.

(The variable google_paths is defined inside of production.rb for brevity. However, I recommend defining it as a constant elsewhere in your own project.)

Using the javascript_include_tag
As mentioned in the introduction, your existing javascript_include_tag calls remain the same:

1 javascript_include_tag 'prototype', 'effects', 'controls', 'application'