Viget Extend : The Development Lab

Rails, Internationalization, and Tú

2009-07-02T07:00:00Z

Internationalization (or I18n if you’re hep) is a bear of a problem to deal with in software development. I’ve had to work with a multi-lingual site in PHP before. It wasn’t painful, but it was constantly annoying. I just got a chance to work with internationalization in Rails for the first time, and I was pretty excited to see how it’s been handled.

The good news that I found is that Rails’ I18n support is pretty great. The bad news is that it is overly integrated with the rest of Rails, making changing it in isolation difficult.

The Basics of I18n in Rails

There’s a great Rails Guide on internationalization that’s required reading for anyone trying this at home. My brief nutshell of an explanation is that you create a Ruby or YAML file under config/locales that has a set of hashes of hashes pointing to a specific key. An example would be { 'en' => { 'activerecord' => { 'errors' => {'blank' => 'is required' } } }. That sets the English translation for the key activerecord.errors.blank to “is required.”

You can set your Rails application’s default locale like so:

I18n.default_locale = :es

That will set it to Spanish. There’s not a definitive list of locales, so you could use :lolcatz or :morlock and it will work fine. You set the locale in a specific request like this:

I18n.locale = :es

When you do that, all keys called via I18n.translate (accessible as t in views because of helpers) will return the Spanish translation.

Overall, this is simple and easy. We found this to be helpful not only for Spanish translations, but setting English text as well. You can set the human-readable names of your ActiveRecord models and attributes through this facility, which is very nice for creating forms, especially with Formtastic.

Extending I18n

Reading these YAML and Ruby files and holding the translations in memory is the job of the class I18n::Backend::Simple, which is what’s written on the box: a simple backend for the I18n service in Rails. If you want translations to be editable by non-programmers, though, YAML and Ruby files won’t cut it.

It is easy to extend Simple or write a new backend. Once you’ve written one, you can tell `I18n` to use it:

I18n.backend = I18n::Backend::SnippetBackend.new

I wanted to store my translations in the database and knew I’d only have English and Spanish translations forever. If you’re supporting multiple languages, you’ll obviously want to add another model to the following code. Here’s the backend I wrote:

module I18n
  module Backend
    class SnippetBackend < Simple
      protected

      def init_translations
        load_translations(*I18n.load_path.flatten)
        load_translations_from_database
        @initialized = true
      end

      def load_translations_from_database
        data = { :en => {}, :es => {} }

        Snippet.all.each do |snippet|
          path = snippet.name.split(".")
          key = path.pop

          en = data[:en]
          es = data[:es]

          path.each do |group|
            en[group] ||= {}
            en = en[group]

            es[group] ||= {}
            es = es[group]
          end

          en[key] = snippet.english
          es[key] = snippet.spanish
        end

        data.each do |locale, d| 
          merge_translations(locale, d)
        end
      end
    end
  end
end

This code pulls all the translations I have stored as Snippet models out of the database and merges them into the translations already loaded from the YAML and Ruby files. The only piece of magic here is splitting the key's name. The period syntax to indicate nesting is used in the keys normally, so I continued it in my snippets. To make this match the data from the YAML and Ruby files, I have to transform it into nested hashes.

Where My Problems Began

I also used a plugin called Typus for an administrative interface in this application. It’s a great plugin, and I can recommend it if you want a Django-style interface for editing content without a lot of fuss.

When displaying a list of your editable models, it pluralizes those models’ human names; for example, EmergencyContact becomes “Emergency contact” and is then pluralized to “Emergency contacts.” This works perfectly with English. However, when using Spanish, I ended up with a slight problem:

>> EmergencyContact.typus_human_name
=> "Emergency contact"
>> EmergencyContact.typus_human_name.pluralize
=> "Emergency contacts"
>> I18n.locale = :es
=> :es
>> EmergencyContact.typus_human_name
=> "contacto de emergencia"
>> EmergencyContact.typus_human_name.pluralize
=> "contacto de emergencias"

“Contacto de emergencias” means something like, “a contact for lots of emergencies,” which wasn’t my intention. What I need is “contactos de emergencia,” which would require Spanish pluralization rules. The Rails pluralization rules are based on English. That made sense to me, so I could go add pluralization rules for Spanish, just like you can add translations for Spanish, right?

Unfortunately, that’s not true. Inflection rules in Rails don’t take multiple languages into account, so you can’t define new ones for a particular locale.

Quixotically, I decided to fix this. Here’s the code I ended up with:

module Inflector
   # Yields a singleton instance of Inflector::Inflections so you can specify additional
   # inflector rules.
   #
   # Example:
   #   ActiveSupport::Inflector.inflections do |inflect|
   #     inflect.uncountable "rails"
   #   end

    # Original method
   def inflections
     if block_given?
       yield Inflections.instance
     else
       Inflections.instance
     end
   end      

    # My crazy method
   def inflections(locale = nil)
     locale ||= I18n.locale
     locale_class = \
       if locale.to_s == I18n.default_locale.to_s
         ActiveSupport::Inflector::Inflections
       else
         ActiveSupport::Inflector.const_get("Inflections_#{locale}") rescue nil
       end

     if locale_class.nil?
       ActiveSupport::Inflector.module_eval %{
         class ActiveSupport::Inflector::Inflections_#{locale} < ActiveSupport::Inflector::Inflections
         end
       }
       locale_class = \
         ActiveSupport::Inflector.const_get("Inflections_#{locale}")
     end

     if block_given?
       yield locale_class.instance
     else
       locale_class.instance
     end
   end

   # Returns the plural form of the word in the string.
   #
   # Examples:
   #   "post".pluralize             # => "posts"
   #   "octopus".pluralize          # => "octopi"

    # Original method
    def pluralize(word)
     result = word.to_s.dup

     if word.empty? || inflections.uncountables.include?(result.downcase)
       result
     else
       inflections.plurals.each do |(rule, replacement)| 
         break if result.gsub!(rule, replacement)
       end
       result
     end
   end

    # My method
   def pluralize(word, locale = nil)
     locale ||= I18n.locale
     result = word.to_s.dup

     if word.empty? || inflections(locale).uncountables.include?(result.downcase)
       result
     else
       inflections(locale).plurals.each do |(rule, replacement)|
         if replacement.respond_to?(:call)
           break if result.gsub!(rule, &replacement)
         else
           break if result.gsub!(rule, replacement)
         end
       end
       result
     end
   end
end

This code is pretty complex, but there's only three major changes to the original methods. First, I added the ability to pass in a locale to use on each method, and made the current locale the default one. Second, I changed the behavior of the Inflections class. Instead of having one singleton class, I have a lot of singleton classes, each associated with a locale. These classes are created dynamically through a module_eval. Lastly, I added the ability for inflection transformations to be lambdas instead of just strings.

If you’re shaking your head and asking, “did he just create a new class for each different locale?”, I don’t blame you. It’s a little crazy. Believe it or not, this works magically, though! Here it is at work:

ActiveSupport::Inflector.inflections(:es) do |inflect|
  inflect.plural /$/, 's'
  inflect.plural /([^aeioué])$/, '\1es'
  inflect.plural /([aeiou]s)$/, '\1'
  inflect.plural /z$/, 'ces'
  inflect.plural /á([sn])$/, 'a\1es'
  inflect.plural /í([sn])$/, 'i\1es'
  inflect.plural /ó([sn])$/, 'o\1es'
  inflect.plural /ú([sn])$/, 'u\1es'
  inflect.plural(/^(\w+)\s(.+)$/, lambda { |match|
                   head, tail = match.split(/\s+/, 2)
                   "#{head.pluralize} #{tail}"
                 })
end

You'll notice that lambda in there: I added that ability so that I could transform the first word in a string instead of the last.

My Problems Compound

The above works a little too magically, however. The pluralize method is used everywhere, particularly in changing model class names to table names. As you can imagine, I started seeing errors like “No table found with name emergency_contactes.” I spent a good day running into similar issues and fixing it with code like:

# Create the name of a table like Rails does for models to table names. This method
# uses the +pluralize+ method on the last word in the string.
#
# Examples
#   "RawScaledScorer".tableize # => "raw_scaled_scorers"
def tableize(class_name)
  pluralize(underscore(class_name), I18n.default_locale)
end

Note that I called pluralize explicitly specifying that I want to use the default locale, which would be English. This is a simple fix, but I ended up having to repeat it multiple places in ActiveRecord, ActiveSupport, and ActionController.

Lessons Learned

First, even though this post’s talked about issues I had, the I18n support in Rails rocks, and I’ve found it to be a bit of a universal solvent for form issues, which I hope to talk about in another post. The ability to add new backends is nice, and the combination of my backend and Typus is hot.

I’ve learned that Rails’ inflections are used throughout the framework and trying to reuse them for automatic pluralization and singularization of non-English languages is hard. I think the results might be worth the effort, though. The translation system does have the ability to define singular and plural names for things, but they don’t work through pluralize, so you’d have to re-write how you generate plural names for model objects. That’d be more local than my sweeping monkey-patches, however. Looking back over my code, I’ve decided that it is full of red flags, but not altogether misguided.

How are you using Rails’ I18n? Have you had problems? Are the above fixes terrible hacks or awesome hacks? Let’s talk in the comments.

(You can see all the code from this post at http://gist.github.com/138956.)

How (& Why) to Run Autotest on your Mac

2009-06-19T12:09:00Z

If you aren’t using Autotest to develop your Ruby application, you’re missing out on effortless continuous testing. If you’d like to be using Autotest, but can’t get it running properly, I’ll show you how to set it up.

Autotest is a fantastic way to do TDD/BDD. Here’s a rundown of the benefits from the project homepage:

Improves feedback by running tests continuously.
Continually runs tests based on files you’ve changed.
Get feedback as soon as you save. Keeps you in your editor allowing you to get stuff done faster.
Focuses on running previous failures until you’ve fixed them.

Like any responsible Ruby citizen, Autotest changes radically every month or so. A few weeks ago, some enterprising developers released autotest-mac (now autotest-fsevent), which monitors code changes via native OS X system events rather than by polling the hard drive, increasing battery and disk life and improving performance. Here’s how get Autotest running on your Mac, current as of this morning:

Install autotest:
```
gem install ZenTest
```
Or, if you’ve already got an older version installed:
```
gem update ZenTest
gem cleanup ZenTest
```
Install autotest-rails:
```
gem install autotest-rails
```
Install autotest-fsevent:
```
gem install autotest-fsevent
```
Install autotest-growl:
```
gem install autotest-growl
```

Make a ~/.autotest file, with the following:

require "autotest/growl"
require "autotest/fsevent"

Run autotest in your app root.

Autotest is a fundamental part of my development workflow, and well worth the occasional setup headache; give it a shot and I think you’ll agree. These instructions should be enough to get you up and running, unless you’re reading this more than three weeks after it was published, in which case all. bets. are. off.

Unfuddle User Feedback

2009-06-02T18:32:00Z

Recently, we wanted a better system for managing feedback from SpeakerRate users. While we do receive some general site suggestions, most of the feedback we get involves discrete corrections to data (a speaker who has been entered into the system twice, for example). We started to create a simple admin interface for managing these requests, when we realized that the ticket tracking system we use internally, Unfuddle, already has all the features we need.

Fortunately, Unfuddle has a full-featured API, so programatically creating tickets is simply a matter of adding HTTParty to our Feedback model:

class Feedback < ActiveRecord::Base
  include HTTParty
  base_uri "viget.unfuddle.com/projects/#{UNFUDDLE[:project]}

  validates_presence_of :description

  after_create :post_to_unfuddle, :if => proc { Rails.env == "production" }

  def post_to_unfuddle
    self.class.post("/tickets.xml",
      :basic_auth => UNFUDDLE[:auth],
      :query => { :ticket => ticket })
  end

  private

  def ticket
    returning(Hash.new) do |ticket|
      ticket[:summary]      = "#{self.topic}"
      ticket[:description]  = "#{self.name} (#{self.email}) - #{self.created_at}:\n\n#{self.description}"
      ticket[:milestone_id] = UNFUDDLE[:milestone]
      ticket[:priority]     = 3
    end
  end
end

We store our Unfuddle configuration in config/initializers/unfuddle.rb:

UNFUDDLE = {
  :project    => 12345,
  :milestone  => 12345,  # the 'feedback' milestone

  :auth => {
    :username => "username",
    :password => "password"
  }
}

Put your user feedback into Unfuddle, and you get all of its features: email notification, bulk ticket updates, commenting, file attachments, etc. This technique isn’t meant to replace customer-service oriented software like Get Satisfaction (we’re using both on SpeakerRate), and if you’re not already using a ticketing system to manage your project, this is probably overkill; something like Lighthouse or GitHub Issues would better suit your needs, and both have APIs if you want to set up a similar system. But for us here at Viget, who manage all aspects of our projects through a ticketing system, seeing actionable user feedback in the same place as the rest of our tasks has been extremely convenient.

Developer Day DC, Done!

2009-06-01T14:00:00Z

Developer Day DC took place this weekend, and by all accounts everyone had a great time. Bookended by Jay Virdy’s story about the rise of Summize (and its sale to Twitter) and Russ Olsen’s survey of emerging languages, we had great presentations on a wide variety of topics. Keeping to our technology-agnosticism, we had JavaScript (twice!), Ruby, and Python on the schedule, as well as a more general session on testing and a mind-expanding demonstration of the Monome and other fringe technologies. (If you couldn’t make it, Peter Harkins posted his notes here.)

After conquering Durham and DC, we’ve decided to widen our range a bit, so if you’re eager for a fun, local, cheap conference of your very own, keep watching the Developer Day site. Next up: Boston!

Reusing Contexts in Shoulda with Context Macros

2009-05-26T18:08:00Z

Last month David wrote up a good explanation of how to create shoulda macros with blocks. Recently, I needed to reuse context behavior across a few different tests as well. Out of curiosity, I went in search of a more idiomatic solution and was able to find this ticket and its associated conversation. From the discussion surrounding that ticket, I learned that you can use the merge_block method to create nestable context macros.

class Test::Unit::TestCase
  def self.context_with_an_object(&block)
    context "With an object" do
      setup do
        @object = {:rock => 'on'}
      end

      should "do something fantastic" do
        assert @object[:rock], 'on'
      end

      merge_block(&block) if block_given?
    end
  end
end

You can use the context macro in one of your tests and it will accept a block as well as respect context nesting.

class ModelTest < Test::Unit::TestCase
  context "A great and wonderous test" do
    setup do
      @thing = {:creature => 'lagoon', :rock => 'on'}
    end

    context_with_an_object do
      should "do something specific" do
        assert_equal @thing[:rock], @object[:rock]
      end

      context "And a friend" do
        setup do
          @friend = {:rock => 'on'}
        end

        should "respect some nested context insanity" do
          assert_equal @friend[:rock], @thing[:rock]
        end
      end
    end
  end
end

You can also create your context macros such that they accept arguments.

class Test::Unit::TestCase
  def self.context_with_a_modified_object(modifier, &block)
    context "with a modified object" do
      setup do
        @object = {:mod => modifier}
      end

      merge_block(&block) if block_given?
    end
  end
end

Modifying the context behavior via an argument allows you to test a handful of edge cases without having to duplicate context code.

class ModelTest < Test::Unit::TestCase
  context_with_a_modified_object("dance!") do
    should "be modified" do
      assert_equal @object[:mod], "dance!"
    end
  end
end

I recommend using this technique sparingly and only to remove unnecessary duplication among your tests. It is possible to hide too much context behavior behind these macros and end up with tests that are difficult to understand and maintain. For contexts that aren't reused outside of a single file consider defining them at the top of the test file.

Rack Support in Rails: Why It Matters

2009-05-19T12:00:00Z

As I’ve mentioned elsewhere, my RailsConf session on Rack support in Rails went very well. I’m not planning on giving this talk again, however, so I thought I’d write a quick summary of it for people who are interested in a little more detail than the slides can provide.

My main motivation in giving this talk was to call attention to what I feel has been the most important change in Rails over the past year – its move to become Rack-compatible. The initial work on this was done by Ezra almost a year ago, but it only made it into a stable release with 2.3. This delay meant that it stayed a bit under the radar, and even the addition of Rails Metal didn’t really pull it into the spotlight. Then, in December of 2008, the announcement of the Merb-Rails merger pretty well decided the story of the year for Rails.

So What?

So, why is Rack support so important? Why should it be recognized as more vital to the success of Rails than the merger of the two most popular Ruby web frameworks? Rack is just a common API for Ruby web frameworks, after all, and as such isn’t as flashy as, say, Merb’s router.

The answer is just that Rack is a common API for Ruby web frameworks. It’s essentially the Ruby version of Python’s WSGI, specifiying a simple interface for handling HTTP requests in Ruby. The simplest Rack-compatible web application would look something like this:

class SimpleRackApp
  def call(env)
    [       
      status,  # 200
      headers, # {"Content-Type" => "text/html"}  
      body     # ["..."] 
    ] 
  end
end

To be Rack-compatible, an application need only have a call method that accepts a hash of environment data (including, for instance, the HTTP headers included in the request) and returns an array of an HTTP status, a hash of headers, and a body that responds to .each.

Standardizing the interface for web applications means that you can link them together in interesting and useful ways – leading us to Rack middleware. Middlewares are just Rack-compatible applications that intercept and manipulate a request or response; a middleware could, for example, rescue any unhandled exceptions from your application and display a friendly error page.

Rack in Rails

As one of the results of the integration of Rack into Rails, many of the functions of ActionController have been refactored into middlewares, which means they can be required into non-Rails applications and used. As of today, these middlewares live in the ActionDispatch module, and include classes like ActionDispatch::Failsafe, ActionDispatch::ShowExceptions, and ActionDispatch::ParamsParser (among others). Normally, these will be automatically required when you run your Rails application through script/server or Passenger, but you can control their inclusion or replace them entirely with environment.rb; see the Rack guide for more details.

Rails Metal is another of the results of the move to Rack. Metal provides a way to bypass some of the normal overhead of the Rails stack, allowing a developer to specify certain routes and code to execute when those routes are hit. Metal actions avoid the entirety of ActionController, and so are somewhat faster than standard Rails actions (though this performance benefit may not always be significant).

Tools and Techniques

All of this is just background, though; the real argument in favor of the importance of Rack is found when you look at what you can now do. In the session at RailsConf, I presented two such examples: Rack::Bug and progressive caching. Rack::Bug is a middleware, and some great things have been written on it recently. Progressive caching is a technique that uses Rails Metal to respond to AJAX requests, making page caching a more generally useful strategy (it is also one of the other talks I give, and I’ve written on it in a couple of places).

During the final part of the section, I showed how to write a couple of middlewares that might be generally useful: Rack::Embiggener and Rack::Hoptoad. The latter is obviously a port of the hoptoad_notifier plugin to a Rack middleware (and indeed, this is a direction that the thoughtbot team is considering – something I’m really looking forward to!). The former, however, may be a more interesting example. Rack::Embiggener is a middleware that intercepts the response from a Rack application, detects any TinyURLs embedded in it, and replaces them with the expanded URL. An obvious use-case for this is Twitter, where tweets could be stored with TinyURLs in place (remaining under the 140 character limit), while they are displayed with the full URL visible regardless of the length¹.

module Rack
  class Embiggener
    def initialize(app)
      @app = app
    end
    
    def call(env)
      status, headers, body = @app.call(env)
      headers.delete('Content-Length')
    
      response = Rack::Response.new(
        Rack::Embiggener.embiggen_urls(body),
        status,
        headers
      )
    
      response.finish
      return response.to_a
    end
        
    def self.embiggen_urls(original_body)
      new_body = []
      
      original_body.each { |line|
        tinys = line.scan(/(http:\/\/(?:www\.)?tinyurl\.com\/(.{6}))/)
        new_body << tinys.uniq.inject(line) do |body, tiny|
          original_tiny = tiny[0]
          preview_url = "http://preview.tinyurl.com/#{tiny[1]}"
          response = Net::HTTP.get_response(URI.parse(preview_url))
          embiggened_url = response.body.match(/redirecturl" href="(.*)">/)[1]
          body.gsub(original_tiny, embiggened_url)
        end
      }
      
      new_body
    end
  end 
end

Walking through the code:

#initialize stores the Rack application object in a variable for later use.
#call passes the request to the application stored in @app, and records its responses. It then deletes the Content-Length header (since we’ll be enlarging some URLs), and creates a new Rack response with the same information, passing the original body through the embiggen_urls method. The response.finish call ensures that the appropriate headers (including a new Content-Length) are set.
embiggen_urls accepts a body (that, per the Rack specification, responds to .each), and (inefficiently) scans over it to find and replace the TinyURLs².

With this middleware, then, any Rack-compatible application – be it Twitter, some other Rails 2.3 application, a Sinatra microapp, or even something written in Camping – can replace TinyURLs with their expansions.

¹ More interestingly, Twitter or another service might be able to replace the full ‘http:/tinyurl.com/[foo]’ with just ‘[foo]’ and a single-character flag of some sort, providing even more storage savings.

² This code was just meant to inspire people by showing something that could be done, and isn’t meant to be particularly efficient or good. Feel free to fork and improve it!

Wrapup

One of my favorite sayings comes from Socrates: “A list is not a definition.” My presentation (and this post) argue for the importance of Rack in Rails, but all they really do is give a list of examples of useful tools and techniques. My final point, then, is this: Rack compliance makes Rails a first-class member of the community of Ruby web frameworks (contrast that with Django, which implements a non-standard version of WSGI). This means that work done for Rails, such as the refactoring of parts of ActionController into ActionDispatch’s middlewares, benefits both Rails developers and users of other frameworks, and vice versa. Rack unites the fragmented Ruby web framework ecosystem, which means that no matter the community in which we work, our efforts improve the whole rather than just a single piece. As we’ve seen in open source time and again, the more people working on a set of problems, the better.

Provisional and Repo Man: Automated Bootstrapping

2009-05-13T15:10:00Z

If we’ve said it once, we’ve said it many times: Don’t Repeat Yourself.

Ahem. As the kids say, “do-over!”

Really, what we’ve been repeating is “Don’t repeat yourself without automating the repetitive task.” We try to automate everything we do more than once, and as a busy agency, one of the things we do more than once is bootstrap new Rails applications. And we do pretty much the same things every time: we create a new Rails application, we install a few favorite gems and plugins, and we check the lot into a freshly-created Git or Subversion repository. It’s ripe for automation, not just because we repeatedly do it, but because we want to repeat it. We want the same setup every time, so that we know we’re starting from the same base on every project.

Rails application templates arrived as part of Rails 2.3, and they go a long way towards automating the new-Rails-app process. Spend a little time crafting your ideal template, and you can use it to stamp out identical Rails skeletons, ready for development. If you write more than one or two Rails applications a year, and you haven’t written a template yet, you really should.

But what about the other step we mentioned above - setting up our new SCM repository? Surely we could automate that process as well. Several of the popular code-hosting sites have APIs to manipulate repositories. Why not programatically set up our SCM and check in the fruits of our templating labor?

That’s why we wrote Provisional. It picks up where the Rails application generator leaves off, setting up a Git or Subversion repository and putting your code in. It’s one less thing to do by hand, and that’s always a good thing.

Let’s take a peek. Go ahead and gem install provisional, then run provisional --help to check out the options:

$ provisional --help
Options:
      --name, -n <s>:   Name of the project
       --scm, -s <s>:   SCM to use
  --template, -t <s>:   Rails template to use
    --domain, -d <s>:   Domain (for some SCMs, see documentation)
  --username, -u <s>:   Username (for some SCMs, see documentation)
  --password, -p <s>:   Password (for some SCMs, see documentation)
        --id, -i <s>:   Id (for some SCMs, see documentation)
    --config, -c <s>:   Config file (optional)
          --help, -h:   Show this message

The three most important options are at the top: a name for your application; an SCM system to use; and the Rails template to apply. (The other options are used for identifying, and authenticating to, code hosting services. See the README for more information on how to use these.)

By default, Provisional will create a Git repository on your local system, ready to push out to wherever you’d like. If you’re hosting on GitHub, you can specify that, and Provisional will create a new repository on GitHub, set it as origin, and push out to master. Likewise for Unfuddle, a project-management site that can host both Git and SVN repositories; Provisional supports them both. (Have you got your own Git or SVN server inside your firewall? I’ll come back to you later.)

You can specify any Rails application template you like. As with running the generator directly, you can use a local template file, or a URL to a remote file. If you don’t choose, you’ll get the default template we use at Viget, which is a busy template indeed; it:

freezes Rails from your currently installed gems
installs one of our favorite testing stacks: Shoulda, Factory Girl and Mocha
installs the Hoptoad notifier, so we can keep a handle on exceptions
installs jRails, since we roll with jQuery these days
installs our model generator with Factory Girl support
installs our standard Capistrano setup
installs our coverage enforcement plugin
cleans up various things we don’t use, like the default public/index.html and prototype/script.aculo.us

(You’ll find it under lib/provisional/templates/viget.rb in the gem, if you’d like to fork it and write your own.)

And that’s pretty much it. There are some other things it could do, but currently doesn’t. It won’t set up your production server for you; Sprinkle’s got that market covered. It’d be nice if it set up continuous integration, but we haven’t settled on our one true CI server yet. Maybe that’ll be a future add-on. For now, that’s it for Provisional; go forth and bootstrap!

“Hey, wait!” you say. “I’m hosting my own Git or SVN server inside my firewall. You said you’d come back to me later.” We haven’t forgotten you. Actually, at Viget, we’ve had Git and SVN inside our firewall for years. Since we use Provisional for our own projects, we need to handle this kind of setup. Before Provisional, we had a couple of shell scripts on our Git/SVN server that we used to create new, empty repositories with access for everyone in the company. They were handy, and worked well, but weren’t that easy to automate.

So we wrote Repo Man, a very simple (perhaps too simple) Rails application to create repositories. Originally, we had written Repo Man to let our design staff create repositories for non-code purposes. All we had to do is add some controller logic to respond to Active Resource calls, and it became API-capable just like the code hosting sites. And with that, the Provisional-Repo Man bridge was a snap to create.

Repo Man can work with your internal SCM setup if it runs on a Linux or UNIX-like system. You’ll just need to create some scripts like the ones we wrote in the beginning (which are provided with Repo Man as examples.) And you’ll probably want to add some support for your local authentication scheme. But Repo Man will get you pretty close to what you need to work with Provisional.

And there you have it - a complete system for automating those crucial early bootstrapping tests. It’s just one more step in our ongoing quest to automate everything we do more than once. (Except writing these blog posts, of course. That still requires a human touch…)

JSConf 2009 Recap: Javascript at the Edge

2009-05-12T13:00:00Z

Two weeks ago, I spent an amazingly full two days at JSConf 2009 getting my mind blown by some JavaScript developers who are pushing the boundaries of the language. Thanks to having so many great people there, we all had a great time discussing ideas and learning from each other. Having a smaller number of attendees really enhanced this aspect of the conference and allowed people to discuss the ideas more deeply.

Beside the discussions of newer Javascript Frameworks (like Cappuccino/Objective-J and SproutCore), the main theme that I think threaded its way through a lot of the conference was using javascript outside the browser. Also, everyone I met was fired up about Javascript and exploring what new things we can do with it.

CouchDB to the Edge

The technology that probably had the biggest spotlight on it was CouchDB. Chris Anderson & Jan Lehnardt gave a stellar talk on how CouchDB works and what makes it special. As they expressed, CouchDB has 3 big selling points: it's Simple, Scalable, and Distributed. The first point is mainly a result of using JSON, REST/HTTP, and Javascript as the basis for interacting with the DB and building apps on top of it. The last two are mostly due to it being built in Erlang, and using Map-Reduce in the “views.” They also talked heavily about the ease of creating CouchApps, which are web apps that are stored and run on CouchDB. The fact that CouchApps have filesystem mapping and that CouchDB is RESTful allows for very simple cloning of CouchApps. CouchApps appear to be extremely scalable and simple to create, making them the item from the conference I’m most interested in exploring more deeply.

The Real Time Web with XMPP

The other technology I was excited about was Strophe, a Javascript library that allows you to do XMPP calls in the browser. Strophe provides a very simple API for doing XMPP over BOSH communications in the browser. Jack Moffitt explained the basics of how the library implements this and then went on to give some examples of just how easy using the framework is.
The most obvious use of this is adding chat capabilities to any page. This can, of course, be specific to the page, specific to the site, or somewhere in between. The less traditional use for it is as a real-time communication channel between the web page and the server. Using this in conjunction with some of the methodologies discussed by George Malamidis, you could easily have this type of client-server interaction.

Interoperable Javascript-Based Client/Server Web Applications

Kris Zyp of Sitepen presented on Persevere, an object storage server based REST/HTTP and JSON. It allows you to take advantage of a number of cutting edge JSON technologies including JSON referencing, JSONQuery/JSONPath, and JSON Schema. They’ve worked heavily to offer tight integration with Dojo’s Data JsonRestStore functionality, although it can also work with other frameworks (like Jester). It also opens up the possibility to work with other data stores on the server side, including MySQL and CouchDB, opening up all sorts of interesting possibilities. What I found most interesting here was the use of JSON referencing to handle relationships between objects. This is a traditional problem with JSON document based storage in my opinion, but this offers a solution to that.

JazzRecord

Nick Carter gave an excellent hands-on demonstration of the Javascript ORM JazzRecord. It mirrors ActiveRecord’s API and syntax as closely as it can, offering up the ability to store data client side via Gears and other client-side data persistence options. It’s a really great start to offering a single way of interacting with data with-in a Rails app, be it on the server or on the client. The code is up on github, and he’s looking to offer support HTML 5 databases so if you’ve got a chance check it out and help out by forking it.

YQL Execute

Tom Hughes-Croucher gave a demo of what you can do with YQL, a service that offers up SQL like syntax to access data from various web services. He gave us a preview of the now-announced execute functionality. With execute in your YQL table definition, you can take the data coming into the query and modify it with Javascript before sending out the final response. The main advantage YQL offers is offloading the fetching, processing, sorting, filtering, and joining of data to an external service that’s much faster. To quote Tom, “Our [Yahoo’s] pipes are fatter than yours.”

PhoneGap

Every conference needs some comic relief, and that could be found in Brian LeRoux’s hilarious Day 2 morning session on PhoneGap. PhoneGap is a framework and toolset to create apps for the iPhone, Android, and Blackberry using just HTML, CSS, and Javascript. It even offers up an API to get at the devices native technologies like geo location, vibration, accelerometer, and sound. The slides where great and it got everyone excited about building an app with PhoneGap.

And much, much more.

Really, that’s just skimming the surface. John Resig gave a really phenomenal talk on better ways to test Javascript performance and compatibility, and introduced a killer Javascript game. There was also an interesting session on Appcelerator’s Titanium, a way WebKit based platform to create desktop apps with HTML, CSS, and Javascript, Ruby or Python.
Also, E4X was mentioned a number of times. E4X is an extension to Javascript for Rhino that makes XML a native primitive. It allows you to write, access, and manipulate XML in Javascript easier and quicker then with traditional DOM methods.

Overall, it got me very excited to try out all these new up and coming tools. I’d be very excited to attend a JSConf 2010. Javascript as a real programming language is very much a growing topic and I think based on what I saw this year there’ll be even more amazing sessions in the future.

RailsConf 2009 Wrapup

2009-05-11T13:00:00Z

I've been back from RailsConf for a few days now, and I've had a chance to reflect on the things I learned while in Las Vegas for what turned out to be my favorite RailsConf yet. Elsewhere, I wrote about my overall impressions, but for this post I want to focus on several of the sessions I attended and participated in.

Teaching Rails BoF

RailsConf for me really started Monday night, with the Teaching Rails Birds of a Feather session that I moderated. We had a great turnout (somewhere in the neighborhood of 30 people), and the attendees ranged across a number of different disciplines — some people were teaching in a university setting, some were writing books, and others were interested primarily in building the community through educating new developers. Regardless of their backgrounds, though, everyone had good ideas and experiences to share, and I learned about a number of exciting new efforts (including the various components of the RailsBridge effort and the upcoming Rails Tutorials project).

The Even-Darker Art of Rails Engines

Tuesday, I attended James Adam's session on Rails Engines. I heard great things about his presentation on plugins from the 2008 RailsConf, so I had high hopes about this one. As it turned out, the presentation itself was decent, but the content was a little too basic for me — we've looked fairly carefully at the engines support in Rails as the next step for our resourceful plugins, so most of what James talked about I'd already seen. I do have to say that the DHH-headed cherub slide was a real winner, though.

The Women in Rails Panel

Given the recent flap in the community, I was eager to see how the Women in Rails panel would go. Attendance was lower than most everyone expected, though the major players were all present. Desi McAdam, Sarah Mei, and Lori Olson avoided (for the most part) the brouhaha, and ended up having a nice discussion of the various issues surrounding women in technology in general and in the Rails community in particular. As I mentioned to various others at the conference, I think it's good that we're having these sorts of discussions now, and that people are willing to talk about diversity and its value.

Smacking Git Around — Advanced Git Tricks

This was another session that I was looking forward to primarily based on the presenter. Scott Chacon did the session before mine last year, and famously ran through several hundred slides; this year, he had somewhere around 400. Amazingly, his slides aren't even of the Takahashi school — he puts some fairly complex diagrams and animations on there, and does an excellent job of presenting compex issues. I knew of many of the advanced tricks in this presentation beforehand, but a lot of it was new, and I had a great time at the session.

Blood, Sweat, and Rails

I wasn't entirely sure what to expect of Obie Fernandez's presentation; it ended up being a primer on the various issues he's dealt with as the founder of Hashrocket. Viget's similarities to Hashrocket meant that I was familiar with many of these, but Obie's an engaging presenter, and I think he communicated the challenges very well.

Keynote: Chris Wanstrath

For the most part, I'd like to avoid talking about the keynotes. I do, however, want to call specific attention to Chris Wanstrath's. Unlike many presenters, Chris writes essays for his talks, and he reads them live. Unlike some speakers who do this, though, Chris is really good at it. It also helps that his content is excellent. For this talk, Chris told people how to become 1) a famous Rails developer, and 2) a good Rails developer. Unsurprisingly, the two are not synonymous, and his exploration of the processes behind each was inspirational.

Rails Metal, Rack, and Sinatra

I attended this session mainly to make sure that my session on Thursday wouldn't duplicate too much of Adam Wiggins' content. Luckily, Adam provided a much more basic introduction to the issues than I'd planned on, so there was minimal overlap. The most interesting thing I came out of this session with was an understanding of just how pervasive Rack was at this RailsConf — everyone was talking about it to some extent. I almost think there could've been a complete Rack track across all three days...

Starting Up Fast: Lessons from the Rails Rumble

Unfortunately, the Rumble panel didn't go as well as I'd hoped. The more events I attend, the more convinced I am that panels are one of the hardest sessions to do well — and while I really enjoyed the discussions all of us had leading up to the session, the actual on-panel conversations just didn't seem to flow as smoothly as the less formal ones did.

I was very excited to meet the other panelists, though — they've all worked on some remarkable things, and they're great guys. I'm looking forward to seeing what they all do in this year's Rails Rumble (August 22nd-23rd, by the way - mark your calendars!)

And the Greatest of These Is ... Rack Support

My solo session was (as it always seems to be) on the last day of the conference. Luckily, however, this time it was in the morning, so people were still fairly alert, and were only beginning to suffer from learning-overload. I'm very happy with how the overall presentation went, particularly as I was trying out a new technique for taking questions (which I originally heard about from Giles Bowkett).

At the beginning of the talk, I showed the hashtag #?forben, and asked people to tweet their questions as the session progressed. This has the dual benefit of letting people ask questions when they arise, and forcing those questions to a reasonable (140 character) length. When I was ready for Q&A, I fired up my trusty Twitter client and saw what people had asked. The main problem with the technique was the layout of the room, which meant I had to jump down off the stage and run over to the side to see the actual questions (after a few iterations of this, the audience finally just shouted the question out to me and I answered it from the stage).

One other interesting thing came out of this session; while I was talking about all the great changes in Rails over the past year, I put up a slide with the most awesome logo I know of (the one Doug Avery created for Bowling Thunder last year). The logo and website got around, and finally ended up on Reddit, where it apparently generated a small flood of traffic to the website. Let's hear it for Doug Avery, everybody!

And the Greatest of These Is ... Rack Support

View more presentations from Ben Scofield.

The Russian Doll Pattern: Mountable Apps in Rails 3

Going in to RailsConf, this was the session I was most excited about. The problems with integrating disparate Rails applications have always been one of my prime interests, so I'm greatly looking forward to it being much easier in Rails 3. Unfortunately, however, Yehuda's and Carl's talk left me unsatisfied and a little less hopeful. They admitted upfront that there was no code for mountable applications yet, and that they'd be showing us unicorns — the way they hope it'll work — which is perfectly fine. The vision they presented, though, appeared to me to be very incomplete, and to have some fundamental issues that don't appear to have been addressed. At this point, however, it's pretty close to useless to object, since the vision itself is so far from being implemented. I'll be keeping an eye out, however, and will be very interested to see how this progresses in the future.

Incidentally, this talk was also the one that most inspired me to begin working on Athena again, since one of the features I'd originally planned for it was a painless way to bundle multiple applications together.

So, that's a quick review of some of the sessions I attended. All in all, I think this RailsConf was a huge success — I met a ton of new people, and learned some interesting things along the way, which is about all you can hope for out of a conference, isn't it? Let's hope that next year is as good!

Backup your Database in Git

2009-05-08T14:15:00Z

Short version: dump your production database into a git repository for an instant backup solution.

Long version: keeping backups of production data is fundamental for a well-run web application, but it’s tricky to maintain history while keeping disk usage at a reasonable level. You could continually overwrite the backup with the latest data, but you risk automatically replacing good data with bad. You could save each version in a separate, timestamped file, but since most of the data is static, you would end up wasting a lot of disk space.

When you think about it, a database dump is just SQL code, so why not manage it the same way you manage the rest of your code — in a source code manager? Setting such a scheme up is dead simple. On your production server, with git installed:

mkdir -p /path/to/backup
cd /path/to/backup
mysqldump -u [user] -p[pass] --skip-extended-insert [database] > [database].sql
git init
git add [database].sql
git commit -m "Initial commit"

The --skip-extended-insert option tells mysqldump to give each table row its own insert statement. This creates a larger initial commit than the default bulk insert, but makes future commits much easier to read and (I suspect) keeps the overall repository size smaller, since each patch only includes the individual records added/updated/deleted.

From here, all we have to do is set up a cronjob to update the backup:

0 * * * * cd /path/to/backup && \
mysqldump -u [user] -p[pass] --skip-extended-insert [database] > [database].sql && \
git commit -am "Updating DB backup"

You may want to add another entry to run git gc every day or so in order to keep disk space down and performance up.

Now that you have all of your data in a git repo, you’ve got a lot of options. Easily view activity on your site with git whatchanged -p. Update your staging server to the latest data with git clone ssh://[hostname]/path/to/backup. Add a remote on Github and get offsite backups with a simple git push.

This technique might fall down if your app approaches Craigslist-level traffic, but it’s working flawlessly for us on SpeakerRate, and should work well for your small- to medium-sized web application.

SimplestAuth: Gem-ified, with DataMapper Support

2009-05-07T19:00:00Z

A few months ago we released simplest_auth as a stripped-down alternative to authentication plugins such as restful auth. After using it in a few projects we came to the realization that there was no particular reason for it to be a plugin and even less of a reason for it to work only with ActiveRecord.

As a result, the latest version on Github is now built as a gem and can be used in a Rails project that is using either ActiveRecord or DataMapper as its ORM. The same example usage from the README applies to ActiveRecord, but I'd like to give an example which uses DataMapper for the model.

  class User
    include DataMapper::Resource
    include SimplestAuth::Model

    property :id, Serial
    property :email, String, :nullable => false
    property :crypted_password, String, :length => 60

    # validate your model as you see fit
    # these are some sane defaults for password
    validates_present :password, :if => :password_required?
    validates_is_confirmed :password, :if => :password_required?
  end

Or, using a username instead of an email address just change these lines:

  property :username, String, :nullable => false

  # add this line when not using the default of email
  authenticate_by :username

Everything else should remain the same. Install the latest gem using:


sudo gem install simplest_auth

Updated Garb: Even Easier Access to the Google Analytics API

2009-05-04T17:00:00Z

In my introductory post, I explained what Garb was, and how it could be useful to those wishing to get access to their Google Analytics data. In this post, I would like to introduce the latest version of Garb (0.2.1 as of this writing) and explain the many changes and new ways to use Garb to get at your data.

Getting sessions and profiles is still exactly the same.

  Garb::Session.login('username', 'password')
  profile = Garb::Profile.all.first

What's new in 0.2.1 is the way in which reports are built and results are retrieved. Check it out:

As a Report Class

  class Exits
    include Garb::Resource

    metrics :exits, :exit_rate
    dimensions :request_uri
  end

Getting the Results with a Class

  Exits.results(profile, :limit => 10,
                         :offset => 20,
                         :start_date => (Date.today - 30),
                         :end_date => Date.today)

  # With Filtering and Sorting

  Exits.results(profile) do
    filter :request_uri.contains => 'fun', :exits.gte => 1000
    sort :exit_rate
  end

One-off Report

  report = Garb::Report.new(profile)
  report.metrics :exits, :exit_rate
  report.dimensions :request_uri

  # With Filtering and Sort

  report.filter :request_uri.contains => 'fun'
  report.filter :exits.gte => 1000
  report.sort :exit_rate.desc

  # Getting Results

  report.results(:limit => 10,
                 :offset => 20,
                 :start_date => (Date.today - 30),
                 :end_date => Date.today)

The results returned from Garb will be OpenStructs with methods for each of the metrics and dimensions in an array.

  results.exits       #=> 1234
  results.exit_rate   #=> 0.20423810234
  results.request_uri #=> '/some/fun/url/to/a/page'

Overall, we feel that the improvements are solid and make more sense. Be warned: if you've used a previous version of Garb, then updating to the latest version will very likely break most of what was done previously. I hope everyone can find a use for this, and I encourage all to check out the project on Github and to read the documentation in the Wiki.

Shoulda Macros with Blocks

2009-04-29T12:00:00Z

When I’m not working on client projects, I keep myself busy with SpeakerRate, a site that lets conference goers rate the talks they’ve attended. After a number of similar suggestions from users, we decided to display the total number of ratings alongside the averages. Although only talks can be rated, speakers, events and series also have ratings through their associated talks. As you can imagine, calculating the total ratings for each of these required a lot of somewhat repetitive code in the models, and very repetitive code in the associated tests.

Fortunately, since we’re using Shoulda, we were able to DRY things up considerably with a macro:

class Test::Unit::TestCase
  def self.should_sum_total_ratings
    klass = model_class

    context "finding total ratings" do
      setup do
        @ratable = Factory(klass.to_s.downcase)
      end

      should "have zero total ratings if no rated talks" do
        assert_equal 0, @ratable.total_ratings
      end

      should "have one total rating if one delivery & content rating" do
        talk = block_given? ? yield(@ratable) : @ratable
        Factory(:content_rating,  :talk => talk)
        Factory(:delivery_rating, :talk => talk)

        assert_equal 1, @ratable.reload.total_ratings
      end
    end
  end
end

This way, if we’re testing a talk, we can just say:

class TalkTest < Test::Unit::TestCase
  context "A Talk" do
    should_sum_total_ratings
  end
end

But if we’re testing something that has a relationship with multiple talks, our macro accepts a block that serves as a factory to create a talk with the appropriate relationship. For events, we can do something like:

class EventTest < Test::Unit::TestCase
  context "An Event" do
    should_sum_total_ratings do |event|
      Factory(:talk, :event => event)
    end
  end
end

I'm pretty happy with this solution, but having to type “event” three times still seems a little verbose. If you've got any suggestions for refactoring, let us know in the comments.

Introducing Garb: Access the Google Analytics Data Export API with Ruby

2009-04-24T13:00:00Z

Since the moment Google announced that they would be releasing an API for their de-facto analytics tools, I was very excited to dive into the wealth of useful information within. Shortly thereafter, Josh and Jen from our Marketing Lab came to us to develop some tools that would help in collecting the data for our reporting.

To accomplish these goals, I first read through all of the documentation that Google provided with access to the analytics data export beta. After, with some brainstorming with Pat, we decided to create a new gem to handle much of the interaction with the new API. Garb is what we've come up with.

Thankfully, the waiting period is over. Google has announced the release of the API to the public, and Josh has written his take on the announcement.

The API and Design of Garb

Documentation

The documentation for the analytics API is pretty solid, though complex. Sections describe the authentication process, retrieving accounts and profiles, the metrics and dimensions available and how they could be combined, and details for sorting and filtering the data. It is very well written, and very concise. I suppose I should have expected nothing less from Google.

A Word on Dimensions and Metrics

The API provides access to all of the same dimensions and metrics (and more) that are familiar to anyone who has ever used Google Analytics and especially the custom reporting tools within. Through the API there is, however, a substantial set of rules surrounding the selection and combination of dimensions and metrics.

Basically, two dimensions and up to 50 metrics can be selected in any one request. Easy enough? Unfortunately, we hit snags trying to make (what seem like) commonplace combinations. For example, while we were developing Garb, you could not get the "visits" metric scoped to the "requested page URI" dimension. It took a good bit of studying to truly understand the boundaries of these combinations.

So, On To Using Garb in Your Project

There are three parts to using Garb. The first is getting a session, which requires your Google Analytics username and password . Second is the profile (or many profiles) that your authenticated account has access to. Last, given the first two parts, you can create reports and retrieve the data from the API.

Session

Sessions are simple enough to create and are currently stored globally. This is simpler, but limits us to one session at a time.


Garb::Session.login('username', 'password')

Profile


profile = Garb::Profile.all.first # or select from the list

Reports

You can create a simple report.


report = Garb::Report.new(profile, {:metrics => [:visits]})
report.all

Or, an advanced one.


class ExitsReport < Garb::Report
  def initialize(profile)
    super(profile) do |config|
      config.start_date = Time.now.at_beginning_of_month
      config.end_date = Time.now.at_end_of_month
      config.metrics << [:exits, :pageviews, :exit_rate]
      config.dimensions << :request_uri
      config.sort << :exits.desc
      config.max_results = 10
    end
  end
end

ExitsReport.new(profile).all

Note: the date/time related methods are from ActiveSupport.

The Google Analytics Data Export API has come along very quickly. It's already powerful enough to provide the data to generate reports that we've never before been able to create. Soon, we'll only be limited by our imaginations, which is exactly how Google wants it. There have always been many compelling reasons to use Google Analytics. Now, I struggle to find a reason not to.

I encourage everyone interested in using this library to read the documentation, check the code out on Github at http://github.com/vigetlabs/garb/tree/master and, of course, comment, question, and enjoy. If you'd like to go straight to installing it be sure to vigetlabs-garb sourced from Github until I can get it up on Rubyforge!

Progressive Caching In-Depth

2009-04-23T09:01:00Z

Recently, I've been presenting on a technique that takes advantage of Rack support in Rails to revitalize page caching; I've taken to calling it "progressive caching." There are a couple of places around the web where you can find an introduction to the topic, but this post will go into significantly more depth.

Note: This technique isn't particularly new (I've seen at least one blog post from 2007 about it), though some of the advances in Rails make it more effective. As far as I can tell, however, it hasn't really been explored in depth, but I welcome disagreement on that point!

Background

When you implement page caching in Rails, an action is only processed once; after that, the generated markup is saved to a static file in /public. Subsequent requests to the same URL are then processed by the web server (Apache, nginx, etc.), which makes them orders of magnitude faster than if they were processed dynamically.

This performance boost is wonderful in some cases, but comes with two major challenges: cached pages are both public and static. If you present a page only to authenticated users, you can't use page caching — after the first valid presentation, the content would be cached and anyone could see it, because the web server doesn't respect your application's authentication and authorization code. Also, cached pages are (by definition) static, which means that this is unsuitable for pages that change frequently, or change based on who is viewing them.

Progressive caching is intended to overcome both of these challenges by combining page caching with highly-efficient AJAX requests and client-side JavaScript.

Example

To see how this works, let's start with a sample page:

This is a channel page from odeo.com; it shows, among other things, the most recent episodes for the "Ask a Ninja" series. Notice that each episode has a small, pink plus icon. When you add an episode to your quicklist on Odeo, the pink changes to gray and the icon is disabled — and this state persists across all pages where the episode appears.

This practice means that every time a page on Odeo is rendered, the system has to check whether the episodes on the page are in the current user's quicklist, and it has to show the correct icon for each. Let's show some (entirely faked) code for that:

class ChannelsController
  def show
    @quicklist = current_user.quicklist
    @channel = Channel.find(params[:id])
    @episodes = @channel.episodes.recent
    # ...
  end
end

<% @episodes.each do |episode| %>
    <%= image_tag episode.thumbnail_url, :alt => h(episode.title) %>  
    <%= link_to image_tag('quicklist_icon.png',
                        :alt => "Add this episode to your quicklist"),  
                        foo_url,
                        :class => (@quicklist.includes?(episode) ? 'listed' : '')  %>  
<% end %>

In the view, we're changing a class on the link based on whether the episode has been added to the quicklist or not. Through CSS, this changes the icon displayed to the user.

Changing this fairly costly action to use progressive caching is easy. First, we remove the conditional from the view, which leaves us with content suitable for page caching (since it is no longer dependent on the logged-in user):

<% @episodes.each do |episode| %>
    <%= image_tag episode.thumbnail_url, :alt => h(episode.title) %>
    <%= link_to image_tag('quicklist_icon.png',
                        :alt => "Add this episode to your quicklist"),
                        foo_url,
                        :id => "quicklist_#{episode.id}" %>
<% end %>

Next, we cache the page and remove unnecessary code (note: at this point you'd also have to add the appropriate cache-expiration code — say, when new episodes are uploaded):

class ChannelsController
  caches_page :show

  def show
    @channel = Channel.find(params[:id])
    @episodes = @channel.episodes.recent
    # ...
  end
end

At this point, the bulk of the original page's content is being processed through the web server, but we're obviously missing the bits that depend on the logged-in user. To add those, we'll add some Metal:

./script/generate metal Personalizer

# Allow the metal piece to run in isolation
require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails)

class Personalizer
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/personalize/
      [
        200, 
        {"Content-Type" => "application/javascript"}, 
        [User.find(env['rack.session'][:user]).quicklist.episode_ids.to_json]
      ]
    else
      [404, {"Content-Type" => "text/html"}, ["Not Found"]]
    end
  end
end

And finally, a bit of jQuery that runs when the page loads:

$(document).ready(function() {
  $.getJSON('/personalize', function(data) {
    $.each(data, function() {
      $('#quicklist_'+this).addClass('listed');
    });
  });
});

After that, we're pretty much done. When someone visits the page, they see all the episodes as if none were in the quicklist. After a brief delay (during which the AJAX call is sent back to the Metal action), the page updates and the episodes that are in the quicklist change. If users are interrupted or confused by the delay, the original state might instead have activity indicators (e.g., spinners) in place of the quicklist icons, but that all comes down to a UX decision.

By now (if not well before), you may have asked yourself about the payoff. Well, on the sample app I describe in my presentation, using progressive caching took a locally-hosted page from 617ms (including database and rendering time) to 135ms (39ms for the cached, static content + 96ms for the AJAX call), which is a decrease of nearly 80%. A second version (which avoided ActiveRecord in the Metal action) took a total of 66ms (43ms for the page, 23ms for the AJAX). These might seem like small numbers in the absolute sense, but when added up over a day, week, or month of an application, they can become extremely significant.

Problems

Of course, there are no perfect techniques, and progressive caching is no exception. We've already mentioned one issue (the delay before the page updates), but there are also problems with accessibility and testing.

Clearly, this approach requires JavaScript on the client. Some people browse without JavaScript (for a variety of reasons), and there are known accessibility issues with dynamically-updating pages. Progressive caching runs into these headlong, so if your application is one that must degrade gracefully, it may not be the correct approach for you.

Also, the state of testing for Rails Metal is ... primitive, to say the least. As far as I know, the best approach is somewhere between an integration test and a homebrewed unit test. That's far from ideal, which means that using Metal can tempt you into testing less. If you do decide to use this strategy, remain vigilant and keep testing!

So, that's it — progressive caching in one form or another has been around for a while, but with the recent changes in Rails (Rack, Metal, and the like), it's poised to come into its own. If you're working on applications that present mostly-similar content to everyone, with pages that vary only in discrete pieces, it may be a viable option for boosting performance by a noticeable amount.

Look Out! It’s Developer Day in DC!

2009-04-15T10:05:00Z

That's right, folks, we're opening up registration for Developer Day in DC. If you're interested in new and exciting developments in technology, you should mark May 30th on your calendar — we've got talks planned on hot-off-the-code-press Ruby 1.9, Cucumber for acceptance testing, and Lily and the Monome, and we're adding more even as you read this.

The inaugural Developer Day in Durham was a hit, and we expect this one to be even better, with more diverse technologies represented and more people to connect with. So join us on May 30th in Falls Church for a day full of engaging talks and good food, all for the low, low price of $50!

Note: If DC is a bit too far for a day trip, don't despair! We're planning on continuing to expand Developer Day to other cities. Email me at ben.scofield@viget.com if you're interested in more details.

Out, Damned Tabs

2009-04-09T14:06:00Z

Like many developers I know, I’m a little bit OCD about code formatting. While there are about as many ideas of properly formatted code as there are coders, I think we can all agree that code with tabs and trailing whitespace is not it. Git has the whitespace = fix option, which does a fine job removing trailing spaces before commits, but leaves the spaces in the working copy, and doesn’t manage tabs at all.

I figured there had to be a better way to automate this type of code formatting, and with help from Kevin McFadden’s post, I think I’ve found one, by configuring TextMate to strip off trailing whitespace and replace tabs with spaces whenever a file is saved. Here’s how to set it up:

Open the Bundle Editor (Bundles > Bundle Editor > Show Bundle Editor).
Create a new bundle using the “+” menu at the bottom of the page. Call it something like “Whitespace.”
With your new bundle selected, create a new command called “Save Current File,” and give it the following settings:
- Save: Current File
- Command(s): blank
- Input: None
- Output: Discard
Start recording a new macro (Bundles > Macros > Start Recording).
Strip out trailing whitespace (Bundles > Text > Converting/Stripping > Remove Trailing Spaces in Document).
Replace tabs with spaces (Text > Convert > Tabs to Spaces).
Save the current document (Bundles > Formatting > Save Current Document).
Stop recording the macro (Bundles > Macros > Stop Recording).
Save the macro (Bundles > Macros > Save Last Recording). Call it something like “Strip Whitespace.”
Click in the Activation (Key Equivalent) text field and hit Command+S.

Alternatively, we’ve packaged the bundle up and put it up on GitHub. Instructions for setting it up are on the page, and patches are encouraged.

How About You?

This approach is working well for me; I’m curious if other people are doing anything like this. If you’ve got an alternative way to deal with extraneous whitespace in your code, please tell us how in the comments.

Pain-Free Pretty URLs in Rails

2009-04-07T20:18:00Z

When working recently on a client project in a later stage of development, it became clear that the client wanted “pretty” URLs for specific models in the application. In particular, they wanted some URLs without an id, meaning the normal Rails to_param trick was out:

def to_param
  "#{self.id}-#{self.name.parameterize}"
end

We needed this type of URL on the User model. We already had a field that was both unique and “URL safe,” the login field, so for this model we could just do

def to_param
  self.login
end

On another model, I decided to create a slug field just for the purpose of pretty URLs. There was already a name field that had to be unique, so I just created a before_validation method to handle the creation and updating of the slug:

before_validation :update_or_create_slug

def to_param
  slug
end

private
def update_or_create_slug
  if self.new_record? || self.name_changed? || self.slug.blank?
    self.slug = self.name.try(:slugize)
  end
end

I used my own “slugize” method, as it’s a little different from the Rails built-in parameterize, and I believe it offers a few minor advantages. I also check for three conditions where we need to assign a new slug:

If it's on a new record.
If the name has changed.
If the slug currently happens to be blank.

You’ll also notice I use the “.try(:slugize)” in case the model is invalid and has nil for the name.

With all this in place, you’d think we’d be all set -- or at least I did.

There was one problem left, though. Let’s say you have a user who is editing an existing model object in the system, and they try to update it with a blank name. This will result in a user with a blank slug field. This wouldn’t seem too bad, but it results in the edit page not being able to render because to_param returns a blank string to the named route URL helper on the form, which raises an ActionController::RoutingError exception.

The solution is to return the old slug value for to_param:

def to_param
  slug_changed? ? slug_was : slug
end

With that solved, we’re all set to go with pretty, id-free URLs.

If you’re in a situation where you don’t have a required unique field like we did on this project, you might want to look at Patrick’s solution from FeedStitch.

Stop Pissing Off Your Designers

2009-04-01T15:00:00Z

A few weeks ago, our local Refresh group pitted me (representing web developers) against Viget designer Mindy in a battle for the ages. Our talk, “Ten Things Designers Do That Piss Developers Off (and Vice Versa),” offered a back-and-forth look at some of the issues that crop up between web professionals. Despite the overwhelming strength of my arguments, I won’t deny that she got some good shots in. Here are some of the key lessons I took away.

Stay off the bandwagon

One of Mindy’s best points was the tendency of developers, when selecting technologies to use on a project, to go with what’s new and hip rather than what’s the best fit or what will yield the best final result. I think we can all relate to learning a new technology or technique and then wanting to immediately apply it to whatever we’re working on.

Technology bandwagon-jumping goes hand-in-hand with another common problem: over-engineering. In my experience, when a chosen technology is a bad fit for a project, it’s typically because it’s too powerful. An over-engineered solution is a nightmare for the next developer — in a past life, I maintained a Spring-powered, Lucene-searchable monstrosity running on dedicated hardware that would have been better served with a WordPress install on Dreamhost.

When selecting technologies, stick with the best fit, whether that’s what you know best or what will lead to the best final product. If you’re just dying to try out some new technology, do what I do: redo your personal site (in lieu of actually posting any new content to it).

Avoid the knee-jerk “No”

Picture this: you’re sitting at your desk one morning, happily reading Hacker News, when an IM window pops up on your screen. It’s your PM, and she’s got a new feature request from the client. It’s not a major change, but it will involve a substantial overhaul of the messaging system you built. What’s your response? Be honest — you give her seventeen reasons why the requested change is a bad idea.

When discussing feature requests, keep in mind that the ultimate goal is to create the best product possible. Requirements change, and though it sucks to complicate elegant solutions, sometimes change is necessary. As an added benefit, if you avoid staying “no” instinctively, when a truly bad idea lands on your plate, your objections will carry a lot more weight.

Remember: you are not the user

Mindy noted a trait common to many developers: a lack of empathy for the user, or rather, the mistaken idea that we ourselves are the typical user. In other words, developers are prone to creating features that they would want to use, regardless of how well they might serve the actual audience of the site.

When deciding on geeky features, it’s important to keep your audience in mind. If you’re designing a site about web productivity, by all means, go nuts — bookmarklets, keyboard shortcuts, customizable RSS feeds, the whole nine yards. But if your site’s intended audience is, say, gardening enthusiasts, your time would probably be better spent elsewhere.

But in the end

We all want to create the best web sites possible. Disagreements arise about definitions of “best”; while a designer wants a site that’s attractive and intuitive, the developer wants one that is stable and maintainable. In the end, these qualities aren’t mutually exclusive — the highest-quality websites have them all.

Mindy has posted her thoughts on the talk, and our slides are available on SlideShare. And if you’re in Durham (or lesser nearby cities), come on out to the next Refresh meeting.

Conditionally Customize Your URLs in Rails

2009-03-31T17:49:00Z

Our primary goal when building FeedStitch was to encourage users to share their feeds by making the process both simple and easily customizable. In the interest of simplicity, we went with OpenID (using RPX through the rpx_now plugin) to combine the registration and login processes so that users could quickly start creating feeds.

As part of encouraging sharing, we wanted to allow users to customize the URL that they could give out to others. The challenge here was keeping this functionality from interrupting the sign-in process while understanding that users may opt out of the customization step altogether.

We implemented this using a quick to_param trick - here's how you can do the same in your Rails application.

The User Model

Start generating:

$ ./script/generate model user username:string
$ rake db:migrate

Our user model is simple - we only ask for an optional username. There are only a few modifications needed to make it useful:

class User < ActiveRecord::Base
  validates_presence_of :username, :on => :update
  validates_format_of :username, :with => /^[a-z]$/i, :on => :update

  def self.find_by_identifier(identifier)
    find(:first, :conditions => ['id = ? OR username = ?', identifier, identifier])
  end

  def to_param
    username || id
  end
end

This will allow us to find a user by either ID or username - the to_param magic will be required later.

The Users Controller

We'll need a list of users and a way to view an individual user's profile:

$ ./script/generate controller users index show

# config/routes.rb
ActionController::Routing::Routes.draw do |map|
  map.user ':identifier', :controller => 'users', :action => 'show'
  map.root :controller => 'users', :action => 'index'
end

Our controller is pretty standard, the only change is using our custom finder in the show action:

class UsersController < ApplicationController
  def index
    @users = User.all
  end

  def show
    @user = User.find_by_identifier(params[:identifier])
  end
end

Views

The view logic is simple, we can use the user_path helper as usual:

# views/users/index.html.erb
<% @users.each do |user| %>
  <p><%= link_to (user.username || 'N/A'), user_path(user) %></p>
<% end %>

# views/users/show.html.erb
<p>User Id: <%= @user.id %></p>
<p>Username: <%= @user.username || 'N/A' %></p>

Testing it Out

Now that all the pieces are in place, create some users to see how this all comes together:

$ ./script/console
>> User.create!
>> User.create!(:username => 'reagent')

Hit the homepage and click on the links for the individual user pages. You'll see that the URL switches between the ID and username as appropriate.

How Does It Work?

The secret to this is how Rails generates the URL identifier when ActiveRecord instances are passed to a URL helper. The to_param method will return a username first and then fall back to returning the ID. This data is then exposed in the URL and gets picked up by the custom finder that we defined on the User model.

An interesting side effect of this process is that all the user profiles that have usernames are still available using their unique IDs. This can be prevented with a little more work, but the benefits of such an approach are minimal.

Update: If you have a field that's required for URL customization, check out Brian's solution to the problem.