Katz Got Your Tongue?

Automatic Flushing: The Rails 3.1 Plan

2010-09-08T01:08:27Z

preamble: this post explains, in some detail, how we will implement a nice performance boost for Rails developers. Understanding the details might help gain the full benefits of the optimization, but you will gain some benefits even if you have no idea how it works.

As you’ve probably seen, DHH announced that we’d be looking at flushing in Rails 3.1 to improve the client-side performance of typical Rails applications.

The most obvious solution, and one that already exists in plugin form, is to allow a layout to have a new flush method, which would immediately flush the contents of the layout to the browser. By putting the flush method below the JavaScript and CSS includes, the browser could begin downloading and evaluating those static assets while the server continues building the page.

Unfortunately, this solution has a major problem: it requires a fairly significant change in the current model of how people build applications. In general, for performance optimizations (including client-side optimizations), we like to make the default as fast as possible, without asking people to understand a brand new paradigm, centered around the optimization.

The problem lies in the fact that a Rails layout is essentially a template with a bunch of holes to fill in.

<html>
  <head>
    <title><%= yield :title %></title>
    <%= javascript_include_tag :defaults %>
    <%= yield :extra_javascripts %>
    <%= stylesheet_link_tag :defaults %>
    <%= yield :extra_stylesheets %>
  </head>
  <body>
    <%= yield :sidebar %>
    <%= yield %>
  </body>
</html>

I this simple example, each yield is a slot that is filled in by the template (usually via content_for). In order to achieve this, Rails evaluates the template first, which populates a Hash with each piece of content. Next, it renders the layout, and each yield checks the Hash for that content. In short, because of the way layouts work, Rails renders the template first, and then the layout.

To get around this, one option would be to say that everything before the flush must not use yield, and must be able to run before the template. Unfortunately, it’s somewhat common for people to set up a content_for(:javascripts) in a template, to keep the JavaScript needed for a particular snippet of HTML close to the HTML. This means that not only does the user have to be careful about what can go above and below the flush, he can no longer use content_for for things high up in the template, which is a fairly significant change to the overall design of Rails applications.

For Rails 3.1, we wanted a mostly-compatible solution with the same programmer benefits as the existing model, but with all the benefits of automatic flushing. After a number of very long discussions on the topic, José Valim came up with the idea of using Ruby 1.9 fibers to jump back and forth between the template and layout.

Let’s start by taking a look at a very simplified version of the current Rails rendering pipeline. First, we set up a Buffer object purely for logging purposes, so we can see what’s happening as we push things onto the buffer.

module Basic
  class Buffer < String
    def initialize(name, context)
      @name    = name
    end
 
    def <<(value)
      super
 
      puts "#{@name} is pushing #{value.inspect}"
    end
  end
end

Next, we create a simple version of ActionView::Base. We implement the content_for method simply, to print out a bit of logging information and stash the value into the @content_for Hash. Note that the real version is pretty similar, with some added logic for capturing the value of the block from ERB.

module Basic
  class ViewContext
    def initialize
      @buffer      = Buffer.new(:main, self)
      @content_for = {}
    end
 
    def content_for(name, value = nil)
      value = yield if block_given?
      puts "Setting #{name} to #{value.inspect}"
      @content_for[name] = value
    end
 
    def read_content(name)
      @content_for[name]
    end
  end
end

Next, we create a number of methods on the ViewContext that look like compiled ERB templates. In real life, the ERB (or Haml) compiler would define these methods.

module Basic
  class ViewContext
    def layout
      @buffer << "<html><head>"
      @buffer << yield(:javascripts).to_s
      @buffer << yield(:stylesheets).to_s
      @buffer << "</head><body>"
      @buffer << yield.to_s
      @buffer << yield(:not_existant).to_s
      @buffer << "</body></html>"
      @buffer
    end
 
    def template
      buffer =  Buffer.new(:template, self)
      content_for(:javascripts) do
        "<script src='application.js'></script>"
      end
      content_for(:stylesheets) do
        "<link href='application.css' rel='stylesheet' />"
      end
      puts "Making a SQL call"
      sleep 1 # Emulate a slow SQL call
      buffer << "Hello world!"
      content_for(:body, buffer)
    end
  end
end

Finally, we define the basic rendering logic:

module Basic
  class ViewContext
    def render
      template
      layout { |value| read_content(value || :body) }
    end
  end
end

As you can see, we first render the template, which will fill up the @content_for Hash, and then call the layout method, with a block which pulls the value from that Hash. This is how yield :javascripts in a layout works.

Unfortunately, this means that the entire template must be rendered first, including the (fake) slow SQL query. We’d prefer to flush the buffer after the JavaScripts and CSS are determined, but before the SQL query is made. Unfortunately, that requires running half of the template method, then continuing with the layout method, retaining the ability to resume the template method later.

You can think of the way that templates are currently rendered (in Rails 2.x and 3.0) like this:

Unfortunately, this makes it very hard to get any more performance juice out without asking the end-developer to make some hard choices. The solution we came up with is to use Ruby 1.9 fibers to allow the rendering to jump back and forth between the template and layout.

Instead of starting with the template and only rendering the layout when ready, we’ll start with the layout, and jump over to the template when a yield is called. Once the content_for that piece is provided by the template, we can jump back to the layout, flush, and continue rendering. As we need more pieces, we can jump back and forth between the template and layout, flushing as we fill in the holes specified by the yield statements.

The implementation is mostly straight-forward:

require "fiber"
 
module Fibered
  class ViewContext < Basic::ViewContext
    def initialize
      super
      @waiting_for = nil
      @fiber       = nil
    end
 
    def content_for(name, value = nil)
      super
      @fiber.resume if @waiting_for == name
    end
 
    def read_content(name)
      content = super
      return content if content
 
      begin
        @waiting_for = name
        Fiber.yield
      ensure
        @waiting_for = nil
      end
 
      super
    end
 
    def layout
      @fiber = Fiber.new do
        super
      end
      @fiber.resume
      @buffer
    end
 
    def render
      layout { |value| read_content(value || :body) }
      template
      @fiber.resume while @fiber.alive?
      @buffer
    end
  end
end

For our fibered implementation, we’ll inherit from Basic::ViewContext, because we want to be able to use the same templates as we used in the original implementation. We update the content_for, read_content, layout and render methods to be fiber-aware. Let’s take them one at a time.

def layout
  @fiber = Fiber.new do
    super
  end
  @fiber.resume
  @buffer
end

First, we wrap the original implementation of layout in a Fiber, and start it right away. Next, we modify the read_content method to become Fiber-aware:

def read_content(name)
  content = super
  return content if content
 
  begin
    @waiting_for = name
    Fiber.yield
  ensure
    @waiting_for = nil
  end
 
  super
end

If the @content_for Hash already has the content, return it right away. Otherwise, say that we’re waiting for the key in question, and yield out of the Fiber. We modify the render method so that the layout is rendered first, followed by the template. As a result, yielding out of the layout will start the template’s rendering.

def render
  layout { |value| read_content(value || :body) }
  template
  @fiber.resume while @fiber.alive?
  @buffer
end

Next, modify the content_for method so that when the content we’re waiting for is provided, we jump back into the layout.

def content_for(name, value = nil)
  super
  @fiber.resume if @waiting_for == name
end

With this setup, the layout and template will ping-pong back and forth, with the layout requesting data, and the template rendering only as far as it needs to go to provide the data requested.

Finally, let’s update the Buffer to take our fibered implementation into consideration.

module Basic
  class Buffer < String
    def initialize(name, context)
      @name    = name
      @fibered = context.fibered?
    end
 
    def <<(value)
      super
 
      if @fibered
        puts "Flushing #{value.inspect}" if @fibered
      else
        puts "#{@name} is pushing #{value.inspect}"
      end
    end
  end
 
  class ViewContext
    def fibered?
      false
    end
  end
end
 
module Fibered
  class ViewContext
    def fibered?
      true
    end
  end
end

Now that we’re rendering the layout in order, we can flush as we go, instead of being forced to wait for the entire template to render before we can start flushing.

It’s worth mentioning that optimal flushing performance will be based on the order of the content_for in your template. If you run your queries first, then put the expensive template rendering, and only finally do the content_for(:javascript) at the end, the flushing behavior will look like this:

Instead of flushing quickly, before the SQL call, things are barely better than they are in Rails 2.3, when the entire template must be rendered before the first flush. Because things are no worse, even in the worst-case scenario, we can make this the default behavior. Most people will see some benefit from it, and people interested in the best performance can order their content_for blocks so they cause the most beneficial flushing.

Even for people willing to put in the effort, this API is better than forcing a manual flush, because you can still put your content_for blocks alongside the templates that they are related to.

Look for this feature in Rails 3.1!

Small Caveat

For the purposes of this simplified example, I assumed that content_for can only be run once, immediately setting the value in the @content_for Hash. However, in some cases, people want to accumulate a String for a particular value. Obviously, we won’t be able to flush until the full String for that value is accumulated.

As a result, we’ll be adding a new API (likely called provide), which will behave exactly the same as content_for, but without the ability to accumulate. In the vast majority of cases, people will want to use provide (for instance, provide :sidebar), and get all the benefits of autoflushing. In a few cases, people will want to be able to continue accumulating a String, and will still be able to use content_for, but the template will not return control to the layout when that happens.

Also note that this (fairly detailed) explanation is not something that you will need to understand as a Rails developer. Instead, you will continue to go about your development as before, using provide if you have just a single piece of content to add, and content_for if you have multiple pieces of content to add, and Rails will automatically optimize flushing for you as well as we can.

A Tale of Abort Traps (or Always Question Your Assumptions)

2010-08-25T15:13:20Z

For a few months now, the bundler team has been getting intermittent reports of segfaults in C extensions that happen when using bundler with rvm. A cursory investigation revealed that the issue was that the C extensions were compiled for the wrong version of Ruby.

For instance, we would get reports of segfaults in nokogiri when using Ruby 1.9 that resulted from the $GEM_HOME in Ruby 1.9 containing a .bundle file compiled for Ruby 1.8. We got a lot of really angry bug reports, and a lot of speculation that we were doing something wrong that could be obviously fixed.

I finally ran into the issue myself, on my own machine a couple days ago, and tracked it down, deep into the fires of Mount Doom. A word of warning: this story may shock you.

It Begins

I usually use rvm for my day-to-day work, but I tend to not use gemsets unless I want a guaranteed clean environment for debugging something. Bundler takes care of isolation, even in the face of a lot of different gems mixed together in a global gemset. Last week, I was deep in the process of debugging another issue, so I was making pretty heavy use of gemset (and rvm gemset empty).

I had switched to Ruby 1.9.2 (final, which had just come out), created a new gemset, and run bundle install on the project I was working on, which included thin in the Gemfile. I had run bundle install to install the gems in the project, and among other things, bundler installed thin “with native extensions”. I use bundler a lot, and I had run this exact command probably thousands of times.

This time, however, I got a segfault in Ruby 1.9, that pointed at the require call to the rubyeventmachine.bundle file.

Debugging

Now that I had the bug on a physical machine, I started debugging. Our working hypothesis was that bundler or Rubygems was somehow compiling gems against Ruby 1.8, even when on Ruby 1.9, but we couldn’t figure out exactly how it could be happening.

The first thing I did was run otool -L on the binary (rubyeventmachine.bundle):

$ otool -L rubyeventmachine.bundle 
rubyeventmachine.bundle:
  /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/libruby.1.dylib
  (compatibility version 1.8.0, current version 1.8.7)
  /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
  /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)

So that’s weird. I had run bundle install from inside an rvm gemset, yet it was being compiled against the libruby that ships with OSX. I had always suspecting that the problem was a leak from another rvm-installed Ruby, so this was definitely a surprise.

Just out of curiosity, I ran which bundle:

$ which bundle
/usr/bin/bundle

Ok, now I knew something was rotten. I printed out the $PATH:

$ echo $PATH
/Users/wycats/.rvm/gems/ruby-1.9.2-p0/bin:...
$ ls /Users/wycats/.rvm/gems/ruby-1.9.2-p0/bin
asdf*             erubis*           rake2thor*        sc-build-number*
autospec*         prettify_json.rb* redcloth*         sc-docs*
bundle*           rackup*           ruby-prof*        sc-gen*
edit_json.rb*     rake*             sc-build*         sc-init*

In other words, a big fat WTF.

I asked around, and some people had the vague idea that there was a $PATH cache in Unix shells. Someone pointed me at this post about clearing the cache.

Sure enough, running hash -r fixed the output of which. I alerted Wayne of rvm to this problem, and he threw in a fix to rvm that cleared the cache when switching rvms. Momentarily, everything seemed fine.

Digging Further

I still didn’t exactly understand how this condition could happen in the first place. When I went digging, I discovered that shells almost uniformly clear the path cache when modifying the $PATH. Since rvm pushes its bin directory onto the $PATH when you switch to a different rvm, I couldn’t understand how exactly this problem was happening.

I read through Chapter 3 of the zsh guide, and it finally clicked:

The way commands are stored has other consequences. In particular, zsh won’t look for a new command if it already knows where to find one. If I put a new ls command in /usr/local/bin in the above example, zsh would continue to use /bin/ls (assuming it had already been found). To fix this, there is the command rehash, which actually empties the command hash table, so that finding commands starts again from scratch. Users of csh may remember having to type rehash quite a lot with new commands: it’s not so bad in zsh, because if no command was already hashed, or the existing one disappeared, zsh will automatically scan the path again; furthermore, zsh performs a rehash of its own accord if $path is altered. So adding a new duplicate command somewhere towards the head of $path is the main reason for needing rehash.

By using the hash command (which prints out all the entries in this cache), I was able to confirm that the same behavior exists in bash, but it seems that the which command (which I was using for testing the problem) implicitly rehashes in bash.

During this time, I also took a look at /usr/bin/bundle, which looks like this:

$ cat /usr/bin/ruby
#!/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
#
# This file was generated by RubyGems.
#
# The application 'bundler' is installed as part of a gem, and
# this file is here to facilitate running it.
#
 
require 'rubygems'
 
version = ">= 0"
 
if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
  version = $1
  ARGV.shift
end
 
gem 'bundler', version
load Gem.bin_path('bundler', 'bundle', version)

As you can see, executable wrappers created by Rubygems hardcode the version of Ruby that was used to install them. This is one of the reasons that rvm keeps its own directory for installed executables.

A Working Hypothesis

It took me some time to figure all this out, during which time I was speaking to a bunch of friends and the guys in #zsh. I formed a working hypothesis. I figured that people were doing something like this:

Install bundler onto their system, and use it there
Need to work on a new project, so switch to rvm, and create a new gemset
Run bundle install, resulting in an error
Run gem install bundler, to install bundler
Run bundle install, which works, but has a subtle bug

Let’s walk through each of these steps, and unpack exactly what happens.

Install bundler on their system

This results in an executable at /usr/bin/bundle that looks like this:

#!/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
#
# This file was generated by RubyGems.
#
# The application 'bundler' is installed as part of a gem, and
# this file is here to facilitate running it.
#
 
require 'rubygems'
 
version = ">= 0"
 
if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
  version = $1
  ARGV.shift
end
 
gem 'bundler', version
load Gem.bin_path('bundler', 'bundle', version)

There are two important things here. First, it hardcodes the shebang to the system Ruby location. Second, it uses Rubygems to look up the location of the executable shipped with bundler, which will respect GEM_HOME.

Switch to rvm, and create a new gemset

Switching to an rvm gemset does a few relevant things. First, it prepends ~/.rvm/gems/ruby-1.9.2-p0@gemset/bin onto the $PATH. This effectively resets the shell’s built-in command cache. Second, it sets $GEM_HOME to ~/.rvm/gems/ruby-1.9.2-p0. The $GEM_HOME is where Ruby both looks for gems as well as where it installs gems.

Run `bundle install`, resulting in an error

Specifically,

$ bundle install
/Library/Ruby/Site/1.8/rubygems.rb:777:in `report_activate_error': 
Could not find RubyGem bundler (>= 0) (Gem::LoadError)
	from /Library/Ruby/Site/1.8/rubygems.rb:211:in `activate'
	from /Library/Ruby/Site/1.8/rubygems.rb:1056:in `gem'
	from /usr/bin/bundle:18

Most people don’t take such a close look at this error, interpreting it as the equivalent of command not found: bundle. What’s actually happening is a bit different. Since bundle is installed at /usr/bin/bundle, the shell finds it, and runs it. It uses the system Ruby (hardcoded in its shebang), and the $GEM_HOME set by rvm. Since I just created a brand new gemset, the bundler gem is not found in $GEM_HOME. As a result, the line gem 'bundler', version in the executable fails with the error I showed above.

However, because the shell found the executable, it ends up in the shell’s command cache. You can see the command cache by typing hash. In both bash and zsh, the command cache will include an entry for bundle pointing at /usr/bin/bundle. zsh is a more aggressive about populating the cache, so you’ll see the list of all commands in the system in the command cache (which, you’ll recall, was reset when you first switched into the gemset, because $PATH was altered).

Run `gem install bundler`, to install bundler

This will install the bundle executable to ~/.rvm/gems/ruby-1.9.2-p0@gemset/bin, which is the first entry on the $PATH. It will also install bundler to the $GEM_HOME.

Run `bundle install`, which works, but has a subtle bug

Here’s where the fun happens. Since we didn’t modify the cache, or call hash -r, the next call to bundle still picks up the bundle in /usr/bin/bundle, which is hardcoded to use system Ruby. However, it will use the version of Bundler we just installed, since it was installed to $GEM_HOME, and Rubygems uses that to look for gems. In other words, even though we’re using system Ruby, the /usr/bin/bundle executable will use the rvm gemset’s Bundler gem we just installed.

Additionally, because $GEM_HOME points to the rvm gemset, bundler will install all its gems to the rvm gemset. Taken together, these factors make it almost completely irrelevant that the system Ruby was used to install gems. After all, who cares which Ruby installed the gems, as long as they end up in the right place for the rvm’s Ruby.

There are actually two problems. First, as we’ve seen, the version of Ruby that installs a gem also hardcodes itself into the shebang of the executables for the gem.

Try this experiment:

$ rvm use 1.9.2
$ rvm gemset create experiment
$ rvm gemset use experiment
$ /usr/bin/gem install rack
$ cat ~/.rvm/gems/ruby-1.9.2-p0@experiment/bin/rackup
#!/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
#
# This file was generated by RubyGems.
#
# The application 'rack' is installed as part of a gem, and
# this file is here to facilitate running it.
#
 
require 'rubygems'
 
version = ">= 0"
 
if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
  version = $1
  ARGV.shift
end
 
gem 'rack', version
load Gem.bin_path('rack', 'rackup', version)

As a result, any gems that bundler installed will put their executables in the gemset’s location, but still be hardcoded to use the system’s Ruby. This is not actually the problem we’re encountering here, but it could cause some weird results.

More concerningly, since our system Ruby installed the gems, it will also link any C extensions it compiles against its own copy of libruby. This shouldn’t be a major issue if you’re using a version of Ruby 1.8 in rvm, but it is a major issue if you’re using Ruby 1.9.

Now, we have a gem stored in your rvm’s Ruby that has a .bundle in it that is linked against the wrong Ruby. When running your application, Ruby 1.9 will try to load it, and kaboom: segfault.

Postscript

The crazy thing about this story is that it’s a lot of factors conspiring to cause problems. The combination of the shell command cache, $GEM_HOME making it look like SYSTEM Ruby was doing the right thing, and hardcoding the version of Ruby in the shebang line of installed gems made this segfault possible.

Thankfully, now that we figured the problem out, the latest version of rvm fixes it. The solution: wrap the gem command in a shell function that calls hash -r afterward.

Amazing.

Using >= Considered Harmful (or, What’s Wrong With >=)

2010-08-21T16:14:00Z

TL;DR Use ~> instead.

Having spent far, far too much time with Rubygems dependencies, and the problems that arise with unusual combinations, I am ready to come right out and say it: you basically never, ever want to use a >= dependency in your gems.

When you specify a dependency for your gem, it should mean that you are fairly sure that the unmodified code in the released gem will continue to work with any future version of the dependency that matches the version you specified. So for instance, let’s take a look at the dependencies listed in the actionpack gem:

activemodel (= 3.0.0.rc, runtime)
activesupport (= 3.0.0.rc, runtime)
builder (~> 2.1.2, runtime)
erubis (~> 2.6.6, runtime)
i18n (~> 0.4.1, runtime)
rack (~> 1.2.1, runtime)
rack-mount (~> 0.6.9, runtime)
rack-test (~> 0.5.4, runtime)
tzinfo (~> 0.3.22, runtime)

Since we release the Rails gems as a unit, we declare hard dependencies on activemodel and activesupport. We declare soft dependencies on builder, erubis, i18n, rack, rack-mount, rack-test, and tzinfo.

You might not know what exactly the ~> version specifier means. Essentially, it decomposes into two specifiers. So ~> 2.1.2 means >= 2.1.2, < 2.2.0. In other words, it means “2.1.x, but not less than 2.1.2″. Specifying ~> 1.0, like many people do for Rack, means “any 1.x”.

You should make your dependencies as soft as the versioning scheme and release practices of your dependencies will allow. If you’re monkey-patching a library outside of its public API (not a very good practice for libraries), you should probably stick with an = dependency.

One thing for certain though: you cannot be sure that your gem works with every future version of your dependencies. Sanely versioned gems take the opportunity of a major release to break things, and until you have actually tested against the new versions, it’s madness to claim compatibility. One example: a number of gems have dependencies on activesupport >= 2.3. In a large number of cases, these gems do not work correctly with ActiveSuport 3.0, since we changed how components of ActiveSupport get loaded to make it easier to cherry-pick.

Now, instead of receiving a version conflict, users of these gems will get cryptic runtime error messages. Even worse, everything might appear to work, until some weird edge-case is exercised in production, and which your tests would have caught.

But What Happens When a New Version is Released?

One reason that people use the activesupport >= 2.3 is that, assuming Rails maintains backward-compatibility, their gem will continue to work in newer Rails environments without any difficulty. If everything happens to work, it saves you the time of running their unit tests against newer versions of dependencies and cutting a new release.

As I said before, this is a deadly practice. By specifying appropriate dependencies (based on your confidence in the underlying library’s versioning scheme), you will have a natural opportunity to run your test suite against the new versions, and release a new gem that you know actually works.

This does mean that you will likely want to release patch releases of old versions of your gem. For instance, if I have AuthMagic 1.0, which worked against Rails 2.3, and I release AuthMagic 2.0 once Rails 3.0 comes out, it makes sense to continue patching AuthMagic 1.0 for a little while, so your Rails 2.3 users aren’t left out in the cold.

Applications and Gemfiles

I should be clear that this versioning advice doesn’t necessarily apply to an application using Bundler. That’s because the Gemfile.lock, which you should check into version control, essentially converts all >= dependencies into hard dependencies. However, because you may want to run bundle update at some point in the future, which will update everything to the latest possible versions, you might want to use version specifiers in your Gemfile that seem likely to work into the future.

Threads (in Ruby): Enough Already

2010-08-14T20:25:44Z

For a while now, the Ruby community has become enamored in the latest new hotness, evented programming and Node.js. It’s gone so far that I’ve heard a number of prominent Rubyists saying that JavaScript and Node.js are the only sane way to handle a number of concurrent users.

I should start by saying that I personally love writing evented JavaScript in the browser, and have been giving talks (for years) about using evented JavaScript to sanely organize client-side code. I think that for the browser environment, events are where it’s at. Further, I don’t have any major problem with Node.js or other ways of writing server-side evented code. For instance, if I needed to write a chat server, I would almost certainly write it using Node.js or EventMachine.

However, I’m pretty tired of hearing that threads (and especially Ruby threads) are completely useless, and if you don’t use evented code, you may as well be using a single process per concurrent user. To be fair, this has somewhat been the party line of the Rails team years ago, but Rails has been threadsafe since Rails 2.2, and Rails users have been taking advantage of it for some time.

Before I start, I should be clear that this post is talking about requests that spent a non-tiny amount of their time utilizing the CPU (normal web requests), even if they do spend a fair amount of time in blocking operations (disk IO, database). I am decidedly not talking about situations, like chat servers where requests sit idle for huge amounts of time with tiny amounts of intermittent CPU usage.

Threads and IO Blocking

I’ve heard a common misperception that Ruby inherently “blocks” when doing disk IO or making database queries. In reality, Ruby switches to another thread whenever it needs to block for IO. In other words, if a thread needs to wait, but isn’t using any CPU, Ruby’s built-in methods allow another waiting thread to use the CPU while the original thread waits.

If every one of your web requests uses the CPU for 30% of the time, and waits for IO for the rest of the time, you should be able to serve three requests in parallel, coming close to maxing out your CPU.

Here’s a couple of diagrams. The first shows how people imagine requests work in Ruby, even in threadsafe mode. The second is how an optimal Ruby environment will actually operate. This example is extremely simplified, showing only a few parts of the request, and assuming equal time spent in areas that are not necessarily equal.

I should be clear that Ruby 1.8 spends too much time context-switching between its green threads. However, if you’re not switching between threads extremely often, even Ruby 1.8′s overhead will amount to a small fraction of the total time needed to serve a request. A lot of the threading benchmarks you’ll see are testing pathological cases involve huge amounts of threads, not very similar to the profile of a web server.

(if you’re thinking that there are caveats to my “optimal Ruby environment”, keep reading)

“Threads are just HARD”

Another common gripe that pushes people to evented programming is that working with threads is just too hard. Working hard to avoid sharing state and using locks where necessary is just too tricky for the average web developer, the argument goes.

I agree with this argument in the general case. Web development, on the other hand, has an extremely clean concurrency primitive: the request. In a threadsafe Rails application, the framework manages threads and uses an environment hash (one per request) to store state. When you work inside a Rails controller, you’re working inside an object that is inherently unshared. When you instantiate a new instance of an ActiveRecord model inside the controller, it is rooted to that controller, and is therefore not used between live threads.

It is, of course, possible to use global state, but the vast majority of normal, day-to-day Rails programming (and for that matter, programming in any web framework in any language with a request model) is inherently threadsafe. This means that Ruby will transparently handle switching back and forth between active requests when you do something blocking (file, database, or memcache access, for instance), and you don’t need to personally manage the problems the arise when doing concurrent programming.

This is significantly less true about applications, like chat servers, that keep open a huge number of requests. In those cases, a lot of the application logic happens outside the individual request, so you need to personally manage shared state.

Historical Ruby Issues

What I’ve been talking about so far is how stock Ruby ought to operate. Unfortunately, a group of things have historically conspired to make Ruby’s concurrency story look much worse than it actually ought to be.

Most obviously, early versions of Rails were not threadsafe. As a result, all Rails users were operating with a mutex around the entire request, forcing Rails to behave like the first “Imagined” diagram above. Annoyingly, Mongrel, the most common Ruby web server for a few years, hardcoded this mutex into its Rails handler. As a result, if you spun up Rails in “threadsafe” mode a year ago using Mongrel, you would have gotten exactly zero concurrency. Also, even in threadsafe mode (when not using the built-in Rails support) Mongrel spins up a new thread for every request, not exactly optimal.

Second, the most common database driver, mysql is a very poorly behaved C extension. While built-in I/O (file or pipe access) correctly alerts Ruby to switch to another thread when it hits a blocking region, other C extensions don’t always do so. For safety, Ruby does not allow a context switch while in C code unless the C code explicitly tells the VM that it’s ok to do so.

All of the Data Objects drivers, which we built for DataMapper, correctly cause a context switch when entering a blocking area of their C code. The mysqlplus gem, released in March 2009, was designed to be a drop-in replacement for the mysql gem, but fix this problem. The new mysql2 gem, written by Brian Lopez, is a drop-in replacement for the old gem, also correctly handles encodings in Ruby 1.9, and is the new default MySQL driver in Rails.

Because Rails shipped with the (broken) mysql gem by default, even people running on working web servers (i.e. not mongrel) in threadsafe mode would have seen a large amount of their potential concurrency eaten away because their database driver wasn’t alerting Ruby that concurrent operation was possible. With mysql2 as the default, people should see real gains on threadsafe Rails applications.

A lot of people talk about the GIL (global interpreter lock) in Ruby 1.9 as a death knell for concurrency. For the uninitiated, the GIL disallows multiple CPU cores from running Ruby code simultaneously. That does mean that you’ll need one Ruby process (or thereabouts) per CPU core, but it also means that if your multithreaded code is running correctly, you should need only one process per CPU core. I’ve heard tales of six or more processes per core. Since it’s possible to fully utilize a CPU with a single process (even in Ruby 1.8), these applications could get a 4-6x improvement in RAM usage (depending on context-switching overhead) by switching to threadsafe mode and using modern drivers for blocking operations.

JRuby, Ruby 1.9 and Rubinius, and the Future

Finally, JRuby already runs without a global interpreter lock, allowing your code to run in true parallel, and to fully utilize all available CPUs with a single JRuby process. A future version of Rubinius will likely ship without a GIL (the work has already begun), also opening the door to utilizing all CPUs with a single Ruby process.

And all modern Ruby VMs that run Rails (Ruby 1.9′s YARV, Rubinius, and JRuby) use native threads, eliminating the annoying tax that you need to pay for using threads in Ruby 1.8. Again, though, since that tax is small relative to the time for your requests, you’d likely see a non-trivial improvement in latency in applications that spend time in the database layer.

To be honest, a big part of the reason for the poor practical concurrency story in Ruby has been that the Rails project didn’t take it seriously, which it difficult to get traction for efforts to fix a part of the problem (like the mysql driver).

We took concurrency very seriously in the Merb project, leading to the development of proper database drivers for DataMapper (Merb’s ORM), and a top-to-bottom understanding of parts of the stack that could run in parallel (even on Ruby 1.8), but which weren’t. Rails 3 doesn’t bring anything new to the threadsafety of Rails itself (Rails 2.3 was threadsafe too), but by making the mysql2 driver the default, we have eliminated a large barrier to Rails applications performing well in threadsafe mode without any additional research.

UPDATE: It’s worth pointing to Charlie Nutter’s 2008 threadsafety post, where he talked about how he expected threadsafe Rails would impact the landscape. Unfortunately, the blocking MySQL driver held back some of the promise of the improvement for the vast majority of Rails users.

Everyone Who Tried to Convince Me to use Vim was Wrong

2010-07-29T06:27:56Z

A couple weeks ago, I took the plunge and switched to vim (MacVIM, to be precise). It wasn’t the first time I tried to make the switch, and I had pretty much written it off entirely.

Why? Because the past few times I tried switching to vim, I took the advice of a master vim user, and quickly sunk into the quicksand of trying to learn a new tool. In every prior attempt, I gave vim a few days before I gave up. And every time, I managed to get virtually no work done the entire time, spending about 90 percent of my day fighting with my editor (a more charitable way to put it would be “learning my editor”).

Invariably, the master vim users that were helping me make the switch would encourage me to stick it out. “If you just give it a few weeks, you’ll never want to switch back.”

The trouble was, I had work to do. I could only switch editors if the new editor did not significantly impede on my day-to-day work. I can already hear the responses: “That’s simply impossible. It’s a new editor designed for advanced users. You’ll just have to put up with the pain until you get used to it.”

Here’s the thing, though: I didn’t really have to put up with a huge amount of pain when switching to Textmate for the first time. In fact, it was downright pleasant.

The last few times someone tried to get me to switch to vim, I issued them a simple challenge. Can you tell me a way to switch that will not significantly reduce my productivity for the first few weeks. It wasn’t a challenge that was intended to fully shut down discussion. When I really thought about it, Textmate wasn’t doing all that much for me. It was a glorified Notepad which had working syntax highlighting and understand where to put the cursor when I hit enter (most of the time).

I don’t actually use “snippets” all that often, or all that many “commands”. I don’t mind the extensibility of Textmate, but I’m not a hardcore Textmate hacker myself, meaning that I’m ok with any editor that has the same level of extensibility that Textmate has (namely, all of them).

Despite what I considered a relatively reasonable request, my challenge was met with disdain and even anger by most of the people I talked to. “If you feel that way, Vim probably isn’t for you.” “You’re learning a new EDITOR for God’s sakes. Of COURSE there’s going to be a learning curve.”

I had written off the entire sorry affair.

A few weeks ago, Carl told me that he was playing with Vim. His explanation was that he had seen a number of people be really productive with it, and he was curious. Carl is definitely willing to put up with more pain to learn something new than I am, so I issued the same challenge to him.

Perhaps because he wasn’t steeped in hardcore vim hacker lore, he didn’t angrily dismiss the entire premise of my question. Thinking about it a bit more, I realized that most of the people who had tried to get me into vim had suggested that I dive in head first. “First thing: turn off the arrow keys.” “Don’t use the mouse. Force yourself to use the keyboard.”

Carl convinced me to use vim for the first couple of days pretty much exactly as I use Texmate (with the exception of having to switch between normal and insert modes). I installed NERDTree on MacVIM, grabbed the most common vim “packages”, and was off to the races. (I should note that I installed topfunky’s PeepOpen, which definitely helped with a very common workflow that I find it hard to live without).

For the first day, I clunked around by using my mouse’s scroll wheel, clicking and highlighting things, and spending most of my time in insert mode. It was slightly less productive than Textmate, but mostly in the range of what I’d expect switching to a new tool. In short, while I felt a bit out of sorts, I was able to get plenty of work done that first day.

As the days went on, I learned a few commands here and there. The first big one for me was ci as in ci " (it means: replace what’s inside the next set of " and go into insert mode). This singlehandedly made up for most of the productivity losses I was feeling from learning a new tool. Throw in o, O, A, :N and /search and I was already quite a bit more productive than I had been in Textmate.

Sure, I’m still plodding around in some cases, but only a handful of days later, using Textmate for anything feels clunky (most commonly, I try to use o or O to insert a new line above or below the one I’m currently on).

I was able to get here because I used my mouse wheel and button, arrow keys, apple-f to find text, apple-s to save files, and a whole slew of other common idioms, instead of grinding to a halt and trying to switch all of my practices at once.

To those who would say “that’s obvious; of course you learn vim incrementally”, I would simply say that having spoken to a number of vim users in the past, I never got that advice. Instead, I got a lot of advice about turning off my arrow keys, disallowing the use of the mouse, and learning the (MORE EFFICIENT!!!) vim ways to do everything, all at once. People just couldn’t stomach the idea of me continuing to use an outmoded practice (like apple-f) when vim had much better tools available just a (huge volume of) memorization away.

To those who are considering using vim, my recommendation is to use MacVIM, NERDTree, PeepOpen (or command-t), and use the mouse, arrow keys, and familiar OSX’isms all you want. Very quickly, it will become obvious that there’s a better way to do all kinds of things, and you can pile on the newly found efficiency once you’ve successfully made the switch without losing the ability to do work in the short-run.

What’s New in Bundler 1.0.0.rc.1

2010-07-26T19:22:17Z

Taking into consideration the huge amount of feedback we received during the Bundler 0.9 series, we streamlined Bundler 1.0 significantly, and made it fit user expectations better.

Whether you have used bundler before or not, the easiest way to get up to speed is to read the following notes and go to http://gembundler.com/v1.0 for more in-depth information.

(note that gembundler.com is still being updated for the 1.0 changes, and should be ready for the final release).

Starting a new project with bundler

When you generate a new Rails application, Rails will create a Gemfile for you, which has everything needed to boot your application.

Otherwise, you can use bundle init to create a stub Gemfile, ready to go.

First, run bundle install to make sure that you have all the needed dependencies. If you already do, this process will happen instantaneously.

Bundler will automatically create a file called Gemfile.lock. This file is a snapshot of your application’s dependencies at that time.

You SHOULD check both files into version control. This will ensure that all team members (as well as your production server) are working with identical dependencies.

Checking out an existing project using bundler

After checking out an existing project using bundler, check to make sure that the Gemfile.lock snapshot is checked in. If it is not, you may end up using different dependencies than the person who last used and tested the project.

Next, run bundle install. This command will check whether you already have all the required dependencies in your system. If you do not, it will fetch the dependencies and install them.

Updating dependencies

If you modify the dependencies in your Gemfile, first try to run bundle install, as usual. Bundler will attempt to update only the gems you have modified, leaving the rest of the snapshot intact.

This may not be possible, if the changes conflict with other gems in the snapshot (or their dependencies). If this happens, Bundler will instruct you to run bundle update. This will re-resolve all dependencies from scratch.

The bundle update command will update the versions of all gems in your Gemfile, while bundle install will only update the gems that have changed since the last bundle install.

After modifying dependencies, make sure to check in your Gemfile and Gemfile.lock into version control.

By default, gems are installed to your system

If you follow the instructions above, Bundler will install the gems into the same place as gem install.

If necessary, Bundler will prompt you for your sudo password.

You can see the location of a particular gem with bundle show [GEM_NAME]. You can open it in your default editor with bundle open [GEM_NAME].

Bundler will still isolate your application from other gems. Installing your gems into a shared location allows multiple projects to avoid downloading the same gem over and over.

You might want to install your bundled gems to a different location, such as a directory in the application itself. This will ensure that each application has its own copies of the gems, and provides an extra level of isolation.

To do this, run the install command with bundle install /path/to/location. You can use a relative path as well: bundle install vendor.

In RC1, this command will use gems from the system, if they are already there (it only affects new gems). To ensure that all of your gems are located in the path you specified, run bundle install path --disable-shared-gems.

In Bundler 1.0 final, bundle install path will default to --disable-shared-gems.

Deployment

When deploying, we strongly recommend that you isolate your gems into a local path (using bundle install path --disable-shared-gems). The final version of bundler will come with a --production flag, encapsulating all of the best deployment practices.

For now, please follow the following recommendations (described using Capistrano concepts):

Make sure to always check in a Gemfile.lock that is up to date. This means that after modifying your Gemfile, you should ALWAYS run bundle install.
Symlink the vendor/bundle directory into the application’s shared location (symlink release_path/current/vendor/bundle to release_path/shared/bundled_gems)
Install your bundle by running bundle install vendor/bundle --disable-shared-gems

Encodings, Unabridged

2010-05-20T11:29:04Z

I wrote somewhat extensively about the problem of encodings in Ruby 1.9 in general last week.

For those who didn’t read that post, let me start with a quick refresher.

What’s an Encoding?

An encoding specifies how to take a list of characters (such as “hello”) and persist them onto disk as a sequence of bytes. You’re probably familiar with the ASCII encoding, which specifies how to store English characters in a single byte each (taking up the space in 0-127, leaving 128-255 empty).

Another common encoding is ISO-8859-1 (or Latin-1), which uses ASCII’s designation for the first 127 characters, and designates the numbers 128-255 for Latin characters (such as “é” or “ü”).

Obviously, 255 characters isn’t enough for all languages, so there are a number of ISO-8859-* encodings which each designate numbers 128-255 for their own purposes (for instance, ISO-8859-5 uses that space for Russian characters).

Unfortunately, the raw bytes themselves do not contain an “encoding specifier” or any kind, and the exact same bytes can either mean something in Western characters, Russian, Japanese, or any other language, depending on the character set that was originally used to store off the characters as bytes.

As a general rule, protocols (such as HTTP), provide a mechanism for specifying the encoding. For instance, in HTTP, you can specify the encoding in the Content-Type header, like this: Content-Type: text/html; charset=UTF-8. However, this is not a requirement, so it is possible to receive some content over HTTP and not know its encoding.

This brings us to an important point: Strings have no inherent encoding. By default, Strings are just BINARY data. Since the data
could be encoded using any number of different incompatible encodings, simply combining BINARY data from different sources could easily result in a corrupted String.

When you see a diamond with a question mark inside on the web, or gibberish characters (like a weird A with a 3/4 symbol), you’re seeing a mistaken attempt to combine binary data encoded differently into a single String.

What’s Unicode

Unicode is an effort to map every known character (to a point) to a number. Unicode does not define an encoding (how to represent those numbers in bytes). It simply provides a unique number for each known character.

Unicode tries to unify characters from different encodings that represent the same character. For instance, the A in ASCII, the A in
ISO-8859-1, and the A in the Japanese encoding SHIFT-JIS all map to the same Unicode character.

Unicode also takes pains to ensure round-tripping between existing encodings and Unicode. Theoretically, this should mean that it’s possible to take some data
encoded using any known encoding, use Unicode tables to map the characters to Unicode numbers, and then use the reverse versions of those tables to map the Unicode numbers back into the original encoding.

Unfortunately, both of these characteristics cause some problems for Asian character sets. First, there have been some historical errors in the process of
unification, which requires the Unicode committee to properly identify which characters in different existing Chinese, Japanese and Korean (CJK) character sets actually represent the same character.

In Japanese, personal names use slight variants of the non-personal-name version of the same character. This would be equivalent to the difference (in English) between “Cate” and “Kate”. Many of these characters (sometimes called Gaiji) cannot be represented in Unicode at all.

Second, there are still hundreds of characters in some Japanese
encodings (such as the Microsoft encoding to SHIFT-JIS called CP932 or Windows-31J) that simply do not round-trip through Unicode.

To make matters worse, Java and MySQL use a different mapping table than the standard Unicode mapping tables (making “This costs ¥5″ come out in Unicode as “This costs \5″). The standard Unicode mapping tables handle this particular case correctly (but cannot fully solve the round-tripping problem), but these quirks only serve to further raise doubts about Unicode in the minds of Japanese developers.

For a lot more information on these issues, check out the XML Japanese Profile document created by the W3C to explain how to deal with some of these problems in XML documents.

In the Western world, all encodings in use do not have these problems. For instance, it is trivial to take a String encoded as ISO-8859-1, convert it into
Unicode, and then convert it back into ISO-8859-1 when needed.

This means that for most of the Western world, it is a good idea to
use Unicode as the “one true character set” inside programming languages. This means that programmers can treat Strings as simple sequences of Unicode code points (several
code points may add up to a single character, such as the ¨ code point, which can be
applied to other code points to form characters like ü). In the Asian world, while this can sometimes be a good strategy, it is often significantly simpler to use the original encoding and handle merging Strings in different encodings together manually (when an appropriate decision about the tradeoffs around fidelity can be made).

Before I continue, I would note that the above is a vast simplification of the
issues surrounding Unicode and Japanese. I believe it to be a fair characterization,
but die-hard Unicode folks, and die-hard anti-Unicode folks would possibly disagree
with some elements of it. If I have made any factual errors, please let me know.

A Digression: UTF-*

Until now, I have talked only about “Unicode”, which simply maps code points
to numbers. Because Unicode uses counting numbers, it can accommodate as many
code points as it wants.

However, it is not an encoding. In other words, it does not specify how to
store the numbers on disk. The most obvious solution would be to use a few
bytes for each character. This is the solution that UTF-32 uses, specifying
that each Unicode character be stored as four bytes (accommodating over 4 billion characters). While this has the advantage of being simple, it also uses huge amounts of memory and disk space compared to the original encodings (like ASCII, ISO-8859-1 and SHIFT-JIS) that it is replacing.

On the other side of the spectrum is UTF-8. UTF-8 uses a single byte for English characters, using the exact same mapping as ASCII. This means that a UTF-8 string that contains only characters found in ASCII will have the identical bytes as a String stored in ASCII.

It then uses the high bit (the bytes representing 128-255) to specify a series of escape characters that can specify a multibyte character. This means that Strings using Western characters use relatively few bytes (often comparable with the original encodings Unicode replaces), because they are in the low area of the Unicode space, while the large number of characters in the Asian languages use more bytes than their native encodings, because they use characters with larger Unicode numbers.

This is another reason some Asian developers resent Unicode; while it does not significantly increase the memory requirements for most Western documents, it does so for Asian documents.

For the curious, UTF-16 uses 16-bits for the most common characters (the BMP, or basic multilingual plane), and 32-bits to represent characters from planes 1 through 16. This means that UTF-8 is most efficient for Strings containing mostly ASCII characters. UTF-8 and UTF-16 are approximately equivalent for Strings containing mostly characters outside ASCII but inside the the BMP. For Strings containing mostly characters outside the BMP, UTF-8, UTF-16, and UTF-32 are approximately equivalent. Note that when I say “approximately equivalent”, I’m not saying that they’re exactly the same, just that the differences are small in large Strings.

Of the Unicode encodings, only UTF-8 is compatible with ASCII. By this I mean that if a String is valid ASCII, it is also valid UTF-8. UTF-16 and UTF-32 encode ASCII characters using two or four bytes.

What Ruby 1.9 Does

Accepting that there are two very different ways of handling this problem, Ruby 1.9 has a String API that is somewhat different from most other languages, mostly influenced by the issues I described above in dealing with Japanese in Unicode.

First, Ruby does not mandate that all Strings be stored in a single internal encoding. Unfortunately, this is not possible to do reliably with common Japanese encodings (CP932, aka Windows-31J has 300 characters than cannot round-trip through Unicode without corrupting data). It is possible that the Unicode committee will some day fully solve these problems to everyone’s satisfaction, but that day has not yet come.

Instead, Ruby 1.9 stores Strings as the original sequence of bytes, but allows a String to be tagged with its encoding. It then provides a rich API for converting Strings from one encoding to another.

string = "hello"                     # by default, string is encoded as "ASCII"
string.force_encoding("ISO-8859-1")  # this simply retags the String as ISO-8859-1
                                     # this will work since ISO-8859-1
                                     # is a superset of ASCII.
 
string.encode("UTF-8")               # this will ask Ruby to convert the bytes in
                                     # the current encoding to bytes in
                                     # the target encoding, and retag it with the
                                     # new encoding
                                     #
                                     # this is usually a lossless conversion, but
                                     # can sometimes be lossy

A more advanced example:

# encoding: UTF-8
 
# first, tell Ruby that our editor saved the file using the UTF-8 encoding.
# TextMate does this by default. If you lie to Ruby, very strange things
# will happen
 
utf8 = "hellö"
iso_8859_1 = "hellö".encode("ISO-8859-1")
 
# Because we specified an encoding for this file, Strings in here default
# to UTF-8 rather than ASCII. Note that if you didn't specify an encoding
# characters outside of ASCII will be rejected by the parser.
 
utf8 << iso_8859_1
 
# This produces an error, because Ruby does not automatically try to
# transcode Strings from one encoding into another. In practice, this
# should rarely, if ever happen in applications that can rely on
# Unicode; you'll see why shortly
 
utf8 << iso_8859_1.encode("UTF-8")
 
# This works fine, because you first made the two encodings the same

The problems people are really having

The problem of dealing with ISO-8859-1 encoded text and UTF-8 text in the same Ruby is real, and we’ll see soon how it is handled in Ruby. However, the problems people have been having are not of this variety.

If you examine virtually all of the bug reports involving incompatible encoding exceptions, you will find that one of the two encodings is ASCII-8BIT. In Ruby, ASCII-8BIT is the name of the BINARY encoding.

So what is happening is that a library somewhere in the stack is handing back raw bytes rather than encoded bytes. For a long time, the likely perpetrator here was database drivers, which had not been updated to properly encode the data they were getting back from the database.

There are several other potential sources of binary data, which we will discuss in due course. However, it’s important to note that a BINARY encoded String in Ruby 1.9 is the equivalent of a byte[] in Java. It is a type that cannot be reasonably concatenated onto an encoded String. In fact, it is best to think of BINARY encoded Strings as a different class with many methods in common.

In practice, as Ruby libraries continue to be updated, you should rarely ever see BINARY data inside of your application. If you do, it is because the library that handed it to you genuinely does not know the encoding, and if you want to combine it with non-BINARY String, you will need to convert it into an encoded String manually (using force_encoding).

Why this is, in practice, a rare problem

The problem of incompatible encodings is likely to happen in Western applications only when combining ISO-8859-* data with Unicode data.

In practice, most sources of data, without any further work, are already encoded as UTF-8. For instance, the default Rails MySQL connection specifies a UTF-8 client encoding, so even an ISO-8859-1 database will return UTF-8 data.

Many other data sources, such as MongoDB, only support UTF-8 data internally, so their Ruby 1.9-compatible drivers already return UTF-8 encoded data.

Your text editor (TextMate) likely defaults to saving your templates as UTF-8, so the characters in the templates are already encoded in UTF-8.

This is why Ruby 1.8 had the illusion of working. With the exception of some (unfortunately somewhat common) edge-cases, most of your data is already encoded in UTF-8, so simply treating it as BINARY data, and smashing it all together (as Ruby 1.8 does) works fairly reliably.

The only reason why this came tumbling down in Ruby 1.9 is that drivers that should have returned Strings tagged with UTF-8 were returning Strings tagged with BINARY, which Ruby rightly refused to concatenate with UTF-8 Strings. In other words, the vast majority of encoding problems to date are the result of buggy Ruby libraries.

Those libraries, almost entirely, have now been updated. This means that if you use UTF-8 data sources, which you were likely doing by accident already, everything will continue to work as it did in Ruby 1.8.

Digression: force_encoding

When people encounter this problem for the first time, they are often instructed by otherwise well-meaning people to simply call force_encoding("UTF-8") on the offending String.

This will work reliably if the original data is stored in UTF-8, which is often true about the person who made the original suggestion. However, it will mysteriously fail to work (resulting in “�” characters appearing) if the original data is encoded in ISO-8859-1. This can cause major confusion because some people swear up and down that it’s working and others can clearly see that it’s not.

Additionally, since ISO-8859-1 and UTF-8 are both compatible with ASCII, if the characters being force_encoded are ASCII characters, everything will appear to work until a non-ASCII character is entered one day. This further complicates efforts of members of the community to identify and help resolve issues if they are not fluent in the general issues surrounding encodings.

I’d note that this particular issue (BINARY data entering the system that is actually ISO-8859-1) would cause similar problems in Java and Python, which would either silently assume Unicode, or present a byte[], forcing you to force_encoding it into something like UTF-8.

Where it doesn’t work

Unfortunately, there are a few sources of data that are common in Rails applications that are not already encoded in UTF-8.

In order to identify these cases, we will need to identify the boundary between a Rails application and the outside world. Let’s look at a common web request.

First, the user goes to a URL. That URL is probably encoded in ASCII, but can also contain Unicode characters. The encoding for this part of the request (the URI) is not provided by the browser, but it appears safe to assume that it’s UTF-8 (which is a superset of ASCII). I have tested in various versions of Firefox, Safari, Chrome, and Internet Explorer and it seems reliable. I personally thank the Encoding gods for that.

Next, the request goes through the Rack stack, and makes its way into the Rails application. If all has gone well, the Rails application will see the parameters and other bits of the URI exposed through the request object encoded as UTF-8. At the moment (and after this post, it will probably be true for mere days), Rack actually returns BINARY Strings for these elements.

At the moment, Ruby allows BINARY Strings that contain only ASCII characters to be concatenated with any ASCII-compatible encoding (such as UTF-8). I believe this is a mistake, because it will make scenarios such as the current state of Rack work in all tested cases, and then mysteriously cause errors when the user enters a UTF-8 character in the URI. I have already reported this issue and it should be fixed in Ruby. Thankfully, this issue only relates to libraries that are mistakenly returning
BINARY data, so we can cut this off at the pass by fixing Rack to return UTF-8 data here.

Next, that data will be used to make a request of the data store. Because we are likely using a UTF-8 encoded data-store, once the Rack issue is resolved, the request will go through without incident. If we were using an ISO-8859-1 data store (possible, but unlikely), this could pose issues. For instance, we could be looking up a story by a non-ASCII identifier that the database would not find because the request String is encoded in UTF-8.

Next, the data store returns the contents. Again, you are likely using using a UTF-8 data store (things like CouchDB and MongoDB return Strings as UTF-8). Your template is likely encoded in UTF-8 (and Rails actually makes the assumption that templates without any encoding specified are UTF-8), so the String from your database should merge with your template without incident.

However, there is another potential problem here. If your data source does not return UTF-8 data, Ruby will refuse to concatenate the Strings, giving you an incompatible encoding error (which will report UTF-8 as incompatible with, for instance, ISO-8859-1). In all of the encoding-related bug reports I’ve seen, I’ve only ever seen reports of BINARY data causing this problem, again, likely because your data source actually is UTF-8.

Next, you send the data back to the browser. Rails defaults to specifying a UTF-8 character set, so the browser should correctly interpret the String, if it got this far. Note that in Ruby 1.8, if you had received data as ISO-8859-1 and stored it in an ISO-8859-1 database, your users would now see “�”, because the browser cannot identify a valid Unicode character for the bytes that came back from the database.

In Ruby 1.9, this scenario (but not the much more common scenario where the database returns content as UTF-8, which is common because Rails specifies a UTF-8 client encoding in the default database.yml), you would receive an error rather than sending corrupted data to the client.

If your page included a form, we now have another potential avenue for problems. This is especially insidious because browsers allow the user to change the “document’s character set”, and users routinely fiddle with that setting to “fix” pages that are actually encoded in ISO-8859-1, but are specifying UTF-8 as the character set.

Unfortunately, while browsers generally use the document’s character set for POSTed form data, this is both not reliable and possible for the user to manually change. To add insult to injury, the browsers with the largest problems in this area do not send a Content-Type header with the correct charset to let the server know the character set of the POSTed data.

Newer standards specify an attribute accept-charset that page authors can add to forms to tell the client what character set to send the POSTed data as, but again, the browsers with the largest issues here are also the ones with issues in implementing accept-charset properly.

The most common scenario where you can see this issue is when the user pastes in content from Microsoft Word, and it makes it into the database and back out again as gibberish.

After a lot of research, I have discovered several hacks that, together, should completely solve this problem. I am still testing the solution, but I believe we should be able to completely solve this problem in Rails. By Rails 3.0 final, Rails application should be able to reliably assume that POSTed form data comes in as UTF-8.

Moving that data to the server presents another potential encoding problem, but again, if we can rely on the database to be using UTF-8 as the client (or internal) encoding, and the solution for POSTed form data pans out, the data should smoothly get into the database as UTF-8.

But what if we still do have non-UTF-8 data

Even with all of this, it is still possible that some non-BINARY data sneaks over the boundary and into our Rails application from a non-UTF-8 source.

For this scenario, Ruby 1.9 provides an option called Encoding.default_internal, which allows the user to specify an preferred encoding for Strings. Ruby itself and Ruby’s standard libraries respect this option, so even if, for instance, it opens some IO encoded in ISO-8859-1, it will give the data to the Ruby program transcoded to the preferred encoding.

Libraries, such as database drivers, should also support this option, which means that even if the database is somehow set up to receive UTF-8 String, the driver should convert those String transparently to the preferred encoding before handing it to the program.

Rails can take advantage of this by setting the default_internal to UTF-8, which will then ensure that String from non-UTF-8 sources still make their way into Rails encoded as UTF-8.

Since I started asking libraries to honor this option a week ago, do_sqlite, do_mysql, do_postgres, Nokogiri, Psych (the new YAML parser in Ruby 1.9), sqlite3, and the MongoDB driver have all added support for this option. The fix should be applied to the MySQL driver shortly, and I am still waiting on a response from the pg driver maintainer.

In short, by the time 1.9.2-final ships, I don’t see any reason why all libraries in use don’t honor this setting.

I’d also add that MongoDB and Nokogiri already return only UTF-8 data, so supporting this option was primarily a matter of correctness. If a driver already deals entirely in UTF-8, it will work transparently with Rails because Rails deals only in UTF-8.

That said, we plan to robustly be able to support scenarios where UTF-8 cannot be used in this way (because encoding are in use that cannot be transparently encoded at the boundary without data loss), so proper support for default_internal will be essential in the long-term.

TL;DR

The vast majority of encoding bugs to date have resulted from outdated drivers that returned BINARY data instead of Strings with proper encoding tags.

The pipeline that brings Strings in and out of Rails is reasonably well-understood, and simply by using UTF-8 libraries for each part of that pipeline, Ruby 1.9 will transparently work.

If you accidentally use non-UTF-8 sources in the pipeline, Ruby 1.9 will throw an error, an improvement over the Ruby 1.8 behavior of simply sending corrupted data to the client.

For this scenario, Ruby 1.9 allows you to specify a preferred encoding, which instructs the non-UTF-8 source to convert Strings in other encodings to UTF-8.

By default, Rails will set this option to UTF-8, which means that you should not see ISO-8859-1 Strings in your Rails application.

By the time Ruby 1.9 is released in a few months, this should be a reality, and your experience dealing with Ruby 1.9 String should be superior to the 1.8 experience, because it should generally work, but libraries will have properly considered encoding issues. This means that serving misencoded data should be basically impossible.

TL;DR the TL;DR

When using Rails 3.0 with Ruby 1.9.2-final, you will generally not have to care about encodings.

Postscript

With all that said, there can be scenarios where you receive BINARY data from a source. This can happen in any language that handles encodings more transparently than Ruby, such as Java and Python.

This is because it is possible for a library to receive BINARY data and not have the necessary metadata to tag it with an encoding.

In this case, you will either need to determine the encoding yourself or treat it as raw BINARY data, and not a String. The reason this scenario is rare is that if there is a way that you can determine the encoding (such as by looking at provided with the bytes), the original library can do the same.

If you get into a scenario where you know the encoding, but it is not machine available, you will want to do something like:

data = API.get("data")
 
data.encoding #=> ASCII-8BIT # alias for BINARY
 
data.force_encoding("SHIFT-JIS").encode!
 
# This first tags the data with the encoding that
# you know it is, and then re-encodes it to
# the default_internal encoding, if one was
# specified

My Common Git Workflow

2010-05-26T17:16:22Z

A recent post that was highly ranked on Hacker News complained about common git workflows causing him serious pain. While I won’t get into the merit of his user experience complaints, I do want to talk about his specific use-case and how I personally work with it in git.

Best I can tell, Mike Taylor (the guy in the post) either tried to figure out a standard git workflow on his own, or he followed poor instructions that tried to bootstrap someone from an svn background while intentionally leaving out important information. In any event, I’ll step through my personal workflow for his scenario, contrasting with subversion as I go.

Cloning the Repository

The very first step when working with a repository is to clone it. In subversion, this is accomplished via svn checkout svn://url/to/repo/trunk. This retrieves the most recent revision of the trunk branch of the repository.

In git, this is accomplished via git clone git://url/to/repo (the http protocol is also possible). This retrieves the entire repository, including other branches and tags.

Making the Change

In both git and subversion, you make the change using a normal text editor.

After Making the Change

In git, you make a local commit, marking the difference between the most recent pulled version (master) and the changes you made. In subversion, the normal workflow does not involve making a change, but apparently some people make manual diffs in order to have a local copy of the changes before updating from the remote. Here’s an example comment from the Hacker News post:

I’ll tell you what happens when I use svn and there’s been an upstream change: I never update my local tree with local modifications. Instead, I extract all my local changes into a diff, then I update my local tree, and then I merge my diff back into the updated tree and commit.

When I need three-way merging, which isn’t often – usually patch can resync simple things like line offsets – it’s handled by a file comparison tool. I have a simple script which handles this

My personal process for making the commit in git almost always involves the gitx GUI, which lets me see the changes for each individual file, select the files (or chunks in the files) to commit, and then commit the whole thing. I sometimes break up the changes into several granular commits, if appropriate.

Updating from the remote

Now that we have our local changes, the next step is to update from the remote. In subversion, you would run svn up. Here, subversion will apply a merge strategy to attempt to merge the remote changes with the local ones that you made. If a merge was unsuccessful, subversion will tell you that a conflict has occurred. If you did not manually save off a diff file, there is no way to get back to the status from before you made the change.

In git, you would run git pull. By default, git applies the “recursive” strategy, which tries to merge your current files with the remote files at the most recent revision. As with subversion, this can result in a conflict. You can also pass the --rebase flag to pull, which is how I usually work. This tells git to stash away your commits, pull the remote changes, and then reapply your changes on top one at a time.

If you use --rebase, you may get a conflict for each of your local commits, which is usually easier to handle than a bunch of conflicts all at once.

I definitely recommend using --rebase which also provides instructions for dealing with conflicts as they arise.

In either case, in my experience, git’s merging capabilities are more advanced than subversion’s. This will result in many fewer cases where conflicts occur.

Resolving Conflicts

From here on, I am assuming you followed my advice and used git pull --rebase.

If a conflict has occurred, you will find that if you run git status, all of the non-conflicting files are already listed as “staged”, while the conflicting files are listed outside the staging area. This means that the non-conflicting files are already considered “added” to the current commit.

To resolve the conflicts, fix up the files listed outside the staging area and git add them. Again, I normally use gitx to move the resolved files into the staging area.

Once you have resolved the conflict, run git rebase --continue. This tells git to use the fixed up changes you just made instead of the original commit it was trying to put on top of the changes you got from the remote.

In subversion, if you got a conflict, subversion will create three files for you: file.mine, file.rOLD, and file.rNEW. You are responsible for fixing up the conflicts and getting back a working file. Once you are done, you run svn resolved.

NOTE: If you had not used git pull --rebase, but instead did raw git pull, you would fix up the files, add the files using git add or gitx, and the run git commit to seal the deal

Yikes! Something went wrong!

In git, if something goes wrong, you just run git reset --hard, which will bring you back to your last local commit.

In subversion, it’s not always possible unless you manually stored off a diff before you started.

Pushing

Now that you’re in sync with the remote server, you push your changes. In git, you run git push. In subversion, you run svn commit.

One Glossed-Over Difference

Subversion allows you to commit changes even if you haven’t svn uped and there have been changes to the remote, as long as there are no conflicts between your local files and the remote files.

Git never allows you to push changes to the remote if there have been remote changes. I personally prefer the git behavior, but I could see why someone might prefer the subversion behavior. However, I glossed over this difference because every subversion reference I’ve found advises running svn up before a commit, and I personally always did that in my years using subversion.

Comparison

Here’s a workflow comparison between git and subversion:

Operation	git	svn
Clone a repository	`git clone git://github.com/rails/rails.git`	`svn checkout http://dev.rubyonrails.org/svn/rails/trunk`
Preparing changes	`git commit` (using gitx)	nothing or create a manual diff
Update from the remote	`git pull --rebase`	`svn up`
Resolving conflicts	`git add` then `git rebase --continue`	`svn resolve`
Resolving conflicts without –rebase	`git add` then `git commit`	N/A
Yikes! Rolling back	git reset –hard	`svn up -rOLD` then apply diff (only if you manually made a diff first)
Pushing	git push	svn commit

Note that I am not attempting to provide an exhaustive guide to git here; there are many more git features that are quite useful. Additionally, I personally do a lot of local branching, and prefer to be able to think about git in terms of cheap branches, but the original poster explicitly said that he’d rather not. As a result, I didn’t address that here.

I also don’t believe that thinking of git in terms of subversion is a good idea. That said, the point of this post (and the point of the original poster) is that there are a set of high-level version control operations that you’d expect git to be able to handle in simple cases without a lot of fuss.

The How and Why of Bundler Groups

2010-05-09T19:09:30Z

Since version 0.9, Bundler has had a feature called “groups”. The purpose of this feature is to allow you to specify groups of dependencies which may be used in certain situations, but not in others.

For instance, you may use ActiveMerchant only in production. In this case, you could say:

group :production do
  gem "activemerchant"
end

Specifying groups allows you to do two things. First, you can install the gems in your Gemfile, minus specific groups. For instance, Rails puts mysql and pg in a database group so that if you’re just working on ActionPack, you can bundle install --without db and run the ActionPack tests without having to worry about getting the gems installed.

Second, you can list specific groups to autorequire using Bundler.require. By default, Bundler.require requires all the gems in the default group (which is all the gems that have no explicit group). You can also say Bundler.require(:default, :another_group) to require specific groups.

Note the difference between these operations: bundle install is opt-out, while Bundler.require is opt-in. This is because the common usage of groups is to specify gems for different environments (such as development, test and production) and you shouldn’t need to specify that you want the “development” and “test” gems just to get up and running. On the other hand, you don’t want your test dependencies loaded in development or production.

It is also worth noting that all gems that you installed (i.e. not the ones that you excluded at install time with --without) will be available to require. This has no effect unless you actually require them. This means that in development mode, if you explicitly require rspec, it will work.

Rails 3 defaults to mapping groups to environment names, and explicitly autorequiring the implicit default group and the group named the same as the current environment. For example, in development mode, Rails will require the default group and the development group. The code that does this is in your application.rb:

Bundler.require(:default, Rails.env) if defined?(Bundler)

Consistency

In order to ensure consistency across all environments, bundler resolves the dependencies of your application using the gems listed in all groups, even if you specify --without. This means that while you can skip installing the gems listed in the production group by saying --without production, bundler will still download and examine the gems in order to properly resolve all dependencies.

As a result, the dependencies you install in development mode and test with will be compatible with the gems in other environments. In essence, this policy ensures that if your tests pass and run in development, your app will not fail to run in production because the dependencies resolved differently.

Multiple Inconsistent Configurations

Sometimes, especially when developing gems for wider use, you want to test your code against multiple incompatible configurations. At first glance, you might think that you could use groups for this case, but as described above, groups are designed for cases where all of the gems are compatible, but you don’t always want to have to install them in all situations.

Instead, use multiple Gemfiles, one for each incompatible configuration. When installing, do bundle install --gemfile Gemfile.rails2. This will tell Bundler to use Gemfile.rails2 rather than the default Gemfile. As in all cases in Bundler, you can also specify this option globally with an environment variable (BUNDLE_GEMFILE).

Ruby 1.9 Encodings: A Primer and the Solution for Rails

2010-05-07T07:15:33Z

UPDATE: The DataObjects drivers, which are used in DataMapper, are now updated to honor default_internal. Let’s keep this moving.

Since Ruby 1.9 announced support for encodings, there has been a flurry of activity to make existing libraries encoding aware, and a tornado of confusion as users of Ruby and Rails have tried to make sense of it.

In this post, I will lay out the most common problems people have had, and what we can do as a community to put these issues to bed in time for Ruby 1.9.2 final.

A Quick Tour

I’m going to simplify some of this, but the broad strokes are essentially correct.

Before we begin, many of you are probably wonder what exactly an “encoding” is. For me, getting a handle on this was an important part of helping me understand the possible solution space.

On disk, all Strings are stored as a sequence of bytes. An encoding simply specifies how to take those bytes and convert them into “codepoints”. In some languages, such as English, a “codepoint” is exactly equivalent to “a character”. In most other languages, there is not a one-to-one correspondence. For example, a German codepoint might specify that the next codepoint should get an ümlaut.

The list of English characters represented by the first seven bits of ASCII (characters 0 through 127 in “ASCII-7″) have the same representation in many (but not all) encodings. This means that if you only use English characters, the on-disk representation of the characters will often be exactly the same regardless of the source encoding.

However, once you start to use other characters, the bytes on disk mean different things in different encodings. Have you ever seen a page on the Internet filled with something like “FÃ¼hrer”? That is the consequence of the bytes of “Führer” stored as UTF-8 being interpreted as Latin-1.

You can trivially see this problem using Ruby 1.9′s encoding support by running this program:

# encoding: UTF-8
 
puts "hello ümlaut".force_encoding("ISO-8859-1").encode("UTF-8")
 
# Output
# hello Ã¼mlat

First, we create a String (“hello ümlaut”) in the UTF-8 encoding. Next, we tell Ruby that the String is actually Latin-1. It’s not, so an attempt to read the characters will interpret the raw bytes of the “ü” as though they were Latin-1 bytes. We ask Ruby to give us that interpretation of the data in UTF-8 via encode and print it out.

We can see that while the bytes for “hello ” and “mlat” were identical in both UTF-8 and Latin-1, the bytes for “ü” in UTF-8 mean “Ã¼” in Latin-1.

Note that while force_encoding simply tags the String with a different encoding, encode converts the bytes of one encoding into the equivalent bytes of the second. As a result, while force_encoding should almost never be used unless you know for sure that the bytes actually represent the characters you want in the target encoding, encode is relatively safe to use to convert a String into the encoding you want.

You’ve probably also seen the reverse problem, where bytes encoded in Latin-1 ended up inside a page encoded in UTF-8.

# encoding: ISO-8859-1
 
puts "hello ümlaut".force_encoding("UTF-8")
 
# Output
# hello ?mlat

Here, the sequence of bytes that represents an “ü” in Latin-1 could not be recognized in UTF-8, so they were replaced with a “?”. Note that puts will always simply write out the bytes to your terminal, and the terminal’s encoding will determine how they are interpreted. The examples in this post are all outputted to a terminal using UTF-8 encoding.

As you can imagine, this presents quite the issue when concatenating two Strings of different encodings. Simply smashing together the raw bytes of the two Strings can result in output that is incomprehensible in either encoding. To make matters worse, it’s not always possible to represent all of the characters in one encoding in another. For instance, the characters of the Emoji encoding cannot be represented in the ISO-8859-1 encoding (or even in a standardized way onto the UTF-8 encoding).

As a result, when you attempt to concatenate two Strings of different encodings in Ruby 1.9, Ruby displays an error.

# encoding: UTF-8
 
puts "hello ümlaut".encode("ISO-8859-1") + "hello ümlaut"
 
# Output
# incompatible character encodings: ISO-8859-1 and UTF-8 (Encoding::CompatibilityError)

Because it’s extremely tricky for Ruby to be sure that it can make a lossless conversion from one encoding to another (Ruby supports almost 100 different encodings), the Ruby core team has decided to raise an exception if two Strings in different encodings are concatenated together.

There is one exception to this rule. If the bytes in one of the two Strings are all under 127 (and therefore valid characters in ASCII-7), and both encodings are compatible with ASCII-7 (meaning that the bytes of ASCII-7 represent exactly the same characters in the other encoding), Ruby will make the conversion without complaining.

# encoding: UTF-8
 
puts "hello umlat".encode("ISO-8859-1") + "hello ümlaut"
 
# Output
# hello umlathello ümlaut

Since Ruby does not allow characters outside of the ASCII-7 range in source files without a declared encoding, this exception eliminates a large number of potential problems that Ruby’s strict concatenation rules might have introduced.

Binary Strings

By default, Strings with no encoding in Ruby are tagged with the ASCII-8BIT encoding, which is an alias for BINARY. Essentially, this is an encoding that simply means “raw bytes here”.

In general, code in Rails applications should not encounter BINARY strings, except for Strings created in source files without encodings. However, since these Strings will virtually always fall under the ASCII-7 exception, Ruby programmers should never have to deal with incompatible encoding exceptions where one of the two encodings is ASCII-8BIT (i.e. BINARY).

That said, almost all of the encoding problems reported by users in the Rails bug tracker involved ASCII-8BIT Strings. How did this happen?

There are two reasons for this.

The first reason is that early on, database drivers generally didn’t properly tag Strings they retrieved from the database with the proper encoding. This involves a manual mapping from the database’s encoding names to Ruby’s encoding names. As a result, it was extremely common from database drivers to return Strings with characters outside of the ASCII-7 range (because the original content was encoded in the database as UTF-8 or ISO-8859-1/Latin-1).

When attempting to concatenate that content onto another UTF-8 string (such as the buffer in an ERB template), Ruby would raise an incompatible encoding exception.

# encoding: UTF-8
 
puts "hello ümlaut" + "hello ümlaut".force_encoding("BINARY")
 
# Output
# incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)

This is essentially identical to the scenario many people encountered. A UTF-8 String was presented to Ruby as a BINARY String, since the database driver didn’t tag it. When attempting to concatenate it onto UTF-8, Ruby had no way to do so reliably, so it raised an exception.

One reason that many people didn’t encounter this problem was that either the contents of the template or the text from the database were entirely in the ASCII-7 subset of their character set. As a result, Ruby would not complain. This is deceptive, because if they made a small change to their template, or if a user happened to enter non-ASCII-7 data (for instance, they got their first user named José), they would suddenly start seeing an incompatible encoding exception.

When people see this incompatible encoding exception, one common reaction is to call force_encoding("UTF-8") on the BINARY data. This will work great for Strings whose bytes actually are encoded in UTF-8. However, if people whose Strings were encoded in ISO-8859-5 (Russian) followed this instruction, they would end up with scrambled output.

Additionally, it’s impossible to simply encode the data, since Ruby doesn’t actually know the source encoding. In essence, a crucial piece of information has been lost at the database driver level.

Unfortunately, this means that well-meaning people who have solved their problem by force_encoding their Strings to UTF-8 (because the bytes actually did represent UTF-8 characters) become baffled when their solution doesn’t work for someone working on a Russian website.

Thankfully, this situation is now mostly solved. There are updates for all database drivers that map the encodings from the database to a Ruby encoding, which means that UTF-8 text from the database will be UTF-8 Strings in Ruby, and Latin-1 text from the database will be ISO-8859-1 Strings in Ruby.

Unfortunately, there is a second large source of BINARY Strings in Ruby. Specifically, data received from the web in the form of URL encoded POST bodies often do not specify the content-type of the content sent from forms.

In many cases, browsers send POST bodies in the encoding of the original document, but not always. In addition, some browsers say that they’re sending content as ISO-8859-1 but actually send it in Windows-1251. There is a long thread on the Rack tracker about this, but the bottom line is that it’s extremely difficult to determine the encoding of a POST body sent from the web.

As a result, Rack handlers send the raw bytes through as BINARY (which is reasonable, since handlers shouldn’t be in the business of trying to wade through this problem) and no middleware exists (yet) to properly tag the String with the correct encoding.

This means that if the stars align, the raw bytes are UTF-8, end up in a UTF-8 database, and end up coming back out again tagged as UTF-8. If the stars do not align, the text might actually be encoded in ISO-8859-1, get put into a UTF-8 database, and come out tagged as UTF-8 (and we know what happens when ISO-8859-1 data is mistakenly tagged as UTF-8).

In this case, because the ISO-8859-1 data is improperly tagged as UTF-8, Ruby happily concatenates it with other UTF-8 Strings, and hilarity ensues.

Because English characters have the same byte representation in all commonly used encodings, this problem is not as common as you might imagine. Unfortunately, this simply means that people who do encounter it are baffled and find it hard to get help. Additionally, this problem doesn’t manifest itself as a hard error. it can go unnoticed and dismissed as a minor annoyance if the number of non-ASCII-7 characters are low.

In order to properly solve this problem for Ruby 1.9, we need a very good heuristic for properly determining the encoding of web-sent POST bodies. There are some promising avenues that will get it right 99.9% of the time, and we need to package them into up a middleware that will tag Strings correctly.

Incompatible Encodings

If you’ve been paying attention, you’ve probably noticed that while the database drivers have solved one problem, they actually introduced another one.

Imagine that you’re using a MySQL database encoded in ISO-8859-1 (or ISO-8859-5, popular for Russian applications, or any other non-UTF-8 encoding). Now that the String coming back from the database is properly tagged as ISO-8859-1, Ruby will refuse to concatenate it onto the ERB buffer (which is encoded in UTF-8). Even if we solved this problem for ERB, it could be trivially reintroduced in other parts of the application through regular concatenation (+, concat, or even String interpolation).

Again, this problem is somewhat mitigated due to the ASCII-7 subset exception, which means that as long as one of the two incompatible Strings uses only English characters, users won’t see any problems. Again, because this “solution” means that the Ruby developer in question still may not understand encodings, this simply defers the problem to some uncertain point in the future when they either add a non-ASCII-7 character to their template or the user submits a non-ASCII-7 String.

The Solution

If you got this far, you’re probably thinking “Holy shit this encoding stuff is crazy. I don’t want to have to know any of this! I just want to write my web app!”

And you’d be correct.

Other languages, such as Java and Python, solve this problem by encodeing every String that enters the language as UTF-8 (or UTF-16). Theoretically, it is possible to represent the characters of every encoding in UTF-8. By doing this, programmers only ever deal with one kind of String, and concatenation happens between UTF-8 Strings.

However, this solution does not work very well for the Japanese community. For a variety of complicated reasons, Japanese encoding, such as SHIFT-JIS, are not considered to losslessly encode into UTF-8. As a result, Ruby has a policy of not attempting to simply encode any inbound String into UTF-8.

This decision is debatable, but the fact is that if Ruby transparently transcoded all content into UTF-8, a large portion of the Ruby community would see invisible lossy changes to their content. That part of the community is willing to put up with incompatible encoding exceptions because properly handling the encodings they regularly deal with is a somewhat manual process.

On the other hand, many Rails applications work mostly with encodings that trivially encode to UTF-8 (such as UTF-8 itself, ASCII, and the ISO-8859-1 family). For this rather large part of the community, having to manually encode Strings to solve incompatible encoding problem feels like a burden that belongs on the machine has been inappropriately shifted onto Rails application developers.

But there is a solution.

By default, Ruby should continue to support Strings of many different encodings, and raise exceptions liberally when a developer attempts to concatenate Strings of different encodings. This would satisfy those with encoding concerns that require manual resolution.

Additionally, you would be able to set a preferred encoding. This would inform drivers at the boundary (such as database drivers) that you would like them to convert any Strings that they tag with an encoding to your preferred encoding immediately. By default, Rails would set this to UTF-8, so Strings that you get back from the database or other external source would always be in UTF-8.

If a String at the boundary could not be converted (for instance, if you set ISO-8859-1 as the preferred encoding, this would happen a lot), you would get an exception as soon as that String entered the system.

In practice, almost all usage of this setting would be to specify UTF-8 as a preferred encoding. From your perspective, if you were dealing in UTF-8, ISO-8859-* and ASCII (most Western developers), you would never have to care about encodings.

Even better, Ruby already has a mechanism that is mostly designed for this purpose. In Ruby 1.9, setting Encoding.default_internal tells Ruby to encode all Strings crossing the barrier via its IO system into that preferred encoding. All we’d need, then, is for maintainers of database drivers to honor this convention as well.

It doesn’t require any changes to Ruby itself, and places the burden squarely on the few people who already need to deal with encodings (because taking data from outside of Ruby, via C, always already requires a tagging step). I have spoken with Aaron Patterson, who has been working on the SQLite3 driver, and he feels that this change is simple enough for maintainers of drivers dealing with external Strings to make it a viable option. He has already patched SQLite3 to make it respect default_internal.

However you feel about Ruby’s solution to expose String encodings directly in the language, you should agree that since we’re stuck with it for the forseeable future, this solution shifts the burden of dealing with it from the unwashed masses (most of whom have no idea what an encoding is) to a few maintainers of C extensions and libraries that deal in binary data. Getting this right as soon as possible will substantially ease the transition from Ruby 1.8 to Ruby 1.9.

Postscript: What Happened in 1.8!?

When people first move to 1.9 and encounter these shenanigans, they often wonder why everything seemed so simple in Ruby 1.8, and yet seemed to work.

There are a few reasons for this.

First, keep in mind that in Ruby 1.8, Strings are simple sequences of bytes. Ruby String operations just concatenate those byte sequences together without any kind of check. This means that concatenating two UTF-8 Strings together will just work, since the combined byte sequence is still valid UTF-8. As long as the client for the Ruby code (such as the browser) is told that the bytes are encoded in UTF-8, all is well. Rails does this by setting the default charset for all documents to UTF-8.

Second, Ruby 1.8 has a “UTF-8″ mode that makes its regular expression engine treat all Strings as UTF-8. In this mode (which is triggered by setting $KCODE = “UTF-8″), the regular expression engine correctly matches a complete UTF-8 character for /./, for instance. Rails sets this global by default, so if you were using Rails, regular expressions respect unicode characters, not raw bytes.

Third, very little non-English content in the wild is actually encoded in ISO-8859-1. If you were expecting to deal with content that was not English, you would probably set your MySQL database to use a UTF-8 encoding. Since Rails sets UTF-8 as the charset of outbound documents, most browsers will in fact return UTF-8 encoded data.

Fourth, the problems caused when an ISO-8859-1 String is accidentally concatenated into a UTF-8 String are not as jarring as the errors produced by Ruby 1.9. Let’s try a little experiment. First, open up a text editor, create a new file, and save it in the ISO-8859-1 encoding.

$KCODE = "UTF-8"
 
iso_8859_1 = "ümlaut"
 
# the byte representation of ümlaut in unicode
utf8 = "\xC3\xBCmlat"
 
puts iso_8859_1
puts utf8
 
puts iso_8859_1 + utf8
puts utf8 + iso_8859_1
 
# Output
# ?mlat
# ümlaut
# ?mlatümlaut
# ümlaut?mlat

If you somehow get ISO-8859-1 encoded content that uses characters outside of the ASCII-7 range, Ruby doesn’t puke. Instead, it simply replaces the unidentified character with a “?”, which can easily go unnoticed in a mostly English site with a few “José”s thrown into the mix. It could also easily be dismissed as a “weird bug that we don’t have time to figure out right now”.

Finally, Rails itself provides a pure-Ruby UTF-8 library that mops up a lot of the remaining issues. Specifically, it provides an alternate String class that can properly handle operations like split, truncate, index, justify and other operations that need to operate on characters, not bytes. It then uses this library internally in helpers like truncate, transparently avoiding a whole other class of issue.

In short, if you’re dealing mostly with English text, and you get unlucky enough the get ISO-8859-1 input from somewhere, the worst case is that you get a “?” instead of a “é”. If you’re dealing with a lot of non-English text, you’re probably being not using ISO-8859-1 sources. In either case, English (ASCII) text is compatible with UTF-8, and Rails provides solid enough pure-Ruby UTF-8 support to get you most of the rest of the way.

That said, anyone dealing with encodings other than UTF-8 and ISO-8859-1 (Japanese and Russian Rubyists) were definitely not in a good place with Ruby 1.8.

Thanks

I want to personally thank Jay Freeman (aka saurik), who in addition to being a general badass, spent about 15 hours with me patiently explaining these issues and working through the Ruby 1.9 source to help fully understand the tradeoffs available.

Katz Got Your Tongue?

Automatic Flushing: The Rails 3.1 Plan

Small Caveat

A Tale of Abort Traps (or Always Question Your Assumptions)

It Begins

Debugging

Digging Further

A Working Hypothesis

Install bundler on their system

Switch to rvm, and create a new gemset

Run bundle install, resulting in an error

Run gem install bundler, to install bundler

Run bundle install, which works, but has a subtle bug

Postscript

Using >= Considered Harmful (or, What’s Wrong With >=)

But What Happens When a New Version is Released?

Applications and Gemfiles

Threads (in Ruby): Enough Already

Threads and IO Blocking

“Threads are just HARD”

Historical Ruby Issues

JRuby, Ruby 1.9 and Rubinius, and the Future

Everyone Who Tried to Convince Me to use Vim was Wrong

What’s New in Bundler 1.0.0.rc.1

Starting a new project with bundler

Checking out an existing project using bundler

Updating dependencies

By default, gems are installed to your system

Deployment

Encodings, Unabridged

What’s an Encoding?

What’s Unicode

A Digression: UTF-*

What Ruby 1.9 Does

The problems people are really having

Why this is, in practice, a rare problem

Digression: force_encoding

Where it doesn’t work

But what if we still do have non-UTF-8 data

TL;DR

TL;DR the TL;DR

Postscript

My Common Git Workflow

Cloning the Repository

Making the Change

After Making the Change

Updating from the remote

Resolving Conflicts

Yikes! Something went wrong!

Pushing

One Glossed-Over Difference

Comparison

The How and Why of Bundler Groups

Consistency

Multiple Inconsistent Configurations

Ruby 1.9 Encodings: A Primer and the Solution for Rails

A Quick Tour

Binary Strings

Incompatible Encodings

The Solution

Postscript: What Happened in 1.8!?

Thanks

Run `bundle install`, resulting in an error

Run `gem install bundler`, to install bundler

Run `bundle install`, which works, but has a subtle bug