Elevated Code RSS Feed

Speeding up delayed job

Wed, 04 Nov 2009 06:44:28 GMT

Delayed Job is a fantastic plugin for moving work out of the request flow. Unfortunately, it has a few performance problems when running with large numbers of jobs on MySQL. With minor changes, we can remove a large amount of the overhead.

I noticed the performance problem on one of our applications when MySQL was taking large amounts of CPU. The application had been in production for about a year and had been performing well. I had just recently moved image resizing out of the main workflow and into delayed job. When I went to look at the slow query log, I saw this being run frequently:


# Query_time: 0  Lock_time: 0  Rows_sent: 5  Rows_examined: 67604
SELECT * FROM `delayed_jobs` WHERE ((run_at <= '2009-11-04 06:58:49' AND (locked_at IS NULL OR locked_at < '2009-11-04 02:58:49') OR (locked_by = 'host:example.com pid:17731')) AND failed_at IS NULL)  ORDER BY priority DESC, run_at ASC LIMIT 5;

When reading the output of the slow query log, I look mainly at two things. The first is the query time. In this case, the query was being run in less than a second. The second thing I look at is the ratio of the number of rows examined to the number of rows sent back to the app. In this case, the database had to look at 67,604 rows just to give the app 5 rows back. When you see full table scans like this, it is usually a sign of a missing index. I looked at the delayed jobs table and saw that there was only an index on ID. That means the DB was reading every row in the table for each query.

Once I realized that I was seeing a full table scan, I knew I needed to add an index. Normally, when you add indexes to a table, you're doing it to speed up the row selection performed by the where clause. That isn't a good idea here. Delayed job removes successfully completed records by default, and we have very few updates that fail. This means that the vast majority of rows in the table will match the where clause. When this happens, you typically want to add an index to speed up the sort instead of the where clause.

This query sorts on two fields, priority and run_at. We can easily add an index there, but notice the ordering on the sort. It sorts by priority in descending order. Because of the way database indexes work, they are typically only used for ascending order sorts. If you were to create an index on delayed_jobs(priority,run_at) the database won't be able to use the index. (You can try it out. Add the index and explain the query.

Some databases, like Oracle, allow you to create an index with a specified ordering. For example, you can create an index on delayed_jobs(priority desc, run_at asc) which will then allow the query to use the index. While that syntax is supported by MySQL, the ordering is ignored. That means MySQL still won't be able to use the index.

So how did I solve the problem? Simple. I changed the semantics of priority. By default, Delayed Job expects a higher value of priority to be more important. I decided to assign lower numbers to mean more important (similar to Unix nice values.) Once I changed the way I assigned priorities, I simply changed the query to order by priority ASC, run_at ASC and added an index on delayed_jobs(priority,run_at). Now, when the query is run, the log looks like:


# Query_time: 0  Lock_time: 0  Rows_sent: 5  Rows_examined: 5
SELECT * FROM `delayed_jobs` WHERE ((run_at <= '2009-11-04 06:58:49' AND (locked_at IS NULL OR locked_at < '2009-11-04 02:58:49') OR (locked_by = 'host:example.com pid:17731')) AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 5;

That quickly reduced the load on the system and improved application performance.

Slides from Inman Connect 2009

Tue, 11 Aug 2009 06:42:45 GMT

As promised, I wanted to provide my slides from Inman Connect 2009

My favorite links for learning about Agile methodologies are:

There are many more, but those are some of my favorites.

Standing up to be counted

Thu, 30 Apr 2009 09:58:57 GMT

Nick Sieger asked that people stand and be counted and I too will join in. I’m disappointed with many in our community. While I think the presentation that sparked this discussion was in poor taste, it’s not the presentation that bothers me. I’m disappointed by how little empathy some people are showing.

Whether you agree with somebody or not, I think it’s important to try to see things from their perspective. It may not change your mind, but it will hopefully at least give you a better understanding of how your actions affect others.

Speeding up location based searches

Fri, 06 Mar 2009 06:35:51 GMT

With the popularity of Google Maps, more and more web applications are becoming location aware. Rails already has some great tools for working with geodata, such as GeoKit. We've been happily using these tools for years. While GeoKit has some basic optimizations, we recently found a case where lookups were taking close to 8 seconds. We needed a way to speed this up. Read on to learn how we did it.

First, let me explain our domain. We are working on a site that shows the 25 nearest amenities to a location. The application is focused on San Francisco and the surrounding areas. We store the amenities in a table with columns for latitude and longitude. All told, we search about 800,000 amenity records.

To find the nearest 25 amenities, our database has to do a lot of work. For each listing in the database, we need to calculate the distance from our origin. Performing that calculation would be unbearably slow, so geokit speeds up the query by including a square bounds. First, the list of locations is limited by a square 5 miles on a side (we use a big window to make sure we get amenities in more rural areas.) Once we've limited our dataset, the results are sorted by distance. Unfortunately, due to the amenity density in San Francisco, we still have to go through 10,000 records on average.

This calculation and sort is just slow. In version 5.1, MySQL includes some geospatial extensions. These extensions include datatypes for locations and spatial indexes to make spatial searches much faster. Unfortunately, ActiveRecord doesn't support these extensions. That means we'll have to do some things manually.

First, let's look at the three primitives we'll use. The first is a @Point@. Points represent a location in a 2D space. We'll use Points to represent the location of an amenity by it's latitude and longitude coordinates. In MySQL, you can create a point using the SQL: @GeomFromText('Point(38.567 -122.576)')@

The second primitive is a @Polygon@. Polygons include a list of coordinates that make up the vertices. Polygons must be closed. For example, we could create the polygon @GeomFromText('Polygon((0 0,0 1,1 1,1 0,0 0))')@. We'll use a Polygon to represent the boundary inside which we want to search.

The final primitive is a @Line@. We use a Line to calculate the distance between two points. According to the OpenGIS specification, we should be able to use the distance function for this. Unfortunately, MySQL doesn't implement Distance.

We'll also need to create a spatial index. A spatial index is an r-tree index that is optimized for fast distance calculations. Currently, only MyISAM tables support spatial indexes. They also require that the column be defined as NOT NULL. We'll have to keep that in mind as we go along.

Let's start by adding a Point column to our amenities table. Because ActiveRecord doesn't support the GIS extensions, we'll need to use the execute method inside our migration. This looks like:

    execute 'alter table amenities add lat_lng_point point not null'

Because we want to use Spatial indexes, we'll also need to convert our amenities table to MyISAM.

    execute 'alter table amenities engine myisam'

With that in place, we'll need to convert our existing data to include the @lat_lng_point@ column. From what I can tell, Points can only be created using the GeomFromText method. That means we'll need to generate the text representation of a Point during our conversion. This can be done with the following SQL"

update amenities set lat_lng_point = GeomFromText(concat('Point(',lat,' ',lng,')'))

At this point, we now have a table with the location data being stored as a Point. Since all of our records have a Point, we now meet the criteria for adding our spatial index:

    execute 'create spatial index amenities_spatial_idx on amenities(lat_lng_point)'

With that in place, we can need a method that does a few thing. First, we need to compute a bounds to limit our distance calculation. We'll do that using @GeoKit::Bounds@. Once we have that, we need to write a query that uses the @MBRWithin@ function (MBR stands for Minimum Bounding Rectangle) to find the list of points inside our Bounds. Once we have that, we can use a Line to get the distance between our origin and each point and order by that. The end result is a little ugly:

  
  def self.near(mappable)
    bound = GeoKit::Bounds.from_point_and_radius(mappable,5)
    sql = <<SQL
    select *,  GLength(LineStringFromWKB(LineString(AsBinary(lat_lng_point), 
    (AsBinary(GeomFromText('Point(#{mappable.lat} #{mappable.lng})')))))) * 69 as distance
    from amenities where MBRWithin(lat_lng_point,
     GeomFromText('Polygon((#{bound.ne.lat} #{bound.ne.lng}, #{bound.ne.lat} #{bound.sw.lng},#{bound.sw.lat} #{bound.sw.lng},#{bound.sw.lat} #{bound.ne.lng},#{bound.ne.lat} #{bound.ne.lng}))')) 
    order by distance asc limit 25;
SQL
    find_by_sql(sql)
  end

It's ugly, but it is blazingly fast! Our query went from 8s to 0.01s. That's definitely worth a little code ugliness.

While we've handled the inital data conversion and querying, we can't add data to our table. We need to provide a Point for each record, and there's no way to do that using ActiveRecord. Instead, we'll need to use a MySQL trigger.

    execute <<EOT
      create trigger set_lat_lng_point 
      before insert on amenities for each row 
      set  new.lat_lng_point = GeomFromText(concat('Point(',new.lat,' ',new.lng,')'));
EOT

That code will create a Point from the lat and lng when a record is inserted. If you need to be able to update locations, you could make the trigger fire on update as well. You'll want to install the Trigger Happy plugin so that your triggers are dumped and included in your test database.

After all that, can you believe we're still not quite done? Due to a bug in MySQL, inserting a row where @lat_lng_point@ is NULL will give an error. It turns out that MySQL checks the not null columns before triggers are run. Luckily, there is a workaround. If you don't explicitly set the column value to NULL, the trigger will be run before the value is checked. To get ActiveRecord to not explicitly set the column to NULL, we'll use a bit of a hack.

  # Remove lat_lng_point from the list of attributes
  # otherwise, AR writes it as NULL which keeps the trigger from firing
  # and setting the value correctly  
  def before_save
    @attributes.delete("lat_lng_point")
  end

That code will remove the column for AR's list of attributes.

Obviously, this is somewhat complicated. If your searches are running okay using just normal SQL, I wouldn't recommend changing. If you need high performance and are searching large amounts of data, this little bit of code can make a huge difference.

# Our entire migration
class ConvertAmenitiesToUseSpatials < ActiveRecord::Migration
  def self.up
    execute 'alter table amenities add lat_lng_point point not null'
    execute 'alter table amenities engine myisam'
    execute 'update amenities set lat_lng_point = GeomFromText(concat(\'Point(\',lat,\' \',lng,\')\'))'
    execute 'create spatial index amenities_spatial_idx on amenities(lat_lng_point)'
    execute "create trigger set_lat_lng_point before insert on amenities for each row set  new.lat_lng_point = GeomFromText(concat('Point(',new.lat,' ',new.lng,')'));"
  end

  def self.down
    execute 'alter table amenities drop lat_lng_point'
  end
end

Speeding up SOLR indexing

Wed, 14 Jan 2009 12:01:01 GMT

We're finishing up a project that uses SOLR and acts_as_solr extensively. Our dataset isn't large, but it's non-trivial as well. We're managing a search index with a little over 100,000 documents in it. Each document has indexes on 12 fields. In general, our search performance is outstanding. We get results almost instantly. Our indexing performance was another story. I'm not a big fan of premature optimization, but when a full reindex took 4 days, I decided something needed to change

There's some good advice on the web for pulling indexing out of a request flow. Unfortunately for us, this wasn't going to help. Our updates would take several days to process. This delay makes testing changes to search incredibly painful.

After much googling, I found a number of people that recommended allowing SOLR to manage its own commits. It took me a few tries to get this working. In the end, I was making it much harder than it had to be. Disabling autcommit really only takes two steps.

First, update your call to acts_as_solr to disable autcommit:

class MyIndexedModel < ActiveRecord::Base
  acts_as_solr :fields=>[:name,:body], :auto_commit=>false
end

With just that change our index performance went from 2 seconds per record to 50 records per second. A 100x speed up. Unfortunately, our changes weren't showing up in the index. SOLR doesn't add indexed records to the database until a commit is done. SOLR does provide an easy way to tell it to manage the commits itself. To enable this, edit vendor/plugins/acts_as_solr/solr/solr/conf/solrconfig.xml. There is commented out configuration for autoCommit. Remove the XML comments around this area. We use the maxTime parameter to have SOLR update the index every minute. Our configuration looks like:

...
  <updateHandler class="solr.DirectUpdateHandler2">

    <!-- A prefix of "solr." for class names is an alias that
         causes solr to search appropriate packages, including
         org.apache.solr.(search|update|request|core|analysis)
     -->

    <autoCommit>
      <maxDocs>10000</maxDocs>
      <maxTime>60000</maxTime>
    </autoCommit>
...

That's all there is to it! We get 100x faster indexing and our updates still show up within a minute. Suddenly I don't dread making changes to our indexed models!

Announcing Facebooker support for Facebook Connect

Fri, 02 Jan 2009 09:51:20 GMT

Over the last few months I've received quite a few requests for Facebook Connect support in Facebooker. Thanks to Kevin Lochner, I'm proud to announce Facebook Connect support in Facebooker. Read on to learn how to create a simple Facebook Connect application.

The setup of a Facebook Connect application is very similar to that of a Facebook Platform Application. You'll need to create an application using the Facebook Developer tool and then build a Rails application and install the Facebooker plugin. If you need help with this, check out the screencast or My Book. You'll need to set up your facebooker.yml and configure a callback URL on Facebook.

Once you've got a basic application set up and running, we'll need to generate an xd_receiver file. The xd_receiver file allows Facebook to perform cross domain javascript requests. You can generate this file by running:

  script/generate xd_receiver

A file named xd_receiver.html will be generated and placed in your public directory. Make sure this file is added to your source code control system. It must exist for Facebook Connect to work.

With that setup done, it's time to try out Facebook Connect. Before we can do anything else, we'll need to ask our user to login. Let's create a controller and an action for testing. I called my controller "connect" and just used the index action because I'm lazy. Inside the view for action, we'll need to include a javascript file and do a little setup. Add this to your view:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://www.facebook.com/2008/fbml">
<head>
  <%= javascript_include_tag :defaults%>
</head>
<body>
<h1>My Great Application</h1>
<%= fb_connect_javascript_tag %>
<%= init_fb_connect "XFBML"%>
<%= fb_login_button%>

</body>
</html>

Facebook connect requires an XHTML doctype declaration along with a namespace on your HTML element. It also works most reliably if your HTML is valid. If you're getting strange results, it's worth validating your HTML. The three calls to helpers are the actual meat of the view. The first call includes the Facebook Connect javascript. The second line runs a little setup and the final line renders an <fb:login-button> tag.

If all goes well, you should see something that looks like:

If you don't see that, you'll need to verify that things are configured just right. There's a lot of magic going on and it tends to be brittle. Make sure that your callback URL is correctly set and that you are accessing your site using the callback URL. Make sure that your public/xd_receiver.html file exists. Finally, make sure that you have correctly set up your facebooker.yml file.

Take a look at the source of your page. Doesn't that <fb:login-button> tag look strange? To render FBML inside your page, Facebook is using into the HTML equivalents. The login button is just one of many tags we can use. For example, we can include a user's profile photo by calling fb_profile_pic. Add a call to show your own profile. Once you've done that, you should see something like: Notice that Facebook isn't showing your picture. That's because of privacy restrictions. Before we can see information about a Facebook user, we'll need to login. Click the "connect" button and you should see a popup prompting you to log in. After logging in, your profile picture will now be shown.

Getting access to FBML outside of the Facebook canvas is cool, but it's just a start to what we can do. When we login using Facebook Connect, Facebook sets several cookies that our application can use to access a Facebook session. Facebooker knows how to use these cookies to set up a facebook session. Add the following code to your controller:

  before_filter :set_facebook_session
  helper_method :facebook_session

That code sets up the users Facebook session and makes it available to the view. With it, you have access to the normal REST API as you can see in the following code.

<% if facebook_session %>
  <h2>You are logged in as <%= facebook_session.user.name %></h2>
<% else %>
  <h2>You are not logged in!</h2>
<% end %>

This means you can ask for data about a user like their name and their friends. You can also post stories to their feed! You have most of the power of a Facebook Platform application inside your existing code!

This support is still very new, so make sure you join the Facebooker mailing list on Rubyforge if you decide to use it. Let us know what you build with it!

Thanks again to Kevin Lochner who provided the patch to support authentication using cookies. Thanks also to Nate Kontny of Inkling Markets who provided a great overview of how Facebook Connect works! Happy New Year everyone!

Daemonizing Rails

Tue, 02 Dec 2008 12:02:54 GMT

No, this isn't a post about us converting from Linux to BSD. Instead, I'm talk about starting long running processes that use the Rails environment. If you've tried to do this, you've probably seen some obscure errors like:

closed stream 
activesupport-2.1.0/lib/active_support/buffered_logger.rb:105:in `write'
activesupport-2.1.0/lib/active_support/buffered_logger.rb:105:in `flush'
activesupport-2.1.0/lib/active_support/buffered_logger.rb:118:in `auto_flush'
activesupport-2.1.0/lib/active_support/buffered_logger.rb:70:in `add'
activesupport-2.1.0/lib/active_support/buffered_logger.rb:76: ...

Inside, I'll show you how to avoid this

Daemons in UNIX are a wonderful, magical thing. Even though it seems like starting up a background job should be simple, there are a lot of steps involved. You can see them in detail in Tammer Saleh's Angels and Daemons talk. The really interesting issue for us is that processes close all open IO streams when they daemonize. That may not cause a problem for a normal Ruby Application, but it violates some assumptions that Rails makes.

When Rails boots, it opens a few different file descriptors. First, it opens a log file. This is your production.log or similar. It also makes a connection to the database. Rails gets unhappy when these files are closed out from under it. You could write a lot of code to handle these conditions or you can take the simple way out: Delay booting Rails until after you daemonize. Here's a simple script that does that:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
ROOT = File.expand_path(File.dirname(__FILE__)+'/../')
Daemons.run_proc('watcher', :dir_mode => :normal, :dir=>"#{ROOT}/log/",:log_output =>true, :backtrace=>true) do
  require "#{ROOT}/config/environment"
  # Do work here
end

This code is simple, but there are a couple of subtleties. The first is that we need to use File.expand_path to turn the Rails Root path into an absolute path. When an application daemonizes, it changes its working directory to "/". If we use a relative path for our RAILS_ROOT it will no longer point to our Rails installation.

Once we have the absolute RAILS_ROOT path, we can daemonize and then boot Rails. Rails will now start as usual and all files will be opened. That's all there is to it. Now you can easily start long running processes using Rails. You can even use a cap task to easily restart them on deploy:

task :restart_watcher do
  run "#{current_path}/script/watcher restart"
end

It feels good to see your name in print

Wed, 22 Oct 2008 08:11:51 GMT

It was almost a full year ago that I started writing a book on Facebook development. This past Friday I got a very exciting box. Unboxing pictures inside.

A nice note on top of the box.

Carefully wrapped

There it is.

I really underestimated the amount of work it would take to write a book. I had read Dave Thomas’ So You Want to Write a Book so I should have realized what an undertaking it was. I had just completed my first marathon at that point and figured it couldn’t be harder than running 26.2 miles. Boy was I ever wrong. I’m proud of the book. I know it is something I would have loved to have when starting to develop Facebook applications.

Now I’m going to get back to training for the Philly Marathon on 11/23. Training seems much easier now.

Preloading summary data using ActiveRecord

Wed, 08 Oct 2008 07:55:21 GMT

It seems like every application we build has at least one screen that needs to access a lot of data. For example, if we were building a web application for a school we would likely have a page for teachers that shows all of their students including their current grage, number of assignments turned in on time and attendance. Inside, we'll look at how we can make this type of page run quickly without adding much complexity.

Let's set up a couple of example models to demonstrate this case. We'll have a student model. Our student will have a list of absences, a list of assignments and a list of grades. Our model could look something like:

class Student < ActiveRecord::Base
  has_many :assignments
  has_many :grades, :through => :assignments
  has_many :absences
end

In our output, we'll want to show a tabular view of information for each student. A first cut would probably look something like:

  <% for student in @teacher.students %>
    <tr>
      <td><%= student.name %> </td>
      <td><%= student.absences.count %> </td>
      <td><%= student.assignments.select {|a| a.on_time?}.size / student.assignments.size.to_f rescue nil %> </td>
      <td><%= student.grades.sum(:score)/student.grades.count %> </td>
   </tr>
  <% end %>

That code works, but there are a couple of obvious problems with it. First, we have a lot of logic in our views. Second, we'll run 4 queries for each user: 1 to count absences, 1 to get the number of on_time assignements, 1 to get the total score and one to count the number of grades. This is okay for viewing 10 students, but performance will become a serious problem when looking at a couple of hundred students.

So how do we clean this up? Let's start by cleaning up the logic in the views. Hopefully that will make a solution more apparent. The first step is to create methods on student for each of those items. Instead of calling student.absences.count in the view, we can add a student.times_absent method. We can do the same for the percent of on time assignments and for the average score. Our code then becomes:

  <% for student in @teacher.students %>
    <tr>
      <td><%= student.name %> </td>
      <td><%= student.times_absent %> </td>
      <td><%= student.percent_assignments_on_time %> </td>
      <td><%= student.average_score %> </td>
   </tr>
  <% end %>

Our implementation of those methods will look familiar:

class Student < ActiveRecord::Base
  has_many :assignments
  has_many :grades, :through => :assignments
  has_many :absences
  
  def times_absent
    absences.count
  end
  
  def percent_assignments_on_time
    student.assignments.select {|a| a.on_time?}.size / student.assignments.size.to_f rescue nil
  end

  def average_score
    student.grades.sum(:score)/student.grades.count
  end 

end

That cleaned up our code, but it didn't do anything for our performance. We could try to use :include to speed up our query. If we had 200 students each with 200 assignments, this would involve us running a find that returns 200*200 rows. Another solution to our performance problem is to use a raw sql query. We can write a sql query that will give us the percent_assignments_on_time and the average score for our whole list of users at once. I normally avoid doing this because then I can't lose the abstraction that active record provides. What if we could keep the same interface but use a sql query to preload the data? That would be the best of both worlds.

We can do this using memoization. To implement this, we'll start by adding memoization to our methods:

  def average_score
    @average_score ||= student.grades.sum(:score)/student.grades.count
  end

Next, we'll create a method that will retrieve and set the average score for each user that we care about. It will look something like:

  attr_accessor :average_score, :percent_assignments_on_time
  def self.load_assignment_analytics!(students)
    sql =<<-SQL
      select u.id, 
             avg(score) as cached_average_score,
             sum(on_time)/count(*) as cached_on_time_percent
      from students u
       join assignments a on u.id = a.user_id
       join grades g on g.assignment_id = a.id
      where u.id in (?)
      group by u.id
    SQL
    assignment_info = Student.find_by_sql([sql,false,users.map(&:id)])
    user_map = assignment_info.index_by(&:id)
    students.each do |student|
      rec = user_map[student.id] 
      if rec.nil?
        student.percent_assignments_on_time = "None"
        student.average_score = "None"
      else
        student.percent_assignments_on_time = rec. cached_on_time_percent.to_f
        student.average_score = rec. cached_average_score.to_f
      end
    end
    
  end

That methods runs our query for all users that we pass in. For each user, it finds the result from our query. If no result exists (because they don't have any assignments or grades) it will set the values to "None". (If we set them to nil or false our query would be re-run in the interface because of the ||= line. If there are results for a user, we set the value on the object.

Now, when our view iterates over the list of users, it will use the previously stored data. The interface remains the same but our performance improves just by adding a single call to Student.load_assignment_analytics!(teacher.students) in our controller. If we forget to call this method, the view will still work, it will just run more queries. This is definitely less DRY than our original code, as we implement the logic for our counts in two ways. We could redefine our average score method to always call load_assignment_info!, but that might be overkill. There are also a couple of gotchas. Because we use Student.find_by_sql to fetch our data, we need to make sure that we name our calculated constants something that isn't a method name on student. If you don't, the method will be called.

All in all, I think this is a relatively elegant and simple solution to a complex problem.

Developing Facebook Platform Applications with Rails is done!

Thu, 25 Sep 2008 07:04:45 GMT

I've been busy recently, as you may have guessed from the lack of posts. It's time to share the items I've been working on!

First, I’m thrilled to announce that Developing Facebook Platform Applications with Rails is being printed. It should be shipping withing the next couple of weeks. If you haven’t ordered your copy, pick one up now. All told, it took about 10 months to get the book out. That’s definitely longer than I hoped, but there is a good reason. Facebook released a massive overhaul of their platform about 3 months ago. The shipping version of the book has been completely updated to reflect this new API. It took a lot of work, but I’m happy with how it turned out. You can check out the application we build on Facebook. You can also see the code on github. I’d love for some intrepid readers to fork the code and send me patches to make the application prettier!

In other Facebook news, I released a plugin called facebooker_authentication that encapsulates a lot of the code that we use on Facebook applications. It covers user creation, user authentication and some helpers for handling users. I’ve used something similar on every application. In fact, I use this plugin on two forthcoming screencasts for the Pragmatic Programmers. Expect another announcement here when that happens.

Finally, I’m still getting settled into Philadelphia. It’s a great city with some fun people. I’ve spent a couple of days at Indy Hall and really dig it. I’ve also been volunteering with a group called Back On My Feet that supports the homeless population in Chicago. I’m running the Philly marathon on their behalf. Check out my story and make a donation!