Production Hacks

Active Users with Redis

2011-03-22T00:00:00Z

Want to log and display the active users in your web application? Enjoy using Redis? Read on.

Scenario

A user with ID 123 hits our web application. The unix timestamp (seconds since epoch) for the current date and time is: 1300803419.

Logging

We insert the user’s ID as a member of the ‘active-users’ Sorted Set, using the timestamp as the score.

ZADD ‘active-users’ 1300803419 123

If 123 was already a member of the Sorted Set, Redis would only update the score to the new timestamp, effectively setting the user’s last action timestamp.

Display

If you want to display the active users in the last 15 minutes, you will first need to generate 2 timestamps.

timestamp1 = 15 minutes ago

timestamp2 = current date and time

Then return a range of members in the ‘active-users’ Sorted Set using these timestamps as the min and max scores.

user_ids = ZRANGEBYSCORE ‘active-users’ timestamp1 timestamp2

The returned array will contain the user ID’s of all users active in the last 15 minutes.

That’s it.

What I like about this approach is that it’s simple and lightweight; only 1 entry in the Sorted Set per user. You can also query “active users in the last X minutes/days/months” up to the date when you began logging the requests.

ActionMailer MissingTemplate error in Rails 2.3

2010-07-20T00:00:00Z

In Rails 2.3, if you’re using implicit multipart emails for your ActionMailer views (the content type is part of the file name), ActionMailer will have problems locating the correct template during tests and background jobs. I ran into this problem when setting up background emailing with resque_mailer.

ActionView::MissingTemplate: Missing template my_mailer/mailer_action.erb in view path app/views

This is a known and documented bug, but the patch went in after the 2.3.5 release. Adding the following code to a Rails initializer should fix the problem:

Redis API access logger

2010-07-10T00:00:00Z

We just launched a REST API for our production Rails app and we wanted to keep track of the API usage by user. Inserting a row into MySQL or incrementing an existing counter on every request had inevitable scaling issues. We needed something lightweight and fast, Redis was a perfect candidate for the job.

Redis has to keep all stored objects in memory, so just putting all data in there and forgetting about it was out of the question. We decided to only keep a few days of data in Redis and archive the results to MySQL. Daily API usage stats would be served directly by Redis, archived results on date ranges would be fetched from MySQL.

Assuming a User with ID 1337 hits the UsersController #update API call on July 10, 2010 – the process would go like this:

Increment a Sorted Set of daily “Controller#Action” values by 1 (the hit count):

ZINCRBY 'api:requests:2010-07-10' 1 'users#update'

Increment a Sorted Set of daily “Controller#Action” values by User ID:

ZINCRBY 'api:requests:2010-06-30:users#update' 1 1337

Add the current date to a Set of dates for which we’re currently tracking data:

SADD api:requests:dates '2010-07-10'

We are now able to pull a list of Controller#Action values by date, sorted by the hit count (ZSCORE).

controller_actions = ZREVRANGE 'api:requests:2010-07-10' 0, -1

hit_count = ZSCORE 'api:requests:2010-07-10' 'users#update'

Then using the Controller#Action value, generate a key to pull a list of User IDs that have accessed that action, also sorted by their hit count (ZSCORE).

user_ids = ZREVRANGE 'api:requests:2010-07-10:users#update' 0, -1

hit_count = ZSCORE 'api:requests:2010-07-10:users#update' 1337

Finally we archive our daily results to a MySQL table with the following schema:

date | user_id | controller | action | hit_count

On a daily cron, we iterate through the Set of archived dates (skipping the current date) and generate the keys needed to pull the API usage stats from Redis. Once we’ve inserted the daily results into MySQL, we can safely delete the daily Sorted Sets and remove the date from the Set:

dates = SMEMBERS 'api:requests:dates'

… repeat the steps above to pull the usage stats by date, controller#action and user ID …

DEL 'api:requests:2010-07-10

DEL 'api:requests:2010-07-10:users#update'

SREM 'api:requests:dates' '2010-07-10'

Passenger 2.2.14 with bundler08 support

2010-06-11T00:00:00Z

Passenger 2.2.14 now comes with built in Bundler support for Rails 3, but some Rails 2 apps might have explicitly added Bundler support in the “config/preinitializer.rb” file.

To stop Passenger from setting up Bundler (and crashing in the process), just create the “config/setup_load_paths.rb” file and copy your “config/preinitializer.rb” content in there. Passenger will now detect the file and require it instead of running its Bundler setup.

Make sure to keep both files around too make the app backwards compatible with earlier versions of Passenger (for EY-Cloud support).

Domain redirect using rack-rewrite

2010-06-05T00:00:00Z

Stick this snippet in at the top of the “config.ru” file of your Rack app (Rails, Sinatra) to always redirect visitors to the ‘www’ version of your domain name (rack-rewrite gem required):

Engine Yard Cloud, New Relic and Hoptoad deploy notifications

2010-05-12T00:00:00Z

Engine Yard’s Cloud platform uses the chef-deploy gem instead of the more commonly used Capistrano to deploy your Rails app. Both New Relic and Hoptoad have a built in Capistrano task to notify the services on deploy. Using a deploy hook we can achieve the same result with EY-Cloud.

Create the following file in “RAILS_ROOT/deploy/after_restart.rb”

The above deploy hook will only fire off when you’re deploying to a production environment. If you’re using bundler08 instead of Engine Yard’s gem manager, then replace the “newrelic_cmd” with “bin/newrelic_cmd”.

toto and New Relic RPM

2010-04-20T00:00:00Z

This blog is powered by toto, a minimalistic Ruby blogging engine. It’s currently hosted on the Heroku free plan, which is more than enough for now, but I wanted to know when would be a good time crank up the dynos. Heroku allows you to setup the free version New Relic RPM as an addon, but you will have to follow a couple of extra steps to get toto’s requests to show up in New Relic.

First make sure you have added the New Relic RPM addon for Heroku. If you’re not using Heroku or you want you use your own New Relic account, add your newrelic.yml config to the APP_ROOT/config folder.

Checkout the newrelic_rpm gem in the vendor/plugins folder.

Finally, add the following code to your config.ru file (before the toto config block):

The last part of the snippet monkey-patches toto’s Site#call method to report back to New Relic on each request.

MySQL NOT Comparisons and NULL values

2010-04-17T00:00:00Z

If you’re using any of the MySQL NOT logical operators (NOT IN (?), <>, !=) on a column, you should be aware that all records that have a NULL column value will always be excluded by MySQL.

What if you do want to include the NULL value records? You could try something like this:

SELECT * FROM books WHERE author_id IS NULL OR author_id != 123

Memcached and Passenger connection sharing

2010-04-15T00:00:00Z

One of the few gotchas for Passenger’s smart/smart-lv2 spawning methods is the unintentional sharing of file descriptors between workers. This is actually documented at the very bottom of the Passenger docs, but you don’t really understand the impact of it until it starts wreaking havoc on a highly cached production application.

We started by getting random Memcache IO timeout errors, which we initially thought were caused by the strain on the server and Memcached running out of memory. Then the Rails session manager would receive random cached ActiveRecord objects instead of the session hash (we store our sessions in Memcached).

This last error lead me into the right direction and I found similar issues posted on the web. The problems were caused by having all Passenger workers share the same memcache-client connection. They had to be reset when spawned to avoid clashing of cached objects as they get fetched. Putting the following code in your environment.rb file should take care of this:

Rails 2.3, cache_fu and memcached session_store

2010-04-14T00:00:00Z

Many Rails developers mistakenly assume that by installing defunkt’s cache_fu, Rails will be configured to use the settings found in the memcached.yml config YAML installed by the plugin. This was somehow true in earlier versions of Rails 2.*, but a couple of extra steps are required for 2.3.

Tim Connor’s branch has a commit that correctly sets the session_store config in Rails 2.3. I applied Tim’s patch to the most recent version of cache_fu:

http://github.com/jfernandez/cache_fu

If you’re also using Rails abstract cache_store (Rails.cache) and you want it to use the same settings as cache_fu, you will have to use the MemcachedConfig class. Just drop the class file in your RAILS_ROOT/lib folder and add the following code to your environment.rb file (or production.rb)

Now you have Rails 2.3 running with all Memcached stores configured under the same settings and namespace.