Jim Neath - Ruby and Rails and Facebook Developer

Ruby SSL Certificate Verify Failed

Wed Oct 19 13:37:32 +0100 2011

If you’re using Ruby to open connections to an external server over https, eg. the Facebook Graph API, you may run into the following error:

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

This error is due to Ruby not being able to find the certification authority certificates (CA Certs) used to verify the authenticity of secured web servers. The solution is to download the this ca-bundle.crt into your application’s lib/ directory:

Then add the following code to config/initializers/fix_ssl.rb:

require 'open-uri'
require 'net/https'

module Net
  class HTTP
    alias_method :original_use_ssl=, :use_ssl=
    
    def use_ssl=(flag)
      self.ca_file = Rails.root.join('lib/ca-bundle.crt')
      self.verify_mode = OpenSSL::SSL::VERIFY_PEER
      self.original_use_ssl = flag
    end
  end
end

This should force ruby to use the CA bundle from your application’s lib/ directory.

Party on, Wayne!

Serving Your Facebook App's Maintenance Page for POST Requests

Tue Apr 05 13:37:32 +0100 2011

The standard way of deploying rails applications is to serve a static maintenance page while the code is being updated, to prevent errors being thrown by the user and to prevent writes happening to your database. This is all fine and dandy in the real world but in Facebook land we run into issues.

Facebook will always send a POST on the initial iframe request for security reasons, and this can be to any end-point in our application, while in reality, this should be treated as a GET.

So, the first time a page of your app is requested by user, Facebook sends a POST request. If you’re currently in maintenance mode you’re going to get a big 405 Method not allowed error.

To fix this we need to tell nginx to serve your maintenance page to POST requests, as well as GET requests. We can do this by updating your servers config files. Say your server config initially looks like this:

server {
  # all your other server stuff
  # ...

  # show maintenance page if exists 
  if (-f $document_root/system/maintenance.html) {
    rewrite ^(.*)$ /system/maintenance.html break;
    break;
  }
}

Add the following location rule to allow nginx to serve the page for POST requests:

server {
  # all your other server stuff
  # ...

  # show maintenance page if exists 
  if (-f $document_root/system/maintenance.html) {
    rewrite ^(.*)$ /system/maintenance.html break;
    break;
  }

  # serve maintenance page to POST requests
  location = /system/maintenance.html {
    post_to_static on;
  }
}

Reload your nginx config and everything should be rocking as expected.

Rails 3, nginx and send_file

Tue Apr 05 13:37:32 +0100 2011

Trying to use send_file from your Rails 3 app and getting a file with a size of zero bytes? It’s more than likely that you’re using nginx and haven’t setup your app to use send_file properly.

Open up your config/environments/production.rb file and find the following lines:

# Specifies the header that your server uses for sending files
config.action_dispatch.x_sendfile_header = "X-Sendfile"

# For nginx:
# config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect'

Then comment out the first config.action_dispatch.x_sendfile_header line and uncomment out the second. So it should look like this:

# Specifies the header that your server uses for sending files
# config.action_dispatch.x_sendfile_header = "X-Sendfile"

# For nginx:
config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect'

Then just commit, push, deploy and voila!

Using Redis with Ruby on Rails

Thu Mar 24 15:37:32 +0000 2011

Buy The Redis Book!

Written by Redis creator, Salvatore Sanfilippo, and key contributor, Pieter Noordhuis, the Redis Book will show you how to work with different data structures, how to handle memory, replication, and the cache itself, and how to implement messaging, amongst other things! Buy the book

Redis is an extremely fast, atomic key-value store. It allows the storage of strings, sets, sorted sets, lists and hashes. Redis keeps all the data in RAM, much like Memcached but unlike Memcached, Redis periodically writes to disk, giving it persistence.

Redis is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

You can run atomic operations on these types, like appending to a string; incrementing the value in a hash; pushing to a list; computing set intersection, union and difference; or getting the member with highest ranking in a sorted set.

In order to achieve its outstanding performance, Redis works with an in-memory dataset. Depending on your use case, you can persist it either by dumping the dataset to disk every once in a while, or by appending each command to a log.

The above quote was taken from the official Introduction to Redis page.

Redis Data Types
Installing Redis
Starting Redis Server
Redis and Rails
Example Uses in Rails
Using Redis As Your Rails Cache Store
Monitoring Redis
Redis and Unix Sockets
Redis Related Gems

Redis Data Types

Below is a general overview of the data types available to you in Redis:

Data Type	Description
String	A string of characters. The most basic data type. Can also be used as a counter, using `incr`, `incrby`, `decr` and `decrby`
Set	Sets are unsorted collections of strings with no duplicate members.
Sorted Set	Sorted sets (`zset`) are like sets in that each member has to be unqiue. Unlike sets, sorted sets have a score associated with each member
List	Lists are sorted collections of strings, essentially like an `Array` in ruby. You can `push` and `pop` on either end, making them useful for queues.
Hash	Hashes are what you’d expect: keys and string values.

As well as those basic types, there is also the ability to do PubSub with Redis. A full list of Redis commands is available on the official site.

Installing Redis

Installation is simple:

curl -O http://redis.googlecode.com/files/redis-2.2.2.tar.gz
tar xzf redis-2.2.2.tar.gz
cd redis-2.2.2
make
cp src/redis-server src/redis-cli /usr/bin

Starting Redis Server

If you’re using redis-server locally, you can get away with just running the following, which will use the default Redis config file:

redis-server

If you’re running it on a server you’ll probably want to use your own config file:

redis-server /path/to/redis.conf

The default redis.conf looks like this.

Redis and Rails

Now that we have Redis installed and running, we need to get it playing nice with Rails. Enter redis-rb, created by Ezra Zygmuntowicz:

https://github.com/ezmobius/redis-rb

First step, add redis-rb to your Gemfile:

gem 'redis', '2.1.1'

Then install the gem via Bundler:

bundle install

Lastly, create an initializer in config/initializers/redis.rb and add the following:

$redis = Redis.new(:host => 'localhost', :port => 6379)

This will create a new instance of the Redis client, connected to localhost:6379 (the default), and store it in the global variable $redis.

Let’s check that everything is working by firing up rails console:

> $redis
=> #<Redis client v2.1.1 connected to redis://localhost:6379/0 (Redis v2.2.2)> 
> $redis.set('chunky', 'bacon')
=> "OK" 
> $redis.get('chunky')
=> "bacon"

Example Uses in Rails

User Friendships

The first example I’ll show you is modelling friendships using ActiveRecord and Redis. We’ll be using sets to store the friendship data. Here is our User model:

class User < ActiveRecord::Base
  # follow a user
  def follow!(user)
    $redis.multi do
      $redis.sadd(self.redis_key(:following), user.id)
      $redis.sadd(user.redis_key(:followers), self.id)
    end
  end
  
  # unfollow a user
  def unfollow!(user)
    $redis.multi do
      $redis.srem(self.redis_key(:following), user.id)
      $redis.srem(user.redis_key(:followers), self.id)
    end
  end
  
  # users that self follows
  def followers
    user_ids = $redis.smembers(self.redis_key(:followers))
    User.where(:id => user_ids)
  end

  # users that follow self
  def following
    user_ids = $redis.smembers(self.redis_key(:following))
    User.where(:id => user_ids)
  end

  # users who follow and are being followed by self
  def friends
    user_ids = $redis.sinter(self.redis_key(:following), self.redis_key(:followers))
    User.where(:id => user_ids)
  end

  # does the user follow self
  def followed_by?(user)
    $redis.sismember(self.redis_key(:followers), user.id)
  end
  
  # does self follow user
  def following?(user)
    $redis.sismember(self.redis_key(:following), user.id)
  end

  # number of followers
  def followers_count
    $redis.scard(self.redis_key(:followers))
  end

  # number of users being followed
  def following_count
    $redis.scard(self.redis_key(:following))
  end
  
  # helper method to generate redis keys
  def redis_key(str)
    "user:#{self.id}:#{str}"
  end
end

Some sample useage:

> %w[Alfred Bob].each{|name| User.create(:name => name)}
=> ['Alfred', 'Bob']
> a, b = User.all
=> [#<User id: 1, name: "Alfred">, #<User id: 2, name: "Bob">] 
> a.follow!(b)
=> [1, 1] 
> a.following?(b)
=> true 
> b.followed_by?(a)
=> true 
> a.following
=> [#<User id: 2, name: "Bob">] 
> b.followers
=> [#<User id: 1, name: "Alfred">]
> a.friends
=> [] 
> b.follow!(a)
=> [1, 1] 
> a.friends
=> [#<User id: 2, name: "Bob">] 
> b.friends
=> [#<User id: 1, name: "Alfred">]

It is all pretty simple. We’re using the following set commands: sadd, srem, smembers, sinter, scard, sismember and the multi command. An overview of the commands is provided below:

Command	Overview
sadd	Add member to the set stored at key. If member is already a member of this set, no operation is performed. If key does not exist, a new set is created with member as its sole member.
srem	Remove member from the set stored at key. If member is not a member of this set, no operation is performed.
smembers	Returns all the members of the set value stored at key.
sinter	Returns the members of the set resulting from the intersection of all the given sets.
scard	Returns the set cardinality (number of elements) of the set stored at key.
sismember	Returns if member is a member of the set stored at key.
multi	Marks the start of a transaction block. Subsequent commands will be queued for atomic execution using EXEC.

High Score Table

We can construct a simple high score table using Redis sorted sets:

class User < ActiveRecord::Base
  # log high score
  def scored(score)
    if score > self.high_score
      $redis.zadd("highscores", score, self.id)
    end
  end
  
  # table rank
  def rank
    $redis.zrevrank("highscores", self.id) + 1
  end
  
  # high score
  def high_score
    $redis.zscore("highscores", self.id).to_i
  end
  
  # load top 3 users
  def self.top_3
    $redis.zrevrange("highscores", 0, 2).map{|id| User.find(id)}
  end
end

Sample useage:

> a, b, c, d = User.limit(4)
=> [#<User id: 1, name: "Alfred">, #<User id: 2, name: "Bob">, #<User id: 3, name: "Charlie">, #<User id: 4, name: "Derek"">] 
> a.scored 100
=> true 
> b.scored 500
=> true 
> c.scored 25
=> true 
> d.scored 10000
 => true 
> d.high_score
 => 10000 
> d.rank
=> 1 
> c.rank
=> 4 
> c.scored 5000000
=> false 
> c.high_score
=> 5000000 
> c.rank
=> 1 
> User.top_3
=> [#<User id: 3, name: "Charlie">, #<User id: 4, name: "Derek">, #<User id: 2, name: "Bob">]

We’re using the following set commands: zadd, zrevrank, zrevrange and the zscore commands. An overview of the commands is provided below:

Command	Overview
zadd	Adds the member with the specified score to the sorted set stored at key. If member is already a member of the sorted set, the score is updated and the element reinserted at the right position to ensure the correct ordering. If key does not exist, a new sorted set with the specified member as sole member is created.
zrevrank	Returns the rank of member in the sorted set stored at key, with the scores ordered from high to low. The rank (or index) is 0-based, which means that the member with the highest score has rank 0.
zrevrange	Returns the specified range of elements in the sorted set stored at key. The elements are considered to be ordered from the highest to the lowest score. Descending lexicographical order is used for elements with equal score.
zscore	Returns the score of member in the sorted set at key.

More Redis Use Cases

Using Redis As Your Rails Cache Store

If you want to use Redis as your cache store, then look no further than Redis Store by Luca Guidi. Redis Store provides namespaced Rack::Session, Rack::Cache, I18n and cache Redis stores for Ruby web frameworks. It allows you to use Redis as your Rails cache store, session store and integrates well with Rack::Cache.

https://github.com/jodosha/redis-store

First, let’s add redis-store to the Gemfile:

gem 'redis-store', '1.0.0.beta4'

Then install the gem via Bundler:

bundle install

Then update your config/environments/production.rb:

config.cache_store = :redis_store

Then you’re good to go. Your cache store should now be using Redis.

Monitoring Redis

If you’re using Redis on your server and you have daemonize set to yes in your redis.conf, you need to monitor it. Below are an example Monit and God script:

Monit

check process redis
  start program = "/usr/bin/redis-server /etc/redis/redis.conf"
  stop program = "/usr/bin/redis-cli -p 6379 shutdown"
  with pidfile /var/run/redis.pid
  group redis

God

Adapted from this Gist:

%w{6379}.each do |port|
  God.watch do |w| 
    w.name = "redis" 
    w.interval = 30.seconds 
  
    w.start = "/usr/bin/redis-server /etc/redis/redis.conf" 
    w.stop = "/usr/bin/redis-cli -p 6379 shutdown" 
    w.restart = "#{w.stop} && #{w.start}" 
  
    w.start_grace = 10.seconds 
    w.restart_grace = 10.seconds 
  
    w.start_if do |start| 
      start.condition(:process_running) do |c| 
        c.interval = 5.seconds 
        c.running = false 
      end 
    end 
  end
end

Redis and Unix Sockets

Redis 2.2 introduced the ability to connect to Redis via unix sockets. To allow this functionality you need to uncomment the following line in your redis.conf:

unixsocket /tmp/redis.sock

Then you can connect to Redis in your app, change the contents of your config/initializers/redis.rb, to this:

$redis = Redis.new(:path => "/tmp/redis.sock")

Below are some rough benchmarks from my MacBook Pro:

~ redis-benchmark -q
PING (inline): 54347.82 requests per second
PING: 52083.33 requests per second
MSET (10 keys): 28571.43 requests per second
SET: 48309.18 requests per second
GET: 49504.95 requests per second
INCR: 47846.89 requests per second
LPUSH: 48309.18 requests per second
LPOP: 49751.24 requests per second
SADD: 48543.69 requests per second
SPOP: 51282.05 requests per second
LPUSH (again, in order to bench LRANGE): 48076.92 requests per second
LRANGE (first 100 elements): 34129.69 requests per second
LRANGE (first 300 elements): 17064.85 requests per second
LRANGE (first 450 elements): 13661.20 requests per second
LRANGE (first 600 elements): 11074.20 requests per second

~ redis-benchmark -q -s /tmp/redis.sock
PING (inline): 80000.00 requests per second
PING: 82644.62 requests per second
MSET (10 keys): 41841.00 requests per second
SET: 72463.77 requests per second
GET: 81300.81 requests per second
INCR: 63694.27 requests per second
LPUSH: 70422.53 requests per second
LPOP: 80000.00 requests per second
SADD: 72463.77 requests per second
SPOP: 77519.38 requests per second
LPUSH (again, in order to bench LRANGE): 80645.16 requests per second
LRANGE (first 100 elements): 42372.88 requests per second
LRANGE (first 300 elements): 22779.04 requests per second
LRANGE (first 450 elements): 16891.89 requests per second
LRANGE (first 600 elements): 13458.95 requests per second

Redis Related Gems

Redis Objects

Redis objects maps redis data types to ruby objects. Created as a compliment for current ORMs instead of a replacement.

https://github.com/nateware/redis-objects

Resque

Resque (pronounced like “rescue”) is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later. It was written by GitHub and is used in production there, as well as on many other apps. Read their blog post: Introducing Resque

https://github.com/defunkt/resque

Vanity

Vanity is an Experiment Driven Development framework for Rails that uses Redis. It allows you to define A/B tests in your Ruby on Rails application and integrates with Google Analytics via Garb.

https://github.com/assaf/vanity

Ohm

Ohm is a library for storing objects in Redis, a persistent key-value database. It includes an extensible list of validations and has very good performance.

https://github.com/soveran/ohm

Rollout

Rollout lets you roll out new features to select groups of users for testing. Rollout has been covered on the changelog. Also works in tandem with Degrade

https://github.com/jamesgolick/rollout

Soulmate

Soulmate is a tool to help solve the common problem of developing a fast autocomplete feature. It uses Redis’s sorted sets to build an index of partially completed words and the corresponding top matching items.

https://github.com/seatgeek/soulmate

Buy The Redis Book!

Showing Passenger Stats From Inside Your App

Wed Mar 23 12:37:32 +0000 2011

If you’ve ever wanted to know how many active or idle Passenger processes your server has and display them inside your app, you can use the following helper:

def passenger_stats
  @passenger_stats ||= begin
    `passenger-status`.split("\n")[1..4].inject({}) do |memo, line| 
      memo.merge(line[/^\w+/].to_sym => line[/\d+$/].to_i)
    end
  end
end

Calling passenger_stats will return a hash resembling the following:

{ :inactive => 1, :count => 4, :max => 15, :active => 3 }

Which you could then display in your views, thus:

<h3>Passenger Stats</h3>
<p><strong>Total:</strong> <%= passenger_stats[:total] %></p>
<p><strong>Maximum:</strong> <%= passenger_stats[:max] %></p>
<p><strong>Active:</strong> <%= passenger_stats[:active] %></p>
<p><strong>Inactive:</strong> <%= passenger_stats[:inactive] %></p>

Enjoy.

Fixing NGINX 201 Content-Length Header Bug

Tue Mar 22 14:40:26 +0000 2011

NGINX is an awesome server, but unfortunately contains an error in earlier versions (pre 0.8.32), that sends the incorrect headers for 201 (created) responses. It does not set the Content-Length header, which causes modern browsers to keep the connection open and wait until the timeout value is exceeded.

There are two possible fixes for this; Upgrade NGINX to version 0.8.32 or greater, or fix the issue in your server side code. I’ll show you how we can do the latter, using Rack.

Rack provides a minimal, modular and adaptable interface for developing web applications in Ruby. By wrapping HTTP requests and responses in the simplest way possible, it unifies and distills the API for web servers, web frameworks, and software in between (the so-called middleware) into a single method call.

We will use a piece of Rack middleware to set the correct headers for any 201 responses.

Copy the code below into lib/content_length_fix.rb:

module Nginx
  class ContentLengthFix
    def initialize(app)
      @app = app       
    end                

    def call(env)
      status, headers, response = @app.call(env)
      headers["Content-Length"] = response.length.to_s if status == 201
      [status, headers, response]
    end                
  end
end

Then add the following to your config.ru file:

require 'content_length_fix' 
use Nginx::ContentLengthFix

Rejoice and your 201 responses now have the correct headers set!

Fixing Long Key Lengths in MySQL

Thu Jun 17 13:37:32 +0100 2010

While using Query Reviewer, I occasionally come across a warning along the lines of the following:

2 identical queries Table videos: Long key length (768)

With the following explanation:

The key used for the index was rather long, potentially affecting indices in memory

To fix this we need to limit the length of our index. We can do this by creating a migration with :length added to the add_index method:

add_index :videos, :cached_slug, :length => 20

If you’re indexing more than one column, you can use the following:

add_index :stories, [:title, :author_name], :length => { :title => 10, :author_name => 5 }

Hopefully this post will save someone a few minutes.

5 Rails Plugins to Help Optimize Your MySQL

Tue Jun 15 15:03:59 +0100 2010

Optimizing your MySQL queries and performance in your rails applicationcan be a real pain. The plugins below help to make things a little easier.

Bullet

Bullet is a rails plugin, written by Richard Huang (@flyerhzm), that helps to kill N+1 queries. It shows you where you should be using :include in your ActiveRecord calls. Bullet also informs you where you’re missing counter caches as well as warning you of any unused eager loading.

http://github.com/flyerhzm/bullet/

SlimScrooge

SlimScrooge is a plugin by Stephen Sykes (@sdsykes), that restricts the columns retrieved in your MySQL queries by learning which attributes are subsequently called on the ActiveRecord model.

Below is an example, taken from the readme:

# 1st request, sql is unchanged but columns accesses are recorded
Brochure Load SlimScrooged 1st time (27.1ms)   SELECT * FROM `brochures` WHERE (expires_at IS NULL)

# 2nd request, only fetch columns that were used the first time
Brochure Load SlimScrooged (4.5ms)   SELECT `brochures`.expires_at,`brochures`.operator_id,`brochures`.id FROM 
`brochures` WHERE (expires_at IS NULL)

# 2nd request, later in code we need another column which causes a reload of all remaining columns
Brochure Reload SlimScrooged (0.6ms) `brochures`.name,`brochures`.comment,`brochures`.image_height,`brochures`.id,
`brochures`.tel,`brochures`.long_comment,`brochures`.image_name,`brochures`.image_width FROM 
`brochures` WHERE `brochures`.id IN ('5646','5476','4562','3456','4567','7355')

# 3rd request
Brochure Load SlimScrooged (4.5ms)   SELECT `brochures`.expires_at,`brochures`.operator_id,`brochures`.name,
`brochures`.id FROM `brochures` WHERE (expires_at IS NULL)

http://github.com/sdsykes/slim_scrooge/

Query Reviewer

Query Reviewer by David Stevenson (@dsboulder) of Pivotal Labs, is a Rails plugin that adds a div to the top left of the screen that contains lots of useful details regarding the MySQL executed on the current page.

The plugin makes it easy to view the EXPLAIN output for each SELECT query on the page. It also provides ratings and warnings on each query executed on a page.

http://github.com/dsboulder/query_reviewer

Rails Indexes

Rails Indexes is a plugin by Elad Meidar (@eladmeidar), that helps to generate a migration containing suggested indexes for your database.

By running the following rake command:

rake db:index_migration

Rails Indexes will generate the example migration containing suggested indexes. While these indexes are a good start, you will probably find that you need some detailed database indexes.

http://github.com/eladmeidar/rails_indexes/

Ambitious Query Indexer

Ambitious Query Indexer is an index generating plugin by Sam Phillips (@samsworldofno). Like Rails Indexes, it generates a migration containing suggested indexes, but it works by analysing all of the queries it can find in your app.

http://github.com/samdanavia/ambitious_query_indexer

Calculating an Images Orientation using Paperclip

Thu Jun 10 12:52:52 +0100 2010

If you ever need to know if an image is portrait or landscape in orientation, you can do the following (assuming you have an orientation attribute on your model:

class Image < ActiveRecord::Base
  # Paperclip
  has_attached_file :file
  
  # Callbacks
  before_create :set_orientation
  
  protected
  
  def set_orientation
    original_file = self.file.to_file(:original)
    dimensions = Paperclip::Geometry.from_file(original_file)
    self.orientation = (dimensions.width > dimensions.height) ? 'landscape' : 'portrait'
  end
end

How To Speed Up Textmate in Large Projects

Thu Mar 25 11:05:03 +0000 2010

You know those times when Textmate beachballs for about 5 seconds when it regains focus, it’s generally caused by the fact you’ve got about 16 million folders in your public/system folder used by paperclip.

Here’s a quick tip to speed things up by ignoring those folders:

Open Textmate Preferences and goto the Advanced section. Then click on Folder References.

And enter the following in Folder Pattern:

!.*/(\.[^/]*|**log|system**|CVS|_darcs|_MTN|\{arch\}|blib|.*~\.nib|.*\.(framework|app|pbproj|pbxproj|xcode(proj)?|bundle))$

Rejoice in the fact that Textmate doesn’t make you want to cry.