asemanfar

Pandemic -- because information needs to be contagious

Fri, 10 Apr 2009 23:03:50 -0700

Pandemic is a map-reduce framework. You give it the map, process, and reduce methods and it handles the rest. It's designed to serve requests in real-time, but can also be used for offline tasks.

It's different from the typical map-reduce framework in that it doesn't have a master-worker structure. Every node can map, process, and reduce. It also doesn't have the concept of jobs, everything is a request.

The framework is designed to be as flexible as possible, there is no rigid request format, or API, you can specify it however you want. You can send it http-style headers and a body, you can send it JSON, or you can even just send it a single line and have it do whatever you want. The only requirement is that you write your handler to appropriately act on the request and return the response.

Here is how you use it:

Server require 'rubygems' require 'pandemic'



class Handler < Pandemic::ServerSide::Handler
  def process(body)

body.reverse


  end
end

pandemic_server = epidemic! pandemic_server.handler = Handler.new pandemic_server.start.join

In this example, the handler doesn't define the map or reduce methods, and the defaults are used. The default for each is as follows:

map: Send the full request body to every connected node
process: Return the body (do nothing)
reduce: Concatenate all the responses

Client require 'rubygems' require 'pandemic'



class TextFlipper
  include Pandemize
  def flip(str)

pandemic.request(str)

end end

Config

Both the server and client have config files:



pandemic_server.yml

servers: - host1:4000 - host2:4000 response_timeout: 0.5 Each value for the server list is the host:port that a node can bind to. The servers value can be a hash or an array of hashes, but I'll get to that later. The response timeout is how long to wait for responses from nodes before returning to the client.



pandemic_client.yml

servers: - host1:4000 - host2:4000 max_connections_per_server: 10 min_connections_per_server: 1 response_timeout: 1
The min/max connections refers to how many connections to each node. If you're using the client in Rails, then just use 1 for both min/max since it's single threaded.

More Config

There are three ways to start a server:

ruby server.rb -i 0
ruby server.rb -i machine1hostname
ruby server.rb -a localhost:4000

The first refers to the index in the servers array: servers: - host1:4000 # started with ruby server.rb -i 0 - host2:4000 # started with ruby server.rb -i 1
The second refers to the index in the servers hash. This can be particularly useful if you use the hostname as the key. servers: machine1: host1:4000 # started with ruby server.rb -i machine1 machine2: host2:4000 # started with ruby server.rb -i machine2
The third is to specify the host and port explicitly. Ensure that the host and port you specify is actually in the config otherwise the other nodes won't be able to communicate with it.

You can also set node-specific configuration options. servers: - host1:4000:



  database: pandemic_node_1
  host: localhost
  username: foobar
  password: f00bar

host2:4000: database: pandemic_node_2 host: localhost username: fizzbuzz password: f1zzbuzz And you can access these additional options using config.get(keys) in your handler: class Handler < Pandemic::ServerSide::Handler def initialize @dbh = Mysql.real_connect(*config.get('host', 'username', 'password', 'database')) end end

Code: github repository

Install:
sudo gem -a http://gems.github.com
sudo gem install arya-pandemic

Request Queue via Mongrel Proctitle

Wed, 11 Feb 2009 23:04:57 -0800

In our cluster, we run many mongrels on several app servers all behind a load balancer. In order to get an idea of how each app server and its mongrels are doing, we use rtomayko's mongrel_proctitle gem. This gem sets up a mongrel handler that extracts request information from the client's request and sets it as the process title so when you do a ps aux or run top you can see each mongrel's queue length and request info:

mongrel_rails [8000/1/4]: handling 127.0.0.1: GET /users



that's [port/queue length/requests handled]: handling client ip: request

Problem

Last week, we were having some misbehaving mongrels that would lock up and essentially stop serving requests. We'd look at the process titles and see something like this:

mongrel rails [8021/14/6123]: handling 127.0.0.1: GET /status

We have an action designated to be a lightweight health check to let the load balancer know what's up and according to this process title, this action was locking up the mongrel and growing to a queue length of around 14. This didn't make sense, why is the lightest weight action locking up the mongrel?

Solution

With a little digging, I found that the mongrel_proctitle gem is broken. More precisely, it is only accurate when there is a single request being handled and none queued. The handler was setting the process title as soon as it received the request, which then led that client's thread to the next handler, which happens to be the synchronized Rails handler. So the request the process title shows is the most recent received request and not the current request being handled.

With a few small modifications, we now have a more accurate process title that also shows the rest of the queue (up to a character limit) as a comma-separated list:

mongrel rails [8021/2/6123]: handling 127.0.0.1: GET /users, 127.0.0.1: GET /status

The updated version is available on github and also as a gem at gems.github.com as arya-mongrel_proctitle:

gem sources -a http://gems.github.com sudo gem install arya-mongrel_proctitle

Note, as described in the comments in the code, it's still not exact. There is at least one race case where the entire list won't be ordered accurately, but it's unlikely to occur and still gives a good idea of what's going on.

I Went Unobtrusive

Sat, 08 Nov 2008 10:44:24 -0800

So I've been cleaning up my blog's code base and decided to switch to unobtrusive javascript. I've heard many reasons advocating its use but for some reason, they never stuck. Until a couple of weeks ago when I worked on a Railsrumble project with a couple of buddies of mine, one of which was a big advocate of unobtrusive javascript using LowPro. Actually using unobtrusive javascript turned out to be much more gratifying that I anticipated. It satisfied some inner OCD tendencies of mine that a lot of programmers share and it made developing javascript'd UIs much easier to develop and much much easier to maintain.

Looking back, I don't think know why I chose to use the Rails javascript helpers or RJS; I like writing javascript, especially when there are great frameworks out there like LowPro and Prototype. The fact that the Rails javascript helpers generate inline javascript, which sometimes turns out being longer than a simple one-liner, is kind of gross. As for RJS, the fact that it's generates code that contains both the data and the logic makes OCD baby Jesus cry.

The separation of HTML (markup), CSS (presentation), Javascript (interactivity) is almost as awesome as the separation of church and state...oh wait, crap.

If you haven't already switched, do it now!

Factory Girl with Cross-Validating Attributes

Thu, 06 Nov 2008 19:15:47 -0800

Just as a little background, I'm working on adding the concept of rounds to an application. A "round" is simply an organization unit with a start and end date. Each round has_many :events, and each event has an end date, defined as its moment of expiration. The validation logic for these two models is that an event's end date must be between the round's start/end dates and that rounds cannot overlap. This isn't an issue at all with things like fixtures since that's all manually specified. But I'm using FactoryGirl for generating my test data so things get a little tricky.

The problem I ran into is that an event requires a round with valid start/end dates, but the round factory generates its own sequential non-overlapping rounds. There are a few requirements I'm looking for:

Independently generate rounds that do not overlap.
Generate an event given a round, and pick a valid event end date.
Generate an event given an end date, and generate a valid round.

This is what I came up with to do this:



Round Factory

Factory.sequence :round_start_date do |n|
  ((n - 1) * 14).days.from_now
end

Factory.define :round do |round|
  round.starts_at { Factory.next(:round_start_date) }
  round.ends_at { |r| r.starts_at + 14.days }
end

Event Factory

Factory.define :event do |event|
  event.round do |e|

if e.event_ends_at
  e.association :round, 
        :starts_at => e.event_ends_at - 10.days, 
        :ends_at => e.event_ends_at + 1.day
else
  e.association :round
end

end event.event_ends_at { |e| e.round.ends_at - 2.days } end

So the round factory is pretty simple. The round_start_date sequence just insures that rounds generated using that factory do not overlap. This meets the 1st requirement.

To satisfy the 2nd and 3rd requirement, in the event factory, I first check to see if an end date for the event was specified in the factory build options, in which case I pass valid attributes to the round factory. Otherwise, I delegate the start/end date selection to the round factory. I then specify the event end date based on the round end date, relying on the fact that the event end date won't be set if it was specified using the build options hash. Phew! That's a mouth full.

Although this is quite ugly, it seems to work well with one caveat: I'm not guaranteed to have non-overlapping rounds created by the event factory when specified an event end date:

Factory(:event, :event_ends_at => 5.days.ago) Factory(:event, :event_ends_at => 5.days.ago)

This will throw an exception because the second factory call will try to build a round with the same start/end date as the round created by the first. I couldn't come up with an elegant solution for that, the only thing I can think of is querying the database before creating a round and using a pre-existing event with proper date boundaries, but I don't want to do that; this works for me for now since my tests happen not to hit that caveat.

Does anyone have any similar experience or maybe a better solution?