Paul Dix Explains Nothing

Typhoeus, the best Ruby HTTP client just got better

2009-10-29T12:29:29-04:00

I've been quietly working on Typhoeus for the last few months. With the help of Wilson Bilkovich and David Balatero I've finished what I think is a significant improvement to the library. The new interface removes all the magic and opts instead for clarity.

It's really slick and includes improved stubing support, caching, memoization, and (of course) parallelism. The Typhoeus readme highlights all of the awesomeness. It should be noted that the old interface of including Typhoeus into classes and defining remote methods has been deprecated. I'll be removing that sometime in the next six months.

In addition to thanking everyone using the library and everyone contributing, I should also thank my employer kgbweb. If you're a solid Rubyist that likes parsing, crawling, and stuff, or a machine learning guy, or a Solr/Lucene/indexing bad ass, let me know. We need you and we're doing some crazy awesome stuff.

Bypassing wxWidgets error when building Erlang from Source on OS X

2009-10-26T11:23:13-04:00

I had the following error when trying to build Erlang from source on OS X:


make[3]: *** [i386-apple-darwin9.8.0/wxe_events.o] Error 1
make[2]: *** [opt] Error 2
make[1]: *** [opt] Error 2
make: *** [libs] Error 2

After a quick search I uncovered this post about building without wxWidgets support (which is fine by me for running RabbitMQ Server.) So putting it all together, building and installing Erlang looks like this:


# in the erlang source directory
./configure
touch lib/wx/SKIP
make
sudo make install

Detect and Parse Atom and RSS Feed URLs from HTML with TruffleHog

2009-10-23T11:55:31-04:00

TruffleHog is a small library I wrote for detecting and parsing Atom and RSS feeds in a web page. I'm sure there are other libraries out there, but I couldn't find one easily. I found Cory Forsyth's FeedDetector, but there were some cases that it didn't handle.

Anyway, here is everything you need to get running.

gem install truffle-hog --source http://gemcutter.org

# and then in teh rubys
require 'rubygems'
require 'truffle-hog'

# get atom and rss
feed_urls = TruffleHog.parse_feed_urls(some_html)

# get atom if available, otherwise rss
feed_urls = TruffleHog.parse_feed_urls(some_html, :atom)

# get rss if available, otherwise atom
feed_urls = TruffleHog.parse_feed_urls(some_html, :rss)

Behind the scenes it's grabbing the urls through a regex. Really just a few lines of code. I may update it with some fancy Nokogiri parsing and some other stuff, but for now it's just a simple way to parse out feed urls.

Using the Nginx Memcached module with Passenger

2009-10-08T11:28:26-04:00

Nginx, everyone's favorite speedy web server has a module to hook in directly to memcached. For those of us running Ruby servers behind nginx we can avoid hitting our running Ruby processes completely on a cache hit. I'll avoid the details and point you instead to Ilya Grigorik's coverage of the Nginx + Memcached goodness.

My trouble was that I couldn't find a good example of using Passenger and the Nginx memcached module together. The module documentation shows how to proxy cache misses to your running web servers (like Mongrel), but with Passenger you don't know what processes are running. This is because Passenger automatically spawns web server processes for you. Well, after a little trial and error I figured out a config that works.

I'll show the config at the bottom of this post, but first let's look at some performance numbers. For my use case this goes beyond simple page caching. I'm using HTTP services to serve up data. So it's entirely possible that I can achieve a high cache hit ratio and make things really speedy. To check performance I set up two tests. The first returns a small json object and the second returns a collection of objects. The test Passenger application is a simple Sinatra service that looks like this:

require 'rubygems'
require 'json'
require 'sinatra'

get '/api/v1/comments/users/:id/no_cache' do
 ([{:id => 2, 
   :user_id => "paul", 
   :body => "this is a test comment to see how this thing works."}
  ] * 25).to_json
end

get '/api/v1/comments/:id/no_cache' do
 {:id => 2, 
  :user_id => "paul", 
  :body => "this is a test comment to see how this thing works."}.to_json
end

The setup was a single small EC2 instance running nginx and another small instance in the same availability zone running Apache Bench against the server. The first run is for cache misses that fall through to Passenger. The second run is against cache hits that go from nginx straight to memcached. Here's a small section of the results:

# single json object cache miss (passenger)
ab -n 10000 -c 50 http://ec2-some-ip.compute-1.amazonaws.com/api/v1/comments/1/no_cache
Requests per second: 310.08 [#/sec] (mean)
Time per request: 161.249 [ms] (mean)
Time per request: 3.225 [ms] (mean, across all concurrent requests)
Transfer rate: 94.17 [Kbytes/sec] received

# single json object cache hit (nginx + memcached)
ab -n 10000 -c 50 http://ec2-some-ip.compute-1.amazonaws.com/api/v1/comments/1
Requests per second: 2496.64 [#/sec] (mean)
Time per request: 20.027 [ms] (mean)
Time per request: 0.401 [ms] (mean, across all concurrent requests)
Transfer rate: 556.12 [Kbytes/sec] received

# collection of 25 json objects cache miss (passenger)
ab -n 10000 -c 50 http://ec2-some-ip.compute-1.amazonaws.com/api/v1/comments/users/1/no_cache
Requests per second: 225.98 [#/sec] (mean)
Time per request: 221.256 [ms] (mean)
Time per request: 4.425 [ms] (mean, across all concurrent requests)
Transfer rate: 530.31 [Kbytes/sec] received

# collection of 25 json objects cache hit (nginx + memcached)
Requests per second: 2385.22 [#/sec] (mean)
Time per request: 20.962 [ms] (mean)
Time per request: 0.419 [ms] (mean, across all concurrent requests)
Transfer rate: 5434.29 [Kbytes/sec] received

As you can see the time per request and the number of concurrent requests is significantly higher for the cache hits. The performance bump this reflects is actually better than the 400% Ilya mentioned in his blog post (of course, your mileage may vary). If you're interested, I've posted the full ab output in this gist.

And finally, without any more delay, here's the nginx.conf I used.

user nobody nogroup;
worker_processes 4;

error_log logs/error.log;

events {
 worker_connections 1024;
}

http {
 passenger_root /usr/lib/ruby/gems/1.8/gems/passenger-2.2.5;
 passenger_ruby /usr/bin/ruby1.8;

 include mime.types;
 default_type application/octet-stream;
 tcp_nopush on;
 tcp_nodelay on;
 gzip on;

 sendfile on;

 keepalive_timeout 65;

 server {
 listen 80;
 server_name localhost; # put in your IP in place of localhost
 location / {
 set $memcached_key $uri;
 memcached_pass 127.0.0.1:11211;
 default_type application/json;
 error_page 404 502 = @fallback;
 }

 location @fallback {
 root /var/www/public;
 passenger_enabled on;
 }
 }
}

First NYC Machine Learning Meetup

2009-09-10T11:22:35-04:00

I have organized the first NYC Machine Learning meetup. The goal of the group is to have discussions based on recent papers in the machine learning community. This can include topics like search, information retrieval, pattern recognition, natural language processing, artificial intelligence, collaborative filtering, recommendation systems, and other topics related to machine learning.

The format is pretty simple. Pick a research paper. Everyone in the group reads and attempts to understand the paper before the meetup. Meet and discuss with one person leading the discussion. For the first meeting we'll be going over Yehuda Koren's paper "Collaborative Filtering with Temporal Dynamics." This paper is based on his work in the BellKor Pragmatic Chaos team that is in contention for the Netflix prize.

To RSVP go to the NYC Machine Learning meetup page. You don't need to be a machine learning expert to join in the conversation. Just make an honest attempt to read the paper and we can go over questions.