The PubSub specification within XMPP, as defined in XEP-0060, is definitely not as flexible as that of AMQP, but it is often times enough to cover the most popular use cases. However, technical merits aside, one of the key missing components, especially in Ruby, has been the historical lack of functioning libraries - xmpp4r claims to support it, but examples are lacking. Thankfully, after test driving the latest batch of gems, it looks like we're finally there.

Getting off the ground with XMPP

Without a good toolkit XMPP can be a gnarly protocol to get started with - Pidgin IM client has some great tools for spying on the exchange, but monitoring pages of XML scroll by can only get you so far. Thankfully, Seth Fitzsimmons has built switchboard ("curl for XMPP"), which offers a powerful command line tool to greatly simplify the process. Make sure to read the full tutorial, or jump right into it by testing it with the Wordpress XMPP stream:

# list available options, subscribe to a blog, list subscriptions and then open the stream
switchboard disco --target pubsub.im.wordpress.com info
switchboard pubsub --server pubsub.im.wordpress.com --node /blog/icanhazcheesburger.com subscribe
switchboard pubsub --server pubsub.im.wordpress.com subscriptions
switchboard pubsub --server pubsub.im.wordpress.com listen

Based on xmpp4r, switchboard is also a toolkit for assembling your own XMPP clients, which means that it can be easily customized to power a PubSub consumer. From start to finish, and since examples are still hard to come by:

> switchboard-pubsub.rb

require 'rubygems'
require 'switchboard'
 
class WordpressJack
  def self.connect(switchboard, settings)
    switchboard.plug!(PubSubJack)
    switchboard.hook(:post)
 
    switchboard.on_pubsub_event do |event|
      event.payload.each do |payload|
        payload.elements.each do |item|
          on(:post, item)
        end
      end
    end
  end
end
 
settings = Switchboard::Settings.new
settings['pubsub.server'] = 'pubsub.im.wordpress.com'
settings['jid'] = 'user@im.wordpress.com'
settings['password'] = 'password'
 
switchboard = Switchboard::Client.new(settings)
switchboard.plug!(WordpressJack)
 
switchboard.on_post do |post|
  puts "A new post was received:"
  puts post.methods.sort.uniq
  exit
end
 
switchboard.run!
 

XMPP with EventMachine and Nokogiri

If you have an EventMachine stack, or looking for a high performance library, Jeff Smick's blather is definitely a gem to investigate. The combination of the asynchronous nature of EventMachine, a SAX parser within Nokogiri, and a great DSL make it very fast and a pleasure to work with:

> blather-pubsub.rb

require 'rubygems'
require 'blather/client/client'
require 'blather/client/dsl/pubsub'
require 'blather'
 
EventMachine.run {
  host = 'pubsub.im.wordpress.com'
  node = 'blog/icanhazcheesburger.com'
  user = 'user@im.wordpress.com'
  pass = 'pass'
 
  jid = Blather::JID.new(user)
  client = Blather::Client.setup(jid, pass)
  client.register_handler(:ready) {
    puts "Connected. Send messages to #{client.jid.inspect}."
    pub = Blather::DSL::PubSub.new(client, host)
  }
 
  client.register_handler(:pubsub_event) { |event|
    puts event
  }
 
  client.connect
}

PubSub & Event-Driven Architecture

Having personally struggled in the past with XMPP PubSub and Ruby, it's been great to revisit the use case and find a new set of fully functional libraries. The event driven architecture which is enabled by technologies such as XMPP, AMQP, Comet, Webhooks and PubsubHubbub are increasingly becoming the staple of many web applications, and for a good reason. If you haven't already, grab switchboard or blather and take XMPP for a test drive.

Coined by Alex Russell in early 2006, the term Comet is an umbrella term for technologies which take advantage of persistent connections initiated by the client and kept open until data is available (long polling), or kept open indefinitely as the data is pushed to the client (streaming) in chunks. The immediate advantage of both techniques is that the client and server can communicate with minimal latency. For this reason, Comet is widely deployed in chat applications (Facebook, Google, Meebo, etc), and is also commonly used as a firehose delivery mechanism.

Converting Nginx into a Long Polling Comet Server

A large entry barrier to Comet adoption is the implicit requirement for specialized, event driven web servers capable of efficiently handling large numbers of long polling connections. Friendfeed's Tornado server is a good example of an app level server that meets the criteria. However, thanks to Leo Ponomarev's efforts, you can now also turn your Nginx server into a fully functional Comet server with the nginx_http_push_module plugin.

Instead of using a custom framework, Leo's plugin exposes two endpoints on your Nginx server: one for the subscribers, and one for the publisher. The clients open long-polling connections to a channel on the Nginx server and start waiting for data. Meanwhile, the publisher simply POST's the data to Nginx and the plugin then does all the heavy lifting for you by distributing the data to the waiting clients. This means that the publisher never actually serves the data directly, it is simply an event generator! It is hard to make it any simpler then that.

Best of all, it only gets better from here. Both the client and the publisher can create arbitrary channels, and the plugin is also capable of message queuing, which means that the Nginx server will store intermediate messages if the client is offline. Queued messages can be expired based on time, size of the waiting stack, or through a memory limit.

Configuring Nginx & Ruby Demo

To get started you will have to build Nginx from source. Unpack the source tree, grab the plugin repo from GitHub and then build the server with the push module (./configure --add-module=/path/to/plugin && make && make install). Next, consult the readme and the protocol files to learn about all the available options. A simple multi client broadcast configuration looks like the following:

> nginx-push.conf

# internal publish endpoint (keep it private / protected)
location /publish {
  set $push_channel_id $arg_id;      #/?id=239aff3 or somesuch
  push_publisher;
 
  push_store_messages on;            # enable message queueing 
  push_message_timeout 2h;           # expire buffered messages after 2 hours
  push_max_message_buffer_length 10; # store 10 messages
  push_min_message_recipients 0;     # minimum recipients before purge
}
 
# public long-polling endpoint
location /activity {
  push_subscriber;
 
  # how multiple listener requests to the same channel id are handled
  # - last: only the most recent listener request is kept, 409 for others.
  # - first: only the oldest listener request is kept, 409 for others.
  # - broadcast: any number of listener requests may be long-polling.
  push_subscriber_concurrency broadcast;
  set $push_channel_id $arg_id;
  default_type  text/plain;
}
 

Once you have the Nginx server up and running, we can setup a simple broadcast scenario with a single publisher and several subscribers to test-drive our new Comet server:

> comet-push-consume.rb

require 'rubygems'
require 'em-http'
 
def subscribe(opts)
  listener = EventMachine::HttpRequest.new('http://127.0.0.1/activity?id='+ opts[:channel]).get :head => opts[:head]
  listener.callback {
    # print recieved message, re-subscribe to channel with
    # the last-modified header to avoid duplicate messages 
    puts "Listener recieved: " + listener.response + "\\n"
 
    modified = listener.response_header['LAST_MODIFIED']
    subscribe({:channel => opts[:channel], :head => {'If-Modified-Since' => modified}})
  }
end
 
EventMachine.run {
  channel = "pub"
 
  # Publish new message every 5 seconds
  EM.add_periodic_timer(5) do
    time = Time.now
    publisher = EventMachine::HttpRequest.new('http://127.0.0.1/publish?id='+channel).post :body => "Hello @ #{time}"
    publisher.callback {
      puts "Published message @ #{time}"
      puts "Response code: " + publisher.response_header.status.to_s
      puts "Headers: " + publisher.response_header.inspect
      puts "Body: \\n" + publisher.response
      puts "\\n"
    }
  end
 
  # open two listeners (aka broadcast/pubsub distribution)
  subscribe(:channel => channel)
  subscribe(:channel => channel)
}
 

nginx-push.zip (Full Nginx Config + Ruby client)

Downloads: 133 File Size: 2.7 KB

In the script above, every five seconds a publisher emits a new event to our Nginx server, which in turn, pushes the data to two subscribers which have long-polling connections open and are waiting for data. Once the message is sent to each subscriber, Nginx closes their connections and the clients then immediately re-establish them to wait for the next available message. End result, a real-time message push between the publisher and the clients via Nginx!

Long Polling, Streaming, and Comet in Production

Leo's module is still very young and is under active development, but it is definitely one to keep an eye on. The upcoming release is focused on bug fixes, but looking ahead there are also plans to add a streaming protocol: instead of closing the connection every time (aka, long polling), Nginx would keep it open and stream the incoming events as chunks of data to the clients in real-time. Having such an option would make it ridiculously easy to deploy your own firehose API's (ex: Twitter streaming).

Last but not least, don't forget about the growing number of other available modules for Nginx, or if you are so inclined, get a head start on building your own by reading Evan Miller's great guide on the subject.

Originally developed at JP Morgan as a vendor neutral wire and broker protocol, AMQP (Advanced Message Queuing Protocol) is, in fact, a general purpose messaging bus. The protocol itself is still under active development, but there are a variety of open source client and server implementations for it, as well as some big commercial supporters (RedHat, Microsoft, etc). In other words, it works, it is production ready, and I can vouch for it from personal experience - we stream tens of millions of messages through AMQP at PostRank on a daily basis.

AMQP vs XMPP: Features & Architecture

The AMQP vs XMPP debate has been raging for years now. On the surface they both look identical, but in reality there are a number of important distinctions. For example, presence is one of the central components of XMPP, but it is not part of the AMQP specification. XMPP uses XML, whereas AMQP has a binary protocol. AMQP has native support for a number of delivery use cases (at least once, exactly once, select subscribers, persistence, etc) and also a variety of exchange implementations which allow fine-grained control to where and how the messages are routed.

The AMQP spec is a fast and recommended read, but by a way of quick introduction, the core architectural components are: publisher, exchange, queue, and consumer. As you may have guessed, the publisher is the data producer which pushes messages to an exchange. Why is it called an exchange? Because the exchange is a routing engine which is responsible for delivering the messages to the right queues (exchanges never store messages). For example, a message may need to be routed to just a single queue (direct exchange), maybe the message should be forwarded to every queue (pubsub) in the list (fanout exchange), or perhaps the message should be routed based on a key (topic exchange).

Publishing & Consuming AMQP in Ruby

The type of exchange, message parameters, and the name of the attached queue can all contribute to the delivery and routing behavior of the message. However, for the sake of example, let's create a simple pubsub fanout exchange in Ruby:

> amqp-publisher.rb

require 'amqp'
 
AMQP.start(:host => 'localhost') do
  # create a fanout exchange on the broker
  exchange = MQ.new.fanout('multicast')
 
  # publish multiple messages to fanout
  exchange.publish('hello')
  exchange.publish('world')
end
 

In order to consume the messages from an exchange the consumer needs to create a queue and then bind it to an exchange. A queue can be durable (survive between server restarts), or auto-deletable for cases when the queue should disappear if the consumer goes down. Best of all, once the queue is bound to an exchange, the messages are streamed to the client in real-time via a persistent connection (no polling!):

> amqp-consumer.rb

require 'amqp'
 
AMQP.start(:host => 'localhost') do
  amq = MQ.new
 
  # bind 'listener' queue to 'multicast' exchange
  amq.queue('listener').bind(amq.fanout('multicast')).subscribe do |msg|
    puts msg # process your message here
  end
end
 

amqp.git (AMQP client implementation in Ruby/EventMachine)

Downloads: 195 File Size: 0.0 KB

Advanced AMQP Recipes

The flexibility of the message and the exchange model is what makes AMQP such a powerful tool. Whenever a publisher generates a message, he can mark it as 'persistent' which will guarantee delivery through the broker - if there is an attached queue, it will accumulate messages until the consumer requests them. However, if you're streaming transient data (access logs, for example), you can also disable message persistence and not worry about overwhelming your broker. That's how you achieve 'exactly-once' vs 'at least once' semantics.

Trying to build a pubsub hub? Create a fanout exchange and attach as many queues as you want, each consumer will receive a copy of the message. Load balancing? Bind two workers to the same queue and the broker will automatically round-robin the messages (there is no upper limit on the number of workers). Failover? By default the AMQP broker does not require a message to be ACKed by a consumer, but with a simple configuration flag the messages will be kept on the server until the ACK is received. If the consumer goes down without ACKing a message, they will be automatically put back on the queue. Need to route a message based on a key? Topic exchange allows partial matching based on a message key that is set by the producer. Do you want to notify the producer if there are no subscribers attached to a queue? Set the immediate flag on the message and the broker will do all the work.

Best of all, you can also compose these patterns to cover virtually any delivery use case!

AMQP Brokers & Ruby / Rails

There are a variety of available broker implementations: ZeroMQ, ActiveMQ, OpenAMQ, and RabbitMQ. Because the underlying protocol is still in flux, there is definitely some variation between all the implementations - do your homework. If you're looking for a speed demon, ZeroMQ claims a 15.4 microsend routing overhead (4+ million msgs/s). However, RabbitMQ is arguably the most stable and feature complete broker implementation. If you are a CentOS or a Fedora user, you'll be happy to know that it is now part of the distro (yum install rabbitmq-server), otherwise follow the installation instructions.

Once the server is installed, follow the administration guide to start the broker. If you're looking for a RESTful or a GUI tool to help you configure the broker, drop in Alice on the same server. Like the SQL prompt? Install the BQL plugin and familiarize yourself with the syntax.

The AMQP gem is probably the best choice when it comes to Ruby clients - it is asynchronous, it is fast, and it is in use by dozens of companies. If you're looking for a synchronous client, Carrot gem is the answer. If you're using async-observer plugin in your Rails projects, you can drop in async-observer-amqp to migrate to AMQP. In other words, it is easy to get started, it is incredibly powerful, and it has great library support for virtually every language. Give it a try.

Having said that, the auto-scale, load-balancing, geographic redundancy, BigTable datastore, XMPP, memcached and the newer cron and taskqueue services, are definitely all reasons to revisit the platform. John Woodell and Ryan Brown, both Google employees, have been doing terrific work on helping to lower the entry barrier. In fact, while the documentation still needs a lot of work and polish, the deployment story and the API's are basically there - it now takes just a few keystrokes to get your Ruby app on GAE!

Migrating to Google App Engine

The first thing you'll notice with App Engine is that most of your Ruby/Rails applications won't run on the platform out of the box. GAE offers a lot of great benefits, such as free load-balancing and access to the BigTable datastore amongst many other things, but it does so by abstracting or removing some of the interfaces we're all used to: there is no filesystem, BigTable is not a relational database, cron jobs are HTTP calls, and so on. While an inconvenience at first, the functionality is still there but it does require some extra work on behalf of the developer. For example, while ActiveRecord does not work at the moment, there is a DataMapper adapter that can meet all of your needs. Likewise, your code cannot make arbitrary outbound HTTP connections, but you can patch net/http to use the URLFetch API.

The net positive of all of these changes is that your application can then be uniformly scaled across the GAE infrastructure. By default, you receive a free quota for up to (roughly) five million requests, and beyond that a credit card is required. Best of all, you no longer have to think about hardware, databases, or any other supporting services. BigTable guarantees uniform access time for your queries irrespective of the size of the dataset (at a cost of higher overall latency), and CPU, memcached, and XMPP load is automatically spread within the GAE cluster. And if you're feeling adventurous, you could try deploying a distributed file system (GaeVFS), or even build your own GAE hosting environment from scratch with AppScale.

Hello World with GAE

The tooling support for getting your application on GAE is now almost fully automated. Ryan Brown's appengine-api's gem has full coverage of all the essential services, and the google-appengine gem provides a number of essential tools to help you with the generation, setup, and deployment of your applications. A simple "Hello World" Rack application is as simple as:

> rack-gae.rb

# install GAE tools (no JRuby required)
# > sudo gem install google-appengine
 
# create a rackup file: config.ru
require 'appengine-rack'
 
AppEngine::Rack.configure_app(
    :application => "your-app-id",
    :version => 1)
 
run lambda { Rack::Response.new("Hello World!") }
 
# start development server on local server
# > dev_appserver.rb .
 
# deploy application to GAE
# > appcfg.rb update .
 

All the configuration and deployment logic is done on your behalf by the dev_appserver and appcfg executables: packaging JRuby, creating the XML configs, authentication, etc. Deploying a Rails application, takes just a few more steps. Once the app is live, login into your GAE dashboard to scan the logs, query BigTable, or track your quotas - no need for mucking around with any system utilities!

Developing for App Engine

In theory, GAE has all the potential to become a popular deployment platform for Ruby applications (not to mention a nice market for custom Google Apps modules). The lacking resources at this point are the blog posts, developers, and the overall Ruby community mindshare around App Engine. John Woodell and Ryan Brown have done a fantastic job of lowering the entry barriers, but we're still missing the real-life production deployments that would weed out the API bugs and inconsistencies, as well as, build the overall trust in the platform.

Curious to give it a try, I took an evening to try and port a small, but non-trivial app to App Engine. Having settled on watercoolr (webhooks pubsub), I swapped out SQLite for BigTable and a few keystrokes later, I had a public watercoolr service (http://watercoolr.appspot.com) in production. No configuration, no need to worry about overloading the database. Feel free to use it for your own HTTP pubsub needs! GAE is an exciting platform, check out the examples, articles, and make sure to give it a try.

Few people would argue against the idea of consulting several experts on any complex subject - as the saying goes: "two heads are better than one". Having said that, the ensemble techniques which leverage this exact strategy are rarely a hot topic in the machine learning communities, or at least, up until now. Given their recent success this is likely to change, but perhaps even more importantly, their effectiveness may actually force the 'collaborative filtering' space to become, well, a lot more collaborative.

GitHub Contest: The Good, The Bad

Religious arguments about the merits of each distributed version control system aside, one thing GitHub does really well is allow users to follow and discover new open source projects. In that light, a competition for a recommendations engine makes perfect sense. As part of the challenge you get access to records of over 56K users, 120K repositories, and 440K relationships between each. Great data set, immediate feedback via post-commit hooks, all results available in the open, and a few people even shared their code!

If there was one thing to improve about the contest, then it would have to be the ranking methodology. The goal was to "guess" the 4,788 removed repositories, which I would argue, is optimizing for the wrong thing. The goal should have been not to guess what users are already watching, but to produce a more general predictor for what users should be watching - the former is a subset of the latter. To see why, think of the case where a general predictor might actually have a set of better recommendations (based on other user's patterns) than the few repos that the GitHub team hid from the dataset for any user.

Ensemble Techniques & Machine Learning

Ranking methodology aside, perhaps the most interesting result of the GitHub contest is due to how it was setup: each submission is a push into a public Git repo, which means that each entry is automatically in the public domain. Unlike the Netflix contest, where all submissions were private and teams had to agree to merge their results, the GitHub contest became a free-for-all and a fertile crowd for testing ensembles!

While the topic of Ensembles is a rich one, the general idea is remarkably simple: instead of attempting to build one general model to capture all the subtleties of the data, use multiple predictors and combine their results. For an illustrative example, assume that we were trying to build a model that would separate the data in the diagram above. After a brief examination, we form a simple hypothesis (call this hypothesis family H): we could separate the data using simple geometric shapes such as a circle (H1), or a square (H2). Visually we can clearly see that neither one by itself will do a good job, but applying them both gives us a pretty good approximation while the model remains very simple (we want our model to be simple, for computational reasons). So what do we do? We train two independent classifiers and then merge their results, aka, use an ensemble of classifiers.

The description above sweeps a lot of details under the rug, but core of the idea is there. Boosting, bagging, consensus aggregation, dynamic classifier selection, and dozens of other techniques have been developed on top of this process and have proven to be very successful. The overarching principle of ensemble techniques is to make each predictor as unique as possible: use a different learning algorithm (decision trees, svm, svd), or use a different feature (random subspace method). Then, once you have many individual classifiers, determine a mechanism to join the results (simple method: each predictor votes on each point, then tally up the votes) and make a final prediction using the new classifier.

GitHub Ensembles

Perhaps the most illustrative example of this simple technique is John Rowell’s entry into the GitHub contest, which is best described by a snippet right from the source:

> ensemble.rb

require 'nokogiri'
require 'open-uri'
 
class Crowdsource
  def initialize
    load_leaderboard  # scrape github contest leaders
    parse_leaders     # find their top performing results
    fetch_results     # download best results
    cleanup_leaders   # cleanup missing or incorrect data
    crunchit          # build an ensemble
  end
  #...
end
 
Crowdsource.new
 

ghcontest-crowdsourced.git (John Rowell's ensemble predictor)

Downloads: 395 File Size: 0.0 KB

Because all the entries in the GitHub contest are in the public domain (actually, a few people caught on and started deleting their repos immediately after their submissions because of this technique), we can download the best predictions of other users, construct an ensemble, and then submit our own set of results.

Collaborative, Collaborative Filtering: CCF!

While on first examination the ensemble technique may have a flavor of cheating, I think it can actually have a significant effect on how future challenges of this sort could be structured. Don't fight it, embrace it. Instead of focusing on individual effort or insight, it is not hard to imagine much more collaborative, collaborative-filtering competitions (CCF's). Some teams can focus on building great independent predictors, while others can focus on developing and improving the methods for constructing ensembles (a mashup, of sorts). In other words, collaborative filtering has the potential to become a lot more collaborative! And in the meantime, I wish the GitHub guys luck in trying to determine the winner.

Update: GitHub removed most ensemble submissions / released the code to power the challenge.

Javascript is (Almost) Everywhere

Like it or not, disabling JavaScript in your browser today is likely to render a large section of the web as completely unusable, and that is unlikely to change anytime soon. For that reason, over the years there have been many recurring rumors of the Google bot rendering Javascript when it came to your page. Google denies this, but there are also theories that the V8 engine is a direct result of such initiatives - which, I think, is not completely unfounded.

In either case, if you ever attempted this yourself, you quickly realized just how non-trivial this problem actually is. A typical spider just downloads the HTML of the page, it has not knowledge of the DOM, or Javascript. So, at the end of the day, what you need is a browser: the content must be downloaded, the HTML needs to be interpreted into a DOM model, and then finally handed off to the Javascript runtime for further processing. That's a lot of work.

Jaxer, the AJAX Server

Having attempted at this problem many times with Rhino, SpiderMonkey and even raw Webkit bindings, I finally stumbled on Jaxer. Developed at Aptana, it is based directly on Gecko (Firefox rendering engine), minus the graphical rendering, which means that the DOM is already there and Javascript is free of charge. I never bought into the server-side Javascript movement, but seeing Jaxer in this light definitely opened my eyes to a lot of opportunities. The installation is as easy as unpacking the Apache runtime and then proxying your requests through the server. For example, to get the full post-Javascript DOM of any web-page, just drop this template into your Jaxer directory:

> post-js.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
  "http://www.w3.org/TR/html4/loose.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Post JavaScript DOM</title>
  </head>
  <body>
 
    <!-- Render post-JS DOM: http://jaxer/post-js.html?www.google.com -->
    <script runat='server'>
       var url_query = Jaxer.request.queryString;
       var sb = new Jaxer.Sandbox(url_query);
       Jaxer.response.setContents(sb.toHTML());
    </script>
  </body>
</html>
 

post-js.html (Proxy through Jaxer for post-JS DOM)

Downloads: 320 File Size: 0.6 KB

Accessing the Server-Side DOM

The first thing you'll notice is just how slow most of the pages will come back when proxied through Jaxer. However, this has nothing to do with Jaxer and everything to do with the dozens of Javascript files and slow remote calls our browsers have to do on our behalf. Their asynchronous nature just happens to mask the abysmal performance of most pages. Which, incidentally, is also likely the reason why our search engines today do not look at the post-JS DOM.

If you haven't already, take a look at the Jaxer tutorials. You have full access to the DOM on the server, which means that you can use JQuery to alter the response, connect to a database, test your Javascript on the fly, etc. Having said that, I'm still secretly hoping that one day our curl command client will just have a flag to return the final post-Javascript DOM.

Smart(er) Clients: Pushing Complexity to the Edges

Our web-browsers today are, for the most part, passive consumers. This model has worked wonderfully up until now, but as thought experiment, think about the implications on your architecture if the browsers were smarter. For example, what if the browser also contained a web server? For one, this would mean that the browser and the web-server would be peers: they could talk via HTTP, use OAuth, Webhooks, etc!

Brushing NAT and firewall complications aside for a few seconds, this would mean that any browser could register itself via a webhook, and the server would simply propagate the updates to each client whenever new data is available (aka, PubSubHubbub with the client). Fast and efficient.

So how do you go about running a web-server in your browser? It is possible in theory, but the support is lacking. Firefox has a built in Javascript webserver for testing, but it requires XPCOM privileges; there is some talk on the Jetpack mailing lists about supporting this feature, but nothing conclusive; node.js is an evented V8 powered web-server, but it can't run within Chrome. The closest you will get without modify the core of your browser is Firefox POW (Plain Old Webserver) extension, but we need something much more flexible.

Hybrid Clients: ReverseHTTP & Supernodes

The first time you read the proposed ReverseHTTP, or equivalent spec, it will undoubtedly feel like a crazy idea. Having said that, it works. Because most clients are hidden behind personal / corporate firewalls or NAT's, it renders them unreachable for a raw client-side HTTP push model. ReverseHTTP proposes a hybrid solution: another web-server in middle acts a proxy, and the client maintains a persistent connection to it. Whenever a request comes to the proxy, it is relayed to the client via the persistent connection, where the response is determined by the browser and finally relayed back to the originating source. In effect, you are running a webserver in your browser. Give it a try, and if you’re curious, check out the server code as well.

Unfortunately, this is still a half-hearted solution because it introduces yet another layer of infrastructure into the equation, but it also has its benefits. First, the scalability complexity is propagated down to the edges – a client, a hosting provider, or even an ISP can maintain such proxy nodes for their users. In fact, that is exactly what Opera Unite is all about. Blur your eyes to cut through the marketing, and you'll see that, in fact, the "revolution" is the ability for a browser to act as a server, via the Opera Unite proxy service. Sounds crazy? Microsoft's Teredo service (tunneling IPv6 traffic over IPv4) has been providing this very service for free for several years now to all Windows users. Last but not least, this is also the same principle that powers your favorite P2P client (Skype, Kazaa, etc). It is not as crazy as it seems.

Bi-directional Communication With WebSockets API

Of course, we also can't forget Comet, BOSH, or many other attempts at the same problem. Personally, I've been holding off for a simpler solution. All I want is a socket – Flash has it, why can't the web browser too? Thankfully, HTML 5 has an answer: WebSocket API.

> websocket.js

// open a websocket
var conn = new WebSocket("ws://yourwebservice.com/service");
 
// act on incoming data
conn.onopen  = function(e) { ... }
conn.onread  = function(e) { ... }
conn.onclose = function(e) { ... }
 
// push messages back to server
conn.send("Bi-directional!");
 

As more and more browsers start adopting HTML 5 features I'm hoping to see WebSockets become a reality within the next couple of years. Full bi-directional data exchange between the client and server, without a need for a third party in between – that is what Opera Unite announcement should have been. It still means maintaining persistent connections between client and server, which is arguably not as elegant as registering your browser session through a webhook, but there is a need for both solutions. One is great for intermittent updates, the other (WebSockets) are the right answer for high-velocity streams.

HTTP Caching Extensions: stale-*

Both stale-while-revalidate and stale-if-error are the direct results of Mark Nottingham's work at Yahoo. It is clear that Squid plays a big role in the Y! infrastructure and these extensions are undoubtedly deployed in their data centers. Stale-while-revalidate addresses a simple problem: when a record becomes stale, instead of allowing the request to hit the application server, serve the stale data to the client and create an asynchronous request to update the cache (read more & spec). The benefit? All of your customers see consistent performance because the data is always served out of the cache (think RSS feeds, search results, etc).

The second extension (stale-if-error) can help you mask downtime by returning stale data while your ops team resolves the problem. How does it work? You specify a cache-control header (Cache-Control: max-age=600, stale-if-error=1200) which indicates for how long, after the record is expired the cache server can serve this data if the application server is down. Of course, nothing stops you from setting this to a high value like a day or longer! This way, if your server goes down, at least the clients won't timeout on their requests while you're resolving the problem!

Setting up Squid as a Reverse Proxy

Varnish and Squid are the two top choices when it comes to caching reverse proxies. Varnish has been getting a lot momentum but unfortunately the features we're looking for are still in the works. There is an open ticket for stale-if-error support, and stale-if-revalidate implementation unfortunately does not provide the full asynchronous refresh model. For that reason, we're going to use Squid 2.7 (note: Squid 3.0 is a full rewrite of the 2.x branch, and while stale-* patches exist, they haven't officially made it into the tree). A minimal Squid config (if there is such a thing) to get us up and running:

> squid.conf

# http_port public_ip:port accel defaultsite= default hostname, if not provided
http_port 0.0.0.0:80 accel defaultsite=yourdomain.com
 
# IP and port of your main application server (or multiple)
cache_peer 192.168.0.1 parent 80 0 no-query originserver name=main
cache_peer_domain main yourdomain.com
 
# Do not tell the world that which squid version we're running
httpd_suppress_version_string on
 
# Remove the Caching Control header for upstream servers
header_access Cache-Control deny all
 
# log all incoming traffic in Apache format
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
access_log /usr/local/squid/var/logs/squid.log combined all
 

squid.conf (Full Squid 2.7 Reverse Accelerator config)

Downloads: 274 File Size: 167.9 KB

Connecting Squid and your Application

Once Squid is deployed as part of your request chain, taking advantage of its caching mechanism is a trivial matter, just add the "Cache-Control" header! A simple Rack application to make it all work:

> rack-stale.rb

require "rack"
 
app = lambda {
  p [:new_request, Time.now]
 
  headers = {
    # cache for 10 seconds, serve stale for up to 20s, and for up to 30s on errors
    "Cache-Control" => "max-age=10, stale-while-revalidate=10, stale-if-error=20",
    "Last-Modified" => Time.now.to_s
  }
 
  [200, headers, "Hello World @ #{Time.now}"]
}
 
Rack::Handler::Mongrel.run(app, {:Host => "127.0.0.1", :Port => 80})
 

The only requirements are that we set a Last-Modified time, and then provide our preferred caching intervals. Max-age specifies the general lifetime of the object (TTL), stale-while-revalidate indicates for how long after max-age has expired can the server provide the stale data (total lifetime is max-age + stale-while-revalidate). Finally, stale-if-error indicates for how long the server can provide the stale data if the application server is down - it is not unusual to set this higher than stale-if-revalidate. That's it!

Masking Latency and Failures

Of course, these extensions do not excuse any of us from building fast and reliable web-services. In theory, we would have no need for these extensions, but as the saying goes: "in theory, theory and practice are the same, in practice, they are not." If you have data that can be served slightly stale (frankly, the majority of the data falls into this category), then Squid can make all the difference next time your server decides to take a break at 4AM.

Best of all, the rewrite allowed the project to shed its past baggage and build on a more modern stack with a better perspective of the required features for a modern deployment (Drizzle is another recent example from the MySQL camp). Case in point, Tokyo Tyrant supports Lua scripting on the server, allowing us to add arbitrary user defined functions (UDF's in MySQL parlance) and to extend the database itself!

Lowering the Barrier With Lua Scripting

If you have played with Lego Mindstorms, or ever tried to script your phone or a gaming environment (WoW), chances are you've worked with Lua. Due to its extremely small footprint (~212kb for source, documentation, and examples), fast interpreter, portability and easy to understand syntax it has become the de-facto embedded scripting language for many applications. Explore the available tutorials, or pick up "Programming Lua" by the creator of the language for a more in depth look, it's a fun language!

So why is Lua scripting such an exciting feature for Tokyo Tyrant? Because it lowers the barrier for programming and extending the database by an order of magnitude! If you have ever worked with MySQL UDF functions, you'll definitely appreciate the ease and the speed of development. No need for mucking with internal API's, nothing to compile or link against, and execution is done within a stable and sandboxed environment.

Extending Tokyo Tyrant with Lua

Anytime you bootup a Tokyo Tyrant server you can tell it to load arbitrary Lua code alongside which will then be evaluated at runtime if the client requests it. From there, the developer of the extension has full access to the incoming request and the underlying Tokyo Cabinet database, allowing us to inject arbitrary functionality. To get a flavor for the workflow, take a look at some of the examples in the following slides:

tokyo-recipes.git (Lua Extensions for Tokyo Tyrant)

Downloads: 503 File Size: 0.0 KB

Because the Lua scripting interface is relatively new, the number of available extensions and documentation is not very large. For that reason, I've started the tokyo-recipes repo on GitHub, in which I've aggregated some of the available extensions, and added extra documentation and examples to help grease the wheels: TTL functionality (ala memcached), working with Sets (ala Redis), session timestamping and a wordcount map-reduce example just to name a few. Give it a try, it is an extremely powerful feature!

The best part about Webhooks is that most of us are already familiar with them: callbacks over HTTP. Pioneered by PayPal and Subversion as a way to send real-time notifications to the client, they have found their way into many dozens of products we all use every day. Need pre or post commit hooks for your SVN or Git repository? Both GitHub and SVN support HTTP callbacks. Need a payment alert from PayPal, or an alert when a wiki page is modified? There are webhooks for that too. This simple mechanism allows us to build web services that work together via a simple and ubiquitous protocol we can all understand: HTTP!

Working with Webhooks

A good way to think about webhooks is as Unix pipes for the web. While PubSub is a great application of the technology, Webhooks allow bidirectional communication, which opens up our possibilities to: publishing notifications, chaining web services to perform complex actions, or allowing external plugins to enter the workflow. All of this functionality is accomplished via simple HTTP queries (both POST and GET), which means no special libraries or servers to make it all happen. In fact, building your own Webhooks powered PubSub server takes less than a hundred lines with the help of Sinatra, as demonstrated by the Julio Capote and his watercoolr project on GitHub. To see it in action, we can use PostBin as our mock recipient:

[ilya@igvita ~]# ruby postbin.rb http://www.postbin.org/1987c4m
creating channel...
adding subscriber to channel sej7u7
{"status":"OK"}
Post message: Hello Webhooks PubSub!
{"status":"OK"}

In this scenario, our local watercoolr service acts as a PubSub server, accepting requests to create custom channels, maintaining a subscriber list, and pushing out notifications when an update arrives at the channel. Best of all, interacting with the server is as simple as writing a RESTful client:

> postbin.rb

require "rubygems"
require "rest_client"
require "json"
 
puts "creating channel..."
resp = RestClient.post "http://localhost:4567/channels", :data => ""
id = JSON.parse(resp)["id"]
 
puts "adding subscriber to channel #{id}"
resp = RestClient.post "http://localhost:4567/subscribers", :data => {:channel => id, :url => ARGV[0]}.to_json
 
loop do
  print "Post message: "
  msg = STDIN.gets.chomp
 
  resp = RestClient.post "http://localhost:4567/messages", :data => {:channel => id, :message => msg}.to_json
  puts resp
end
 

PubSubHubbub Spec: Auth, Discovery & Protocol Implementation

Brad Fitzpatrick of Livejournal fame and Brett Slatkin, both of whom are currently at Google drafted a PubSubHubbub spec for a "simple web-scale pubsub protocol" which preserves the spirit of HTTP webhooks but adds a much needed layer of basic security, discovery, and coherent protocol definition:

We offer this spec in hopes that it fills a need or at least advances the state of the discussion in the pubsub space. Polling sucks. We think a decentralized pubsub layer is a fundamental, missing layer in the Internet architecture today and its existence, more than just enabling the obvious lower latency feed readers, would enable many cool applications, most of which we can't even imagine. But we're looking forward to decentralized social networking.

The google code project also provides an AppEngine implementation with a public hub, which we can test via an asynchronous (EventMachine based) PubSubHubbub Ruby library:

> pubsub-gae.rb

require "rubygems"
require "pubsubhubbub"
 
EventMachine.run {
  # publish single URL
  pub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/publish').publish "http://www.test.com/"
  pub.callback { puts "Successfully notified hub." }
  pub.errback  { puts "Uh oh, something broke: #{pub.response}" }
 
  # publish multiple URL's to hub
  feeds = ["http://www.test.com", "http://www.test.com/2"]
  pub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/publish').publish feeds
 
  pub.callback { puts "Successfully notified hub." }
  pub.errback  { puts "Uh oh, something broke: #{pub.response}" }
}
 

PubSubHubbub.git (Asynchronous PubSubHubbub Ruby Client)

Downloads: 503 File Size: 0.0 KB

Webhooks & PubSubHubbub

Of course, neither Webhooks nor PubSubHubbub are the answer to every problem. Both XMPP and AMQP will continue to exist alongside, but chances are, will take on the brunt of high-velocity feeds while Webhooks can happily power the remaining 90% of the simpler use-cases. In fact, there is already a RabbitMQ PubSubHubbub hub server implemenation in the works! For further reading, flip through Jeff Lindsay's presentations on webhooks, and if you are interested in PubSubHubbub check out this presentation by Brad Fitzpatrick and Brett Slatkin.