The code has evolved a bit since and proven useful in a number of production systems. I created a gist of Rack::CacheHeaders in case someone else finds it handy. The tool is not exhaustive in terms of policies as found in the HTTP specs, it’s a collection of the ones we needed in the projects it’s been used so far. Consider adding ones you need to the gist to make the code more complete and widely useful.

Rack::CacheHeaders allows configuring HTTP cache policy response headers based on request URI patterns. For example, to set the Cache-Control: max-age header for a /guitars/:id resource to one hour:

Rack::CacheHeaders.configure do |cache|
  cache.max_age(/^\/guitars\/d+$/, 3600)
end

Download/develop Rack::CacheHeaders

WEB_SERVICE_ADDRESS = 'http://www.example.com'

url = URI.parse(WEB_SERVICE_ADDRESS)

Net::HTTP.start(url.host, url.port) do |http|
  http.get('/product-search', 'q' => 'guitar')
end

Inspecting the response headers, we notice the web service instructs consumers that the results of the query will remain the same for one hour.

curl -I "http://www.example.com/product-search?q=guitar"

HTTP/1.1 200 OK
Content-Type: text/html
Cache-Control: max-age=3600, must-revalidate
Content-Length: 32650
Date: Sat, 14 Feb 2009 13:53:31 GMT
Age: 0
Connection: keep-alive

At this point we can choose to ignore the cache control header and keep on querying the service for this specific resource regardless of whether the response is going to be the same. This is suboptimal for the consumer, which will suffer unnecessary latency penalties, the service, which will have to respond to inessential requests, and the network which will be subject to unnecessary bandwidth usage. Another option involves making the web consumer aware of the service’s caching policies so that it only queries for data that it doesn’t have or data that’s become stale. This option remedies the above problems but introduces additional complexity to the consumer.

A third option involves introducing a caching proxy to the web consumer’s stack responsible for mediating the service/consumer interactions solely based on the content’s caching characteristics.

Benefits of this approach include: The consumer never has to deal with any caching logic; No effort is required in re-implementing cache handling code; It is likely that the caching engine will perform better than custom caching code in the consumer because it’s been built and optimized for this purpose; The caching proxy can be re-used by more than one types of consumer or more than one instances of the same consumer in the stack. As a possible side-effect, the caching proxy is an additional layer to the consumer stack and this can result in network (the consumer’s LAN) latency.

Here’s the configuration needed in order to use Varnish as a caching web consumer proxy for the above example.

# varnish.conf

backend default {
  .host = "www.example.com";
  .port = "http";
}

The only thing that changes in the consumer is the address it directs its requests to.

WEB_SERVICE_ADDRESS = 'http://service-proxy'

url = URI.parse(WEB_SERVICE_ADDRESS)

Net::HTTP.start(url.host, url.port) do |http|
  http.get('/product-search', 'q' => 'guitar')
end

By nature, key value stores offer one way of retrieving data, by some sort of primary key which uniquely identifies each entry. But what about queries that require more elaborate input in order to collect relevant entries? Full text search engines like Sphinx and Lucence do exactly this and when used in conjunction with a database will query their indexes and return a collection of ids which are then used to retrieve the results from the database. Full text search engines support indexing data sources other than RDBMSs, so there’s no reason why one couldn’t index a distributed key-value store.

Here, we’ll look at how we can integrate Sphinx with MemcacheDB, a distributed key-value store which conforms to the memcached protocol and uses Berkeley DB as its storage back-end.

Sphinx comes with an xmlpipe2 datasource, a generic XML interface aimed at simplifying custom integration. What this means is that our application can transform content from MemcacheDB into this format and feed it to Sphinx for indexing. The highlighted lines from the following Sphinx configuration instruct Sphinx to use the xmlpipe2 source type and invoke the ruby /app/lib/sphinxpipe.rb script in order to retrieve the data to index.

# sphinx.conf

source products_src
{
  type = xmlpipe2
  xmlpipe_command = ruby /app/lib/sphinxpipe.rb
}

index products
{
  source = products_src
  path = /app/sphinx/data/products
  docinfo = extern
  mlock = 0
  morphology = stem_en
  min_word_len = 1
  charset_type = utf-8
  enable_star = 1
  html_strip = 0
}

indexer
{
  mem_limit = 256M
}

searchd
{
  port = 3312
  log = /app/sphinx/log/searchd.log
  query_log = /app/sphinx/log/query.log
  read_timeout = 5
  max_children = 30
  pid_file = /app/sphinx/searchd.pid
  max_matches = 10000
  seamless_rotate = 1
  preopen_indexes = 0
  unlink_old = 1
}

Following is a Product class. Each product instance can present itself as xmlpipe2 data. The class itself gets the entire product catalog as a xmlpipe2 data source. It also has a search method used for querying Sphinx and retrieving matched products from MemcacheDB. Finally, there’s a bootstrap method for populating the store with some example data.

# product.rb

require "rubygems"
require "xml/libxml"
require "memcached"
require "riddle"

class Product
  attr_reader :id
  MEM = Memcached.new('localhost:21201')

  def initialize(id, title)
    @id, @title = id, title
  end

  def to_sphinx_doc
    sphinx_document = XML::Node.new('sphinx:document')
    sphinx_document['id'] = @id
    sphinx_document << title = XML::Node.new('title')
    title << @title
    sphinx_document
  end

  # Query sphinx and load products with matched ids from MemcacheDB
  def self.search(query)
    client = Riddle::Client.new
    client.match_mode = :any
    client.max_matches = 10_000
    results = client.query(query, 'products')
    ids = results[:matches].map {|m| m[:doc].to_s}
    MEM.get(ids) if ids.any?
  end

  # Load all products from MemcacheDB and convert them to xmlpipe2 data
  def self.sphinx_datasource
    docset = XML::Document.new.root = XML::Node.new("sphinx:docset")
    docset << sphinx_schema = XML::Node.new("sphinx:schema")
    sphinx_schema << sphinx_field = XML::Node.new('sphinx:field')
    sphinx_field['name'] = 'title'

    keys = MEM.get('product_keys')
    products = MEM.get(keys)
    products.each { |id, product| docset << product.to_sphinx_doc }

    %(<?xml version="1.0" encoding="utf-8"?>\n#{docset})
  end

  # Create a some products and store them in MemcacheDB
  def self.bootstrap
    product_ids = ('1'..'5').to_a.inject([]) do |ids, id|
      product = Product.new(id, "product #{id}")
      MEM.set(product.id, product)
      ids << id
    end
    MEM.set('product_keys', product_ids)
  end
end

The sphinxpipe.rb script looks like this.

# sphinxpipe.rb
Product.bootstrap
puts Product.sphinx_datasource

With MemcacheDB (or even memcached for the purpose of this example) running, we can tell Sphinx to create an index of products by invoking indexer --all -c sphinx.conf and then start the search daemon – searchd -c sphinx.conf. Now we’re ready to start querying the index and retrieving results from the distributed store.

puts Product.search('product 1').inspect

It is not uncommon for the database to become a performance hotspot. The integration of a fast, distributed key-value store with an efficient search engine can be an interesting substitute for high throughput data retrieval operations.

Consider online retailer websites which usually operate in two modes, one for visitors and one for logged in users. Logged in users are presented with a customized, session specific experience, yet data like the product catalog is essentially the same regardless of whether one is logged in or not and it makes sense for everyone to be accessing the same cached copy of a common resource.

A possible solution involves creating two separate web applications, one entirely dedicated to stateless interactions and one meant for pages that are rendered as part of a user’s session. This might seem like overkill, but it clearly enforces the divide between what can and what can’t be cached. It also promotes reuse of the system’s web caching layer, which now serves content to site “visitors” as well as to the stateful components. The stateful application can delegate requests for potentially cached content to its stateless counterpart via the caching layer and decorate the responses with session specific data.

Web caching presents but one way to cache data that remains static for predefined periods of time. Apart from harnessing proven existing tools, this form of caching comes with the advantage that its policies are universally understood and can significantly improve a website’s efficiency in ways beyond the maintainer’s control. Retrofitting web caching into an application that hasn’t been designed with it mind can be difficult, therefore it is worth to logically separate cacheable and non cacheable resources early on.

One way of getting around this problem involves creating more than one instances of the shared resource, one of which is considered “live”, the one the system’s clients interact with, and perform expensive operations on a copy which will itself become live the moment these operations conclude. This solution does not apply to every situation but can be useful in scenarios where real time is not a concern. In the example website’s case, we can create a copy of the database on which we run the processing jobs. The front end components run off the “stale”, live database copy whose performance is not affected by the jobs. Once the jobs complete we can switch databases and repeat the live component rotation process as needed. Live component rotation also nicely lends itself to distribution, as component copies can exist on different physical hosts.

Virtualization and cloud computing make this method all the more interesting. Imagine hosting a database server on Amazon EC2 with its static data stored on an EBS volume. We can snapshot the EBS volume, fire up a new EC2 instance, attach the snapshot to it, run the job and rotate live database instances once the jobs are complete with most parts of the system never having to worry about the costly operations taking place.

One of the advantages of offering executable code alongside a service’s data is client simplification by code reuse. For example, we can distribute a library that’s specific to the data on offer, so interested clients can make use of that functionality and avoid having to re-implement it. Another advantage is distributing computational load, which would otherwise have to be handled by the server, to clients.

To put things into perspective, consider a simplistic web API call that lists guitar models. Much like a JavaScript include, the response to http://example.com/guitars contains a line which advertises a guitar model Ruby library available at /libguit.rb.

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
  <head>
    <title>Guitars</title>
    <script type="text/ruby" charset="utf-8" src="/libguit.rb"></script>
  </head>
  <body>
    <ul id="guitars">
      <li>SG</li>
      <li>Les Paul</li>
      <li>Tele</li>
      <li>Strat</li>
    </ul>
  </body>
</html>

The libguit library has one method for iterating over an alphabetically sorted list of guitars.

module LibGuit
  class List
    def initialize(guitars)
      @guitars = guitars
    end

    def each_guitar_alphabetically(&block)
      @guitars.sort.each(&block)
    end
  end
end

Interested clients can load and use the library together with the retrieved data. Code-on-demand is an optional constraint, so clients that cannot interpret the code, Ruby in this case, or are not interested in using the library can safely ignore it without side effects.

require "rubygems"
require "hpricot"
require "open-uri"

doc = Hpricot(open("http://example.com/guitars"))

libguit_address = (doc / 'script[@type="text/ruby"]')[0][:src]
libguit_src = open("http://example.com#{libguit_address}").read
eval(libguit_src)

guitars = (doc / "#guitars li").map { |e| e.html }
LibGuit::List.new(guitars).each_guitar_alphabetically { |g| puts g }

This is a superficial example, but imagine a service which advertises an e-commerce website’s daily updated catalog of products. Instead of clients making queries like /products.xml?category=sports&sort=price, they could once a day download a zipped version of the day’s entire catalog and a library to manipulate its entries, relieving the service from any further requests and at the same time avoid maintenance costs, in case the data’s structure changes, as long as this is well abstracted by the on-demand library.

At this point many would voice well founded, security implication based objections. Although one could propose a security system reminiscent to that of applets, I would opt for a controlled environment where trust is granted, such as inter-department service offer/consumption inside the company. Also, in an Internet where many of us store our private email on Gmail or trust Amazon’s S3 with mission critical data, I wouldn’t have a problem dynamically loading code provided by, say, Amazon. It’s not very difficult to put basic safeguards in place to avoid catastrophic effects and, in any case, every option is viable as long as the benefits outweigh the costs.

Rack supports the notion of Middleware, pieces of code that sit between the HTTP request and response life cycle. Rack::Lint, for example, validates an application’s requests and responses according to the Rack specification.

Rack::Handler::Mongrel.run(
  Rack::Lint.new(app), :Host => "0.0.0.0",  ort => 9999
)

Similarly, if we were to implement a cache header producing layer on top of Rack we’d end up with a construct similar to the following.

Rack::Handler::Mongrel.run(
  Rack::Lint.new(
    Rack::CacheHeaders.new(app)
  ), :Host => "0.0.0.0",  ort => 9999
)

Here’s a possible way of configuring how an application provides HTTP caching headers based on URL path patterns.

Rack::CacheHeaders.configure do |cache|
  cache.max_age("/rock", 3600)
  cache.expires("/metal", "16:00")
end

Following is a potential implementation for the above.

module Rack
  class CacheHeaders
    def initialize(app)
      @app = app
    end

    def call(env)
      result = @app.call(env)
      header = Configuration[env['PATH_INFO']].to_header
      result[1][header.key] = header.value
      result
    end

    def self.configure(&block)
      yield Configuration
    end

    class Configuration
      def self.max_age(path, duration)
        paths[path] = MaxAge.new(duration)
      end

      def self.expires(path, date)
        paths[path] = Expires.new(date)
      end

      def self.[](key)
        paths[key]
      end

      def self.paths
        @paths ||= {}
      end
    end

    class MaxAge
      def initialize(duration)
        @duration = duration
      end

      def to_header
        Header.new("Cache-Control", "max-age=#{@duration}, must-revalidate")
      end
    end

    class Expires
      def initialize(date)
        @date = date
      end

      def to_header
        Header.new("Expires", Time.parse(@date).httpdate)
      end
    end

    class Header < Struct.new(:key, :value);end
  end
end

The code below is a minimal Rack based application.

require "rubygems"
require "rack"

app = proc {|env| [200, {"Content-Type" => "text/plain"}, "hello"]}

Rack::Handler::Mongrel.run(
  Rack::Lint.new(
    Rack::CacheHeaders.new(app)
  ), :Host => "0.0.0.0",  ort => 9999
)

In order to observe the caching related headers the application’s responses are decorated with we can use curl or something similar, i.e curl -I http://0.0.0.0:9999/rock or curl -I http://0.0.0.0:9999/metal. Output should look something like the following.

air:~ gmalamid$ curl -I http://0.0.0.0:9999/rock
HTTP/1.1 200 OK
Connection: close
Date: Sat, 08 Nov 2008 00:51:23 GMT
Cache-Control: max-age=3600, must-revalidate
Content-Type: text/plain
Content-Length: 5

air:~ gmalamid$ curl -I http://0.0.0.0:9999/metal
HTTP/1.1 200 OK
Connection: close
Date: Sat, 08 Nov 2008 00:51:16 GMT
Content-Type: text/plain
Expires: Sat, 08 Nov 2008 16:00:00 GMT
Content-Length: 5

Understanding and employing HTTP cache configuration not only enables harnessing the power of tools like Varnish or Squid, it also makes good citizens in a diverse ecosystem of HTTP aware browsers and caches outside an application’s knowledge or control.

require "rubygems"
require "sinatra"

def guitars
  @@guitars ||= ['Les Paul', 'SG']
end

get "/guitars" do
  guitars * ', '
end

This application exposes a /guitars resource, a request for which will always hit the application server if no caching has been in place. This can prove suboptimal had this been a high traffic website, especially if the operation of generating the content is system resource intensive. Luckily this problem has been solved before. A running instance of Varnish, for example, will only require the following configuration to enable caching of all resources the application serves.

backend default {
  .host = "127.0.0.1";
  .port = "4567";
}

One of the challenges associated with caching has to do with the cached content’s freshness. We want to relieve server stress as much as possible, but we also need our application’s consumers to receive correct data at all times. Let’s assume that the application contacts guitar manufacturers’ websites once a day to refresh its inventory and we have scheduled this operation to complete at 16:00 every day. This suggests that the cached resource should be refreshed every day at four o’clock in the afternoon to reflect the latest list of available guitar models. One of the ways of achieving this in HTTP is by making use of the Expires header, whose semantics are understood by (hopefully) any caching aware HTTP component.

require "time"

get "/guitars" do
  headers "Expires" => Time.parse("16:00").httpdate
  guitars * ', '
end

Things aren’t always as straightforward. In many cases we cannot fully control the exact time or frequency a resource’s content changes. The example application also comes with an admin interface, allowing the guitar list administrators to manually enter new guitar models.

post "/guitars" do
  guitars << params["guitar"]
  redirect("/guitars")
end

It is clear that a means for arbitrary expiration of cached content needs to be available in order to maintain content freshness. With Varnish, this capability comes in two flavors, one of which involves the use of a PURGE HTTP call. The following configuration enables this functionality.

acl purge {
  "localhost";
}

sub vcl_recv {
  if (req.request == "PURGE") {
    if (!client.ip ~ purge) {
      error 405 "Not allowed.";
    }
    lookup;
  }
}

sub vcl_hit {
  if (req.request == "PURGE") {
    set obj.ttl = 0s;
    error 200 "Purged.";
  }
}

sub vcl_miss {
  if (req.request == "PURGE") {
    error 404 "Not in cache.";
  }
}

To natively make use of this in Ruby, we need to extend the Net::HTTP library to support the PURGE method.

require "net/http"
require "uri"

module Net
  class HTTP
    class Purge < HTTPRequest
      METHOD = "PURGE"
      REQUEST_HAS_BODY = false
      RESPONSE_HAS_BODY = false
    end

    def purge(path, initheader=nil)
      request(Purge.new(path, initheader))
    end
  end
end

def purge_cache(u)
  uri = URI.parse(u)
  query = "?#{uri.query}" if uri.query
  Net::HTTP.new(uri.host, uri.port).start {|h| h.purge("#{uri.path}#{query}")}
end

Now we can expire the cached /guitars resource every time the list is amended.

post "/guitars" do
  guitars << params["guitar"]
  purge_cache("http://localhost/guitars")
  redirect("/guitars")
end

Although this method is effective, there can be cases where the bidirectional coupling between the application and caching layers might be undesirable. With the fundamental functional pieces in place, however, it is not hard to implement a more elaborate strategy such as the one described in Cache Channels in order to reduce the application layer’s knowledge of the caching infrastructure.

Memory usage is often a headache with large dataset computations. While memory optimization is something to be sought after, tracking down memory leaks can become tedious and time consuming. We can decrease the chances of a heavy job running a system’s memory dry by coming up with a strategy for fragmenting the job into a number of shorter running processes. By doing so, any memory used by a worker process will be released the moment the process completes. Additionally, we can run job fragments in parallel, allow ourselves to harness the operating system’s multi-core capabilities and potentially distribute worker processes over a number of physical hosts and scale out when the need arises. Smaller processes also dictate more manageable chunks of code which are easier to maintain, optimize and test.

Let’s look at an example where a job involves fetching a large number of categorized products from various sources and processes them for use by our own application.

class Job
  def perform
    ADDRESSES.each do |address|
      category = load_category(address)
      category.products.each { |product| process(product) }
    end
  end

  def process(product)
    #some intensive computation
  end

  def load_category(address)
    #load an addressable category dataset
  end
end

Let’s assume that the ADDRESSES constant in the example is a list consisting of entries such as example.com/toys, example.com/phones, example.org/guitars, etc. The job fetches the addressable by category product datasets, iterates over the products and performs a long processing operation on each. Supposing that after every possible optimization the job takes three hours to complete, we can at best run the job eight times a day. What happens if the product categories are updated more often than eight times a day and a requirement in order for our application to be successful suggests that it needs to deal with fresh data all the time?

One natural split can involve creating a worker process for each address entry. We can do so by extracting the majority of the code from the Job class into a Worker class meant to run as a standalone process.

class Worker
  def self.process_category(address)
    category = load_category(address)
    category.products.each { |product| process(product) }
  end

  def self.process(product)
    #some intensive computation
  end

  def self.load_category(address)
    #load an addressable category dataset
  end
end

Worker.process_category(ARGV[0]) if ARGV.size == 1

Each worker will operate on a significantly smaller dataset and will complete much faster than the initial long running job. Any memory used by each worker will be immediately released the moment the process finishes execution.

After the latest change, Job can take on the role of instrumenting the worker processes. We start by only allowing an arbitrary maximum number of concurrent workers, three in this case.

require "thread"

class Job
  def initialize
    @worker_count, @mutex = 3, Mutex.new
  end

  def perform
    ADRESSESES.each do |address|
      sleep 0.1 until @worker_count > 0
      @worker_count -= 1
      Thread.new do
        system("ruby worker.rb #{address}")
        @mutex.synchronize {@worker_count += 1}
      end
    end
  end
end

At this point it is a good idea to run the job and monitor the time it takes for it to complete while also measuring system resource usage. This way we can determine the optimal number of concurrent worker processes based on the system’s specs. Once available resources have been exhausted and both Job and Worker have been sufficiently optimized, we can start thinking about running workers on separate physical nodes.