Since then we’ve done pretty well. On average we have about 4-8 hours of downtime per year. So today was especially nasty.

The first sign of trouble

Around 11:01 p.m. GMT I get a mail about an unexpected reboot. One of our other guys had caught an issue and was looking into it, but I hadn’t noticed the SMS he’d sent me saying such was coming up.

We hop on Skype and start going through what’s going on. Our web services process was alternating between crashing and hanging every few minutes. Usually that means some corrupted indexes in one of our databases. We get that once or twice a year. It’s an annoyance (and let me tell you about the fun of trying to track down a bug that happens twice a year), but we have a script that can rebuild the indexes, depending on the size of the database, in somewhere between 10 seconds and 10 minutes.

A Heisenbug

The trick is figuring out which database is causing the problem. Aside from our enterprise customers, which are usually on their own hardware, we run a bunch of our small and medium sized databases off of the same hardware. Our web services setup is heavily multithreaded, so there are usually some 40-ish threads that are writing to the logs in parallel; figuring out which request was the one triggering the problem can sometimes be a bit tricky.

But we’ve been there before too. We have a switch we can use to flip on function tracing which usuall makes it clear pretty quickly — or at least has in every instance up until now.

This time it didn’t. We kept watching our logs, trying to catch the offending item and guessing the wrong one. We have a script to pull down databases from the production to our development machines, where we check the indexes for corruption. For about an hour we kept missing.

We hadn’t been offline for a full hour in more than a year, so we were getting antsy at that point. We’d thought about this situation before too though — what if we couldn’t find the offending database?

A not-so-super super-tool

The last time we had a similar issue (which was sorted out in under 15 minutes), I wrote a small program to go through and check every single database on a system. It was, as we soon will learn, not so aptly called, “IntegrityChecker”. We’d never run it in production, but it had been tested on dev machines. Basically it would disable an account for a few seconds, try to get a file lock on its files, and then run the integrity checker that we usually use offline to test such things.

Except that for some reason it was given a file lock when the OS should have rejected it. So, now we have mmap’ed files that are being written from two process. Not good. In fact, very, very bad.

Unfortunately, we didn’t notice this immediately. In fact, things looked great — it identified the database with a corrupt index, we rebuilt it, waited 15 minutes to make sure everything was kosher, and headed off to snooze. It was about 4:00 a.m. local time.

This isn’t supposed to happen

We have two redundant monitoring systems that live in a separate network and check all of our servers once per minute. Both of them send out a barage of SMSes if the servers stop responding to requests. But most of the API requests that were coming in were actually being handled — probably 80% or so — and our monitoring systems retry a second time if there’s a failure just so that we don’t get notified every time there’s a minor network glitch somewhere. So we weren’t notified that our servers were still sickly.

The score

Now, if you’re keeping score, here are the points so far of this perfect storm:

• Relatively minor issue of database index corruption
• IntegrityChecker that’s actually corrupting databases (and not just their indexes)
• Monitoring not catching the problems

Digging deeper

9:30 a.m. rolls around and we’re up and trying to see if there’s anything we can piece together about the issues from the night before. Pretty soon one of the other guys that’s working with me on the problem notices that the server process actually kept crashing all night, albeit at a somewhat reduced frequency (every 10 minutes or so).

Since the IntegrityChecker had worked so well the night before (as we still thought), I gave it another go. It turned up this time six corrupt databases. We’d never seen that. I rebuilt them, and again, in perfect storm fashion, ran it again. Two more. Something was obviously very wrong.

A little while later the database process is back to restarting every minute. Panic ensues.

Our monitoring turns on itself

See, we not only have out-of-band monitoring, we also have monit running to restart things on our machines if something goes wrong.

We didn’t notice for about an hour that the errors in the monit log weren’t the usual ones. They were 404 — “not found” — errors. That was weird. It turned out the account that we use for status checking was corrupt. We fixed that and were back down to crashes every 10 minutes or so, but still had no idea what could possibly be causing the problems.

A couple of heavily caffinated hours of shooting in the dark passed. Eventually we started to suspect the IntegrityChecker. And we noticed that a few accounts (about a dozen — the ones that had the misfortune of writing while the IntegrityChecker was checking them) had been effectively nulled out. We started getting a sense of how widespread the damage was.

The low-down

There was a lot that came together to make this as bad as it was. To recap, we started off with a relatively minor issue, which was made into a major issue by a tool that was built specifically for dealing with the exact situation we were facing, and those large issues were not caught by our generally quite good monitoring, which eventually even became part of the problem itself.

The last resort: the backups

Fortunately we have backups, so we started going through accounts manually, offline this time, checking almost every database to try to find corruption. We found a few more.

At that point we started restoring the broken databases to their last known good state — about 17 hours before all of the mess started. We’ve done that now and things look pretty stable. It’s now been an 90 minutes since we had a crash or hang and all accounts are back online. In some cases we’re trying to merge data that came in later in the day where we can get to it, but that’s a bit spotty.

An apology

This is the first time in three and a half years of being in production that we lost data and we feel terrible about it. We’ll be doing some work on our internal systems to help us monitor, diagnose and fix these issues in the future. Thanks so much for your patience.

Edit: One additional note: if you’re one of our customers that uses our Shopify app and your data seems to be out of date, it’ll automatically be updated the next time that we pull your shop’s data in with our importer. There was no data loss for Shopify customers.

Per the HTTP spec, when a request uses HTTP BASIC authentication, as our API does, the client is supposed to first send a request without credentials. When it receives a 401 error, then it should retry with the credentials supplied and repeat that process for each and every request.

While that makes sense for interactive use from a browser, for use in a web services API, it makes much more sense to avoid the extra round-trip and send the credentials already in the first request, thus significantly speeding up request times.

You can get the updated versions of our bindings on Github.

So, it’s been a gazillion years since we posted updates here, but there have been a number of things shaking out of the woodwork.

First, we just did the biggest update to our Shopify app since we first launched it a couple years back.  It features a new Bootstrap-ified configuration interface and a whole bunch of new recommendations types.

New Ruby Bindings in Beta

Whilst working on our Shopify app, which itself is a Rails app, we got frustrated with the current state of our Ruby bindings.

You see, our Ruby bindings were written back in the dark days before we actually used Ruby at all inside of the company.  Truth be told, the original version was written by yours truly a couple hours after I started learning Ruby.

These days, along with Java and C++ (which our lower-level, performance critical bits are written in), we write quite a bit of Ruby as as such, our tastes have become more refined.

Here are a couple of quick examples:

Old:

require 'rubygems'
require 'directed_edge'
 
DE_USER = ENV['DIRECTEDEDGE_TEST_DB']
DE_PASS = ENV['DIRECTEDEDGE_TEST_PASS']
 
# Import some data
 
DE_FILE = 'temp.xml'
 
exporter = DirectedEdge::Exporter.new(DE_FILE)
 
(1..10).each do |i|
  item = DirectedEdge::Item.new(exporter.database, "item_#{i}")
  item.add_tag('item')
  item.link_to("item_#{rand(10) + 1}")
  exporter.export(item)
end
 
exporter.finish
 
database = DirectedEdge::Database.new(DE_USER, DE_PASS)
database.import(DE_FILE)
 
# Get recommendations
 
item = DirectedEdge::Item.new(database, 'item_1')
item.related([ 'item' ]).each { |i| puts i }
 
# Update item
 
item.add_tag('foo')
item.save

New:

require 'rubygems'
require 'directededge'
 
DE_USER = ENV['DIRECTEDEDGE_TEST_DB']
DE_PASS = ENV['DIRECTEDEDGE_TEST_PASS']
 
database = DirectedEdge::Database.new(DE_USER, DE_PASS)
 
DirectedEdge::UpdateJob.run(database, :replace) do |update|
  (1..10).each do |i|
    update.item("item_#{i}") do |item|
      item.tags.add('item')
      item.links.add("item_#{rand(10) + 1}")
    end
  end
end
 
# Get recommendations
 
item = DirectedEdge::Item.new(database, 'item_1')
item.related(:tags => :item).each { |i| puts i }
 
# Update item
 
item.tags.add('foo')
item.save

The changes are most obvious in the importer above, but there are a lot of little subtle improvements in the API’s semantics.

If you’re up for beta testing the new bindings, drop us a line.  The documentation is sorely lacking at present (ahem, meaning non-existent), but that’s the main thing that we know of that’s missing at the moment:  the new bindings are already in use by our Shopify app and the code is up on Github.

Web services updates on the way

In a similar vein, we’ve got some stuff that we added to the web services for the Shopify app that still have some rough corners that we want to smooth out before pushing them out to the unwashed masses, but we’ll soon have explicit API support for tracking product view histories and showing recommendations based on them.  It’s been possible to do such in a hackey way in the past, but the new stuff adds some special math-fu for time-clustered recommendations and storing high-traffic queues of updates.  More on that soon.

So, there are two things I want to talk about. The first is how our new beta hotness, acts_as_edgy, just made it super easy to add Directed Edge recommendations to Rails 2 apps. How easy? One line easy.

Recommendations the Easy Way

Let’s suppose for example that you’ve got a model called User and another called Product. And let’s also suppose that you support “likes” in your app, so you’ve got a model called Like too that just maps from a user ID to a product ID. Now, you want to show people personalized recommendations when they log in and related products when they’re viewing product pages. Nothing too exotic there.

So, here’s what you add to your app in that case:

class User < ActiveRecord::Base
  acts_as_edgy(:like, Like, Product)
  ...
end

The acts_as_edgy line is the whole shebang. That’s it. After that’s in there you’ve got rake tasks that let you do rake edgy:export and it’ll push your data over to our servers. You can also have it automatically update your data with our web service on saves to your model. Did I mention it’s easy?

Okay, okay. So, yeah, I skipped the part about where you actually do something with the recommendations. But that’s not rocket surgery either. Now that you’ve got that in your model you can do:

User.first.edgy_recommended # returns a list of recommended products based on likes
Product.first.edgy_related  # returns a list of related products based on likes

You can do much more complicated stuff too. Our Black Magic figures out the route between models that you provide a list of. One of the apps that we’ve been testing with is Spree. Let’s say that we want to do product recommendations based on sales. In Spree that means that we have to map a User to a Product, but there are a bunch of intermediaries: Orders belong to a User, LineItems belong to an Order, LineItems also belong to a Variant and Variants belong to a Product. Whew. What’s that look like in acts_as_edgy nomenclature?

class User < ActiveRecord::Base
  acts_as_edgy(:purchase, Order, LineItem, Variant, Product)
  # ...
end

You just list the models that it has to pass through on its way to its target and it figures out (or, well, tries to, this assumes you’ve used nice consistent column names, which you of course did, didn’t you?) how to get there.

And then, once again, we can just query for related and recommended stuff like it’s nobody’s business.

You also have access to the regular stuff that our API supports. So if you had both likes and purchases in the same application, that’s where that handle sitting up there right at the front of the acts_as_edgy line comes in handy. You can chose how you want to weight those, e.g.:

Product.first.edgy_related(:like_weight => 0.1, :purchase_weight => 0.9)

And there you’ve got recommendations based on some mix and match weights that seem appropriate to you.

I wanted to get the cool stuff out first, but there are naturally a couple of set up steps that have to happen beforehand:

THIS IS THE PART YOU ACTUALLY HAVE TO READ

1. Sign up for a Directed Edge account
2. Install the directed-edge and will_paginate gems
3. Install the plugin
4. Run rake edgy:configure to enter your login info

There’s more on the nuts and bolts of that over on the Github page, and just let us know if you get stuck. But here’s a real world example. Wait. I’ve just realized I lied to you. There are three things I want to tell you about.

We have a Spree plugin.

Yes, yes, we do. Hell, we even have a full tutorial on working with Directed Edge and Spree! It only works with Spree 0.11 at present since we haven’t ported this baby to Rails 3 yet. Mostly we needed something real to test this thing on, and Spree seemed like a nice way to test with data models that weren’t tied to our own assumptions.

You can get at the full plugin on Github.

And here are the guts — the example I mentioned above:

1. An initializer to add the line above to the User model
2. A helper to figure out if we’re doing personalized recommendations, related products or basket recommendations
3. A partial template to handle showing the results

That’s pretty lean for all of the glue and display code for adding recommendations (and instant updating) to a full-blown e-commerce thingereedoo.

The nerdy bits.

There are a few neat technical things that are happening behind the scenes to make all of this stuff easy from a user’s perspective.

SQL Generator

One of them is a fancy custom SQL generator that builds optimal queries for all of this stuff. Entire Rails models get exported with one query. The generated SQL can get hella ugly, but it offloads most of the dirty work to the database rather than having to do it all in Ruby code.

The above Spree example (the User to Product mapping) generates this SQL monstrosity:

SELECT users.id AS from_id, 'user' AS from_type, variants.product_id AS to_id,
CASE WHEN variants.product_id IS NOT NULL THEN 'product' END AS to_type,
CASE WHEN variants.product_id IS NOT NULL THEN 'purchase' END AS link_type FROM users
LEFT OUTER JOIN orders ON users.id = orders.user_id
LEFT OUTER JOIN line_items ON orders.id = line_items.order_id
LEFT OUTER JOIN variants ON line_items.variant_id = variants.id
WHERE users.id IS NOT NULL ORDER BY from_id;

We build up a thing we call a “connection” which is a number of “bridges”. A bridge is just a route between two models, including the foreign keys and so a connection is a full path from one model to another one via a chain of foreign keys. In the simple case this is detected based on the model’s name and built in ActiveRecord methods for reporting foreign keys, but you can also specify a bridge manually for foreign keys that are created that don’t match the typical nomenclature. That isn’t documented at the moment, but shout if that’s something that you need.

Triggers

Another neat thing is the automatic installation of model triggers. So when it’s building that connection mentioned before, our system knows which models trigger updates which need to be sent over to Directed Edge to keep your data in sync.

So if acts_as_edgy is set up to automagically send updates (a config parameter that can be called in the config blog that gets written when you call rake edgy:configure) then as soon as a model changes anywhere along that path, we get the goods. And this triggers a different code path that leads to our SQL generator just pulling out the stuff that needs to be updated in a single query.

Future: Asynchronous Web-service Calls

And since those updates are hitting a remote web service, it’s ideal if they’re not blocking execution. We make liberal use of a class we call Future (conceptually borrowed from QFuture from Qt) which executes a block of code in a background thread and only forces synchronization when its data is accessed. Here’s what it looks like:

class Future
  def initialize(postprocessor = nil, &finalize)
    if ENV['EDGY_SYNCHRONOUS']
      @data = postprocessor ? postprocessor.call(finalize.call) : finalize.call
      return
    end
    @postprocessor = postprocessor
    @future = Thread.new do
      begin
        finalize.call
      rescue => ex
        warn "Exception in background thread: #{ex}"
      end
    end
  end
 
  def method_missing(method, *args, &block)
    data.send(method, *args, &block)
  end
 
  def to_s
    data.to_s
  end
 
  private
 
  def data
    @data ||= @postprocessor ? @postprocessor.call(@future.value) : @future.value
end

Since we implement method_missing on the Future class to first block and then forward the call on, you can use it just like a normal value, e.g.:

foo = Future.new { sleep 1; 42 }
puts foo # prints 42

Altogether, while weighing in at a slim 382 lines of code the plugin is fairly light, but it’s pretty dense code with quite a bit of interesting stuff going on.

It’s still quite beta, so we expect there will be a handful of kinks to iron out and there are features we’d still like to add, but hopefully we’ll be doing that iteratively as you wonderful folks start building things with it and telling us what you need.

The answer turns out to be surprisingly easy. Who has an incentive to spam Google? People living from advertising. Who owns the largest online display ad network? Google.

Unlike with email, where there are heuristics at work to guess the intentions of the sender based on the content, Google has that data right in front of them.

So, here’s the heresy: the spamminess of a web site is inversely proportional to its ad click-through.

Think about it — in a typical internet search, a navigation path terminating at that page is the best result. If they click on an ad, it probably means you missed serving up the right page in the first place. As a corollary, the pages best optimized to pull you in via a search term and send you back out via a related ad are among the worst results.

So if you’re Google which value do you optimize for? More ad clicks or better search results? I’m a big enough Google fan that I believe that they’d mostly want to optimize for good search results since that’s what made them the company that they are. But what do you do if there’s an inverse correlation between the two? Bite the hand that feeds you?

Thinking about things this way in my opinion makes the issue even more interesting, because it seems to hint at something systemic — i.e. that there there might be something deeper problematic in financing search through display advertising.

Obviously this is a massive oversimplification of the problem of spam, but the paradox intrigued me.

So, the blog has been a bit silent of late, mostly because there’s been a lot of incremental stuff that’s been happening. We shout that stuff out over at Twitter and Facebook, but we’ll pull some of the last couple-o-months together here. But while I still have you attention:

We need Rails beta testers. If you’ve been thinking about adding recommendations to your Rails app, drop us a line now.

So, with that out of the way, a few of the things that have gone down in Directed Edge-land of late:

• We’ve been ramping up the features in our Shopify app based on things that we’ve been hearing from customers. The Shopify folks covered our new automagic bundling feature here.
• New version of our Ruby gem, 0.2.1 out, and a couple of features added to our Java bindings so that they approach parity with the other bindings.
• Wener Vogels, CTO of Amazon, has been spreading the good news about Directed Edge and we caught up with him when he was presenting about AWS in Berlin.
• Yours truly will be doing a big tour out through New York, Austin and San Francisco / Silicon Valley in December and January. Holler if you want to meet and talk about recommendations.

Werner Vogels presenting Directed Edge as "one of the building blocks of Cloud Computing"

While a lot of the stuff in the last few months has been focused on internal tools and run-of-the-mill growing pains stuff, we’re getting pretty close to a series of announcements about new stuff coming down the pipe. (For some reason those always seem to come in batches.) We’re hoping the first of those will be ready in the next week or so.

What astounded me was how many of my customers (not just visitors) clicked on recommendations. Indeed the ExpressRex referrer is responsible for a full third (33%) of my revenue in the last quarter and has a 36% conversion rate.

You can read the full post in his blog.

Eric Houtkooper of www.PupLife.com helps customers find and discover other products they may want. He has increased his customers’ average order size by recommending products as they browse the store and go through checkout. He suggests you do the same…

More there.

But, you see, I was none-too-keen on having another long-running Ruby process, not to mention an open port with production database data lumbering around on it, so I thought I’d let you guys in on a little hack we produced internally to let you get all of the fun of taps, but without the taps server.

Basically it starts up the taps server on the remote server, tunnels the transfer over SSH, then sends a ctrl-c to the server to kill it’s done. It’s pull-only very intentionally — I want to push from a development database to a production database about like I want a hole in my head.

#!/usr/bin/env ruby
 
require 'rubygems'
require 'active_support/secure_random'
require 'net/ssh'
 
SSH_USER = '[sshuser]'
SSH_HOST = '[dbhost]'
 
LOCAL_DB = 'mysql://[dbuser]:[dbpass]@localhost/[dbname]'
REMOTE_DB = 'mysql://[dbuser]:[dbpass]@localhost/[dbname]'
 
TAPS_USER = ActiveSupport::SecureRandom.hex(16)
TAPS_PASS = ActiveSupport::SecureRandom.hex(16)
 
URL = "http://#{TAPS_USER}:#{TAPS_PASS}@localhost:5000"
 
Net::SSH.start(SSH_HOST, SSH_USER, :compression => true) do |ssh|
  ssh.forward.local(5000, 'localhost', 5000)
 
  ready = false
 
  channel = ssh.open_channel do |c|
    c.request_pty
    c.on_data { |c, data| ready = true if data =~ /port=5000/ }
    c.exec("taps server #{REMOTE_DB} #{TAPS_USER} #{TAPS_PASS}")
  end
 
  finished = false
 
  Thread.new do
    sleep 0.1 until ready
    system "taps pull #{LOCAL_DB} #{URL}"
    finished = true
  end
 
  ssh.loop(0.1) do
    channel.send_data(Net::SSH::Connection::Term::VINTR) if finished
    !finished
  end
end

Substitute in the right values in the constants up at the top and you’ve got a nifty way to securely use taps without leaving a server running.

No huge shockers, but a few places where it’s interesting to compare perceptions to reality:

• PHP, for better or worse, still dominates web development.
• Python’s much closer to Ruby in the usage we see, despite making less noise.
• Scala, despite being the new hotness has very little actual uptake.
• Perl is all but out of the game (though still more than Scala)