So, there are two things I want to talk about. The first is how our new beta hotness, acts_as_edgy, just made it super easy to add Directed Edge recommendations to Rails 2 apps. How easy? One line easy.

Recommendations the Easy Way

Let’s suppose for example that you’ve got a model called User and another called Product. And let’s also suppose that you support “likes” in your app, so you’ve got a model called Like too that just maps from a user ID to a product ID. Now, you want to show people personalized recommendations when they log in and related products when they’re viewing product pages. Nothing too exotic there.

So, here’s what you add to your app in that case:

class User < ActiveRecord::Base
  acts_as_edgy(:like, Like, Product)
  ...
end

The acts_as_edgy line is the whole shebang. That’s it. After that’s in there you’ve got rake tasks that let you do rake edgy:export and it’ll push your data over to our servers. You can also have it automatically update your data with our web service on saves to your model. Did I mention it’s easy?

Okay, okay. So, yeah, I skipped the part about where you actually do something with the recommendations. But that’s not rocket surgery either. Now that you’ve got that in your model you can do:

User.first.edgy_recommended # returns a list of recommended products based on likes
Product.first.edgy_related  # returns a list of related products based on likes

You can do much more complicated stuff too. Our Black Magic figures out the route between models that you provide a list of. One of the apps that we’ve been testing with is Spree. Let’s say that we want to do product recommendations based on sales. In Spree that means that we have to map a User to a Product, but there are a bunch of intermediaries: Orders belong to a User, LineItems belong to an Order, LineItems also belong to a Variant and Variants belong to a Product. Whew. What’s that look like in acts_as_edgy nomenclature?

class User < ActiveRecord::Base
  acts_as_edgy(:purchase, Order, LineItem, Variant, Product)
  # ...
end

You just list the models that it has to pass through on its way to its target and it figures out (or, well, tries to, this assumes you’ve used nice consistent column names, which you of course did, didn’t you?) how to get there.

And then, once again, we can just query for related and recommended stuff like it’s nobody’s business.

You also have access to the regular stuff that our API supports. So if you had both likes and purchases in the same application, that’s where that handle sitting up there right at the front of the acts_as_edgy line comes in handy. You can chose how you want to weight those, e.g.:

Product.first.edgy_related(:like_weight => 0.1, :purchase_weight => 0.9)

And there you’ve got recommendations based on some mix and match weights that seem appropriate to you.

I wanted to get the cool stuff out first, but there are naturally a couple of set up steps that have to happen beforehand:

THIS IS THE PART YOU ACTUALLY HAVE TO READ

1. Sign up for a Directed Edge account
2. Install the directed-edge and will_paginate gems
3. Install the plugin
4. Run rake edgy:configure to enter your login info

There’s more on the nuts and bolts of that over on the Github page, and just let us know if you get stuck. But here’s a real world example. Wait. I’ve just realized I lied to you. There are three things I want to tell you about.

We have a Spree plugin.

Yes, yes, we do. Hell, we even have a full tutorial on working with Directed Edge and Spree! It only works with Spree 0.11 at present since we haven’t ported this baby to Rails 3 yet. Mostly we needed something real to test this thing on, and Spree seemed like a nice way to test with data models that weren’t tied to our own assumptions.

You can get at the full plugin on Github.

And here are the guts — the example I mentioned above:

1. An initializer to add the line above to the User model
2. A helper to figure out if we’re doing personalized recommendations, related products or basket recommendations
3. A partial template to handle showing the results

That’s pretty lean for all of the glue and display code for adding recommendations (and instant updating) to a full-blown e-commerce thingereedoo.

The nerdy bits.

There are a few neat technical things that are happening behind the scenes to make all of this stuff easy from a user’s perspective.

SQL Generator

One of them is a fancy custom SQL generator that builds optimal queries for all of this stuff. Entire Rails models get exported with one query. The generated SQL can get hella ugly, but it offloads most of the dirty work to the database rather than having to do it all in Ruby code.

The above Spree example (the User to Product mapping) generates this SQL monstrosity:

SELECT users.id AS from_id, 'user' AS from_type, variants.product_id AS to_id,
case when variants.product_id IS NOT NULL then 'product' end AS to_type,
case when variants.product_id IS NOT NULL then 'purchase' end AS link_type FROM users
LEFT OUTER JOIN orders ON users.id = orders.user_id
LEFT OUTER JOIN line_items ON orders.id = line_items.order_id
LEFT OUTER JOIN variants ON line_items.variant_id = variants.id
WHERE users.id IS NOT NULL ORDER BY from_id;

We build up a thing we call a “connection” which is a number of “bridges”. A bridge is just a route between two models, including the foreign keys and so a connection is a full path from one model to another one via a chain of foreign keys. In the simple case this is detected based on the model’s name and built in ActiveRecord methods for reporting foreign keys, but you can also specify a bridge manually for foreign keys that are created that don’t match the typical nomenclature. That isn’t documented at the moment, but shout if that’s something that you need.

Triggers

Another neat thing is the automatic installation of model triggers. So when it’s building that connection mentioned before, our system knows which models trigger updates which need to be sent over to Directed Edge to keep your data in sync.

So if acts_as_edgy is set up to automagically send updates (a config parameter that can be called in the config blog that gets written when you call rake edgy:configure) then as soon as a model changes anywhere along that path, we get the goods. And this triggers a different code path that leads to our SQL generator just pulling out the stuff that needs to be updated in a single query.

Future: Asynchronous Web-service Calls

And since those updates are hitting a remote web service, it’s ideal if they’re not blocking execution. We make liberal use of a class we call Future (conceptually borrowed from QFuture from Qt) which executes a block of code in a background thread and only forces synchronization when its data is accessed. Here’s what it looks like:

class Future
  def initialize(postprocessor = nil, &finalize)
    if ENV['EDGY_SYNCHRONOUS']
      @data = postprocessor ? postprocessor.call(finalize.call) : finalize.call
      return
    end
    @postprocessor = postprocessor
    @future = Thread.new do
      begin
        finalize.call
      rescue => ex
        warn "Exception in background thread: #{ex}"
      end
    end
  end
 
  def method_missing(method, *args, &block)
    data.send(method, *args, &block)
  end
 
  def to_s
    data.to_s
  end
 
  private
 
  def data
    @data ||= @postprocessor ? @postprocessor.call(@future.value) : @future.value
end

Since we implement method_missing on the Future class to first block and then forward the call on, you can use it just like a normal value, e.g.:

foo = Future.new { sleep 1; 42 }
puts foo # prints 42

Altogether, while weighing in at a slim 382 lines of code the plugin is fairly light, but it’s pretty dense code with quite a bit of interesting stuff going on.

It’s still quite beta, so we expect there will be a handful of kinks to iron out and there are features we’d still like to add, but hopefully we’ll be doing that iteratively as you wonderful folks start building things with it and telling us what you need.

The answer turns out to be surprisingly easy. Who has an incentive to spam Google? People living from advertising. Who owns the largest online display ad network? Google.

Unlike with email, where there are heuristics at work to guess the intentions of the sender based on the content, Google has that data right in front of them.

So, here’s the heresy: the spamminess of a web site is inversely proportional to its ad click-through.

Think about it — in a typical internet search, a navigation path terminating at that page is the best result. If they click on an ad, it probably means you missed serving up the right page in the first place. As a corollary, the pages best optimized to pull you in via a search term and send you back out via a related ad are among the worst results.

So if you’re Google which value do you optimize for? More ad clicks or better search results? I’m a big enough Google fan that I believe that they’d mostly want to optimize for good search results since that’s what made them the company that they are. But what do you do if there’s an inverse correlation between the two? Bite the hand that feeds you?

Thinking about things this way in my opinion makes the issue even more interesting, because it seems to hint at something systemic — i.e. that there there might be something deeper problematic in financing search through display advertising.

Obviously this is a massive oversimplification of the problem of spam, but the paradox intrigued me.

So, the blog has been a bit silent of late, mostly because there’s been a lot of incremental stuff that’s been happening. We shout that stuff out over at Twitter and Facebook, but we’ll pull some of the last couple-o-months together here. But while I still have you attention:

We need Rails beta testers. If you’ve been thinking about adding recommendations to your Rails app, drop us a line now.

So, with that out of the way, a few of the things that have gone down in Directed Edge-land of late:

• We’ve been ramping up the features in our Shopify app based on things that we’ve been hearing from customers. The Shopify folks covered our new automagic bundling feature here.
• New version of our Ruby gem, 0.2.1 out, and a couple of features added to our Java bindings so that they approach parity with the other bindings.
• Wener Vogels, CTO of Amazon, has been spreading the good news about Directed Edge and we caught up with him when he was presenting about AWS in Berlin.
• Yours truly will be doing a big tour out through New York, Austin and San Francisco / Silicon Valley in December and January. Holler if you want to meet and talk about recommendations.

Werner Vogels presenting Directed Edge as "one of the building blocks of Cloud Computing"

While a lot of the stuff in the last few months has been focused on internal tools and run-of-the-mill growing pains stuff, we’re getting pretty close to a series of announcements about new stuff coming down the pipe. (For some reason those always seem to come in batches.) We’re hoping the first of those will be ready in the next week or so.

What astounded me was how many of my customers (not just visitors) clicked on recommendations. Indeed the ExpressRex referrer is responsible for a full third (33%) of my revenue in the last quarter and has a 36% conversion rate.

You can read the full post in his blog.

Eric Houtkooper of www.PupLife.com helps customers find and discover other products they may want. He has increased his customers’ average order size by recommending products as they browse the store and go through checkout. He suggests you do the same…

More there.

But, you see, I was none-too-keen on having another long-running Ruby process, not to mention an open port with production database data lumbering around on it, so I thought I’d let you guys in on a little hack we produced internally to let you get all of the fun of taps, but without the taps server.

Basically it starts up the taps server on the remote server, tunnels the transfer over SSH, then sends a ctrl-c to the server to kill it’s done. It’s pull-only very intentionally — I want to push from a development database to a production database about like I want a hole in my head.

#!/usr/bin/env ruby
 
require 'rubygems'
require 'active_support/secure_random'
require 'net/ssh'
 
SSH_USER = '[sshuser]'
SSH_HOST = '[dbhost]'
 
LOCAL_DB = 'mysql://[dbuser]:[dbpass]@localhost/[dbname]'
REMOTE_DB = 'mysql://[dbuser]:[dbpass]@localhost/[dbname]'
 
TAPS_USER = ActiveSupport::SecureRandom.hex(16)
TAPS_PASS = ActiveSupport::SecureRandom.hex(16)
 
URL = "http://#{TAPS_USER}:#{TAPS_PASS}@localhost:5000"
 
Net::SSH.start(SSH_HOST, SSH_USER, :compression => true) do |ssh|
  ssh.forward.local(5000, 'localhost', 5000)
 
  ready = false
 
  channel = ssh.open_channel do |c|
    c.request_pty
    c.on_data { |c, data| ready = true if data =~ /port=5000/ }
    c.exec("taps server #{REMOTE_DB} #{TAPS_USER} #{TAPS_PASS}")
  end
 
  finished = false
 
  Thread.new do
    sleep 0.1 until ready
    system "taps pull #{LOCAL_DB} #{URL}"
    finished = true
  end
 
  ssh.loop(0.1) do
    channel.send_data(Net::SSH::Connection::Term::VINTR) if finished
    !finished
  end
end

Substitute in the right values in the constants up at the top and you’ve got a nifty way to securely use taps without leaving a server running.

No huge shockers, but a few places where it’s interesting to compare perceptions to reality:

• PHP, for better or worse, still dominates web development.
• Python’s much closer to Ruby in the usage we see, despite making less noise.
• Scala, despite being the new hotness has very little actual uptake.
• Perl is all but out of the game (though still more than Scala)

So I set off yesterday to rewrite our importer in C++ using Qt (and libcurl to grab the data). The result is a nice Qt-ified ActiveResource consumer that throws things into a collection of QVariant / hashes / lists.

Once I had that done I wondered, “What would the performance look like if I wrote a Ruby wrapper for the C++ implementation?” The answer was, fortunately, “Great!” meaning fortunately that the application logic can stay in Ruby with QActiveResource doing the heavy lifting.

It’s still relatively light on features — it just supports the find method, but the numbers speak for themselves:

The first column is the default pure-Ruby ActiveResource implementation, the second is with the same, but using the implemented-in-C Nokogiri backend. The third is just using my C++ backend directly and the fourth is with that bound to Ruby objects.

Methodology:

The data set is the set of products that we have in our test shop on Shopify. There are 17 of them, for a total of a 36k XML file. Each test iterates over reading that set 100 times. To remove other bottlenecks, I copied the file to a web server on localhost and queried that directly.

So, then that’s reading 1700 records for each backend over 100 request, with that the average times were:

• Ruby / ActiveResource / REXML: 34.60 seconds
• Ruby / ActiveResource / Nokogiri: 12.87 seconds
• C++ / QActiveResource: 0.79 seconds
• Ruby / QActiveResource: 1.01 seconds

All of the code is up on GitHub here, including the test data and the raw results.

API in Ruby and C++:

The Ruby API is very similar to a limited version of ActiveResource and supports things like this, for example:

resource = QAR::Resource.new(ENV['AR_BASE'], ENV['AR_RESOURCE'])
resource.find(:all, :params => { :page => 1 }).each { |record| puts record[:title] }

The C++ implementation also follows the same basic conventions, e.g.

#include 
#include 
 
int main()
{
    QActiveResource::Resource resource(QUrl(getenv("AR_BASE")), getenv("AR_RESOURCE"));
 
    foreach(QActiveResource::Record record, resource.find())
    {
        qDebug() << record["title"].toString();
    }
 
    return 0;
}

At present this naturally requires Qt and libcurl to build. I may consider building a version which doesn’t require Qt if there’s general interest in such (we use Qt in the C++ bits of our code anyway, so there was no extra cost to schlurping it in).

There are more examples in the Ruby directory on GitHub. Once it matures a wee bit we’ll get it packaged up as a Gem.

Edit:

The API’s already been going through some changes, now it can be easily used as a mixin, a la:

require 'rubygems'
require 'active_resource'
require 'QAR'
 
class Product < ActiveResource::Base
  extend QAR
  self.site = 'http://localhost/'
end

The above is not an exercise in geometric abstraction. It’s an attempt to do something seemingly simple, but to the best of my knowledge, impossible in CSS. The challenge is this:

Given a container, fill the container with n items of variable width, with the leftmost item flush with the left border, and the rightmost item flush with the right border, and an equal amount of space between each.

Unpossible! Try it.

This isn’t a contrived example. Being a pixel pedant, when we started in on our app for Shopify, where images may be any size within a bounding square, distributing them equidistantly seemed The Thing To Do. Here’s a shot from our test shop with the items properly distributed and the text properly centered under them:

So, let’s step through the options:

.container {
    border: 1px solid #333;
    width: 520px;
    margin-bottom: 1em;
 }
 
.container ul {
    padding: 0;
    margin: 0;
}
 
.container ul li {
    margin: 0;
    padding: 0;
    display: inline;
}

<div class="container">
  <ul>
    <li><img src="80.png" /></li>
    <li><img src="100.png" /></li>
    <li><img src="80.png" /></li>
    <li><img src="100.png" /></li>
    <li><img src="60.png" /></li>
  </ul>
</div>

That gives us the first box, things are messily clumped along the left side of the container. I quickly threw in the towel and went for a table. With a table it was easy to get things distributed evenly, but with ragged extreme borders:

table.container {
    width: 522px;
    border-spacing: 0;
}
 
.container td {
    text-align: center;
    padding: 0;
    margin: 0;
}

<table class="container">
  <tr>
    <td><img src="80.png" /></td>
    <td><img src="100.png" /></td>
    <td><img src="80.png" /></td>
    <td><img src="100.png" /></td>
    <td><img src="60.png" /></td>
  </tr>
</table>

But if we want to make the extremities flush, we’re stuck with a larger-than-mean gap between the first two and last two boxes:

.flush td.first {
    text-align: left;
}
 
.flush td.last {
    text-align: right;
}

<table class="container flush">
  <tr>
    <td class="first"><img src="80.png" /></td>
    <td><img src="100.png" /></td>
    <td><img src="80.png" /></td>
    <td><img src="100.png" /></td>
    <td class="last"><img src="60.png" /></td>
  </tr>
</table>

It turns out that the only way to do this seemingly simple task is to compute the margins ourselves and set them via Javascript. Here we take a table just like our second one, but with a “adjust” ID so that we can find it easily:

function adjust(id)
{
    var table = document.getElementById("adjust");
    var row = table.firstElementChild.firstElementChild;
    var cells = row.children;
    var totalImagesWidth = 0;
 
    for(var i = 0; i < cells.length; i++)
    {
        var image = cells[i].firstElementChild;
        totalImagesWidth += image.offsetWidth;
    }
 
    var extra = row.offsetWidth - totalImagesWidth;
 
    for(var i = 0; i < cells.length; i++)
    {
        var image = cells[i].firstElementChild;
        var padding = extra / (cells.length - 1);
        var buffer = Math.floor(padding * i) - Math.floor(padding * (i - 1));
 
        if(i == 0 || i == cells.length - 1 && cells.length >= 2)
        {
            cells[i].style.textAlign = ((i == 0) ? "left" : "right");
            cells[i].style.width = (image.offsetWidth + Math.floor(buffer / 2)) + "px";
        }
        else
        {
            cells[i].style.textAlign = "center";
            cells[i].style.width = (image.offsetWidth + buffer) + "px";
        }
    }
}
 
if(window.addEventListener)
{
    window.addEventListener("load", adjust, false);
}
else if(window.attachEvent)
{
    window.attachEvent("onload", adjust);
}

Now things get adjusted just as we’d want them. Let’s walk through the steps:

    var table = document.getElementById("adjust");
    var row = table.firstElementChild.firstElementChild;
    var cells = row.children;

Here we just get a handle to our building blocks, the table, row and cells.

    var totalImagesWidth = 0;
 
    for(var i = 0; i < cells.length; i++)
    {
        var image = cells[i].firstElementChild;
        totalImagesWidth += image.offsetWidth;
    }

Now we compute the total width of all of the images used in the table. We’ll use that to figure out how much left-over space we have to distribute:

    var extra = row.offsetWidth - totalImagesWidth;

From there we go in to assign this to each cell:

    for(var i = 0; i < cells.length; i++)
    {
        var image = cells[i].firstElementChild;
        var padding = extra / (cells.length - 1);
        var buffer = Math.floor(padding * i) - Math.floor(padding * (i - 1));

The padding is simply the extra space divided by the number of gaps that we have, but we compute the buffer for each iteration so that all of the left over pixels don’t accumulate at the end — i.e. if we had 7 items and the space between each should be in theory 10.333… pixels, that would leave us with 3 extra pixels stuck into the last gap. By doing the floor of each iteration and subtracting the previous value, we end up with appropriately distributed spare pixels.

        if(i == 0 || i == cells.length - 1 && cells.length >= 2)
        {
            cells[i].style.textAlign = ((i == 0) ? "left" : "right");
            cells[i].style.width = (image.offsetWidth + Math.floor(buffer / 2)) + "px";
        }
        else
        {
            cells[i].style.textAlign = "center";
            cells[i].style.width = (image.offsetWidth + buffer) + "px";
        }
    }

The second half of the loop just sets the cells at the extremities to be left / right aligned and gives each cell half of the buffer space for each of their internal gaps. The ones on the edges only need half of the buffer allocation since they don’t have a buffer on their outside edges.

The sad thing is this is actually a much simplified version of the real code. The real code also accounts for:

• Centering text under these images (Sounds easy right? No. You have to compute the margins manually also.)
• Dealing with stylesheets where images have margins, padding and borders and still making the right thing happen.
• Only one item being shown.
• Minimum sizes for images, centering them if they’re smaller (you don’t want to try to center text below a 5 pixel wide image that’s right aligned)

I can get into the mechanics of those in another post if necessary and have been considering stripping the real implementation out of our widget and throwing it up on Github where it can take a nice, clean JSON structure and turn it into a table of images, text, prices and links. Give a holler in the comments if you’d find such useful.

And with that, I’ll leave you with a gallery of what-this-looks-like-in-real-life shops, from the annals of our Shopify customers.

The thing is, POSIX provides a more useful mechanism for recovering from null pointer dereferences using signal handlers.

See, signals in UNIX can be caught and you can do things with them. Let’s say that we’ve got an event-driven application and we want to move on to the next event if the handler for one event caused a segfault. We can do that by catching the SIGSEGV and SIGBUS signals and returning to our event loop to process the next event.

Let’s look at some code:

#include <stdio.h>
#include <signal.h>
#include <setjmp.h>
#include <string.h>
 
static jmp_buf event_loop_jump;
 
static void process_event(int event_id)
{
    if(event_id % 10 == 0)
    {
        /* Boom. */
 
        int *i = 0;
        ++*i;
    }
    else
    {
        printf("Processed Event: %i\n", event_id);
    }
}
 
static void handler(int signum)
{
    printf("Caught %s\n", strsignal(signum));
    longjmp(event_loop_jump, 1);
}
 
static void run_event_loop(void)
{
    int event_id = 0;
 
    for(event_id = 1; event_id <= 20; event_id++)
    {
        if(setjmp(event_loop_jump))
        {
            printf("Couldn't process event: %i\n", event_id);
        }
        else
        {
            process_event(event_id);
        }
    }
}
 
int main(void)
{
    signal(SIGSEGV, handler);
    signal(SIGBUS, handler);
 
    run_event_loop();
 
    return 0;
}

And that give us:

Processed Event: 1
Processed Event: 2
Processed Event: 3
Processed Event: 4
Processed Event: 5
Processed Event: 6
Processed Event: 7
Processed Event: 8
Processed Event: 9
Caught Bus error
Couldn't process event: 10
Processed Event: 11
Processed Event: 12
Processed Event: 13
Processed Event: 14
Processed Event: 15
Processed Event: 16
Processed Event: 17
Processed Event: 18
Processed Event: 19
Caught Bus error
Couldn't process event: 20

What’s happening, Peter

So, let’s break that down a bit — we’ve created a function called handler that matches the signature required for signal handlers. man signal knows more, or check out the GNU documentation. In main we specify that we want to use that function to handle SIGSEGV and SIGBUS, the typical things you’ll be looking at when dereferencing null pointers. From there we call run_event_loop, whose function should be obvious.

Jump Around

run_event_loop just loops through 20 fake events, but the interesting bit is setjmp. That saves our stack (the variables in use, basically). During normal execution it just returns 0, so we fall through to our process_event call below, which just pretends to handle and event and prints out a message.

We’ve wired our event handler to blow up once every 10 times. When it does execution is suspended and we cut over to our signal handler. Our signal handler, in turn, prints a little message and then breaks out of the signal handler and back into the normal event handling flow with a longjmp call that corresponds to our setjmp from before.

So once we call that we’re back at our line with the setjmp call, only this time it returns the value that we passed to longjmp — 1, and since that evaluates to true, we end up printing a message saying that we weren’t able to process the event.

In Context

On the whole this seems far more useful than mapping address zero. However, There Be Dragons in these waters, and it no doubt opens up a can of worms in terms of what’s done with static variables in the functions you’re breaking out of and if they’re left in a secure state — in other words, this does open up another potential security vector.

I’ll also note that we don’t actually do this in any of our stuff at Directed Edge, going for the less exciting, but more predictable process watchers to handle restarts when our C++ engine falls foul of memory protection (which, blessedly, is a rare occurrence).

Update:

For posterity, and based on some of the comments here and on Hacker News, it seems apt to note that this method isn’t being seriously suggested for production code; the idea behind the post was more as a novelty and a brief exploration in signal handling.