RailsTips by John Nunemaker

Misleading Title About Queueing

2012-03-05T13:37:12-05:00

I don’t know about you, but I find it super frustrating when people blog about cool stuff at the beginning of a project, but then as it grows, they either don’t take the time to teach or they get all protective about what they are doing.

I am going to do my best to continue to discuss the strategies we are using to grow Gauges. I hope you find them useful and, by all means, if you have tips or ideas, hit me. Without any further ado…

March 1st of last year (2011), we launched Gauges. March 1st of this year (a few days ago), we finally switched to a queue for track requests. Yes, for one full year, we did all report generation in the track request.

1. In the Beginning

My goal for Gauges in the beginning was realtime. I wanted data to be so freakin’ up-to-date that it blew people’s minds. What I’ve realized over the past year of talking to customers is that sometimes Gauges is so realtime, it is too realtime.

That is definitely not to say that we are going to work on slowing Gauges down. More what it means, is that my priorities are shifting. As more and more websites use Gauges to track, availability moves more and more to the front of my mind.

Gut Detects Issue

A few weeks back, with much help from friends (Brandon Keepers, Jesse Newland, Kyle Banker, Eric Lindvall, and the top notch dudes at Fastest Forward), I started digging into some performance issues that were getting increasingly worse. They weren’t bad yet, but I had this gut feeling they would be soon.

My gut was right. Our disk io utilization on our primary database doubled from January to February, which was also our biggest growth in terms of number of track requests. If we doubled again from February to March, it was not going to be pretty.

Back to the Beginning

From the beginning, Gauges built all tracking reports on the fly in the track request. When a track came in, Gauges did a few queries and then performed around 5-10 updates.

When you are small, this is fine, but as growth happens, updating live during a track request can become an issue. I had no way to throttle traffic to the database. This meant if we had enough large sites start tracking at once, most likely our primary database would say uncle.

As you can guess, if your primary says uncle, you start losing tracking data. In my mind, priority number one is now to never lose tracking data. In order to do this effectively, I felt we were finally at the point where we needed to separate tracking from reporting.

2. Availability Takes Front Seat

My goal is for tracking to never be down. If, occasionally, you can’t get to your reporting data, or if, occasionally, your data gets behind for a few minutes, I will survive. If, however, tracking requests start getting tossed to the wayside while the primary screams for help, I will not.

I talked with some friends and found Kestrel to be very highly recommended, particularly by Eric (linked above). He swore by it, and was pushing it harder than we needed to, so I decided to give it a try.

A few hours later, my lacking JVM skills (Kestrel is Scala) were bearing their head big time. I still had not figured out how to build or run the darn thing. I posted to the mailing list, where someone quickly pointed out that Kestrel defaults to /var for logging, data, etc. and, unfortunately, spits out no error on startup about lacking permissions on OSX. One sudo !! later and I was in business.

3. Kestrel

Before I get too far a long with this fairy tail, let’s talk about Kestrel — what is it and why did I pick it?

Kestrel is a simple, distributed message queue, based on Blaine Cook’s starling. Here are a few great paragraphs from the readme:

Each server handles a set of reliable, ordered message queues. When you put a cluster of these servers together, with no cross communication, and pick a server at random whenever you do a set or get, you end up with a reliable, loosely ordered message queue.

In many situations, loose ordering is sufficient. Dropping the requirement on cross communication makes it horizontally scale to infinity and beyond: no multicast, no clustering, no “elections”, no coordination at all. No talking! Shhh!

It features the memcached protocol, is durable (journaled), has fanout queues, item expiration, and even supports transactional reads.

My favorite thing about Kestrel? It is simple, soooo simple. Sound too good to be true? Probably is, but the honeymoon has been great so far.

Now that we’ve covered what Kestrel is and that it is amazing, let’s talk about how I rolled it out.

4. Architecture

Here is the general idea. The app writes track requests to the tracking service. Workers process off those track requests and generate the reports in the primary database.

After the primary database writes, we send the information through a pusher proxy process, which sends it off to pusher.com, the service that provides all the live web socket goodness that is in Gauges. Below is a helpful sketch:

That probably all makes sense, but remember that we weren’t starting from scratch. We already had servers setup that were tracking requests and I needed to ensure that was uninterrupted.

5. Rollout

Brandon and I have been on a tiny classes and services kick of late. What I am about to say may sound heretical, but we’ve felt that we need a few more layers in our apps. We’ve started using Gauges as a test bed for this stuff, while also spending a lot of time reading about clean code and design patterns.

We decided to create a tiny standardization around exposing services and choosing which one gets used in which environment. Brandon took the standardization and moved it into a gem where we could start trying stuff and share it with others. It isn’t much now, but we haven’t needed it to be.

Declaring Services

We created a Registry class for Gauges, which defined the various pieces we would use for Kestrel. It looked something like this:

class Registry
  include Morphine

  register :track_service do
    KestrelTrackService.new(kestrel_client, track_config['queue'])
  end

  register :track_processor do
    KestrelTrackProcessor.new(blocking_kestrel_client, track_config['queue'])
  end
end

We then store an instance of this register in Gauges.app. We probably should have named it Gauges.registry, but we can worry about that later.

At this point, what we did probably seems pointless. The kestrel track service and processor look something like this:

class KestrelTrackService
  def initialize(client, queue)
    @client = client
    @queue  = queue
  end

  def record(attrs)
    @client.set(@queue, MessagePack.pack(attrs))
  end
end

class KestrelTrackProcessor
  def initialize(client, queue)
    @client = client
    @queue = queue
  end

  def run
    loop { process }
  end

  def process
    record @client.get(@queue)
  end

  def record(data)
    Hit.record(MessagePack.unpack(data))
  end
end

The processor uses a blocking kestrel client, which is just a decorator of the vanilla kestrel client. As you can see, all we are doing is wrapping the kestrel-client and making it send the data to the right place.

Using Services

We then used the track_service in our TrackApp like this:

class TrackApp < Sinatra::Base
  get '/track.gif' do
    # stuff
    Gauges.app.track_service.record(track_attrs)
    # more stuff
  end
end

Then, in our track_processor.rb process, we started the processor like so:

Gauges.app.track_processor.run

Like any good programmer, I knew that we couldn’t just push this to production and cross our fingers. Instead, I wanted to roll it out to work like normal, but also push track requests to kestrel. This would allow me to see kestrel receiving jobs.

On top of that, I also wanted to deploy the track processors to pop track requests off. At this point, I didn’t want them to actually process those track requests and write to the database, I just wanted to make sure the whole system was wired up correctly and stuff was flowing through it.

Another important piece was seeing how many track request we could store in memory with Kestrel, based on our configuration, and how it performed when it used up all the allocated memory and started going to disk.

Service Magic

The extra layer around tracking and processing proved to be super helpful. Note that the above examples used the new Kestrel system, but that I wanted to push this out and go through a verification process first. First, to do the verification process, we created a real-time track service:

class RealtimeTrackService
  def record(attrs)
    Hit.record(attrs)
  end
end

This would allow us to change the track_service in the registry to perform as it currently was in production. Now, we have two services that know how to record track requests in a particular way. What I needed next was to use both of these services at the same time so I created a multi track service:

class MultiTrackService
  include Enumerable

  def initialize(*services)
    @services = services
  end

  def record(attrs)
    each { |service| service.record(attrs) }
  end

  def each
    @services.each do |service|
      yield service
    end
  end
end

This multi track services allowed me to record to both services for a single track request. The updated registry looked something like this:

class Registry
  include Morphine

  register :track_service do
    which = track_config.fetch(:service, :realtime)
    send("#{which}_track_service")
  end

  register :multi_track_service do
    MultiTrackService.new(realtime_track_service, kestrel_track_service)
  end

  register :realtime_track_service do
    RealtimeTrackService.new
  end

  register :kestrel_track_service do
    KestrelTrackService.new(kestrel_client, track_config['queue'])
  end
end

Note that now, track_service selects which service to use based on the config. All I had to do was update the config to use “multi” as the track service and we were performing realtime track requests while queueing them in Kestrel at the same time.

The only thing left was to beef up failure around the Kestrel service so that it was limited in how it could affect production. For this, I chose to catch failures, log them, and move on as if they didn’t happen.

class KestrelTrackService

  def initialize(client, queue, options={})
    @client = client
    @queue  = queue
    @logger = options.fetch(:logger, Logger.new(STDOUT))
  end

  def record(attrs)
    begin
      @client.set(@queue, MessagePack.pack(attrs))
    rescue => e
      log_failure(attrs, e)
      :error
    end
  end

  private

  def log_failure(attrs, exception)
    @logger.info "attrs: #{attrs.inspect}  exception: #{exception.inspect}"
  end
end

I also had a lot of instrumentation in the various track services, so that I could verify counts at a later point. These verifications counts would prove whether or not things were working. I left that out as it doesn’t help the article, but you definitely want to verify things when you roll them out.

Now that the track service was ready to go, I needed a way to ensure that messages would flow through the track processors without actually modifying data. I used a similar technique as above. I created a new processor, aptly titled NoopTrackProcessor.

class NoopTrackProcessor < KestrelTrackProcessor
  def record(data)
    # don't actually record
    # instead  just run verification
  end
end

The noop track processor just inherits from the kestrel track processor and overrides the record method to run verification instead of generating reports.

Next, I adjusted the registry to allow flipping the processor that is used based on the config.

class Registry
  include Morphine

  register :track_processor do
    which = track_config.fetch(:processor, :noop)
    send("#{which}_track_processor")
  end

  register :kestrel_track_processor do
    KestrelTrackProcessor.new(blocking_kestrel_client, track_config['queue'])
  end

  register :noop_track_processor do
    NoopTrackProcessor.new(blocking_kestrel_client, track_config['queue'])
  end
end

With those changes in place, I could now set the track service to multi, the track processor to noop, and I was good to deploy. So I did. And it was wonderful.

6. Verification

For the first few hours, I ran the multi track service and turned off the track processors. This created the effect of queueing and never dequeueing. The point was to see how many messages kestrel could hold in memory and how it performed once messages started going to disk.

I used scout realtime to watch things during the evening while enjoying some of my favorite TV shows. A few hours later and almost 530k track requests later, Kestrel hit disk and hummed along like nothing happened.

Now that I had a better handle of Kestrel, I turned the track processors back on. Within a few minutes they had popped all the messages off. Remember, at this point, I was still just noop’ing in the track processors. All reports were still being built in the track request.

I let the multi track service and noop track processors run through the night and by morning, when I checked my graphs, I felt pretty confident. I removed the error suppression from the kestrel service and flipped both track service and track processor to kestrel in the config.

One more deploy and we were queueing all track requests in Kestrel and popping them off in the track processors after which, the reports were updated in the primary database. This meant our track request now performed a single Kestrel set, instead of several queries and updates. As you would expect, response times dropped like a rock.

It is pretty obvious when Kestrel was rolled out as the graph went perfectly flat and dropped to ~4ms response times. BOOM.

You might say, yeah, your track requests are now fast, but your track processors are doing the same work that the app was doing before. You would be correct. Sometimes growing is just about moving slowness into a more manageable place, until you have time to fix it.

This change did not just move slowness to a different place though. It separated tracking and reporting. We can now turn the track processors off, make adjustments to the database, turn them back on, and instantly, they start working through the back log of track requests queued up while the database was down. No tracking data lost.

I only showed you a handful of things that we instrumented to verify things were working. Another key metric for us, since we aim to be as close to realtime as possible, is the amount of time that it takes to go from queued to processing.

Based on the numbers, it takes us around 500ms right now. I believe as long as we keep that number under a second, most people will have no clue that we aren’t doing everything live.

7. Conclusion

By no means are we where I want us to be availability-wise, but at least we are one more step in the right direction. Hopefully this article gives you a better idea how to roll things out into production safely. Layers are good. Whether you are using Rails, Sinatra, or some other language entirely, layer services so that you can easily change them.

Also, we are now a few days in and Kestrel is a beast. Much thanks to Robey for writing it and Twitter for open sourcing it!

More Tiny Classes

2012-02-06T07:00:10-05:00

My last post, Keep ’Em Separated, made me realize I should start sharing more about what we are doing to make Gauges maintainable. This post is another in the same vein.

Gauges allows you to share a gauge with someone else by email. That email does not have to exist prior to your adding it, because nothing is more annoying that wanting to share something with a friend or co-worker, but first having to get them to sign up for the service.

If the email address is found, we add the user to the gauge and notify them that they have been added.

If the email address is not found, we create an invite and then send an email to notify them they should sign up, so they can see the data.

The Problem: McUggo Route

The aforementioned sharing logic isn’t difficult, but it was just enough that our share route was getting uggo. It started off looking something like this:

post('/gauges/:id/shares') do
  gauge = Gauge.get(params['id'])

  if user = User.first_by_email(params[:email])
    Stats.increment('shares.existing')
    gauge.add_user(user)
    ShareWithExistingUserMailer.new(gauge, user).deliver
    {:share => SharePresenter.new(gauge, user)}.to_json
  else
    invite = gauge.invite(params['email'])
    Stats.increment('shares.new')
    ShareWithNewUserMailer.new(gauge, invite).deliver
    {:share => SharePresenter.new(gauge, invite)}.to_json
  end
end

Let’s be honest. We’ve all seen Rails controller actions and Sinatra routes that are fantastically worse, but this was really burning my eyes, so I charged our programming butler to refactor it.

The Solution: Move Logic to Separate Class

We talked some ideas through, and once he had finished, the route looked more like this:

post('/gauges/:id/shares') do
  gauge    = Gauge.get(params['id'])
  sharer   = GaugeSharer.new(gauge, params['email'])
  receiver = sharer.perform
  {:share => SharePresenter.new(gauge, receiver)}.to_json
end

Perfect? Who cares. Waaaaaaaaay better? Yes. The concern of a user existing or not is moved away to a place where the route could care less.

Also, the bonus is that sharing a gauge can now be used without invoking a route.

So what does GaugeSharer look like?

class GaugeSharer
  def initialize(gauge, email)
    @gauge = gauge
    @email = email
  end

  def user
    @user ||= … # user from database
  end

  def existing?
    user.present?
  end

  def perform
    if existing?
      share_with_existing_user
    else
      share_with_invitee
    end
  end

  def share_with_existing_user
    # add user to gauge
    ShareWithExistingUserMailer.new(@gauge, user).deliver
    user
  end

  def share_with_invitee
    invite = ... # invite to db
    ShareWithNewUserMailer.new(@gauge, invite).deliver
    invite
  end
end

Now, instead of having several higher-level tests to check each piece of logic, we can just ensure that GaugeSharer is invoked correctly in the route test and then test the crap out of GaugeSharer with unit tests. We can also use GaugeSharer anywhere else in the application that we want to.

This isn’t a dramatic change in code, but it has a dramatic effect on the coder. Moving all these bits into separate classes and tiny methods improves ease of testing and, probably more importantly, ease of grokking for another developer, including yourself at a later point in time.

Keep 'Em Separated

2012-02-04T16:14:59-05:00

Note: If you end up enjoying this post, you should do two things: sign up for Pusher and then subscribe to destroy all software screencasts. I’m not telling you do this because I get referrals, I just really like both services.

For those that do not know, Gauges currently uses Pusher.com for flinging around all the traffic live.

Every track request to Gauges sends a request to Pusher. We do this using EventMachine in a thread, as I have previously written about.

The Problem

The downside of this, is when you get to the point we were (thousands of a requests a minute), there are so many pusher notifications to send (thousands of a minute) that the EM thread starts stealing a lot of time from the main request thread. You end up with random slow requests that have one to five seconds of “uninstrumented” time. Definitely not a happy scaler does this make.

In the past, we had talked about keeping track of which gauges were actually being watched and only sending a notification for those, but never actually did anything about it.

The Solution

Recently, Pusher added web hooks on channel occupy and channel vacate. This, combined with a growing number of slow requests, was just the motivation I needed to come up with a solution.

We (@bkeepers and I) started by mapping a simple route to a class.

class PusherApp < BaseApp
  post '/pusher/ping' do
    webhook = Pusher::WebHook.new(request)
    if webhook.valid?
      PusherPing.receive(webhook)
      'ok'
    else
      status 401
      'invalid'
    end
  end
end

Using a simple class method like this moves all logic out of the route and into a place that is easier to test. The receive method iterates the events and runs each ping individually.

class PusherPing
  def self.receive(webhook)
    webhook.events.each do |event|
      new(event, webhook.time).run
    end
  end
end

At first, we had something like this for each PusherPing instance.

class PusherPing
  def initialize(event, time)
    @event         = event || {}
    @time          = time
    @event_name    = @event['name']
    @event_channel = @event['channel']
  end

  def run
    case @event_name
    when 'channel_occupied'
      occupied
    when 'channel_vacated'
      vacated
    end
  end

  def occupied
    update(@time)
  end

  def vacated
    update(nil)
  end

  def update(value)
    # update the gauge in the
    # db with the value
  end
end

We pushed out the change so we could start marking gauges as occupied. We then forced a browser refresh, which effectively vacated and re-occupied all gauges people were watching.

Once we new the occupied state of each gauge was correct, we added the code to only send the request to pusher on track if a gauge was occupied.

Deploy. Celebrate. Booyeah.

The New Problem

Then, less than a day later, we realized that pusher doesn’t guarantee the order of events. Imagine someone vacating and then occupying a gauge, but receiving the occupy first and then the vacate.

This situation would mean that live tracking would never turn on for the gauge. Indeed, it started happening to a few people, who quickly let us know.

The New Solution

We figured it was better to send a few extra notifications than never send any, so we decided to “occupy” gauges on our own when people loaded up the Gauges dashboard.

We started in and quickly realized the error of our ways in the pusher ping. Having the database calls directly tied to the PusherPing class meant that we had two options:

Use the PusherPing class to occupy a gauge when the dashboard loads, which just felt wrong.
Re-write it to separate the occupying and vacating of a gauge from the PusherPing class.

Since we are good little developers, we went with 2. We created a GaugeOccupier class that looks like this:

class GaugeOccupier
  attr_reader :ids

  def initialize(*ids)
    @ids = ids.flatten.compact.uniq
  end

  def occupy(time=Time.now.utc)
    update(time)
  end

  def vacate
    update(nil)
  end

private

  def update(value)
    return if @ids.blank?
    # do the db updates
  end
end

We tested that class on its own quite quickly and refactored the PusherPing to use it.

class PusherPing
  def run
    case @event_name
    when 'channel_occupied'
      GaugeOccupier.new(gauge_id).occupy(@time)
    when 'channel_vacated'
      GaugeOccupier.new(gauge_id).vacate
    end
  end
end

Boom. PusherPing now worked the same and we had a way to “occupy” gauges separate from the PusherPing. We added the occupy logic to the correct point in our app like so:

ids = gauges.map { |gauge| gauge.id }
GaugeOccupier.new(ids).occupy

At this point, we were now “occupied” more than “vacated”, which is good. However, you may have noticed, that we still had the issue where someone loads the dashboard, we occupy the gauge, but then receive a delayed, or what I will now refer to as “stale”, hook.

To fix the stale hook issue, we simply added a bit of logic to the PusherPing class to detect staleness and simple ignore the ping if it is stale.

class PusherPing
  def run
    return if stale?
    # do occupy/vacate
  end

  def stale?
    return false if gauge.occupied_at.blank?
    gauge.occupied_at > @time
  end
end

Closing Thoughts

This is by no means a perfect solution. There are still other holes. For example, a gauge could be occupied by us after we receive a vacate hook from pusher and stay in an “occupied” state, sending notifications that no one is looking for.

To fix that issue, we can add a cleanup cron or something that occasionally gets all occupied channels from pusher and vacates gauges that are not in the list.

We decided it wasn’t worth the time. We pushed out the occupy fix and are now reaping the benefits of sending about 1/6th of the pusher requests we were before. This means our EventMachine thread is doing less work, which gives our main thread more time to process requests.

You might think us crazy for sending hundreds of http requests in a thread that shares time with the main request thread, but it is actually working quite well.

We know that some day we will have to move this to a queue and an external process that processes the queue, but that day is not today. Instead, we can focus on the next round of features that will blow people’s socks off.

What a Year

2012-01-01T14:13:58-05:00

The last 12 months have been nuts. My health and professional/personal life were completely at odds.

Between January and August, I had three hernia surgeries. As if that wasn’t enough for one year, the last few months of the year I’ve been plagued by a few other ailments (which are still giving me a hard time). Definitely a rough stretch. I will never take health for granted again and really look forward to getting back to “normal”.

Quite the contrary to my health, Ordered List grew from 2 to 5 people, helped Zynga launch Words with Friends on Facebook, launched Gauges and Speaker Deck while improving Harmony, and, finally, was acquired by the only other company in the world I wanted to be a part of, GitHub.

Here is to a healthy 2012.

Acquired

2011-12-05T13:03:56-05:00

Several times over the past few years, I have stated that GitHub is probably the only other place I could see myself working. Today, it is official. All of Ordered List has joined GitHub.

Maybe someday I’ll write about what Ordered List has meant to me, but today I am going to fully enjoy the present, instead of rambling about the past. I have no doubt great things will come of this.

You can read more at GitHub and Ordered List.

Creating an API

2011-12-01T12:43:33-05:00

A few weeks back, we publicly released the Gauges API. Despite building Gauges from the ground up as an API, it was a lot of work. You really have to cross your t’s and dot your i’s when releasing an API.

1. Document as You Build

We made the mistake of documenting after most of the build was done. The problem is documenting sucks. Leaving that pain until the end, when you are excited to release it, makes doing the work twice as hard. Thankfully, we have a closer on our team who powered through it.

2. Be Consistent

As we documented the API, we noticed a lot of inconsistencies. For example, in some places we return a hash and in others we returned an array. Upon realizing these issues, we started making some rules.

To solve the array/hash issue, we elected that every response should return a hash. This is the most flexible solution going forward. It allows us to inject new keys without having to convert the response or release a whole new version of the API.

Changing from an array to a hash meant that we needed to namespace the array with a key. We then noticed that some places were name-spaced and others weren’t. Again, we decided on a rule. In this case, all top level objects should be name-spaced, but objects referenced from a top level object or a collection of several objects did not require name-spacing.

{users:[{user:{...}}, {user:{...}}]} // nope
{users:[{...}, {...}]} // yep
{username: 'jnunemaker'} // nope
{user: {username:'jnunemaker'}} // yep

You get the idea. Consistency is important. It is not so much how you do it as that you always do it the same.

3. Provide the URLs

Most of my initial open source work was wrapping APIs. The one thing that always annoyed me was having to generate urls. Each resource should know the URLs that matter. For example, a user resource in Gauges has a few URLs that can be called to get various data:

{
  "user": {
    "name": "John Doe",
    "urls": {
      "self": "https://secure.gaug.es/me",
      "gauges": "https://secure.gaug.es/gauges",
      "clients": "https://secure.gaug.es/clients"
    },
    "id": "4e206261e5947c1d38000001",
    "last_name": "Doe",
    "email": "john@doe.com",
    "first_name": "John"
  }
}

The previous JSON is the response of the resource /me. /me returns data about the authenticated user and the URLs to update itself (self), get all gauges (/gauges), and get all API clients (/clients). Let’s say next you request /gauges. Each gauge returned has the URLs to get more data about the gauge.

{
  "gauges": [
    {
      // various attributes
      "urls": {
        "self":"https://secure.gaug.es/gauges/4ea97a8be5947ccda1000001",
        "referrers":"https://secure.gaug.es/gauges/4ea97a8be5947ccda1000001/referrers",
        "technology":"https://secure.gaug.es/gauges/4ea97a8be5947ccda1000001/technology",
        // ... etc
      },
    }
  ]
}

We thought this would prove helpful. We’ll see in the long run if it turns out to work well.

4. Present the Data

Finally, never ever use to_json and friends from a controller or sinatra get/post/put block. At least as a bare minimum rule, the second you start calling to_json with :methods, :except, :only, or any of the other options, you probably want to move it to a separate class.

For Gauges, we call these classes presenters. For example, here is a simplified version of the UserPresenter.

class UserPresenter
  def initialize(user)
    @user = user
  end

  def as_json(*)
    {
      'id'          => @user.id,
      'email'       => @user.email,
      'name'        => @user.name,
      'first_name'  => @user.first_name,
      'last_name'   => @user.last_name,
      'urls'        => {
        'self'    => "#{Gauges.api_url}/me",
        'gauges'  => "#{Gauges.api_url}/gauges",
        'clients' => "#{Gauges.api_url}/clients",
      }
    }
  end
end

Nothing fancy. Just a simple ruby class that sits in app/presenters. Here is an example of the the /me route looks like in our Sinatra app.

get('/me') do
  content_type(:json)
  sign_in_required
  {:user => UserPresenter.new(current_user)}.to_json
end

This simple presentation layer makes it really easy to test the responses in detail using unit tests and then just have a single integration test that makes sure overall things look good. I’ve found this tiny layer a breath of fresh air.

I am sure that nothing above was shocking or awe-inspiring, but I hope that it saves you some time on your next public API.

Stupid Simple Debugging

2011-09-01T11:49:52-04:00

There are all kinds of fancy debugging tools out there, but personally, I get the most mileage out of good old puts statements.

When I started with Ruby, several years ago, I used puts like this to debug:

puts account.inspect

The problem with this is two fold. First, if you have a few puts statements, you don’t know which one is actually which object. This always led me to doing something like this:

puts "account: #{account.inspect}"

Second, depending on whether you are just in Ruby or running an app through a web server, puts is sometimes swallowed. This led me to often times do something like this when using Rails:

Rails.logger.debug "account: #{account.inspect}"

Now, not only do I have to think about which method to use to debug something, I also have to think about where the output will be sent so I can watch for it.

Enter Log Buddy

Then, one fateful afternoon, I stumbled across log buddy (gem install log_buddy). In every project, whether it be a library, Rails app, or Sinatra app, one of the first gems I throw in my Gemfile is log_buddy.

Once you have the gem installed, you can tell log buddy where your log file is and whether or not to actually log like so:

LogBuddy.init({
  :logger   => Gauges.logger,
  :disabled => Gauges.production?,
})

Simply provide log buddy with a logger and tell it if you want it to be silenced in a given situation or environment and you get some nice bang for your buck.

One Method, One Character

First, log buddy adds a nice and short method named d. d is 4X shorter than puts, so right off the bat you get some productivity gains. The d method takes any argument and calls inspect on it. Short and sweet.

d account # will puts account.inspect
d 'Some message' # will puts "Some message"

The cool part is that on top of printing the inspected object to stdout, it also logs it to the logger provided in in LogBuddy.init. No more thinking about which method to use or where output will be. One method, output is sent to multiple places.

This is nice, but it won’t win you any new friends. Where log buddy gets really cool, is when you pass it a block.

d { account } # puts and logs account = <Account ...>

Again, one method, output to stdout and your log file, but when you use a block, it does magic to print out the variable name and that inspected value. You can also pass in several objects, separating them with semi-colons.

d { account; account.creator; current_user }

This gives you each variable on its own line with the name and inspected value. Nothing fancy, but log buddy has saved me a lot of time over the past year. I figured it was time I send it some love.

Counters Everywhere, Part 2

2011-08-01T10:44:26-04:00

In Counters Everywhere, I talked about how to handle counting lots of things using single documents in Mongo. In this post, I am going to cover the flip side—counting things when there are an unlimited number of variations.

Force the Data into a Document Using Ranges

Recently, we added window and browser dimensions to Gaug.es. Screen width has far fewer variations as there are only so many screens out there. However, browser width and height can vary wildly, as everyone out there has there browser open just a wee bit different.

I knew that storing all widths or heights in a single document wouldn’t work because the number of variations was too high. That said, we pride ourselves at Ordered List on thinking through things so our users don’t have to.

Does anyone really care if someone visited their site with a browser open exactly 746 pixels wide? No. Instead, what matters is what ranges of widths are visiting their site. Knowing this, we plotted out what we considered were the most important ranges of widths (320, 480, 800, 1024, 1280, 1440, 1600, > 2000) and heights (480, 600, 768, 900, > 1024).

Instead of storing each exact pixel width, we figure out which range the width is in and do an increment on that. This allows us to receive a lot of varying widths and heights, but keep them all in one single document.

{
  "sx" => {
    "320"  => 237,
    "480"  => 367,
    "800"  => 258,
    "1024" => 2273,
    "1280" => 10885,
    "1440" => 6144
    "1600" => 13607,
    "2000" => 2154,
  },
  "bx" => {
    "320"  => 121,
    "480"  => 390,
    "800"  => 3424,
    "1024" => 9790,
    "1280" => 11125,
    "1440" => 3989
    "1600" => 6757,
    "2000" => 301,
  },
  "by" => {
    "480"  => 3940,
    "600"  => 13496,
    "768"  => 8184,
    "900"  => 6718,
    "1024" => 3516
  },
}

I would call this first method for storing a large number of variations cheating, but in this instance, cheating works great.

When You Can’t Cheat

Where the single document model falls down is when you do not know the number of variations, or at least know that it could grow past 500-1000. Seeing how efficient the single document model was, I tried to store content and referrers in the same way, initially.

I created one document per day per site and it had a key for each unique piece of content or referring url with a value that was an incrementing number of how many times it was hit.

It worked great. Insanely small storage and no secondary indexes were needed, so really light on RAM. Then, a few larger sites signed up that were getting 100k views a day and had 5-10k unique pieces of content a day. This hurt for a few reasons.

First, wildly varying document sizes. Mongo pads documents a bit, so they can be modified without moving on disk. If a document grows larger than the padding, it has to be moved. Obviously, the more you hit the disk the slower things are, just as the more you go across the network the slower things are. Having some documents with 100 keys and others with 10k made it hard for Mongo to learn the correct padding size, because there was no correct size.

Second, when you have all the content for a day in one doc and have to send 10k urls plus page titles across the wire just to show the top fifteen, you end up with some slowness. One site consistently had documents that were over a MB in size. I quickly realized this was not going to work long term.

In our case, we always write data in one way and always read data in one way. This meant I needed an index I could use for writes and one that I could use for reads. I’ll get this out of the way right now. If I had it to do over again, I would definitely do it different. I’m doing some stupid stuff, but we’ll talk more about that later.

The keys for each piece of content are the site_id (sid), path (p), views (v), date (d), title (t), and hash (h). Most of those should be obvious, save hash. Hash is a crc32 of the path. Paths are quite varying in length, so indexing something of consistent size is nice.

For writes, the index is [[‘sid’, 1], [‘d’, -1], [‘h’, 1]] and for reads the index is [[‘sid’, 1], [‘d’, -1], [‘v’, -1]]. This allows me to upset based on site, date and hash for writes and then read the data by site, date and views descending, which is exactly what it looks like when we show content to the user.

As mentioned in the previous post, I do a bit of range based partitioning as well, keeping a collection per month. Overall, this is working great for content, referrers and search terms on Gaug.es.

Learning from Mistakes

So what would I do differently if given a clean slate? Each piece of content and referring url have an _id key that I did not mention. It is never used in any way, but _id is automatically indexed. Having millions of documents each month, each with an _id that is never used starts to add up. Obviously, it isn’t really hurting us now, but I see it as wasteful.

Also, each document has a date. Remember that the collection is already partitioned by month (i.e.: c.2011.7 for July), yet hilariously, I store the full date with each document like so: yyyy-mm-dd. 90% of that string is completely useless. I could more easily store the day as an integer and ignore the year and month.

Having learned my lesson on content and referrers, I switched things up a bit for search terms. Search terms are stored per month, which means we don’t need the day. Instead of having a shorter but meaningless _id, I opted to use something that I knew would be unique, even though it was a bit longer.

The _id I chose was “site_id:hash” where hash is a crc32 of the search term. This is conveniently the same as the fields that are upserted on, which combined with the fact that _id is always indexed means that we no longer need a secondary index for writes.

I still store the site_id in the document so that I can have a compound secondary index on site_id (sid) and views (v) for reads. Remember that the collection is scoped by month, and that we always show the user search terms for a given month, so all we really need is which terms were viewed the most for the given site, thus the index is [[‘sid’, 1], [‘v’, -1]].

Hope that all makes sense. The gist is rather than have an _id that is never used, I moved the write index to _id, since it will always be unique anyway, which means one less secondary index and no wasted RAM.

Interesting Finding

The only other interesting thing about all this is our memory usage. Our index size is now ~1.6GB, but the server is only using around ~120MB of RAM. How can that be you ask? We’ve all heard that you need to have at least as much RAM as your index size, right?

The cool thing is you don’t. You only need as much RAM as your active set of data. Gaug.es is very write heavy, but people pretty much only care about recent data. Very rarely do they page back in time.

What this means is that our active set is what is currently being written and read, which in our case is almost the exact same thing. The really fun part is that I can actually get this number to go up and down just by adjusting the number of results we show per page for content, referrers and search terms.

If we show 100 per page, we use more memory than 50 per page. The reason is that people click on top content often to see what is doing well, which continually loads in the top 100 or 50, but they rarely click back in time. This means that the active set is the first 100 or 50, depending on what the per page is. Those documents stay in RAM, but older pages get pushed out for new writes and are never really requested again.

I literally have a graph that shows our memory usage drop in half when we moved pagination from the client-side to the server-side. I thought it was interesting, so figured I would mention it.

As always, if you aren’t using Gaug.es yet, be sure to give the free trial a spin!

Counters Everywhere

2011-06-28T11:57:32-04:00

Last week, coming off hernia surgery number two of the year (and hopefully the last for a while) I eased back into development by working on Gaug.es.

In three days, I cranked out tracking of three new features. The only reason this was possible is because I have tried, failed, and succeeded on repeat at storing various stats efficiently in Mongo.

While I will be using Mongo as the examples for this article, most of it could very easily be applied to any data store that supports incrementing numbers.

How are you going to use the data?

The great thing about the boon of new data stores is the flexibility that most provide regarding storage models. Whereas SQL is about normalizing the storage of data and then flexibly querying it, NoSQL is about thinking how you will query data and then flexibly storing it.

This flexibility is great, but it means if you do not fully understand how you will be accessing data, you can really muck things up. If, on the other hand, you do understand your data and how it is accessed, you can do some really fun stuff.

So how do we access data on Gaug.es? Depends on the feature (views, browsers, platforms, screen resolutions, content, referrers, etc.), but it can mostly be broken down into these points:

Time frame resolution. What resolution is needed? To the month? Day? Hour? Which piece of content was viewed the most matters on a per day basis, but which browser is winning the war only matters per month, or maybe even over several months.
Number of variations. Browsers is a finite number of variations (Chrome, Firefox, Safari, IE, Opera, Other). Content is completely the opposite, as it varies drastically from website to website.

Knowing that resolution and variation drive how we need to present data is really important.

One document to rule them all

Due to the amount of data a hosted stats service has to deal with, most store each hit and then process them into reports on intervals. This leads to delays between something happening on your site and you finding out, as reports can be hours or even a day behind. This always bothered me and is why I am working really hard at making Gaug.es completely live.

Ideally, you should be able to check stats anytime and know exactly what just happened. Email newsletter? Watch the traffic pour in a few minutes after you hit send. Post to your blog? See how quickly people pick it up on Twitter and in feed readers.

In order to provide access to data in real-time, we have to store and retrieve our data differently. Instead of storing every hit and all the details and then processing those hits, we make decisions and build reports as each hit comes in.

Resolution and Variations

What kind of decisions? Exactly what I mentioned above.

First, we determine what resolution a feature needs. Top content and referrers need to be stored per day for at least a month. After that, probably month is a good enough resolution.

Browsers and screen sizes are far less interesting on a per day basis. Typically, these are only used a few times a year to make decisions such as dropping IE 6 support or deciding to target 1024×768 instead of 800×600 (remember that back in the day?).

Second, we determine the variations. Content and referrers varies greatly on a per site basis, but we can choose the browsers and screen dimensions to track. For example, with browsers, we picked Chrome, Safari, Firefox, Opera, IE and then we lump the rest of the browsers into Other. Do I really care how many people visit RailsTips in Konquerer? Nope, so why even show it.

The same goes for platforms. We track Mac, Windows, Linux, iPhone, iPad, iPod, Android, Blackberry, and Other.

Document Model

Knowing that we only have 6 variations of browsers and 9 variations of platforms to track, and that the list is not likely to grow much, I store all of them in one document per month per site. This means showing someone browser and/or platform data for an entire month is one query for a very tiny document that looks like this:

{
  '_id' => 'site_id:month',
  'browsers' => {
    'safari' => {
      '5-0' => 5,
      '4-1' => 2,
    },
    'ie' => {
      '9-0' => 5,
      '8-0' => 2,
      '7-0' => 1,
      '6-0' => 1,
    }
  },
  'platforms' => {
    'macintosh' => 10,
    'windows'   => 5,
    'linux'     => 2,
  },
}

When a track request comes in, I parse the user agent to get the browser, version, and platform. We only store the major and minor parts of the version. Who cares about 12.0.1.2? What matters is 12.0. This means we end up with 5-10 versions per month per browser instead of 50 or 100. Also, note that Mongo does not allow dots in key names, so I store the dot as a hyphen, thus 12-0.

I then do a single query on that document to increment the platform and browser/version.

query  = {'_id' => "#{hit.site_id}:#{hit.month}"}
update = {'$inc' => {
  "b.#{browser_name}.#{browser_version}" => 1,
  "p.#{platform}" => 1,
}}
collection(hit.created_on).update(query, update, :upsert => true)

b and p are short for browser and platform. No need to waste space. The dot syntax in the strings in the update hash tell Mongo to reach into the document and increment a value for a key inside of a hash.

Also, the _id (or primary key) of the document is the site id and the month since the two together are always unique. There is no need to store a BSON ObjectId or incrementing number, as the data is always accessed for a given site and month. _id is automatically indexed in Mongo and it is the only thing that we query on, so there is no need for secondary indexes.

Range based partitioning

I also do a bit of range based partitioning at the collection level (ie: technology.2011, technology.2012). That is why I pass the date of the hit to the collection method. The collection that stores the browser and platform information is split by year. Maybe unnecessary looking back at it, but it hurts nothing. It means that a given collection stores number of sites * 12 documents at a maximum.

Mongo creates collections on the fly, so when a new year comes along, the new collection will be created automatically. As years go by, we can create smaller summary documents and drop the old collections or move them to another physical server (which is often easier and more performant than removing old data from an active collection).

Because I know that the number of variations is small (< 100-ish), I know that the overall document size is not really going to grow and that it will always efficiently fly across the wire. When you have relatively controllable data like browsers/platforms, storing it all in one document works great.

Closing Thoughts

As I said before, this article is using Mongo as an example. If you wanted to use Redis, Membase or something else with atomic incrementing, you could just have one key per month per site per browser.

Building reports on the fly through incrementing counters means:

less storage, as you do not need the raw data
less RAM, as there are fewer secondary indexes
real-time querying is no problem, as you do not need to generate reports, the data is the report

It definitely involves more thought up front, but several areas of Gaug.es use this pattern and it is working great. I should also note that it increases the number of writes. Creating the reports on the fly means 7 or 8 writes for each “view” instead of 1.

The trade off is that reading the data is faster and avoids the lag caused by having to post-process it. I can see a day in the future where having all these writes will force me to find a different solution, but that is a ways off.

What do you do when you cannot limit the number of variations? I’ll leave that for next time.

Oh, and if you have not signed up for Gaug.es yet, what are you waiting on? Do it!

EventMachine and Passenger

2011-05-04T11:20:00-04:00

In order to fully explain this post, we first need to cover some back story. Originally, Gaug.es was hosted on Heroku. Recently, we moved Gaug.es to RailsMachine (before the great AWS outage luckily), where we are already happily hosting Harmony.

At Heroku, we were running on 1.9.2 and thin. The most common RailsMachine stack is REE 1.8 and Passenger. Sticking with the common stack meant it would be a far easier and faster transition to Railsmachine, so we tweaked a few things and switched.

Heroku, Thin, and EventMachine

While at Heroku, we had been testing using PusherApp for live updating of analytics as they occurred. The pusher gem has two ways to trigger notifications, trigger (net/http) and trigger_async (em-http-request).

Since Heroku runs on thin, we used trigger_async. This meant that sending the PusherApp notifications in the request cycle was fine, as they did not block.

One of the changes when moving to RailsMachine was switching from trigger_async to trigger. Obviously, having an external HTTP request in your request path is less than ideal, but backgrounding it seemed to go against the whole idea of “live”.

Our response times for Gaug.es average around 5ms, so even with 75-100ms for each PusherApp request, we were still in a normally acceptable response time range (not ok with me, but ok for now).

Pusher Conversation

I contacted the fine folks at Pusher and asked if they had any suggestions. One suggestion Martyn mentioned was Thread.new { EM.run }.

Given my lack of experience with threads and event machine, this at first this struck me as dirty/scary and I was not sure if he was serious.

I did a bit of research and discovered he was not only serious, but people we doing it. The AMQP gem even recommends it in the Readme.

Hmmm, This Might Actually Work

After a bit of googling and scouring code on Github I found a few different solutions. I started hacking and got something that was “working” pretty quickly. Quite intrigued I decided to hit up someone smarter than I, Aman Gupta, who maintains the EventMachine and AMQP gems.

He confirmed that it would work and recommended a few tweaks. Yesterday, I pushed it to production and thus far it is working great. Below is the code needed to make the magic happen.

module GaugesEM
  def self.start
    if defined?(PhusionPassenger)
      PhusionPassenger.on_event(:starting_worker_process) do |forked|
        if forked && EM.reactor_running?
          EM.stop
        end
        Thread.new { EM.run }
        die_gracefully_on_signal
      end
    end
  end

  def self.die_gracefully_on_signal
    Signal.trap("INT")  { EM.stop }
    Signal.trap("TERM") { EM.stop }
  end
end

GaugesEM.start

Gaug.es is 100% Sinatra, so I just put this in the file in Gaug.es that works similar to environment.rb or an initializer would in Rails.

There are two key parts. First, if we are running on Passenger and using smart spawning, we need to stop the event machine if it is started. Second, we create a new thread and start the event machine loop.

Now, in the Notification class that we have in Gaug.es, I can do the following to make the Pusher request not block the main request.

EM.next_tick {
  Pusher[channel].trigger_async('hit', doc)
}

The main request carries on as usual and does not wait for the Pusher to request to finish. In the background, event machine is sending all these notifications. Once again, even on Passenger, we now have non-blocking pusher notifications.

Hmm, This Does Work

Since it took me a bit to figure it out, I thought I would post it here for everyone to benefit from and maybe to start some discussion. If you have suggestions or see glaring issues, please let me know.

I have no assumptions that I am wise or that this is perfect, but thus far it is getting the job done with no adverse affects.

Misleading Graph of Proof

Below is a graph of response times for Gaug.es thanks to New Relic. Seriously, where would we be without New Relic! Green is the time spent in external requests. I am sure you can tell at which point I pushed out the event machine integration.

That said, don’t think that all that time is instantly gone. It is still happening, just in a thread in the background without much affect, if any, on our normal response times.

Demo, Plz!

If you are curious about what the live updating looks like currently in Gaug.es, I posted a short video a few weeks back.

Also, if you too are addicted to analytics, you should definitely sign up and try it out. Lots of good stuff coming down the pipe!

SSH Tunneling in Ruby

2011-04-19T10:42:44-04:00

The other day I wanted to do some queries in production, but our servers are pretty locked down to the outside world. I was well aware that I could just make an ssh tunnel to connect to the database server, but I decided I wanted to do it in Ruby.

I am not the brightest of crayons in the box, so it took me a bit. Since I struggled with it for a few, I figured others probably will someday as well and decided to post my solution here.

Obviously, replace the strings with <…> with your own information and change the host port information in the gateway.open call.

require 'net/ssh/gateway'

gateway = Net::SSH::Gateway.new('<myremotehostorip.com>', '<remote_user>')

# Open port 27018 to forward to 127.0.0.1:27017
# on the remote host provided above
gateway.open('127.0.0.1', 27017, 27018)

# Connect to local port set in previous statement
conn = Mongo::Connection.new('127.0.0.1', 27018)

# Just printing out stats to show that it works
puts conn.db('<database_name>').stats.inspect

gateway.shutdown!

With just a few lines of Ruby, I can make scripts that use my local ssh key to talk to production. Thanks go to Jamis Buck for all the heavy lifting of writing net-ssh and company.

Hi My Name is John...

2011-03-22T09:16:04-04:00

…and I am addicted to analytics. It all started when I was a wee lad. I quite enjoyed playing Tecmo NBA Basketball, among other games. One day, while rocking the house with Shawn Kemp and the Seattle Supersonics, I noticed that Tecmo NBA basketball did not seem to be correctly recording rebounds.

Obviously, this kind of egregious error was unacceptable. With pad and paper, I began to keep track of rebounds on my own. After each rebound, I would record the stat for the player grabbing it. Yes, I actually paused game play so that I could have correct analytics on rebounds.

The Joys of Blogging

Anyway, fast forward to 2011 where I now operate as a programmer. I could tell you that I grew out of that phase in my life, but alas I have not. From Shortstat, to Mint, and now on to Gaug.es, I have maintained quite a fascination with analytics.

If I am being completely honest, one of the main reasons I blog is to see the views come in after a new post. And oh the joys when it lands on Reddit or HN and brings me people in excess (and lame comments covering how stupid I am).

Graphite and Statsd

The great thing is that on top of websites, I now help maintain several applications. Applications are a fun and tricky beast full of opportunities to record metrics. Most of the time though, these metrics go unrecorded because it is too much work to store and maintain them.

After reading measuring anything and everything by the fine folks at Etsy, I decided it was time to get dirty. I spent a few hours this weekend setting up Graphite and statsd on a small VPS.

Graphite is “enterprise scalable realtime graphing” and statsd, built by Etsy, is a “network daemon for aggregating statistics, rolling them up, then sending them to Graphite”.

Stealing pieces of a gist, I fumbled my way through, and with a little help from Kastner, I was good to go.

UDP

Once I was past the I feel stupid because I have never really setup python or node.js apps before, it was time to start sending my setup some data. statsd speaks UDP, which I have certainly heard about, but never before actually looked into.

UDP is an unreliable, unordered, lightweight protocol for slinging messages around the interwebs. The best way to think of it for those that are unfamiliar is fire and forget. The huge upside of UDP for analytics is that the effect of sprinkling it all over your app is minimal.

You lose a millisecond constructing and sending the message, but if statsd ever goes down, your app does not. You simply lose statistics until it comes back up. Lets look at a simple example.

require 'socket'
socket = UDPSocket.new
socket.send('some message', 0, '127.0.0.1', 33333)

Go ahead and run that. Notice how it doesn’t error? No, it does not magically spin up something in the background. It is fire and forget. The message is sent, but whether or not it makes it to its destination does not matter. Most of the time it will, sometimes it won’t.

I read somewhere that TCP is like a phone call and UDP is like a letter in the mail. Good analogy.

Statsd from Ruby

I started to work on a UDP client for statsd and then realized I should probably check Github before getting too far in. Thankfully, Rein already had a nice little statsd library created.

I felt like it was missing a few things, so I forked it and added a time method that works with blocks and namespacing (so I could track multiple apps from same graphite/statsd install). I have already talked with him and he plans on pulling both. Until then, you can checkout the mine branch on my fork.

Now that I had the server side setup and was armed with a client library, I started to think about what kind of stats I would like to add to Gaug.es. The first thing I could think of was recording each track. I already store an all time number in Mongo, but minute/hour/day data could not hurt.

I created a tiny wrapper around Rein’s library so things would only be tracked in production. I certainly could do this other ways, and probably will, but it worked good enough to get things out the door.

class Stats
  cattr_accessor :client

  def self.record_stats?
    Gauges.environment == 'staging' || Gauges.environment == 'production'
  end

  def self.increment(*args)
    client.increment(*args) if record_stats?
  end

  def self.decrement(*args)
    client.decrement(*args) if record_stats?
  end

  def self.timing(*args)
    client.timing(*args) if record_stats?
  end
end

Stats.client = Statsd.new(ipaddr, port)
Stats.client.namespace = 'gauges'

Using this, I added a increment to the track route Stats.increment('routes.track'), deployed, and instantly had graphs to play with. Below is tracks per second since last night when I first added the tracking.

Fun Use Case

In Gaug.es, about 75% of the storage is in the contents collection. This collection tracks the views, titles and paths for each site. I was curious what was taking up more space, titles or paths.

Abusing the timing method in statsd, I was able to send the length of the path and title for each piece of content as it was tracked and then get a nice graph of the lower, upper, mean, and upper 90 percentiles.

I noticed right away that some pieces of content were over 600 characters long. This seemed odd, so I started logging the offending pieces of content. I tailed the log for a while and saw that it was Facebook’s fault. :)

For some reason sites using Facebook’s “like” tools end up getting a querying string parameter named fbc_channel, which has a value that is hundreds of characters of json. Awesome.

I created a test case out of the misbehaving content, stripping the fbc_channel param, and deployed a fix. Based on the graph below it is obvious when I pushed out the change.

From adding the analytics, to detection, to deploying a fix, only a few minutes flew by. Note that previously I would not have even tracked content path length. I would have never discovered the issue and the sites that had this going on would have continued to have jacked up stats, probably never mentioning it to me.

You have no excuse

I spent a few hours getting things running, but oh the joy I have now. Setup a small VPS or an EC2 micro instance. Install graphite and statsd. Never again wonder. Graph all your theories and improve your apps. That is all for now, I have more metrics to track!

Give Yourself Constraints

2011-02-21T10:11:51-05:00

Recently, I had a hernia and surgery to fix it. This knocked me out of the game and onto the couch for a couple weeks. During my recovery, I had a lot of time to think. I also had a lot of time to miss what I do every day.

This was the longest period in several years for me without creating. Once I felt good enough to get back at it, even if only for a few hours, I decided to focus all this pent up energy on something new.

What I wanted to do, was to think through a problem different than I ever have. I have been creating applications pretty much the same way for quite some time. Sure, MongoDB changed my methods a bit, but I knew I had not used it to its full potential, as I typically start all new Mongo projects with MongoMapper.

What to Build

First, I thought about what to build. I have a plethora of “someday” ideas that have never made it out of that stage. One of those ideas was to build a simple analytics program.

Sure, there are a crap ton of analytics apps out there, including Google Analytics, but not a one thus far has hit the sweet spot I am looking for.

The Old Constraint

Back in the day, I would entertain every whim I had. This is great for learning a lot of new things, but I never really focused and finished anything. What I had was a project directory full of half (or less) finished ideas.

When I actually forced myself to work on only a project or two (I chose MongoMapper and Harmony), I noticed that I actually finished things and had something to show for myself.

Important: The constraint of what I could work on made me more productive than working on whatever I was inspired to work on.

That said, rules are made to be broken, right? Plus, this new project was not entirely misguided. Harmony manages websites. Websites need analytics. There is definitely a benefit to Harmony in building an analytics system that can be deeply integrated, so I began work.

The New Constraints

Since I was bending the rules a bit, I decided to give myself different constraints this time. Rather than what I would work on, I focused on what tools I could use to do the work. The thought was that forcing myself to avoid my comfort tools would lead to thinking outside of my usual box.

The first constraint I made was that I could only use the Mongo Ruby driver. No ActiveRecord, MongoMapper or any other object mapper.

Second, no aggregate querying for reports. All of the analytics had to be built on the fly as each hit came in. I told myself I could not even store the individual hit data that was generated each time a page was tracked.

Third, no signup or authentication. I wanted to focus on the core functionality (tracking views) instead of spending my time authenticating users and all that crap.

Fourth, I cannot remember, so lets move on.

The Prototype

Within a few hours, using Sinatra and the MongoDB Ruby driver, I had a little prototype working. Each hit was a single MongoDB operation, an upsert based on the host, with year, month, day, and hour information stored in nested hashes. The nested hashes were updated in the operation using $inc. It did not do much, but it was pretty cool.

I am aware, especially now, how simple that first prototype was, but it felt great to create again. Next, I threw the app up on Heroku and MongoHQ, so I could try it out tracking this site you are reading.

Both are free to try, so it cost me nothing to get something up that I and others could react to. I showed it to the rest of the Ordered List team and every one shared the excitement.

The Result

I am amazed at what we have come up with in such a short amount of time (4 weeks using occasional evenings/weekends). In fact, the constraint of time is why Gauges is where it is today. We were really productive in the hours we snatched to work on it, because they were few and far between.

Steve came up with a great name, found a matching domain (which is a miracle these days) and Gaug.es was born.

Home Page

Dashboard

Note: Holy crap did Steve (@orderedlist on Twitter) knock the design out of the park. Dude has skills!

Conclusion

The last 4 weeks working on Gauges has made me feel, at times, like a kid again. It is almost like we were racing to the next fence post (sue me, I grew up on a farm). Sure, I broke the original constraints I set, but without them, I would never have started.

To break constraint numero uno, I used ToyStore, with my mongo adapter. Any service that involves user input needs validations and the like, and I was not in the mood to build them from scratch.

I actually stuck with constraint number two. All of the tracking code uses upserts with MongoDB modifiers to create and update reports on the fly.

Constraint number three also fell to the wayside. You cannot very well make money on a service that requires no signup (or at least it is increasingly difficult), so you do in fact have to sign up and in to use Gauges. That said, right now we are just testing things out, so we are controlling things with a code.

Whether it be the tools that you use, the projects that you work on, or the people you work with, give yourself constraints. Create something that you have always wanted. Get other people excited about it. Work cannot always be “work”.

P.S. If Gauges is something you are interested in and you would like to be an early tester, let me know. The main goals right now are hosted (no setup/servers), easy sharing with other people, how much traffic, where was it from and where was it to.

Plus, we have a few great ideas for down the road, once we get the basics ironed out. Thus far, Gauges is really scratching an itch I have had for a while and I am stoked.

Data Modeling in Performant Systems

2011-02-21T22:16:07-05:00

I have been working on Words With Friends, a high traffic app, for over six months. Talk about trial by fire. I never knew what scale was. Suffice to say that I have learned a lot.

Keeping an application performant is all about finding bottlenecks and fixing them. The problem is each bottleneck you fix leads to more usage and a new bottleneck. It is a constant game of cat and mouse. Sometimes you are the cat and sometimes, well, you are not.

Most of the time, the removal of those bottlenecks is about moving hot data to places that can serve it faster. Disks are slow, memory is fast, enter more memcached.

Over time, you work and work to move hot data into memory and simplify your data access to fit into memory. Key here, value there. Eventually, you get to a place where you have simplified how you access your data into simple key/value lookups.

Games get marshaled into a key named "Game:#{id}". Joins are simplified to selecting ids and caching the array of ids into a key such as "User:#{id}:active_game_ids" or "User:#{id}:over_game_ids". In turn, those arrays are turned into objects by un-marshaling the contents of "Game:#{id}", etc.

Your data model morphs from highly relational to key/value because key/value is fast and memcached can withstand a bruising.

Do it once, and you know how to do it in the future. The problem is by the time you get to this data model, it is kind of bolted on/in to your app.

What if you could design it this way from the beginning? What if you had no option but to think through your data model in keys and values? Need your data in two different ways? Put it in two different places, etc, etc.

I have good news. Now you can.

A Little History

Not long into my tenure with WWF, we were hitting a lot of walls and there was a lot of talk about NoSQL. Mongo? Membase? Cassandra? Riak?

Which one will work best for the problem at hand? What if we could try them all really easily by just changing which place the data went to? What if we could try out more than one at once?

I sat down one weekend and started thinking about the app and realized what I just talked about above. Along the way, our data access changed from relational to key lookups. This made me think about a hash.

Hashes are so versatile, and yet, so constrained. Hashes are for reading, writing and deleting keys, just like key/value stores. I did a bit of GitHub searching and stumbled across moneta, by Yehuda Katz.

Moneta immediately struck me as brilliant. I was shocked there was no activity around it. If you only allow yourself to read, write and delete with the same API, you can make nearly any data store talk the correct language.

I fiddled with it and forked it, but in the end, it was not quite what I was looking for. I liken it to my first house. I like the house, but having lived in it for six years, I know exactly what I want out of my next house.

The folks at Newtoy (now Zynga with Friends) had mentioned that they wanted to build their own object mapper and name it ToyStore—such a great name.

In a fit of inspiration over the 4th of July weekend, I cranked out attributes and initialization, relying heavily on ActiveModel. It was really fun. I emailed the crew when the next work day came around and they were stoked.

It began to occupy some of my work-related time and Geoffrey Dagley started helping me with it. Over the next few weeks, Geof and I hammered out validations, serialization, callbacks, dirty tracking, and much more.

Everything was built on the premise that the only acceptable methods that could be used to read, write and delete data were read, write and delete.

Adapter: The Common Interface

Over time Brandon Keepers got involved and ToyStore started looking pretty legit. We switched from using Moneta as the base to something I whipped together in a few hours, Adapter.

Defining an adapter is as simple as telling it how the client reads, writes and deletes data. You also have to define a clear method for convenience and to stick close the Ruby hash API.

The client can be anything that you want to have a unified interface. For example, this is how you would create an adapter to store things in a ruby hash.

Adapter.define(:memory) do
  def read(key)
    decode(client[key_for(key)])
  end

  def write(key, value)
    client[key_for(key)] = encode(value)
  end

  def delete(key)
    client.delete(key_for(key))
  end

  def clear
    client.clear
  end
end

key_for ensures that most things can work as a key. encode and decode allow one to hook some kind of serialization in, whatever you fancy, be it Marshal, JSON, or whatever you can imagine.

By defining those methods, we can now get an instance of this adapter and connect it to a client. In the example above, the client is just a plain ruby hash, but in other adapters, it could be an instance of Redis (adapter), Memcached (adapter), or maybe a Riak bucket (adapter).

adapter = Adapter[:memory].new({}) # sets {} to client
adapter.write('foo', 'bar')
adapter.read('foo') # 'bar'
adapter.delete('foo')
adapter.fetch('foo', 'bar') # returns bar and sets foo to bar

# [] and []= are aliased to read and write
adapter['foo'] = 'bar'
adapter['foo'] # 'bar'

Adapters can also be defined using a block (like above), a module, or both (module included first, then block so you can override module with block).

Adapters can also define atomic locking mechanisms, see the memcached and redis adapters for their locking implementations. The more opaque the object, the more you need to lock. Or, in the case of riak, the adapter can handle read conflicts.

ToyStore: The Mapper Fixings on top of Adaper

Once you have secured how your data layer speaks the adapter interface you can use the real power, ToyStore.

Lets say you want to store your users in redis. Create your class, include the Toy::Store, and set it to store in redis.

require 'toystore'
require 'adapter/redis'

class User
  include Toy::Store
  store :redis, Redis.new

  attribute :email, String
end

From there, you can go to town, defining attributes, validations, callbacks and more.

class User
  include Toy::Store
  store :redis, Redis.new

  attribute :email, String
  validates_presence_of :email
  before_save :lower_case_email

private
  def lower_case_email
    self.email = email.downcase if email
  end
end

user = User.new
pp user.valid?

user.email = 'John'
pp user.save

pp user
pp User.get(user.id)

user.destroy
pp User.get(user.id)

Change your mind? Decide that you do not want to use Redis? Fancy Riak? Simply change the store to use the riak adapter and you are rolling.

require 'toystore'
require 'adapter/riak'

class User
  include Toy::Store
  store :riak, Riak::Client.new['users']

  attribute :email, String
end

Boom. You just completely changed your data store in a couple lines of code. Practical? Yes and no. Cool? Heck yeah.

What all does Toy::Store come with out of the box? So glad you asked.

Attributes – attribute :name, String (or some other type) Can be virtual which works just like attr_accessor but all the power of dirty tracking, serialization, etc. Also, can be abbreviated which means :first_name could be the method you use, but in the data store the attribute is :fn. Save those bytes! Allows for default values and defaults can be procs.
Typecasting – Same type system as MongoMapper. One day they will share the exact same type system in its own gem, for now duplicated.
Callbacks – all the usual suspects.
Dirty Tracking – save, create, update, destroy
Mass assignment security – attr_accessible and attr_protected
Proper cloning
Lists – arrays of ids. If user has many games, user would have list :games which stores in game_ids key on user and works just like an association.
Embedded Lists – array of hashes. More consistent than MongoMapper, which will soon reap the benefits of the work on Toy Store embedded lists.
References – think belongs_to by a different (better?) name. Post model could reference :creator, User to add creator_id key and relate creator to post.
Identity Map – On by default. Should be thread-safe.
Read/write through caching – If you specify a cache adapter (say memcached), ToyStore will write to memcached first and read from memcached first, populating the cache if it was not present.
Indexing – Need to do lookups by email? index :email and whenever a user is saved the user data is written to one key and the email is written as another key with a value of the user id.
Logging
Serialization (XML and JSON)
Validations
Primary key factories

It pretty much has you covered. Adapters for redis, memcached, riak, and cassandra already exist. Expect a Mongo one soon. Have to make a few tweaks to adapter. Yep, even Mongo.

What are other adapters that could be created? Membase? Just start with the memcached adapter and override key_for. Git? File system? REST? MySQL?! I love it!

The Future

The future is not picking a database and forcing all your data into it. The future (heck, now even) is the right database for the job and your application may need several of them.

All this said, in no way do I think ToyStore is going to take the world by storm. It is a different way to build applications. This way comes with great power, but great confusion as well.

Currently, each model is serialized into one key in the store, based on how the adapter does encode/decode. Eventually, I would like to add the ability to store different attributes in different keys. For example, maybe you want active_game_ids to be stored in a key by itself so you don’t have to constantly save the entire user object.

I can also see a use for being able to store an attribute not just a different key, but a different store entirely. Store your user objects in Riak, but active_game_ids in a Redis set. This is where it would get really powerful.

At any rate, I am very excited about this project and I think it has a lot of potential. I would also like to add that MongoMapper is here to stay.

In fact, I learned from my mistakes on MongoMapper when building ToyStore and will be back-porting those learned experiences very soon. Expect a flurry of activity over the next little while.

Closing Thanks

Huge thanks to Newtoy (now Zynga with Friends) for allowing Geof and I to open source this. Several pieces of ToyStore were built on their dime and I really appreciate their contribution to the Ruby and Rails community!

As is typical with new projects, there are probably rough spots and good luck finding documentation. I have included a bevy of examples and the tests do a superb job at explaining the functionality of each method/feature.

Let me know what your thoughts are and be sure to kick the tires!

Roundup of Links

Year In Review

2010-12-31T13:56:57-05:00

One of the main reasons that I write is for reflection. Blogging gives me a history of what I was interested in and when. In 2008, I posted the Smörgåsbord. I skipped 2009, for whatever reason, but 2010 will not suffer the same fate.

Bang!

The first post of the year was pretty huge for me. It was the first post using Harmony, the website management system that we are building at Ordered List. We even released it for public consumption on August 3rd.

Post number two was my most popular post of all time. It took off so fast that I could not even enjoy it. I immediately had to add page caching to Harmony so it could handle the load.

I Have No Talent had 25k views in the first two days, but that did not prepare me for day three. Day three ended with over 60k views and by the time the dust settled on day four, it was over 100k views. Crazy!

Below is a screenshot of views for the entire year. Note how most of the year looks like a flat line next to the no talent post and the other hit of the year, Stop Googling.

Thoughts

Every year I try to do a few posts that are just thoughts I have been having or that someone else had and I respect.

This year I posted on Just in Time, Not Just in Case, Correct, Beautiful, Fast, In That Order, and Improving Your Methods. Each of these posts are something I believe really strongly in.

Presentations

This was the year of speaking for me. There were a few stretches where I was prepping for a presentation every other week. That is definitely a bit hectic when each presentation is on a new and different topic.

A few of the highlights were keynoting the Great Lakes Ruby Bash on my lack of talent, talking about MongoMapper at MongoSF, and teaching how to steal at RailsConf 2010.

I even posted on how to Improve Your Presentations in Under $50. If you are presenting any time soon, definitely check that post out.

Projects

This was the year of the tiny project for me. I have worked on so many applications now that I am beginning to see a lot of patterns. Each time I repeated myself, I did my best to move the abstraction into its own gem and share it for all or none to use.

This year, I released Canable to help with permissions, Joint to make file uploads with GridFS easy, Whois to aid in naming your project, Bin for caching in Mongo, Plucky for sexy Mongo querying, Hunt for easy search with Mongo, and Scam a simple enum/fake model helper.

In addition to the aforementioned new projects, I released several updates to MongoMapper, including plugins, identity map, and scopes.

I also tried to share some of what I learned building MongoMapper in Creating Duplicable Objects and Over-ridable Accessors, and The Chain Gang.

Numbers

Though I posted about half as many times as in 2009, I feel the quality went way up. In total, RailsTips garnered over 700,000 views in 2010. Blows my mind that it has grown to that in three and half years.

Even more mid blowing is how much I have grown in that time. From barely knowing Rails, to creating one of the more popular object mappers. From working at Notre Dame, to owning my own business.

I feel that all of this really is a testament to what a lot hard work can result in. I cannot wait to see what 2011 brings.