Gowalla Engineering

Scaling at Gowalla: Databases & NoSQL

2011-11-17T00:00:00-08:00

This is an introduction to relational and non-relational databases as they relate to scalability of web apps. The recording is for a guest Lecture I gave at the University of Texas School of Information.

In this talk I address the storage technologies and tools Gowalla uses including PostgreSQL, Memcached, Redis and Cassandra.

Video

Slides

Open Source Libraries

Gowalla open-sourced a few NoSQL libraries that you can use in your own apps. Check out Likeable & Chronologic.

Scraping Wikipedia

2011-11-16T00:00:00-08:00

Here at Gowalla we use Campfire extensively for internal communication. We’ve always had some sort of chat robot hanging out in our rooms with us, providing simple abilities, but recently GitHub released Hubot and we moved our bot functionality over to Hubot scripts.

Can We Query Wikipedia?

Just the other day, our own Andy Ellwood asked if we could have a script that would query Wikipedia and return article summaries. Already looking for an opportunity to become more comfortable with CoffeeScript and Hubot, I said “Sure.”

So, I asked myself, does Wikipedia have an API? Well, it turns out that they do, but it’s not terribly helpful. For the task at hand—pulling a summary from any given article—the best their API can do is return the entire article as a huge chunk of malformed HTML. Not exactly ideal, but this is what we have tools for.

Scraping Article Pages

So I looked through GitHub’s repository of existing Hubot scripts from the community for ideas on how to proceed. I found web.coffee, which uses HTMLParser and Soupselect to parse arbitrary HTML into a workable DOM. This makes dealing with Wikipedia’s malformed documents somewhat easier.

So long as we’re going to parse all of the HTML in articles anyway, we may as well just load the full article pages instead of dealing with their complicated API:

makeArticleURL = (title) ->
  "http://en.wikipedia.org/wiki/#{encodeURIComponent(title)}"

Then, in order to get a summary for a given article, I noticed that each article has a series of <p> elements on a given page, the first one of which is usually the summary, so long as it contains enough text. First I made a utility method to give me arbitrary DOM objects from HTMLParser for ease of use:

parseHTML = (html, selector) ->
  handler = new HTMLParser.DefaultHandler((() ->),
    ignoreWhitespace: true
  )
  parser  = new HTMLParser.Parser handler
  parser.parseComplete html

  Select handler.dom, selector

Then, pulling all paragraphs on the page was simple:

paragraphs = parseHTML body, "p"

Finding a Summary

In order to find the right paragraph—a summary—I tried playing with data from many articles and came up with a simple heuristic that does quite well:

findBestParagraph = (paragraphs) ->
  return null if paragraphs.length is 0

  childs = _.flatten childrenOfType(paragraphs[0], 'text')
  text = (textNode.data for textNode in childs).join ''

  # remove parentheticals (even nested ones)
  text = text.replace(/\s*\([^()]*?\)/g, '').replace(/\s*\([^()]*?\)/g, '')
  text = text.replace /\s{2,}/g, ' '                # squash whitespace
  text = text.replace /\[[\d\s]+\]/g, ''            # remove citations
  text = _s.unescapeHTML text                       # get rid of nasties

  # if non-letters are the majority in the paragraph, skip it
  if text.replace(/[^a-zA-Z]/g, '').length < 35
    findBestParagraph paragraphs.slice(1)
  else
    text

Essentially, I collect all of the text under the element and strip out some undesirable bits, then look to see if the remainder has at least 35 letters in it. I can’t give a compelling explanation of why this works well, other than that I tried several methods and limits before landing on this one. Sometimes there are deceptively simple heuristics that can accomplish tasks that may seem difficult at first. It can’t hurt to just dive in with some real data and see what happens.

Find All Text Nodes in a DOM Tree

The only other bit of difficulty came when trying to extract the text from the DOM tree generated by HTMLParser. For that, I wrote a recursive method:

childrenOfType = (root, nodeType) ->
  return [root] if root?.type is nodeType

  if root?.children?.length > 0
    return (childrenOfType(child, nodeType) for child in root.children)

  []

When dealing with trees, recursion often works quite well. In this case, I just weed out text nodes from all elements in the DOM subtree rooted at root, returning empty array tombstones for everything else, which are later discarded with _.flatten.

The Final Product

The final script can be found in GitHub’s official hubot-scripts repo. Hopefully the simple tactics in this script will help you put together your own interesting Hubot script, or just help improve your CoffeeScript skills. Please also let us know if you find ways to improve the script—we’re always looking to improve!

Likeable: Love your Objects with Redis

2011-11-03T00:00:00-07:00

Likeable is a new open-source Ruby library that we built here at Gowalla to power all of the "loves" on stories, comments, highlights and photos that you see. It's built on top of Redis to be extremely simple, isolated and fast. Likeable plays well with ActiveRecord objects, but can be used with any Ruby object that implements an #id method.

To see it in action, head over to my Gowalla profile and love the first thing you see. Immediately you'll see the heart change colors and the love count increase by one. If you click on that number you'll see everyone who has loved that object.

This might seem trivial at first glance. But consider that dozens of elements on the page must pull this information: how many people have loved it, whether the viewer has loved it, and (on some pages) which of your friends have loved it. Using Redis allows us to stay simple, flexible and fast.

The Code

Let's say we've got a comment made on a recent Gowalla story.

@comment = Comment.last
@comment.like_count  #=> 0

If our user wants to like this comment, he can do so with ease:

@user = User.find_by_username('schneems')
@user.like! @comment

And that's it. The comment now reports the details of the like.

@comment.like_count      #=> 1
@comment.likes           #=> [#<Likeable::Like ... >]
@comment.likes.last.user #=> #<User username: "schneems" ...>

The Install

For Rails apps, Likeable is extremely simple to set up.

First, add Likeable to your Gemfile:

gem 'likeable'

Set up your Redis connection in (e.g.) config/initializers/likeable.rb:

Likeable.setup do |likeable|
  likeable.redis = Redis.new
end

Next, add the Likeable::UserMethods module to your User model:

class User < ActiveRecord::Base
  include Likeable::UserMethods
end

Finally, add the Likeable module to any model you want to be likeable:

class Comment < ActiveRecord::Base
  include Likeable
end

To learn more, check out the Likeable source, look at the example Rails app or watch the Likeable screencast.

Contributing

We'd love to get contributions to Likeable. If you want it to do something it doesn't already, open an issue on GitHub. Or, if you feel so inclined, create a patch and submit a pull request.

AFNetworking FAQ

2011-11-01T00:00:00-07:00

This article is out of date. Please find the most recent version on the AFNetworking Wiki.

Boxer: Generating Sane JSON from Complex Objects for Our API

2011-10-24T00:00:00-07:00

The Problem

Here at Gowalla we have a large API with many different types of objects distributed across many different resources. So composing and showing views of those objects for API responses has been a challenge.

We started out by just rendering ActiveRecord models directly to JSON:

class Api::UsersController < ApiController
  def show
    @user = User.find(params[:id])
    render :json => @user
  end
end

But that quickly became unwieldy as we wanted full and simple control over which attributes are returned, and to start returning method values alongside model attributes. So we moved on to writing .json.erb files, which allowed us to specify exactly what appears in a response:

# app/views/users/show.json.erb
<%= raw({
  :name => @user.name,
  :recent_spots => @user.recent_spots(30),
  :is_private => @user.private?
}.to_json) %>

Though after a long while, these views ended up growing out of control, with conditionals based on the authenticated user requesting the information, whether they're friends, whether the user resource is a business, whether a promotion is running, etc.

# app/views/users/show.json.erb
<%
  hash = {
    :name => @user.name,
    :is_private => @user.private?
  }
  if user.public?
    hash.merge!({
      :recent_spots => @user.recent_spots(30),
      :recent_highlights => @user.recent_highlights(10),
    })
  end
  if user.friends?(current_user)
    hash.merge!({
      :is_friend => true,
      :mutual_friends => user.mutual_friends(current_user).map(&:to_hash),
    })
  end
%>
<%= raw(hash.to_json) %>

The Solution

So during our developing of Gowalla 4, we came across Nathan Esquenazi's RABL library, which is a great attempt at attacking this problem. For our purposes, though, we went with a simpler and less robust framework that would do exactly what we wanted.

Our simple framework, Boxer, knows about different object representations, the various views each of our objects can have and will let us compose objects together in responses that satisfy the complexity of our large API.

Instead of defining messy conditional views full of merged hashes, Boxer provides a clean framework for describing the representation and views of an object as hashes:

Boxer.box(:user) do |box, user|
  box.view(:base) do
    {
      :name => user.name,
      :age  => user.age,
    }
  end

  box.view(:full, :extends => :base) do
    {
      :email      => user.email,
      :is_private => user.private?,
    }
  end
end

Then you can "ship" the contents of a box with a model object (or any other arguments you want to pass in) and get a JSON-ready hash:

>> Boxer.ship(:user, User.first, :view => :full).to_json
=> "{"name": "Bo Jim", "age": 17, "email": "b@a.com", "is_private": false}"

In our case, our controllers do more or less a straight render of a shipped box:

def show
  @object = Object.find(params[:id])
  render :json => Boxer.ship(:object, @object, :view => :full)
end

For more examples and a more in-depth explanation, see the Boxer README and the project wiki.

Boxer also goes through the trouble of solving small and common problems that creep up when using this style of framework. Things like preconditions, helper methods, including methods for use in boxes and multiple box inheritance.

We have released Boxer as a gem and, as noted, an open-source project on GitHub. Let us know what interesting uses you find for Boxer, in addition to any suggested improvements.

AFNetworking: A Delightful Networking Library for iOS and Mac OS X

2011-10-24T00:00:00-07:00

This article is out of date. Please find the most recent version on the AFNetworking Wiki.

Classifying Puppet Nodes with EC2 Security Groups

2011-10-21T00:00:00-07:00

Introduction

One of the tools we are using on the operations side of things here at Gowalla is the configuration management tool Puppet. Many of you are likely already familiar with the project, but if not, I definitely recommend that you check it out over at Puppet Labs. It has matured quite a bit since its beginnings, and is now fairly easy to get up and running, especially with all of the open source modules being shared on PuppetForge and GitHub.

After using Puppet for a bit, you will likely come to know the tedious task of adding and removing node definitions. This is especially true if your infrastructure is hosted on a cloud provider such as EC2, as ours is. We bring down and spin up new instances fairly often, and having to manually add and remove these definitions becomes time-consuming and hampers automation.

Puppet ENC and EC2 Security Groups

This is where Puppet's External Node Classifier feature comes to the rescue. It allows you to delegate node classification, and therefore which classes are applied to each node, to an external script or application. As you can imagine, this provides all kinds of possibilities for integrating with other systems that may already know about your nodes.

But what if you don't already have this information in another system? This is where we found ourselves. After doing a bit of research, we stumbled across a presentation that suggested using EC2 security groups to define nodes. Bingo! This seemed like a great solution. We already had our nodes (mostly) separated into different security groups based on their purpose, as this allowed us to easily define firewall rules between our different types of instances. Why not re-use information that's already there?

The implementation was fairly straightforward. We hacked together a quick Ruby script (ec2_enc.rb) that would query the EC2 API with a node's hostname to determine its security group. With that information in hand, the script loads a YAML file (node_groups.yml) containing all of the security groups mapped to the classes and parameters to be applied, and passes the proper data back to the puppetmaster. After creating the YAML file with the required mappings, we were up and running.

It's a simple solution that has worked quite well for us so far. Of course there's always room for improvement—please feel free to use, modify and send us feedback at will!

Files

So You Want to Hire an Intern?

2011-10-21T00:00:00-07:00

This past summer, Gowalla brought on its second intern, Michael May, a Computer Science undergrad from the University of Texas. In turn, Michael brought us a great amount of enthusiasm and ability.

I had the pleasure of mentoring Michael over his internship. It led me to reflect on my days of interning when I was still at Georgia Tech. Having seen both sides of the intern/employer relationship, I'd like to clear up some common misconceptions about interns, and offer suggestions on how to keep the relationship fruitful and symbiotic.

Myth: Interns are cheap labor! Replace all your employees weith interns!

I hear this one all the time from developers — Instead of hiring one employee, they'll hire three interns: instant three-fold increase in output. Sadly, The Mythical Man-Month applies at least as much for interns as for employees.

Reality:

Interns are new employees. Like any new employee, they'll need a certain amount of ramping-up time before they can start contributing. Others from the team will have to invest time to get them up to speed. And unlike a full-time employee, you lose all that training time when they leave to go back to school. They may be "cheap" on the pocketbook, but their cost goes far deeper than that. The good news: it forces your organization to get good at bringing people up to speed, and to put the right processes and tools in place to do so — an asset for a quickly-growing company.

Myth: Interns are great for doing degrading or extremely repetitive work!

In my first internship, I didn't have to make coffee, but I did have to make thousands of manual z-folds for brochures we sent out. I also had to wash the bosses' grill, and even disassemble and fix a toilet. This might have been great for the company I was working for, but it wasn't exactly the work experience I would have liked to put on my post-college résumé.

Reality:

Interns offer the chance to hire someone fresh out of college who was trained on the cheap and already knows your organization inside and out. If you insist on using your interns for nothing but cheap, mindless labor, you won't find them knocking down your door after graduation. (Even worse: they might quit on you mid-term.) But even if they're willing to come back to work for you, how have those chores prepared them to be valuable contributors?

Give your interns the tools and resources they need to do their jobs — not just a desk and an internet connection, but also direction and feedback. The more you treat them like real employees, the more you'll get back. If your employees aren't learning from their mistakes, and celebrating their successes, how can the company know if it's headed in the right direction? Be forward with expectations, and be vocal when those expectations aren't met, or when they've been surpassed. This isn't easy, but it has to be done.

Myth: Interns are lazy! They're not capable of doing "real" work.

When I worked for a major American appliance manufacturer, I played Flash games for a whole week because no one could find me anything "I could do." While it did wonders for my high scores, it was a suck on my morale, and a drain on company funds.

Reality:

I've seen interns do incredible things. Once he was comfortable with the code and the team, Michael started knocking out features and closing quite a few GitHub issues. Just like full-time employees, interns want to do work that impresses and amazes. Just like full-time employees, interns work best when they can see how the work they're doing affects the big picture. Pair them up with different members of your team and come up with a number of different projects your intern can work on. Don't underestimate their abilities.

Sum it all up:

Interns take a lot of time and energy, but they can be incredibly rewarding to you and your company. If you want to hire one, make sure they have a dedicated mentor, someone who can and should regularly ask: "Do you have enough to do?", "Is there anything you need?", "What are you working on?" — and simply, "Are you happy?". The mentor should have informal meetings with the intern on a regular basis — at least once a week, but perhaps as often as once a day.

You will also need several people interested in spec-ing out tasks for your junior employee — not just for the initial assignment, but to answer the occasional question during the task. And you'll need employees to do pair programming with the intern until they're familiar enough with your codebase to take on tasks by themselves. When your intern is busy, your intern is happy. And so is your company.