opensoul.org by Brandon Keepers

The Plight of Pinocchio: JavaScript's quest to become a real language

2012-05-16T08:08:32-04:00

JavaScript is no longer a toy language. Many of our applications can’t function without it. If we are going to use JavaScript to do real things, we need to treat it like a real language, adopting the same practices we use with real languages.

This framework agnostic talk takes a serious look at how we develop JavaScript applications. Despite its prototypical nature, good object-oriented programming principles are still relevant. The design patterns that we’ve grown to know and love work just as well in JavaScript as they do any other language. Test driven development forces us to write modular, decoupled code.

Here are the slides for my JSDay talk.

Walking

2012-02-03T10:51:17-05:00

I had to answer the question “What inspires you?” for a TEDx application. Every answer to that question sounds cheesy. Here’s my cheesy answer.

I am inspired by walking; steady, continuous forward progress. Fast enough to get where I want to go in time, but not too fast.

Maintaining a fast pace while walking requires enormous concentration on the act of walking. It is extremely difficult to focus on anything else and maintain a fast pace. I only walk fast when arriving at my destination is of the utmost of importance.

Walking slow is often enjoyable, but neglects the fact that the point of walking is to go somewhere. I walk slow when I have a particularly difficult problem to solve, or I only want a slight change of scenery in hopes of inspiration.

Walking naturally requires almost no concentration. It allows me to focus on what is important, adjust my path when necessary, and arrive at my destination in a reasonable time.

I am inspired by progress. I love the continuous process of taking something that is awkward, uncomfortable or even painful and making it incrementally better.

Getting Started with Sublime Text 2

2012-02-02T00:56:41-05:00

I recently switched to Sublime Text 2 from TextMate. I did not love it at first. As it goes with many of the finer things in life, Sublime is an acquired taste. It was not until I paired with John Nunemaker for a few days that I really started to see its power. I have been using it for several weeks now and have not been tempted to open TextMate for anything.

Here are a few tips that made it easier for me to get into it. Thanks to Anthony Garand for providing feedback and contributing to some of these tips below.

Beat the ugly out of it

I cannot figure out a nice way to say this, so I will just say it: out of the box, Sublime is hideous. We are pompous programmers, so we cannot let that stand.

Update: The new default theme that was was released today is actually much better. I still prefer the light Soda theme linked to below, but it’s no longer a necessity.

Start by installing the Soda UI theme. This is not a theme in the syntax highlighting sense, but for the UI widgets in Sublime. Note that when plugins talk about the Packages directory in Sublime, on OS X that is in ~/Library/Application Support/Sublime Text 2/Packages.

Ahh, that is better. But the icon is ugly too. There are several replacement options on dribbble. This one is my favorite, although there is still much room for improvement.

The syntax highlighting theme that comes with Sublime is alright, but everyone has their favorite theme. Mine is Made of Code. You can use any TextMate theme by copying it to the Packages directory and going to “Preferences » Color Scheme”.

Ok, now we can stand to look at it.

Opening a project

Sublime has support for project files, but I do not use them. I prefer to just open projects from the command line. Sublime comes with a command line interface, but you need to add it to your path.

sudo ln -s "/Applications/Sublime Text 2.app/Contents/SharedSupport/bin/subl" /usr/bin/subl

Now you can open a file or directory from the command line.

subl -n ~/blog

I almost always edit from the current directory, so I set up a bash alias.

alias e="subl -n ."

Ok, now we have Sublime looking pretty and our project open.

Customizing Settings

Sublime is extremely configurable, but the configuration is in a few different JSON files that you have to edit by hand. But once you get familiar with them, they are not that scary.

On OS X, go to “Sublime Text 2 » Preferences”.

For each type of setting, Sublime has at least two files: defaults and custom user settings. You edit the User versions to specify the things you are overriding.

File Settings affect the file that you are currently editing. This includes things like your color scheme, fonts…basically, anything related to editing files.
Global Settings affect how Sublime itself behaves, including the UI theme, files to ignore, and window behavior.
Key bindings are…well, key bindings. 614 lines of glorious shortcut keys.

Take time to browse through the defaults to see all the levers and dials you can tweak. Check out my settings.

Command palette

Think of the command palette as Alfred for your code editor. Quick access to pretty much anything. Pressing ⌘⇧P will bring up the command palette where you can then scroll through a few of the options available to you: snippets, convert case, sort lines, and toggle various interface options.

Package Control

If you feel like something is missing from Sublime, then there is probably a plugin for it. The most essential plugin is Package Control, which allows you to install pretty much anything.

To use package control, press ⌘⇧P to bring up the command palette, begin to type “install”, and select “Package Control: Install Package”. This will bring up a new palette to search for packages. Find your packages and hit enter to install it. You can see in the status bar at the bottom that it is installing. Most packages will just work, but occasionally you might have to restart Sublime for the package to work.

⌘p is the new ⌘t

⌘p (which is also mapped to ⌘t) brings up the Go To File palette. In addition to giving you quick access to your files, it has several features to allow you quickly jump into your files. After typing the file name, use @ in the palette to jump directly to a specific method. You can also use ⌘r to bring up the palette prefaced with @ to jump to a method in your current file.

You can also use : (colon) in the palette to go to a specific line number, and ^g will open the paletted prefaced with colon to jump to a line in the current file.

Favorite Plugins

Here is a list of some of my favorite plugins. Most of these should be available to install through Package Control. Check them out and don’t forget to browse around Packgae Control to see what else is available. Here is a complete list of packages available.

One more thing

I highly recommend you go over and read the Sublime article on NetTuts+ for many more tips and tricks to integrate into your workflow as you migrate to this new editor.

I started tweeting tips from @sublimetips on Twitter. Tweet your tips mentioning @sublimetips and I will likely retweet them.

Happy Coding!

The $40 Standup Desk

2012-03-02T17:35:31-05:00

I spent weeks searching for the ideal standup desk. I shopped around online, visited every furniture store within 10 miles no fewer than 3 times, and almost ordered a Geek Desk. I could not find something that was exactly what I wanted and I could not justify paying $800 or more for a desk that I did not love.

Ultimately, I decided I to do something cheap for now; I built my own. I had been thinking about doing it for a while when my coworker Steve Smith built his desk. After seeing his, I was convinced this was what I wanted to do.

It turns out, the cheap option is pretty awesome.

Parts

Heavy-duty 19" shelf brackets – These should cost $8-$10 each at the local hardware store. A set of 2 brackets claims to be able to hold 1000 pounds if properly secured, so it should be more than enough, even for extra-curricular activities (good luck).
Desktop surface – I found a local furniture outlet that sells assorted desktop surfaces for $3-$4 per foot. I picked up a 4’ desktop for $16.
Screws – I needed 6 larger screws (about 2 1/2") to attach the brackets to the wall, and 6 smaller screws (less than the thickness of the desktop) to attach the desktop to the brackets.

Tools

Drill for creating pilot holes for the screws. If you have powerful drill/screwdriver, you may not need to drill holes.
Level, because nobody wants a crooked work surface.
Tape measure to get the height right and measure the distance between the studs.
A stud finder or old man that knows how to knock on the wall and tell where the studs are. Since this is a desk and will be supporting a lot of weight, the brackets need to be screwed into the studs in the walls. If you do not have a stud finder or an old man, I provide some tips below in guessing where the studs are.
A Pencil, which is ridiculously hard to find around my house in this day and age.

Installation

Hanging a shelf is taught in Adulthood 101. We are hanging a really big shelf, but it is not much harder than that.

Finding the studs

Finding the studs was my biggest challenge. My house has this crazy cement board instead of typical drywall that is in most houses, so a stud finder does not work. Which means I had to guess. Studs are typically placed 16" apart (or 24" in some houses) on the center, meaning it is 16" from the center of one stud to the center of the other.

Outlets are attached to studs when the house is built. Find an outlet on the wall and on either side of it is likely a stud. The first stud will likely be in one of the corners. Measure from the corner of the wall to the outlet, and the distance between the corner to one side of the outlet should be about 3/4" off of a multiple of 16. If that is not the case, then try measuring from the other corner. If neither of them are, then just try drilling a hole with the smallest drillbit you can find on either side of the outlet to see if you can find the stud. Once you find one stud, it is pretty easy to find the rest.

From there, measure 16" and try drilling another small hole. If you hit another stud, then your studs are 16" apart. If not, then try 24". I ended up with about 6 tiny holes in my wall just trying to find the studs. They are easily filled and painted over.

Getting the height right

After I found the studs, I was ready to hang the brackets. But first I had to figure out how high to hang them. Which means I had to decide how high I wanted my desk to be. Every standup desk I have ever used felt too low, so I wanted to aim high. My first attempt I aimed too high.

standupdesk.com has some measuring tips. In the end, I found their suggestion of 1" below the elbow to be just right for me. My desk ended up at 45".

After hanging the brackets on the wall, place your desktop on them and try it out for a day. If this is your first experience with a stand up desk, your first day will probably feel uncomfortable in your legs and lower back. If it is uncomfortable in your shoulders and upper back, then the height is probably not right.

Hanging the desk

I wanted my desk to be at 45", so I subtracted 1 1/2" for the thickness of my desktop, and another 1 1/2" for the distance from the top edge of the bracket to the first screw hole and I drilled my first pilot hole. I measured over in multiples of 16" to where I wanted the next bracket and marked it with a pencil, then used the level to make sure it is even with the first pilot hole.

I attached the brackets with just one screw and laid my desktop to test the height. I worked at the desk for a day like this to make sure I liked the height. The next day, I drilled the remaining pilot holes and screwed the brackets into the desktop.

Happy ending

I am thrilled with how the desk turned out. It is extremely stable and takes up considerably less space than my previous desk.

Wasting Time

2012-01-06T17:30:19-05:00

I remember the day I bought TextMate. Like everyone, I loved TextMate. I loved it so much that I procrastinated learning anything else while I waited for TextMate Forever to be released. The new release that fixed everything was going to be out any day, so why should I waste time learning something new?

Why should I waste time learning something new? That question echoed in my head when I explained to a friend why I hadn’t tried another editor yet. Why should I waste time learning something new? Since when is learning something new a waste of time? I can think of many ways I have wasted time, and none of them involve learning something new.

72 hours. That’s how long I wasted watching four and a half seasons of 24. I will never get those hours back. Now that’s a waste! It was not relaxing (it was actually quite stressful and anxiety inducing), it wasn’t intellectually stimulating and I didn’t learn anything. Fortunately I came to my senses and stopped watching it.

From now on, when I waste time, I’m going to try to do it by learning something new, even if it’s something I won’t ever use.

On a related note, next week I will be blogging about my experience with learning Sublime Text 2

GitHub Acquires Ordered List

2011-12-05T13:02:46-05:00

I am thrilled to announce that I am now a GitHubber. Check out the official announcements on both the GitHub and Ordered List blogs.

When I joined Ordered List, we had grand plans of building a great company on the back of some amazing products. GitHub is already doing that, so when the opportunity came to join forces with them, I had almost no hesitation.

Personally, I look forward to bringing the products we built and the experiences we gained building them to make GitHub even better. GitHub has an amazing team and I cannot wait to inspire and be inspired by them.

Tunneling to Production

2011-12-01T17:17:29-05:00

One of the great features of SSH is its ability to tunnel any port to a remote server. I’ve used this SSH command often to connect to a production database:

$ ssh -N -L 27018:localhost:27017 example.com

That SSH command takes all traffic to port 27018 on your local host and tunnels it to 27017 on the remote server.

If you rarely use this command, it’s easy to forget it. So why not create the tunnel directly in the scripts that need it using net-ssh-gateway? We’ve been using this on Gaug.es and it works great.

require 'net/ssh/gateway'

gateway = Net::SSH::Gateway.new(host, username)
gateway.open('127.0.0.1', 27017, 27018)

db = Mongo::Connection.new('127.0.0.1', 27018).db('myapp_production')
# …

HAML: the unforgivable sin

2011-12-09T10:16:19-05:00

The other day I posted this on twitter: “there are many mistakes I’m willing to forgive, but using haml is not one of them”. Many people replied asking me to explain why I don’t like it, so here it goes…

I have used HAML on several client projects, and every time it drives me nuts. While there are certainly some things that are nice about it, overall it is a net loss.

For abstraction’s sake

Abstractions are a beautiful thing. The goal of an abstraction is to reduce or factor out details that you don’t care about. By removing details, an abstraction allows you to focus on the problem at hand and not the underlying implementation.

But an abstraction fails when it doesn’t remove the details. HAML is purely an abstraction of syntax. I still have to think about tags, ids and classes while I’m writing HAML. It simply provides a more concise syntax.

So you hate SASS and CoffeeScript too?

No. While SASS and CoffeeScript do abstract the syntax of CSS and JavaScript, they provide other great features.

SASS (or even better, SCSS) is a great abstraction because it enables features that would be nearly impossible to maintain by hand. It makes it easier to break your styles up into small, reusable components, and encourages good design patterns.

Likewise, CoffeeScript adds “features” on top of JavaScript that are extremely cumbersome to code by hand. CoffeeScript guards against common coding mistakes, like ensuring variables are properly declared within lexical scope.

Easy to write, hard to read

I agree that HAML is easier to write than HTML, simply because it involves less typing. But I feel it is infinitely harder to read. While the indentation makes it easy to see the nesting, the extremely overloaded synax requires careful attention to each line.

If HAML provided some value other than saving a few keystrokes, I would likely become a believer. But it doesn’t.

Typing is not my bottleneck in coding. Thinking is. And forcing my brain to parse HAML when it is occupied by more important things is not conducive to productivity.

Startup Fatigue

2011-11-21T21:50:29-05:00

People in West Michigan are doing startups completely wrong. I’m sure this is relevant elsewhere, but I can only speak for the context in which I live.

The problem with the startup fad is that people like Rick Devos are throwing money at people that have only sold them on an idea and have no history of execution. I’m sure the presenters at 5×5 have a lot of interesting ideas. I’ve heard many good ideas in the past year for business that I would gladly give my money to if they existed. But they don’t, because nobody has been able to execute on the idea. What the startup culture doesn’t understand is that ideas don’t matter. Execution matters.

In his now classic post, Derek Sivers succinctly articulates that ideas are just a multiplier of execution. Or, as Steve Jobs put it, “the problem with that is that there’s just a tremendous amount of craftsmanship in between a great idea and a great product.”

If you want to fund the next great software company, then take an idea—any idea because it doesn’t really matter—and partner with someone that can execute. There are several great software companies in the area that have a history of execution. Pick one and give them enough resources to deliver.

Update: A few people have taken the previous paragraph as me saying “contract” with one of these companies. I don’t mean that at all. I literally mean partner with them. Find a company that’s interested in your idea, give them an equity stake, fund them, and help them find people that can execute in areas that they can’t.

Sinatra and SSL

2011-11-17T16:23:27-05:00

I recently rolled out SSL support on Gaug.es. Gaug.es is a fairly simple Sinatra application. Most of the app requires authentication, but there are a few public URLs that are unauthenticated.

There two essential steps in securing an app which requires user authentication and then gives them unlimited access to their data:

Only set and use session cookies over secure connections
Mark all cookies as Secure so the browser doesn’t transmit them when requesting a non-secure URL

I evaluated a couple options, but quickly settled on Josh Peek’s simple rack-ssl gem. Rack::SSL is a middleware that just redirects http:// requests to https:// and marks any cookies set by your app as secure.

I wanted to create a clear separation between the parts of Gaug.es that require authentication and those that don’t to avoid accidentally leaking cookies (yeah, that’s a technical term), so I moved all the non-authenticated routes to a separate Sinatra application and insert it into the main app. Here’s essentially what I ended up with:

class Application < Sinatra::Base
  set :ssl, lambda { !development? }

  use PublicApplication
  use Rack::SSL, :exclude => lambda { !ssl? }
  use Rack::Session::Cookie, :expire_after => 1.week, :secret => '…'

  get '/dashboard' do
    'Secure!'
  end
end

class PublicApplication < Sinatra::Base
  get '/' do
    'Hello world'
  end
end

Why another queuing library?

2011-10-19T22:06:26-04:00

Along with the release of Speaker Deck, we released Qu, a Ruby library for queuing and processing background jobs. Why another queuing library? I briefly explain why I wrote Qu in the README, so read that first. But I wanted to give a little more of the backstory.

Reason #1: Resque issues on Heroku

Speaker Deck makes heavy use of background processing to turn PDFs of slides into the gorgeous player that you see on the site. Speaker Deck was using Resque, but unfortunately there a few issues with Resque on Heroku (#368, #319). The mishandling of signals described in #368 was especially an issue for us, because any time we deploy or scale the workers up or down, we would lose jobs; they wouldn’t show up as failed, they would just completely vanish.

So the logical approach would be to just fork resque, reverse the signals, and use that version, right? If that was my only issue with resque, then yes, but…

Reason #2: Infrastructure dependencies

At Ordered List, all of our products are built on top of MongoDB. Both Harmony and Gaug.es already have infrastructure set up to properly scale, replicate, and back up MongoDB. Both apps currently use resque minimally. Gaug.es literally only uses it to send email in the background since everything else is done in real-time. Harmony uses it slightly more.

So to run resque, we have to administer 4 additional Redis instances (master and slave in both production and staging) for both products. If we were at a scale where we needed Redis, the dependency would be warranted. But mongo is currently more than capable of doing what we need.

And I know we’re not alone. I’ve had many consulting clients in the past on shoe-string hosting budgets that didn’t need the robustness of Redis. delayed_job, were it more stable, was perfect for them.

Reason #3: Cheap thrills

Having been the primary maintainer of a popular fork of delayed_job for a couple years, I feel like I formed a lot of ideas about how I would build it differently. I made a lot of mistakes with delayed_job (let’s not talk about YAML), but I also learned a lot of lessons.

Supporting multiple backends is one of the most important things I think Qu can provide. Resque has a thriving ecosystem that is all built around Redis. As I said in Reason #2, Redis often makes sense, but there are times when it doesn’t. I find it extremely disappointing that the flourishing Resque ecosystem can’t be leveraged with another database backend. I think Qu can create an ecosystem with the benefit of being able to use it in every case.

One criticism of this approach is that databases have different features and strengths, so it will be impossible to make the backends behave the same. Each backend will certainly have it’s quirks, but fundamentally, I think they can all behave the same. The current Redis and Mongo backends are perfect examples of this. Internally, they both operate very differently, but their API is the same and they both pass the same specs.

So far, I’m very pleased with how Qu is turning out. There is still a lot of work to be done, but I think I have a good start and have been pleased by the intrest in it already.

bundle exec fatigue

2011-09-15T10:15:07-04:00

Tired of typing bundle exec rake? Me too. I even aliased it to bx, but that still breaks autocomplete and annoys me in subtle ways. So this week I finally remedied that situation, and thought I would share my setup since I haven’t seen anyone else with as nice of a setup.

First, bundler has a handy feature to install stubs for all of the bins in your gem dependencies.

$ bundle install --binstubs

This is great, but it creates them all in a bin directory in your project, so not only do you have to type `./bin/rake`, but you also have to ignore them from git. I’m lazy and that’s too much work!

The Final Solution

My bundler setup has 3 essential components: 1) an alias for bundle install which always installs bin stubs into .bin, 2) adding .bin to PATH, and 3) globally excluding the .bin directory from all git repos.

First, put the following in .bashrc, .profile, or .bash_profile to alias bundle install and add .bin to your PATH. :

alias bi="bundle install --binstubs=.bin"
export PATH="./.bin:$PATH"

Now every time you run bi, it will install the gem dependencies and add the bin stubs in your path so you can just run rake like the good ol’ days. Now all you need to do is create a global ~/.gitignore that always excludes .bin from every project, and tell git to use it.

$ git config --global core.excludesfile ~/.gitignore

This approach has been working wonderfully for me. Now I just need to get out of the habit of typing bx.

Update: Travis Tilley pointed out rubygems-bundler in the comments. It looks like another unobtrusive (and more secure) way to accomplish this same thing.

Sortable lists in key-value data stores

2011-09-06T12:43:45-04:00

Note: This post contains illustrations that may not appear properly in your feed reader. You may want to view the original post in your browser

Last week I blogged about using Git as a key-value database as I experiment with using Git as a database backend for an application.

One of the first challenges I encountered is not unique to Git, but to key-value stores in general; allowing users to manually sort a list of items without the niceties of atomic operations.

The most common way to allow users to manually sort a list is simply to store a position along with each record. When the user reorders the records, then you update the position on each record. This works well as long as the data set is small and there are guards in place to prevent two users from sorting at the same time.

However, with large amounts of data or a data store that doesn’t support atomic operations on multiple records, this doesn’t scale well. And specific to Git, updating every single record pretty much guarantees a merge conflict when the Git repos are pushed to a common origin.

I needed a simple way to create a manually sortable list that avoided merge conflicts. Preferably, a manual reorder would only update the one record whose order changed, and leave the others untouched.

Floats between 0 and 1

The solution I arrived at is to specify order as a float between 0 and 1. When you move an item, it’s position becomes a float that is half-way between the item before and the item after it. Items moved to the top of the list will get closer and closer to 0, and items to the bottom of the list get closer to 1.

Let me illustrate with some examples. The first item added to the list will get a position of 0.5, half-way between 0 and 1.

Item A position: 0.5

The second item added will be half-way between the last item, 0.5, and 1.

Item A position: 0.5
Item B position: 0.75

The third item added will be half-way between the last item, 0.75, and 1.

Item A position: 0.5
Item B position: 0.75
Item C position: 0.875

Now, if the user wanted to move Item C in between Item A and Item B, then its new position would be half way between the position of the item before and after it.

Item A position: 0.5
Item C position: 0.625
Item B position: 0.75

If the user wanted to move Item B to the top of the list, then its new position would be half way between the first item in the list and 0.

Item B position: 0.25
Item A position: 0.5
Item C position: 0.625

And so on and so on…

This solution allows the user to reorder items without updating every record in the data store. As more items get added to the list and moved around, the position just becomes more and more precise. There is a theoretical limit to the number of times items can be moved, but it is very high.

Git: the NoSQL database

2011-12-09T10:27:07-05:00

We all know that Git is pretty amazing. It’s fast, reliable, flexible, and it keeps our project history safely nuzzled in it’s cozy object database while we sleep soundly at night. But I’m curious to see if it can be used for more than code. I’ve had a few apps in the back of my mind for a while now that would be really interesting if the data was stored in Git.

If only there was an easy way to read and write a Git repo from Ruby…

Toystore & adapter-git

Toystore is an ActiveModel-based object mapper for key-value data stores. The beauty of Toystore is that it doesn’t care what the backend is. It uses Adapter to abstract the connection to any data store that can set, get, and delete keys.

Well, Git is a key-value store; it supports set, get and delete on keys (a.k.a. paths). So I sat down with Scott Chacon’s Git Internals Peepcode PDF and put together adapter-git, built on top of Grit.

Now I can create pretty models that are stored in Git.

class Item
  include Toy::Store

  store :git, Gaskit.repo,
    :branch => 'content',
    :path   => 'items'

  attribute :title,       String
  attribute :description, String
  attribute :created_at   Time, :default => lambda { Time.now }
end

Toystore uses conventions that will be familiar to anyone that has used Active Record or MongoMapper.

item = Item.create!(:title => 'Git: the NoSQL database')
item.update_attributes(:description => "OMG this is awesome!")

The biggest difference is that you can’t “find” records. The data stored in a key-value store is opaque, so all you can do is get it by key.

item = Item.get!('3FB053FA-0A3B-4903-9CE0-2A8A964E0F37')

Caveats

I have no idea if Git will work as a data backend for an application. I’m sure GitHub has solved many of the problems with concurrency and scaling a filesystem.

There is still a lot of room for improvement on adapter-git too. Here are just a few things I’d like to add soon:

Locking – I wouldn’t want to use adapter-git for anything with a lot of concurrency at the moment. Git commits are atomic; you’ll never corrupt a repo by a failed commit, but if you have a lot of concurrent access, you might loose commits.
Custom commit messages – Currently adapter-git just uses the ID of the key being set in the commit message. In the app I’m experimenting with, I’ve already had a desire to set custom commit messages.
Update working copy – adapter-git currently just works against the git repo itself. It doesn’t update your working copy. So `git status` will currently tell you that you’ve deleted files from your working copy after you update records with adapter-git.
Merge conflicts – I’m looking forward to being able to programmatically resolve merge conflicts. Riak has a cool pattern for resolving conflicts on read, so I’d love to see if I can build something into adapter-git and toystore to work in a similar fashion.

Check out adapter-git on GitHub and try building an app backed by Git!

Pusher notifications with EventMachine

2011-08-30T19:31:24-04:00

Gaug.es has been a fun exercise in building an app to scale from the start. John Nunemaker previously posted about our quest to send live updates to the browser without slowing down the request cycle.

Backstory

We use Pusher to send realtime updates of stats to the Gaug.es dashboard. We started out on Heroku–which uses Thin, which runs on EventMachine–so we could just use EventMachine and Pusher’s trigger_async method to queue the Pusher update on the next tick, thus not affecting response times. But then we moved from Heroku to RailsMachine and their Passenger stack, so we wanted to find a way to keep the Pusher update out of the request cycle but still “realtime”.

So John devised a genius plan of starting up a thread in Passenger that runs EventMachine. We can still use the trigger_async method and take advantage of EventMachine’s IO goodness, and only suffer hits in response time occasionally when Ruby’s naive thread scheduling interrupts the passenger process in the middle of a request.

It didn’t did work

This ran great in production for a few weeks. But as our traffic started to increase thanks to launching several new features, we started to run into problems. Passenger processes frequently started hanging and gobbling up ungodly amounts of CPU. The only way to recover was to snipe the processes with the friendly kill -9.

So we tried to concoct a theory: since most of our hits are ridiculously fast (5-10ms), and we were running multiple threads but weren’t doing anything fancy to set priority or control scheduling, the passenger thread must be handling requests faster than the EventMachine thread can send them to Pusher, causing the Event Machine thread to get backed up to the point where it has so many IO connections open that it thrashes the CPU.

It turns out our theory was wrong (we were actually hitting a crazy bug with the regex engine in all major versions of Ruby), but we didn’t know that at the time. So we concluded that we had to take the Pusher notification out of the request cycle.

Redis and EventMachine to the rescue

With help from Chris Gaffney, I came up with a different approach that pushes hits on a Redis list and uses a separate EventMachine process to send them to Pusher.

In our Sinatra application, we just push JSON from the notification onto a list.

Gauges.redis.rpush('notifications', notification.to_json)

Redis list operations are fast, even with a decent-sized blob of text. Now our app doesn’t have to worry about the overhead of sending the notifications.

Next we needed background process to process the Redis list and communicate with Pusher. EventMachine is well-suited for all of this IO, so we wrote this simple script:

module Gauges
  module Pusher
    def self.redis
      @redis ||= EM::Hiredis.connect("redis://localhost:6379")
    end

    def self.next
      redis.blpop('notifications', 0).callback do |list, data|
        notification = JSON.parse(data)
        channel = notification.delete('channel')
        ::Pusher[channel].trigger_async('hit', notification)
        EM.next_tick(&method(:next))
      end
    end
  end
end

EM.run do
  Gauges::Pusher.next
end

It uses em-hiredis to connect to Redis and run a blocking left pop, which will wait until there is an item on the list and then send the notification to Pusher with trigger_async. It then re-schedules itself to run on the next tick, causing it to continuously loop.

This has been working really well for us in production for a couple months now. The biggest disadvantage currently is if the process ever dies and notifications start to build up, Pusher will get inundated with requests form us because it will send them as fast as Redis can pop them off the list. We will eventually look into adding some rate-limiting.