Software By Josh

Fault Tolerant MongoDB on EC2

2012-03-11T00:00:00-08:00

Fault Tolerant MongoDB on EC2

While working on a project at Bizo I needed to connect a Rails app to a MongoDB backend both of which run in Amazon’s Cloud (EC2). At Bizo we have a policy to not use non Amazon services when possible (to limit risk) – so we normally run most of our services straight off of EC2. I’d like to share what I’ve learned as best practices throughout the experience as I hope it might save some time and frustration for others.

Primer

Replica sets are the preferred way to run a distributed, fault tolerant MongoDB service. But as with any distributed system, nodes will eventually fail. Now replica sets are pretty good at handling failures, but they can’t save you if too many nodes fail.

Specifically a replica set requires a minimum of two nodes to function at all times (1 primary and 1 secondary node). Thus a good rule of thumb is to run at least 3 nodes in a replica set, that way if a node fails your database service doesn’t go down with it. The Rails app I was working with doesn’t experience enormous amounts of traffic so 3 m1.large (64bit) nodes were sufficient for my needs. What follows is a rundown of our setup and how it handles common needs of fault tolerant systems.

Best Practices

Minimize Failure with AutoScaling, Availability Zones, CloudWatch and EBS Volumes

Use Autoscaling Groups, CloudWatch and EBS Volumes to replace failed nodes as soon as they go down. Since we run three nodes, our replica set is insulated from failure due to a single node crashing. But if two nodes crash the replica set goes with them. To solve this we use Cloudwatch alarms to trigger the Autoscaling Group whenever a node goes down – that way a new replacement node is automatically brought online within a few minutes of a failure to reduce the risk of nodes sequentially failing. Additionally each node stores it’s data on an EBS Volume (network attachable hard drive) – that way when a node fails, it’s replacement doesn’t startup with missing data – it simply mounts the previous node’s EBS.

To protect against multiple nodes failing simultaneously run each node in a separate availability zone. The above isn’t sufficient to protect against things like hardware failures as all 3 instances could wind up on the same hardware. Running each node in a separate availability zone guarantees that our mongo instances run with a reasonable amount of separation (eg. they don’t all end up on the same hardware box). Ideally you’d run each node in its own region (separate data center), but this causes headaches trying to configure firewalls as Amazon does not allow security groups to be used across multiple regions (see security below). So unless you want to setup a VPN for cross region communication – you’re probably better off just running in separate availability zones.

Assuming you’ve created the group and are running three nodes, each in a separate availability zone you can configure the auto scaling group using Amazon’s command line tools like so:

as-update-auto-scaling-group my-mongo-service \
--region us-east-1 \
--availability-zones us-east-1a us-east-1b us-east-1c \
--max-size 3 \
--min-size 3 \
--desired-capacity 3

Now if any node fails then a new one will startup to take its place in the proper availability zone.

If Everything Fails, have backups

It’s always good to have backups just in case something really bad happens. Fortunately since we use EBS Volumes this is really easy – we create nightly snapshots of the primary node’s EBS Volume on our cron server using Amazon’s command line tools (ec2-create-snapshot). These snapshots are persisted to S3 and we can easily restore our replica set from these backups.

Use Elastic IPs

As nodes fail and are replaced you want both your Replica set and your database clients to be able to find and connect to the new nodes. The easiest way to do this in Amazon is to use Elastic IPs – special* static ip addresses that can be assigned to individual instances. Since each instance runs in a separate availability zone we just need one Elastic IP per zone. When a new node starts up to replace a failed instance, it checks which zone it was started in and assigns itself the matching Elastic IP. Both the client and replica set configuration should point at the Elastic IP address – that way failures and startups of new nodes will be seamless to your app. This is because cross-security group openings in the firewall need to use internal (not external) addresses and the DNS url in the console will resolve to an internal ip address from an EC2 instance.

Security

This is where the headaches can start. Ideally you want to restrict access to your MongoDB instances to just your client application using Amazon’s security groups. The way we normally set this up is to give your mongo instances a security group, say mongo-db-prod and your client app a security group, say cool-app-prod. Then mongo-db-prod would grant access on port 27017 (default mongodb port) to security group: cool-app-prod. Unfortunately what’s not documented very well is that if you use the external Elastic IP addresses in your configuration it will not work with security groups! Instead you have to use the Elastic IPS DNS url (found in Amazon’s web console) for security groups to work properly.

A Final Caveat

One thing to be careful of is if you require more than 5 nodes in a replica set you’ll run into a problem using Elastic IPs – Amazon by default only allows 5 eips per region. You’ll need to either ask Amazon to increase this limit on your account or seek out an alternative setup.

Well, that’s it but If you have another setup for running MongoDB on EC2 I’d love to here it. Until next time.

Design Pattern: Threaded Requests with Retries

2012-02-29T00:00:00-08:00

Design Pattern: Threaded Requests with Retries

Imagine you have an amazing idea for the next great photo sharing app with Facebook integration. Never mind that there are already 10,000 more photo apps than the world needs right now – to hell with the haters. You will make millions, people will love you and the world will be your oyster (err..figuratively speaking).

But before you can ascend to the peak of photo sharing world domination – you need to build the app. Naturally you’ll need to communicate with Facebook, so you’ll probably wind up using their Graph API along with some wrapper library for your favorite programming language. Let’s assume you write the app in Ruby on Rails and you have some really simple Ruby class that wraps the api, like so:

class FacebookApi
  def initialize(auth_credentials)
   # authentication stuff
  end

  def upload_photo(image_data)
    # logic to upload a photo to a user's stream
  end

Great, your app can upload photos to Facebook now. But you can’t just go running around willy-nilly calling the upload_photo() method – everybody knows that networks are unreliable and can’t be trusted to work all the time. Some of those photo uploads are bound to fail and you need to handle that somehow.

If at First You Don’t Succeed, Try, Try Again

Lucky for you Facebook is a pretty reliable service – any failures in your api requests are likely going to be fleeting – simply retrying a failed upload a few times will probably work 99% of the time. Of course you want to wait a bit before retrying, eg. maybe an api method stops responding for a minute or two, or you temporarily lose your network connection.

Ideally we’d do this work in a background thread so we don’t block the ui and we’d have some kind of exponentially increasing delay between retries (because by the 3rd retry the chances of a 4th retry succeeding are minuscule).

Generalizing

Gee this situation seems familiar doesn’t it? This kind of scenario (at least for me) crops up all the time in web apps – ex. make a call to some webservice, if it didn’t work try again a few times and if that doesn’t work log something useful and give up. It’d be great to have some reusable design pattern for this type of situation. Fortunately this turns out to be pretty trivial to do in Ruby (or most languages actually).

I’ll show you a simple example of what this might look like in Rails and break down what’s happening below the code:

# lib/threaded_api_call.rb

class ThreadedApiCall
  def initialize(options, &block)
    execute_in_thread(DEFAULT_OPTS.merge(options), &block)
  end

  def join
    @thread.join
  end

  def result
    @result
  end

  private

  def execute_in_thread(opts, &block)
    @thread = Thread.new do 
      attempts = 0
      wait_time = 1

      while (attempts < opts[:max_attempts])
        begin
          attempts += 1
          @result = block.call
        rescue Exception => e
          opts[:logger].error "request failed for: #{opts[:name]}"
          opts[:logger].info "will retry in #{wait_time} seconds" if attempts < opts[:max_attempts]
          sleep wait_time

          # exponential backoff between retries
          wait_time *= opts[:wait_interval]
        ensure
          opts[:logger].flush
        end
      end
    end
  end

  DEFAULT_OPTS = {
    :name          => "generic api request",
    :max_attempts  => 2,
    :logger        => Rails.logger,
    :wait_interval => 2 # seconds
  }

Basically we just create a class that takes in a hash of options and a block (similar to a function pointer or lambda in other languages – if you’re unfamiliar with Ruby the differences are immaterial for this example). When the class is initialized the block is executed inside of a new thread (to avoid blocking our main thread), if anything goes wrong we simply try again with an exponentially increasing cooldown interval (options[:wait_interval] is the base) up to a maximum number of retries (options[:max_attempts]). Finally if we exceed the maximum retries, we give up and log everything to options[:logger] which defaults to the Rails.logger (make sure to remember to always call flush!).

If for some reason you need to stop the main thread and wait for results to come back, you still can by calling join (just like a normal Ruby thread) and then call result().

Testing

Since we keep our concurrency and api integration code separate this ends up being trivial to test. For our ThreadedApiCall class all we have to do is use a mock/spy or a simple counter. For example:

  describe ThreadedApiCall do
    it "executes the block passed to it one time on success" do
      test_method = mock() 
      test_method.expects(:call).with(:true).exactly_once

      ThreadedApiCall.new :name => :test_api do
        test_method.call(true)
      end.join
    end

    it "retries the block passed to it if failed the first time" do
      num_calls = 0 

      api_call = ThreadedApiCall.new do
        # pretend we fail on first try
        if num_calls == 0
          num_calls += 1 
          raise "FAIL!"
        else
          num_calls += 1
          num_calls
        end
      end

      api_call.join
      api_call.result.should == 2
    end

    # etc for other test cases 
  end

Using and Abusing

The great thing about this approach is when you add Google+ and Flickr integration into your amazing web app you can easily reuse this class to wrap your API calls – so you get threaded retries, logging and exponential backoff all for free!

Building a Product in Just 8 Hours

2012-02-17T00:00:00-08:00

Building a Product in Just 8 Hours

Cross posted on the Bizo Dev Blog

Recently at my employer, Bizo, we decided to try a new kind of hack day. Previously during hackdays our engineers worked individually on their own project(s). But on our last hack day we decided to try something new – The 8 Hour Product Challenge.

We would build and launch a completely new product in the course of a normal workday (9-5pm). “Launching” meant this product had to be running publicly on the internet by 5pm – no excuses, no “Wait! I need 5 more minutes” – whatever was there had to be deployed. In short the experience was fantastic and I can’t wait to do it again. Here’s a breakdown of the experience:

Initial Meeting 9:30am

Organizing developers for a meeting of any kind is like trying to heard cats. But if it’s a meeting before 11:00am you’re not herding regular cats, you’re herding sleepy, fat cats with one leg and half an ear. Somehow after a lot of cattle prodding by our VP of engineering our team eventually managed to shuffle its way into the conference room like the decaffeinated zombies we were and get to work.

We decided to build a stealth product. The system would use Bizo’s rich business data to personalize special content for visitors based on things like their industry, company size and seniority. The goal for the end of the day was to have a small webapp up and running on Amazon’s servers.

After a bit more discussion, we decided to split tasks up into five groups of two engineers:

Data Discovery Team – Find relevant items for users by using our B2B business data & network.
Scraping Team – given a url representing an item scrape the page contents and store them for later use
Data Classification Team – extract relevant data from the HTML source of the previously scraped urls.
Backend Team – backend architecture for the webapp that fetches serves the content from that was generated in the previous steps
Frontend Team – frontend design + javascript that makes the app functional

Engineers were assigned more or less randomly, with the exception of myself – I was assigned to the frontend team directly.

During the course of our initial planning meeting, we (myself included) often found ourselves becoming sidetracked with feature bloat, premature scalability concerns and a myriad of other things not essential to our MVP. Fortunately one of my coworkers (Stephen) was smart enough to enforce timeboxing the meeting to one hour – eventually we got things back on topic and designed the critical components before time ran out.

Start Work 10:30am

Once the teams were assigned, we all jumped in and started to work on our relevant tasks. My partner, Darren and I immediately started out by sketching ideas on paper for our design. I can’t stress enough how important sketching is for being able to rapidly prototype a product – a trick I picked up back when I interned over at ZURB. Only after we had some solid sketches did we move into Photoshop mockups. Meanwhile the other teams were all furiously programming their parts of the application:

Data Discovery Team Worked out a simple ranking algorithm for data and started writing the Hive script to extract it.

URL Scraping Team Started out in Scala hacking up a script to scrape & download url content.

Data Extraction Team Decided to try out the Pismo gem to extract summaries and titles from scraped html content.

Backend Team Was working on getting a sweet Scalatra webapp up and running

Lunch Time & Status Updates 12:30pm

By lunchtime everything seemed to be coming along nicely. On the frontend had completed our Photoshop mockup and had just began writing some basic css styles. All the other teams reported making good progress on their tasks, with no major snags in the foreseeable future (betcha you never heard that one before…).

Afternoon 1:30pm

My frontend partner and I powered through our post lunch food coma and were able to move into begin wiring up the ui using CoffeeScript in conjunction with Dependence.js. My teammate and I decided to give pair programming a shot. He has always been more of an Emacs kind of guy, while I prefer vi, but in the interests of learning I decided to try Emacs for the rest of the day – I now know why he’s always so worried about contracting carpel tunnel syndrome :).

Using some sample data generated by the first three teams we were able to get a rough ui working pretty quickly. Our side of things turned out to be pretty straightforward and involved three AJAX requests. One was to retrieve a list of items grouped by segment from the Scalatra web server, one to get a list of the current targetable segments from the Bizo API and another call to the API to retrieve a visitor’s business segments (bizographics).

The only snag we hit was running into a race condition – originally we attempted executing all three requests simultaneously. In reality we had to wait to get the list of segments before getting the visitor’s profile. Darren and I just looked at each other and shrugged, then we indented the third API request a few spaces in our CoffeeScript code – race condition solved! Yes that’s correct we fixed a race condition by indenting some code, don’t judge – it was a hackday.

The other teams all seemed to be doing well, the scraping team discovered the Scala Collection’s magical par(), which turns normal data structures into parallel ones, they almost peed their pants with joy. At this point the backend team had completed the Scalatra app and was working on setting up our eventual deployment to EC2 using our custom infrastructure, cowboy.

The Home Stretch & Deployment 4:30pm

Right around 4:30 we ran into a major problem. There had been a miscommunication regarding the necessary format of the JSON file needed by the frontend, and our data was coming through to the app in a format that just wouldn’t work. We had to scramble and hash things out with the other teams quickly before we hit our 5pm deadline. Thanks to a major push by the Data Extraction Team we were finally able to get everything in place. The product worked! – it wasn’t the most polished app ever, but what we had accomplished in just one day was pretty amazing. The product was deployed on EC2 and presented internally within Bizo – it was met with a lot of excitement and Bizo will probably be releasing it publicly in the weeks to come.

Closing Thoughts

Overall the experiment turned out to be a smash hit. Looking back on the experience there are a bunch of things we could have done better. In retrospect we got sidetracked a bit too much on non essential feature ideas when we really should have been spending time clarifying the format of the data as it passed through each team’s project – this was something that came back and bit us near the end of the day. But mishaps aside it’s really amazing what you can accomplish with 9 other talented people in a single day.

I can’t recommend trying this with your team enough – what do you think, is your engineering team up for The 8 Hour Product Challenge? If you’re interested in the technologies we used throughout the day, see below.

Appendix, Technologies Used

News Discovery Team – Hive, Amazon EC2, Amazon S3
URL Scraping Team – Scala
Data Extraction Team – jRuby, gems of note: pismo, right_aws (for Amazon S3)
Backend Team – Scalatra, Amazon S3
Frontend Team – Photoshop, HTML5, Sass, CoffeeScript, jQuery, dependence.js

Stop Wasting Time Setting Up Frontend Projects

2012-01-29T00:00:00-08:00

Stop Wasting Time Setting Up Frontend Projects

Setting up a new project and its dependencies in most languages is a pain, but particularly so on the frontend where you have lots of separate tools like CoffeeScript, Sass, HTML5 and various 3rd Party JS libs like jQuery all (supposedly) working together. A lot of programmers I’ve met dismiss the time wasted while setting up a new project as a “one time fee”. But in reality most programmers set up new projects all the time – work projects, projects for fun, side projects etc. We shouldn’t be repeatedly wasting time setting up a new project when we could be writing something relevant instead. This kind of problem is ripe for automation.

That’s why tonight I hacked together a little Ruby script Frontend-Launchpad to setup new frontend projects with all the bells and whistles I normally use. While most of the tools it sets up are my own – I’m a eat my own dog food kind of guy (figuratively, not litterally) – you could easily modify the script to suit your own needs.

Features

Out of the box, you get the following:

CoffeeScript + Sass

Frontend-Launchpad generates a build script for you that compiles your .coffee and .sass files. CoffeeScript files are compiled and minified using Dependence with no setup required beyond installing the gem.

Optional Modules

The launchpad script will optionally install the following modules into your new project:

1. jQuery the dom library we all know and love
2. Raphy Charts beautiful HTML5/SVG/VML charts, if I do say so myself
3. StateMachine.js easy custom events finite state machines
4. JasmineTests superfast command line Jasmine tests with PhantomJS
5. PrettyPieces like Twitter Bootstrap, but for Sass

Setup

Installation should be really easy, just grab the script from the link below and run the code below. Note* you’ll probably want to put this script on your path for easy reuse and tab completion.

git clone https://jcarver989@github.com/jcarver989/frontend-launchpad.git

cd frontend-launchpad 
./launch.rb foo_project 

# follow the prompts to install modules...

cd foo_project && ./build.sh

# Your shiny new project is ready to go

Get Frontend-Launchpad

Extension

This script only me a short time to hack together so if this script doesn’t suit your needs (ex you want to use Bootstrap), feel free to extend it – send me pull requests and/or fork it for your own usage. Whatever you do, don’t waste time manually setting up another frontend project manually again! Until next time, cheers.

Pirating Ruby Methods For Fun And Profit

2012-01-21T00:00:00-08:00

Pirating Ruby Methods For Fun And Profit

With all the SOPA coverage lately I wanted to post about software piracy (yarrrr!). But this post isn’t about the kind of software piracy you’re used to hearing about – it’s about pirating object methods using Ruby.

Sometimes while programming you might want to do what I call pirating a method – where you augment an object by borrowing a method from another module or class. Method pirating is fairly common to see in JavaScript code, especially to augment array-like objects like arguments which don’t have some useful methods that Arrays have, ex: Array.slice. Rubyists however aren’t quite the scallywags that JavaScript programmers are and don’t tend to do much pirating, but there’s times in Ruby where method pirating can be useful.

Including only what you need, nothing you don’t

Suppose that we have an extremely simple LinkedList implementation:

class LinkedList
  def add(value)
    return @head = Node.new(value) if @head.nil?

    current = @head
    while (!current.next.nil?)
      current = current.next
    end

    current.next = Node.new(value)
  end

  def each
    current = @head
    while (!current.nil?)
      yield current
      current = current.next
    end
  end

  def print
    each { |node| puts node.value }
  end

  private

  class Node
    attr_accessor :next
    attr_reader   :value

    def initialize(value)
      @value = value
    end

    # comparison operator
    def <=>(node)
      @value <=> node.value
    end
  end
end

Now imagine we’d like to add a map (collect) method onto our LinkedList. We could implement this ourselves, but the Enumerable module already has the exact method we want. There is no need to reinvent the wheel – so we should find a way to make use of it.

Option #1: Mixin Enumerable

We could easily add Enumerable’s map() method to our LinkedList implementation by simply including the Enumerable module into our class like so:

class LinkedList
  include Enumerable
  # rest of implementation
end

Since we already implemented an each() method, Enumerable’s map() methods will automatically work with our LinkedList class – nice! The downside to this approach is that a bunch of other methods we didn’t need came along for the ride like min(), max(), reduce() (inject) – in fact every instance method defined in Enunmerable is now a part of LinkedList.

Now in Enumerable’s case this isn’t a disaster – we only had to define one method (each) to make Enumerable work with LinkedList. But more complex mixins might require several helper methods to be defined to work with classes that include them. If we only want one or two methods from a module that don’t require extra helper methods – that could be a lot of extra work defining methods for functionality we don’t need.

Option #2: Pirating, Yarrr!

If you only want a few methods that a module provides, then a better option might be to pirate that method instead.

In my last post on writing DSL’s I showed how we could do this using instance_eval to change the execution context of a block. This seems like a reasonable approach to pirating methods as we could turn a regular module method into a block and then attempt to execute it in the context of the class stealing the method. Let’s give it a shot:

module Foo 
  def hi
    puts @words 
  end
end

class Boo
  def initialize
    @words = "boo"
  end 

  def hi
    method = Foo.instance_method(:hi)
    instance_eval &method
  end
end

boo = Boo.new
boo.hi

# wrong argument type UnboundMethod (expected Proc) (TypeError)

Uh-oh, looks like we wound up with an UnboundMethod, which isn’t associated with an object and thus uncallable. That’s ok though since in this case we’d like to bind our new unbound hi() method to the the Boo class so Boo.hi functions properly.

There is one issue though, UnboundMethods can only be bound to the same object type that they were created from. Ex. since we created an unbound method from Foo, we can only bind that method to another instance of Foo. Boo does not use Foo as a mixin so we’re currently unable to bind an UnboundMethod from Foo to an instance of Boo.

Fortunately Ruby allows to cheat. We can create a temporary Boo object that extends Foo without altering any other Boo instances using Object.extend. This then allows us to create an UnboundMethod that can be bound to a Boo instance even though it doesn’t mixin the Foo module!

Then all we have to do is bind our stolen method to the self variable in Boo and call that method.

class Boo
  def hi
    # temp class
    pirate = Boo.new.extend(Foo)

    # change execution context to this instance of Boo
    stolen_method = pirate.method(:hi).unbind().bind(self)

    # use our stolen method, prints "boo"
    stolen_method.bind(self).call
  end
end

We can easily apply the same technique to our LinkedList implementation to pirate the map() function off of Enumerable. Notice how we can pass parameters to call just like a normal method:

class LinkedList
  def map(&block)
    pirate = LinkedList.new.extend(Enumerable)
    stolen_method = pirate.method(:map).unbind.bind(self)
    stolen_method.call(&block)
  end
end

Generalizing

It would be quite tedious to have to pirate mulitple methods this way, but fortunately it’s relatively easy to abstract this technique into a module method that we can mixin and call inside our LinkedList class:

module MethodPirate
  def pirates(the_module, method)
    pirate_class = self.new.extend(the_module)
    stolen_method = pirate_class.method(method).unbind

    send :define_method, method do |*args, &block|
      stolen_method.bind(self).call(*args, &block)
    end
  end
end

class LinkedList
  include MethodPirate
  pirates Enumerable, :map
  pirates Enumerable, :inject
  # rest of implementation 
end

You do wan’t to be careful here since this implementation isn’t very performant, notice how we create a new object and closure everytime we call pirate(). So for production use you’ll most likely want to cache the the generated classes/unbound methods.

Wrapping Up

Hopefully you’ve learned something about how to pirate methods in Ruby, metaprogramming and how it pirating methods from modules might be useful in situations where you only want to mixin parts of a module (or group of modules). Questions, comments or ideas feel free to post.

How To Write Your Own DSL

2012-01-08T00:00:00-08:00

How To Write Your Own DSL

DLS’s or Domain Specific Languages have become quite popular over the past few years. Perhaps you’ve heard of DSLs before, but even if you haven’t chances are you’ve come into contact with one before. Ruby On Rails, for example can be seen as a DSL for writing web apps, jQuery is a DSL for manipulating the DOM and SQL is a DSL for writing database queries.

Why Bother With DSLs?

DSLs are useful tools – they allow us to easily express logic specific to a particular problem (domain) that would be otherwise difficult (or verbose) to write in another language. Usually this boils down that to using a grammar and syntax that more closely resembles the lexicon used by the target domain. For example a mathematician working with matrices doesn’t think in loops, iterators or arrays but instead thinks in terms of vectors, dot products and transformations. Using a general purpose language with only arrays and iterators would require a fair amount of mental gymnastics for our mathematician as he would have to mentally translate between his problem domain (matrices) and the language he writes code in (ex. c++). Using a DSL designed for matrix operations would eliminate this mental translation while simultaneously providing code that is both more terse and (hopefully) less error prone.

Types of DSLs

DSLs come in two forms – external and internal. External DSL’s exist independently from any other language, SQL is a good example of an external DSL. Internal DSL’s on the other hand live inside another programming language – for example Rails is an internal DSL which is hosted within the Ruby programming language.

Typically internal DSLs are easier to create but aren’t as flexible as external DSL’s. Internal DSL’s need not worry about parsing or grammars but must conform to valid syntax within the host language (eg. all Rails code is valid Ruby syntax), conversely an external DSL can have any syntax its creator wishes at the cost more work to build a parser and grammar (again think SQL).

One final thing to keep in mind is that internal DSLs allow you to take full advantage of the host language. Large applications often touch across multiple domains (database, business logic etc.) and passing data between multiple DSLs in a single host language tends to be much less of a headache than juggling multiple external DSLs – which is why I prefer internal DSLs to external ones.

Making Your Own DSL

At first glance DSLs can be daunting – but fear not they don’t require magical unicorns or superhuman powers. Actually they often turn out to be simple to create. This post will show you how to make your very own internal DSL using Ruby.

Picking a Domain

All DSLs start with a domain – so let’s pick something – say Amazon’s CloudWatch service. If you aren’t familiar with CloudWatch it allows you to store, retrieve and aggregate metrics pertaining to servers you run in their cloud (EC2) service. Currently Amazon provides a command line tool (mon-get-stats) for querying CloudWatch, but it’s fairly low level. Often times you want to aggregate metrics about servers running in multiple regions – the current tool only allows you to query one region at a time, so for multi-regional metrics you’re forced to manually merge the data. Having a nice DSL to help us view CloudWatch data across multiple regions would be great, so let’s get to it!

Where to Start

Often when writing a new DSL it’s helpful to start by imagining how we’d like the ideal syntax to look. To have a good idea of what this ideal syntax might be, we should examin our requirements:

We know we want to retrieve CloudWatch metrics from multiple regions.
We’ll want to look at multiple metrics from the server’s load balancer (abbreviated elb) instead of a specific server instance (instances tend to be transient, sitting behind load balancers which are available).
CloudWatch stores data for up to two weeks, so our DSL will need a way of expressing what time frame we’re interested in seeing.

Something like the following would be a great start:

  # Assume DSL defined above

  cloudwatch = CloudWatchClient.new

  # returns our aggregated data
  cloudwatch.stats do
    elb           "my-app-server"
    regions       :us_east_1, :us_west_1
    metrics       :request_count, :other_metric
    units         :count 
    aggregation   :sum  
    start_time    Time.now.beginning_of_day 
    period        1.hour.to_s
  end

Using our proposed syntax we instantiate an CloudWatchClient class whose stats() method serves as the entry point into our DSL. The above might seem a bit magical, but I promise it’s nothing crazy. We simply execute the stats method and pass in a block (unexecuted function, similar to a lambda in other languages). All the special syntax inside the block will turn out to be just plain ol’ ruby method calls.

Implementing

Let’s sketch out what our entry point class, CloudWatchClient will look like:

class CloudWatchClient
  include MergeData

  def initialize
    # authenticate with CloudWatch's api
  end

  def stats(&block)
    # generate query parameters (heart of our DSL)
    queries = Query.new(&block)

   # make cloudwatch api request(s)
    get_stats(queries)

   # merge the regional data 
   # from MergeData module
    merge_data(results)
  end

  private

  def get_stats(queries = [])
    # make necessary requests to CloudWatch's api
  end
end

Following best practices for object-oriented programming, our client class has one responsibility – making requests to the CloudWatch api. CloudWatch’s api only allows querying for one metric and one region at a time, so we have to make multiple requests to the api and then merge them into a single data structure. Generating requests parameters and merging data are two separate tasks and thus live in their own classes/modules.

The DSLs heart

The task of merging regional data is delegated to a module called MergeData (left unimplemented) while the task of generating the query parameters is contained in the Query class. The Query class is really the heart of our DSL so I’ll be focusing on that class and leave the nuts and bolts of making requests to the CloudWatch api along with merging the results as an exercise to the reader.

Here’s a first crack at our Query class that contains our DSL methods:

class Query 
  def initialize(&block)
    # defaults
    @elb             = "Foo" 
    @regions         = ["us-east-1"]
    @metrics         = ["RequestCount"]
    @aggregation     = "Average"
    @units           = "Count"
    @start_time      = Time.now.beginning_of_day
    @period          = 1.hour.to_s

    instance_eval(&block)
    to_cloudwatch_query_format
  end

  def elb(elb)
    @elb = elb
  end

  # varargs
  def regions(*regions)
    @regions = regions
  end

  def metrics(*metrics)
    @metrics = metrics
  end

  def aggregation(aggreation)
    @aggregation = aggregation
  end

  def units(units)
    @units = units
  end

  def start_time(start_time)
    @start_time = start_time
  end

  def period(period)
    @period = period
  end

  private
  def to_cloudwatch_query_format
   # map data into format cloudwatch api understands
  end
end

Instance eval

Notice how our initialize method takes in a block and converts it to a proc with: instance_eval(&block). This line is key to understanding the whole class. Ruby’s instance eval takes a block and executes within the context of the class instance_eval was called from. This has makes all of Query’s methods callable in the block passed to Query’s initialize method.

So we can write:

Query.new do
  elb           "my-app-server"
  regions       :us_east_1, :us_west_1
  metrics       :request_count, :other_metric
  units         :count 
  aggregation   :sum  
  start_time    Time.now.beginning_of_day 
  period        1.hour.to_s
end

Which really is just syntax sugar for regular method calls on the Query object, eg.:

query = Query.new do 
end

query.elb           "my-app-server"
query.regions       :us_east_1, :us_west_1
query.metrics       :request_count, :other_metric
query.units         :count 
query.aggregation   :sum  
query.start_time    Time.now.beginning_of_day 
query.period        1.hour.to_s

Refactoring The Query Class

One issue with our Query class is that it’s more verbose than it needs to be, we’re defining one method per field – ex elb, period, start_time. This gets worse as we add more fields to our DSL. Instead we can use Ruby’s meta programming features to create a macro which creates these setters for us, like so:.

class Query
  def self.setter(*method_names)
    method_names.each do |name|
      send :define_method, name do |data|
        instance_variable_set "@#{name}".to_sym, data 
      end
    end
  end

  def self.varags_setter(*method_names)
    method_names.each do |name|
      send :define_method, name do |*data|
        instance_variable_set "@#{name}".to_sym, data 
      end
    end
  end

  setter :elb, :aggregation, :units, :start_time, :period
  varargs_setter :metrics, :regions 

  def initialize(&block)
    # defaults
    @elb             = "Foo" 
    @regions         = ["us-east-1"]
    @metrics         = ["RequestCount"]
    @aggregation     = "Average"
    @units           = "Count"
    @start_time      = Time.now.beginning_of_day
    @period          = 1.hour.to_s

    instance_eval(&block)
    to_cloudwatch_query_format
  end

  private
  def to_cloudwatch_query_format
   # map data into format cloudwatch api understands
  end

end

Here we’re adding two static methods on Query, setter and varags_setter. These methods are simple macros which creating setter methods automatically for us. You could also use Ruby’s built in attr_accessor method here instead of rolling our own but that causes setter methods to be named like elb=() rather than elb() which is extra syntactic noise and you’d have to deal with setters taking multiple parameters another way.

Putting It All Together

At this point we’ve created our Query class which contains all of our DSL methods. The Query class calls instance_eval on the block passed to it, which gives us a nice syntax for the CloudWatch domain, by allowing Query’s methods to be called inside the block. Then our CloudWatchClient class handles passing our configured queries off to the CloudWatch api finally the data is merged via the MergeData module.

We can generalize what we’ve done with our CloudWatch DSL to a blueprint for writing other DSL’s in Ruby:

1. Pick a problem domain that would benefit from a DSL
2. Invent an ideal syntax using your domain’s lexicon
3. Create an entry point to your DSL (class or method)
4. Create a DSL class with method names corresponding to your chosen syntax
5. DSL class should accept a block that is instance_eval’ed for syntax sugar
6. World Domination!

Get Your Company To Blog More

2011-12-19T00:00:00-08:00

Get Your Company To Blog More

Cross posted on the Bizo blog

At Bizo we try to blog fairly often. But writing blog posts with any degree of frequency at a startup is tough – there are often ten other important tasks that needed to be done yesterday. Finding the time to sit down and write a post when you have features to build, code to review and the occasional meeting is difficult to say the least.

We needed something that would encourage us to blog more frequently, and there’s no better way to motivate a bunch of engineers than a game. So during our last Open Source Day we built a Blog Scoreboard that ranks authors based on the number of posts and comments they have. It’s setup on our office big screen tv serving as constant reminder that Larry is by a landslide, the king of blogging. For a live demo of the scoreboard and to see just how much Larry is dominating us, checkout the live example below:

Currently we just optimize towards the volume of posts so the points are currently assigned as: 10 points per blog post and 1 point per comment. The weighting scheme is intentionally naive due since this had to be built in a few hours and will undoubtably change as time goes on.

The best part about our scoreboard is that it is an open source sinatra (ruby) app, and it works with any Blogger blog! All you have to do is edit a few lines of YAML and you’ll have your very own big screen blog scoreboard. You can grab the source code and install instructions on github

Live demo Get your own scoreboard

The Ultimate Javascript Testing Setup

2011-12-11T00:00:00-08:00

The Ultimate Javascript Testing Setup

Any programmer worth his (or her) salt knows that testing is an important part of software development. No I’m not talking about mashing refresh on your browser and seeing if your code threw an exception – I’m talking about automated unit tests. Setting up unit tests for your front end JavaScript is actually pretty easy these days. You can pick from a multitude of drop in unit testing frameworks like Jasmine or Google’s Js Test (but you really should use Jasmine).

Loading your unit tests inside a standard browser works well for a short time on small projects with single developers. But when it comes time to integrate with the team and your build system (ex. Jenkins, Hudson, Cruise Control etc.) that approach isn’t going to cut it – for the ultimate testing setup you need to be able to execute your unit tests from the command line so your builds can pass/fail appropriately.

What follows is what I consider to be the ultimate JavaScript test setup and is what I use everyday at work.

Test Framework: Jasmine

Jasmine is by far my favorite unit testing framework. It combines a nice rspec like syntax along with facilities for testing asynchronous and delayed (timeout) functions. There have been many articles written like this one which explain more about Jasmine’s features and how to set it up, so I wont go into too much detail here. Suffice to say out of all the testing frameworks I’ve tried (which is a lot) – I like Jasmine the best by far.

Execution Environment: PhantomJS

For running tests on the command line I’ve yet to find anything better than PhantomJs which is a headless (~ non gui) fully-fledged webkit browser. Running a headless browser like PhantomJs gives you the benefit of testing in a real (webkit) browser on the command line with very, very good performance. For example I can run over 220 unit tests in around 1/10th of a second. You might also have heard of Zombie.js but unfortunately Zombie.js isn’t a complete browser and In my experience has a tendency to choke trying to execute valid JavaScript.

Because of this I’d recommend using PhantomJs over something like Zomebie.js. You can install PhantomJs for most platforms here You’ll probably also want to put phantomjs on your path for convenience

Hooking it all up: Phantom-Jasmine Bridge

Perhaps I’ve sold you on using Jasmine and PhantomJs together – but how do you set the two up and get them working together to bring you ultimate testing awesomeness? Lucky for you I’ve already done the hard work.

Assuming you already have PhantomJs installed all you have to do is download my phantom-jasmine repository over on Github. It comes with a Jasmine plugin along with a PhantomJs script that allows you to execute your suite of Jasmine tests directly on the command line with appropriate exit statuses and colored output! There’s also some simple examples to help you get started.

If you use the included TestRunner.html, running your tests becomes as easy as:

phantomjs run_jasmine_test.coffee TestRunner.html

If you aren’t starting from scratch and have an existing Jasmine Test Suite that you’d like to use – it’s still easy, just add the following to your Jasmine initialization code in your html test runner:

<!-- JasmineTestRunner.html -->

<!-- load this file, from phantom-jasmine repo -->
<script type="text/javascript" src="console-runner.js"></script>

<script type="text/javascript">
  // standard Jasmine init code
  jasmine.getEnv().addReporter(new jasmine.TrivialReporter());

  // what you need to add
  var console_reporter = new jasmine.ConsoleReporter()
  jasmine.getEnv().addReporter(console_reporter);

  // make sure you add the above before execute() is called
  jasmine.getEnv().execute();
</script>

Get the ultimate test setup for yourself »

Promises in Javascript/Coffeescript

2011-12-10T00:00:00-08:00

Promises in Javascript/Coffeescript

Cross posted on the Bizo Dev Blog

This happens often. You’re humming along writing some awesome Javascript code. At first everything is neat and organized. Then you add a feature here, an AJAX call there and before your once lovely codebase has turned into callback spaghetti.

One scenario where this can become particularly nasty is when you have an arbitrary number of asynchronous tasks, one of which cant run until all the others have completed. Suppose for example you are creating a page that displays information on your organization’s engineering team and open source projects hosted on Github. This might require a few api calls to Github (eg. one to get the repositories, another to get the team members) that can be executed in parallel. We could just append the information to the DOM as each api call finishes, but it would be nice to avoid pieces of the site popping in at different times. Instead it would be ideal if we made the api calls to Github and only after both api calls had completed would we render the information to the DOM. The problem is that in Javascript trying to implement this can get quite messy.

A Bad Solution

The issue here is knowing when it is safe to execute the task that renders the Github information to the page. One strategy might be to store the results from each api call in an array and then wait for a set amount of time before drawing the page.

results = []

githubApiCallOne (response) -> 
  results.push(response)

githubApiCallTwo (response) ->
  results.push(response)

setTimeout(() ->
  drawPage(results)
, 5000)

This turns out to be a poor solution as we can either end up waiting too little, in which case the program would fail or we could set the timeout too high making our users wait longer than necessary to load our page.

Promises to The Rescue

Fortunately there is a better way to implement our Github page. Using a construct called a Promise (sometimes called a Future) allows us to elegantly handle these types of situations. Using promises we can turn our code into something like this:

Promise.when(
  githubApiCallOne(),
  githubApiCallTwo()
).then((apiCallOneData, apiCallTwoData) ->
  renderPage(apiCallOneData, apiCallTwoData)
)

The basic idea is that our async api calls will now return a promise object that functions much like an IOU – they can’t give us the results of the api call immediately but they (probably) can at some time in the future. The Promise.when method takes an arbitrary number of promise objects as parameters and then executes the callback in the “then” method once every promise passed to “when” has been completed.

To do this, our api calls would have to be modified to return promise objects, which turns out to be trivial. Such an implementation might look like so:

githubApiCallOne = () ->
  promise = new Promise()

  # async call
  ajaxGet("/repositories", (repository_data) ->
    # fulfill the promise when async call completes
    promise.complete(repository_data)
  )

  return promise

githubApiCallTwo = () ->
  promise = new Promise()

  ajaxGet("/users", (user_data) ->
    promise.complete(user_data)
  )

  return promise

The githubApiCallOne and githubApiCallTwo make their ajax calls but return a promise object immediately. Then when the AJAX calls complete, they can fulfill the promise objects by calling “complete” and passing in their data. Once both promise objects have been fulfilled the callback passed to Promise.then is executed and we render the page.

With jQuery

The good news is if you’re already using jQuery you get Promises for free. As of jQuery 1.5 all the $.ajax methods (eg. $.get, $.post etc) return promises which allows you to do this:

promise1 = $.get "http://foo.com"
promise2 = $.post "http://boo.com"

$.when(promise1, promise2)
 .then (promise1Result, promise2Result) ->
  # do something with the data

What if I cant use jQuery?

Rolling a custom implementation of Promises isn’t recommended for production code but might be necessary if you write a lot of 3rd party Javascript and/or just want to try it for fun. Here’s a very basic implementation to get you started. Error handling, exceptions etc are left as an exercise to the reader.

class Promise
  @when: (tasks...) ->
    num_uncompleted = tasks.length 
    args = new Array(num_uncompleted)
    promise = new Promise()

    for task, task_id in tasks
      ((task_id) ->
        task.then(() ->
          args[task_id] = Array.prototype.slice.call(arguments)
          num_uncompleted--
          promise.complete.apply(promise, args) if num_uncompleted == 0
        )
      )(task_id)

    return promise
    
  constructor: () ->
    @completed = false
    @callbacks = []

  complete: () ->
    @completed = true
    @data = arguments
    for callback in @callbacks
      callback.apply callback, arguments

  then: (callback) ->
    if @completed == true
      callback.apply callback, @data
      return

    @callbacks.push callback

Sharp eyed readers might notice that the code inside the for loop in the Promise.when method looks a bit strange. You might notice that I’m wrapping the promise’s “then” method call inside of a self executing function that passes in the task_id variable. This funkiness is actually required due to the way that closures work in Javascript. If you attempt to reference the task_id without the self executing closure, you’ll actually get a reference to the task_id iterator instead of a copy – which means by the time your “then” methods execute the loop will have finished iterating and all the task_ids will share the same value! To get around this you have to create a new scope and pass in the iterator so we end up with a copy of the value instead of a reference.

And Finally an example using the supplied Promise class to prove it works:

delay = (string) ->
  promise = new Promise()
  setTimeout(() -> 
    promise.complete string
  ,200)
  return promise

logEverything = (fooData, barData, bazData) -> 
  console.log fooData[0], barData[0], bazData[0]

window.onload = () ->
  Promise.when(
    delay("foo"),
    delay("bar"),
    delay("baz")
  ).then logEverything

How to Write Large JavaScript Projects

2011-12-04T00:00:00-08:00

How to Write Large JavaScript Projects

Client-side Javascript can have a nasty tendency to balloon out of control in a large project. Since Javascript has many quirks and no built in module or dependency system things can quickly become a tangled, bug-ridden mess if you aren’t careful. Over the past year and change I’ve had the opportunity to work on several large JavaScript projects at Bizo – and these are what I have found to be best practices.

Testing

It should be obvious that testing is by far the most important thing to pay attention to when working on a large project – especially with JavaScript due to browser idiosyncrasies. You can get by without tests on small personal projects but for high quality production code – testing is a must.

I’ve found that a combination of fast unit tests using PhantomJs and Jasmine for development plus a suite of cross browser Selenium tests is ideal. While you’re coding you can continuously run your unit tests, which with my setup takes less then 1/10th of a second for 200+ tests. Then when it’s time to deploy you run the entire suite of (slower) selenium tests. This way you maintain a high quality test suite without disrupting your normal development cycle. I’d also highly recommend checking out SauceLabs – they allow you to run cross browser Selenium tests, which means you don’t have to fuss with multiple IE vms. SauceLabs even gives you 200 minutes a month of test time free, which should be more than enough for most users needs.

Modules

Most front end developers will already be familiar with the module pattern, but if you aren’t you should familiarize yourself with it. Basically we wrap a group of related functions/objects in a self executing function that returns a “public” object. This allows us to only occupy one symbol in the global namespace (our module’s name) rather than one per function/variable. Here’s a basic implementation of the module pattern with both private and public components:

var module = (function() {
  var private_variable = [1,2,3];
  var private_function = function() {
    private_variable.push(4);
  };

  var public_interface = {};
  public_interface.public_function = function() {
    private_function();
  };

  return public_interface;
})();

You should make use of modules throughout your project(s) to reduce the chance of accidentally overwriting variables.

Dependencies

Modules are great, but as they grow eventually you’ll want to begin to split up a module into multiple files. Smaller files are easier to reason about and easier to test in isolation. Ideally you’d have one file per object/group of related functions. But as soon as we begin splitting up modules into multiple files we have to manage dependencies. For example if we had a charting module called “Charts” we might split this up into files LineChart, BarChart and AbstractChart. Now suppose that LineChart and BarChart both depend on AbstractChart – unfortunately Javascript doesn’t a means to resolve this dependency. Instead we have to handle this ourselves or via a 3rd party tool.

One option is we could simply arrange the order in which we include scripts like so:

<script type="text/javascript" src="abstract_chart.js"></script>
<script type="text/javascript" src="line_chart.js"></script>
<script type="text/javascript" src="bar_chart.js"></script>

Sadly, this approach wont scale as we add more files to our project manging dependencies like this becomes a headache at best and an head-splitting migraine that makes you want to kill kittens at worst.

Another approach is to use a script loader like RequireJS which will handle loading modules for us. This works well if you only have a hand-full of files to load – but in a large project a single module could have upwards of ten files, multiply ten by the number of modules you have and it’s easy to see that this doesn’t scale nicely either.

Instead, I’ve found the best way to manage dependencies for production is to resolve them in a compile step, such that each module ends up as a single JavaScript file concatenated in dependency order. Using our previous example of the “Charts” module we would end up with a “Charts.js” file whose contents might look like this: AbstractChart + BarChart + LineChart. This allows you to test everything in isolation and resolve dependencies in a way that still plays nicely with manual script insertion or RequireJs.

I’ve written a ruby gem, Dependence.js that makes this kind of dependency resolution for JavaScript and/or CoffeeScript trivial. In the Charts example all you would have to do is this:

AbstractChart.js

function AbstractChart() {
  // code ..
}

BarChart.js

// @import AbstractChart.js
function BarChart() {
  this.prototype = new AbstractChart();
  // code ...
}

LineChart.js

// @import AbstractChart.js
function LineChart() {
  this.prototype = new AbstractChart();
  // code ...
}

Build Everything

dependence path_to_src_dir -o output_dir

What about Backbone, Spine, jQuery and other frameworks?

It goes without saying that if you’re able to use a nice framework to abstract away some common functionality you want to use in your projects – you should. Working at Bizo most of our code runs on 3rd part websites with an enormous amount of variation in the environment our tags are loaded in – hence the majority of my experience has been writing raw JavaScript. But don’t fret there have been many, many blog posts covering how to use these frameworks and all of the above principles still apply.

That’s it for now. Thoughts, comments or suggestions for important things to consider when building large JavaScript projects please share!

Kickass HTML5 SVG Charts

2011-12-03T00:00:00-08:00

Kickass HTML5 SVG Charts

December 03, 2011

Let’s face it, most charting libraries suck. Often times the default charts look awful. You need to set seemingly hundreds of options to get something that looks halfway decent. Other times the api is wonky. You might have to do extra work to 0 pad or transform data into an intermediate data structure (I’m looking at you Google Visualizations).

It’s 2011, charts should look good by default, not require developers to do extra data transformation and (gasp!) work in IE. But after looking around for a charting library that could do all three of these things (well) – I was left found wanting.

Lacking a good solution, I decided to write my own open-source charting library – Raphy Charts. Raphy Charts draws vector charts in SVG and is built ontop of the excellent Raphael – it even works in IE due to Raphael’s fallback support of VML. Time series (line, bar) charts, smoothing, tooltips, axis labeling, and sparklines are all currently supported, and I have many more features planned for the near future.

The best part is Raphy Charts is open source and super easy to setup.

Get Raphy Charts