However, if your API sucks, you can rest assured that nobody will touch it. We’ve all had to deal with crappy web APIs, the ones that make you jump through hoops in order to perform a task that should be dead simple to do. Web APIs should make the simple tasks easy, and the hard tasks possible. To add to the challenge, APIs are notoriously difficult to change. Even with a solid versioning scheme, it is often a real chore to get your users to stop using the deprecated API in favor of the new version. So, it’s important to do a good job the first time.

When building a web API, identifying the tasks that one might want to perform can sometimes be difficult to see when you’re surrounded by JSON, XML, GETs, POSTs, PUTs, DELETEs, and HTTP status codes. While it can be easy to see what single actions you would want to expose, seeing how those actions may interact with each other can be much more difficult. Sometimes you need to take a step back, away from the land of HTTP, in order to see your API as another programmer would see it.

Building a client library that wraps your web API is a great way to do this. It’s relatively easy to imagine how your requests and responses could be represented as objects. The largest benefit of this exercise is to take it a step further, and give the user of your client library the ability to determine what they should do next. Simply knowing if an API call succeeded or failed is usually not enough. Users of your client library need to be able to determine why the request failed, and understand what they can do about it. This extends well beyond the lifecycle of a single HTTP request and response.

Communicating errors

There are several different ways to communicate errors to the user. The proper use of HTTP status codes is one such way. The 4xx class of status codes are specifically intended to be used to communicate that something was wrong with the client’s request. If your API methods are simple, and specific in their purpose, you may be able to rely on HTTP status codes alone to communicate the various causes of failure to the client.

If your API method is complex, and could result in many different failure scenarios, you should first try to break it down into smaller, more specific API methods :) If that can’t be done, then another option is to return some easily parseable text in the response body (JSON or XML) that includes an error code that identifies the specific failure scenario. The response body could be as simple as:

{ "error_code" : 123 }

You could also provide a description of the error in the response as well. This helps users getting started with the API, saving them from having to constantly refer to your API’s documentation every time they get an error:

{
  "error_code" : 123,
  "error_message" : "A widget with that name already exists"
}

The important thing is that all failure cases be easily identifiable via a specific, documented code (HTTP status code or custom error code). Error messages should be seen as purely supplemental information. At no point should your users have to parse the error message to determine what happened, or what they should do next.

Isn’t this the same as “dogfooding”

Not exactly. Dogfooding simply involves using what you have created. You could easily dogfood your web API by firing HTTP requests at it using a simple HTTP client library. It is not until you need to take different actions based on different responses that you really start to see if you are properly communicating the result of the request. Building a client helps with this because it forces you to think about the different results and error scenarios in order to decide how your client should handle them. Which failures should raise exceptions? What sort of exception should be raised? How should non exceptional failures be communicated to the caller?

The next step in this process would be to build an application that uses your client library. That step could help identify issues with your client library, just like building the client library helps identify issues with your web API.

The client library

Oh, and don’t forget. At the end of the day, you’ll end up with a better designed web API, AND a great client library that your users can use to interact with your system. Not a bad deal!

Every week (at least it feels that way) some new drama rears its ugly head in the ruby community. Petty arguments via twitter, one ranting blog post after another, people mocking ideas they consider less than ideal, and even some personal attacks thrown in the mix. There’s so much drama in fact that there is now a site out there that lists it all for the rest of the world to see.

Seriously? Are we all still in junior high?

Just think for a minute about all of the time and energy we are wasting here. Instead of igniting these flame wars, from which nothing productive is ever achieved, we could be growing as a community. We could be bringing up the next generation of software developers. We could be positively encouraging others to build better software. We could be sharing our experiences with others. We could be leading by example.

For a community of people complaining that they’re not treated like professionals, we sure don’t act very professional. If this is the way we behave, can we honestly expect people to treat us with the respect that they treat doctors, accountants, teachers, and members of other professions?

If you want to be treated like a professional, it’s best to start acting like one first.

Take the high road for once. The view is much nicer.

One of the best things about custom promotion web forms is that it allows our customers to completely control what the web page looks like. The Signal application allows for the creation of a web form theme that can be used as a template for a given promotion. The specific promotion can then customize the web page further by specifying the copy that appears on the web page, the data attributes that should be collected, and more.

The web form themes are managed by the Signal application, and are saved to disk as a view (an ERB template) when created or updated. Our customers can edit these themes at any time. When a theme is updated, we need to tell Rails to clear the cache for these specific views, so our customer will see their changes the next time they visit a web page that uses the updated theme.

In Rails 2, this was done using ReloadablePath.

class SomeController < ApplicationController
  prepend_view_path
    ActionView::ReloadableTemplate::ReloadablePath.new(
      "/path/to/my/reloadable/views")
end

However, ReloadablePath is no more in Rails 3. So, we needed to find a new solution to this problem.

Rails 3 introduced the concept of a Resolver, which is responsible for finding, loading, and caching views. Rails 3 also comes with a FileSystemResolver that the framework uses to find and load view templates that are stored on the file system.

FileSystemResolver is very close to what we want. However, we need the ability to clear the view cache whenever one of the web form themes has been updated. Thankfully, this was fairly easy to do by creating a new Resolver that extends FileSystemResolver, which is capable of clearing the view cache if it determines that it needs to be cleared.

Looking at the code for the Resolver class, you can see that it checks the view cache in the find_all method. If it does not have the particular view cached, it will proceed to load it using the proper Resolver. So, we simply have to override find_all to clear the cache if necessary before delegating the work to the super class to find, load, and cache the view.

class ReloadablePathResolver < ActionView::FileSystemResolver

  def initialize
    super("/path/to/my/reloadable/views")
  end

  def find_all(*args)
    clear_cache_if_necessary
    super
  end

  def self.cache_key
    ActiveSupport::Cache.expand_cache_key("updated_at",
      "reloadable_templates")
  end

  private

  def clear_cache_if_necessary
    last_updated = Rails.cache.fetch(ReloadablePathResolver.cache_key) { Time.now }

    if @cache_last_updated.nil? ||
        @cache_last_updated < last_updated
      Rails.logger.info "Reloading reloadable templates"
      clear_cache
      @cache_last_updated = last_updated
    end
  end

end

Since we're running multiple processes in production, we need a way to signal all processes that their view caches should be cleared. So, we're using memcache to store the time that the web form themes were last updated. Each process then checks that timestamp against the time that particular process last updated its cache. If the timestamp in memcache is more recent, then the ReloadablePathResolver will clear the cache using the clear_cache method it inherited from Resolver.

Next, we need to add some code that will update memcache any time a web form theme has been updated and saved to disk.

class WebFormTheme < ActiveRecord::Base
  after_save :update_cache_timestamp

  private

  def update_cache_timestamp
    Rails.cache.write(ReloadablePathResolver.cache_key, Time.now)
  end
end

The final step is to simply prepend the view path with the new ReloadablePathResolver.

class SomeController < ApplicationController
  prepend_view_path ReloadablePathResolver.new
end

References

We’ve all been there. We have a product that works great. It solves a certain problem incredibly well. Then, a well meaning customer comes along and says, “This is fantastic. It almost solves my problem perfectly. Is there any way you can modify it slightly to do X instead of Y.”

Sometimes this is no problem at all. Sometimes the design of the product is flexible in ways that make it a breeze to add this functionality. But, sometimes the request comes out of left field, and takes your product in a direction you never anticipated. While we strive to build software that is extensible and adaptable (it is software after all, isn’t it?), none of us can see the future, nor anticipate every possible customer request.

About this time you start to hear a little voice inside your head. “Well, I suppose I can hard code this, or add an if statement here, or write a one-off script to do X”. After all, you don’t want to tell your customer “No, sorry, we can’t do that”. And, they certainly don’t want to hear “Sure, we can make that modification, but it will require a significant amount of refactoring in order to ‘do it right’”.

Acting on these thoughts births a tiny baby Hack. The Hack is little when it’s born, but it certainly doesn’t stay that way. Once the Hack is born, it is much easier to add to the Hack, or feed it. With everybody modification to the Hack, it gets bigger, and bigger. Pretty soon you have a large, ugly Hack with a nasty attitude on your hands. And, despite you being it’s “mommy”, it doesn’t like you, at all. Not one bit.

Hacks are dangerous for several reasons.

First, they almost always live outside the main execution path of the code. This means they’re not executed nearly as often as the other code. Even if you have a series of tests for the Hack, nothing exercises code like constant execution by your customers. Also, because they’re not really “part of the application”, Hacks are often forgotten about when updating or fixing code.

Second, they’re usually created to quickly get around some issue. And by “quickly”, I mean “didn’t totally think this though, but I’m fairly certain that if I tweak X, alter Y, and drive it with a custom script, it should work just fine”. And, usually it does work just fine…at least in the beginning. But, this is when the Hack is still young, and under your control. Adult Hacks are not nearly as cooperative.

Third, they’re usually only known about (at least in detail) by the members of the team that created them. A Hack is like a big, puss filled pimple on your ass. You don’t go around showing those to your friends and co-workers, do you? Hacks, by definition, are quick and dirty solutions to problems. They’re not elegant, or sexy. So, developers tend to keep their Hacks to themselves. At most, a developer will mention that they hacked around a problem, but rarely do they go into details. The other team members are largely left in the dark. Not knowing where a Hack lives or how it behaves is a sure fire way to get bitten by it down the road.

Always remember, that little baby Hack…it will grow up. It will get nasty. It will bite. It’s just a matter of when and where. Those who have been programming for long enough know this to be fact. And, being nasty little creatures, Hacks usually wait until the worst possible time to bite.

So, beware the Hack! They are big, ugly, mean, have teeth, and will most certainly bite.

It feels as though, over the past couple of years, the pace at which I have been learning has slowed. This troubles me. One of the reasons I love the software industry is that there is always something new to learn. I love learning about new ways to solve problems with technology. And, learning is what you must continue to do if you are to stay relevant as a software engineer.

Now, this doesn’t mean that I haven’t learned anything over the past couple of years. Quite the contrary actually. I’ve really been enjoying the boom our industry is seeing in non relational data storage systems. We’ve been utilizing many of these newer technologies at Signal with great success. I’ve been an active member of the ChicagoDB users group, attended a few conferences on this topic, and even gave a few talks…a first for me. I’ve also been learning first hand how to scale a system as we’ve taken the Signal application from one capable of sending thousands of messages per “blast” to one that regularly sends millions of messages per “blast”.

However, something is missing. I used to read more. I used to blog more. I used make a sincere effort to learn a new programming language every year. I used to constantly download and tinker with new tools. These things have fallen to the wayside. And, I really miss them.

Instead, over the past couple of years, I have been spending the majority of my time creating. When creating a new project, there is usually something new to learn. That’s what drives me to create the project in the first place. But as time goes on, the cost of the time I spend working on the project starts to outweigh the benefit. I reach a point where I am no longer learning anything while working on the project. If I’m not benefiting from working on the project in any way (now that the learning part has mostly dried up), there is little motivation for me to keep working on it.

Part of me feels obligated to keep these projects going, and to get them to a point that I would consider “done”. I’ve never been a quitter, and stopping work on these projects sounds like quitting to me. However, it makes little sense to continue working on something that gives me nothing in return. I will still fix bugs that are reported, but any new development on most of my active projects is likely going to cease. Time is precious, and I need to make the most of it.

So as of right now, I’m making an adjustment in priorities. I’m using my free time to invest in myself. To read more. To blog more. To get back to learning a new programming language a year (Clojure is on tap for 2012…I’ve been wanting to learn it for 3 years now). To resume downloading and tinkering with tools that spark my curiosity. I want to start practicing how to write code, like I learned at the code retreat I attended over the summer.

I’ll still be creating, as I’m someone who learns by doing, but I have a feeling from now on my projects will be much, much smaller.

It is impossible to “find” time to do something. You have to make time.

One Monday morning about a month ago, I was browsing through open issues in our bug tracker looking for something to work on. It was my week on “technical support”. At Signal, the engineer on technical support spends a week working on non customer facing issues with our infrastructure, process, etc, that are too large to simply knock out as you encounter them.

An issue entitled “Add monitoring to detect jobs that do not start” caught my attention. We already had monitoring in place to alert us when a job fails. However, we had nothing to alert us if that job never starts in the first place. We have many scheduled jobs that run in the background, some pretty important, and have been bitten on more than one occasion by reoccurring jobs that had not run for quite some time.

I bounced some ideas off the team, and we all agreed that that it made sense to create a new application to monitor our scheduled tasks. The app would know what is supposed to run, and when. We would change all of our jobs to “ping” the app when they start and finish. If the app detects that something did not run when it was supposed to, or did not finish when it was supposed to, it could alert the team.

And just like that, Proby was born.

“What’s with the name?”, you ask? I likened the application to a probation officer, keeping an eye on wayward tasks.

Since going live, Proby has caught several issues with our scheduled jobs that would have gone unnoticed…many more than I would have anticipated. It has also evolved into a fairly complete application. It is easy to setup new tasks. It holds onto the execution history of each task, allowing you to see a trend of the run time for each task, and which runs resulted in an alarm. It supports alarms via email or SMS, using Signal’s SMS messaging API. It has several settings that allow you to tweak exactly when an alarm is sent, cutting down on the number of false alarms. And, thanks to designer extraordinaire Drew Myler, it looks amazing!

Shortly after creating Proby, we realized that there is no way we are the only ones who have had scheduled tasks go days without running, unnoticed. So today, we are making Proby available to the general public. It is currently in closed beta, and we are letting people in slowly to flush out any bugs or other issues. Use of Proby is also FREE while it is in beta.

Interested? Head on over to http://probyapp.com/signup and signup to participate in the closed beta! And, don’t forget to follow @probyapp on Twitter for updates!

What the hell is a code retreat?

For those who have never attended a code retreat, the format is simple. The group is given a problem. In our case the problem was Conway’s Game of Life. The code retreat is then broken up into several 45 minute sessions. Each session you pair with somebody new, and you take a stab at solving some portion of the problem. All code is written using TDD. At the end of each session you delete all of the code you created. The problem, and the time limit on the sessions, are specifically chosen so that there is little to no chance that you will come up with a complete solution in the allowed time. Solving the problem is not the point. The point is to practice.

We as software developers don’t give ourselves enough time to practice. At work, or even with open source or side projects, we’re always trying to get something done. Rarely, if at all, do we practice solely for the sake of practice. This is what we focused on at the code retreat. At the end of each 45 minute session, we deleted our code, and discussed what we learned. We were encouraged to try new ideas, to really push ourselves outside of our comfort zone. Knowing that the code would be deleted in a matter of minutes, there was no need to worry about screwing up the design, or creating something that would ultimately become unmaintainable. We had a safety net, and were encouraged to use it.

Corey said that we never get the chance to write “perfect code”. Perfect code doesn’t exist in the real world. In the real world products need to ship. It is simply not practical to spend the amount of time on a piece of code that is required to make it “perfect”. There are always trade offs of one form or another. However, code retreats give you the opportunity to practice writing perfect code. There is nothing to ship, nothing to “get done”. The code lives for 45 minutes, and then it disappears, giving you a unique opportunity to practice writing tiny pieces of perfect code.

Structured chaos

Sounds pretty hectic, right? But, the sessions were far from unstructured. Throughout the retreat, we were encouraged to keep in mind the four rules of simple design:

• Runs all the tests
• No duplication
• Proper naming (expresses ideas and reveals intent)
• Small (small methods, small classes)

The morning sessions were primarily focused on getting familiar with the problem. As the day progressed, we were encouraged by Tyler and Corey to try out certain ideas, or to think about tackling old problems in new ways. Can you solve the issue using polymorphism instead of if statements? Can you do it by capping all of your methods to 3 lines? What happens when you focus on proper naming of your tests and your methods? These “challenges” were designed to push us out of our comfort zone. It worked.

Some thoughts

I quickly realized that 45 minutes was not enough time to solve the problem. However, it took me almost the entire day to finally get comfortable with the fact that even solving a significant portion of the problem was out of reach. About half way through the day, I caught myself trying to solve a chunk of the problem that was way too large for the amount of time we were given. This resulted in me rushing a solution in an attempt to get my way too large of scope test to pass. At the end of the session, I was left confused with a pile of crappy code (unnecessary abstractions, no encapsulation, etc), at least half of which I wasn’t sure why we wrote.

I love that feeling you get when you create something, or solve a problem. This is why I am a programmer. Not being able to finish the problem, or even a significant portion of it, was incredibly frustrating. It took me the entire day to finally let go of that.

It was also great to see the different ways people were approaching the problem. At the start of the code retreat most pairs were pursuing a similar approach. But by the end of the day, the variety of approaches was simply amazing. With each session, everybody learned something new. Since we were required to find a new partner for each session, the combined experience of each new team seemed to produce a new approach, slightly different from the approaches each of the partners had tried in previous sessions. It was almost like looking at an old tree. Each session resulted in the forking of each branch on the tree. By then end of the day, there were a whole lot of branches.

Practice, practice, practice…

At the end of the code retreat, we all briefly shared with the group what we learned that day. Everybody had something new to say. The code retreat had a way of showing all of us what we needed to work on and what we needed to explore.

When writing this blog post, it suddenly hit me.

This is exactly what practice is designed to do!

When I practice something else (karate basics or kata for example), it has the same effect. Serious practice makes it very clear what needs to be developed. It makes no difference what it is that you’re practicing. The key to getting better at something is to practice, and to continue developing the “problem areas” identified by your practice. But it can’t be just any practice. As Corey stated during the code retreat:

Practice doesn’t make perfect. Perfect practice makes perfect.

Perhaps the single biggest thing I took away from the code retreat is that it showed me how to practice writing code.

What I learned

First off, I learned that I need to slow down…way down. All too often I think I know the solution to a problem before my fingers even touch the keyboard. And, perhaps worse, is that I am so focused on implementing that solution, that I will often ignore warning signs (code smells) that should be alerting me that my solution may perhaps not be the best solution. This code retreat made me realize that I need to do a much better job of listening to my tests, and listening to my code. I need to do a better job of keeping my nose open at all times for code smells, and to stop when I smell something. Something that doesn’t feel right in the tests is a sure sign that something is wrong with the design. Stop, take a step back, think for a minute about what is causing the smell, and fix it.

There is nothing wrong with having a solution in mind before you start coding. In fact, I think that most times you want to have a solution in mind before your fingers touch the keyboard. However, you should keep an open mind as you start to build your solution. Don’t assume your initial solution is the best solution. In fact, it’s probably safer to assume that it is not, and allow your tests to guide you to a better one.

Go to a code retreat!

If you’re serious about getting better as a programmer, and have never been to a code retreat, your missing out on an incredible learning experience. Make it a point to go to one. Since code retreats are essentially group coding practice, you can attend them over and over and be sure to walk away with something new every time. I’m already looking forward to the next one.

I have so much to learn.

To address this issue, we decided to create daily summaries of the subscription list data, and report off of the summary data instead of the raw data. The vast majority of our list reporting is already broken out by day, so this seemed logical. And, we decided to use MongoDB to store the summary data.

Why MongoDB?

We are already running MySQL and CouchDB in production. Why not use one of those? Why introduce a 3rd database product to the architecture?

The summary data is live data, not archived data (like the data we are storing in CouchDB). As subscriptions are created/destroyed for the current day, we need to increment/decrement the appropriate metrics. There are also some use cases where we need to alter summary documents for days in the past. This means we need support for atomic operations in the underlying database, to prevent race conditions from skewing the stats. CouchDB does not have the ability to atomically update a value in a document with a single call, so that took it out of the running. Some of our more active lists see several new subscriptions a second. Dealing with document update conflicts would have been a nightmare.

We decided to go with MongoDB over MySQL because of the structure of the summary data. We had several metrics that we wanted to keep tabs on (opt ins per day, opt outs per day, etc). Some of these metrics had a nested nature to them. For example, in addition to keeping track of the number of opt-outs per day, we also want to keep track of the reasons those subscriptions were canceled. Also, there are several ways that a user can opt-in to a subscription list. For example, each list can have several different keywords for entry via SMS. We needed a way of breaking all of these metrics out by method of entry.

Also, the main data structure containing the stats is repeated twice within the document. Once for the current day’s stats, and once for the totals for that subscription list up until the date in the document. This “to_date” data structure keeps us from having to evaluate ALL of a list’s documents in order to determine how many subscribers there are on a given date. With the “to_date” data structure, the specific day’s document is all we need.

We decided that this data structure was better represented as a single JSON document instead of a series of tables in MySQL. It seemed much cleaner to have a single document per list, per day that contained all of the information for that list’s daily activity than to have it scattered about in a series of relational tables. Choosing MongoDB meant giving up on SQL’s aggregate functions (AVG(), MAX(), MIN(), SUM(), etc), but the benefits provided by the simplicity of the data structure seemed to make up for the loss of these functions.

An example summary document

{
  "campaign_id": 1,
  "subscription_campaign_id": 2,
  "account_id": 1,
  "date": ISODate("2010-10-13T00:00:00Z"),
  "stats": {
    "sms_101": {
      "confirmed_opt_ins": 100,
      "unconfirmed_opt_ins": 15,
      "unconfirmed_opt_outs": 10,
      "unconfirmed_opt_out_reasons": { "UI": 5, "CNF": 5 },
      "confirmed_opt_outs": 20,
      "confirmed_opt_out_reasons": { "UI": 10, "CNF": 10}
    }
    "sms_102": {
      "confirmed_opt_ins": 200,
      "unconfirmed_opt_ins": 30,
      "unconfirmed_opt_outs": 20,
      "unconfirmed_opt_out_reasons": { "UI": 5, "CNF": 15 },
      "confirmed_opt_outs": 35,
      "confirmed_opt_out_reasons": { "UI": 15, "CNF": 20 },
    }
    "email": {
      "confirmed_opt_ins": 300,
      "unconfirmed_opt_ins": 35,
      "unconfirmed_opt_outs": 25,
      "unconfirmed_opt_out_reasons": { "UI": 20, "BULK": 5 },
      "confirmed_opt_outs": 30,
      "confirmed_opt_out_reasons": { "UI": 20, "BULK": 10 }
    }
  }
  "to_date": {
    "sms_101": {
      "confirmed_opt_ins": 1100,
      "unconfirmed_opt_ins": 1000,
      "unconfirmed_opt_outs": 200,
      "unconfirmed_opt_out_reasons": { "UI": 100, "CNF": 100 },
      "confirmed_opt_outs": 400,
      "confirmed_opt_out_reasons": { "UI": 300, "CNF": 100 }
    }
    "sms_102": {
      "confirmed_opt_ins": 2200,
      "unconfirmed_opt_ins": 200,
      "unconfirmed_opt_outs": 150,
      "unconfirmed_opt_out_reasons": { "UI": 100, "CNF": 50 },
      "confirmed_opt_outs": 400,
      "confirmed_opt_out_reasons": { "UI": 250, "CNF": 150 }
    }
    "email": {
      "confirmed_opt_ins": 2050,
      "unconfirmed_opt_ins": 125,
      "unconfirmed_opt_outs": 75,
      "unconfirmed_opt_out_reasons": { "UI": 75 },
      "confirmed_opt_outs": 750,
      "confirmed_opt_out_reasons": { "UI": 600, "BULK": 150 }
    }
  }
}

A possible relational database schema

MongoDB’s Atomic Operations

As I mentioned above, MongoDB’s atomic operations were key to us choosing MongoDB for this task.

MongoDB does not support locking or transactions like you would find in a traditional relational database. However, it does support a number of operations that are guaranteed to be atomic on a single document. If your data is designed so that all related data (or at least data that would need to be updated at the same time) is contained within a single document, then the atomic operations supported by MongoDB should be more than enough to handle the majority of use cases.

For this project, we made heavy use of the $inc operation. $inc will atomically increment or decrement a value anywhere in the document by the specified value. If no property exists in the document with that name, $inc will create it, and set its initial value to the value you wanted to increment it by. For this project, we simply initialized the data structure holding the metrics to an empty JSON hash when the summary document is created. The first time $inc is used to increment or decrement some metric, it will insert the metric into the hash, along with the proper initial value. Subsequent calls to update the document using the $inc operator will then update that value accordingly.

Using the $inc operation also meant we didn’t have to read the document to get the current value of the field in order to increment/decrement its value. We were simply able to increment/decrement the value by making one call to the database, keeping things nice and simple.

Atomically incrementing a document’s confirmed subscriptions count

collection = db.collection('subscription_statistics')
today = Date.today.to_time
collection.update({'campaign_id' => 1, 'date' => today},
  {"$inc" => {"stats.sms_101.confirmed_opt_ins" => 1}},
  {:safe => true})

MapReduce

MongoDB also supports MapReduce, providing you the ability to evaluate your data in ways that simply can’t be done using their standard queries. For this project, we needed to support summing the values of specific keys across several documents (to calculate the total opt-ins over a date range, for example). Initially, this sounded like a good fit for MapReduce. However, MapReduce will run the specified map function against each document in the database. The more documents your database has, the longer it will take MapReduce to run. I hoped that since we had built an index for the fields that MapReduce was using to determine if a document should be selected, that MongoDB would utilize that index to help find the eligible documents. Unfortunately that was not the case.

In our case, since we are only dealing with 365 documents for a year’s worth of statistics, it was considerably faster for us to find the documents using MongoDB’s standard queries and sum the data in ruby code, than to use MapReduce to do the same. If we were evaluating ALL of the documents in the database, then MapReduce would have been a much better option. I understand that 10gen is hard at work on making MapReduce faster for MongoDB 2.0. Having a strong MapReduce framework can be a powerful tool for a statistics database. Hopefully we’ll be able to utilize it in the future.

The Results

The results were staggering. On our largest list, the overview chart we display showing the current number of subscriptions per day over the last 30 days went from taking 37726ms to load to just 502ms. And the summary report for the list, which contains a wealth of statistics for the list including opt-ins / opt-outs per day, subscriber count by opt in keyword, and a series of summary statistics, went from taking 64836ms to load to just 322ms.

Dramatically reducing the size of the data being evaluated had an equally dramatic effect on the amount of time it took to evaluate that data. And, MongoDBs atomic operations and dynamic queries made this project a blast to work on.

def some_method(required_1, required_2, options={})
  # Do something awesome!
end

some_method("foo", "bar")
some_method("foo", "bar", :option_1 => false, :option_2 => true)
some_method("foo",
            "bar",
            :option_1 => false,
            :option_2 => true,
            :option_3 => "something",
            :option_4 => "something else")

This may not look like much. However, this feature alone is capable of producing some very readable code, and is used extensively in APIs throughout the Ruby ecosystem. Consider for a moment what these APIs would look like if Ruby did not have this capability, which isn’t hard to imagine for those of us with a background in a language like Java. You would either be forced to require that each parameter be specified:

# What is this code doing?  What do the nil values,
# or even the true and false values map to?
some_method("foo", "bar", false, nil, true, nil)

or accept a hash or a request object that contains all of the necessary parameters:

# This is much more readable, but requires that the
# options hash be created on its own line.
options = {:option_1 => true, :option_2 => false}
some_method("foo", "bar", options)

Providing optional parameters via hash key/value paris at the end of a method call produces code that is incredibly readable. You have the names of the attributes right next to their corresponding values! There is no ambiguity whatsoever as to which values match up with which parameters.

It is also very flexible. The order of the attributes in the hash does not matter, like it does for required attributes. And, it is very easy to add new options, or delete old ones.

This approach also makes it easy to specify default values for options that were not specified when calling the method:

def some_method(required_1, required_2, options={})
  defaults = {
    :option_1 => "option 1 default",
    :option_2 => "option 2 default",
    :option_3 => "option 3 default",
    :option_4 => "option 4 default"
  }
  options = defaults.merge(options)

  # Do something awesome!
end

There are however a few minor drawbacks to this approach. The first is documentation. Methods that take a hash of options as a parameter do not convey any information about the valid options in the method definition alone. And, it is possible that the method in question simply forwards the options to another method, sending you on a wild goose chase to determine the set of valid options the code supports.

# Looking for a list of valid option keys...no help here.
def some_method(required_1, required_2, options={})
  do_something_awesome_with_the_options(options)
end

This is why it is so important do document your public API if you are using this approach. Take a look at the ActiveRecord::Associations::ClassMethods documentation. This page documents, in a very clear and easy to read mannor, all of the supported options for each method.

It is also worth pointing out that while this approach is great for optional parameters, it is ill suited for required parameters. Required parameters should be specified outside of the options hash, making it clear that values for the required parameters must be provided. While it’s true that stuffing all of your parameters inside a hash means you’ll never have to look at another wrong number of arguments error again, it will make your code difficult to understand, and easy to misuse.

This was our first ‘bot’ in the room, and it felt a little weird. My teammate Ray created a persona for the Jenkins bot by associating it with the one and only Leroy Jenkins, changing the name of the bot from “Jenkins” to “Leeerrroooy J”. We also changed the sender name of the emails that Jenkins delivers to match.

But something was missing. Every time Leeerrroooy J. popped into our chat room with something to say, we’d all scream “Leeerrroooy Jenkins!”. I thought it would be awesome if we could have Campfire do this for us! But sadly, I couldn’t find any scripts to play audio in Campfire in response to a certain message.

So, I created one. Now, every time Leeerrroooy J. lets us know the status of a build, this will play. Makes me laugh every time.

Realizing this script could be useful in other contexts, I made it a tad more configurable, and put it on github. Right now, you can setup the script to play the sound at a given URL any time a specific word is said in the room, or when a certain user posts a message.

Hope you enjoy it as much as I do.

• Mean reads / second
• Mean writes / second
• Mean requests / second for DELETE, GET, HEAD, POST, and PUT requests
• Mean view requests / second
• Mean bulk HTTP requests / second
• Counts for various HTTP response codes

In addition, there is a plugin for individual CouchDB databases and individual couchdb-lucene indexes. The database plugin will report:

• Database size
• Number of documents
• Number of deleted documents
• Number of update operations

The couchdb-lucene plugin will report:

• Size of the index
• Number of documents indexed
• Number of deleted documents

The kind folks over at Scout have just released two new, official plugins based on the ones I created. The CouchDB Overall plugin combines some of the more important CouchDB metrics into a single plugin, and the CouchDB Database plugin reports the same set of the stats as the database plugin listed above.

The original plugins can be found at https://github.com/signal/scout-plugins/tree/master/couchdb. More information can be found here. I hope you find them useful.

Thanks to Doug Barth for some help on the plugins, and Derek over at Scout for putting together the official plugins.

However, polyglot persistence comes with its own set of drawbacks. One of those drawbacks is the loss of foreign keys, which are very important in maintaining data integrity. Another drawback is that Object/Relational Mapping (ORM) libraries typically focus on a specific database, or type of database. So, writing code that manages relationships between objects backed by different databases hasn’t been nearly as easy as writing code to manage relationships between objects in the same database.

Tenacity’s goal is to address some of these issues. Tenacity is a ruby gem that provides an ORM independent way of managing relationships between models backed by different databases.

Tenacity works by extending popular Ruby ORM libraries to respond to a set of methods that the tenacity core uses to build and manage relationships between objects. By extending the ORM libraries to implement this interface, tenacity is able work with the objects in a generic way, without having to know what database is backing the given objects. This approach also allows you to continue using your favorite ORM libraries. To use tenacity, you simply need to include Tenacity inside your model.

Tenacity is heavily based on ActiveRecord’s associations, and aims to behave in much the same way, supporting many of the same options.

This initial release of tenacity supports belongs_to, has_one, and has_many associations, and the ActiveRecord, CouchRest, and MongoMapper ORMs. However, there is still plenty of work to be done. Feedback, bug reports, and code contributions are always welcome.

Tenacity is free and open source, and can be found on GitHub at https://github.com/jwood/tenacity.

Example

Download

gem install tenacity

The Rise of the App

However, the web does have limitations. One such limitation has long been the ability to interact with the hardware you are using to browse the web. Over the years technologies have been developed to help with this limitation (Java, Flash, etc), but I see those technologies as extensions the web, and not part of the web itself. Those technologies have carried their own set of drawbacks as well, such as requiring the installation of client software, performance issues, and security vulnerabilities.

Nowhere is this limitation more obvious then in the mobile arena. Today’s smartphones are amazing. You have a camera, a GPS, a MP3 player, a web browser, oh…and a phone too, all wrapped up in one device. It’s like walking around with a tiny computer in your pocket. But currently, the web is only able to utilize a tiny fraction of this functionality. Its document centric design, which has been stretched, twisted, and pulled in directions that its creators never could have imagined, appears to have finally reached its limit.

In addition to hardware access limitations, may web applications were not designed to be viewed on a device with such a small screen and no mouse. It can be very difficult to view a 900 pixel wide web page on a screen that is only a couple of inches wide. And while you can zoom in over certain areas of the page, not being able to view and interact with the page in its entirety introduces some big usability issues. Also, many web applications rely on mouse over events to interact with the user. Drop down menus are a perfect example of this. There is no mouse on today’s touch screen smartphones, making these sites very difficult to navigate when viewed on a mobile device.

These limitations have given rise to “the app”. The app, a native application that runs on the smartphone itself, it not a new idea. Smartphones have been around for quite some time, and these phones have always had native applications. But only recently did Apple change the game by providing independent application developers with a way to easily sell and distribute their applications. As a result, there are now hundreds of thousands of applications out there for various platforms, many of which do incredible things with the hardware on the mobile device. Things that simply could not be done via a web application.

The rise of the app came on so quick and so strong that many began to ask if the app killed the web. This is certainly a reasonable question to ask. A native application has access to all that the device has to offer; the camera, the GPS, and yes, even the web. Apps are also designed to run on a mobile device. So, the text is easy to read, the controls are easy to use, and the user experience is generally much better than a web application.

Problems with Native Applications

While native applications have full access the the hardware and address many usability issues that web applications have on mobile devices, they introduce a new set of problems.

Some platforms charge program fees that must be paid to develop applications for the particular platform, even if you plan on creating a free app, or an app you never plan on releasing. While generally inexpensive, it is a new cost that web application developers have never had to deal with.

Deployment is more complicated for native applications than it is for web applications. Even though Apple’s App Store, Google’s Android Market, and other application marketplaces have made the distribution and installation of native applications much easier, it still does not compare to the ease of deployment of a web application. On the web, you have complete control over what you deploy, when you deploy it, and to whom you deploy it to. This is not the case with native applications, which can be subjected to drawn out, illogical review processes before being made available to the public. If you have a critical bug that needs fixing, or you’re trying to beat your closest competitor to market with some killer new feature, this process can be agonizing. And, even after publishing the latest version of your application to the marketplace, there is no way to force your users to upgrade. This could leave you supporting older, buggy versions of your application, or maintaining old services still in use by those old versions of the application.

If you charge for your application, distribution of that application is not free, or even cheap. Apple’s App Store, Google’s Android Market, and Palm’s App Store all take a 30% cut of the sale price of your application. Some may argue that this is a small price to pay for distribution to such a large audience. However, the audience is larger on the web, and distribution is piratically free.

The development of native applications is largely platform dependent. Programming languages and APIs vary significantly from platform to platform, making it practically impossible to develop a single application capable of running on multiple platforms. Cross platform development tools, such as PhoneGap and Rhomobile, address this issue by allowing you to develop native applications using HTML, Javascript, and CSS. However, these tools don’t address the other issues mentioned in this section. In addition, you also run the risk of some App stores outright banning applications not built using their tools (looking at you Apple). Though Apple has recently softened their stance on this issue, I would not rule out future restrictions.

A Mobile Web App in Native App’s Clothing

The sad part is, the majority of today’s apps don’t need to be native applications. They do nothing to utilize the features exclusive to native applications. Most of these apps simply fetch data from a web service and display it nicely to the user. Some take advantage of the GPS, or store data locally on the device. However, these apps could just as easily be mobile optimized web applications.

“Look and feel” is often cited as a reason to develop a native application instead of a mobile optimized web application. Though this argument certainly has its merits, there are plenty of options out there to help make a web application look and feel great on a mobile device. A bit of mobile optimized CSS (fluid layouts, properly sizing images, etc) can go a long way in terms of making a web application look awesome on a mobile device. And, there are projects out there, such as jQTouch, jQuery Mobile, and Sencha Touch that can help your web application look and feel more like a native application if that’s what you are looking for.

Web applications also have access to a few features that, just a short time ago, were only available to native applications. The device’s location is accessible via Javascript’s Geolocation API, eliminating the need to build a native application solely for access to the device’s GPS. Web applications also have access to persistent local storage on the mobile device via the HTML5 Web Storage API. While the Web Storage API falls short of a being a true database (it’s more like a hash, allowing you to store key/value pairs), it can go a long way in addressing the storage needs of most applications.

A good comparison between mobile apps and native apps can be found here.

Making the Web a Friendlier Place for Mobile Web Apps

The mobile web is growing by leaps and bounds, but much work is needed in order to make the web a friendlier place for mobile web applications. The W3C realizes this, and have started a Device APIs and Policy Working Group to create a specification that will allow web applications to access device hardware and services (contacts, calendar, camera, microphone, SMS, MMS, email, etc) in a standardized way. The specification is far from complete, and it will likely be quite some time before these APIs have been implemented by the various web browsers. But, the mere fact this is being worked on is a great sign for the future of mobile web applications.

There are non-technical hurdles preventing the web from being a more mobile friendly place as well.

First, browsing the web on a mobile device isn’t a very pleasant experience right now. Having “real” web browsers on mobile devices, such as WebKit, have gone a long way in fixing this. However, many sites are not optimized for mobile devices, making them difficult to use. I think this prevents the majority of people from browsing the web on their mobile device like they would on their desktop or laptop computer.

Second, we’ve been told for years that “There’s an app for that”. Personally, when I’m looking for a mobile application take care of some task, the first place I hit is the Android Market. Having one place, where apps are broken up into categories, and searchable, makes finding the right app for my needs incredibly easy. There is no such place for web applications. This used to be the case for desktop applications as well, but not anymore. The Ubuntu App Center, and now the Apple Mac Store, aim to serve the same purpose as the App Store or the Android Market, but for desktop computers instead of mobile devices. Perhaps it won’t be long before we see a categorized, searchable index for mobile web applications as well.

Conclusion

Opting to build a mobile optimized web application instead of a native application is not a new idea. However, the idea has been slow to catch on, due to some of the reasons I’ve outlined above.

Obviously, even after the Device APIs being worked on by the W3C have been implemented, there will always be a need for native applications. Games that need hardware accelerated graphics, or apps with complex user interaction will most likely stay native applications for the foreseeable future. But I hope that the other applications, especially the applications that are simply a front to a web service, are re-born as web applications at some point in the near future. For these apps, it’s time to come home.

Kudos to Heroku for continuing to find ways to make their platform easier to use.

The script is below. Hope you find it useful.

After graduating from college I immediately started work at a very large, multi-national communications technology company. At the time, I liked the idea of working for a large company. I thought that it would provide room to grow, job security, and the ability to switch roles easily without leaving the company. Over the past 10 years, I have moved from a company with about 100,000 employees, to a company with about 2,000 employees, and then to a company with 4 employees. Every time I moved to a company with substantially less employees, I noticed that my job satisfaction increased.

I have never been more happy with a job that I am at Signal. And, there are several reasons for this.

The ability to wear all hats

When I started at Signal there were only 2 other engineers (3 if you include the technical co-founder who was looking to focus less on technology and more on growing the business). With only three people on the engineering team, you need to be able to work on any part of the system, at any time. Application design and development, user interface design and development, database administration, system administration, IT, etc. In an environment like this there are many opportunities to learn something new. Database administration and user interface design were not strong areas of mine, but working at a startup gave me the opportunity to develop those skills. Much more so than at company where teams are segmented based on the functional areas of the application. And, even after a year and a half, I’m still learning new stuff like crazy.

This isn’t to say that some engineers aren’t stronger than others in certain areas, and you can’t play to those strengths. You’d be foolish not too. But at this size, it’s simply not practical to have one engineer do all the UI work, one do all of the database work, and one do all of the application development.

The chance to make a really big impact

I was hired as Signal’s fifth overall employee. When there are only five people in the entire company, everybody has the chance to make a really big impact. The harder you work, the bigger contribution you get to make. And contributions can take many forms, like a newly implemented feature, providing awesome support to a customer, a new idea for a product, or a new way of resolving a problem that has been facing the company. There is no product group or customer service team at a very small startup, but the needs traditionally serviced by those roles still exist. In addition, you are able to see the impact of your contribution immediately, be it positive or negative.

You also have the opportunity to play a bigger role in how your product or service develops. If you see an area that needs improvement, a bug that needs fixing, or a gaping hole in the product that could be addressed by a new feature, it’s up to you to address it. There is no committee or change board that needs to approve the change, and no need to line up resources from the database and UI teams to help with the work. You are expected to drive the change. This, for me, is a major source of inspiration and motivation.

The ability to choose your tools

One of the things a software engineer likes to complain about the most are the tools they are forced to use at their company. And with good reason. Using tools that you are comfortable with can make you more productive, and quite frankly, happier. Little is more frustrating than being forced to solve a problem one way when you know you can solve it quicker, and better, another way. This is generally not a problem at startups. While you rarely have the ability to choose all of your tools (programming language for example), you will generally have the ability to choose your work machine, operating system, editor/IDE, web browser, mail client, etc. And, since you are your own IT department, there is no question that you will have root access to your work machine. These may sound like minor details to most people, but any geek will assure you how important this actually is.

The ability to choose the right technology for the job

Many companies standardize on technologies. There are many valid reasons for this. However, sometimes the standardized option makes little sense for the task at hand, and you are forced trying to fit a square peg through a round hole. This generally doesn’t happen at startups. One of the best things about my job is that the engineering team gets to decide which technologies to use after evaluating the technical details of the problem it is trying to address. One might think that a stance like this would lead to an explosion of tools and technologies in our stack, but this is hardly the case. Being responsible and professional developers, we understand that we will have to live with and support the decisions we make. So, we carefully evaluate any significant new technology before bringing it into the stack.

A voice in choosing your teammates

When working on a small team, everybody needs to pull their weight in order for the company to be successful. Bringing new people onto the team is a big deal. A single bad team member can be a cancer to an otherwise strong and healthy team. Therefore, people interviewing for the company talk with everybody. This may be a bit overwhelming for candidates, but I think most appreciate how closely we protect our team dynamic. Now, this isn’t to say that everybody “interviews” each candidate. Our VP of marketing isn’t going to ask a candidate for an engineering position to solve a coding problem on the whiteboard. Instead, a few people will determine if the candidate is qualified for the position, and the rest will determine if the candidate would be a good fit, personality wise, on the team. As an engineer, I think this is great. When hiring a new engineer, we all get to ensure that the candidate is smart, loves technology, knows how to solve problems, and will get along with the rest of the team. And when hiring for a non engineering position, we get to ensure that the person we’re bringing onto the team isn’t a jerk.

A great work culture

Startup culture is great. I’m not talking about $1,000 desk chairs or having a company masseuse. Hell, two other engineers and I were coding around what the CEO called a “nerd campfire” the other day (can’t say that I disagree with that terminology) because we currently have no desks. I’m talking about the camaraderie; eating lunch together, getting to meet each others spouses and children, chilling out and having a few drinks with each other. I think this is very hard to find outside of small company. It seems to me that most startups have a casual, relaxed, and fun work environment. Flexible hours. No dress code to speak of. Signal even has a kegerator in the office which is always fully loaded with great beer. They do pretty much anything they can to make their employees feel comfortable so they be productive.

Flexible work options

Flexible work hours are very popular now-a-days. Even at the larger companies where I have worked, it wasn’t uncommon for some people to come in at 11am and stay until 7pm or 8pm. Having the ability to fit work around your personal schedule is a big plus, and I feel it makes for a much happier employee.

Even more important to me is the ability to work from home. I live about 30 miles from the office, which takes about 90 minutes to travel during rush hour by car or train. I’ve tried not too complain too much about my commute, because nobody forced me to live where I do. It was my choice, and I’d make the same choice again. But, when I was given the opportunity to work from home, I jumped on it. Working from home saves me 3 hours a day. That is huge for me, as it always feels like there are not enough hours in the day. Working from home 4 days a week has given me the ability to see my wife and kids more often, start running, and to spend a little more time on some side projects to improve and expand my skills as a developer. Seeing my kids more often is by far the biggest benefit of working from home. Now that my oldest is in school, I barely see him when I go into the office. Our schedules only overlap for about 30 minutes in the morning, and 30 minutes at night. This is not the case when I work from home.

No bullshit politics

When there are only 13 people in the entire company, there are no politics. There is no jockeying for position. There is no posturing to give the impression that you are an all star employee when you’re really just a leech. There is no room to hide. If you’re not contributing, everybody knows it. If you are trying to exploit others for personal gain, everybody sees it.

Your opinion matters

It’s hard to make a decision that impacts the company in complete isolation at a startup. The founders at Signal always consult the team when it comes to making a big decision. Make no mistake, they have the final say. But they value the opinions of the team and constantly request feedback. This gives the employees a sense of ownership, which is great for morale and can be very motivational.

Conclusion

I think many of these points are the product of a few major differences between a big company and a startup. First, productivity is paramount, and anything that can be done to increase productivity is done. If employees aren’t producing at their highest level, it greatly increases the risk of the startup folding (delaying time to market or being surpassed by a competitor). Second, startups run on the bare minimum. There is no extra weight to carry around. There are no employees that need to justify their existence by adding unnecessary overhead. Everybody has more than enough “real work” to do in order to move the company forward.

Working for a startup does have some drawbacks as well, but the positives far outweigh the negatives. The comparison is not even close as far as I’m concerned. As I mentioned at the beginning of the post, each company I have worked at has been substantially smaller than the previous one, and my job satisfaction has increased with each move. While some large companies can have some really interesting problems to solve (especially when it comes to problems of scale), I’m not sure I could ever go back to working for a large company.

Yo Ho, it’s a startup’s life for me.

See everybody next month!

Information about the meeting can be found here. I hope to see you there!

First and foremost, this was an incredibly well organized and wonderfully run conference. Ray Hightower and the WindyCityDB team did an amazing job. The space at IIT was perfect. The food and drink was ample and tasty. And the talks were interesting and thought provoking. To top it off, all proceeds from the conference, approximately $1,000, was donated to the Greater Chicago Food Depository.

One theme that took me by surprise was the broad support for relational databases. Almost all of the speakers stated that the relational database still has a place in this new world, along side all of the NoSQL technologies that we had gathered to discuss. This is something I completely agree with, but being a NoSQL conference, I had imagined that there would be quite a bit of RDBMS bashing going on. Nothing could have been further from the truth. Everybody seemed to agree that the RDBMS is a very functional and reliable piece of software that is still very much needed. Everybody also seemed to agree that there are some things the RDBMS doesn’t do too well based on how it is designed, and that the different NoSQL technologies aim to fill this gap. Nobody was advocating their particular NoSQL database as a replacement to the RDBMS. Instead, all were pitched as alternatives, created to solve different problems.

Another theme that sort of caught me off guard was the wide acceptance of Polyglot Persistence as “the future” of application persistence methods. I say “sort of” because I was there to speak about Polyglot Persistence, which of course means that I think it can be a great tool if used properly. However, I thought it was a bit more controversial of a topic. As it turns out, there are many already taking a polyglot approach to solving their persistence needs. This was great to hear, as made me feel a little less insane for wanting to use multiple databases within a single application. Well, I could still be insane, but at least now I know I’m in good company :) Several speakers mentioned Polyglot Persistence in their talks, and I spoke with several individuals at the conference, and after the conference, who are actively using a polyglot approach to address their persistence requirements.

I also found John Nunemaker’s prediction that hosted database solutions are going to be the future of application data persistence to be very interesting. If you think about it, this is exactly what happened with web hosting. Most companies, especially small companies, now use a 3rd party to host their production environments. In fact, I’d bet that the thought of hosting your own environment doesn’t even cross the mind of the vast majority of small companies now-a-days. I can easily see how hosted database solutions, such as MongoHQ or Cloudant, could follow a similar path. You would no longer need to worry about hardware issues, scaling issues, backups, maintenance, etc. It would allow you to simply focus on your data.

Even though I didn’t get a chance to play with it, the WindyCityDB lab was a nice touch, especially for those who had never tinkered with some of the NoSQL databases on display. The lab consisted of several iMacs, each loaded with a particular NoSQL database. All of the databases were preloaded with the Stack Overflow data set. Sharing a data set was a great idea, as it gave you a good way to compare and contrast how the different databases handled the same set of data.

I really enjoyed myself at this conference, and am already looking forward to next year’s. At the rate the NoSQL movement is moving, I can only imagine what the landscape will look like a year from now.

The case for Polyglot Persistence was made throughout the day by several of the speakers. Most people seemed to acknowledge the fact that it is very likely one database may not have all of the tools you need to get your job done. Instead of coercing a single database to try and do things it was never designed to do, it is becoming more common for applications to use multiple databases, utilizing each for their respective strengths. But, of course, using multiple databases in a single application comes with its own set of issues, and you should make sure there is a real need for Polyglot Persistence before making that leap.

Thank you very much to the WindyCityDB organizers for putting on such a great event. I had a great time, learned a ton, met some interesting people, and participated in some great conversations. What more can you ask for in a tech conference?

So I started taking notes, documenting everything I could regarding what we learned about CouchDB, the design decisions we made for our application (and their respective trade-offs), and the migration of our application code to use CouchDB. I organized those notes, and posted them on this blog as a case study, hoping it would help others looking into CouchDB.

I received a lot of positive feedback from the posts, making me feel like I had in fact filled that need, at least to some degree. Even better, earlier this year I was contacted by Couchio about combining the series of posts into a white paper that would be posted on their site. This would put the case study in front of a larger audience, potentially helping even more people. I was thrilled.

Today, that white paper was released as the “Epic Interactive Mediums Whitepaper” (love the Epic :)). You can get it here. Many thanks to the kind folks over at Couchio for putting this together. I hope people will find it helpful.