andrewberls.github.io

Introducing Overseer - data pipeline management in Clojure (Pt. 1)

2016-04-14T00:00:00+00:00

Overseer is a library for building and running data pipelines in Clojure. Workflows are designed as a graph (DAG) of dependent tasks, and it handles scheduling, execution, and failure handling among a pool of workers. It ran the vast majority of the Framed predictive analytics platform for over a year up to the acquisition by Square, and today we’re happy to release the project as open source to the community.

In simpler words, Overseer lets you run a bunch of batch jobs that have dependencies on each other, i.e. a given job can’t run until all the jobs it depends on have completed successfully. This has broad applications for ETL, data ingestion, building complex reports, and more. At Framed, we used it for a wide variety of tasks including pulling data from external providers, computing aggregations and metrics, generating machine learning training sets, running models, and parsing/persisting their output.

This is the first post in a two-part series. In this post, we’ll explain the rationale for building Overseer and a bit about mechanics - how the system works, key terms, and what it’s like to operate in production. In the next post, we’ll discuss principles and best practices that shaped our approach to simplifying data crunching at scale.

Overseer operates in a crowded space of so-called workflow management engines, and is conceptually similar to Azkaban, Airflow, and Luigi. From the respective descriptions:

Azkaban [LinkedIn] is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.

Airflow [AirBnB] is a system to programmatically author, schedule and monitor data pipelines.

Luigi [Spotify] is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc.It also comes with Hadoop support built in.

We’ve also seen mention of similar projects at Medium, IFTTT, and the McDonnell Genome Institute, and surely more exist. So why in the world did we develop our own solution?

Our primary reason for developing something new was simplicity. Framed did not use Hadoop or any of the related ecosystem projects - we believed they were heavy, complex tools, and the operations side is extremely daunting for a very small team lacking lots of ops experience. Thus, the existing workflow managers that plugged into Hadoop weren’t a good fit for our needs. Several other solutions were written in Python, and we preferred to remain in Clojure or at least on the JVM. The result is that Overseer is written in Clojure for Clojure, and is not tied to Hadoop or any other ecosystem.

From a design standpoint, Overseer favors ordinary Clojure data structures and functions over all else. We didn’t want to to have to twist our code to match a particular framework, so no special classes or InputFormats are required here. Overseer imposes very little structure and overhead - jobs and their dependencies are specified in plain Clojure maps, and handlers are ordinary functions that can be run locally and tested like any other; these are incredibly important aspects for keeping things simple and reasoning about the system.

Running Overseer

From an operational standpoint, Overseer keeps track of all jobs in Datomic, which is a fantastic database from the same people who developed Clojure, and reflects many of the same values. Jobs in Overseer are durable; they are not de-queued or deleted after they complete, meaning the framework operates in a slightly different space than background job processing systems like Sidekiq. Overseer integrates into application code as library, and we tend to call each node running an instance of the library a worker. Each worker will continuously resolve dependencies to find an eligible job, run it, and track its status, gracefully handling failures. Overseer is a masterless system, i.e. every worker has an identical role, and clusters can be seamlessly scaled out horizontally. Framed’s primary cluster elastically varied from fewer than 10 to dozens of nodes and Overseer never knew the difference.

A brief example

So what does Overseer look like? Spoiler: most of this code is lifted from our quickstart and there’s an extensive user guide.

Here’s an entire example namespace that defines a job graph and fires up a worker:

(ns myapp.core
  (:require [overseer.api :as overseer])
  (:gen-class))

(def job-graph
  {:start []
   :result1 [:start]
   :result2 [:start]
   :finish [:result1 :result2]})

(def job-handlers
  {:start (fn [job] (println "start"))
   :result1 (fn [job] (println "result1"))
   :result2 (fn [job] (println "result2"))
   :finish (fn [job] (println "finish"))})

(defn -main [& args]
  (let [config {:datomic {:uri "datomic:free://localhost:4334/myapp"}}]
    (overseer/start config job-handlers)))

There are a few important components at play here. First is job-graph: this is an ordinary Clojure map that abstractly describes the types of jobs and dependencies between them. Each job describes the jobs it depends on, i.e. its “parents” (this may be somewhat the reverse of other graph notations where each node describes its children, i.e. all arrows going downwards). Here the :start job type has no dependencies, and so is eligible for execution as soon as we create an instance to be run. The :result1 and :result2 jobs both depend on :start, and they will not run until the :start job successfully completes. Similarly, the :finish job depends on both :result1 and :result2. Overseer handles scheduling and execution, so if any job fails unexpectedly, its dependent children will not run.

There is some opportunity for magic behind the scenes - since :result1 and :result2 do not depend on each other, as soon as the :start job finishes, both :result1 and :result2 may start executing immediately in parallel on different machines in your cluster.

The next important concept is job-handlers. This is a Clojure map where the keys are job types corresponding to the dependency graph from before, and the values are ordinary Clojure functions to run. Overseer will automatically call these functions and pass in a job argument, which is a map of information about the current job. These examples just print a message to stdout and don’t do anything meaningful; real jobs of course will likely perform computation and persist their results to external storage, enabling data dependencies between jobs.

What Overseer is not

Overseer is not a distributed data processing library like Spark or MapReduce. Each job runs on a single worker, meaning the practical limit on input size is the worker’s disk size (you can push a terabyte surprisingly far!), and workers have no communication with each other. Nor does it even run on multiple CPU cores per-worker, although it could choose to easily; instead it leaves it to each job to maximally utilize all the cores and resources available. These decisions and tradeoffs worked extremely well for our use cases, but it’s not for everyone; the project doesn’t aim to be the one true solution for every situation.

As a side note, we wrote all of our jobs as native Clojure functions, but in theory if one so desired Overseer could be used purely as a thin coordination layer and shell out a process for code written in some other language or system (e.g. a Spark job)

Wrapping up

To recap, Overseer was the backbone of the Framed processing pipeline for over a year, and ran hundreds of thousands of jobs during that time. Its focus on simplicity made it trivial for us to define new jobs, modify an existing dependency graph, test job code thoroughly, and scale up with more machines. In the next post we’ll dive into how we were able to avoid Hadoop and other heavyweight frameworks and keep our jobs simple at scale. In the meantime, be sure to check out the code on Github, or try the quickstart and user guide.

Ready for more? Read Part 2 → Go!

Introducing Overseer - data pipeline management in Clojure (Pt. 2)

2016-04-14T00:00:00+00:00

In the last post, we introduced the Overseer workflow engine that ran Framed, and saw how to use plain Clojure data structures and functions to wire up an example pipeline. In this post, we’ll talk about simplicity and how we were able to leverage Overseer and a variety of techniques to do away with Hadoop and other heavyweight processing frameworks.

tl;dr: We didn’t have some stealth in-house distributed data processing layer. You can push flat files on S3 amazingly far.

That’s pretty much it, actually. The reality is that most data is not that big and relatively few people truly need a system like Hadoop. This has been written about before, sometimes humorously so. It’s really something that needs to be shouted from the rooftops, though. S3 is a truly stellar service that served us very well, and binary serialization formats (we used Nippy heavily) helped keep file sizes down and parsing snappy.

As mentioned in the last post, Overseer jobs run entirely on a single worker - even without any effort spent on compression, our job inputs fit on a 1TB local disk. Disks on AWS are not the fastest thing in the world, but this has the added benefit of skipping lots of network chatter and data transfer incurred by the distributed frameworks.

So, from an operations perspective we didn’t need much. In addition to all of this, we were able to develop a set of best practices on top that kept our job code simple and testable.

Controlling effects

Overseer doesn’t have any opinions on what your jobs should do or return. However we observed that most of our jobs ran a computation over some inputs and uploaded their outputs to S3 at the end, so we developed ‘artifacts’, which are just data structures that specify a target to upload to S3, roughly:

{:bucket "io.framed.foobar"
 :key "/foo/bar/baz.csv"
 :file #<File /tmp/...>}

Most of our jobs would return a sequence of these artifacts, and we had a single shared generic piece of code that would actually go upload all the artifacts to S3. This is way more important than it might seem at first glance! It completely decouples the processing and computation steps from the actual side effects of persisting the results. Our job handlers became pure functions, for some sense of the word; they derive the required inputs they need to grab based on the job maps they’re passed, then they compute a result, and return a description of the output value and where to put it, without mutating the state of the outside world. This is trivial to unit test like any other function! As a bonus, we only have to write and verify a single shared persistence function.

We used artifacts primarily for S3, but it’s easy to imagine how the idea could be extended to broad applicability: rows to be saved in MySQL, facts in Datomic, or anything else you like.

This is like a poor-man’s effect system; using simple data structures we have a first-class description of side effects, and can verify and validate them all we like before executing them. This had a tremendous impact for us, and I have a feeling it will shape lots of code in my future.

Idempotence

All of our jobs were idempotent, meaning they could be run and re-run with no harmful side effects. This makes operations a breeze. If something goes wrong we know can always safely retry, and in fact Overseer does not go to great lengths to try and ensure exactly-once execution guarantees.

We were able to accomplish idempotence because all of our job inputs and outputs were fully deterministic and immutable. Our job handlers could compute their required inputs based on the job maps they were passed, which contained things like the company ID a report belonged to and the date the report was being generated for, etc. They would download their inputs from S3, run a pure transformation / computation over them, and return a result to be persisted. If the output already existed in S3, we could stop, or overwrite it to no ill effect.

Side note: every Overseer job has a unique ID. We used this to compute deterministic java.util.Random seeds per-job, so even our randomness was deterministic!

First-class failure

Overseer provides the user a number of controls around failure handling, and maintains a few of its own. It was an important design principle for us that an unhandled exception in a job not topple the worker process - this was inspired by Sidekiq (although we continue to debate the merits of an Erlang-style die-and-restart approach…). Overseer catches these exceptions and neatly marks the job as failed before moving on to the next job. In addition, if a job ever encounters a known bad configuration or otherwise “shouldn’t happen” scenario, users can call (overseer.api/abort), which will halt execution of the job and its dependent children. Finally, Overseer recognizes that transient errors happen; networks may be wonky, an external service may be down, etc, so (overseer.api/fault) will halt a job and mark it as eligible to be attempted again in the future. We configured a number of our functions to retry their operation a certain number of times before ultimately giving up and registering a fault.

These things combined meant that our Overseer workers virtually never unexpectedly fell over, with the notable exception of hard OutOfMemoryError JVM crashes from incorrectly-written job code (and ideally, a future iteration would recover even from those!). On any given day our job code could have been slow and the queue backed up, but the worker cluster kept plugging away basically no matter what and we didn’t have to do much in the role of operator, especially not at 2am.

Conclusion

A variety of techniques enabled Framed to keep data processing ridiculously simple, even at scale. Overseer was our coordination backbone that would kick off functions that in turn would download a bunch of flat binary files from S3, compute interesting results, and return a description of the result side effect: a value and a destination to upload it to. Idempotence and a first-class failure model made operations a no-brainer, and since workers didn’t have to communicate with each other or a master node of any sort, we could scale our cluster up and down to match demand easily. All in all, we believe the experience was about as simple as it could be. If you’re using Clojure or looking to start, we hope you give Overseer a shot. All the code is on Github, and there’s a quickstart and user guide. Happy processing!

BTables: A fast, compact disk format for machine learning

2016-04-13T00:00:00+00:00

Note: This is an unmodified re-publication of a post that originally appeared on Medium.com in October 2015.

At Framed, machine learning is our bread and butter. Every day, we run a lot of models over a lot of data, and recently ran up against some limits in the disk serialization format we were using to store our training datasets, which models use to discover patterns between input attributes and the outcome being predicted. The result is an efficient new serialization format we’re excited to introduce.

The input to our models is a 2D matrix of labeled numeric features describing the activity of a given set of users. For example, we may want to record event counts for users browsing an e-commerce site. An example dataset containing three labeled features and one row per user may look like this:

Login  View_Cat_Food  Purchase_Cat_Food
5      3              1
2      1              0
0      0              0
10     2              2
1      0              0

Here, the first user logged in 5 times and viewed a listing for cat food 3 times before eventually buying some, while the second user logged in twice and viewed the product page but stopped before purchasing, and so on.

Previously, we were storing these datasets in CSV files, which are slow to parse (using the Pandas CSV parser in Python) and very inefficient in terms of disk space. This was frustrating in development, as iteration cycles were very slow, and problematic in production as model runs were consuming minutes or an hour (!), or crashing entirely as the model choked attempting to load input files that exceeded available memory. Thus we set out to find a better disk format to represent our input data, aiming to maximize both space efficiency and performance for reads/writes, as well as being cross-platform (our main backend is all Clojure, but some of our models are in Python).

We first investigated the HDF5 format, which is a cross-platform format popular for scientific datasets. Our initial investigation was not promising, as we found the code (wrapping a Java library) clunky and complex to deal with, and in some cases even slower to write or less space-efficient than an equivalent CSV! It is unclear if we wrote our prototype code in an incorrect way or didn’t properly fully utilize the library, but regardless we felt the format was heavier-weight than we were looking for. HDF5 allows for modeling sophisticated arbitrary data structures/metadata using datasets in groups similar to UNIX directories/files. We knew exactly the data we need to specify, as well as the specific features we care about, and felt we could find something simpler.

First, we knew we only cared about row-by-row access over the entire file; we do not need things like random row or column reads. Second, we knew that our input data was very sparse, meaning most of the elements are zero. It seemed a trivial space optimization to only store the nonzero values, and an easy perf win to store values directly in binary to avoid continuous wasteful string parsing. So, we created BTables, a binary serialization format for sparse, labeled 2D numeric datasets (binary tables).

Conceptually, BTables are similar to a DataFrame, albeit with much fewer operations defined. As mentioned, our datasets are sparse and so we only need to store the values of nonzero cells, along with their location. We based much of our approach on the Compressed Row Storage (CRS) format, which describes arbitrarily sparse matrices with 3 vectors: vals for all the nonzero values (row-wise), col_ind for the column index of each value, and row_ptr denoting the indices of values in vals that start a row. To recount the example from the linked page, given an input matrix like

10 0  0  0 -2  0
3  9  0  0  0  3
0  7  8  7  0  0
3  0  8  7  5  0
0  8  0  9  9  13
0  4  0  0  2  -1

It would be described in CRS as:

vals: [10 -2 3 9 3 7 8 7 3 8 7 5 8 9 9 13 4 2 -1]
col_ind: [0 4 0 1 5 1 2 3 0 2 3 4 1 3 4 5 1 4 5]
row_ptr: [0 2 5 8 12 16 20]

(Note that this has been translated to 0-based indices rather than the 1-based provided in the article)

Again, the row_ptr corresponds to indices in the vals vector, so here index 0 (value = 10) begins a row, followed by index 2 (value = 3), index 5 (value = 7), and so on, with the final value containing the number of non-zeroes + 1. This is significantly more space-efficient than a CSV, and not terribly difficult to understand. We took the ideas of CRS and designed a disk layout that stores data in a single self-describing stream of values, rather than making use of multiple vectors. First, we store the length-prefixed string of labels joined together. Then, for each row we first store the number of nonzero values in that row, followed by that many (column index, value) pairs. That means the same table would be represented in a BTable with a single flat sequence (spacing for clarity):

2 0 10 4 -2
3 0 3 1 9 5 3
3 2 7 3 8 4 7
4 0 3 2 8 3 7 4 5
4 1 8 3 9 4 9 5 13
3 1 4 4 2 5 -1

This can be easily parsed in a single scan over a binary file. In addition, we can fiddle with numeric types to maximize space efficiency: all values are stored as 8-byte doubles (big-endian), and all row counts/indices are stored as 4-byte integers (big-endian).

(In fact, if we let n represent the number of rows in the matrix, CRS requires (2 * num-nonzeroes) + n + 1 values stored to reconstruct the matrix, whereas BTables require (2 * num-nonzeroes) + n. Assuming equivalent numeric encodings for values/metadata, thats a massive 4 bytes of space savings over CRS! Needless to say, we went with our format because we believed it to be simpler/easier, not because were attempting to shave off every single byte we can)

The astute reader may notice that this same matrix written as a CSV only takes up 76 bytes on disk! Ignoring the additional cost of parsing values from a text file, it would seem this is a loss in terms of space efficiency. This brings us to our next point: BTables are not a drop-in replacement for all datasets otherwise represented as CSV! The efficiency gain is proportional to the sparsity of the dataset; for a very large, very sparse dataset, BTables are dramatically more space-efficient than CSVs, but for a pathological fully-dense dataset their space usage can be much worse!

Satisfied that we had designed what we believe to be a relatively simple disk format, we prototyped an initial reader/writer implementation using Java’s DataOutputStream and DataInputStream classes within Clojure. This worked well and was quick to develop, but still didn’t achieve the performance we were looking for. We went one level lower, and began looking into Java’s NIO packages, which provide access to low-level IO operations with very high performance. We found that converting our Clojure prototype to write data to NIO buffers (containers for primitive data) and channels (connection to e.g. a file) from Java directly was a mostly painless process, and yielded a significant performance boost (in fewer than 100 lines of Java, no less).

Our current implementation is able to write a sample table with 50,000 rows and 500 features per row (25 million cells in total) in 5.8 seconds on average; the same dataset takes 16.4 seconds to write as CSV. Additionally, the CSV data files take 118 MB on disk, whereas the BTable files are only 20 MB! The gains only become more pronounced as the input scales; our production datasets may contain upwards of 1000 features per row and hundreds of thousands or millions of rows. The disk format of BTables easily allows for streaming writes and reads for datasets that are larger than available memory (this combines extremely well with lazy sequences in Clojure), but their compact representation also allows many datasets to fit in memory for model training.

Using BTables is easy! The library provides a single function to write a table to a file, and two functions to read the labels or rows from a given table file. Here’s an example in Clojure:

No HDF5IntStorageFeatures$HDF5IntStorageFeatureBuilder in sight here.

Right now, we have open-source libraries available for Clojure and Python (we’ve even written an experimental Haskell binding for fun), with documentation and examples available in each repository. We’ve been successfully running BTables in production for just over six months and were excited to share our work with the community! If you’re working with sparse datasets we hope you give BTables a shot!

Simple software

2016-03-31T00:00:00+00:00

Now that the dust has settled from Square’s acquisition of Framed, I wanted to take some time to talk about the building blocks of simple software, and the things I think we got right at Framed.

The Framed platform was written entirely in Clojure, and at its peak it was processing hundreds of millions of analytics events per day and running machine learning models over tens of millions of input points with thousands of features. Despite being a full-fledged production-grade data platform, a number of techniques enabled us to keep the codebase refreshingly simple and easy to work in. I think a lot of the complexity we face every day in software is incidental (as opposed to inherent to the particular domain), and can be reined in.

The examples and arguments in this post will mostly be contrasting Clojure and Ruby as those make up the bulk of my professional experience, although the point is not really to argue the specifics of either or write an ode to Clojure. Instead, this is really about doing more with less. I don’t expect many people to be intimately familiar with Clojure, so I’ll try and explain things as I go.

Namespaces / functions

In Clojure, code is organized using namespaces (ns) which contain functions, and that’s pretty much it. Namespaces are explicitly imported in code that requires them, and functions or definitions within them are referred to with a qualified name.

A trivial example:

(ns myapp.calculator)

(defn add [x y]
  (+ x y))

(ns myapp.core
 (:require [myapp.calculator :as calc]))

(defn do-important-work
  (let [x (rand 100)
        y (rand 100)
        sum (calc/add x y)]
    (process-result sum)))

Here, myapp.core loads our calculator namespace and chooses to call it calc; from then on any functions within are explicitly referred to, as in calc/add and so on. All that’s required for this is some extremely basic classpath config in the top-level project file (saying that Clojure code is in the src directory, for example), and then the problem is basically solved across our entire project. foo/bar/quux.clj defines (ns foo.bar.quux) and is imported as such; all external dependencies are also explicitly imported and used. This is in stark contrast to Ruby’s LOAD_PATH modification, Rails’ magical autoloading, disgusting ../../../ guesswork with require_relative and all the rest.

I have to mention here that it's technically possible to dodge the explicit namespace references in Clojure with things like `use` and `refer`. There is virtually never a good reason to do this. Clojure namespaces do have their own flaws, but they're pretty damn good.

Using explicitly-specified dependencies and namespace/package names is admittedly just an opinion of mine rather than an objectively-better approach; modern IDEs/ctags/etc make jumping around codebases and locating things very feasible, but honestly doing things this way in Clojure reduced mental overhead to zero and was a total joy. I’d develop all my software this way if I could. Surprisingly, I can think of very few modern languages that enable this workflow. Haskell does if you choose to import qualified everything, although that doesn’t seem to be the widespread approach. In Scala/Java/Ruby, about as close as you can get is a container object with a bunch of static methods, e.g. Calculator.add(x, y), which is an okay approximation at best.

It’s admittedly very easy to think in terms of OOP, and view this approach as almost primitive. I used to think to myself “well, in a real app you’d need X”, where X is classes or a complicated inheritance hierarchy or design patterns and the like. However it’s shocking the level of sophistication one can achieve with these simple building blocks alone. Clojure provides some nice abstraction capabilities on top, but building complex things from simpler component parts is what good software design is all about; suddenly classes holding tons of state and diving up and down method hierarchies feels like an unnecessary exercise.

“Dependency Injection”

At Framed, our software was modularized using a lightweight version of dependency injection. I use those words carefully, because I think for a lot of people those words conjure mental images of factories and @Inject and Spring and all sorts of unpleasant things (unpleasant to me at least). We wrote thin clients for external services using Clojure protocols (think Java interfaces), and passed them around as parameters. No magic required. In cases where we had a bunch of dependency instantiations, we’d put them in a map and pass it in as a single unit:

(ns storage.core)

(defprotocol Storage
  (put [this k v])

(deftype S3Client []
  storage.core/Storage
  (put [this k v]
    ; Call out to real S3 SDK...
  ))

(let [system
      {:storage (S3Client.)
       :redis (RedisClient.)
       :conn (datomic.api/connect my-uri)}]
  (do-work system foo bar))

(defn do-work [system foo bar]
  (let [storage-client (:storage system)]
    (storage.core/put storage-client "key" "value")))

Given a Storage protocol with a put function and an S3Client type that implements that protocol, the use of the storage.core/put function here dynamically dispatches to the proper implementation; the code has no idea if its going to hit the network, or if its just a mock version, or anything else.

(This is especially nice with Datomic. I highly recommend The database as a value. I digress.)

Testing code like this is trivial - we would write mock clients satisfying our protocols that operated purely on in-memory data or returned preset values, and pass them in; our code wouldn’t know the difference. No more mocking out the network or just hoping it works on staging or production.

This approach isn’t perfect, certainly. The overhead of writing a new protocol and wiring up real/mock clients for new services is surprisingly trivial and didn’t really factor into our decision process. However when you’re passing around dependency maps in Clojure, you have no guarantees about what it does or doesn’t contain, so its up to you to carefully construct them and check things at runtime; that’s kind of the rub with dynamic languages though. All in all this worked fantastically for us and was probably one of the most important technical decisions we made.

This section is slightly Clojure-specific, although similar things can be achieved with Java interfaces as mentioned, traits, or duck-typing depending on your language’s toolbox choice. Design for swappability!

Testing

Testing is a breeze when you program in a functional style, where a function operates solely on its inputs and produces a deterministic output*. Lightweight dependency injection as mentioned allowed us to control the state of the external world and interactions with it. Taken together, pretty much all you need to write tests is ‘given this input, I get this output’. This equals that. Mocking, stubbing, doubles, fancy DSLs, and all the rest quickly become bloated, fragile, overbearing constructs. The built-in clojure.test ships with is, so you’d write

(deftest test-add
  (is (= 5 (calc/add 2 3))))

This scales up remarkably well.

* Clojure is not a pure language and we are not purists, but you can push this surprisingly far. We even did some fancy work with deterministically computing random number seeds and passing around java.util.Random instances so even our randomness was deterministic!

A huge red flag for me is if code needs to be run in some implicit context to be tested, because it is incredibly opaque and difficult to figure out exactly what that context needs to look like. By this, I mean everything from needing to freeze the clock instead of just passing a timestamp as a parameter, mocking out every detail of network requests, or stubbing a dozen methods in advance before you can call the code under test.

We still had a few pretty hairy tests at Framed where we hadn’t aggressively split apart certain parts of the code and it took a fair amount of legwork to set up the various states of our mock dependency world just so. Despite this, the fact that you could test an entire production pipeline without ever interacting with the real outside world or reaching in to stub any functions remained very impressive to me.

Technical debt

It’s not to say that we didn’t have any technical debt. In fact, it’s been claimed that it’s “irresponsible for a startup not to have any technical debt”. Especially in the early days of the company when things were uncertain and the future was hazy, a much more cavalier attitude was taken, and some things were done hastily or some suboptimal approaches taken that stuck with us for a while. Once things settled down a bit though, we took careful thought much more seriously and really took the time to get things right; very little tech debt was introduced after those early days. I believe this led to massive overall time savings (as well as making the codebase a joy to work in!). All technical debt is born with the phrase “we’ll just ship this hack/temporary solution/quick first version and revisit it later”. Later never comes, of course. By the time you’ve rolled out your quick fix or hasty first attempt, there’s a million other things to do and you can’t waste time doing silly things like paying down technical debt. And that means you’re stuck trying to build on top of a subpar foundation, stacking hacks and workarounds on each other and suddenly you’re spending more time wrangling this mess than if you had just spent the few extra minutes to get it right at the start.

If things get bad enough, you might even end up with a Grand Rewrite, which will totally-solve-everything-this-time-I-promise (tm). Grand Rewrites are messy and error-prone beasts that can take weeks or months or longer to complete, way more time than that initial refinement phase that got skipped in the name of “ship it”. Rewrites and migrations of large systems from old to new are black holes of complexity and error - avoid them like the plague. The “long time” in “eh, we won’t need that for a long time” is often not so far off.

Most surreal of all is that in my experience this rinse-and-repeat cycle seems to be generally accepted as the cost of doing business. I feel like I’ve seen the same situation a dozen times over where everyone half-jokingly groans about the state of affairs, which is a looming quickly-cobbled together ball of mud and twine plagued by the occasional big migration of some component. It’s possible to avoid a great deal of this by moving deliberately and thoughtfully; this is usually obvious in hindsight, and tremendously difficult to commit to at the outset. As a CTO or tech lead or what have you, it might feel contrary to every fiber in your being to say things like “no, take more time on this” or “we’ll ship it when we get it right”. It’s also frustrating to hear this in review as an engineer itching to release their feature into the wild. Fight your instincts and move fast by moving slow.

There’s a line of course. There are plenty of cases where you really don’t need a fully-generic infinitely-scalable tower of abstraction up front. Sometimes its okay to ship now and defer things until later, or migrate an old system to something newer based on evolving requirements. It requires judgement, like everything else. All I’m saying is the need to ship that quick hack right-this-second-now is often less pressing than it seems.

Wrapping up

These few points had massive influence on the simplicity of the Framed platform. I’d like to write more in the future about some more Clojure-specific things we did, but I think these ideas are broadly applicable to software in any language. It’s sometimes surprising what you can achieve with a small number of well-designed building blocks, and my perspective has completely changed looking back on large object-oriented systems that rely heavily on stubbing and the like. I’d be fascinated to hear any opinions on the subject. Happy coding!

On Anxiety

2016-01-14T00:00:00+00:00

I have always been an anxious person. It’s a pervasive undercurrent, the constant voice of ‘no’ in my head; No, I don’t want to go to that party where I only know one person. No, I’ll talk to that girl tomorrow. It’s all too easy to succumb to this voice, and I’ve been spending a lot of time thinking of what to do about it.

It goes without saying that there is no rhyme or reason to these anxieties. I’ve spoken in public successfully many times before, so why do I feel the same deep pangs of fear every time even the possibility arises? I’ve met new people lots of times, so why am I nervous about this time? I am well aware that the worst possible outcome of many things I worry about day to day is really not that bad, and that my life will be better if I just first put myself out on a limb. But even with that seemingly fully internalized, it’s often very difficult to actually move past and act. Why is that?

By the start of 2015 I was sick of it. My resolution that year was to stop living under the constant cloud of anxiety and nervousness. I used to think anxiety was something to be eliminated, through some combination of discipline and confidence, real or not. Fake it ‘till you make it, right? I was convinced with some steely mental fortitude and a few changes in behavior, I would eventually cease to be an anxious person. Looking back a year later, 2015 was a fantastic year in many ways and certainly my best yet, but I still get anxious, same as anyone else. Instead of viewing the resolution as a failure though, I’ve instead changed my opinions a bit.

Note: this post is merely personal anecdotes and opinions. If you're concerned about your well-being, seek advice from a medical professional!

The single biggest conscious decision I made in 2015 was to say yes more often - that is, unequivocally saying yes to invitations and opportunities that arise on my path (and actually following through!) despite every internal scream that I could alternatively not do that. I still felt as anxious as ever when putting myself into new situations, meeting new people, or doing anything that made me feel the same way before, but maybe I was making progress? It takes a huge amount of effort to fight your instinct and follow through, but as expected I had a great number of positive experiences as a result of doing so. I plan to keep doing this for the foreseeable future.

Near the end of the year I kicked a longtime hesitation and started exercising regularly at the gym. Even that took a healthy nudge from my cardiologist. Unsurprisingly, it’s been hugely beneficial both mentally and physically, so why such an enormous initial hump to overcome? With all of these things going so well for me, why was I still feeling the usual anxiety about these things?

For the longest time, I thought confidence was the key to eliminating anxiety. Surely once I’m more confident, I won’t be nervous anymore about that new social situation / talking to that girl / interviewing that person, I thought. Believe it or not, some of the best advice I’ve heard comes from 2 Chainz on dating:

Q: 2 Chainz, I've never been in a relationship; how do I find the right one?

A: Maybe instead of trying to find the right one, you should try to become the right one.

I worked on my confidence by investing in myself. I focused more than ever on developing personal interests, mastering my hobbies and my professional craft, liking myself, and trying to be generally an interesting person to spend time around, all while saying yes to new things. As a result, I am much more comfortable in my own skin. Still no luck in escaping anxiety though.

The result of all my efforts is that I am no longer convinced that the anxiety I feel in these situations is something that can be overcome entirely through sheer force of will, or something that will just go away after some point. Instead of focusing on eliminating anxiety, I think we ought to be measured by how we respond to it, and how we overcome the mental roadblocks that come up day to day.

A friend of mine has told me a bit about Stoicism, which I’m interested in studying more in depth. Ironically it teaches avoiding anxiety through control over one’s emotions, which I am so far unsuccessful at, but it also stresses the importance of understanding the obstacles we face and not hiding from them. The things we avoid thinking about rule us most! When I feel nervous or generally uneasy, I try to clearly figure out and address why I feel that way, rather than trying to shy away from it. I might never stop being nervous about things, but I won’t let it stop me, either.

At this point I should say it’s okay to not be “on” 24/7. There’s nothing wrong with enjoying time spent alone and taking a break from it all. I still choose to spend time by myself and recharge my batteries sometimes, and that’s fine. You don’t have to try to change the nature of who you are.

I’ve said nothing about the constant pressure to hide any notion of internal anxiety and fear from the outside world, and to appear as if everything is better than ever. This is something that particularly plagues startup founders and people in the tech industry, and fortunately is a subject I’m slowly starting to see more written about. Just this week Evgeny Tchebotarev, founder of 500px, wrote about the founder’s battle for mental health. I don’t think there’s any amount of zen or confidence in the world I could possess that would prevent me from feeling anxious were I to start my own company one day. But I don’t think that’s important anymore. Here’s to another year of confronting our obstacles head on instead!

On Practice

2015-08-21T00:00:00+00:00

When I moved into a short-term apartment in January, my then-roommate had an espresso machine, a gorgeous Breville BES900XL. He taught me how to use it, and I immediately fell in love with making cappuccinos in the morning. My roommate knew all about the different types of art, and taught me how to froth the milk and make a simple tree shape. It’s trickier than it looks, and most of the time I would under-froth or over-froth the milk and pour it too fast or slow, yielding nothing at all or a vaguely-misshapen blob if I was lucky.

I kept at it, making myself a cappuccino every morning - it’s a very relaxing morning ritual, and fun to try and improve a bit every day. I moved into my current apartment several months later, and of course the first thing I bought is a shiny new Breville BES900XL. Fast forward to now - I’ve been making a cappuccino more or less once a day, and I recently got frustrated because, for the most part I still suck at latte art. I can pretty reliably produce misshapen blobs, or even something that looks like it was done on purpose, but I’m far from whatever your average barista can do easily. I thought to myself, “how is this possible? I’ve been making drinks and practicing every day for 7 months, but I’m still not that much better than when I started out”.

One of my better attempts. Not terrible, but a far cry from what my barista does in 3 seconds.

Practice is the operative keyword here. By one measure, I’ve been practicing drawing shapes with milk every day for 7 months and I still can barely produce anything. That’s embarrassing, and not a good sign god forbid I attempt something complex like programming. It’s a popular saying that the key to achieving mastery in some skill is practicing for 10,000 hours. Here I am marching towards my 10,000 hours but the outlook is not good. That’s because the other part of the saying is that you must practice correctly. I can sit and make cappuccinos the way I’ve been doing it for an infinite number of hours, but I’ve been doing it wrong every time and not improving from my mistakes - I certainly won’t magically become a master once I reach the 10k hour mark.

This is a silly example of course, and seems obvious. There’s no happy ending to this story - I still do mostly suck at latte art, but that’s because I haven’t sat down and invested the few minutes it takes to really get everything down properly - other things on my mind. However, it’s sometimes less obvious that you’re spending a bunch of time “practicing” some skill and racking up hours without becoming noticeably better.

In a previous life, I was a Rails developer, and indeed I wrote about it in depth. I enjoyed it immensely, but I’ve since left that field and never looked back, because despite having however many years of experience and studying under my belt, I felt like I had reached a plateau. I was shipping features every day, but one day’s code and designs were not discernibly more brilliant than the previous day’s or week’s, and the reading material I came across was mostly targeted at beginners. I was practicing every day and working towards that 10,000 hours but not really improving at the same rate.

To be clear, this is not a unique thing to Rails. Had I truly wanted to, I could have aggressively pursued the absolute cutting edge of the framework and pushed it and myself to their limits. Regardless, I decided that was no longer the career path I wanted to go down, so I joined a company where I hack on Clojure and hard data analytics / machine learning problems every day, and once more feel that I’m practicing correctly and heading down the road to mastery. Ask yourself if you’re practicing your craft correctly and meaningfully every day, and if you know how to make a latte bear get in touch!

Why do you want to start a startup?

2015-02-16T00:00:00+00:00

For a brief period of time after college, I ran a small consulting firm with a partner. During that time we had a short but insightful conversation with the CEO of a client company, asking for his advice and talking about general motivations. Several things he said really stuck with me - this post is about one of those things.

~

My partner and I incorporated an LLC for our firm. We had not attempted to hide the fact that consulting was not our end goal as a company; we hoped to consult to get a cash ‘safety net’ in the bank and make some contacts in industry with the ultimate goal of spinning off into a startup / product company, à la 37Signals (now Basecamp). We told the CEO this, and one of the first questions he asked us both was simple and direct:

"Why do you want to start a startup?"

Despite this seeming like a trivial question, I was stunned I didn’t had a ready-made answer to give. I had always assumed starting some sort of venture was just ‘in my blood’, without really pinpointing any one particular reason close to my heart. I could think of a couple possible answers, although none of them truly described me.

Some ostensible reasons to start a startup:

1) I don’t want to work for anyone else

This one resonated with my partner more than me. At the end of the day he couldn’t stand the idea of building someone else’s dream rather than his own, and wanted total control over his own business, product, and team without having to answer to a higher authority (it was quickly pointed out that if you take investment this does not necessarily continue to be true). I’ve never liked the idea of working for “the man” as a nameless engineer at a multi-thousand person corporation, so I’ve always been drawn to smaller environments and companies, but this was not my sole motivating factor above all others.

2) Money / glory

It’s hard to watch the Social Network without wanting a piece of the action. Startups today are glamorized to no end. Who wouldn’t want to scale a darling startup to 100MM+ users before selling out for umpteen billion dollars and retiring carefree at 28? Although tempting, this was not a driving factor for me. Most startups fail. Out of those that succeed, some miniscule fraction spiral into insanely profitable behemoths. If it’s solely money you’re after, your odds are better taking a job at one of the big players and negotiating for a very cushy (and stable) salary. By doing this you lose your shot at walking away from an exit with millions in your pocket, but for some this may be a perfectly acceptable tradeoff to make. Consulting pays well enough as is, but I didn’t want to stay in it forever, and I’m not terribly compelled to try and push a product in front of as many eyeballs as humanly possible.

3) Why not?

Fresh out of college, this was not a terrible answer. When you’re young and carefree the amount of risk you can realistically take on is close to an all-time high. Consulting was paying our rent for the time being, and were we to ever spin off into a full-blown startup, the worst possible outcomes did not seem disastrous. Neither of us had families to tend to, nor significant debts looming over our heads at the outset (which is an incredibly fortunate position to be in). If our attempt at a company were to never even get off the ground, we would pick up our shattered egos and take a steady job in industry. This helped my confidence, although I don’t think it’s the most solid answer to the question.

(in addition, Paul Graham has posted on reasons to not not start a startup.)

I told the CEO that I wanted to make a real impact on a company. Not just from a technical perspective, but on a holistic level. I want to shape the direction of a business and craft a product and a team from the ground up. I want to build something truly great, make customer’s lives better, and solve difficult problems along the way. This is true, but at the time it felt like I was making it up on the spot!

The CEO asked me if I would ever be the second employee/engineer at somebody else’s startup. I thought about it and admitted that I probably would, assuming I felt it was the right team and a good product. At that point, I felt like I could still make a significant impact, and I don’t have a hard idealistic opposition to being anything but #1. He then asked if I would be the 20th, to which I hesitated and said probably not. He asked where the line was for me, and I couldn’t answer. I think he was mostly proving a point. Could I still have that degree of control and impact as employee number 12? Number 7? At what point do you go from being an integral early employee to just another hire? There is certainly not a number that answers that for all people. For many people it does not matter.

After the conversation I was not fully satisfied with the answer I had given. It really sent me into a spin - maybe I’ve just been telling myself I’m an entrepreneur at heart this whole time, but not really meaning it. I spent a bunch of time on forums and sites for entrepreneurs, searching for inspiration or answers perhaps, but came away with little. I found many people who made their entrepreneurial living selling books/courses on entrepreneurship, doing silly things like starting X “companies” in as many months, or selling to-do list software aimed at insanely specific verticals. My lack of interest in these things made me question even more whether or not I was an ‘entrepreneur’, which I suppose is silly in hindsight. Not to say that there’s anything wrong with these - these people seemed to be die-hard self-starters, and free as a bird. For some, their ventures enabled them to live and travel all over the globe, without being constrained by a 9-to-5 schedule. That I can admire, but truthfully I am not that person, and that is not my ultimate goal. I know at least that I am certainly not an aspiring “serial entrepreneur”. I’d rather nurture a single venture to something great in my lifetime rather than looking for an exit right out the gate and circling back for round two. Of course, this conversation had me wondering if that notion was just a work of fiction in my head.

I still question my motives sometimes. Time passed, and I took a job as an early engineer at a startup, and I’m very happy with the decision. True to his word my partner from the firm is currently pursuing his own software venture. There’s no telling what life holds down the line, and maybe my answer to the question will change as time goes on. Maybe our firm will rejoin to form a world-changing startup, and maybe it won’t. If you’re an entrepreneurial spirit at heart, I suggest you really ask yourself: “why do you want to start a startup?”

Partial function application for humans

2014-11-08T00:00:00+00:00

If you’ve ever spent any time reading about functional programming, you may have heard about “partial function application” or “currying”. For the longest time these were just big scary words to me, but it turns out they’re relatively simple concepts. I’ve read plenty of posts describing what they are , but it wasn’t until recently that I really came to grasp why one would ever need them. In this post, I’ll briefly go over how partial application/currying work (and the subtle difference between them!), but I mostly want to talk about why you should care and when they can be used.

The what

Recall that arity refers to the number of arguments that a function accepts - add(a, b) has an arity of 2. From Wikipedia, partial application is the process of fixing some number of arguments to a function, thus producing a function of smaller arity. Let’s illustrate this in Ruby with procs and the Proc#curry method. As I mentioned, there is a subtle difference between partial application and currying, which will be explained later.

Here’s a simple proc in Ruby which adds two numbers:

add = proc { |x, y| x + y }
add.call(2, 3) # => 5

This is a proc taking two arguments. However, using the Proc#curry method, we can call it with fewer than two arguments, and it will return a proc that takes the rest of the arguments. Here’s what that looks like:

add = proc { |x, y| x + y }
add_one = add.curry.call(1) # => #<Proc:0x00...>
add_one.call(3) # => 4
add_one.call(9) # => 10

By supplying only one of the arguments to the curried add proc, we get another proc that in turn accepts the last argument and adds the two numbers for us. Since add expects two arguments, and we only supplied one, this is called partially applying the add function (in mathematics, calling a function is referred to as “applying the function to its arguments”). This works for any number of arguments - if we had a function taking 3 arguments, we could supply 2 and get back a function expecting 1, and so on.

Now, most of the posts I’ve read about partial application stop there. At this point, we get the gist of what partial application is, but we haven’t accomplished anything since nobody uses procs like this and adding two numbers is not a very revolutionary takeaway.

Let’s nail down the difference between partial application and currying before we dive into why any of this matters.

Currying

For the longest time, I thought partial application and currying were the same thing - the terms seems to show up interchangeably in functional programming literature, and the difference is not huge. We already saw that partial application refers to supplying some number of arguments to a function, and getting back a new function that takes the rest of the arguments and returns the final result. Currying refers to the process of taking some function that accepts multiple arguments, and turning it into a sequence of functions, each accepting a single argument. In notation, a function f might look like

f: (x, y, z) -> n

f accepts 3 arguments (x, y, and z), and returns some value n. Currying f would yield

curry(f): x -> (y -> (z -> n))

The curried version turns f into a sequence of 3 separate functions, each taking 1 argument. While we might call the uncurried version as f(1, 2, 3), the curried version would be called as f(1)(2)(3).

The difference is this: recall that for partial application, we could supply f with one argument, and get back another function that would accept the last 2 arguments and return the result immediately when called with those 2 arguments. Currying refers to the process of breaking f down into a chain of 3 functions, each taking only a single successive argument, before returning the final answer. Like I said, the difference is subtle. In Ruby, the Proc#curry method does both of these things. Wikipedia has a good contrast between the two, and Reginald Braithwaite has a great post which explains the differences in more detail.

The why

I hope I haven’t lost you at this point - as I said, the purpose of this post is to explain why you might care about any of this. The examples we’ve seen so far have not been terribly compelling. Here’s my opinion on the subject: if you’re programming in a language like Ruby or Javascript, these concepts are of limited use to you. Most of the examples I’ve read have been in these languages, and so it took me a long time to really get it. In a more functional language, the story is different.

There are many benefits to adopting a functional style in your code (in any language) that I won’t enumerate here, but in a language that’s not functional-first, these things will seem pretty forced. That’s because these languages have different approaches and conventions for managing state and computation, and trying to curry a bunch of procs in Ruby for example goes against the more “normal” way of doing the same task.

One of the primary focuses of functional programming is state - it tries to avoid maintaining global state and mutating any state as much as possible, and instead operates by composing ‘pure’ functions, which operate only on their inputs. These functions are then like easy-to-understand building blocks that can be used to construct a more complex program. Yada yada.

Back to partial application. Here’s the realization that got me on board with everything: in functional programming, state is often maintained in function parameters. This is why recursion shows up more in FP - it represents an alternative way to ‘iterate’ and build up some state stored in an accumulator parameter. This also makes things like partial application make a whole lot more sense.

Let’s see an example! At Framed, we write a lot of Clojure, so the examples will be in Clojure but you can probably follow along even if you’re not familiar with the language.

Framed computes a lot of things related to retention. For example, a customer might want to know, how many users visited my site, and then visited again 7 days later? We say that these users retained over a 7 day period. Let’s write some functions to summarize this information.

We can write a function to check if any given user retained like this:

(defn user-retained? [retained-user-ids user-id]
  (contains? retained-user-ids user-id))

All this does is check if a given user id exists in some set of retained user ids. Here, the retained user id set is a parameter, even though it’s really more like fixed state. See where this is going?

We can partially apply this function with some set of retained user ids, and get back a new function that will take a user id and tell us if they retained. If we make up sample data for retained users and some set of users in question, we can run this function over the whole sequence like so:

(def retained-ids #{3 4 5}) ; #{} constructs a Set
(def user-ids [1 2 3 5 6])

(map (partial user-retained? retained-ids) user-ids)
; => (false false true true false)

In Clojure, partial takes a function and some arguments, and works just like Proc#curry in Ruby. Here we partially apply it with the retained-ids, and get back a new function that will take a single user id, passed to it as we map over the sequence (the signature of map is (map fn collection)).

For fun, we can call the built-in frequencies function to count up the results:

(frequencies (map (partial user-retained? retained-user-ids) user-ids))
; => {false 3, true 2}

If any hardcore Clojure-ists are reading this, there are even more elegant ways of writing that, but that’s not really the point of this post.

For another example, if we wanted to fetch a bunch of user data given their ids out of Redis, we might write

(defn fetch-user [conn user-id]
  (redis/get conn user-id))

(defn fetch-users [conn user-ids]
  (map (partial fetch-user conn) user-ids))

In Ruby we might have had a global $redis object hanging around, whereas here we pass a connection around as ‘state’, and thus it’s handy to partially apply things with it.

The list of examples goes on and on. Anyway, so far we’ve seen a very long-winded explanation of how partial application is useful in functional programming to fix some ‘state’ parameters in a function, and get back a new function that can operate on variable data. That’s not the only way in which its useful, but its the pattern I’ve seen most often so far, and which really made things click for me.

I hope I’ve succeeded in explaining when and why you might care about things partial application and currying. If you are a developer in a non-predominantly-functional-language, fear not; these may be of use of you yet, and I’d be very interested to see examples in practice. And for what it’s worth, Clojure is a very fun language worth checking out. Happy currying!

Rails from Request to Response: Part 3 - ActionController

2014-01-02T00:00:00+00:00

This is part 3 of a series taking an in-depth look at how the internals of Rails handle requests and produce responses - be sure to catch up on previous parts!

Part 1 - Introduction + Unicorn
Part 2 - Routing

Last time we focused on how requests are routed to controller actions through Journey and the ActionDispatch stack. This time, we’ll look at ActionController and how controller actions turn into a rendered view.

The last post ended with the #dispatch method of the ActionDispatch::RouteSet::Dispatcher, which actually calls the controller action. Here’s that method as recap:

# rails/actionpack/lib/action_dispatch/routing/route_set.rb
  
def dispatch(controller, action, env)
  controller.action(action).call(env)
end

Remembering that controller is a class reference such as PostsController, we need to find the definition of the class method action. That’s actually defined in the ActionController::Metal class, so before we look into that, let’s go over the controller hierarchy. In this series, we’ve been working with a simple PostsController that inherits from the ApplicationController, a pretty standard setup for most Rails controllers. However, there’s several more levels to the controller hierarchy. If we take a look at app/controllers/application_controller.rb, we see that it inherits from ActionController:Base, which itself has several parent classes. Here’s what the whole chain looks like:

Click for larger image

The bottom two are user-facing classes, while the top three are internal to Rails. Let’s examine them top down. The root, AbstractController::Base, is a self-proclaimed low-level action API that should not be used directly. Its subclass, ActionController::Metal provides the simplest-possible Rack interface, without all of the helpers you get from ActionController::Base. From the docs, a metal controller allows us to write something like:

class HelloController < ActionController::Metal
  def index
    self.response_body = "Hello, world!"
  end

# And in config/routes.rb:
get 'hello', to: HelloController.action(:index)

So as we saw before, metal controller actions act as extremely simple Rack endpoints. It’s not until we get down to ActionController::Base that we get all the goodies such as rendering helpers, cookies, flashes, instrumentation and more (in fact, if you look at the source for the class, you’ll see it’s mostly just including a ton of modules!).

We’ll dive into ActionController::Base in great detail later, but in the meantime let’s return to the action dispatch code we saw previously. The action method which was called is defined in the ActionController::Metal class, and returns a Rack endpoint. Here’s its definition:

# rails/actionpack/lib/action_controller/metal.rb
  
def self.action(name, klass = ActionDispatch::Request)
  middleware_stack.build(name.to_s) do |env|
    new.dispatch(name, klass.new(env))
  end
end

Here, middleware_stack is a class attribute that’s an instance of ActionController::MiddlewareStack. In the block passed to middleware_stack#build, we create a new controller instance and call its dispatch method, passing in the name of the action, and an ActionDispatch::Request constructed from the current environment hash. In our simple app example, the middleware app is actually empty so #build just returns the block we passed in (recall that the RouteSet #dispatch method invokes this block with controller.action(action).call(env)).

The #dispatch method invoked on the controller instance first goes through the RackDelegation module, which is included by ActionController::Base:

# rails/actionpack/lib/action_controller/metal/rack_delegation.rb
  
def dispatch(action, request)
  set_response!(request)
  super(action, request)
end

All we’re doing here is initializing the internal @_response instance variable to a new ActionDispatch::Response in set_response! and calling super, which takes us up to the #dispatch method in ActionController::Metal.

# rails/actionpack/lib/action_controller/metal.rb
  
def dispatch(name, request)
  @_request = request
  @_env = request.env
  @_env['action_controller.instance'] = self
  process(name)
  to_a
end

So we set some instance variables, call process, and return to_a, which returns our beloved [status, headers, response_body] trio. The #process method takes us back up to AbstractController::Base, where we first call #process_action:

# rails/actionpack/lib/abstract_controller/base.rb
  
def process(action, *args)
  @_action_name = action_name = action.to_s

  unless action_name = method_for_action(action_name)
    raise ActionNotFound, "The action '#{action}' could not be found for #{self.class.name}"
  end

  @_response_body = nil
  process_action(action_name, *args)
end

method_for_action checks to see if the action is actually defined in the controller, and in the case it returns nil we get an ActionNotFound error.

process_action

The call to #process_action kicks off a fairly complex series of method calls that traverses a large class/module hierarchy. ActionController::Base includes a large number of modules that define the #process_action method, which do some work and then call super. Here’s a snippet of code from the ActionController::Base class (comments from source):

# rails/actionpack/lib/action_controller/base.rb
  
module ActionController
  class Base < Metal

    MODULES = [
      # (n.b. bunch of modules omitted here ...)

      AbstractController::Callbacks,

      # Append rescue at the bottom to wrap as much as possible.
      Rescue,

      # Add instrumentations hooks at the bottom, to ensure they instrument
      # all the methods properly.
      Instrumentation,

      # Params wrapper should come before instrumentation so they are
      # properly showed in logs
      ParamsWrapper
    ]
    MODULES.each do |mod|
      include mod
    end

  end
end

(you can see a list of all the modules included here.)

It helps here to have an understanding of how including a module affects the ancestor chain in Ruby. In short, including a module into a class actually inserts that module as a direct superclass of the including class, with the module’s superclass set to the original parent class, thus preserving the method lookup chain. I put together this gist as a simple example to help visualize the call stack. This means that method calls will proceed backwards up that list: #process_action in the ParamsWrapper module will come first, followed by Instrumentation, Rescue, and AbstractController::Callbacks before finally arriving at AbstractController::Base#process_action. I’ll gloss over each step, but the source is recommended reading for anyone looking to understand deeply - all of the modules live in the actionpack/lib/action_controller/metal directory in the Rails source.

The ParamsWrapper module wraps parameters into a nested hash, with the key name matching the controller’s name. For example, if you’re POSTing to the UsersController, you can just use data like

{"name": "Bob"}

and it will automatically be wrapped into

{ "user" => {"name" => "Bob"} }

The Instrumentation module provides pub/sub hooks into controller action processing (which will come up later in this post) using ActiveSupport Notifications, as well as including some view runtime benchmarking information. I recommend the Rails Guide on the notification/instrumentation system for more information.

#process_action in the Rescue module wraps the call to super in an exception handler, and enables the rescue_from helpers.

The AbstractController::Callbacks module uses ActiveSupport’s callback mechanism to hook into #process_action - this is where before/after filters (before_action and after_action in Rails 4 terminology) are run. Defining a before/after/around filter simply inserts the method as a callback to be executed. Here’s what that looks like:

# rails/actionpack/lib/abstract_controller/callbacks.rb
  
module AbstractController
  module Callbacks
    include ActiveSupport::Callbacks

    def process_action(*args)
      run_callbacks(:process_action) do
        super
      end
    end

  end
end

Finally, ActionController::Rendering defines #process_action in order to set the request formats (such as :html or :json).

Here it’s worth mentioning the ActionController::LogSubscriber. This uses the ActiveSupport notification system mentioned previously to hook into several events such as start_processing.action_controller and process_action.action_controller (both of which are triggered in the Instrumentation module). This is what’s responsible for generating the log messages such Processing by PostsController#index as HTML or Completed 200 OK in 27ms. You can find its source here.

At this point, we’ve made our way through the entire module chain and arrived at AbstractController::Base#process_action, which is refreshingly simple:

# rails/actionpack/lib/abstract_controller/base.rb
  
def process_action(method_name, *args)
  send_action(method_name, *args)
end

Of course, send_action goes through a module chain of its own! Don’t panic - only one module in ActionController::Base defines send_action - the ActionController::ImplicitRender module. It’s pretty simple:

# rails/actionpack/lib/action_controller/metal/implicit_render.rb

module ActionController
  module ImplicitRender

    def send_action(method, *args)
      ret = super
      default_render unless response_body
      ret
    end

  ...

  end
end

This module is what lets you write controller actions without an explicit call to render - it causes the view matching the action to be rendered, assuming no render has been performed already. This also handles the case if there is a template without a corresponding action. The call to super in the method goes up to AbstractController::Base#send_action, which is just an alias for send - finally we’ve actually called the controller method!

Conclusion

This is all a lot to take in - to recap, we’ve covered:

The ActionController / AbstractController hierarchy
How ActionController::Metal provides a barebones Rack action interface
How ActionController:Base includes a ton of modules that provide useful features such as parameter wrapping and before/after filters by implementing #process_action
How views get rendered by default without having to call render in the controller every time, thanks to the ImplicitRender module

At this point in the series, we’ve seen everything from reading clients from a socket, sending the request into Rails, the routing process, and controller dispatch. In a future post we’ll examine ActionView and the view rendering process!

Rails from Request to Response: Part 2 - Routing

2013-12-29T00:00:00+00:00

This is part 2 of a series taking an in-depth look at how the internals of Rails handle requests and produce responses - you can read part 1 here.

Last time we took a brief look at how Unicorn workers accept clients from a shared socket and call our Rails application, and how requests bubble down through the middleware stack before arriving at Blog::Application.routes. We’re now at the routing stage, where we need to match the request URL to a controller action and invoke it to get a response.

As mentioned, the #call method will be invoked on whatever the Blog::Application.routes method returns (as it happens, most of this post is actually just tracing the #call method to the point where we get a response).

The definition of Blog::Application.routes is located in the same engine.rb file:

# rails/railties/lib/rails/engine.rb

def routes
  @routes ||= ActionDispatch::Routing::RouteSet.new
  @routes.append(&Proc.new) if block_given?
  @routes
end

So Blog::Application.routes returns an instance of ActionDispatch::Routing::RouteSet. This is the first reference we’ll see to ActionDispatch, which is a module that’s part of ActionPack. ActionDispatch handles tasks such as routing, parameter parsing, cookies, and sessions. ActionPack is one of the top-level gems in the Rails source code, and encompasses the ‘VC’ in Rails MVC - it handles routing as mentioned previously, as well as controller definitions (ActionController) and view rendering (ActionView). The RouteSet class appears to be the pairing of a table of routes with a router to interpret and dispatch requests to controller actions - we’ll see more on these parts later.

Back to our engine - we need to examine the #call method for the RouteSet. The code lives in actionpack/lib/action_dispatch/routing/route_set.rb. Here’s the relevant code:

# rails/actionpack/lib/action_dispatch/routing/route_set.rb

module ActionDispatch
  module Routing
    class RouteSet

      def call(env)
        @router.call(env)
      end

      ...

    end
  end
end

So the RouteSet hands things off to whatever’s in @router. If we look at the RouteSet constructor, we can see that @router is an instance of Journey::Router.

Journey

Journey is the core routing module in ActionDispatch. Its codebase is fascinating to browse as it actually uses a generalized transition graph (GTG) and non-deterministic finite automata (NFA) to match URLs and routes. Pull out those theoretical CS textbooks! Journey even includes a full-blown yacc grammar file for parsing routes! I’ll be writing more about Journey in future posts.

The #call method in Journey::Router is a bit lengthy, but here are the relevant bits:

# rails/actionpack/lib/action_dispatch/journey/router.rb

module ActionDispatch
  module Journey
    class Router

      def call(env)
        find_routes(env).each do |match, parameters, route|
          env[@params_key] = (set_params || {}).merge parameters
          status, headers, body = route.app.call(env)
          return [status, headers, body]
        end
      end

      ...

    end
  end
end

This is the same Rack interface we saw in the Unicorn #process_client code in part 1 - this is the heart of Rails, and all other Rack-compatible frameworks.

#find_routes runs the URL through the GTG simulator using the routing table - it’ll take a separate post to explain the mechanism, but as its name suggests it returns a list of routes that match the requested URL. The routing table contains all of the routes of the system (instances of Journey::Route), and is itself an instance of Journey::Routes, which is constructed by the code you define in config/routes.rb. The construction of the routing table and the ActionDispatch::Routing::Mapper class could fill a post alone, and I highly recommend RailsCasts #231 and #232 for anyone looking for a detailed look at route construction.

Once we’ve found a matching route, we call #call on the app associated with the route. Now, whenever we see a reference to app in the Rails source, it’s a safe bet to assume that it refers to a Rack app. As it turns out, controller actions act like Rack apps! We’ll see more on this later on.

Tracking down where the app associated with a route gets initialized is a little tricky. The short of it is that when constructing the routing table, the ActionDispatch::Routing::Mapper class calls add_route, which will construct a Journey::Route instance associated with a controller endpoint (the Rack app) and add it to the routing table. However, the class definition in mapper.rb is almost is almost 2000 lines! Here’s the shortened definition of Mapper#add_route:

# rails/actionpack/lib/action_dispatch/routing/mapper.rb

module ActionDispatch
  module Routing
    class Mapper

      def add_route(action, options)
        path = path_for_action(action, options.delete(:path))

        ...

        mapping = Mapping.new(@set, @scope, URI.parser.escape(path), options)
        app, conditions, requirements, defaults, as, anchor = mapping.to_route
        @set.add_route(app, conditions, requirements, defaults, as, anchor)
      end

      ...

    end
  end
end

Here, @set is the RouteSet we were dealing with previously, and a little digging reveals that the app referenced here is an instance of ActionDispatch::Routing::Dispatcher. The Dispatcher class is defined back in route_set.rb - here’s its (simplified) definition of #call. To make things more clear, imagine that we’re browsing to the /posts URL of our blog app, which routes to the index action on the Posts controller.

# rails/actionpack/lib/action_dispatch/routing/route_set.rb

module ActionDispatch
  module Routing
    class Dispatcher

      def call(env)
        params = env[PARAMETERS_KEY]
        prepare_params!(params) 
        # params = {:action=>"index", :controller=>"posts"}

        controller = controller(params, @defaults.key?(:controller))
        # controller = PostsController

        dispatch(controller, params[:action], env)
      end

      ...

    end
  end
end

We’re getting to the good stuff! At this point the controller method has taken the interpreted controller parameter (such as "posts") and given us a reference to the class (PostsController), and we have a basic params hash. We’re ready to call the action on the controller and get our full-blown response - here’s the definition of the dispatch method:

# rails/actionpack/lib/action_dispatch/routing/route_set.rb
  
def dispatch(controller, action, env)
  controller.action(action).call(env)
end

At this point we’ve successfully matched the route to a controller/action, and the request is sent off into the ActionController stack, which will be the subject of the next post. Take a deep breath - routing is a particularly complex topic within Rails and the method chain can get pretty huge. Of course, as mentioned it’s not necessary to fully understand every detail of Rails routing. Part of the magic of Rails is that it hides all of these messy details from you! However, I find it fascinating to trace through and see the full flow of execution.

Some key takeaways from this post are the role of ActionPack in handling web requests from start to finish, and how controller actions behave like simple Rack endpoints. You can test this one for yourself by opening up rails console and running something like:

2.0.0-p247 :001> PostsController.action("index")
 => #<Proc:0x007fa9fd356d40@rails/actionpack/lib/action_controller/metal.rb:269>

If you were to #call that with the proper env hash, you’d get the usual array of [status,headers,body]!

In the next post, we’ll dive into ActionController, metal and all. Until next time!

Ready for more? Read Part 3 - ActionController → Go!