You work for me, Computer

Personal Update: NYC

2013-02-09T08:50:00.004-08:00

I've moved to New York City! The plan is to bootstrap my next company, so stay tuned for a lot more details about that in the near future. Meanwhile, I'll be available for contract work, so shoot me an email at brandon@brandonbloom.name if you're in NYC and would like to chat.

Fipp: Fast Idiomatic Pretty-Printer for Clojure

2013-01-11T18:10:00.001-08:00

I decided to create a new pretty printer in the spirit of "Data All The Things!" And it's fast too!

Fipp, the Fast Idiomatic Pretty-Printer for Clojure, is a pretty printer with linear runtime and bounded space requirements. Unlike clojure.pprint's side-effectual API, Fipp is configured by pretty print documents that are similar to Hiccup HTML templates. It's based on some fancy pants research pretty printers from the Haskell world.

You can see my implementation and lots more notes at https://github.com/brandonbloom/fipp

Factjor: Concatenative Clojure

2013-01-01T18:07:00.002-08:00

I've released another small Clojure project: Factjor

Head on over to GitHub for a tiny intro in the README. Lots more explanation and context will follow in a future post, after I release the project that motivated Factjor.

The Node.js REPL is Broken

2012-12-21T00:05:00.000-08:00

Not only is the Node.js REPL broken, but so are the stock REPLs for Python and Ruby too! In fact, for many of you, every single REPL you've ever used has been fundamentally flawed. Moreover, even if you used a fancy replacement REPL, such as UltraREPL, IPython, or Pry, you'd only be mitigating the problem: JavaScript, Python, and Ruby are all fundamentally flawed languages from a REPL-lover's perspective.

Now, I realize that this is a bold and, thus far, unsubstantiated claim. Python and Ruby are revered for interactive, dynamic development and Firebug's JavaScript console changed the game for web apps. So with so many happy REPL users, what are these fundamental flaws? Since it's the worst offender, let me pick on JavaScript for a moment...

Node's stock REPL provides a fragile and awkward notion of an evaluation context. A fresh REPL is configured with an evaluation context as if by both with(context) and .call(context) to enable unqualified access to context variables, as well as introspection of the this context. This, however, is a convenient illusion. If you evaluate var x = 1; this, you'll note that x is added to this. If you were to use var inside a with block, you wouldn't get that same behavior. The top-level inside the REPL is somewhat magical.

CommonJS Modules provides the blueprint for Node's module system. The key idea is that Plain Old JavaScript Objects can be used to describe modules and their exports. Simple, but perhaps too simple. Consider if you were working on some function f in module m. You boot up your trusty REPL, evaluate var f = require('m').f; and try out some corner cases. Ugh oh, f has a bug, so you modify src/m.js and then you.... promptly restart your REPL! The require function doesn't offer any sort of {reload: true} option, and even if you did, you'd still be left with a reference f pointing to an old version of the function. That might not be a big problem if you have just one function, but what if you have several objects and functions in a larger system? The too-simple plain old JavaScript objects approach results in the copying of references and assigning those copies to local names. It's extremely difficult to reason about changes. Even if you knew to cheat by reaching inside of require.cache with a delete, you'd still be better off restarting your REPL to preserve your sanity. That's a real shame if you had a bunch of interesting test cases lying around in memory, or if there is an expensive initialization computation you need to wait through each time.

Now that I'm done picking on JavaScript, I'll point out that the situations in both Python and Ruby aren't much better. Python doesn't have a notion of a current module, although it does provide the builtin locals() function, as well as lets you call code.interact(locals={...}) to start a nested sub-REPL with something akin to Node's magical top-level. Ruby fairs marginally better by providing a reasonably well behaved implicit self, which can be shadowed in a nested sub-REPL via an argument to the confusingly named irb method. Python's modules have a similar issue to plain old JavaScript objects with from m import f and Ruby's metaprogramming facilities add more than enough complexity to make REPL-coding hair-raising.

It doesn't have to be this way. There are languages that are much better suited to iterative REPL development. If you've been following my blog, then this is where you'd expect me to praise Clojure. I'll save that for a future post in which I'll discuss what is really necessary for a successful REPL. And while Clojure's REPL is far better than any other I've ever used, it's got it's own shortcomings to discuss as well.

Clojure's multimethod dispatch as a library

2012-12-10T00:24:00.003-08:00

[In keeping with the theme, this post motivates dispatch-map]

Clojure's multimethods embody a powerful polymorphic dispatch mechanism. With multimethods, you can define a function which is polymorphic with respect to a given hierarchy and arbitrary dispatch value. The hierarchy is actually a directed acyclic graph, but you can adjust method priority to address the diamond problem.

Unfortunately, Clojure's dispatch mechanism is baked into the defmulti and defmethod forms. If you want multimethod-style dispatch, you need a Var to hold your multimethod. Similar to the circumstances which prompted the creation of backtick, I discovered that there was an opportunity to decomplect a part of the core library to extract a useful subcomponent. In this case, the datastructure which describes dispatch rules was complected with the identity which points to those dispatch rules. As a result, it's impossible to treat a multimethod as an immutible value.

So I created dispatch-map. A dispatch map is just like a regular Clojure map, but it comes coupled with a dispatch function and hierarchy. Looking up keys in a dispatch map has the same dispatch behavior and caching functionality of multimethods. However, unlike a multimethod, a dispatch-map is a true value. The standard map operators assoc, dissoc, etc, return a new dispatch-map, leaving the original unmodified. This enables you to manipulate dispatch maps like any other Clojure data structure, without the need for a named, mutable Var. Clojure's multimethods and all related functions could trivially be reimplemented in terms of a dispatch-map and an atom (edit: and now are!).

One use for a dispatch-map as a value is to store them within some other data structure, without breaking the value guarantees of the enclosing data structure. In my case, I've got a dispatch-map of data types to their GUI representations. In a hierarchical definition of a GUI, each node can have a :templates key which is a (dispatch-map class). If you want to render a (Person.) inside a list box, you can rely on automatic dispatch by type to find the appropriate template: Simply walk up the GUI hierarchy and look in the :templates map. Thanks to the powerful dispatch functionality, this will also work if you have an (Employee.) too!

Dataflow Solver (qplan for Clojure)

2012-12-01T19:38:00.001-08:00

[This post describes the motivation for my next Clojure library: qplan]

The overwhelming majority of graphical applications are built using ad-hoc event callbacks. Some portion of graphical applications, however, use a databinding model of varying expressivity. Often, these databinding models are simply shorthand for ad-hoc dependency graphs composed of event callbacks. Inevitably, the databinding system will be insufficiently expressive, necessitating manual callback wiring. Even with the most expressive data binding systems available in application frameworks these days, there is a shockingly fragile rat's nest of control flow and execution order.

Quite a bit of activity has occurred in the research community surrounding more formal, more composable, and less fragile event handling. The buzzword de jour is "Functional Reactive Programming" (FRP) which, although quite varied by author, generally involves the concepts of continuous "signals", discrete "events", and some evaluation strategy. FRP was born of animation systems and is a fertile research area. More generally, FRP is a type of dataflow programming. The "reactive" moniker also applies to the area of push-sequences, such as .NET's IObservable and associated Reactive Extensions (Rx).

As fun as it has been to study FRP, there has been little success building real applications using its techniques. In my opinion, this is because FRP encourages a continuous model of your application, but most graphical applications are much more discrete in nature. Multimedia systems, like the animation system that motivated the first FRP paper, and games have quite a few continuous use cases. Point-and-click software, however, generally has discrete controls, with discrete values, and discrete commands. Consider, for example, the undo-able discrete mutations in the most broadly deployed dataflow application on the planet: spreadsheets. Although capable of representing discreet events, FRP systems project the mental framework of water flowing through pipes. This mental framework clashes badly with the more familiar event callback model. Furthermore, even for truly continuous use cases, discretization is essential to the nature of computation; every game fundamentally distills down to an update and render loop.

I'm still contemplating whether or not it is possible to build a databinding system that is an order of magnitude more expressive and composable than the existing state of the art. However, I'm certain that the underlying architecture will be a discrete system which embraces a formal notion of time through an explicit notion of identity and value. Consider again the spreadsheet, where cells have an identity composed of their row and column labels. As you change the spreadsheet, the document updates the values in all cells synchronously in response to a discrete command. This approach can be extended to an arbitrary graph of identities connected by dataflow constraints.

Given a sprawling graph of identities, you need some evaluation system for assigning values to each identity. Unlike a spreadsheet, many applications demand sophisticated and potentially cyclical bindings. In the face of multi-input, multi-output, and multi-directional dataflow, there are many potential evaluation strategies yielding many different graph linearizations. You need a constraint solver suited to the task. Luckily, Texas A&M has studied this problem in collaboration with Adobe. Their approach is referred to as Property Models and addresses many of the issues with relying on solvers for user-facing constraint systems. The Property Models approach is much broader than simply databinding, it also considers predictability and effectively synthesizing commands, such as those recorded in Photoshop's History pane.

At the core of the Property Model system, is a variation of Zanden's Incremental Algorithm for Satisfying Hierarchies of Multiway Dataflow Constraints as described by Freeman, et al. I've produced a Clojure implementation of this algorithm called qplan.

My intention is to use qplan as part of a larger approach to building graphical user interfaces. I expect a that a complete system requires a variety of algorithms and solvers tuned for particular use cases. For example, Adobe's Adam and Eve, has both a Quickplan solver for property models and a linear constraint solver for layout. An even more complete system may have an FRP evaluation strategy for animations.

Templating: Clojure's Backtick

2012-11-23T04:34:00.001-08:00

[This post describes the motivation for my new Clojure library: Backtick]

There is no shortage of template languages. In fact, there are probably far too many of them. Template languages fall into two major categories that I'll call procedural and structural. Procedural templating languages operate by emitting code as a side-effect. Structural templating languages operate on trees. For example, Ruby's ERB is procedural, where as HAML is structural.

The primary advantage of a text-based procedural template language is reach. ERB can be used to generate any type of text file, while HAML can only really be used to emit HTML. Reach is not without disadvantages, however. For one, it's quite easy to write incorrect ERB templates which emit invalid syntax, such as missing a close tag. Additionally, code with side effects can be difficult to refactor because execution order is critical. You can trivially generate SQL statements with an ERB template, but you'll quickly accumulate an incomprehensible mud ball, so you're better off working with a system that treats queries as data and lets you manipulate them structurally. Unfortunately, you'll need to implement a unique structural tool for each target data structure. If you're generating a lot of HTML, then using HAML is a great choice. But if you're only generating one HTML file among a sea of dozens of other arbitrary config file formats, then you really want ERB.

Generalizing, structural templating systems are really just plain old pure functions. You take some data, let's say describing a blog post, and then you return some other data describing the rendered representation of that same post. This is a powerful realization because it enables you to utilize your language's full set of utility functions for refactoring your templates. For an HTML example, see Noir's partials and their use of Hiccup.

When you get into the wonderful world of metaprogramming, templating systems encounter a series of new problems. First, there is multi-level escape character hell. Anyone who has ever tried to write a shell script, which substitutes arguments into an Awk script, which generates a config file, knows what I'm talking about. Second, there is the issue of names. Unlike HTML or many simple config files, code generally have one or more notions of context, such as scoping or namespaces. You need to worry problems like variable capture. Yikes!

Early lisps discovered a mechanism for minimizing the pain of escape character hell: quoting. Quoting is similar to escaping, in much the same way that structural templates relate to procedural ones. Escaping lets one combine any two languages, provided the language on top provides a uniform escaping policy. However, with each new layer of languages, there is a new layer of escaping rules and, if those escape sequences conflict, you'll have to double-, or even triple-escape common metacharacters like apostrophes, backslashes, and dollar signs. There's no protection against incorrectly escaped strings and refactoring is thwarted by the need to increment and decrement escaping levels. In contrast, quoting takes advantage of Lisp's homoiconicity to simplify template indirection into four composable, primitive operators: quote, syntax-quote, unquote, and unquote-splicing.

[If you're already a Clojure syntax-quote pro, you may want to skip down to the next section]

Due to their high frequency of use, both Common Lisp and Clojure reserve some of their limited syntax for these primitives. Common Lisp uses the apostrophe ('), backtick (`), comma (,), and comma-at (,@) character sequences as shorthand for quote, syntax-quote, unquote, and unquote-splicing respectively. Because Clojure treats commas as whitespace, it substitutes tilde (~) and tilde-at (~@) sequences for the unquoting operators. I'll use Clojure's notation for my examples.

The quote operator is used to protect symbols from evaluation. If I have a symbol x with value 5, then the expression (quote x) or 'x does not evaluate to 5, it evaluates to the symbol x. Quoting is distributed recursively throughout a form. '(f x) will protect both f and x from evaluation. It's this distributive property that allows quoting and unquoting to compose across multiple levels of templating.

You might guess that (unquote 'x) or ~'x would return 5, but that operation is performed by the eval function: (eval 'x) does return 5. Trying to unquote here will generate an error. This distinction is because eval is generally implemented as a function, not a macro or special form. It does not alter the interpretation of its inputs, so it's behavior doesn't play nice with the distributive property of quoting.

Unquoting comes into play once you introduce syntax-quoting. Syntax-quotes are demarcated by a backtick, which can be thought of as switching Lisp into template mode. Like regular quotes, syntax-quotes are distributive. Unlike regular quote, which leaves its input form alone, syntax-quote transforms its input in two important ways: First it enables unquoting. You can think of unquoting like template variable substitution. Second, it resolves symbols. We'll come back to this second point later.

Let's say that you want to generate the code (f 1 (g 2 3)) where the numbers are provided by variables x and y holding the number 1 and the vector [2 3] respectively. You can accomplish this with the expression (syntax-quote (f (unquote x) (g (unquote-splicing y)))). Well, sort of. Clojure does not provide direct access to syntax-quote as a symbol. You need to use the shorthand form: `(f ~x (g ~@y)). Luckily, that's nicer to look at anyway.

Unquote substitutes in a single value, like a dollar-sign in a shell script. Unquote-splicing is a little bit more interesting. When a syntax quote is transforming its inputs, it looks for unquotes and evaluates them. The evaluated result is substituted into the parent expression. Regular unquotes are substituted directly, but splicing unquotes cause the parent expression to be rewritten as a concatenation. The expression `(a b ~@x c d) is transformed into an expression resembling (concat '(a b) x '(c d)). The splicing operation is a more powerful alternative to a traditional templating language's looping constructs. It can also emulate a traditional templating language's if statement. The expression `(f ~@(when b [x])) will return code that passes the x argument to f only when b is true.

As discussed previously, quoting and unquoting are distributive. This is what allows you to escape from escape character hell. The expression `(f (g x) ~(h y `z)) distributes the quoting behavior down to f, g, and x, but the unquoting operator decrements the quoting level, such that neither h nor y are quoted. The z is explicitly quoted, re-incrementing the quoting level. You can't do that with escape sequences, because, like a procedural template language, there is no explicit notion of the inherent tree structure. It's worth noting that most lisps don't have an explicit quoting level, instead syntax-quote processes its inputs recursively; the quote level is maintained implicitly by the execution stack.

That's not all you need to know about syntax-quote; there's also symbol resolution. Now that you've escaped escape hell and you're writing structural templates, you're able to do bigger and better metaprogramming. If you're writing Common Lisp, you'll quickly run into the aforementioned variable capture problem: You'll generate code that includes name collisions. Clojure's syntax-quote differs from Common Lisp's by providing two mechanisms to combat this problem. First is namespace resolution. Clojure will expand `inc into clojure.core/inc and `foobar into user/foobar. This protects you from free symbol capture. Secondly, Clojure provides automatic gensyms. Symbols ending with a # character inside a syntax-quote are replaced by an automatically generated name. So the expression `(x# x# y#) will expand into something like (x__1__auto__ x__1__auto__ y__2__auto__), which is virtually impossible to cause macro argument capture. As linked several times, Paul Graham's On Lisp is the best source for understanding these issues more deeply.

[Experienced Clojurians can start reading from here]

OK, so now that you have an extremely powerful template language baked into Clojure's syntax-quoting mechanism, you still have the problem of reach: What good is a template language if you can only generate Clojure code with it? What if you need to generate some HTML? Or what about that Awk script? Unfortunately, the template language can only really yield Clojure forms. Luckily, Clojure forms are relatively rich with a variety of primitives and collection types, so much of the Clojure community uses them for configuration files and DSLs. If you're generating input to these Clojure systems, you're in good shape. You don't have quite the same reach as "all tools that operate on a stream of characters", but "any library in the Clojure ecosystem" is a pretty damn good start. If you're going to create a system to structurally generate SQL queries, you'll need to build some kind of query representation that can be compiled to a SQL string. In a non-homoiconic language, you'd eventually need to come up with a new structural template engine too. In Clojure, your query library will get that template engine for free!

However, there is still a problem. When you first encountered code duplication, you reached for a simple procedural text template. Now you have the escape sequence problem. Lisp mitigates the escape sequence problem with quoting. But once you have larger scale templates, you start running into the variable capture problem. So Clojure solves the variable capture problem with an enhanced syntax-quote. Well just as each prior solution yields a new problem, so too does Clojure's enhanced syntax-quote.

The problem is that Clojure tightly couples the syntax-quote symbol resolution to Clojure's namespace system. This isn't usually a problem if you're always using syntax-quotes to generate code that will ultimately be executed in your local Clojure environment. However, consider the case of generating code to be executed in a *remote* Clojure environment. Or what about generating Clojure forms that will be compiled to SQL procedures to be executed on your database server? In those two cases, it's less likely that resolving symbols against the locally loaded namespaces is a useful behavior. Suddenly, Clojure's powerful template language is unavailable to you! Either you have to piece together your own template system, just as a non-homoiconic language, or you need to play tricks by manually or procedurally defining vars. Ouch.

Enter Backtick.

Backtick is an absurdly simple library. It provides Clojure's syntax-quote reader macro as a normal macro. You can create new template quoting macros by providing a custom symbol resolver. Check it out!

ClojureScript Projects

2012-11-22T03:51:00.002-08:00

I stumbled across Clojure a few times, but I've fallen in love with it sometime over the past year. I've always loved the idea of Lisp, but I never loved any particular Lisp until Clojure.

The name "CLoJure" comes from C#, Lisp, and Java. The best parts of each of community clearly shine through. Bolstered by the Java community, Clojure combines all the expressiveness of a Lisp in a package that makes everything seem as easy as C#. That last point deserves elaboration: C# has managed to empower the average code monkey with lambdas, closures, generics, expression trees, and futures. Similarly, Clojure empowers the average senior developer with immutability, macros, flexible dispatch, software transactional memory, interactive development, and a whole lot more! Plus, it's free and open source! Rich Hickey has an incredible talent for distilling good ideas down into understandable bite sized chunks, making agreeable tradeoffs, and then deftly guiding the community.

Lately, I've been spending a lot of my spare cycles working on ClojureScript. Hacking on the ClojureScript compiler and runtime has been more fun than I can ever recall in my programming career. It's an incredible toolkit that I intend to utilize heavily in my next startup, so I feel like my hours and hours of fiddling with cljs.core is extremely worthwhile. I've already contributed 20+ patches!

As fun as hacking on ClojureScript is, it's probably a better idea to hack on something that uses ClojureScript, rather than ClojureScript itself. Paul Graham described Hacker News as an application to sharpen Arc on. Paul is right in saying that you need the top down pressure of a real application to motivate the design and implementation of the language from the bottom up. Although where his goal is brevity, Rich's goal is simplicity. While Paul has seemingly effortlessly scaled Hacker News to a 1M+ daily pageview community, Rich has seemingly effortlessly reinvented the database.

ClojureScript hasn't found its killer application yet. There is a lot of interest around the project, but only a handful of active contributors and production users. People like Chris Granger are pushing down on the top with ambitious applications like Light Table, but we need others. Chris is starting to make re-inventing the IDE look effortless, but his application prefers to be installed outside the browser, generally on Macs, so it isn't clear to me if his vision remains squarely in the HTML/CSS/JavaScript-targeting, client/server sweet spot.

It has been over seven years since Gmail and Google Maps has made us re-think what is possible in the browser. I sense that ClojureScript holds the key to the next generation of re-thinking what's possible in client-side applications. As I ponder my next startup, I'm going to be on the lookout for just such an application. In the meantime, I'll be open sourcing some of the technology I'm building as I explore that path.

Personal Update

2012-11-02T15:53:00.001-07:00

Thinkfuse was acquired by Salesforce in June. I spent a few months working at Salesforce's office here in Seattle, but I badly felt the need to get back into the startup world. I'm not really sure what I'm going to do with myself quite yet, but feel free to send me an email, if you want to grab a coffee.

Learn to Read the Source, Luke

2012-04-16T14:13:00.002-07:00

Apparently, I'm Jeff Atwood's guest blogger today: Learn to Read the Source, Luke

Source code access is near and dear to my heart, so head on over there and read it!

REST: One Thousand Inconsequential Decisions

2012-02-26T19:23:00.001-08:00

Hardly a week goes by without yet another Hacker News front page post by a blogger making a bold declaration about some important thing you absolutely must do if you want your API to be "RESTful". With Rails' recent PATCH announcement, REST is all over the front page again.

In my experience designing and consuming APIs of varying levels of RESTfulness, I've concluded that most of the decisions involved in designing a RESTful API are completely inconsequential. PUT vs POST vs PATCH? HATEOAS vs manual URL construction? JSON vs custom Mimetype? It's all the same to an API consuming developer. It might matter to some kind of generic crawler, but I'm not building an API for crawlers, so I don't care.

What follows is a series of rants about various parts of the grand REST debate. This is by no means comprehensive. My goal with this post isn't to discourage you from writing a REST API. I simply want you to spend your time on the API decisions that matter, not the cosmetic, inconsequential ones.

HTTP Verbs & Status Codes

The Rails PATCH announcement states "both PUT and PATCH are routed to update". So what's the difference? Apparently PATCH is for partial updates, where as PUT is for complete replacement of resources. Here's the funny bit: Both of these are implemented via POST with a _method parameter for support of older browsers. The only two HTTP Verbs that actually matter are GET, for idempotent, cachable requests; and POST, for side-effects. Even DELETE is insufficiently defined for any real application. What if you want to support both Archive and Delete like Gmail?

Which response codes should you use? Those which have specific meaning to HTTP clients. Specifically: those treated differently by browsers. Should you use 201 on create instead of 200? Who cares? Should you use 301 or 302 for redirects? In an API, it doesn't matter. Pick one. Document it. For your actual web page, consult an SEO guide.

What about errors? If you don't have permission to see a resource, is that a 403? Or do you hide the existence of the resource and return a 404 (like GitHub)? It doesn't matter. Return a 4xx for expected caller errors and a 5xx for unexpected errors. Pick one. Document it.

The smart thing to do is to always return your resources within one level of nesting. That is an item key in JSON or an item element in XML. Do this so that you can add an error key or other metadata without breaking changes. For example, you may want a timings key for perf debugging. For lists, use an items key because pretty soon, you're going to want a paging key. Also, for errors, make sure you include a non-localized (and hence, Google-able) error string; a status code is insufficient.

Hating On HATEOAS

HATEOAS, an absurd mouthful of an acronym, is a complete waste of time outside of read-only semi-structured datasets. If you're FreeBase, then go to town with HATEOAS conventions that are useful to you. However, for your typical webapp API, you simply cannot build a useful API client application without deep knowledge of the problem domain and the available API calls. You're going to have API documentation and you're going to have to read it.

By all means, feel free to use HTTP 201 and a Location header to return the location of a newly created resource. Or you could just return the resource as JSON with an href or id key. That's yet another inconsequential decision. Might as well go with the path of least resistance: return interesting data in the body of the response (ie. in the JSON) where it can be accessed without additional effort to parse the headers. As long as your response code is in the 200 range, no one is going to look at it further.

I've seen people advocate Link headers such as rel="newcomment". It's weird to me because you still need to know which rel to lookup. The same thing applies to returning a bunch of extra URLs as metadata in your JSON. You still need to know which key to look up for the post-new-comment URL. You might as well just appended "/comments" and avoid the indirection. Link and rel are useful to generic tools that want to treat a wide range of resources the same. There aren't enough common aspects between the domain models of a typical API to justify the spare brain cycles. Again, if you're Wikipedia, your use case is different. You have generic data that you want to traverse generically. And again, that's a mostly read-only use case.

Versioning

Some REST fans advocate the use of custom MIME types for API versioning. These folks believe that version numbers in URLs are ugly. The usual objection is that resources should be equitable by URL and that /v1/users/5 and /v2/users/5 are different resources. The problem here is that this resource is actually identified by just the number. You don't need the whole URL and practical constraints are going to make it impossible to equate resources by URL. Consider the constraint of browsers' Same Origin Access Policy. You need api.example.com to run your public API on it's own IP address. And you need example.com/api/ to make calls from your web frontend without jumping through crazy IFrame interop hacks.

There are two types of API changes: 1) Backward Compatible, and 2) Breaking Changes. In the case of backwards compatible changes, you don't have to, nor should you, change version numbers. Simply add the additional method or additional return data. Nothing to worry about. In the case of breaking changes, you want to break clients as loudly as possible. If you simply change the MIME type, most consumers of the API are totally going to ignore it. If you deploy your new API at api2.example.com, then there is no risk of breaking older clients and they have to make a conscious decision to upgrade.

In general, you're probably going to want to protect yourself with API keys. API keys provide the best way to version your API: record the desired API version with the generated key. Just stick an api_version column on your api_keys table. Expose that column as a drop down box in your API key management UI. This solution let's you leave the "ugly" version number out of the URL, but still enforce breakage on breaking changes. The best part is that you can totally ignore API keys and versioning when you start: Simply default unregistered API consumers to use v1 when you do implement v2.

Last word on custom MIME types: One of the great promises of REST is the ability to treat generic things, generically. If your MIME type isn't a standard JSON MIME type, then how can you expect your browser to render it nicely with syntax highlighting and code folding? Your data is structured as JSON or XML; if you want to version the schema, use the facilities provided by XSDs or JSON Schemas, etc. But you're better off with documentation written by humans, for humans, than you are with formal schema.

REST: The Good Parts

There are a lot of good reasons why REST has gained popularity. For one thing, it isn't SOAP.

The good ideas in REST are primarily predictable URLs, contractual idempotence, the preference of nouns over verbs, and CURL as a client library. In practice, you can get contractual idempotence via GET vs POST. Predictable URLs are no different than predictable function names in all typical library code.

All this other stuff, the finer points of URIs and HTTP headers, are completely inconsequential to a developer programming against a well documented API.

Plugging a hole in Microsoft's hiring pipe: IE Frame

2012-01-14T18:22:00.000-08:00

You're probably familiar with Google Chrome Frame, Google's clever workaround to combat Internet Explorer dependency in enterprises. You're probably also aware that Microsoft is not a fan.

Microsoft desperately needs to implement an equivalent "Internet Explorer 10 Frame" mass deployed as a high-priority Windows Update. Failure to do so all but guarantees that Microsoft will become completely incapable of hiring a sufficient supply of talented graduates.

The key issue is that two of the major vectors for aspiring engineers are no longer dominated by Microsoft technology.

Javascript is the New Basic

Microsoft QBasic was my first experience with programming, but I got started a few years earlier than most of my peers. The students I graduated college with largely had their first foray into text editing via HTML. Browsers are installed on every computer on the market and web development skills are vital for nearly all professional developers (even those not doing full-time web development). This puts Microsoft in a really bad spot: every budding developer's first impression of Microsoft is "IE SUCKS!!!!11!!one!".

Microsoft's Stranglehold on Gaming is Loosening

When I was employed by Microsoft, I took an informal survey of my recently hired peers. I worked in the gaming division, so my view was skewed: they almost all got into programming via PC gaming. If you loved games, you ran Windows. If you wanted to make games, you learned Microsoft Visual C++. However, times are changing. Apple's iPhone is the biggest gaming platform on the planet. Your family computer is probably a Mac by now and your Xbox doesn't come with a compiler. Why not learn Objective-C? Furthermore, canvas games are popping up everywhere and it won't be long before WebGL is common place. Soon enough, a significant subset of aspiring game developers only have to boot Windows to test something in Internet Explorer. sigh.

One year after accidentally falling in love with Rails

2011-09-24T17:40:00.000-07:00

About a year ago, I wrote a post titled How two Pythonistas accidentally fell in love with Rails. Some people predicted that the honeymoon would be over quickly and that I would soon hate it.

So, do I still love Rails?

Yes. Absolutely.

The prototype and first beta version of my startup was built entirely on Rails. The framework enabled us to build our site significantly faster than we would have been able to without it. For that, I still love Rails. It was critical in helping us get to where we are today.

It's the perfect hammer for the v1, nebulous direction, rapid development, nail.

That said, we're not using Rails for any new components.

After getting a lot of feedback from our first beta, we realized we had the wrong approach to the UI. Now, in our second attempt, we've got a much richer client-side front-end, written in CoffeeScript and bootstrapped by Node.js. Ruby/Rails is simply not the right tool for that job. In the new world of the browser app, even the folks of 37signals have been accused of propping up their successor.

Luckily, we had a pretty good handle on the models in the problem domain. So I ran `git rm -r app/views` and effectively turned the Rails app into a Json API backend. The models have evolved, but not by much. In fact, there were a few months where the Ruby code remained practically unchanged while we developed the frontend.

As time goes on, we've been breaking our API layer apart into several services. It makes development, deployment, and versioning easier. Service Oriented Architecture is working great for us... now that we know what services we need. These services are each too simple to justify the overhead of Rails. Some are Node.js, some are Sinatra, some are cron jobs and shell scripts. We're polyglots, so we're careful to choose the right tools for each job.

We'd have gotten it very wrong if we tried to build services from the start. The monolithic Rails app was the correct evolutionary intermediate form.

Given a time machine, would I have done anything differently?

Only one thing: I wouldn't have used ActiveRecord.

I guess I should have seen that one coming. ORMs are an insidious, broken abstraction. At first, they save you time and make you feel productive. They work out splendidly for a while, but like a virus, they infect every part of your application. One day, you hit that abstraction brick wall and it's too late to do anything about it without significant effort.

Are there things I hate about Rails?

Ignoring ActiveRecord, only one thing really stands out: Performance.

I'm not even talking about page load times. I'm talking about development time.

Even with our modestly sized application, the impact on productivity is atrocious. Our test suite runs far too slowly to be waited on for every checkin. Rake's startup time is too great to for its usage to be considered "interactive". Simple shell scripts have replaced practically all of our custom tasks; they are instantaneous.

Isn't this basically the same story I told about Django?

Yeah, I guess it is.

The story basically goes:

Use a framework
Be productive
Encounter shortcomings
Stretch it to work
Watch it break
Replace components
Loop steps three to six
Be productive
"Where's the framework?"

I still stand by my original criticisms of Django (although, I'm sure much as changed in the last year). I'm also re-affirmed in my belief that big frameworks are generally bad. It just so happens that Rails is good enough and matches my mindset well enough to justify the framework pains.

Stupid brain trick: Tests and vertical splits

2011-08-16T23:57:00.000-07:00

Recently discovered a stupid brain trick that I thought was worth sharing.

When writing code with unit tests, I find it helpful to view the tests and the implementations side by side. I use vertical splits in Vim to do this.

The stupid brain trick is this: Put your tests in the left-hand split!

Why? I traditionally used the left-hand split for the "main" thing I'm working on and the right-hand split for reference. I've watched many other developers do the same thing. The right split is constantly sub-split horizontally and always changing to various different files.

In trying to be better about writing tests first, I created a test code file before the implementation file, so it just happened to be in my left-hand split. After an hour or so of development, I realized that the "main" file placement of the tests meant that I focused more on them. I viewed the actual implementation file as just some ancillary file I had to tweak to get my tests to pass. It changed the focus of what I was doing from "Write a method that does X" to "Describe X". Describing something involves breaking it up in to smaller and smaller pieces until the implementations are so trivial that they basically write themselves.

Thinkfuse is hiring!

2011-03-28T15:08:00.000-07:00

If you caught the recent TechCrunch coverage, then you know that things are heating at my startup, Thinkfuse :-)

We've got lots to do, so we're looking for a kick-ass engineer who can get up and running quickly. My co-founder, Steve, and I are full-stack developers who really give a shit, so we're looking for a Seattle-based developer who can keep up, or better yet, force us to struggle to keep up. Rails experience is a plus. Good sense of humor and beer drinking skills, a bigger plus. We're offering a solid salary and meaningful equity.

If you're interested, reach out to me at brandon@thinkfuse.com

Where Rails Rocks (and where Python doesn't)

2010-09-17T11:31:00.000-07:00

Here’s a list of things I like about Ruby on Rails. Almost every single one of these are available in at least one Python framework in some form, but the best thing about Rails is how polished the pieces are and how well they all fit together. The whole is truly greater than the sum of its parts.

Fantastic OOBE
Rails has a fantastic out-of-box experience. Simply execute rails new and you’re off to the races with a directory full of goodies that every real web app needs: dependency management, build system, multi-environment configuration, development storage, logging, custom error pages, robots.txt file and more. See section 3.2 of the getting started guide for a complete list. Hell, they even fill out a .gitignore file for you! A place for everything and everything in its place.

Dependency Management: RubyGems
In the Python world, easy_install and its ilk are all kinds of broken. You can’t even uninstall a package! Luckily, there is Pip and Virtualenv, but assuming Pip supports all your packages, you’ve still got to make sense of dependency freezing and manually setup a virtual environment. With Rails, that all just works: rails server runs in a virtualized environment and loads gems as defined by your Gemfile; rails console for a repl. No need to muck with sys.path! Any seasoned developed has spent many an hour in dependency hell, but it is still hard to justify the infrastructure investment before writing a line of HTML. Bonus points for the “vendor” directory, in case you need to patch a third party package (or Rails itself) and want to keep it under version control.

Build System: Rake
I’ll admit it: I like Makefiles. A lot of people really hate them, but they are just so damn useful. Frequently, you need to transform some set of files into some other set of files, but only when they change. Makefiles help you do that reliably. They accomplish their feat explicitly by not being a programming language, but a dependency and build rule declaration language. I’ve seen a lot of attempts at build systems that aim to be idiomatic with respect to their host languages. Java has Ant and Maven. Python has Scons and others. Scala has Simple Build Tool. None of them are as good as plain old Makefiles. Then, I discovered Rake. Rakefiles rock. They are 90% Makefiles, and 90% Ruby. Rake expertly uses the features of Ruby to craft an API/DSL that looks a lot like Make declarations without compromising on general purpose programmability.

Markup and Style Sheets: Haml and Sass
Haml and Sass are so amazing, they ought to be illegal. Just click those links and check out the HTML and CSS comparisons. It is no contest. I never want to write HTML or CSS ever again. As far as I’m concerned, HTML and CSS are object code: to be generated by a compiler. Whenever I have to look at the generated code, it feels like I am reading assembly. Simply add gem ‘haml’ to your Gemfile, then start writing .html.haml files instead of .html.erb and .sass files instead of .css. Use html2haml and sass-convert to upgrade your crusty old markup and style sheets instantly.

Object Relational Mapper: ActiveRecord
I’ve discussed my preference for database schema reflection at great length, so I won’t reiterate here. I will, however, say that ActiveRecord makes simple things simple and complex things possible while embodying successful best practices in its various subsystems, like migrations. It doesn’t try to be everything to everyone, unlike SqlAlchemy. And it seems to be nicely onion layered for pealing back when necessary. For example, ActiveModel validations can be used on plain old Ruby objects. Which brings me to...

Form Handling: FormBuilder, params and mass assignment
I don’t need to learn and manage three different data marshaling techniques to map a database record to a web form. In fact, I don’t need to manage any. Schema reflection spares me in the model layer, FormBuilder/FormHelpers in the views, and the hash & array structured params in the controllers layer. Almost every user interaction boils down to interpreting a form’s worth of JSON-like data structures as an API call, implemented via a (secured) mass assignment to virtual attributes.

Templating: Layouts, Partials, and Helpers
When I’m getting my Haml on, I frequently want to run some non-trivial code. Feel free to shout about your separation of concerns religion, but I’ve got real things to worry about. Furthermore, we as engineers should be writing our production views/templates, not some hypothetical designer who is technical enough to write loops and branching statements, but not to be trusted with a function definition. Views are composed of markup and presentation declarations in the templates, presentation logic goes in helpers (which conveniently share scope with templates and controllers), and are neatly organized into partials. Simple, but effective.

URL Design: Routes
Rail’s routing system makes clever use of blocks to provide a hierarchical, declarative URL map that just makes sense to read. I used to design URLs in an indented text file and then map it to a list of regexps, but the routes.rb file is close enough to a spec and doesn’t require me to even think about regexes, ‘nuff said.

Authentication: Authlogic
Authentication is hard. Really, really hard. And boring. Really, really boring. But important. Really, really important. Everyone knows that the sign-up funnel is critical to get right and security is easy to get wrong. Most quality sites have a little bit of domain-specific special sauce in their authentication model. Trivial sign-up and log-in forms just don’t cut it. One size simply does not fit all for authentication front-ends, but the authentication back-ends tend to be pretty repeatable. Authlogic turns repeatable into re-usable, without the baggage of views you’re going to need to rewrite anyway. Thinking about authentication as CRUD operations on a user-session model blew my mind slightly. I had a full authentication system tuned to my needs up and running within two hours.

Authorization: CanCan
I hate ACLs. Role-based authorization simply has too much impedance mismatch for most problem domains. CanCan is a clever little library which makes zero assumptions about how permissions are represented. Simply declare a list of everything a user can do and how to tell if they may. Query with the can? function, or assert with the authorize! method. Beautiful.

Rope to Hang Yourself: Ruby Itself
One keyword in my previous post triggered much discussion: magic. The concept of magic is a topic for a whole ‘nother blog post (as I’ve already been hung a few times), but as several commenters pointed out: it’s all just Ruby. Between mixins, include, require, blocks, method_missing, symbols, and many other features, Ruby is a very capable internal domain specific language development toolkit. Most of the time, you don’t want a DSL. If you’re not an expert in both the host language and the DSL, there is almost certain confusion and maintenance danger. Most projects are either small or distinct enough to justify writing more verbose code that can more readily be understood by successors or even your future self. The gains of a DSL do not always outweigh the costs. Python excels at not supporting DSLs; it is basically executable pseudocode. However, when it comes to a task as common as web application development, it’s worth the time to develop or learn a DSL or two. Rails is loaded with DSL features and “magic” behavior that increases the learning curve, but also increases peak productivity.

Community
These sites have been invaluable: The Ruby Toolbox, RailsPlugins, Railscasts. The quantity and quality of Ruby libraries on Github continues to impress. The educational infrastructure is extensive. Somehow, the Ruby community seems to consistently be at the forefront of the biggest trends in software development. Much like Rails compared to other frameworks, the community distinctions are subtle, but meaningful. For example, Rubyists championed Git while Python adopted Hg. To the untrained eye, they are virtually identical tools, but extensive experience with both has convinced me that Git is significantly superior. I’m new to the community, but it seems like Rubyists tend to insist on quality and bet on the winning horses.

Other Random Things

Javascript and CSS minification are trivially easy with plug-ins
Static files are served with version stamps for better caching behavior
Console and file logging are usefully configured by default
git push heroku
rails.vim
ActiveMerchant looks awesome
Only been working with Rails for two weeks; lots more goodness to uncover

Just who the hell do I think I am?
I’m nobody. Just some opinionated computer geek working on a start-up. Not unlike many of you! My co-founder and I are currently participating in TechStars Seattle and hope to launch something killer come demo day, November 11th, 2010. We’ve got a grand vision for next generation enterprise collaboration software, but we’re starting small by focusing on making weekly status reports less painful and more useful. More details soon, but please visit ~~http://www.thisweeklastweek.com/ to register for a beta invite~~http://www.thinkfuse.com. Thanks!

How two Pythonistas accidentally fell in love with Rails

2010-09-13T02:33:00.000-07:00

One week ago, my co-founder and I found ourselves staring down a thick stack of paper UI prototypes; almost 50 pages total. One week before that, we threw out a perfectly good, working beta product because customer feedback suggested a significantly better path. The tale of how we ended up at that point is entertaining as well, but it will have to wait for another time. This is the tale of how our paper prototype turned a pair of long time Python developers (and documented framework haters) into Ruby on Rails developers.

Paper prototyping proved to be an extremely worthwhile exercise, but we really wanted something we could click around. We devised a strategy to build a clickable prototype that would evolve into production code. The plan was to stub out all of the views with static HTML and CSS. The CSS could be directly carried over to the product and the HTML could be retrofitted as templates. Having become religious about Sass about a year ago, I fired up the file watcher (which automatically rebuilds CSS) and set to work filling a directory with sass and html files.

After some time, I asked myself “I wonder if Haml has a --watch argument?” (it doesn’t) because Sass makes me hate Html’s verbosity. Armed with practically zero Ruby knowledge, I hobbled together a simple Rakefile and Haml watcher script and got back to work. A few Git pushes later, my co-founder, being the clever bastard that he is, simply installed Rails and the Haml plugin to replace my hacky scripts. “Neat,” I said and thought nothing of it.

Then we designed our URLs in an indented plain text file, which pretty much directly mapped into the routes.rb file and some trivial (empty method) controllers. Whoops! Now we’re writing Ruby.

“At this point, we might as well try Rails proper.”

Generated some models and corresponding migrations: “Wow, explicit schema with reflective model objects. None of this declarative schema bullshit.”

Wired up some controllers. “The structured params infrastructure is pretty cool.”

“Ruby is kinda fun. Blocks are super useful.”

Integrated Authlogic. “Wow, that was easy. I really like how it does not force any front-end decisions on me.”

“Rails is full of evil black magic that I do not understand, but it seems to read my mind, so I don’t really care.”

Started to apply some polish and work on the 10% that takes 90% of the time. “I think I can safely add Ruby to my resume now.”

Late at night, at the very end of the week, exhausted, we stared at each other. “Dude, I think we can show this off to people tomorrow.” Walking out the door of the office, my co-founder said to me “I feel so dumb for having ignored Rails for so long. I just assumed it was Ruby’s Django”.

We accidentally implemented our entire beta product in one week, with zero prior Ruby or Rails experience.

When I have some more time (and experience), I’ll try to write about specific design decisions that set Rails apart.

NoScript Add-on Install Base

2010-06-04T19:37:00.001-07:00

The official NoScript site downloads via the Mozilla Add-ons page, so its download count is probably accurate: 67,616,402

NoScript does not publish their individual add-on usage statistics, but the global download/usage ratio can be calculated from the statistics on the Firefox Add-on home page:

1,962,617,946 add-ons downloaded

157,090,095 add-ons in use

About 8% of downloaded add-ons are still in use. Assuming NoScript's usage ratio is comparable to the average, approximately 5.4 million installations of FireFox are running NoScript. Let's ignore the fact that NoScript's usage ratio is probably much lower than the average, due to the fact that it breaks most web pages.

I'd wager that the average NoScript user has at least two machines, so the total number of NoScript users is probably less than 2.7 million.

There are over 230 million internet users in the USA.

Even if 100% of NoScript users were Americans, they form 1% or less of the general population. If you, like me, believe that I have been generous to NoScript here, it is likely that no script users are no more numerous than 1 in 1,000.

Even if the user is using NoScript, they can whitelist your site. You can probably add <noscript>WARNING: THIS SITE IS BUSTED WITHOUT JS</noscript> to the top of your page and call it a day. If you are feeling generous, redirect no-script users to the mobile version of your site and tell them why.

tldr: Assume that human user agents have Javascript.

Leaving Microsoft

2010-04-14T14:55:00.001-07:00

April 30th will be my last day on the Microsoft XNA team. I'm leaving to start work on my own startup full-time. I also hope to invest some more effort into this blog. Exciting times!

ORMs and Declarative Schemas

2009-10-06T10:32:00.001-07:00

My prior post was more controversial than I anticipated. In hindsight, I should have realized what a hot button issue web frameworks are. One assertion went all but unnoticed. I expect it may be even more controversial: the schema-generative ORM paradigm is fundamentally flawed.

Disclaimer

I came to this conclusion while working with Django's ORM, but this post is completely object-relational mapper agnostic. We are now using SqlAlchemy, but we are not using any of the many available declarative layers. Instead, we are using schema reflection and semi-automatically configured mappers. This is not an argument against ORMs. It is an argument against generating database schemas from ORM declarations. By extension, this is an argument against Django's ORM because Django uses an exclusively schema declarative model. That said, Django's ORM is far from alone in this camp.

Data Outlives Code

When code is dead and gone -- be it through rewrite, obsolescence, or by other means -- the data will still be there. Longevity implies slower evolution; data is always more difficult and riskier to change. Data is also more valuable. What if Facebook rebooted their database? They've already rebooted their software several times.

Schemas are data. As data, schemas are longer lived, less flexible, and more valuable than code. These factors alone suggest that the database itself should hold the authoritative schema, not a class declaration in the code.

If you have inherited data from another project, you already know this lesson. You can't generate the schema from code because the schema already exists. You can mimic the authoritative schema in your declarations, but it is easier and more accurate to use reflection.

Ineffective Domain Objects

Object relational mapping is primarily a serialization problem. Every serialization solution has its quirks. The scale and number of quirks seem directly proportional to the absolute difference between the runtime and storage representations. Since a database is a completely distinct type system, rather than an opaque byte array, serialization to a database can have particularly quirky quirks.

Modeling is one of the fundamental challenges of software development. Capable developers prefer highly expressive or unconstrained type systems to aid with modeling. Generally, runtime type systems are more expressive than those found in databases. Declarative relational mappers, however, constrain the programming language type system to its less expressive counterpart. When building domain objects, the developer must think in terms of the database's type system, not the programming language's.

While using a declarative object relational mapper, developers are effectively trying to design storage and runtime models simultaneously. On average, superior results are achieved by modeling these two concerns separately and then solving an additional subproblem: serialization mapping. You might wind up with slightly more code, but it will be easier to understand and maintain.

SQL Is Not Going Away

Despite forcing a less expressive type system on to the developer, declarative ORM layers attempt to treat the database as an implementation detail. However much we wish this were true, the database is not a detail which can be ignored. Sooner or later, you are going to have to open your database shell and write a SQL expression. This requires knowledge of your database's particular SQL dialect and idiosyncrasies. Exacerbating this issue, generated schemas are typically full of name mangling and other ugliness. It is far more pleasant to work with a carefully designed schema than one that compromises for the ORM or the runtime type system.

Schema and Declarations Diverge

Data migrations present the most pressing need to work with SQL directly. As an widely unsolved problem, automation can not be trusted. Unless you are painstakingly simulating the schema generator, the production database schema will slowly diverge. Most deviations are tolerable as they will not affect the runtime behavior, but it is wise to minimze differences between production, staging, and development environments. To faciliate this, store migration scripts and backed-up schemas in version control.

Some deviations will directly affect runtime behavior. For example, consider the case where two versions of an application are running in production. An is_read boolean column was added to a message table in the database. It's default value is false. When a row's page is viewed, the is_read column is set to true. The old version of the application doesn't know about the column, so it can not set the flag. When the new version is rolled out to everyone, affected user will see a bunch of read items marked as unread! The solution is to set the column's default to true, but initialize it to false in the application. Declarations must either deliberately deviate from the schema or present a misleading default value to be overridden during initialization. This is just a simple example, real world schema migrations can be significantly more complex and suffer from numerous more subtle problems.

URL routing and views

2009-08-24T22:44:00.000-07:00

As promised, I'd like to elaborate on the URL routing system I came up with.

Weighing at less than 200 lines of code (including example), I'll let it speak for itself: download it.

This approach seems to be working great for us. Love it? Hate it? Feel free to let me know what you think.

Dropping Django

2009-08-19T00:07:00.001-07:00

My partner and I built a non-trivial web site on Django. When the next version ships, there might not be a single Django module imported.

We're not trying to drop Django; it is just sort of happening. Piece by piece, it is failing to meet our needs. Despite the marketing copy on the Django site, most components of the framework are tightly coupled enough to make customization frustrating. It is often easier to rewrite core framework components than to implement them on top of the existing extensibility points.

What follows is a loose chronology of our migration away from Django.

URL Routing

A flat list of patterns violates the DRY principal when creating nested URLs. Trees are a superior representation. Having a tree of views also enabled us to optionally associate a "binder" function with each node. These bind functions are executed for each URL component from left to right, filling the template context as they go. Breadcrumbs are automatically generated as each node binds, but only the last node executes its full view logic.

Authorization

Our site enforces permissions on every resource, but Django's database ACLs would have been prohibitively numerous. Instead, views or their URL binders may raise an AccessDenied exception. Upon catching such an exception, a middleware layer serves a login form. This ensures users have permission to access the current resource, as well as all ancestor resources bound to the URL.

Authentication

Both of Django authentication's key extensibility points are flawed. These two extensibility points are "user profiles" (storing additional per-user data) and custom credentials (such as for logging in via email address instead of username). Django's documentation and numerous internet sources cover both topics, but all of the guidance lacks important caveats. The admin UI, in particular, is very easy to break with either extensibility mechanism.

Extending the User model with the ORM requires a one-to-one database relationship. This relationship can be implemented with a "user profile" setting, an explicit foreign key, or model inheritance. Each approach has its own strengths and weaknesses in terms of performance, API semantics, subtle behavioral changes, and outright bugs.

Enabling custom credentials requires implementing a trivial authorization "backend" object. Unfortunately, it is non-trivial to replace usernames with email addresses. The admin UI's login form refuses to accept email addresses without hacking the template. Even if you hacked the template, the User model would still enforce a non-null constraint on the username field and the generated database schema enforces a uniqueness constraint as well. It turns out to be easier to fill the username field with a dummy value and "support" both forms of authentication with your backend, but you won't come to that conclusion until your head has already bore a hole in your desk.

Templating

We do our best to keep view and template logic separate. Django's templates are targeted at designers, who aren't implementing any real logic anyway. However, we're a pair of hackers. Sometimes it is just more convenient to put a little bit of logic in the views. Besides, templates are code; code needs to be reviewed and tested. We wouldn't ever hire a designer who couldn't pass a code review for some trivial template logic.

We needed a pragmatic template language to replace Django's idealistic one. Any template language with greater expressive power would have been welcome, but Jinja2 fit the requirements and provided the easiest migration path. Ultimately, we'd prefer to use something like HAML, but there doesn't seem to be a Python equivalent besides the inactive GHRML project. We are, however, using SASS. I will never write CSS by hand again.

ORM and Admin UI

One of Django's most touted features is the Admin UI. For simple "active record" style database models, the Admin UI is a huge time saver. Sadly, it struggles a little bit with nullable fields and is tricky to customize. You'll definitely need to write custom UI for complex models, but by and large the admin solves the problem at hand: viewing, creating, updating, and deleting database rows.

After using the Admin for a little, I found myself missing Microsoft Access. I never thought I'd say that, but it is true. ~~Django's admin does not support sorting, filtering, or other impromptu queries.~~ Edit: It turn's out I was mistaken about sorting and filtering, but I stand by the core message of this section. I found myself writing impromptu queries in the database and Python shells. After a while, I just gave up and installed a desktop client. I haven't visited the Admin UI since.

Django's ORM has shortcomings with respect to querying, especially for joins and aggregation. It has been improving over time, but it will likely never reach the capability of projects solely focused on databases, such as SqlAlchemy. With the admin having fallen into disuse, the Django ORM lost all advantage. Beyond Django's specific weaknesses, I've come to believe that the schema-generative ORMs paradigm is fundamentally flawed. That is a topic that deserves an entire (Django-agnostic) post of it's own. We are now using SqlAlchemy via schema reflection; no declarative layer.

Form Validation and Generation

Here is where our chronology meets present day. We are still using Django form validation, but never used form generation beyond scaffolding. Nearly all of our templates customize labels and display of errors. Additionally, embedding widget information in the Python code is cumbersome during template development. Django forms is a quality validation library, but there are some inconsequential style things that I like better about FormEncode. Preferences aside, the difference isn't large enough to justify switching.

While I like FormEncode, I'm still not sold on its anti-form-generation companion, htmlfill. I think there is a middle ground with form generation that provides scaffolding during development, smoothly transitions to production use, and cooperates with validation. As we implement more complex client views, I'll be on the lookout for ways to improve our form development toolbox.

So, ugh... What's left?

Besides a few isolated helper functions, not much is left of Django.

The last big ticket item is the HTTP framework and WSGI server. We could continue using Django as if it were CherryPy or Paste, but Django has this nasty habbit of insisting on running your code for you. The settings and manage.py infrastructure are fiddly for deployment and don't really add any value over simple scripts using our application like a library. Might as well use a simpler WSGI library, and replace those over-engineered management/commands/foo.py files with vanilla scripts/foo.py files.

Moral of the Story

I'm sure there are numerous lessons to be generalized from this journey. Personally, I've developed a moderate fear of the word "framework", as well as altered the way I think about software abstractions. I think the most important lesson, however, is one I already knew: choose the right tool for the job. Unfortunately, we had no idea what the right tool was when we started. I'm not sure we know any better now.

AppWeek

2009-06-12T20:02:00.001-07:00

Shawn's AppWeek post inspired me to write one too. AppWeek is our chance to be creators for a little while and it was a lot of fun. I didn't set out to build something nearly as ambitious as "Super Avatar Sample Smashup EXTREME! - 'Capture the Cat' edition", but I did get to take a swing at a game I've wanted to build for a while: ~~Rock'em Sock'em Avatars~~ Avatar Boxing. Avatars, being a new feature in this release, were an unwritten requirement for all of the AppWeek games. Between SASSECTCE, my game, and the many others, Avatars were chasing cats, beating each other up, play futuristic sports, falling off buildings, dancing in a cloud of gems, being launched from canons to save the world, and much more. All this excitement was almost too much for a bunch of exhausted engineers, but that's what the beer was for during the game unveilings.

Here's what the game looked like with the basic animations wired up. You'll notice that the avatars have been hitting the gym. That's because their arms were too short to reach each other! I added a little extra bulk because I was laughing too hard not to. I directly bound the game pad triggers to the shoulder and elbow joints and rigged up the chase camera sample to inspect my work. There wasn't much game play yet, but it was already fun. That's always a good sign.

Even with just one week, I decided to invest some time into debugging visualizations. That turned out to be a really great idea.

Then, I added some collision spheres for the heads, hands, and upper bodies. This was a hacky, trial and error process. Thankfully, C# compiles quickly.

At this point, I spent an entire day working on the physics. I wanted the avatars to bounce/wobble when they got hit, so I rigged up some complex spring systems. Things were starting to work, but I'm generally pretty bad at this sort of thing and my simulation routinely exploded. The avatars arms went shooting off into space and I was getting pretty frustrated. No screen shots of that chaos because I am embarrassed.

With half a day to go, I added the obligatory damage bars and some rudimentary hand-to-head collision detection.

I was feeling pretty good about the game, despite my physics failures, it was pretty fun anyway. I wondered down the hall to chat with Jace, who had just added sound effects to his game. His game was hilarious before, but the sound effects were priceless. I ejected the sound effect CD out of his machine, yoinked it, and took off running. An hour later (and 10 minutes after the deadline), my game had some sweet punch and miss sounds. I also made the avatars' heads pop up when their damage bar was full, accompanied by an awesome zip-tie sound.

At our team happy hour, I'd like to think Avatar Boxing was a fan favorite. I certainly had fun making it! I hope everyone enjoys Avatar support in the new XNA Game Studio.

PowerShell: condemned to reinvent

2009-04-20T02:33:00.001-07:00

I tried PowerShell when it was first released, but never used it for real work. I recently attended a "brown bag" presentation about PowerShell. This presentation spurred me to augment our team's environment with PowerShell and I have been using it every day since.

In the past weeks using and abusing PowerShell, I have drawn two conclusions:

PowerShell has a killer set of standard tools with brilliantly designed usability.
The PowerShell team doesn't understand UNIX and therefore were condemned to reinvented it, poorly -- with apologies to Henry Spencer.

First things first: if you spend any time working with Windows, get PowerShell. Now. Stop reading my blog and go download it immediately. It mops the floor with cmd.

The key premise behind PowerShell is that it operates on live .NET objects. This is beneficial because it eliminates a lot of the text cutting and manipulation common in shell scripts. Additionally, it puts the full .NET Base Class Library into your scripting toolbox. PowerShell tools, known as commandlets, typically only render the most common fields for their objects, but the less common fields are easily available in memory. By convention, Commandlets are named with a verb-noun pattern and support a common command line parsing behavior. The repository of commandlets and the command line options of each are easily queried and highly consistent. All this meta-data makes PowerShell a breeze to learn.

I fell in love with the the discoverablity and ease of use when I tried to kill a collection of runaway processes:

PS> get-command -noun process

CommandType   Name           Definition
-----------   ----           ----------
Cmdlet        Get-Process    Get-Process [[-Name] <String[]>] [-Verbo...
Cmdlet        Stop-Process   Stop-Process [-Id] <Int32[]> [-PassThru]...

PS> get-process notepad | stop-process
PS> get-alias | where { $_.definition.contains("Process") }

CommandType   Name   Definition
-----------   ----   ----------
Alias         kill   Stop-Process
Alias         ps     Get-Process

PS> ps someotherapp | kill

OK, that's pretty cool and oh-so-very Unixy -- right? Wrong. Notice the "CommandType" column in the results of get-command. There are many other types of commands besides commandlets: functions, filters, scripts, applications, etc. Each of these has slightly different semantics for pipes and parameters. Applications, for example, have no way of accepting .NET object pipes. You must develop a separate commandlet. Yikes!

Compare to Unix: all commands are applications which accept a command line and pipe byte streams in and out. Much simpler, but byte streams aren't as friendly, discoverable, and maintainable as object streams. However, the brilliantly simple thing about Unix is that, when you get right down to it, object streams are just byte streams! There is absolutely nothing stopping you from implementing get-process and stop-process as Unix programs which pipe object references, JSON, pickled Python objects, XML, S-expressions, or any other data format you fancy. Doug Mcllroy, the inventor of Unix pipes, was right: text streams are the universal interface.

Actually, this is no different on Windows. All of the PowerShell commandlets could have been implemented as applications which import a library. This library would replace main in much the same way as winmain, provide a metadata enriched implementation of getopt, man, etc. There is no need to invent a new shell in order to acquire the power of piping objects. Sure, cmd is old and needed to be retired for many other reasons, but it is a real shame that the PowerShell toolset is not available to those of us stuck in batch scripts.

Personally, I would really like to see such a library developed. Microsoft has certainly proved one thing with PowerShell: steep learning curves are not intrinsic to command line interfaces. Unfortunately, commandlets are two steps forward and one step backwards. I have no doubt that we can retake that forward step.

Language-oriented programming: too much, too fast

2009-02-09T02:49:00.001-08:00

There was some recent discussion on Hacker News about the 2004 article on "Language Oriented Programming" by Sergey Dmitriev. With my growing interest in programming languages, this article, and the pending release of JetBrain's Meta Programming System, I have been thinking a lot about the future of programming.

As Sergey points out, MPS is not the first entry into this paradigm of software development. Intentional Programming was being demonstrated by Microsoft Research as early as 2000. Wikipedia lists several implementations of the language-oriented programming concept. (OK, that is enough links for now!) Sadly, none of these systems have been met with wide spread success. Despite my unlimited respect for the JetBrains team and love of their products -- especially Resharper -- I expect MPS to fail to achieve critical mass. I don't think many programmers would disagree. It's just too much, too fast.

Software development needs to evolve, not start anew. If you change too much at once, the common developer simply won't accept it. Language designers are still re-inventing Lisp piece by piece in an effort to make sense of all the different concepts for Mort. In order to improve adoption of language-oriented programming concepts, the critical path must be identified and the programming community must be lead down that path one step at a time.

I believe that the first stepping stone is the box editor.

Both MPS and Intentional Programming provide a visual tree of boxes editor and store the abstract syntax tree in a sort of source code database. Both of these concepts are critical to power of language-oriented programming, but changing the storage of source code is simply too radical of a change. Well, actually, source code is routinely stored in databases for IDE features, but the authoritative storage always remains text files. All of a programmers most trusted tools operate at the string level and it is simply impractical to throw everything out at once.

This is why I propose the creation of a new editor which uses an abstract syntax tree box editing model, but preserves the source as text. This editor would have to play nice with other developers who are using traditional text editors. It should be possible for a single developer to try the editor for a day or two without another collaborator even noticing. There are clear benefits to a box editor even in the absence of language-oriented features. For example, navigation of source code by parent, child, or sibling relationships instead of by word or by line. Advanced renders can be used to layout math expressions or to show a referenced image file in a comment.

Editors are a religion, start a new one. If some box editor becomes popular enough, other tools would spring up around them. To help the process, the editor should have well exposed extension and composition mechanisms. For example, it should be trivially easy to add new renderers for various nodes. It should be equally trivial to utilize the abstract syntax tree system to create stand alone code analysis tools.

If box editors garner enough of a following, then the language-oriented programming advocates can seek the next stepping stone.