You work for me, Computer

Learn to Read the Source, Luke

2012-04-16T14:13:00.002-07:00

Apparently, I'm Jeff Atwood's guest blogger today: Learn to Read the Source, Luke

Source code access is near and dear to my heart, so head on over there and read it!

REST: One Thousand Inconsequential Decisions

2012-02-26T19:23:00.001-08:00

Hardly a week goes by without yet another Hacker News front page post by a blogger making a bold declaration about some important thing you absolutely must do if you want your API to be "RESTful". With Rails' recent PATCH announcement, REST is all over the front page again.

In my experience designing and consuming APIs of varying levels of RESTfulness, I've concluded that most of the decisions involved in designing a RESTful API are completely inconsequential. PUT vs POST vs PATCH? HATEOAS vs manual URL construction? JSON vs custom Mimetype? It's all the same to an API consuming developer. It might matter to some kind of generic crawler, but I'm not building an API for crawlers, so I don't care.

What follows is a series of rants about various parts of the grand REST debate. This is by no means comprehensive. My goal with this post isn't to discourage you from writing a REST API. I simply want you to spend your time on the API decisions that matter, not the cosmetic, inconsequential ones.

HTTP Verbs & Status Codes

The Rails PATCH announcement states "both PUT and PATCH are routed to update". So what's the difference? Apparently PATCH is for partial updates, where as PUT is for complete replacement of resources. Here's the funny bit: Both of these are implemented via POST with a _method parameter for support of older browsers. The only two HTTP Verbs that actually matter are GET, for idempotent, cachable requests; and POST, for side-effects. Even DELETE is insufficiently defined for any real application. What if you want to support both Archive and Delete like Gmail?

Which response codes should you use? Those which have specific meaning to HTTP clients. Specifically: those treated differently by browsers. Should you use 201 on create instead of 200? Who cares? Should you use 301 or 302 for redirects? In an API, it doesn't matter. Pick one. Document it. For your actual web page, consult an SEO guide.

What about errors? If you don't have permission to see a resource, is that a 403? Or do you hide the existence of the resource and return a 404 (like GitHub)? It doesn't matter. Return a 4xx for expected caller errors and a 5xx for unexpected errors. Pick one. Document it.

The smart thing to do is to always return your resources within one level of nesting. That is an item key in JSON or an item element in XML. Do this so that you can add an error key or other metadata without breaking changes. For example, you may want a timings key for perf debugging. For lists, use an items key because pretty soon, you're going to want a paging key. Also, for errors, make sure you include a non-localized (and hence, Google-able) error string; a status code is insufficient.

Hating On HATEOAS

HATEOAS, an absurd mouthful of an acronym, is a complete waste of time outside of read-only semi-structured datasets. If you're FreeBase, then go to town with HATEOAS conventions that are useful to you. However, for your typical webapp API, you simply cannot build a useful API client application without deep knowledge of the problem domain and the available API calls. You're going to have API documentation and you're going to have to read it.

By all means, feel free to use HTTP 201 and a Location header to return the location of a newly created resource. Or you could just return the resource as JSON with an href or id key. That's yet another inconsequential decision. Might as well go with the path of least resistance: return interesting data in the body of the response (ie. in the JSON) where it can be accessed without additional effort to parse the headers. As long as your response code is in the 200 range, no one is going to look at it further.

I've seen people advocate Link headers such as rel="newcomment". It's weird to me because you still need to know which rel to lookup. The same thing applies to returning a bunch of extra URLs as metadata in your JSON. You still need to know which key to look up for the post-new-comment URL. You might as well just appended "/comments" and avoid the indirection. Link and rel are useful to generic tools that want to treat a wide range of resources the same. There aren't enough common aspects between the domain models of a typical API to justify the spare brain cycles. Again, if you're Wikipedia, your use case is different. You have generic data that you want to traverse generically. And again, that's a mostly read-only use case.

Versioning

Some REST fans advocate the use of custom MIME types for API versioning. These folks believe that version numbers in URLs are ugly. The usual objection is that resources should be equitable by URL and that /v1/users/5 and /v2/users/5 are different resources. The problem here is that this resource is actually identified by just the number. You don't need the whole URL and practical constraints are going to make it impossible to equate resources by URL. Consider the constraint of browsers' Same Origin Access Policy. You need api.example.com to run your public API on it's own IP address. And you need example.com/api/ to make calls from your web frontend without jumping through crazy IFrame interop hacks.

There are two types of API changes: 1) Backward Compatible, and 2) Breaking Changes. In the case of backwards compatible changes, you don't have to, nor should you, change version numbers. Simply add the additional method or additional return data. Nothing to worry about. In the case of breaking changes, you want to break clients as loudly as possible. If you simply change the MIME type, most consumers of the API are totally going to ignore it. If you deploy your new API at api2.example.com, then there is no risk of breaking older clients and they have to make a conscious decision to upgrade.

In general, you're probably going to want to protect yourself with API keys. API keys provide the best way to version your API: record the desired API version with the generated key. Just stick an api_version column on your api_keys table. Expose that column as a drop down box in your API key management UI. This solution let's you leave the "ugly" version number out of the URL, but still enforce breakage on breaking changes. The best part is that you can totally ignore API keys and versioning when you start: Simply default unregistered API consumers to use v1 when you do implement v2.

Last word on custom MIME types: One of the great promises of REST is the ability to treat generic things, generically. If your MIME type isn't a standard JSON MIME type, then how can you expect your browser to render it nicely with syntax highlighting and code folding? Your data is structured as JSON or XML; if you want to version the schema, use the facilities provided by XSDs or JSON Schemas, etc. But you're better off with documentation written by humans, for humans, than you are with formal schema.

REST: The Good Parts

There are a lot of good reasons why REST has gained popularity. For one thing, it isn't SOAP.

The good ideas in REST are primarily predictable URLs, contractual idempotence, the preference of nouns over verbs, and CURL as a client library. In practice, you can get contractual idempotence via GET vs POST. Predictable URLs are no different than predictable function names in all typical library code.

All this other stuff, the finer points of URIs and HTTP headers, are completely inconsequential to a developer programming against a well documented API.

Plugging a hole in Microsoft's hiring pipe: IE Frame

2012-01-14T18:22:00.000-08:00

You're probably familiar with Google Chrome Frame, Google's clever workaround to combat Internet Explorer dependency in enterprises. You're probably also aware that Microsoft is not a fan.

Microsoft desperately needs to implement an equivalent "Internet Explorer 10 Frame" mass deployed as a high-priority Windows Update. Failure to do so all but guarantees that Microsoft will become completely incapable of hiring a sufficient supply of talented graduates.

The key issue is that two of the major vectors for aspiring engineers are no longer dominated by Microsoft technology.

Javascript is the New Basic

Microsoft QBasic was my first experience with programming, but I got started a few years earlier than most of my peers. The students I graduated college with largely had their first foray into text editing via HTML. Browsers are installed on every computer on the market and web development skills are vital for nearly all professional developers (even those not doing full-time web development). This puts Microsoft in a really bad spot: every budding developer's first impression of Microsoft is "IE SUCKS!!!!11!!one!".

Microsoft's Stranglehold on Gaming is Loosening

When I was employed by Microsoft, I took an informal survey of my recently hired peers. I worked in the gaming division, so my view was skewed: they almost all got into programming via PC gaming. If you loved games, you ran Windows. If you wanted to make games, you learned Microsoft Visual C++. However, times are changing. Apple's iPhone is the biggest gaming platform on the planet. Your family computer is probably a Mac by now and your Xbox doesn't come with a compiler. Why not learn Objective-C? Furthermore, canvas games are popping up everywhere and it won't be long before WebGL is common place. Soon enough, a significant subset of aspiring game developers only have to boot Windows to test something in Internet Explorer. sigh.

One year after accidentally falling in love with Rails

2011-09-24T17:40:00.000-07:00

About a year ago, I wrote a post titled How two Pythonistas accidentally fell in love with Rails. Some people predicted that the honeymoon would be over quickly and that I would soon hate it.

So, do I still love Rails?

Yes. Absolutely.

The prototype and first beta version of my startup was built entirely on Rails. The framework enabled us to build our site significantly faster than we would have been able to without it. For that, I still love Rails. It was critical in helping us get to where we are today.

It's the perfect hammer for the v1, nebulous direction, rapid development, nail.

That said, we're not using Rails for any new components.

After getting a lot of feedback from our first beta, we realized we had the wrong approach to the UI. Now, in our second attempt, we've got a much richer client-side front-end, written in CoffeeScript and bootstrapped by Node.js. Ruby/Rails is simply not the right tool for that job. In the new world of the browser app, even the folks of 37signals have been accused of propping up their successor.

Luckily, we had a pretty good handle on the models in the problem domain. So I ran `git rm -r app/views` and effectively turned the Rails app into a Json API backend. The models have evolved, but not by much. In fact, there were a few months where the Ruby code remained practically unchanged while we developed the frontend.

As time goes on, we've been breaking our API layer apart into several services. It makes development, deployment, and versioning easier. Service Oriented Architecture is working great for us... now that we know what services we need. These services are each too simple to justify the overhead of Rails. Some are Node.js, some are Sinatra, some are cron jobs and shell scripts. We're polyglots, so we're careful to choose the right tools for each job.

We'd have gotten it very wrong if we tried to build services from the start. The monolithic Rails app was the correct evolutionary intermediate form.

Given a time machine, would I have done anything differently?

Only one thing: I wouldn't have used ActiveRecord.

I guess I should have seen that one coming. ORMs are an insidious, broken abstraction. At first, they save you time and make you feel productive. They work out splendidly for a while, but like a virus, they infect every part of your application. One day, you hit that abstraction brick wall and it's too late to do anything about it without significant effort.

Are there things I hate about Rails?

Ignoring ActiveRecord, only one thing really stands out: Performance.

I'm not even talking about page load times. I'm talking about development time.

Even with our modestly sized application, the impact on productivity is atrocious. Our test suite runs far too slowly to be waited on for every checkin. Rake's startup time is too great to for its usage to be considered "interactive". Simple shell scripts have replaced practically all of our custom tasks; they are instantaneous.

Isn't this basically the same story I told about Django?

Yeah, I guess it is.

The story basically goes:

Use a framework
Be productive
Encounter shortcomings
Stretch it to work
Watch it break
Replace components
Loop steps three to six
Be productive
"Where's the framework?"

I still stand by my original criticisms of Django (although, I'm sure much as changed in the last year). I'm also re-affirmed in my belief that big frameworks are generally bad. It just so happens that Rails is good enough and matches my mindset well enough to justify the framework pains.

Stupid brain trick: Tests and vertical splits

2011-08-16T23:57:00.000-07:00

Recently discovered a stupid brain trick that I thought was worth sharing.

When writing code with unit tests, I find it helpful to view the tests and the implementations side by side. I use vertical splits in Vim to do this.

The stupid brain trick is this: Put your tests in the left-hand split!

Why? I traditionally used the left-hand split for the "main" thing I'm working on and the right-hand split for reference. I've watched many other developers do the same thing. The right split is constantly sub-split horizontally and always changing to various different files.

In trying to be better about writing tests first, I created a test code file before the implementation file, so it just happened to be in my left-hand split. After an hour or so of development, I realized that the "main" file placement of the tests meant that I focused more on them. I viewed the actual implementation file as just some ancillary file I had to tweak to get my tests to pass. It changed the focus of what I was doing from "Write a method that does X" to "Describe X". Describing something involves breaking it up in to smaller and smaller pieces until the implementations are so trivial that they basically write themselves.

Thinkfuse is hiring!

2011-03-28T15:08:00.000-07:00

If you caught the recent TechCrunch coverage, then you know that things are heating at my startup, Thinkfuse :-)

We've got lots to do, so we're looking for a kick-ass engineer who can get up and running quickly. My co-founder, Steve, and I are full-stack developers who really give a shit, so we're looking for a Seattle-based developer who can keep up, or better yet, force us to struggle to keep up. Rails experience is a plus. Good sense of humor and beer drinking skills, a bigger plus. We're offering a solid salary and meaningful equity.

If you're interested, reach out to me at brandon@thinkfuse.com

Where Rails Rocks (and where Python doesn't)

2010-09-17T11:31:00.000-07:00

Here’s a list of things I like about Ruby on Rails. Almost every single one of these are available in at least one Python framework in some form, but the best thing about Rails is how polished the pieces are and how well they all fit together. The whole is truly greater than the sum of its parts.

Fantastic OOBE
Rails has a fantastic out-of-box experience. Simply execute rails new and you’re off to the races with a directory full of goodies that every real web app needs: dependency management, build system, multi-environment configuration, development storage, logging, custom error pages, robots.txt file and more. See section 3.2 of the getting started guide for a complete list. Hell, they even fill out a .gitignore file for you! A place for everything and everything in its place.

Dependency Management: RubyGems
In the Python world, easy_install and its ilk are all kinds of broken. You can’t even uninstall a package! Luckily, there is Pip and Virtualenv, but assuming Pip supports all your packages, you’ve still got to make sense of dependency freezing and manually setup a virtual environment. With Rails, that all just works: rails server runs in a virtualized environment and loads gems as defined by your Gemfile; rails console for a repl. No need to muck with sys.path! Any seasoned developed has spent many an hour in dependency hell, but it is still hard to justify the infrastructure investment before writing a line of HTML. Bonus points for the “vendor” directory, in case you need to patch a third party package (or Rails itself) and want to keep it under version control.

Build System: Rake
I’ll admit it: I like Makefiles. A lot of people really hate them, but they are just so damn useful. Frequently, you need to transform some set of files into some other set of files, but only when they change. Makefiles help you do that reliably. They accomplish their feat explicitly by not being a programming language, but a dependency and build rule declaration language. I’ve seen a lot of attempts at build systems that aim to be idiomatic with respect to their host languages. Java has Ant and Maven. Python has Scons and others. Scala has Simple Build Tool. None of them are as good as plain old Makefiles. Then, I discovered Rake. Rakefiles rock. They are 90% Makefiles, and 90% Ruby. Rake expertly uses the features of Ruby to craft an API/DSL that looks a lot like Make declarations without compromising on general purpose programmability.

Markup and Style Sheets: Haml and Sass
Haml and Sass are so amazing, they ought to be illegal. Just click those links and check out the HTML and CSS comparisons. It is no contest. I never want to write HTML or CSS ever again. As far as I’m concerned, HTML and CSS are object code: to be generated by a compiler. Whenever I have to look at the generated code, it feels like I am reading assembly. Simply add gem ‘haml’ to your Gemfile, then start writing .html.haml files instead of .html.erb and .sass files instead of .css. Use html2haml and sass-convert to upgrade your crusty old markup and style sheets instantly.

Object Relational Mapper: ActiveRecord
I’ve discussed my preference for database schema reflection at great length, so I won’t reiterate here. I will, however, say that ActiveRecord makes simple things simple and complex things possible while embodying successful best practices in its various subsystems, like migrations. It doesn’t try to be everything to everyone, unlike SqlAlchemy. And it seems to be nicely onion layered for pealing back when necessary. For example, ActiveModel validations can be used on plain old Ruby objects. Which brings me to...

Form Handling: FormBuilder, params and mass assignment
I don’t need to learn and manage three different data marshaling techniques to map a database record to a web form. In fact, I don’t need to manage any. Schema reflection spares me in the model layer, FormBuilder/FormHelpers in the views, and the hash & array structured params in the controllers layer. Almost every user interaction boils down to interpreting a form’s worth of JSON-like data structures as an API call, implemented via a (secured) mass assignment to virtual attributes.

Templating: Layouts, Partials, and Helpers
When I’m getting my Haml on, I frequently want to run some non-trivial code. Feel free to shout about your separation of concerns religion, but I’ve got real things to worry about. Furthermore, we as engineers should be writing our production views/templates, not some hypothetical designer who is technical enough to write loops and branching statements, but not to be trusted with a function definition. Views are composed of markup and presentation declarations in the templates, presentation logic goes in helpers (which conveniently share scope with templates and controllers), and are neatly organized into partials. Simple, but effective.

URL Design: Routes
Rail’s routing system makes clever use of blocks to provide a hierarchical, declarative URL map that just makes sense to read. I used to design URLs in an indented text file and then map it to a list of regexps, but the routes.rb file is close enough to a spec and doesn’t require me to even think about regexes, ‘nuff said.

Authentication: Authlogic
Authentication is hard. Really, really hard. And boring. Really, really boring. But important. Really, really important. Everyone knows that the sign-up funnel is critical to get right and security is easy to get wrong. Most quality sites have a little bit of domain-specific special sauce in their authentication model. Trivial sign-up and log-in forms just don’t cut it. One size simply does not fit all for authentication front-ends, but the authentication back-ends tend to be pretty repeatable. Authlogic turns repeatable into re-usable, without the baggage of views you’re going to need to rewrite anyway. Thinking about authentication as CRUD operations on a user-session model blew my mind slightly. I had a full authentication system tuned to my needs up and running within two hours.

Authorization: CanCan
I hate ACLs. Role-based authorization simply has too much impedance mismatch for most problem domains. CanCan is a clever little library which makes zero assumptions about how permissions are represented. Simply declare a list of everything a user can do and how to tell if they may. Query with the can? function, or assert with the authorize! method. Beautiful.

Rope to Hang Yourself: Ruby Itself
One keyword in my previous post triggered much discussion: magic. The concept of magic is a topic for a whole ‘nother blog post (as I’ve already been hung a few times), but as several commenters pointed out: it’s all just Ruby. Between mixins, include, require, blocks, method_missing, symbols, and many other features, Ruby is a very capable internal domain specific language development toolkit. Most of the time, you don’t want a DSL. If you’re not an expert in both the host language and the DSL, there is almost certain confusion and maintenance danger. Most projects are either small or distinct enough to justify writing more verbose code that can more readily be understood by successors or even your future self. The gains of a DSL do not always outweigh the costs. Python excels at not supporting DSLs; it is basically executable pseudocode. However, when it comes to a task as common as web application development, it’s worth the time to develop or learn a DSL or two. Rails is loaded with DSL features and “magic” behavior that increases the learning curve, but also increases peak productivity.

Community
These sites have been invaluable: The Ruby Toolbox, RailsPlugins, Railscasts. The quantity and quality of Ruby libraries on Github continues to impress. The educational infrastructure is extensive. Somehow, the Ruby community seems to consistently be at the forefront of the biggest trends in software development. Much like Rails compared to other frameworks, the community distinctions are subtle, but meaningful. For example, Rubyists championed Git while Python adopted Hg. To the untrained eye, they are virtually identical tools, but extensive experience with both has convinced me that Git is significantly superior. I’m new to the community, but it seems like Rubyists tend to insist on quality and bet on the winning horses.

Other Random Things

Javascript and CSS minification are trivially easy with plug-ins
Static files are served with version stamps for better caching behavior
Console and file logging are usefully configured by default
git push heroku
rails.vim
ActiveMerchant looks awesome
Only been working with Rails for two weeks; lots more goodness to uncover

Just who the hell do I think I am?
I’m nobody. Just some opinionated computer geek working on a start-up. Not unlike many of you! My co-founder and I are currently participating in TechStars Seattle and hope to launch something killer come demo day, November 11th, 2010. We’ve got a grand vision for next generation enterprise collaboration software, but we’re starting small by focusing on making weekly status reports less painful and more useful. More details soon, but please visit ~~http://www.thisweeklastweek.com/ to register for a beta invite~~http://www.thinkfuse.com. Thanks!

How two Pythonistas accidentally fell in love with Rails

2010-09-13T02:33:00.000-07:00

One week ago, my co-founder and I found ourselves staring down a thick stack of paper UI prototypes; almost 50 pages total. One week before that, we threw out a perfectly good, working beta product because customer feedback suggested a significantly better path. The tale of how we ended up at that point is entertaining as well, but it will have to wait for another time. This is the tale of how our paper prototype turned a pair of long time Python developers (and documented framework haters) into Ruby on Rails developers.

Paper prototyping proved to be an extremely worthwhile exercise, but we really wanted something we could click around. We devised a strategy to build a clickable prototype that would evolve into production code. The plan was to stub out all of the views with static HTML and CSS. The CSS could be directly carried over to the product and the HTML could be retrofitted as templates. Having become religious about Sass about a year ago, I fired up the file watcher (which automatically rebuilds CSS) and set to work filling a directory with sass and html files.

After some time, I asked myself “I wonder if Haml has a --watch argument?” (it doesn’t) because Sass makes me hate Html’s verbosity. Armed with practically zero Ruby knowledge, I hobbled together a simple Rakefile and Haml watcher script and got back to work. A few Git pushes later, my co-founder, being the clever bastard that he is, simply installed Rails and the Haml plugin to replace my hacky scripts. “Neat,” I said and thought nothing of it.

Then we designed our URLs in an indented plain text file, which pretty much directly mapped into the routes.rb file and some trivial (empty method) controllers. Whoops! Now we’re writing Ruby.

“At this point, we might as well try Rails proper.”

Generated some models and corresponding migrations: “Wow, explicit schema with reflective model objects. None of this declarative schema bullshit.”

Wired up some controllers. “The structured params infrastructure is pretty cool.”

“Ruby is kinda fun. Blocks are super useful.”

Integrated Authlogic. “Wow, that was easy. I really like how it does not force any front-end decisions on me.”

“Rails is full of evil black magic that I do not understand, but it seems to read my mind, so I don’t really care.”

Started to apply some polish and work on the 10% that takes 90% of the time. “I think I can safely add Ruby to my resume now.”

Late at night, at the very end of the week, exhausted, we stared at each other. “Dude, I think we can show this off to people tomorrow.” Walking out the door of the office, my co-founder said to me “I feel so dumb for having ignored Rails for so long. I just assumed it was Ruby’s Django”.

We accidentally implemented our entire beta product in one week, with zero prior Ruby or Rails experience.

When I have some more time (and experience), I’ll try to write about specific design decisions that set Rails apart.

NoScript Add-on Install Base

2010-06-04T19:37:00.001-07:00

The official NoScript site downloads via the Mozilla Add-ons page, so its download count is probably accurate: 67,616,402

NoScript does not publish their individual add-on usage statistics, but the global download/usage ratio can be calculated from the statistics on the Firefox Add-on home page:

1,962,617,946 add-ons downloaded

157,090,095 add-ons in use

About 8% of downloaded add-ons are still in use. Assuming NoScript's usage ratio is comparable to the average, approximately 5.4 million installations of FireFox are running NoScript. Let's ignore the fact that NoScript's usage ratio is probably much lower than the average, due to the fact that it breaks most web pages.

I'd wager that the average NoScript user has at least two machines, so the total number of NoScript users is probably less than 2.7 million.

There are over 230 million internet users in the USA.

Even if 100% of NoScript users were Americans, they form 1% or less of the general population. If you, like me, believe that I have been generous to NoScript here, it is likely that no script users are no more numerous than 1 in 1,000.

Even if the user is using NoScript, they can whitelist your site. You can probably add <noscript>WARNING: THIS SITE IS BUSTED WITHOUT JS</noscript> to the top of your page and call it a day. If you are feeling generous, redirect no-script users to the mobile version of your site and tell them why.

tldr: Assume that human user agents have Javascript.

Leaving Microsoft

2010-04-14T14:55:00.001-07:00

April 30th will be my last day on the Microsoft XNA team. I'm leaving to start work on my own startup full-time. I also hope to invest some more effort into this blog. Exciting times!

ORMs and Declarative Schemas

2009-10-06T10:32:00.001-07:00

My prior post was more controversial than I anticipated. In hindsight, I should have realized what a hot button issue web frameworks are. One assertion went all but unnoticed. I expect it may be even more controversial: the schema-generative ORM paradigm is fundamentally flawed.

Disclaimer

I came to this conclusion while working with Django's ORM, but this post is completely object-relational mapper agnostic. We are now using SqlAlchemy, but we are not using any of the many available declarative layers. Instead, we are using schema reflection and semi-automatically configured mappers. This is not an argument against ORMs. It is an argument against generating database schemas from ORM declarations. By extension, this is an argument against Django's ORM because Django uses an exclusively schema declarative model. That said, Django's ORM is far from alone in this camp.

Data Outlives Code

When code is dead and gone -- be it through rewrite, obsolescence, or by other means -- the data will still be there. Longevity implies slower evolution; data is always more difficult and riskier to change. Data is also more valuable. What if Facebook rebooted their database? They've already rebooted their software several times.

Schemas are data. As data, schemas are longer lived, less flexible, and more valuable than code. These factors alone suggest that the database itself should hold the authoritative schema, not a class declaration in the code.

If you have inherited data from another project, you already know this lesson. You can't generate the schema from code because the schema already exists. You can mimic the authoritative schema in your declarations, but it is easier and more accurate to use reflection.

Ineffective Domain Objects

Object relational mapping is primarily a serialization problem. Every serialization solution has its quirks. The scale and number of quirks seem directly proportional to the absolute difference between the runtime and storage representations. Since a database is a completely distinct type system, rather than an opaque byte array, serialization to a database can have particularly quirky quirks.

Modeling is one of the fundamental challenges of software development. Capable developers prefer highly expressive or unconstrained type systems to aid with modeling. Generally, runtime type systems are more expressive than those found in databases. Declarative relational mappers, however, constrain the programming language type system to its less expressive counterpart. When building domain objects, the developer must think in terms of the database's type system, not the programming language's.

While using a declarative object relational mapper, developers are effectively trying to design storage and runtime models simultaneously. On average, superior results are achieved by modeling these two concerns separately and then solving an additional subproblem: serialization mapping. You might wind up with slightly more code, but it will be easier to understand and maintain.

SQL Is Not Going Away

Despite forcing a less expressive type system on to the developer, declarative ORM layers attempt to treat the database as an implementation detail. However much we wish this were true, the database is not a detail which can be ignored. Sooner or later, you are going to have to open your database shell and write a SQL expression. This requires knowledge of your database's particular SQL dialect and idiosyncrasies. Exacerbating this issue, generated schemas are typically full of name mangling and other ugliness. It is far more pleasant to work with a carefully designed schema than one that compromises for the ORM or the runtime type system.

Schema and Declarations Diverge

Data migrations present the most pressing need to work with SQL directly. As an widely unsolved problem, automation can not be trusted. Unless you are painstakingly simulating the schema generator, the production database schema will slowly diverge. Most deviations are tolerable as they will not affect the runtime behavior, but it is wise to minimze differences between production, staging, and development environments. To faciliate this, store migration scripts and backed-up schemas in version control.

Some deviations will directly affect runtime behavior. For example, consider the case where two versions of an application are running in production. An is_read boolean column was added to a message table in the database. It's default value is false. When a row's page is viewed, the is_read column is set to true. The old version of the application doesn't know about the column, so it can not set the flag. When the new version is rolled out to everyone, affected user will see a bunch of read items marked as unread! The solution is to set the column's default to true, but initialize it to false in the application. Declarations must either deliberately deviate from the schema or present a misleading default value to be overridden during initialization. This is just a simple example, real world schema migrations can be significantly more complex and suffer from numerous more subtle problems.

URL routing and views

2009-08-24T22:44:00.000-07:00

As promised, I'd like to elaborate on the URL routing system I came up with.

Weighing at less than 200 lines of code (including example), I'll let it speak for itself: download it.

This approach seems to be working great for us. Love it? Hate it? Feel free to let me know what you think.

Dropping Django

2009-08-19T00:07:00.001-07:00

My partner and I built a non-trivial web site on Django. When the next version ships, there might not be a single Django module imported.

We're not trying to drop Django; it is just sort of happening. Piece by piece, it is failing to meet our needs. Despite the marketing copy on the Django site, most components of the framework are tightly coupled enough to make customization frustrating. It is often easier to rewrite core framework components than to implement them on top of the existing extensibility points.

What follows is a loose chronology of our migration away from Django.

URL Routing

A flat list of patterns violates the DRY principal when creating nested URLs. Trees are a superior representation. Having a tree of views also enabled us to optionally associate a "binder" function with each node. These bind functions are executed for each URL component from left to right, filling the template context as they go. Breadcrumbs are automatically generated as each node binds, but only the last node executes its full view logic.

Authorization

Our site enforces permissions on every resource, but Django's database ACLs would have been prohibitively numerous. Instead, views or their URL binders may raise an AccessDenied exception. Upon catching such an exception, a middleware layer serves a login form. This ensures users have permission to access the current resource, as well as all ancestor resources bound to the URL.

Authentication

Both of Django authentication's key extensibility points are flawed. These two extensibility points are "user profiles" (storing additional per-user data) and custom credentials (such as for logging in via email address instead of username). Django's documentation and numerous internet sources cover both topics, but all of the guidance lacks important caveats. The admin UI, in particular, is very easy to break with either extensibility mechanism.

Extending the User model with the ORM requires a one-to-one database relationship. This relationship can be implemented with a "user profile" setting, an explicit foreign key, or model inheritance. Each approach has its own strengths and weaknesses in terms of performance, API semantics, subtle behavioral changes, and outright bugs.

Enabling custom credentials requires implementing a trivial authorization "backend" object. Unfortunately, it is non-trivial to replace usernames with email addresses. The admin UI's login form refuses to accept email addresses without hacking the template. Even if you hacked the template, the User model would still enforce a non-null constraint on the username field and the generated database schema enforces a uniqueness constraint as well. It turns out to be easier to fill the username field with a dummy value and "support" both forms of authentication with your backend, but you won't come to that conclusion until your head has already bore a hole in your desk.

Templating

We do our best to keep view and template logic separate. Django's templates are targeted at designers, who aren't implementing any real logic anyway. However, we're a pair of hackers. Sometimes it is just more convenient to put a little bit of logic in the views. Besides, templates are code; code needs to be reviewed and tested. We wouldn't ever hire a designer who couldn't pass a code review for some trivial template logic.

We needed a pragmatic template language to replace Django's idealistic one. Any template language with greater expressive power would have been welcome, but Jinja2 fit the requirements and provided the easiest migration path. Ultimately, we'd prefer to use something like HAML, but there doesn't seem to be a Python equivalent besides the inactive GHRML project. We are, however, using SASS. I will never write CSS by hand again.

ORM and Admin UI

One of Django's most touted features is the Admin UI. For simple "active record" style database models, the Admin UI is a huge time saver. Sadly, it struggles a little bit with nullable fields and is tricky to customize. You'll definitely need to write custom UI for complex models, but by and large the admin solves the problem at hand: viewing, creating, updating, and deleting database rows.

After using the Admin for a little, I found myself missing Microsoft Access. I never thought I'd say that, but it is true. ~~Django's admin does not support sorting, filtering, or other impromptu queries.~~ Edit: It turn's out I was mistaken about sorting and filtering, but I stand by the core message of this section. I found myself writing impromptu queries in the database and Python shells. After a while, I just gave up and installed a desktop client. I haven't visited the Admin UI since.

Django's ORM has shortcomings with respect to querying, especially for joins and aggregation. It has been improving over time, but it will likely never reach the capability of projects solely focused on databases, such as SqlAlchemy. With the admin having fallen into disuse, the Django ORM lost all advantage. Beyond Django's specific weaknesses, I've come to believe that the schema-generative ORMs paradigm is fundamentally flawed. That is a topic that deserves an entire (Django-agnostic) post of it's own. We are now using SqlAlchemy via schema reflection; no declarative layer.

Form Validation and Generation

Here is where our chronology meets present day. We are still using Django form validation, but never used form generation beyond scaffolding. Nearly all of our templates customize labels and display of errors. Additionally, embedding widget information in the Python code is cumbersome during template development. Django forms is a quality validation library, but there are some inconsequential style things that I like better about FormEncode. Preferences aside, the difference isn't large enough to justify switching.

While I like FormEncode, I'm still not sold on its anti-form-generation companion, htmlfill. I think there is a middle ground with form generation that provides scaffolding during development, smoothly transitions to production use, and cooperates with validation. As we implement more complex client views, I'll be on the lookout for ways to improve our form development toolbox.

So, ugh... What's left?

Besides a few isolated helper functions, not much is left of Django.

The last big ticket item is the HTTP framework and WSGI server. We could continue using Django as if it were CherryPy or Paste, but Django has this nasty habbit of insisting on running your code for you. The settings and manage.py infrastructure are fiddly for deployment and don't really add any value over simple scripts using our application like a library. Might as well use a simpler WSGI library, and replace those over-engineered management/commands/foo.py files with vanilla scripts/foo.py files.

Moral of the Story

I'm sure there are numerous lessons to be generalized from this journey. Personally, I've developed a moderate fear of the word "framework", as well as altered the way I think about software abstractions. I think the most important lesson, however, is one I already knew: choose the right tool for the job. Unfortunately, we had no idea what the right tool was when we started. I'm not sure we know any better now.

AppWeek

2009-06-12T20:02:00.001-07:00

Shawn's AppWeek post inspired me to write one too. AppWeek is our chance to be creators for a little while and it was a lot of fun. I didn't set out to build something nearly as ambitious as "Super Avatar Sample Smashup EXTREME! - 'Capture the Cat' edition", but I did get to take a swing at a game I've wanted to build for a while: ~~Rock'em Sock'em Avatars~~ Avatar Boxing. Avatars, being a new feature in this release, were an unwritten requirement for all of the AppWeek games. Between SASSECTCE, my game, and the many others, Avatars were chasing cats, beating each other up, play futuristic sports, falling off buildings, dancing in a cloud of gems, being launched from canons to save the world, and much more. All this excitement was almost too much for a bunch of exhausted engineers, but that's what the beer was for during the game unveilings.

Here's what the game looked like with the basic animations wired up. You'll notice that the avatars have been hitting the gym. That's because their arms were too short to reach each other! I added a little extra bulk because I was laughing too hard not to. I directly bound the game pad triggers to the shoulder and elbow joints and rigged up the chase camera sample to inspect my work. There wasn't much game play yet, but it was already fun. That's always a good sign.

Even with just one week, I decided to invest some time into debugging visualizations. That turned out to be a really great idea.

Then, I added some collision spheres for the heads, hands, and upper bodies. This was a hacky, trial and error process. Thankfully, C# compiles quickly.

At this point, I spent an entire day working on the physics. I wanted the avatars to bounce/wobble when they got hit, so I rigged up some complex spring systems. Things were starting to work, but I'm generally pretty bad at this sort of thing and my simulation routinely exploded. The avatars arms went shooting off into space and I was getting pretty frustrated. No screen shots of that chaos because I am embarrassed.

With half a day to go, I added the obligatory damage bars and some rudimentary hand-to-head collision detection.

I was feeling pretty good about the game, despite my physics failures, it was pretty fun anyway. I wondered down the hall to chat with Jace, who had just added sound effects to his game. His game was hilarious before, but the sound effects were priceless. I ejected the sound effect CD out of his machine, yoinked it, and took off running. An hour later (and 10 minutes after the deadline), my game had some sweet punch and miss sounds. I also made the avatars' heads pop up when their damage bar was full, accompanied by an awesome zip-tie sound.

At our team happy hour, I'd like to think Avatar Boxing was a fan favorite. I certainly had fun making it! I hope everyone enjoys Avatar support in the new XNA Game Studio.

PowerShell: condemned to reinvent

2009-04-20T02:33:00.001-07:00

I tried PowerShell when it was first released, but never used it for real work. I recently attended a "brown bag" presentation about PowerShell. This presentation spurred me to augment our team's environment with PowerShell and I have been using it every day since.

In the past weeks using and abusing PowerShell, I have drawn two conclusions:

PowerShell has a killer set of standard tools with brilliantly designed usability.
The PowerShell team doesn't understand UNIX and therefore were condemned to reinvented it, poorly -- with apologies to Henry Spencer.

First things first: if you spend any time working with Windows, get PowerShell. Now. Stop reading my blog and go download it immediately. It mops the floor with cmd.

The key premise behind PowerShell is that it operates on live .NET objects. This is beneficial because it eliminates a lot of the text cutting and manipulation common in shell scripts. Additionally, it puts the full .NET Base Class Library into your scripting toolbox. PowerShell tools, known as commandlets, typically only render the most common fields for their objects, but the less common fields are easily available in memory. By convention, Commandlets are named with a verb-noun pattern and support a common command line parsing behavior. The repository of commandlets and the command line options of each are easily queried and highly consistent. All this meta-data makes PowerShell a breeze to learn.

I fell in love with the the discoverablity and ease of use when I tried to kill a collection of runaway processes:

PS> get-command -noun process

CommandType   Name           Definition
-----------   ----           ----------
Cmdlet        Get-Process    Get-Process [[-Name] <String[]>] [-Verbo...
Cmdlet        Stop-Process   Stop-Process [-Id] <Int32[]> [-PassThru]...

PS> get-process notepad | stop-process
PS> get-alias | where { $_.definition.contains("Process") }

CommandType   Name   Definition
-----------   ----   ----------
Alias         kill   Stop-Process
Alias         ps     Get-Process

PS> ps someotherapp | kill

OK, that's pretty cool and oh-so-very Unixy -- right? Wrong. Notice the "CommandType" column in the results of get-command. There are many other types of commands besides commandlets: functions, filters, scripts, applications, etc. Each of these has slightly different semantics for pipes and parameters. Applications, for example, have no way of accepting .NET object pipes. You must develop a separate commandlet. Yikes!

Compare to Unix: all commands are applications which accept a command line and pipe byte streams in and out. Much simpler, but byte streams aren't as friendly, discoverable, and maintainable as object streams. However, the brilliantly simple thing about Unix is that, when you get right down to it, object streams are just byte streams! There is absolutely nothing stopping you from implementing get-process and stop-process as Unix programs which pipe object references, JSON, pickled Python objects, XML, S-expressions, or any other data format you fancy. Doug Mcllroy, the inventor of Unix pipes, was right: text streams are the universal interface.

Actually, this is no different on Windows. All of the PowerShell commandlets could have been implemented as applications which import a library. This library would replace main in much the same way as winmain, provide a metadata enriched implementation of getopt, man, etc. There is no need to invent a new shell in order to acquire the power of piping objects. Sure, cmd is old and needed to be retired for many other reasons, but it is a real shame that the PowerShell toolset is not available to those of us stuck in batch scripts.

Personally, I would really like to see such a library developed. Microsoft has certainly proved one thing with PowerShell: steep learning curves are not intrinsic to command line interfaces. Unfortunately, commandlets are two steps forward and one step backwards. I have no doubt that we can retake that forward step.

Language-oriented programming: too much, too fast

2009-02-09T02:49:00.001-08:00

There was some recent discussion on Hacker News about the 2004 article on "Language Oriented Programming" by Sergey Dmitriev. With my growing interest in programming languages, this article, and the pending release of JetBrain's Meta Programming System, I have been thinking a lot about the future of programming.

As Sergey points out, MPS is not the first entry into this paradigm of software development. Intentional Programming was being demonstrated by Microsoft Research as early as 2000. Wikipedia lists several implementations of the language-oriented programming concept. (OK, that is enough links for now!) Sadly, none of these systems have been met with wide spread success. Despite my unlimited respect for the JetBrains team and love of their products -- especially Resharper -- I expect MPS to fail to achieve critical mass. I don't think many programmers would disagree. It's just too much, too fast.

Software development needs to evolve, not start anew. If you change too much at once, the common developer simply won't accept it. Language designers are still re-inventing Lisp piece by piece in an effort to make sense of all the different concepts for Mort. In order to improve adoption of language-oriented programming concepts, the critical path must be identified and the programming community must be lead down that path one step at a time.

I believe that the first stepping stone is the box editor.

Both MPS and Intentional Programming provide a visual tree of boxes editor and store the abstract syntax tree in a sort of source code database. Both of these concepts are critical to power of language-oriented programming, but changing the storage of source code is simply too radical of a change. Well, actually, source code is routinely stored in databases for IDE features, but the authoritative storage always remains text files. All of a programmers most trusted tools operate at the string level and it is simply impractical to throw everything out at once.

This is why I propose the creation of a new editor which uses an abstract syntax tree box editing model, but preserves the source as text. This editor would have to play nice with other developers who are using traditional text editors. It should be possible for a single developer to try the editor for a day or two without another collaborator even noticing. There are clear benefits to a box editor even in the absence of language-oriented features. For example, navigation of source code by parent, child, or sibling relationships instead of by word or by line. Advanced renders can be used to layout math expressions or to show a referenced image file in a comment.

Editors are a religion, start a new one. If some box editor becomes popular enough, other tools would spring up around them. To help the process, the editor should have well exposed extension and composition mechanisms. For example, it should be trivially easy to add new renderers for various nodes. It should be equally trivial to utilize the abstract syntax tree system to create stand alone code analysis tools.

If box editors garner enough of a following, then the language-oriented programming advocates can seek the next stepping stone.

Break the cycle of broken builds

2009-02-03T23:07:00.001-08:00

Broken builds suck. Breaking the build sucks. No one wants to have their work stopped by others' build breaks and no one wants to cause work stoppage on account of their own build breaks.

So then why do builds break so often? You'd think that after so many years of software development, someone would have solved this problem. Microsoft employees tens of thousands of engineers across hundreds, if not thousands, of code projects. Many of those code projects are development tools for the engineering community at large as well as engineers within Microsoft. Yet, still, all of the tools at the disposal of those engineers are highly prone to build breaks.

Below, are three ideas for reducing build breaks. Together, they could all but eliminate the problem.

Use a better version control system

Everyone keeps raving about distributed version control, and for good reason. I won't rehash the argument here, but I will say: branch early and merge often. Perforce (better known as Source Depot within Microsoft) and it's spiritual successor Team Foundation Server are completely unbearable. A great majority of build breaks can be contained if people are only pulling known good changes from dedicated integrators. Everyone checking in all at once to a main branch simply doesn't work with anything but the most tightly knit of teams. Even if the does build break, it should only block the person waiting on that particular change.

Isolate build breaks

`sd submit -c 12345` translates roughly to "I'm 100% confident that my changes are solid and am equally confident that I am submitting the changes that I think I am." You never can be 100% confident, don't take any chances: submit to a private branch which is being monitored by a build service. The server should detect your submission, run a full buddy build (with unit tests!), and then submit it to the shared branch only after it has passed.

If your builds take too long for this, people should be able to grab changes down from your unverified branch on an as-needed basis. If this happens too frequently: your builds take too long. Refactor your system into smaller components and formalize the contractual interface between those components. Then buddy build just the components which have changed. Verify that the interface hasn't changed with a unit test. This will stop compile errors at the boundaries.

Go lazy, be late

This one is a big of a pipe dream... software systems developed with dynamic languages do not experience build breaks nearly as often as those developed with static languages. This is because dynamic languages are typically lazy-loaded and utilize late bounded method invocations. If someone adds a new button to the UI which calls a method in a file they forgot to add to source control, that should only block people who want to click the button!

There is no firm technical reason why statically or JIT compiled systems can't simulate this. Most compile errors could theoretically be treated as a runtime error. Clearly, compile errors are valuable, but should be ignorable whenever possible. Don't ignore them in your known good builds, but ignore them when they stand between you and forward progress.

Since most of us aren't writing our own compilers are full tool chains, here is some more practical advice for projects using statically compiled languages: write code which fails gracefully. Runtime errors can be build breaks just the same as compile errors, but try to defer the negative consequences as much as possible. One broken feature shouldn't break the whole project; unless, of course, you are about to ship. In production builds, perform verification code at startup or in unit tests, but try to be lazy and late in development builds.

Contributions

2009-01-26T03:49:00.001-08:00

The modern, read/write web thrives on processes which accept and coordinate a large set of contributions from a set of contributors that is almost as large. Before this was common on the web, open source software capitalized on the aggregate efforts of communities. However, not every web 2.0 site or open source project succeeds in attracting or retaining contributors. In the case of web sites, this is often due to poor user interface or other highly visible problems

Software projects, however, typically suffer from a different problem: the cost of creating and integrating contributions. This ailment also afflicts proprietary software, where it is actually rarely diagnosed, and even more rarely treated.

I argue that all substantial software projects should strive to minimize two variables:

Aggregate cost adding a contributor [1]
Maximum individual cost of accepting a contribution

If either of these variables are not minimized, a project will fail to fully utilize the efforts of its key members, never mind begin to capitalize on the long tail of contributions. High contribution costs waste countless millions of hours of productivity.

In a company, the easiest way to measure the cost of adding a contributor is to measure the time from signing the employee agreement until the moment that the software builds and runs on the new-hire's development machine. Don't forget to count this in man-hours because this will surely require the efforts of a manager, a more senior engineer, or both.

Additionally, you should also count the time before the new developer feels some level of confidence in the codebase. Can this new person dive right in and start hacking on a new feature? Could they fix a bug without everything falling apart? How many times will they need to knock on the door of the expert? [2]

Measuring the cost of accepting a contribution is subtly different. The aggregate cost is not of critical importance as long as it is justifiable given the benefit of accepting a contribution. However, minimizing the aggregate cost of accepting a contribution is already the goal of any successful software project. Teams are always striving to be more efficient, but it is a local optimization. If you want to increase the number of contributors and contributions, you need to minimize the maximum cost to maintainers, newcomers, other team members, and communities.

As any good manager knows, they must suffer the pain of meetings, accept frequent context switching, and act as a shield from politics. They must do all of this to ensure their reports are as productive as they possibly can be. Similarly, senior engineers should donate some of their time to analogous tasks in component and systems ownership. Although these tasks introduce a cost which causes an individual contributor to spend less time actually contributing, it opens the door to greater net contributions.

So here come's the concrete advice part of the post...

Some ways to reduce the cost of adding a contributor

Buddy builds - One button should instantly and automatically allocate a fresh machine, enlist in the source tree, build without additional dependencies, execute all tests, and provide detailed status and results.
Standardized tools - Allow contributors to transfer their skills in version control, issue tracking, building, testing, and communication. Every team is different, but the more you can reasonably share, the more you simulate one cohesive team.
Encourage branching
Understand that contributions can come from anyone [3]
Keep documentation up to date and complete - Hold new project members accountable for the correctness and completeness of the newcomers wiki page.

Some ways to reduce the cost of accepting a contribution

Automated unit tests - Insist on a high code coverage and a low flakiness rating. Require a 100% pass-rate before committing to the main branch.
Explicit coding standards
Code reviews
"Lieutenants" - Put an "owners" file in each directory of your source tree. Fill it with the usernames of those who should perform the code reviews for those files and sub-directories. Keep these files up to date.
Widely and clearly communicate project vision and direction - This ensures misguided contributions will be rejected swiftly.
Regular release cycle - Do I really need to say "release early, release often"? Releasing often enough will force you to reduce the cost of your release process as well.

Got any more?

[1] This post was prompted by my ~~startup~~ hobby project's failure in this respect. Adding a third team member took several hours of my attention, but should have taken several minutes.
[2] Sorry Stephen!
[3] Some discussion lists at Microsoft are notorious for responding to internal feedback with the URL http://career/ and possibly a snarky comment. These people are artificially setting the cost of adding a contributor to be a career left-turn and the cost of accepting outside contributions to infinity. I hope these dinosaurs go extinct.

Rift is available on NXE!

2008-11-20T20:07:00.001-08:00

You can download the demo or purchase the game on the web or from the Community Games section of the Games Marketplace on your NXE dashboard. If you get it on the web, it will automatically be downloaded to your Xbox 360 the next time you sign in.

Sure, the game isn't worth the $1.75, but you can still pay for it anyway because you are super generous. And sure, the game has serious design flaws because it was put together in three days as a homework assignment. And sure, the game is loaded with performance issues because it was ported to XNA Framework 3.0 and "polished" for release in two evenings involving beer. But you are super generous.

Definitely check it and all the other great XNA Community Games launch titles. They all have free demos. Exciting times!

Rift submitted for play test

2008-11-08T17:48:00.001-08:00

I've uploaded Rift for play test by XNA Creators.

If you are an XNA Creators Club premium member, please download the game and post your comments in this forum thread. Later this week, I will be submitting the game for final peer review. Hopefully, the game will make it through the pipes in time for the New Xbox Experience and Xbox LIVE Community Games launch on November 19th!

Control flow has to flow

2008-10-27T18:45:00.001-07:00

I've been experiencing MSBuild a bit at work lately. Ignoring the mortal sin of designing a programming language within XML, MSBuild fails on usability in another fundamental way: control flow has to flow.

In general, there are two key approaches to control flow. One approach is the procedural "jump" approach. This includes all manner of conditionals, loops, and more. The second approach involves functional constructs such as recursion and pattern matching. These two approaches are blended between procedural languages, functional languages, and everything in between and beyond.

The procedural and functional mechanisms can be visualized in terms of forwards and backwards. Procedural control flow moves forward in a set of steps which are executed in sequence as defined. Functional control flow begins with the results and works its way backwards pulling in bits and pieces of data and logic along the way.

MSBuild uses neither of these approaches.

That was a lie. MSBuild targets are executed in sequence procedurally. Targets can also define a "DependsOn" property which works functionally to pull in pre-requisite tasks.

The truth is that MSBuild introduces (and often requires) a very unwelcome third approach in the form of the "Condition" property. The "Condition" property is neither forwards nor backwards control flow. It is sideways. MSBuild has only one global namespace of variables (confusingly, called "properties") which can be set or tested at any time.

With either a procedural or functional control flow, you can follow the flow simply in your text editor. In a structured procedure, you simply trace through the code pretending to be a human interpreter. In a functional method or when dealing with sub-procedures, you can trivially search for functions by name as they are called. In a sense, you are "flowing" through the code like water though pipes. You are either pumping water through or sucking water out.

With sideways control flow, you need to search for all references to every variable at every step of both the procedural and functional code flows. This is a laborious and error prone operation because after finding a relevant "Condition", you need to analyze the forward and backwards control flow which led to that sideways control flow construct! There is no "flowing".

Sideways control flow leads to a combinatorial explosion of forward and backwards control flows. The developer is required to keep all of these flows in their heads all at once. Meanwhile, in normal programming languages, you only ever need to deal with just two: forwards and backwards.

Blog resolution

2008-10-18T13:42:00.001-07:00

I now have 9 blog subscribers. Maybe more; if you aren't subscribed to the http://feeds.feedburner.com/YouWorkForMeComputer version, please switch. However, my site gets a pretty strong amount of other traffic from recruiters and other people more interested in my resume than my blog. While feed subscribers never even see the site in its original layout, about 65% of the visitors have a horizontal resolution greater than or equal to 800 pixels. The previous template was designed for a minimum horizontal resolution of 600 pixels, so I added another 100 pixels to the content.

I've been writing my posts with Windows Live Writer. It has a really cool feature:

"Weblog" menu
"Edit Weblog Settings..." menu item
"Editing" tab
"Update Style" button

Poof! Now my editor has the same width as my Blogger template. I love it when software just works. Now if only it could do this automatically...

Imaginary APIs

2008-10-08T01:48:00.001-07:00

As promised, I will explain my favorite way to design APIs. I call it "imaginary APIs". I'm far from the first person to have thought of this and I've even seen this used in practice by others, but I have no idea if there is an official name for it.

In short, the idea is that you write code which consumes an imaginary API and then you imagine it works. Then, you write some more code which does something more advanced with your imaginary API and imagine that works too. Then, you figure out how to get your magic code to compile, run, and work.

Success is measured by how closely your final API matches your imaginary API and by how much you need to modify your magic code to be a working sample you can ship along side your API. That's it.

As an example, let's imagine an API for a simple home movie editing program. I'm sorry about the excessively narrow blog formatting.

var video = new Movie();
video.StoryBoard.Add(
    Image.FromFile("title.png"));
video.StoryBoard.Add(
    new CrossFade(TimeSpan.FromSeconds(2.0f)));
video.StoryBoard.Add(
    Movie.FromFile("scene1.wmv"));
video.StoryBoard.Add(
    Movie.FromFile("scene2.mpg"));
view.Export("myMovie.avi");

A lot of voodoo is going to have to happen behind the scenes to make this code work! But you will thank yourself when you need to implement the UI and your client developers will thank you when they want to generate a highly customized slide show or develop a nifty flip book drawing application.

Simplicity belongs in APIs

2008-10-08T01:01:00.001-07:00

In a recent post, I insisted that flexibility belongs in APIs.

Simplicity also belongs in APIs. There are two key things to simplify in APIs:

Common tasks
Surface area

Since you are jam packing your APIs with loads of powerful and flexible functionality, you might have a lot of functions and data structures. You may have hoards of configuration options or capability bits, and a whole mess of operations to perform on them. However, your UI is simple and has picked sensible defaults for every operation. Your APIs should do the same. In fact, you should expose the same simplified operations your UI uses, so that developers do not have to guess what arguments you pass or in what sequence you composed more complex operations. If you have widely used private helper methods you've developed for your UI, you have identified common tasks for which your helper methods should be promoted to address.

Developers should be unwittingly led to these simplified methods, they shouldn't have to go searching for them. For example, the .NET framework provides a significant number of overloads for opening a file. The most complex of these takes parameters for file access, sharing, creation behavior, and more. The simplest of these, just opens the damn file. Reading all of the lines from a file is a very common task that involves opening a file, iterating over each line, accumulating the results, and then closing the file. Or you could just call File.ReadAllLines and simply get an array of strings.

ReadAllLines is completely implemented using StreamReader, but was placed in the File class instead of the StreamReader class because more people would look there first. Organize APIs by how naive developers would look for them, not experts. Remember that developers, like users, are perpetual intermediates. They are only going to write the code once (as beginners) and iterate on it several times until it works (as intermediates). After that, they will move on to something else. There won't be time for them to become experts.

You have limited screen space for features, but you can always fit more APIs. You develop a great UI by removing the features you don't need. However, you develop a great API by avoiding adding the features that you don't need. If there is a set of parameters no one will ever change, remove the ability to change them. If there are multiple approaches to the same common task, choose one. The less surface area there is for a developer to search and understand, the more likely they are to quickly choose and implement an effective solution.

In a future post, I will discuss my favorite technique for designing APIs.

Why so serious?

2008-09-16T20:37:00.001-07:00

Software doesn't need to be serious all the time. Of course, this applies to games, software toys, and consumer focused products, but it also applies to enterprise class software.

Compare these two screen shots:

"Hurray, no spam here!" versus

"There are no items to show in this view."

Which message would you prefer to see each time you open your spam folder? Google's, of course.

Does Google's choice of words cause you to question the quality of their software? Do they cause you to question whether or not you can do serious work with their tools? Of course not.

Just because you are doing serious work, doesn't mean you need to be serious all the time. Research, anecdotes, and all the other evidence clearly show that a little bit of humor and personality injected into the workplace improves productivity and moral. Same goes for your software. Inject some personality. Include some humor. Everyone will be better off.