You work for me, Computer

NoScript Add-on Install Base

2010-06-04T19:37:00.001-07:00

The official NoScript site downloads via the Mozilla Add-ons page, so its download count is probably accurate: 67,616,402

NoScript does not publish their individual add-on usage statistics, but the global download/usage ratio can be calculated from the statistics on the Firefox Add-on home page:

1,962,617,946 add-ons downloaded

157,090,095 add-ons in use

About 8% of downloaded add-ons are still in use. Assuming NoScript's usage ratio is comparable to the average, approximately 5.4 million installations of FireFox are running NoScript. Let's ignore the fact that NoScript's usage ratio is probably much lower than the average, due to the fact that it breaks most web pages.

I'd wager that the average NoScript user has at least two machines, so the total number of NoScript users is probably less than 2.7 million.

There are over 230 million internet users in the USA.

Even if 100% of NoScript users were Americans, they form 1% or less of the general population. If you, like me, believe that I have been generous to NoScript here, it is likely that no script users are no more numerous than 1 in 1,000.

Even if the user is using NoScript, they can whitelist your site. You can probably add <noscript>WARNING: THIS SITE IS BUSTED WITHOUT JS</noscript> to the top of your page and call it a day. If you are feeling generous, redirect no-script users to the mobile version of your site and tell them why.

tldr: Assume that human user agents have Javascript.

Leaving Microsoft

2010-04-14T14:55:00.001-07:00

April 30th will be my last day on the Microsoft XNA team. I'm leaving to start work on my own startup full-time. I also hope to invest some more effort into this blog. Exciting times!

ORMs and Declarative Schemas

2009-10-06T10:32:00.001-07:00

My prior post was more controversial than I anticipated. In hindsight, I should have realized what a hot button issue web frameworks are. One assertion went all but unnoticed. I expect it may be even more controversial: the schema-generative ORM paradigm is fundamentally flawed.

Disclaimer

I came to this conclusion while working with Django's ORM, but this post is completely object-relational mapper agnostic. We are now using SqlAlchemy, but we are not using any of the many available declarative layers. Instead, we are using schema reflection and semi-automatically configured mappers. This is not an argument against ORMs. It is an argument against generating database schemas from ORM declarations. By extension, this is an argument against Django's ORM because Django uses an exclusively schema declarative model. That said, Django's ORM is far from alone in this camp.

Data Outlives Code

When code is dead and gone -- be it through rewrite, obsolescence, or by other means -- the data will still be there. Longevity implies slower evolution; data is always more difficult and riskier to change. Data is also more valuable. What if Facebook rebooted their database? They've already rebooted their software several times.

Schemas are data. As data, schemas are longer lived, less flexible, and more valuable than code. These factors alone suggest that the database itself should hold the authoritative schema, not a class declaration in the code.

If you have inherited data from another project, you already know this lesson. You can't generate the schema from code because the schema already exists. You can mimic the authoritative schema in your declarations, but it is easier and more accurate to use reflection.

Ineffective Domain Objects

Object relational mapping is primarily a serialization problem. Every serialization solution has its quirks. The scale and number of quirks seem directly proportional to the absolute difference between the runtime and storage representations. Since a database is a completely distinct type system, rather than an opaque byte array, serialization to a database can have particularly quirky quirks.

Modeling is one of the fundamental challenges of software development. Capable developers prefer highly expressive or unconstrained type systems to aid with modeling. Generally, runtime type systems are more expressive than those found in databases. Declarative relational mappers, however, constrain the programming language type system to its less expressive counterpart. When building domain objects, the developer must think in terms of the database's type system, not the programming language's.

While using a declarative object relational mapper, developers are effectively trying to design storage and runtime models simultaneously. On average, superior results are achieved by modeling these two concerns separately and then solving an additional subproblem: serialization mapping. You might wind up with slightly more code, but it will be easier to understand and maintain.

SQL Is Not Going Away

Despite forcing a less expressive type system on to the developer, declarative ORM layers attempt to treat the database as an implementation detail. However much we wish this were true, the database is not a detail which can be ignored. Sooner or later, you are going to have to open your database shell and write a SQL expression. This requires knowledge of your database's particular SQL dialect and idiosyncrasies. Exacerbating this issue, generated schemas are typically full of name mangling and other ugliness. It is far more pleasant to work with a carefully designed schema than one that compromises for the ORM or the runtime type system.

Schema and Declarations Diverge

Data migrations present the most pressing need to work with SQL directly. As an widely unsolved problem, automation can not be trusted. Unless you are painstakingly simulating the schema generator, the production database schema will slowly diverge. Most deviations are tolerable as they will not affect the runtime behavior, but it is wise to minimze differences between production, staging, and development environments. To faciliate this, store migration scripts and backed-up schemas in version control.

Some deviations will directly affect runtime behavior. For example, consider the case where two versions of an application are running in production. An is_read boolean column was added to a message table in the database. It's default value is false. When a row's page is viewed, the is_read column is set to true. The old version of the application doesn't know about the column, so it can not set the flag. When the new version is rolled out to everyone, affected user will see a bunch of read items marked as unread! The solution is to set the column's default to true, but initialize it to false in the application. Declarations must either deliberately deviate from the schema or present a misleading default value to be overridden during initialization. This is just a simple example, real world schema migrations can be significantly more complex and suffer from numerous more subtle problems.

URL routing and views

2009-08-24T22:44:00.000-07:00

As promised, I'd like to elaborate on the URL routing system I came up with.

Weighing at less than 200 lines of code (including example), I'll let it speak for itself: download it.

This approach seems to be working great for us. Love it? Hate it? Feel free to let me know what you think.

Dropping Django

2009-08-19T00:07:00.001-07:00

My partner and I built a non-trivial web site on Django. When the next version ships, there might not be a single Django module imported.

We're not trying to drop Django; it is just sort of happening. Piece by piece, it is failing to meet our needs. Despite the marketing copy on the Django site, most components of the framework are tightly coupled enough to make customization frustrating. It is often easier to rewrite core framework components than to implement them on top of the existing extensibility points.

What follows is a loose chronology of our migration away from Django.

URL Routing

A flat list of patterns violates the DRY principal when creating nested URLs. Trees are a superior representation. Having a tree of views also enabled us to optionally associate a "binder" function with each node. These bind functions are executed for each URL component from left to right, filling the template context as they go. Breadcrumbs are automatically generated as each node binds, but only the last node executes its full view logic.

Authorization

Our site enforces permissions on every resource, but Django's database ACLs would have been prohibitively numerous. Instead, views or their URL binders may raise an AccessDenied exception. Upon catching such an exception, a middleware layer serves a login form. This ensures users have permission to access the current resource, as well as all ancestor resources bound to the URL.

Authentication

Both of Django authentication's key extensibility points are flawed. These two extensibility points are "user profiles" (storing additional per-user data) and custom credentials (such as for logging in via email address instead of username). Django's documentation and numerous internet sources cover both topics, but all of the guidance lacks important caveats. The admin UI, in particular, is very easy to break with either extensibility mechanism.

Extending the User model with the ORM requires a one-to-one database relationship. This relationship can be implemented with a "user profile" setting, an explicit foreign key, or model inheritance. Each approach has its own strengths and weaknesses in terms of performance, API semantics, subtle behavioral changes, and outright bugs.

Enabling custom credentials requires implementing a trivial authorization "backend" object. Unfortunately, it is non-trivial to replace usernames with email addresses. The admin UI's login form refuses to accept email addresses without hacking the template. Even if you hacked the template, the User model would still enforce a non-null constraint on the username field and the generated database schema enforces a uniqueness constraint as well. It turns out to be easier to fill the username field with a dummy value and "support" both forms of authentication with your backend, but you won't come to that conclusion until your head has already bore a hole in your desk.

Templating

We do our best to keep view and template logic separate. Django's templates are targeted at designers, who aren't implementing any real logic anyway. However, we're a pair of hackers. Sometimes it is just more convenient to put a little bit of logic in the views. Besides, templates are code; code needs to be reviewed and tested. We wouldn't ever hire a designer who couldn't pass a code review for some trivial template logic.

We needed a pragmatic template language to replace Django's idealistic one. Any template language with greater expressive power would have been welcome, but Jinja2 fit the requirements and provided the easiest migration path. Ultimately, we'd prefer to use something like HAML, but there doesn't seem to be a Python equivalent besides the inactive GHRML project. We are, however, using SASS. I will never write CSS by hand again.

ORM and Admin UI

One of Django's most touted features is the Admin UI. For simple "active record" style database models, the Admin UI is a huge time saver. Sadly, it struggles a little bit with nullable fields and is tricky to customize. You'll definitely need to write custom UI for complex models, but by and large the admin solves the problem at hand: viewing, creating, updating, and deleting database rows.

After using the Admin for a little, I found myself missing Microsoft Access. I never thought I'd say that, but it is true. ~~Django's admin does not support sorting, filtering, or other impromptu queries.~~ Edit: It turn's out I was mistaken about sorting and filtering, but I stand by the core message of this section. I found myself writing impromptu queries in the database and Python shells. After a while, I just gave up and installed a desktop client. I haven't visited the Admin UI since.

Django's ORM has shortcomings with respect to querying, especially for joins and aggregation. It has been improving over time, but it will likely never reach the capability of projects solely focused on databases, such as SqlAlchemy. With the admin having fallen into disuse, the Django ORM lost all advantage. Beyond Django's specific weaknesses, I've come to believe that the schema-generative ORMs paradigm is fundamentally flawed. That is a topic that deserves an entire (Django-agnostic) post of it's own. We are now using SqlAlchemy via schema reflection; no declarative layer.

Form Validation and Generation

Here is where our chronology meets present day. We are still using Django form validation, but never used form generation beyond scaffolding. Nearly all of our templates customize labels and display of errors. Additionally, embedding widget information in the Python code is cumbersome during template development. Django forms is a quality validation library, but there are some inconsequential style things that I like better about FormEncode. Preferences aside, the difference isn't large enough to justify switching.

While I like FormEncode, I'm still not sold on its anti-form-generation companion, htmlfill. I think there is a middle ground with form generation that provides scaffolding during development, smoothly transitions to production use, and cooperates with validation. As we implement more complex client views, I'll be on the lookout for ways to improve our form development toolbox.

So, ugh... What's left?

Besides a few isolated helper functions, not much is left of Django.

The last big ticket item is the HTTP framework and WSGI server. We could continue using Django as if it were CherryPy or Paste, but Django has this nasty habbit of insisting on running your code for you. The settings and manage.py infrastructure are fiddly for deployment and don't really add any value over simple scripts using our application like a library. Might as well use a simpler WSGI library, and replace those over-engineered management/commands/foo.py files with vanilla scripts/foo.py files.

Moral of the Story

I'm sure there are numerous lessons to be generalized from this journey. Personally, I've developed a moderate fear of the word "framework", as well as altered the way I think about software abstractions. I think the most important lesson, however, is one I already knew: choose the right tool for the job. Unfortunately, we had no idea what the right tool was when we started. I'm not sure we know any better now.

AppWeek

2009-06-12T20:02:00.001-07:00

Shawn's AppWeek post inspired me to write one too. AppWeek is our chance to be creators for a little while and it was a lot of fun. I didn't set out to build something nearly as ambitious as "Super Avatar Sample Smashup EXTREME! - 'Capture the Cat' edition", but I did get to take a swing at a game I've wanted to build for a while: ~~Rock'em Sock'em Avatars~~ Avatar Boxing. Avatars, being a new feature in this release, were an unwritten requirement for all of the AppWeek games. Between SASSECTCE, my game, and the many others, Avatars were chasing cats, beating each other up, play futuristic sports, falling off buildings, dancing in a cloud of gems, being launched from canons to save the world, and much more. All this excitement was almost too much for a bunch of exhausted engineers, but that's what the beer was for during the game unveilings.

Here's what the game looked like with the basic animations wired up. You'll notice that the avatars have been hitting the gym. That's because their arms were too short to reach each other! I added a little extra bulk because I was laughing too hard not to. I directly bound the game pad triggers to the shoulder and elbow joints and rigged up the chase camera sample to inspect my work. There wasn't much game play yet, but it was already fun. That's always a good sign.

Even with just one week, I decided to invest some time into debugging visualizations. That turned out to be a really great idea.

Then, I added some collision spheres for the heads, hands, and upper bodies. This was a hacky, trial and error process. Thankfully, C# compiles quickly.

At this point, I spent an entire day working on the physics. I wanted the avatars to bounce/wobble when they got hit, so I rigged up some complex spring systems. Things were starting to work, but I'm generally pretty bad at this sort of thing and my simulation routinely exploded. The avatars arms went shooting off into space and I was getting pretty frustrated. No screen shots of that chaos because I am embarrassed.

With half a day to go, I added the obligatory damage bars and some rudimentary hand-to-head collision detection.

I was feeling pretty good about the game, despite my physics failures, it was pretty fun anyway. I wondered down the hall to chat with Jace, who had just added sound effects to his game. His game was hilarious before, but the sound effects were priceless. I ejected the sound effect CD out of his machine, yoinked it, and took off running. An hour later (and 10 minutes after the deadline), my game had some sweet punch and miss sounds. I also made the avatars' heads pop up when their damage bar was full, accompanied by an awesome zip-tie sound.

At our team happy hour, I'd like to think Avatar Boxing was a fan favorite. I certainly had fun making it! I hope everyone enjoys Avatar support in the new XNA Game Studio.

PowerShell: condemned to reinvent

2009-04-20T02:33:00.001-07:00

I tried PowerShell when it was first released, but never used it for real work. I recently attended a "brown bag" presentation about PowerShell. This presentation spurred me to augment our team's environment with PowerShell and I have been using it every day since.

In the past weeks using and abusing PowerShell, I have drawn two conclusions:

PowerShell has a killer set of standard tools with brilliantly designed usability.
The PowerShell team doesn't understand UNIX and therefore were condemned to reinvented it, poorly -- with apologies to Henry Spencer.

First things first: if you spend any time working with Windows, get PowerShell. Now. Stop reading my blog and go download it immediately. It mops the floor with cmd.

The key premise behind PowerShell is that it operates on live .NET objects. This is beneficial because it eliminates a lot of the text cutting and manipulation common in shell scripts. Additionally, it puts the full .NET Base Class Library into your scripting toolbox. PowerShell tools, known as commandlets, typically only render the most common fields for their objects, but the less common fields are easily available in memory. By convention, Commandlets are named with a verb-noun pattern and support a common command line parsing behavior. The repository of commandlets and the command line options of each are easily queried and highly consistent. All this meta-data makes PowerShell a breeze to learn.

I fell in love with the the discoverablity and ease of use when I tried to kill a collection of runaway processes:

PS> get-command -noun process

CommandType   Name           Definition
-----------   ----           ----------
Cmdlet        Get-Process    Get-Process [[-Name] <String[]>] [-Verbo...
Cmdlet        Stop-Process   Stop-Process [-Id] <Int32[]> [-PassThru]...

PS> get-process notepad | stop-process
PS> get-alias | where { $_.definition.contains("Process") }

CommandType   Name   Definition
-----------   ----   ----------
Alias         kill   Stop-Process
Alias         ps     Get-Process

PS> ps someotherapp | kill

OK, that's pretty cool and oh-so-very Unixy -- right? Wrong. Notice the "CommandType" column in the results of get-command. There are many other types of commands besides commandlets: functions, filters, scripts, applications, etc. Each of these has slightly different semantics for pipes and parameters. Applications, for example, have no way of accepting .NET object pipes. You must develop a separate commandlet. Yikes!

Compare to Unix: all commands are applications which accept a command line and pipe byte streams in and out. Much simpler, but byte streams aren't as friendly, discoverable, and maintainable as object streams. However, the brilliantly simple thing about Unix is that, when you get right down to it, object streams are just byte streams! There is absolutely nothing stopping you from implementing get-process and stop-process as Unix programs which pipe object references, JSON, pickled Python objects, XML, S-expressions, or any other data format you fancy. Doug Mcllroy, the inventor of Unix pipes, was right: text streams are the universal interface.

Actually, this is no different on Windows. All of the PowerShell commandlets could have been implemented as applications which import a library. This library would replace main in much the same way as winmain, provide a metadata enriched implementation of getopt, man, etc. There is no need to invent a new shell in order to acquire the power of piping objects. Sure, cmd is old and needed to be retired for many other reasons, but it is a real shame that the PowerShell toolset is not available to those of us stuck in batch scripts.

Personally, I would really like to see such a library developed. Microsoft has certainly proved one thing with PowerShell: steep learning curves are not intrinsic to command line interfaces. Unfortunately, commandlets are two steps forward and one step backwards. I have no doubt that we can retake that forward step.

Language-oriented programming: too much, too fast

2009-02-09T02:49:00.001-08:00

There was some recent discussion on Hacker News about the 2004 article on "Language Oriented Programming" by Sergey Dmitriev. With my growing interest in programming languages, this article, and the pending release of JetBrain's Meta Programming System, I have been thinking a lot about the future of programming.

As Sergey points out, MPS is not the first entry into this paradigm of software development. Intentional Programming was being demonstrated by Microsoft Research as early as 2000. Wikipedia lists several implementations of the language-oriented programming concept. (OK, that is enough links for now!) Sadly, none of these systems have been met with wide spread success. Despite my unlimited respect for the JetBrains team and love of their products -- especially Resharper -- I expect MPS to fail to achieve critical mass. I don't think many programmers would disagree. It's just too much, too fast.

Software development needs to evolve, not start anew. If you change too much at once, the common developer simply won't accept it. Language designers are still re-inventing Lisp piece by piece in an effort to make sense of all the different concepts for Mort. In order to improve adoption of language-oriented programming concepts, the critical path must be identified and the programming community must be lead down that path one step at a time.

I believe that the first stepping stone is the box editor.

Both MPS and Intentional Programming provide a visual tree of boxes editor and store the abstract syntax tree in a sort of source code database. Both of these concepts are critical to power of language-oriented programming, but changing the storage of source code is simply too radical of a change. Well, actually, source code is routinely stored in databases for IDE features, but the authoritative storage always remains text files. All of a programmers most trusted tools operate at the string level and it is simply impractical to throw everything out at once.

This is why I propose the creation of a new editor which uses an abstract syntax tree box editing model, but preserves the source as text. This editor would have to play nice with other developers who are using traditional text editors. It should be possible for a single developer to try the editor for a day or two without another collaborator even noticing. There are clear benefits to a box editor even in the absence of language-oriented features. For example, navigation of source code by parent, child, or sibling relationships instead of by word or by line. Advanced renders can be used to layout math expressions or to show a referenced image file in a comment.

Editors are a religion, start a new one. If some box editor becomes popular enough, other tools would spring up around them. To help the process, the editor should have well exposed extension and composition mechanisms. For example, it should be trivially easy to add new renderers for various nodes. It should be equally trivial to utilize the abstract syntax tree system to create stand alone code analysis tools.

If box editors garner enough of a following, then the language-oriented programming advocates can seek the next stepping stone.

Break the cycle of broken builds

2009-02-03T23:07:00.001-08:00

Broken builds suck. Breaking the build sucks. No one wants to have their work stopped by others' build breaks and no one wants to cause work stoppage on account of their own build breaks.

So then why do builds break so often? You'd think that after so many years of software development, someone would have solved this problem. Microsoft employees tens of thousands of engineers across hundreds, if not thousands, of code projects. Many of those code projects are development tools for the engineering community at large as well as engineers within Microsoft. Yet, still, all of the tools at the disposal of those engineers are highly prone to build breaks.

Below, are three ideas for reducing build breaks. Together, they could all but eliminate the problem.

Use a better version control system

Everyone keeps raving about distributed version control, and for good reason. I won't rehash the argument here, but I will say: branch early and merge often. Perforce (better known as Source Depot within Microsoft) and it's spiritual successor Team Foundation Server are completely unbearable. A great majority of build breaks can be contained if people are only pulling known good changes from dedicated integrators. Everyone checking in all at once to a main branch simply doesn't work with anything but the most tightly knit of teams. Even if the does build break, it should only block the person waiting on that particular change.

Isolate build breaks

`sd submit -c 12345` translates roughly to "I'm 100% confident that my changes are solid and am equally confident that I am submitting the changes that I think I am." You never can be 100% confident, don't take any chances: submit to a private branch which is being monitored by a build service. The server should detect your submission, run a full buddy build (with unit tests!), and then submit it to the shared branch only after it has passed.

If your builds take too long for this, people should be able to grab changes down from your unverified branch on an as-needed basis. If this happens too frequently: your builds take too long. Refactor your system into smaller components and formalize the contractual interface between those components. Then buddy build just the components which have changed. Verify that the interface hasn't changed with a unit test. This will stop compile errors at the boundaries.

Go lazy, be late

This one is a big of a pipe dream... software systems developed with dynamic languages do not experience build breaks nearly as often as those developed with static languages. This is because dynamic languages are typically lazy-loaded and utilize late bounded method invocations. If someone adds a new button to the UI which calls a method in a file they forgot to add to source control, that should only block people who want to click the button!

There is no firm technical reason why statically or JIT compiled systems can't simulate this. Most compile errors could theoretically be treated as a runtime error. Clearly, compile errors are valuable, but should be ignorable whenever possible. Don't ignore them in your known good builds, but ignore them when they stand between you and forward progress.

Since most of us aren't writing our own compilers are full tool chains, here is some more practical advice for projects using statically compiled languages: write code which fails gracefully. Runtime errors can be build breaks just the same as compile errors, but try to defer the negative consequences as much as possible. One broken feature shouldn't break the whole project; unless, of course, you are about to ship. In production builds, perform verification code at startup or in unit tests, but try to be lazy and late in development builds.

Contributions

2009-01-26T03:49:00.001-08:00

The modern, read/write web thrives on processes which accept and coordinate a large set of contributions from a set of contributors that is almost as large. Before this was common on the web, open source software capitalized on the aggregate efforts of communities. However, not every web 2.0 site or open source project succeeds in attracting or retaining contributors. In the case of web sites, this is often due to poor user interface or other highly visible problems

Software projects, however, typically suffer from a different problem: the cost of creating and integrating contributions. This ailment also afflicts proprietary software, where it is actually rarely diagnosed, and even more rarely treated.

I argue that all substantial software projects should strive to minimize two variables:

Aggregate cost adding a contributor [1]
Maximum individual cost of accepting a contribution

If either of these variables are not minimized, a project will fail to fully utilize the efforts of its key members, never mind begin to capitalize on the long tail of contributions. High contribution costs waste countless millions of hours of productivity.

In a company, the easiest way to measure the cost of adding a contributor is to measure the time from signing the employee agreement until the moment that the software builds and runs on the new-hire's development machine. Don't forget to count this in man-hours because this will surely require the efforts of a manager, a more senior engineer, or both.

Additionally, you should also count the time before the new developer feels some level of confidence in the codebase. Can this new person dive right in and start hacking on a new feature? Could they fix a bug without everything falling apart? How many times will they need to knock on the door of the expert? [2]

Measuring the cost of accepting a contribution is subtly different. The aggregate cost is not of critical importance as long as it is justifiable given the benefit of accepting a contribution. However, minimizing the aggregate cost of accepting a contribution is already the goal of any successful software project. Teams are always striving to be more efficient, but it is a local optimization. If you want to increase the number of contributors and contributions, you need to minimize the maximum cost to maintainers, newcomers, other team members, and communities.

As any good manager knows, they must suffer the pain of meetings, accept frequent context switching, and act as a shield from politics. They must do all of this to ensure their reports are as productive as they possibly can be. Similarly, senior engineers should donate some of their time to analogous tasks in component and systems ownership. Although these tasks introduce a cost which causes an individual contributor to spend less time actually contributing, it opens the door to greater net contributions.

So here come's the concrete advice part of the post...

Some ways to reduce the cost of adding a contributor

Buddy builds - One button should instantly and automatically allocate a fresh machine, enlist in the source tree, build without additional dependencies, execute all tests, and provide detailed status and results.
Standardized tools - Allow contributors to transfer their skills in version control, issue tracking, building, testing, and communication. Every team is different, but the more you can reasonably share, the more you simulate one cohesive team.
Encourage branching
Understand that contributions can come from anyone [3]
Keep documentation up to date and complete - Hold new project members accountable for the correctness and completeness of the newcomers wiki page.

Some ways to reduce the cost of accepting a contribution

Automated unit tests - Insist on a high code coverage and a low flakiness rating. Require a 100% pass-rate before committing to the main branch.
Explicit coding standards
Code reviews
"Lieutenants" - Put an "owners" file in each directory of your source tree. Fill it with the usernames of those who should perform the code reviews for those files and sub-directories. Keep these files up to date.
Widely and clearly communicate project vision and direction - This ensures misguided contributions will be rejected swiftly.
Regular release cycle - Do I really need to say "release early, release often"? Releasing often enough will force you to reduce the cost of your release process as well.

Got any more?

[1] This post was prompted by my ~~startup~~ hobby project's failure in this respect. Adding a third team member took several hours of my attention, but should have taken several minutes.
[2] Sorry Stephen!
[3] Some discussion lists at Microsoft are notorious for responding to internal feedback with the URL http://career/ and possibly a snarky comment. These people are artificially setting the cost of adding a contributor to be a career left-turn and the cost of accepting outside contributions to infinity. I hope these dinosaurs go extinct.

Rift is available on NXE!

2008-11-20T20:07:00.001-08:00

You can download the demo or purchase the game on the web or from the Community Games section of the Games Marketplace on your NXE dashboard. If you get it on the web, it will automatically be downloaded to your Xbox 360 the next time you sign in.

Sure, the game isn't worth the $1.75, but you can still pay for it anyway because you are super generous. And sure, the game has serious design flaws because it was put together in three days as a homework assignment. And sure, the game is loaded with performance issues because it was ported to XNA Framework 3.0 and "polished" for release in two evenings involving beer. But you are super generous.

Definitely check it and all the other great XNA Community Games launch titles. They all have free demos. Exciting times!

Rift submitted for play test

2008-11-08T17:48:00.001-08:00

I've uploaded Rift for play test by XNA Creators.

If you are an XNA Creators Club premium member, please download the game and post your comments in this forum thread. Later this week, I will be submitting the game for final peer review. Hopefully, the game will make it through the pipes in time for the New Xbox Experience and Xbox LIVE Community Games launch on November 19th!

Control flow has to flow

2008-10-27T18:45:00.001-07:00

I've been experiencing MSBuild a bit at work lately. Ignoring the mortal sin of designing a programming language within XML, MSBuild fails on usability in another fundamental way: control flow has to flow.

In general, there are two key approaches to control flow. One approach is the procedural "jump" approach. This includes all manner of conditionals, loops, and more. The second approach involves functional constructs such as recursion and pattern matching. These two approaches are blended between procedural languages, functional languages, and everything in between and beyond.

The procedural and functional mechanisms can be visualized in terms of forwards and backwards. Procedural control flow moves forward in a set of steps which are executed in sequence as defined. Functional control flow begins with the results and works its way backwards pulling in bits and pieces of data and logic along the way.

MSBuild uses neither of these approaches.

That was a lie. MSBuild targets are executed in sequence procedurally. Targets can also define a "DependsOn" property which works functionally to pull in pre-requisite tasks.

The truth is that MSBuild introduces (and often requires) a very unwelcome third approach in the form of the "Condition" property. The "Condition" property is neither forwards nor backwards control flow. It is sideways. MSBuild has only one global namespace of variables (confusingly, called "properties") which can be set or tested at any time.

With either a procedural or functional control flow, you can follow the flow simply in your text editor. In a structured procedure, you simply trace through the code pretending to be a human interpreter. In a functional method or when dealing with sub-procedures, you can trivially search for functions by name as they are called. In a sense, you are "flowing" through the code like water though pipes. You are either pumping water through or sucking water out.

With sideways control flow, you need to search for all references to every variable at every step of both the procedural and functional code flows. This is a laborious and error prone operation because after finding a relevant "Condition", you need to analyze the forward and backwards control flow which led to that sideways control flow construct! There is no "flowing".

Sideways control flow leads to a combinatorial explosion of forward and backwards control flows. The developer is required to keep all of these flows in their heads all at once. Meanwhile, in normal programming languages, you only ever need to deal with just two: forwards and backwards.

Blog resolution

2008-10-18T13:42:00.001-07:00

I now have 9 blog subscribers. Maybe more; if you aren't subscribed to the http://feeds.feedburner.com/YouWorkForMeComputer version, please switch. However, my site gets a pretty strong amount of other traffic from recruiters and other people more interested in my resume than my blog. While feed subscribers never even see the site in its original layout, about 65% of the visitors have a horizontal resolution greater than or equal to 800 pixels. The previous template was designed for a minimum horizontal resolution of 600 pixels, so I added another 100 pixels to the content.

I've been writing my posts with Windows Live Writer. It has a really cool feature:

"Weblog" menu
"Edit Weblog Settings..." menu item
"Editing" tab
"Update Style" button

Poof! Now my editor has the same width as my Blogger template. I love it when software just works. Now if only it could do this automatically...

Imaginary APIs

2008-10-08T01:48:00.001-07:00

As promised, I will explain my favorite way to design APIs. I call it "imaginary APIs". I'm far from the first person to have thought of this and I've even seen this used in practice by others, but I have no idea if there is an official name for it.

In short, the idea is that you write code which consumes an imaginary API and then you imagine it works. Then, you write some more code which does something more advanced with your imaginary API and imagine that works too. Then, you figure out how to get your magic code to compile, run, and work.

Success is measured by how closely your final API matches your imaginary API and by how much you need to modify your magic code to be a working sample you can ship along side your API. That's it.

As an example, let's imagine an API for a simple home movie editing program. I'm sorry about the excessively narrow blog formatting.

var video = new Movie();
video.StoryBoard.Add(
    Image.FromFile("title.png"));
video.StoryBoard.Add(
    new CrossFade(TimeSpan.FromSeconds(2.0f)));
video.StoryBoard.Add(
    Movie.FromFile("scene1.wmv"));
video.StoryBoard.Add(
    Movie.FromFile("scene2.mpg"));
view.Export("myMovie.avi");

A lot of voodoo is going to have to happen behind the scenes to make this code work! But you will thank yourself when you need to implement the UI and your client developers will thank you when they want to generate a highly customized slide show or develop a nifty flip book drawing application.

Simplicity belongs in APIs

2008-10-08T01:01:00.001-07:00

In a recent post, I insisted that flexibility belongs in APIs.

Simplicity also belongs in APIs. There are two key things to simplify in APIs:

Common tasks
Surface area

Since you are jam packing your APIs with loads of powerful and flexible functionality, you might have a lot of functions and data structures. You may have hoards of configuration options or capability bits, and a whole mess of operations to perform on them. However, your UI is simple and has picked sensible defaults for every operation. Your APIs should do the same. In fact, you should expose the same simplified operations your UI uses, so that developers do not have to guess what arguments you pass or in what sequence you composed more complex operations. If you have widely used private helper methods you've developed for your UI, you have identified common tasks for which your helper methods should be promoted to address.

Developers should be unwittingly led to these simplified methods, they shouldn't have to go searching for them. For example, the .NET framework provides a significant number of overloads for opening a file. The most complex of these takes parameters for file access, sharing, creation behavior, and more. The simplest of these, just opens the damn file. Reading all of the lines from a file is a very common task that involves opening a file, iterating over each line, accumulating the results, and then closing the file. Or you could just call File.ReadAllLines and simply get an array of strings.

ReadAllLines is completely implemented using StreamReader, but was placed in the File class instead of the StreamReader class because more people would look there first. Organize APIs by how naive developers would look for them, not experts. Remember that developers, like users, are perpetual intermediates. They are only going to write the code once (as beginners) and iterate on it several times until it works (as intermediates). After that, they will move on to something else. There won't be time for them to become experts.

You have limited screen space for features, but you can always fit more APIs. You develop a great UI by removing the features you don't need. However, you develop a great API by avoiding adding the features that you don't need. If there is a set of parameters no one will ever change, remove the ability to change them. If there are multiple approaches to the same common task, choose one. The less surface area there is for a developer to search and understand, the more likely they are to quickly choose and implement an effective solution.

In a future post, I will discuss my favorite technique for designing APIs.

Why so serious?

2008-09-16T20:37:00.001-07:00

Software doesn't need to be serious all the time. Of course, this applies to games, software toys, and consumer focused products, but it also applies to enterprise class software.

Compare these two screen shots:

"Hurray, no spam here!" versus

"There are no items to show in this view."

Which message would you prefer to see each time you open your spam folder? Google's, of course.

Does Google's choice of words cause you to question the quality of their software? Do they cause you to question whether or not you can do serious work with their tools? Of course not.

Just because you are doing serious work, doesn't mean you need to be serious all the time. Research, anecdotes, and all the other evidence clearly show that a little bit of humor and personality injected into the workplace improves productivity and moral. Same goes for your software. Inject some personality. Include some humor. Everyone will be better off.

Bug farms

2008-09-10T00:36:00.001-07:00

I recently learnt a new microspeak phrase that I love: bug farm.

Some features are bug farms.

This means that these features are either poorly specified or highly unmaintainable. As a result, they produce a larger than average portion of the product's bugs. They can get into either state though a variety of ways, but by far the most common way is insufficient thought during design or implementation. Additionally, it is also possible to establish a bug farm by building a feature on top of an existing bug farm.

If you encounter a feature which is covered in larvae, you should spray it with insecticides immediately. It is better to use too much insecticide and kill the feature than it is to deliver an infested product to your customers. That is, if the bugs don't rise up and eat you before you get the chance to do either.

Flexibility belongs in APIs

2008-08-28T21:44:00.001-07:00

Software comes in a large spectrum of power and flexibility, as well as an equally large spectrum of complexity and usability. Users also come in a wide range of intelligence and skill. However, if you create software of average power and average complexity, you may satisfy the average users' needs, but they won't love you for it.

One of my favorite books on software, The Inmates Are Running the Asylum, describes the concept of "perpetual intermediates". The book explains that beginners are only beginners for a little while, then they become intermediates. Most users are intermediates for as long as they use the software. Very few users become true experts. At any given time, virtually all of your users will be varying intermediates.

With this powerful realization in mind, you may be tempted to design exclusively for these intermediates. When designing your software you can spin the flexibility and complexity dials to try to hone in this average user. Unfortunately, you would wind up pleasing more users by cranking these dials down much lower.

The fundamental problem is that adding a bit of power and flexibility also adds a bit of complexity. Since most users are intermediates, they can tolerate some degree of complexity, but would honestly prefer not to. They avoid the complexity by only using the small subset of features they have come to understand and only venturing beyond their comfort zone when they absolutely have to. They quickly become intermediates and are no longer troubled by complexity.

When users have to venture into uncharted features, they are effectively beginners, not intermediates. However, the more esoteric or powerful the feature, the more likely it was designed for that small percentage of experts. When the infrequent need arises for these features, users would rather force the problem into the features they know than learn new ones.

Given that users actively avoid complexity and more powerful features, it becomes apparent that these powerful features should be excluded to keep complexity to a minimum. In doing so, you may annoy the 1% of your users who are truly experts and need the power. If you have a million users, 1% -- 10,000 users -- can not be forgotten. You need to give these users this power somehow.

Put the power and flexibility into APIs! APIs enable power users to automate frequent tasks and create the unique functionality they need. As a bonus, new tools will pop up to fill needs you didn't even consider when you originally designed your software.

"Wait a minute!" I hear your cry. "Expert users are not necessarily programmers", you say. This is very true, but there are two factors which mitigate this problem.

First, most expert users of any software have at least some technical ability with which to hack together solutions. If you don't believe me, take a look at the popularity of Visual Basic. Despite the relatively small number of professional software engineers using Visual Basic, it still has very strong search volume and book sales. This is presumably due to its inclusion in Microsoft Office. However, the "real" software engineers view it as not worth their time and these are the people producing technical content on the web. As such, there is less online discussion of the language. Even if an expert is unable to put something together, they probably know someone who can. And even if they don't know someone who can, then they probably have just identified a need to hire someone.

Second, the needs of experts are so specialized that they need to exert a large amount of effort bending flexible software to their needs. Sometimes, no amount of flexibility will solve their problem. Other times, the flexibility is hidden behind so much complexity that it is not cost effective to uncover it. Still other times, users bend software until it breaks. Again looking at Visual Basic, it is quite common for Office users to create macros or scripts to solve problems that built in tools could have solved. Users do this because they didn't know they could or couldn't figure out how to utilize the existing features in the way they wanted to.

The bottom line is that software should be as simple as possible, not simpler. Treat users as intelligent, capable intermediates; do not engage in excessive hand-holding or assume expertise. Where applicable, provide plug-in or extension mechanisms, so that power and flexibility can be leveraged by APIs. Your users will love you for it.

Updated feed url

2008-08-19T22:51:00.001-07:00

If you are subscribed to my blog, please re-subscribe to:

http://feeds.feedburner.com/YouWorkForMeComputer

Thanks!

Computers are everywhere, you no longer have an excuse

2008-08-18T23:17:00.001-07:00

I work in an environment full of brilliant, technically minded people. Try as I might, sometimes I simply forget how little most people know about computers.

In IKEA, I saw a odd wire metal rack which mounted to the side of a desk and held a computer tower suspended off the ground. Why anyone would attach such an ugly thing to a desk is beyond me. What is wrong with the computer being four inches lower, on the ground?

Moments later, an young, female employee pointed to the tower rack and referred to it as a "CPU holder". Never mind that she was actually trying to sell this silly thing to someone, she just called a car engine by the name of a fuel injector!

Microsoft manages interns better than Google

2008-06-24T21:25:00.000-07:00

In the interest of full disclosure: I was not offered a full-time position at Google. I was encouraged to apply again after earning a masters degree or after a few more years of experience. Please do not misconstrue this post as sour grapes; I mainly applied for a full-time conversion at Google to improve my Microsoft offer.

The title of this post is hyperbole. Both companies are full of brilliant engineers, managers, and HR staff. Both do an excellent job of managing interns; they really do care. However, my experiences at Microsoft made it impossible for me to choose Google. Both sets of experiences were overwhelmingly positive, but several individuals at Microsoft went that extra mile and won me over in a big way.

Early during my first internship at Microsoft, it was apparent that my skills were being misappropriated. I wasn't having a whole lot of fun and my direct manager quickly identified the problem during one of our weekly 1-on-1 meetings. He radically adjusted the course of my internship to fit my skills and personality. Soon, things were going much better. I was more productive, having more fun, and learning a lot more.

Great experiences didn't end with just my direct manager. Each intern also gets a mentor for additional job and team related support as well as a "coach" from a different part of the company to jump start your personal network. My mentor helped me understand the politics and personalities of our team and team members. My coach convinced me to tell my recruiter about the changes going on with my internship. My recruiter encouraged me to conduct informal interviews and have casual lunches with various people around the company. These "informationals" led to an awesome position for my second internship that was a perfect fit for me.

Internships at Microsoft are as much about growing people as they are about getting engineering done. There are a lot of people around to support you, so you receive a lot of support. If you are lucky -- like I was -- all of these people are exceptional. Even if you are unlucky, at least one of the your many network seeds may be helpful. Additionally, recruiters are more personally involved. Our recruiter took us out for dinner and drinks in small groups and got to know us quite well.

At Google, I once again felt as if my skills were being misappropriated, but enjoyed far less success in correcting it. For the majority of the internship, I only interacted with my direct manager. He lacked experience mentoring interns and was a little surprised when I raised concerns over the quality and pace of my own work. With no one else to turn to, I contacted my recruiter who was alarmed by my frustration. Although my manager and recruiter had the best of intentions, they choose to reassure me that I was doing good work rather than reposition me so that I could do outstanding work.

I am more vocal and assertive than many engineers, and especially interns, at both Microsoft and Google. Speaking up at Microsoft did wonders for my career, but I didn't have to say a word for people to ask "what's the matter?" At Google, I spoke up and I felt as though I was the first to ever do so -- people were surprised and unsure how to react.

It should be stressed that these were personal experiences. Some other fresh college graduate may have a completely symmetric experience.

Rift

2008-05-18T17:03:00.001-07:00

Rift is a top-down shooter with a time-stopping twist. It was written in XNA by three Drexel students in three days:

Brandon Bloom - Lead Programmer
Lee Baker - Programmer and Sound
Charbel El-Beyrouty - Artist

Watch the video:

(low quality, no sound, obvious recording artifacts -- I might have to buy Fraps)

Or download the game (with source):

http://brandonbloom.name/rift.zip

Hope you enjoy it!

A googol at Google

2008-05-12T01:04:00.000-07:00

Ok, well maybe not quite a Googol (1x10E^100), but certainly 2^63-1 (9,223,372,036,854,775,807). This quote was taken from a talk by Steve Yegge:

We overflowed a long at Google once. Nobody thought that was possible, but it actually happened.

Working at Google has irreversibly damaged my ability to reason about numbers. One moment you are talking about petabytes of data and the next moment you are talking about nanoseconds. It is truly mind numbing, but it has become hard not to laugh when someone talks about a huuuuge dataset of a few gigabytes or a fast process at a few seconds.

It is easier to pirate Windows than to install it legally

2008-05-10T14:26:00.001-07:00

My desktop machine is totally hosed. I don't know why or what is wrong, but it is really bad. So I want to re-install Vista, but I can't find the DVD or product key.

I currently own several Windows Vista licenses:

Gift for being a Vista beta tester
Drexel CS Department
From interning at Microsoft for a combined 9 months (and full-time this July)
Microsoft Dreamspark
Probably more

I expected to use my beta tester Ultimate license via logging into connect.microsoft.com, downloading the DVD, and getting the product key from Connect as well. Unfortunately, I can't find the product key or download anywhere on the new Connect. Additionally, I'm blocked from all MSDN subscriber downloads because, well, I'm not a subscriber. Strangely enough, I vividly remember being able to download Vista from MSDN at one point...

I'm looking high and low for a spot to download Windows Vista legally. It is laughably easy to download it illegally, and the download speed will be better than Microsoft's servers anyway.

The next version of Windows should be freely available for download and the installer should ask for my Windows Live ID, not some cryptic 5x5 product key.