Thoughts on Systems

SCNA 2012 summary

2012-11-23T00:36:00-05:00

Software Craftsmanship North America is an annual conference bringing together software craftsmen—developers who are interested in improving their own ability to program. In his opening remarks at SCNA 2012, 8th Light’s co-founder, Micah Martin described the conference as the “yearly attitude adjustment” for software craftsmen.

The speakers covered topics ranging from professional and product development, to engineering practices like testing and architecture, to theoretical CS concepts like monoids and logic programming. I have a complete-ish set of notes on my Flickr but here are some highlights.

Cory Foy talked about a model for teaching programmers (slides) that starts with work that has low context and low cognitive demand (such as katas, koans) and brings them up to doing work with high context, high cognitive demand (such as adding features and listening to code). This mirrors closely the 8th Light apprenticeship model. He also talked about how we need to learn to listen to the code and not try to force it to do things that it is not suited for; to listen requires understanding, to understand requires practice, and to practice requires context.

There were several discussions about apprenticeship. My sense is that 3 months is enough time to train people in basic craftsmanship suitable for basic web development (the equivalent of a semester, so maybe 4 courses worth). It obviously isn’t the ten thousand hours necessary to produce a master. The successes described also suggests that apprenticeship is not necessarily good at producing developers that can be hired for other companies. Of the 20 or so apprentices trained by apprentice.io (a program at Thoughtbot to try to commercialize apprenticeships), only one has been actually placed in an external company despite over a hundred companies interested in hiring out of the apprentice pool. On the other hand, they’ve hired about eight themselves. 8th Light has similarly grown much of its current 20+ craftsmen through its internal apprenticeship program.

8th Light has shared their internal syllabus for training craftsmen. Thoughtbot, the team behind apprentice.io, has also produced a set of basic trailmaps for learning basic techniques that the community can contribute to on GitHub.

I’m curious about adopting a more formal apprenticeship/mentoring program at places not primarily doing web app development, and in particular at systems-y companies like Hadapt (where time and money is limited) and VMware (where there is both more existing training, and where resources are less scarce). Certainly, some of the basic skills and culture do need to be acquired, but so does the knowledge necessary to build a distributed query execution engine or a shadow page table walker.

Uncle Bob’s talk (video/summary) spoke more broadly. He argued that we need to behave professionally because, one day, some software glitch will result in lots of deaths (think Therac-25) and the world will demand an answer from the tech industry. If we don’t want government regulation, we better behave professionally. As Uncle Bob put it, to be professional means that we do not ship shit. That we say no. That we work as a team. And that we learn continuously: Uncle Bob proposed upwards of 20 hours a week on our own.

There were many talks about aspects of testing. Michael Feathers gave a talk that sort of questioned some of ones assumptions about testing by focusing on the value delivered by tests. He talked about, for example, deleting tests—if they no longer provide value. Value of tests can come from many places: guiding the design of objects, detecting changes in behavior, acting as documentation, guiding acceptance criteria. The value of a test can change over time and we should not over-venerate any specific test. He argued that it is more appropriate to set a time budget for testing.

Gary Bernhardt gave a beautiful talk about mixing functional programming and object oriented programming. He noted that mocks and stubs cause tests to become isolated from reality but that purely functional code does not require mocking: it always behaves the same way given the same inputs. Thus, he argued that code should be structured to have a functional core surrounded by a more imperative/OO shell that sequences the results with actions, a style he called “Faux-O”. By focusing on providing values (functional results), we free the computation from the execution model (for example, how Java Callable’s can be plugged into a variety of ExecutorServices).

Justin Searls took a different tack to testing, bridging Michael and Gary’s talks in a sense. His big picture observation is that different kinds of testing deliver different amounts of reality and we should choose tests that give us the amount of reality we need. (He has a nice taxonomy of tests on his blog.) One takeaway from his talk is that we should adopt a standard for what kind of testing we do and stick to it: he liked the GOOS style of using isolation tests to guide design and more end-to-end acceptance tests to prove functionality, but listed a few others.

Drilling down into more specific tools/techniques, Brian Marick gave a talk about generating data for tests using logic programming, using an example in Clojure. His goal was to ensure that he only says as much about the data used for a test as is absolutely necessary for the test and to allow other aspects of that data to vary; this can be achieved by writing a logic program to state the test’s requirements and allowing the runtime to solve for the right data. In fact, you could imagine automatically testing all valid values that the logic program generated, instead of just one (much like Guava’s Collections test suite builder does more imperatively). We have explored this idea for system-level testing at both VMware and Hadapt, where it would be useful for tests to declare their dependencies on the system (e.g., requires a system configured in a particular way) and have the test framework automatically satisfy those dependencies in some way that the test does not care about. Logic programming would provide a way to bind the resulting dependencies to variables that could be used by the test.

Susan Potter gave a talk about monoids at a very theoretical level, but they have a practical impact on code expressiveness. A nice way to understand monoids is to see how monoids apply to FizzBuzz. At a more systems level, monoids are used by Twitter in their services stack to compose asynchronous results. As we develop tools at Hadapt for provisioning systems or manipulating internal plan trees, I expect to apply monoids to help ensure composable abstractions.

The last talk of the conference was by Leon Gersing and was a great motivational talk about personal development. You should watch it.

The talks were only half the time at SCNA. Networking with other developers made up the rest, as well as being intermixed with fun activities like kata battles (wherein two developers race to complete a basic coding kata live on screen in front of the audience) and Jeopardy. There was also a re-factoring kata fishbowl where I narrowly missed an opportunity to pair with Uncle Bob. While I got a lot of value from the talks, I wished there had been more time for pairing and working on code with the other developers there. On the last day, I got a tutorial from Randy Coulman, who has been programming in SmallTalk for 10 years, as he did the coin changer kata in SmallTalk. More explicit time for that sort of impromptu practice (not just chatting about work) would have made the conference even better.

Overall, SCNA was a great conference and I hope to be able to spend more time with software craftsmen in the future.

Growing a Software Craftsman Engineering Organization

2012-11-09T08:23:00-05:00

One of the hallmarks of a software craftsman is the desire to improve and hone one’s abilities. Certainly, this is one of the reasons that I am attending Software Craftsmanship North America (SCNA) this year. As a leader in an engineering organization, however, I am also curious about how to grow an engineering organization that is focused on not only delivering value, but doing so in a way that values well-crafted software.

The population of people who are already craftsmen (outside of conferences such as this) is somewhat limited, so hiring solely craftsmen is not likely to be scalable. At the SCNA mixer last night, I heard two basic approaches to developing a team of craftsmen.

8th Light uses an apprenticeship model. 8th Light hires people in as an apprentice: there is a clear understanding that an apprentice is learning about the craft and how they work at 8th Light. There is a good ratio of craftsmen to apprentices and everyone is invested in teaching and learning. During the apprentice period, the apprentice may be unpaid or paid at below-market rates as they finish training/learning. (I’m sure this is done in a fair way and everyone gets value from the arrangement.) What was surprising to me was that not only do they hire in experienced developers (who have self-selected as being interested in improving/learning), but they hire people with aptitude but relatively little programming experience. Over the course of a year, these true apprentices grow into journeymen and craftsmen. It appears one successful model is to budget time and money into training up your own pipeline of craftsmen.

A second approach I heard about was through injection of a leader/manager who drove craftsmanship into the organization. I spoke with people at a financial services company and at a publishing company; in both cases, about a year ago, someone was brought in who drove the engineers in the direction of craftsmanship. Today, those teams practice TDD/BDD, watch Clean Coders videos to learn, and attend conferences like SCNA.

I hope over the next few days, and through continuing conversations afterwards, to get more insight into organizations that successfully balance the need for delivery with training its team to deliver high quality code, and what principles and tactics they use to transition to a high productivity state.

If you have any thoughts, please share them!

Developing Cloudera Applications with Gradle and Eclipse

2012-09-02T00:08:00-04:00

This post is a translation/knock-off of Cloudera’s post on developing CDH applications with Maven and Eclipse for Gradle. It should help you get started using Gradle with Cloudera’s Hadoop. Hadapt makes significant use of Gradle for exactly this purpose.

Gradle is a build automation tool that can be used for Java projects. Since nearly all the Apache Hadoop ecosystem is written in Java, Gradle is a great tool for managing projects that build on top of the Hadoop APIs. In this post, we’ll configure a basic Gradle project that will be able to build applications against CDH (Cloudera’s Distribution including Apache Hadoop) binaries.

Gradle projects are defined using a file called build.gradle, which describes things like the projects dependencies on other modules, the build order, and any other plugins that the project uses. The complete build.gradle described below, which can be used with CDH, is available as a gist. Gradle’s build files are short and simple, combining the power of Apache Maven’s configuration by convention with the ability to customize that convention easily (and in enterprise friendly ways).

The most basic Java project can be compiled with a simple build.gradle that contains the one line:

apply plugin: "java"

While optional, it is helpful to start off your build.gradle declaring project metadata as well:

// Set up group and version info for the artifact
group = "com.mycompany.hadoopproject"
version = "1.0"

Since we want to use this project for Hadoop development, we need to add some dependencies on the Hadoop libraries. Gradle resolves dependencies by downloading jar files from remote repositories. This must be configured, so we add both the Maven Central Repository (that contains useful things like JUnit) and the CDH repository. This is done in the build.gradle like this:

repositories {
    // Standard Maven 
    mavenCentral()
    maven {
        url "https://repository.cloudera.com/artifactory/cloudera-repos/"
    }
}

The second repository enables us to add a Hadoop dependency in the dependencies section. The first repository enables us to add a JUnit dependency.

dependencies {
    compile "org.apache.hadoop:hadoop-client:2.0.0-mr1-cdh4.0.1"
    testCompile "junit:junit:4.8.2"
}

A project with the above dependency would compile against the CDH4 MapReduce v1 library. Cloudera provides a list of Maven artifacts included in CDH4 for finding HBase and other components.

Since Hadoop requires at least Java 1.6, we should also specify the compiler version for Gradle:

// Java version selection
sourceCompatibility = 1.6
targetCompatibility = 1.6

This gets us to a point where we’ve got a fully functional project, and we can build a jar by running gradle build.

In practice, it’s good to declare the version string as a property, since there is a high likelihood of dependencies on more than one artifact with the same version.

ext.hadoopVersion = "2.0.0-mr1-cdh4.0.1"
dependencies {
    compile "org.apache.hadoop:hadoop-client:${hadoopVersion}"
    testCompile "junit:junit:4.8.2"
}

Now, whenever we want to upgrade our code to a new CDH version, we only need to change the version string in one place.

Note that the configuration here produces a jar that does not contain the project dependencies within it. This is fine, so long as we only require Hadoop dependencies, since the Hadoop daemons will include all the Hadoop libraries in their own classpaths. If the Hadoop dependencies are not sufficient, it will be necessary to package the other dependencies into the jar. We can configure Gradle to package a jar with dependencies included by adding the following block:

// Emulate Maven shade plugin with a fat jar.
// http://docs.codehaus.org/display/GRADLE/Cookbook#Cookbook-Creatingafatjar
jar {
    from configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
}

Unfortunately, the jar now contains all the Hadoop libraries, which would conflict with the Hadoop daemons’ classpaths. We can indicate to Gradle that certain dependencies need to be downloaded for compilation, but will be provided to the application at runtime by augmenting the Hadoop dependencies. The code then looks like this, with an added dependency on Guava:

// Provided configuration as suggested in GRADLE-784
configuration {
    provided
}
sourceSets {
    main {
        compileClasspath += configurations.provided
    }
}

ext.hadoopVersion = "2.0.0-mr1-cdh4.0.1"
dependencies {
    provided "org.apache.hadoop:hadoop-client:${hadoopVersion}"

    compile "com.google.guava:guava:11.0.2"

    testCompile "junit:junit:4.8.2"
}

Gradle also has integration with a number of IDEs, such as Eclipse and IntelliJ IDEA. The default integrations can be provided by adding

apply plugin: "eclipse"
apply plugin: "idea"

to add support for generating Eclipse .classpath and .project files and IntelliJ .iml files. The default build output locations may not be desirable, so we configure Eclipse as follows:

eclipse {
    // Ensure Eclipse build output appears in build directory
    classpath {
        defaultOutputDir = file("${buildDir}/eclipse-classes")
    }
}

For Eclipse, simply run gradle eclipse and then import the project into Eclipse. As you update/add dependencies, re-run gradle eclipse to update the .classpath file and refresh in Eclipse. Gradle automatically handles generating a classpath, including linking to source jars.

Recent versions of IntelliJ and the SpringSource Tool Suite also support direct import of Gradle projects. When using this integration, the apply plugin lines are not necessary.

Gradle represents a well-documented and powerful alternative to developing projects in Maven. While not without its quirks, I am significantly happier maintaining an enterprise build in Gradle at Hadapt, compared to the complex Maven build I maintained at VMware. Give it a try.

Let's improve our code

2012-01-01T23:00:00-05:00

New Year’s is a good time to set intentions for the coming year. Many people come off the holidays with the intention to exercise more, but if you’re reading this blog, you’re probably a programmer (if you’re not, consider signing up for Code Year…), so let’s set an intention about our programming. But first, a musical interlude.

Earl Hines was a jazz pianist; in this 9 minute video, he describes how his early playing evolved.

As you watch it, notice how he not only describes and demonstrates how his style evolved, he also describes why. For example, he talks about how his melodic line was drowned out in the larger bands so he picks up playing in octaves (doubling up the notes).

In his TED talk, David Byrne generalizes the idea of environment influencing music by talking about how music has always evolved to fit the architecture in which it was performed: from how the ethereal sounds of early church music were driven by the open acoustics of churches to how the smaller rooms of the 18th and 19th centuries allowed for the more complex rhythms and patterns of classical music to be heard. (Watch it here.)

Can we as programmers reflect similarly about our programming styles? What influences the way our programs look? And more importantly, perhaps, why should we care?

For music, Byrne argues that the evolution of styles was driven by the needs of the audience and the acoustics of the performance hall. Understanding these consciously allows contemporary musicians to make more informed choices about what and how they perform.

As programmers, our programs must communicate: with the compiler, of course, so that it will render our code executable, but also with the human readers of our code, be that our future selves or our colleagues. So to write better programs—programs that communicate their intent more concisely and clearly, as opposed to those that execute more efficiently or that are more clever—we should consider what affects the structure and readability of the programs we write.

The frameworks and mechanisms available to us most obviously affect the structure of code. Write a program in a system based on callbacks, such as the async XML HTTP request that underlies AJAX, and you will find yourself with code that chains callbacks together, preserves state in various heap objects, and is requires that callbacks be called from the right contexts to work properly. Write code for a threaded system and your code will have all manner of locks and constructs to control memory write visibility. Regular expressions can be called from Perl with the overhead of only m// so it is easier to write text munging code in Perl than almost any other language.

Our methodologies, tools, and processes—how we program—also determine how our code looks. Test-driven development will tend to produce stronger and more usable abstractions. Stream of consciousness programming results in a mess. Using an editor that supports refactoring patterns will make it more likely that you will refactor. Code review or pair programming will similarly result in code improvements, simply because you had to communicate while writing the code. (Even just commenting your code helps in this regard.) The end result of these practices is code that is more understandable.

Our audience (that is, our teammates) also affects our code. This is the role of engineering culture. What will your teammates accept versus some ideal? To get code committed to the Linux kernel requires detailed commit messages, a well structured patch series and surviving code review on the kernel mailing list. To get code committed to your personal project requires nothing outside of what you ask of yourself.

We have control over these factors. We can vary our tools, our practices, our choice of frameworks, and influence our team culture. If we are framework or API developers, we can consciously evaluate what code we induce our users to write and improve on what we provide to simplify their lives, and facilitate their communication and self-expression.

This year, let’s set an intention to examine our code and improve how it reads. Let’s experiment and play with the factors under our control to see which choices work better for our teams. Ask your teammates whether one way or another works better for them. Spend some time analyzing your own code and consider how it got that way.

I’ll try to share some of what I learn from my team at Hadapt and I’m curious to hear what you learn from yours.

Git is more usable than Mercurial

2011-12-05T22:49:00-05:00

Once upon a time, I used Mercurial for development. When I moved to VMware, people there seemed to favor Git and so I spent the past few years learning Git and helping to evangelize its use within VMware. I have written about why I chose Mercurial, as well as my initial reactions upon starting to use Git. Hadapt happens to be using Mercurial today and so I have been re-visiting Git and Mercurial.

What I wrote about Git and Mercurial in 2008 is still true: Git and Mercurial are similar in may respects—for example, you can represent the same commit graph structure in both—and they are both certainly better than Subversion and CVS. However, there are a lot of differences to appreciate in terms of user experience that I am now in a better position to evaluate.

In using Mercurial, I find myself oddly hobbled in my ability to do things. At first, I thought that this might simply be because some things are simply done differently in Mercurial but at this point, I actually think that Git’s design and attention to detail result in it actually being more usable than Mercurial.

There are three “philosophical” distinctions that are in Git’s favor:

Git has one branching model. Mercurial has several that have evolved over time; Steve Losh has a comprehensive essay describing ways to branch in Mercurial. The effect of this is that different Mercurial users branch in different ways and the different styles don’t really mix well in one repo. Git users, once they learn how branching works, are unlikely to be confused by branches.
Git has names (refs) that don’t change unexpectedly. Every Git commit you care about has a name that you can choose. Some Mercurial commits that you might care about do not have a name. For example, the default branch in Mercurial can have multiple heads, so it interprets -r default as the tip-most commit. Unfortunately, that commit will vary depending on who has committed what to which head (and when you see it).

Further, Git exposes relative naming by allowing you to refer to the branches in remote repositories by name, without affecting your own names.

Putting this together, consider what happens after you pull in Mercurial. Your last commit used to be called default but after the pull, default is something from the upstream. Your commit is a separate head that now has no name. In Git, your master doesn’t move after a fetch and the remote’s branch is called origin/master.

Git even tracks the changes what commit each name refers to in a reflog. You can easily refer to things that the name used to refer to. In Mercurial, branch names don’t have reliable meanings, and it doesn’t track them.
Git commands operate in a local context by default. Mercurial commands often operate on a repository context. For example, git grep operates on the current sub-directory of your work tree, hg grep operates on your history. The Git analog of hg grep is using the log pick-axe; the Mercurial analog of git grep is to use ack, or if you must, something like hg grep -r reverse(::.) pattern . (Seriously?)

Another example is the log command. Git’s log command shows you the history of the commit you are on right now. Mercurial’s log command shows you something about the whole repository unless you restrict with some combination of -b and -f. Combined with Mercurial’s way of resolving branch names to commits, it becomes very difficult to use hg log to compare two heads or explore what has changed in another head of the same branch.

More often than not, I care about things in their current tree more than how things are in some random other branch that I am not working on and Mercurial makes it hard to do that.

There are other usability issues that I’ve found that are more detail-oriented than philosophical. I’ll note a few here.

hg log doesn’t display the full text of the commit message unless you hg log --debug. This is an unfortunate disincentive to writing good commit messages.

hg log -p doesn’t pay as much attention to merge commits as Git does; the help for hg log reads:

log -p/–patch may generate unexpected diff output for merge changesets, as it will only compare the merge changeset against its first parent. Also, only files different from BOTH parents will appear in files:.

whereas git log has a variety of options to control how the merge diff is displayed, including showing diffs to both parents, removing “uninteresting” changes that did not conflict, or showing the full merge against either just the first or all parents of the merge commit.

Both Mercurial and Git have lots of configurable options; Git has a thin veneer over editing a config file in the form of the git config sub-command. Mercurial involves editing a file even if just setting up your initial username or enabling extensions. I often wound up editing Git config files directly, but having the commands were nice for sharing instructions with others.

Git support for working with patches natively is better. Mercurial supports e-mailing and applying patches, but oddly the extension for sending out patches is built in (patchbomb) but the extension for importing from an mbox (mbox) is not. There’s no direct analog of git apply; instead you have to use a patch queue. Patch queues are okay, but branches and well-integrated rebase/e-mail/apply support are much nicer than patch queues: you don’t need to manual find some .hg/patches/series file and edit it to re-order stuff.

I could write more and indeed many people have written about Git and Mercurial—you can explore my bookmarks about git for some of the better ones. Let me close here with three interesting features in Mercurial 2.0:

the new largefiles extension allows users to not transfer large files down until they are needed;
subrepos can be Git or Subversion in addition to Mercurial;
revsets allow you to search your history in very flexible ways.

Overall, I feel that Git is significantly more usable for day-to-day development than Mercurial. I’d be curious to hear if you think the opposite is true.

A new adventure

2011-11-06T21:10:54-05:00

Friday, 4 November, was my last day at VMware.

I started at VMware in 2008, working on a project that has now become VMware’s Horizon Mobile. Last year, I switched to working on the latest release of VMware’s vCloud Director.

VMware has a lot going for it as a place to work. For example, it has:

Executive leadership with a clear vision to make VMware a leading player in the cloud era;
Experienced and friendly engineers to work with: within 50 feet of my VMware office there were two (other) MIT PhDs, numerous people from IBM and Sun with deep Java experience, and a former Usenix tutorial instructor;
An R&D organization dedicated to internal innovation and making some of them public (with VMware Flings); not to mention,
Great perks (comparable to perks at other top companies).

It wasn’t an easy choice to leave.

Last month, I became aware that a startup in the big data space was moving to Boston. I’d been wondering about life outside VMware and this opportunity seemed just about perfect. So I’m beginning a new adventure at Hadapt. As an early employee, I imagine I’ll be doing a little bit of everything. I hope to combine the skills and knowledge I built up from my graduate work and the practical experience of delivering enterprise software at VMware to help Hadapt build a powerful, scalable, data analytics platform and make Hadapt a successful company.

I’m excited to get started and I hope to share here with you some of my experiences as I go.

Rules for Development Happiness

2011-10-20T09:20:49-04:00

Inspired by Alex Payne’s Rules for Computing Happiness, some rules for having happy developers and being happy as a developer.

Use version control. (See The Joel Test.) In particular, use a distributed version control system (like Mercurial or Git). This ensures you can commit offline and also conduct code archaeology offline.
Have a correct and fast incremental build (e.g., non-recursive Make or Gradle) to avoid this.
Have a system for testing your changes in a safe environment prior to code submission.
Avoid dependencies on system tools. Different developers tend to have different systems and hence different versions of tools.
Be able to work offline. Offline may mean when you’re on a plane, but it may also happen when the office network goes down. Both happen. Notably, the latter happens even when you work on a desktop with a wired connection. (It’s been pointed out to me that the network going down can be a good team-building experience.)
- Be able to build offline. That means having all build dependencies cached locally.
- Have all e-mail cached locally. Don’t be unable to find those key instructions some mailed you just because GMail is restoring your mail from tape. Helpful tools here are isync or offlineimap. Index your mail with mu. (Or configure Thunderbird/Apple Mail/etc to keep everything offline.)
- Be able to send mail offline; e.g., have it queue locally for deliver when the network comes back. But, make sure you keep a copy locally in case the hotel’s WiFi is transparently re-directing out-bound SMTP connections to /dev/null. (This really happened to me.)
- Have other documentation cached locally. (Use something like gollum for your wiki.)
- If you work somewhere with a shared-storage home directory, make sure you can login when the network is down!

Store Hudson configuration in Git

2011-01-14T15:13:40-05:00

For any kind of server, it’s a good idea to keep its configuration in some sort of version control system. Hudson is a pluggable continuous integration system. Recently, I was trying to set one up and was wondering the best way to store Hudson’s configuration in version control (StackOverflow summary). The most complete answer is a post on the Hudson blog about how to keep Hudson’s configuration in Subversion; there are also plugins like a nascent SCM Sync configuration plugin. But, the former is very Subversion specific and the latter does not seem particularly mature. So, to understand how to do it in your workflow, there are two things to consider.

First, which files are relevant? Hudson puts configuration, run-time state, source code and build output all in the same sub-directory (called HUDSON_HOME). Second, relatedly, since normally you edit Hudson’s configuration through the GUI, when should you commit changes? Should it be automated (e.g., nightly at midnight) or manual (e.g., ssh into the server and manually commit)? I’ll answer those questions with an implementation in Git but you can translate the information easily to your preferred VCS.

Identify relevant files by using the following .gitignore file:

This ignores the uninteresting files and will allow git status to show you interesting new files. Note that I prefer to actually commit the binaries of plugins since I don’t want to rely on outside sources (namely, the mirror network) having the particular version of the plugin that I was using for the given configuration files. To use this if you are installing a new Hudson server, you can just

cd $HUDSON_HOME/.. # Default is /var/lib
rm -r hudson
git clone git://gist.github.com/780105.git hudson
# Don't forget to chown hudson hudson as appropriate for your environment

before starting Hudson for the first time. Then once it has started, run git commit to track the default config that Hudson creates.

The second question is when. The Hudson blog’s recommendation is to create a Hudson job that runs nightly at midnight to check for differences and automatically commit them. I prefer manually committing the changes on the server and then pushing it. This allows me to identify specific functional changes (using git add -p) and commit them individually. If you want to do it automatically, simply write a script or add a job that will

git commit -a -m "Automated commit of Hudson configuration"
git push

once you set up an appropriate origin.

Once you have this set up, you can even use something like Chef to automatically pull down updated configuration that you manage and test elsewhere and restart the Hudson server when necessary. Then you can re-create your Hudson server in case of failure at any time!

Programming without fear

2010-11-16T01:06:39-05:00

This past weekend, I attended Gil Broza’s seminar on Programming Without Fear, organized by the Greater Boston Chapter of the ACM’s Journeyman Programmer initiative. The seminar was as advertised, and covered:

Code smells;
Test-driven development, with a focus on micro-testing (aka unit-testing);
Common refactoring patterns and techniques for refactoring using a unit-test base for safety;
Testing in isolation with dependency injection with stubs and mocks.

For anyone with more than a passing interest in agile, the material Gil presented (covered in the links above) will not be new.

The benefits of the seminar came from two things. First, Gil presented the information in a somewhat “formal” framework: a taxonomy of code smells, a set of refactoring patterns, a pair of mnemonics (PRICELESS unit tests and TRUST your refactoring process) to help remember basic techniques. This gives someone new to the material an organized set of knowledge to internalize. Second, Gil has prepared a series of exercises, interspersed with the lecture-y sections, that seminar participants work through in pairs, designed to reinforce the theoretical frameworks with practical experience. Even as someone moderately experienced with these concepts, the exercises are useful in that they focus on the fundamentals and force you to actively strengthen those fundamentals. (The weakest section, I thought, was the one on mocking which received insufficient exposition and dropped the class directly into jMock, which was a bit opaque.)

Gil is not the most exciting or funny teacher but he kept the attendees engaged by teaching with a Socratic flavor—he presented examples and solicited audience evaluations, allowing the audience to interact to reach conclusions. The practical exercises were followed by group de-briefs. This encouraged the audience to stay engaged and better absorb the material.

My main worry about the techniques is the overall reliance on Eclipse (or other IDE) as a developer’s assistant: while certainly the tooling is convenient, they make me worry about Java and whether the use of tools and wizards weaken developers who may never learn how to do things themselves.

What I really enjoyed was the experience of actually developing and refactoring with the protection of a unit test suite and learning techniques to perform refactoring without more than a moment or two of compiler errors. This was in sharp contrast to my normal refactoring experience of making a top-level change and then following all the compiler warnings until the work is done. Now if only every codebase I worked on came with such a set of tests…

How to install ThinkUp on NearlyFreeSpeech

2010-09-03T23:50:21-04:00

Gina Trapani and ExpertLabs have put ThinkUp, a cool tool for tracking replies to your posts on Twitter. As of September 2010, ThinkUp has a nifty drop-in web-based installer, much like WordPress. Simply grab ThinkUp 0.007 or later, unzip it somewhere that your PHP/MySQL-enabled web server can get at and it’ll prompt you through the installation.

In response to Gina’s post asking for help testing/hacking this long-weekend, I ran through this in about 10 minutes on NearlyFreeSpeech. Here are some quick tips where NFSN’s set-up is a bit different than what is expected by the default installer:

After you unzip the ThinkUp dist, run chgrp -R web _lib/view/compiled_view and chmod -R g+w _lib/view/compiled_view so that the templating engine can cache its views.
Make sure you have a MySQL process enabled in your NearlyFreeSpeech control panel. A basic MySQL process costs $0.02/day but you can share the process with your WordPress database. Spin up phpMyAdmin in the right-hand sidebar and create a user called thinkup and make sure to check off the option to create a database with the same name and grant all rights to that user. Generate a random password, and copy that password.
In the ThinkUp database configuration section, enter thinkup as the user name and database name and paste your generated password. Open the advanced section and change the database host from localhost to your database host name. It’ll be something like username.db; mine, for example, is sit.db.
ThinkUp will fail to write the configuration file due to perms but helpfully offers the ability to copy and paste a file. Select the text in the config text box and go back to the terminal where you unzipped the dist. In the thinkup directory, cat > config.inc.php, paste and then Ctrl-D to save the file.

Check your e-mail for the activation link and configure your account. You’ll need to register your installation as a Twitter application and paste in the consumer key and consumer secret. The config page will send you to the Twitter registration page and tell you the callback URL to provide. Leave ThinkUp as a read-only application and leave the ‘Use Twitter for login’ unchecked.

That should do it!

For more details, check out the ThinkUp wiki for more up-to-date instructions.

Understanding SpringSource and the Spring Framework

2010-06-01T22:35:44-04:00

In light of recent announcements like vmForce or the SpringSource/Google App Engine integration, you may be wondering, what the Spring Framework is, precisely. What does the SpringSource company provide?

According to their homepage, SpringSource is in the business of “eliminating enterprise Java complexity” and is a leader in Java application infrastructure and management. That’s not very concrete, and so I don’t feel it is particularly helpful, particularly if you are not an J2EE/JEE (Java Enterprise Edition) developer. In this post, I’ll talk about SpringSource in general and focus on the Spring Framework. Note that while I work for VMware (which owns SpringSource) and use the Spring Framework (commonly referred to as Spring) at work, I am not part of our SpringSource division nor do I have any particularly special access to the innards of SpringSource. I did get to take the Core Spring training for free, but it is only after 5 months of programming with Spring that I’ve started to understand the SpringSource philosophy.

SpringSource products let you write code that is as focused as possible on the needs of your application, and as little as possible on the boilerplate or hassle of dealing with different underlying environments (e.g., dev, test, production may have different database backends) or infrastructures (e.g., GAE, vCloud). This is the core value that underlies SpringSource, but it is only explored indirectly via its various instantiations in the SpringSource literature and product line.

The Spring Framework (aka Spring) provides glue. Spring provides glue in a relatively uniform manner, so that once you understand the basic approach(es), you can apply it to interfacing with different components. From the documentation, Spring seems to do everything, but at the same time, when you try to use it, you may feel that it seems to do almost nothing. It may be useful to compare the Spring Framework to the Debian Linux distribution: Debian provides a nice out-of-the-box experience with a uniform mechanism for managing software, and in particular, alternative software packages that can provide a common service. But to get at the power of the underlying packages, you must learn how to configure and use them. Likewise, Spring does not actually provide many services on its own. It does not free you from having to learn how to write a unit test, access a database, manage a messaging system, or implement security. Instead, it makes it possible for you to write code to do these things in a somewhat generic manner, so that your code can be as generic as possible.

Understanding these two key points will help you make sense of the variety of things written about Spring.

The core glue provided by Spring is its dependency injection support, also known as the “inversion of control” or IoC container. This means that, instead of class Foo explicitly instantiating an object implementing interface Bar, Foo will have a constructor argument or setter that accepts a Bar. The inversion of control container lets you specify the right kind of Bar for Foo in a given environment and handles constructing that Bar, and injecting it by calling the setter. This makes code less fragile because it no longer needs special-casing for testing (e.g., a stub Bar) or anything else. The mapping of a particular Bar to Foo becomes part of a configuration file that also captures all of Foo’s other dependencies.

Spring also provides more special-purpose glue; the Spring Framework page writes:

Spring provides the ultimate programming model for modern enterprise Java applications by insulating business objects from the complexities of platform services for application component management, Web services, transactions, security, remoting, messaging, data access, aspect-oriented programming and more.

For example, your application can use the Spring Framework with straight JDBC, or with generic object-relational mappers (ORMs) like JPA through to highly specific ones like iBatis or Hibernate. You can configure your application to then talk to a variety of database back-ends, with minimal changes in the actual configuration files, and write minimal code related to setting up database connections and processing error cases. Spring provides wrappers and translators to help unify service-specific method names, such as the one that causes an ORM system to generate database tables, and exceptions into more generic expressions of those concepts. This means you might be able to switch between JPA providers, for example, without changing too much configuration. However, you still have to configure your JPA provider correctly.

In line with allowing you flexibility from the infrastructure, Spring also provides flexibility of mechanism, so that the code and configuration that you write to integrate with Spring’s services are at a level that you are comfortable with. You can configure the IoC with XML or with Java; you can use annotations or you can use explicit configuration. To specify which of your business methods should be in a database transaction, you can annotate with @Transactional in your source code, or you can use an aspect-oriented programming filter to tag the relevant methods in an external configuration file.

All of this glue and the flexibility of mechanism contribute to making Spring hard to understand; however, their presence emphasizes the idea that Spring wants to get out of your way so that you can focus on application development. Other SpringSource tools such as Roo and Insight work similarly: they simplify development and debugging (respectively) without requiring that you make extensive changes to your source, and respecting current best-practices.

I’ve left out various components of Spring to focus on the core philosophy of SpringSource, but this background should help you make sense of resources like the Wikipedia article on Spring, the Spring Framework documentation, and books like Spring in Action. If you’re running into problems, however, the best place to get concrete questions about Spring answered is Stack Overflow.

Examining your personal programming style

2010-05-02T23:23:29-04:00

When I was growing up, we would listen to classical music stations in the car and try to figure out the composer and sometimes even the performer. Both musicians and composers often have their own distinctive style: you can hear the mathematical precision of Gould, or the clarity of Horowitz, whether they are interpreting Bach or Mozart. My last post started me thinking about a musician or composer’s style and drawing a parallel in the context of computer programming.

When thinking about music, one’s style is a matter of personal expression, but if you say “coding style” to a programmer (or really, to Google), you’ll find rules about whitespace, variable naming, plus some proverbs about how to write maintainable code (e.g., “avoid global variables”). Overall, I don’t think these are particularly relevant to the art of programming.

For example, formatting and naming conventions are important in a codebase only that a properly followed convention becomes invisible—just like your nose becomes acclimated to a smell, your brain quickly learns to recognize a formatting convention and ignore it. Having a convention (any convention!) allows you to focus on what the source code is really doing. Following a convention is good for everyone reading your code, even you. Automate your coding conventions and forget about it. (In the extreme, check out what the Go Language formatter can do.)

Similarly, coding style proverbs, like “write tests before code” or “keep code in a function at one level of abstraction”, are like any other proverb: these statements capture an element of experience from programmers past, but are often blindly followed by people new to the practice. It takes significant time before a programmer can truly internalize the reasons for and nuance behind any proverb. (Incidentally, if you are interested in studying proverbs, I highly recommend you examine the game of Go.)

What I am interested in exploring is personal expression and style in programming, outside of language/library/tools, proverbs or code formatting. Having a personal style is not a concept that we as computer programmers are generally exposed to. School focuses almost exclusively on the technical, ignoring both the practice (i.e., the stuff of proverbs) and the art (the subject of this post). Indeed, I am only beginning to be able to express what my personal style might be.

So, how do you express yourself in code? To begin exploring our artistic programming style, let’s continue to draw an analogy from the arts—instead of music, let’s look at the process of establishing a personal photographic style. The author talks about the importance of the choices—the choice of equipment (camera), subject matter, the approach to making a picture. You come upon a way of doing things that you believe is right, that supports your personal values. When looking at myself:

Equipment: Linux/Vim/Mutt/Xmonad/Git.
Subject matter: I care strongly about the process with which you build programs, and so I wind up working a lot on tools and scripting, but I’m also interested in distributed systems problems.
Approach: I like (re)using what is present; I strive for consistency, simplicity, elegance. I am somewhat inclined towards using functional constructs in imperative languages (though I never did like OCaml’s syntax). I always look at a diff of my code to ensure it is minimal before committing it, and I like to write verbose log messages.
Examples: Some things I’ve worked on that you can look at include of course Chord, and some contributions to Gina Trapani’s todo.txt tool.

I’m not entirely happy with this “approach” because it feels like mostly a list of platitudes. But some of my difficulty, I think is in not having thought about this specifically as I look at other people’s code, not even having the words that I can use to compare and contrast my approach with that of others. So, I’d like this post be a start for each of us to explore our own style.

Spend a few minutes thinking about what you value in your own code, and how you define yourself as an artistic programmer and write about it in a comment, perhaps using the template I’ve set for myself above. I hope we’ll each learn something!

Music education versus computer science education

2010-04-24T23:41:58-04:00

My mother recently forwarded me this interview of the pianist Glenn Gould:

I encourage you to watch this (and all 6 parts), even if you know nothing about classical music.

What stood out to me in viewing this series of videos was the fluidity with which Gould is able to discuss a piece of music, in its historical context, and to simply jump in to play a phrase from a different piece to call out a point for discussion. This demonstrates an incredible mastery of the subject matter, unifying history and context, theory, and practical implementation.

From my experience at the Manhattan School of Music, musical training seeks precisely to bring this unification into its students. As you study, you learn to play a variety of pieces or styles, from different time period. You are taught some history, to be able to understand the evolution of styles; you are taught the underlying theory, to be able to discuss this evolution using precise terms; and then you practice the mechanics needed to actually play the pieces.

How does this compare to computer programming? Computer “science”, as you may know, has a fair amount of artistry to it.

We are simply not trained to have these kinds of discussions. How many people do you know who, during a code or design discussion, might say, “Oh, this is very similar to System X, in contrast to how System Y did things,” and then pull up the source code (or architecture diagram) for System X and Y and compare them with the relevant pieces under discussion?

At MIT, the classes are (were?) organized around ideas and then around technical implementation (leading, hopefully, to understanding). Little emphasis is placed on how to be a programmer, such as prototyping, testing, revision control, or code-review; that is, these things rarely factor significantly into your grade. Even less emphasis is given to the ability to discuss ideas in context, even at the graduate level. Undergraduate classes at most schools seem to focus on learning best practices for a particular programming language. As graduate students, only the extremely motivated would explore beyond the papers presented in the course syllabus, or the immediate related work for a given project; tracking down the source code of other systems is almost never done (perhaps simply because it is not often available).

In the professional world, no one teaches you how to do code or design reviews at this level either. Just like in school, professional programmers are constantly subject to deadlines which override just about any extra-curricular work. Reviews are often focused on mechanics or on vague, unsubstantiated worries. Again, extreme personal motivation is required to move beyond this.

How can we improve this situation? Is there room at the undergraduate level for more capstone projects that unify the theory, the history, and the mechanics with the craft of programming? What about at the graduate level? How about in a professional environment? What has been your experience?

I’d like to find programmers that work this way, that are excited and passionate about the craft of programming. Are you one? Get in touch.

Repeated and Reproducible Systems Research

2010-04-20T14:52:36-04:00

Daniel Lemire (@lemire) recently posted on the Mythical Reproducibility of Science, noting that sharing code also makes it easier to spread your ideas:

The reproducibility that matters is getting people to use your ideas. Merely proving you are honest falls short of your potential!

He’s also written on this in the past with statistics on downloads of his software.

Here are some more writings on reproducibility:

The case for repeated research in operating systems from ACM SIGOPS 2004.
The Many Faces of Systems Research—And How to Evaluate Them from HotOS 2005.
Tools for repeatable research, that I wrote four (!) years ago, has some pointers.

At PDOS, Frans Kaashoek and Robert Morris definitely encourage us to build real systems and make the system available. I definitely like this approach and never found that publishing the full source to Chord, including our work-in-progress/submission, caused any problems. It has also meant that a lot of people still play with Chord, even if I no longer actively maintain it.

Interview non-questions

2010-04-18T23:42:24-04:00

Once you get a job at a company, you move from one side of the interview table to the other. My ideal candidate for just about any engineering position:

has the ability to present technical ideas on the fly;
has practical Unix knowledge;
can write clearly and concisely in English and in code;
has a strong technical background.

Knowledge of particular technologies or programming languages is generally not interesting. Rather, the candidate should be smart and passionate.

One way I’ve started looking for passion is to see if the candidate has been involved in any volunteer work or open source projects. But it can be hard to assess the other qualities, even in an hour long interview. Typically, an interview assesses your ability to solve a particular problem, possibly in code, but not much about how it would be actually work with you.

As we hire some more people for MVP, I’m considering changing up our standard “bring in candidate for a series of 45m one on ones” to include some ways to probe for my desired qualities before the interview. I’d like to have candidates perhaps:

Send in a dot-file of some sort (e.g., .bashrc, .vimrc, .emacs, httpd.conf, etc.) That is: does the candidate use Unix and customize it? Does the candidate comment dot-files?
Prepare and deliver for the interview panel, a 5m presentation on some (any!) technical topic. Ensures the ability to communicate ideas clearly and answer questions.
Provide some samples of bug reports the candidate has filed or technical discussions that the candidate has had on a mailing list. (Say, 3 from the past 3 years, ideally from an open source project.) Alternately, provide a pointer to the candidate’s blog. That is, can this person write cogently? “Excellent communication skills” anyone?
Provide a code sample, something the candidate has had primary responsibility for developing, on the order of 100–500 lines of code. Two interviewers will review the code with the candidate.
Provide a commit, i.e., a diff to existing code (perhaps the code sample provided) and a commit message. This would demonstrate the ability to provide a clean functional change and document it for the team.

Most companies I interviewed with (at the PhD level) required a presentation, but only one asked for code samples. I’ve not seen any requests personally for anything else. Have you?

Incidentally, the Mobile Virtualization Platform team at VMware is hiring (mostly for our Cambridge office). Get in touch if you’re interested.

Getting started with Git

2010-03-18T11:12:11-04:00

Because of our work with the Linux kernel and with Android, we have started using Git more extensively at work, and my colleagues often have questions about how to get things done with Git. While the every-day command lists are helpful, most of the time, people would benefit more from getting a fundamental understanding of how git works. Here is a brief list of useful resources to help achieve that:

Why Git is Better than X, where X is one of hg, bzr, svn, perforce. Also, why git can be more confusing.
Git’s data model in a nutshell.
Scott Chacon’s ProGit book is a good tutorial and reference.
Scott Chacon’s Git screencasts, if you’re a visual learner.
Like presentations? Slides from an MIT SIPB Understanding Git class [PDF, 1MB], if you’re in a hurry, or Getting Git from Scott Chacon if you’re not.
Use Mark Lodato’s Visual Git Reference to help you understand how commands interact with history, the staging area and your working directory.

For me, the core difficulty is that people have to explicitly think about the history of their code and how they would like to share it with others. Git gives you many options whereas tools like Subversion and Perforce don’t; this plethora of options can make things confusing. In fact, it can lead to very philosophically different approaches for all aspects of your development process, ranging from shared-repository vs integrator to whether or not to merge frequently (yes? or no!). Here are a few useful readings on how people actually can use git:

Oliver Steele’s Git Workflow and commit policies. Helpful for understanding how people use the index and remotes.
Ryan Tomako has another way to use the index to help you split logically separate work into separate commits. Though some people don’t like the index.

Once you’re ready for more “tips”, be sure to follow:

Junio Hamano’s blog. Junio is the maintainer of git and writes deep and useful technical posts.
Scott Chacon’s ProGit blog. Scott has done more for advocacy of Git than anyone else.
Git Ready; an inactive but very useful set of tips from Nick Quaranto.

For more regularly updated and possibly esoteric resources, check out my git bookmarks on delicious. Do you have some useful resources for getting people started with Git? Leave them in the comments.

Update: Added a link to the Visual Git Reference.

StackOverflow DevDays Boston 2009, Afternoon

2009-10-12T07:21:55-04:00

The afternoon of Boston DevDays 2009 was, in my opinion, not as broadly appealing as the morning sessions (see my writeup of the morning here). However, there was still a lot of interesting material presented.

Joel welcomed us back from lunch by plugging StackExchange and how it’ll mean the end of “crappy old copies of Usenet” (by which he meant phpBB). He showed a pretty graph of StackOverflow edging out ExpertsExchange in traffic. He also announced a new job search site called careers.stackoverflow.com that charges job seekers some money and asks you what your favorite editor is. There was also a video ad for the FogCreek training videos. This man knows how to monetize.

Patrick Hynds and Chris Bowen

The first technical session of the afternoon was on ASP.NET MVC. Patrick started the session with an explanation of ASP.NET MVC’s history relative to ASP Classic and ASP.NET, and why one might want to use a model-view-controller (MVC) architecture for a website: for example, much finer control over generated HTML compared to traditional ASP, test-driven development, and better URLs for SEO.

The rest of the talk was a demo of creating a hello world MVC application in Visual Studio. The presenters walked through updating models and view and controllers, setting up some basic routing. It seems that ASP.NET MVC is a fine re-implementation of Ruby on Rails or Django for the Microsoft world. One concrete tip I learned was that in Visual Studio, Control-. will offer you some completions or other shortcuts.

Reception to this talk was somewhat mixed, at least as far as I can tell from the blogs and Tweets about it. The talk itself could have been improved, of course; for example it would have helped for Patrick to have explained what MVC stood for (with a few architecture diagrams) before plugging its advantages for ten minutes. My take is that if you knew nothing about MVC, it was a straightforward talk that gave an introduction to the concepts and the implementation in .NET. If you were already familiar with MVC, I think you would have thought it pretty content-free as there wasn’t a tremendous amount of focus on the ASP.NET side of things.

John Resig

John Resig, the creator and lead developer of jQuery, a very popular JavaScript library, next took the stage to talk about JavaScript testing.

“Developing for JavaScript is a lot like whack-a-mole,” John reported. The large space of operating systems, browsers, browser versions, JavaScript engines, and browser plugins mean that typically if you fix one thing you’re more than likely to break something else. And so, in some informal studies, John found that people just don’t test. This is something John would like to change.

A unit test suite for JavaScript apparently isn’t that hard to write. John threw up a bunch of increasingly feature-rich test harnesses—with asserts, grouping by role, and a test runner web-page—using a few dozen lines of JavaScript. The hardest part of writing a test suite is likely to be adding support for asynchronous events (e.g. XMLHttpRequests). Fortunately, there are several pre-built suites such as QUnit, JSUnit, YUI Test, and Selenium. John spent a bit of time talking about the differences between these frameworks, and particularly plugged QUnit and YUITest.

Selenium is of particular interest since, unlike the others, it is not just a unit test framework. It also has plugins to allow recording and scripting events to a browser, so you can do whole site testing. It even comes with Selenium Grid which will let you distribute and automate testing. This seems like a big win.

There are JavaScript engines like Rhino. To help test code in a browserless environment, John wrote env.js, which is a browser-like environment that runs in pure JavaScript. He talked about how this could be used for screen scraping.

Finally, John introduced testswarm.com. This is a SETI@Home style site where anyone can visit the site, download some tests to run and report back the results. This should give very broad coverage and allow developers to get feedback from a wide range of real environments (e.g. mobile!).

Overall, John’s talk was a rapid-fire overview of JavaScript testing resources from a JavaScript ninja. It was very practical, easy to follow and probably great for anyone who does JavaScript development. However, it lacked the “Python is awesome!” feel of Ned Batchelder’s morning talk and so for a non-Javascript developer such as myself, it was not as appealing.

Miguel De Icaza

Miguel De Icaza closed out the day with a talk on Mono. Miguel explained that he wasn’t really sure what he should talk about—Mono is a giant universe and explaining “Mono” it is “kind of like explaining God”—and his informal survey didn’t really provide a mandate. He would up giving some nice technical demonstrations of some recent Mono developments, with a fair amount of personal flair to keep the audience engaged.

The core of Mono is an implementation of the Common Language Runtime for Linux. One of the goals of this project was to bring the best development tools to Linux.

The first demonstration Miguel gave was an impressive combination of tools. Using a plugin to Visual Studio, Miguel demonstrated that you could develop a Linux port of a .NET application entirely using Visual Studio on Windows, and seamlessly testing and packaging on a Linux machine (or VM), by walking through a live example of porting BlogEngine.NET. This made use of Bonjour to dynamically discover the Linux machines, pushing execution to the selected machine, and viewing the debugging results in Visual Studio.

Miguel then decided that a developer might want to publish their application as a software appliance, so he walked through a complete demonstration of using SUSE Studio. He seamlessly built an RPM on his Linux box from Visual Studio and pushed it into “the cloud” of SUSE Studio. From his browser, he configured an appliance with that RPM, baked it the way they do on cooking shows, booted the virtual machine in the cloud, accessed it using a Flash-based console in the web browser, and accessed the port of BlogEngine.NET that he had just booted.

For his second major demonstration, Miguel moved over to MonoTouch. This showed using MonoDevelop, an IDE for Mono developers, running on a Mac, to work with the iPhone interface builder application, to build a simple flashlight application (i.e., a giant white button) for the iPhone simulator. He talked a little also about the technical work involved here, which was to compile the developer’s Mono code into ARM assembly and link it into the Mono runtime, to create an iPhone application. This gets around Apple’s “no interpreters” rule.

Miguel’s talk was easily the most entertaining one of the day. It was perhaps most entertaining because, in addition to his wry humor (check out the pictures from Ian Robinson’s Tweet, for example), as he performed his live demonstrations, things would break, whereupon Miguel would think, realize what was wrong, pop open a Terminal and fix it. That’s not something you see in the usual carefully scripted demos at most shows. Of course, Miguel was also demonstrating some interesting technical features and giving an advertisement for a wide-range of Mono-related tools as well, so there was something for everyone.

Wrap-up

After Miguel’s talk, Joel suggested that the audience break up into informal groups and get dinner, loosely organized around seven areas that he suggested. I hadn’t planned for that and had to get home; not sure how many people went.

Overall, I think DevDays was well worth attending. On the networking side, I got to meet a few local developers (of whom I’ve posted a few pictures on Flickr) and catch up briefly with some acquaintances from school. On the technical side, I got a broad overview of several popular technical areas from leading figures in those areas.

StackOverflow DevDays Boston 2009, Morning

2009-10-08T09:47:44-04:00

Boston DevDays kicked off a month-long tour of technical talks aimed at programmers, organized by StackOverflow and Carsonified. I had the good fortune to attend, meet a few interesting people and see some fun talks. I tried to write a bit in real-time (search Twitter here) but the WiFi was pretty over-subscribed and there was no cell coverage to speak of so eventually I gave up. Here are some more detailed notes, starting with the morning sessions.

The day ran very much on schedule, and after some very loud music, DevDays opened with a funny video demonstrating the competent technical leadership of the keynote speaker…

Joel Spolsky

Joel opened the day by giving a demonstration of the tyranny of computers: you are constantly interrupted with questions asking for decisions, like “Do you want to install these 10 updates to Windows?” or “Enable caret browsing?”, that can be really hard to answer. He argued that one of the reasons that computers ask us so many questions is that programmers want to give users the power (and control) to do what they want. But that’s another way of saying that programmers don’t want to make the decisions to keep things simple. The rest of Joel’s somewhat meandering but always entertaining talk was a discussion of how programmers (us!) should approach decision making, framed as a trade-off between simplicity and power.

Decisions are ultimately hard to make—there have been many studies that demonstrate that when people have too many choices, they freeze up and choose nothing. Thus, we’ve seen strong push towards simplicity in recent years; one clear example of that has been the Getting Real book from 37signals. Joel (jokingly?) points out that three of the four 37signals apps are in fact just one big textarea tag that you type into. Other examples, given later, include of course Apple’s products and Google.

But why do we wind up with programs with lots of options? Well, if you have a simple program, you find that most people won’t buy your product if it doesn’t have feature X (or Y or Z or …). So you wind up adding features over time, as you get more customers and more experience. Thus, removing simplicity often happens as a side-effect of making money.

How to balance these? If we don’t want to take away all decisions from the user, we need a rule to guide us in what to remove and what to keep. One rule to follow is that the computer does not get to set the agenda. Good decision points are those that should help the user achieve what they want to do. Bad decision points are those that interrupt the user, that the user really isn’t equipped to answer (e.g., should GMail automatically display inline images in HTML?), that are things that the programmer cares about.

To decide what is good or bad, developers need a good model to understand what the user is trying to do—Joel says every user is ultimately trying to replicate their DNA, but you may have some more refined model. Joel gave the example of Amazon’s 1-Click purchasing where the user should just be able to buy something with a single click. Apparently, early drafts of 1-Click really weren’t one click: programmers kept wanting to put in things like confirmation pages. Eventually, they arrived at just one click—by not immediately starting the order processing, but holding it for a few minutes to consolidate related 1-Click orders and allow for cancellation of errors. This was more work for the developers, but simpler for the user. This is what we want to see happen.

Overall, I think Joel’s talk set a nice tone for how we should think as developers but didn’t offer anything particularly ground-breaking.

Ned Batchelder on Python

Ned Batchelder presented Python as a “Clean noise free environment to type your ideas.” Or, alternatively, “Python is awesome.” With a few bullets to lead-off (e.g., the REPL, duck typing, and “batteries included” nature of Python), we dove into code to really understand what Python is capable of doing. Ned’s slides are online if you want to take a look. I won’t cover his talk in much detail since it was largely explaining the Python language but will list some high points.

The first example was Peter Norvig’s concise spell corrector. This code makes ample use of list comprehensions and sets, plus functions like re.findall, collections.defaultdict, and parameterized max, which were all new to me.

The second example was a custom build-up of a simple Python templating engine, loosely based on Django’s template styles. This example demonstrated simple formatting with the percent operator, but then quickly moved into more advanced features like duck-typing (by implementing __getitem__) and callable.

One of the nice things about Ned’s presentation is that he demonstrated the power of Python in two short but extremely powerful examples that left people who didn’t know Python thinking, “Wow! That is really amazing!”

Ned recommended reading Dive into Python to learn more, and if you are in the Boston-area, checking out the Cambridge Python meetup.

Dan Pilone

Dan Pilone’s talk was an overview of iPhone development, first giving a quick market overview, then giving a broad overview of the nitty-gritty of developing in Objective C, and finally diving into the practical economics and realities of selling iPhone apps. The iPhone app market (as of Apple’s September numbers), consists of something like 85,000 applications, of which 75% are pay applications. Some 35% of applications are games, whereas 6% fall into more social applications. There are 5 or 6 different iPhone/iTouch hardware platforms, and something like 20 million plus devices sold. The iPhone has a great user experience and comes with a great delivery model (the iTunes app store). This combined with some of the numbers make it a good platform to develop for, with significant amounts of money that can be earned. (Dan emphasized that you really have to work to develop and market the application, hence the “earned”.)

The iPhone development environment is very shiny. I’m sure Ars Technica has a much better overview but in short Dan demonstrated some tools like XCode, CoreData (a graphical SQLite data modeler), reference counting support (aka retain), and Instruments (a memory profiler). Dan suggested that this shininess is to make up for some of the oddities of Objective C that you’ll have to live with. He also demonstrated some of the interface builder tools and how they link up.

Testing turns out to be quite interesting; the simulator is okay but limited and often your app will work in the simulator but fail on real devices. For example, your app on a real device might “run so slow you wish it had crashed”. The simulator also doesn’t enforce sandboxing as strictly as real devices, where each app has its own uid and single directory where it can store data. There are also many different hardware variants that you have to support that limit you in different ways: for example, early iPhones only give you 40MB of memory to play with whereas newer ones give you almost 120MB. This is not reflected well in the simulator either.

Shipping an iPhone app on the app store requires approval, a process that can take two weeks per round-trip. There’s no way to get around it so you must budget in time for that in development. The approval store helps guarantee a minimal level of quality of apps—they will verify that your app indeed works on all different hardware, and they will (eventually) catch any licensing violations but they’re overall pretty reasonable.

Once you get approval, you show up in the recent releases section of the app store, and there you have about 24 hours to get popular or else you will fade into the long tail of oblivion. In fact, if you are in the top 50 apps, you will easily get double the sales (presumably relative to the 51st app); if you are in the top 10, you’ll be getting an order-of-magnitude more sales. So, make sure you get your approval/release-date lined up with your marketing blitz. The alternative is to charge a bit more than $0.99, and go for slow but steady sales.

As a Blackberry owner and Linux user, I found Dan’s talk to be a great introduction to iPhone development. Presumably his new ORA book, Head First iPhone Development, would be a good buy if you are in to that sort of thing.

Joel on FogBugz

Before lunch, Joel took the opportunity to give a pitch for his company’s FogBugz product, and announce some new features. He gave us a walk-through of its capabilities, from organizing documentation, to project planning, to bug tracking, to user support. New features announced are a rich plugin architecture, plus support for Mercurial and code reviews in a new hosted plug-in to FogBugz called Kiln. He spent a fair amount of time on that, demonstrating calling hg push from the command line. He also demonstrated the evidence-based scheduling features of FogBugz.

Nothing too exciting for me working in a big company using Perforce, but a good marketing opportunity for FogCreek and a nice chance to see how some other people do scheduling and bug tracking. I was a bit disappointed that there’s no direct way to do pre-commit (i.e. pre-push) reviews a la Gerrit, but @jasonrr says you can set up a branch repo, push to that and then review there before merging to main. I expect this means that GitHub will be getting code review support soon.

Lunch!

With that, we broke for lunch. @jldio has the scoop on lunch, and his own write-up of the day too. More to come later, thanks for reading.

Systems Researchers: Mike Freedman

2009-03-25T06:00:49-04:00

Mike Freedman and I have known each other since we were Masters students at MIT, working on things like the Tarzan anonymizing network (a parallel, pre-cursor to Tor). He went on to build the hugely successful (“as seen on Slashdot”) Coral content distribution network, which figured largely in his dissertation. It’s a great treat to have him talk here about how Coral was built and deployed. Be sure also to check out his research group blog for more interesting thoughts from him and his students!

What did you build?
CoralCDN is a semi-open content distribution network. Our stated goal with CoralCDN was to “democratize content publication”: namely, allow websites to scale by demand, without requiring special provisioning or commercial CDNs to provide the “heavy lifting” of serving their bits. Publishing with CoralCDN is as easy as slightly modifying a URL to include .nyud.net in its hostname (e.g., http://www.cnn.com.nyud.net/), and the content is subsequently requested from and served by CoralCDN’s network of caching web proxies.

Our initial goal for deploying CoralCDN was a network of volunteer sites that would cooperate to provide such “automated mirroring” functionality, much like sites do somewhat manually with open-source software distribution. As we progressed, I also imagined that small groups of users could also cooperate in a form of time-sharing for network bandwidth: they each would provide some relatively constant amount of upload capacity, with the goal of being able to then handle any sudden spikes (from the Slashdot effect, for example) to any participant. This model fits well with how 95th-percentile billing works for hosting and network providers, as it then becomes very important to flatten out bandwidth spikes. We started a deployment of CoralCDN on PlanetLab, although it never really migrated off that network. (We did have several hundred users, and even some major Internet exchange points, initially contact us to run CoralCDN nodes, but we didn’t go down that path, both for manageability and security reasons.)

CoralCDN consists of three main components, all written from scratch: a special-purpose web proxy, nameserver, and distributed hash table (DHT) indexing node. CoralCDN’s proxy and nameserver are what they sound like, although they have some differences given that they are specially designed for our setting. The proxy has a number of design choices and optimizations well-suited for interacting with websites that are on their last legs—CoralCDN is designed for dealing with “Slashdotted” websites, after all—as well as being part of a big cooperative caching network. The nameserver, on the other hand, is designed to dynamically synthesize DNS names (of the form .nyud.net), provide some locality and load balancing properties when selecting proxies (address records it returns), and ensure that the returned proxies are actually alive (as the proxy network itself is comprised of unreliable servers). The indexing node forms a DHT-based structured routing and lookup structure that exposes a put/get interface for finding other proxies caching a particular web object. Coral’s indexing layer differs from traditional DHTs (such as MIT’s Chord/DHash) in that it creates a hierarchy of locality-based clusters, each which maintains a separate DHT routing structure and put/get table, and it provides weaker consistency properties within each DHT structure. These latter guarantees are possibly because Coral only needs to find some proxies (preferably nearby ones) caching a particular piece of content, not all such proxies.

Tell us about what you built it with.
CoralCDN is built in C++ using David Mazieres’ libasync and libarpc libraries, originally built for the Self-certifying File System (SFS). This came out of my own academic roots in MIT’s PDOS group, where SFS was developed by David and its libraries are widely used. (David was my PhD advisor at NYU/Stanford, and I got my MEng degree in PDOS.) Some of the HTTP parsing libraries used by CoralCDN’s web proxy were from OKWS, Max Krohn’s webserver also written using SFS libraries. Max was research staff with David at NYU during part of the time I was there. It’s always great to use libraries written by people you know and can bother when you find a bug (although for those two, that was a rare occurrence indeed!).

When I started building CoralCDN in late 2002, I initially attempted to build its hierarchical indexing layer on top of the MIT Chord/DHash implementation, which also used SFS libraries. This turned out to be a mistake (dare I say nightmare?), as there was a layering mismatch between the two systems: I wanted to build distinct, localized DHT clusters in a certain way, while Chord/DHash sought to build a single, robust, global system. It was thus rather promiscuous in maintaining group membership, and I was really fighting the way it was designed. Plus, MIT Chord was still research-quality code at the time, so bugs naturally existed, and it was really difficult to debug the resulting system with large portions of complex, distributed systems code that I hadn’t written myself. Finally, we initially thought that the “web proxy” part of the system would be really simple, so our original proxy implementation was just in python. CoralCDN’s first implementation was scrapped after about 6 months of work, and I restarted by writing my own DHT layer and proxy (in C++ now) from scratch. It turns out that the web proxy has actually become the largest code base of the three, continually expanded during the system’s deployment to add security, bandwidth shaping and fair-sharing, and various other robustness mechanisms.

Anyway, back to development libraries. I think the SFS libraries provide a powerful development library that makes it easy to build flexible, robust, fast distributed services…provided that one spends time overcoming their higher learning curve. Once you learn them, however, they make it really easy to program in an event-based style, and the RPC libraries prevent many of the silly bugs normally associated with writing your own networking protocols. I think Max’s tame libraries significantly improve the readability and (hopefully) lessen the learning curve of doing such event-based programming, as tame removes the “stack-ripping” that one normally sees associated with events. Perhaps I’ll use tame in future projects, but as I’ve already climbed the learning curve of libasync myself, I haven’t yet.

That said, one of my PhD students at Princeton, Jeff Terrace, is building a high-throughput, strongly-consistent object-based (key/value) storage system called CRAQ using tame. He’s seemed to really like it.

How did you test your system for correctness?
I deploy it? In seriousness, it’s very difficult to test web proxies, especially ones deployed in chaotic environments and interacting with poorly-behaving clients and servers.

I did most of my testing during initial closed experiments on about 150-300 PlanetLab servers, which is a distributed testbed deployed at a few hundred universities and other institutions that each operate two or more servers. Testing that the DHT “basically” worked was relatively easy: see if you actually get() what you put(). There are a lot of corner cases here, however, especially when one encounters weird network conditions, some of which only became apparent after we moved Coral from the network-friendly North American and European nodes to those PlanetLab servers in China, India, and Australia. Always be suspicious with systems papers that describe the authors’ “wide deployment” on “selected” (i.e., cherry-picked) U.S. PlanetLab servers.

Much of the testing was just writing the appropriate level of debug information so we could trace requests through the system. I got really tired of staring at routing table dumps at that time. Last year I worked with Rodrigo Fonseca to integrate X-Trace into CoralCDN, which would have made it significantly easier to trace transactions through the DHT and the proxy network. I’m pretty excited about such tools for debugging and monitoring distributed systems in a fine-grained fashion.

Testing all the corner cases for the proxy turned out to be another level of frustration. There’s really no good way to completely debug these systems without rolling them out into production deployments, because there’s no good suite of possible test cases: The potential “space” of inputs is effectively unlimited. You constantly run into clients and servers which completely break the HTTP spec, and you just need to write your server to deal with these appropriately. Writing a proxy thus because a little bit of learning to “guess” what developers mean. I think this actually has become worse with time. Your basic browser (FireFox, IE) or standard webserver (Apache) is going to be quite spec-compliant. The problem is that you now have random developers writing client software (like podcasters, RSS readers, etc.) or generating Ajax-y XmlHttpRequest’s. Or casual developers dynamically generating HTTP on the server-side via some scripting language like PHP. Because who needs to generate vaguely spec-compliant HTTP if you are writing both the client and server? (Hint: there might be a middlebox on path.) And as it continues to become even easier to write Web services, you’ll probably continue to see lots of messy inputs and outputs from both sides.

So while I originally tested CoralCDN using its own controlled PlanetLab experiments, after the system went live, I started testing new versions by just pushing them out to one or a few nodes in the live deployment. Then I just monitor these new versions carefully and, if things seemed to work, slowly push them out across the entire network. Coral nodes include a shared secret in their packet headers, which excludes random people from joining our deployment. I also use these shared secrets to deploy new (non-backwards-compatible) versions of the software, as the new version (with a new secret nonce) won’t link up with DHT nodes belonging to previous versions.

How did you deploy your system? How big of a deployment?
CoralCDN has been running 24/7 on 200-400 PlanetLab servers since March 2004. I manage the network using AppManager, built by Ryan Huebsch from Berkeley, which provides a SQL server that keeps a record of current node run state, requested run state, install state, etc. So AppManager gives me a Web interface to control the desired runstate of nodes, then all nodes “call home” to the AppManager server to determine updated runstate. You write a bunch of shell scripts to actually use these run states to start or stop nodes, manage logs, etc. This “bunch of shell scripts” eventually grew to be about 3000 lines of bash, which was somewhat unexpected. While AppManager is a single server (although nodes are configured with a backup host for failover), CoralCDN’s scripts are designed for nodes to “fail same”. That is, requested runstate is stored durably on each node, so if the management server is offline or returns erroneous data (which it has in the past), the nodes will maintain their last requested runstate until the management server comes back online and provides a valid status update.

How did you evaluate your system?
We performed all the experiments one might expect in an academic evaluation on an initial test deployment on PlanetLab. Our NSDI ‘04 paper discusses these experiments.

After that stage, CoralCDN just runs – people continue to use it, so it provides some useful functionality. My interest transitioned from providing great service to just keeping it running (while I moved onto other research).

I probably spend about 10 minutes a week “keeping CoralCDN running”, which is typically spent answering abuse complaints, rather than actually managing the system. This is largely because the system’s algorithms were designed to be completely self-organizing – as we initially thought of CoralCDN as a peer-to-peer system – as opposed to a centrally-managed system designed for PlanetLab. System membership, fault detection and recovery, etc., is all completely automated.

Unfortunately, dynamic membership and failover doesn’t extend to the primary nameservers we have registered for .nyud.net with the .net gTLD servers. These 10-12 nameservers also run on PlanetLab servers, so if one of these servers go offline, our users experience bad DNS timeouts until I manually remove that server from the list registered with Network Solutions. (PlanetLab doesn’t provide any IP-layer virtualization that would allow us to failover to alternate physical servers without modifying the registered IP addresses.) And I have to admit I’m pretty lazy about updating the DNS registry, especially given the rather painful web UI that Network Solution provides. (In fairness, the UIs for GoDaddy and other registrars I’ve used are similarly painful). I think registrars should really provide a programmatic API for updating entries, but haven’t found one for low-cost registrars yet. Anyway, offline nameservers are probably the biggest performance problem with CoralCDN, and probably the main reason it seems slow at times. This is partly a choice I made, however, in not becoming a vigilant system operator for which managing CoralCDN becomes a full-time job.

There’s a lesson to be had here, I think, for academic systems that somehow “escape the lab” but don’t become commercial services: either promote lessened expectations for your users (and accept that reality yourself), build up a full-time developer/operations staff (a funding quandary), or expect the project to soon die-off after its initial developers lose interest or incentives.

Anything you’d like to add?
My research group actually just launched a blog. In the next few weeks, I’ll be writing a series about some of the lessons I’ve learned from building and deploying CoralCDN. I want to invite all your readers to look out for those posts and really welcome any comments or discussion around them.

Systems Researchers: Justin Cappos

2009-03-16T06:00:22-04:00

Justin Cappos received his PhD from the University of Arizona under the supervision of John Hartman. I met Justin several years ago at a PlanetLab Consortium meeting when he was starting to work on Stork, a system to simplify package deployment. He is currently a Post Doc at the University of Washington working with Tom Anderson and Arvind Krishnamurthy.

What did you build?
The most relevant / longest term projects are: Stork, San Fermin, and Seattle.

Stork is a package manager that has security and functionality improvements over existing Linux package managers. Some of the advances we made in Stork have been adapted by APT, YUM, YaST and other popular Linux package managers.

San Fermin is a system for aggregating large quantities of data from computers. San Fermin provides the result faster and with better fault tolerance than existing techniques.

Seattle is an educational testbed built from resources donated by universities all around the world. The universities run a safe, lightweight VM that students from other universities can run code in.

Tell us about what you built it with.
I used Java for San Fermin because I needed to leverage existing Pastry code that was in Java. I used Python for Stork and Seattle. I found Python to be far superior for large projects (other languages I’ve used are Java, C, C++, QBASIC, and Pascal). Python has been a dream come true because it’s great for prototyping, easy to learn, and the resulting code is readable (so long as you have sensible style constraints on the written code).

Perhaps the most useful thing is getting other developers involved. I like to do the initial prototyping myself, but after that it is great to have others helping out.

How did you test your system for correctness?
There are whitebox / unit tests as well as blackbox / integration tests for most parts of the systems. The time that we spent building thorough test cases really paid off because it simplifies debugging.

I like to use my systems in real world environments with outside users, so the system is never done thus correctness is an iterative process. If I’m aware of bugs, we fix them. If I’m not aware of bugs, we’re adding features based upon user requests (and therefore may be adding more bugs for us to fix later). In general, the fact that we have users that rely on the software over long time periods is a testament to its stability which is related to correctness.

To more specifically answer the question you are really asking, I usually run my code by hand and evaluate it in small / constrained environments (turning these test runs into unit tests). I also follow a philosophy where I try to “detect and fail” as much as possible. I care more about correctness than performance (at least initially) and so add many redundant checks in my code to catch errors as soon as possible.

I find that if I’m careful and thorough when writing my code, I spend very little time debugging. I probably spend about 30% of the time writing code, 40% writing comments / docs (which I normally write before / during coding), 20% of the time writing test code for individual modules, and about 10% debugging after the fact. I think part of this is I’m really careful about checking input and boundary conditions and so I can normally pin-point the exact cause of a failure.

In terms of problems when writing code, I generally only use standard libraries and code I’ve written. I don’t depend on third party code because I don’t know what level of support it will have. Also, since I usually code in Python, it’s easy for me to add functionality to do whatever I need.

How did you deploy your system? How big of a deployment?
Stork has been deployed on PlanetLab for about 6 years. For the majority of the time we’ve been deployed on every working PlanetLab node. Stork has managed > 500K VMs and when I last checked was used daily by users on around two dozen sites. We initially used AppManager to deploy Stork, but since have been using PlanetLab’s initscript functionality.

San Fermin was deployed on PlanetLab for use in combining and managing logs from Stork. However, we found that San Fermin was dysfunctional due to difficulties in getting Pastry to start reliably and work when non-transitive connectivity occurs. As a result, we mainly ended up using San Fermin as a publication vehicle.

Seattle is currently deployed on more than 1100 computers around the world. We have done our initial deployment by encouraging educators to use our platform in networking and distributed systems classes. Our longer term plans involve using Seattle to build a research testbed of 1 million nodes.

How did you evaluate your system?
With San Fermin (and other systems research I haven’t mentioned), there is fairly clear related work to compare against. In some cases, the biggest challenge has been getting the existing research prototype code from another project to run well enough for comparison.

For the work that I’ve done where I’ve focused on impact over publication impact (Seattle / Stork), evaluation is much more difficult because these aren’t incremental improvements over existing models and systems. These systems break the mold in terms of security and / or functionality so sometimes it’s difficult to know how to compare them to existing work.