RustyKlophaus.com

How to Practice a Technology Talk

Technology conferences combine two very disparate skills: code and public speaking. I code, and I've also presented at nine different tech conferences in the past two years, across the US and internationally, with an average audience of about 140 people per talk. Each time I try to tweak the way I practice to maximize the quality of the presentation while minimizing preparation time and effort.

In the beginning, I wasn't great. Now I'm much, much better. I attribute a large part of the improvement to better practice methods. The standard advice on giving a good presentation is "Practice, Practice, Practice." But nobody says how to practice. This is what works for me:

Practice Fast
Practice Slow
Practice Out Loud

Practice Fast

Long distance runners have a training technique called "Fartlek" which means "speed play" in Swedish. A Fartlek workout consists of running at a normal pace with sudden bursts of sprinting. This does two things: 1) it gets the runner's legs used to a variety of different paces, and 2) it simulates high-pressure race conditions, such as when a runner needs to kick to the finish to beat an opponent.

There are benefits to practicing a presentation at a fast pace as well. If you have a 50 minute presentation, try to speed through it in just 5 minutes. Speak quickly, be brief, drop any extraneous chatter, but make sure to hit all of your important points.

Practicing at 10x the normal speed has three benefits:

Just like a Fartlek workout, it simulates a high-pressured environment. This mimics the panic that you sometimes feel on stage, and gives you an opportunity to get used to it and overcome it. Moving through your talk in fast-forward helps you internalize the order of your points, allowing you to transition more smoothly on presentation day.
It quickly highlights what is important and what is not. You are forced to streamline in order to meet the new time constraints.
It will allow you to recover more smoothly from a late start, technical glitches, or a long question earlier in your presentation. When you can give your entire talk at warp speed, you are ready for anything.

Practice Slow

Novice guitar players often try to learn a complicated solo by playing it over and over as quickly as possible. Counter-intuitively, this isn't the optimal way to learn a solo. Experienced guitarists know that the best way to really hone something is to practice slowly. Very slowly. Like one-half or one-third normal speed. This forces each note to be clean, clear, and deliberate.

Again, this same principle applies to presenting. Run through your presentation as slowly as possible. Talk a little more slowly, exhaust every point, use metaphors, embellish your language, phrase things in different ways, etc. Draw things out as much as possible.

Practicing your talk at slow speed has three benefits:

It helps you find useful metaphors. By stretching out your talk, you are forced to brainstorm for new, creative ways to phrase things. Some of those new ways will be better than what you currently have.
It reveals rabbit holes. Some presentations contain small asides or tangents. This is fine, as long as they remain small asides. But in the nervousness of presenting, there is a tendency to turn that small aside into a deep rabbit hole, completely derailing the presentation. It's important to identify the potential rabbit holes early so that you don't fall down them on presentation day.
It will allow you to extend your talk if necessary. An earlier speaker may have already covered parts of what you wanted to say, or the audience may have fewer question than you anticipated. Practicing at slow speeds allows you to adapt.

Practice Out Loud

Reading your presentation silently, or whispering quietly at a desk, is not effective practicing. Practicing should be a dress rehearsal. You actually need to stand up, walk around, gesture, and project your voice to a pretend audience.

Practicing out loud has three benefits:

Train your emotional energy. Unlike normal conversation, a presentation is mostly a one-way exchange of emotional energy. A polite audience is silent and still. When you are speaking to an audience, you need to get comfortable providing all of the emotional energy for a sustained period of time. The best way to prepare is to get comfortable talking to an empty room.
Acclimate your voice and body to speaking loudly and clearly. Many of the symptoms that we blame on "nervousness", (dry mouth, tight throat, breathlessness, etc.) are really just the physical demands of presenting.
Get comfortable moving around. A speaker who stands still behind a podium is visually boring. Moving is an easy way to add variety to your presentation, and a good way to feel in control, like the master of the stage. It's important to practice purposeful movement in advance because you can't think about it too much during your presentation.

How Long Should You Practice?

Two years ago, I budgeted 15 minutes of practice for every minute of presentation time. This does not account for time outlining the presentation or making slides, just practice time. Now, I'm down to a ratio of 5 minutes of practice for every minute of presentation.

I try to have the slides done four days to a week in advance, and then alternate between practicing at fast, normal, and slow speeds, depending on how much time I have, and always out loud. I also spend more time on the introduction and first few slides, because a smooth start carries through to the rest of the talk.

Last Minute Preparation

On the day of the presentation, I follow roughly the same routine:

Sit with good posture before the talk. This sounds weird, but it helps maximize oxygen to the lungs and brain.
Set up a little early if possible, and then walk to both for sides of the stage, looking out at the audience. It gives me a chance to adjust to the size of the room.
Chat with somebody in the front row for the last minute or two before its time to start. It gets the words flowing.
Carry around a cup of water while presenting. Keeps you hydrated, and also makes things feel less formal.

Finally, when I start, I try to give the presentation room to be an interactive, two way exchange of energe. I let the audience know they should interrupt for questions, and I stop for questions at the end of every section.

Good luck!

GNU Screen / Pecho

GNU Screen is like Emacs for your console: Lots of power at the top of a steep learning curve. And just like Emacs, screen can be customized to fit your individual workflow. It's helpful to refer to others' configuration files when you first start using and customizing screen, so I've posted my screen configuration file below.

I also developed a simple utility called pecho that allows you to send keystrokes to multiple screens at the same time, introduced at the end of this post.

What is GNU Screen?

GNU Screen is a full-screen window manager for your terminal. Some features:

Start up multiple shell instances (called windows) inside a single shell. Navigate between them with the keyboard.
Divide your window into multiple regions, with a separate shells running in each region. Again, with keyboard navigation.
Detach from a Screen session, and have the applications keep running. (You can re-attach at a later point.) Very useful for keeping things alive between ssh sessions.

This is not a tutorial. I'm assuming you already know how to run screen and do some simple commands, if not, there are many great tutorials out there.

My Screen Configuration File

I've modified screen to use the same keybindings as Emacs when possible. For example, C-x C-f will create a new shell, and C-x C-b lists all shells. Copy the text below into ~/.screenrc and then run screen.

Pecho - Parallel Echo

A few weeks ago I needed to perform a series of roughly identical operations tasks across 5 machines in a way that couldn't be easily scripted. "This would be so much easier", I thought, "if I could type in one terminal and have my keystrokes broadcast to the other terminals." Thus, Pecho was born.

Here is a short video demonstration:

To install Pecho (short for Parallel Echo), copy the code below to pecho file in your path, and change its attributes to executable. This has been tested on Mac OSX.

To run Pecho (Parallel Echo), simply type pecho followed by the Screen window numbers that should receive the output, ie: pecho 1 2 3 4. Then, type a key in the the pecho window, and it will appear in the other windows.

It works by reading a keystroke on the command line and then iterating over each window number, calling the Screen "stuff" command to transmit the character. If you're careful, you can actually use it to edit multiple remote files in vi at the same time.

Erlang Factory London - Recap

Erlang Factory London gathers Erlang pioneers from across the world—Berlin to Boston, Krakow to Cordoba, and San Francisco to Shanghai—for a two-day conference of innovative Erlang development.

The summaries below are just a small sampling of the talks at Erlang Factory London. There were three tracks running back-to-back for two days, and I often couldn't decide which of the three to attend. Slides and videos will be released by Erlang Solutions, and can be found under individual track pages on the Erlang Factory website.

Opening Session
The History of the Erlang Virtual Machine
Riak from the Inside
Mnesia for the CAPper
Riak Search
Building a Scalable E-commerce Framework
Clash of the Titans: Erlang Clusters and Google App Engine
Using Open-Source Trifork QuickCheck to test Erjang
Efene: A Programming Language for the Erlang VM
Erlang in Embedded Systems
Zotonic: Easy Content Management with Erlang's Performance and Flexibility
Closing Session
Final Thoughts

Please email me with any corrections.

Day 1 - June 10, 2010

Opening Session

Francesco Cessarini (Chief Strategy Officer, Erlang Solutions Ltd.), began the conference with a warm welcome and a quick review of progress made by Erlang-based companies in the last year.

Some highlights:

LShift (RabbitMQ, built in Erlang) acquired by VMWare
Couchio formed and funded to develop and support CouchDB
MochiMedia (Flash game ad network using Erlang stack), acquired by Shanda Games
Basho received two rounds of funding to develop and support Riak
Klarna received a round of investment from Sequia Capital, and Michael Moritz previous board member of Google, Yahoo, and Paypal, joined the Klarna board.

The History of the Erlang Virtual Machine - Joe Armstrong, Robert Virding

Joe Armstrong and Robert Virding gave a colorful, back-and-forth history of the Erlang's birth and early years. A few notable milestones and achievements:

Joe's early work on reduction machines. Robert's complete rewrite of Joe's work. Joe's complete rewrite of Robert's work. (etc.)
How Erlang was almost based on Smalltalk rather than Prolog
The quest to make Erlang 1.0x 80 times faster
Experiments with different memory management and garbage collection schemes
The train set used demonstrate Erlang, now in Robert's basement
The addition of linked processes, distribution, OTP, and bit syntax

It's easy to take a language like Erlang for granted and assume that its builders followed some well-known, pre-ordained path. Hearing Erlang's history from two of its main creators provided an excellent reminder that building software is both an art and a science, uncertain and exciting like any creative process.

Riak from the Inside - Justin Sheehy

Justin Sheehy (CTO of Basho Technologies) opened his talk by introducing Riak, "a scalable, highly-available, networked, open-source key/value store." He then very quickly announced that he wasn't there to talk about using Riak, he was there to talk about how Riak was built using Erlang and OTP

There are eight distinct layers involved in reading/writing Riak data:

The Client Application using Riak
The client-side HTTP API or Protocol Buffers API that talks to the Riak cluster
The server-side Riak Client containing the combined backing code for both APIs
The Dynamo Model FSMs that interact with nodes using Dynamo style quorum behavior and conflict resolution
Riak Core provides the fundamental distribution of the system (not covered in the talk)
The VNode Master that runs on every physical node, and coordinates incoming interaction with individual VNodes
Individual VNodes (Virtual Nodes) which are treated as lightweight local abstractions over K/V storage
The swappable Storage Engine that persists data to disk

During his talk, Justin discussed each layer's responsibilities and interactions with the layers above and below it.

Justin's main point is that carefully managed complexity in the middle layers allows for simplicity at the edge layers. The top three layers present a simple key/value interface, and the bottom two layers implement a simple key/value store. The middle layers (FSMs, Riak Core, and VNode Master) work together to provide scalability, replication, etc. Erlang makes this possible, and was chosen because it provides a platform that evolves in useful and relatively-predictable ways (this is a good thing, a surprising evolution is bad).

Mnesia for the CAPper - Ulf Wiger

Ulf Wiger (CTO of Erlang Solutions) discussed where Mnesia might fit into the changing world of databases, given the new focus on "NoSQL" solutions. Ulf gave a quick introduction to ACID properties, Brewer's CAP theorem, and the history of Mnesia, and then dove into a feature level description/comparison of Mnesia with other databases:

Deployed commercially for over 10 years
Comparable performance to current top performers clustered SQL space
Scalable to 50 nodes
Distributed transactions with loose time limits (in other words, appropriate for transactions across remote clusters)
Built-in support for sharding (fragments)
Incremental backup

The downsides are:

Erlang only interface
Tables limited to 2GB
Deadlock prevention scales poorly
Network partitions are not automatically handled, must recombine tables automatically

Ulf and others have done work to get around some of these limitations. Ulf showed code for an extension to Mnesia that automatically merges tables after they have split, using vector clocks.

Riak Search - Rusty Klophaus

I presented Riak Search, a distributed indexing and full-text search engine built on (and complementary to) Riak.

Part one covered the main reason for building Riak search: clients have built applications that eventually need to find data by value, not just by key. This is difficult, if not impossible, in a key/value store.

Part two described the shape of the final solution we set out to create. The goal of Riak Search is to support the Lucene interface, with Lucene syntax support and Solr endpoints, but with the operations story of Riak. This means that Riak Search will scale easily by adding new machines, and will continue to run after machine failure.

Part three was an introduction to Inverted Indexing, which is the heart of all search systems, as well as the difference between Document-Partitioning and Term-Partitioning, which forms the ongoing battle in the distributed search field. Part three continued with a deep-dive into parsing, planning, and executing the search query on Erlang.

Slides: http://www.slideshare.net/rklophaus/riak-search-erlang-factory-london-2010

Building a Scalable E-commerce Framework - Michael Nordström and Daniel Widgren

Michael Nordström and Daniel Widgren presented an Erlang-based e-commerce framework on behalf of their project team from Uppsala University (Christian Rennerskog, Shahzad Gul, Nicklas Nordenmark, Manzoor Ahmad Mubashir, Mikael Nordström, Kim Honkaniemi, Tanvir Ahmad, Yujuan Zou, and Daniel Widgren) and their industrial partner, Klarna AB.

The application uses a "LERN stack" (Linux, Erlang, Riak, Nitrogen), to provide a reusable web shop that can be quickly set up by clients, customized via templates and themes, and extended via plugins to support different payment providers.

The project is currently going a rewrite to update to the latest versions of Riak and Nitrogen.

GitHub: http://github.com/mino4071/CookieCart-2.0

Twitter: @Cookie_Cart

Clash of the Titans: Erlang Clusters and Google App Engine - Panos Papadopoulos, Jon Vlachoyiannis, Nikos Kakavoulis

Panos, Jon, and Nikos took turns describing the technical evolution of their startup, SocialCaddy, and why they were forced to move away from the Google App Engine. SocialCaddy is a tool that mines your online profiles for important events and changes, and tells you about them. For example, if a friend gets engaged, SocialCaddy will tell you about it, and assist you in sending a congratulatory note.

Google App Engine imposes a 30 second limit on requests. As SocialCaddy processed larger and larger social graphs, they bumped into this limit, which made GAE unusable as a platform. In response, the team developed Erlust, which allows you to submit jobs (written in any language) to a cluster. An Erlang application coordinates the jobs, and each job should read from a queue, process messages, and write to another queue.

Using Open-Source Trifork QuickCheck to test Erjang - Kresten Krab Thorup

Kresten Krab Thorup (CTO of Trifork) stirred up dust when he originally announced his intention to build a version of Erlang that ran on the JVM. Since then, he has made astounding progress. Erjang turns Erlang .beam files into Java .class files, now supporting a broad enough feature set to run Mnesia over distributed Erlang. Kresten claimed performance matching (or at times exceeding) that of the Erlang VM.

Erjang is still a work in progress, there are many BIFs that still need to be ported, but if a prototype exists to prove viability, then this prototype was certainly a success. One slide showed the code for the spawn_link function reimplemented in Java in ~15 lines of simple Java code.

For the second half of his talk, Kresten showed off Triq (short for Trifork Quickcheck), a scaled-down, open-source QuickCheck inspired testing framework that he built in order to test Erjang. Triq supports basic generators (called domains), value picking, and shrinking. Kresten showed examples of using Triq to validate that Erjang performs binary operations with the exact same results as Erlang.

More information about Erjang here: http://wiki.github.com/krestenkrab/erjang/

Day 2 - June 11, 2010

Efene: A Programming Language for the Erlang VM - Mariano Guerra

Mariano Guerra presented Efene, a new language that is translated into Erlang source code. Efene is intended to help coax developers into the world of Erlang who might otherwise be intimidated by the Prolog-inspired syntax of Erlang. We've heard about a number of other projects compiling into Erlang byte-code (such as Reia and Lisp-Flavored Erlang), but Efene takes a different approach in that the language is parsed and translated using Leex and Yecc into standard Erlang code, which is then compiled as normal. By doing this, Mariano manages to leave most of the heavy lifting of optimizations to the existing Erlang compiler.

Efene actually supports two different syntax flavors, one with curly brackets, the other without, leading to a syntax that feels vaguely like Javascript or Python, respectively. (The syntax without curly brackets is called Ifene, for "Indented Efene", and is otherwise identical to Efene.)

In some places, Efene syntax is a bit more verbose than Erlang. This is done to make the language more readable than Erlang. ("if" and "case" statements have more structure in Efene than Erlang.) In other places, Efene requires less typing, multi-claused function definitions don't require you to repeat the function name, for example.

Code samples and more information: http://marianoguerra.com.ar/efene

Erlang in Embedded Systems - Gustav Simonsson, Henrik Nordh, Fredrik Andersson, Fabian Bergstrom, Niclas Axelsson and Christofer Ferm

Gustav, Henrik, Fredrik, Fabian, Niclas, and Christofer (Uppsala University), in cooperation with Erlang Solutions, worked on a project to shrink the Erlang VM (plus the Linux platform on which it runs) down to the smallest possible footprint for use on Gumstix and BeagleBoard hardware.

The team experimented with OpenEmbedded and Angstrom, using BusyBox, uClibc, and stripped .beam files to further decrease the footprint. During the presentation, they played a video showing how to install Erlang on a Gumstix single-board computer in 5 minutes using their work.

More information about Embedded Erlang here: http://embedded-erlang.org

Zotonic: Easy Content Management with Erlang's Performance and Flexibility - Marc Worrell

Marc Worrell (WhatWebWhat) breaks CMSs into:

1st Generation - Static text and images
2nd Generation - Database- and template-driven systems (covers current CMS systems)
3rd Generation - Highly interactive, real-time, personalized data exchanges and frameworks

Zotonic is aimed squarely at the third generation, Zotonic turns a CMS into a living, breathing thing, where modules on a page talk to each other and other sessions via comet, and the system can be easily extended, blurring the line between CMS and application framework.

This interactivity is what motivated Marc to write the system in Erlang; at one point he compared the data flowing through the system to a telephone exchange. Zotonic uses Webmachine, Mochiweb, ErlyDTL, and a number of other Erlang libraries, with data in PostgreSQL. (Marc also mentioned Nitrogen as an early inspiration for Zotonic, parts of Zotonic are based on Nitrogen code, though much has been rewritten.)

The data model is physically simple, with emergent functionality. A site is developed in terms of objects (called pages) interlinked with other objects. In other words, from a data perspective, adding an image to a web page is the same as linking from a page to a subpage, or tagging a page with an author. Mark gave a live demo of Zotonic's ability to easily add and change menu structures, modify content, and add and re-order images. Almost everything can be customized using ErlyDTL templates. Very polished stuff.

Marc then introduced his goal of "Elastic Zotonic", a Zotonic that can scale in a distributed, fault-tolerant, "buzzword-compliant" way, which will involve changes to the datastore and some of the architecture.

Marc is now working with Maximonster to develop an education-oriented social network on top of Zotonic.

More information: http://zotonic.com

Closing Session

Francesco (CSO, Erlang Solutions, Ltd.) thanked the sponsors, presenters, and audience. Frank then gave a big special thanks to Frank Knight and Joanna Włodarczyk, who both worked tirelessly to organize the conference and make everything go smoothly.

Final Thoughts

Erlang is gaining momentum in the industry as a platform that enables you to solve distributed, massively concurrent problems. People aren't flocking directly to Erlang itself, they are instead flocking to projects built in Erlang, such as RabbitMQ, ejabberd, CouchDB, and of course, Riak. At the same time, other languages are adopting some of the key features that make Erlang special, including a message-passing architecture and lightweight threads.

Berlin Buzzwords Recap, Day 2

BerlinBuzzwords has a stellar venue and talks describing cutting edge developments on all things search, scalability, and storage. Day 2 is below, click here for Day 1.

Please email me with any corrections.

Day 2 - June 8, 2010

Keynote - Pieter Hintjens

Pieter (iMatix) started his talk with a series of high-level questions, developer-to-developer, intended to focus the audience on the fact that multi-core processing across multiple computers is the new norm, and (most) programming tools haven't yet evolved to meet the challenge.

He then identified and discussed some of the natural patterns in software development that make things simpler. After a few examples relating to the NoSQL world, he identified three that led into his introduction of 0MQ (Pronounced Zero-M-Q):

Asynchronous processing is a natural pattern, single threads that read from a queue, do work, and write to another queue.
Choosing reliability over persistence is a natural pattern. If you don't crash, you don't need to worry about persisting data. (Not sure how I feel about this one. Just gonna roll with it for the sake of the talk.)
Being agnostic to lines of communication is a natural pattern, what you send is orthogonal to how you send it.

Pieter then introduced the 0MQ library, which attempts to be a simple and lightweight message queue following these natural patterns. It takes care of defining queue endpoints, connecting (and re-connecting) the endpoints, buffering messages in memory, and not much else. The data format is simply a length and a binary blob, that's all.

According to Pieter, 0MQ should be thought of as a protocol, just like TCP or UDP. In other words, 0MQ is the sort of thing you embed in your database application.

With 0MQ, you can safely create multi-threaded applications that safely leverage multiple cores by making each worker process single threaded, and have it read from a queue, perform some unit of work, and write to another queue. (In other words, the Actor model, concurrency by message passing.) The idea is not new, but it bears repeating as often as possible because it's far simpler than multithreaded systems with locking, 99% of the time it's the right solution, and many people still don't know it.

Pieter had two choice quotes that drove home the main goals of 0MQ:

"If [a developer] can't pick it up in a weekend, it's not going to work."
"Cheap effective messaging changes the way we think about architecting applications."

Hypertable - Doug Judd

I unfortunately missed the first few minutes of Doug's talk. When I arrived, Doug (Hypertable) was in the midst of an architectural overview of the Google stack, BigTable architecture, the ideas behind a log-structured merge tree, and examples of Hypertable optimizations, including bloom filters and using different compression algorithms in different parts of the system.

The money slides came toward the end, with performance comparisions claiming Hypertable to be 70% faster than HBase on random reads and sequential writes. Another chart claimed Hypertable to be multiple times faster than HBase when doing only random reads of different distributions.

Apart from the previously mentioned optimizations, there seem to be two main reasons for Hypertable's speed: It's based in C vs. HBase's Java, and it is smart enough to dynamically adjust memory between caching reads and buffering writes according to the read/write distribution of the data.

And yes, Hypertable works with Hadoop for Map/Reduce-ing goodness...

Apache Cassandra Revisited - Eric Evans

Eric quickly focusing the talk by narrowing from All-of-NoSQL to Just-the-Large-Data-Projects, and then finally Just-the-BigTable-or-Dynamo-projects, which means Cassandra, HBase, Hypertable, Riak, and Voldemort.

Given these criteria, Eric called Cassandra the love-child of BigTable AND Dynamo, having influences from both. As such it has Dynamo staples like homogonous nodes, P2P-routing and partitioning (though not VNodes), and things like SSTables and (optionally) ordered data and range queries, similar to BigTable. (His slides contained a humorous, yet distrurbing picture showing a Brad Pitt/Angelina Jolie mutant child. See pics.)

Eric described the bootstrap process, the Cassandra data model (Keyspace, Column Family, Record, Column), and the interface (Thrift), and showed API examples.

He then highlighted a few key Cassandra developments and features:

Cassandra does now support batch Map/Reduce via Hadoop.
Cassandra comes with rack awareness, and this can be customized.
Keyspaces and Column families, currently defined in XML, will soon be configurable via an API without a restart.
Vector clocks will be added in the future. (But in his view, the hype outweights the benefit.)
SuperColumns may be phased out in the future, as they are not widely used, and lead to more confusion than they are worth.

According to Eric, the largest Cassandra instance that he knows of is Twitter, with around 100 nodes holding about 170TB of data.

Massively Parallel Analytics Beyond Map/Reduce - Fabian Huske

Fabian (TU Berlin) began by describing some of the challenges behind Map/Reduce, namely that it does make big data processing more simple than it used to be, but it still requires a developer to fit his problem into something Map/Reduce shaped, and this is exacerbated by the complexities of the various Map/Reduce frameworks out there.

Fabian then introduced Stratosphere, which is a combination programming model (PACT) and execution engine (Nephele) that provides additional blocks beyond Map and Reduce that can be used instead of a simple Map or Reduce, with the dual goals of making it easier to program as well as require fewer execution phases leading to higher performance. Stratosphere is a result of combining Map/Reduce with parallel database technology.

As an example, Fabian showed a SQL task that could be converted to two Map/Reduce jobs that with Stratosphere could be made simpler using Stratosphere.

A few examples: with PACT, you have new second-order functions in which to put your user code such as operations for "cross" (compute a cartesian cross-product of inputs), "match" (compute only where input keys from both sources match), and "cogroup" (missed this one). Building more complex second-order functions allows for less user code.

Next steps for the projcet are more input contracts, flexible checkpointing and recovery, and robust and adaptive execution, with a goal of going open-source by the end of 2010.

Sqoop - Database Import and Export for Hadoop - Aaron Kimball

Aaron (Cloudera) set the stage with a quick run-down of the limits of the SQL world, and the plusses and minuses of Hadoop, which lead to the introduction of Sqoop (SQL in Hadoop).

Sqoop provides a suite of tools to connect Hadoop to a JDBC-compliant SQL database, extract data and schema information, import the data into Hadoop, auto-generate code to parse the data, and export any results back into the SQL database.

The goal is to make it easier to pull SQL-hosted data into your Hadoop-cluster for the purpose of having the data available while doing other processing. For example, clickstream data might be in Hadoop, while profile information is in SQL. With Sqoop, you can get the data into Hadoop in an efficient way to support analysis. Copying the data from SQL in one operation is better than repeatedly hitting the database while running analysis because a big Hadoop cluster can easily hose a SQL machine.

Sqoop has some complexity under the hood:

It can export data definitions to Hive (see writeup next.)
It reads from/write to SQL in parellel.
You can use a SELECT/WHERE query to get data, which allows you to run Sqoop incrementally, fetching new data since the last run.
Supports mysqldump and mysqlexport.

Hive: SQL for Hadoop - Sarah Sproehnle

Sarah (Cloudera) described Hive, a parser, optimizer, compiler, and shell for transforming SQL-like queries into Map/Reduce. With Hive, you think of your data as being in tables rather than files, so you create tables, load data from a local file or Hadoop file into the table, and can then run SQL-like queries.

(I used the word "SQL-like" above, but Hive queries are actually standards compliant SQL, with just a few limitations/twists. Anyone who knows SQL at any level can pick up the changes in just a few minutes.)

In other words, with Hive you can:

CREATE and DESCRIBE tables, and ADD and DROP columns on tables.
EXPLAIN queries.
Query Hadoop data using SELECT, TOP, FROM, JOIN, WHERE, GROUP BY, and ORDER BY.
Write Hadoop data using INSERT (though the insert actually means "clobber the old data and replace it with this new data")
In a twist on standard SQL, run a multi-table INSERT, where you split a single SELECT/FROM into multiple output streams, allowing you to write different columns to different tables. You can also, further filter, group, or transform the data in each stream independently before writing to the final table.
Run data through a custom shell script, expecting lines of data on stdin, results on stdout.
Partition and bucket data, allowing for easy ways to drop a subset of data, or take a sampling of data.

Hive gives you the convenience of SQL, but at the end of the day it's still running as a Map/Reduce job on Hadoop, which means:

No transactions.
Latencies measured in minutes, not milliseconds.
No indexes, think of everything as a full-table scan.

Not surprising, and not bad considering you can run a SQL query across Petabytes of data.

The Hive install is installed on the client, so you don't need to do anything to the Hadoop cluster to run it. Hive keeps schema information in a Metastore, which can be kept on the local machine without any special configuration, or shared in a central repository allowing multiple users to share Hive table definitions. The schema is verified at data read time, not when the schema is created. Again, this makes sense given Hadoop's execution model.

1,000 points to Sarah for running a live demo during the presentation. Gutsy, but always a crowd pleaser.

Talks I Wished I Had Attended

The conference schedule today had two tracks, so there were a number of talks I was not able to attend. I would have liked to see the talks below, and look forward to the conference video:

Hadoop: An Industry Perspective - Aaron Kimball
Behemith: A Hadoop-based platform for large scale document processing - Julian Nioche
Introduction to Collaborative Filtering using Mahout - Frank Scholten

Closing Session

Isabel Drost, Jan Lehnardt, and Simon Willnauer kept the wrap-up short, thanking the other organizers, the tech staff (who gave a quick, fun recap of network usage), the venue, the presenters, and the audience.

When Jan asked who wanted to go to BerlinBuzzwords 2011 next year, every hand in the room shot up.

Final Thoughts

BerlinBuzzwords was an amazing conference. Half of the credit goes to the organizers for picking a great venue and interesting presenters. The other half goes to the largely German/European audience, who, 99% of the time, were focused on the presentation with laptops closed and (often) paper notepads open. This level of engagement lead to great questions from the audience after each presentation, and lots of hallway interaction. Sign me up for next year!

Berlin Buzzwords Recap, Day 1

BerlinBuzzwords has a stellar venue and talks describing cutting edge developments on all things search, scalability, and storage. Day 1 is below, click here for Day 2.

Please email me with any corrections.

Day 1 - June 7, 2010

The conference check-in was seamless, with much swag including messenger bags and notebooks. The venue itself--Kosmos Club--was amazing. Kosmos Club was the biggest movie house in East Germany before the fall of the wall, and has since turned into an event venue. Lots of metallics, varied textures, and swank chandeliers, with two of the biggest presentation rooms I've ever seen (they used to be movie theatres.) Check out pictures below to see what I mean.

Isabel Drost, Jan Lehnardt, and Simon Willnauer kept the opening remarks light and intimate. Overall, there were about 350 people in attendance.

Keynote: Grant Ingersoll from Lucid Imagination

Grant focused his talk around the words "Open", "Scalable", and "Intelligent". He described how a number of things, such as big-data storage, search, and distributed computing have become commodities, but required a staff of Ph.D. level employees just a few years ago.

The main point of his talk is that openness and open-source, plus scalability, have turned these things into commodities, and that the next big, interesting thing to work on is data intelligence.

Data (produced per year, I believe) has grown from 161 exabytes in 2006 to ~1000 exabytes in 2010.
85% of data we produce is unstructured, where unstructured may mean that we just aren't yet smart enough to parse the data.

There are multiple levels to intelligence:

Level 1 - Finding, organizing, discovering and associating data
Level 2 - Collecting and personalizing data
Level 3 - Mining data for sentiment and semantics
Level 4 - Learning from data, extracting ideas
Level 5 - Reasoning about data

At this point in the talk, he switched to a description of Apache Mahout, a machine-learning engine. Mahout can do things such as recommendations, collaborative filtering, Bayesian analysis, Random Forests, discovery, and pattern matching. At some point, he mentioned things like Restricted Boltzmann Machines, Stachastic Gradient Dsecents, and Vector Machines as upcoming features.

The takeaway is that using Mahout, you can build open, scalable, intelligent apps right now. In practical terms, this means things like auto-suggest, auto-complete, content clustering, clickstream analysis, etc.

NoSQL: The Definitive Guide - Mathias Mayer

Mathias Meyer (Peritor Solutions, @roidrage) gave a very balanced view of the current state of NoSQL. Mathias gave history its proper respect, saying that much of what we view as "new" in NoSQL is actually older ideas in a prettier package. He mentioned things like Berkely DB (K/V store), Sybase (Column Store), Versant Object DB (Graph/Object Database), and Lotus Notes (Peer to Peer Document Database) as throwback examples.

Mathias said that relational databases tried to be a one-size-fits all solution, and that NoSQL is about removing constraints to speed up performance and development.

Mathias himself is a fan of CouchDB, Redis, and Riak, and wisely avoided giving specific recommendations on what projects someone should use. Each datastore handles different use cases, so use what is right for your data.

Mathias briefly touched on Voldemort, Tokyo, Redis, S3, Scalaris, Couhdb, Riak, Mongo, BigTable, Cassandra, HBase, HyperTable, Core Data, Neo4J, and HyperGraphDB. (In order of mention.) He then gave a slightly more in depth view of the replication and scaling models of both CouchDB and Dynamo.

One of the key things that NoSQL gets right, he said, is being constructed of open web technologies such as JSON, HTTP, links, and textual protocals.

What is hard for NoSQL now? Range queries, ad-hoc queries, and transactions, mainly because the NoSQL space is focused on scalability as a major goal.

One of the big points that I'm glad he brought up is, "As a developer, how do I know I'm not wasting my time on a NoSQL solution?" The key is that each of the different NoSQL projects was built to solve a real-world problem, so trust that somebody found it useful and needed it built.

His main point: NoSQL is not the Holy Grail. NoSQL should not be about replacing SQL. Instead, you need to be okay with having polyglot data storage. The data itself should dictate the datastore.

Making Software for Humans - CouchDB and the Usable Peer-To-Peer Web - Jan Lehnhardt

Jan Lehnardt (@janl, Couch.io), after a brief introduction of himself and Couch, led off with the statement that 80% of all NoSQL projects do the same thing as flat files. It's the differences in the last 20% that really differentiate the projects. Therefore, NoSQL is about choice to build better systems. Each NoSQL project starts with a main idea.

According to Jan, CouchDB's main idea is being ready to scale up, in that each node functions independently, and also ready to scale down, in that Couch is a great candidate for running on embedded mobile phones, routers, and other devices, as a way of synchronizing user content.

CouchDB's synchronization is it's killer feature. With Couch, you can subscribe to events like data updates that send out HTTP based notifications to other parts of your application. As an example, the Couch.io team has built a chat service based solely on writing Couch objects and receiving updates.

Jan wrapped up by touching on projects like Opera Unite as ahead-of-the-parade examples of what CouchDB is aiming to do, and is already doing, in projects like UbuntuOne--P2P data synchronization. He mentioned Facebook as an example of a centralized, closed web, Flickr as a centralized, but open service (since you can pull your data out), and Diaspora as the open web that follows the true vision of Tim Berners-Lee.

Riak from Small to Large - Rusty Klophaus

In my talk (Rusty Klophaus, Basho Technologies), I gave a brief description of how Riak differentiates itself in the NoSQL market. It was built first with the operations folks in mind, which makes sense given the Akamai background of the core developers, who understand and embrace the uncertainties of a distributed system.

I then described which features of Riak become important in single-node Riak clusters, three-node clusters, and ten-plus node clusters. Just like different features of your car are important going 50 m.p.h. vs. 100 m.p.h., different features of Riak are important at different cluster sizes.

In single node clusters, Riak provides a simple data model, with key/value access, a variety of client libraries (Python, PHP, Java, Javascript, Erlang), and configurable replication settings and backing datastores.

In small-to-medium sized clusters, Riak provides a way to take advantage of hardware in parallel, with Javascript-based Map/Reduce, well-behaved HTTP (allowing easy placement of proxies and caches), and Google Protobuffs support.

And in large-clusters, Riak provides an extremely easy operations story that can survive server outages and network partitions, and scale out by just running a few commands.

Finally, I ended with a 5-minute run through of how to use the Python client API to read/write an object, run a linkwalking operation, and run a Javascript-based map/reduce operation.

Slides are available on Slideshare.

Realtime Search with Lucene - Michel Busch

Michael Busch from Twitter discussed some upcoming changes to Lucene that allow it to search on data that has not yet been committed to disk. From my understanding, when Lucene commits a change it creates a segment and possibly merges multiple segments together. At that point, a reader can access the newly created segment.

For real-time search, the process needed to be shortened. The first attempt simply involved syncing out the changes when a reader was created. This didn't work well. The next step was to actually search on the uncommitted index in memory. This was a challenge for a few reasons: first, Lucene uses multiple threads to update the index, so synchronizing those threads to provide the correct read-isolation is a problem. Second, the index maintains a large number of long lived objects in memory, and this causes inefficient garbage collection that kills performance.

Michael described a number of fixes that have already been written or are on their way, mostly around making multiple single-threaded index writers and changing the way postings are stored in memory which changes their structure from an unbounded number of objects to instead use a finite number of arrays. Effects on performance were amazing, especially for small memory sizes. A JVM with ~200 MB of RAM allocated was something like ~80% more performant.

A Twitter prototype with simultaneous indexes and queries showed that query performance, with the new modifications, is almost completely independent from query load, which is impressive. That said, the Twitter index has 32-bit postings (24 for DocID, 8 for term postition) so these results may not be the same for everyone.

ElasticSearch - Shay Banon

I unfortunately missed the first few minutes of Shay's talk on ElasticSearch. ElasticSearch automates the partitioning, sharding, and replication of documents into Lucene indexes, and provides a unified interface for searches.

Shay described the JSON model that ElasticSearch uses for API access, which includes everything from queries and filters to creating new indexes.

The ElasticSearch distribution model works by posting a JSON document with the new index definition. ElasticSearch automatically balances the shards across available nodes, and it sounded like this is done in a node-aware way, so that replicas are stored on different nodes if possible. At index or query time, you can hit any node, and the node itself is responsible for directing the operation to the right place(s).

ElasticSearch supports per-document consistency (in other words, no commit/flush support.) The most interesting thing, to me, is that ElasticSearch embraces the idea of transient storage. In other words it was written with the intent of running on something like EC2, where you can't trust local storage, and writing to remote storage is expensive, both computationally and monetarily.

To get around this potential bottleneck, ElasticSearch still supports reliable (or "somewhat reliable", if you want to split hairs) persistence by assuming that not all replicas containing a node will fail at the same time. New documents are lazily logged to the backing store, and if the master node doing the logging dies, then one of its slaves will detect the death and finish the logging. When a new node starts up, it reads from the backing store.

The other interesting thing is that ElasticSearch is aware of different cloud providers (it sounded like EC2 and Rackspace Cloud) and consult the cloud provider itself to get a list of potential nodes, allowing it to self-assemble.

Key differences between ElasticSearch and Solr? Distributed model, different API, no facet support yet.

Nutch as a Web Mining Platform - Andrzej Bialecki

Andrzej Bialecki (SIGRAM) described Nutch, a distributed web-crawler/search engine built on top of Lucene. It provides the standard framework of a distributed search engine that is extensible by plug-ins, and handles things such as URL filtering, normalizing, depth vs. breadth first crawling, etc.

The presentation was eye-opening just to see all of the things that make web-crawling difficult that are NOT about storing data and serving up queries. The Nutch team has spent a large amount of time going into things like what happens when you encounter auto-generated sites, buggy web-servers, link-spammers, or other tar-pits during a crawl.

Some common techniques for "bootstrapping" a web crawler are to start with some high quality seed sites, which may be well-known, authoritative resources, reference sites, or even the top-N results from an existing search engine like Google.

Once you have your search data, Andrzej described ways to mine the data, such as using keyword, phrase, or anchor search, using facets to find latent topics, using top-N results to prioritize future crawling, mining incoming links, treating the web as a corpus of sample textual data, associating concepts, uncoving gossip, opinions, and sentiments from data.

Nutch is currently under a redesign, attempting to share more code with common crawler libraries and other projects. Part of this will be converting data storage to a well-defined layer, allowing for pluggable backends so that users can take advantage of native data tools for those backends.

Riak Search - Rusty Klophaus

In my second talk, I discussed Riak Search, a distributed indexing and full-text search engine built on (and complementary to) Riak.

The tradeoffs are that Document-Partitioning generally has lower latency queries, but lower overall throughput due to it requiring a disk-seek on each partition. For this reason, Riak Search uses Term-Based partitioning, with some special optimizations using term-splitting, bloom filters, and result batching.

Slides: http://www.slideshare.net/rklophaus/riak-search-berlin-buzzwords-2010

Talks I Wished I Had Attended

The conference schedule today had two tracks, Search and NoSQL, plus I presented two talks, so there were a number of talks I was not able to attend. I would have liked to see the talks below, and look forward to the conference video:

Finite-State Queries in Lucene - Robert Muir
Text and metadata extraction with Apache Tika - Jukka Zitting
MetaCarta GeoSearch Toolkit for Solr - James Goodwin
The return of the Hierarchical Model - Jukka Zitting
Five cool problems you can solve with Neo4J - Peter Neubauer

Submodules and Subrepos Done Right

An approach to managing Git or Mercurial sub-repos easily, safely, and simply, while allowing you to embed Git projects in a Mercurial repo and vice-versa.

Background

Most software projects rely on other software projects to function. For example, Nitrogen depends on SimpleBridge, Coverize, and Mochiweb or Yaws. Riak depends on Webmachine and Mochiweb.

In the name of simplicity and ease of use, it's generally a good idea for the parent repo to contain the source code of any sub-projects it uses.

But then you are faced with a decision:

Should the parent project include the full history of the sub-project as well?

You currently have three options, all with tradeoffs:

Remove revision history in your sub-projects by deleting the .git/.hg directory, and experience pain when you want to pull the latest updates or commit a patch on the sub-project.
Track the entire .git/.hg directory for each sub-project, and accept that the .git/.hg directory of your parent project will now be huge.
Use Git submodules (or Mercurial subrepos), and hope that you never have to include a Git project inside of a Mercurial project, or vice versa. Also, accept the unfortunate fact that your build process now requires a working Internet connection.

The Search for a Better Way

What do I really want out of sub-repo support?

I want the sub-repos to seem like part of the parent project to the user, while still seeming like distinct repositories to me.
I want to be able to work with the full history of the sub-repo, but I don't want this included in the parent repo, or sent out to anyone who downloads the code.
I want an easy process for contributors to get the history of the sub-repos, so that they can commit patches.

Furthermore:

I only want to use tested, core features of Git or Mercurial.
I want the solution to be "cross platform", so that I can stick Git repos in Mercurial and vice versa.
I want the solution to be simple to use and easy to understand for the most common use case. (In other words, hide complexity from the non power-users.)

Introducing subgit and subhg

After much thought and frustration, I think I've finally found a solution that meets all of my needs. It lets me work with the parent repo as I normally would, using the git or hg command. Furthermore, it gives me a different command to work with the sub-repos. Finally, it is cross platform, allowing me to mix and match Git and Mercurial projects. The only downside is that a contributor needs to jump through an extra hoop or two in order to get the history of a sub-repo.

In practice, the solution looks like this:

Change to the directory of a sub-project...

> cd ParentProject/SubProject1

Operate on the parent project...

> git status
...status info...

> git commit
...commit code...

Operate on the sub-project. Notice the use of the 'subgit' command...

> subgit status
...sub-project status info...

> subgit commit
...commit sub-project code...

Best of all, the subgit and subhg commands are just thin wrappers around git and hg. Each is about 25 lines of shell script.

Installation

To try this out on your computer:

Save the following scripts to a location in your PATH: subgit and subhg. (Remember to run chmod 755 to make them executable.)
Create a global excludes file for both Git and Mercurial, and add .subgit and .subhg to it.

~/.gitconfig
```
[core]
excludesfile = "~/.gitignore"
```
~/.gitignore
```
.subgit
.subhg
```
~/.hgrc
```
[ui]
ignore=~/.hgignore
```
~/.hgignore
```
.subgit
.subhg
```

That's all!

How Does It Work?

The core concept behind this approach is to store the version history of your sub-project in a non-standard directory, and then use special wrapper scripts when you want Git or Mercurial to operate against that directory.

In other words, your projects won't have a .git or .hg directory. Instead, they will have a .subgit or .subhg directory, which is not tracked by the parent repo.

|--ParentProject    <-- a Git repository
   |
   |--.git
   |
   |--SubProject1   <-- a Git sub-repo
   |  |--.subgit    
   |
   |--SubProject2
   |  |--.subhg     <-- a Mercurial sub-repo
   |
   |--src

This tricks Git or Mercurial into tracking the files inside of your sub-repo, even though the files actually belong to a different repository. (Normally Git won't track a Git repo nested inside of another Git repo.)

The wrapper scripts--subgit and subhg--do the heavy lifting to make Git or Mercurial use the .subgit and .subhg directories.

subgit simply searches upward for the closest parent directory that contains a .subgit directory. Once found, it calls git, telling git to use the .subgit directory for repository information. subhg works the same way.

Usage

To create a sub-repo that can be managed with subgit or subhg:

Inside of an existing, clone a repository like normal:

git clone git://hostname.com/repository.git sub_project

Then, change to the new repository's directory and run subgit setup. This simply renames .git to .subgit:
```
cd sub_project
subgit setup
```

Now, test that it worked by viewing the parent's Git log and the sub-repo's git log:

git log
...print out log for the parent project...

subgit log
...print out log for the sub-project...

subhg works the same way.

Some Final Thoughts

First, this approach is intended for all of the projects out there using GitHub, BitBucket, Google Code, etc. as their main distribution channel. Most of these projects have a small group of contributors, and a much larger group of users.

If you distribute your project via a tar'd, gzip'd file, then this blog post is not for you.

Second, in order for other contributors to submit patches to the sub-project code, they will first need to obtain the full history of the sub-project. (Which makes sense, because the whole point of this was to NOT transfer the full history during a clone.)

As far as I know, the best approach to get the history is to just pull it from the sub-project's remote URL into a tmp directory:

git clone git://hostname.com/repository.git tmp
mv tmp/.git sub_project/.subgit
rm -rf tmp

-or-

hg clone http://hostname.com/repo/path/ tmp
mv tmp/.hg sub_project/.subhg
rm-rf tmp

Then, switch to the sub_project directory and checkout the right version. (This assumes your sub-project is in a directory named sub_project.)

Downloads:

Source code for subgit.
Source code for subhg.

Nitrogen/Riak Video from EUC 2009 Stockholm

The fine folks over at Erlang Solutions, Ltd. just released the video of my talk "Nitrogen and Riak by Example" from Erlang User Conference 2009 in Stockholm.

Nitrogen, Riak, and 1,000 Lines of Erlang

UPDATE: See the video of my talk on Nitrogen, Riak, and SlideBlast.com.

Check out SlideBlast.com, a tool I created that lets you share and control a slide presentation on the web. SlideBlast was built using Nitrogen and Riak, and is an example of exactly how much you can do with the right tools and 1,000 lines of code. (Ok, it's more like 1,130 lines, but who's counting?)

The full source code is available on GitHub: http://github.com/rklophaus/SlideBlast

How it Works

Upload a .pdf or .zip file.
Share a link with your remote audience.
Start presenting. As you flip through slides, your attendees' slides change, too.

Components

SlideBlast runs on Erlang and uses the following projects:

Nitrogen to create a slick, Comet-based user interface.
Riak to store slide show data (including the images) with a flexible schema.
Mochiweb as the underlying HTTP server.
Ghostscript to split and convert .pdf files into images.
Imagemagick to create image thumbnails.
SyntaxHighlighter to transform uploaded code files into beautiful HTML.
(Try uploading an .erl, .cs, .cpp, .js, .java, .sh, or .sql file!)

Why I Built SlideBlast.com

I built SlideBlast for my talk Nitrogen and Riak by Example, presented at the Erlang User Conference 2009 in Stockholm, Sweden. In the presentation, I briefly cover both Nitrogen and Riak, and then describe some of the techniques used to build SlideBlast. The video should be online soon, check it out.

The Bilski Case and Software Patents

Something happened in the second week of November that could forever change the face of the software industry. (No, I'm not referring to the release of Go Lang.)

On November 9th, the Supreme Court heard oral arguments in re Bilski. (In re Bilski means "in the matter of Bilski." I looked it up for you.)

The outcome of this case determines the future of software patents.

What's a "Bilski"?

Back in 1997, Bilski and Warsaw filed a patent for managing risk in commodities trading through hedging. Using their process, for example, an oil company would be able to offer a locked-in rate to customers, and do some fancy purchasing behind the scenes to cover their asses so that if oil prices spiked, they would still turn a reasonable profit.

The US Patent Office (USPTO) denied the patent because it did not pass the "machine-or-transformation" test. This test says that for a business process to be patentable, it must either be implemented with a particular machine designed or adapted to carry out the process, OR it must transform/reduce an article into a different state.

The problem with the USPTO's decision is that it contradicted precedent set by the Federal Circuit Court.

"Ruh Roh"

Ruh Roh indeed, Scoob. The Federal Circuit Court previously held in State Street, that an invention is patentable if "it produces a useful, concrete and tangible result" and applies to a specific application, tests that Bilski's patent should have passed.

The Bilski team appealed to the Board of Patent Appeals, with no luck. So they appealed again to the Federal Circuit Court, who you would think would quickly overrule the Board's decision, citing State Street. The Circuit Court, however, agreed with the USPTO, saying in effect that they were wrong before, that the State Street ruling no longer applied, and that the machine-or-transformation test is now the one true test.

Interestingly, the Circuit Court included language in the decision stating more or less "holy crap guys, ever since the State Street decision we've been overwhelmed with business process patents, these damn patents are getting more and more abstract, and we need to streamline this process lest we get buried in paperwork."

When all else fails, talk to "Los Jefes"

Bilski appealed again, to the Supreme Court, which granted certiorari in June. (Certiorari means that they would hear the case. Again, I looked it up for you.) The surprisingly readable oral arguments occurred on November 9th, which brings us to the present day.

(See a timeline of these events.)

So, how does this relate to software?

Software was seen as patentable as a business process because it produced a useful, concrete and tangible result. By discarding the State Street precedent, the Federal Circuit Court questioned the foundation upon which business process patents, and thus software patents, are built.

If the Supreme Court upholds the Federal Circuit Court's decision, it means that all existing US software patents will be very, very suspect, and RMS will be a happy, happy man.

If the Supreme Court completely overrules the Federal Circuit Court's decision, it will mean we keep the status quo, a continuation of the muddy quagmire that surrounds software patents today.

This is a black-and-white summary of the outcomes, of course. It is likely that the Supreme Court will shoot for the gray area in between.

Don't Forget The Economy

As exciting as it may be, the Supreme Court won't shake things up too much. Technology is one of the few industries in the United States that hasn't been gutted by the recession. At best, sweeping away software patents will plant some serious concerns in the minds of investors. At worst, their concerns will be justified. I suspect that of all of the factors the Supreme Court considers, the health of the economy will be what guides them the most.

It's Broke, Fix It

Rather than rehash all of the arguments for and against software patents, I'll provide you with a Google search for software patent debate.

To be honest, I'm still not sure which side I support on the issue. I do know that software patents--in their current form--are broken, because:

Software patents alone don't do much. You need a patent plus gobs of money.
A good chunk of innovation in software industry comes from teams of smart people at startup companies.
Startup companies don't have gobs of money, and if they did, they'd prefer to spend it building things, not fiddling with patents.

Can't wait to hear what the Supreme Court decides.

GProc - Erlang Global Process Registry

First blog post from my new position at Basho Technologies, home of Riak, a fast, flexible, scalable, operationally-easy data store written in Erlang. In this post, I examine gproc, an improved global process registry written in 2007 by Ulf Wiger, for possible use by Riak and Nitrogen.

The Erlang Process Registry

Very often in Erlang, you want to create a process and give it a name. This allows you to send messages to that process from other processes without passing the Pid around. Erlang supports named processes using erlang:register(Name, Pid), but there are limitations.

The Name must be an atom.
A process can only have one, and a name can be associated with only one process.
The local and global registries are separate. (A process is registered globally via global:register_name(Name, Pid).)
There can be a lag between when a process is registered, and when it can receive messages under its registered name. ie: calling "register(myprocess, self())", and then immediately calling "myprocess ! message", can cause an error.

These limitations constrain the way Erlang developers architect and debug applications. With a more robust process registry capable of storing additional metadata about each registered process, a developer can better decouple the responsibilities of an application, or gather information about running processes for debugging or reporting purposes.

Some concrete examples:

Riak spins off a new process for each get, put, and delete operation. There is currently no way to tell how many data operations are running at any one time, or to look inside a process once it is running to see what bucket it will modify. Ideally, this could be attached as metadata to the process, and a separate monitoring application could fetch this data and display it on the Riak Web dashboard.

Nitrogen spins off a process for a user's session. Ideally, on subsequent requests Nitrogen could look up the session process in the registry based on the user's cookie. For now, a separate lookup structure is used.

GProc

Ulf Wiger, back in his days at Ericcson (he is currently the CTO of Erlang Training and Consulting, noticed that he and other engineers were repeatedly (re)creating their own process registry structures in the form of lists and lookup tables as a way of working around the limitations described above.

To solve the problem in a more general fashion, Ulf created gproc. (Well, he first created proc, and then--with the insight he gained--created gproc.)

gproc lets you:

associate one or more names with a process, where a name can be any Erlang term
attach one or more arbitrary properties to a process, where the key or value can be any Erlang term
associate a counter with a process (with an optional rollup to aggregate counters across multiple processes), where the process name can be any Erlang term
retrieve any of these keys and values using ETS style wildcard matching

gproc can be confusing at first because naming processes, applying properties, and creating counters are all accomplished using gproc:reg(GProcKey, Value). This was done in the name of speed, consistency, code-reuse -- all of these operations store data in the same ETS table, and so gproc makes functions double up.

Furthermore, to ensure thread safety, a process can only set it's own values. To put it another way, it's as if erlang:register(Name, Pid) became erlang:register(Name) and can only act on the current process.

Finally, it helps to think of "registering" a process as instead "the process sets a globally unique key". Similarly, "setting a property" means "the process sets a non-unique key".

You'll see what I mean in the examples below.

Start a shell with GProc

To get a gproc playground, run the following:

    svn checkout http://svn.ulf.wiger.net/gproc/branches/experimental-0906/gproc
    cd gproc
    erlc -I include -o ebin src/*.erl
    erl -pa ebin
    &gt; application:start(sasl).
    &gt; application:start(gproc).

Storing a value

So let's play. To store a value in gproc, call "gproc:reg({n, l, Key}, Value)"

Again, both Key and Value can be any Erlang term. With a type of n (for name) you can only associate one value with a key. Try it again, and gproc will throw an exception.

Removing a Value

To remove something from gproc, call "gproc:unreg({n, l, Key})". Alternatively, kill the process. gproc monitors all processes that have stored values. When a process dies, gproc automatically removes any values associated with that process. This can lead to some confusion when testing gproc in the Erlang shell. If you store some values in gproc, and then run a gproc operation that throws an exception, Erlang restarts your shell process, so you lose any gproc values associated with that process.

Though this can be confusing in the shell, this is exactly the behaviour we want from a process registry.

Retrieving a Value

To get your stored value, call "gproc:get_value({n, l, Key})". Note that this will only retrieve values stored by the current process. You can also use "gproc:get_value({n, l, Key}, Pid)" to get values from another pid.

Who put the cookie in the cookie jar?

In addition to your value, gproc stores the Pid responsible for the value. You can retrieve this by calling "gproc:lookup_pid({n, l, Key})" or "gproc:lookup_pids({p, l, Key})". (You use the former when working with registered properties and aggregated counters, and the latter when working with properties and local counters.)

Taking a step back

So you are probably wondering about the significance of the 3-tuple that we are passing to these functions. This tuple, of the form {Type, Scope, Key} tells gproc a few things about your value:

Type is either 'n', 'p', 'c', or 'a' (n = name, p = property, c = counter, a = aggregate counter)
Scope is either 'l' or 'g' ('l' = local, 'g' = global)
Key is any Erlang term.

So far, we have been using a type of 'n', a globally unique key, which gives us a good mechanism for uniquely naming a process. This is similar to erlang:register/2.

Specifying 'p', for property, lets us set a key/value pair for a process, but other processes can also set a name and value of their own.

Specifying 'c', for counter, lets you set and increment a local counter for a process.

And specifying 'a' for aggregated counter, lets you access the sum of all local counters by the same name.

Note: we're always using 'l' for local scope. See the 'G is for Global' note below on why I haven't written anything about global scope.

Registering a Process

To register a process in gproc, call "gproc:reg({n, l, Key}, ignored)". The 'ignored' atom is irrelevant here, but could be used to store some extra metadata about the process.

Now you can use:

gproc:lookup_pid({n, l, Key}) or gproc:where({n, l, Key}) to return the Pid of the process.
gproc:send({n, l, Key}, Msg) to send a message to the process.

These functions are also available for registered processes, but less useful:

gproc:get_value({n, l, Key}) gets the value from the process that set it.
gproc:select/1 to gets the value from any process (more below).

Setting a Property

To set a property, call "gproc:reg({p, l, Key}, Value)".

Now, you can use:

gproc:get_value({p, l, Key}) to get the value from the process that set it.
gproc:select/1 to get the value from any process (more below).
gproc:lookup_pids({p, l, Key}) to return a list of processes matching the key.
gproc:send{p, l, Key}) to send a message to all processes matching the key.

Creating a Local Counter

To create a local counter, call "gproc:reg({c, l, Key}, Integer)".

Now, you can use:

gproc:update_counter({c, l, Key}, Integer) to increment or decrement the counter.
gproc:select/1 to get the counter value from any process (more below).
gproc:lookup_pids({c, l, Key}) to return a list of processes matching the key.
gproc:send{c, l, Key}) to send a message to all processes matching the key.

Creating an Aggregated Counter

To create an aggregated counter, call "gproc:reg({a, l, Key}, Ignored)"

Now you can use:

gproc:get_value({a, l, Key}) to get the counter value from the process that set it.
gproc:select/1 to get the counter value from any process (more below).
gproc:lookup_pid({n, l, Key}) or gproc:where({n, l, Key}) to return the Pid of the process that set the counter.
gproc:send({n, l, Key}, Msg) to send a message to the process that set the counter.

It's Really a Shared Dictionary

As you can see, gproc is really a shared dictionary with some limitations on who can store what when. Again, this leads to some confusion at first, but means that gproc can be applied to a wider variety of problems.

gproc:select/1

So far, we've glossed over gproc:select/1, and for good reason: it's complicated. But, it's also where gproc derives most of its power, so it pays to understand it.

With gproc:select/1, you can query the gproc process registry using roughly the same semantics as an ETS MatchSpec. Here is a quick (and far from complete) tutorial:

The basic form is gproc:select(Scope, MatchSpec).

Scope can either be 'all', 'names', 'props', 'counters', or 'aggr_counters'. MatchSpec is of the form [{MatchHead, Guard, Result}]. (Yes, it must be wrapped within a list.)

The MatchHead is a 3-tuple of the form {GProcKey, Pid, Value}. GProcKey is the 3-tuple key we've seen earlier, ie: "{n, l, Key}". By leaving an empty Guard ([]) for now, and using a catch-all Result (['$$']), we can already do some interesting things.

This will pull out every record in the process registry:

    MatchHead = '_',
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

This will pull out every record where value == value1:

    MatchHead = {'_', '_', value1},
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

You can specify a more complicated keys. This example will pull out any entry whose key would match "{myrecord, _}"

    Key = {myrecord, '_'},
    GProcKey = {'_', '_', Key}
    MatchHead = {GProcKey, '_', '_'},
    Guard = [],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

MatchHead can also use '$1' variables, which lets you start using guards to shape your select. For example, this will pull out any gproc entry whose key is a list:

    GProcKey = {'_', '_', '$1'},
    MatchHead = {GProcKey, '_', '_'},
    Guard = [{is_list, '$1'}],
    Result = ['$$'],
    gproc:select([{MatchHead, Guard, Result}]).

See this gist for more examples. http://gist.github.com/188032.

Normally, you would be able to use the Result property of the MatchSpec to select out a subset of the data you want to capture. In gproc this is slightly broken. You can choose which result you want to pull out by setting result to a '$1' variable (for example, ['$1']), but currently you can only pull out one result from each match. In other words, ['$1', '$2'] does not work. (It only returns '$2'.)

GProc and QLC

It appears that GProc began to have support for QLC, which is pretty friggin' sweet, but unfortunately I could not seem to make it work.

G is for Global

GProc automatically detects whether gen_leader is available on the system. If it is, then gproc will run in "global mode" meaning that all of the functions above work across your entire Erlang cluster rather than on a single node, which is also pretty friggin' sweet. Unfortunately, gproc currently uses an obsolete version of gen_leader, so running gproc in global mode is not recommended.

Evaluation

GProc is extremely powerful, but the interface could use simplification. Convenience methods around gproc:reg/2 specifically shaped for registering processes would go far to help smooth the learning curve for newcomers.

In addition, gproc:select/1 has a few small bugs. The MatchSpec must be passed as a list, even though it will only work with one MatchSpec, and the Result portion of the MatchSpec can only be used with one '$N' variable.

Update from Ulf Wiger:

Gproc has now been moved to Github: http://github.com/uwiger/gproc.

I have added some functions, such as gproc:await(UniqueName, Timeout), which will suspend until the name key UniqueName becomes available, and then return a {Pid, Value} tuple representing the registration info. One way to use this is as a simple resource broker, or to manage dependencies during system start.

I have done some initial testing of the global parts of gproc. They seem to work. I used the hanssv+sergeversion at http://github.com/uwiger/genleaderrevival in which I have made small changes. The other genleader versions should work as well, but I haven't tested them. See gprocdist:elected/[2,3] to understand the differences (the hanssv+sergeversion calls elected/3, while the others call elected/2).

Nitrogen at the Erlang Factory Across the Pond

In less than a month I'll be heading out to London to speak about http://nitrogenproject.com at the http://erlang-factory.com/conference/London2009. I've booked a little extra time before and after the trip to go sip some tea, see Big Ben, high five the Queen, and http://news.bbc.co.uk/1/hi/sci/tech/677048.stm.

In the meantime, Nitrogen is undergoing some big changes. The improvements were inspired by conversations and presentations at Erlang Factory San Francisco as well as http://www.brics.dk/~hosc/local/HOSC-12-3-pp221-236.pdf by Guy Steele.

The main goals are to:

Allow Nitrogen to cooperate with--rather than compete against--some of the other great work out there, including http://bitbucket.org/justin/webmachine/wiki/Home, http://www.erlang-web.org, http://erlyweb.org, etc.
Make it easier for the Nitrogen community to customize caching, sessions, state management, and many of the other things that get more complicated as a site grows.

Early prototypes are looking good. Hopefully I'll have some great stuff to show off by the time the 'Factory starts.

Nitrogen at the Erlang Factory

Very excited to be presenting at the Erlang Factory in Palo Alto, CA at the end of April. I will be giving an hour long talk about Nitrogen, a cutting-edge web framework that I created that is quickly building momentum in the Erlang community.

Mindsets and How to Predict the Future

Some things that are hard to learn. Good posture, for instance.

It usually goes like this: "Put your shoulders back, no wait, not that far back, now lift up your chin higher, but don't tilt it, your shoulders moved, keep them still, argh, forget it."

It's a lot easier to say, "Picture that you have an imaginary string attached to the top of your head. Now what would your body do if somebody pulled up on the string." The head automatically rises, the body falls in line, everything clicks.

It's fascinating to me that such a simple concept can aim and focus your thoughts in the right direction. That simple concept, or mindset, is a very powerful thing. Good posture is just a simple example. When applied to more complicated skills, the right mindset goes by many different names: "intuition", "going with your gut", "muscle memory", even "wisdom", and "genius".

We don't need to have as clear a picture as "a string tied to your head". Most of us have mindsets about everything at which we excel, we just haven't thought about defining them. Think of something that you do well, and now think of the last time you tried to teach it to somebody. You can quickly see their mistakes; it's like you have an internal compass telling you whether they are pointed in the right direction. That compass is evidence that you have the right mindset, or at least a mindset.

Conversely, trying to learn a skill without the right mindset is very difficult. Without the right mindset to pull everything together, advice or direction doesn't help all that much because it doesn't fit into a bigger picture. It's like trying to learn a language by memorizing sounds without knowing the meaning behind the words. You may be reciting Spanish, but you aren't speaking it.

In Blink, Malcolm Gladwell talks about how the human brain switches to a different mode when faced with a decision having too many data points. Once your brain has had enough time to distill the information, something clicks deep inside, and suddenly your brain begins to steer you in the right direction. I think it's the same process that lets your brain distill information about a skill and develop a mindset to guide you in the right direction.

Wouldn't it be great if somebody took a macro view, conducted years of research, reduced the research down to the most potent mindsets about business and world events, and wrote about it?

Somebody has: John Naisbitt wrote the book Mind Set! in 2006. The book is revelational. John has made a career of discovering the right mindset to use when thinking of society and it's response to changing conditions, and he has successfully applied this to management, business, and politics to predict how today's developments will impact tomorrow's reality.

Mind Set! covers 11 mindsets that hold true on many levels. A few of my favorites chapters are:

Don't get so far ahead of the parade that people don't know you're in it

Resistance to change falls if benefits are real

Don't forget the ecology of technology

Planning Poker: Software estimation for fun and profit

Ever since hearing about Planning Poker (an excellent description is here: http://www.planningpoker.com/detail.html) I've wanted to try it out with my team. We started holding twice weekly planning poker meetings back in August 2007, and now, approximately 30 meetings later, we've established a good system.

We have adapted the rules slightly from what is described in the link above.

The process is:

15 minutes before the meeting begins, I set up a room so that Jira (our issue tracking software) displays the list of pending items on the wall through and a projector.

Anyone who is not engaged in some sort of urgent activity files in (usually between 5 and 8 people).

Everyone grabs a deck of cards, described below.

Starting with the first item, the person who knows the most about the item gives a brief description and calls out any 'gotchas' that may be involved. This takes 5 to 7 minutes.

Once discussion starts to die down, either I (or whoever is leading the meeting that day) or the person describing the item says, "Ready to vote?"

On three, everyone throws down a card from their deck of cards (Details below).

If the estimates are wildly different, then the two people with the lowest and highest estimates describe their thoughts, and we revote.

If the estimates are roughly the same, we'll settle on one and update Jira with the estimate.

After the meeting, I compile the estimates for all of the items into a project schedule, add some time for testing and deployment, and then run it by the project team for a "gut check". For example, "These estimates have us hitting Milestone 1 on Friday, Milestone 2 next Wednesday, and then finishing the project the Tuesday after that. Is that reasonable?"

The Deck of Cards

Each person has a deck of cards with '1 hour', '2 hours', '1/2 day', '1 day', '2 days', 'Breakdown', 'Needs Design'.

Each person votes by throwing down one card. We decided that we didn't need to get very granular in the task breakdown because who really wants to argue over whether something will take 10 or 11 hours.

'Breakdown' is used whenever you think something will take longer than two days. This signals that we need to break the item into sub-tasks and estimate them individually.

'Needs Design' is used whenever more information is needed before an item can be actionable.

Short Circuiting

We have invented two new rules to speed up the process.

Rule: The 'Needs Design' card can be played by anyone at any time without waiting for voting.

We put this in place because when an item needs design, discussions around it have a way of being very long and circular. This rule gives anyone in the meeting the power to recognize this and prevent a bunch of wasted time.

Rule: You can skip discussion of an item by saying, "I think this will take 2 hours, assign it to me."

The Benefits of Planning Poker

Planning Poker has a lot of obvious and non-obvious benefits:

Everyone has well-defined, scoped, immediately actionable items assigned to them (GTD-ers should see the value in this)

Schedules are more trustworthy

The discussions spread historical knowledge about the system

The discussions keep the team informed of changes happening to the system

Junior team members learn how to calibrate their estimates

All team members grow more confident in their estimates

The person doing the work has a voice in the estimate

The estimate feels fair, and the process is transparent

Because the estimate is fair, you feel healthy peer pressure to meet your schedule

The team gets a good sense for how difficult a task will be

When someone takes on a difficult task, the whole team knows it and appreciates it

Guards against "optimistic" estimates that are way to low

I've tried to gather software estimates in a few other ways that don't work as well. Each one has a few fatal flaws, for example:

Having one senior person estimate items for the whole team puts a lot of pressure on the senior person, and also leads to resentment if the estimates are too low.

Having a junior person estimate their own items usually results in the person either estimating wildly high and losing the confidence of their coworkers, or estimating wildly low and screwing themselves with an unrealistic schedule.

Function Points (http://www.functionpoints.com/) are complicated and time consuming.

A 10 Second Guide to Smoother Projects: Urgent vs. Important

A little background information: I've decided to leave my job to launch a technology start-up company. (Talk about big news buried in an unsuspecting article, eh?)

As a result, I've spent most of this week transitioning my responsibilities to someone else. The handoff feels somewhat like a death-bed situation. Not because I think it won't go well (I think it will go perfectly fine), but rather because I feel like I have a small window of time to think about and pass on the important advice I'd like to leave behind.

My position was Director of Software Development. This involved managing multiple teams responsible for new development and maintenance of a system with over one million lines of code. The single most important piece of knowledge I want to pass on to my successor is this:

Separate urgent items from important items.

Urgent items are those things that need to get done today or else the system will die. For example, if a client starts calling into the system with incorrectly formatted data, you urgently need to fix the situation, either by working with the client or adding code to accept the new format.

Important items are the longer term projects, for example: the new feature that increases profits by 10%, or the feature necessary for the big trade-show demo a month from now.

No matter how hard you try, there is no good way to prioritize one of these items against the other.

As a result, a team (or individual) can't effectively work in both "urgent item" mode and "important item" mode at the same time.

Over the last few months, by far the most effective processes I've put into place is to assign all urgent items to a dedicated team. This enabled our group to handle almost any emergency thrown at us without having to re-adjust long term schedules. This saved a bunch of time that we used to waste jostling things about in order to accomodate the urgent item.

This also makes it very clear who will work on the item. When an emergency arose, I immediately sent it to the Firefighting team; nobody else has to be distracted. The Firefighting team might need to shuffle some priorities around, but they are used to that, because that's the mode they work in all day.

I would imagine that as the group evolves, they will need to add a third type of team, resulting in:

A Firefighting team
Multiple Project teams
A Research team

The Research team works on technology gambles. The project may not work out, but if it does, then it will revolutionize the business. I would staff this team with senior people who are strong individual contributors who can investigate hazy ideas and make them reality.

Manager Points

I have a list I call "Manager Points". I keep it in a folder that I carry everywhere at work and refer to often (but not often enough.) It consists of short reminders to myself of knowledge pulled from books, blogs, and personal experience. It is constantly evolving, and it is not always gramatically correct, but here is a snapshot of it as of now:

Manager Points

Rely on group wisdom, believe that the team is smarter than the individual

Drive toward a common understanding of a problem. It will save time in the long run and prevent future roadblocks

You can't fake clarity on the desired end result

Look for responsibilities to give to people

Focus on What is right, not Who is right

Give people a chance to feel like big contributors, and praise people as a spokesman for the team. "Thank you for... The whole team appreciates..."

People are fitter, happier, and more productive when they claim work, rather than you assigning it to them. The people we hire are motivated, they will claim work hard if given the chance and shown appreciation

All meetings need a purpose and agenda

Welcome smart ideas, don't be afraid of someone doing "your job". This is how people grow, and it's always productive if working toward a common goal

Encourage good / difficult communication. There is no substitute for a face-to-face conversation

To make sure that difficult (ie: painful) conversation happens when it needs to, make communication routine

Stay calm, look at challenges as a change in circumstances. Focus on adapting

Look for what motivates people - everyone is different - Challenge, Attention, Excitement, Friends, Money, Deadlines

Everyone has ups and downs. Learning new things is hard. Be patient.

Look for opportunities for change, excitement, and learning.

Think often about what you could be overlooking and what you should be preparing for

Think just as often about what you are learning and what you have done correctly

Hold people to a schedule by asking questions, not making demands

Your main responsibility is no longer "you". It is now your team. If they do not learn and advance in their careers throughout the project, you are not doing your job.

The Klophaus Equation of Bug Difficulty

The difficulty of a bug can be measured as the distance, in lines of code, from the cause of a bug to the visible symptom of a bug.

Good developers know this equation intuitively and find a way to whittle down the playpen of a bug. When coding, they create simple and easily understood interfaces between code so that the bugs are trapped in small playpens from the start. When debugging, they create experiments to trap the bug into smaller and smaller snippets of code. They get turned on by Object Oriented Programming, Service Oriented Architecture, Inversion of Control, Plug-Ins, and other things that create walls between code.

In the easiest case, the symptom and cause of the bug are on the same line of code. In the hard case, the location of the cause is unknowable, making the bug infinitely hard to solve. The hard case happens when the system is non-deterministic or random. This could be caused by problems internal to the system, such as a threading race condition or deadlock situation. It can also be caused by problems outside of the system, such as faulty hardware, or an unstable network, or by a bug in a third party component or the underlying framework or operating system.

An unknowable cause is very different from an unknown cause, and this is why a good QA team helps. A good QA team will help leverage a good developer by telling the developer exactly what inputs or actions cause the bug--quickly turning the cause from unknowable to unknown--and providing the developer with all of the information necessary to nail down the line of code causing the bug. A bad QA team will create bugs so vague that the cause is impossible to find, ie: "The application crashed."

I'm still undecided as to whether the Klophaus Equation of Bug Difficulty should take into account the amount of code required to fix the bug. I don't think it matters, and here is why. The difficult part of debugging is understanding what is going wrong and how it should be corrected. In most cases the fix, even for the hardest of bugs, only involves a few lines of code. This is because most bugs are typos--the developer meant one thing but typed another.

Bugs that require a large amount of code to fix are usually no longer bugs, they are tasks. They are the signs that the software's design was incomplete, that scope changed, that the developer didn't fully understand the functionality he was developing, that the developer cut corners, or that a code merge somewhere went horribly awry. In other words, they are an indication of a glitch in the software lifecycle, a bug in the process or the team.

And a bug in the process or the team is the same as any other bug... the difficulty of a bug can be measured as the distance from the cause of a bug to the visible symptom of a bug. But causes and symptoms are more difficult to pin down in processes or teams than in code.

Resumes: Don't Move Bricks, Build a Cathedral

There is a story about a traveler who passes three men moving bricks at a construction site. The traveler asks the first worker, "What are you doing?" The worker says, "I'm moving bricks." The traveler asks the second worker the same question, "What are you doing?" The second worker replies, "I'm building a wall." The traveler asks the third man the same question. "I," the third man replies, "am building a cathedral."

I reviewed a resume for a friend that described one of his contributions as "extracting and parsing program output from XML files." Technically, this is what he did, but he's describing his work in terms of the first or second man.

When you write a resume, don't just write what you did. Also write how it helped the company from a business standpoint. The real point of "extracting and parsing program output" was to support a company initiative to port one of their best selling products to a Windows platform. The new software was expected to drive millions of dollars in sales to new clients. This sounds much better.

I have to be honest; I grapple with giving this advice because on the surface it looks inflationary. It seems like "spin." But I think the benefits go beyond just making your resume look better. I think that a resume written in this style also benefits the company doing the hiring for three reasons:

Some people make a career of avoiding important projects and doing "sidelines" work. They won't have much to write. The best people-the clutch people who seek out important projects- will have a lot to write.

It is a valuable skill to be able to effectively communicate how contributions fit into the strategic objectives of a company. Anybody who writes their resume in this way shows that they are a better hire just by the act of doing it.

Hopefully, this sort of high level thinking carries over into other responsibilities long after the resume is written. If you are thinking in terms of cathedrals, then you are looking for risks, opportunities, problems, and solutions on a larger scale.

My second bit of advice is to include specifics about your contributions. Was the project big? How many man hours in total? How much revenue did it generate? What was the budget? How many people?

Vagueness, on a resume, looks like you made stuff up. Being specific is the best way to add credibility to your resume. This is especially important because 95% of the time, someone is reading your resume when you aren't there to explain it.

Just a reminder: Use your judgement here, don't include any confidential information.

Three Common Flaws of Good Software Developers

Software developers are a smart bunch. Smarter than average, I'd venture. (Patting myself on the back, here, cause I am one.) Seriously though, here's my reasoning: computers are the most complex tools on earth, and we the software developers tell the computers what to do.

Despite being smart, however, we developers are sometimes frustrated in the business world. And I believe that, like almost all frustrations, this stems from feelings of being ineffective or misunderstood.

I think there is a simple explanation-a common set of flaws shaped by nature and environment-that explain why this is the case. The main theme is that we have developed one-dimensional work habits, and to solve this, we need to learn to think and work in different ways when the situation calls for it.

Here are our flaws:

We honed our craft by working alone.

We probably excelled in school, a system where self-promotion is, for the most part, unnecessary.

We expect consistency.

Flaw #1: We honed our craft by working alone.

Like mastering an instrument or writing a book, mastering the details of a computer requires a lot of time spent alone. As a result, we are trained to solve problems as follows:

Step 1 - Find a keyboard.

Step 2 - Work really hard.

This individualistic style of working isn't always a bad thing. Being able to work by yourself on a task for a long time is a valuable skill. It is a problem, however, if this is the _only_ way that you know how to work. Why?

Business applications generally require a team of people to create. If you can only work alone, it means that you don't trust a team.

Not all problems can be solved with a computer. This applies to even the most high-tech projects.

Too much "holing up" to solve problems does not exactly reinforce positive impressions to your coworkers. What to you may feel like dogged determination can look like avoidance or absentee-ism to everyone else.

Flaw #2: We probably excelled in school, a system where self-promotion is, for the most part, unnecessary.

How does a teacher know which students are the good ones? Tests. Teachers have a fair, objective, emotion-free way of measuring effort and intelligence. From kindergarten through 12th grade, school is structured so that an all-seeing judge awards you the exact credit that your work deserves. In school, all that mattered were results. Now, after many years of being in a system that focuses 100% on results, software developers jarringly enter a system that prizes effective communication of results as much as the results themselves.

It's true that this is a transition through which every professional goes, but this transition is more difficult for software developers than other professions because the disparity between the work we produce and the results others see is large. Almost nobody sees the real work of a software developer except for other software developers. The hundreds of thousands of lines of code that form the results of a software developer's time are paramount to his small circle of teammates, but they are gibberish to anyone outside of the circle.

What the rest of the business really cares about are the implications of the software developer's work. What problem have you solved? What money have you saved? What problem will you solve next, how soon, and how certain are you that you will be able to solve it?

This flaw is painfully obvious to anyone who has left a conversation with a software developer thinking, "Are we talking about the same project? I'm more confused now than before." For software developers in a business world, it's no longer enough to study hard, keep your head down, and quietly put out good work, because the work is hidden. Whether you call it self-promotion or just effective communication, a software developer has to actively talk about their work in the context of the rest of the business.

Flaw #3: We expect consistency.

You never have to convince or persuade a computer. It just does exactly what you tell it. Unless software is buggy, it should be deterministic: given the same input, it should function the same way every time. In fact, being deterministic is pretty much the main job of a computer.

As a result, there is a "best" solution for almost any computer problem, a specific lever to pull that will return the optimal result. Being a good software developer, to a certain point, means that you know how to pull a lot of levers. We create libraries and invent patterns to apply to common problems. There are so many prescripted solutions available that it's considered foolish to invent one when there is already a well-known, pre-existing lever that solves problem.

People and teams, on the other hand, are inconsistent. Morale and mood are constantly swinging in orbit around every person and every team, propelled by accomplishments, criticism, and praise, and affecting happiness and productivity. Personal lives are an inestimable and uncontrollable line item on every project. With people, an effective approach one day can be useless the next.

For software developers who count on deterministic behavior, this can feel like lunacy.

The key is to understand that not everything is as precise as a computer. With people, the levers are less like levers and more like rudders. Pulling the lever won't solve your problem, but (hopefully) it will at least point you a little more in the right direction. So the real work, and the fun, is to constantly try new approaches to solving the problems that you have. Maybe you will stumble across the right one, but more likely, your solution requires pulling a bunch of them and seeing what works in that moment.

Career Fair Advice

Part 1 - Introduction

College career fair (and interviewing) advice that is more sophisticated than the standard "dress well" and "shake hands firmly."

If you know any teachers, observe their first week back in school. It will likely be easy-- after school you will find them slumped in a chair, eyes glazed, feet sore, hoarse, and exhausted. It's a hard job-- an eight hour performance-- and you set the tone. Being a recruiter at a career fair is comparable to teaching.

For the past three years, twice a year, I have been very involved with my company's on campus recruiting. Every year it is a mix of excitement, anticipation, and utter exhaustion. And my daily office routine does very little to prepare me for the workout.

Yes, as recruiters we are the first step in the employment journey. I understand this, so I strive for perfection: high energy, perfect recall of every candidate, etc. But, it is impossible. The typical day at a career fair is full of dozens of conversations. Only the best ones stand out.

This series of articles is both a glimpse into what I look for as a recruiter, and a guide to what works and what doesn't. I have gathered and refined this advice over the last three years of recruiting. I write it down now because, honestly, I wish I had read something like this when I started my job search.

Some background about me. Out of college, I worked for an international pharmaceutical company for nine months, and then left to start a consulting company with two college friends. About three years later, our consulting company had grown to 20 people and was acquired by our largest client, an internet marketing company that is turning heads in the industry. I work in the Reston, VA office where we develop the software and data analysis tools that drive the company. As a result, my recruiting is focused on CS majors for software development and business analyst positions. Adapt this to your own pursuits accordingly.

Part 2 - We Are Keeping Score

The only sane way to organize the myriad resumes that we gather at a career fair is to score them. Every company has a different method of scoring resumes.

We use a five point scale, where five is the highest. A five is the candidate with the right mix of strong academic performance, real work experience, strong communication skills, a dash of wit, and a sense of modesty. A one is missing a few of these things in a big way. A zero is reserved for the guy who decided to come to the career fair in bare feet. (Ah, Carnegie Mellon, you produce some odd creatures.)

After each conversation, I write the candidate's score and my initials on the front of the resume, and, on the back, I write a summary of our conversation to help me remember the candidate.

At the end of the career fair, my team takes the stack of resumes to the nearest restaurant. We sort the resumes over dinner, focusing on the fours and fives. If we have enough fours and fives, then we don't ever get to the ones, twos, and threes. Usually, the goal is to narrow down the candidates to the top 10 or 15 whom we will then invite to interview with us.

I admit that sorting and dissecting the resumes like this may sound dry and clinical. What if we miss that hidden gem? I will say three things in the system's defense:

It's fair, everyone gets an equal shot to be a 4 or 5.
It lets my company find the best candidates without having to interview everybody.
It's better than a lot of other companies.

There's an interesting follow-up to that last item. Before you approach a company at a career fair take a look at how they manage resumes. The good recruiters will have some sort of system for recording the information that they get out of your conversation. The bad recruiters, usually the ones at companies that are in super high demand, will just serve as the frontmen to a resume drop. Usually, in this case, the people doing the recruiting are not the people lining up the interviews. So basically they take your resume, drop it in the box, and never think of you again.

If you see the latter situation, all is not lost. You can change your approach to attempt to be more memorable, or you can just save yourself a lot of time by skipping the conversation and dropping your resume into the box. Because if they aren't making any notes, there is a very good chance that they won't remember your conversation.

Part 3 - Be Concrete

Ad agencies, Hollywood directors, and product designers use mockups for a reason: it's a lot easier to have a conversation about something than about nothing. When you've got something to talk about-even if it's not perfect- you have something to compare, contrast, and mold into shape. You have a starting point.

Remember this at the career fair. Before the fair, think about your ideal job. Be able to visualize it, and verbalize it. Some attributes for consideration:

Small / Medium / Large Company

In small companies you wear more hats. You learn mainly by doing things.

In large companies you have a more well-defined career path. Large companies have generally been around longer, so you can learn from coworkers with more experience.

In most cases, the company's business has a greater impact on job security than a company's size.

Initial Responsibilities

Application design, software development, testing, operations, strategy, sales.

Type of work

Generally, any paying software development job fits into one of three categories:

IT Department - You are helping others at your company achieve their business objectives. For example, you work at Target corporate head writing sales reports and HR applications. Generally, these jobs will have the smallest upside, but the best work/life balance, and the best opportunity to jump around to new jobs outside of computers.
Consulting Company - You are helping some other company achieve their business objectives. For example, you work at Accenture, your team is responsible for optimizing Target's inventory management system. Generally, these jobs have the longest hours (because consulting companies are in the business of selling people's time), but are high paying. You may also bounce around to many different clients, so it's sometimes difficult to develop a sense of unity with your coworkers.
Technology Company - You are the business objective of the company. For example, you work on Microsoft Office. These jobs usually require a slightly high but reasonable number of hours. If the company is small, it's probably high risk, but also high reword.

Industry Vertical

Some examples: technology and communication, financial services, retail, transportation, insurance, food and agriculture, healthcare, manufacturing, education, etc.

Geographic Location

North, south, east, west. City, suburbs.

To really impress a recruiter, get clear on your ideal job attributes, then use them as a framework for discussion at the career fair. Don't be scared to assert a preference that does not align with whatever company you are talking to. If you've always pictured yourself at a small company and you are talking to IBM, tell them you've always pictured yourself at a small company. Ask the recruiter why they chose a big company. Talk about why you are leaning toward a small company. Above all, convey an open mind. Just like a visual mockup, the idea is to make progress in your thinking, not to limit yourself to a specific idea.

Part 4 - Plan Effectively

I have known that a would work with computers since 4th grade. But even so, if you compared my effective career planning during college to the amount of planning I put into any given term paper, my career planning comes up short.

Notice I say "effective" career planning. I put a lot of effort into looking for a job, but not much time thinking about how or why I was looking for it. Honestly, it felt like when you're in a car on a trip, and you've possibly missed your turn, so you have to decide whether to keep going or turn around... and I just kept going.

I don't feel too bad now, because I see many candidates following the same strategy that I did. They know they need a job, but they don't know the right series of turns to get one. Drawing from the candidates who we have hired and their results on the job, I've noticed three common patterns that signal a problem:

The Shotgun Approach

Some candidates are so worried about missing an opportunity that they throw their resume at every single job out there, even if it goes against the grain of their major. This is foolish. If you studied chemical engineering, or romance languages and literature for the past four yours, you hopefully did it for some sort of reason. You may be smart, but are you REALLY sure that a career in software engineering is for you?

The Perpetual Student

Another common scenario is the candidate who has spent the last 20+ years in school, achieving degree after degree, spending every summer writing intense theoretical papers on quantum cryptography. I am impressed, and my field will always need pioneers to move things forward. But be honest- 20+ years in academia means that you love academia. Are you REALLY going to be happy working with us to make practical business applications?

The Misguided Programmer

The saddest, to me, is the Computer Science major who is in it for money. When you think of the dot-com boom, you immediately think of the founder of a start-up out in Silicon Valley who made 10 million dollars overnight. Because of these paper millionaires, the boom created the idea that programmers are rolling in easy money. Mothers started instructing their daughters, "marry a doctor, a lawyer, or a computer programmer!" (Or, "BECOME a doctor, a lawyer, or a computer programmer!") When it came time to choose a major, a bunch of kids who should have remained "undecided" instead marked the CS box.

One day, these misguided programmers will wake up to realize that they spent four years studying something that they will never be passionate about. When they look for a job, the only logical place to look is in software development. They are easy to spot: apart from the major and a list of classes, there is no other mention of computers or technology. In their list of computer skills, they write C, Database Design, and Javascript/HTML because they took Intro to Computer Science, Intro to Databases, and Intro to Web Design. No exploration or tinkering, nothing beyond what is required.

These are all intelligent people. If you described any of these candidates to themselves, if you asked them to write a critical paper comparing possible alternatives, I am confident that any one of them would be smart enough to navigate their future. But job market pressure has a way of clouding vision.

So the advice? As a recruiter, it is unbelievable rare to find someone who has examined their own interests, weighed the possibilities, and then fiercely pursued the opportunities that fit them the best. The candidates who do this are the ones I want to work with. In the end, these candidates will also be the happiest.

Part 5 - Two Questions

The whole point of resumes, career fairs, and interviews is to answer two questions:

What can you do for the company?
What can the company do for you?

Most people only think about the first question. It's understandable, you're at a career fair, you need a job, and you have a short amount of time to find one. But the second question is just as important as the first one. Recruiters are people, too, and they are probably not that different from you. If you guys went to the same school you may have been good friends. Heck, if you end up working at his company you may be good friends. The recruiter, if he's good, is trying to create a win/win situation, a situation where both of you are better off if you take the job.

So while you are chatting with the recruiter, try to get both questions answered. One of the best things to convey here is that you have all, or almost all, of the skills that the company needs, but that you have also gotten a little exposure to some other advanced skills. You are interested in developing these other skills, and something about that specific company would provide you with a great opportunity to do it. A knowledgeable candidate who's focused on his own career development? I'll take two!

A note about the first question:

Some individuals have attempted to answer this question, to unfortunate effect, by walking up to me, handing off a resume, and then reciting a prepared monologue, usually directed at my feet, about how their experience could help my company. There is really nowhere to go from here. The candidates usually have some impressive talents on paper, but they just exhibited a lack of social skills (talking at my feet?), communication skills (conversations involve two people), and business savvy (do they even know anything about my company?). This is not the way to go.

Part 6 - The College of ??

Let's talk about career fair physics. Because of career fair physics, five minutes researching a company before the career fair carries as much or more weight than 200 hours volunteering at the local hospital.

To understand why, I suggest flipping the script around. You would be unimpressed, and rightfully so, by a recruiter who was completely clueless about your school.

"University of Virginia? Hmm... never heard of it. Is it big? Is it any good? I've heard a lot about University of Phoenix. They have classes online. I saw them on a billboard. Is your school basically like them?"

It would be laughable. But that's what candidates regularly do at career fairs. The reason that five minutes is equal to 200 hours is because if a candidate can talk intelligently about your company it signifies that he or she is specifically interested in your company for reasons deeper than a paycheck.

Compound this with the fact that most recruiters-setting aside the ones who just wanted a free hotel room-are there for a reason. Either they have enough experience to pick out good candidates, or they are passionate enough about the company to want to represent it. In either case, the company is important to them, way more important than a paycheck. They want to find people who think the company is important, too, just like them.

Part 7 - Ask for the Job

When I rented my last apartment, there were so many potential tenants that I actually had to stand in line to fill out an application. It was toward the end of a scheduled open house, so I was the last one in the room when I finished writing.

After some small talk, I turned to the owner and said, "Chad, I really like this place. I just want to let you know how excited I am to possibly live here." He smiled, thought for a moment, glanced at the stack of other applications, and said, "If your credit report checks out, then it's yours."

When my company was young, we had a senior adviser who had the same advice for us again and again: "You significantly increase the chances of getting what you want if you ask for it."

Part of being human is that something compels us to help our fellow man. We pull together. Most of us, at a deep level, want to make each other happy.

More than anything, this applies to getting a job. So many candidates try to act so professional that they come off as cold. As a result, they remain faceless resumes. But the rare candidate who really lets his or her guard down, expresses excitement about the company, and asks for the job immediately enters a different category. Their excitement mirrors the excitement I felt when I first joined my company. Their anxiousness reminds me of the anxiousness I felt when searching for a job. They become like me, I want to help them succeed.

Pushing is Better Than Pulling: A Discusson of Status Reports

Summary: Don't require stakeholders to pull information from you. Change Modes. Push it to them instead.

Think Like a Doctor

"Bedside manner" is important in healthcare. A doctor with a good bedside manner reassures a patient by acting in a calm, deliberate manner and explaining his actions. He doesn't soften his message or bend the facts to put the patient at ease-- he just makes the information transparent. A good doctor may reach the exact same diagnosis as a bad doctor; the difference is all in the bedside manner.

Stakeholder Concerns

Project management needs to borrow this concept. Every project addresses an ailment or pain within the company. Every project manager is a doctor who can either choose to spread calm or spread fear throughout the organization by how transparent he or she make the information.

To be specific, here are some things that keep stakeholders up at night:

Is your team working on the right things...
...in the right order...
...and making progress...
...quickly enough?

Push, Don't Pull

And just like a worried patient, if a stakeholder is worried about these things, it is their natural inclination -- and professional duty -- to do whatever they can to make sure that the project is on the right track. If you, as the project manager, aren't pushing this information to them, then any good stakeholder will try to pull it from you in any way possible. This is a good thing, because keeping this information hidden is risky. But it can be frustrating because it feels like prying. And it is: underneath every email, conversation, and phone call the stakeholder is trying to pull more information from you.

The solution is to change modes. Don't require stakeholders to pull information from you. Push it to them instead. Be like a doctor with a good bedside manner. The effect is almost immediate. It reassures the stakeholders that you are running things well.

Learning From Experience

I learned this by experience. Back in 2004, on a year-long project for a financial services client, the project coordinator for the client would habitually hound me with questions and change requests to the project. It was taking up a significant portion of my time, and the unscheduled effort spent on responding to her was actually a risk to completing the project on schedule. As the emails piled up, it became clear that her objective in this was not to understand more about the project, it was to make sure that I was paying attention.

Honestly, she had good reason to be worried, but not because of performance. All she saw was that we delivered milestones on time, but I'm sure that--for her--every deadline was a nailbiter because the only visibility she had into the process was what she pried from me. I was inexperienced, and I did not recognize this pattern of communication as a problem. I just assumed that this was her style.

The Status Report

At a certain point I matured and understood both the situation and a possible solution. I begin sending a weekly friday status report at 4pm. Each report followed the same format:

At the top, a green checkmark, yellow square, or red X to indicate how I felt the project was going with regards to schedule, risk, technical challenges, and outstanding issues.
A high level schedule with the major milestones, dates, and percent complete.
A more detailed schedule for the current milestone, containing projected and actual completion dates.
A list of risks, issues, and things we needed to talk about.

The weekly status report was much less effort than I expected. Realistically, the reports didn't change that much--80% of the report's content stayed the same. The effect, however, was dramatic. The client stakeholder backed off, only dipping in when things she had truly important concerns, and our conversations became less frantic, more friendly, and more productive.

Post-Its and a Folder

In my last post I talked about working in an Operations team to help the care and feeding of a large distributed software application. The challenge was making sure every bolt was tightened and nothing fell through the cracks. Lots of firefighting, "remember to check this in 3 hours", "email so-and-so and wait for a reply", "update that batch file", "don't forget to call Frank", etc.

The first two days of this had me floundering. I tried using GTD techniques, emailing myself every time a task came up, processing things when I got back to my desk, scheduling reminders on my calendar, etc. None of it worked.

Why? Well:

Many of the tasks needed drop-everything immediate attention, often for an hour or two
I was forced to switch between tasks by things outside of my control
I would have to wait for a person/computer to finish something before I could continue

As a result, I got very frustrated and forgot to do a lot of things.

So I came up with the Manilla Folder method. It started with sticky notes at the bottom of my monitor of things I had to remember to do. Then, to be more mobile (and professional) it moved to the inside of an open manilla folder.

Who needs it?

First, think of everything you should be doing Right Now. If you feel pulled in 5 different directions (like a working mother with triplets) then you need it. (If, on the other hand, you are getting pulled in one direction, and your problem is that you need more speed, then this will not help you.)

For most people, having more than one task that must be done Right Now is frustrating because our brains just weren't built to juggle so many things at one time. We can handle one or two tasks. Once we hit three or four tasks we panic. We feel suffocated. Things slip out of the mind. Just like humans can remember 7 digit numbers relativey easily, but stumble at 8 or 9, we do well with two tasks that must be done ASAP but stumble at three or four. We get anxious, frustrated, and forgetful.

This is where the manilla folder comes in. The goal is to aid your brain in coping with your high priority tasks so that you can focus on actually doing the work quickly. Just like GTD, it will give you a sense of enormous well-being by freeing up your psychic RAM. Or whatever.

How It Works

First, grab a file folder and open it. Hold it horizontally (wider rather than taller).

Label the left side of the folder "Active". Here you will stick the tasks you are actively working on.

Label the right side "Waiting". Here you will stick tasks that someone else needs to finish before you can act on them. Examples: You are waiting for an email from Jones before you go ahead with deployment. You are waiting for the 14 GB database backup file to finish copying to another server. You are waiting for Bob to return from his afternoon meeting to ask him a question.

Note, when I say "stick tasks" above, I mean it literally. Grab a stack of 3" x 3" Post-It notes.

Write one task per note (just 3 to 5 words, and today's date) and put it on the Active side. When the task becomes something you are waiting for then move it to the Waiting side.

If your manilla folders are the same size as mine, you will quickly notice that you've only got room for 6 active tasks and 6 waiting tasks. This is all part of the plan. If you find yourself loading up more than 6 tasks on the Active side, you are deluding yourself. Either you are not working on the task right now, or you are wasting time by switching tasks too frequently. If this is the case, hold a quick prioritization session, put tasks 7, 8, and 9 into your Outlook Tasks list, and work on the six important tasks. Once you clear those out, hold another prioritization session. Forcing yourself to work on a reasonable number of things at one time is half the battle in reducing frustration.

For the technically inclined, you want to picture this manilla folder as the Windows Taskbar of your brain.

Also, when you are done with a task, don't toss it. Stack up completed notes on your desk and flip through them before your next status meeting. If you are like me, when you are in this mode it feels like you spent all day working furiously with very little tangible results, or at least less tangible results than a day's worth of coding gives you. Saving a list of your completed tasks will help you with this.

The Laws of the Manilla Folder (Summary):

The left side (Active side) is for tasks you are working on. (Or will be working on very, very soon.) Things should rarely be in here for more than a day. Each task should be between 30 minutes to 2 hours, but it's a guideline, you know?
The right side (Waiting side) is for tasks you are waiting on. Things sometimes stays here for multiple days.
Write 3 to 5 words per task and the date. It's a reminder to yourself. Be minimal. (Note: I often find myself jotting an additional reminder note when I move things over to the "Waiting" side.)
Use 3" X 3" Post-It notes, nothing smaller. And don't overlap notes. Limit yourself to 6 notes per side. Trust me. Otherwise you end up writing too many unrealistic tasks on your list, and the folder becomes visually confusing and useless.
If you have too many tasks, hold a quick prioritization session, and only use the 6 most important tasks. Put the rest into Outlook Tasks.

Finally, remember that this will not work for everything. It is best for situations where your schedule is not your own, but rather dictated by multiple outside forces. If every day feels like a Whack-a-mole game at an arcade, this will be helpful. Otherwise, it probably won't.

Flavors of Software Development

I've been at a small software consulting company for almost 3 years. Recently, after over two years of working on what I thought of as "standard" software projects, I switched projects at my software consulting company twice in 5 months, and switched to a completely different mode of development both times. The lesson? There is no such thing as a "standard" software project, and any tools, tips, and magic cures that work on one flavor of project will not work on another.

Projects can be categorized in a few different ways: New Standalone Development, Enhancements/Maintenance, and Operations. There are more, but these are the ones I have experience in. Here are some of the challenges I faced in each type of project.

New Standalone Development

Basically, a client needs a new web application, and we'd build it. Project were divided into a number of phases:

Phase One: Conceptual and logical design
Phase Two: Implementation of the framework and prototyping risky parts of the system
Phase Three - Phase N: Implementation of each subsystem
Deployment
Handoff

Everything could be pinned down and scheduled. Progress was indicated by an steadily growing line of checkmarks down the left side of a Gantt chart.

The challenge was understanding what the client needed, and finding the quickest way to produce the code and make sure it was high quality. For this, we had a variety of artifacts including use cases, use case realizations (quick writeups of the technical details necessary to implement a use case), and Gantt charts. To make things high quality we created unit tests and conducted peer code reviews.

Enhancements / Maintenance

I then switched to a team responsible for making enhancements and fixing bugs to a big part of an even bigger distributed enterprise application. A lot of the best practices on the previous project were no longer necessary.

The challenge here was making sure that we had gathered enough information from other teams about how a change would impact the system (pulling information) and then communicating the design quickly to everyone involved (pushing information). For the most part, strict, well formatted use cases were tossed in favor of team design around a table. We documented and explained designs with CRC cards (http://www.extremeprogramming.org/rules/crccards.html).

Another challenge was getting a grip on the incoming requests. We settled on a first come first serve solution only to have a critical bug knock everything aside. It felt disorganized and frustrating to both us and the client until we settled on the severity rating system below.

Sev 1: The production system is down, fix it immediately (within a day). Drop everything to get these fixed.
Sev 2: The system is crippled, but has a workaround, fix it within a week. Prioritize in daily meetings.
Sev 3: Short project or enhancement, deliver in a month. Prioritize in weekly meetings.
Sev 4: Long project, deliver in 3 months.
Sev 5: Nice to have, deliver in 4+ months .

In practice, Sev 5 items get moved into Sev 3 or Sev 4 before they get worked on, but it makes people feel good to have things in a bin somewhere. Longer Sev 3 and Sev 4 items would still have use cases and code reviews, but anything else was done with as little overhead as possible.

Operations

I then switched to a project that was a combination of new development and operations. There were hour long stretches of development interspersed with managing emails, checking running processes, exporting and importing data, copying files, etc.

The big challenge with operations was making sure that no task was forgotten. I tried keeping it all in my head for a day or two. That didn't work. I tried GTD, but I had to act on a lot of things immediately (that took way longer than 2 minutes), so I couldn't just throw it in a processing bin. I tried making task lists on paper or blackberry, but it quickly became a list of all current and future tasks and grew to unwieldy proportions, and in the case of paper I hated rewriting things every few hours.

I needed something that was portable, quick to update, efficient to read, traceable (for status meetings) and where nothing fell through cracks.

The solution I came up with involves a manilla folder and is described in the next post.