Kent R. Spillner's Blog - Work

Make All Environments Consistent

2009-12-30T00:00:00-08:00

Consistent environments are important on software projects. They increase the team’s productivity, make it easier to diagnose, replicate and resolve bugs, make it easier to tune performance and scale the application, and reduce the cost of administration and maintenance. Building consistent environments is a great candidate for automation; in fact, tools like Chef and Puppet were written for exactly this purpose. But Chef and Puppet are complicated, heavyweight solutions. They also require a considerable amount of infrastructure themselves (git, Ruby, etc.)!

For lighter, simpler configuration needs I prefer Makefiles. Makefiles are great for this sort of thing because versions of Make exist for every operating system, and most UNIX systems come with some version already installed. I use Make to install my personal configuration settings for common utilities on each machine I use (see http://github.com/sl4mmy/dotfiles), for example.

Makefile syntax is familiar to system administrators and easy to learn, and because Makefiles are shell scripts (with a few format constraints) they map directly to the sequence of steps you would perform manually to setup your environments. But arguably the biggest advantage of Makefiles is they won’t do anything when the file about to be installed or modified exists. This is critical in shared production environments where services might already be installed and configured for other applications, and you don’t want to risk accidentally overwriting something important.

Consider a project dependent on Java and JRuby, and a convention that third-party applications should be installed under /opt/apps/<name>/<name>-<version>. The following Makefile installs and configures this project’s environment accordingly:


# Makefile

JAVA_VERSION=1.6.0_18
JRUBY_VERSION=1.4.0

# TARGET 0:
all: java jruby

# TARGET 1:
java: /opt/apps/java/java-${JAVA_VERSION}/

# TARGET 2:
jruby: /opt/apps/jruby/jruby-${JRUBY_VERSION}/

# TARGET 3:
/opt/apps/java/java-${JAVA_VERSION}/: /opt/apps/java/ /tmp/java-${JAVA_VERSION}.tar.gz
	(cd /opt/apps/java; tar czf /tmp/java-${JAVA_VERSION}.tar.gz)

# TARGET 4:
/opt/apps/jruby/jruby-${JRUBY_VERSION}/: /opt/apps/jruby/ /tmp/jruby-${JRUBY_VERSION}.tar.gz
	(cd /opt/apps/jruby; tar czf /tmp/jruby-${JRUBY_VERSION}.tar.gz)

# TARGET 5:
/opt/apps/java/: /opt/apps/
	mkdir /opt/apps/java

# TARGET 6:
/opt/apps/jruby/: /opt/apps/
	mkdir /opt/apps/jruby

# TARGET 7:
/tmp/java-${JAVA_VERSION}.tar.gz:
	# Shell commands to download an archive of Java and save it as /tmp/java-${JAVA_VERSION}.tar.gz

# TARGET 8:
/tmp/jruby-${JRUBY_VERSION}.tar.gz:
	# Shell commands to download an archive of JRuby and save it as /tmp/jruby-${JRUBY_VERSION}.tar.gz

# TARGET 9:
/opt/apps/:
	mkdir -p /opt/apps

Starting from the top:


# Makefile

JAVA_VERSION=1.6.0_18
JRUBY_VERSION=1.4.0

You should always name your Makefiles as Makefile because Make will look for that file in the current directory by default when it runs. You are free to name your Makefiles anything you wish, but if they are not named Makefile then Make will require additional configuration telling it which file to load.

Makefile comments start with #, just like shell scripts. The # Makefile comment in this example is not necessary, it is only for illustrative purposes.

The first two non-comment lines define variables that will be used throughout this Makefile. You may safely assume the syntax for defining and referencing variables in Makefiles is identical to shell scripting.


# TARGET 0:
all: java jruby

Target 0, named all here but could be named anything, is the default target because it is the first target declared in this Makefile. Running make in the directory containing this Makefile is equivalent to running make all.

Make is a file-centric build tool; it assumes the purpose of each target is to produce (or “make,” natch) a single file. By convention, the name of a target is the name of the output file produced by invoking that target. In our example, Make assumes that the all target will produce a file in the current directory named all. Make uses this convention to determine whether or not it actually needs to execute that target: if a file named all already exists in the current directory, Make won’t execute the all target.

Of course, targets are not required to actually produce a file with the same name as the target, but if they don’t Make will never think such targets are up-to-date and will execute them every time. In this case, we want Make to do exactly that; our goal is not to produce a file named all in the current directory, our goal is to set up a fully working environment on any machine by simply copying this Makefile and running make. We are using all as a convenience target so that we can make both java and jruby by default. If there were more dependencies, we could create separate targets for each of them and add them as additional prerequisites of all.

Our all target is declared with two prerequisites: java and jruby. Make executes all prerequisites before executing the target, so our java and jruby targets will execute before all.


# TARGET 1:
java: /opt/apps/java/java-${JAVA_VERSION}/

# TARGET 2:
jruby: /opt/apps/jruby/jruby-${JRUBY_VERSION}/

The java and jruby targets are declared in Target 1 and Target 2, and each have a single prerequisite of the Java and JRuby installation directories defined in our convention. Absolute paths are valid target names in Makefiles, and since they are used as prerequisites here, Make will search for targets with those names before it executes the java or jruby targets.


# TARGET 3:
/opt/apps/java/java-${JAVA_VERSION}/: /opt/apps/java/ /tmp/java-${JAVA_VERSION}.tar.gz
	(cd /opt/apps/java; tar czf /tmp/java-${JAVA_VERSION}.tar.gz)

# TARGET 4:
/opt/apps/jruby/jruby-${JRUBY_VERSION}/: /opt/apps/jruby/ /tmp/jruby-${JRUBY_VERSION}.tar.gz
	(cd /opt/apps/jruby; tar czf /tmp/jruby-${JRUBY_VERSION}.tar.gz)

The /opt/apps/java/java-${JAVA_VERSION}/ and /opt/apps/jruby/jruby-${JRUBY_VERSION}/ targets are declared in Target 3 and Target 4. They both have two prerequisites, and one step. Notice that the trailing slash in each target’s name makes it explicit to Make that these targets produce directories.

Steps are shell commands Make runs when it executes a target, and virtually any valid shell script is also a valid step. Steps are associated with the target immediately above them in the Makefile, and they must begin with a TAB (no spaces!). Multiple steps can be associated with a target and will run in order, but each line must begin with a TAB (no spaces!). Makefiles are very touchy about whitespace: every line beginning with a TAB below a target definition is a step for that target, and the first line that does not begin with a TAB is the separator between targets.

When Target 3 is executed, Make will spawn a temporary subshell, change the current working directory to /opt/apps/java and expand /tmp/java-${JAVA_VERSION}.tar.gz there. The parenthesis around the step definition are not required by Make, they are used here as in a regular shell script: the enclosed commands will run in a subshell, returning to the parent shell when finished. I did this so that changing directories in this step won’t affect the rest of the Makefile. Target 4 does the same for JRuby.


# TARGET 5:
/opt/apps/java/: /opt/apps/
	mkdir /opt/apps/java

# TARGET 6:
/opt/apps/jruby/: /opt/apps/
	mkdir /opt/apps/jruby

...

# TARGET 9:
/opt/apps/:
	mkdir -p /opt/apps

Target 3 (/opt/apps/java/java-${JAVA_VERSION}/) depends on /opt/apps/java/, declared as Target 5, and Target 4 (/opt/apps/jruby/jruby-${JRUBY_VERSION}/) depends on /opt/apps/jruby/, declared as Target 6. Both targets depend on /opt/apps/, declared as Target 9. These three targets ensure that the correct directory structure exists according to our convention, creating missing directories as necessary.


# TARGET 7:
/tmp/java-${JAVA_VERSION}.tar.gz:
	# Shell commands to download an archive of Java and save it as /tmp/java-${JAVA_VERSION}.tar.gz

# TARGET 8:
/tmp/jruby-${JRUBY_VERSION}.tar.gz:
	# Shell commands to download an archive of JRuby and save it as /tmp/jruby-${JRUBY_VERSION}.tar.gz

Targets 7 and 8 are placeholders showing how to download the necessary files before building an environment.

So, what happens when you run make on a clean environment without any of the directory structure or tarballs necessary for this project? Follow along by tracing the prerequisites in the Makefile.

Since all is the default target, Make starts by looking for a file named all in the current directory; unable to find that file, Make tries to execute the prerequisites of all, java and jruby. Since no file named java exists in the current directory either, Make looks up the target named java and then tries to execute its prerequisite, /opt/apps/java/java-${JAVA_VERSION}/. Since that directory does not exist, Make looks up the target named /opt/apps/java/java-${JAVA_VERSION}/ and tries to execute its prerequisites, /opt/apps/java/ and /tmp/java-${JAVA_VERSION}.tar.gz. Since the directory /opt/apps/java/ does not exist, Make looks up the target named /opt/apps/java/ and tries to execute its prerequisite, /opt/apps/. Since that directory does not exist, Make looks up the target named /opt/apps/, sees that it has no prerequisites, and executes the steps associated with that target (mkdir -p /opt/apps) which produces the directory /opt/apps/.

Make then backtracks to the /opt/apps/java/ target, determines that its prerequisites were previously executed, and executes its steps (mkdir /opt/apps/java) which produces the directory /opt/apps/java/. Make backtracks again to the /opt/apps/java/java-${JAVA_VERSION}/ target, sees that it still has one unsatisfied prerequisite, and looks up the target named /tmp/java-${JAVA_VERSION}.tar.gz. Make sees that /tmp/java-${JAVA_VERSION}.tar.gz has no prerequisites, and executes its steps to download an archive of Java and save it as /tmp/java-${JAVA_VERSION}.tar.gz. Make backtracks to the /opt/apps/java/java-${JAVA_VERSION}/ target again, sees that all of its prerequisites are now satisfied, and executes its steps which produce the directory /opt/apps/java/java-${JAVA_VERSION}/.

Make then backtracks all the way back to the all target, and does the same for its other prerequisite, jruby.

Now, consider what happens when you re-run make in the directory containing this Makefile (or run it for the first time in an environment which was previously setup manually). Make sees that all is the default target, but cannot find a file named all in the current directory, so it tries to execute its prerequisites, java and jruby. There is no file named java in the current directory, either, but the java target’s only prerequisite is /opt/apps/java/java-${JAVA_VERSION}/, and that directory does exist, so Make skips the prerequisite’s steps and goes straight to executing the steps associated with the java target. Since there are none, nothing happens, and Make backtracks to the all target and tries to execute its other prerequisite, jruby. Similarly, there is no file named jruby, but the directory corresponding to the jruby target’s only prerequisite does exist, so Make skips executing the prerequisite target, going straight to the steps associated with jruby. Again, there are none so nothing happens, Make backtracks back to the all target, determines its prerequisites were previously executed and so tries to execute the steps associated with all. Since there are no steps associated with all, nothing happens, and Make finishes successfully without having done anything. Perfect for production environments!

What happens if you delete /tmp/java-${JAVA_VERSION}.tar.gz and re-run make in the directory containing the Makefile? Well, nothing. Make never sees that file is missing since the directory /opt/apps/java-${JAVA_VERSION} exists. But if you delete the directory /opt/apps/java-${JAVA_VERSION}/ and re-run make, then /tmp/java-${JAVA_VERSION}.tar.gz will be downloaded again and /opt/apps/java-${JAVA_VERSION}/ will be re-created. The point is: you must carefully ensure all of your application’s dependencies are properly exposed as prerequisites in your Makefile, not hidden as nested prerequisites of prerequisites.

I love Make as a low-touch solution for setting up consistent environments! It’s simple to learn and easy to use, and it is low-risk for shared production environments because it reduces the likelihood of modifying or deleting files used by other applications. And when you outgrow Make’s capabilities and need a more powerful tool, Makefiles are the perfect way to consistently install and configure Chef or Puppet across multiple machines.

Java Build Tools: Ant vs. Maven

2009-11-14T00:00:00-08:00

TRANSLATIONS: Spanish (gracias, José!)

UPDATED 2010-01-06: linked to demonstration of the “10 minute mvn clean build” problem, and added notes about: slow build times, excessive memory use, bad test result output, untrusted repository artifacts, and external configuration files.

UPDATED 2010-02-21: linked to Spanish translation (courtesy of José Manuel Prieto)

The best build tool is the one you write yourself. Every project’s build process is unique, and often individual projects need to be built multiple different ways. It is impossible for tool authors to anticipate every build’s requirements, and foolhardy to try (Apache developers: take note). The best any tool can do is provide a flexible library of reusable tasks that can easily be adapted to your needs, but even that is insufficient. Off-the-shelf tasks never suit your project perfectly. You will waste countless hours struggling to make those tasks do exactly what you need, only to give up and write a plugin instead. Writing your own custom build tool is quick and easy, and requires less maintenance than you fear. Don’t be afraid: builds should fit your project, not the other way around.

If you don’t want to write your own build tool, then you should use Rake. Rake is the best existing build tool for Java projects. Rake provides a bunch of standard methods to perform common build tasks, and anything else can be quickly implemented in Ruby. Writing build scripts in a real programming language gives Rake a huge advantage over other tools. There are other advantages, too, but none are as important.

So, you should write custom build tools for your projects. If you don’t want to, then you should switch to Rake. If you can’t switch, you should lobby for the right to switch. If politics drives technology decisions, if you will never be allowed to switch, then quit your job or leave the project.

If you lack the courage to quit, then use Ant. Ant is the second best existing build tool for Java projects. Although inferior to Rake, Ant is still a great build tool. Ant is mature and stable, it is fast, and it comes with a rich library of tasks. Ant makes it possible (but not at all easy) to script rich, complex builds processes custom-tailored to your project.

So, write your own build tool, or else switch to Rake, or fight to switch to Rake, or quit and go some place where you can use Rake. And if all else fails, use Ant until you can find a new job somewhere else that uses Rake.

That’s it! Those are the only choices I can recommend! Because you never, ever, under any circumstances want to use Maven!

Maven builds are an infinite cycle of despair that will slowly drag you into the deepest, darkest pits of hell (where Maven itself was forged). You will initially only spend ten minutes getting Maven up and running, and might even be happy with it for a while. But as your project evolves, and your build configuration grows, the basic pom.xml that you started with will prove inadequate. You will slowly add more configuration to get things working the way you need, but there’s only so much you can configure in Maven. Soon, you will encounter Maven’s low glass ceiling for the first time. By “encounter,” I mean “smash your head painfully against.” By “for the first time,” I mean “you will do this repeatedly and often in the future.” Eventually, you’ll figure out some convulted pom.xml hackery to work around your immediate issue. You might even be happy with Maven again for a while… until another limitation rears its ugly little head. It’s a lot like some tragic Greek myth, only you are the damned soul and the eternity of suffering is your build process.

Seriously. Maven is a horrible implementation of bad ideas. I believe someone, somewhere had (perhaps still has) a vision for Maven that was sensible, if not seductive. But the actual implementation of Maven lacks any trace of such vision. In fact, everything in Maven is so bad that it serves as a valuable example of how not to build software. You know your build is awesome when it works the opposite of Maven.

Consider the test results output from Maven’s Surefire plugin. Everything seems fine as long as all of your tests are passing, but Surefire reports are a nightmare to debug when things go wrong! The only information logged to the console is the name of the failing test class. You must manually cross-reference that name with a log file written in the target/surefire-reports/ directory, but those logs are written one per test class! So, if multiple test classes fail, you must separately check multiple log files. It seems like a minor thing, but it quickly adds up to a major annoyance and productivity sink.

Maven advocates claim their tool embraces the principle of Convention Over Configuration; Maven advocates are liars. The only convention Maven supports is: compile, run unit tests, package .jar file. Getting Maven to do anything else requires configuring the conventions. Want to package a .war file? Configure it. Want to run your application from the command line? Configure it. Want to run acceptance tests or functional tests or performance tests with your build, too? You can configure it, but it involves not running your unit tests, or not running them during the conventional unit test phase of your build process, or… Want to generate code coverage metrics for your project? You can configure that, too, but your tests will run twice (or only once, but not during the conventional unit test phase), and sometimes it reports 0% code coverage despite the comprehensive test suite.

Speaking of configuration, Maven has the worst configuration syntax since Sendmail: alternating normal form XML. As a consequence, Maven configuration is verbose, difficult to read and difficult to write. Things you can do in one or two lines of Ruby or XML with Rake or Ant require six, seven, eight lines of pom.xml configuration (assuming it’s even possible with Maven).

There’s nothing consistent about Maven’s configuration, either. Some things are configured as classpath references to .properties files bundled in .jar files configured as dependencies, some things are configured as absolute or relative paths to files on disk, and some things are configured as system properties in the JVM running Maven. And some of those absolute paths are portable across projects because Maven knows how to correctly resolve them, but some are not. And sometimes Maven is smart enough to recursively build projects in the correct order, but sometimes it’s not.

And some things aren’t even configured in the pom! Some things, like Maven repositories, servers, and authentication credentials, are configured in settings.xml. It is perfectly reasonable to want to keep user’s passwords out of pom.xml files which will be checked into the project’s version control repository. But Maven’s solution is terrible: all this configuration goes in a settings.xml file that lives outside of any project’s directory. You can’t directly share any of this configuration between your desktop and laptop, or with other developers, or with your project’s build servers. But it is automatically shared with every single Maven project you work with, and potentially every single Maven project every user on that machine works with. When a new developer joins your project, they must manually merge the necessary configuration into their existing settings.xml. When a new agent is added to your build server farm, the necessary configuration is manually merged into its existing settings.xml. Ditto for when you migrate to a new machine. And when any of this configuration needs to be updated, it must be manually updated on every single machine! This was a solved problem before Maven came along, too: properties files. Project teams can put generic configuration like this in a properties file which is checked in to version control, and individual developers can override that information in local properties file which are not checked in to version control.

All this stuff in Maven — the conventions, the configuration, the process — is governed by “The Maven Way”. Unfortunately, “The Maven Way” is undocumented. You can catch fleeting glimpses of it by trawling the Maven documentation, searching the Google, or buying books written by Maven developers. The other way you encounter “The Maven Way” is by tripping over (or smashing against) its invisible boundaries. Maven was not built to be flexible, and it does not support every possible build process. Maven was built for Apache projects, and assumes every project’s build process mirrors Apache’s own. That’s great news for open-source library developers who volunteer on their own time and to whom “release” means “upload a new .zip file to your website for others to manually find, download, and add to their own projects.” It sucks for everyone else. While Rake and Ant can accommodate every build process, Maven can’t; it is possible, and in fact quite likely, that Maven just doesn’t support the way you want to build your software.

And Maven’s dependency management is completely, entirely, irrevocably broken. Actually, I take that back; Maven’s strategy of downloading ibiblio to the user’s home directory and then dumping everything on the classpath is incredibly stupid and wrong and should never be confused with “dependency management.” I recently worked on a Maven project which produced a 51 MB .war file; by switching to Ant with hand-rolled dependency management, we shrunk that .war file down to 17 MB. Hrmmm… 51 – 17 = 34 = 17 × 2, or: 2/3 of the original bulk was useless crap Maven dumped on us.

Extraneous dependencies don’t just eat up disk space, they eat up precious RAM, too! Maven is an all-around memory hog. Relatively simple projects, with only a parent pom and a few sub-modules, require extensive JVM memory tuning with all those fancy JAVA_OPTS settings you typically only see on production servers. Things are even worse if your Maven build is integrated with your IDE. It’s common to set your JVM’s max heap size to several hundred megabytes, the max permgen size to a few hundred megabytes, and enable permgen sweeping so classes themselves are garbage collected. And all this just to build your project, or work with Maven in your IDE!

Funny story: on that same project I once endured a ten minute “mvn clean” build because Maven thought it needed yet more crap in order to “rm -rf ./target/” (see a similar example: http://gist.github.com/267553). Actually, there’s nothing funny about that story; trust me: you don’t want a build tool which automatically downloads unresolved dependencies before cleaning out your build output directories. You don’t want a build tool which automatically downloads unresolved dependencies, PERIOD! Automatically downloading unresolved dependencies makes your build process nondeterministic! Good ol’ nondeterminism: loads of fun in school, not so fun at work!

And all that unnecessary, unwanted network chatter takes time. You pay a performance penalty for Maven’s broken dependency management on every build. Ten minute clean builds are horrible, but adding an extra minute to every build is even worse! I estimate the average additional overhead of Maven is about one minute per build, based on the fact that the one time I switched from Maven to Ant the average build time dropped from two and a half minutes to one and a half. Similarly, the one time I switched from Ant to Maven the average build time increased from two minutes to three.

You have no control over, and limited visibility into, the dependencies specified by your dependencies. Builds will break because different copies of Maven will download different artifacts at different times; your local build will break again in the future when the dependencies of your dependencies accidentally release new, non-backwards compatible changes without remembering to bump their version number. Those are just the innocent failures, too; the far more likely scenario is your project depends on a specific version of some other project which in turn depends on the LATEST version of some other project, so you still get hosed even when downstream providers do remember to bump versions! Every release of every dependency’s dependencies becomes a new opportunity to waste several hours tracking down strange build failures.

But Maven is even worse than that: not only does Maven automatically resolve your project’s dependencies, it automatically resolves its own plugins’ dependencies, too! So now not only do you have to worry about separate instances of Maven accidentally downloading incompatible artifacts (or the same instance downloading incompatible artifacts at different times), you also have to worry about your build tool itself behaving differently across different machines at different times!

Maven’s broken dependency management is also a gaping security hole, since it is currently impossible in Maven to determine where artifacts originally came from and whether or not they were tampered with. Artifacts are automatically checksummed when they are uploaded to a repository, and Maven automatically verifies that checksum when it downloads the artifact, but Maven implicitly trusts the checksum on the repository it downloaded the artifact from. The current extent of Maven artifact security is that the Maven developers control who has write access to the authoritative repository at ibiblio. But there is no way of knowing if the repository you download all your dependencies from was poisoned, there is no way of knowing if your local repository cache was poisoned, and there is no way of knowing which repository artifacts in your local repository cache came from or who uploaded them there.

These problems are not caused by careless developers, and are not solved by using repository managers to lock down every artifact Maven needs. Maven is broken and wrong if it assumes humans never make mistakes. Maven is broken and wrong if it requires users to explicitly specify every version of every dependency, and every dependency’s dependencies, to reduce the likelihood of downloading incompatible artifacts. Maven is broken and wrong if it requires a third-party tool to prevent it connecting to the big, bad internets and automatically downloading random crap. Maven is broken and wrong if it thinks nothing of slowing down every build by connecting to the network and checking every dependency for any updates, and automatically downloading them. Maven is broken and wrong if it behaves differently on my laptop at the office and at home. Maven is broken and wrong if it requires an internet connection to delete a directory. Maven is broken and wrong.

Save yourself.

And Now for Something Completely Different

2009-11-13T00:00:00-08:00

Today is my last day working for ThoughtWorks. After four and a half years, I decided to hang up my consultant’s cape and try something new: I accepted an offer from DRW Trading Group starting next week.

I am extremely excited about my new job at DRW, but the decision to leave ThoughtWorks was difficult and I am sad to say “goodbye!” ThoughtWorks was an important part of my life these past several years, and I am leaving behind a lot of great friends. I will miss working with ThoughtWorkers every day, and I wish them all the very best.

Ultimately, my decision to leave stems from my frustration with consulting. Ostensibly, companies hire consultants for their expertise in helping solve difficult problems. Typically, companies select consultants based on past experience and expertise in successfully solving similar problems. But always companies ignore or push back against anything running counter to their current way of doing things, no matter how much they paid for the suggestion.

I accept that compromise, education and negotiation are important aspects of consulting, and I agree it is unreasonable – and would be unwise – to expect clients to do everything exactly as they are advised. I also enjoy helping others solve problems. I am just tired of arguing about the same things with different clients over and over again. And hence the change.

I am not bitter or jaded, I bear no grudge; it is just time for me to try something new away from client politics. ThoughtWorks is a great company, and I can’t wait to see what things lie in store for them in the future! But I won’t miss my cape.

Private Methods are a Code Smell

2009-11-12T00:00:00-08:00

Private methods are a code smell. They should be moved to collaborating classes and made public.

Private helper methods indicate classes are doing too many things. Moving private helper methods to different classes, including creating new classes if necessary, splits the original responsibilities across multiple classes leading to simpler, better designs.

Sometimes, private methods are created to split complex logic or processes into small, easily digested pieces. Often, such private methods have gnarly dependencies because they directly access or modify internal state. Moving these methods to appropriate collaborators (again, creating new classes as necessary) exposes such dependencies. Eliminating these dependencies simplifies the new API, which improves readability and understanding.

Sometimes, private methods are created just to give pieces of functionality more descriptive names. Although descriptive names are desirable, creating private methods to provide descriptive names for things is still a smell. Moving these methods to collaborators and making them public creates opportunities for future reuse without reducing the clarity of the original code.

Taking small steps to improve design leads to flashes of brilliant design inspiration. As code slowly evolves into better shape, bits and pieces fall into place until another, superior design becomes clear. Making private methods public and moving them to (perhaps missing) collaborators is a simple and effective way to quickly improve design. The resulting code is simpler, more testable, more reusable, more cohesive, and less coupled. And when a superior design suddenly presents itself, a few public methods on many classes are easier to refactor than a few classes with many private methods.

The Tao of Test-Driven Development

2009-11-11T00:00:00-08:00

I gave my Tao of Test-Driven Development tutorial at OOPSLA 2009 in Orlando, FL. The tutorial began with a quick lecture-style introduction to test-driven development, and concluded with a hands-on coding dojo for attendees to practice what they learned.

The slides from my presentation are now available online in Keynote, PowerPoint and PDF formats.

I also uploaded starter code for parts 1 and 2 of the dojo. Although the dojo was technology-agnostic and attendees free to use any language, the starter code uses Java, JUnit and Ant.

The starter code for part 2 includes my solution to part 1, but there are no right or wrong solutions. My solution is illustrative of my approach to solving part 1, but is by no means the only solution. Indeed, I practiced part 1 three times in preparation for the tutorial and ended up with a different solution each time.

Introducing Hippie

2009-11-10T00:00:00-08:00

My current project integrates with a lot of external services. Unfortunately, our service-level agreements do not extend to our development and testing environments, so our application breaks frequently because of service problems. Services go down, services have bugs in the latest development versions deployed in our environments, our environments are upgraded to new versions of services that are not backwards compatible with the old versions, etc.

We write JUnit tests to verify each service is running correctly in our environments, so we do not waste time debugging problems caused by service failures. These tests make my team more productive, but unfortunately they do nothing to make the services more reliable. At least, not by themselves…

Enter hippie

Hippie is an open source tool that automatically sends Nagios passive checks based on the result of running JUnit tests.

hippie bridges the divide between developer tools and system administrator tools. JUnit tests automatically notify Nagios which automatically notifies other teams that their services are broken.

It is an extension of the automatic build notifications our continuous integration tools provide, only at much, much finer resolution. Hippie makes it possible to notify Alice on “Team: Tiger” when TigerTests#shouldRespondWithOkStatus() fails, but notify Bob in production support when LegacyTests#shouldAlwaysBeUpAndRunning() fails.

Those JUnit tests we wrote no longer just prevent us from wasting time debugging service problems, they now kick off a chain of events culminating (hopefully) with someone else fixing the problem. And that is the ultimate goal of every programming endeavor.

GitHub Pages

2009-11-09T00:00:00-08:00

UPDATED 2010-09-18: Added a new category to my site, Books, with its own separate feed.

I finally found a blogging platform to love: GitHub Pages.

GitHub Pages rocks! It gives me control over every aspect of my site: directory structure, page layout, permalink format. Everything is stored in plain text, versioned (courtesy of Git, natch), and cloudified. There is no dumb blog name or stupid subtitle, no biography or mugshot, no comments or tag clouds. I especially like the lack of comments because the best conversations happen when everyone just listens to me.

The sheer awesomeness of GitHub Pages and its simplicity of use compel me to blog; they also compelled me to upgrade my GitHub account and wire-up a vanity domain. I resisted lending my voice to the symphony of idiots blogging on the internets for many years, but no more! GitHub Pages makes it so damn easy, I just can’t help myself. If I fail in this endeavour, if I fail to update regularly or at all, it will be for lack of trying, not inadequacy of the tool.

The world suffers for lack of access to my opinions. To make it easier to learn my opinion on a particular subject, blog entries will be categorized as one of: life, politics, or work. This entry itself is a special case and will be categorized as all three to seed my syndication feeds. There are separate feeds for each category, and a fourth feed with everything, to provide the illusion of choice. In reality, all of my opinions are equally valuable.

You’re welcome.