Why Be Paranoid and Pessimistic

Being paranoid and pessimistic will save your ass constantly. Before even talking about actual code, backups, debuggers, source control, error logging, automated deployment scripts, and almost all tools of the programming trade are predicated on the pessimistic idea that stuff goes wrong often, and there must be procedures in place to identify, correct, and prevent errors.

Another side effect of being paranoid is coding defensively. No matter if you are coding against libraries, code written by other developers on your team, or code written by yourself, it is important to code defensively, because everybody and everything is fallible.

Defensive code makes fewer assumptions and carefully handles error cases instead of failing silently or producing undefined behavior. This leads to code that is easier to maintain and less prone to bugs.

Libraries are updated, frameworks evolve, requirements change. Programmers are constantly fighting against entropy. Defensive programming guards against the effects of software entropy by reducing the possibility that changes in one part of a system will cause subtle bugs somewhere else.

It is important expect the unexpected in all things.

It seems that there’s a mantra that every programmer must memorize early in his or her career. It is a fundamental tenet of computing, a core belief that we learn to apply to requirements, designs, code, comments, just about everything we do. It goes

THIS CAN NEVER HAPPEN…

- The Pragmatic Programmer by Andrew Hunt & David Thomas

Embracing paranoia and pessimism will automatically lead to wonderful things like:

• Explicitly stating preconditions, postconditions, and invariants
• If it can’t happen, using assertions to ensure that it won’t
• Always using source control
• Crashing early – a dead program normally does a lot less damage than a cripple one

All of the above are widely considered good practice, and techniques recommended in The Pragmatic Programmer.

When to stop being paranoid and pessimistic

You should never stop being paranoid or pessimistic, but when interacting with non-technical people, you need to change your game face.

Let’s say you get a phone call from your boss who is a search engine optimization guy. He is wondering if you can build a tool to check how a website ranks in the search engines for various keywords.

You are a good programmer. A paranoid programmer. A pessimistic programmer. You immediately start thinking about the most significant challenges involved. That to do this project right, you will need to account for differences in search results across geographic areas. That means a distributed architecture of some kind, and most likely reports with combined rankings plus rankings broken out by geographic area. Now, if we use too many geographic areas and check them too often, we will significantly change the search volume for a particular term, which can have indirect effects on rankings. Plus scraping results is probably against Google’s ToS…

Swallow those words.

Your boss is a marketer at heart. He is an optimist. He needs to be an optimist in order to sell stuff. He doesn’t want to hear about worst case scenarios or most significant challenges. He needs to know about them, but wrap them up in a language he is better equipped to understand.

“Hrmmm. There are already similar tools out there, so I’m sure this is something we can do. However, I don’t think creating this tool will be as simple as it first appears. The most straightforward way to collect the data is to do some searches programmatically and then scrape the page to get rankings, but I think this is against Google’s ToS, and could get us in hot water with Google. We will also have to account for things like geographical rankings differences. I’m sure there are ways around these issues, but I’m not sure what the solution will look like yet without doing a bit more research.”

See the difference? Instead of your boss getting what sounds to him like a litany of problems, he is now getting some reassurance that it can be done, it just isn’t as easy as he probably thinks it is, and will take some thought to get everything right.

The only problem with pessimistic problem solving is that it is often misunderstood by optimists (read: management, sales, marketing, etc.). You don’t want to be known as the person who spits out a list of problems at every new idea. Try instead to be known as the person who solves problems before anyone else realizes that a problem exists.

So unless you are communicating with optimists, go forth and spread paranoia and pessimism.

The promises of cloud computing

Its hard to do any reading on cloud computing without running into promises that sound too good to be true: “With the cloud, individuals and small businesses can snap their fingers and instantly set up enterprise-class services.”^[1] Or to hear people proclaiming that the cloud is the only solution: “The future is the cloud. Not because it is trendy but because only the cloud can scale.”^[2]

The reality

Despite the cloud being billed as the next evolution of the internet, the reality is that cloud computing has been around for a long time under different monikers: hosted services, Software as a Service (SaaS), shared infrastructure. For the most part, the cloud is made up of all the same technology that existed before the cloud hype got started. It is composed of standard operating systems, databases, load balancers, management tools, and so on.

The innovation isn’t in the tech at all – it is in the operations model.

The innovation is that individuals and small businesses now have access to resources on-demand and at scale without having to maintain the hardware and the expertise to run it in-house. It allows them to experiment and iterate cheaply and quickly.

Individuals have the ability to quickly provision a ruby on rails setup, deploy their application, and then crank up the power if the application becomes popular, or simply throw the instance away if it doesn’t work out. That is the true power of cloud computing. It gives individuals and small businesses access to resources that were previously only cost-effective for the enterprise.

Confusion and angst in the tech sector

There is tension in the tech sector because cloud computing is such a poorly defined concept, and it is usually defined in terms of the technology. When looking only at the tech involved, the amount of hype around cloud computing is completely disproportionate to any advancements in tech needed to create cloud products.

The cloud hype is getting thicker and smellier every day. All the cloud excitement is coming from those who hope to profit from it, the vendors and breathless tech journalists who can’t think of anything worthwhile to write about. They’re working very hard to make it sound like a wonderful thing, a miracle of rare device that will transform life as we know it.Carla Schroder, Editor – LinuxToday.com on Keep your cloud

Even worse, “cloud” has become just a catchy word that is sprinkled over marketing copy to boost customer acquisition and increase buy-in (or something).

The interesting thing about cloud computing is that we’ve redefined cloud computing to include everything that we already do. I can’t think of anything that isn’t cloud computing with all of these announcements. The computer industry is the only industry that is more fashion-driven than women’s fashion. Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s insane. When is this idiocy going to stop? We’ll make cloud computing announcements. I’m not going to fight this thing. But I don’t understand what we would do differently in the light of cloud.Larry Ellison, CEO – Oracle Corporation

Anyway, that’s my take on the cloud. The innovative part has nothing to do with the tech and everything to do with the operations model. I suppose Cloud Computing For Dummies will tell you something very different.

1. Roy Stephan, Director of IT architecture and engineering – Intelligent Decisions in CNNMoney Why the cloud is now the hottest trend in tech.
2. Sean McGrath in ITWorld Only the cloud can scale

Always use source control

There is no reason not to use source control. Even for a solo project. Even for a toy project. Even if you never make mistakes. Just use it. Source control offers so many benefits like rollbacks, code diffs, backups, and commit logs that every project will benefit from it.

Don’t break the build

Don’t commit code if it breaks the build. You don’t want to force other people to sift through your code looking for the problem that broke the build before they can continue with their own work.

Commit early, commit often

Don’t go dark. Committing early and often gives you a backup, an incremental log of progress, easier merges, and lets the team know what you are doing.

Remove unused code immediately

Don’t commit code with large blocks of old code commented out or unused, deprecated functions. If you need to get this code back at some point, you can always get it from the repository. Keep the current working copy of the source as clean and cruft-free as possible.

Keep a good commit log

Leave good comments in the commit log. Searching the commit log should allow you to find which commits implemented a specific feature or fixed a specific bug. Skimming the commit log should provide a good history of the project and give some hints as to it’s maturity.

Whitespace

Don’t commit whitespace, formatting, or code style changes at the same time as other changes. If you must make these types of changes to the code base, commit them separately. This will keep the diffs vastly more readable.

Minimal Changeset

When doing bugfixes, make the minimal set of changes needed to fix the bug. If a rewrite of a module is really in order, do the rewrite, but err on the side of making the absolute minimal set of changes needed to fix the bug. This way the changelog will clearly reflect the exact source of the bug and the changes needed for the fix.

One bugfix per commit

Again, keep the changelog clear by doing only one bugfix per commit (unless multiple bugs are very closely related). One bugfix per commit makes rollbacks easier and leaves a clear track of how the bug was fixed that will be valuable later if a regression test fails or if the fix created unintended side effects.

Notify the team of new commits

Other team members working on the code base should be immediately and automatically notified of any changes that have the potential to affect them. This could be setup in different ways depending on the project and the workflow used by the team. One of the easiest and most effective ways to set this up is an automated email including the commit message and the diff to all members of the team working on that particular module. Notifications keep all team members appraised of the current state of the code and give them a heads up if they are likely encounter problems merging their own code.

Integrate source control with other tools

Source control should be integrated with other tools used on the project like bug tracking, continuous integration, and IDEs. This makes it easy to cross-reference the commit that fixed a bug, created a bug, or broke the build. Integration with an IDE gives you the option to do source control operations like diff, status, and commit within the IDE without having to drop to the command line or a separate tool.

The above basic principles of version control will make you effective with any type of version control system out there. Have more source control tips? Leave them in the comments.

1. The terms ‘version control’, ‘source control’, and ‘revision control’ are used interchangeably here.

The hosting platforms covered below provide a Rails environment right out of the box. It is certainly possible to run directly on AWS (or another cloud platform) with a bit more setup work, but that is beyond the scope of this article.

Engine Yard

Setup and deploy process: After creating an Engine Yard account, setup and deploy is as simple as gem install engineyard then ey deploy from within the app directory.

Starter plan: Engine Yard offers 500 free compute hours on a High-CPU Medium instance to test out the service.

Other plans: Plans are completely customizable. You choose the number and type of instances, and then pay per hour for instances, plus extra storage and bandwidth costs. Full pricing details are here.

Version Control: Must use Git for version control.

Conclusion: Engine Yard’s pricing structure is geared towards projects that require compute time, not 24/7 availability, making it most appropriate for data crunching. For web apps that need to be available all the time, but may be idle or nearly idle much of the time, another provider will be more cost effective.

Heroku

Setup and deploy process: Simple git based workflow. sudo gem install heroku then heroku create sushi will create an app called sushi. Deploying is as easy as git push heroku master.

Starter plan: The free starter plan includes one worker (a background process running your code and processing jobs from a queue) and a 5MB shared database.

Other plans: The number of dynos and workers can be adjusted dynamically to handle traffic spikes or provide more proccessing power. Dynos and workers are priced by the hour, and dedicated databases are priced based on the size and number of connections required. See the full pricing details for your particular situation.

Version Control: Must use Git for version control.

Conclusion: Heroku’s free starter plan makes it an ideal choice for bootstrapping a web application and scaling up as needed.

Winnie Cloud

Setup and deploy process: Push to a Git repo managed by Winne Cloud.

Starter plan: Free 30 day trial of the staging environment.

Other plans: Winnie Cloud offers either a production working instance or a staging working instance. PostgreSQL, MongoDB, and Redis are available at extra cost.

Version Control: Must use Git for version control.

Conclusion: It appears that Winnie Cloud started as an internal product of Rails programming shop Ragnarson. It doesn’t seem to be as mature as the other Ruby on Rails platforms, but might be a good option for those who are looking for both Ruby on Rails consulting and a cloud hosting platform in one package.

DotCloud

Setup and deploy process: After creating a DotCloud account, setup and deploy operations are done via the DotCloud command line interface. The CLI handles setting up the stack, deploy, rollback, and a number of other operations. The CLI integrates with Git and Murcurial.

Starter plan: DotCloud offers a free starter plan with two services, where a service is a single connection to a stack component. Or if you were lucky enough to get in during the private beta, there is a free VIP plan.

Other plans: The pro plan at $99 per month allows four services, and a flexible enterprise plan is also available with pricing depending on the requirements of the project.

Version Control: Works with all types of version control, although Git and Mercurial have the best integration with DotCloud’s CLI.

Conclusion: DotCloud is a more general platform than the others; it allows developers to mix and match stack components. Currently in beta: Java, PHP, PostgreSQL, Python, Redis, Ruby, and SMTP. There are a number of other components in alpha or on the roadmap for future support. It’s most appropriate for people who need support for a variety of stack components and don’t mind running on a beta platform.

Final Words

Each RoR platform has its particular strength, and picking the right one at the beginning will save headaches and money further down the road. This article covers the technical details of the different cloud platforms, but glosses over other important considerations like the reliability, transparency, and quality of support offered by the platform.

Please chime in with a comment describing your experiences if you have used any of the above cloud platforms.

There are a few different ergonomic keyboard designs, the most popular are shown below from left to right: the split keyboard, the angled split keyboard (sometimes called a Klockenburg keyboard), and the contoured keyboard.

So which of these will make you the most productive? The most likely to prevent carpal tunnel and other injuries dreaded by programmers? Probably none of them.

There is research showing that ergonomic keyboards increase typing speed. There is research showing that the same keyboard decreases typing speed. Studies show that ergonomic keyboards increase productivity, that they decrease productivity, that they decrease injuries, and that they have no effect on the number or severity of injuries. Whatever conclusion you wish to show regarding ergonomic keyboards, you are bound to find at least a few studies supporting that conclusion.

Sifting through the research, the only clear trend that emerges is that much of the research showing the benefits of ergonomic keyboards was done by manufacturers of ergonomic keyboards, by companies who are in the business of injury prevention consultation, or companies that have some other vested interest in the success of ergonomic keyboards. Further muddying the waters, Google Scholar search results on the topic of ergonomic keyboards are polluted by marketing documents disguised as research, such as this document by Goldtouch, a popular brand of adjustable split angled keyboards.

So which keyboard should you buy? Scientifically speaking, it probably doesn’t make a damn bit of difference as long as you are comfortable and have had sufficient time to adapt to any peculiarities of your keyboard. Test a few out and then get a keyboard that you like. No matter if that is an ergonomic keyboard, a cheapo minimalist keyboard, an old school mechanical keyboard with backlighting, a medical grade waterproof keyboard, or whatever keyboard makes you happy.

There are plenty of other tweaks you can make to your work environment that do increase productivity (and are backed by solid scientific evidence). Read more about how to increase productivity by reordering your office, but forget about ergonomic keyboards.

Bring Nature into the Office

Studies have shown that certain types of environments are restorative: they reduce mental fatigue, increase the ability to concentrate, and improve health, leading to an increase in productivity. One of the most effective restorative environments is nature. An office with a window that has a natural view is enough to trigger the benefits.

Those with a view of nature felt less frustrated and more patient, found their job more challenging, expressed greater enthusiasm for it, and reported higher life satisfaction as well as overall health.^[1]

Colorful, diverse landscapes with natural composition are preferred, and views of trees, water, and sky can increase the restorative effect.

Take action to increase productivity

Take micro-breaks throughout the day to gaze at the natural elements around you. If you aren’t lucky enough to have an office with a window, or if your view is a cityscape instead of a landscape, you can still bring touches of nature to your office. Live plants, tabletop fountains, or posters of nature scenes will do the job as well.

Use High Color Temperature Light Sources

Research has shown that working under light with a high color temperature has a number of benefits such increased mental acuity, vitality, and alertness, increased work performance, reduced fatigue, reduced daytime sleepiness, and improved mental health.

“Color temperature” refers to the color of the light produced; it is independent of the brightness of the bulb. Bluer lights increase productivity and alertness, while yellower lights promote relaxation. Most compact fluorescent bulbs have a color temperature of 2700K or 2850K, and the highest color temperature that is readily available commercially is 6500K.

Studies have shown that the 7500K light sources are correlated with higher mental acuity compared to 3000K light sources.^[2] 5000K sources are correlated with reduced drowsiness compared to 3000K.^[3] 17,000K sources are correlated with improved concentration and thought clarity compared to 4000K.^[4]

Take action to increase productivity

Choose light bulbs that have a high color temperature and a comfortable brightness. Try these 23 watt bulbs, or these 26 watt (equivalent to 100w incandecents) bulbs.

Use a Highly Adjustable Chair

A highly adjustable chair combined with ergonomics training has been shown to improve health by reducing the incidence of musculoskeletal pain an injury. A reduction in bodily pain was observed throughout the day, and the study found a significant economic benefit in investing in an ergonomic chair.

[...] the chair-with-training intervention is associated with productivity improvements of $354 per worker per day and has a benefit-to-cost ratio of 22:1^[5]

An ergonomic char is one of the more expensive workspace changes to implement, but the benefit-to-cost ratio clearly shows that it is worth it.

Take action to increase productivity

Invest in a highly adjustable chair and learn how to use it properly. Have a look at Herman Miller’s line of highly adjustable Aeron chairs. Use a chair with adjustable armrests in height, width, and pivot, adjustable chair height, a flexible back support, adjustable firmness support in the lower back, adjustable seat depth, and a gliding mechanism that allows the seat to glide forward as you recline.

Keep the Ambient Temperature Comfortable

There has been quite a bit of research on the effect temperature has on productivity. The graph below is taken from a meta-study which summarizes the findings of a number of studies on the relationship between temperature and performance.

There is a lot of variation in the optimal temperature in terms of productivity. This may be caused by personal variations in comfort zone temperature, seasonal variations, or climate differences between the locales where the studies were conducted.

Further muddying the issue is that some studies provide evidence that for optimal work performance the temperature should be in the comfort zone, but other studies show that temperatures outside of the comfort zone cause arousal effects that increase performance.

Based on our review, available data do not provide compelling or consistent evidence that temperature variations within the comfort zone significantly affect worker performance. However, performance decrements are more clearly established for temperatures outside of the comfort zone. Decrements are most clearly documented for high temperatures.^[6]

Take action to increase productivity

Maintain an ambient temperature that is comfortable or slightly cooler than your comfort zone. Avoid working in high temperatures.

1. Rachel Kaplan, The role of nature in the context of the workplace, Landscape and Urban Planning, Volume 26, Issues 1-4, Special Issue Urban Design Research, October 1993, Pages 193-201, ISSN 0169-2046, DOI: 10.1016/0169-2046(93)90016-7.
2. Deguchi T, Sato M: The effect of color temperature of lighting sources on mental activity level. Ann Physiol Anthropol 1992, 11:37-43.
3. Noguchi H, Sakaguchi T: Effect of illuminance and color temperature on lowering of physiological activity. Appl Human Sci 1999, 18:117-123.
4. Peter R Mills, Susannah C Tomkins, and Luc JM Schlangen, The effect of high correlated colour temperature office lighting on employee wellbeing and work performance, Journal of Circadian Rhythms 2007, 5:2 DOI:10.1186/1740-3391-5-2
5. Amick, Benjamin C. III, PhD; Robertson, Michelle M. PhD; DeRango, Kelly PhD; Bazzani, Lianna MPH; Moore, Anne PhD; Rooney, Ted MPH; Harrist, Ron PhD, Effect of Office Ergonomics Intervention on Reducing Musculoskeletal Symptoms, Spine, Volume 28, Issue 24, 15 December 2003, Pages 2706-2711
6. Seppanen, Olli, Fisk, William J., & Faulkner, David. (2003). Cost benefit analysis of the night-time ventilative cooling in office building. Lawrence Berkeley National Laboratory: Lawrence Berkeley National Laboratory. LBNL Paper LBNL-53191.

This is the third design since GrokCode went live in 2007. Unfortunately I don’t have a screenshot of the first design, but it was ugly enough that perhaps it is better off forgotten.

What’s different?

There are quite a few changes, but they all tie back to 3 main goals: keep the emphasis on the articles, include more information for prospective clients, and try not to be so ugly.

Emphasize the articles

The programming and software development articles are the heart of GrokCode, and I wanted that to be obvious. The article pages are now less cluttered and there are fewer distractions.

GrokCode has never been ad-heavy, and the new design further reduces the number of ads. Many pages are add-free, and there is never more than one ad per page. If you would like to support the continued creation of new articles, you can buy yourself a few computer science and programming books (or anything else) from Amazon.

Other tweaks to improve readability and keep the layout neat include cleaning up the sidebar, moving comments to their own page, and switching to a fixed-width layout.

Meet new clients

The old design had virtually no information for prospective clients. There was a portfolio of work, but no information about how I work, what I do, and nothing to even really suggest that I am taking new projects. The new design remedies this with new information on working with me, a better about page, and an updated portfolio. The idea is to make this info accessible and obvious, but not in-your-face.

Don’t be ugly

Even though I am not a designer or a frontend developer, I do all of the GrokCode designs myself from scratch to get a peek into how the other half lives. I was aiming for a beautiful, minimalist, and a slightly retro look, and I spent quite a bit of time reading and learning about logo design, color selection, typography, copy writing, etc. In the end I’m satisfied that it’s not ugly.

Design critiques are welcome in the comments.

What’s broken?

The switch went off without too many problems. Other than a problem with the feeds, all of the remaining issues that I am aware of are all “won’t fixes” or “can’t fixes.”

Feeds

The only lingering issue that I have on my list to fix is that the feeds aren’t working quite right yet. If you are subscribed to the feed via RSS or email, you probably saw that a bunch of old articles were marked as new and resent. You may also be seeing an excerpted feed. Sorry about that. If you haven’t yet subscribed to the feed, you can subscribe here; things will be back to normal shortly.

Details

A few of the details are “broken by design,” which is to say that some of the nice extras degrade gracefully on older browsers and some browsers don’t properly render valid HTML5. Since this is a personal site, I have the luxury of saying that I don’t care too much if things are a little wonky on browsers that don’t comply with standards.

If you are one of the 17% of GrokCode visitors using an outdated or non-standards compliant browser (*cough* IE) things just won’t look as nice. Have a look at Firefox 3.5 vs. IE 8.

The retro style is a bit spoiled by the boxy links, and the some of the finer details like rounded corners and transparency are lost.

Fonts

Again, the fonts aren’t exactly broken, but some people will have a better experience than others.

If you are one of the 32% of GrokCode readers on Windows XP, the fonts will be rendered poorly. This isn’t just a problem with GrokCode or web pages in general – Windows XP is just bad at fonts. I had no idea that the problem was this bad until doing cross-browser testing, and I used to use XP quite often. I had to modify the original design so that it would be readable on XP (I believe Vista and 7 are better).

Have a look at the side by side of Ubuntu and Windows XP. You are looking at the difference between aliased and anti-aliased text.

Other than that, I think everything is working fine. If you see anything strange or something doesn’t seem to be working, please let me know via comments or email. Friendly advice and constructive criticism on the redesign are also welcome.

Just a quick refresher, TDD looks like this:

1. Create a test
2. Run all tests, making sure the new test fails
3. Write the minimum amount of code to make the test pass
4. Run all of the tests to ensure they pass
5. Refactor the code, making sure that the tests still pass
6. Repeat for each new code module being developed

TDD has strong supporters and detractors. At one extreme, agile consultants like Robert Martin (Uncle Bob) fiercely advocate a TDD process in all cases, asserting that unit testing saves time during development. On the other side, grey beards like Knuth talk about using a large up front design process with absolutely no testing until the end of a project. These are the outliers though – not many of us make our living as an Agile consultant and author, nor are we writing the definitive text on Computer Science.

Peter Seibel used excepts of his book, Coders at Work, to put together a nice synopsis of both sides of the TDD debate. The central criticism of TDD is that it slows developers down, while TDD proponents claim that unit testing does not slow a project down, and that in cases TDD actually speeds things up. So who is right here?

Research done by IBM and Microsoft indicates that TDD teams took 15-35% longer than teams using more traditional development practices. However, the bug density of projects developed using TDD decreased 40-90% relative to similar projects that did not practice TDD. There is nothing here to support the claim that Test Driven Development is faster, but the study leaves an important question unanswered.

If we control for defect rate, is TDD faster? Could the non-TDD teams have produced code with a similar defect rate if they had used as much development time as the TDD teams? I would be willing to bet that they could. So why didn’t the non-TDD teams take more time in order to reduce their defect rate? Why did they declare the project done and ship with so many outstanding bugs?

The IBM-Microsoft research used case studies of real projects developed in a professional setting. Accordingly, these projects experienced the same schedule pressures and race-to-the-finish bug fixing sessions that professional programmers are all too familiar with. TDD developers iterate rapidly between test and code. Since testing is such an integral part of the development methodology, they are unlikely to skimp on test coverage even when deadlines loom. However non-TDD teams are more likely to back load testing, which allows them to met unrealistic deadlines by sacrificing testing time. Anecdotal evidence certainly supports this scenario.

There’s bound to be stuff where this would have gone faster if we’d had unit tests or smaller modules or whatever. That all sounds great in principle. Given a leisurely development pace, that’s certainly the way to go. But when you’re looking at, “We’ve got to go from zero to done in six weeks,” well, I can’t do that unless I cut something out. And what I’m going to cut out is the stuff that’s not absolutely critical. And unit tests are not critical.-Jamie Zawinski in Coders at Work

Shunting the testing phase off until the end of a project makes it particularly vulnerable to schedule pressure, because the code is in some sense done when all of the features have been implemented. Even though the code is buggy, there is in fact a shippable product. It may be very tempting to just ship it, especially if the project has already experienced scheduling delays. In contrast, the same application developed with TDD won’t be nearly as vulnerable to schedule pressure because the test-up-front methodology delays the point where there is a shippable product.

So perhaps the most valuable effect of TDD is just a side effect of upfront unit testing: it relieves schedule pressure and allows teams to delay the point at which code can be called done.

In the beginning there was Make.

But Makefiles aren’t natively compatible with Windows, nor are they inherently portable across Unixes. For a platform-independent language like Java, this was a big problem. Even for applications that did not require platform independence, Makefiles were perhaps an overly complex solution for Java builds.

Then there was Ant.

Ant stepped in and did a good job of addressing the major problems with Make for Java builds. Since Ant itself is written in Java and it abstracts away many of the details of the underlying OS, build files are for the most part platform-independent. Ant was also written specifically to work with Java, so it has some built in assumptions about the build process which reduce the number of things that need to be stated explicitly.

So Ant was an improvement over Make in some ways, but it introduced its own set of problems.

XML and the angle bracket tax.

Ant build files are pure XML. The unavoidable verbosity of XML compared to flat files, JSON, YAML, or virtually anything else is a clear violation of DRY (Don’t Repeat Yourself). This verbosity is expensive in terms of readability, clarity, and ultimately the time it takes humans to parse and extract the precious few bits of information hidden amongst the garbage. I won’t go into this too much as it has already been discussed to death.^[1]

I’m sure some will say that nobody should be mucking about in the build files anyway. In this day and age, the IDE should handle all of that ickiness. Fair enough. Until Netbeans creates a build file that can’t be read by the next Netbeans version, or until my team switches IDEs, or until I decide I want to use Emacs instead, or until I want to set up a continuous integration server and have to add some custom targets not handled by the IDE. Eventually something will happen and you will need to peak peek under the covers.

And build files created by IDEs are not pretty. They work nicely when they work, but they are far more complicated than they need to be since they are designed to be general enough to support all kinds of crap.

The reality is that build files need to be hand-crafted for all but the simplest use cases, and the angle bracket tax must be paid.

XML as programming language

XML isn’t a language, it is a way of marking up documents. It is completely inappropriate for a build system where you often need the flexibility of a programming language. Lets look at conditionals as an example. Sure, there are ways of faking conditionals in XML markup, but which would you rather read?

A conditional in any sane language (here we use Ruby):

1
2
3

if prop1 and prop2 and not prop3
    print "Doing stuff\n"
end

Or a pseudo conditional in Ant:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

<target name="cond" depends="cond-if"/>
 
<target name="check-cond">
  <condition property="cond-is-true">
    <and>
      <not>
        <equals arg1="${prop1}" arg2="$${prop1}" />
      </not>
      <not>
        <equals arg1="${prop2}" arg2="$${prop2}" />
      </not>
      <equals arg1="${prop3}" arg2="$${prop3}" />
    </and>
  </condition>
</target>
 
<target name="cond-if" depends="check-cond" if="cond-is-true">
  <echo message="Doing stuff"/>
</target>

The above example is straight out of the official Ant FAQ. Yuck.

Or maybe you want to loop through a set of files and perform a set of operations on them. XML (and by extension Ant) is not the tool of choice for this. We really want a simple Turing-complete language, not an ad hoc set of XML tag soup for a build system. A complex build process is a dynamic series of operations; it is unwieldy to shoehorn this into the XML syntax which was intended for a different purpose.

Ant’s implementation

Even ignoring for a moment the gross inappropriateness of XML, Ant’s implementation leaves much to be desired.

Take as an example a problem I ran into recently: use a set of jar files in the classpath, and then copy the jars and package them up with an application. Seems fairly simple right? Wrong. I could set up the classpath as a property, but Ant’s copy task requires a FileSet. OK. But creating a FileSet requires a base directory containing all of my jars. Some of my jars live with my project, and others are part of Tomcat on the machine I am using to build. So now what? Use root as the base directory? That seems ugly. Surely there must be a way to convert my classpath property to a FilesSet? Or to use Ant’s Path which can contain multiple files. Nope. If there is a way, it is well hidden. Maybe I could just shell out and use copy on the local machine. In that case I loose the platform independence Ant gives me; why didn’t I stick with Makefiles? So in the end the solution is to list the files twice: once in a format that I can use in a classpath, and again in a format that can be used for copy. So now I’m violating DRY again, but on a whole different level.

Ant doesn’t just encourage the creation of unmaintainable copypasta, it requires it.

We need something better.

Along comes Maven.

Maven’s philosophy is convention over configuration. So far so good. But Maven uses XML too, so it comes with all of the problems inherent in that choice. In fact, Maven completely ignores all of the pain points created by Ant, and aims to solve a completely different problem.

Maven’s “killer feature” is that it will check your project dependencies and pull new versions from upstream. Its design encourages you to ride the upgrade treadmill. Maybe I don’t want to put my application at risk of breaking every time upstream does something clever. Some applications don’t need shiny features, they need stability. This is something Java people should understand; the whole Java ecosystem revolves around an excellent set of standard libraries that are well tested and solid.

But even more troubling is that Maven introduces non-determinism into the build process.

You have no control over, and limited visibility into, the dependencies specified by your dependencies. Builds will break because different copies of Maven will download different artifacts at different times; your local build will break again in the future when the dependencies of your dependencies accidentally release new, non-backwards compatible changes without remembering to bump their version number. Those are just the innocent failures, too; the far more likely scenario is your project depends on a specific version of some other project which in turn depends on the LATEST version of some other project, so you still get hosed even when downstream providers do remember to bump versions! Every release of every dependency’s dependencies becomes a new opportunity to waste several hours tracking down strange build failures.

But Maven is even worse than that: not only does Maven automatically resolve your project’s dependencies, it automatically resolves its own plugins’ dependencies, too! So now not only do you have to worry about separate instances of Maven accidentally downloading incompatible artifacts (or the same instance downloading incompatible artifacts at different times), you also have to worry about your build tool itself behaving differently across different machines at different times!-Kent R. Spillner on Ant vs. Maven

We still need something better.

So whats next?

There aren’t any clear successors to Maven; certainly there is nothing on the horizon that looks like it is on the way to attracting the level of popularity currently enjoyed by Ant or Maven.

One interesting option comes full circle: Rake – Ruby Make. Rake looks quite promising; it uses Ruby syntax, not XML, and comes with a library of prepackaged tasks to make build files simpler. Other tools to investigate are Make in Java – JMK and APB.

Update: Several people in the comments are recommending Buildr or Gradle.

1. See Jeff Atwood’s XML: The Angle Bracket Tax and a rebuttal by Norman Walsh Defending the Tax for a nice overview of the discussion.

If Computer Science has one definitive text, then TAOCP is it. So if you don’t have a copy in your library already, now is as good a time as any to pick it up.

You can pre-order Volumes 1-4A of TAOCP from Amazon now. All Amazon pre-orders come with a price guarantee, so if the price drops before the book ships, you will automatically get the lower price.

The boxed set includes Volume 1: Fundamental Algorithms, Volume 2: Seminumerical Algorithms, Volume 3: Sorting and Searching, plus the new Volume 4A: Combinatorial Algorithms.

The 4th volume is the first part in a series on Combinatorial Algorithms. Its coverage begins with Boolean functions and bitwise tricks and techniques, then treats in depth the generation of all tuples and permutations, all combinations and partitions, and all trees.

Here is the full table of contents for the new volume:

Volume 4A: Combinatorial Algorithms, Part 1

• 7. Introduction
  • 7.1. Zeros and ones
    • 7.1.1. Boolean basics
    • 7.1.2. Boolean evaluation
    • 7.1.3. Bitwise tricks and techniques
    • 7.1.4. Binary decision diagrams
  • 7.2. Generating all possibilities
    • 7.2.1. Combinatorial generators
      • 7.2.1.1. Generating all n-tuples
      • 7.2.1.2. Generating all permutations
      • 7.2.1.3. Generating all combinations
      • 7.2.1.4. Generating all partitions
      • 7.2.1.5. Generating all set partitions
      • 7.2.1.6. Generating all trees
      • 7.2.1.7. History and further references