The Cafes

Would You Entrust Your Data to These Yokels?

2009-12-31T12:40:19Z

securely to www.delldatasafe.com, but we can't confirm that your connection is secure.

Normally, when you try to connect securely,
sites will present trusted identification to prove that you are
going to the right place. However, this site's identity can't be verified.

What Should I Do?

If you usually connect to
this site without problems, this error could mean that someone is
trying to impersonate the site, and you shouldn't continue.

www.delldatasafe.com uses an invalid security certificate.

The certificate is not trusted because the issuer certificate is unknown.

(Error code: sec_error_unknown_issuer)" title="delldatasafe" width="811" height="602" class="alignnone size-full wp-image-1002887" />

If Dell can’t even manage their public key certificates, how can I trust them to keep my data safe and secure?

Technical Details

www.delldatasafe.com uses an invalid security certificate.
The certificate is not trusted because the issuer certificate is unknown.
(Error code: sec_error_unknown_issuer)

SourceForge for the 21st Century

2009-12-31T12:39:46Z

Lately I’ve been thinking a lot about continuous deployment for reasons I’m not quite yet at liberty to disclose. This has inspired me to improve the XOM release process, to make it more of a one click process, or, to be more accurate, a one ant target process. I can now release a new version simply by typing:

$ ant -Dpassword = secret -Dwebpassword=other_secret release

This not only builds the entire project. It tags the release in CVS, uploads the zip and tar.gz files to IBiblio, and uploads the documentation to my web host. It doesn’t yet file a bug to upload the maven files, but I’m working on that.

During the process of setting this up, I realized that my organization is a little backwards. In particular, I’m pushing all the artifacts from my local system. Instead, I should merely be committing everything to the source code control repository; tagging a release; and then having the further downstream artifacts like the zip and tar.gz files and documentation pulled from source code control onto the Web servers.

There are some commercial products that are organized like this, including ThoughtWorks’s Cruise, but none of the major open source hosting sites such as SourceForge and java.net work like this. Certainly, SourceForge and similar sites have been major contributors to the open source revolution. They have enabled hobbyist developers working in their garages to use tools and techniques of software development that were previously limited to corporations. They have it enabled far-flung developers around the world to collaborate with each other far more effectively than they could do by e-mailing each other tar files. They have removed the burden of system administration from many programmers, thus enabling them to devote more time to writing code. Make no mistake. SourceForge et al. are real force for good in the community.

That said, the state of the art in software development has moved forward significantly since these sites were founded. CVS has mostly been replaced by Subversion. On some projects, Subversion has been been replaced by distributed version control systems such as git and Mercurial. Unit testing and test driven development have moved from extreme practices to standard operating procedure. Continuous integration using products like Hudson and Cruise Control is routine. Nonetheless, most project hosting sites still offer little beyond a source code repository, a bug tracker, and some webspace. Not that that’s not important, but we can do so much more.

It’s time to think about what a modern project hosting site might want to offer and what it might look like.

Continuous Integration

The first step forward, and possibly the hardest, is to add continuous integration capabilities to the existing project hosting repositories. Every time code is checked into SourceForge or Java.net or code.google.com, the project should be built and the tests should be run. Technically, the hard part is understanding every project’s unique build infrastructure. Some projects use ant; some use make; some use maven; and some roll their own. Maven is probably the most constrained of a lot. For the others, it will be necessary to ask project owners which targets to run for which tasks. It’s probably a good idea to auto generate basic ant or maven or make scripts for new projects.

Beyond merely building the code, running the tests has some very serious security implications. Currently project hosting sites do not run third-party code. They store it; they display it; they make it available and bundle it up as tar and zip files; but they don’t even compile it, much less run it. Running arbitrary third-party Java and C code submitted by any random teenager with attention deficit disorder somewhere on the Internet is begging for trouble. 10 years ago I would’ve thought this was insane and impossible. But now, just maybe we can do it.

In fact, there are several services on the Internet today that will run arbitrary third-party code for all comers. Amazon’s EC2 service lets anybody with a credit card run what amounts to a complete rooted Linux box on Amazon’s network. Google’s AppEngine let’s more or less anyone, credit card or no, run Python and Java code inside Google’s cloud. And these are hardly the only such services. Advances in virtualization and security sandboxing have made this possible. That said, it certainly helps to have a real user attached to any code that you run so you know who to blame when it starts spamming the world. However when an application goes rogue whether through malice or incompetence, it is possible to shut it down quickly. It is possible to limit the resources used by anyone test suite, and to limit what else I can see on the same filesystem and the same network.

Codehaus uses Atalassian Bamboo to provide continuous integration, including test running, for their projects. However they’re a relatively small site that’s somewhat picky about the projects they host. They do use a separate server for the continuous integration. I’m not sure what other, if any, security precautions they put in place. Launchpad builds Ubuntu packages, but I’m not sure of they run tests. JavaForge builds Java code and runs the unit tests, apparently on top of Amazon EC2. Assembla will build and run tests, and also uses Amazon EC2. Both of these thereby delegate some of the security issues to Amazon’s virtualized systems.

submit queue

Once we’ve solved the problem of running continuous integration servers on project hosting sites, the next step is to flip them around. The usual process is to commit code to the repository and have the continuous integration server pull the code out of the repository. Then, if the build or tests fail, the continuous integration server goes into red mode and sends out alerts. Wouldn’t it be better if the server never turned red in the first place?

What should happen is that new code gets sent directly to the continuous integration server rather than to the source code repository. The continuous integration server pulls the latest known good build from the repository. then it patches the new code into the build and runs the tests. If the tests pass, the continuous integration server commits the code to the repository. If the tests fail, the code is never committed at all.

So far as I know, no current project hosting sites offer this; and it’s a relatively uncommon feature even among self hosted projects. However, it’s a critical one, especially when accepting contributions from the wide world of programmers, not all of whom have yet learned the importance of test driven development. I suppose such a site could also perform other checks on the source code. For example, it could verify coding conventions or measure the incremental code coverage before and after the check-in. It could automatically reject any patches that did not meet some predetermined measures of quality. That said, automated checks tend to be better used as additional data for humans to evaluate rather than as hard and fast rules. One way this can happen is by offering code metrics to code reviewers. This brings us to the next improvement in the code hosting ecosystem.

Code Reviews

Committing code, even assuming all the tests pass, is still a serious operation. Most open source projects don’t want to allow just anyone to commit code willy-nilly. Usually there’s a core group of committers that reviews all incoming patches and decides whether or not to accept them, to reject them, or to send them back for further work. This is somewhat labor-intensive both on the reviewer and the reviewee.

However, if we move to a submit queue-based system, this can become somewhat more straightforward. The continuous integration server can check every incoming patch regardless of the submitter’s status. If the tests pass, it can send an automatic request for review to a project commiter. If the commiter approves the change, then the continuous integration server can commit it to the source code control repository.

Indeed, it’s probably a good idea to require code reviews for all submitted changes, not just those from new users. After all, it’s not like the project’s owners are immune from introducing bugs. In fact, they probably introduce more than anybody else, if for no other reason than that they commit more code than anyone else. Code reviews are well known for increasing the quality of a code base and avoiding stupid errors, yet they’re one of the lesser used software development practices among open-source programmers. It’s time for that to change. Web-based code review interfaces such as Guido von Rossum’s Rietveld have the potential to really move the community forward here. We should integrate this technology or something equivalent into project hosting sites. code.google.com already offers code review, and a few others like BitBucket do too. The rest should follow.

One-button deployment

The final stage of software development is deployment. Eventually the software has to ship to and be installed by its intended users. Here is one area where open source projects have a significantly easier time than a lot of commercial projects, especially enterprise projects. The deployment process for many open source projects consists of little more than uploading a few jar files and some documentation to the right directories on the right Web servers. This should become a one-button operation.

All of project owners should have to do to release a new version is choose a version number and push a button. The server should pull all the code, documentation, and configuration information out of the source code repository; build everything; and put all the finished artifacts in the right locations. No further manual work should be required. This does require that absolutely everything needed to release goes into the repository; not only code but also HTML files, images, config files, and more. The only things that don’t go into the repository are the artifacts that are built from these components: jar files, zip files, Javadoc, etc.

Maven comes close to this, but it still builds and deploys from a local system rather than from the version control repository. This should be turned around. Ideally, maven deploy should work with nothing more than a pom.xml file. Deploying shouldn’t need to access the local maven repository or the local copy of the source code at all.

Summary

There might even be a startup idea in here somewhere. Open source projects aren’t the only ones that would like to offload some of the routine system administration tasks involved in running source code control repositories, continuous integration servers, bug trackers, and deployment pipelines. More likely, what’s really needed are some tweaks in and additions to the existing project hosting services. Or perhaps we can even take advantage of the advances in virtualization technology to install these services on top of Amazon EC2 and similar platforms.

But one thing is for certain: if open source projects are to keep pace with and surpass closed systems, then their software development practices need to be at least as good and probably better than the state-of-the-art in the overall software development community. In order to do that, it’s time to upgrade our tools.

Why Don’t Macs Support Multiple Monitors?

2009-09-27T20:12:05Z

I have two 23″ monitors on my desktop at work, and have worked that way for about three years now (aside from a brief flirtation with a single 30″ monitor in California). On Windows and Linux this is an incredibly productive setup. I can have a full screen IDE open on one and a full-screen web browser open in the other. The web browser gives me a huge reference library and easy access to a lot of apps including e-mail, calendar, and more, and the IDE lets me do my work. I can easily switch back and forth between them to surf or edit. It’s a smooth and fluid workflow. Even a single monitor twice the size doesn’t work as well since you can’t easily organize the two applications on the screen.

I’m a programmer but the same is true for anyone who works primarily in one large application. For instance, for designers it might be Photoshop or QuarkXPress. For writers it may be Microsoft Word. For business folks it could be Excel. We all need a web browser open and we all need our main productivity app. On Windows and Linux these days, this just works. You plug-in two monitors. You open two apps. You move between them as you feel like it, and do your work. This is what it looks like:

On the Mac, however, it doesn’t work. The Mac, which was perhaps the first platform to support multiple monitors, certainly the first consumer platform, a two monitor setup looks like this:

Do you see the difference?

The problem is that the Mac always and only puts the menubar on one screen. That means using any app on the second screen means constantly redirecting your attention back to the first screen. Furthermore, you have to constantly move the mouse between screens, rather than just slamming the mouse to the top as Mac users are accustomed to doing. It may not seem like much, but after you’ve grown accustomed to a multiple monitor setup on Windows or Linux, trying to use one on the Mac is jarring. Not only does it not just work. It just doesn’t work.

The solution should be simple: put a menu bar on each monitor, at least optionally. The menu bar can even be disabled until the app is focused with a click. Why Apple hasn’t done this, I don’t know. For single screen computers, the Mac still has by far the most efficient menubar design and the best multi-application UI ever invented. Windows and Linux still haven’t caught up with where the Mac was in 1984. A menu bar belongs at the top of a screen, not the top of a window. But unfortunately the Mac doesn’t put a menu bar belongs at the top of a screen. It puts the menu bar belongs at the top of one screen, and that makes all the difference.

A few applications can actually use multiple displays for their own needs without the single menu bar approach crippling the user. For instance, PowerPoint can show the slides on one display (typically a laptop) and speaker’s notes on the other (typically a projection system). The key here, though, is that such applications are tiling a single application across several monitors, and are designed for this environment. However this is rare. Most applications are designed to run on a single screen, and can’t really make use of a second monitor. To gain extra productivity with extra monitors, you need to be able to show different applications on different monitors and still have each application be fully functional.¹ Without that ability the second monitor is just eye candy. In fact, maybe it’s not even that. I’ve never even found something as simple as a slideshow program or screensaver that can take over one monitor without blacking out the second.

Virtual desktops are not a solution either. The goal is to be able to see the web browser and the IDE (or whatever applications you happen to want to run simultaneously) at the same time, not merely to be able to switch back and forth between them. The eye can move faster than mouse, and sometimes that’s important.

I suspect this problem just wasn’t obvious circa 2000 when the last major refresh of the Mac UI was contemplated. At the time, multi-monitor setups were still rare and pricey things. However it’s 10 years later now. Video cards that support multiple displays are stock equipment in every Mac, and large flat screen monitors can be had for the price of a nice meal in Palo Alto. It is well past time to fix this. Until Apple does, the Mac is, at least in this respect, a distinctly inferior platform for software development, photo editing, page layout, spreadsheet analysis, and other tasks that require a large, full-screen interface.

A Square Is Not a Rectangle

2009-09-11T15:07:34Z

The following example, taken from an introductory text in object oriented programming, demonstrates a common flaw in object oriented design. Can you spot it?

public class Rectangle {

  private double width;
  private double height;

  public void setWidth(double width) {
    this.width = width;
  }

  public void setHeight(double height) {
    this.height = height;
  }

  public double getHeight() {
    return this.height;
  }

  public double getWidth() {
    return this.width;
  }

  public double getPerimeter() {
    return 2*width + 2*height;
  }

  public double getArea() {
    return width * height;
  }

}

public class Square extends Rectangle {

  public void setSide(double size) {
    setWidth(size);
    setHeight(size);
  }

}

(I’ve changed the language and rewritten the code to protect the guilty.)

There are actually several problems here. Thread safety is one, but let’s assume the class doesn’t need to be thread-safe. Another is that it’s possible to give the sides negative lengths. That we could fix with a couple of judiciously thrown IllegalArgumentExceptions. However the problem that most troubles me is demonstrated here:

Square s = new Square();
s.setSide(5.0);
s.setHeight(10.0);

In object oriented programming, it is necessary that a subclass be able to fulfill the contract of its superclass. In this case, that means the square has to respond to setHeight() and setWidth() calls. However doing so enables the square to violate the nature of a square. A square cannot stand in for a rectangle.

You can try to work around this by overriding the setHeight() and setWidth() methods. For example, one might call the other:

public class Square extends Rectangle {

  public void setWidth(double width) {
    super.setWidth(width);
    super.setHeight(height);
  }

  public void setHeight(double height) {
    super.setWidth(width);
    super.setHeight(height);
  }

  public void setSide(double size) {
    super.setWidth(width);
    super.setHeight(height);
  }

}

However this is fundamentally unsatisfying because there is no reasonable expectation that calling one of setHeight() on a Rectangle object will also invoke the setWidth() method or vice versa. The contract of the Rectangle class is that you can set the width and the height independently, and the Square subclass violates that. Setting width when you’re setting height is an unexpected side effect. It violates the single responsibility principle¹.

We could instead just forbid setHeight() and setWidth() completely by throwing UnsupportedOperationException:

public class Square extends Rectangle {

  public void setWidth(double width) {
    throw new UnsupportedOperationException();
  }

  public void setHeight(double height) {
    throw new UnsupportedOperationException();
  }

  public void setSide(double size) {
    super.setWidth(width);
    super.setHeight(height);
  }

}

However, this is really just a louder way of warning the client that the Square class does not fulfill the contract of the Rectangle class. It doesn’t address the fundamental problem that, in object oriented terms, a square is not a rectangle. The geometric nature of a square is incompatible with the object-oriented definition of a rectangle given above.

There is, however, a way out of this conundrum. Our problem only arises because of the setter methods. If constructor were used instead, and no setters were exposed, then it would be possible to make a square a subclass of rectangle without violating any contracts. For example,

public class Rectangle {

  private double width;
  private double height;

  public Rectangle(double width, double height) {
    this.width = width;
    this.height = height;
  }

  public void getHeight() {
    return this.height;
  }

  public void getWidth() {
    return this.width;
  }

  public double getPerimeter() {
    return 2*width + 2*height;
  }

  public double getArea() {
    return width * height;
  }

}

public class Square extends Rectangle {

  public Square(double size) {
    super(size, size);
  }

}

As long as the Rectangle class is immutable, we can define subclasses that are limited to a particular subset of rectangles, such as squares. This is one more reason to prefer immutability. To the extent possible, define the public interface in terms of what an object is and what it does rather than what you can do to it. However that’s not always possible, and in those cases you need to be extremely careful around inheritance. Otherwise constraints can be violated when you least expect, thus introducing subtle and potentially dangerous bugs in your code.

¹ Actually the single responsibility principle is usually understood to apply to classes, but it’s even more critical that it apply to methods. each method should do exactly one thing and one thing only. Side effects should be avoided.

Why Pair Programming Works

2009-07-01T01:26:50Z

Pair programming is like magic in more ways than one. It dramatically improves programmer productivity and reduces bug count, and yet it does so through a technique that’s completely counter-intuitive. You can’t help but think that there’s some trick yet to be exposed; that pair programming is just slight of hand. In this article, I will endeavor to pull back the curtain and reveal the secrets of the pair programming magicians.

Specifically, I identify six reasons pair programming succeeds:

Continuous Code Review
Fewer blockages
Masking distractions
Guaranteed focus
Multiple points of view
Reduced training cost and time

Continuous Code Review

One of the maxims of extreme programming (XP) is to turn all the dials to 11. Code review is good, so continuous, instantaneous code review is great; and that’s exactly what pair programming delivers. No more back and forth over a week. No more waiting for feedback on a code review. Instead, feedback comes immediately. This means less time waiting for reviews and less time going down the wrong path. The time savings compared to classic code reviews is dramatic. In fact, this savings alone is likely to pay the cost of pair programming.

Furthermore the feedback is more likely to be useful that that from classic code reviews. When presented with a 60 file change list, the tendency for most reviewers is either to give it a perfunctory once over, or to merely focus on the little details like indentation without really seeing the bigger picture. However, when pair programming, both developers are actively involved with the code and can see what’s happening as it happens. Code reviews start and continue one line at a time and grow gradually with the code. It’s much easier to understand code this way than by having 60 classes and 3000 new lines of code dumped on you in one huge batch.

Finally, feedback can be incorporated more quickly and easily when it’s given earlier in the process. It’s bad enough having to review 60 classes and 3000 new lines of code. It’s much worse having to throw all that code away after a month’s work and start over because the reviewer has uncovered a serious design flaw. The sooner the flaws are noticed, the cheaper they are to fix. Pair programming is much less likely to produce misguided code.

And if you’re not already doing code reviews? Then introducing them through pair programming is going to improve your code quality and programming velocity that much more.

Fewer blockages

We’ve all experienced the annoying bug that just won’t die, no matter what we try. Sometimes we work on it for a day and then immediately see the answer when we wake up the next morning. Sometimes we work on it for two days or more until we finally break down and ask for help, at which point our cube mate says, “Oh, you’re implementing getProjectId() when you should be overriding getProjectID().”

Why wait?

Stupid bugs like these waste a lot of time; but blind spots are rarely shared. What one programmer doesn’t see, the other one likely does. Find one of these per day by pair programming, and you’ve paid for the extra pair of eyes. You may not even realize how much time you’re saving here because the mistake will be found so fast, and once you see it it will be so obvious, that no one will even consider they might have wasted a day on the problem had their partner not spotted it. Nonetheless the effect is very, very real.

On a related note, two minds are better than one. What one programmer doesn’t know the other one likely does. You may not know the details of cuckoo hashing (or even that it exists); but if your partner does, you’ll be able to use it when appropriate. On the other hand, you may be the world expert in the idiosyncrasies of cross-browser JavaScript DOM and save your partner hours of web surfing reading up on how to work around IE6.

As a special bonus, when pairs match programmers with different experiences and knowledge domains, each partner learns from the other. Knowledge effortlessly percolates across the team and the company. This doesn’t speed you up on the immediate project, but it does increase everyone’s skills and improve your velocity on future projects. This is especially important for growing companies.

Try to pair programmers with different skill sets and experience levels rather than similar ones. A pair
is as strong as the strongest pair member, not as weak as the weakest. In fact, it’s probably stronger. The harder and more urgent the problem, the more critical it is to use pairing. If you have some really thorny problem, for instance involving multithreading, don’t pair your top experts in multithreaded optimization. Pair one expert with the lowest intern on the totem pole, so you grow more experts.

If it’s a critical issue and you aren’t sure how or whether you can fix it, pair each expert with a lesser star, and have the pairs both try to solve the problem. Then the pick the best solution, or merge the best features of each. (Caveat: I only recommend this for really important work items that are critical, not for mere grunt coding. For most problems the quality of the solution doesn’t matter so much as just having some solution.)

Mask out distractions

Microsoft and Fog Creek give their programmers private, quiet, distraction-free offices. The rest of us work in cubicles where we have to listen to the web designer two cubicles over yell at Verizon for two days trying to get out of a cell phone contract. Noise is a real problem for many, many programmers. You just can’t think when your brain is in a blender. Your brain does not filter out conversations happening around you. In fact, when no one else is talking directly to you, it deliberately focuses in on these external conversations, whether they’re relevant to you or not, the exact opposite of what it needs to do. It’s like sitting at a table alone in a restaurant, and not listening to the conversation at the next table. You simply can’t do it. White noise would be better.

This is where pair programming really does seem like magic. Your brain has evolved to focus preferentially on a nearby conversation instead of a distant one. When you are focusing on the person next to you, your brain masks out all other conversations at any volume below a shout. The interaction between the pair is so high-intensity that low intensity noise from a cubicle environment is much less problematic for a pair than for a programmer working alone. You can still be distracted from the task at hand by your own name or a few other special cues, but it’s surprising how much noise you can ignore that would completely break your concentration when you’re working alone. That egregious cell phone contract the HTML jockey keeps shouting about? Not your problem any more.

Guaranteed Focus

The eight hour day is a myth. Between meetings, lunch, e-mail, water cooler chatter, personal errands, paperwork, birthday celebrations, collections for going away presents, and more, you’re lucky to get two hours a day of solid coding time. Four hours is great. And even if you have four actual hours to sit in front of the keyboard, how much of that is going to be wasted on e-mail and the Web? Even if the e-mails sent and pages googled are related to the immediate task at hand, it’s likely there’s another e-mail or page just a click away that isn’t work related but is interesting enough to distract you from the real job.

When pairing, programmers actually program. They don’t check e-mail, read blogs, play Scrabulous, and generally goof off. If they have to check a web page to look up the exact behavior of a class (which they’re less likely to do in the first place because one of them may know the answer even if the other doesn’t) they come right back to the IDE after finding the answer. Four hours spent pairing is much closer to an actual four hours spent coding.

One common objection to pair programming is that it’s hard to find four hour blocks in which to pair, especially in larger, more bureaucratic organizations where meetings occupy a large part of the day. But here too pairing can help. Put the pairing time on the shared calendar as another meeting so no one can schedule the pair into meetings when they plan to be coding. Reserve your time for pairing before others take it away from you for less productive activities.

Multiple points of view

Programming has two modes: writing code and reading code. Pair programming has three: writing code, reading code, and explaining code. Explaining what code does makes you look at it differently than simply reading or writing it. When explaining code you are more likely to notice that what you think code does isn’t what it really does.

How often have you discovered bugs in your program when explaining a detailed algorithm to a colleague? Often you see the problem before the person you’re explaining it to does. Not too surprising given that usually the person doing the talking is the one who’s most interested in what they’re saying. There’s nothing like trying to explain something to force you to realize that you don’t really understand what you thought you knew, or that the code doesn’t work quite like you thought.

When pairing, a programmer is always explaining. You’re not just typing while your mind’s really on the World Cup finals or the morning antics of the Zoo Crew on KUPD. The pair members type as they explain what they’re doing and why they’re doing it. Thus, they’re less likely to do something stupid in the first place. If the words coming out of your mouth don’t match the characters emitting from the keyboard, then your partner will likely notice and call you on it.

Reduced training cost and time

Some organizations spend significant (> 5% of annual salary) amounts of money on conferences, onsite classes, online courses, books, and other training experiences for their developers. They may spend several thousand dollars per programmer just to send them to one out-of-town conference, and hope they learn something valuable enough to be worth the airfare, hotel, conference admission, and lost time. Or they bring in a trainer for 1 to 5 days to run a class on a new technology, which can be somewhat more cost effective if you have 5 or more developers and specific subject matter that they need to learn in depth. However you still lose multiple developer-weeks of work to training. Other organizations spend hardly any money on training, and just struggle along with whatever skills the programmers hired on with, as well as whatever they can pick up by reading blogs like this one (though again usually read on company time).

Pair programming doesn’t eliminate the cost of the training, but it can eliminate the lost time and productivity. Instead of purchasing classroom instruction, onsite or online or off, hire a contractor experienced with the technology to pair with your programmers. (The contractor should also be experienced and comfortable with pair programming.) Rather than learning in a classroom (which only works for about 5% of people anyway) help your team learn on the job. Informal lunchtime discussions led by the trainer can cover broader issues and general principles, but the real training takes place as your team builds the system they need training for. Shuffle the trainer between different developers every half day or so, while the ones who’ve already spent time with him or her pair with as yet untrained programmers.

On your balance sheet, it looks like you’ve hired an extra developer rather than a trainer. On your schedule, you’ve replaced training time with development time. Nonetheless, you’ll get a better trained team that can move faster when the contractor leaves.

Although I know a few consultants who specialize in this sort of pair training, I admit they can be a little hard to find. If your team has enough experience with pair programming, you may just want to hire a classic code jockey contractor with experience in the technology you need to learn, rather than an explicit trainer or coach. Pair the contractor up with one of your programmers, and let the knowledge flow naturally. This is a much more effective use of the contractor’s time than just putting them in a room by themselves, and getting an incomprehensible black box out after a few weeks. If the hired guns work directly with your permanent developers, then your team will be ready to take over, maintain, and grow the code after the mercenary leaves. You won’t be trapped in a situation where you have to keep calling the contractor back every two months to spend a few days at $300 an hour fixing the bugs they put into the code in the first place.

Just Do It

Pair programming isn’t magic, but it is effective. Very, very effective. Smart managers and anyone else who cares about code quality and programmer productivity should demand pair programming. In the best code shops, no one codes alone.

In Praise of Draconian Error Handling, Part 2

2009-06-05T14:21:15Z

The fundamental reason to prefer draconian error handling is because it helps find bugs. I was recently reminded of this when Peter Murray-Rust thought he had found a bug in XOM. In brief, it was refusing to parse some files other tools let slip right through. In fact, XOM’s strict namespace handling had uncovered a cascading series of bugs that had been missed by various other parsers including Xerces-2j and libxml.

But before I describe what happened, let’s see if you can eyeball this bug. I’ll make it easier by cutting out the irrelevant parts so you know you’re looking right at the bug. Here’s the instance document we start with:

<!DOCTYPE svg SYSTEM
"http://www.w3.org/TR/2000/03/WD-SVG-20000303/DTD/svg-20000303-stylable.dtd">
<svg/>

And the referenced DTD is:

<!ENTITY % StylableSVG "INCLUDE" >
<!ENTITY % ExchangeSVG "IGNORE" >
<!ENTITY % SVGNamespace "http://www.w3.org/2000/svg-20000303-stylable" >
<!ENTITY % Shared PUBLIC "-//W3C//DTD SVG 20000303 Shared//EN" "svg-20000303-shared.dtd" >
%Shared;

Then in svg-20000303-shared.dtd we find this:

<!ATTLIST svg
  xmlns CDATA #FIXED "%SVGNamespace;"
  %stdAttrs; >

Not obvious, is it? In fact, I looked at this one for quite a while, and consulted several spec documents before Tatu Saloranta figured out what was actually wrong here. If it helps the relevant part of the XML specification is Section 4.4, XML Processor Treatment of Entities and References.

Give up? OK. Here’s what’s happening:

Although the parameter entity reference %SVGNamespace; appears in a DTD. it appears inside a default attribute value. The parameter entity reference is therefore not resolved until the attribute actually appears in the document. However, parameter entity references are not resolved in document content (including attribute values). There it’s just literal text. In essence this document is:

<svg xmlns="%SVGNamespace;"/>

Although this bug was nigh-on impossible to find by eye, XOM picked it right up. It noticed that Xerces was setting a namespace URI to %SVGNamespace;. XOM didn’t know what the namespace was supposed to be, but it knew that %SVGNamespace; was not a legal absolute URL, and it refused to allow that. Hence the bug was flagged, and we eventually figured out where and how in the DTD the bug really was. And then we figured out that this was an old DTD from a working dfraft spec, and the tool that was having trouble should be upgraded to the new DTD anyway.

However no other XML library I’m aware of would have caught this because none of them make the check that a namespace URL should be absolute. Technically that’s only good practice, not a strict requirement of the spec. However it is a strict requirement for Infoset based specs such as XInclude and XProc. A document that uses non-absolute URLs does not have an Infoset, and therefore one cannot use XInclude or XProc on them, at least not with any confidence that the results will be consistent from tool to tool because the spec simply does not define how to handle this case.

In practice, I have yet to encounter an XML document that did use a relative namespace URI for any good reason. I have had people request an option not to reject documents that use non-absolute URIs for namespaces, and merely allow any string to be passed in. However, if I accepted that request, then XOM would no longer catch real bugs like this. Furthermore, document producers would have less incentive to generate correct documents. The more error correction tools perform, the more error correction they have to perform. Documents gradually deviate further and further from the spec, and tools try to keep up. However different tools implement different forms and amounts of error correction, until eventually we might as well not have a spec at all.

Instead of participating in this race to the bottom, XOM refuses to play. It will parse namespace well-formed XML documents only. If you want to parse something else, you can’t. Fix the documents or fix the process generating the malformed documents. XOM isn’t changing.

Imagine There’s No Null

2009-05-28T04:20:43Z

A couple of weeks ago I spent a considerable amount of time chasing down bugs involving null in a large code base: null checks after a variable had already been dereferenced, nulls passed to methods that would immediately dereference them, equals() methods that didn’t check for null, and more. Using FindBugs, I identified literally hundreds of bugs involving null handling; and that got me thinking: Could we just eliminate null completely? Should we?

What follows is a thought experiment, not a serious proposal. Still it might be informative to think about it; and perhaps it will catch the eye of the designer of the next great language.

To explore this, let’s reverse perspective and think about primitive types for a bit. I’ve long advocated a completely object-based type system. The distinction between primitive and object types is a relic of days when 40 MHz was considered a fast CPU; and even then it reflected more the lack of experience with OO optimization than any real issue. Languages such as Eiffel perform quite well without separating out primitive types. But if we were to make int and double and boolean full object types, would it then be necessary to allow null to be assigned to these types as well:

int i = null;
double x = null;
boolean b = null;

I suppose we could just rule out assigning null to these types but this would be an unnatural distinction between primitive and object types, which is precisely what we’re trying to avoid. Still I’d hate to give up the default values for unassigned primitives, 0 for numbers, false for booleans. After all it would be really annoying to have to write a method to add two numbers like this:

public static double sum(double x, double y) {
  if (x == null) throw new NullPointerException();
  if (y == null) throw new NullPointerException();
  return x + y;
}

One of the nicest characteristics of primitive data types is precisely that you don’t have to worry about this. You know they’re always initialized.

So suppose we go the other way instead. Let’s allow or even require each class to define a default value object that will be used whenever a null variable of that type is dereferenced. For instance, for the String class the empty string is the obvious choice. Perhaps it could be defined by overloading the existing default keyword:

public final class String {
  default = "";

  // rest of String class here...
}

Or for a ComplexNumber class, the default might be 0+0i:

public class ComplexNumber {

  private realPart;
  private imaginaryPart;

  default = new ComplexNumber(0, 0);

  // rest of class here...
}

Then any dereferences would simply use the default value instead of throwing a NullPointerException.

public class ComplexValue {

  private ComplexNumber z;

  public void ComplexNumber increment(ComplexNumber delta) {
    z = z.add(delta);
  }

In the current regime, this throws a NullPointerException unless some other code has initialized z and delta is not null. But in this scheme it would simply adds 0+0i if either is null.

Sometimes, though, you might really want to forbid uninitialized values. Or perhaps there is no sensible default value. For instance, what would be a sensible default value for a java.net.Socket? To indicate that no default is available, simply omit the default declaration from the class. There are two incompatible ways we could interpret this: either this means a null dereference throws a NullPointerException (current behavior); or it means the compiler forbids any declaration of such a variable without an immediate assignment. Pick one. I’m not sure which of these makes more sense, though I think I prefer the latter. No accidental NullPointerExceptions is the goal.

Of course, sometimes three-valued logic is sensible. After all, very few things are really true or false. True, false, and unknown is a much more accurate division. Integers and floating point values can also benefit from an otherwise impossible value that can represent unknown, unset, or uncertain. To enable this, we can allow any value to be tested for equality with default like so:

if (x == default) {
  // Special handling for the default case.
  // We could even throw an exception here,
  // but that would only be by deliberate choice,
  // not something that happens unexpectedly.
}

This comparison would return true if and only if the variable x were set to the default object for its type. It would not return true if x merely had the same value. For example,

String s1; // assumes the default value
String s2 = ""; // explicitly set to the default value
if (s1 == default) {
  System.out.println("This branch is executed.");
}
if (s2 == default) {
  System.out.println("This branch is not executed.");
}

This proposal could even be added to the existing Java language. However backwards compatibility would only make it feasible for new types. Unless we can apply this to existing types like String, Integer, and Document, I don’t think it’s a strong enough idea to carry its own weight. However in a new language without any such constraints, it could dramatically increase program robustness by eliminating an entire class of common errors.

Cue John Lennon.

How To Shutdown a Computer

2009-01-16T15:33:32Z

Sometimes I still feel like we’re in 1982 when it comes to really basic things like turning off a computer. Why do we have to select shutdown from a menu? Why do we have to carefully save each open file? Why don’t programs stop when we tell them to? (Time Machine has now been spinning for hours, and won’t stop even though I’ve told it to.) Why is this so much more complex than it needs to be?

In the future, here’s how shutdown should work:

You flip the power switch.

That’s it. No shutdown menu item. No wait for the system to hibernate. No opportunity for applications to save data. Nothing.

Then, when you turn the power switch back on, the entire system is restored with all data, windows, and programs intact (optionally after typing in the usual username/password credentials.)

In 2009 there is no excuse for an application to lose data because it hasn’t been recently and explicitly told to save. Nor is there any excuse for an application to block the system from shutting down. When I turn my computer off, it should go off, and when I tell an application to quit it should quit. More to the point, the operating system should forcibly terminate that application. I am thoroughly sick of applications that think File/Quit (File/Exit for Windows users) is a suggestion. It’s not. It’s a command, and one they should immediately respond to.

Few applications today get this right: not on the Mac, not on Windows, not on the Web. (Web apps such as WordPress do actually get a little closer to this, but many still launch popups telling you you may lose data if you close a window. There should be no such popups because they shouldn’t lose data, ever.) Firefox helpfully offers to restore all the open windows when the applications is relaunched, including after a force quit. More applications should do this. However even Firefox occasionally gets wedged and refuses to quit when told to.

There are actually a few steps to reaching this simplified nirvana, so maybe I’m asking for more than one thing here. Most laptops and applications can handle a system going to sleep. We just need to make sleeping a little faster (or at least less intrusive to the user) and possibly hook that up to the power switch in the hardware.

However, when waking from sleep, the operating system restores each application to the state it was in previously. We also need to make sure that applications are ready to be terminated with extreme prejudice in circumstances where the operating system will not restore their stack and heap. In such circumstances a character or two of data loss may be acceptable, but no more. Applications should save constantly to persistent storage without waiting for the user to tell them to do so. Furthermore, when writing complicated data structures they should maintain integrity even when they’re interrupted in media res. The techniques to do this are not complicated are hard to understand. They have been implemented by server software like Oracle for years now. We just need to demand the same sort of reliability from our desktop software.

A little more bleeding edge (but only because no one has bothered to do this yet) applications should maintain data integrity even in the face of flaky and unreliable hardware. Filesystems should not be tied to any particular physical drive. (No, RAID helps but doesn’t quite achieve this. RAID controller failures still cause data loss.) Better operating system support and more abstract filesystems that aren’t tied to particular disks would help here. However in the absence of this, applications could still store their data across multiple file systems, and they should do this without asking the user where to save or how to configure the disks. (Whether a traditional hierarchical file system is still the right metaphor for storage is a subject for another article.)

We take too many things for granted merely because that’s the way they’ve always been. But what made sense in an era of 64K floppy disks that could write a hundred bytes a second no longer makes sense in an era of terabyte hard drives that write a hundred megabytes a second. It is time to make saving files as obsolete as remembering to fill your fountain pen.

In Praise of Draconian Error Handling, Part 1

2009-01-12T16:22:27Z

I’m doing a bit of work on XOM, trying to optimize and improve some of the Unicode normalization code. A lot of this is autogenerated from the Unicode data files, and I’m actually working on the meta-code that parses those files and then generates the actual shipping code. In this code, I’m setting up a switch statement like this one:

       switch(i) {
          case 0:
            return result + "NOT_REORDERED";
          case 1:
            return result + "OVERLAY";
          case 7:
            return result + "NUKTA";
          case 8:
            return result + "KANA_VOICING";
          case 9:
            return result + "VIRAMA";
          case 202:
            return result + "ATTACHED_BELOW";
          case 216:
            return result + "ATTACHED_ABOVE_RIGHT";
          case 218:
            return result + "BELOW_LEFT";
          case 220:
            return result + "BELOW";
          case 222:
            return result + "BELOW_RIGHT";
          case 224:
            return result + "LEFT";
          case 226:
            return result + "RIGHT";
          case 228:
            return result + "ABOVE_LEFT";
          case 230:
            return result + "ABOVE";
          case 232:
            return result + "ABOVE_RIGHT";
          case 233:
            return result + "DOUBLE_BELOW";
          case 234:
            return result + "DOUBLE_ABOVE";
          case 240:
            return result + "IOTA_SUBSCRIPT";
          default:
            return result + "NOT_REORDERED";
        }

And then I stop myself. Do you see the bug? Actually it’s a meta bug that leads to the true bug.

The problem I’ve noticed is that I really have no justification for supplying a default value. If I parse the Unicode data files, and the Unicode consortium has added a new combining class, this code should fail instead of merely assigning that new class to an existing class. So I change the default block like so to throw an exception instead; not that this will ever happen of course.

          default:
            throw new RuntimeException("New case! " + i);
        }

You can probably guess what happens next: I run the code and the very first time, it throw the exception. It seems I forgot to handle the case of 560. OK. Simple enough to fix. I check the data files to see what case 560 means. Strange, it doesn’t seem to be there. Where’s 560 coming from? And that’s when I realize the problem is much, much worse than I thought.

The real bug is that I’ve been treating hexadecimal constants from the Unicode data as decimal constants in my Java code without conversion. Almost everything the code is doing is wrong. It should look like this:

       switch(i) {
          case 0x0:
            return result + "NOT_REORDERED";
          case 0x1:
            return result + "OVERLAY";
          case 0x7:
            return result + "NUKTA";
          case 0x8:
            return result + "KANA_VOICING";
          case 0x9:
            return result + "VIRAMA";
          case 0x202:
            return result + "ATTACHED_BELOW";
          case 0x216:
            return result + "ATTACHED_ABOVE_RIGHT";
          case 0x218:
            return result + "BELOW_LEFT";
          case 0x220:
            return result + "BELOW";
          case 0x222:
            return result + "BELOW_RIGHT";
          case 0x224:
            return result + "LEFT";
          case 0x226:
            return result + "RIGHT";
          case 0x228:
            return result + "ABOVE_LEFT";
          case 0x230:
            return result + "ABOVE";
          case 0x232:
            return result + "ABOVE_RIGHT";
          case 0x233:
            return result + "DOUBLE_BELOW";
          case 0x234:
            return result + "DOUBLE_ABOVE";
          case 0x240:
            return result + "IOTA_SUBSCRIPT";
          default:
            throw new RuntimeException("New case! " + Integer.toHexString(i));
        }

Now the code runs and no classes are missing. But then I get a sinking feeling in the pit of my stomach. What about the other XOM file where I’m using these values? Are those hexadecimal or decimal? Damn it, they’re decimal!

OK. Don’t panic yet. They’re private named constants. The code has been shipping in production for 3+ years now, and no one’s reported a bug here. Maybe it doesn’t matter? It looks like most of the uses of these constants just involve equality comparisons of named constants, so the bug tends to cancel itself out. There are a couple of places where I compare a class to zero, but 0×0 == 0 so that’s safe. And there’s one line that gives me the willies where I compare to classes for relative size, but again if 0×230 > 0×154, then it’s also true that 230 > 154. Whew, that was a close one. I probably would have noticed this years ago except that by freak accident I was working with over 70 different hexadecimal constants, none of which used the digits A-F.

Wait a minute. Is that really plausible? Either someone deliberately set up those constants that way, or the bug isn’t what I thought it was. Maybe they aren’t really hexadecimal constants after all? But then where was that 560 coming from when the data files have 230? Wouldn’t you know it? I was parsing two values out of a line, treating both as hexadecimal, when one is hexadecimal and one isn’t. Does it really make sense to include both hexadecimal and decimal constants in the same line? But there’s not a lot I can do about that, so I change Integer.parseInt(m.group(2), 16) to Integer.parseInt(m.group(2)) and run the program again. At least this means the existing code is safe with its decimal constants.

The problem turns out not to have been as bad as I thought it was. However, it was still a real problem, and one I would not have noticed or found had I simply filled in a default value. The moral of the story is, don’t try to account for code errors you don’t expect. If something “can’t happen”, make sure you throw a RuntimeException when it does. Don’t just silently ignore it. Your intuition about what a program will do is vastly inferior to the experimental observation of what it does do.

Some programmers hate adding uncaught, unhandled runtime exceptions that may surface at unexpected times. (This one, at least, surfaced immediately.) The usual claim is that it’s unnecessary because the problem “can’t happen”. However, if it’s really true that the problem can’t happen, then it won’t happen, and the exception won’t harm anyone. And if they’re wrong and the problem does happen anyway, well then, at least the debugging will be a lot easier. Silent failure is still failure, and is much harder to diagnose and cure than screaming program crash due to an unexpected runtime exception.

I did have one advantage in this case. My program was a one-off piece of throw-away code run only by me. It has to give me the answer once, and then I don’t have to bother with it again. The only user is me, and I know what to do if it breaks. Code that has clients other than the programmer who wrote it, and especially non-programmer end users, may need to be more careful than this. It’s often wise to put a generic catch (RuntimeException ex) block or even a catch (Throwable ex) block in your main() method to more carefully handle unexpected exceptions. Such a block can log and report the problem, while still shielding the end user from the issue.

However an exception caused by a case that “can’t happen” but does is no different in this respect than a library bug or a null pointer exception from your own code. And it is still preferable to have the application fail as early, as fast, and as close to the actual bug as possible rather than silently ignore the failure until some other block of code fails as a result further down the line. Worst of all, the code could well run but generate incorrect data that might even be passed through multiple systems in multiple continents in multiple time zones and organizations before someone notices. Try debugging that some time.

Prefer Multiline if

2009-07-05T21:33:07Z

C-family languages including Java, C#, and C++ do not require braces around single line blocks. For example, this is a legal loop:

for (int i=0; i < args.length; i++) process(args[i]);

So’s this:

for (int i=0; i < args.length; i++)
    process(args[i]);

However both of these are very bad form, and lead to buggy code. All blocks in C-like languages should be explicitly delimited by braces across multiple lines in all cases. Here’s why:

The most dangerous form is a multiline block that doesn’t use braces. The problem is that when you start with this:

if (p.needsGiftWrapping())
    wrap(p);

sooner or later some programmer–perhaps you, perhaps someone else–is going to discover a need to add a second line. For example:

if (p.needsGiftWrapping())
    wrap(p);
   ribbon(p);

Bang! That code is now buggy. It looks like the if block applies to both statements, but in fact it only applies to one. The indentation is lying about the intent of the code. By contrast, this form is less dangerous:

if (p.needsGiftWrapping()) wrap(p);

In this example, it’s obvious to any programmer who comes along and adds a line, that they need to add braces too. It’s less likely to cause bugs down the road, but it should still be avoided, and here’s why.

The statement

if (p.needsGiftWrapping()) wrap(p);

is really two statements: one that calls needsGiftWrapping() and one that calls wrap(p). These are independent statements, and may need to be treated separately. In particular, you may want to mark a breakpoint on one and not the other. For instance, I sometimes like to put a breakpoint in the body of an if or for or while, just to make sure that this code is really being executed when I think it is. If the program doesn’t stop, then the code isn’t being executed; and I have a big clue where the bug is.

This is also important for code coverage tools. Most tools such as Cobertura measure the coverage of lines of code, not statements. Even if they measure lines of code coverage and statement coverage separately (and some tools do this) they still display the coverage with lines of code. This statement can be marked covered even if the body of the block is never entered:

if (p.needsGiftWrapping()) wrap(p);

However, if the block is rewritten like this:

if (p.needsGiftWrapping()) {
    wrap(p);
}

it now becomes obvious if the tests are never testing the case where p.needsGiftWrapping() returns true.

This applies equally to all block statements: if, for, while, do while, and any others you may encounter. I’m beginning to believe this is actually a special case of a general principle for C-like languages, and perhaps others:

Each line of source should contain exactly one statement

That is, avoid lines like this:

int i = 7, j = 18;

int i = j = 18;

Similarly, avoid the ?: operator.

if (a > b) {
  max = a;
}
else {
  max = b;
}

is easier to read than

max = a < b ? a : b;

For instance, did you even notice the bug in the above line? If the verbosity bothers you, try something like this instead:

max = Math.max(a, b);

The problem may reflect a misdesign in C-family languages. The compiler only pays attention to the semicolons and braces while ignoring the line breaks and indentation, but humans usually only pay attention to the line breaks and indentation while ignoring the semicolons and braces. This gives the code the opportunity to lie about what it’s really doing. Consequently we need to take extra care when writing in C, Java, C++, C#, etc. not to lie to ourselves. If you place exactly one statement on each source line, you can be reasonably confident the code isn’t lying to you, and you’ll have a much easier time debugging.

Compact code is fun, but it’s not maximally readable, and more lines don’t really cost you anything after the compiler is finished anyway. Save yourself the hassle and for 2009 resolve to put one statement on each line.

P.S.

If you want to comment on this article, remember that the <pre> tag is allowed in comments. Otherwise your point may get lost along with your indentation.

The Cafes

Would You Entrust Your Data to These Yokels?

Technical Details

SourceForge for the 21st Century

Continuous Integration

submit queue

Code Reviews

One-button deployment

Summary

Why Don’t Macs Support Multiple Monitors?

Further Reading

A Square Is Not a Rectangle

Why Pair Programming Works

Continuous Code Review

Fewer blockages

Mask out distractions

Guaranteed Focus

Multiple points of view

Reduced training cost and time

Just Do It

In Praise of Draconian Error Handling, Part 2

Imagine There’s No Null

How To Shutdown a Computer

In Praise of Draconian Error Handling, Part 1

Prefer Multiline if

Each line of source should contain exactly one statement

P.S.