What should go into a source code repository?

The short answer: the repository should contain everything necessary to perform a clean build of your system. In most cases, this includes the code, third-party binaries necessary for building, tests, and documentation.

It’s ok to assume that everyone has their build environment “properly configured” for building. To make sure, make a list of everything that must be setup in the environment to build the software and put it on the team wiki. These things don’t need to be stored in the repository but you should at least write down what the standard build environment is supposed to look like. Depending on what the required software is, it might also be a good idea to keep a copy of it, just in case something happens to vendor in the future. The last thing you want is for a vendor to stop supporting the version of something you need, forcing you to upgrade because your hard drive crashed and you had to setup a new environment.

Include at least the following in your standard build environment list:

• Compiler versions
• Team sanctioned IDEs
• Required frameworks, toolkits, and build tools
• IDE extensions that the team has decided are so critical/awesome to the project they have to be used. Critical/awesome IDE extensions might enable a required tool-kit (such as GWT in Eclipse) or configure the IDE is specific ways (such as coding styles or static analysis settings)

Putting code and tests in the repository is fairly obvious, but third-party binaries (e.g. libraries) might not be. Put these in version control so that it’s easy to check out a project from source control and build without monkeying around with anything. I’ve found it best to create an “ext_lib” folder for storing all the external libraries. This way there is no confusion over what versions to use, and all the build paths can be set so that anyone can build just by checking out the code.

Here’s a real life example. Let’s say you’re writing a web application using the Google Web Toolkit and you rely on a caching library. The caching library should go into your ext_lib folder and you should tuck a zip of the GWT version you use away in a safe place just in case you need it later. Say your team is also using JUnit. Put the version you use in the ext_lib folder. This way everyone can build and use whatever GUI they want to run tests, be it the JUnit GUI or an Eclipse Plug-in.

Another real life example. Let’s say you use the excellent Sharp AutoUpdated component. Should you version the binary or the source? That was a trick question since it depends. The best answer is to only keep the binary of the library, but this isn’t always possible. One of the awesome things about open source software is that you have access to the source if you need it. So, let’s say you find a bug in the AutoUpdater and for some reason the maintainers aren’t responding quickly enough for your immediate needs. You can’t live with this bug so you have no choice but to fix it yourself. Congratulations, you just took ownership over your own fork of the AutoUpdater component. You now are responsible for maintaining the code – either in your version control library or in a public fork, and merging with the original code base may be more difficult in the future.

What doesn’t go into the source code repository?

Remember the DRY Principle for writing code (Don’t Repeat Yourself)? Well, that applies to your version control system too. Anything that can be derived shouldn’t be held under version control. Since your source code is already in the repository, storing the built binary is a violation of the DRY Principle. The penalty? Confusion, mistakes, and avoidable headaches. Third party libraries in the ext_lib folder don’t violate DRY since you can’t build them – you don’t own the source.

Also, do your fellow developers a favor and keep your personal stuff out of the repository. If you’re testing, nobody else wants to see your test reports. When you run the application, keep your logging messages to yourself. Also keep anything related to how you set up your personal environment in your personal environment. The last thing I want is to open up my IDE and see the last tabs you had open because you committed your personal user settings.

The easiest way to keep these sorts of undesirables out of the repository is by setting up an ignore list. Share it among the team.

How often do I commit?

Generally you should commit your changes anytime you think you’ve finished something useful that doesn’t introduce problems into the system. On the average, you should be committing changes at least once a day.

There’s two parts to this commit rule. “Finished something useful” might mean many things. This is by design. When you’ve finished a logical chunk of code that does something, feel free to commit it. “Doesn’t introduce problems” is a common courtesy to your fellow developers. Make sure, at a minimum, the system builds and passes any automated tests you have. And always update before you commit. Depending on your team size and how important the code is, you might establish a checklist for committing. Google has theirs automated. Every change the system has to build, pass tests, and pass a peer review before it can be committed.

Remember this mantra: Commit early, commit often.

But if everyone is committing all the time, isn’t that going to cause problems?

When you’re working with people and coordinating effort, problems will inevitably arise. Just remember, if you’re going to fail, fail early. It’s better to cause a conflict today through miscommunication while there’s plenty of time to fix it than the day before it’s time to deploy. Why? The conflicts will be smaller since you’re incrementally growing your code base. Also, since you made the changes recently they are fresh in your head and easier to work with. Code more than a week old might as well have been written by someone else.

Taking a risk management approach makes mitigating this easy. The risk: “Developers use a shared repository and commit changes frequently; might cause code conflicts that break the build.” The source of this risk is communication; therefore anything which helps facilitate communication can reduce the likelihood of this risk becoming a problem. Daily stand-up meetings are perfect for getting the word out about what everyone is working on. Automatically generated email updates from the version control system keep folks abreast throughout the day as changes are made. Continuous integration acts as a smoke test for uncovering integration problems while they’re small. Good merge tools can help reduce the impact of the consequence.

Once everyone gets used to the update-then-commit cycle, most of these problems go away. In my experience, big problems with code in the repository are usually a symptom of larger problems such as poor communication or failing processes.

What are some of your version control philosophies? What helps you keep things organized so you can get things done?

The first step when performing a SWOT analysis is to define the business objectives. This is very similar to defining a threshold of success in software risk management. The main difference is a business objective takes the form of the desired end state whereas the threshold of success is the minimum objectives necessary for the project to be successful. For example, a perfectly valid business objective might be to deliver all 100 story points by the end of the year while the threshold of success might be to deliver the core functionality (worth only about 50 story points). Would more stories completed be better? Of course, but what if you end up only completing 75 story points by the end of the year? How did you do? You missed your goal, but you still succeeded right? It’s difficult to tell without understanding the difference between wants and needs.

The main part of a SWOT analysis consists of a group session where strengths and weaknesses internal to the group and opportunities and threats external the group are identified. People like to put SWOTs into a 4×4 grid so it’s easier to look at. While there is some great advice out there for understanding what goes into a SWOT, the analysis is largely subjective, relying on a teams’ gut feelings to know the strengths from the weaknesses, the opportunities from the threats. Software risk management can be a much more systematic approach to understanding the potential dangers that face a project based on known facts when tools such as the SEI’s Taxonomy Based Questionnaire for risks (pdf) are used. Guts still come into play, but there is enough engineering in place to help people make the right decisions.

Risks are specifically actionable – depending on the risk you might be able to mitigate it by manipulating the timeline, impact of the consequence, probability of the risk occurring, or by addressing the condition. You might transfer the risk to someone else or simply accept the risk. SWOT by itself is merely a collection of statements relative to internal or external entities which may or may not actually be true. Are you good at testing? How do you know that? Is Bing really a threat to Google Search? Should you do anything about your weaknesses? Will they prevent you from achieving your business objectives? Without further analysis there really is no way to know and other than prioritizing there really is no way to analyze a SWOT, nor is there any clear direction for next steps.

Look, when planning a project you really need both SWOT analysis and risk management. SWOT is a tool for assessing capabilities while risk management is a tool for assessing the likelihood of success. Each technique serves a very different purpose. SWOT is most useful at the beginning of a project to help you figure out what you’re doing and come up with an overall strategy. Risk management, though is an ongoing activity that makes sure you don’t fall flat on your face in trying to achieve your business objectives.

My wife takes a slightly different tact. After the meal is finished she immediately cleans all dishes, pots, pans, and utensils used while she was cooking. She has the forethought to run the dishwasher beforehand so there is plenty of room to load dirty dishes after the meal. She even wipes down the counter and stove so everything is ready for the next meal we cook. She’s quite amazing actually and a good cook to boot (especially when she’s following a recipe).

Professional chefs take matters a step further still. They clean as they cook.

I procrastinate doing something I hate and the result is a monumental, exhausting chore which takes an hour or more to finish. My wife spends 10, 15 minutes tops a night “tidying up” and though she hates doing dishes just as much as I, she makes it seem effortless. Professional chefs make miracles in the kitchen minutes at a time.

So it is with software.

Integrating software, even with a small team can be a chore.  Which would you prefer?  Approach A: write a lot of code, get everything working individually, and then do a big bang integration at the end; or Approach B: write a little code and integrate a little. While putting off integrating might satisfy your immediate needs, much like skipping dishes and moving straight to dessert, Approach A is going to cost more than Approach B in the end. Why?

• Integration problems aren’t uncovered until you integrate (profound, I know) so the longer you wait to integrate, the longer it takes to find out if there is a problem. Of course, no one ever plans for problems…
• Conflicts have further reaching consequences the longer you wait to fix them. Modern version control systems usually do a pretty good job merging changes but even magic has its limits.
• The full power of refactoring can’t be realized because the turnaround time on changes is too long.  The side effect is that you don’t refactor which means the code becomes more brittle over time.
• More code changes means more time to bring it all together and a higher likelihood of introducing defects through integration.  Unless you’ve planned knowing that integration will take time, chances are good you’re going to ship late.

Even better than big bang and nightly builds: continuous integration, cleaning as you code. Automated build servers have made huge advancements over the past few years. I highly recommend Hudson. It’s super easy to install and get started and has plug-ins for practically everything. There are even Hudson plug-ins for C# and just about every version control system you could want (and even some you don’t).

Doing chores sucks, but don’t make it worse than it has to be by putting things off. Continuous integration is a no brainer. If I had a dishwashing machine that constantly washed dishes as I finished using them I would dance naked in the streets, celebrating the marvels of modern technology. (You should be thankful such a machine doesn’t exist.)

You really don’t have an excuse for not using continuous integration.

Kid’s stuff right?

A lot of software projects with teams made up of working adults still play this game. The scenario goes something like this. A team is put together to build some software. Neither the clients nor the team talk about the objectives of the project other than building “some software”. After a few months, something goes wrong or someone doesn’t like what’s happening so someone changes the rules. Before too long, one side or the other is upset that they can’t win, somebody throws a fit, and goes home. Instead of summoning invisible armor, software projects change the rules by cutting features, adding more requirements, moving due dates, wasting resources, and things like that.

We make believe that we’re software engineers.

While Make Believe was a fun game as a kid, changing the rules when there’s real money on the line isn’t as fun. My brother and I ran into problems as kids because we got the objectives of the game wrong. Actually, there were no common objects, which is why we could change the rules so easily. The same thing happens on a software project when the objectives aren’t well known.

Defining and committing to a clear picture of success establishes the common ground rules for a project by making the basic project goals explicit. The technique is known as Threshold of Success.

Defining What Success Looks Like

The Threshold of Success for a project is the minimum set of conditions that must be met for the project to be considered successful. If the team fails to meet even one of the conditions then the project is a failure. A good Threshold of Success is made up of about 3-4 SMART goals (no more than a few bullets on a single PowerPoint slide). SMART is a mnemonic which stands for Short/Specific, Measurable, Achievable, Relevant, and Time bound.

Some other pointers for defining a Threshold of Success:

• The Threshold of Success should be built as a team. Since this is the measure by which you will define success or failure, everyone on the team must buy into it. If you can include your client that’s even better.
• Threshold of Success goals should be challenging, but it’s important that they are achievable. If the goals are too easy, victory will be meaningless, too difficult, elusive.
• Once the Threshold is established, don’t change it! The only reason to modify the Threshold of Success is if the project has changed so drastically that the Threshold no longer makes sense (for example if someone leaves the project).
• Revisit the Threshold of Success regularly (a good time is when planning iterations) so everyone remembers what success looks like. Put it on your team wiki so that it’s readily accessible.
• Be sure that the goals in your Threshold are SMART! The point of defining a Threshold of Success is to take away the wiggle room for defining what it means to succeed or fail. The goals you define should make this black and white. The more specific the goal is the better.

Building a Threshold of Success

The easiest way to create a Threshold of Success is to first create a minimum picture of failure, then convert failure into success. Here’s an example:

Failure for my current project might look something like this.

• Essential features are not ready by the end of the second quarter.
• Team members are dissatisfied or bored with their jobs.
• Newly hired team members don’t feel like they’re part of the team by March 31.
• There isn’t enough money to continue development after this fiscal year and we have to fire people.

Now that I know what failure looks like, seeing success is easy. I don’t want any of these things to happen. The threshold of success for my current project might look something like this.

• By the end of the second quarter, all “Must Have” features are implemented and pass acceptance tests with no known critical defects.
• All team members give average score of 5 or better on a job satisfaction survey taken quarterly.
• By March 31, the team has successfully executed at least three team building activities with all team members present.
• Funds of at least $1 million are secured by December 31 to allow for future development without a reduction in team size.

Notice that only 1/4 of the success goals in this example are related to software functionality. While goals might come from anywhere, teams traditionally focus on goals related to people and relationships, process, resources (such as budget or schedule), and product (software functionality and quality).

As this technique originated with the Software Engineering Institute (pdf), nearly every studio team in the Carnegie Mellon Master of Software Engineering program creates a Threshold of Success for their projects. The MSE Studio Archive has extensive examples of both good and bad pictures of success that teams have created. The Square Root Team’s threshold (my team) is a good place to start, but there are plenty of other examples.

There might be many goals for a project. In the Team Software Process you actually identify at least three different kinds! But there is only one threshold of success for a project. Knowing what success looks like gives you a better chance of actually achieving it.  Without it, you’re just pretending that you know what’s going on.

After the power went out, the heat did not fire up. The tree went dark. So did the TV, DVD player, and Wii.

Using a pair of LED headlamps my wife and I received for Christmas the 20 of us took turns rolling and stuffing homemade ravioli dough for dinner. Christmas ravioli making is extremely serious business and I had finally been promoted to “unmonitored ravioli stuffer” this year. Luckily we had manual pasta rollers to flatten the dough. There’s always talk of “upgrading” from the hand cranked system but this year, tradition trumped technology.

Once ravioli are stuffed, they have to dry for a few hours before cooking. In the years past this was the time to play with new games or watch a new movie. Of course, without electricity most of our new toys were rendered useless.

Since the heat was off and the vacation house was gigantic, the fireplace in the living room was our only option for warmth. Lighting the room was the handful of candles we had, originally intended for dining ambiance, and the low glow from the fire.

To pass the time, we sang Christmas carols in the near dark. It was all sort of surreal, a setting I am certain we would not have created on our own had we not lost electricity, had we not been so completely pushed out of our comfort zone. Even more fantastic was that after over 20 years of putting it on my wish list, someone actually gave me a pair of night vision goggles for Christmas. Surprisingly, this “toy” is the real deal. I was able to wander around a completely pitch dark house with no problems. The night vision, along with the head lamps (which single handedly saved dinner), were by far the best gifts of the day. The electricity turned on around 1:00am that morning and by the next morning, everything went back to the way it was.

Sitting in the dark on Christmas night got me thinking. This was an experience that I never would have chosen, a situation I would never have intentionally embraced, but it turned out to be pretty fun. How many opportunities have I missed because my default attitude was to stay inside my comfort zone? While happy accidents are great, can’t I do more to create opportunities rather than relying on happenstance? Given the time of year it only seems appropriate to ask these sorts of questions.

I’m a technologist, a scientist, and an engineer at heart. I love playing with gadgets, tinkering with software, and working on interesting and challenging problems. Sitting in the dark with no power, my normal pursuits removed, it was easy to remember that people are important too. Sure, when building software everyone always talks about how people are important, but when building software we call people “users” essentially reducing their humanity. After all, there are only two industries where the customers are called “users” – software is one of them.  Is it really appropriate to use the same term to describe software clients and drug addicts?

My lesson from the night: the technology and gadgets and programming and processes and everything else are awesome, but they are meaningless if they fail to create a genuine relationship with people.  I do believe that relationships matter and that people are important. I’d like to do a better job of thinking about people first this year. In the rush of excitement surrounding every new technological achievement, it’s sometimes easy to forget that helping people is why I build software.  It’s too bad it took a harsh winter snow storm to remind me of this.

For almost every project in the world, I think formal methods should be generally avoided. Given the option of spending money and time on mathematicians or extremely smart coders I would chose the latter. With smart coders, code inspection is a fun and effective defect filtering option. And let’s face it. Why would you have your amazing coders do something other than write amazing code?

But what if formal methods didn’t have to be so…formal?

While traditional formal methods such as Z might be a little difficult to pick up if your predicate logic is rusty, applying formalisms that are close to the code doesn’t seem that outrageous in perspective. Design-by-contract language extensions such as Spec# for C# or Plural for Java have been stealthily making their way into IDEs for years now without anyone really knowing the wiser, without anyone really thinking about these tools as being formal methods. State machines in UML and even some uses of domain specific languages might be considered formal methods when these tools are used for specification and analysis purposes.

Creating formal methods that are close to the code is one of the best ideas to come out of the formal methods arena in the past 20 years. Just like how adding cherry flavor makes it easier for kids to take icky tasting medicine, putting formal methods in the code makes it more likely that I, as a programmer, will actually use them. Because no matter how good for me something is, if it’s difficult to use or generally unpleasant, it won’t get used.

This is why domesticating formal methods (pdf) is extremely important for formal method adoption.

Wild formal methods can be difficult to work with. No matter what anyone tells you it does take time to get the hang of Z. Formal methods such as Z have limited use when talking with customers and, though a formal specification is an excellent tool for gaining better understanding of functional requirements, the payoff for creating the specification isn’t seen for weeks or months making it difficult to justify their use in Agile development environments.

But is turning what was once a ferocious and unpredictable wolf into an obedient and trusted canine companion safe? In selecting the features for our domesticated formal methods, did we accidentally breed out the most important benefits?

Anthony Hall outlined seven myths of formal methods (pdf), ideas about formal methods erroneously taken as truth.

Myth 1: Formal methods can guarantee that software is prefect
Myth 2: Formal methods are all about program proving.
Myth 3: Formal methods are only useful for safety-critical systems.
Myth 4: Formal methods require highly trained mathematicians.
Myth 5: Formal methods increase the cost of development.
Myth 6: Formal methods are unacceptable to users.
Myth 7: Format methods are not used on real, large-scales software.

Any formal method worth using should not uphold any of Hall’s seven myths, but domesticated formal methods have the additional burden of usability. Just like we expect certain behavior from our four-legged friends, I don’t think it’s too much to ask that a domesticated formal method be friendly and obedient. Any method which bites the master’s hand obviously can’t be trusted. Crashing, making it easy to make mistakes, poor affordances, difficult to read output, impossible to maintain specifications – these are all reasons for not trusting a domesticated formal method.

I think informal formal methods are great. When done well, it doesn’t even feel like I’m using a formal method and I get many of the same benefits a wild formal method would give me – clarity, understanding, and maybe even a little automated verification depending on the language and method. Domain specific languages and state charts are great for working with end users and clients. It’s even plausible to skip other testing or verification measures such as unit testing or inspection.

And the best part is that since many domesticated formal methods are close to the code, I’ve already got the necessary training on how to use the methods because I already know how to program.  Domesticated formal methods are a win-win for programmers who want to do more engineering and another practical tool for your silver toolbox.

Kathy Sierra explains it best:

Any time a new idea is brought to the table, especially a Big Idea, the group acts like white blood cells, attacking the Big Idea as if it were a foreign invader, reducing the idea to a benign and much less exciting version of itself. The end result is something that no one can really get all that excited about. It may get the job done but it certainly doesn’t inspire anyone.

It’s really difficult to fight off this group tendency. It’s much, much easier to say no to something than yes, especially when there’s an established status quo. Once a group (company, club, team, non-profit, whatever) feels safe, group risk aversion magnifies problems into insurmountable barriers. Kathy Sierra (boy, do I miss her blog!) talked about this too, avoiding risks leads to the safest route, but the safest route, ultimately will only yield incremental improvements:

In most cases, the barriers preventing new Big Ideas from being achieved are no bigger than the barriers overcome by the group when it formed. The main difference is that there’re more people to get over the wall now than when it was just you and your Big Idea which brought the group together.

As a group it’s important to fight risk aversion which is often reinforced by groupthink. It’s a difficult job, but if growth and innovation are goals, taking risks is an essential part of success. Figuring out how to work with Big Ideas rather than fight against them (and in turn deal with risk as an organization) makes it easier to jump the wall as a group when and if the time comes. Delegating is a great way to build practice taking risks If you’re able to trust someone else to do something that you could reasonably do yourself you’re on your way to letting others run with their Big Ideas.

As an individual with a Big Idea, it’s important to quickly figure out whether your idea is something that has legs or whether it’s something better left to the back burner. Seth Godin’s The Dip has some simple advice but ultimately it’s up to you to determine. If you yourself are not prepared to pull each and every group member of the group over the wall that is blocking you from realizing your ultimate vision then it may not be time to unveil your idea to the group.

Both the group and the innovator have a part in killing a Big Idea or making it fly. Recognizing that you have to work together is the first step toward achieving greater success.

The Megalomaniac Architect: “You will implement the system as I have designed it because I am the most important person on this team, the project will fail without me, and I am the only one smart enough to know everything that needs to be known!”

The Benevolent Dictator: “Implement the system precisely as I have designed it and I will make your work easier.” [via Dennis Mancl]

The “Everything’s a Nail!” Architect: All problems can be solved by a single tool, idea, or way of thinking. He’s got a hammer…and everything he sees is a nail.

The Diva: “I’m the architect of this system! Are you questioning my ability to design? If you’re not going to appreciate my talents, maybe I’ll just go to another team.”

The Magician: Excellent at high level abstraction, vague on how the design actually achieves architectural drivers.

The Over-Accommodating Architect: “You want that change? No problem.” There are no trade-offs to consider when making architectural decisions.

The Chess Master: To achieve victory, it is simply a matter of being able to see enough moves ahead. Every element in the architecture can be strategically placed in a perfect position to checkmate your opponent: requirements changes.

The PowerPoint Architect: “Architecture is just pretty pictures!”

The Ninja: Infiltrates a project from the outside, crafts an amazing design, then disappears into the shadows before anyone realizes he’s gone. Ninjas do not suffer fools and their designs are technically correct, thorough, and beyond your ability to comprehend.

The Navigator: Creates a map (with legend) and uses it to plot a course through implementation, testing, and deployment.

The Movie Producer: Leads the team indirectly by providing technical design support. [via George Fairbanks]

The Coach: Teaches the team about architectural practices and concepts as well as the design.

The Puppeteer: Able to manipulate multiple levels of abstraction simultaneously to design the architecture, the Puppeteer effortlessly manages the various threads influencing the design (commonly known as architectural drivers).

The Seasoned Veteran: He’s tried a lot of things and is pretty good at all of them including programming, design, processes, and management. Thanks to his experience, having walked 10,000 miles, he understands the role of architecture across the lifecycle, has seen many different situations, and is an excellent technical practitioner. [via Bill Opdyke]

The Long-Bearded Wise Man: A little philosophical but always willing to share an enlightened thought that will help resolve whatever concerns are ailing the project, though somewhat indirectly.

The Team Captain: A bona fide member of the team, playing on the field with everyone else, leading the team from the field.

The Design Evangelist: Excited about the architectural drivers and architecture to the point of near fanaticism. His enthusiasm for architecture and the system’s design is infectious and helps maintain conceptual integrity of the system.

The Student: Takes time to learn about the problem domain and the customer’s needs to the greatest extent possible so that he understands and solves the right problems. [via Gail Harris]

What other archetypes have you worked with? Which archetypes do you think might combine to make the perfect software architect?

The biggest assumption in XP (and indeed most software processes) is about bug tracking. Common sense dictates that you will create some kind of bug database. Hopefully it will at least be some kind of third party bug tracker such as Bugzilla or Bug Genie. Excel will work in a pinch but quickly becomes unsuitable for teams larger than one developer.  But how does the bug tracking actually work? What bugs get reported? Will you record issues from inspections in your bug tracker or only “true” bugs? What is the process for fixing a bug? What is the process for closing a bug? Who has access to the bug tracker? What information is required in your bug database and what information is optional? How do you determine defect priorities? Or the severity of bugs?

The majority of software processes provide answers for almost none of these questions. You are largely on your own to make up whatever you think makes the most sense for your development environment based on the best practices for your software process and your understanding of “good” software quality assurance practices.

No matter what quality process you follow, you will need a defect control philosophy. Once again, in absence of guidance I turn to the Team Software Process, one of the few processes to define what it means to track defects. In the Team Software Process, defects are treated as blight, a horrific mistake injected through the ineptitude of a developer. To remove these blights, the TSP relies on a series of filters in the form of code reviews, code inspections, unit tests, function tests, and so on. XP has a similar, though somewhat less rigorous set of filters in the form of pair programming, TDD, continuous integration, and acceptance tests. Each filter is meant to remove more and more defects, until finally “all” defects are removed from the system by the time the software has passed through all the filters. Generally each filter is intended to remove different types of defects, though it is conceivable to capture escaped defects from a previous filter in a later filter.

Just as water passing through layers of sand and rock will remove debris, so too will code passing through layers of unit tests and inspections sift out injected defects.

Defect Data

With these ideas in mind, bug tracking has three basic goals.

1. Record defects so they can be analyzed and fixed.
2. Identify the means by which defects are injected.
3. Identify the means by which defects are removed.

We write down bugs so we can go back and fix them later. This is obvious. But bug data can be used to measure process improvement also.

Since many parts of a software process are dedicated to filtering out the defects we’ve injected, understanding how defects are injected is essential to preventing similar defects from being seen in the future. The idea is that we want to learn from our mistakes. To achieve this, record the type of defect and the reason it was injected. The TSP gives us a good starting point for each of these, shown in the tables below. You should feel free to modify the types and reasons so they make sense for you, your team, and your project.

The defect type characterizes what kind of defect is injected and captures the essence of what is needed to fix the defect.

The defect’s reason characterizes why the defect was inject.

Assigning the reason can be a little tricky. I have found it is best if the person who injected the bug assigns the reason. On small teams (or if you’re following the PSP), this is generally pretty easy. It’s not about rubbing their nose in the problem – well, actually it is. Ok, it’s not about embarrassing or punishing the person but creating an opportunity for learning from our mistakes. If you can understand the reason why a defect was injected, it’s possible to prevent the defect from occurring again. For example, if there seems to be a rash of education related defects in a particular module, perhaps some training is in order.

Ideally we’d also like to know when the defect was injected, as in at what phase of development. It is possible to realize this information with additional analysis but I’ve found the return to be rather diminutive. Basically, you’ll learn what we’ve known all along – that the longer a defect is in the system, the harder it is to get out and that the most expensive defects are injected during the earlier phases of development (e.g. design defects are costly). Rather than track when things were injected I think it makes more sense to track when defects are detected. The point is to gain an understanding of how well the quality process filters out defects. To accomplish this, simply write down what you were doing when you found the bug. If you’re using XP, the list might include designing, writing new code (in a pair), writing new code (alone), refactoring, unit testing, integration, and acceptance testing. With this information you should be able to determine how effective each activity filters defects and over time whether the quality process is having issues. For example, I would expect interface defects to be detected during integration. If they are being detected earlier, say during unit testing, or later, say during acceptance testing, then my continuous integration and smoke test suite might not be as robust as it should be.

Better Bug Tracking

The strategies I’ve outlined here are a little more sophisticated than your average bug tracker, but add a lot of punch for very little effort. Tracking defect type, reason injected, and phase detected allow you to get a better handle not only on how defects are being injected into the software, but also how they are being detected. Both these chunks of information are necessary for understanding how defects are making their way into the system and how your process is helping you ferret them out of your system.

The contract rider read like a version of the Chinese Yellow Pages because there was so much equipment, and so many human beings to make it function. So just as a little test, in the technical aspect of the rider, it would say “Article 148: There will be fifteen amperage voltage sockets at twenty-foot spaces, evenly, providing nineteen amperes . . .” This kind of thing. And article number 126, in the middle of nowhere, was: “There will be no brown M&M’s in the backstage area, upon pain of forfeiture of the show, with full compensation.”

So, when I would walk backstage, if I saw a brown M&M in that bowl . . . well, line-check the entire production. Guaranteed you’re going to arrive at a technical error. They didn’t read the contract. Guaranteed you’d run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.

In economics, signals are indicators that convey specific meaning between producers and consumers. For example, when you see THX on the side of a set of speakers, you know the speakers are going to probably be of audiophile quality. The THX logo is the speaker manufacturer’s signal to you, the consumer, that these speakers are really good. To David Lee Roth and the Van Halen road crew, the presence of brown M&Ms indicated that the hosting venue had not understood all details of the contract and had very likely made a mistake in configuring the set. One mistake in this case could cause malfunctions during the show or even the death of a crew member.

As it turns out, signaling software projects isn’t that difficult. The 12 step Joel Test is a reasonable signal for software development companies. While the Joel Test is nice for getting a feel for a company before you work for them, the concept is still useful once you’ve got the job and the project is in full swing.

Ultimately signals, also known as tripwires or triggers, are really just binary metrics for uncovering potential problems your project might be facing before the problems explode in your hands. When some condition is met (the signal), you know it has specific significance and prompts certain actions to prevent a problem from occurring. Triggers are most often used with risk management but their use should not be exclusive to that practice. In fact, if you’re collecting real data, you have even more opportunities for identifying signals outside of risk management.

On past projects I’ve used signals for a variety of issues. Here are some examples.

• During the past 3 iterations the team identified between 15 and 20 defects. I expect a similar number of defects to be detected for this iteration. If more defects are detected, there may be a disconnection in understanding between requirements, design, and implementation. If fewer defects are detected, tests may not have been as rigorously defined as they should have been.
• A Fagan inspection completed in less than one hour with a rate of 400 LOC/hour. Since most inspections have covered only 250 LOC/hour it is likely that this inspection was not effective and the results not reliable since the inspection team sped through the code.
• When evaluating potential open source libraries, Source Forge projects without a website shows a general lack of dedication to the project and indicates that the software is probably of poor quality or ill-maintained; the library is worth neither the time nor effort to use.
• Tasks that have been estimated to require longer than 9 hours have probably not been thoroughly thought through.
• No risks have been identified for this project or risks have not been updated for several iterations. This implies that the team doesn’t have a realistic understanding of what problems the project faces.

In each of these examples, when the signal is heard, I knew there was going to be a problem on the project.

Work with your team to establish signals for your project. The best part is that once you’ve decided on the signals for your team, when triggers are tripped you can throw a Van Halen sized rock star fits in your cubicle! Well, try to resist throwing your monitor out the window anyway.

Why then do most software developers treat software development differently?

Tom DeMarco recently brought his own famous quote into question (pdf), musing that not only is it possible to control what you can’t measure, but the most important stuff you need to control on a software project is impossible to measure. Once again, DeMarco is wrong (in my opinion anyway).

Wikipedia is one of the most controlled projects on the planet

One the surface, Wikipedia is the Wild West of online content. Not only can anyone edit any page, but content from Wikipedia is widely proliferated in the media and (sadly) school reports. Wikipedia is the single greatest success of user generated content in the history of mankind (”The Internet,” as the medium, doesn’t count). What started with a dozen humble articles has evolved into the most comprehensive encyclopedia ever created and includes everything from the fundamentals of science to the definitive source on Babylon 5.

What folks seem to forget is that even in the Wild West, there were laws and there were lawmen. Though we love to think romantically about such brigands and gunslingers as Jesse James, Billy the Kid, and Butch Cassidy, most stories about these historic figures are greatly exaggerated. So too is the case with Wikipedia.

Let’s take a closer look at the Wikipedia entry for Billy the Kid. This article belongs to a number of internal WikiProjects, visible from the top of the article’s talk page. The WikiProject Biography is not unlike most projects in Wikipedia. There defined processes for assessing articles, conducting peer reviews. There are rubrics defined for assessing the quality of articles within the project. People even take on specific roles and responsibilities within the project. The collection of processes and information serves as the main means of coordination for content contributors and helps the group control articles within the scope of the project.

The WikiProject Biography even collects metrics on articles which it then uses to make decisions concerning the articles under the project. The metrics are derived from quantifiable data and help control the project.

As it turns out, Wikipedia is not the lawless territory of the internet it has been made out to be.

You can measure the immeasurable

Wikipedia works because people were able to figure out ways to measure things that usually can’t be measured. The fundamental principle that many people overlook is that binary is a metric too. Yes or no questions can be just as effective a measure as any complex metric. Did everyone fill out their task data today? Yes or no. Did the estimate match the actual? Yes or no. Did the test pass? Yes or no. Is the project done? Yes or no. Have we identified risks? Yes or no. Has this risk become a problem? Yes or no.

At the heart of every complicated metric is really a series of yes or no, binary questions. When considering whether the project is done, you have to define done. One way of defining done is in terms of a checklist. Is feature 1 done? Is feature 2 done? Defining done for a feature could be as simple as checking whether all the tests have passed for the feature, again a binary measure.

For more subjective assessments, you can rely on observation-based, experience-defined rubrics. Does the team get along with one another? In the simplest form, this could be a binary metric (Am I friends with everyone on the team?) but it could also be more complicated relying on gut feelings and a guiding rubric (”we never hang out together and don’t trust one another” might indicate low harmony while “we hang out often and feel comfortable sharing personal stories” could indicate high harmony). Teachers use rubrics and experience to judge subjective assignments everyday. The difference is that they slap a grade on it and send it home as a report card.

While DeMarco is correct that many of most critical things in a project are the most difficult to measure, it is possible to create measurements if you feel it is important enough to do so. How would you assess whether you have a good architecture that solves the problem at hand? Rubrics might play a part but so too might binary gates based on quality attribute scenarios or intricate observations concerning design trends over time. If you think hard enough, you’ll find that it’s extremely easy to find measuring points for nearly every aspect of a software project.

Whatever you do, don’t become a mindless, data-driven robot

I love data and I know you do to. While it’s tempting to inject data collection and derive metrics for every aspect of a project (because it’s fun and informative!) don’t. Collecting data and calculating metrics can be expensive. Not so expensive that you shouldn’t use it, but expensive enough so that you shouldn’t use it on everything. I like to compare using metrics to eating out at restaurants. Once or twice a week isn’t that big a deal, but it’s not something you should do every day if you’re trying to watch your budget.

DeMarco is right about one thing: control is not the end-all-be-all of software engineering. Consider carefully, what are the most risky parts of my project? What are the parts of my project that even require control? What are the parts in which I need more insight or want to improve? Strategically develop metrics for these areas and don’t worry about measuring the rest. Trust me, the world won’t end. If you don’t know what you’re doing, start with a simple binary measure. And above all, if something isn’t working, change it.

With all due respect, Tom DeMarco is wrong.

And Jeff Atwood made things worse.

According to Atwood’s interpretation of DeMarco, since we can’t control software projects, there is no sense in trying to engineer software.

What DeMarco seems to be saying — and, at least, what I am definitely saying — is that control is ultimately illusory on software development projects. If you want to move your project forward, the only reliable way to do that is to cultivate a deep sense of software craftsmanship and professionalism around it.

Atwood’s conclusion simply is not supported by DeMarco’s article. DeMarco made two points in his piece.

1. We don’t have as much control over software as we think we do — even when we can measure the software on which we work.
2. We should be focusing more on the upfront “conception” activities than the areas that currently receive the most attention, construction.

My interpretation of “conception” activities are things like requirements, architecture, and design — details that ultimately help you figure out whether it makes sense to build the thing you think you want to build. By framing DeMarco’s argument as “craftsmanship” vs. “engineering” Atwood misses the whole point and reopens the tired art or engineering debate.

Overlooked by Atwood, DeMarco never questioned the idea that software should be engineered.

I’m gradually coming to the conclusion that software engineering is an idea whose time has come and gone. I still believe it makes excellent sense to engineer software. But that isn’t exactly what software engineering has come to mean. The term encompasses a specific set of disciplines including defined process, inspections and walkthroughs, requirements engineering, traceability matrices, metrics, precise quality control, rigorous planning and tracking, and coding and documentation standards. All these strive for consistency of practice and predictability.

DeMarco is really saying that the engineering part of software engineering has become overshadowed by a collection of best practices for building software. In my mind this isn’t necessarily a bad thing. All it means is that what has become known as “software engineering” is different than the original definition intended by the NATO Conference on Software Engineering.

But by discounting current software engineering practices, DeMarco dismisses the real engineering that went into advancing the field to where it is today.

DeMarco seems to imply that what we really want software engineering to be–the application of systematic, disciplined, quantifiable approaches to the development of software–and what software engineering has become cannot coexist. Essentially, to reach a state where metrics and measures, quantifiable approaches, are used correctly and consistently by the software development community we must stop using the term “engineering” to describe the current set of practices.

This is backwards thinking.

Engineering is more than something you do; it’s also a way of thinking about problems and solutions. Reaching the point in software engineering that we are today required systematic, disciplined, and quantifiable thinking. Over time results of this thinking have been codified into the set of best practices that most developers now take for granted.

For example, we know that there is a 100x or more difference in costs between defects discovered later in the software lifecycle than earlier. We know that certain practices can effectively remove defects at different costs and at different times throughout the lifecycle (for example, inspection vs. prototyping vs. unit testing vs. system testing). We also know that historical data is an excellent indicator of future performance on software projects.

Systematic, disciplined, quantifiable thinking was required to make these discoveries.

Because of these codified best practices, it is not always necessary to conduct experiments on a project to trust that they are working. I know unit testing combined with regular system integrations will flush certain defects from my software before those defects become a problem during system testing. I know that statistical analysis of collected task tracking data will help me better predict how long future tasks of a similar size will take. It doesn’t matter whether I completely understand the engineering behind the practice or whether I simply follow the process or use the tool. The benefits will be the same.

Does that make me less of an engineer? I don’t think so.

Using best practices codified as processes, methods, or tools on a software project means you are engineering software whether you like it or not. With many of these practices, the control mechanisms are already built in so you don’t realize that you’re already controlling your project. As DeMarco points out, it simply isn’t necessary that every engineering detail be painstakingly scrutinized for a project to be successful. For many projects, the essence of the project is sufficient to overcome the accidents encountered when engineering by coincidence.

But just because you engineer by coincidence it doesn’t make you a software craftsman. To prove it, I’m calling Jeff Atwood out. Jeff, I dare you and the Stack Overflow team to take the PSP Challenge. Take a course on the Personal Software Process, honestly give it a try — use actual software engineering for a few weeks — then tell me that software engineering is dead. But don’t knock it until you’ve tried it.

To prove this I’m going to use a tangentially-related personal experience that has absolutely nothing to do with software. In this case, it’s ok, because realistically, this bit of reflection actually has nothing to do with software – remember it’s about relationships. Let’s talk about two concerts I’ve seen this summer.

Coldplay: “You’re Awesome!”

I’ve heard Coldplay’s radio hits but I was never really a huge fan of the band. Had it not been for a friend’s invitation I would never have gone to this concert and I would have missed out on one of the best live shows I’ve seen. Acceptable radio hits became handcrafted masterpieces, more amazing live than on FM. The most interesting part of the show, and the thing that left the greatest impression on me, was the band’s absolute humble devotion toward the fans in attendance. At one point the band walked out to mini-stages set up in the seated area under the pavilion and way out in the lawn, giving all fans an equal chance to be a part of the show, not just the diehards that purchased overpriced tickets.

Coldplay’s attitude created a completely positive vibe that made me want to enjoy myself.  After they had exhausted themselves for close to two hours, doing everything possible to create a positive experience for the audience, Coldplay thanked us, not for coming to see them, but for allowing them to play for us, “You’re awesome, thank you for letting us play for you tonight!”

Dave Matthews Band – “We’re Awesome!”

I didn’t realize that Dave Matthews Band live shows are really just one long meandering jam session.  Every one of the band members is an amazing musician, but 14+ minute songs start to wear after a while.  Dave Matthews rarely spoke (when he did it was incomprehensible) and most of the band wore sunglasses throughout the entire show. Overall, it was a very mediocre concert that was better suited to an afternoon picnic setting than an evening at a large outdoor amphitheatre.  From a technical perspective it was amazing but overall it failed to entertain me.

The most interesting thing about this concert was the crowd. In spite of the fact that the band seemed to be playing solely for their own amusement, the audience was full of absolute fanatics, complete with their own dress code – plaid shorts and tie dyed shirts (though rarely together) with sandals – and a shared emotional connection to every song the band played. Dave Matthews Band is here to play music – If you like it, whatever, if you don’t, whatever.  “We had a fun time tonight and I hope you did too!”

It’s all About the Relationship

It turns out that the relationship the bands built with the audience during the show was directly related to how much I enjoyed it. It should be pretty obvious how this extends to software.  The relationship your software builds with your users will determine how much they enjoy using it. Is your software helping users be amazing or does your software expect users to change their behaviors to use it? Are you building something for yourself that others might happen to like or are you targeting a specific audience?

Software that is technically the best will always fall short to similar software that builds a relationship. Kathy Seirra expressed a similar sentiment with her Dating Rules for Software. But there’s a small catch to Kathy’s rules as exhibited by the Dave Matthews Band. Apple tends to act this way too. Sometimes if you do your own thing and you actually are great, you can afford not to focus on the relationship. It’s extremely difficult but sometimes it works. Sacrificing the relationship creates a razor thin margin for error but when you get it right, you’ll breed fanatics that will love you (almost) unconditionally and critics who will absolutely hate you – there is no mediocrity.

Sacrifice the relationship and you could be the best, but only until someone almost as good comes along who thinks about others before themselves.  I think Microsoft has learned that lesson the hard way.  No matter what you do, relationships rule the day.

You’ve got to know when to hold ‘em

When you’ve got a process that works, keep using it.  It’s a great feeling and there isn’t much better than finding a process that works well for you and your team.  When you finally settle on a process that works it can lead to a boon of productivity.  If you’re disciplined enough to capture data, use it to prove that your processes are actually working.  If you aren’t the data type you’ll have to rely on heuristics and intuition and the best heuristic I’ve found is the KA Index.  The KA Index is a subjective measure of how much your team Kicks Ass.  The best processes are the ones that make you awesome – fewer defects discovered in the field, better predictions of when you’ll finish tasks, fewer fires to fight – all of these are “gut measurable” data points indicating that your process is probably working.

Your gut can get you far but eventually you’ll need data to convince your boss to make the investment.  I can highly recommend the Personal Software Process if you’re interested in learning how to use data to objectively measure your processes.

Know when to fold ‘em

When you’ve got a process that isn’t working, drop it.  Data can be used here too if you’re collecting it.  If you’re not, I’ve found carefully examining process affordances to be helpful in uncovering bum processes but affordances can be extremely subtle.  The trick when giving up a process is to quit strategically – not too early, before you might see benefits, and not too late, after you’re experiencing severe problems.  (The Dip by Seth Godin is an excellent, short read on this subject.)  The other thing to look out for is the cost of dropping the process.  Even if your process is terrible, if worse habits fill the process vacuum you could be worse off.

Know when to walk away

Once you’ve decided to abandon a problem process it usually pays to gently transition to something else.  Changing processes can be expensive between learning curve, tool changes, and the requisite burn-in period when trying out something new.  If your bum process is only killing your team slowly, take a little time to read up on a replacement and avoid setting yourself up for failure.  For example, when my team switched to Scrum, we finished the iteration we were in when we decided to make the switch before making drastic changes to our operating procedures.

Know when to run

If your process is absolutely killing you, drop it.  Immediately.  When considering this route I recommend a thorough impact analysis including some back-of-the-envelope ROI calculations.  Changing a horse mid race will be expensive, I’ve seen it happen enough times in the DoD to know that you will pay dearly in the short term, but if the long term costs justify the change and you’re willing to go through the pain, do not hesitate to make the change.

Never Count your money when you’re sitting at the table

Generally I’ve found that processes out of a book need a few iterations of development before I’m able to understand how they really work and how I’m supposed to use them.  If you’re following an established process (XP, Scrum, TSP, ACDM, and so on), then try to follow it to the letter before tailoring it.  Most processes have built in mechanisms for changing them once you’ve gained some experience and by all means, tailor once you know something doesn’t work.

There’ll be time enough for counting when the dealings done

Postmortems are one of the most powerful tools in your silver toolbox.  No matter how the project goes down, always take the time to reflect on what happened, why, and what you should have done so you can use that knowledge next time.  Personally, I hate the idea “lessons learned” and much prefer recommendations for next time.  “Lessons learned” implies that you’ve learned the lesson and no further action is required but this is hardly the right attitude for effective software process improvement.

Every process is a winner and every process is a loser

So every software engineer knows that the secret to surviving, is knowing what to throw away and knowing what to keep.  And in the right hands, the right process can be a savior while under the wrong circumstances the same process might kill a project.  I can’t offer any concrete advice other than to not be afraid of data, listen to your instincts, and look back on what you’ve done.

I’m not saying there isn’t a creative component to software engineering. In fact, there is a huge creative component and creative problem solving is essential for building great software. I’m also not saying that code and the resultant software can’t be aesthetically pleasing. It absolutely can and should have good aesthetics. I enjoy reading well written, elegant code and always do the best I can when writing software myself. What I am saying is that art is more than just creativity and elegance and in almost all cases, software falls well short of art. Since the vast majority of software is not art, why should the people who wrote it be considered artists?

My neighbor upstairs is an artist in the traditional sense. He writes and plays music, he paints, he makes stained glass, and he’s a professor of eurhythmics (more specifically Dalcroze Eurhythmics) at Carnegie Mellon University. As you can imagine, he’s a really cool guy. He’s actually one of a handful of eurhythmics grand masters in the world and attended the world eurhythmics conference in Tokyo this past summer. From what I gathered from talking with him, eurhythmics is a way of studying art through the body’s motions. The most obvious application is music and dance but it’s not too difficult to imagine a painter creating a masterpiece; moving with the flow of the painting as it is created.

Some have even tried applying eurhythmics to study other, non-traditional art forms such as architecture and even mathematics. He explained it to me something like this. A brilliant mathematician working on an elegant proof might be considered more of an artist. But only someone truly brilliant who is in the zone, able to see all the connections of the work with unbridled clarity, tuning out the outside world (much like flow when programming), with body and mind working as one to solve a proof might be considered to be making art rather than doing math. It’s an extreme extension of eurhythmics theory, but it seems plausible.

The argument for treating software engineering as an art form is similar. Writing software could be considered art but only in the hands of truly brilliant software developers (Paul Graham calls them hackers) and only when those truly brilliant software developers are in the zone, building something amazing. Everyone else is just writing code. Much like how no one would consider multiplication tables artistic, anyone who considers churning out code without intention or understanding of what is written as art is kidding themselves. Looking at software engineering from the perspective of eurhythmics lets us state this more as a fact than an opinion.

Making this distinction is important because treating software like art and software engineers like artists is damaging to the industry. Only the greatest, most brilliant developers might ever have the ability to build software as artists. Everyone else is just trying to get by, much like the many thousands of starving artists who are never discovered, never hit it big, or whose muse never makes it around for a visit. Treating software like art allows developers who don’t have the artistic chops to have the excuse of creating poor quality software that is not fit for the purposes of its creation.

Fitness for purpose is another key distinction. Art exists to evoke emotion but software has a utility. In many cases, software is too important for it to not work correctly. Approaching software development from an engineering perspective creates an environment where it is possible for the rest of us to create working software that is still fit for purpose. Again, I’m not saying that software can’t be have good aesthetics, it should, I’m simply saying that a systematic, critical, methodical, mathematical, scientific approach to building software is required for most people to succeed – much like how architects need a solid foundation in physics and engineering to build beautiful buildings.

I like the idea of software as art. Artists are cool and thinking of myself as an artist allows me to justify extravagant behavior that, as a creative knowledge worker, I know I need. On the other hand, I rarely care about using software as a medium for creating an emotional experience for my users. I only really care about creating something great that helps users kick ass while accomplishing whatever it is that they want to do. Following sound engineering principles helps me meet this goal consistently.

Building software is a creative endeavor and the engineering part of the field is still very young compared to other engineering disciplines. But treating software engineering like an art form is a copout, an excuse for ignoring the sound engineering practices that do exist because they’re difficult or boring or unappealing. There are important creative components to building software but that shouldn’t hide the fact that proven engineering practices can and should be used too.

First, a little history

Security was never considered when the internet was originally conceived.  I guess that’s not entirely accurate, security was considered but quickly dismissed in favor of “the smallest set of tools that would give us the largest set of possibilities for the future,” according to Mockapetris.  Considering the original operating environment, the closed ARPANET/DARPANET, it makes sense that security was traded in favor of other quality attributes such as performance, reliability, and modifiability.  In the early 1980s, hosts numbered in the hundreds and there was an established and respected community of trust among the fraternity of hosts.

Once the internet became a public entity things changed.  Slowly, the community of trust that kept everything together began to unravel.  It started with news group flame wars.  Shortly after viruses started showing up.  There was even some direct hacking, albeit primarily by misled white hat hackers trying to help.  Then on November 2, 1988 the Morris Worm was released by a researcher from MIT and within 72 hours most of DARPANET was taken offline by the resulting accidental denial-of-service attack.  The community of trust that kept the internet running was permanently shattered.

CERT was created to deal with the problems that resulted from the world’s first denial-of-service attack.  While CERT’s mission is noble and its function critical to dealing with all the threats faced on the internet today, CERT is not the panacea required to save the world from the ever increasing internet threats coming from all angles in the form of botnet armies, distributed denial-of-service-attacks, and targeted attacks by sophisticated hackers motivated purely by profit.  The main problem is that CERT, by nature, is a reactive organization put in place to respond to emergencies as they happen.

As Vinton Cerf pointed out at the symposium, “the internet is always under attack.  It’s not enough to respond to each of the different incidents as they happen.  Reactivity is not enough.”  To save the world, we need to proactively address the security holes that were originally designed into the internet.

Short Term Solutions

Fixing the internet is an architectural problem.   It’s impossible to focus on a single layer of the protocol stack to make the internet more secure.  Each layer has different kinds of vulnerabilities and each layer can be exploited using different methods to achieve the same ends.  Unfortunately, replacing the entire protocol stack is difficult, thanks in a large part to how successful the internet has been.  But there are several short term strategies we can begin working toward today that work within the constraints of the current system.

One of the easiest and most significant things we can do is take humans out of the security loop.  It’s been proven time and again that people don’t follow instructions for securing systems.  Both John Gilligan and Vinton Cerf stated that improperly configured software accounted for over 80% of attacks on the internet.  Fix this problem and we’ve essentially removed 80% of cyber security threats.  Fixing the way we identify ourselves online (i.e. passwords) is another easy way to take humans out of the security loop.  A recurring theme from most speakers but most prominently discussed by Scott Charney was redefining identity online.  There are solutions that can work now to solve each of these problems.

It turns out that fixing the improper configuration problem is easy but it’s going to be a difficult sell.  The basic idea is that a computer or device has to start with a good configuration and then control mechanisms have to be put in place to prevent the configuration from getting hosed.  Some vendors such as Apple, Tivo, and Blackberry are already doing a great job at keeping their device configurations under control.  The trade-off for security is going to be freedom (modifiability if you prefer quality attributes).  In exchange for a usable, safe, and enjoyable internet experience that even your grandmother can handle you have to give up the ability to tweak anything you want in the system.  If iPod, iPhone, Tivo, and Blackberry sales are any indicator, people are willing to exchange freedom for something that just works.

Is it really fair to compare devices such as phones and PDAs with full blown computers?  Absolutely.  The key phrase is “Internet Enabled Device.”  An iPhone or Tivo can be hijacked and recruited into a botnet army just as easily as a standard desktop PC.  If even only 1% of the more than 4 billion mobile phones in use throughout the world were compromised, that would make 40 million bots ready to unleash mountains of spam and unending DDoS attacks.  The only reason this doesn’t happen is because mobile devices are effectively locked down to prevent such compromises from occurring.

With regards to passwords, I hate to say it but I think the Department of Defense is actually on the right track.  Every DoD employee and contractor is issued what is known as a Common Access Card (CAC), an ID card attached to a digital identification certificate issued by a third party authority that has verified that the person to whom the certificate was issued is indeed that person.  Both Vinton Cerf and Scott Charney were very clear that online identity can only be solved by combining something you know, usually a password, with something you have, a physical thing.  This something you have could be a digital identification certificate issued by an authority.

While it would be awesome to issue CACs to every citizen in the United States, perhaps attached to their driver’s licenses as Charney suggested, a staged success approach is possible.  PayPal has taken the initiative with a hardware device called Security Key.  Security Key is small enough to fit on a keychain and it generates a secret token (in this case a string of numbers) every thirty seconds.  When you purchase through PayPal you enter your username, password, and the token shown on the device.  Even if you accidentally gave this information to a phishing website, the token will be worthless after 30 seconds protecting your online wallet from fraud.

Though Vinton Cerf didn’t know the first thing about what I was talking about when I asked him about it, I see a lot of potential with OpenId.  OpenId is a protocol that provides something akin to a universal password that you can use to access thousands of websites and, with a little more work, could solve the identification problem across the internet.  First, OpenId providers would need to take on the role of verifying user’s real life identities and connecting the physical identity with the digital identity.  Second, OpenId providers have to offer some kind of physical verification when logging in, the something you have.  This could be in the form of a CAC, something biometric like a fingerprint reader, or even something as simple as PayPal’s Security Key.  In my opinion, OpenId is poised to become the third-party identification authority the world needs.

The Next Step

Another big theme at the symposium was trying to come up with a plan of action for moving forward, for fixing these problems.  Most speakers agreed that it would be unfortunate if it took a Pearl Harbor-like event to be the catalyst that motivated people to action.  Given the sophistication of attacks and the amount of value on the internet, the magnitude of damages could bring the United States, and in turn the world thanks to the global economy, to its knees.  The most frustrating part is that we know about the problems and I think there are a lot of smart people who know how to technically solve the problems.

By the end of the second day of the symposium I was left with more questions than answers and I am completely convinced that there will be a cyber security failure of a magnitude we have not yet seen in the near future.  I’m not suggesting that everyone start stocking canned goods in the cellar, but now is the time to proactively start working for change.  For as much flack as Microsoft has taken over the years, they seem to be on the forefront of effecting change both in terms of technology and public policy.  It’s difficult to know where to start, but Microsoft’s Trustworthy Computing resources are both comprehensive and practical.  The knowledge to fix the problems seems to be out there, the trick is motivating the world to change.

Obviously the idea of affordances is directly applicable to devices as well as software usability but it wasn’t until I read about the urinal flies that I realized affordances don’t always have to have a physical representation. For example, a well designed software process should gently nudge a team to do the right thing. Since there is no one-size-fits-all process that works for all teams it is essential that the process complements the team and that the process’s affordances nudge team members to do what’s best for the project and the team.

Using a process that lacks the right affordances could have one of two possible outcomes. In the best case, the team abandons the process because they realize subconsciously that it is telling them to do the wrong things at the wrong times. This is bad because it sacrifices repeatability; you’ve regressed back to an ad hoc, “make it up as we go” state. In the worst case, the team sticks with the process and it leads them astray. This introduces risks into the project and could lead to complete project failure.

Software is already difficult enough to build successfully and processes are supposed to make software development easier. Unfortunately, knowing when something isn’t working is not an exact science, but with a dash of experience and little team reflection (for example from regular postmortems) it is possible to figure out when you are working for your process instead of your process working for you. To demonstrate this I am going to tell you a story.

Our Process

My studio team in the Carnegie Mellon software engineering program is charged with building a web-based requirements elicitation tool that helps users follow the SQUARE process out of the SEI. About halfway through the Elaboration Phase of the project (sometime in the spring semester) the project was going downhill. The warning signs were fairly apparent, we were missing milestones, tasking priorities were confusing, and a lot of work was stalling out at different levels of partial completion. Though we knew there was something wrong we weren’t really sure what was causing it, what we were doing wrong in our planning and tracking process.

The planning process we were using was fairly simple. At the beginning of the phase we looked at all the activities and artifacts that need to be completed by the end of that phase. For each identified milestone we enumerate specific entry criteria, general tasking, validation procedures, and exit criteria. This is a technique known as ETVX (entry, tasking, validation, and exit). Next we used planning poker to estimate how long we thought each milestone would take to complete. Finally, with this information we created a phase timeline which includes known due dates and dependencies between milestones.

Since we’re using an iterative approach to complete work in a phase, iterations follow largely the same planning process on a smaller scale. As a team we identify the milestones on which we will work during the iteration. Each milestone is assigned an owner whose job it is to ensure the milestone is completed by either delegating tasks or working on it themselves. The planning poker estimate is used to determine the approximate workload allocation on the team. This estimate is validated with bottom-up estimates that team members create based on their individual tasking.

There are several good things about this process. First, it’s written down and the team follows it. This is good because it means we can produce repeatable results over time. Second, this process makes use of several practices that are generally considered “good” by software experts. ETVX is a great way to clearly identify project milestones. Planning poker is similar to the wide-band Delphi estimation technique. Third, we’re using two forms of estimation to validate the plan as more information becomes known. Finally, the engineers responsible for the work determine the specific tasking and creating the bottom-up estimates.

You’re Good, but not That Good

In spite of all the good things we were doing, something still wasn’t connecting. The big aha! moment occurred about two weeks into the second iteration. Up to that point I had been working on my tasks that had carried over from the first iteration. The team leader noticed that almost no work had been started on the milestones I owned. [An aside: this, to me, says that at least our tracking process works somewhat well.] During the discussion that followed I became extremely defensive when the team leader asked me to shift priorities for the rest of the iteration. What should have been a simple request turned into a heated debate over tasking. I felt compelled to complete the past due work and here was this jerk trying to stop me. “Sure,” I thought, “I’ll do what you ask, buddy, but when this whole project comes crashing down it’s on your head, not mine.”

Later, as I looked back at the incident, I wondered to myself, “Why was I so defensive in light of such a simple request?” The reality was that the project wouldn’t come crashing down if I shifted priorities and I knew that. So why defend these older tasks when it was obvious that there were more immediate needs?

It turns out that the affordances built into the planning process were encouraging my behavior. There were a few simple things at play that, when combined, decreased our ability to plan effectively.

First, our process encouraged us to plan more work than time allowed. This was due to there being a missing connection between day-to-day progress and the “big picture,” the overall plan. Second, though the new team leader may have believed there was consensus, the team in fact did not wholly agree with the priorities for iterations. This behavior was not specifically discouraged by our planning process and so allowed to persist. Third, leftover work was not addressed during planning. Some tasks might simply expire while others may change priority, becoming more or less important with a new iteration. Since this wasn’t addressed it created a sense of urgency for individuals carrying over work from iteration to iteration. Finally, assigning milestone owners had unanticipated side effects. The goal was to ensure that someone was taking responsibility for coordinating and monitoring milestone work. This worked so effectively that milestone owners exhausted themselves attempting to finish milestones and resisted changes to the plan that prevented them from finishing what was promised.

When it came time to make a necessary modification to the plan, our process encouraged us to fight against the best course of action for the team. We didn’t have the level of flexibility needed due to our process’s affordances nudging us to do the wrong things. Milestones were slipping and people wanted to finish what they started. Project priorities were shifting as the project matured but team members were wearing blinders, ignoring the changing facts around us. To stand a chance at success we had to change the affordances in the planning process. We had to nudge the team in a new direction.

Our Solution

To try to solve this problem we decided to incorporate some of the planning principles from Scrum, specifically the product backlog, sprint backlog, and sprint planning meeting, into our planning process. Scrum takes a more task-oriented approach when planning iterations and correlates the sprint backlog with the product backlog. This better encourages the team to not plan more work than there is time to complete while connecting day-to-day work with the overall plan. Scrum also requires that the team reprioritize work when planning iterations and that we agree on the resulting priorities. This will hopefully eliminate the prioritization conflicts we experienced during iterations. With Scrum, leftover work from iterations is saved in the product backlog. This change decreases the anxiety team members feel when work is left undone (because the work is not forgotten) while simultaneously giving the team more flexibility to change direction as the project progresses. Finally, the team, rather than individuals, takes ownership over the milestones held in the product backlog. With each commitment made during iteration planning, the whole team buys in effectively shifting the passion and dedication individuals held for owned milestones to the commitments we agreed on as a team.

I’m not really sure how Scrum is going to turn out for us. I think the most important thing is that we recognized that something was not working and took action to correct it. I personally would rather see the team fail in a new and spectacular way rather than repeating the same mistakes again and again.

Add This to Your Silver Toolbox

Unfortunately, I don’t think there is a trick for detecting these sorts of process failures. Data and metrics can help but only if the process is repeatable and the team has the knowledge and discipline to collect the data in the first place. Team postmortems can help but if individuals are afraid to raise concerns, you’ll find yourself on a trip to Abilene before you realize it. In many cases, if you think something isn’t going well, others are probably thinking the same thing. Once I spoke up I found out that others thought something wasn’t working also. I was just the first person who was able to articulate it.

Affordances are powerful but subtle mechanisms. In well designed things, we aren’t supposed to be consciously aware of them. But that doesn’t mean they always nudge us to do the right thing.

“Does anyone have any other risks they’d like to bring up?” I asked, confident that, as the team leader, I had a firm grasp on how the team was doing and where our current problems lay.

I waited a few seconds.

“OK, if nobody’s got anything I’ll go ahead and email this out…”

“Communication,” the quietest member of the team chimed in. “We have problems with our team communication.”

I was in shock. Surely this must just be an isolated problem. “Do you mean team communication or do you mean you don’t understand some aspect of the project?”

“I agree. Team communication seems to be problematic.” Now there were two dissenters.

A few seconds later there were three. Three fifths of the team felt we had team communication problems that were directly impacting the outcome of the project. I sat there shocked for a moment. I had just been blindsided by a team communication problem, therefore, obviously, we have a team communication problem.

Looking back at this incident I have decided that leadership, my leadership, was to blame. From the beginning, I had led the way I liked to be led: hands-off with enough space to make my own decisions and get things done the way I wanted to do them, the polar opposite of micromanagement. Of course, for me this worked out really well. This was the way I liked to work and the way I was used to working. Unfortunately, for a new team, a team lacking in trust, whose members didn’t know one another and had other commitments and priorities (school work) my ideal working environment and preferred leadership style was disastrous.

To prevent utter team destruction I changed perspectives. From what I could tell, team members didn’t know what they were supposed to be doing. I had assumed that this team would operate similarly to my last team where everyone would come together like a well-oiled machine to get work done. Shortly after the incident in the library I remembered that my last team took almost a year to become that awesome. Of course there were going to be problems in my new team.

Since my team seemed not to know how to take initiative in completing tasks and getting work done I thought I’d set an even better example for them. By doing so, maybe they’d get the hint and follow suit. The next day I kicked things into gear. I took on more work. I picked up all the slack I could and then some. I took on tasking outside of work specifically assigned to my role. I set high standards for myself and my team and we were going to meet those standards if I had anything to do with it.

Leadership disaster number two was prevented thanks to a timely assignment in my Managing Software Developers course. Leadership that Get’s Results by Daniel Goleman defines six distinct leadership styles and gives a little advice on when each style is appropriate. Up to that point I had no knowledge of such styles and had been flying on instincts. The following table is taken from Goleman’s paper but I encourage you to read the full paper to gain a better understanding of these ideas in a more complete context. There’s also tons of information on the web, a search away.

As it turns out, I started with the right idea by using an authoritative style of leadership even though I didn’t know the name for it. I failed with this style because the team wasn’t ready for it yet. We didn’t have a clear vision or clear goals. We weren’t yet working well together. There was little buy-in to my leadership or the few goals the team did have. We were a full-fledged dysfunctional team. Switching to a pacesetting style was just about the worst possible thing I could have done. If I had gone on for too long I am fully confident that I would have destroyed the team and we wouldn’t have gotten anything done during our first semester. With my new found knowledge of the different leadership styles I now had new options. I immediately put three styles to use: affiliative, coaching, and coercive.

The team was clearly broken. My hope was that affiliative leadership might help build trust and better bonds among teammates. Being a programmer I tend to be a little more logical than emotional so trying to tune in more to how people felt was tough. I found that affiliative leadership goes hand in hand with vulnerability and trust and that individually thanking someone for their hard work, even if it’s relatively small, and meaning it is one of the most important things you can do as a leader.

In addition to team harmony, it was obvious that some team members were struggling with the tasks they had been given. Rather than taking over those tasks as I had been doing, I decided to try taking on more of a coaching role. In some cases I would help team members directly, other times I encouraged other team members to work together on tasks. The end result was a team better able to work together to accomplish tasks.

In spite of all this, I had the feeling that if we didn’t do something immediately, the entire semester would be a wash. To prevent this from happening I used coercive leadership in an attempt to get us back on track quickly even though the overall impact could be negative in the long term. In a sense, I was willing to be a bad guy so the team had a chance of meeting its goals. The gamble paid off and the way the team operated turned around almost instantly.

About two months after I had been blindsided by unseen communication problems, my team seemed to be working together much better. Problems were being flushed out into the open more quickly and everyone on the team seemed to enjoy working with one another on the project. I am not going to take full credit for the change but I will take credit for being the catalyst that put the change into motion. All because I changed how I led the team.

There is a small downside to the changes, but it’s only really a downside because I enjoy doing technical work. The leadership role on my team has evolved. By the end of the semester I found myself directly responsible for almost no work but rather, I was the go to guy for all problems the team was facing. I had my hands in everything but I wasn’t able to really sink my teeth into anything. If this is what management is like and I ever do become a project manager I will need to find ways to remain technically involved in something or I think I’d eventually go insane.

The biggest lesson I’m taking away from this experiences is that sometimes the team’s needs and my needs aren’t going to line up. Recognizing when this is occurring and finding a balance between these conflicting needs is critical to team success.

First, let me clarify what I mean by formal methods.  Formal methods use mathematical representations of software to formally prove that the software behaves as expected.  Most formal methods use predicate logic and set theory along with other mathematical notations to describe software in a formal specification.  Examples include Z (pronounced zed), state machines (such as the finite state process algebra FSP), Petri Nets, the use of pre and post conditions, and the Object Constraint Language used in conjunction with class diagrams in UML.  Most formal methods have some kind of automated proof capability be it by brute force or through the use of more sophisticated theorem proving programs.

Let’s take a quick look at Z which is probably the most widely available formal specification method in terms of books, online documentation, and automated theorem provers.  Here’s a simple example of the birthday book taken from “An introduction to Z and formal specifications” by Mike Spivey.

Looks like gibberish, right?  It isn’t actually important that you understand what this means.  In fact, if you don’t understand it then I’ve already made my point.  Writing formal specifications using a formal modeling language is time consuming, costly, and in the end, not effective since it is too difficult to understand what the specification is actually telling me, the programmer, even if I’m trained in Z.

Formal methods get a lot of traction from the ridiculously high cost of defect injection early in the software lifecycle and the unacceptably high cost of defects in safety critical systems.  Injecting a bug in the requirements phase that is not caught and fixed until testing has been estimated to cost upwards of 100 times the cost of finding and fixing the same bug while the requirements are being written.  In safety critical systems, where human life is literally in the hands of software, deployed defects could result in injury or death (the Therac-25 accidents being some of the most famous).

Formal methods folks argue that a clear, precise, unambiguous, and proven formal specification can prevent many of these early defects from making their way into the software and thus prevent these problems.  Economic claims aside, formal methods fail because they still ignore the weakest link in building software: the human programmers.

Mathematics tells us that it is possible to create functions that map one representation of a set to another equivalent set.  For example, given a cube in three-dimensional space, a rotation function will maintain the dimensions of the cube while rotating the shape about a desired axis.  A similar principle is applied when considering Z and other formal methods.  Mapping functions are used to transform a high-level Z specification into a more concrete representation of the same specification.  Eventually the specification reaches a level of detail that is easily translated to code.  In practices this might mean that a set, such as birthday in the BirthdayBook schema above might be represented by a pair of arrays in a more concrete version of the schema.

While theoretically this sounds great, the problem is that the mapping cannot be automated in most cases so we’re essentially back to square one in terms of what we can say about the more concrete version of the specification.  This is thanks to the inevitable mistakes injected during the transformation process.  Kennedy-Carter’s iUML from the Model Driven Architecture camp is the closest thing to an automated transformation tool I’ve yet seen and even that did not include OCL or any sort of pre and post condition analysis.

So formal methods are difficult to write, difficult to understand, few people know them, and they don’t translate easily into code.  Is there a silver lining?  After all, a lot of time and research has gone into devising formal methods and some of the greatest minds in software engineering are behind them.

I think the answer is a big fat maybe.

Formal methods, in the right hands on the right project can make it easier to understand the system being specified.  A Navy track manager might be one good example.  Rules for how ships assume reporting responsibility might be easily recorded as a predicate.  Safety is certainly a concern and though the validation process will have to begin anew once code is written at least the basic algorithms will be clear and unambiguous.  That is of course assuming the right people are working on the project.  Then again, in the case of the IABM, any specification would have been better than the big plate of nothing the developers had to work with.

For almost every project in the world, I think formal methods should be generally avoided.  Given the option of spending money and time on mathematicians or extremely smart coders I would chose the latter.  With smart coders, code inspection is a fun and effective defect filtering option.  And let’s face it.  Why would you have your amazing coders do something other than write amazing code?