Our latest code-quality theme, the Interface Segregation Principle, says your classes and interfaces shouldn’t try too hard, either. If one object really is going to do two jobs, give it two interfaces and let other objects talk to whichever interface they actually need. This gives you the flexibility to split the object into its constituent parts or replace some of those parts in the future.

We found several examples relevant to the Interface Segregation Principle in our code, including

• a PriceManager object that manages many different types of market data in disparate ways (classes with “manager” in the title are particularly susceptible to double-wide-interface disease)
• a single Services object that implements interfaces for various families of actions (creating, updating, and closing)
• a Builder object that had an extraneous getter floating around in its interface (we moved the getter to a more appropriate object)

You may enjoy the demotivational posters we came up with to illustrate the Interface Segregation Principle:

All I wanted was a volume control.

I said a pizza with everything, dammit!

Two great things must work great together.

Bach compared the situation in testing today to early days of software development where compiling and writing code were all called programming, then compiling got separated and renamed to reflect the fact that it is completely done with a machine. Checking can be automated to a great degree and delegated to the machine, said Bach, offering an example of classic TDD unit tests. Testing as a concious process needs to be done by a trained specialist. This allows us to put more emphasis on exploratory testing, which Elisabeth Hendrickson listed as one of the key seven practices for successful agile testing.

On a similar note, Mary Poppendieck talked about how testing is a learning activity at the Agile Testing Days conference in Berlin, contrasting it to checking the correctness using known expectations.

See other articles from the Oredev conference.

While XMLUnit is a hip JUnit-style framework, you can certainly leverage it outside of JUnit. What’s more, via easyb’s plug-in framework, using XMLUnit just got a lot easier! For instance, leveraging XMLUnit without the new plug-in required a bit of manual manipulation of XMLUnit’s Diff class and then a boolean verification like so:

import org.custommonkey.xmlunit.XMLUnit
import org.custommonkey.xmlunit.Diff XMLUnit.setIgnoreWhitespace(true) scenario "the XMLRepresentationBuilder should build XML", { given "a table name and collection of name value pairs", { lst = [new ColumnNameValue("LN_ID", 10002130), new ColumnNameValue("NIBA", "99.99"), new ColumnNameValue("EFF_DATE", "2009-04-01")] map = ["LN_ID":"id", "NIBBA":"nb","EFF_DATE":"effectiveDate"] } then "the XML produced should be valid", { rep = new DatabaseTableResource() rep.columnNameValues = lst rep.columnNameMappings = map rep.identifiers = ["LN_ID"] out = XMLRepresentationBuilder.buildRepresentation(rep) control = """<cmnres id='10002130'> <nib>99.99</nib><effectiveDate>2009-04-01</effectiveDate> </cmnres>""" diff = new Diff(control, out) diff.identical().shouldBe true }
}

Note how in this code it’s necessary to:

• Ensure white space doesn’t cause comparison errors via the XMLUnit.setIgnoreWhitespace(true) call
• Use XMLUnit’s Diff class to compare two documents and then call the identical (or similar) method to actually verify the documents

With easyb’s XMLUnit plug-in, things are a lot simpler! All you need to do is insert a using clause and ensure the plug-in is in your classpath. Watch:

using "xmlunit" scenario "XML documents are compared easier via xml unit plug-in", { given "some xml document", { control = """<account><id>3A-00</id><name>acme</name></account>""" } then "the plugin should enable easy comparisons", { testXML = """<account><id>3A-00</id><name>acme</name></account>""" testXML.shouldBeIdenticalTo control testXML.shouldBeIdenticalWith control //same behavior different call testXML.shouldBeIdentical control //ditto testXML.identical control //ditto } and "the instance of XMLUnit should be available for use", { XMLUnit.version.shouldBe "1.3alpha" XMLUnit.getIgnoreWhitespace().shouldBe true }
}

Notice that there isn’t any need to alter the white space setting (it’s been done for you in the plug-in — yo can undo it easily too — see the and clause?), nor is it necessary to use XMLUnit’s Diff class — it’s all done under the covers. While this story uses the identical call, there is a corresponding similar call (and related pattern — that is, shouldBeSimilarTo and the like).

Also, the initialized instance of XMLUnit is available should you need further configurations, etc — the plug-in inserts the copacetic instance into the binding available to your behaviors as shown in the and phrase above.

As you can see, the XMLUnit plug-in for easyb makes workin with xml (that is, comparing documents) a lot easier. While there’s plenty of ways to skin a cat, so to speak, this is just one of them — I’ve certainly found it handy from time to time — I think you will too!

You can download the current version of the plug-in at easyb’s Google code site along with a few other hip gems like easyb’s DbUnit plug-in.

One perennial issue - and this isn’t new with agile - is that testers and teams still struggle with how to design and write automated tests that have a good return on investment. That is, the tests are reliable, stable, maintainable, and provide good value for the amount of work that goes into them. We also all struggle with how much to automate, what to not automate, what to keep in our regression suite.

Janet and I devoted a section of our book to automation, and we presented principles, techniques and strategies that will help. We also include this in our Agile Testing course, and we both do conference tutorials on how to succeed with automation in agile projects. I think in addition to that, a lot of people need some practical hands-on experience. They need to be guided through an example of automating tests for a story, then they need to try it for themselves.

I’m developing an internal class for people at my company, and we also have ideas for workshops and other ways for teams who have solved some automation problems and developed good practices to share with other teams. I just mind mapped the ideas we have so far. At Adam Goucher’s suggestion, I’m posting my first draft here.

It’s my hope that I could extend this to a more generic class that would be appropriate for a conference tutorial. I’ve talked with some folks about collaborating on something like this and kind of making it open source, with several of us who could teach it, like the Watir tutorial that Bret Pettichord, Charley Baker and others have taught. But, others better than I have tried to teach hands-on courses like this and run into issues. What tools would we use that can easily be installed on all participants’ laptops? What if the participants have different skill levels? So, I don’t know if this is really possible. I’m interested in ideas, or if anyone has done this successfully.

Actually I’m sure this has been done successfully by people like Elfriede Dustin who have been specializing in test automation for years - I need to get her book out and re-read it. I’m not trying to steal anyone’s idea or say that this is a new original idea I thought up. I just see a gap in many teams’ ability to create the ‘right’ automated regression tests.

So, here is the mind map, you’ll have to open it in a new window (I’m sure there’s a better way to do this but I have to catch a plane), feedback welcomed.

Make it visible

AppStore is now saturated with games, with over a hundred new applications appearing weekly. Schade advised first carefully choosing the category where you want to compete. Putting this into context, he said that there are over 6000 puzzle games in the store, 1107 card games, 1014 role-play games, 679 casino games, coming down to less than hundred in high-end 3D simulations.

To make the application visible in AppStore, Schade advised choosing a catchy name and creating an icon that presents unique game features. For example, the icon for their Rally Master Pro game shows dirt on tires and lights to demonstrate that they have levels with different tracks and weather conditions. He also advised keeping the product description short and snappy as customers mostly won’t read that first but look at screenshots. This also means that you have to choose screenshots wisely to demonstrate unique features of the game. For keyword selection, he advised applying traditional SEO techniques. “Pick keywords wisely as you won’t be able to change them without resubmission. Ask yourself what people would look for without knowing your application. Those who know your application will find it anyway”, said Schade. He also advised making price drops very clear and visible because some customers wait for price drops to buy games.

Marketing schedule

Fishlabs has a very aggressive marketing schedule for games. The key phases there are:

• Announce the title early and focus on things that make the game special (eg. number of tracks in the game, weather)
• Contact journalists to provide them with unique identifiers (UDIDs) to be able to deliver adhoc beta builds to them. Schade said that developers get only up to 50 UDIDs yearly, so they are careful about who they give them to. He suggested contacting people on Touch Arcade forum (US), Pocket Gamer (UK) and Slide To Play
• At the alpha stage, they spread first screenshots
• At the beta stage, they release the game-play trailer on YouTube and provide beta builds to journalists for initial feedback
• Two weeks before submission, they release the official game trailer on YouTube to generate buzz on forums. Schade said that these trailers can get up to 100k views.
• When submitting the application to AppStore, they spread the final screenshots and story about the game online.
• They they send the final version to journalists for review and chase those who haven’t published a review. Schade said that most journalists know that UUIDs are scarse, so they will respect that and publish a review.
• Once the game is live, they send out the press releases. They need to be sent to Apple upfront according to the developer agreement, so Schade advised taking extra care to get the spelling of Apple products right.
• They then provide promo codes to journalists and enthusiastic gamers with blogs and youtube channels that will review the game. To startups without such connections, he suggested announcing promo codes on the promo-code section on Touch Arcade.
• They recently started running online and mobile banners banners for the first time, so it’s still early to know the effects of this.
• Then they carefully monitor user reviews, YouTube comments and special interest groups, replying to complaints and offering advice, making sure that comments from spammers and competitors are removed where possible.

Prepare for promotion

The US customers represent 50% of the market, UK customers 30%, so releasing only the English language initially first makes sense. Schade also advised considering are French, Italian, Spanish, German, Dutch and Japanese. Even if you release only English initially, Schade advised to be prepared to localise very quickly if needed. Apple might decide to promote the application internationally if it becomes popular, said Schade, and they might ask you for example to give them a Japanese localised version that same day. In that case, you’ll have to act very quickly. “If Apple asks you to jump, don’t ask why but how high”, said Schade, because being promoted helps a lot with sales. You also need to have hi-resolution screenshots ready in layered PSD format for this. Although there is no guarantee that the application will be promoted, you can increase your chances by monitoring applications that are currently promoted and trying to understand why they are there. As some good ideas that might get the game promoted, Schade mentioned utilising latest technologies, having very polished games or a distinctive style. Schade advised not launching the light game while being promoted or dropping the price.

Online CRM

A key marketing tool for application developers is online CRM, but not in a traditional sense. Schade advised being very active on forums and monitoring YouTube channel comments and Twitter feeds for people talking about your application, especially those that are complaining. “Watch out for major complaints and fix them ASAP”, advised Schade, saying that developers should be responsive and seek out clarification for any complaints in order to either provide advice or reply constructively to diffuse potential bad reputation. The most important period to monitor are the first two to four weeks after a release, especially the weekends when the traffic goes up three times. Schade said that monitoring online channels is now a 24/7 job for him, and that other team members do that occasionally as well.

They also make an effort to broadcast news on Twitter. Schade advised posting anything relevant to the game, especially new reviews and price drops because people retweet these things. Again, Twitter feeds should be monitored closely and any complaints should be dealt with straight away. He also advised following journalists and game reviewers because they might also follow you then.

Game commercials

Schade categorised applications in two groups: Premium (costs between $4.99 and $9.99) and Freemium ($0.99 to $3.99). For premium games, he advised launching the full version to early adopters and then launching a light version when the full version failed to make it to charts or dropped out. “If you’re not in top 50 within the first week, you’re never going to make it [to the chart].”, said Schade. The light version should be fully playable on at least one level in proper game mode, not just be a 30 second preview. In-application purchase (available in 3.0) should be used to unlock full version without download. For freemium games, Schade advised offering a free download and then using in-game purchase to unlock it. The purpose of this strategy is to maximise reach and make people addicted to the game before they purchase. “Allow people to feel the game and tan ask for money”, said Schade.

See other news from the Oredev conference.

The day featured work on things like Google App Engine and the Android application platform. I took this chance to get myself out of the PHP world I’ve been stuck in for about a year now and get back into some Rails development. I had a little tool in mind that I wanted to develop which would help us internally do some things a bit more efficiently. I used that as an excuse to try out some of the Rails plugins or gems that I have wanted to use for a while, which are:

• Mocha (http://github.com/mislav/rspec-rails-mocha http://mocha.rubyforge.org/)
• Factory Girl (http://github.com/thoughtbot/factory_girl)
• GChart (http://github.com/jbarnette/gchart)
• Open Flash Chart (http://github.com/pullmonkey/open_flash_chart)

I have to say, the buzz around Mocha and Factory Girl is very old and I wish I had the opportunity to use these tools before now. New things come out everyday and it is tough to keep up with everything. My advise to you is to dedicate a “playing time” often for you to try out the newest and greatest thing that comes out. Do this in groups where each one can be working on a different things and share your findings at the end of the day. Even though I didn’t work on the Android app, I did get a feel for it from Arin through his experience with it. One of the things that helped take this experience further is our presence all together in one conference room feeding out of each other.

Looking forward to the next OSC hackathon.

One thing we talked about was the “Cone of Uncertainty”, as described in Steve McConnell’s book “Rapid Development.” You can read the full description of the cone of uncertainty on the Construx website. Here is a drawing of it as depicted on that website.

The Cone of Uncertainty is a convenient way to think of how software estimation works. When a project is first beginning, you know very little about it and so you should never give customers a single point estimate. A single point estimate just means that you supply one number, like saying “this project will take 6 months.”

Giving estimates in ranges instead is much more valuable and realistic. This more accurately conveys to your customer the uncertainty of an estimate, and therefore the risk. For example, if I give you a wide estimate like “this will take 3-12 months”, that is a very clear way to communicate that we don’t know enough yet to give a firm estimate.

The Cone of Uncertainty shows that the more you know about a project, the more you can narrow that range. So while I might initially think it’s 3-12 months, after spending time analyzing the project, I can revise that to a more narrow range, like 6-8 months perhaps, or 10-12 months.

But what about the end of a project? The Cone of Uncertainty says that the closer you get to the end of a project, the more likely you are to know exactly when it ends. That makes sense in theory, but is it realistic?

I would suggest that many times, there is a large amount of uncertainty at the end of a project, even if it’s well run. Imagine that you are two weeks away launching a new site, and you are still working out some bugs that your testers have found. You know you only have a few bugs left, so you confidently assert to your boss or customer that you’ll be ready to launch in two weeks. Each day you are fixing bugs, but the testers keep finding new bugs. Perhaps they are even finding new bugs faster than you are fixing them, or their rate of finding bugs is not decreasing.

When you had a short bug list in front of you, you felt very confident about two weeks, but now you are less certain. How many more bugs will testing uncover? Your confidence about the time left to launch is now decreasing, and you have to go back to your boss or client and ask for additional time.

The Cone of Uncertainty captures risk at the front end of a project very well, but it doesn’t capture the risk and uncertainty at the back end of a project.

There’s one other thing that also makes me feel like the Cone of Uncertainty is perhaps a bit simplistic. In my classes, we discussed using the Cone of Uncertainty as a way to educate our bosses and clients about why estimates are inaccurate. The idea was to sketch it out on a napkin or whiteboard, and use it to convince them why you can’t give them a set estimate up front (”6 months”), but instead need to estimate in a range (”3 – 9 months”).

However, I believe the problem is that the Cone of Uncertainty implies that you will converge on a middle point most of the time. While that is not the intention of the cone, it still can appear that way because of the way the cone is drawn. So if your boss wants to be tricky, he can play games with the numbers. If you tell him that a project will take 3-9 months, then he may just translate that to mean 6 months (the midpoint of your range.)

But how often have your range estimates actually converged on the midpoint? I would speculate that most of the time, they actually converge on something above the midpoint. That’s because our initial reactions are almost always optimistic, so as we learn more about the project’s complexity which we had not originally anticipated, then we are likely to settle on a point higher in our initial range estimate.

So I would further suggest that it’s better to draw the cone as not converging on a midpoint. At some point in time, your estimates are likely to converge on an estimate that is higher than your original midpoint. This leads me to my humbly suggested variation of the cone of uncertainty, which I like to call “The Wormhole of Reality.”

Why the wormhole of reality? When the cone is redrawn like I’ve done above, it just looks more like a wormhole than a cone. I’m referring to the wormholes of physics and astronomy, where it is a shortcut through space and time. There’s no correlation between the meanings of wormholes in space and my estimation wormhole, it’s just that they look similar. Plus I think it’s a cool name.

But more importantly, how do we avoid getting sucked into “The Wormhole of Reality”? If the Cone of Uncertainty is ideally the way we want our estimates to work, then we want to avoid the “fuzzy back end” of a project. As I already suggested, the fuzzy back end is driven by continual bug reports that seem like they may never end. Here are some tips to try and close out your projects more predictably:

• Use test driven development to write your unit tests early.
• Use browser based automated testing tools like Selenium to automate user level testing.
• Keep your continuous integration server happy and well-fed with frequent code check ins, so it can feed you bug information earlier rather than later.
• Engage users in your system testing early on. Make sure that you deliver incremental releases of your code, at least to internal users and testers, well before the final weeks of your project.
• Engage customers in those incremental releases too. If your internal users are not an accurate representation of external users to your code, then identify some beta customers who would be willing to try out future versions of your product.
• Consider incorporating regular user-experience testing with beta releases of your product, on a feature by feature basis. This is another way to get additional feedback about the code you are going to deploy.
• Use agile methodologies to drive all of the above recommendations.

While the “Cone of Uncertainty” is an easy way to explain to managers and clients how to estimate software development, too often we fall into development traps and end up in dangerous territory like “The Wormhole of Reality.” But by following best practices, and clearly communicating to customers the risk at the front and back end of a project, you will be more likely to avoid these pitfalls and deliver your software on time, on scope, and on budget.

“We get what we measure”, said he, adding that the system is going to game the metrics. “Does finding a lot of bugs really mean that you have good testers, or does it mean that you have crap code?”, asked North. He also mentioned several examples of how people game the system in software development, including commenting out failing tests to increase test pass rates and generating tests without assertions to increase test coverage. Improving the efficiency in these parts leads to inefficiencies on the global level. “When you focus on efficiency of a part, you lose sight of the big picture. [...] Efficiency is the enemy of effectiveness.”, said North.

Likewise, things that seem very inefficient can produce great effect on the global scale. As some examples of that, North mentioned cross-functional teams, pair programming, parallel spikes and the theory of constraints. Using a less controversial topic, he said that experimentation and innovation are very inefficient if considered on their own because they lead to lots of work that will necessarily be thrown away, but the global results of that are very effective.

North suggested that teams shouldn’t care about effort, but about the effect. Functional points, velocity, man hours and similar are easy to game and can lead to local optimisations that hurt the big picture. It is crucial to measure the correct thing and North suggests measuring cycle time for end-to-end customer satisfaction or cost of the end-to-end process, including opportunity costs and factoring in time.

So how can we improve the effectiveness? North suggests these pointers:

• Understand that you’re working in a system, and that it is likely that there is a system above that system and a system above that one
• Identify the goal of that system – sometimes it’s just its own survival
• Step outside to observe the effects – how well does its goal align with yours?
• Think about what’s stopping the system from giving you what you want, and improve that.

I’ll be posting many more articles on Oredev in the next few days. Subscribe to my rss reed to get notified about updates

There were so many good sessions and I’ll only mention a few here. Mark your calendars for next year’s Agile testing Days, Oct. 4 - 6.

My slogan after this conference comes from Dave Evans (and from Mike Scott - not sure which one of them coined the phrase but they both talked about it): Respect the Tests. More about that in a minute.

Elisabeth Hendrickson always inspires me, and her keynote didn’t let me down. I’ve heard her definition of agile before, but it bears repeating: Delivering value in the form of releasable software at frequent, regular intervals at a sustainable pace, while adapting ot the changing needs of business. She quoted Mary Poppendieck as saying, if you want to go very fast, you have to be very disciplined. Sustainable pace means that we keep technical debt down so that the code doesn’t impede us.

Elisabeth presented her four sources of technical risk: Ambiguity, assumptions, dependencies and capacity, and her 7 key testing practices: TDD, ATDD, Exploratory Testing, Automated System Tests, Collective Test Ownership, CI, and one more that I think is automated unit tests (I can’t read my notes!)

Elisabeth also quoted Tobias Mayer, who refers to the Product Owner as the Voice of the What, and the dev team as the Tribe of the How. I like this, but I take issue with it a bit. The Poppendiecks have influenced me to experiment with having the dev team learn the business inside out, so the dev team itself can help the business prioritize and even think of features. This worked well for my team at ePlan, and I have to agree that funneling all the business requirements through the Product Owner has a lot of drawbacks.

What was new to me in this talk was something that several of us had talked about a night or two before, the idea of Collective Test Ownership. This ties in with the Whole Team Approach. Everyone on the team takes responsibility for quality, testing, and test automation.

Another great session, again from someone whose work I’m familiar with, was David Evans‘ talk. One big takeaway for me was this: If a test is worth writing, it’s worth automating, and it must always pass in the future. I often see teams that ignore failures in their CI tests because “We have to get these new stories out, this is the PO’s priority.” So the PO doesn’t care whether what she prioritized in previous sprints still works or not? When the CI breaks, the team should stop the line and get it green.

Dave pointed out that if all tests pass, you don’t need extra metrics. Defects are evidence of missing tests. So don’t call it a defect - call it a failing test, write the failing test and fix it right away. If the customer really doesn’t want it fixed now, write a story for later.

Dave recommend fixing time and quality, but don’t fix functional scope. If time is running out, thin the requirements. It’s not like leaving a wheel off or leaving out the engine. Respect the tests - any test worth writing is worth running all the time.

Dave presented his Agile Quality Framework. I won’t go into all the details, but one of my favorite parts is applying balloon patterns. An empty balloon is still a balloon, but if you don’t have the rubber outside, it’s hard to have the air and then add the rubber later. Do the rubber first - the CI, the test framework.

Mike Scott elaborated on the balloon pattern in his demo of Testify Wizard. In one hour, Mike went from zero to a full CI process, including the project two continuous builds, one running unit tests, and one running FitNesse tests. It even generated code coverage reports! I think he did this with both Java and .Net during the same session. He did this with a couple of pushes of the button,some minor configuration and a wave of his crutch.

This shows how easy it is to respect the tests. It’s not that hard to build a framework to integrate our code and run our automated tests so that we get quick feedback all the time.

The Build Doctor is now running a new version of Wordpress, Thesis and some other plugins, on a new VPS. I have plans to make the blog more useful now that it’s not running an ancient version of Wordpress. On the positive side of things I was very impressed that none of the vulnerabilities hit the blog. Having your entire Wordpress install owned by the root user helps, certainly.

Anyway, I’ll be pretty busy for the first part of November. Until I manage to get some of my devopsdays notes up, here’s some links to keep you going:

• The Ship’s Blog fires a shot across Maven’s bows. The Sonatype guys respond with all cannons blazing.
• There’s yet another Ruby CI server. Looks better than some of the alternatives.
• Erik visualizes build pain
• Damon Edwards is working on a Creative Commons licensed whitepaper on automated provisioning.

Related posts:

1. links for 2009-01-18 Sonatype
2. links for 2009-01-31 A Univer
3. links for 2008-12-24 collect

Related posts brought to you by Yet Another Related Posts Plugin.

I chose to try StAX rather than SAX because it was developed to be a middle ground between DOM and SAX, having the memory efficiency of a streaming parser like SAX whilst retaining a simpler API like DOM.

A quick note: This is not an exhaustive analysis of the pros and cons of the various XML APIs as this has been done before. Rather, this is a comparison of some of the things I made a mental note of whilst doing the conversion.

DOM

• Works with an in memory tree representation of the XML document, and therefore has high memory requirements for large documents
• Easy to use API that allows you to navigate the XML document in whatever way is most appropriate to. You can process a element as often as you like, go forwards and backwards as well as search through the document.

This is what our code looked like before the conversion.

Note that using dom is simply manipulating an in memory tree.

SAX

• Implementations manage the state in the form of instance variables. This works well for simple documents, but becomes more difficult to manage as the document gets more complex.
• Processing of an element typically occurs when you encounter the elements end tag, as only then is all of the elements content available. Until you reach an end tag, you have no real idea of how far through an element you are.
• You only need to respond to elements that are of interest. Ie, when you receive a callback for one of these elements, just do nothing and return.

StAX

• Implementations typically manage the state on the execution stack, with a new method call for each element that is encountered. This makes the code pretty easy to read as it is self documenting.
• You need to process each and every (unfiltered) tag and event in the xml document. This is rather low level, and without care can lead to confusion and complications.

And this is what the code looked like after the conversion:

My use of StAX is a little more regimented. I begin and end each method with an assertion that I am at the element I expect to be (this has the advantage of documenting the implementation). The rest of the implementation is similar to its DOM counterpart, expect that rather than simply asking for the elements I want to process, I need to loop over all the elements, skipping over those that are not of interest.

Summary

Overall, I am happy with the way the conversion has turned out. A couple of things were unexpected. The StAX API did not include any higher level utility functions that allowed you to move around at the element level, only between the end tag of one element and the start tag of the next. The other was that it required a fair amount of effort to write the code such that it was resilient to unexpected data in the reports. Every tag has to be processed after all.

“Unfortunately, there have been misconceptions and misinterpretations of the existing laws, policies and regulations that deal with software and apply to OSS, that have hampered effective DoD use and development of OSS.”

I wonder how long it will take for contracting officers to get the memorandum (so to speak) and to try to find ways around the guidance, spelled out in Attachment 2:

“In almost all cases, OSS meets the definition of “commercial computer software” and shall be given appropriate statutory preference in accordance with 10 USC 2377 (reference (b)) (see also FAR 2.101(b), 12.000, 12.101 (reference (c)); and DFARS 212.212, and 252.227-7014(a)(1) (reference (d))).”

We also eagerly await broader acceptance within the DoD information assurance community, but do not expect rapid change, despite the following guidance:

“DoD Instruction 8500.2, “Information Assurance (IA) Implementation,” (reference (g)) includes an Information Assurance Control, “DCPD-1 Public Domain Software Controls,” which limits the use of “binary or machine-executable public domain software or other software products with limited or no warranty,” on the grounds that these items are difficult or impossible to review, repair, or extend, given that the Government does not have access to the original source code and there is no owner who could make such repairs on behalf of the government. This control should not be interpreted as forbidding the use of OSS, as the source code is available for review, repair and extension by the government and its contractors.”

In sum, this is good news for those of us who believe that open source has a place in the Department of Defense. However, word spreads slowly in the Government, and I expect that this policy will take many months, if not years, before gaining significant traction in DoD contracting policy. A good place to start would be providing correct links in the policy memo. Clicking on the link currently leads to a 404 error.

Thanks to the Powdermonkey blog and Dr. Mark Drapeau for sharing this!

I flew up to Boston on Monday to lead some workshops on Continuous Integration. I checked into my room at the Hyatt and then went downstairs to see who was around. I ran into a couple of speakers milling about, and eventually joined one of them, and we headed over to the MIT Press bookstore, me to look for my Solr book. I wasn’t too sure of the name of the other speaker I was with (I asked once, but couldn’t quite remember what it was…). So we got to the book shop, I ask my fellow speaker again: James Bach. The name I was familiar with, but couldn’t quite place it… I ended up buying Parentonomics, and then we go for coffee.

So, over coffee, he asks me about what my topic is, and I gave him the brief summary of my two CI related workshops. Wow.. Little did I realize that I was sitting with the guy who rails against the “fetish” that Agile folks have for automated testing! That his entire approach to “testing” is to use skilled, motivated folks who do “sapient testing”. And I’m the guy who’s selling an approach that REQUIRES automated tests! That encourages expanding the use of automated testing!

He actually walked me through a process of talking about how to “think like a tester”, and it was really great mini-workshop.. He definitely subscribes to the socratic approach, and believes in his message, I was sweating at the end of it! That chat probably sparked more ideas in less time over that coffee then anything else this week. I also heard a lot of ideas and phrases that were echoed in Michael Bolton’s keynote later on in the week. Clearly a lot of collaboration between the two!

Probably the biggest idea that James and chatted about was the idea that automated tests really aren’t automated tests, they are automated checks. They verify that the expected behavior of the code was met. His argument that if you want to do testing, real testing, then computers, automated processes can’t meet that need, only people can.

Now, I don’t know if I believe that is completely true, but I am very aware that the “manual testing” where long test scripts written as Word documents are executed by human beings by hand are really a waste of human potential. And that those test scripts are really, to use James terms, “check scripts” because the people are not using any creativity! In fact, a lot of my interest in CI comes from the idea that people should not do monkey testing, that machines can do it much better, and my frustation with the perception that testing is a low value activity and can be easily shipped off to low skilled folks.

I think that this shift away from the term “test” for automated tests is actually happening in many places. In the Ruby world, we have libraries like Shoulda that are moving from using words like assert to other words like should. A Cucumber test really shows how controlled the space that a test needs to be to work well in an automated fashion:

Scenario: See all vendors
Given I am logged in as a user in the administrator role
And There are 3 vendors
When I go to the manage vendors page
Then I should see the first 3 vendor names

So while I don’t know if I am bought in on the idea that only people can do “testing”, and machines can only do “checking”. Tools like Heckle try to simulate aspects of what a human can do. While not suggesting that we can automate the “does my website look okay after someone changed the CSS” type of work today, in the future our automated testing will be more capable then just “checking” because we will move beyond the very constrained tests we have today to ones that mimicing the richness of the simulators that Airline Pilots use. Instead of testing the training given to pilots, we’ll be testing the robustness of software via simulations!

At any rate, James Bach, while taking a rather provocative approach to sharing his ideas, does subscribe to my favorite bullet in the Agile Manifesto: Individuals and interactions over processes and tools.

Here is him giving a great presentation with the subversive title of “How to Fake a Project” that was incredible entertaining, and also quite thought provoking:

What do you think? Automated testing is a fetish of the Agile community?