Mark Waite

Team Centered Bug Finding

2010-10-01T15:29:00.000-07:00

We've agreed to deliver a "release ready" version of our internal use components every three weeks. That is a nice step from the "deliver once a year" model we've used in the past, and we're trying to make the transition to the new release cycle successfully.

As part of that transition, we've returned to a practice we had before. We hold once a week test days for the entire development team. "Test Friday" is dedicated to testing.

Positives

Test day happened! It feels good to dedicate one day per week to whole team testing
Color printer sitting in the pairing area and pens for handwritten notes on the sheets was a great way to rapidly record bugs as they were detected, without too badly interrupting the flow of testing for entering bugs into a bug tracking database. It made a difference that the printer was within a few steps of the developer desks
Bug triage at the end of the testing day with the papers on tables in a large conference room seemed to work reasonably well
Bug pages with pictures were much quicker to process than bug pages which only contained words
Chance to use products "end to end" increases our exposure to customer experiences, helps us understand how the products feel, and ultimately will help us make better products
Native language speakers make localization testing much more effective (German and Russian native speakers in the team are a great help)
Having a "reference machine" with a previous version of the products was a good idea

Negatives

System setup too time consuming - some of us spent most of the test time configuring systems to meet the base requirements before we could install and test our own code
Builds had to be taken from a "temporary branch" because one of the other teams in the company had provided code which was not ready to install. Building on that temporary branch meant using a temporary machine, and the temporary machine was not configured the same as the standard build machines, so it did not sign the executables
Inexperienced with bug detection techniques to rapidly detect if there is a problem in "known hot spots". Log file analyzers and other bug detection oracles need to be added to the code and used in our investigations
The "reference machine" did not have enough configuration to use as a reference in the most important areas

64 bit Debian Testing on a low power desktop machine

2010-08-28T15:50:00.000-07:00

I've been experimenting with a cheap, low power, quiet computer running the 64 bit Debian testing distribution ("Squeeze"). The experience has included delights, several positives and some negatives.

Delights:

Fanless Intel Atom D510 processor and motherboard works great, even in my bedroom (quiet enough that I leave it on while sleeping)
Google Chrome beta is fast and impressively stable on Debian Testing. The Debian IceWeasel (rebranded Firefox) is acceptable, but Chrome makes web browsing pleasant and snappy, even on the Intel Atom D510 processor in my low power box
XMarks for Google Chrome beta works smoothly and well, copying my bookmarks between computers and operating systems seamlessly and easily
Hudson continuous integration server upgrades are trivial with the Debian repository provided by the Hudson team
Google desktop search has been interesting on Debian (works, behaves as I expected
Debian testing has been very usable

Positives:

No virus scanner performance penalty for my disc accesses
Frequent software updates keep the system running smoothly
Interesting new things to learn in an unfamiliar environment
Ample memory (4 GB), a dual core processor and hyperthreading keep the computer feeling responsive even when I run several different demanding tasks

Negatives:

My experimental code was not ready for 64 bit, and is still not ready for 64 bit, so it has taken some work to get the experimental code running on this new environment
Wireless networking has not been successful on this low end machine. Part of that is my own fault because I can only invest small amounts of time in the setup

My most sincere thanks to the Debian project team, the Google development team, the Hudson development team, and so many others. They've provided a free software stack that looks and feels smooth, interesting, and fun.

Prepared or Intentionally Unprepared for a Meeting?

2010-07-26T13:27:00.000-07:00

I dislike disorganized, poorly structured, or meandering meetings. Poor meetings treat the participants disrespectfully by wasting time and energy.

Yet, today I consciously chose to enter a meeting unprepared. I made that decision because the preparation for this specific meeting (a technical estimation meeting between a very few well aligned, frequently interacting, highly skilled people) would have used their time even less efficiently than performing that preparation during the meeting itself.

More typically I've felt the other way, that there were plenty of more effective ways I could have prepared for those meetings than spend the time in the meeting doing the preparation. This was a special (and rare) case.

I suspect there are "investment" heuristics which can be applied to the subject, the participants, the duration, my role, and the relative business value of a meeting. I envision the heuristic would provide a suggestion of how much time I should spend on the meeting based on those attributes. Unfortunately, google search did not show me that heuristic, or even a list of attributes of meetings which would help me decide how much to invest in preparation.

Windows Home Server - Shock & Surprise & a happy ending

2010-05-05T07:11:00.000-07:00

My home computers connect to an HP MediaSmart Server LX-195. That little server performs nightly backup, uploads files to Amazon S3 for backup, and has generally been a delightful little box (after overcoming the initial installation hurdles).

A week or so ago, the client computers were displaying a terrifying message that there were conflicts on the server, and backup was offline. When I examined the console, it showed a new disc added and listed the same exact disc as "removed" and "unhealthy". The disc it was say had a problem was a new 2 TB Western Digital USB drive.

I was confident the drive was not dead, but something had caused Windows Home Server to decide the drive was dead, then had come back to life as something different.

I rebooted the server. No change in behavior. I experimented with other small (non-damaging) tests. No change in behavior. I left it alone and assumed I would need another hour or two on the phone to resolve the problem.

One day I had the ingenious idea to shutdown both the server and the USB hard disc. I then switched on the USB hard disc and after it had some time to fully awaken, I switched on the Windows Home Server machine. SUCCESS!

I have no idea why the system decided the hard drive was "healthy" and "ready to add" in one list, and later in the same panel should be "removed". If I'd followed those steps (add and remove), I suspect I would have lost all the data on that hard disc.

It is hard work creating user interfaces which don't risk user data, and don't perplex the user when exceptional conditions occur.

Delighted by Seamless Operation

2010-04-17T04:54:00.000-07:00

I needed a particular software utility (a hard disk partition archiving program). I was in the middle of some other work on my machine running the Debian "testing" distribution. I didn't want to interrupt my other work, but I had a few minutes to download and burn the needed software to a bootable CD.

I hadn't used Debian to burn a CD before, so I worried was going to demand I use various command line arguments with the usual reading of manuals. I was wrong.

I found the ISO image (thanks google), downloaded it with the web browser, clicked "open" from the web browser, and it offered to open it in "brasero". I inserted a CD-R in the CD drive, and brasero happily burned that CD with no complaints.

Thanks to the open source community! That was a great experience, simple, elegant, and did what I expected.

Conversational Productivity Improvement

2010-03-13T05:41:00.000-08:00

While sitting with a colleague at work, I was complaining that my favorite bookmark management tool (GMarks) was no longer working reliably with Firefox 3.6. He had an improvement suggestion, that I might try the same bookmark sync method he is using. I've learned that when my colleagues have found a better way to do things, I will usually benefit by trying the same technique. This was one of those cases

A Bit of History

I started managing and synchronizing my bookmarks with the Google toolbar a while ago, then migrated to using a Google notebook during the short life of that tool, and from there had moved to using GMarks because it could read and write the bookmark definitions I had on Google notebook. My bookmarks have had a long and varied life as they traveled from place to place...

He suggested that I might try "Xmarks", since that was the bookmark sync solution he'd been using and preferred for a while. I'm a "slow study", but decided I'd try it as an experiment.

XMarks installed easily into Firefox and accepted my manually entered bookmark just fine. I didn't see any obvious options to sync my GMarks bookmarks, but was willing to experiment without that feature initially. It seemed to work smoothly and well.

Chrome and Internet Explorer

I prefer to switch between browsers frequently in hopes that will lead me to discover bugs in our software more quickly. I move between IE, Firefox, and Chrome frequently. I installed the XMarks add-on to Chrome and it offered to sync my Chrome bookmarks to XMarks. That was a great offer, since Chrome had pulled all my bookmarks from the Google bookmark location where I had stored them. I now had all my bookmarks in XMarks, without entering them myself. Thanks Chrome!

The Internet Explorer XMarks add-on was just as simple to install, and just as well behaved. Now my bookmarks are the same on any browser I use, or at least any browser I use where I can control the browser add-ons installed.

HP MediaSmart LX 195 - Victory At Last

2010-03-13T05:25:00.000-08:00

Two months ago I purchased a new HP MediaSmart Server LX195. It is a low power computer and disc drive in a small box, with Windows Home Server, an operating system intended to help with some of the common home computing chores, like:

Automatic backup (both onsite and offsite)
Photo upload
Media storage
Media streaming

The initial purchase price was slightly over $200 for a unit with a 640 GB hard disc. That was close enough to the purchase price for a comparable sized hard disc that I was willing to experiment with the system just for the value of experimentation (and hope for easier backup).

Miserable Initial Experience

The initial installation experience was poor. The computer has no display adapter so it must boot, connect to the home network, and be reachable from within that home network without any interaction with the user. The total user interaction is through the power switch (on or off) and three front panel LED's.

I was fascinated by the challenges hiding in making Microsoft Windows boot and configure itself on a network without user interaction. There are so many ways to configure a network, so many conditions which could fail, so many potential problems, and a general assumption among the Windows development teams that a Windows server always has a display.

The first time I switched on the power, the lights went red (disconcerting), then eventually glowed a yellow color. That was not a good sign. I pressed the power button to stop the machine and nothing seemed to happen. I pressed and held the power button for 5 seconds and the power went off (learning to control the box...)

The second time I switched on the power was similar to the first. I was short of time, so I left the box on for a period and did other things. It stayed in the same condition. I didn't have time to fight with the box, so I sent a message to the seller asking for instructions to return the device.

The seller was very considerate and offered to accept the return (impressive by itself, since this appeared to be a small shop selling through amazon.com). The "I'll accept the return" message also included a suggestion that I contact HP.

A week or so later, I had time again, so I went searching for the 195LX Windows Home Server support team on the HP web site. After some digging, I found a phone number (800-474-6836). I had initially been unwilling to spend any technical support time with the device, but decided I'd "try the experiment".

Considerate Technical Support

I assumed the technical support would be provided by someone lacking native language skills, and the support call would be a deep struggle. I was wrong. The technical support call was handled by a well spoken, considerate individual with clear communication skills, a cheery attitude, and no detectable accent in their speech. I appreciated that!

I didn't have much time that day, and we quickly exhausted that time before the problem was solved. However, the experience was so positive (compared to my expectations), that I decided I might try again another day.

Home Networking Complications

The second technical support call was just as pleasant an experience as the first call, and was much more successful at solving the root problem. The technical support person asked some questions about my home network configuration, and I offered some things I had observed. With the "back and forth" of that conversation, I think I persuaded him that I knew enough about networking to be interesting, and he persuaded me that he knew enough about the typical behaviors of the box that I should trust him.

The tech support person had me insert the IP address of the 195LX into the hosts file of the Windows computer I was using as the client and display for the little server. Apparently, Windows Server 2003 and Windows client operating systems did not like my attempt to use purely the IP address to configure the computer. I don't have a DNS server at home, I don't have a WINS server at home, and at the time I was not running my own DHCP server either, choosing to use the DHCP server provided by my Linksys wireless router.

Inserting the IP address was the "magic key". The computer was now reachable, and ready for configuration.

Success At Last

Once that initial installation hurdle was overcome, the rest of the experience was great. I'm a software tester type, and I expect software to disappoint me. That's the nature of software testing. In this case, rather than disappoint me, the configuration experience delighted me. It has been a long time since I felt delight like that in working with software. Thanks Microsoft!

I configured the server to backup the computer I used for initial installation. After installing a little client software and entering a password, the backup hid itself and ran very nicely. I moved to the other 3 computers in the house and did the same thing, with the same easy experience. That was already impressive, since those computers have operating systems of widely varying ages. Two older Windows XP machines, a Windows Vista x64, and a Windows 7 x64 machine were all easily added to the backup process, and safely copied their files to the server.

The server also has places for shared folders. One of my concerns has been losing our digital photo collection, so I copied files from the client computers to the digital photos folder on the server. That was easy, and well behaved. There was also a setting to allow the server to search for photos on its own (thanks!).

The same for music, although the music is a more replaceable item, since we should have the CD's for all that music anyway.

Offsite Backup

A week or two after that positive experience, I remembered that I had seen mention of being able to use the MediaSmart to send backup data "offsite" for safety. I found the configuration panel easily, and answered a few prompts for the offsite backup process. It configured an Amazon simple storage service instance for me and started pushing my data to that offsite backup location.

Several days later, I checked the status and it was still pushing my data to Amazon.

I was curious why it was taking so long, so I checked the hard disc capacity of the server. I had already consumed about 500 MB of the 640 MB disc. Most of that was backup. Apparently my computers contained more data than I realized.

The backup process eventually completed successfully, and has been running happily since then.

Searching for a Lost File

In the midst of all this fun with the server, my son lost a homework assignment. He had forgotten to save it from within Word, and was desperately trying to find any remnants of the assignment. I opened the restore wizard and was able to comfortably navigate around the files in the backup device. Unfortunately, the file was not there, but my confidence was increased that the backup might actually be usable, not just a "write only" device like most backups I've had in the past.

Adding Storage

Since the home computers were needing more backup space than I expected, I could see that I would exhaust the disc space on the server in a week or two. That was a nice opportunity to expand the disc storage. I purchased a 2 TB USB disc drive and connected it to one of the 4 USB ports on the back of the server. The server recognized the drive, and offered me choices in how I'd like to use the drive. I used it to add additional file system space to the server, and then configured the existing folders on the server to "mirror" themselves.

The "mirror" concept seems to be a "poor man's RAID", an operating system provided option that will periodically copy the contents of selected folder from its original drive to another drive on the system.

That worked well also.

Changing Passwords

One of the low points in my life is the requirement from work that I must change my password every three months. Changing my password has been a hassle on a number of computers, and I worried this would be a similar experience. I was wrong. It was an easy process. I changed my password on one of the client computers, and the Windows Home Server icon prompted a short time later that it is best if the client and server passwords match. It let me make the change, with options to force the client and server passwords to be the same (nice touch!)

Impressed

I'm thoroughly impressed with the device. The initial install experience was "rocky", but the experience with Windows Home Server has been delightful ever since that initial rough spot.

As a caveat to software creators, the initial install experience is high risk that you will lose the customer before they've even seen your product's features.

Stone Age Productivity Improvement

2010-03-08T16:50:00.000-08:00

Sometimes we spend more effort on presentation than on content, even when the real value is in the content, not in the presentation.

Some examples:

Why spend time creating a PowerPoint slide deck for an informal discussion? Why not scribble notes on a page, scan the page to PDF, then annotate the hand-written PDF notes during the meeting?
Why spend time transcribing a white board to a computer when a digital photo of the white board captures the arcs, arrows, angles, and phrases on the board, without wasting the effort to form those into a digital document?
Why enter requirements into a computer system when the team is local? Use scraps of paper (3x5 cards work for me) and move them around on a table to prioritize and understand them
Why not perform bug triage by rapidly moving pieces of paper around on a table instead of forcing everyone in the room to focus their attention on a screen where one and only one person can make progress?
Why not have a face to face discussion or a phone call instead of composing an e-mail message?
Why not talk about an issue instead of spending time to document it?

I listened to a programmer this morning noting that he was unable to find documentation on something. The thought that kept rolling around in my head while he said that was, "what you need documented does not exist, and the waste of creating documentation for all the things in order to not be missing that one thing was just not worth it".

Expressing My Priorities

2010-02-13T05:07:00.000-08:00

We've had a set of tests failing on our continuous integration server for a week or more. That's a bad thing, since we tend to ignore continuous integration jobs which are not clean almost all the time. Continuous integration failures need to be infrequent and interspersed with extended periods of "clean", or at least need to be viewed as a trustworthy development information source. Otherwise, the team ignores the continuous integration server and it becomes another wasted and ignored feedback system.

The same tests work fine on the developer machine where we've run them (in a slightly different configuration). The "builder" attempts to diagnose the root cause of the failures have been unsuccessful.

I've scheduled a Monday brainstorming session with a larger group of people. We'll discuss what we know about the failures, what we know about matching successes, and then decide on a course of action.

A colleague asked why I was the one convening the meeting, instead of someone in the affected team. His logic was that several different people on the development team should have been intensely interesting in the root cause of the failure, and have "run it to ground".

I think there are several reasons why individual developers are not chasing this problem:

The problem crosses boundaries (only visible from an installed version, only visible when other components installed, only visible from the continuous integration machine, etc.)
There is no clear correlation between the first failure and a commit from a developer
Installation related problems are more difficult and painful to diagnose. Setup and teardown time for the test is signficantly more than the setup and teardown time for most of our other automated tests
Diagnosing this failure will reduce the energy available to work on other things, like new backlog items and new tests

I'm expressing my priorities by scheduling the meeting and bringing a group of people together to work on the problem. Since I'm a manager, my priorities have a little more weight, and I'm "throwing that weight around" a little for this case.

Eventually I'm confident others in the organization will recognize my focus on keeping the continuous integration servers "clean". Until then, I'll continue working with people to persuade them.

Microsoft Mesh - Synchronizing Files Between Computers

2010-01-24T08:35:00.000-08:00

I work with many different computers between home and work and have found that Microsoft Live Mesh has been a nice addition to my computers. Mesh allows me to specify one or more directories which should be synchronized between all the various computers I'm using.

Thus far, I've used it to allow me to write performance reviews on any of my typical computers, knowing that the written results will be synchronized with my other computers so I can continue writing elsewhere.

I'm now attempting to use it to synchronize my "3x5" cards electronically. I usually carry a stack of 3x5 paper cards in my pocket for idea and task capture. When those cards need to live a little longer than they would normally sit in my pocket, I'm writing them to a file (in emacs "org mode") which let's me shuffle, stack and move them almost as easily as I can move them on a table. With that file synchronized between my various computers, I hope to lose fewer cards, and have a more clear idea of priorities at any computer I use.

Google Alerts - Monitoring Your Online Reputation

2009-12-22T05:36:00.000-08:00

Google alerts delivers periodic search results to my e-mail box. I've been impressed at how it helps me detect new bloggers that might be worth reading, and helps me manage my own reputation.

I have several topical search terms that I use to watch for interesting new authors and sites. I currently have alerts set for "Association for Software Testing", "exploratory testing" and Debian Linux testing. I'll eventually add search terms for SharePoint 2010 and evidence based management.

I also have search terms which search for specific authors I've found interesting in the past. The search results on those authors have lead to other interesting writing on topics I follow.

I even search for my own name. I realize that is more than just a little vain, but it provides one way to monitor how others might perceive me. I use an exclusion term or two to avoid the writing of the journalist who shares my name and writes at a newspaper in Nevada. Someday I may add an exclusion term for the person who shares my name in Georgia.

Simplifying is Hard Work

2009-12-09T04:57:00.001-08:00

One of our QA managers asked a very simple question yesterday that took me through a series of twists and turns trying to find a reasonable answer. It reminded me that seeking and finding solutions to problems is hard work, and finding a workable solution can be exhilarating.

The question was "How can a tester know if a specific fix is included in this build?"

The question hides contextual information like "source code is stored in git", "bug reports are stored in a customized Siebel database", "Siebel interface or API changes are not feasible", "testers are typically a long distance (geographically) from developers", "builds are accessible from a web page of hyperlinks" and "the recent builds page also has links to gitweb".

Challenges hidden in the question:

No API connection between git and Siebel means we need to rely on people as the carriers of information, the "information transport"
Time and space separation between developers and testers complicates communication, yet complicated forms or processes are more likely to be bypassed than simple ones
Complicated user interfaces are more expensive to develop than simple ones, and complicated user interfaces tend to get less interest from users
Persuading people to enter accurate data into forms, fields, or even free form text is difficult. It can be made much easier if they see immediate benefit from that entry, if the format is simple, and if the results of that data entry help them

My proposal to my colleagues (after an embarrassing amount of thought) was:

When a developer fixes a bug, they paste the SHA1 hash of the fix into the bug report. The SHA1 hash is a unique identifier of that commit in git, so recording it in the bug report provides a "link" between the bug reporting system and git
When a build runs, it records the short form of the "git log" for that build (the list of checkins which were in that build) with the identifiers (SHA1 hashes) of each checkin and the first line of the checkin comment
When a tester verifies a bug, they first verify that the log file which came with the build contains the SHA1 hash from the bug report. If it does, then they perform the verification. If it doesn't, they don't waste time performing that verification

During the thought process before arriving at that idea, I wandered several different blind alleys, thinking of one complicated solution or another. For example, the early wrong or overly complicated ideas and questions included:

Why do they need this data, don't the testers "trust" developers? The question is not about trust, it is about communicating intent, and preventing the wasted time associated with testing a specific fix, reporting it is not fixed, then being told "Oh, it wasn't in that build"
How can we ask git to tell us which changes are in a build when there is no API connection between the build and the source repository? I envisioned complicated sequences trying to map a build date or time or version number or branch back to the git repository to generate the list of changes
How do we connect the build (performed at some independent time) with the bug report (submitted at a different time on a different system) and the source master (yet another time and system)? Programs, links, pages, all came to mind as ways to make that connection, before I realized that the unique identifier of the commit is one of the few data items which might travel reliably between all those systems

Then again, looking at the idea now in the "cold light of a new day", maybe I was just too tired to think clearly.

Small Barriers == No Socialization

2009-11-28T12:32:00.000-08:00

I was again reminded today that very small barriers can kill a conversation, or cause a contributor to stop contributing. I wanted to post a follow-up comment to David Chadwick's exploratory testing blog posting. I was on a different computer than I originally used to post the first comments.

Unfortunately, the IBM developerworks site is unfamiliar enough that I don't remember my password. I remember that it had a more restrictive password policy than other sites, so I was unable to apply my common password rules to their site. That made the site "unique" in a negative sense, something I had to remember outside my normal patterns of memorization.

The key point: I had a few minutes to contribute something to a conversation. I was not willing to spend more than a minute or two, including the time to login, post the comment, and verify the comment was posted. I abandoned the comment because:

Restrictive password rules were outside my typical rule set
Time was limited, I was unwilling to spend more time trying to login again
Prior experience with the site made me distrust its use of my time

The relationship between a contributor and the forum to which they are contributing seems to be very fragile, at least for me. The newer I am to a forum, the more fragile the relationship. The less value I perceive from the forum, the more fragile the relationship. Even seemingly insignificant hurdles may be enough to stop a contribution and disengage a contributor.

Do Not Discard My Data

2009-11-26T08:09:00.000-08:00

I just wasted 40 minutes trying to post a comment to an IBM developerworks blog posting from David Chadwick.

That wasted 40 minutes was a flawed attempt to describe the simple changes I think could be made to David's list of things which he believes exploratory testing is "not". I believe with relatively simple wording changes, I could modify each of his "not" descriptions to instead be descriptions of high value, useful exploratory tests. The wording changes are so small that I believe they hint David may not yet understand exploratory testing and how to apply it. However, this posting is not about exploratory testing, it is about a site that lost my data, twice.

My first comment was lost after 20 minutes of writing, thinking, and editing. The comment was lost because I clicked the "Add Comment" link, entered my comment, then pressed the "submit" button. The page which was returned politely informed me that my comment was rejected, and listed several possible reasons for the rejection. Unfortunately, IT DISCARDED MY DATA.

Don't discard my data! It frustrates me as a user and makes me unwilling to return to the site. If I must be logged in to submit a comment, force the login before accepting my input.

I registered on developerworks, logged in, and clicked the "add comment" link again. I added a short, dummy comment to assure that I was now able to add comments. I was able to add the comment.

I wrote a new response (I assume it was a little better than the first comment, since second drafts are commonly better than first drafts). After about 20 minutes of working on that response, I clicked the "submit" link. My comment was rejected again, with the same list of possible reasons for the rejection. Unfortunately, IT DISCARDED MY DATA AGAIN.

Don't discard my data! It makes me feel stupid, and then I need to remind myself that I'm not stupid, the software which should be working for me is instead making me do the work.

I assume the second failure was either due to my inserting URL's into the text, or due to the length of the text I was trying to post. I don't know which it is, and I'm frustrated enough with the developerworks site to not care which it is.

I gave up on trying to post a useful comment. I left a short note that my comments had been rejected twice, and if the author wanted my comments, he would need to send me e-mail.

Systematic Source Formatting

2009-11-12T05:47:00.001-08:00

When my team switched to Extreme Programming in March 2003 (XP by 3/03), we decided to try all the practices and all the recommendations. There were plenty of bumps and bruises as we learned what worked for us and what didn't work for us. We learned that the original XP descriptions were too light on testing for our business needs, and we learned that integration tests worked better for us than pure unit tests with mock objects. Those can be topics for another time.

One of the surprisingly effective results of that adoption of all the practices and all the recommendations was using automated source code formatting to assure all our code looked the same. We had a diverse team of people developing the code, some in the U.S., some in Germany. That diverse team of people had different opinions and attitudes about the "one true way" to format source code. Those different attitudes and opinions lead to annoying little changes from one file version to another as the files changed hands from one person to another.

Since XP espouses "no code ownership" and allows anyone to be anywhere in the code, those formatting differences tended to worsen as people moved through the source tree.

We took a "top down edict" approach to implementing "no code ownership". I decided (as the manager) that we would use a program to format all our Java source code as part of each compile. The concept was that developers would no longer need to think about formatting their code, a program would do it for them. We inserted the open source version of the "jalopy" source code formatter into our standard build process and then added an additional step to systematically format the code once a day with the same process, in case someone forgot to format before they committed changes to the master.

The initial adoption of the change caused some angst, since the chosen format (Sun Java code formatting guidelines) disagreed with the "one true way" which some of the team expected. We pushed our way through that initial hurdle and found over the course of years that automated source formatting freed us from a number of problems.

Positive results of our switch to automated source code formatting:

Code looks the same no matter who wrote it
Changes in the source master are "real changes", not white space shuffling
Developers don't need to think about source code format, they can insert code however they want and the formatter will fix it to meet the standard

Negative results of our switch to automated source code formatting:

A "diff wall" was created at the point where we first ran the automated source code formatting tool. The automated formatting created a single massive change to the source master to convert from the old ad-hoc format to the new consistent format. I worried at the time that the "diff wall" was going to be a real problem, but it never appeared as a problem in our work
Command and control style management is frequently a bad pattern and should be reserved for rare occasions (our major process change from waterfall to Extreme Programming was done as a manager dictate from above, and this was "hidden" in that process change)

There are newer tools to support automatic source code formatting (astyle seems to be popular) and I hope to persuade my new organization to adopt automated source code formatting. I'm not in a position to be the dictator in the new organization, so changes like that are more difficult to create.

Deliberate Practice - Mary Poppendieck Talk

2009-11-08T06:03:00.000-08:00

As a manager at PTC, I need to write a performance appraisal for each of my direct reports. I will also receive a performance appraisal from my manager to help me identify strengths and weaknesses and set goals for the coming year. While preparing my own self-evaluation, I remembered reading about the concept of "deliberate practice". I'd already been trying to apply my understanding of the idea, but decided to listen to an original source before including more about it in my self appraisal.

Mary Poppendieck's Agile 2009 deliberate practice talk describes deliberate practice as practice which is intentionally focused on improving performance. I want to improve my performance. Mary highlights four key components that are required for a person to be using deliberate practice:

Mentor - a high skills expert to review, critique, and highlight flaws

Challenge - tasks that require greater skill than we currently possess

Feedback - review and analysis of results used to improve future attempts

Dedication - hard work, time and energy applied diligently

I don't have a mentor for the things I'm trying to improve! At least, I haven't identified someone as my "coach" and had them agree to review, critique, and highlight flaws in my performance. Flaw number 1 (and the most glaring flaw) in my recent improvement attempts.

Many of my tasks and assignments are the same assignments I've had before, coordinating, planning, discussing, and interacting with others. However, I always find those tasks challenging because I don't feel like a naturally social person. Negotiating, discussing, persuading, and debating do not come naturally to me. There is room to improve in the "challenge" area, although I don't see it as making as much an improvement as the absence of a coach in my improvement efforts.

Most of my activities have feedback built into the activity in one form or another. Meetings which start on time, stay on task, complete their objectives, and end on time leave me feeling refreshed, invigorated, and useful. Meetings which start late, wander aimlessly, don't have objectives, or extend beyond their scheduled end leave me frustrated and edgy. That's a form of feedback. However, there are other feedback forms (like team retrospectives) which I've not been using faithfully lately, and need to start using again.

Dedication may actually need some "negative attention", since lately I've been spending too many hours at work and not enough hours with my wife and children. Mary's talk notes that expert performers in other fields (like music) have discovered that they cannot apply more than 3 hours of deliberate practice at a time because it is too tiring. They stop, change tasks, take naps, or otherwise refresh themselves rather than continuing, and risking developing bad habits by practicing poorly.

With the identified gaps in my efforts to improve, now it is time to choose a mentor and start hearing the performance critiques, then acting on them. Finding a mentor seems like a difficult task for a manager. I want a mentor that is

Regularly and naturally exposed to my performances, without requiring that I present them a summary of what I did. Swimming coaches do not ask the swimmer to describe the most recent swim, they watch the swimmer and then tell the swimmer directly and openly what could have been improved in that swim. My mentor needs to be someone who "watches my performances" on a regular basis as part of their normal work

Able to spend time and energy critiquing my performances with focus on improving them

Credible as an expert. My request for coaching is an act of trust and that extension of trust requires that I believe in the skills of the person providing the coaching. I may disagree with their perspectives, challenge their ideas, and still want their coaching

Interested in my success. Without interest in my success, I doubt the coach can be trusted to provide excellent feedback

I'm sure there are other things I need in a mentor as well, but that list already worries me. Those who are regularly involved in my work tend to be my direct reports, my peers, and my manager. My direct reports aren't very interested in coaching their manager (other than possibly to smile with him about his many failings). My manager is already a mentor by being my manager. My peers and I frequently disagree on methods and techniques and so I'm not sure peers are the best source of mentors either.

All this musing might also be less useful if instead of a mentor, I need to become a "buccaneer scholar" as suggested by James Bach. James suggests that I should take responsibility for my own education and for my own learning. That may make seeking a single perfect mentor a waste of effort, rather I could accept that there are mentors all around me and those mentors can provide useful information at times, and information to be ignored at other times.

Another alternative is to consider Bob Sutton's questioning of the value of annual performance appraisals. So many ideas to consider, so many things to learn (the act of writing this has already taken me on a different path than I expected when I began...).

Was Our Switch to Git a Mistake?

2009-11-05T04:42:00.000-08:00

My team is part of a larger, multi-team, multi-site organization in our company. The corporate choice for software configuration management is ClearCase. Unfortunately, my team is "remote", and "small". ClearCase does not handle remote teams well. I could find more polite ways to say it, but that is the simple result of our experiment. ClearCase was costing us far too much time due to its poor performance over a wide area network.

We solved the ClearCase problem by creating a bidirectional bridge between ClearCase and Subversion. My team's interface to the source master became Subversion. We saw great performance improvements, and were seeing the ClearCase updates within a few minutes of their arrival on the ClearCase master. The bridge was conceptually very simple, and it worked very, very well for our needs. Life was good.

We were then faced with a new, even more challenging problem. The new development work needed to be spread even more widely than the previous development work, with a larger number of teams involved and more dispersed geography. The new work would create multiple products, each with their own release cycle and their own development lifetime.

At the time we were making this transition, I'd been experimenting with the git version control system. Git is version control software created by Linus Torvalds when he needed to switch from BitKeeper for Linux kernel development. It runs very, very well on Linux. It is significantly faster than Subversion, which (in our environment) is significantly faster than ClearCase. Performance looked like a big winner.

Another challenge of the new environment was branch related. The new teams thought their development model would likely be "branch intensive". My prior experiences with branches have been with CVS and Perforce, where branches are globally visible and merging things between branches is a hassle. I hate branches. However, considering that the new world would be "branch intensive" and Subversion is generally not perceived favorably for branch management, we didn't want to use Subversion in a "branchy" environment.

With those two needs, distributed teams and branch intensive environment, we skipped Subversion and went looking. My recent git experience (and recent Mercurial experience) biased me in favor of a distributed version control system. A key opinion leader in the company had also been using git in a subteam of a very large project, pushing their results back to ClearCase. They reported positive results. My experience had also been positive while I was experimenting with taking a work project off on a "tangent". Git worked well for me, sitting on my underpowered Linux box doing my personal "skunk works" project.

We chose git as the team source control system.

Unfortunately, I had failed to detect my own biases, and the biases of the other early adopters of git. Those biases were very different than the biases of my co-workers.

I'm a command line fan. I'm old enough that my first high school experience programming computers was with the newly installed terminals to the school district mainframe (thanks Davis School District and Layton High for spending the money, the time, and the pain to install those machines!), then I moved to a University that required I submit programs on punch cards (makes me sound old). Before I left the University, they had upgraded to dumb terminals communicating with a DEC minicomputer.

As a command line fan, I found the git "user interface" perfectly comfortable and very similar to CVS, Subversion, and Perforce. There were a few surprises while I tried to understand distributed version control, but those surprises were related to version control concepts, not the specifics of git.

Unfortunately, many in my team and in other teams are not command line fans. They are accustomed to productivity accelerators like graphical user interfaces, integrated development environments, and mouse clicking to perform work much faster. The transition to git has been painful for them. In addition to my transition experience (centralized vs. distributed, new commands, new concepts), they've also had to deal with transitions from robust GUI tools (TortoiseSVN, Perforce Windows client, etc.) to weak and brittle GUI tools (GitSVN, gitk, git gui, etc.).

The challenge has been made worse by our decision as a management team to isolate teams on branches. Two of the managers in the team come from a large scale development organization (5-10x larger than our current organization) and they are accustomed to requiring branches as a way to isolate one team from the potentially breaking changes made by another team. The price of that branch isolation is that we now are required to perform more frequent merges of work, with the resulting complexity and frustrations which come from merging with conflicts. It gets worse when the files to be merged are coming from the Visual Studio IDE, and the meaning of the contents of the files is not always clear.

I think the branch configuration decision has done more damage than the choice of git, but that is probably biased (again) by my command line centric mindset. Unfortunately, we're far enough into the project that we aren't willing to switch SCM systems. We'll remain with git for at least the duration of this project, glad to have a source master, glad to have it connected to our continuous integration servers, and glad to not have the awful performance of remote ClearCase.

In all fairness to git, I still remember the growing pains when we switched from CVS to Perforce. I whined mightily at paying hundreds of dollars per developer for our corporate standard SCM system. Then I whined mightily at the tool changes and use model changes forced upon us by Perforce's way of thinking. After 6 months or a year, I discovered that I had changed my way of thinking, and was now very comfortable using Perforce, getting value from its way of branching, and being very grateful that it was so fast.

Maybe 6-12 months from now I'll say the same things about git. Maybe it is a part of "climbing the learning curve", and unfair to judge our experience this early. Or maybe not...

I still don't know what we should have chosen instead of git, since it is not clear to me that there were any better alternatives for my team at that time. The company was not willing to purchase another SCM system, since they were already paying for ClearCase. That excluded all the purchased SCM systems (Perforce, Microsoft Team System, Accurev, BitKeeper, etc.). The teams were known to be widely distributed, so that pushed us towards distributed SCM. The benchmark comparisons suggested that Git was faster than Mercurial in many operations, and the Bazaar people were still not settled on their final "on disc" format. Subversion was not well perceived for handling "branchy" development, and CVS was worse than Subversion.

The Linux kernel handles massive amounts of change (averaging 2-4 changes per hour continuously for the last 4 years) from many, many developers. It scales well for that widely distributed, branch intensive team, yet we're struggling with it. Of course, Linux kernel developers are even more likely to be command line biased than I am, and scaling the tool is not the issue that is getting in our way, it is more our choice to be "branchy" and the user interface weaknesses in git.

So many things to learn, so little time...

Dangerous Interruptions

2009-10-29T03:23:00.001-07:00

Bob Sutton's blog post on reducing medical interruptions reminds me of Sunday mornings when I take my mother-in-law from her nursing home to church. I frequently interrupt or disrupt the nursing staff with my out of sequence, unpredictable arrival, and with my desire to get "Nana" to church on time.

Nana's trip from the nursing home to church starts from her room at the nursing home. I arrive at her room between 10:20 AM and 10:40 AM (depending on how late I arrive from home). Sunday is the only day we do this, so I tend to disrupt all sorts of people at the nursing home, including the nursing staff.

My mother-in-law is a diabetic, and church runs over the noon hour when she would normally receive her medications. The nurse would normally check her blood sugars right before lunch, then based on the results of that blood sugar test, she would select the proper dose of insulin, draw that dose, and administer the dose to Nana. Each of those steps has a potential for error, and each of those steps needs careful thought and attention to detail by the nurse.

Because I arrive as much as 90 minutes prior to lunch, and Nana will be gone for the three hours of church, the nurse is required to interrupt her current medication process, test Nana, medicate Nana, and then return to her previous task. The nurses are always very kind about handling the interruption, and they provide great care. I worry that my interruption may cause them to make unnecessary mistakes...

The study which Bob Sutton references was performed in the UCSF hospital system in San Francisco and is described in a San Francisco Chronicle article. The study was an attempt to reduce the frequency of medication errors at hospitals. They used both low-tech solutions and high tech solutions to reduce medication errors by nurses.

The low tech solutions described in the article focused on reducing nursing interruptions when administering medication. The article describes "do not interrupt" sashes and vests, closing blinds to prevent distractions, and other relatively simple techniques to reduce interruptions during the crucial activity of administering medication. The article noted that the nursing teams were encouraged to develop their own solutions, within their own working environment (own your process). It appears from the study that nurses administer medications (a detailed technical task) less accurately when they are interrupted than when they are undisturbed. It also appears that nurses allowed to explore improvement techniques tend to improve.

The study may not directly apply to my software development team, but I think there are several lessons I should take from the study. They are lessons others have noted, but the article serves as a good reminder.

Interrupting technical work (pair programming, software design, software testing, etc.) increases the chances for error. I need to interrupt my people less
Allowing and encouraging people to improve their own processes, their own ways of working is likely to generate improvement. I need to find ways to acknowledge my mistakes openly, learn from those mistakes, and encourage others to do the same. A software bug is a late manifestation of a mistake, mistakes will happen in human endeavors, and we want to learn from those mistakes, not hide them until later
Fear of failure tends to hide those failures, particularly in organizations with a culture of fear. Sutton's posting notes that hospitals which acknowledge and seek to reduce their drug administration errors tend to report 10x more drug administration errors than units with a more punitive attitude towards errors. The failures will still occur, but they will be discovered later, and likely be discovered with more damage done, or higher costs incurred from the failure. It would be a gross mistake to declare the nursing unit which reports 10x the drug administration errors as a failed unit without further investigation. If the clinical results of the unit are better (fewer deaths, fewer injuries, lower costs, etc.), then the larger number is actually highlighting their good practice of learning, rather than the bad practice of medication errors. Don't worry about bug counts, worry about what bugs can tell us about how to be better
"Best practices" at one location were not necessarily "best" at another location, although sharing practice based experiences seems like a good way to learn from mistakes and thus make fewer mistakes in the future.

Learning and Responding

2009-10-23T20:59:00.000-07:00

A mistake was made today. Code was merged from one branch to another branch and the destination branch was broken in its intended use. The break was detected late in the day of the team that caused the break, and they had already mostly left for the day. The break highlighted all sorts of weaknesses in how I was handling things, including things like

Why didn't I make it clear to everyone both the purpose and the target configuration of each branch? Poor communication I had not made it clear what the purposes and expected configuration was for each branch, and they assumed that since they could see the branch building in one case, that was sufficient
Why didn't the person who performed the merge detect the broken build on the continuous integration server? Unclear information sources. We had configured 3 different continuous integration servers because we needed three different configurations. Unfortunately, I then "muddied the waters" by having one of the branches made compatible with all three configurations, and actively visible on all three configurations. When the developer performed the merge, they saw that it was "green" on the screen they were watching, and thought they were done. It had gone "red" on the other two servers, and those two were the most important to my team
Why wasn't the team which performed the harmful merge able to repair their damage? Unavailable spare configuration They had no machine available to them which matched the problem configurations and could be used for diagnosis and development. Their machines were all configured for their needs, and the break was in an area needed by other teams
Why did it take half a day to recover from the damage? Inexperience with our tools We recently switched from Perforce to Subversion to Git and the transition has left us less skilled in dealing with the complexity of this type of failure.

All told, the damage cost my team less than a day to recover, and because we're using a distributed version control system, they were able to continue their work locally, but they were not able to push to the central repository.

Moral of the story: Communicate clearly, listen carefully, and be willing to change as better ideas arrive

Ask More Questions - Get More Answers

2009-10-10T14:33:00.000-07:00

I had been tolerating a nagging problem with my web browser on multiple machines and now have a solution, because I had the presence of mind to finally ask a question.

Firefox is my preferred web browser because it includes the Gmarks plugin. The Gmarks plugin brings my google bookmarks into the Firefox menu. That makes portable bookmark management easier (all my bookmarks are stored at Google, visible from any web browser on the internet).

I also prefer the Foxit PDF reader instead of the Adobe reader. It feels faster, cleaner, and seems less likely to be attacked by malware (smaller installed base, newer code base).

Unfortunately, Firefox would report "OCX failed to load" when I tried to open a PDF file. I had found all sorts of strange alternatives for opening PDF files in Firefox. For example, sometimes I would download the PDF file, then open it in Foxit from the local machine. Other times I would copy and paste the URL from FIrefox to Internet Explorer, then use Internet Explorer to open the PDF file.

All those strange alternatives (work arounds, fixes, etc.) have now stopped. I was weary of the alternatives, so I used Google to search for the error message. In classic Google search fashion, the first page had a perfect match for my needs, a post which described my problem and an easy solution to my problem.

The moral: Ask questions sooner, don't be afraid of questions or their answers. I'll need to think more about why I didn't ask the question sooner...

Feedback Junkie

2009-09-24T09:35:00.001-07:00

I was reminded again this morning that I am a "feedback junkie". My team is on a new project and the backlog items are starting to appear as small working pieces of code. In addition, we're connecting with other groups in the company and connecting them to our continuous integration server, showing them how it works, showing them why it helps, and hoping that together we can arrive at working faster because we have better feedback systems.

We've already exceeded my 5 minute threshold on the feedback system. My arbitrarily chosen rule of thumb is that we need to know a checkin is "good" or "bad" within 5 minutes of that checkin. I know that is not always possible. I know there are many tests and sets of tests which will take more than 5 minutes. Still, that 5 minute barrier is the goal, and we'll keep splitting, partitioning, and refining to keep the feedback system within those 5 minutes.

Now, we continue fomenting revolution, getting other people just as addicted to feedback as I am...

Experimenting with Windows Live Writer

2009-08-24T11:40:00.001-07:00

I heard about Windows Live Writer in the context of Windows 7. It proclaimed itself as able to help me do a better job posting to my blog. Thus far I’ve struggled to post multiple pictures and to have the formatting I want.

The idea is that I should be able to easily embed multiple pictures, wrap text around the pictures, and generally manage the blog more directly with their editing tool rather than using the HTML editor provided by blogspot.com.

We’ll see…

Python WIN32 extensions on Windows 7

2009-08-13T07:55:00.001-07:00

I started using Windows 7 a few days ago and needed the Python interpreter. I also needed the WIN32 extensions for Python.

The Python installation worked just fine, I downloaded Python 2.6.2 and it installed with the expected Microsoft UAC prompt confirming that I truly intended to install.

The WIN32 installation prompted for UAC as well, but then failed with an obscure error message. However, I was able to successfully install if I opened a command processor window with "Run as Administrator", and then ran the pywin32 installer from there. With that "magic", the installer succeeded and had the happy message "The pywin32 extensions were successfully installed".

CAST 2009 - Red, Yellow, Green Facilitation

2009-07-16T05:52:00.001-07:00

CAST 2009 - Red, Yellow, Green
Originally uploaded by Mark Waite

An amazing conference has ended its formal sessions. The last 3 days have been filled with insights about software testing, management, and measurement with a group of serious thinkers and practitioners of software testing.

I was fascinated by the facilitation technique used to support the high amount of conversation that occurs at CAST. The facilitation technique allowed a group of engaged listeners to discuss, debate, question, and interact with the presenter in a fluid but surprisingly orderly fashion.

Imagine the types of problems which are likely to occur when a presenter brings a controversial topic to a group of highly engaged, forward thinking professionals. The audience (software testers) consider it their professional role to challenge the status quo, see things differently, and understand those differences more deeply. With that type of audience, a typical presentation would rapidly lose focus as many of the testers challenge, question, and discuss their insights.

The CAST 2009 facilitators used "K-cards" and some agreed principles to assure that the presentations and resulting conversations are a good balance between allowing the presenter to complete their ideas and allowing the audience to interact with the presenter. K-cards are a trio of colored 3x5 cards with a unique number assigned to each member of the audience.

The red card is the "Burning Issue" card. An audience member raises their red card to interrupt the presenter or any current discussion. The red card is used to raise points of order, to flag blocking problems (like poor facility acoustics), or to allow a meaningful interruption of the presentation with key information. The red card can be confiscated by the facilitator if the facilitator feels it is being misused or abused.

The yellow card is the "On Stack" (or current topic) card. It signals that you have a question or comment related to the current thread of discussion.

The green card is the "New Stack" (or new topic) card. It signals that you have a question or comment which is not related to the current discussion. The facilitator keeps a list of numbers on a sheet of paper which reminds the facilitator the expected order in which new topics will be addressed.

The interaction during the sessions was orderly, insightful, and well managed. Ideas were presented, disagreed upon, discussed, and then new ideas were managed as well. The general format of a presentation allowed the first half of the allotted time to be dedicated to the presenter, while audience members would listen and if necessary, would raise a red card to flag a point of order or question which justified interrupting the speaker. The facilitator during my first few sessions was quite patient with my tendency to interrupt and did not take my red card.

Once the presentation was complete, or presentation time expired, the session switched to "open session". The facilitator called the number of the first "new topic" card they had seen during the presentation. The audience member whose number was called (I was number 15 throughout the sessions) asked their question or made their comment and the presenter responded, with some "back and forth" dialog between questioner and presenter.

If someone else had an "on topic" comment or question, they would raise their yellow card. Throughout the session the facilitator is noting the order of appearance and resolution of green ("new topic") and yellow ("on topic") cards on a notepad. "On topic" comments and questions take priority over new topics, and burning issues take priority over same thread topics.

With that simple mnemonic device, a skilled facilitator, an engaged audience, and a presenter ready to engage in dialog about their topic, the conference moved forward very well.

Paul Holland, the lead facilitator of CAST 2009, noted that there are some other subtle techniques which the facilitator can use to further improve the meeting. For example, if there are especially strong or high expertise individuals in the room, it is OK (and useful) for the facilitator to place those individuals at the "bottom" of the "on topic" stack, even if that is not the order in which they raised their card. By placing the experts at the bottom of the on topic stack, it allows the chance for others to present the question or observation which the expert would have presented, and involves other less expert people in the discussions more effectively. I believe some of the facilitators even chose consciously to place experts at the bottom of their "new topic" stack so the less expert would be involved in the conversation.

That stacking system worked well in the session I attended with experts. There were cases where the expert would be called upon and would call "pass" because their idea or comment had already been covered in the discussion.

The system is called "K-cards", named after Paul Holland's wife Karen. Before K-cards, Paul facilitated by having people learn three hand signs to signal the same meaning as the 3 colored K-cards. One of the attendees complained that the hand signs were too complicated. Paul was complaining to Karen in mock outrage that someone would not be able to learn 3 simple hand gestures. Karen suggested, "Why not use different colored cards". They made the switch, and they are now named "K-cards".

Paul noted that the Los Altos Workshop on Software Testing (LAWST) pioneered the original format which has been improved with K-cards.

I will attempt to use K-cards in sessions where I facilitate a discussion with a large group, and I may discuss the idea with others. We have a user experience workshop coming soon, and that workshop seems like an interesting place to try this technique as a way to manage the many opinions, discussions, and conversations which will naturally arise.

Thanks to Paul Holland and to the rest of the CAST 2009 facilitation team for showing how effectively a simple device can encourage interesting, effective, actively progressing conversation!

Another Mistake - Inconsistent Systems

2009-05-12T05:02:00.001-07:00

The build broke again. It seems to have been the same root cause as the last break. The build machine has something different about its configuration which caused it to reject code which was allowed on a developers clean installation of Visual Studio 2008 and the Microsoft.NET framework.

The moral of this story: Don't ignore the first failure, since the root cause of the failure won't "just go away" until it is understood and repaired.