Abakas

Project Doldrums

2009-11-13T12:48:00.001-05:00

Sometimes a project gets into a pretty frustrating state, in which:

it's "mostly" done
it's highly visible
it's just starting to get tried by a broad audience, and not all of them know the background and details of the project, just this thing they have been asked to try.

If you're not careful, this is where you get stuck in the project doldrums. Now is the time to avoid getting stagnated on the project. You're getting feedback, which is probably introducing new requirements or ideas. You're probably finding a few issues. It's likely that you have one or two things you already knew you needed to do. And those things just sort of keep piling on each other.

It's now up to you to get control of it and get momentum again. (I could keep the doldrums metaphor going and say that you have to turn on the motor and get out of the listless winds.)

Getting momentum isn't hard, really. There are only three key things that you must do:

Time box it. You should be happy to take feedback, but you're only giving people a set amount of time to provide it. After that, no new requirements, no whining. It will go into production as it is spec'd.
Make your task list public. You have a set of things you now need to do (fix bugs, update config, add a few features). Publish it, and publish where you are on that list. That way you don't get the same complaints over and over, and when it's fixed, you can tell people to try again. It's a way to show that feedback is not ignored, that you will get to it, and that you are making progress.
Do only your tasks. Don't make random changes or other changes. Every change you make should be based on a task in your list. It is imperative that your task list be complete. If you find something else, add it to the task list, then do it. You don't want to give your (now very public) audience the impression that you're flailing around making random changes. It makes them lose confidence in you.

Projects can hit the doldrums. It will happen eventually. Don't worry about it overly; you can get out of them. Just do it with momentum, and do it with confidence.

Rituals

2009-11-12T14:32:00.004-05:00

After you work in a team for long enough, you start to develop rituals, formal and informal. A standup is a daily ritual. An iteration retrospective is a ritual. The guy who brings in donuts most Wednesdays is a ritual.

Rituals are great. They are the affirmation that you're a team, and that the team is almost a living organism. It has a heartbeat and habits - and those are your rituals.

But...

Rituals are only affirming if they continue to have meaning. There's no point to having a retrospective if you're no longer coming to small stopping points every iteration. Otherwise it's just a standup, only longer; you can't retrospect in the middle of something. There's no point to having donuts on Wednesdays if you're forced to bring them in; the beauty of that ritual is the small thrill of informality.

Embrace the rituals you have. But evaluate them to make sure there is still meaning behind your rituals. As soon as they lose their meaning, stop doing them. There is no affirmation in empty rituals.

Break It Down

2009-11-11T10:30:00.003-05:00

We're working on an internal project that involves (among other things) sending notifications programmatically to Jabber users. It's at that stage where it works for some people and not for others. There are two versions of code doing the sending (different OSes). There are two OSes on the clients, and there are about 10 different clients.

ACK!

So it's time to break it down. It's overwhelming to try to tackle it all at once, but if we make a table we can see what works and what doesn't, and start to try to get all "Y"s into the table, and see if there are any patterns. It gives us a base to work from.

When you're lost, write down what you know, manipulate the data to visualize it, and you'll see a way out.

Sufficient Quality

2009-11-10T11:52:00.006-05:00

How do you measure yourself? How do you know your release is of acceptable quality? You've found a lot of bugs, and you've fixed a lot of bugs. You have a set of great new features, and you've done all sorts of interesting security and usability testing. It's a great release! Or is it?

Your release is of sufficient quality if your customers are sufficiently happy.

The real trick here is to define "sufficient". You could have hundreds or thousands of bugs in the product, and if your customers don't hit them or don't mind them (or think a bug is a feature!), then it's still a release of acceptable quality. You could have a total of 5 bugs, but if your customers hit them a lot and they're bad, then this is not a release of sufficient quality.

So if you want to know how you as an engineering (and requirements gathering and sales) team are doing, ask your customers. They're the ultimate arbiters.

Grunt Work

2009-11-09T11:07:00.002-05:00

I've been working a bit on some data analytics projects. I've been looking at two major things:

(1) what kinds of issues we find in the field; and (2) what kinds of issues we find late in a release. To do this, I go diving through our defect tracking system. We use Jira, so this is mostly creating filters, and generally runs along the lines of "show me all the issues by client in the customer escalations project".

The problem - and this happens with many things - is that our reporting now has more data than it used to. For example, we didn't used to track which client an escalation was open at as a query-able field (it was just in the text). We now track it as a separate field, but that means that all the old issues were never updated. So I have two choices: I can either construct a special query that pulls the info out of the comments; or I can move the data into the field where that field is not populated.

The advantage to a special query is that I can construct it and I don't have to touch a lot of bugs. The disadvantage is that I have to reuse and maintain that fairly complex query every time I need the information. (And if someone else wants to use it, well I hope they can figure out how!) So instead I'm going to make our old issues comply with our new practices - and populate the field we're now using.

The moral of today's story is:

Sometimes, you just have to do the grunt work.

It's not fun, and sometimes it's more manual than I'd like, but your future self will thank you.

Window of Opportunity

2009-11-05T11:22:00.008-05:00

It's relatively easy to decide to change things. It's even fairly easy to decide what you're going to do differently, generally. In order to be successful, though, you also have to consider when you make the change.

Any change has a window of opportunity, time period during which it is most likely to be effective.

For example, let's say you decide to change your release process. Instead of simply sending an email to operations letting them know that the release is ready, you're going to appoint a "development liaison" who will work with operations getting the release in production. The goal of this change is to prevent unintentional misconfigurations (which you've had a problem with in the past). You could do this change right at the beginning of your development effort, but it wouldn't really buy you a lot - after all, you're not releasing so you're not going to try your great new change. No, instead your window of opportunity is a bit before release.

As another example, let's say you're doing iterations and you're not quite perfect at it yet, so the end of an iteration is a bit... frantic. Don't introduce change when you're frantic - it'll only make you more frantic. Your window of opportunity is earlier in the iteration.

So describe your goal, describe your change, and then think of your window of opportunity. All those together will help you gain success.

State Your Purpose

2009-11-04T08:24:00.008-05:00

Being a tester, I see a lot of tickets. Some tickets, unfortunately, hang around for a while, and tend to be worked on by multiple people. These wind up with the basic ticket writeup and a series of comments by different people. Particularly when the ticket is a difficult one, there are theories being tried and discarded.

Let's use an example:

We have an issue where access to the system, either for standard use (reading, writing data over mounts) or for diagnosis (logging in to the box) was slow. The system had several exported mounts, was performing replication, and was deployed in our lab. That's about all we knew going in to it.

As we're working on the ticket, a lot of theories came up, ranging from load on the box to a kernel problem to a network issue (turned out to be saturation of the switch when other system using that same switch we engaging in network-intensive operations).

So the question becomes, how do we talk about this in the ticket? There are good and bad ways to write this up.

A Poorly Written Comment

The replication schedule is:

- 20:00 (average duration: 90 min)

- 07:00 (average duration: 45 min)

- 13:15 (average duration: 80 min)

A Well Written Comment

We noticed that the slowness described only occurs sometimes. Looking at what the box is doing at the time, it always seems to be replicating.

The replication schedule is:

- 20:00 (average duration: 90 min)

- 07:00 (average duration: 45 min)

- 13:15 (average duration: 80 min)

We've seen slowness at:

9/12 21:10

9/13 07:08

9/16 07:14

Earlier comments in this ticket indicate that load average is not the problem, but what else might replication be triggering? Early thoughts: increased threads, increased memory use, increased network use...

The Six Month Test

A good comment is one that makes sense six months later, after you've forgotten all the details. This means it needs to:

describe how it relates to the issue as a whole
describe what the reader is intended to do or take away from the comment

Just like your bugs, write your comments for posterity. Future you will thank you.

Paralysis

2009-11-03T11:34:00.002-05:00

There are days when I walk into work and have a whole lot of different things that need doing, none of them short. A typical list would look like:

- reinstall an object store (1 hour)

- finalize a task list (45 min, and needs people)

- run a scanner utility against a test data set to gather a baseline (2 hours)

- write up how to do a big configuration I've been working on (3 hours)

- provide feedback on a document (1 hour or so)

And I don't want to start any of 'em because that means I'm not making progress on the others! This is a form of paralysis. Fortunately, it's mild.

There's only one way I know to get out of it, and that's to write down my task list for the day, pick one, and start. It doesn't matter which one I pick, as long as it's one single item.

In this case, I invoked my particular prioritization method:

first the stuff that's blocking other people
then the stuff I'm going to forget if I don't do
then the stuff that others are waiting for but not blocked by
then everything else.

In this case, I did the feedback on the document, followed by the task list (also needed by others). After that, I did the configuration writeup, and then started on the scanner utility (and then the day was over!).

Does this ever happen to you? How do you deal with the "too much to do to even get started" problem?

The Rest of the Product

2009-11-02T08:31:00.010-05:00

We have a test plan, and it's great. It covers all the features, and all the workflows of the application. We've got stories, we've accepted them. We've written some automated tests.

Congratulations, you're now half done with the product.

The product is not useful until you can actually use it in production. So now that you've built the darn thing, it's time to think about:

What's production going to look like? How many machines? What configuration?
How are you going to get the software into production?
How about the config info? How're you going to go from dev to test to prod? (hope it's not hard coded into the war file or rpm somewhere!)
Okay, once you got it into production, how're you going to start it?
Come to think of it, how're you going to stop it?
One day you're going to have to maintain this thing. Got a plan for that? Is down time okay? Can you do some rolling upgrades or maintenance?
How are you going to see what's going on? Got logs? Got a way to get logs back to dev for analysis?
How will you know it's running? Any monitoring? Notifications?

There are a lot of questions to answer once you've done the basic implementation. Don't forget to include those when you're thinking about testing it, too.

Taking Notes

2009-10-30T15:18:00.002-04:00

I tend to take notes in meetings. As I was doing that today, it occurred to me that there are different kind of notes that I take:

All notes all the time. These are extremely extensive notes. It almost gets everything (but not quite - I'm not that fast). I take these kind of notes when I'm not sure what's going to be important. Typically these are Q&A sessions with customers, or a presentation for which I'm totally unprepared. I try not to do this often because it's really hard to listen and take these kind of notes at the same time. If I'm going to have to share notes with people not in the meeting, these are the notes I take.
Reminder notes. These are much more outline-like notes that I take for most meetings. These are intended to just provide triggers for my memory. These are the notes I take if I need to share them with people who are in the meeting.
No notes. I do this for a lot of meetings. If I need to be actively participating (or leading) a meeting, I generally don't take notes. I'd rather be fully engaged while I'm there.

Do you take notes?

Choosing

2009-10-29T12:39:00.000-04:00

We make tool choices constantly, sometimes explicitly and sometimes implicitly. For example:

I write a bash script to grab some network info off multiple machines. Tool chosen: bash. Didn't even think about it, just did it.
We're moving parts of our test plan into Jira. Tool chosen: wiki + Jira. This one we discussed for a while, and eventually made our choice based on some cruft with the wiki. I'm not sure it's going to work, but we're giving it a shot.
I burned a CD of our latest installer. Tool chosen: Disk Utility on my mac. This one is quick and handy, and I haven't gotten a bad burn off it yet.

As we make all these tool choices, we're implicitly considering the properties of the tool and comparing that to the requirements of the task. We have to think about only a few things:

What is the tool good for? Jira, for example, is good for workflows. It's horrible for documentation. A wiki is good for documentation but workflow is simply awful. Some tools are more equipped for long term projects and growth than others. Other tools are a lot lighter and good for quick or small projects.
How convenient is it? The tool I already have will usually trump the tool I don't have, just because of setup overhead. It's not universally true, but it takes a really great feature - or a seriously large annoyance with what I have - for me to switch.
How accessible is it? Whatever tool I use needs to be accessible to everyone who needs it. IMing out info is no good for my boss, for example, who doesn't use IM. If he needs to know, then I can't use the IM tool.

Many times tool choice is a really quick, almost unconscious decision. Other times it takes a lot of evaluation and explicit consideration (especially when it's expensive or has far-reaching ramifications). In the end, though, what tool you choose really only comes down to a few simple questions. So don't stress about it too much. In the end, it is just a tool.

Postmortems

2009-10-28T14:04:00.006-04:00

After a release goes out the door, we hold a postmortem. It's pretty standard stuff, usually. We talk about what we did well, what really didn't work out, and what we didn't anticipate.

Timing is an issue, though. You can hold a postmortem right after release, or you can wait to see how it actually does in the field and then hold a postmortem. They each have benefits.

When you hold a postmortem right after you release, you get:

Motivation. People are still stinging from the things we didn't do so well, and are generally aching to fix them. If it was a good release, getting together to remember it will also provide a good boost to people's egos (rightly so!).
Recency. Memories are better right after the release, and you'll be able to have a better discussion about why you did things and what specifically didn't work about them. You'll have a much more precise discussion while everyone's memories are fresh.
Ability to change. If you want to make changes, the sooner you start them, the better chance you have.

When you hold a postmortem after you have some field experience with it, you get:

Perspective. That process ya'll tried that seemed painful right after you did it might not be so painful now. Maybe you've learned it better, maybe you've started reaping benefits, maybe it wasn't as bad as it seemed.
Field Experience. Maybe that release that seemed really shaky has performed like a champion in the field. Perhaps that awesome release ya'll tested extensively has had all sorts of problems. These are things you don't know until it's been out for a while.

My not-so-innovative solution is to realize that our postmortems take us about 45 minutes. That's not very long, so we do both! We hold a postmortem within a week after we release. Then, we hold another one about 6 months later, when the release has been in the field for a while, and we ask ourselves how we really did.

In the land of making things better for ourselves, postmortems are a valuable tool. Holding them twice just lets us learn more from the software and from our development process than we could with just one. Give it a shot.

The Other Fence

2009-10-27T13:29:00.001-04:00

A lot of derision has (rightly) been spilled on the idea that development writes code and chucks it over the fence at QA. Fortunately, at least in the places I've worked, this doesn't really happen any more. That development-QA fence is basically gone (hooray!).

Now maybe we should start working on the other fence.

Other fence?

Think for a second about what happens when you're done testing (and developing) on a release. What do we do? We chuck it over the fence at operations (or professional services, depending on how installations and upgrades get done).

Oh, that other fence.

I've been thinking about this fence, and wondering if it's bad. After all, we didn't used to think it was bad that development finished and chucked the code to QA for work! Now we know better. Maybe we should be starting to learn better as we chuck a build out of engineering and into Operations.

Let's posit for the moment that the engineering-ops fence is bad. What kinds of things might we do to break down the fence, and how might that help?

Change how we structure our builds to make them more releasable. This is somewhat analogous to writing more testable code.
Help deploy. Just like our developers do some testing now, maybe we can help deploy, or create some utilities to help. Rake tasks, deployment scripts, hand installations - how does this stuff get deployed, and can we make it easier or better?
Get help building and packaging. Just like development sometimes asks a tester how best to approach a TDD problem, engineering can get some advice from operations on how to handle a configuration issue, or a packaging question.
Pair on problems. When there's a problem in the field, we don't have to look in isolation, or bounce questions back and forth. We can work on it together. With two different views and skills looking at the problem, you're more likely to figure out a problem that has a foot in both worlds.

Depending on your current organization, and on who receives your code, your list may be different. Maybe you're working with support right after release, or sales. Maybe engineering owns operations, so you don't have this problem. At this point, this is just something to think about.

What do ya'll think? Is there a fence after engineering? And is it time to start talking about that fence?

Org Charts

2009-10-26T11:37:00.003-04:00

There are a lot of different ways to set up an engineering organization. Generally, they fall into one of two categories: function-oriented, and team-oriented.

A function-oriented structure says that people with similar skills and responsibilities should be a team. That team then provides their collective services to other teams. A team-oriented structure says that everyone working on one goal (project, product, feature area, whatever) should be on the same team.

A Function-Oriented Organization

This is a simplistic example of a function-oriented organization. You have your basic disciplines (development, test, product management) as separate groups, and within those groups you have different breakdowns based on the projects that you do.

Pros:

In-discipline learning is easier and more fruitful. Devs will feed off what other devs are doing, testers will see test innovation and build on it - all because you're working with people who are thinking about the same things you are.
Allows dynamic resource allocations. If you need an extra tester on a project, great, we can add a tester.
Explicit thought leadership. You have a head developer who is explicitly charged with improving architecture and development practices. You have a QA manager who is explicitly responsible for evaluating and refining test practices.

Cons:

People are serving multiple masters. They're trying to help their teams and also to conform to their (function-oriented) organization structure. This leads to some conflicts of interest.
Higher risk of silos. If it's a separate group, then you're more likely to have problems with communication.

Use this when:

You lack predictability in your projects. This happens in consulting a lot, but it can happen in other places, too. If you can't predict how many devs you'll need, it helps to have a pool of devs to draw from.
You have unusual requirements on one or more of your groups. If you're doing some really unusual testing, for example, you may need to keep your testers together so you pick up the learning and innovation effect.

Warning:

Avoid this if you're attempting to embrace SCRUM or some other cross-functional team ownership mentality. The "multiple master" problem will get you in this case.

A Team-Oriented Organization

This is a simplistic example of a team-oriented organization. Each team is a group, and contains members from all relevant disciplines.

Pros:

Unity of purpose. The team is all working toward the same goals. There is no secondary or other goal.
Breakdown of silos. If you can get true team ownership, you start to find developers testing, and testers helping with product management, etc.
No need for functional management. The role of "QA Manager" goes away here. Instead you have team leads.

Cons:

Harder to drive functional change. When you have several teams with a few testers each, it's a lot harder for testers to innovate or learn from each other. The same goes for developers. The groups are simply too small to get that kind of momentum.
Hard to handle changing needs and moving through software development phases. You run the risk of having idle testers as you start an effort, and idle developers at the end of an effort. This is something that can be overcome, but you have to encourage cross-functional work, and be sure to plan appropriately.

Use this when:

You're using SCRUM.
You can have generally stable teams. This implies your projects (or products) are pretty consistent in size and resource needs.

Warning:

Avoid this if you have a particularly weak functional area (or more than one). There's a large risk that isolation within a stronger team will make them even weaker.

So Which To Use?

I've seen organizations of both types - functional and team - work great. And I've seen them both fail spectacularly. The trick is to align your teams with your development and business philosophies. Have you embraced SCRUM? Are your projects generally consistent in size and skill sets needed? Cool - you probably want a team-oriented structure. Do you have highly specialized needs in one or more areas, or an extremely lumpy (in terms of resources wanted) plan? Consider a functional-oriented structure.

In the end, pick the structure that works for you. Just do yourself a favor and pick a single structure. Trying to mix and match will lead to heartache, but pick a single way and you'll give yourself a good chance at success.

Merging

2009-10-23T14:19:00.001-04:00

We work in what I imagine is a fairly typical environment. We code away on HEAD for a while. Once we're feature complete, we branch (so now we have HEAD and RELEASE). Then we fix stuff on HEAD and merge it to release until we hit code complete. We also go ahead with the next features on HEAD, but that's not currently the point.

The closer you get to code complete, and particularly after code complete, things get tricky. What do you merge?

There are, after all, several kinds of changes that might be candidates for merging into your branch:

Code changes to production code. Bug fixes, new features, etc.
Code changes to test code. Change the tests, not the code that actually ships.
Infrastructure changes. Change something underlying about the lab or environment (e.g., update the default fstab that gets installed)

So what do we do?

Code changes to production code tends to be the most commonly considered case. You evaluate the risk of the change, how much retesting you need to do, the benefit of the change (and how many of your customers are likely to benefit), and the amount of time left before you really really have to ship. Based on that, you choose to take it or not.

Code changes to test code are trickier. On the one hand, change is change and all change introduces risk. Sure, this code doesn't ship with your product, but it's still change. Plus, you have to consider risk here, too. If your test change breaks something, you might get less information out of your tests in the future, and that would be bad. On the other hand, it probably has some benefit, too: tests run faster so you can do more of them; or a test passes a failure point and lets you expose any other problems that occur later in the test; or perhaps you just spend less time looking at the error that isn't telling you anything new. For me, the bar to test code is lower than the bar to taking production code, simply because the risk to our actual (field) customers is lower, but there are some things that I try to consider:

Is the change caused by a failure or just a cleanup/enhancement/nice to have? The former is more likely to get put in than the latter.
Is the change going to fix something that causes problems for other tests? (e.g., a hang that stops all later tests in the suite from executing). A bad citizen like that is more likely to get fixed.
Is the change risky? The same types of analysis apply here as for product code. Avoid big, sweeping, likely-to-break-something changes.
How many more test runs are we going to have? The closer we get to the end, the longer we can just deal with the problem and not bother to fix it.

Infrastructure changes are generally not really optional. If you want your release tests to keep running in your infrastructure, they have to keep up with changes to your infrastructure. That being said, make sure you really need that infrastructure change, and be mindful of making the changes as small and safe as possible.

Merging is a tricky business, and the closer you get to a release the more of a "gut feel" kind of thing it turns into. So before you get into the thick of it, think about what you will merge and why. It'll save you some arguments later!

All The Other Tests You Did

2009-10-22T13:13:00.001-04:00

I've been verifying bugs for the past day or so. It's actually work I really enjoy. The vast majority of the time, it's concrete evidence of the product being better, which is awesome. Plus it's very easy to see the progress I'm making, which appeals to the list maker in me (I love checking things off!).

Here's the thing, though: I'm not just verifying bugs. I'm performing lots of other tests at the same time.

For example, a bug I verified was a display problem with replication progress. This is a small issue, but, hey, it's fixed, so we'll verify it.

To verify it, I had to:

install two systems
create a volume on one system
configure replication between the two systems
configure replication on the volume

So, just to verify one bug, I had to do an installation test, a volume creation test, and a replication test. All I had to do was a quick check to confirm these weren't throwing errors not visible to the end user, and then I have a small other thing done. Repeat enough time, and this adds up to rather a lot.

So next time you're verifying a simple bug, ask yourself what other tests you're doing. You may be accomplishing more than you think!

Today's Top N

2009-10-21T16:51:00.001-04:00

I don't drive very often - I live and work within the city, and I tend to take the T to work and pretty much everywhere else, too. Consequently, I don't listen to the radio very often. But this weekend I was on a road trip and I had the radio going. I kept hearing the same theme over and over as I flipped around the stations:

"Today's Top 10"

"the Weekly Top 20"

"Top 5 Countdown"

And I remember thinking, "What a good idea!"

After all, the top 10 songs, or top 20 albums, or whatever, are in some ways like the parts of our test plans:

some of them are the same over and over again: how many weeks is one song at the top of the charts, or at position two or five? Same thing with areas of code.
some of them change each time: eventually a song falls off the list, and eventually we're comfortable enough with an area of our test plan that we move on.
they sound a little different every time: maybe this week it's the dance mix and next week it's the acoustic version - same underlying thing, just a bit different.

So why not? Let's embrace the theme!

We happen to be really close to a release. So for this week, we're having the QA Top 4 (there are four of us, so this makes it easy). Every morning, we come in and pick the four areas we're currently least comfortable with, as a group. We all throw ideas around until we agree on the four. Then we go work mostly on those four items - they're the top of our list for the day. The next day, we repeat the procedure. Maybe it's four different items, maybe some of them are the same and some new - doesn't matter, really. But that's the new QA Top 4. So we work those new four for the day.

The idea here is that we get a chance, every day, to re-identify the scariest areas of the code. And then we work on them. If they're still scary, we'll work on them again the next day. If not, we'll work on the new scariest areas.

There are a lot of ways to prioritize things, but having new ways to think about it sparks new ideas. This is just a new (to me, anyway) way to present that old risk evaluation, and hey, it's kind of fun.

(And by the way, you should see the jokes.... "Replication, by the Again Agains" at #1, and "Defect Verification" off the "Oh Boy It Worked!" album at #2. We're pretty easily amused!)

Good Citizen

2009-10-20T11:34:00.001-04:00

As we're testing our software, we have lots of different kinds of requirements. We have use cases, functional requirements, performance requirements, usability requirements, testability requirements, etc. One of the requirements that we don't usually talk about explicitly is the good citizen requirement.

Wait, what's a "good citizen"?

A bit of definition:

Software that is a good citizen behaves in a manner consistent with other software, with regard to interaction with other assets with which it interacts.

That's kind of a pompous way of saying that software is behaving like a good citizen when it does what the systems around it expect (e.g., log in a way that centralized logging tools can handle it) and does create excessive load or resource usage (e.g., doesn't attempt to create hundreds of DNS entries when one will do). In other words, this is software that does what it ought to do, and doesn't behave badly.

Software that is being a good citizen does:

support logging in a common format (e.g., NT Event logs, etc)
use centralized user or machine management (e.g., Active Directory or NIS)
does automatic log rolling
can be configured to start on its own after a power outage or other event
can be disabled or somehow turned off cleanly (to allow for maintenance, etc)

Software that is being a good citizen does not:

log excessively (at least, except maybe in debug, which should be used sparingly)
create excessive traffic on infrastructure servers (DNS, Active Directory, mail, firewall, etc)
send excessive notifications (e.g., a notification for every user logging in would probably be overkill)

Normally, the good citizen requirement is not explicit. Sometimes you'll find mention of it in requirements indirectly (e.g., must support Active Directory for user interaction), but sometimes you won't. You usually won't find the negative requirements (e.g., doesn't renew its DHCP lease too often) at all. But if you miss one and your software misbehaves, you can bet you'll hear about it! Good citizen requirements are generally assumed, even though they're often not mentioned directly.

As you're testing, ask yourself, "is my software being a good citizen?"

"Certifying" Clients

2009-10-19T17:34:00.002-04:00

We're a storage company. Lots of people write to us with lots of different programs. These run the gamut from drag-and-drop to homegrown bash script to full HSM solutions, and lots of points in between. Sometimes we'll get asked to "certify a client" application. What's going on here?

Let's break it down:

Who's asking

Usually for us sales or support is asking. Sometimes sales has a client who wants to use a program and wants a guarantee it will work. Other times sales has a client who has not picked a client and wants to know what we recommend. Alternatively, support might have a client who's attempting to use a program and finds something they don't like about it (doesn't work, works too slowly, etc).

Let's say you're not in a storage company (some of us aren't!). This could be a browser, if you're a web app. It could be a reporting or monitoring tool (anyone else ever had a client ask to point Crystal Reports directly at your database?).

Either way, someone's now looking for a guarantee that a client program will work with our software.

"Guarantee"

I'm usually afraid of the word "guarantee". You can do all the testing in the world, and a new patch of a client program will come out and break in a truly spectacular manner. Or the customer will use an obscure undocumented flag you didn't test and... kablooey! (tm Calvin and Hobbes). Or the client will install it on some totally unsupported hardware and scratch his head when it doesn't work. "Guarantee" is a very strong word that means "it's totally my problem to fix".

I usually get around this by saying, "here's what we've tested" rather than "we guarantee this".

Certification Levels

There are a number of different things you can do and call it certification.

The standards approach. This is where you point to some external standard and say, "we conform to this. Any client that works with this will work with us." By external standard, you should make sure you choose a public standard: NFS v3, or W3C compliance, or whatever's appropriate for you. In this case, you don't actually have to test the client. However, you'd better be darn sure you conform to the standard, or this one may eventually bite you.
The "we test this" approach. This is where you offer up the version and configuration you test, and you say that has been tested and will work. Any deviance from that configuration or version may work but isn't guaranteed.
The "certification program" approach. This is where you turn it around on the client application, and offer a certification program. The idea is that they conform to you, rather than the other way around. You offer a set of criteria, test systems (or a lab for people to come test in), possibly scripts and reporting mechanisms, and you let people run your tests. Then you analyze their results, and either put your stamp of approval on or not (think "runs on Vista", etc). If you're large enough and important enough, people will do the compatibility testing for you. This doesn't work so well if you're kind of a tiny nobody in your industry. I've not done this one personally.

So What to Do?

In the end what you do is driven by how much team, time and sensitivity you have. The real goal here is customer (or potential customer) comfort. So you do what you have to do to achieve that customer comfort, within the bounds of the worth of that customer.

My first approach generally is to do a test for that client. If this is an important client, we can get from them (or create if they don't know), a configuration that will work for their situation. Then we test this (and retest it on new versions of our code). It's client-specific, but it gives us the comfort to go to the client and say, "follow our advice and this will work."

My next approach, if this starts to get to be too much volume, is to publish a "known good" configuration (including version) of the client software. We test that client config with every release we do. We tell clients what works, and then let them experiment from there, if they need to.

These two approaches have gotten me far enough, so far. In the end, there's no substitute for trying it at the customer, but short of that, you can give them comfort. And all "certification" really means, at least in this sense, is comfort.

Strict

2009-10-14T10:18:00.003-04:00

We're making a big switch in the lab; we're upgrading the underlying operating system on our machines. This is something that we kind of have to do - wouldn't want to be on an ancient OS because it only makes things like security patching harder. I can't say it's my idea of a good time, though - it's a lot of work!

Anyway, once we've done all the basics - setting up FAI, setting up build and test systems, getting the lab migrator to run (so we can move machines from old to new and back again), etc. - then we can start the tests.

We start slowly - one night on the new OS, and then not again for a while. This way teams get a chance to go through their problem areas and fix them before they get hit with the same ticket again. At this point, too, our number of failures is generally quite high, and one or two problems will take out entire swaths of tests ("can't talk to the NTP server: 42 machines can't be cleaned up!" or "stunnel configuration is different: killed two entire suites!"). It's a fairly quick and easy way to find problems that affect each of the teams. It's a learning experience, it makes a bit of a mess, and we all join in cleaning up after it.

At some later point, then, we make the decision that it's time to switch. At this point it's time to get strict. So now we worry about things in the following order:

First, resolve compilation errors. If it doesn't compile, not much is going to run.
Second, resolve bugs that cause machines to leak. If a test causes machines to not clean up after it's done, then they're not available for other later tests. This causes the entire lab to grind to a halt. Generally this is accompanied by cries of "we're out of machines!" and tests just not finishing because there's no machine for them to run on.
Third, resolve bugs that are likely to hide other bugs. If I have a bug in my test setup, who knows what will happen when I get to actually exercising the thing I thought I was testing!
Fourth, handle everything else. Once you've gotten through the first three items, then just start fixing bugs according to your preference.

The overall goal is to expose the bugs. Get things running, then follow up with getting them running right. Hopefully this whole process doesn't take you too long, but sometimes when you're mired in the land of "this underlying thing broke a lot!" it helps to step back and think about what to prioritize. You can make the whole process go a bit more smoothly if you think for a minute, then leap in and start fixing.

Good luck, and happy resolution!

Just Do It

2009-10-13T22:09:00.004-04:00

At any point in time, when I sit down at my desk, there are about fifteen things I could do. For example, when I sat down after today's dev leads meeting, I had my choice of:

verify some bugs targeted for the next release
verify some bugs on head
answer a question from a sales engineer
review a potential client sizing worksheet
fix one or more of about three small-ish bugs assigned to me ("fetchlogs doesn't fetch when... whoops!")
read a white paper
work on an in-progress FAI server I'm configuring

In the end, which one I pick isn't nearly as important as one thing:

Just pick something already. Then just do it.

To a certain an extent, what I do doesn't matter as much as simply accomplishing something. I have a general priority list:

active fires
stuff for clients
stuff for sales
stuff other people on my team can't do
other stuff

Based on that, I chose to answer the email first (and do the analysis that I needed to do to provide a good answer). But I could have chosen almost any one of these, and it all would have helped our current situation. And that's the point - just do something. That'll help.

Chatting

2009-10-12T11:32:00.007-04:00

I just read this article by Joanne Rothman, talking about a "low level buzz" with chats, emails, and talking going on all the time. She basically says that maybe having a chat open all day, or high email traffic with quick responses might work for some people but not for her.

Our team is colocated, and we still chat all the time. It's a little odd. Basically, most of us have a chat window open on our desktops constantly, and we take a look at it when we're waiting (for compilation, a grep to finish, etc). It's that low-level background buzz. In addition we have, and frequently use, the option of simply walking over and talking to each other.

We tend to use chat for things that the entire group might be interested in, like:

build failure notifications, and their causes
discussion of which branch to run in the nightly automated tests
notification when weekly lunch arrives (the important stuff!)
code review requests
heads up about interesting or risky checkins
pairing requests
general pleas for help (e.g, "how do I do X in perl?")

We get together from there. Someone will wander over and answer how to do X in perl, if it's non-trivial, or will start pairing. It generally works for most of us. I like it in particular because that way you don't have to feel bad about asking questions - you just ask the collective group and whoever is least concentrating at that point, or currently waiting for something, will answer. I'm not then interrupting someone who's really deep in something.

Use for yourself with caution!

Magic Words

2009-10-08T11:41:00.001-04:00

Just like there are some magic numbers that can point you to the source of a problem, there are some magic words that also have power. The words you use to describe a problem will color how other engineers (developers, other testers, managers, even you) look at the problem and what they do to track it down.

Just like with the numbers, some of these are common to many systems:

Crash. Most people will be looking for a core file and an uncontrolled shutdown in this case.
Hang. Often you'll get asked how long before you mark something as being hung. Also, this word means that absolutely no progress is being made. If it's progressing at a snail's pace, it's still not a hang; it's just really slow.
Failed. This one means completed with error. It doesn't mean "still going".
Wedged. This is imprecise but generally seems to make people think deadlock.

Other magic words are more specific to your product. For example, in our product we discuss:

Data loss. The system could lose data in this scenario (YIKES!). This one raises red flags all over the place.
Failure. This word means system change that resulted in the loss of availability of a system component. Generally its used for hardware failure (of a disk, node, power supply, network connection, etc). Generally "failure" does is modified by some component name (e.g., power supply failure), and is not used for the more general "software didn't work" case.
Simultaneous Failure Vs Sequential Failure. In a redundant self-healing system like ours, number of failures (see above) matters, as does whether the second one occurred at the same time as the first or after. Depending on which one happened, a whole different debug path will be invoked.

What other magic words are there? I'm pretty sure I've missed some.

Magic Numbers

2009-10-07T23:34:00.010-04:00

There are some numbers that I call magic numbers. These are the special numbers that have meaning in the context of your system. These numbers are typically diagnostically important, or triggers to identify problems or potential problems.

Some of these are common on many systems:

86400 seconds: As in, "the test timed out after 86400 seconds". This is a day. As in, the darned thing didn't finish up in a whole day. Oops.
2^32 or 2^32 -1: If you're on a 32 bit system, you're starting to wrap address space here. Look for really large negative numbers where you're expecting positives, etc.
45 seconds: the default connection timeout for network mounts in Windows. If something fails after this long, you're looking at a timeout probably.

Some of these are specific to your system. For example, in our system we know:

6 minutes: The RPC timeout for a query to a remote system is 2 minutes, and it does 3 tries. 6 minute waits mean you're hitting this.
5 minutes: The frequency of heartbeat checks for certain operations. Failover after 5 min means these never succeeded.
25: default max number of simultaneous connections. Can be increased indefinitely, but if you start to see slowdowns or connection timeouts and you have 25 clients in use, you're probably going to want to change it.
XX: Java heap size. (I don't remember this one off the top of my head, but I know it when I see it)

What is interesting is not simply knowing the numbers. What is interesting is the shorthand debugging that it offers you. For example, if support calls up and says that a customer is complaining that the management functions are "very slow to load", and it turns out to be about 6 minutes, then the first place I'll look is to see if it's trying to talk to a remote system, and if there's some sort of problem in that communication.

It's not perfect, but knowing your system's magic numbers can often be a shortcut to finding its problems.

Real Time Feedback

2009-10-06T10:10:00.004-04:00

This is a trick I use in particular for UI bugs.

The Problem

I'm working on a web-based application and am testing it across five browsers - Safari, IE 7/8, and Firefox 3/3.5. At the moment I'm working on layout and styling.

I have identified an issue related to buttons. Specifically, they look like this on IE7:

Normally I would do what I generally do when I find a bug: log it. I'd attach a screenshot, explain which browser(s) were affected, and move on. There'd be a cute title like "buttons look funny", but most of the explanation would be in the screenshot.

And then a developer would decide to work on it, and he'd move pixels around, and he'd recut the button so his CSS sprites were lined up. And he'd generally getting it looking okay. Mark the bug resolved, and it's time to go to lunch!

So along I'd go, and great, it looks fine on IE7 but now the text is way too high and falling off the button on Firefox 3. I reopen the bug (or log a new one), and kick it back to the developer.

Launder, rinse, repeat.

The Approach

Let's bypass the defect tracking system as communication tool technique that we're currently using. The developer and I set up a time to properly fix the buttons. He's got Safari and Firefox 3.5. I launch IE7, IE8, and Firefox 3. We're going to sit next to each other, make a change, and check it until we're both happy with those darn buttons in all browsers.

Here's how it goes:

Both of us open our various browsers
Both of us point all those browsers to the developer's machine. Now we can see everything in all browsers pretty quickly.
We make sure no one's caching CSS or JS (Often this is a configuration setting on the development environment, and probably already set up, but it's worth checking.)
Developer makes a change.
We both reload all browsers. Nope. Not there yet.
Developer makes a change.
We both reload all browsers. Success on IE7, but we broke Firefox 3. Darn.
Developer makes a change.
We both reload...
... etc etc etc ...
Developer makes a change.
We both reload all browsers. Success!
Check in.

What We've Done

By sitting side by side and knocking out the problem, we've substantially reduced the feedback loop duration. It's easy for the developer and for QA both to see the behavior in all the browsers we care about; everyone can look at everyone's machine and we don't have any lost information in terms of visual behavior. I should note that this can work virtually, but in that case it's easier if you can see each other's screens in some way.

Obviously, this technique won't work for all bugs. But for the visual ones that are mostly a matter of messing around with CSS or with JavaScript, it tends to work really well. The total time to get rid of the bug is much much shorter than if you'd both sat at your respective desks and bounced the bug back and forth.

Give it a shot!