Abakas

Are You Really a Team?

2010-07-30T11:53:00.004-04:00

These days, many of us work in Agile teams or in integrated SCRUM teams. We're no longer the development team and the test team and the design team. Instead, we're one team working together to ship product. At least, that's what we tell people.

If this is you, though, are you really a team? Is this really all one group going toward a single goal?

Three simple questions will help figure this out:

Do you assign tasks to individuals?
Do developers do test tasks when the team is crunched? Do testers help with system configurations when the team is trying to get set up for new development?
When a bug is found in the field, who gets yelled at and tries to figure out how to prevent that kind of problem from making it out to customers again? Your testers? Or your team?

If you're still playing "developer" and "tester" roles, then it doesn't matter whether you call yourself one team or two, you're still restricting yourself. You're still saying that you do your bit, the other guy will do his bit, and if we're all good and lucky, then good things will happen. That's not a team. That's a group of people.

A team is a group of people working toward the same goal. Some have more experience than others in different things, but if you're really a team, then each member will happily do whatever is standing between the team and release.

The difference between a group and a team is very simple:

Teams succeed together and teams fail together.

Groups pass or fail, each individual alone.

If you want to be a team, be a team, not a group.

Tales from the Phone Screen

2010-07-28T13:26:00.004-04:00

I've been doing a lot of phone screening lately, and there have been some real doozies. I've heard all sorts of things:

"I left my last job because everyone was incompetent but me. It went through two whole teams like this."

Wow - ego coming through! There are definitely incompetent people out there but to run into such a high proportion says that you either have truly terrible luck .... or that there's something wrong with your idea of competence.

"Oh, my weakness is I don't do automation."

This is only strange once you know that the position is for a QA Automation Engineer.

"I'm just looking for a job before my skills get too rusty."

On the one hand, this one is kind of sad. On the other hand, he's not doing anything to keep up or expand his skills, apparently, and that's a problem. I don't want to have to push for every new learning experience.

There are a lot of decent resumes out there. Phone screens really weed a whole lot out, though. Be careful what you say; the hiring manager is probably listening.

Jira and SOAP: Part III

2010-07-26T15:19:00.004-04:00

The saga of using Perl code to communicate via Jira's SOAP API. See part I and part II here. In part I, we logged and updated issues. In part II, we were able to get status information about an issue. In our final part, we'll actually reopen the issue:

#####################################################################

# Reopen a Jira issue

# @param params{issue} The issue key (e.g., PERMA-12345)

# @param params{assignee} The intended assignee

# @return The jira issue number

sub reopenIssue {

my ($self, %params) = assertMinArgs(1, @_);

$params{issue } ||= '';

my %issue;

$issue{issueKey} = SOAP::Data->type(string => $params{issue});

$issue{assignee} = $params{assignee};

$issue{summary} = $self->_getIssueSummary(issue => $params{issue});

$issue{newStatus} = SOAP::Data->type(string => "3"); #reopened

my $result = $self->_request('progressWorkflowAction', 1, \%issue);

return $result;

}

It's pretty simple, in the end. We're using the progressWorkflowAction call, which changes an issue from one status to another. We simply create our issue object with all the required attributes (in our case, summary, assignee, the new status, and the issue key) and then call progressWorkflowAction. This call will fail if the user doesn't have permission to make that transition, or if the transition doesn't apply (e.g., attempting to close a closed issue).

And that ends the Jira and SOAP tutorial, at least for now. Good luck!

Interview Questions: Junior Candidates

2010-07-22T10:34:00.000-04:00

When I'm interviewing candidates, I like to ask a lot of questions and present a lot of different types of problems to see how the candidate thinks. There are some things that I look for in more junior candidates specifically. For junior candidates, I'm interviewing for potential more than for what they've done, and that shades what I ask.

"You've been dropped in the middle of the Linux kernel, never having worked on kernels before, and asked to fix a thread deadlock. How do you go about it?"

With this question, I'm looking to see how the candidate learns. Our system is not particularly small, so I want to see how they approach it. Answers that imply they will try to learn the entire system before doing anything are generally not good; this isn't something you can hold entirely in your head without living in it for a while, and I want someone who's going to jump in. Incidentally, we use Linux, so this question isn't totally off base.

"When you're working on a project in a team, how often do you check in? Any tricks or preferences there?"

With this question, I want to see how the candidate works in a team, and how the candidate approaches source control. Some candidates will go away for weeks or months and build a huge, beautiful, complete thing, and then have a heck of a time checking in because all the other code has changed in the meantime (this is a red flag). Other candidates check in near constantly, which means lots of reviews (also a red flag). Every team falls somewhere along this spectrum; I want to see if the candidate matches our rhythm, or is even aware there is a rhythm to match.

"Your team of five has been handed a feature to build: you're going to build the crash recovery mechanism. How do you split up the work?"

Junior candidates won't actually be dividing work, but they'll be on a team, and I want to know how they're going to work with that team. One danger of really junior candidates is that they mostly have experience working on school projects or isolated code areas, and often alone. Our company doesn't work that way, and we need to know that a candidate is going to be comfortable building part of a feature when the other part is being built by a teammate at the same time.

Most of these questions revolve around seeing how a junior candidate works in a team environment, since they likely don't have this much experience. We can teach coding skills, we can teach learning techniques, and we can teach development practices, but only to someone who is aware there is something to learn.

Find the Oracle

2010-07-20T09:13:00.001-04:00

In testing, an oracle is something we look to that can define the "should" of an application. The oracle (or oracles) tell us what an application should do, how the UI ought to look, how the application ought to perform, etc.

Oracles can take many forms:

requirements specifications
screen mockups
other (competing or complimentary) products
style guides (yours or someone else's)
similar or related functions in the same application
a previous or alternate version of the application being tested (think porting projects)

When you start to test your application, one of your first jobs is to find the oracle(s). This will help guide your future tests, bugs identified, and behavioral expectations.

Sometimes finding the oracle is easy; for example, if you're handed a requirements specification, then you've got at least one of your oracles. Other times, you'll have to get creative.

For example, a friend recently approached me with a problem:

"We're processing files, and we discovered that on this file server with 573million files, we detected 19 million ZIP files. Is that reasonable?"

This is a problem in search of an oracle. It's infeasible at that scale to hand-check all of the files. We're not yet sure we can trust our detection software, though; after all, that's what we're testing. So what's our oracle?

Seems hopeless. "Hopeless" is just another word for "needs creativity". We can do this.

Our problem revolves around the percentage of files on a file server that are of a certain type. There are two ways I think to go about this:

1. run one or more other file type detectors and see how much they agree with my results

2. find other file servers (or reports about other file servers) with a typical file type breakdown. Academic papers, storage vendors, and OS vendors are great sources for this kind of information.

What would you use for an oracle in this situation?

Perl SOAP Interface to Jira (part II)

2010-07-14T16:48:00.009-04:00

A while ago, we wrote a bit of Perl code that uses Jira's SOAP interface to log bugs, add comments, update tickets, and do other fun stuff. I recently extended this to get the current status of an issue out of Jira. We use this to look for bugs that are closed that we're still updating (whoops!).

Here's the code:

######################################################################

# Issue a request to the Jira server and get a response.

# @param cmd The name of the command being sent

# @param checkResult Whether or not to check the result of the command

# @param params The parameters of that command

# @return The result of the command

sub _request {

my ($self, $cmd, $checkResult, $params) = assertMinArgs(3, @_);

my $soap = SOAP::Lite->proxy(

"https://jira.permabit.com/rpc/soap/jirasoapservice-v2?wsdl");

my $auth = $soap->login($self->{jiraUser}, $self->{jiraPwd});

my $s = SOAP::Serializer->new;

my $doThis;

if ($cmd eq "addComment") {

$doThis = $soap->$cmd($auth->result(), $params->{'issue'},

$params->{'comment_obj'});

} elsif ($cmd eq "getComponents") {

$doThis = $soap->$cmd($auth->result(), $params->{'project'});

} elsif ($cmd eq "getStatuses") {

$doThis = $soap->$cmd($auth->result());

} elsif ($cmd eq "createIssue") {

$doThis = $soap->$cmd($auth->result(), $params->{'issueDef'});

} elsif ($cmd eq "updateIssue") {

$doThis = $soap->$cmd($auth->result(), $params->{'issue'},

[$params->{'rfv'}]);

} elsif ($cmd eq "getIssueById") {

$doThis = $soap->$cmd($auth->result(), $params->{'issueId'});

} elsif ($cmd eq "getIssue") {

$doThis = $soap->$cmd($auth->result(), $params->{'issueKey'});

} elsif ($cmd eq "getStatuses") {

$doThis = $soap->$cmd($auth->result(), $params->{'issueKey'});

} elsif ($cmd eq "getVersions") {

$doThis = $soap->$cmd($auth->result(), $params->{'project'});

} elsif ($cmd eq "getIssuesFromFilter") {

$doThis = $soap->$cmd($auth->result(), $params->{'filterId'});

} else {

croak("Unexpected command: $cmd");

}

if ($checkResult && defined $doThis->faultcode) { # whoops something went wrong

croak("Error running command: $cmd\nGot: " . $doThis->faultstring);

}

return $doThis;

}

#####################################################################

# Get issue Status by key

# @param params{issueKey}

# @return The jira issue number

sub getIssueStatus {

my ($self, %params) = assertMinArgs(1, @_);

$params{issueKey} ||= '';

my %issue;

$issue{issueKey} = SOAP::Data->type(string => $params{issueKey});

my $jStatuses = $self->_request('getStatuses',1);

my $jIssue = $self->_request('getIssue', 1, \%issue);

my $jIssueStatusId = $jIssue->result()->{'status'};

my $jIssueCurrStatus;

foreach my $status (@{$jStatuses->result()}) {

if ($status->{'id'} eq $jIssueStatusId) {

$jIssueCurrStatus = $status->{'name'};

}

return $jIssueCurrStatus;

}

The first method is what actually submits the SOAP-encoded request to Jira. There are different request types with different parameters, but the ones were interested in are:

getStatuses: takes in the authentication token only, and returns all statuses defined in Jira (regardless of project)

getIssue: given the authentication token and an issue key (the thing you see in the UI: e.g., "QA-1"), this returns the status. Jira returns the ID of the status - for example, "1" - and we use the statuses we grabbed to translate that to a human-readable status name - for example, "open".

Once we have the status we can do things like automatically reopen issues if we're getting logs for an occurrence (hey, it must have happened again, so why not reopen the issue?).

You may choose to do this differently, but hopefully it will give you some pointers on how to handle the Jira interface.

Sprint Renegade

2010-07-12T09:27:00.001-04:00

When you're constructing a sprint backlog, there are almost always more things to do than you could possibly handle. Marketing wants some things, sales could close a big deal if only they had some new feature, product management feels like this other thing would be a great differentiator, support has a few requests for additional logging, oh, and development also has a few technical tasks (build restructuring, etc.) they really ought to handle.

So much to do, and unfortunately, those technical tasks often simply don't make it into the sprint. Over time, not having that technical keep up causes increases in technical debt and slows down delivery. Eventually, you have a lot of technical tasks, and yet everyone else still wants new features. Velocity is slowing down, builds are taking longer, and there's more general muttering about things not being clean in the development and test environments.

It's time to introduce the sprint renegade.

The sprint renegade is someone who effectively leaves the team for a sprint. He goes off and simply works on tech tasks to make it easier for the rest of the team to meet the obligations of the backlog. This reduces your potential velocity for that sprint by one person, but it boosts your long-term velocity by taking care of some of the drag on the team. Repeat this for each sprint until velocity is back up where you want it to be.

The general idea of a sprint renegade is that internal tech tasks are very hard to sell onto a backlog, and they will often lose in the face of customer-facing tasks. If that's true in your company, sometimes drastic measures are in order, temporarily.

If your technical debt has crept up to a point where it's interfering with delivering, find yourself a renegade. You'll be glad you did.

Around the Software

2010-07-08T14:35:00.000-04:00

When we're estimating a feature or a story or a software change, we almost always think of the following:

time to design the feature
time to implement the feature
time test the feature

And we come out with an estimate: "2 engineers and 1 tester for a week" (or whatever).

That estimate fails to account for all the other work that needs to be done. We have yet to estimate:

test infrastructure extensions
other infrastructure extensions (e.g., do we need to provision a hosted site for centralized client logging?)
documentation modification or creation
configuration analysis and best practices development (to answer the questions customers have)
marketing, presentation, and other collateral updates

Just because the product code has been written, and even if the tests have been done, a feature isn't done yet. Don't forget to plan for - and actually do - all the stuff around the software.

Code Smell: Model Mismatch

2010-07-06T09:57:00.001-04:00

For any system, there are several potential models of that system:

the user's mental model of how the system works and how its different parts relate
the model in the code, implied by the objects and their relationships
the design of the system (hopefully this matches the code, but not always!)

When all the models match, you have an effective system. When the models don't match, that's a code smell. At it's core, a code smell is a warning of future difficulty. And when the user's model doesn't match the code, it's going to be hard to add features the user wants because the code has different ideas.

For example, I'm working on a project that is a basic web application. Users sign up and are put into teams for contests. Everyone on the team that wins the contest gets a prize, and then the whole process starts over with new contests and new prizes. A user made a request to be able to see how much their team had won, total. Sounds fairly simple. It wasn't.

There was a discrepancy in the models. The user's mental model of the system said that "teams win prizes". The underlying system code was that "contests have prizes". This model mismatch meant that instead of simply getting all prizes for a team, the developer had to get all the contests a team had participated in and then get all the prizes awarded for that contest and then filter by team. It was real work!

So the developer tried to refuse to do the request because "the system isn't designed that way." Now we've got a real problem. There are three ways forward:

Refuse to fix it. This is tempting sometimes, but it's an unacceptable answer in many cases. It's a user request, and assuming it gets approved, you'll need to implement it. Features aren't chosen by how hard they are, but by their value. Get over it - "it's hard" is not an excuse.
Fix it. It's possible to make it work, even if it's in a roundabout way. Note that this does make your code more closely resemble a mountain of spaghetti.
Reconcile models. Whether you change the user's model (often implied through the user interface) or change your underlying system design, the right thing to do often is to reconcile the models. This is how you ultimately clean up the code smell - by rearchitecting, refactoring, or modifying the UI and documentation so that what the user understands matches the design and matches the code.

When you get a request or a bug, and your first response is "oh, that's weird" or "oh, that's a lot harder than it sounds", ask yourself what the model is that came to that request. You may have just found a code smell.

Severity What?

2010-07-02T09:01:00.003-04:00

I've seen a lot of project orders that include language like this:

"The product shall not ship with any known unresolved severity 1 or severity 2 defects."

Severity 1 is usually defined as crash, data loss, and/or total system unavailability. (In my team, we call this "the big oops".) Severity 2 is usually defined as loss of core functionality of the system.

But when the project's late, those definitions sometimes get changed because, "darnit, we've got to ship!"

And then something like this happens.

Beware changing definitions - there be dragons!

Up To Date

2010-06-30T10:28:00.004-04:00

I've been in more than one series of events like this:

"We're starting to put down some dates for the Sweetums* release."

"Okay. What's in it?"

"Stories X, Y, and Z. We've put a brief writeup of each in Jira.**"

(pause while we go look at the stories and estimate the test effort on each as well as the estimates on how many bugs will come through, etc.)

"All right, we're probably looking at 4 weeks between feature freeze and code freeze, and 2 weeks after code freeze for final verification."

And then time passes. We start coding tests, looking at early builds, etc. Development, well, develops.

Inevitably, something changes. We might get a new customer who really wants a feature, or we might find a bug that needs to be fixed for the next release, or we decide to drop a story that seemed important but turned out not to really matter. Net net, something changes. So we have another conversation:

"What's the new estimate for this release?"

"Umm.... is everything updated in Jira?"

"Yeah, except we're not doing story Y, and we're doing story W."

At this point the conversation needs to stop. We have a process. We have a place that defines what's in a release. Let's use it.

If you've defined a place to keep information - the contents of a release, bugs and what's been done on them, employee phone numbers - then that place should be kept up to date. It seems like a lot of work sometimes, to keep updating the information location. It's worth it, though. It saves a lot of other people time, and it works to train them to trust the information location. If it's up to date always, then people will come to trust it. You'll spend less time explaining status and more time working.

And less time on status and more time working sounds great to me.

* We name our releases after muppets where I work. This is Sweetums:

** We keep our stories in Jira so we can track them and do things like attach bugs to stories, etc.

An Error

2010-06-28T10:25:00.004-04:00

What's wrong with this test code?

def testFailureMode

my result = runMyGreatTestCommandHere

if result.error

puts "Test PASSED. We wanted to make sure it errored properly"

else

puts "Test FAILED. Expected to error."

end

On the surface, it's not so bad. It runs a test that should produce an error, and makes sure an error is produced. Let's dig one step deeper.

We're testing a failure mode, which means we're expecting not just any error, but an error we induced. For example, let's say we're testing login with a bad password, and want to see that there's an error "incorrect password". However, this test would pass if any other error occurred. If the page never loaded, or returned a stack trace instead of a login page, this test would pass just fine. After all, it got an error!

When you're testing failure modes, make sure not only that you get an error, but that you get the error you meant to get.

Reboot Heuristic

2010-06-25T11:45:00.006-04:00

Let's say you had to reboot your primary work computer, right now. How much would you lose?

How many windows do you have open with things half done?

How many emails partially written?

How many IMs to read? To respond to?

How many browser tabs with articles to read?

How many IDE windows with various code changes?

How many tests running in various terminals?

And now you have to reboot.

How many different tasks do you have to finish or at least get to a stopping spot before you could reboot?

Your primary workstation shows you how much of a multitasker you think you are. You're not really multitasking, though. You're simply not finishing things - maybe a lot of things.

I'm not particularly paper dependent - there are zero pieces of paper on my desk as I write this - so I use the reboot heuristic to remind myself to finish things. Ready?

If I have to touch more than three things before I would be willing to reboot, then I can't start something new until I've finished at least one.

The reboot heuristic is a reminder to myself that I need to finish things, not just start them. If I write an email and get partway through before I remember that test I wanted to start, that's okay. Repeat several times, though, and I'm spending more time switching between tasks than actually doing tasks. So three is my limit: when I'm doing more than three things, it's time to finish something.

Starting things is great. Finishing things is better. Multitasking is part of working these days, but not letting it get out of control - that's a heuristic I need.

On Telling Somebody

2010-06-23T18:25:00.000-04:00

Testing is about information. We do things to systems in order to get information about the system. From there, we make educated guesses about the systems likely behavior in the future. For example, I might test an installation on a certain hardware configuration, and then from there make statements about the installer's likely behavior on the same hardware configuration at a customer site. Such statements probably take the form, "yup, it'll work", and they can be made explicitly or they can be made implicitly ("Pass").

Testing can be broken down into those two parts:

doing something and seeing what happens
predicting future events

If you haven't done something and seen what happens, you haven't tested. If you haven't told someone about it, you haven't tested.

It's easy to get caught up in the doing, but don't forget the communication side of it. Tell someone - in a test plan, in an email, in a green bar in your IDE, in conversation. You're not done until you've told someone.

"Why" Is a Question

2010-06-22T08:45:00.000-04:00

"Why don't we test X?"

"Why do we wait to check in until the tests are done?"

All too often, we take these questions as a suggestion that we change our behavior. "Why don't you foo?" is something we take as "Do foo." "Why do we do bar?" is a question we take as "Don't do bar."

But it is a question, not a directive.

Sometimes the correct response really is, "No good reason. Let's change that." Sometimes, though, there is a reason why we do things. Maybe we don't test X because we're getting rid of that feature, or we don't have the hardware to do it. Maybe we do wait to check in until the tests are done to make sure our coverage stays high and we don't create technical debt.

Answer the question. Then change your behavior - but only if that's the right answer to the question.

Test Near Development

2010-06-17T09:13:00.004-04:00

Test Driven Development (TDD) is a great idea. You write some tests that say, basically, "this is what should happen." Then you write some code to make it so. It's really rather like waving your magic wand (or keyboard).

Repeat this often enough, the idea goes, and you have yourself a pretty good system, and a pretty good amount of code coverage so you can refactor safely.

All great things.

But.

TDD is hard. It's really difficult to do once you get beyond the first little bit. After all, as you keep adding and modifying tests, every change starts to take a little longer. Not much longer, but enough you notice and start to lose focus. And if it takes more than 8 seconds for something to happen - say, your tests to finish - you're likely to lose focus. (See David Cornish and Dianne Dukette: The Essential 20).

The benefits of TDD, however, accrue as long as the tests exist when the feature exists. This holds true no matter if the feature is a single method or an entire new component. The goal of having tests to enable safe refactoring, to guide the implementation, is still met. We're just not dogmatic about product code vs test code: both product and tests are required; get there in the way that works for you as an engineer.

So on our team we practice test near development. We don't force people to write their tests first. Instead, we ensure that tests exist when the feature is complete. In practice, some engineers write a little code, then write some tests, then refactor a little code, then refactor some tests, then write some more code. The last thing written is usually the acceptance tests. We end up in the same place, but we've allowed the tests to come before, during, and after the development effort. I've taken to calling this "Test Near Development".

And you know what? It works for us.

Give it a shot sometime, if you're having trouble with TDD. It might be a gateway to help get you to full TDD, or it might just give you better test coverage that your team can sustain.

Asking

2010-06-16T16:23:00.002-04:00

Software is a community phenomenon. For almost everything, no matter what you're doing, someone else out there is doing or has done something similar. Maybe they're not doing the same big-picture thing, but someone else is doing that small task you have before you now.

You can ask for help. And you're really very likely to get an answer, but only if you ask the right way.

Whether you ask your team a question, or send out a request to a mailing list, or hit up a support forum, or hop onto an IRC channel, you need to remember a few basic things:

Describe what you're trying to do. Give the user some context so they don't suggest something that makes no sense. Don't take more than about three sentences to do this, at most.
Indicate you've tried something. This does two things: (1) shows you're trying; and (2) prevents people from suggesting something you've tried.
Ask a question. Make sure you actually ask a question. Be as specific as possible in describing what you need to move forward.

Doing these things alone will greatly increase your chances of getting an answer. You've made it easier to understand your question, and therefore to answer it. Asking seems so easy, doesn't it?

No, We Shouldn't Document That

2010-06-14T06:15:00.003-04:00

"We should document that" is not an uncommon phrase. It usually means "something is a little weird and our users might take it the wrong way, or maybe they already have taken it wrong." It's a place where you're doing something your users will find confusing or non-intuitive. Consider these examples:

Our login page asks for "username" but we expect you to provide an email as your username. People keep clicking "Forgot username", so we should document that your username is the email you signed up with.
When patching the high-availability system, support must patch the passive half of the pair first. Patching the active half first leads to system downtime. We should document that for support.
We can register our system in DNS at installation time (go us!) but only if you provide the username and password of someone with permission to do so.

Every "we should document that" is a bug until proven otherwise.

If you have to document it, most of the time that means you've designed it or implemented it in a confusing way. So yes, please document it. Then log it as a bug and make it less confusing. Sometimes that's as simple as changing a label - "email" instead of "username". Sometimes it's harder, like making a high-availability configuration patchable from either half with transparent failover. Sometimes, it's even dependent on a third-party system, and involves moving the documentation into the product (e.g., "please enter the username and password for a user with permission to create DNS entries") rather than removing documentation entirely.

So no, we shouldn't document that. Instead, we should fix that. Let's quit with the post facto workarounds and start actually fixing the weirdness in our application up front.

Found Time

2010-06-10T17:39:00.001-04:00

Unfortunately, often things take longer than planned. A meeting runs over, a quick bug fix takes hours, a feature that should have taken a few days takes nearly a week. Sometimes, though, the inverse is true. That hour-long meeting breaks up in 15 minutes. The bug fix that looked hard turned out to be a simple one-line change (and was elegant, too!). The time you would have spent is found time, and found time is a gift.

So what do you do with your found time?

You can go straight on to the next thing - your next backlog item, the next bug, the next meeting.

But you don't have to.

I use found time to do the nagging little things. These are things that are so unimportant you won't get to them normally. Nonetheless, they're small bumps in my day, things that don't go quite as smoothly as I wish. These are things that when I'm sitting at my desk, I won't schedule them because there's so much more important to do. But they're things that I run into every day.

I had a few hours today of found time. I used them to:

Update a script that spit out information in rows, and I always wind up sending that information to other people in columns. I made it spit out columns. Total time saved: 30 seconds, tops, but boy was that annoying having to transpose it. No more transposing!
Upgrade all my Rails gems. I'm just starting on a new project, and using the most current ones will be useful later on.
Update the Abakas web site with some clearer services information.
Install Transmit 4, which is a lot faster than the previous version.

Not one of these things was hugely important. Now that I've done a few of them, though, my day will be a bit smoother, and I'll be better able to concentrate on the truly important things.

Found time is a free gift. It's unscheduled and its yours. Use some of it for doing the nagging little things. Think of it as spring cleaning for your day!

My Current Test Stack

2010-06-08T15:04:00.004-04:00

I've been doing some functional testing on web applications. The applications are relatively simple, written in Ruby on Rails, and provide the following user interfaces:

a GUI
SOAP and REST APIs

My current testing stack for functional tests is:

Rspec. I use this for testing models primarily, and some of the controllers.

Cucumber. This is where I do most of my testing of workflow, business logic, and integration between pieces. A lot of the tests take the form of "sign up through the GUI, register a machine through an API, upload user data through the API, view usage data through the GUI" (and variations thereon).

Selenium. I use this for the very few tests that cucumber can't handle because of JavaScript, rendering niceties, or other browser-type concerns.

I'm in the market to learn a new toy (I mean... tool!), so I'm turning it over to you, readers. What about this stack should I swap out and for what kinds of tests?

Analyze With Purpose

2010-06-07T11:32:00.001-04:00

Analysis is a double-edged sword. While in theory a quest for understanding is laudable, understanding rarely actually fixes a problem. In the end, our focus has to be on fixing a problem; analysis is one step on the way.

Analysis is a means to an end, not an end by itself.

Instead, we must perform analysis with an eye toward what we want to accomplish. Don't say, "we'd like to understand behavior X". Instead say, "the customer is seeing behavior X and wants behavior Y. Let's figure out how those two behaviors are different under the covers."

Don't say, "that unexpected performance improvement should be analyzed." Instead say, "we'd like to make sure that the performance increase we're seeing isn't due to measurement error, some change in system input we haven't noticed, or caused by some transient state."

In other words, don't simply set analysis or even understanding as a goal. Instead, set a goal of what you will do with that analysis and that understanding. That way you know when to stop analyzing and when to start doing.

"Can't"

2010-06-04T16:09:00.000-04:00

How many of us have heard this phrase?

"We can't do that!"

Or even better...

"You can't do that!"

Almost every time, the person (or us) is perfectly capable of doing that, whatever "that" is. What we really mean is, "please oh please don't do that!" or "I'd really rather not do that". We say "can't". We mean "won't".

The trouble is that "can't" turns it into a dare. "You can't do that!" "Oh yeah? Watch me!". Whoops.

Telling someone they can't do something often has two effects:

Makes you sound rather whiny
Makes the person more likely to do it if only to prove that yes, they can
Makes the person more defensive of his position

Take away the dare. Not "you can't do that!" but "here's why this should be done another way". You're much more likely to prevent "that" from happening (whatever it is) if you can provide an alternative rather than a dare.

CAST 2010

2010-06-03T09:43:00.004-04:00

I try not to do these shameless plugs too often, but I'm excited about this one.

I'm presenting a session at this year's CAST 2010 on Communicating with NonTesters. We'll talk about how to present information in a way that whoever you're talking to really understands it. Throughout the session we'll cover the dreaded "status report", the art of writing a defect explanation for a (probably angry) customer, and discuss how to write a risk assessment that doesn't make you look like you're panicking over nothing.

I'd love to see you there!

Here's the conference info:

Attend CAST 2010

The 5th annual Conference of the Association for Software Testing

August 2-4, 2010, Grand Rapids, Michigan, USA

"Skills in Testing"

About CAST

CAST reflects the AST's core mission: to build community amongst scholars, practitioners, and students for the advancement of the practice of software testing. In 2010, CAST aims to leverage peer collaboration to build an enhanced understanding of how various skills influence tester effectiveness.

CAST offers a unique opportunity to learn and confer with others that simply isn’t found at other conferences. Each scheduled session allocates time for facilitated “open season” discussions that encourage participants to question and challenge the presentation. What takes place in the hallways, at receptions, and during meals and lightning talks truly sets CAST apart; for many attendees, the greatest value is derived from the opportunity to discuss and delve into the topics that matter to them.

More information and Registration: www.CAST2010.org

Change Tactics

2010-06-02T08:43:00.004-04:00

Given a problem, there are generally several ways to approach it. The problem might be an issue occurring at a customer site, or a test we want to conduct, or a scaling design goal, etc. In any of these situations, we have lots of choices in how we tackle it. We might start by looking at what changed to cause the issue to occur. We might begin with reviewing the architecture of massively scaling services similar to ours. We might use code review to identify possible flaws that we want to exercise with unit testing code.

All those are great, and worth trying.

But what if it's not working?

Is that "what changed?" question you've been asking yourself about a problem that showed up at a client site not getting you any answers? Did that massively scalable architecture that seems like it would provide some good ideas lead you down a rathole of trying to map your square peg to their round role?

Time to change tactics. If it's not working, stop doing it, and try something else.

So how do we know when to change it up versus when to pursue a given tactic?

As a rule of thumb, I try to give no more than 20% of my estimate to any given tactic.

Let's say, for example, that I'm designing a test of a new module. If I estimated that the whole test would take me 5 days, and after one day I'm nowhere, it's time to rethink my tactics. If it's only been an hour or two, well, I might just not have taken it far enough yet or given the tactic enough time. If it's been two days and I'm still nowhere, I'm ratholing.

As with any heuristic, there are exceptions. Sometimes it's worth spending longer on a particularly tricky, slow or likely tactic. Just make that a conscious decision. About 20% of the way through your overall estimate, stop. Ask yourself if you're really getting anywhere with this tactic. If not, time to change.

We'd all like our first tactics to work consistently. Often, they will. For those times it won't, pull yourself out of the rathole and change tactics.

"Technical"

2010-05-28T16:30:00.000-04:00

"I'm looking for a technical tester." This is code.

This really means one of several things:

I want someone who can code (developer-lite)
I want someone who will create a robust automated test suite and tools to run it
I want someone who knows what a mouse is and has a prayer of understanding a command line.
I want someone who knows how to understand and manipulate a system, and to apply thoughts derived from computer science to a mental model of how it works.

These are all legitimate needs. It's up to you to figure out which one is your true "technical tester".

Testers who are primarily coders are great. They make wonderful tools engineers (really just a name for someone who writes code used by testers). They are great for testing APIs and other things where interaction with the system under test is done through code.

Testers who don't code much are fine. They can still think deeply about a system. Some of them are great with usability and user interface because they don't think about the code but about the system. Testers like this can also usually put together scripts and other helper code snippets when they need to as well.

Testers who don't code at all are fine. They're more limited in what they can do, but the tester who can't work on Linux is limited as well, just in a different way.

Don't tell me you need a technical tester. Tell me you need a tester who can do X, Y, and Z. Only if you're specific about your needs can I find you a tester who actually meets those needs.

I'd like to thank Matt Heusser for the inspiration for this post.