Quality Frog - Questioning Software

What is Performance?

2012-05-23T11:35:00.002-06:00

As we design and test for performance, let's look beyond speed. Let's look beyond basic stability. Let's look at the many facets of performance. Consider in which ways your system needs to perform. How fast does it need to be? How long does it need to keep running? Does it need to grow? Does it need to be available at all times? How much can we spend? Can we make it faster?

There are endless questions we could ask. Therefore, categorizing facets of performance and creating tests for each category can be helpful. However, let's not fail to look at the interaction between these categories.

OFAT (One Factor At a Time) testing (as exampled in the above performance testing checklist excerpt) often fails to provide information related to the interaction between the categories. Let's do some MFAT (Multiple Factors At a Time) testing and analysis. Let's look at the system as a whole. Let's mix it up. Let's consider how these facets interact. Let's create test scenarios that include multiple facets.

Time Machine: Devops, 45 Years Ago

2012-05-17T10:14:00.001-06:00

"Two major inputs are required

to provide computer service:

equipment and manpower.

For development,

employees such as

programmers and systems analysts are needed;

for operation,

the services of managers, operators, etc., are essential.

In practice the two activities usually coexist.

Some sort of computer installation

is required

to check out development efforts.

And almost every installation

engages in continuing development

as old systems are modified and new ones begun."

- William F. Sharpe, Economics of Computers, 1969

The above suggests that Devops is not at all a new concept. Forty-five years ago, computer rental cost slightly more than the Devops staff required to use it. Non-personnel operations cost typically rounded out the final third of the total cost. Today, hardware is cheap and people are expensive.

So, what did a Devops organization look like forty-five years ago? The table below presents some data gathered in a "nation-wide census of data processing personnel" in 1967.

William F. Sharpe, Economics of Computers, 1969

How do these wages add up in today's dollars?

Check out

The Co$t of Living.

Compared to my experience over the past couple decades, this seems to be heavy on management -- both in cost and percent of personnel. The roles of computer operator and librarian are pretty much gone. Programming and analyst roles have since blurred. Software has replaced many things that people used to do. As the user base has moved from specialist operators to the general public, new testing and user experience design roles have evolved.

What else has changed?

What do you think devops will look like forty-five years into the future?

The Right Stuff?

2012-05-16T15:00:00.000-06:00

You cannot be anything you want to be

-- but you can be a lot more of who you already are.

- Tom Rath, StrengthsFinder 2.0

Given that I resigned a couple weeks ago and am looking for new work, I've been asking myself a lot of questions about what I want to do next. Do I want to do detailed technical work? Do I want to lead people? Do I want to consult? Do I want to be someone's employee, or do I want to do short term contract work? Do I want to stay in Arizona, go back to Colorado, or go somewhere new? I've not yet decided. I am considering a number of options.

What I do know is that I want to find work that is a good match for my strengths and mindset. When looking for new work for myself, and when I've been involved in hiring, matching strengths and mindset has been more important to me than matching specific technical skills. People who are enabled to exercise their strengths and philosophy tend to have the intrinsic motivation needed to update their skills as each situation demands.

Rightshifting:

Improving the effectiveness

of knowledge-work businesses.

I am intrigued by the Rightshifting ideas coming from Bob Marshall, the Flowchain Sensei. The basic idea of the Rightshifting model is that the curve shown below indicates that the majority of organizations (peak of the curve) are ineffective compared to the few effective organizations on the right side of the curve. While organizations to the right of Bob's model tend to better match my personal mindset, finding work with such organizations is difficult -- there aren't many of them. My experience suggests that organizations further to the left can better benefit from what I have to offer than organizations to the right. However, if such organizations have no desire to improve their effectiveness, I am likely to get frustrated. Therefore, one more question I am asking myself is whether I prefer to be more of a hero in the midst of analytical dysfunction or more of a team member in a synergistic organization. I suspect my place is somewhere in the middle -- with an organization that is shifting right.

Side Note: One way in which I think I may disagree with Bob Marshall's model is the assertion that the chaotic organizations on the left are less productive than the analytic organizations to their right. I suspect many of the majority (the analytical organizations) are just better at measuring (or pretending to measure) their productivity than the organizations to their left. What we label chaos is often order we don't yet understand well enough to describe. I'm also not convinced that organizations need to take a trip through the dehumanizing methods of many analytical organizations in order to become more productive.

I recently watched a video in which Bob Marshall recommended the Strengths Finder book and the associated online assessment as a useful tool in discovering your strengths. I am typically skeptical of such assessments. I fear they tend to provoke the Forer effect in which people tend to demonstrate a confirmation bias and attribute accuracy to generalized information about themselves when it is presented as being specifically tailored for them. However, I decided to give Strengths Finder a try. Despite my reservations and desire to critically evaluate the timed questionnaire as I completed it, I found the resulting top "themes" to be a surprisingly accurate description of my strengths. Perhaps this only demonstrates that even a skeptic like me can succumb to the Forer effect.

According to Strengths Finder, my top five strengths are:

Ideation
Command
Activator
Individualization
Relator

Whether this is based on a meaningful algorithm or is total BS, I found the results useful. (All models are wrong, but some are useful.) These five strengths seem to example what I believe can make a good context-driven tester; but then, I'm biased. :)

So, back to what do I want in my next job or independent business venture: I want to exercise my strengths in an environment where I don't have to suffer cognitive dissonance in order to serve my employer or clients.

In thinking about this, I created a mind map of the strengths from Strengths Finder and things I like, as well as things I don't like. Following up on my desire to find a mindset match over simply matching skills, I've decided to ignore typical job hunting advice and share this.

Think I might be a match for your organization? Hire me. :)

Hire Ben Simo!

2012-05-03T17:49:00.000-06:00

[Updated]

I resigned this week.

Therefore, I am now available and looking for new testing work.

I prefer to stay in the Phoenix, Arizona, area but may be willing to move if the opportunity is right. (I dislike the disruption of moving more than I dislike the idea of living elsewhere.)

I am an amphibious time-traveling context-driven cyborg software tester.

What does this mean?

I am amphibious. Although I am a software tester first, I’ve spent much of my career straddling roles and environments. It is common for testers to view me as a very technical person while programmers view me as one more focused on value. I check existing beliefs and test to find new information. I script and I explore. I test to help understand whether the right thing is being done and I check to help verify that what is being done is being done correctly. I've often found myself playing the role of liaison between techies and regular people -- helping disparate stakeholders understand one another.

I am a time-traveler. However, my time machine only goes forward. Over 20 years ago, I fell into a testing role with the illustrious title of “data collector”. I was one of a few hired to execute test cases created by others and report the results. However, from day one, things didn’t go as planned. Executing the test cases required deeper understanding than was communicated in the documentation. I quickly learned that testing can be fun and mentally challenging work. In these past 20 years, I've seen many technologies and practices come and go, and then come back again. One thing remains constant: the need to test whatever it is that us fallible humans try to create in the virtual worlds inside our machines -- virtual worlds that have real impact on real people in the real world.

I am context driven. This means I reject the concept of best practices. Instead, I try to first understand, and then select practices that fit each situation. I adapt my testing techniques and practices to the context. While my context-driven worldview is often at odds with those of others with whom I work, I recognize that people are the most important part of any context, and therefore try to find ways that I can contribute within whatever environment I find myself.

I am a cyborg. I have both human and artificial parts. No, I am not referring to the hardware used to reconstruct my wrist last year. I like to practice sapient brain-engaged testing and enhance it with automation. Rather than try to replace sapient testing, I use automation to enhance and extend my capabilities as a tester.

I am a tester. Most of all, I am a tester. While I have many other skills, I generally use them to support my testing and help others test well. I like to develop software that helps solve testing problems, but I don't want to spend the majority of my time with code. I like people more than I like code. I enjoy getting to know stakeholders and learning what matters to them. I enjoy exploring software in search of threats to value. I enjoy communicating and helping apply my discoveries to make things better.

Want to know more about me?

Check out my profile on Linked In.
Read this blog.
Read my interview with UTest: Part 1, Part 2, Part 3.
Follow me on Twitter.
Send me an email.
Check out the experimental mind map below. This highlights some of my experience; and also contains a bunch of details that I typically think of as resume clutter.

Target Tested ✔

2012-05-03T00:27:00.000-06:00

Computers empower us to create and manipulate things in ways that don't produce quite what we intended. This requires testing for what we don't expect.

There is a problem with this advertisement from Target Australia. Can you spot it?

Do the math!

2011-04-17T11:24:00.000-06:00

Sometimes numbers that make no sense are presented in ways that seem to make sense... or not. :)

uTest Interview

2010-08-25T16:47:00.004-06:00

uTest invited me to be interviewed for Testing the Limits series this month. They have graciously referred to me as "one of the most insightful and entertaining testers in the business". Please give the interview a read. And if you have additional questions, please send them my way.

Part I. Worst bug, testing philosophy, defensive pessimism, certifications
Part II. Trusting automation, why Quality Assurance is a bad title, overstructured managemenet
Part III. How I got into testing, what would I be doing if there were no such thing as software

What Sesame Street taught me about problem solving

2010-02-19T17:13:00.000-07:00

A checklist for buying a new computer

2010-02-07T15:16:00.000-07:00

Buying a computer has changed significantly in the past 25 years. Or not?

From Melbourne's The Age, 18 June 1984

Some features that are really important to me:

on-off switching (the most important feature)
security against blow-up (because I fear burns and shrapnel from exploding computers)
position of reset key (a reset key next to the shift key would be a pain)
big yellow bus (aren't computers good for education?)
radiation shielding (don't want to be sterilized while playing Pole Position)
movable screen (I might want to take the screen home with me)
ease of plug-in ROM installation (cartridges are better than tapes)
recognize commands in u/l case (I expect the computer to understand me -- EVEN WHEN I YELL AT IT!)
hard disc option (I prefer to save data to tape, but a hard disc would be a nice option)
stand-alone capability of slaves (I want slaves that can work without being micro-managed)
octave range (I need a computer that can sing soprano)
RS-232C port (In case I want to interface it to my lawn sprinkler system)
Paddles (to spank the computer when it misbehaves)
key-word dictionary (to understand this check list)
country of origin (can I get a machine made in the USA?)
dynamic debugging tool (certainly don't want a static debugging tool)
Logo (because I just love that little turtle)

Wait != Hesitate

2010-01-07T06:19:00.000-07:00

Metrics from where the sun don't shine

2009-12-06T15:20:00.000-07:00

Numbers don't lie, people do.

2009-12-05T15:23:00.000-07:00

“despite its mathematical base,
statistics is as much an art as it is a science.
… Often the statistician must choose among methods,
a subjective process,
and find the one
that he will use to represent the facts.
This suggests giving statistical material …
a very sharp second look
before accepting any of them. …
But arbitrarily rejecting statistical methods
makes no sense either.

That is like refusing to read
because writers sometimes
use words to hide facts and relationships
rather than to reveal them.”

- Darrell Huff,
How To Lie With Statistics, 1954

“No doubt
some graphics do distort
the underlying data,
making it hard for the viewer
to learn the truth.
But data graphics are no different
from words in this regard,
for any means of communication
can be used to deceive.”

- Edward Tufte,
The Visual Display of Quantitative Information

Talkin' 'bout test in a differrent light

2009-11-16T12:33:00.006-07:00

Pullin' out my big black book
Cause when I need a word defined that's where I look
So I move to the L's quick, fast, in a hurry
Threw on my specs, thought my vision was blurry
I look again but to my dismay
It was black and white with no room for grey
Ya see, a big "V" stood beyond my word
And yo that's when it hit me, that luv is a verb.

Words come easy but they don't mean much
When the words they're sayin' we can put trust in
We're talkin' 'bout love in a different light
And if we all learn to love it would be just right.

- DC Talk

The DC Talk song "Luv is a Verb" points out that love is something to be acted out. Real love is action, not just words and feelings. Love is expressed through action.

Like love, the word test is both a noun and a verb. Also like love, test requires action. Even the noun definitions for test describe action.

test

noun

trying something out to find out about it

any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality, etc.

a set of questions or exercises evaluating skill or knowledge

the act of undergoing testing

the act of testing something

verb

put to the test, as for its quality, or give experimental use to

test or examine for the presence of disease or infection

examine someone's knowledge of something

show a certain characteristic when tested

achieve a certain score or rating on a test

determine the presence or properties of (a substance)

undergo a test

from Princeton WordNet

While I don't think we'll find anyone that argues that test is not a verb, people involved in software development seem to use it primarily as a noun. I have nothing against the many things we create that we call tests. Our test cases, test code, test charters, and whatever test things we create can be useful tools -- but they are not the test.

Let's think about test in a different light.

Several years ago, Shrini Kulkarni challenged me in questioning whether there can be such a thing as an automated test. I don't think I disagreed with Shrini. I've not been one to trust testing to machines, but I've been a fan of automation throughout my testing career. I've automated many testing tasks, but not believed I can automate the testing itself.

Earlier this year, Michael Bolton told me of a distinction he was thinking about between checking and testing. While I had no disagreement with this distinction, I wasn't thrilled with the terms. I wanted something more descriptive. I thought Michael was making a distinction I had been trying to make: a distinction between validation and investigation. I've since come to understand that Michael is making a slightly different distinction. Michael has recently written a series of blog posts better describing the Checking vs. Testing distinction. Michael has limited the scope of checking to observations and decisions rules that can be executed without sapience -- without a brain-engaged human. If something requires human sapience, it is testing, not checking.

Yesterday, an insightful tester, Lanette Cream, made a nice attempt at defining test on her blog. In her latest revision, she defines test as follows.

A test is
an action
which produces discoveries
that can be used to evaluate product quality.

I like that this definition identifies action, discovery, and evaluation as being core to testing. However, I'm thinking of pushing, or rather constraining, this just a bit further.

What if we were to say that the evaluation is the action and discovery is the goal?

A test would then be the sapient part of validation or investigation -- the thinking and learning that cannot be automated. All those other things we do to test are really support activities that help us evaluate.

Test is not a document. Test is not code. Test is not executing a program. Test is not applying a procedural decision rule. Test is not anything that can be done by a machine. Test is the act of evaluating that requires sapience.

Test is thinking and learning that leads to discovery. We may test by evaluating existing data. We may test by running experiments that produce new data. We may take the output of automated checks to test. We may provide what we learn as input to coding new automated checks. The test is the action we perform in our minds.

This may come across as nitpicking vocabulary. That's not my intent. My goal is not to limit anyone's definition of test, but rather to shed a different light on what I believe sets testing apart from checking, and gives both checking and testing value.

If a check fails in a forest and no one is around to hear it, does it make a sound?

The true value of our checking and testing is in the mind of a sapient tester. What value is there in all the things we call checks and tests without a tester (whatever their role or title) evaluating information and learning?

Test is sapient evaluation that leads to discovery

I'm not quite comfortable with this. I want the emphasis to be on the sapient activity; and not generating and collecting data to support the thinking without ignoring that it is a necessary part of testing.

Regardless of where we shine the light or draw lines, let's keep in mind that test is a verb.

What do you think? Testing of my half-baked ideas is welcome and appreciated.

Comprehensive Understanding?

2009-11-12T22:52:00.000-07:00

"Anyone who feels
that he
can precisely define
the boundaries of his profession
either
possesses a skill of little importance
or
is incredibly naive."

- William F. Sharpe,
The Economics of Computers, 1969

Marginal value & cost of software

2009-11-12T08:31:00.000-07:00

* Image from the Economics of Computers, by William F. Sharpe.

Cost of additional copies approaches zero ONLY IF no maintenance or customer support costs are associated with sales of additional copies.

If your quality stinks, expect that supposedly 99% profit copy you sold me to increase your costs.

Quality improvement that reduces maintenance and support costs can have great value!

If scripted tests can't find bugs...

2009-11-11T17:33:00.000-07:00

Shrini Kulkarni (@shrinik) tweeted that a friend told him if exploratory testing finds bugs not found by scripted testing, it may be due to insufficient or incorrect test planning and review.

Perhaps there aren't enough test cases. Perhaps testing techniques weren't applied properly. Perhaps the review of tests was done wrong.

However, perhaps -- just perhaps -- people are fallible and can't completely understand or work through the complexity of their customer's problems and the technical implementation of a solution with finite knowledge, finances, and time.

If it is reasonable to expect testers to design enough tests to exercise near-infinite paths with near-infinite data variations in a software system, then I think it should be reasonable to expect software designers and coders to anticipate everything that could go wrong and prevent all threats to the value of the software they produce -- all with finite finances and time.

If this were reasonable, then it would be reasonable to skip testing altogether.

In the real world, it is just as unreasonable to expect perfection from test case designers as it is to expect perfection from all the other people involved in developing software. Is it not?

One of the things exploratory testing helps avoid is locking in our level of ignorance that exists at the start. An explorer can use information gained as they explore to help guide where they go and what they do next.

So you think you've achieved maturity?

2009-11-11T00:35:00.000-07:00

Some people say that Ada may be the last major high level language that will ever be developed, since automatic program generation techniques may be available in the not too distant future. Thus it seems fitting that the last major programming language be named in honor of the first female prorammer.

- Richard Wiener and Richard Sincovec,

Programming in Ada, 1983

Arrogance combined with foolish optimism? :)

Freedom & Responsibility Culture

2009-08-05T00:37:00.000-06:00

Cool. Based on this slide deck, it appears that Netflix has a good understanding of developing and sustaining a corporate culture of Freedom & Responsibility.

Culture

View more presentations from reed2001.

F.A.I.L.U.R.E.

2009-05-07T00:40:00.000-06:00

F.A.I.L.U.R.E.

A mnemonic for testing error handling & reporting.

F.A.I.L.U.R.E.

View more presentations from Ben Simo.

Is There A Problem Here?

2009-02-15T18:12:00.005-07:00

Over the years, I have collected a number of examples of software failures in the wild -- some I've encountered myself, some were shared by others. I've had intentions to create a blog for sharing these software failures, and new ones as they are discovered, with hope that software designers, developers, and testers can discuss and learn from them. I have finally launched that blog. It is titled Is There A Problem Here?

I invite you to visit the blog and contribute at http://IsThereAProblemHere.com.

I'm helping you. I'm helping you.

2009-01-05T22:56:00.009-07:00

A few months ago, I enlisted my 11 year old son to help me with some work around the house. After a short while, he was doing something other than what I had asked him to do.

I told him, "You're not helping me."

"But I am helping you.", he replied.

"No you're not."

"I'm helping you. I'm helping you.", he shot back. He was frustrated. He really thought he was helping me; and I was putting down his work. I was frustrated too. From my view, his helping was creating more work for me. I did not feel helped.

Then it hit me. I've heard this argument before -- from software testers.

I've seen testers, and test managers, attempt to justify their work by telling team members and stakeholders "I'm helping you. I'm helping you." We QA and tester people develop metrics and reports to help us demonstrate how helpful we are. We talk about our quality assurance and testing processes. We talk about all the test cases we develop and execute. We like to show off our test automation that spits out impressive color-coded results.

However, we still encounter unhappy team members and stakeholders. We develop adversarial relationships with developers. We have to explain ourselves to project leads that question the value of our testing. We hear people tell us we're not helping and we keep saying "I'm helping you. I'm helping you."

Maybe, just like my son, we're not giving our stakeholders what they need. Maybe we aren't really helping. So instead of shooting back the "I'm helping you." line, we can stop and listen. Find out what our stakeholders want from us. Listen and ask clarifying questions to better understand how we can help.

I'm not advocating that we just give in and do whatever we're asked without defending our positions. However, we can be willing to adjust our positions to better serve our stakeholders. (Joining an overly optimistic rush to release poor quality software usually doesn't serve them.) If there is disagreement, work to resolve it. Sometimes we may need to educate others on our areas of expertise. Yet we testers also need to respect others' roles and expertise. Listen and learn.

So, the next time you feel like screaming "I'm helping you. I'm helping you.", try to better understand how you can help before turning up your defenses.

Serve your stakeholders.

The Antonym of Testing

2008-09-30T22:23:00.002-06:00

"... one usually encounters a definition such as, 'Testing is the process of confirming that a program is correct. It is the demonstration that errors are not present.' The main trouble with this definition is that it is totally wrong; in fact, it almost defines the antonym of testing."

- Glenford Myers,
Software Reliability: Principles & Practices, 1976

People keep telling me that testing is a validation activity -- that the purpose of testing is to validate that the software meets all the specifications, has no errors, meets performance SLAs, meets expectations of anonymous users, or some other lofty goal.

I read about testing processes designed to validate software. I use testing tools built to support validation. I listen to service companies pitch testing services to validate software. I read about testing metrics built on the assertion that software systems can be proved correct. I attend testing presentations explaining the presenters' best practices for validation.

The trouble is that we cannot prove software correct. We cannot prove the absence of bugs. We cannot test every possible state and input. We cannot evaluate every possible output. We cannot fully understand the desires of stakeholders. We cannot prove that customers will be happy. We cannot prove that a software product will solve the problems it was built to solve. If all this were possible, I suspect insurance companies would find a way to make a profit selling software quality insurance.

"If you think you can fully test a program without testing its response to every possible input, fine. Give us a list of your test cases. We can write a program that will pass all your tests but still fail spectacularly on an input you missed. If we can do this deliberately, our contention is that we or other programmers can do it accidentally."

- Cem Kaner, Jack Falk, and Hung Quoc Nguyen,
Testing Computer Software, Second Edition, 1999

Now, thirty-two years since Glenford Myers called testing to prove correctness the opposite of testing, we're surrounded by testing practices and tools based on proving correctness. The myth of proving correctness is alive and well.

Activities designed to try to prove correctness are the antonym of testing.

So if testing is not validation, what is testing? Testing is investigation; and communicating useful information about quality to decision makers.

"Testing is the process by which we explore and understand the status of the benefits and the risk associated with release of a software system."

- James Bach,
James Bach on Risk-Based Testing, STQE Magazine, Nov 1999

"Testing is done to find information. Critical decisions about the project or the product are made on the basis of that information."

- Cem Kaner, James Bach, Bret Pettichord,
Lessons Learned In Software Testing: A Context-Driven Approach, 2002

"A software tester’s job is to test software, find bugs, and report them so that they can be fixed. An effective software tester focuses on the software product itself and gathers empirical information regarding what it does and doesn’t do. This is a big job all by itself. The challenge is to provide accurate, comprehensive, and timely information, so managers can make informed decisions."

- Brett Pettichord,
Don't Become the Quality Police, StickyMinds.com, 2002

Once we admit that we cannot prove the software correct, we can refocus our efforts on finding useful quality-related information. Instead of pretending to assure quality or validate correctness, we can gather and communicate useful information. Investigate the software. Find information about threats to the quality of the systems under investigation. Communicate that information in terms that matter to stakeholders. Help managers make informed decisions.

Pause at the Pump

2008-07-26T22:34:00.007-06:00

My fuel gauge is on empty.

I don't want to stop, but pull into the gas station.

I tell the children to stay in the car.

I get out of the car.

The children are asking me questions from inside the car that I can't hear well enough to understand.

I swipe my credit card in the pump's card reader.

The pump responds by prompting me to "SELECT WINDOW OR OUTSIDE".

The children are still talking to me. I still don't understand.

I pause and stare at the keypad.

Is there a problem here?

I pause to think for a moment.

I hear the children asking me questions that I still don't understand through the closed car windows.

I scan the keypad again.

I cant find the "OUTSIDE" button.

Recognizing Bugs

At CAST last week, Pradeep Soundararajan gave a Lighting Talk about the importance of testers being able to recognize a bug. Tests may be of little use if the tester doesn't recognize the bugs triggered by the test.

Sometimes bugs are obvious. Sometimes bugs are not clearly violations of requirements documents. This is especially true when it comes to human computer interaction problems.

Requirements are not always clear and objective.

So, do you recognize why I may have paused when I read the prompt on the gasoline pump? It wasn't the price of the $4 per gallon fuel. It wasn't because I had to think about how I wanted to pay.

I paused because I didn't see a button labeled "OUTSIDE". Plus, I already swiped my credit card indicating that I wanted to pay at the pump. And even if I were to pay at the cashier window, I would still be outside.

I wonder how many minutes are wasted each month prompting customers to select where they want to pay after they have swiped their credit card. I wonder how many other people pause and read twice in search of the button to indicate that they want to pay outside.

The developers and testers of the software in this pump may have not recognized this problem. Maybe the makers executed test scripts -- either automated or manual -- and were blind to the problem. Or maybe they didn't deem it important enough to change.

Sometimes familiarity with the technical details of a system can hide problems that are obvious to those that don't know the technology, the requirements documents, and the test scripts. As testers it is important that we be careful not to let our familiarity with a system make us blind to to bugs -- things that bug our users.

Will you recognize a problem if you see it?

Images of CAST 2008

2008-07-19T10:18:00.007-06:00

The third annual Conference of the Association for Software Testing (CAST) happened this past week in Toronto, Ontario. This was not your typical sit-down, shut-up, and listen conference. Everywhere I looked, I saw testers conferring -- and that makes a good conference.

Day 1 - Tutorials

Day 2 - Conference

Day 3 - Conference

Announcing FROSST 1 - Front Range Open Space Software Testing Conference

2008-07-12T08:14:00.006-06:00

ThinkTalkTest, Ltd., co-founded by Ben Simo and Heidi Harmes-Campbell, is pleased to announce the first FROSST (Front Range Open Space Software Testing) conference.

FROSST 1 will be held on October 10th and 11th in Westminster, Colorado. The conference will be held on a Friday evening and all day Saturday.

The purpose of this conference is to facilitate collaboration amongst software testers in Colorado and beyond. Invite your friends and colleagues.

FROSST is free to all participants. Anyone with something to share or learn about software testing is welcome. Participation is limited to the first 150 testers that sign up. Sign up now to claim a spot.

FROSST is an open space conference. This means that the participants will create the agenda for the conference. This will be done leading up to the event through a yahoo group and discussions the first evening of the conference. If you have something to present, propose it. If there is something you want to learn, ask about it. This is a participatory conference.

Please see www.FROSSTCON.com for more information. Join the FROSSTCON Yahoo Group for the latest information and to participate in planning for the event.