My Technology Fetish

NRG Global Test Competition Retrospective

2013-05-07T23:49:00.003-07:00

Roughly two and a half weeks ago I competed in the first NRG Global Test Competition. The idea behind the competition was simple: get a bunch of people/ teams together to test a few products, split the competition into two days, one with functional testing and another with performance testing, and based on the reports submitted judges would award points and announce winners. The full details are available here and here.

This was the first online testing competition I'd tried but thanks to my experiences with testing challenges and rapid testing online I knew I'd have fun once I got past the quirks. By quirks I mean it can take time to get comfortable with the discussion format, figure out how to ask questions, how best to communicate with my fellow team members, etc. The competition took place at 10 am Eastern which sucks if you live on the west coast and have to wake up before 7 am like I did. It was all for fun anyways.

For as early in the morning and as new as the competition was I think I did reasonable. Not great, not even good, but reasonable. I think the best way to phrase it is: I'm not happy with my work. (I might be overly critical here but still.) Now we only had 3 hours from introduction of the products under test, to learn the product, ask questions of the "owner", test it, ask more questions, file bugs and write a report. Yet when I think back at what we turned in I'm not happy with it. Let me explain.

My team member and I barely communicated with one another. We were using Skype but we didn't do much planning ahead of time (not like we could have because nothing was public) so when it came time for the competition it was a simple "hi", "what are you working on" and "I'll look at x". That was it. We each went to different applications. Thinking back on it now I think we would have done much better if we were on the same application, talking to one other about what we were seeing. My experience has been any collaboration no matter how small results in finding and learning amazing things.

At the time of competition I considered using Bach's HTSM to map out the application but didn't. I wish I had. Even though its a bit detailed and were we on a short deadline I think heuristics would have lead me to think about and discover even more potential problems. At the very least I'd feel more confident in what I had tested.

It took me an hour or so to really get started looking at the products, deciding which one to test and with what equipment (iPad), browsers, etc. I'm still blaming the time of day. I started with some simple touring of this "home built" application that must have been built specifically for the competition because it was really simple and full of problems. Even though I saw lots of problems initially I took note and kept searching until I felt I had covered the entire application as best as I could. Then I circled back, asked a few questions from the "product owner" Matt Heusser and began testing the problems I saw. By that time I had just enough time to get my bugs written up in the tracking system (I had maybe 5) and started working on our Test Report. I think we got our report in right at the deadline.

I wasn't able to commit to the second part of the competition, for performance testing and I'm not sure if my team member was able to either. I knew going in I couldn't commit time for it however I'm still holding on to hope that I'll get a chance to play with the AppLoader tool. Despite my displeasure with my performance I'm glad I joined, in fact I'm looking forward to getting the feedback on how other's think we did. =)

First principle reasoning

2013-04-03T08:06:00.000-07:00

When I was young I remember wanting to be an awesome football player like Joe Montana, or an FBI agent working on the X-Files like Fox Mulder. These days I want to have the skills to identify and solve problems like Elon Musk.

Musk is an interesting person. He’s created and sold numerous companies and with the profits he’s created a rocket building / space exploration company that is now the first (private) company to make it to space. He’s also built an American electric car company. While all these things make Musk an interesting person on the surface, its his approach that makes him enviable.

In his TED talk Musk credits his training in physics with his ability to see and understand difficult problems. He says physics is about how to discover new things that might seem counterintuitive and physics first principle provides a good framework for thinking. The video is a good conversation between Elon Musk and TED curator Chris Anderson, I recommend watching it. Musk mentions first principle reasoning at about 19:37:

According to Wikipedia first principle in physics means you start directly at the lowest levels – at the laws. Musk provides a slightly easier understanding saying first principle is reasoning from the ground up as opposed to reasoning by analogy, which is copying what other people do with slight variations.

Musk further elaborates on first principle reasoning in this video. To summarize he says you look at the fundamentals of things, make sense of it, construct your reasoning and conclusion and (if possible) compare that to whatever is the current understanding. A part of that process involves questioning conclusions, asking whether or not something could be true. Sounds like Musk’s constantly modeling, learning, testing and re-modeling.

In thinking about my job there seems to be more reasoning by analogy than perhaps there should be (or at least its obvious to someone new). Whenever one of my developers or I ask why something is, why some conclusion has been reached – the typical response is “that’s how its done”. If I ask my test team why they do something a certain way its always “that’s how we’ve always done it” and there seems to be no desire (at least I haven't seen it yet) to know whether something makes sense or is based on a real understanding of the problem. Perhaps there should be more modeling, learning and testing?

We all do some reasoning by analogy, in many ways it’s a much simpler way to communicate and learn but for many of us in the software engineering fields (testers and developers) perhaps we confuse reasoning methods? So how do we determine when we need to use first principles and when its ok to use analogy in reasoning? That’s the million-dollar question. I think we do like Musk: create a model, ask questions to help us learn, test and when we aren’t satisfied with the answer, we reason from the ground up.

Low and High Intensity Learning

2013-03-24T22:22:00.001-07:00

Paul Graham in his essay Wealth says startups are a way of compressing a whole working life into a few years. You work at a very high intensity for a short period of time (say four years) instead of the normal low intensity for a long period of time (say forty years), in other words startups are a way of increasing your productivity exponentially. In my experience there is a correlation between high intensity working and high intensity learning.

The potential relationships between high-intensity work and learning have a lot of appeal because it provides a chance to leapfrog our understanding of several domains in a short period of time.

In his essay Startup = Growth Graham defines a startup as “… a company designed to grow fast". My last company was small, we considered ourselves a startup (although according to Graham’s definition we were not) but we worked considerably faster (higher intensity) than a larger company would have. I can say that with some certainty now because a 15,000-person company acquired our small (maybe 10-person) company and the differences are pretty dramatic.

To be fair there is a difference between the learning that occurs when a person is working for a high intensity company as opposed to doing their own high intensity work. With a high intensity company people learn whatever they have to in order to solve the problems in front of them, then they move on. A person working intensely on their own has the freedom to focus on what they want but they run the risk of never finding focus.

I’m struggling with the second part. I’m back to a low-intensity company but I don’t want to be pulled into a low-intensity learning situation. Part of me says that won’t happen because of my own internal drive but another part of me is worried the low-intensity rhythm of the company will make it hard to find focus. (I’m not saying my small company was a good example of a high intensity work / learning environment but as of right now it seems better than the corporate world.)

One solution is to create my own startup – something designed to grow fast which would force fast learning. It would be amazing for many reasons including getting back to a higher-intensity learning situation but I don’t know where to start. Another solution is to find a person or person(s) with the same interests or goals as I have and work together to learn. Maybe the small team size would have the effect of pushing each other into a higher intensity work and learning environment? Now where do I begin?

Export iBooks to your PC for reading

2013-03-12T23:13:00.001-07:00

As I consider switching from an iPad to a Nexus 7, one of the main things I do on my iPad is read books. I try not to purchase media, like Books and Music, through iTunes because Apple is overly restrictive in terms of DRM and accessibility - why isn't there an iBooks Windows or Mac app? Naturally I want to migrate the books I've purchased to a format I can read on another device. (It's a shame this option isn't provided).

So how do you export iBooks purchases to your PC? It's quite simple:

In iTunes, go to your purchased items
Click Books
Download the books you'd like

The downloaded content will be dropped into the default iTunes folder which for me was Music\iTunes\iTunes Media\Books; you can then browse your downloaded book(s) and grab the .epub file. If, like me, you are considering moving to a new device you can then use Calibre to change the format from a .epub to a .mobile or PDF.

Naturally some of the books I've purchased have DRM on them so you'll have to work around that. Personally I find it quite frustrating to spend more than $10 on an ebook only to have it locked into one format so I scoured the net and found a way to remove the DRM from the books I've purchased. If you are interested in doing the same this worked for me.

With my iBooks purchases transferred into .mobi formats I can read them on my desktop, laptop and other mobile devices.

Taking the RTI Online

2013-03-03T12:27:00.001-08:00

In February I attended an online training course where participants test a software product using the Rapid Testing methodology called Rapid Testing Intensive (RTI) Online taught by James Bach. I found it to be a great way to test a product, get feedback on your work, build a software testing portfolio and learn about the Rapid Testing methodology.

Last July I took a similar in person training course appropriately called Rapid Testing Intensive Onsite. I meant to write about my experience but never did so allow me to describe it now:

The onsite version was an intense four and a half days of lecture, learning, testing and other team activities. From survey testing, to group stand up presentations, to the occasional after hours (with beer in hand) dice game it was a week of mental challenges with quite a bit of fun mixed in. After some encouragement from a few twitterers I shared my notes in the form of a live blog: (day 1, day 2, day 3, day 4, day 5).

During that time I was the “lone tester” in my company and taking a week to work on a team with other testers from around the world was a welcome change and an enjoyable experience. When a team member found something interesting or became confused the rest of the team became involved in the discussions which lead to new ideas about where to test. If someone didn't clearly understand something someone else in the group could help. All this team work lead to some exciting discoveries.

Coming back to my original story I can do a little bit of comparison:

The online version was much more concentrated than its onsite counterpart but maintained all of the important aspects with an opening lecture, followed by a 90 minute testing assignment and then a break. During the break James would do the assignment himself and when the participants were back he’d show what he did, explain his approach and then review the work of those who were brave enough to ask for it.

Asking for your work to be reviewed live (webcast) can be a little intimidating. I remember doing the stand up presentation for my group onsite and I made a lot of mistakes in front of others but the result was I learned about safety language. This time around I knew mistakes would mean I’d learn more so I didn't hesitate to ask. After the feedback session there was a little more lecture and then the day is done. This was the schedule for the rest of the course except with each day the assignments build off of one another and get deeper into the product.

Having essentially taking this course twice (first onsite, second online) my approach was to focus on understanding the assignments while also building examples of my work that I would be proud to share afterwards. I used James Bach’s Heuristic Test Strategy Model to generate ideas for covering the product we were testing that would go into my Product Coverage Outline. Through both experiences I got new inspiration / ideas on how to organize and document my deep / combination testing. I played around with the way I take notes and continued to experiment with a method James calls concise documentation (however I prefer the catchier term “lean documentation”) which means there should be no fluff in your documentation, just the important parts.

Perhaps the most visible output from the RTI online are the work examples you build while documenting your testing – the Product Coverage Outline, deep testing matrices, testing notes, etc. These documents, after some cleaning and framing, will become the basis for my software testing portfolio – a public example of the work I've done and am capable of doing.

Not so visible are the truly important parts from the RTI online: learning about testing and deliberate practice. After all you can only get better at something if you practice it. Rapid Testing teaches us how productive and exciting testing can be by focusing on personal skill and the thinking part of testing. In my experience testing has always been about documentation (think test cases) and not about thinking / questioning what needs to be done. Every time I've taken a course in modern software testing practices (like RST) I feel like I start to understand more of the things that effect what I do and I start to question the assumptions involve. With the RTI I get to learn and practice which makes me feel like I’m growing and getting better.

I've found the product of those learning experiences, of the direct feedback to my work has improved how I test at work and how I view my product and the value of my labor. I’m not satisfied with where I’m at now, I might never be, but I do understand how far I've come and how far I have to go. See you at the next RTI.

Creating value by writing

2013-02-05T21:56:00.002-08:00

Different perspectives help us model problems and ideas in new and sometimes exciting ways. They can lead us to new evidence or help to re-engage our thinking about things we've taken for granted. Things like testing software or writing.

Writing isn't easy but I find it helps clarify my thoughts. Unfortunately putting those thoughts out for public scrutiny sometimes has the effect of keeping me from publishing something that is less than perfect. I know it won't be perfect but I drag my heels anyways. It doesn't help that a few of the blogs I follow and admire are written either by PhD's who write at great length and depth or who are just good writers... But I digress.

A new perspective on writing is just what I got from Steve Corona: writing creates value. It switches us from the consumer to the producer of information, information that can be shared with others or used by us years later. Here are the last few paragraphs of his post on Creating Value by Writing:

The biggest dividends paid from my writing has been the feeling of creating, becoming a producer of information, instead of continuing down the path as a chronic consumer. When I publish my thoughts, I create tangible value that can be shared with others.

And I never run out of things to write about. By keeping my idea muscle strong and writing down 10 brand new ideas everyday, I’ve amassed two entire notebooks filled with topics to write about.

Creating value through writing has been a huge eyeopener. The people I look up to the most, the Elon Musks and Steve Jobs of the world, they are chronic producers. They make things, execute outlandish ideas, and produce something valuable for the world. For me, writing was just the first step to becoming a producer - maybe it will be yours too?"

I think Steve hit the nail on the head. Many of the people in and out software testing that I look up to create value - either by writing or teaching or building fascinating things. I enjoy the feeling of being a producer of information and I hope it's my first step as well. Thank you Steve.

Looking ahead at 2013

2013-01-27T21:56:00.004-08:00

2012 was a good year in my development as a software tester. I've slowly started writing about my experiences. Some articles were better than others (I’m not as concerned with popularity to an external audience as I am about my own internal, undefined standards). I’m making progress in my ability to detect and tell stories which helps me gain confidence as both a tester and writer. This year should see me writing more articles about testing and although it’s not my primary goal eventually perhaps other people will find my writings helpful or informative.

Here are a few of the things I did in 2012 to develop my testing skills:

Traveled to Orcas Island in July for the first ever onsite Rapid Testing Intensive. I got to do some testing, learned a few things I’m applying today in my work, blogged a bit about the days and got to meet some testers from around the world.
Joined the Association for Software Testing on June 1st!

Through the AST I enrolled in the Black Box Software Testing Foundations class which I completed in December.

Joined Stack Exchange, specifically SQA forums.
Joined Miagi-Do school of testing as a Student in November.
Read some good books and added dozens more to my Read List

I need to finish a few books including Lean Startup, Code and An Introduction to General Systems Thinking.

I've read more articles and publications this year about testing than all my other years combined. It was pretty cool to go through the BBST Foundations class and have already read an assigned paper. My circle of testers has grown considerably by joining several online communities and forums. I’m subscribed to about 20 testing blogs and through twitter, G+ and the aforementioned communities I now have a much wider base for learning and discussing problems.

So what does 2013 have in store?

Tomorrow I start the Rapid Testing Intensive Online which will be my first training session of the New Year. I expect to do quite a bit of testing over those 3 days and be able to use that information / work to build my testing portfolio. (One of the outputs from the first RTI onsite was supposed to be a report built by James using the participant’s content but it seems like the amount of work involved was a bit much for James to do on his own. This time around I’ll see what I can put together myself.) In fact I've got a cool bug report I’m hoping to publish very soon (thanks for sucking Microsoft).

I’m not sure when (in terms of dates) it will happen but I want to get through the additional BBST courses this year: Test Design and Bug Advocacy before moving on to the Instructor class. Through Miagi-Do I’m signed up for an online test competition in April and I plan to ask for a few test challenges through Miagi-Do and maybe via Skype with James or Michael.

An important goal this year is to become comfortable coding so I can break through this barrier of understanding how to do test automation. I’m thinking Python and Java are good languages with Python being my primary language. I want to get to a point where I’m comfortable enough that I can build some type of application and then test it along the way. As I said before my reading list is quite extensive (long) so I’d like to dedicate more time to reading and I think I can do that by teach scuba diving a little bit less. Reading lots of books sounds great but I think it’s more important to be able to relate what I learn through reading back to testing through writing.

It’s almost the end of January and I've got my goals set for the next 11 months. Here I go.

All it takes is time

2013-01-17T08:23:00.006-08:00

All it takes is time to write a few posts a month, time I haven't had which is unfortunate since I've got a lot of ideas in my head that I want to get written down. Whenever I start thinking of time used I'm reminded of the quote from Harvey Mackay:

"Time is free, but it's priceless. You can't own it, but you can use it. You can't keep it, but you can spend it. Once you've lost it, you can never get it back."

I like being able to write and reflect on the things I've done or plan to do as it becomes a time capsule of my thoughts and actions that I can reference at a later point. I can look at how I've progressed in writing, testing, my ability to express myself; I can even write about interesting and tough topics and I can also see how I've progressed, regressed or see the things I wanted to do but abandoned.

One of my resolutions for 2013 is to write more. Stay tuned.

I'm a Black Box Software Tester

2012-12-29T23:07:00.000-08:00

More accurately I should say I've passed the Association for Software Testing's Black Box Software Testing Foundations class.

Here's the proof:

What does this mean? It means the instructors think I understood enough of the material to pass me based on the work I did throughout the course which includes discussion forums, assignments, quizzes and a final examination. I feel like I learned a lot from the exercises, readings and watching Cem Kaner's videos.

For those who don't know, Foundations is an intense 4 week class covering the basics of black box testing including the mission of testing, the oracle problem, the measurement problem and the impossibility of complete testing. I'd definitely recommend the class as long as you can spare at least 12+ hours a week to commit to watching the videos, reading the required and recommended readings and participating in the assignments. You must take the Foundations course before you can take any further classes so you learn how they are run.

One of my goals for 2013 is to take the other courses: Bug Advocacy, Test Design and then Instructor!

The Bug Bounty Hunter

2012-12-13T23:19:00.000-08:00

I haven’t been writing as much as I'd like thanks to an intense software testing class, travel, a few too many energy drinks which lead to some bad sleep and other life musings. New writing topics keep coming to mind and although I haven't had the time to complete my thoughts yet, I plan to get a few more things out before the end of the year.

In the meantime I thought it would be fun to share a graphic my friend Jeremy created for me a few years ago called The Bug Bounty Hunter:

Boba Fett is my favorite Star Wars character so the combination of Q.A. (Quality Assistance or Software Testing) and Boba as a bug bounty hunter was too awesome to pass up. This photo is so awesome that I enlarged it and hung it up over my test lab a few years ago.

Naturally In my attempt to share this image I came across a little usability problem with Blogger's upload file function. To insert an image into a blog post a user clicks on the Insert Image icon. The upload feature gives the user no indication of limits just that "[y]ou may upload multiple files at once. Use JPG, GIF, or PNG files." I became a bit confused when I got an error message saying "Upload failed: Image too big".

Wait a minute I don't see a limit posted anywhere. Once I get past the confusion I start to think, what about the image is too big? I assume blogger means the file size is too big so I make the file size smaller and try again. I get the same problem. The size of the picture is 10800 x 7200 which is large enough to be blown up and printed out. I think the error message meant to say was my resolution was too high.

Bugs are fun. Enjoy the photo courtesy of the Bug Bounty Hunter. =)

A Test Group of One

2012-11-09T22:39:00.002-08:00

I started the Foundations Black Box Software Testing class earlier this week and we are now into lesson two of six. I thought the discussion question was interesting so I'd like to share it along with my response.

Question:

[P]lease describe the role of the test group (services and responsibilities) in your organization. How do you think this mix differs from what you think of as the "typical" test group? How would you change this?

Answer:
Up until a month ago I was the sole tester in small software company (our entire company was about 11 people). Since then we've been purchased by a large multinational company with over 15,000 people world-wide and I’m still not quite sure what exact group I belong to, what my overall responsibilities are so I’ll focus my answer on the company prior to this acquisition.

Since our company was small so was our technical team, which consisted of 3 full time developers and me. For a few years we also had a CTO who acted as a development manager and occasionally for big projects we’d hire contractors. Like others on our technical team I essentially had two roles in the company: first as a tester of our products and secondly as a jack of all trades, used where-ever I’m needed. My responsibilities and services include:

Support our existing Clients deployments

Questions about software behavior
Debug problems with existing systems

Install and Configure customer deployments (production systems)
Install and Configure test and development labs
Build the releases for our software (release management)
Document installation procedures, update existing online help system
Test software releases and patches which includes:

Make sense of the changes development made
Understand what was asked of our development team (no formal specs) by our customers or executives
Make sure the company knows the risks and stability of the release in both functional and para-functional areas.

Anything else asked of me by management

While these responsibilities can limit the amount of time I can spend on a particular release testing, they provide different points of views which feed into what I know and learn about the product. Those influences then get reflected in my testing approach. This holistic view can help identify pain points which I might otherwise not get to see.

I’m not really sure what a “typical” test group is but in my experience working for large and small companies, each seem to have services and responsibilities that suit what someone thinks the companies needs are. I think the difference between mine and other testing groups has been the amount of time devoted to learning about the craft of software testing and how that can be applied to helping the business. The only way to change this is to set a good example for others to follow.

A Google "All Access Disabled" Issue

2012-10-21T11:23:00.003-07:00

A few days ago I got this weird "AllAccessDisabled" message when trying to access Google.com from my MacBook Pro. I couldn't reproduce this weirdness but I feel it was interesting enough to raise as a possible issue (a potential problem).

The message appeared in IE9 after resuming my Windows session (computer was set to sleep). At first I thought it might have something to do with the computer not being on the Internet, as you can see from the screen shot I had to re-select my network location. After selecting Home network I attempted to refresh and I got the same message.

Maybe it was a problem with my browser? I cleared IE's cookies and cache, closed and restarted the browser but the problem didn't go away. I tried accessing Google.com from my other browsers, in Chrome 22 and in Firefox 15, but both returned the same All Access Disabled message. Chrome provided a little more information than the other browsers, something about an XML style sheet not found but nothing helpful.

Since I'm on a MacBook I decided to boot into the MacOS X 10.8 and see what happened. I opened Chrome for Mac (I assume the same version 22) but I got the same problem. At this point I was getting a little upset so I didn't bother to check what Safari would do. I booted back into Windows but Google.com continued to show All Access Disabled in IE9. I didn't have any additional networks to connect to rule out my connection. I seemed to have ruled out browsers being the problem.

Maybe the problem was with my Google account? I went to my desktop that had IE9 open on Google.com, refreshed and my account seemed fine. When I went back to my laptop and refreshed my browsers all was well.

Random.

How do you handle regression testing?

2012-10-13T14:21:00.000-07:00

Matt Heusser sent the context-driven-testing email group a series of questions about handling regression testing. Specifically he asked:

How do your teams handle regression testing? That is, testing for features /after/ the 'new feature' testing is done. Do you exploratory test the system? Do you have a standard way to do it? Super-high level mission set? Session based test management? Do you automate everything and get a green light? Do you have a set of missions, directives, 'test cases', etc that you pass off to a lower-skill/cost group? (or your own group)? Do you run all of those or some risk-adjusted fraction of them? Do they get old over time? Or something else? I'm curious what your teams are actually doing. Feel free to reply with /context/ - not just what but why - and how it's working for you. :-)

I thought it was a good line of questioning so I responded to Matt:

I worked for a small software company where I was the only tester among (usually) 3-4 developers. Our waterfall-ish development cycles were a month+ in development with the same time given to testing. After the new features were tested, if we had time to do regression testing it was done through exploratory testing at a sort of high level. I don't think I ever wrote out a "standard way" to do it but it fit into my normal process of trying to anticipate where changes (new features, bug fixes) might have affected the product. If I had understood SBTM at the time I would have used it.

We've never gotten around to automating regression testing. A part of that has to do with time constraints - small company I wear multiple hats, could have managed my time better, etc. Other reasons involve not really knowing how to approach designing a regression suite. I've used Selenium IDE in the past but automating parts of our web applications GUI isn't possible without changes.

When I've had test contractors we used a set of missions to guide the group so everyone could hit parts of the application using their own understanding and skill (although this was our approach I don't think it was necessarily an understood approach =).) In all of the testing / regression or otherwise its all based on some sort of risk - based fashion.

Mostly I feel like I don't know enough about automation checking and how to design my testing approach to include an appropriate amount of automation. It seems reasonable to assume some automated regression checking could help provide some assurance in the build not changing for the worse (at least for the areas you've checked). Although I continue to commit time to learn more about testing I haven't committed much time to learning about automation and I believe its to my detriment. I guess I know where to focus more.

Feedback from a Developer (without knowing it)

2012-09-19T14:28:00.000-07:00

Recently someone asked one of my developers if we created formal test plans. Since the conversation was in an email, my developer cc’d me on it and responded saying he wasn’t sure but he had *seen* me create test cases in our bug tracker, SpiraTeam. He wasn’t sure if that qualified as a formal test plan.

Upon reading the email I responded asking what the questioner considered a formal test plan. Then I explained how we use Mind Maps to detail test designs and that works for us as a “test plan”. Yet I kept wondering when the last time I wrote a test case was so I went through our system and found a timestamp on the last created case. It read July 7th, 2011.

Curious still, I sent an email to my developer and asked when was the last time he *saw* me create a test case in Spira. His response was:

“I don’t know, didn’t you create some for the release before the DW or something? Maybe it wasn’t test cases, but I’ve seen you do things that take forever in Spira, I always thought they were test plans or test cases.”

“…I’ve seen you do things that take forever!" Yup that’s what writing out test cases will do. They take time to write out, time to “execute”, don’t necessarily help the tester plan their testing, and are abandoned after their use. After numerous years it was time to move onto something more effective.

This conversation was interesting for a few reasons:

First my developer confused a test case with a test plan. Not a big deal and not unexpected but interesting just the same. I’m sure many people in my organization would answer the question the same way.
Second my developer seemed to remember me writing test cases from over a year ago but didn't recall that a week prior I sent him a Mind Map to review. I wonder if he remembers *seeing* me work on the Mind Map?

Perhaps writing test cases seemed so strange (or perhaps wasteful or tedious?) to him that he couldn't help but remember it? Like when your parents remember something you did wrong and don’t remember things you did well.

Perhaps he didn't connect the Mind Map with testing, or to test planning? If my developer’s response is the consensus across my organization then it says the visibility and/or understanding of my work isn't where it needs to be.

Thanks for the feedback. =-)

MacBook Pro with Retina Benchmarks

2012-09-13T10:58:00.001-07:00

I was really impressed with the latest MacBook refresh so much so that I purchased the new 15" MacBook Pro with Retina a few months after its release. Owning a portable computer is a lot more convenient for writing and testing – far more than any tablet could be. (I do like using my iPad for reading, testing iOS apps and taking notes.) As I’ve done in the past let’s look at the benchmarks for this new computer.

The specs of my new 15” MBP w/Retina:

15.4” LED display with 2880-by-1800 resolution
2.3GHz quad-core Intel Core i7 processor
256 GB Solid State Drive
16 GB of 1600MHz DDR3L onboard memory
Weight: 4.46 pounds
Intel HD Graphics 4000 and NVIDIA GeForce GT 650M with 1GB of GDDR5 memory
Mac OS X 10.8
7 Hours battery life
… and many other impressive attributes

First on the Windows side (I always boot camp my MBPs) I ran AS SSD Benchmark 1.6:

The new MBP has sequential read speed of almost 464 MB/s and sequential write speed of roughly 415 MB/s. If you compare those numbers to the MBP’s from two years ago with the 128 SSD you’ll see a big difference: sequential read speed was only 196 MB/s and sequential write speed was only 160 MB/s. For reference look at the post I wrote two years ago.

On the Mac OSX side things are a bit different in terms of how I can measure performance (SSD or other), so this time I used Xbench (without the thread test) and these are the performance numbers I got:

Overall results 207.40
CPU test 236.61
Memory test 590.69
Quartz Graphics test: 156.41
OpenGL Graphics test: 143.42
User Interface test: 125.37
Disk test: 599.37

I don’t have any comparison numbers on older MBPs with SSDs for Xbench. The closest thing I have are Xbench scores for my old 13” MBP with the following specs:

13" MacBook Pro Core i5
2.3GHz dual-core Intel Core i5 processor
320 GB Hard Drive (I think 5400 RPM)
4 GB DDR3 RAM
Intel HD Graphics 3000 with 384MB of DDR3 SDRAM shared with main memory
Mac OS X 10.7

The Xbench scores for this machine are:

Overall results 137.52
CPU Test 217.26
Memory Test 535.29
Quartz Graphics Test 322.97
OpenGL Graphics Test 130.36
User Interface Test 122.29
Disk Test 54.91

The thing I take from these Xbench scores is the consistency of the quality of the components Apple uses (maybe I'm wrong since, technically I don't know much about the components). CPU and Memory scores are very similar across generations of machines. The major differences between my old 13” MBP and the new 15” MBP w/Retina are the disk scores and possible the graphics. I wonder why the graphics got lower scores on my new machine? Perhaps the new retina display is harder to measure?

Rapid Testing Intensive 2012: Day 5

2012-07-28T09:06:00.000-07:00

The final day of the Rapid Testing Intensive #1:

The Group photo - taken on the 4th day (I'm in the 2nd row behind the #1)

9:00 AM - We all picked up our Certificates of Satisficity - basically saying we completed Rapid Testing Intensive #1

9:05 AM - Jon, as PM, is starting us off with the RTI Project Status with a background about getting started at eBay. He was forced to do metrics he didn't like, getting bogged down in a ton of meetings and he got the opportunity to train new hires so Jon created a slide deck which he is showing. Going over highlights of the week with some screen images - the first bug filed was eBay Motors experiencing technical difficulties.

9:12 AM - Mark (part of team TRON) continued to get the experiencing technical difficulties problem up until yesterday - it was tied to his account. Wheel Center had about 39% of the bugs, Light Center had 35% of the bugs and the Tire Center had 43% of the bugs. Jon claims that the MyVehicles section only had 1 issue but that's unlikely and that's a reason why metrics need a context and a story before they make sense.

9:15 AM - We had opening activities (preparing the coverage) like creating teams, establishing JIRA and confluence, usability surveys, test cover outline, risk list. Jon has a list of more usability questions and it sounds like we can do more testing later today. Then we had the Coverage (test!) like photo upload, international compatibility, performance testing, severity 1 bug drill down and combinatorial testing. Finally we had closing activities (test your testing!) where we made a punch list, bug follow up, etc.

9:19 AM - Dwayne, Mark and I are the top 3 bug reporters for the exercise and we are all on the same team. This metric doesn't really matter but it made Jon question why the top 3 would be all from the same group and Dwayne said it probably had something to do with internal competition. Now we are going to go through our bugs, read comments if any and label with categories. We can then nominate anything we'd like reviewed both good and bad.

9:55 AM - Done with bug triage / updating our bug categories. For any areas that we think might need more coverage we've got 30 minutes for a final session.

10:30 AM - Break time. During our break I was part of a conversation with Andrew, Thomas and Dwayne where James reviewed some of our session notes. It was an interesting debrief because he pointed out information that was unclear, I put a line "that was weird" but I didn't explain what it was. Considering he was the audience for the report, the report to him was confusing. Good information.

10:48 AM - We are back and Jon is reviewing one of the bugs nominated in the chat room. Jon is going through and cleaning up the bug and James recommends trying to anticipate what the developer is going to ask - in the case of eBay if you can include URLs and links to the particular auction items.

11:00 AM - There was a lot of information in a TCO James was working through, it was a mix of parts of the product that might be tested, requirements that might be tested against and test ideas. It's a good idea to keep things separate because it frees you to do more things. They aren't the best place to combine or brainstorm ideas, they should be categories. If you have questions in the TCO you should be trying to answer them and if you can't remove them.

11:10 AM - The Open Questions and Risks sections of this persons document was good according to James, it means they have a lot of questions and were probably paying attention - as long as they didn't copy if from someone or somewhere else. James put his feedback in as a summary on the JIRA page.

11:15 AM - James likes to see expected and actual results in a bug report because it helps identify what the person thinks the issue is. James is comparing a bug of mine to a bug filed by Paul Holland. You don't need to always stick to a template especially with steps to reproduce - if the steps are obvious.

11: 20 AM - James is talking about the debate he and Andrew had, which I mentioned above at 10:30 AM. James says when Andrew gave his oral testing story, yesterday, he was very effective in telling that story but when he wrote up his session notes it didn't tell that same story. The interesting part of the testing story (according to James) was not recorded in the notes which was when Andrew, Mark and Thomas followed up on an interesting artifact which turned out to be a privacy issue and to James it meant the guys didn't feel it was an interesting story. The debrief of the testing, talking to each other was very important for the knowledge transfer.

11:35 AM - James and Jon are reviewing another bug that was nominated. So far most of the bugs have been written well so they are trying to find something written badly. I think they finally found a bad TCO because it looks like some person just copied the eBay Motors homepage because they didn't filter enough information and relied just on the visible.

11:50 AM - Jame and Jon are done evaluating the information for today but they will have to continue doing so as he prepares the report because the bug list has to be fully scrubbed. Problems found for each of the areas of concern for eBay. Don't be afraid to try things and fail because we get better and better, the learning happens all along.

11:53 AM - That's a wrap.

Photos from the event have been posted on Flickr: http://www.flickr.com/photos/83337442@N02/
Check out the other days:

Rapid Testing Intensive 2012: Day 4

2012-07-27T09:07:00.000-07:00

9:02 AM - James starts us off on Day 4. We are going to look at the status of the test project in terms of what we need to accomplish and look for the holes. This is a typical rapid testing management maneuver. James is showing a graph and reiterates he doesn't believe in fake metrics. The pink bands represent off hours and the clear bands represent on hours. At the beginning there is a very big jump in the number of bugs and then it flattens out.

9:08 AM - Turns out Paul Holland is going slow with checking the bugs - he claims its because there are only 3 people checking the bugs while 100 are reporting them. James wants to go through the bugs and check the risk areas to get general impressions so one of the activities together today or tomorrow might be to place risk measures on beach bug. The graph may make it on the front of the bug report. Dwayne says he isn't sure of the value of the graph and James says he also isn't sure but he doesn't need to know the value because he thinks it will provoke interest of the reader - in this case eBay.

9:12 AM - In rapid testing we don't put up graphs that give the impression we want to give, which is why James will filter out all duplicates and clean out the rest of the noise that could mislead readers. The graph could give the general impression of industriousness of the group over the four days we were here. Keep that skepticism in mind before considering showing metrics like this. You should always have someone doing bug triage otherwise you get a lot of noise in your reports and nothing gets corrected - no pressure. If you don't have a big team, if you can't dedicated someone, you can do it one at time at the time of the reporting.

9:22 AM - If you don't do bug triage then you get a lot of complaints from developers and managers even though they've never looked at them. It takes time but its worth it. James says at Borland they could do 20 bugs an hour and they determined out of the 800 bugs about 400 were legit bugs. You've got to maintain the quality of the list. After the first triage you get a much better feedback loop from that information. After the scrubbing we will want to see what eBay's final decisions are about the bugs. How do they rate the bugs we've created, what do they think of the bugs we've reported, how many do they end up fixing? That's the big thing.

9:26 AM - To clarify test strategy James starts making a test report. He pulls up the Test Report for eBay Motors that he and Jon are working on building for this Intensive. This is going to be a professional and comprehensive test report because James is aware of the multiple constituencies for this report. eBay Motors has a number of groups who may look at the report. Apparently Jon had to pay for his trip here and maybe other groups in eBay will want to use this event next year and perhaps pay for Jon and his families trip. Jon wants to get away from test cases and automation as a first path at eBay. One constituency is eBay Motors, another is eBay's other groups, another is us - because we will get a professional artifact to go on our CV. James is going to have to edit extensively because there are so many people reporting artifacts and there are so many overlaps but he's going to get everyone in the report who contributed.

9:35 AM - Different sections of the report for different constituencies which makes it comprehensive. Some parts of the report will be comprehensive and others will not. If James forces himself into thinking - what do I need for a final final report what holes do I have? That can focus him. It points out what is not getting done. It's called a forward backward method, from a book called how to read and do proofs chapter 2.

9:40 AM - Remember the three levels from yesterday? The same thing goes into the test report. It's a challenge to identify all of the testing that has been done, especially from James point of view, because there are so many groups doing things outside of James and Jon's view.

9:45 AM - Jon is showing his screen which is a to do list - he calls it a punch list, apparently it's a home building term? James is reading out of his report from the risk area section. Karen says she uses her low tech testing dashboard and she can use that information to contribute to the lean areas of the report.

9:52 AM - James took a poetry class to meet women when he was 24 and it turns out only middle aged women take that class. (The entire class started laughing.) The upside from taking the class was he learned poetry and he learned consensus can happen which brings people together. The risk areas of the test report are phrased in terms of a question - for this specific list. James ask the class why he changed from a statement to a question? He is trying to name the problem he is interested in without making a statement that it's actually a problem. To make it less confusing James turned the risk areas into a question.

10:15 AM - Looking and talking about test coverage outline of things we accomplished during our sessions that James put together and include in the test report. James films all his testing that he does professionally, he does note taking which can creates time stamps so its easy to refer back. He also does session notes which are crude but can help you locate the relative area for referring back to your videos.

10:24 AM - Both part 1 and part 2 of our reports are available on the internal site, which I'll try and download and post online later. This is one of the reasons James went offline yesterday so we could get feedback on our reports. You can use screen capture tools that take regular, automatic screen shots or a video capture tool to watch you test.

10:30 AM - James is showing us a video of how he records his testing. He's got a small tripod for his camera, the camera is placed under his left shoulder, the screen is zoomed in, and you can see he is using some log to record the inputs. With a detailed recording you can have confidence in what you tested. Scripts aren't the answer because no one really follows scripts which invalidates the script - they didn't setup and follow an oracle.

10:40 AM - Rapid testing is based on the idea of skill, testing credibility and trust and without that all these things are empty documents. You can't stand behind a report you created using other peoples work unless you've done the testing work. This is the reason why Jon and James have to examine the work we've done this far before they can include our work in a testing report. Most of us don't have the reputation with James and Jon where they can except without question the work.

10:48 AM - Break time. Jon and James are going to try to make their punch list a bit bigger.

11:06 AM - We are back and apparently are going to do some triage with Jon leading. I found the camera that James uses to record his meetings: Samsung HMX-Q10 camcorder. We are looking one of the bugs from JIRA and trying to figure out how many of the steps are relevant. Jon is editing the bug, it looks like this particular bug is not actually a bug because its mis-categorized, however Jon is taking notes in a separate document.

11:30 AM - Still discussing this one particular bug.

11:40 AM - How can you make the bug triage meetings go faster? We are still having a conversation like this. But as you develop as a team and as the project proceeds conversations like this go away because people understand the process. Unfortunately when you add or change team members you have to bring them up to speed. The culture can perpetuate itself as long as the project and people stay together. Process does not improve when someone writes a document, it improves when everyone adjusts what they say or are about to say based on something they learn.

11:47 AM - We have moved on to another bug. Except now James is questioning why we moved on. James and Jon have reproduced the bug and as Andrew is pointing out we may have a data consistency error with the criteria for vehicle compatibility in eBay Motors. James mentions black flagging: a situation where you see a bug, it means reporting is not enough for this bug because if you report it the developer will fix it. As James puts it we want its whole family hunted down. According to Jon Black Flag is a racing term that can mean get all the cars off the road because a car is causing damage than can affect other cards on the road. This is the type of bug you want to have a meeting on to understand its consequences.

12:02 PM - With eBay we should pay attention to the URL to check, for searches, whether the URLs are similar or different because the system could be passing different variables despite the same interface selections. James says he uses burp and other proxies to record this type of information.

12:05 PM - Andrew Prentice (part of team TRON) recommended we talk through the bug list and the real value "as we come together as a family and have family time" and agree on things. We are taking a group photo and then lunch time now!

1:15 PM - We are postponing Matt Johnston of uTest and James is talking about deep testing in rapid software testing. Apparently Michael Bolton's "daughter" found a bug in this game: http://www.horse-games.org/Horse_Lunge_Game.html and James is going to talk about state based testing in this game. We get to think about what state based testing might be - we don't have to know what it is exactly.

1:23 PM - What is x based testing? Put anything in for x - for example state based test. Any testing organized around x model. You change the state of something when you test it but that's state related testing, state based testing means focusing on the states on purpose. We need to know what the states are. There will always be questions about what the states are and you need to make a practical decision on what they are. State based testing is deep coverage testing.

1:32 PM - (Combination testing slides.) What if you have a lot of variables, they interact, and you want to test them systematically? It's called combinatorial testing. The first step is to identify the variables that might interact in a way you need to worry about. Remember all testing is based on models. Actually the first step, or step 0, is to learn enough about the product to identify interacting variables - survey the product, interview people, exploratory testing, etc.

1:38 PM - James says testers need to understand Cartesian products. Testing something that has no risk is called inexpensive testing or free time and you do that because your idea of a model or risk might be wrong. We talked about Ashby's Law of Requisite Variety and galumphing which all fit into combinatorial testing by helping us pay attention to strange and subtle outputs. Combinatorial testing goes hand in hand with tools. In combinatorial testing you use test cases and not test activities because they are mainly the same but slightly varied. This is one of the rare times you can count metrics because in combinatorial testing they are comparable.

1:52 PM - A derangement in combinatorics is a dis-ordering of a set to make sure the set isn't in it's natural state. I've found it here on WolframAlpha. James talking about gray code - arranging combinations so that only one thing changes between each test case. In fact one of the participants, Leslie, pointed out we do this in the dice game. It's a focusing concept to reduce chaos.

2:10 PM - A de Bruijn sequence packs combinations into a sequence and James is again showing a slide. You don't use de Bruijn sequences or gray code very often but its a tool for combinatorics testing. James is now talking about pairwise testing where there is a slide with 27 combinations which you can reduce to 9 test cases. Another slide shows a Microsoft Word Options Panel with 12,288 tests and using an ALLPAIRS tool to reduce to 10 tests - but those 10 tests may not include some important things like the defaults, all on all off (the Christmas tree heuristic), popular settings, etc.

2:20 PM - We have Matt Johnston from uTest on the phone. Talking about the differences between beta testing and crowd sourcing. How do you know when people who say they've tested something have actually tested something? uTest will suspend user who falsify work, it can affect their rating and the uTest system is built to monitor those kinds of patterns. People are paid for approved bugs, reports, etc. Customers pay per cycle (I've blogged about this earlier) or monthly. James said he was negative and doubtful of uTest and now is coming around - he likes uTest for the fact people with no experience can come and get experience. Matt mentions, which I think is the best reason for uTest, you get variety in the things you test.

2:40 PM - If someone wants to get into testing, they can sign up for uTest, fill out a tester profile, go to the forums, they will get invited to the sand box (which is unpaid) to try it out. James said he might sign up except he's worried that his reputation would take a hit from it. He also says he can demonstrate to European people who want to get into testing that they don't have to become certified to get into testing, they can go home, join uTest and start testing.

2:45 PM - Break time.

3:00 PM - Finally a session! We are going to do a search on My Vehicles that returns little returns and a few that returns a lot of results. This is an informal combinatorial testing session on factors of the left filter and we are going to file our session report in JIRA.

4:18 PM - Session time is up even though I'm not quite done working on a problem. James is thanking everyone now for coming because a few people are leaving before the final session tomorrow. It took a lot of James' friends to bail him out and help get this Intensive done - Scott you aren't getting paid. It takes a lot of people to keep up with the onliners - we had James and Jon Bach, Karen Johnson, Paul Holland and Rob Sabourin with Scott Barber and Michael Bolton, etc. online. Jon is very happy to have everyone. James wants feedback (for his wife to review) on whether we felt this was a good thing for people. An email will be sent out asking for that feedback.

4:25 PM - Debrief on our sessions based on our groups. Jon and James came up with a combinatoric testing charter for the session we completed, it seemed like a good idea to them and then through our onsite people it turned into something else. We did some combinatoric testing but Andrew, Mark and Thomas switched into a privacy testing mode off an inspiration from Andrew and Dwayne and I focused on filters once we started seeing problems.

5:11 PM - Done!

Photos from the event have been posted on Flickr: http://www.flickr.com/photos/83337442@N02/
Check out the other days:

Rapid Testing Intensive 2012: Day 3

2012-07-26T09:14:00.001-07:00

9:02 AM - Jon kicks off the Intensive with his project meeting. Talking about the communication between us and his eBay team.

9:07 AM - James talking about the upcoming assignments which will be split between onsite and online. Each table will get a 30 min test session. Later today we will be working / testing with tools since both James and Rob has some experience with this.

9:10 AM - James is talking about sympathetic testing.

9:15 AM - James is answering a question about knowledge transfer for regression tests when someone leaves a company. James uses the analogy of someone driving a car, if someone comes in and wants to drive his car he doesn't write down his driving procedures. He assumes that driver has driving skills. A tester should be good at rapid learning, skilled in testing and since most testers are untrained much of the documentation is of poor quality anyways. In Rapid Testing you create concise documentation, take test notes, you can take video but skilled testers should be able to pick up things fast.

9:22 AM - Pay attention to the test coverage outline - maintain it. Maintain the risk coverage outline.

9:24 AM - Jon is talking about a dice game we played last night and how important reputation is. Everyone here is building a reputation. James says things get easier with a better reputation - less annoying things are asked of you, less questioning of your work.

9:28 AM - Feature capability - is it capable of doing what its supposed to do? Correct output? Feature consistency - if they are similar features do things similarly? Example: if you save a Word document in .rtf and then you save a Wordpad document in .rtf - will they both save in the same?

9:30 AM - Are we going to do any bug reviews here? James wishes he thought of that idea he just has to figure out where we can fit it in.

9:36 AM - Rapid Software Testing and Agile fit in perfectly because RST all happens in your head. Agile has a strong emphasis on automated testing and tester's need to be weary of automated testing because it isn't actually testing - it's fact checking. Automated testing / checks are happy path tests, you aren't finding bugs that could be found through more aggressive testing. Rapid testers love using tools if they make you a better tester.

9:45 AM - James talks about Cucumber being a waste of time. James uses his programming skills to improve his testing skills and building tools but he sees a lot of Agile people are tool happy. Andrew points out those using Cucumber may use it because its not an obsession with tools but a way to leverage or improve the quality of the communication they receive.

9:50 AM - James is sharing his experience on using video to communicate findings instead of documentation.

10:00 AM - James is recapping what we've done the last 2 days. Did some testing, worked on risk lists, did some specification reviews, some people ignored the specifications, demonstration test sessions, going forward we will look at tools, maybe do some bug triage and we've blown eBay away with the amount of bugs we produced.

10:02 AM - Online people will work with Scott Barber. The online people are offline until 1PM. James is describing how much they've learned and are adjusting to this format of online and offline. Today we are doing a local activity for the next 30 minutes we are doing a test session on the add photos part of My Vehicles.

10:40 AM - Test session is over, filling our bugs now.

10:45 AM - Break time.

11:03 AM - Each table is going to prepare a professional test report on what we did during the most recent test session. In written and oral form. Written will consist of 2 flip chart pages. 20 minutes. It must relate to the entire work of the entire table and then we will give the report to the entire room. James in basically interested in what we did, the story, specifically the conditions that we tested.

11:08 AM - Working my with team - TRON on building our professional test report.

11:26 AM - We are still working on reports. James says he wants decent reports and you can't do it without practice in only 20 minutes. He is going to add 10 more minutes so we can get a good report down.

11:36 AM - Time to review our reports so we are putting them on the wall. James is video tapping this so it might be available online at some time. He is introducing the video and we are up first.

11:56 AM - Took 20 minutes to give our report - which was the first stand up test report I've ever given. (Apparently I came across as a little nervous - mostly nervous about the material) Some of the terminology I used was rather vague which caused James to ask numerous questions to help identify what I actually meant. Anything we say that has some inherent meaning is immediately questioned by James because the customer may not understand what that means.

12:00 PM - Another group (S-Table) is up talking. Simon is presenting for S-Table group and he's doing a pretty good job by speaking very clearly. (Everyone who goes after the first team should be better! hahaha) James has got some questions and feedback for Thomas but his board / post it sheets are much clearer than ours; again you'll be able to see and his group but he cuts them off because now it's time for lunch.

1:00 PM - James is giving a presentation on How to Give a Professional Test Report. He's going to talk about tools in a bit. Test reporting is the heart of testing, its helpful for managing yourself. Reports aren't just facts, its a choice of which facts matter and you are shading reality. No one who gives a useful report is revealing facts, they are always pruning picking and choosing. Suppress silly metrics - it's like counting how many unicorns will fit into your cubicle? We are back with the online people.

1:08 PM - In the absence of context test case counts have no meaning - so don't include them. Pass rate is a stupid metric without context. James is showing slides from his How to Give a Professional Test report but I'm not sure where the slides are coming from.

1:17 PM - You can't give a form like test case counts, pass rate fails, etc. to managers because they don't know what any of that stuff means. You need to produce a report that tells them something. You can list your bugs because the titles can give them context and you can talk about it. Pie charts, graphs don't tell anyone anything.

1:20 PM - Talking to people is an alternative to counting test cases! You can summarize certain ideas that people need to know and an example of that is a Low Tech Testing Dashboard.

1:27 PM - A professional test report is one that fits the context for the customer or the person who matters. It could be nice to list the things that are important to do but haven't gotten to yet. James is explaining what his dashboard means. He recommends not using happy faces when writing on a white board.

1:35 PM - Your dashboard should have structured subjectivity, a human judgement. No raw data on the dashboard because management can't process raw data when outside the context of the team. When testers do that we are abdicating our responsibility. Management needs to glance at the test report and make decisions very quickly about what needs to be done.

1:36 PM - Test report should be 3 levels. A story about the status of the Product, a story about how you Tested it, a story about the Value of the testing. Apparently our test reports contained all 3 of these levels. Yay!

1:40 PM - Use safety language - phrasing that qualifies or otherwise draft statements of fact so as to avoid false confidence like: I think, so far, apparently, I assumed, it appears, etc. When you are trying to communicate something dramatically (and not factually) be careful of your use of safety languages. Sometimes when you are pressed for time or need to give your message a rhetorical punch or for ritual practice (like at the altar) you probably want to avoid safety languages.

1:48 PM - James is showing examples of reports. He shows the most expensive report he has ever constructed at roughly $250,000 of labor charges for a patent infringement lawsuit. It shows each of the claims in the patent and the proof as demos and exhibits something violates this patent. This was the lawyers test case outline. The second report is an exploratory report that was filmed and contains details about what was done, what was asked and what was thought of by the people doing the test. The third report was one back from 1994 that was based on an idea James had about "rapid testing".

2:00 PM - Test reporting is fundamental. Practice this even though management is not going to force you to do this. Jon talks about how he does reports at eBay and he does several variations.

2:05 PM - We are going to talk about tools and Robert Sabourin. The typical automation formula is purchase an expensive tool, hand them to a tester (forced on them), testers are forced into test case-ism which changes how the tester thinks or forces the company to hire automation testers. This can work if your product is easy and doesn't change much. Tremendous amounts of automation effort and it keeps braking and then you have to spend an increasingly larger amount of time fixing things.

2:15 PM - Snap out of that routine and use free tools. As James calls it guerrilla test automation - quick and dirty tools that can help. If they don't help then abandoned them. Not all testers should be programmers that's a bad idea. You need one tester who is a good programmer so they can build tools, but you want a wide variety of testers who are interested in a number of areas.

2:30 PM - James is answering questions. Ajay asked one about safety language. We are breaking for root-beer floats. Cool.

2:51 PM - We are back to doing test reports with Thomas and his team "7 Up" I think they are called.

3:03 PM - Susan is up for team Coho talking about their test report.

3:09 PM - James is reviewing, talking about watching our team (TRON). We called out facts, we would get quiet, then we started discussing problems and struggles, and would then focus on specific deep issues. We have to have a foot in both worlds:

the mission, risks, or TCO, the overview.
focusing on the deep specific issues or the path you are on.

Testers need to be able to bounce back and forth at certain intervals. These are two types of thinking that can be incompatible at certain points. We are done with test reports.

3:15 PM - Rob is up and he's wondering around tell a story about coming to RTI and talking to James about bring a tool. Rob has a picture with 4 elements (a quick and dirty framework) showing his system under test. Rob is showing and explaining how his ruby tool works. Rob has an example framework of Ruby on his website Amibugshare.com

3:30 PM - James said if enough people are interested in doing combination testing he will do a webinar on it. James and Rob are talking about the results the program returned which measured the time in milliseconds. Now Paul Holland is up helping to figure out what the data means and re-plotting it. One of Rob's undergraduates is a programmer and now his toolsmith who created this application to help us test eBay motors. It performs massive searches with lots of combinations.

3:42 PM - We are going to install the tool and try it. We are using tools to supercharge the human mind, give us more abilities.

3:51 PM - James wants us to brainstorm what we can do with this tool, how we might or how we think it should be used. Then he's going to get another tool up and running.

4:10 PM - We are now reviewing the ideas and of course James comes to our group first. Dwayne and James are discussing the ideas we came up with on how to use Rob's tool. Andrew figured out there were quite a few bugs in the tool and a discussion over it began.

4:35 PM - Other groups are talking about running the tool at different times and on international sites. One of the other participants found a rather interesting bug with a tool designed to check all listings in categories or filters and compared them to non-categorized listings.

4:43 PM - James is showing an example of a blink test of eBay using a tool called IECapt. There are many oracles for this: a juxtaposition blink test oracle, zoom blink oracle, speed blink oracle and a noise zoom oracle.

4:50 PM - James shows a mind map of eBay's hosts which he made through a web crawl. James says its big like a city and Jon says he prefers the term "death star".

4:55 PM - Done. We are going kayaking.

Photos from the event have been posted on Flickr: http://www.flickr.com/photos/83337442@N02/
Check out the other days:

Rapid Testing Intensive 2012: Day 2

2012-07-25T11:00:00.000-07:00

9:00 AM - Start of the day. James doing some talking about what we did yesterday, he's built a mind map. James and Jon are going over our schedule - gonna try to stick to it better than we did yesterday.

9:23 AM - Jon is doing a de-brief from yesterday / Project Check in. Talking about how good the bugs are that were filed. Nice job.

9:30 AM - Reviewing the TCOs from yesterday. Don't update them if it's going to cost too much. Being critical of one TCO that according to James could be affected by Visual Bias - only test the things they see. This is why we use heuristics strategies.

9:40 AM - Lighting and Wheel center for eBay Motors have been added to the scope of My Vehicles and Tire Center. Session of survey the functionality and do until 10:45 AM. Modify your TCOs.

10:45 AM - Break time!

11:00 AM - eBay has been making updates to our bugs so make sure you take a look. eBay says they are seeing some coool bugs! The instructors will make comments if people post in Jira on pages. You can start TCOs in a brainstormy kind of way. You can make edits later.

11:06 AM - You can assign risks to your TCO. You can assign risk to your survey of the product based on probability and impact. Make a list of the possible bugs and then you can group them together. Are they bad or not so bad?

11:10 AM - Risk brainstorming time. Fist make a list of the kinds of bugs you worry about and then summarize them into risk areas (James says 5-12). Then create a component risk analysis based on your TCO.

11:45 AM - Debrief on the risk brainstorming. If you get stuck with a blank slate you can use the Heuristic Test Strategy - Quality Criteria Categories to get un-stuck. Rob is describing how his team built their mind map, how they created their risk outline. The artifacts you come up with aren't as important as the mental preparation you go through and how it gets you ready.

12:01 PM - Time for Lunch.

1:12 PM - James talking about models and his Test Testing Framework diagram that he started talking about yesterday. Deriving test cases from requirements is as if there is no thinking going on, makes it sound like there is a mathematical procedure, which there isn't. Shouldn't talk about testing that way. Different kinds of modeling and designing experiments are conceived which drives learning and new test cases. Confusion and struggling is a normal part of the learning experience but James' diagrams help him get out of the confusing things.

1:29 PM - Risks from eBay Motors according to James' mind map: Usability, Feature Capability, Performance (page load times), International Consistency, Compatibility, Feature Consistency and Data Integrity. Participants will have risk categories that James missed.

1:38 PM - Rapid Testing focuses on Test Activities - the types of things you do when testing. Those things could map to test cases as long as its something done by a tester and not a tool. A human using a tool is a test activity but a tool by itself is not.

1:45 PM - James will talk about his and Michael Bolton's new ideas on what an Oracle means - a medium. Interpret the product for the people whose opinion matters, you are an agent.

1:48 PM - James made a note that he needs to put up the most recent up to date slides for RST. They aren't online yet.

2:03 PM - Jon is trying to determine a Risk Exercise. A search result within one of the centers: Wheel, Light, Tire which is not relevant to the query. Maybe the seller has put it in the wrong category. Online participants get to check UK and DE sites versus US and perform an international comparison.

2:45 PM - End of risk exercise and time for a break.

3:00 PM - Back from break and James is doing a brief on a participants search. The person who likes learning something new is going to be better the next time.

3:13 PM - Talking about Oracles - most specifications will not contain Oracles. Here comes the calculator question. "What do you expect from a calculator when someone enters 2 + 2?" There is a difference between expected and unexpected. You may expect the calculator to return a number 4, you may expect the calculator to remain on long enough so you can read the answer, you may expect the calculator not to blow up in your face, etc. Many expectations are inherit and you aren't aware until of them until they violate your expectations.

3:26 PM - 10 Consistency Heuristics from the RST slides. The purpose isn't to teach someone test but to help someone explain - you can push back against the process bullies. There is no Oracle guaranteed to always solve a problem, they can't be perfect.

3:32 PM - Test activities in which we will use Oracles. Jon Bach and I are going to do a pair test effort.

4:18 PM - Done! As Jon says it seemed like we were going for less than 10 minutes! Jon is debriefing our live paired exploratory session test.

4:21 PM - James says we are going to use these reports to document test sessions. SBTM packages exploratory testing into session units because if they have a fixed time you have a meaningful way to compare test activities. Sessions are relatively stable compared to a test case - no orders of magnitude differences. Sessions should be logically uninterrupted instead of physically uninterrupted; Jon and James had a little bit of an argument about this but if you get interrupted get that time back and continue.

4:35 PM - If you get chronically interrupted you can't do SBTM but you can do thread based test management which is testing with check lists. TBTM includes a list of test activities, arranged in a mind map and you service the threads as you go. You can't count threads but together they define the testing story. Artifact based test management is where you test based on counting test cases - something James tries to get companies away from. You also have activity based test management (which includes SBTM and TBTM) and people based test management.

4:40 PM - We will have test charters and will work from them tomorrow.

4:43 PM - James talks about humility, specifically epistemic humility.

4:45 PM - James says the way you manage these sessions is through managing the charters. You can list the charters, mind map them, organize them based on activity or risk, etc. Jon shows his and James set of charters which they've created a grid with details on the sessions.

4:50 PM - How do you ensure that the notes for sessions are readable? You train them. Tell the story of your testing briefly in a reasonably sharp way. The last 10 minutes you can focus on finishing up your notes, if you are taking longer than that you are taking too many notes.

5:02 PM - Done. Dice game tonight!

7:00 PM - 9:30 PM - After hours, game night! James and Paul (Holland) started us off on the "mind reading" game and between the 10 of us around the time we figured it out, according to James, "the quickest of any of his students". It took maybe 20 minutes. Then we moved on to a series of dice games with escalating challenges where we tried stumping Jon Bach and Paul Holland by creating our own!

Photos from the event have been posted on Flickr: http://www.flickr.com/photos/83337442@N02/
Check out the other days:

Rapid Testing Intensive 2012: Day 1 Recap

2012-07-25T10:57:00.000-07:00

This is a previous day recap of the things we did in Rapid Testing Intensive #1, Day 1 on Orcas Island, WA in 2012. I hope I remember everything. Some of the information I took from Karen Johnson's internal micro-blog. We've got eBay's support, developers online, ready to help with any bugs we find.

9:00 AM - Description of what we will be doing from Jon and James. How Jira works for reporting bugs, find project documentation, etc.

9:31 AM - First assignment is a Usability Test of eBay Motor's My Vehicle section. Session testing in pairs. There is a script to follow, find and report bugs as well as fill out the script. Post it to Jira when done.

11:35 AM - Wrap up of the first assignment. Get all bugs in.

11:47 AM - Usability test debrief from Jon and James.

12:00 PM - Lunch time.

1:10 PM - James is talking about Rapid Software Testing.

1:50 PM - James talking about the Law of Requisite Variety.

2:05 PM - Going over notes about RST, James is discussing a style of testing called galumphing. There was a blog post on this awhile back.

2:15 PM - James talking about the heuristic test strategy which can be found here: http://www.satisfice.com/tools/satisfice-tsm-4p.pdf

2:18 PM - Talking about Test Coverage Outlines. This leads to our next assignment.

3:00 PM - Working on the second assignment which is to be grouped into tables. My table is called Tron. We are working on the "data surface" and our job is to create a TCO or Test Coverage Outline.

3:06 PM - Done creating TCOs.

3:10 PM - Jon and James are doing a de-brief and showing off a few of the TCOs.

4:00 PM - James talking about his note-taking while harvesting test ideas.

4:06 PM - Talking about alternation: read the spec, play. Repeat and continue.

4:50 PM - Done for the day.

Photos from the event have been posted on Flickr: http://www.flickr.com/photos/83337442@N02/
Check out the other days:

Enrolled in BBST Foundations

2012-07-11T14:02:00.000-07:00

It's official. I'm enrolled in the BBST Foundations course for November through AST.

I joined AST (Association for Software Testing) with the end-goal to enroll in the BBST (Black Box Software Testing) Foundations course. I've read about the classes, seen a number of experts whom I trust recommend them and also heard good things from my post on SQA StackExchange. BBST.info which is home to Cem Kaner and Rebecca Fiedler (the creators of the material) BBST consulting practice say this:

Too many testing courses emphasize a superficial knowledge of basic ideas. This makes things easy for novices and reassures some practitioners that they understand the field. However, it’s not deep enough to help students apply what they learn to their day-to-day work.

The BBST series will attempt to foster a deeper level of learning by giving students more opportunities to practice, discuss, and evaluate what they are learning. The specific learning objectives will vary from course to course (each course will describe its own learning objectives).

The Black Box Software Testing course information is all freely available online at TestingEducation.org but it seems like I'd get more information going through the class with people in a more collaborative environment. More importantly it's taught by teachers who have been trained by Cem Kaner and Rebecca Fiedler themselves.

There are several levels to the BBST coursework including:

Foundations
Bug Advocacy
Test Design
and eventually... Instructor

Just the list of required and recommended readings at the bottom of the BBST Foundations website has me excited. It's going to be a lot of work but probably well worth it.

In the video below Dee Ann Pizzica explains the BBST courses at CAST 2011 (Conference of the Association for Software Testing). It's worth a watch:

15" MacBook Pro with Retina

2012-07-07T17:24:00.001-07:00

I decided against waiting for Christmas and a week ago ordered a new 15" MacBook Pro with Retina display with 16GB of RAM and a 256GB Solid State Drive. What can I say other than I was impressed with the 2012 refresh.

When it does arrive (in about a month or so) I intend to write about my current 13" MacBook Pro's (last years model) performance scores (xbench, SSD benchmarks, etc) and the new 15" MBP with Retina's scores.

In the past I've written a few times about Solid State Drive performance in Apple computers:

I do quite a bit of software testing on a number of different platforms - Mac, Windows - and I think this new computer will do nicely.

Do Software Testers Need a College Education

2012-07-06T11:14:00.000-07:00

I came across an old blog post on the uTest blog Do Software Testers Need a College Education. The author says:

Depending on who you ask this question to, you’re likely to receive various degrees (pardon the pun) of yes and no. Or you may find many others who answer in a noncommittal way: “it depends.”

Having worked closely with thousands of software testers in the uTest community, I can attest to the fact that many testers do in fact have impressive resumes with regard to higher education (master’s degrees, PhD.s, etc.). However, there is also convincing evidence that demonstrates quite the opposite. So if you let the data speak for itself, what is one to believe?

The article goes on to list a few explanations from the For, Against and It depends camps. I was trying to comment on the article but it wasn't working. If they turned off the comments for this article then why is the comment box still available? I must have found a bug!

Here is my input:

You don't need a college education to learn technical knowledge / skill, just time and desire. You might be able to make the argument a college education makes Testers and Developers better but it would depend on the school, the person's learning habits and how much they learned.

I've seen quite a few reports that indicate Software Engineering (which includes Testers) has the highest concentration of non-college educated engineers in all of the engineering disciplines. This might to suggest the industry is more open to taking chances. It might also suggest if you are qualified it, have the technical skills, it doesn’t really matter how you learned as long as you can do the work.

Those hiring might prefer employees to be college educated because it's a safe bet. Especially for those who don't know how to hire, have an HR dept who screens candidates, etc. Rather I believe most companies are better off with someone with relevant experience and the desire to learn more.

I learned everything I know about Software Testing after I graduated from college. Today there are hardly any colleges that offer classes or degrees in software testing and plenty of scammers trying to offer Certification as a replacement for learning to test. Luckily there are people like James Bach who try to help train testers.

Check out what James thinks software testers need to learn.

Anyone Can Test, right?

2012-07-04T10:19:00.000-07:00

In this video from StarEast Rob Sabourin talks about his experience with just anyone testing.

Anyone can test, right?

An Impressive 2012 MacBook Pro Refresh

2012-06-14T13:54:00.003-07:00

Last year I wrote the 2011 MacBook Pro refresh was a disappointment. Apple updated the graphics, CPU and added the Thunderbolt port but no redesign and no upgrade to an SSD. This year came the changes I was hoping for last year.

I'm not at home often enough to use a desktop which means I need (want) portability with power. Apple's move to a reasonable size solid state drive and 8GB of RAM ensure power. 8GB of RAM is a minimum these days with 16GB a better option. 256GB seems like a reasonable move; I have SSDs in my work and home desktops, both are 128GB and almost full.

The changes I care about: (not in any particular order)

No Optical Drive (DVD)
Base RAM at 8GB
256GB SSD Hard Drive
Decreased the size and weight
Updated the screen resolution
Increased battery time

I'm sure the most noticeable and impressive thing about the new 15" MacBook Pro is the retina display (I can't say for sure because I've never seen it). Apple's displays have always been the envy of the PC user with their bright colors, glass screens, high resolutions - something PC manufactures never cared to put enough time or effort into. AnandTech has a good review of the Retina display.

Some of the smartest and most demanding companies like Google, Amazon and Facebook custom build their own server hardware because they can't afford the inefficiencies build into standard equipment. Apparently Apple is following in that direction. Check out the insides of the new 15" MBP:

The photo is from PC World's Review. As you can see the RAM is built directly onto the motherboard as is the Flash for the Solid State Drive. Any upgrades you want should be done when you purchase it. The battery takes up the most space but the fact a reasonable user can get 7+ hours of battery adds while two asymmetrical fans keep the laptop cool when crunching hard numbers.

For full review and benchmark information I'd recommend checking out Engadget's review. It's official I've found my next gadget to lust over. Maybe I'll get one for Christmas?