The first incident was a common one. My son asked me what the circumference of the earth was (ok – he didn’t use the word ‘circumference’, but that’s what he asked). Like a trained seal, I grabbed my smart-phone and looked it up. This was stupid, because a) I knew the answer, and b) even if I didn’t, I know enough about distances between locations and longitude to come up with a fairly accurate approximation.

Just a bit later in the morning, I had a short interview with a journalist from my university – apparently they’re hard up for stories and are reaching out to alumni for gossip. At one point, the journalist asked something like, “What advice do you have for students today?”. I gave a long answer that said (more or less), “A university isn’t a vocational school. Although you may get caught up in the rat race of learning facts and writing papers, to be successful in the ‘real world’, you need to know how to learn.” I also (once again) credited my Methods of Musical Research course and Dr. Snedeker for teaching me how to learn and for fueling my own passion for learning.

Less than thirty minutes later, I received an email response to a verbal request. I asked someone to research a topic and see if they could find any interesting papers that could foster additional discussion on the topic. They replied with a link to paper that shows up as the top link in Bing when searching on the exact phrase as the topic I presented. I expected my colleague to look at academic papers, and perhaps explore the references in those papers, or to find experts in the field and see what they wrote. I expected them to research…but I guess to them, ‘search’ was enough.

Perhaps I’m just being an old fart, but the serendipity of these three events launched a thought that scared the crap out of me. In these days of search engines and Wikipedia, I wonder if anyone knows how to think anymore. Knowledge is much more than learning or regurgitating facts. The learning that happens when you move knowledge from something you don’t know to something you do know is trivial compared to when you add discovering what you don’t know you don’t know to the top of the stack (1). I’m scared that too many people (not just testers) are happy with on-demand fact finding rather than true learning and discovery.

It could be just me – perhaps I spend so much time reading and trying to learn because that’s what works for me, and that on-demand learning is the true future. But if that’s true, I worry what kind of future that will be.

(1) For more on this theory of knowledge acquisition, read Philip Armour’s original paper on the topic here, or my takes, here, here, and here.

If you haven’t read this book, I bet it’s because the title scared you away. You’re probably thinking, “I’m not a consultant, this book is irrelevant”. For those thinking that, or for any others reading right now, I’ll tell you this. I have read (and in some cases studied), over 40 books on leadership – and this is one of the best I’ve ever read on the subject.

When people get to “Senior” roles at Microsoft, one of the things we expect is an ability to influence without authority, work through others, and basically make those around you better. For our introverted tech geeks, this is often a big wall to climb. Inevitably, when one of these Senior non—manager folks tracks me down and asks, “How can I get better at influence?”, or “How can I get others to accept my ideas?”, I hand them a copy of this book (I’ve done my best to pad Mr. Weinberg’s royalties with the dozen or so copies of this book I’ve purchased over the years). You see, being a good consultant, and being a good leader are pretty much the same thing. Some of my takeaways are below (adapted heavily to my usage over the years).

Good consultants know how to define the real problem at hand – they see what’s there, and what’s not there, and come up with a solution (or multiple solutions). I would hope, that if you’re a leader, you’d do the same.

Good consultants are humble – they don’t pretend to know it all, and know when to admit when they don’t. Your fee (or your title if you’re not a consultant) doesn’t mean you’re always right. It does mean, however, that you always need to help the team improve.

Establish and maintain trust. I can’t say enough about how much I value trust as a leadership attribute. The chapter on trust is critical for any leader.

Worry about the accomplishment, not the credit. This one rings home at MS, where our (ahem) review system often drives people to focus on credit rather than outcomes. The punch line, is that in my experience, the weight of the accomplishment always outweighs the weight of who did what. Just worry about making your projects successful, and the rewards will follow.

And given the hundred or so bullet points of wisdom sprinkled throughout the book, I’d say the meta-takeaway is that you have a good leadership toolbox. If something doesn’t work, doing it again louder won’t help – do something different. Use another leadership tool, or experiment. Nobody has time (or money) for a one-trick pony.

Now – to be fair, several points in the book are specifically for consulting – but if you read this book looking for leadership advice, that’s exactly what you’ll find (and you’ll barely notice the n/z sections).

What are you waiting for – go learn how to be a leader.

I try to read a lot about testing in blogs, articles, books etc. A few days ago, I came across this quote, and it struck me in an odd way.

“Commonly, test automation involves automating a manual process already in place that uses a formalized testing process”

The source doesn’t matter, as it turns out that sentence was copied directly from the Wikipedia article on Test Automation. I’ve been at Microsoft for a long time now, and although I do try to stay connected with testing outside of the Borg, sometimes I notice that *my* view of a subject is quite different than what’s “commonly” understood.

Or, maybe not – but let me try to explain my concerns here a bit more.

I’m all for saving time and money, but I have concerns with an automation approach based entirely (or largely) on automating a bunch of manual tests. Good test design considers manual and computer assisted testing as two different attributes – not sequential tasks. That concept is such an ingrained approach to me (and the testers I get to work with), that the idea of a write tests->run tests manually->automate those tests seems fundamentally broken. I know there are companies that have separate test automation team that do exactly this, and I think it’s a horrible approach.

Let’s use the Numberz Challenge as an example. I know right away that I’m going to perform some manual tests (e.g. make sure the app can launch and close (original version had a bug here), verify look and feel, color choices, etc.). These sorts of tests could be automated, but I generally don’t for a few specific reasons.

• The oracle for look and feel is difficult. Automated testing based on comparing screen colors or app dimensions is fragile. A pair of eyes for a few seconds every once in a while is much cheaper. There are, of course, exceptions, and if you’re convinced you can pull this off, just make sure you calculate the time spent investigating failures (and potential false positives) into your ROI calculation. I know many teams who have successful UI automation systems, but many more who have a bunch of test auto-crapation.
• Most systems I work on – and certainly everything that has a UI, gets at least a tiny bit of usage from the product team or a small pilot group before rolling out widely. If the screen is pink, or the font somehow changed to Comic Sans, it will be noticed without the need for an automated test.

When I look at something like Numberz, I also see where I need to use the power of a computer to answer some of the questions I have. For example, I know I need a large enough data set to be confident of the random algorithm, and I know I need extensive pattern matching to ensure that the next number is never predictable. Doing this manually is impossible (or a waste of time depending on what you try to do).

Now I imagine this scenario in the write-test-then-automate-it workflow.

   1: Test Case Steps:

   2: 1) Launch App

   3: 2) Press the Roll! Button 

   4:  

   5: Verify:

   6: 1) Ensure that none of the numbers is less than 0 or greater than 9

   7: 2) Ensure that the total field is correct

   8:  

   9: Repeat this test at least 10 times

Now, the “automator” comes along and writes this test:

   1: Loop 10 times

   2:   App.Launch (“numberz.exe”)

   3:   Button.Click(“Roll”)

   4:   // Validate numbers are within range 

   5:   // Validate correct total

   6: End Loop

From what little context I have from the interwebs, this appears to be a common scenario — but we’ve missed the critical aspects of testing this app (testing randomness / predictability). What we haven’t done at all in this situation is test design. To me, test design is far more holistic than thinking through a few sets of user tasks. You need to ask, “what’s really going on here?”, and “what do we really need to know?”. Automating a bunch of user tasks rarely answers those questions. (Note – it does answer some questions, so if you have a good system and tests you can trust, by all means don’t stop what you’re doing).

Where I think I have my big disconnect is in the definition of test automation. When I think of test automation, I don’t think of automating user tasks. I think, “How can I use the power of a computer to answer the testing questions I have?”, “How can I use the power of a computer to help me discover what I don’t know I don’t know?”, and “What scenarios do I need to investigate where using a computer is the only practical solution?”.

Perhaps test automation is purely the automation of manual tasks, and I’m attempting to overload the word. I know some folks prefer the term “computer assisted testing”, and I suppose that’s fine too.

To me (and I’m sure I’ve used this line before), it’s just testing. But please stop thinking of test automation as the step that follows test design, and start thinking of test design first.

But it’s easy to go overboard with skepticism. Time and time again, I hear good testers apply their skepticism broadly and liberally. Some (paraphrased) quotes I’ve heard recently include:

• “Model-based testing is interesting, but it doesn’t work in a lot of places”
• “I’m skeptical of static analysis tools – sometimes they have false positives”
• “Metrics are evil, because someone may use them to measure people”

I agree whole heartedly with each of these quotes. However, I worry that folks are throwing the baby out with the bathwater. Model-based testing (just an example), is a wonderful test design technique for stateful test problems. Although occasionally someone will screw up the value of MBT by claiming that it’s the only test approach you’re ever going to need, it’s just another technique (and the perfect technique given the proper context). Static analysis tools are also awesome, but aren’t perfect It’s good to measure some things too, but sure, one can screw it up.

I’m trying to think of anything in my life that works perfectly in every situation, but I’m coming up empty. I run into situations nearly every day where someone has a good idea that will obviously work most of the time – but not always. Given these situations, we could just send them back to the drawing board telling them, “I’m skeptical of your approach, because it won’t work in situation z”, but it’s probably a better idea to have a conversation about the limitations, understand where and when the approach may fail, and discuss mitigation or workarounds. Instead of throwing out the idea of running static analysis tools because of the potential false positives, discuss the false positive problem. Find out what causes them. Tweak the configuration. Do whatever you need to do to ensure the value of the approach.

Over the years, I’ve found value in some pretty stupid approaches. It seems that we should be able to find more value from some the ideas frequently discounted.

Even if we’re skeptical.

In my quest for knowledge, I discovered a few interesting things.

I discovered the Spinellis book, Code Quality: The Open Source Perspective. I haven’t read this yet, but I ordered it, and looks like it will be interesting.

I found an article on coupling metrics on the IBM site. The reading here was good, as I discovered they have a series of articles on code quality.

I found a few references to analysis tools for JavaScript (some lint-like stuff)

Over in the high brow world of the linkedin groups, I saw a bunch of people referencing code coverage as an indicator of code quality.

Nothing earth shattering, but some good information, and enough to get me thinking about what *I* think code quality is – and what it is not. I think it may be impossible to define code quality in a way that everyone will agree with – but that doesn’t mean I shouldn’t try.

We have some metrics that can give us an idea about code quality. Quality code doesn’t have any compiler warnings (ideally at the highest warning level), and it’s clean from static analysis errors and warnings. Complexity metrics such as cyclomatic complexity (basically, the number of paths through a function), or dependency metrics such as fan-in and fan-out can also be indicators of code quality.

But – let’s make it clear that while I’m a (big) fan of those indicators, that’s all they are. Running clean with static analysis tools does nothing to ensure code quality – or product quality either (I ranted a bit about this a few months back). In fact, I guarantee you that I could write (or show you) a piece of software that compiles with the highest warning levels set and passes any static analysis tool with flying colors…that still sucks. 

One thing definitely unrelated to code quality is code coverage. All code coverage tells you is what percentage of your code you haven’t tested at all. If you squint and have a few drinks, you can pretend a little that high coverage rates from unit tests prove code quality…but once you sober up you’ll realize that all you have are unit tests that exercise some amount of your code.

But we can go beyond code analysis tools. Testability (can I observe and control program flow), and Maintainability (“Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.” ref) are critical to code quality. Given a good reviewer, Thom Holwerda’s comic is also spot on – the only valid measurement of code quality is WTF’s per minute.

As I think about this, I find my mind wandering back to Pirsig. Quality and care are two different ways of looking at the same thing. Quality comes from care, so perhaps what I want to measure is the care put into coding. The fact that a developer wanted to compile at a higher warning level or turn on more static analysis checks, or that they cared enough to get the testers input on testability, or a peer’s advice on maintainability may be the best ways to measure code quality of all.

So now, perhaps, I’m in search of a way to discover and measure care.

Historically, most of my blog posts come from experiences or inspiration from my day job (despite those who seem to think that every snide comment I make is about them, my snide comments are almost entirely about people I work with). While I’m sure that will continue to be the case, I’m still pretty deep in ramp up mode, and it turns out that the bulk of the work I need to do right now is solving a variety of test infrastructure and code issues. While I find this sort of work exciting to perform, I find it dreadfully boring to write about (apologies to all those who enjoy writing about these things).

But I do enjoy writing, and even as I write this trivial little blurb (a blurb that will turn into a blog post in about five minutes), I realize that I have a huge backlog of items not contrived from my day-to-day work that I can draw from in order to satisfy my need to write and share a few times a week.

Meanwhile, some chunks of my work are about things that I can’t blog about. In some ways, those are the most exciting. I have plans to share some of my thoughts on this when it’s appropriate, and I think it will be an interesting exercise (and hopefully, interesting reading when I can share). We’ll see what happens.

I wrote 60 blog entries in 2011. Some of the more popular posts (in no particular order) were:

• My job as a Tester
• What is Testing?
• Test Design for Automation
• Numberz Challenge
• Beyond Regression Tests
• R-E-S-P-E-C-T
• Judgment in Testing
• Lost in the weeds

I did a bit of speaking in 2011 (although, as usual, more than I planned). I was a pnsqc, at Intel in Israel, and at the Swiss and German Testing Day conferences. I also attended the second Writing About Testing conference (and had a great time).

Next year, I’m trying (again) to speak and travel less, blog about the same amount, and keep having fun working on the Xbox team. For everyone who’s read or posted a comment, thank you for supporting my silly addiction to writing. I hope I’ve given you something to think (or complain) about.

Best wishes for the holidays and 2012.

I’m a Senior Tester, don’t you know

Been testing software for five years or so

I got bills to pay and need a better job…

Looking for jobs with ‘Senior’ in the name

Gotta’s admit, they’re all pretty lame

Looks like I’m over-qualified…

Someone (non-Microsoft) asked me recently what a “Senior Tester” does. I told them that I knew in the context of Microsoft, but really had no idea if the term meant anything in the industry. After a quick (and painful) trip to a jobs website, I was able to confirm that the ‘Senior’ title doesn’t carry a lot of weight. Consider some of these quotes – all from positions advertised as “Senior Tester”, or “Senior QA Tester (on a side note, wtf is a QA Tester?):

• With minimal direction writes scripts and executes test cases during functional/system/integration testing.
• Modify existing test plans and testing information to correct errors, to adapt to new test scenarios, upgraded interfaces and performance enhancements.
• Monitor bug resolution efforts and track successes.
• Participates in analysis and design walkthroughs, as well as team meetings
• The Senior Tester will work extensively with the Test Lead to provide test estimates
• Complete knowledge of QA Best Practices and the full Testing SDLC

Let me rant on the last bullet point first. Actually, never mind – anything I can say is something you’ve probably already thought.

As for the first five bullet points, are those really considered “Senior” activities? I was surprised that being able to cross the street by themselves or washing their hands after using the bathroom weren’t required attributes. Then again, I’ve been sheltered in the world of Microsoft for the past 17 years (and as a noob in a startup for 2 years before that), so my world view of testing has limitations.

So, let me talk about what I do know. Here at the evil empire, we have standard titles for all roles (more details on test titles are here). As I mentioned in the link (which I know you haven’t read yet), we have a “Senior SDET” title at MS, and this is where we expect testers to exhibit some leadership skills (they learn how to find the bathroom by themselves at a much lower level). To be clear, leadership does not mean management. ‘Senior’ is the minimum level where we like people (who are interested) to move into management roles, but we have hundreds of non-managers at the Senior level (about 8% of the total test population if you’re curious).

My random search for Senior test roles wasn’t entirely in vain. Consider the following:

• Experience with leading testing efforts on integrated teams and developing and implementing sound testing methodologies
• This is a senior role that includes evaluating existing automated testing process and recommending improvements

Now those seem a little better – now the role requires some leadership and decision making, so let’s look a bit closer at these a few other attributes I expect to see in Senior testers at MS.

• Decision Making – senior testers make confident decisions, and do not rely on consensus for making (most) decisions.
• Influence – Dwight Eisenhower said, “Leadership is the art of getting someone else to do something you want done because he wants to do it.”  Senior testers make their team better through influence rather than mandates – they are the point guard of the test team.
• Network – senior testers use their network to find answers (or knowledge), and connect their teammates with their network (i.e. tester matchmaking)
• Credibility – senior testers don’t proclaim they’re a leader, and no title demands respect. Leaders know that respect as a leader comes through establishing credibility with their team and peers. For me, the moment you demand respect as a leader is the moment you lose my respect as a leader.

That’s not an exhaustive list, and each of those points could be (and may be) a post unto its own. But that’s what ‘Senior Tester’ means to me (In my sheltered life). Undoubtedly, your definition is different, and that’s fine. Regardless of where your opinions lie, I hope this provides some food for thought on your own views of what ‘Senior’ means.

Admittedly, I’m still in the honey-moon phase, but I’m having a great time. But life just doesn’t seem to be as fun without a surprise here and there, and this career move wasn’t without exception. More on that later – but let’s first consider a bit of career advice.

One tidbit of career advice I often give is “The 3 Ps”. When you’re looking for a job (or evaluating your current position), you need to consider the Product, the People, and the Person.

• Consider the Product – You’ll be more motivated and excited about your work if you have interest in the product (or the technical challenges regarding that product). On the corporate applications side of Microsoft, I honestly think that Lync (my old group) is freakin’ cool, but there’s no understating the sexy coolness of Xbox and Kinect either.
• Consider the People – Nobody wants to work with a bunch of jerks. I don’t want to work with people who don’t care about their jobs. I happened to know dozens of people working on XBox (I worked with many of them on Windows 95 & Windows 98), and all are great people. When I was considering the move, I chatted with people on the team that I didn’t know – and they all had this vibe of loving what they do. To me, that’s the sign of a good team to work on.
• Consider the Person – We all have to work for someone. Take time when considering a new position to make sure that whoever you report to (and ideally, their manager) are on the same page as you; and will support you and help you grow.

To me – the Person part of the equation is critical. I’m in a rare position at Microsoft – we don’t have many testers at my level who do not manage teams. Managing someone like me, and communicating my value (or lack of value) appropriately is a challenge. Because there are few of us extra-senior testers, there are very few managers who have experience using us effectively. I selected my last manager based on this criteria, and when I moved to Xbox, I used the same criteria. It was hard to leave a team where I had the 3 Ps, but the time was right, and I took the time to ensure that I would have the 3 Ps on my new team as well.

Three days into my new job, my manager called me into his office and said, “Welcome to the team – we’re having a re-org”. The details of the re-org aren’t important, but they make perfect business sense, and I’m excited about the change. Unfortunately, the manager I had carefully selected wasn’t going to be my manager any more – and with the upcoming release of the new dashboard, I wasn’t sure where I was going to land until a day or two ago.

So – with one manager on the way out, and another yet to be defined, I fell back on what I seem to do best – I did whatever I wanted. In the land of Xbox, I suppose that could mean a whole lot of Skyrim and Forza, but I dug deep into some testing challenges, looked at engineering systems, got to know people and basically spent a whole bunch of time figuring out how the org and technology work. I once considered a position at a company full of young folks where someone suggested that I spend at least a month or two “doing whatever I wanted” until I figured out what I really wanted to do. My recent experience wasn’t quite as directed as that, but I find it interesting that concept seems to repeat throughout my career.

The good news is that I’ve met with my new boss a few times, and I think we’ll work very well together. We’re still figuring out exactly what I’ll do (which is perfectly ok), but I’m completely comfortable with how he wants to use me, and although he’s still a bit of an unknown to me, I’m pretty confident my 3 Ps are still in place. I’m excited, and a little scared (another bit of good career advice), so I know I’m in the right place.

I see this concept come up frequently, and I’ve noticed a bit of a trend. Teams that follow Just Fix It love it. Teams that prefer to fix bugs later are sure that the concept won’t work for their team, and that they need the curve and tracking data in order to ship their (undeniably unique) product. As a side note, one fun thing I’ve talked a few teams at Microsoft into doing is to pair on bug fixes – when Joe-Tester finds a bug, he Tells Kathy-Developer about the issue, and then the two of them pair-program the fix. I could write a whole post on why this is so cool, but I’ll leave it at that for now.

In short, I’m a huge fan of Just Fix It, but as usual, the totality of overlap of agreement between Chris and I is about 93%.

For example, let’s say on Monday, Alex-Tester notices that the froobnozzle isn’t working. He tells the developer, who fixes the problem immediately. Tuesday, Beth-Tester notices that the froobnozzle doesn’t work when interacting with another part of the system. She tells the developer, who fixes the problem right away. Over the following days and weeks, a lot of problems are discovered with the froobnozzle. The froobnozzle, is, in fact, a piece of crap held together by the 1s and 0s version of spit and duct tape. A bug tracking system let’s you see (I hate to say this out lout…) trends of where errors are. Bugs don’t appear randomly sprinkled throughout a product – they tend to congregate in clumps. Knowing where the clumps are can guide further testing, or risk decisions.

Source control almost mitigates this concern, but unless you have a diligent comment policy for check-ins, you probably won’t be able to differentiate between a “I was adding new code or functionality” check-ins vs. “I was fixing shit I broke” check-ins.

But I won’t go as far as to say you need a bug tracking system. As Chris describes it, and as most people use them, a bug system is really a work-item tracking system anyway. If you track work items on post-its or notecards, bugs should work the same way. As far as trends go, I think a simple tic-mark system would work just as well. When you discover a problem with the froobnozzle, write it on the board and put a tic-mark next to it. When a component gets n number of tics, schedule refactoring time, or do a design review (or both). Alternately, look at the components with the highest number of tics during the retrospective or sprint planning and review those for potential re-work.

But other than that, yeah. Just Fix It!

Note: I would have posted this as a comment on Chris’s blog, but I, for one, find the comment forms on blogspot completely unusable.

Let me outline the problem. I don’t look at email or twitter or my rss feed all of the time – in fact, most of my electronic communication time occurs when I’m compiling / building (I could use that time to have a sword-fight, but I find it better to at least pretend I’m productive). I love to group the “non-essential” stuff together, and when I’m waiting on a build, it’s the perfect time to write email, documents, tweet (or sometimes blog).

The problem is, that when I’m compiling, my 12 cores are pretty busy. This makes reading email, or doing pretty much anything else that would like more than a few time slices of cpu power sort of suck.

I have two computers (a desktop and a laptop), so I have an option of using my laptop for those tasks while the compiler elves are taking over my computer, but given the choice of using a 24” monitor and a full size ergonomic keyboard over a 13” monitor and compressed keyboard, which would you choose? To be fair, I adore my laptop (Lenovo X200s) – but I adore it because it’s small and I can use it anywhere (including on an airline tray table while the monster man in front of me leans his seat all the way back for 8 hours straight). But it’s not a computer I like to write large documents or presentations on.

The solution, as many have already guessed, is to connect to my laptop via terminal server and use it from my desktop machine. The overhead of TS is small enough, that I barely notice a perf problem even at peak memory and cpu usage. The TS session even stretches my desktop to 1920×1080, so it’s a pretty sweet setup. All that is good (and somewhat obvious), but few things in life ever go without a hitch.

Despite how much I wanted to love the setup, I noticed that my connection would drop frequently – and then it would take a minute until I could reconnect. I powered through it for a day or two (I tell myself that I was subconsciously gathering clues). Then finally, I found the clue that led me in right direction. I listen to music on Zune quite a bit while I work (I configured terminal server to play audio on the server machine (my laptop), so I select songs from the TS session on my desktop, and listen through headphones connected to my laptop). So…I noticed that whenever my computer disconnected, I lost audio a few seconds later. The first few times this happened, I (ignorantly) assumed that because I lost the TS connection, I also lost the audio. Then it hit me that since the audio is playing on my laptop, I should never lose audio…unless the network connection was lost.

Losing the TS connection and losing audio were both symptoms of the same root cause (lost network connection) – the only thing left to figure out was why two computers on the same subnet were losing their connection.

I took a look at power management settings to ensure that my laptop wasn’t suspending, but all was good. My next step was to look at my adapter settings to see if the network adapter was powering down for some reason. Sure enough, there was a setting named “System Idle Power Saver” that was enabled. I disabled the setting, and I’ve only had one dropped connection all week.

This example may sound more like yak-shaving than testing, but I’ve always liked the trouble shooting aspect of testing. Going beyond “here’s a problem”, to “here’s what (probably) caused the problem” is, in my opinion) an important skill for testers.

Oops – my build is done – time to get this posted.

So – let me throw a few examples of potential testing activities out there for discussion or comment. For the scenarios below, consider two questions:

• Does this describe a testing activity?
• What role typically performs this activity?

Alex practices Test Driven Development. He consistently writes unit tests before implementing the code those unit tests verify. Over time, he’s built up a large set of unit tests that verify the correctness of all of the modules he owns. His unit tests give him the confidence to refactor or add functionality at any time.

Beth works closely with another teammate to ensure that the code implementation is highly testable. This includes pairing on many of the coding tasks, as well as adding hooks or accessible interfaces to the implementation when necessary. Her work results in code that the team finds straightforward and easy to test.

Chuck spends a lot of time analyzing code. Mostly, he’s a rock star code reviewer, and has found dozens and dozens of critical security and reliability issues in his team’s code base during the review process. He often uses tools and techniques like static analysis and complexity measurements in order to target his efforts. The whole team looks to him for critical code reviews, and he’s a mentor for many of the new team members.

Denise painstakingly enters test cases into a test case management system, and then regularly runs subsets of those tests as dictated by her management. She likes the predictability and structure of this approach and his attention to detail makes her successful in her job.

Frank finds product bugs by the handful. He uses a variety of test techniques, approaches, and tools to find several high quality bugs every day. Everyone on the team values his efforts, and many people on the team look to him for coaching on test approaches. His approaches are unscripted, but nobody doubts the effectiveness of his approach.

Fiona spends most of her day writing automated regression tests, and is constantly learning new approaches to writing automation. She writes high quality, trustworthy code that has few false positive / false negatives, and has earned the respect of everyone on the team. She enjoys writing automated tests and is proud of her accomplishments.

Greg works closely with the user experience and project management teams to ensure that the customer experience of his team’s product is of high quality. He frequently analyzes reports from customers and makes sure that all relevant team members know what the most critical issues customers are facing. He uses his broad knowledge of customer data to be influential on both product and test design.

Helen is a bit of a tool-smith. Although her team uses a lot of open source tools, she writes the applications and utilities that stitches the tools into an end-to-end testing framework. She loves thinking “big picture” and ensuring that all of the components in the system work together seamlessly. She doesn’t perform any product testing, but her work enables many of the testing activities to be successful.

To me, I could make a case that each of the scenarios above describe a testing activity – but I could also make a case that many do not. I’ll leave my observations at that for now and see what happens in the comments.

For those interested, here’s a quick recap of the trip.

November 6-7

I flew from Seattle to Detroit, then on to Frankfurt. I just reached “Silver Medallion” status on Delta, so I got a free bump to economy-comfort class and an extra few inches of room on the Detroit to Frankfurt leg of the trip. I’ve been to Frankfurt at least a dozen times, but this was the first time I’ve left the airport (travellers in the audience already know that Frankfurt is a busy airline hub).

I arrived at my hotel (~10 kilometers or so outside of the downtown area) around noon. In an attempt to fight off jet lag, I dumped my bags and then set off to explore. The train station was a 5-minute walk away, and within 20 minutes or so I was in downtown Frankfurt. I took a break at a Starbucks (who apparently have free wireless worldwide these days) to sync my mail and do a bit of work before grabbing some dinner and heading back to my hotel to sleep.

November 8

After a pretty good night of sleep, I had just 3 things on my agenda. 1) Explore more of Frankfurt; 2) Prep for my presentation; 3) Meet the conference board for dinner at 8:00pm.

I combined the first two by working at a coffee shop until I couldn’t stand the wooden seat anymore, then walking around Frankfurt until I couldn’t stand that anymore. Try as I might to park myself somewhere other than Starbucks, it was the only place I could find that had free Wi-Fi access. Coffee Fellows and O’Reilly’s pub both had free Wi-Fi, but they both required that I have a working mobile number (I asked about this at O’Reilly’s, and they said it was so they could track me down if I downloaded exceptionally large amounts of data).

One interesting discovery was a “Hammering Man” sculpture, since we have something almost exactly the same in Seattle (I’ve discovered these are two of three Hammering Men in the world).

By 4:00 or so, I was feeling a little drained, so I went back to my hotel to dump my laptop and take a quick nap before heading off to dinner. It was great seeing Adrian Zwingli from Swiss-Q again and catching up with him. He’s such a passionate person and the perfect person to run these events. After a few beers and a plate of cheese Spätzle that was just a bit too rich to finish, we all headed back to the hotel.

November 9

After another fairly good night of sleep, I got up, prepped a bit more, then took a taxi to Jahrhunderthalle – the site of the conference. I gave the opening keynote of the conference (no pressure!) on Test Innovation at Microsoft. I haven’t given a talk on this topic before, and I had a huge number of insights and revelations during my preparation. I think this is a topic I will continue to develop (both test innovation in general, and sharing the amazing test innovations at Microsoft). The GTD folks got the first glimpse into some of the cool things MS is doing in testing – as well as some tips on how to make innovation happen (hint – you don’t start by saying, “I think I’ll innovate today!”). I felt like the talk went well, but also think that this is a talk I can repeat and improve on in the future as my own thoughts on the subject evolve.I did have quite a few people give me compliments on the talk during the day, but I also realize that people generally don’t track down a speaker to tell them that their talk sucked.

I spent the rest of the day listening in on some of the English talks at the event, and getting a bit of work done. A pair of presenters from eBay talked about test automation principles, and they had a good message (meaning that I agreed with what they had to say, and think they have a good approach). I talked to them a bit afterwards too, and they seem like great people too.

After the event, Sebastien Kernbach (Conference Head from Swiss-Q) one of his colleagues (a woman whose name I have, unfortunately, forgotten) and I had dinner together at the hotel before heading off to bed. Not a crazy night at all, but I was beat, and ready for sleep.

November 10

Time to fly home. I took the hotel shuttle to the airport, had some breakfast, and got on my flight. The connection at JFK went smoothly, and I was home, in my own bed by 9:00pm Seattle time.

Overall, I had a great time on the trip, and once again, the Swiss-Q folks showed me that they know testing and know how to create test events. I was honored to have been invited, and hope I have the opportunity to present at another of their events sometime in the future.

But for now, my travel schedule is completely empty, and I plan to keep it that way for as long as I can get away with it.

But sometimes, even tenured professors are drawn by the lure of an insanely hard challenge. If bigger challenges weren’t available, I suppose tenured professors would stay in their jobs for decades (and I know that many of them do). However, I know of several of my own former professors who found new roles lured entirely by the challenge of the unknown. Even if Indiana Jones would have been tenured, I’d bet he’d have left his position if he had the opportunity for the right archaeological challenge.

So – even though I’ve only spent a relatively small amount of time with the Lync team at Microsoft, the lure of the insanely hard problem has pulled me away. About a week ago, I began a new position on the Xbox team where I will work on some of the interesting challenges in testing the xbox platform, including a deep-dive into testing the Kinect device.

The Lync team was (and still is) awesome, and will always be one of my best memories of Microsoft. They will continue to be wildly successful without me (and to tell you the truth, the Xbox team would be successful without me too), but it’s the right move for me to make right now.

Along with my new job, I have a new office, a new desk, a new computer, and a new Xbox – and testing challenges I have no idea how to solve…yet. And I’m pretty excited about all of it.

More to come.

At this point, I’ll give you a few moments to chuckle, laugh, or make snide remarks about the topic. I think I’ve heard it all before; “testers at MS just write automation all day”, “testers at Microsoft don’t care about quality”, or “testers at MS don’t get testing” (all quotes I’ve heard or read about MS testing – and all, interestingly, by people who have never worked at Microsoft). This post is, I suppose, inspired by my talk tomorrow, but is more of a general example than a reflection of the topics of tomorrow’s presentation.

I think our lack of open-ness in how we test and innovate in testing at Microsoft is (IMO) one of our biggest flaws as a company – or at least one of our biggest flaws in being a good member of the worldwide testing community. We tried to make a dent with testercenter, but that project is all but dead. A few of us who test at MS share what we do, but none of us do a great job sharing our innovations. I hope we can get better at that soon. The short and honest truth is that I can easily name at least fifty testers at MS whose testing skills and ability to innovate are equal or better than any tester you know. I suppose in a company of nearly 10k testers that’s not really a big deal, but it’s really hit home as I’ve prepared for this presentation. I have some ideas for getting some of these people and their ideas on the global radar, but for now, you’ll just have to trust me.

I’m probably not the right person to lecture (or write) about innovation, but I can tell you what kinds of innovative testing solutions tend to be successful (at MS at least). A lot of innovation starts with the “There’s got to be a better way” approach. The worst rut to get into in software development is the “we’ve always done it that way” syndrome. Innovative testers stop, look at the big picture, and ask, ‘”is there a better way to do this?” Some of the biggest test innovations at MS have been solutions that take something complex or difficult (or “impossible”), and making it brain dead simple. One tester created automatic filtering of code coverage reports based on a specific change set – in other words, he made it brain dead simple to know if you (or any tester) tested each changed line in a specific fix or feature addition. Some of our games testers have created a system that allow the bug triage team to navigate to the exact place in the game where an error occurs from the bug tracking system (and conversely, the ability to create a bug from the place in the game where the tester observed the error).

Other innovation happens when someone finds a way to implement a known idea into a testing process. Testing localization is a huge (and hugely expensive) task. Ross Smith (and a few of his cohorts) had the idea to optimize the task via crowdsourcing and gaming elements.The results have been phenomenal.

Eric Sevareid once said,

The chief cause of problems is solutions.

Many years ago, test systems at MS reached a level of maturity and scale that enabled teams to easily run millions of tests. These systems[1] were the culmination of several innovations, but eventually led to other problems. The prime example of a problem created by the test system innovation is the challenge of what to do with all of those failed tests. Think for a moment. If you run a million tests and have a 99% pass rate (I know, but this is just an example), you have ten-thousand failures to investigate. Automatic failure analysis (discussed in HWTSAM) is an innovation that greatly reduces the time testers need to investigate automation failures (my stance is that automation is worthless unless every aspect of automation beyond test authoring is not also automated). We’ve also made great strides in test selection (let’s say you only had time to run five thousand of those million tests – which would you run?).

Of course, innovation occurs in the testing itself (as well as in the systems surrounding the testing). We’ve done amazing things with test generation, test design, and run time analysis.

Some of which I’ll share with the audience tomorrow – and likely in this blog sometime in the future.

[1] I described a test automation system in Beautiful Testing.

Late Saturday night, I returned from my first trip to Israel. I was invited to present at the Intel Software Professional’s conference in Tel Aviv a few months back, and I gave an updated version of my STAR East 2010 keynote on improving methods of gathering customer usage data. I also spent some time at Microsoft’s office and did a bit of sightseeing. It was a quick, but fun trip that I would do again in a heartbeat.

The long version follows.

Monday & Tuesday, October 24 and 25.

My flight out of Seattle was mostly uneventful.As someone who hates to check bags, I was a bit bummed to find out that Lufthansa had a carry-on limit of 8 kilos (my non-full and size-appropriate bag was 10 kilos). I’ve had enough bad experiences with lost or delayed luggage that I prefer to carry-on for anything less than a week. In the end, it didn’t matter (more later).

I flew from Seattle into Frankfurt, and arrived with just enough time to run to a different terminal and catch my flight. I had 5 or so minutes to spare, so I suppose I could have just jogged, but after the 11 hour flight, I needed a bit of exercise anyway.

I landed in Tel Aviv mid-afternoon on Tuesday. Since I sat in row 54 of an HFP (huge plane), I was just about the last person on our flight to arrive in a packed passport control room. After spending some time in the second slowest line, I switched to the actual slowest line and eventually was allowed into the country proper. By this time, my bag was off the plane and was one of the few left on the carousel, so I grabbed it, found a taxi, and went to my hotel.

I stayed at the Dan Panorama in Tel Aviv. After admiring my beach view and unpacking, I found something to eat and went to bed.

Wednesday, October 26.

I used to never have a problem with jet lag. As yet another sign of my age, I didn’t get a great night of sleep, but at 5:30am I got out of bed and went for a run on the beach (or next to the beach). It was fantastic – perfect weather, the sound of the ocean, and a bit of blood pumping through my system gave me energy to face the day.

Microsoft Israel has offices in Haifa and Herzelia and I spent the day in Herzelia talking with testers in a variety of settings. Surprisingly, I had energy all day and had some good conversations (from my side at least – I hope the folks in Israel found the discussions helpful).

Then it was back to the hotel for dinner, a bit of work, and bed.

Thursday, October 27.

Once again, I didn’t sleep well (I can’t believe I claimed for years that jet lag was something I never worried about). The good news is that I finally finished reading Life -  the Keith Richards autobiography. It’s not the most well written book, but to me, it was a fascinating page turner. I’m not a huge fan of the entire Stones’ backlog, but to me, the 10-year period from Beggar’s Banquet through Some Girls is some of the best rock music ever made.

Since I’d given up on sleep, I loaded a dozen or so of the Rolling Stones singles from that period and went for another run on the beach. When I got back, I had breakfast, checked in with Shmuel (my host from Intel), and …went back to bed.

My presentation at Intel was in the afternoon, so I had time to get some rest, wake up, and prepare a bit more. From my point of view, the presentation went well – the audience seemed engaged and receptive, and I received good comments afterwards. I feel fortunate to have been invited and hope the folks at Intel found my presentation useful.

Shmuel and I went out to dinner, then we both called it a night.

Friday, October 28.

Shmuel picked me up Friday morning for some sight-seeing in Jerusalem. First stop was the fascinating (and somewhat depressing) Yad Vashem – the Holocaust Memorial. From there, we went to an open market, wandered a bit and had lunch. The final stop of the tour was Old Jerusalem and the Western Wall. I was fortunate enough to meet Shmuel’s wife and five children there as well. After a bit more wandering, I headed  back to Tel Aviv.

I had a plan for my Friday night, and I’m happy to say it worked out well. When I got back to my hotel, I took a 5-6 hour “nap”, and woke up (as I hoped), around 1:00am. After freshening up a bit, I took a 15 minute walk up to Mike’s Place to watch game 7 of the baseball world series. It’s a bit strange watching a baseball game that starts at 2:00am, but it was fun to be able to watch the game (go Phillies), and sort of surreal to walk back to the hotel along the beach at dawn after the game.

Saturday, October 29.

After showering at the hotel and packing, I left for the airport. Leaving Israel is an interesting experience. My “exit interview” was 20-30 minutes long and reminded me a lot of the police interviews you see on television and the movies. I pretty much told them what I did every minute I was in Israel and they doubled back once in a while to check my answers for consistency. I was tired, but had nothing to hide, so found the experience kind of fun. Once the interview was over, I checked in and made my way to the gate.

The flight home was long (made longer by delays in Newark due to the weekend snow storm), but I did eventually make it home and spent Sunday relaxing with my family. Monday would bring a whole new adventure (and stage) to my life, but I’ll save that for the next post.

In my last post, I reflected on a wonderful article from Phillip Armour. His breakdown of “tests that tell us what we know” (0OI tests), and tests that tell us what we don’t know we don’t know (2OI tests) aligns perfectly with something that’s been gnawing at me for a long time.

You see, I’m also bothered (and have been for years) to some extent by one of the more popular testing debates – the scripted vs. exploratory tests…and to top it off, I’m completely confused by the variety of reactions to the “test is dead” statement one of my old friends has been spouting about recently (I warned you about random thoughts floating in my head…)

But it’s kind of coming together for me – some of it as I write these paragraphs.

A part of the problem is that what I call testing is probably different than what you call testing. That’s also true if I turn it around – what you call testing is probably different than what I call testing. The difference is, that I don’t care what you call testing, and some of you care a lot get really pissed off when somebody calls something testing that doesn’t match your definition.

Testing, in some intersection of our combined testing definitions, contains a lot of 0OI tests. “When you enter 2 + 2 in the calculator, the display shall enter 4”, or “Save the file and ensure it is created”, or “When the user clicks the submit button without a user name, they shall receive an error message”. Many in the test community refer to these as “checks”, and I’m ok with that term, but to me, these are a type of test rather than something different than tests.

Whether you call these tests, checks, or 0OI tests (my current favorite), I don’t want to do them. It’s not just that 0OI tests bore me (because they bore me to tears), but because I think it’s fiscally irresponsible to push verification of mind-numbingly basic functionality so far downstream. In my opinion, developers should do this work, but I guess if any testers find 0OI tests entertaining, I’m sure there’s a lazy developer somewhere who’d like to hire you as his cabin boy.

Short story is that 0OI tests are dead to me.

And then there’s 2OI work – the kind of work good testers love to do. 2OI testing is the land of “I wonder what happens when…”, and “How can I get the program to…” tests. 2OI tests can be manual or leverage tools or automation. Any form of discovery (or new knowledge acquisition) about the software is 2OI testing.

Testers not infatuated with Armour the way I am typically call 2OI testing exploratory testing (true for both manual or automated 2OI tests). In Armour’s recent article on testing that I mentioned in my previous post, he suggests that the sweet spot for 2OI (exploratory) effectiveness is about 50% – meaning that about half of your tests should discover product issues. If you’re finding less, you could vary your techniques, and if you’re finding more, you should back off the effort. I haven’t applied that concept yet when performing testing, but I plan to the next time I conduct a testing session.

But, the way I see it, exploratory testing is dead too. OK – that statement was overly controversial. What’s dead to me, is the term exploratory testing. To me, it’s just testing. It’s all just testing. I wrote a blog post once about this subject (see last three paragraphs to avoid most of my snarkiness) – then one of the top practitioners of ET wrote this post with a similar stance – but it seems every tester in the world is still compelled to proclaim they are “exploratory testers”, because they like to use their brain while they’re testing. All of this is fine, of course, one can use whatever terms they like, but to me, at least, it’s all just testing.

Testers do some of it. Others do some more of it. Some of it’s dead to me, and perhaps none of it is dead to you. It doesn’t matter.

It’s all just testing.

…the paradox of testing: We want to find defects in the software under test, but we do not really want to find them".”Phillip Armour

I often speak of Phillip Amour’s Five Orders of Ignoranceas they relate to software engineering (note; as of this writing, that link doesn’t work, but I don’t have another link to his paper). I’ve mentioned the orders of ignorance on this blog, and also in this article for Better Software.

Although I respect my readers, I know you won’t read those links, so let me briefly cover the orders of ignorance.

0OI – Zero-Oh-I is lack of ignorance. It’s when you know something. I know, for example, that the first track on the Rolling Stone’s Sticky Fingers album is Brown Sugar.

1OI – lack of knowledge. I don’t know (or remember) what the track is following Brown Sugar, but I could find the answer quickly.

2OI – lack of awareness. You have 2OI when you don’t know what you don’t know. I know that there are Stones tunes that I’ve never heard before, but it would be impossible for me to make a list of Stones songs I’ve never heard before.

30I – lack of process. You have 3OI when you don’t have a suitable method for discovering 2OI (for discovering what you don’t know you don’t know).

4OI – meta-ignorance. You have 4OI when you don’t know about the five levels of ignorance.

In the Better Software article, I attempted to stress how important the levels of ignorance are to software testers and software testing, and I frequently bring them up in conversations about testing. Recently, my mind was blown when Phillip Armour wrote an article specifically about the application of the levels of ignorance to software testing. A direct link (for those who have access to the communications of the ACM) is here. If not, I found a link here which should get you to the article.

And for those of you who don’t click through, here’s a summary.

When we test to ensure requirements were implemented, or the stories function as expected, we are testing for 0OI. We know what pass and fail looks like, and we can create tests accordingly. The 0OI testing set may be large, but it’s bounded. A passing 0OI test doesn’t expose any new knowledge – it just proves what we thought we already knew.

Testers also perform a lot of 2OI testing. We explore or run dynamic tests on a system to help discover what we don’t know we don’t know. Testing for 2OI is an unbounded test set – the things that a system might do is infinite. This is where we need to devise tests when we don’t know what we’re looking for. (Note – we don’t test for 1OI because if we truly knew in advance what we didn’t’ know, we’d resolve the ignorance and conduct 0OI testing.

Another (shorter) way to think about this is that 0OI tests are knowledge-proving tests, while 2OI tests are knowledge-acquiring tests.

Armour goes on to discuss the theoretical information content for a test and optimal failure rates (which I’ll leave to you to read, as I’ve probably already butchered the content already). It’s good stuff full of great quotes, and I hope you check it out.

With about five minutes left in the match (the Sounders were ahead 1-nil, and closing in on their second goal), a group of men came into the bar and filled 10-12 seats at a neighboring table. One of the men – I’ll call him Frank looked at the three of us watching the match, looked at the screen and claimed loudly, “That’s not a game!”. Apparently, that’s all of the commentary vocabulary Frank had, because he repeated the claim to his table-mates several more times during the remaining minutes of the match.

A part of me (perhaps the soccer hooligan in me?) wanted to confront Frank and ask him why he didn’t consider the most popular sport in the world to be a game, but I held my tongue and put my focus on watching the sounders score late to seal the game, and on their subsequent celebration.

But then I thought about Frank.I wondered what circumstances would lead to his statement. First off, soccer isn’t that popular in the US, so he’s certainly not the only one around who doesn’t value (what I see as) the beauty of the game. It also looked as if Frank and his friends came from a sporting event (it’s not softball season, but it could have been flag football). Perhaps Frank’s team lost, or he was frustrated with his performance. I don’t know many people who would purposely act as he did without some sort of other motivation. In short, there’s a lot going on with Frank I don’t know about, so I gave him the benefit of the doubt. Over time, I’m sure I’ll see Frank again, and as I get to know him (even solely via observation), I’ll get more insight into his motivations and know what drives his statements and social behavior.

This is how I try to behave at work. One of Microsoft’s core employee competencies is something called Interpersonal Awareness (which I prefer to call IPA). Most people think they’re pretty good at IPA because they get along with others, but that’s only a tiny scratch in the surface of IPA. In fact, I’ve stated on a few occasions that IPA is one of those skills that you’re not any good at until you realize that you’re not any good at it. When you think you have it down, you probably suck at it.

IPA is more than playing nice with others. It’s more than paying attention to how you come across. It’s reading people (including yourself), and adjusting to how you’re coming across based on reactions and words, while taking into account whatever may be motivating the people you’re interacting with – in real time. Yeah – I suck at it too, but that doesn’t mean I don’t work on it.

Let me try to bring it home. Think of your favorite “stupid” manager, “dumb” tester, or “egotistical” freak. Chances are, that those people are neither, stupid, dumb, nor egotistical. They have, however, acted that way and it caused you to react. There are two things going on here – you’re reaction and their behavior. Take a moment to think about why you had a negative reaction to whatever they did or said. Does it create more work for you? Does it violate your values? There’s a reason, and it’s worth it for you to take the time to understand why you reacted the way you did.

Now dig into the “stupid” manager’s statement. Perhaps he asked you to increase your code coverage percentage to some arbitrary number, and you think (rightfully) that increasing code coverage just to hit a number is s dumb idea. But first, take some time to think about why Mr. Pointy-Hair would have asked you that. Do you think his manager asked him? Do you think he thinks the number means something. Do you think he’s under any pressure himself that may have triggered him to ask, or is it something you think he’s valued in the past? Chances are, that Mr. PH isn’t stupid (despite his request). But until you know what motivated him to make the request, you can’t actually give him what he needs vs. what he (thinks) he wants. You probably won’t guess motivations right away, but without practice, you’ll never figure it out.

Think about this the next time you read (or say) that something is dumb, stupid, idiotic, or even careless. I may be overly optimistic, but I don’t think there are nearly as many idiots in the world as some would lead you to believe. But smart people say dumb things all the time – and without understanding the motivation behind their statements, we can’t grow – as professionals, or as humans.

I’m off to Portland, Oregon tomorrow. I’m planning to spend the weekend goofing off before frantically updating my slides Sunday night. My PNSQC presentation is Monday at 11:00, and I’ll spend the remainder of the conference trying to attend as many presentations as I can.

I return home on Tuesday evening, and on Wednesday morning, I’ll be giving a presentation on build verification tests for the STP automation summit.

Then – after ten or so days of time at home, I’m off to Israel. I’ll be spending a day at Microsoft’s Israel office, and then giving a keynote at an Intel internal software conference. I’m honored to have been invited and am looking forward to my first trip to Israel.

I get home just in time for Halloween, but depart a week later for a keynote at German Testing Day. Again, I’m honored to have been invited and I’m excited both to attend, and about the topic I’ll be speaking about.

Frankly, this is probably more travel and presenting than a non-consultant should be doing, but I do believe there’s value in establishing and maintaining industry relationships, and I’m grateful that Microsoft supports my participation (and time away from work). As much as I know I’ll enjoy the travel and meeting so many testers, I know I’ll be glad when the flurry is over.

In between all of that, I have some interesting projects brewing at work – and I can’t wait to share more details…stay tuned.