FrazzledDad

Sample Code & Slides from StarEast 2013 Talk

2013-05-15T09:59:00.001-04:00

I gave a talk on Four Tips for Web Automation at StarEast 2013 this April. I’ve posted up my slides on SpeakerDeck

You can also find the demo code at my GitHub repository.

I’ll be doing a similar talk at StirTrek this Friday!

Slides From Quest 2013 Posted

2013-04-17T13:59:00.001-04:00

Thanks to the folks who attended my talk at Quest 2013! I appreciated the interaction, and I hope you found the talk useful.

You can find my slides, and the mind map, here on SpeakerDeck.

Exciting News: I’m Now an MVP for ALM!

2013-04-01T11:23:00.001-04:00

I just got news today that Microsoft has changed my Most Valuable Professional award from C# to Application Lifecycle Management (ALM). This is great news for me because this is much more in my wheelhouse both technically and passion-wise. Moving to ALM means I’ll be much more inline with things I work on regularly and care intensely about: testing, process, and making teams awesome.

I’m also very excited because I get to hang out with a bunch of awesome ALM geniuses like Esteban Garcia, Ryan Cromwell, and others. This is going to be awesome!

Oh, you thought my last post meant I wasn’t an MVP at all anymore? Oh, terribly sorry for that, but I figure there had to be something evil I could do on April Fool’s Day…

I’m No Longer a Microsoft MVP for C#

2013-04-01T11:03:00.001-04:00

Yes, I realize it’s April Fools Day; however, this isn’t a joke. After a wonderful string of years, I got notification from Microsoft that I’m no longer an MVP for Visual C#. Frankly, I’d figured this was coming some time ago and I’m surprised it took this long.

Why? Because I don’t code for a living, and I don’t focus on C#. I don’t know anything about C# 4.0, contravariance, or covariance. Ask me to write a lambda, or explain how it differs from a delegate, and you’ll likely see my head explode or me head for the nearest bottle of scotch. I was put into the C# category in 2005 because they didn’t have a better spot to put me for all the community work that I was being recognized for, and it’s always been awkward for me.

My award wasn’t so much due to specific things I’ve done as it was due to all the awesome folks in the community, and I want to make sure to thank all the wonderful people who made the last seven or eight years great. Well, I can’t thank all y’all by name, but I hope you get the idea…

My time as a C# MVP has been awesome. I’ve gotten some wonderful opportunities, and I’ve met some amazing folks.

And by the way, no, I’m not going to go on an anti-MVP rant about how the program leeches off everyone and makes MVPs lose their morals and self-worth. The MVP program is what individuals make of it. I got a tremendous amount out of it, and again, thanks to everyone for their support over the years.

Nor am I going to give up .NET and go to Ruby. Duh.

Despite this news I’m sure cool things will come along, so please don’t get your undies in a wad over my case. I’m good with it. It’s been a great experience.

I’m No Longer the Evangelist for Test Studio

2013-02-21T10:20:00.001-05:00

It’s true. I’ve been asked to give up my dream job that I’ve loved working for the last 20 or so months.

Instead, I’m now the Director of Engineering for Test Studio and am going to help with delivery of the automation tool I love so much.

Yes, I know. That was a bad trick to play. You’ve been reading my blog and haven’t figured this part of my “humor” yet?

In the new role I’ll be working closely with the teams in Austin and Sofia on planning and execution for future releases. It’s a scary but wonderfully exciting opportunity, and truth be told I had been missing being part of actively building software.

I’m not vanishing from the testing community—that involvement is a critical part of better understanding problems that we can help solve with Test Studio—but the amount and shape of my involvement will change as I move my focus to the teams at Test Studio.

One thing this brings up in a big way: Telerik needs another Evangelist to fill my position. We’re already looking for one, so this means we’ve got a second slot to fill, too!

Want to know more? Have a look at the job description and drop me a line to chat me up!

Cleaning Up Windows 2008 OS Drive

2013-02-06T10:32:00.001-05:00

I try to keep the virtual machines I use for development, demos, and testing fairly lean. I use a smallish C: partition (60GB or so) and force everything to install to a second D: drive. All my projects and data end up on D:. This has saved me several times in the past when I’ve needed to rebuild a crashed VM—just reinstall the OS and off you go. I also like to think that it helps performance, even on a VM, by keeping IO off the operating system drive. No, I’ve never bothered to actually measure performance for this. Why do you ask?

Today I ran into a rather interesting set of problems when trying to run code coverage metrics in Visual Studio 2012. Of course my tester homies all this this is epically funny and cool.

A quick bit of troubleshooting found the following, uh, minor problem:

WinDirStat is the next tool to jump on, and it quickly shows where the problem lies. The winsxs folder, where backup copies of every DLL dating back to Windows 3.01 are stored. OK, so it’s not quite that bad, but every time anything’s impacted by Windows Update or a Service Pack then cruft gets thrown in there.

Disk Cleanup is a handy utility to clean out unneeded backups of these, plus lots of other goo; however, it’s not installed by default on a Windows 2008 server OS. You’ll need to go add the Desktop Experience Feature (Server Manager | Features | Add Feature | Desktop Experience), then Disk Cleanup will be available under Start | Accessories | System Tools.

I ran Disk Cleanup and got back a huge chunk of disk real estate – Disk Cleanup also let me know I had an extraordinary amount of temp file usage, too. And here I thought I’d been doing better regular maintenance on my server…

Next maybe I should go check the last time I changed the oil in the car.

(PS: Turns out this didn’t fix the epic crashes, but it was still needed. Hopefully it’s useful to you.)

Learning JustMock for Working with Nasty Dependencies

2013-01-30T22:36:00.001-05:00

Update: I totally forgot to give mad props to colleague and good pal Phil Japikse for walking me through a couple things over IM in the course if noodling all this out.

I’ve given my Unit Testing 101 talk ten or 20 times over the last four or five years and I love it. I enjoy getting folks introduced to unit testing without having to deal with dogma issues around Test Driven Development, Test Occasionally Development, etc.

Part of the talk hits up working with mocking in order to get past a dependency on external providers. The example I use is a security provider used to check whether or not the current user’s authorized to update info for an employee.

In the past I’ve used Rhino.Mocks to work through this. I’ve used Rhino in a few projects over the years and it’s been a great tool for me.

Being the rather dense guy I am, I forgot that the awesome company I work for has its own mocking tool: JustMock. I figured it made sense to alter my tests to work with JustMock, if for no other reason than I could speak halfway intelligently to questions on it from other folks I interact with in the community.

One of the things I was most interested in was dealing with badly handled dependencies in code. The example I used previously had a nice constructor-injected interface-based security provider, so it was easy and clean to handle with Rhino. Here’s how it looks:

public class EmployeeUpdater

    private readonly IUpdateEmployeePermissible _updateEmployeeSecurityProvider;

    public EmployeeUpdater(

        IUpdateEmployeePermissible updateEmployeeSecurityProvider)

        _updateEmployeeSecurityProvider = updateEmployeeSecurityProvider;

    public Employee UpdateEmployeeRate(Employee targetEmployee, float newRate)

        Employee updatedEmployee;

        if (_updateEmployeeSecurityProvider.

            CanUpdateEmployeeRate(targetEmployee.EmployeeId))

        //if (true)

            updatedEmployee = new Employee(

                targetEmployee.EmployeeId,

                targetEmployee.FirstName,

                targetEmployee.LastName,

                targetEmployee.Address, newRate);

        else

            throw new SecurityException(

                "Invoking user doesn't have permissions to update target target employee. " +

                                        targetEmployee);

        return updatedEmployee;

A test using Rhino, complete with interaction verification, is straightforward:

[Test]

public void Verify_security_provider_is_called()

    int userId = 1;

    int oldRate = 5;

    int newRate = 10;

    var mockProvider = MockRepository.GenerateMock<IUpdateEmployeePermissible>();

    mockProvider.Expect(x => x.CanUpdateEmployeeRate(userId)).Return(true);

    Employee current = new Employee(userId, "Jim", "Holmes", "Doghouse", oldRate);

    EmployeeUpdater updater = new EmployeeUpdater(mockProvider);

    Employee updated = updater.UpdateEmployeeRate(current, newRate);

    mockProvider.VerifyAllExpectations();

I took and muddied up the EmployeeUpdater a bit to mimic code we’ve either written ourselves or have seen: “new”ing up objects instead of injecting them, and using calls to static methods or classes. These situations make it tough to mock things out and will force you to move to a commercial mocking tool like TypeMock or JustMock. [1] (JustMock has an extremely powerful free version, but mocking concrete classes, “future mocks” of items “new”ed up instead of injected, or static requires the commercial version.)

Here’s my uglified EmployeeUpdater class. I simply created two different methods, one which newed up the security provider in it, another which hit the provider via a static call.

public class EmployeeUpdaterLegacyStyle

    public Employee UpdateEmployeeRate_ProviderNewedUpHere(Employee targetEmployee, float newRate)

        UpdateEmployeeSecurityProvider _updateEmployeeSecurityProvider =

            new UpdateEmployeeSecurityProvider();

        Employee updatedEmployee;

        if (_updateEmployeeSecurityProvider.

        CanUpdateEmployeeRate(targetEmployee.EmployeeId))

        //if (true)

            updatedEmployee = new Employee(targetEmployee.EmployeeId,

                targetEmployee.FirstName,

                targetEmployee.LastName,

                targetEmployee.Address, newRate);

        else

            throw new SecurityException(

                                        "Invoking user doesn't have permissions to update target target employee. " +

                                        targetEmployee);

        return updatedEmployee;

    public Employee UpdateEmployeeRate_ProviderIsStaticHere(Employee targetEmployee, float newRate)

        Employee updatedEmployee;

        if (UpdateEmployeeSecurityProvider_Static.CanUpdateEmployeeRate(targetEmployee.EmployeeId))

        //if (true)

            updatedEmployee = new Employee(targetEmployee.EmployeeId,

                targetEmployee.FirstName,

                targetEmployee.LastName,

                targetEmployee.Address, newRate);

        else

            throw new SecurityException(

                                 "Invoking user doesn't have permissions to update target target employee. " +

                                        targetEmployee);

        return updatedEmployee;

JustMock handles both quite nicely. First, working with “future” mocking to handle the variant that instantiates the object in the method. Note I’m working with a concrete class. No interfaces, no virtual, just a straight concrete class that’s instantiated right in my method above. The final mockedProvider.Assert() validates the security provider’s method was indeed called by the updater. (Comment out the //if(true) portion in the updater to see that test fail!)

public void Mocking_provider_newed_in_method()

    int userId = 1;

    int oldRate = 5;

    int newRate = 10;

    var mockedProvider =

        Mock.Create<UpdateEmployeeSecurityProvider>();

    //Standard syntax for mocking

    //Mock.Arrange(() =>

    //    mockedProvider.CanUpdateEmployeeRate(userId))

    //        .IgnoreInstance()

    //        .Returns(true)

    //        .Occurs(1);

    //using the fluent interface from JustMock.Helpers

    mockedProvider.Arrange(x => x.CanUpdateEmployeeRate(userId))

                  .IgnoreInstance()

                  .Returns(true)

                  .Occurs(1);

    Employee current =

        new Employee(userId, "Jim", "Holmes", "Doghouse", oldRate);

    EmployeeUpdaterLegacyStyle legacyUpdater =

        new EmployeeUpdaterLegacyStyle();

    Employee updated =

        legacyUpdater.UpdateEmployeeRate_ProviderNewedUpHere(current, newRate);

    Assert.AreEqual(updated.HourlyRate, newRate);

    //Standard assert

    Mock.Assert(mockedProvider);

    //Fluent from Helpers

    mockedProvider.Assert();

Here’s a test handling the static call variant above:

[Test]

public void Mocking_provider_called_statically()

    int userId = 1;

    int oldRate = 5;

    int newRate = 10;

    Mock.SetupStatic(typeof(UpdateEmployeeSecurityProvider_Static),

        StaticConstructor.Mocked);

    bool called = false;

    Mock.Arrange(() =>

        UpdateEmployeeSecurityProvider_Static.CanUpdateEmployeeRate(userId))

        .Returns(true)

        .Occurs(1);

    Employee current =

        new Employee(userId, "Jim", "Holmes", "Doghouse", oldRate);

    EmployeeUpdaterLegacyStyle legacyUpdater =

        new EmployeeUpdaterLegacyStyle();

    Employee updated =

        legacyUpdater.UpdateEmployeeRate_ProviderIsStaticHere(current, newRate);

    Assert.AreEqual(updated.HourlyRate, newRate);

    Mock.Assert(() =>

        UpdateEmployeeSecurityProvider_Static.CanUpdateEmployeeRate(userId),

        Occurs.Exactly(1));

I’m sure there are better ways to leverage JustMock, and I’m sure there are cleaner ways to write the test. This exercise was more about me learning what capabilities JustMock has so I could keep it somewhat clear in my own head.

I’d love to hear feedback from folks who’ve worked with both mocking tools! Have you worked with Rhino.Mocks and JustMock? Have any thoughts?

[1]Perhaps Rhino.Mocks now handles these situations. I fully admit I haven’t kept up with it…

CodeMash Financial Planning Sheet

2013-01-14T14:27:00.001-05:00

UPDATED

Interested in more detail on how we plan CodeMash? Want to see all the gory (and scary!) financial figures behind the scenes?

We’ve posted up the exact Excel sheet I use to plan everything financial around CodeMash. If you’re a conference organizer, or considering organizing one, then this sheet may be of great use to you. Yes, it’s tailored to CodeMash, but it’s full of information that may be very useful to you. There are a number of things it helps me with every year:

Meal planning: I’m able to keep track of the various meals, plus I have some notes around what I need to make sure gets covered. (Think proteins for vegetarians, greens at each meal, etc.)
- Fixed versus consumable costs: Planning conferences starts off as “We’ll get awesome sessions and have fantastic people!” and inevitably turns to “Crap. How am I going to organize and pay for all this food?” If you’re organizing a conference you absolutely have to keep a careful eye on your food costs. I break out fixed costs (per head things like meals and party food) versus consumable (variable) costs like soda, coffee, and drinks. Those consumable costs terrify me because the impact can be thousands of dollars.
AV costs: Tracking Kalahari and external vendor AV costs helps me ensure I’ve got the right count of projectors, microphones, and screens ordered to cover every day.
Attendee gifts: I use this as a checklist to help ensure I’ve got shirts/hoodies, USB sticks, etc. ordered up.
Psuedo-TODO List: All the items on the various tabs serve to remind me of some of the various things we have to ensure get covered.

The workbook isn’t perfect. I refactor it every year based on things I’ve learned (usually the hard way). Just this year I finally deleted out columns I’d had hidden which contained revenue and estimated expenses for attendee headcounts of 250, 400, 500, and 700. It’s been a few years since we were that small…

Still, it’s invaluable for me in planning. It may be of interest to you, and it follows CodeMash’s policy of being as transparent as possible around our finances.

Grab it here! I hope you find it useful and informative!

UPDATE: I didn’t make clear that this workbook is not finalized for 2013. Consumables (soda, coffee, cocktails and beer) always are a huge variable, and there are always a bunch of last minute charges/changes. That $31K surplus will be nowhere near that by the time things are said and done.

My Trip to India

2012-12-16T12:22:00.001-05:00

I’ve returned this last Wednesday from 12 days in India and I thought I’d post a few musings on the trip here. (I’ve already posted up some of the work-related things over at my “day job” blog.)

Without a doubt this was one of the most impactful overseas trips I’ve ever had, and I’ve been to a fair number of places outside North America over my life: Okinawa, France, Saudi Arabia, Panama, the Philippines, St. Martin, England, Italy, Hungary, Austria, Poland, Bulgaria, and a likely a few others I’m forgetting.

I did zero preparation for the trip. None. Normally my wife and I are huge planners. We get all kinds of research done for big trips and have lots of things lined up. This time I figured I’d just go over and jump in the deep end and float wherever currents took me. Good choice.

First some of the bad things: India is a country where 300 million people live below poverty, and in India, unlike the US, poverty means poverty. People sleeping on blankets in the dirt with a torn tarp over their head poverty. 20 story five star hotels with tent cities next to them poverty. Hideous water giving you instantaneous dysentery and hepatitis poverty. It’s staggering if you’ve been in similar situations before, mind-numbingly shocking if you haven’t. The piles of trash across the entire country are sad—India is a land of amazing beauty and deep, deep spirituality. It was heartbreaking to see such a wonderful place being used as a dumping ground.

But still…

You get past that, or at least find yourself able to move beyond it, and wow, what India has to show you! Noise, color, smells, sounds, noise, people, food. The list goes on and on and on.

The Food

Food in India is a wonderful experience. India, like Italy, Spain, and Mexico, isn’t one style of cuisine. How could it be? Like those other nations India comes from a very diverse, fragmented history. The food in the south was greatly different from that of the north, and all of it was great. I was lucky to have friends to explain some of the differences to me. As you’ve likely noticed if you’ve followed this blog for long, I’m a pretty serious foodie, so when I run across stuff like this in the street I’m in love!

Or when my colleague takes me to a spot like this for dinner on the trip back from Agra:

This was a dinner from a central region of India which wasn’t typical of Bangalore, but my pal DJ thought I’d enjoy it. How right he was!

Here’s a milk drink we had at a diner-like place in Delhi.

My stomach was in fairly good shape the entire trip, but then I eat a lot of spicy food already so my stomach was somewhat preconditioned. It was pretty amusing how so many of my colleagues and fellow diners there would continually ask “Is this too spicy for you? Are you OK with this?” I suppose I’m a bit atypical in this area, though… (That said, the street food at the cart above did do me in a bit. I knew better, but didn’t care.)

The People

The people are perhaps the happiest, most open, most hospitable I’ve ever come across in any of my travels. Bavarians previously held that spot, but my experiences in India blew even that away. Indians are gracious and full of laughter. I was honored to be invited into two different homes for meals, and they were some of the best times I had. (Being invited into someone’s home for food has a special significance for me, one I’m not able to explain well. Suffice it to say it means a lot.)

Indians are incredibly communal people. You might be taken aback at the depth conversations go between complete strangers. During our 16 hour day trip to Agra, my bachelor colleague got a grilling on his bachelorhood status, advice on why he should marry, and insight on a successful life—all from the middle-aged driver who we’d just met that morning at 6am. Get over the personal boundaries you might have and just enjoy the fact that Indians love people and want to know more about you.

They’re also extraordinarily happy. I had some great meals with various people through the developer and tester communities, and all were filled with laughter and jokes. And good food.

I was also amazed at the forward thinking mindset of the testing professionals I spoke with. India earned a reputation (partially justly) for cheap labor of poor quality. You need to lose that perception of them, right now. The people I spoke with at conferences, user groups, and customer sites were, for the vast majority, serious about taking their work to the next level. They’re looking at the long game, and they’re committed to making serious value-based transformations in how they do their work. I had some amazing conversations there that got me fired up and excited.

the traffic

Traffic in India is non-stop. Traffic is simply an insane, chaotic, wonderful experience if you can let go of any notion of patience. It will take you two hours to travel 60Km in Delhi, and 1.5 hours to go 15Km in Bangalore. Getting frustrated and angry won’t solve a thing, so just sit back and enjoy the tuk tuk (powered three wheeler rickshaw) or cab ride. Don’t freak out that you’ve got a two inch space between your vehicle and the huge dump truck next to you. The driver’s thinking “I had an inch and a half to spare!”

Scooters are family transportation vehicles in India, and you’ll see amazing sights: pipes, furniture, groceries, and more folks than you would think could fit on a two wheeler. Personal favorite: Dad driving, mom on back, puppy in mom’s lap. Personal record: Dad driving, mom near back, four other kids scattered from handlebars to tail end. Yes, six on a two wheeler…

It seems insane to someone from the West, but it’s similar to what I’ve experienced in my previous trips to Panama, the Philippines, etc. You’d think there would be non-stop wrecks, mayhem, and fatalities, but I didn’t see a single accident while I was there. I saw a lot of scratched up, dinged up cars, but not one wreck. Indians understand the implicit system with the traffic, and they, as with so many other things in their nation, just figure out how to have huge numbers of people co-exist in small spaces. With cars. And scooters. And trucks. And pedestrians. And dogs. And tuk tuks.

The horns

Horns require a section of their own. Honking was non-stop in Bangalore, prevalent in my countryside drives, and moderate in Delhi. If you sit back and listen the honking has an entertaining language all of its own.

There’s the single, quick “toot” which is the equivalent of “Coming up on your right/left. Make way, please!” A bit longer “honk” might be used of the person the driver’s passing didn’t move out of the way quickly. Toots escalate through honks up through blaaats to the rarely used Angry Honk where the driver’s really frustrated. My first driver for my day trip to Mysore had a very light, happy honk. I didn’t like the driver for our Agra trip very much. He was Angry Honk right off the bat all day.

The Sights

Where to start? India is full of so many strange, wonderful, overwhelming things. Yes, yes, the Taj Mahal is teh awesum, but you expect that. It’s the Taj Mahal.

What was more impactful to me were some of the more intimate, less visited temples. I was at a temple near Mysore that was built in 849 that really moved me. This palace below with a shrine to a holy man is a short hour from Agra, and I had one of the best times there.

There’s also the crazy IT market in Delhi that is straight out of some cross between a peyote-hammered steampunk and MC Escher.

All of this can just drive you crazy trying to take it all in. I stopped taking pictures very early in my trip, choosing to just absorb a lot and get a few pictures in here and there. I think that was a pretty smart choice.

Wrapping Up

I’m really thankful I got the chance to go to India. I’m already trying to line up a return trip or two. I can’t wait to get back and see new old friends and new old sights.

Handling Rejection (From Conferences)

2012-10-16T09:53:00.001-04:00

Sometimes it seems like a significant part of my day job is having my submissions to various conferences rejected. Off the top of my head, here’s an incomplete list of conferences where I’ve had submissions rejected from over the last year(ish)

TechEd
DevConnections
DevLink
Agile Testing Days
Agile Dev Practices
StarEast/StarWest
Some testing conference in London whose name I’ve forgotten

There are a number of other conferences as well, but frankly I’ve lost track.

Rejection stings, for certain, but I have also come to view these rejections as a pretty good learning opportunity. After I get over the pain of rejection, that is.

First off, I always thank the organizers for considering my submissions. I can’t imagine how many hundreds of submissions conferences like DevConnections or StarEast/West get. Taking a moment to thank the content selection crew is simply good manners. (TechEd is different. It’s a total black box, impersonal process, so I never get any contact with humans.)

Secondly, I sit back and think about what might have been the cause for getting passed over. If possible, I try to get feedback directly from the selection folks; however, that’s not always possible.

I’ve found there are a number of useful aspects to consider:

Content doesn’t fit. Maybe you’ve just missed the mark with your submissions to that conference. Some years ago I tried wedging a testing talk to an open source conference targeted more to business application developers. My abstract simply didn’t make a good case why the talk would fit in their conference. Make sure what you’re submitting will be useful to the conference organizers.
Content lost in the chaff. You need to submit talks that stand out from all the others. “Intro to MVC” is outdated and doesn’t offer up anything unique from the 20 other MVC talks the organizers are looking through. Make a clear case of what value your session brings to the attendees.
Content selection crew was overwhelmed. Poorly organized conferences might have too few folks on staff to get a good review in. If you’re not known to the organizers, then they may have simply lost you in the tidal wave of submissions. Networking matters. Experience matters. (I’m very thankful that the CodeMash content chairs works hard to scale out the selection crew every year to avoid just this problem. They still have huge amounts of work.)
Poorly written abstract. It happens, even to someone who’s polished and submitted hundreds of abstracts over the last ten years. I’d like to think I’ve learned and don’t do this anymore, but it’s possible. I once wrote another blog post with some thoughts about writing a good abstract.
Better submissions from other folks. That happens on occasions, particularly for really large conferences. Look at what did get picked up and compare your submissions to those. Note that you’ll have to do some serious stepping back and viewing things with complete self-honesty and detachment. You can’t let your own pride get in the way with false impressions. Which brings me to…
Ego. Yes, sometimes my own ego gets in the way of submissions. Last year I put in four testing talks to a regional conference. None got accepted. Looking back I think I seriously slacked off when writing the abstracts because I felt I was well-known enough that the talks would get picked up anyway. That one stung but good—however, it was a good lesson learned. Respect yourself enough to put aside your ego and care about what you’re putting in. Remember, it’s not about you.
Drama. Conference organizers are horribly, insanely busy during planning and especially during execution of the conference. The last thing they need is drama or worries about unreliability. If you’re a Drama Queen or King, or if you’re a flake, then you’ve got a deep, deep hole to dig yourself out of. Getting over that can take years because that sort of trust is hard to rebuild. (As a personal note, seven years ago I bailed from a half-day workshop at a conference put on by my pal Chris Woodruff. I bailed the day before the conference. It’s perhaps the worst I’m a Douchebag moment of my adult life, and I’m still beating myself up about it. Thankfully Chris is an awesome guy who was extraordinarily gracious about it, and I think I’ve somewhat atoned for it by now.)

Rejection’s not easy. I’ve gotten three rejection notices in the last two weeks alone. That said, view it as an opportunity to avoid lashing out and instead consider how you can improve for the next conference you target.

I’m already working on a few more submissions now…

Consolidating Drawings from Your iPad’s Paper app

2012-10-09T18:14:00.001-04:00

I really like doodling around on the Paper app from 53. It’s a lot of fun, and lets me come up with some funky, off-the-beaten-path presentations.

The work involved in getting those images out of Paper and into Keynote or PowerPoint is a hassle, though. You have to fire up iExplorer and dive into the Apps folder on your iPad when it’s docked.

You’ll have to figure out which node under that Journals folder is the book you want. There’s some goo in the model.json files that can help you out, or just look at the Date Modified fields.

Drag that folder over to a working directory on your Mac, then you can start to pull the graphics out. They’re beautiful, high-rez PNG files with transparent backgrounds. Unfortunately, they’re all stored as the same filename in GUID-named folders. <sigh/>

The following Ruby script helps you pull the files out and drop them in a common folder, renaming them on the fly. Run it from the folder holding all the subdirectories with the files. You’ll need to edit sourceDir and targetDir as appropriate for your environment.

require "FileUtils"

count = 0

sourceDir = "/Users/jimholmes/Documents/tmp/Pages"

targetDir = "/Users/jimholmes/Documents/tmp/Consolidated/"

dirContents = Dir.entries(sourceDir)

dirContents.each do |folder|

  next if folder == '.'

  next if folder == '..'

  next if folder == '.DS_Store'

  next if not File.directory?(folder)

  source = File.join(sourceDir, folder, "sketchLayer@2x.png")

  target = File.join(targetDir, "Pages-#{count}.png")

  puts "Source: #{source}"

  puts "Target: #{target}"

  FileUtils.cp_r(source, target)

  count += 1

end

Mad props to @rubyist who dug me out of some issues with how I was handling the filenames.

My next step some day would be to figure out how to pull this stuff straight from the iPad’s filesystem; however, I’ll have free time for that in approximately 246 years, I think.

Video on “Solving Common Web UI Automation Problems”

2012-09-17T09:11:00.001-04:00

Looking to learn more about creating flexible locators, ease your frustrations over dynamic content, and see some flexible ways to pull data from tables in a flexible fashion that’s not tied to sort or column order?

Check out the recording of a webinar I hosted last week on these common web automation problems.

This webinar’s very generic in content. As a matter of fact, most of the examples are in C# using WebDriver.

I’ve got a wrapup posting too listing the various demo code I used during the webinar.

I hope you find it useful!

Love Training & Coaching Software and Test Teams? Come Work With Me!

2012-09-12T09:52:00.001-04:00

Telerik’s Test Studio is looking for someone to help us make teams using Test Studio be awesome. We’re extremely passionate about ensuring our customers succeed over the long-term with test automation – and a huge part of that success hinges on getting Test Studio users great training in the early phases of their adoption efforts.

We’re looking for someone to join the Telerik team full time to help deliver training and coaching engagements for our customers. As a trainer you’d be spending 25% – 50% of your time on the road at customer sites. We’ve got upcoming training engagements all across the US and internationally as well. You’d also be delivering training via online webinar sessions, and you’d be helping to evolve our existing training curriculum and materials. You’d also be responsible for helping create other content like blog posts, white papers, and videos.

We’ve got a very different philosophy about training for our customers. Training is NOT a revenue stream for us; it’s explicitly about ensuring our customers’ long-term success with Test Studio. Moreover, our onsite training engagements quickly move from an overview of Test Studio into working with the customer teams to write tests in their own environment. You’ll quickly be flipping the switch from trainer to coach during these sessions!

It’s also not just all about Test Studio, either. It’s about becoming an influencer and thought leader in the automated test space. The folks here in this corner of Telerik are awesome about helping improve testing overall, not just via our product. This means you’ll have the opportunity to stay (or get) familiar with other testing tools. (I still spend plenty of time talking about and presenting on WebDriver/Selenium as well as gabbing other automation tools, too.)

What kind of person are we looking for? You’ll need to be passionate about helping customers succeed long term with automation. You’ll need to have enough self-confidence to say “I don’t know” when confronted with something new to you – and the self-motivation and drive to follow up on the accompanying “but I’ll find out and get back to you!” You’ll need to have exposure to development and testing in real environments – but you don’t need to be an expert in either. (Obviously expertise helps…)

We’re looking for this position to be staffed up in either Austin, Texas, or Hudson, Ohio. Preference would be Austin – the ability to coordinate closely with our Test Studio team down there is a tremendous advantage.

Working for Telerik is amazing. Working with Test Studio at Telerik has evolved into pretty much my dream job. I get to train and coach customers, I get to learn from wicked smart folks in the industry, and I work along with some incredibly awesome product and sales folks.

Interested? Ping me directly at my work email address. I’d love to chat with you.

Are You a Geek In Dayton? Fill an Opening with the Dayton .NET Developers Group!

2012-08-11T10:16:00.001-04:00

The Dayton .NET Developers Group has a need for an enthusiastic person to step up and help lead the group. It’s time for me to step aside after seven or so years with the group, and the current Board of Directors is looking for someone to fill the soon-to-be empty position.

Helping run the DevGroup is a great experience. It’s a couple hours a month of your time, but you’ll get to meet a lot of great folks, and you’ll get some tremendous satisfaction in knowing you’re helping people make tremendously positive impacts in their careers through networking and skills development.

Moreover, as part of the group’s leadership you have an active hand in determining what the group’s direction for topics and special events is. That’s pretty exciting stuff! You’ll also build out your network of contacts to include speakers, sponsors, and companies at the regional and national level.

A few of the things you might be involved with during a regular month:

Helping run the monthly meetings (raffles, meeting logistics, greeting, etc.)
Helping plan out future topics
Helping locate speakers for future meetings
Helping grow the group’s membership through drives, media contacts, etc.

Getting involved with running a user group is an extraordinarily rewarding experience. You’ll make a lot of new friends and contacts throughout the Heartland region (or farther afield!), you’ll get even more exposure to career-broadening ideas, and you’ll get opportunities you never imagined. I’d never have started my speaking “career” were it not for having to jump in to fill a meeting when the planned speaker had to reschedule at the last minute. Somehow the group survived me…

Interested, or know someone who’d be a good fit? Drop me a mail via the contact link at the blog’s upper right corner.

Get involved! It’s totally worth it.

Hear Jim Holmes talk Testing with Jesse Liberty

2012-07-19T12:56:00.001-04:00

Jesse Liberty, one of Telerik’s great evangelists, had me on his Yet Another Podcast show to talk about, wait for it, testing. Jesse and I cover a lot of things about testing at all levels. It’s a wandering conversation that I think you’ll find entertaining and useful, too.

Yet Another Podcast Episode 70: Jim Holmes & Automated Testing.

Honest Dialog with Stakeholders on Distributed Teams’ Constraints

2012-06-11T08:03:00.001-04:00

My talk on “Effective Distributed Teams” always garners polarized feedback. I get kudos from folks who’ve left with a few new ideas, and I always get one or two evals where I’m skewered for talking too much about the human side of teams and the criticality of working hard, VERY HARD, to ensure you get the right folks on your distributed teams. In my talk I spend nearly 30 minutes at the start focusing on hiring your team members, assembling your team, and dealing with the inevitable issue of offshore workers.

(If you’re interested, here’s my deck from StarEast.)

One attendee at my StarEast talk wrote that I was naïve in my assertions about controlling offshore hiring. The same attendee also felt I was unrealistic in my position on working with the stakeholders/business partners to ensure the team has proper time to get tasks done. I’ve been called a lot of things in life, many of which are I earned, but I’m not sure “naïve” and “unrealistic” are labels which stick…

Here’s why I think it’s absolutely critical for you as a team lead/member to have open and honest dialog with your stakeholders/management/whatever: stakeholders/management/whatever need to understand the risks to the success of their project. They’re putting constraints on your team by asking you to work in a distributed fashion. They’re also potentially putting additional constraints on you by expecting to realize cost savings by using offshore help.

If they’re putting those constraints on you, then they must be made aware of the impacts of those actions: reduced velocity, and potentially reduced quality of work due to all the things inherent with offshored work. (Differing skills, communication difficulties, cultural differences, and of course timezone barriers.)

If your team members aren’t working out, regardless of whether they’re direct reports, distributed employees, or offshored subcontractors, then you need to be empowered to change the makeup of that team. That should include having input on changing a business partner relationship and terminating contracts with poorly performing individuals or companies. Be clear: this is NOT an easy effort, nor is it generally a quick one. I’ve never said otherwise. To the contrary, I emphasize it may take months to terminate a relationship with a poorly performing team member or subcontractor.

The same concept applies to unreasonable expectations around timelines. There’s a huge amount of writing on this topic by folks who are much smarter and wiser than I. The bottom line is you can’t simply accept the situation; you’ve got to ensure some rational discussion happens around scope, effort, and dates.

You need to make sure you’re clear on what YOU need to flex on too: just because you’re the QA lead doesn’t mean you get to hold up the project until you feel comfortable about quality. Your job is not to be the final voice on shipping or not. Your job is to ensure the stakeholders have a clear picture of the overall quality and risk at any given time. Stakeholders make shipping decisions, not you!

Again, I’ll repeat myself: these conversations are rarely easy, but you need to have them, and you need to have them earlier in the project versus later. (Actually, these conversations need to happen constantly through a project’s lifecycle.)

Your project’s stakeholders, believe it or not, really are looking for a project to succeed and help out their bottom line. If something is jeopardizing the project’s success, then they’re going to want to know about the issue. [1] The conversation may not be an easy one to have, and it will likely take several attempts to get clear, but you need to step up to the plate and get the issues out in the open. (You may also find you’re tilting at windmills with management who’s unwilling to support you. In that case, remember you can change where you work or you can change where you work.

I’ve had these conversations a number of times in different organizations and roles. I’ve been part of efforts getting rid of poorly performing workers and subcontractors, and I’ve been part of efforts getting poorly performing workers/subcontractors up to being productive members of our teams. It’s hard work, but it’s worth the effort.

It’s not naïve or unrealistic to think you can change these bad situations. It’s defeatist to think you can’t. Step up to the plate, do the work. You’ll be happy you did.

Really.

[1]Yes, yes, there are rare environments where an organization’s politics are so horrific that management actually looks to sabotage teams and projects. All I can say is that if you’re working in such an environment you’re there by choice. Flee that environment immediately or accept you’re making a choice to stay where you’re at. Leave right now or deal with it.

Three Day “Lunch & Learn” on Web UI Automation

2012-06-06T08:00:00.001-04:00

Are you looking to start working with web UI automated testing? Are you or your team currently working with functional testing for your web apps, but struggling with writing maintainable, valuable tests? Join me June 26th, 27th, and 28th for a three day series “Getting Started with Web Automated Testing” and I’ll lay out a few pointers to help you through the rough spots.

I’ve written a bit about the series on my blog at Telerik, but the gist of the series is this: three days of webinars, each lasting 30 – 45 minutes. Through the series you’ll learn about fundamentals for web testing:

What a web page’s lifecycle looks like, and how automation frameworks monitor that
The Document Object Model (DOM)
Understanding how a test framework interacts with the page
What element locators are, and how it’s critical to carefully choose them
Dealing with dynamic content like AJAX
Good test case design
Good test suite organization
Backing APIs

I’ll be using Telerik’s Test Studio to work through most of the examples, but the fundamental points apply regardless of whether you’re using WebDriver, Watir, or another commercial tool.

You can register here for the series!

It’s Not About You

2012-04-24T09:20:00.001-04:00

Too often we get hung up on ourselves. PMs, developers, testers, whomever. We forget, or chose to ignore, the critical fact that we (almost always) do our work for someone else: a stakeholder, a customer, a user.

In a significant number of cases, the person we’re doing our work for may have put their entire career or business on the line. We need to get over ourselves and remember that we’re working to solve a problem, sometimes a crucial one, for those folks.

Instead of focusing on that, too often we get hung up on the stakeholder’s inability to understand technology (developers), think that we’re the final line of death who decides what ships (QA/testers), or get frustrated when the schedule/scope needs to change (PMs).

This deck is something that’s been noodling around in my head for a long time. I finally got my thoughts to Paper on a new iPad I got for work. It’s not perfect, but it sort of lays out my thoughts on this a bit more clearly. I hope you enjoy it!

My Slides from #StarEast Distributed Test Team Talk

2012-04-19T21:32:00.001-04:00

I’ve uploaded my slides from my StarEast talk “Making Distributed Test Teams Work” to Speakerdeck. This deck is slightly different from the earlier version of the talk. I’ve iterated a few things in the talk and deck based on feedback from the last couple presentations.

If you were at StarEast and attended the talk, then thanks very much! I think this was the best session I’ve had for this talk, and I really appreciated all the audience interaction – that always makes talks so much better!

31 Days of Testing—Hiatus This Week

2012-01-09T10:46:00.001-05:00

No #31DaysOfTesting posts this week. I’m a mite busy with a few other things.

More testing stuff starting next Monday!

31 Days of Testing—Day 25: Performance Testing, Part 2

2012-01-06T16:25:00.001-05:00

Index to all posts in this series is here!

My past post laid out some overview and planning issues around performance testing. This post points out what you might be interested in, and lays out some resources I’ve found very useful.

What do I Monitor?

Figuring out which metrics, measurements, and counters to monitor can be extremely daunting—there are hundreds of individual counters in Performance Monitor alone! In most cases you don’t need anywhere near the entire set of metrics. A few counters will give us all the information you generally need for starting your performance testing work.

Most performance testing gurus will tell you just a few items will get you started in good shape:

Processor utilization percentage
ASP.NET requests per second
SQL Server batch requests per second
Memory usage (total usage on the server, caching usage)
Disk IO usage
Network card IO

If you’re doing load testing you’ll likely be interested in errors per second and queued requests. Often times soak or endurance testing will look to counters associated with memory leaks and garbage collection too—these help you understand how your application holds up over a long period of stress. However, those are different scenarios. The few counters mentioned above will get you started in good shape.

Where Can I Learn More?

Microsoft’s “Performance Testing Guide for Web Applications” is somewhat older, but remains a tremendous resource for learning about performance testing. It’s an extensive, exhaustive discussion of everything around planning, setting up for, executing, and analyzing results from your performance testing. The guide is freely available on Codeplex.

Steve Smith of NimblePros in Kent, Ohio, has been extremely influential in my learning about performance testing. Steve’s been appointed by Microsoft as a Regional Director because of his technical expertise in many areas. He blogs extensively on many software topics and has great practical examples for performance testing. He also has an online commercial course offered through Pluralsight that’s well worth checking in to.

The website Performance Testing has a great number of references to performance testing information across the Web. The site lists blogs, articles, training material, and other highly helpful information.

I’ve recently come across two folks on Twitter who I’ve found a wealth of information from:

Ben Simo, aka Quality Frog, writes and Tweets extensively about testing, but also talks specifically about performance issues regularly.
Scott Barber has an amazing blog with scads of information on it, plus he Tweets amazingly good reads on a regular basis.

One of the things Scott Tweeted recently was this nice series on web performance optimization. There’s some tremendously valuable information in its articles.

Go! Get Started!

Spend some time planning out your performance testing effort. Make sure you work HARD to only change one variable at a time. Don’t get flooded with information; more often less information can be more helpful at the start.

Performance testing is a tremendous asset to your projects, and it can also be an extremely fun, interesting, and rewarding domain to work in.

Go! Get started!

31 Days of Testing—Day 24: Getting Serious About Performance

2012-01-05T22:48:00.001-05:00

Updated: Fixed wrong day # in title. Duh.

Index to all posts in this series is here!

In this post I’d like to cover something that too often gets ignored: performance testing. I thought I’d take some time to lay down some of my opinions and experiences around performance testing in general.

The phrase “performance testing” can mean a great many things to different people in different scenarios, so covering a few of the different types of tests may be helpful.

Performance Testing is generally an umbrella term covering a number of different, more complex test environments. I’ve also used the term to describe a very simple set of scenarios meant to provide a baseline for performance regressions.

Load Testing generally uses a number of concurrent users to see how the system performs and find bottlenecks

Stress Testing throws a huge number of concurrent users against your system in order to find “tipping points” – the point where your system rolls over and crashes due to a huge amount of traffic

Endurance/Soak Testing checks your system’s behavior over long periods to look for things like degradation, memory leaks, etc.

Wikipedia’s Software Performance Testing page has some very readable information on the categories.

You can also look at performance testing as a slice of your system’s performance. You can use a specific scenario to dive down in to specific areas of your system, environment, or hardware.

Load, stress, and endurance testing are all that, but turned up to 11. (A reference to Spinal Tap for those who’ve not seen the movie.)

With that in mind, I generally think of performance testing in two categories: testing to ensure the system meets specified performance requirements, and testing to ensure performance regressions haven’t crept into your system. Those two may sound the same, but they’re not.

Performance testing to meet requirements means you’ll need lots of detail around expected hardware configurations, baseline datasets, network configurations, and user load. You’ll also need to ensure you’re getting the hardware and environment to support those requirements. There’s absolutely no getting around the need for infrastructure if your customers/stakeholders are serious about specific performance metrics!

Performance testing to guard against regressions can be a bit more relaxed. I’ve had great successes running a set of baseline tests in a rather skimpy environment, then simply re-running those tests on a regular basis in the exact same environment. You’re not concerned with specific metric datapoints in this situation – you’re concerned about trends. If your test suite shows a sudden degradation in memory usage or IO contention then you know something’s changed in your codebase. This works fine as long as you keep the environment exactly the same from run to run—which is a perfect segue into my next point.

Regardless of whether you’re validating performance requirements, guarding against regressions, or flooding your system in a load test designed to make your database server weep, you absolutely must approach your testing with a logical, empirical mindset. You’ll need to spend some time considering your environment, hardware, baseline datasets, and how to configure your system itself.

Performance testing isn’t something you can slap together and figure out as you go. While you certainly can (and likely will!) adjust your approach as you move through your project, you do indeed need to sit down and get some specifics laid out around your testing effort before you begin working.

First and foremost: set expectations and goals.

Ensure everyone’s clear on why you’re undertaking the performance testing project. If you are looking to meet specific metrics for delivering your system then you’ll need to be extremely detailed and methodical in your initial coordination. Does your system have specific metrics you’re looking to meet? If so, are those metrics clearly understood – and more importantly reasonable?

Keep in mind that your customer/stakeholder may be giving you metrics you think are unreasonable, but it may fit business needs of their which you’re unaware of. You have to put in the extra effort to ensure you understand those higher-level needs.

Your customer may also be giving you vague requirements simply due to their lack of experience or understanding. “We want the page to load fast!” is an oft-heard phrase from stakeholders, but what do they really mean?

Define your environment

If those same metrics are critical to your delivery, then they will also need to be defined based on a number of specific environment criteria such as exact hardware setups, network topologies, etc. These environments should be the same exact environment you recommend to your customers. If you’re telling your system’s users they need a database server with four eight-core CPUs, 32 GB of RAM, and a specific RAID configuration for the storage, then you should look to get that same hardware in place for your testing.

(A tangential topic: it’s happened more than once that a server and environment acquired for performance testing somehow gets borrowed or time-shared out to other uses. Timesharing your performance environment can be a highly effective use of expensive resources, but you’ll need to ensure nothing, absolutely nothing, is being utilized on that server once your performance runs start – you have to have dedicated access to the server to ensure your metrics aren’t being skewed by other processes.)

Agree on baseline data

Something that’s commonly overlooked is the impact of your system’s baseline dataset on your performance tests. You likely won’t get anything near an accurate assessment of a reporting or data analysis system if you’ve only got ten or thirty rows of data in your database.

Creating baseline data can be an extremely complex task if your system is sensitive to the “shape” of the data. For example, a reporting system will need its baseline data laid out across different users, different content types, different date patterns.

Often the easiest route to handle this is to find a live dataset somewhere and use that. I’ve had great success coordinating with users of systems to get their datasets for our testing. You may need to scrub the dataset to clear out any potential sensitive information such as e-mail addresses, usernames, passwords, etc.

If using a live dataset isn’t an option, you’ll need to figure out tooling to generate that dataset for you.

Determine your usage scenarios

Talk through the scenarios you want to measure. Make sure you’re looking to measure the most critical scenarios. Your scenarios might be UI driven, or they could be API driven. Steve Smith has a terrific walkthrough of a real world scenario that gives a great example of this.

Set up your tooling

Once you’ve got a handle on the things I’ve discussed above, look to get your tooling in place. Performance testing utterly relies on an exact, repeatable process. You’ll need to do a large amount of work getting everything set up and configured each time you do a perf run. Avoid doing this work manually; instead, look to tooling to do this for you. You shouldn’t rely on doing the setup manually for two reasons. One: automating setup ensures you’ll cut out any chance of human error. Two: it’s really boring.

Build servers like Hudson, Team City, or TFS can interface with your source control and get your environment properly configured each time you need to run a perf pass. Scripting tools like PowerShell, Ruby, or even good old command files can handle tasks like setting up databases and websites for you.

You’ll also need to ensure you’re setting up your tooling to handle reporting of your perf test runs. Make sure you’re keeping all the output data from your runs stored so you can keep track of your trends and history.

Change only one variable at a time. Compare apples to apples!

It’s critical you take extraordinary care with the execution of your performance testing scenarios! You need to ensure you’re only changing one variable at a time during your test passes, or you won’t understand the impact of your changes.

For example, don’t change your database server’s disk configuration at the same time you push a new build to your test environment. You won’t know if performance changes were due to the disk change or code changes in the build itself.

In a similar vein, ensure no other folks are interacting with the server during your performance run. I alluded to shared servers earlier; it’s great to share expensive servers for multiple uses, but you can’t afford for someone to be running processes of any shape or form while you’re doing your performance passes.

Profiling: Taking the simple route for great information

All the work above can seem extraordinarily intimidating. There’s a lot to consider and take in to account when moving through some of the more heavyweight scenarios I laid out in my introductory post.

That said, you can look to simpler performance profiling as a means to get great insight in to how your application is behaving. Profiling enables you to use one scenario, or a very small set, and see in a slice how your application’s behaving. Depending on the tooling you can see results of performance back to the browser, dive in to performance metrics on the server (think CPU or disk usage, for example). You may even be able to dig down in to the application’s codebase to see detailed metrics around specific components of the system.

Profiling is a great way to start building a history of your application’s performance. You can run regular profiling tests and compare the historical performance to ensure you’re not ending up with performance regressions.

Start small, start smart

As you’ve seen in this post, performance testing can be particularly complex when you’re looking to ensure high performance, reliability, and scalability. You need to approach the effort with good planning, and you need to ensure you’re not changing variables as you move through the testing.

Make sure your performance efforts get you the information you need. Start with small environments and scenarios, ensure you’ve clearly laid out your goals and expectations, and keep a careful eye out as you’re running your tests.

31 Days of Testing—Day 23: Acceptance Tests & Criteria in the Real World

2012-01-04T13:05:00.001-05:00

UPDATED: I goofed and Andy caught it, thankfully. They’re using Watir, not WatiN in their work. I knew that and still fat-fingered the post. Fixed!

Index to all posts in this series is here!

Today’s post is by Andrew Vida, another smart pal in the Heartland region. I’ve chatted with Andy a number of times at various conferences, and I’ve enjoyed hearing about the work he and Bramha Ghosh do at Grange Insurance in Columbus, OH.

We three have spent a pretty good amount of time moaning about our shared pain in getting great, reliable, valuable functional test suites in place. Andy and Bramha are working in Ruby and Watir, but their issues are my issues are the same issues seen in any technology: dealing with data, environments, timing, and of course the inevitable hardest part: “soft” problems in ensuring clarity of communication between folks on the project team.

Andy offered up the following article for my series based on the work they’ve done trying to get a smooth flow around well-defined acceptance criteria. This is a perfect follow on to yesterday’s post by Jon Kruger!

Using Acceptance Tests to Define Done

Have you ever been on a team and was asked "What is the definition of done?" You respond by saying, "When all of your automated tests pass, and there are no bugs, then you have satisfied the acceptance criteria. Done!" Which then is responded to by "Well, how do I define the acceptance criteria?" Good question!

Understanding the Feature

First things first - you have to understand what feature you'll be building. Building the right product and building the product right takes communication and collaboration between your product owner and your team.

The reason for all of the collaboration is that we're trying to build a shared understanding of what needs to be done and also produce examples that are easy to maintain. There are many ways to work collaboratively and ultimately, you have to decide what works best for your team.

The team I'm currently on has found that smaller workshops work best for us. Those workshops, otherwise known as "Three Amigos", include a business analyst, a developer and a tester who share a similar understanding of the domain.

Lets hypothetically say you're discussing a shopping cart feature for your site. Start by defining the goals of this feature. By starting with the goal, you'll let everyone know why they're spending their time on implementing the feature. If you can't come up with a good reason why, then maybe the product owner is wasting everyone's time.

We've used the Feature Injection template from Chris Matts and Liz Keogh to help us successfully describe why:

As a <type of stakeholder>

I want <a feature>

So that <I can meet some goal>

Here's our feature description:

As an online shopper

I want to add items to my shopping cart

So that I can purchase them

Determining Acceptance Criteria

Next, your team needs to determine what the system needs to do to meet those goals-the Acceptance Criteria.

In your Three Amigos meeting, be sure to ask questions to clear up assumptions, such as "Are there any products that cannot be purchased online?" or "Does the shopper need to be authenticated to purchase?”

Remember, the scope of feature should be high level as we only want to identify what the application needs to do and not how it's implemented. Leave that part to the people that know how to design software. It was determined by the team that the following are in scope:

Only authenticated shoppers can add items to the shopping cart.
Cannot add refrigerators to shopping cart.
Only 25 items can be added.
Shopper can remove items from shopping cart.
Shopper can change quantity of items after adding it to the cart.

Hey, now we have some acceptance criteria!

Acceptance Criteria lead to Acceptance Tests

We've used communication and collaboration to determine why a feature is necessary and what the system needs to do to at a high level, so now we can come up with some examples to test our acceptance criteria.

To do this, we'll write some Cucumber scenarios. We've chosen Cucumber for all of the reasons mentioned in Tim Wingfield's post on Day 15. If you haven't read it, go back and check it out. It's an excellent post on the benefits of employing Cucumber.

Here are a few scenarios that were created:

Given the shopper is a guest

When they try to add an item to their shopping cart

Then they will receive the error "Only authenticated shoppers can add items to their shopping cart."

Given an authenticated shopper

When they click the "Add Item to Cart" button

Then they will have an item in their shopping cart

Given an authenticated shopper with an item in their shopping cart

When they click the "Remove Item" button

Then that item is no longer in their shopping cart

These are only a few of the examples that were developed as part of the Three Amigos meeting. On our team, the output of the Three Amigos is a Cucumber feature file. We now have a shared understanding and a definition of done! We can pass on our failing acceptance tests to the Dev team to begin their work. They will begin by creating failing unit tests and writing enough code to make them pass. Once they are passing, they can then run the acceptance tests. Once those are passing then the feature is complete. We're done! Those acceptance tests will be added to the regression suite to be ran anytime to ensure that the feature remains done. Now the feature can be demonstrated to the product owner at the next review.

What we've just done is taken a trip around the Acceptance Test Driven Development cycle. Just remember, it's not about the tools or the technology, but rather the communication and collaboration. Our ultimate goal is to deliver high quality software that functions as the product owner intended. By including QA in the entire process, we can eliminate many of the problems that plague us earlier so that they don't make it to production. Quality is not just a QA function, it's a team function.

31 Days of Testing—Day 22: Why Collaboration Matters (A Real World Example)

2012-01-03T12:19:00.001-05:00

Updated: Index to all posts in this series is here!

Today’s post is reposted from Jon Kruger’s blog. Jon is a tremendously smart, passionate indie working out of Columbus, Ohio. I was lucky enough to work with Jon some years back, and I’ve always had great regard for his views and thoughts.

Jon’s post today really hit home for me because it’s all about communication and collaboration early in the cycle. I can’t jump up and down enough about how critical this is--and Jon’s post is a real-world example of why it’s so important.

I read Jon’s blog this morning and immediately pinged him on IM to see if he’d let me drop his article in to my #31DaysOfTesting series. Thankfully he agreed!

Follow Jon on Twitter, and definitely bookmark or subscribe to his blog. Lots of great stuff in both spots!

Just Another Run of the Mill Wednesday

On my current project, we release every 2 weeks. We do the push to production on Saturday, so we set a deadline of Wednesday night for everything to be developed and tested so that we can have two days for demos and UAT.

I remember a certain Wednesday a couple of months ago where things were chaotic to say the least. We looked at the board on Wednesday in the early afternoon and there were 20 items where testing was not complete. We were running around trying to make sure that everything got tested. The entire development team was helping out with testing. Many people stayed past dinnertime to get everything done.

This past Wednesday was much different. Everyone was very relaxed. There was only one item on the board that was still being tested. We were all working on getting stuff ready for the next iteration. And oh by the way, one of the QA testers was out on vacation and another one had been moved to another project.

I immediately thought back to that chaotic Wednesday a few months ago and thought about everything that has happened since then. We certainly had come a long way to get to the point where things were much more relaxed. So what happened?

The Three Amigos

Before development can start on a feature, we have a “three amigos” meeting where developers, business analysts, and QA people get together and decide on the acceptance criteria for the feature. This helps us all get on the same page and make sure that we know what we’re building. It also gets the QA team involved very early in the process, so when it comes time for them to manually test the feature, they already know it inside out.

Automating acceptance tests

The outcome of the three amigos meeting is acceptance criteria. Developers take these and automate them whenever possible (we use a combination of unit tests and acceptance tests using SpecFlow). The development workflow now looks something like this:

Work with the QA team to write out the acceptance tests in SpecFlow
Develop all of the components needed to make the feature work, writing unit tests along the way
Try and get the acceptance tests to pass, fixing any problems we find along the way

When I’m working on the “try and get the acceptance tests to pass” phase, I’m going to find pretty much all of the coding errors that we made during development. The development ticket is still marked as “In Development” at this point, which is very important. We all take quality seriously, both QA testers and developers. I’m not going to hand it over to be tested by the QA team until I can get all of the acceptance tests to pass.

Almost no bugs

Because we knew what we were building up front and because we automated pretty much all of the testing, the QA team is finding very few bugs in the new features that we’re developing. One of our testers brought this up in our retrospective this past week and mentioned how they got everything tested so much faster because they weren’t finding bugs, writing up bugs, waiting for bugs to be fixed, and retesting bug fixes.

We had looked at the schedule earlier in the week and we had thought that developers might have to help out with testing because one of the testers was on vacation and they had some items to test that we thought would take a long time. In the end, no developers had to help with testing and testing got done ahead of schedule!

Everyone talks about how bugs are a waste of time, how they slow you down, etc., but it was really cool to see it play out. Yeah, getting those acceptance tests to pass takes a little extra time, but now I can hand a completed feature over to QA and have a good chance of not having any bugs. We had two developers working for a week on the feature that we completed, and we did it with no bugs. Not only that, we have automated acceptance tests that will do our regression testing for us.

Recap

A lot of the changes that we’ve made seem to be relatively minor, but they’ve produced huge dividends. Much of it comes down to discipline, not cutting corners, communicating effectively, and taking pride in your work. I’m really excited about what we’re going to be able to do from here and I expect to have even more stories to tell in the near future.

31 Days of Testing—Day 21: Data Driving Your Functional Tests

2011-12-28T01:20:00.001-05:00

Index to all posts in this series is here!

This post shouldn’t be confused with Seth’s awesome post he lent me on Rules for Effective Data-Driven Tests. He was talking about testing interactions with the database. This post will show you how to push sets of data through a functional test.

This post’s examples are in C# with Selenium. If you’d like to see how the same test rolls in Test Studio, please go check out the short video I recorded on Data Driving Dynamically Loaded Elements for Telerik TV.

I’m going to refer back to the post I wrote on day 13: Functional Test 201 (Common Problems), specifically, case 3 where the elements you are working with are loaded up in the DOM, but have no content. That example used the ASP.NET AJAX Cascading Drop Down site. Feel free to go explore the original post and AJAX site if you need to refresh yourselves. I’ll wait.

The example site I used offers up three option lists (make, model, color) and gives you a matrix of combinations to deal with. Writing up separate test scripts for each combination would be insanity, so let’s not. Instead, we’ll create a list of items to pass through one test, and iterate that test repeatedly through the list.

First off, I’ve refactored the original example from day 13 to make it more modular and readable. Day 13 worked for an elementary example, but let’s move on to something more real-world-ish. Here’s the crux of the new test:

 Test: Working_with_no_content_data_driven

   1: [Test]

   2: public void Working_with_no_content_data_driven()

   3: {

   4:     IList<Car> cars = CarFactory.Return_three_valid_cars();

5:

   6:     browser.Navigate().GoToUrl(

   7:         "http://www.asp.net/ajaxLibrary/AjaxControlToolkitSampleSite/CascadingDropDown/CascadingDropDown.aspx");

   8:     WebDriverWait wait = new WebDriverWait(browser, TimeSpan.FromSeconds(10));

9:

  10:     foreach (Car car in cars)

  11:     {

  12:         browser.Navigate().Refresh();

13:

  14:         Select_make(car, wait);

  15:         Select_model(car, wait);

  16:         Select_color(car, wait);

  17:         Validate_message(car, wait);

  18:     }

  19: }

First I’m using a factory to build a list of three cars. Here’s what that looks like:

 Class: CarFactory

   1: public static class CarFactory

   2: {

   3:     public static IList<Car> Return_three_valid_cars()

   4:     {

   5:         return new List<Car>

   6:         {

   7:             new Car { Make = "Acura", Model = "Integra", Color = "Sea Green",

   8:                 Message = "Sea Green Acura Integra" },

   9:             new Car { Make = "Audi", Model = "S4", Color = "Metallic",

  10:                 Message = "Metallic Audi S4" },

  11:             new Car { Make = "BMW", Model = "7 series", Color = "Brown",

  12:                 Message = "Brown BMW 7 series" }

  13:         };

  14:     }

  15: }

Here I’m simply creating a list of Cars right in the method. This factory method could just as easily reach out to a database, read from an Excel file, etc., etc. The point being, the test itself has no idea what the data source is—and that’s exactly how it should be! Hiding the data source in the Factory lets me change the source as needed without impacting any of the tests which rely on that Factory.

Here’s the inspiring Car class:

 Class: Car

   1: public class Car

   2: {

   3:     public string Make { get; set; }

   4:     public string Model { get; set; }

   5:     public string Color { get; set; }

   6:     public string Message { get; set; }

   7: }

Back to the actual test now!

Lines 6-8 navigate to the site and set up our wait, quite similarly to day 13’s example.

Lines 10-18 let us iterate through our list of Cars. If you’re working in Python, Java, Ruby, or some other platform then obviously things will look different. The idea is we loop through our cars and run the same test each time.

Note that line 12 explicitly refreshes the browser each time through. Because we’re pulling data back from service calls, I’ve found the page DOM can hold old contents around. Refreshing each iteration ensures I have exactly the DOM I expect to work with.

Line 14, Select_make() calls a newly extracted method to interact with the page’s drop down for Make.

   1: private void Select_make(Car car, WebDriverWait wait)

   2: {

   3:     var listOfMakes = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList1"));

   4:     wait.Until<IWebElement>((d) =>

   5:     {

   6:         return d.FindElement(By.XPath(

   7:                                       "id('ctl00_SampleContent_DropDownList1')/option[text()='" +

   8:                                       car.Make + "']"));

   9:     });

  10:     var makeOptions = new SelectElement(listOfMakes);

  11:     makeOptions.SelectByText(car.Make);

  12: }

This works exactly the same as described in the previous post: get the drop down list, wait until its contents populate with the desired make for this iteration. I’m passing in the Car class here, which as I’m writing this makes me realize I should instead be passing in only the Make property from the Car, not the entire Car. Select_make shouldn’t have to know how to deal with a Car, only with what it expects to.

Enough ponderings on software design for now. I’ll refactor later.

The calls to Select_model() and Select_color() work in exactly the same fashion:

   1: private void Select_model(Car car, WebDriverWait wait)

   2: {

   3:     var listOfModels = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList2"));

   4:     wait.Until<IWebElement>((d) =>

   5:     {

   6:         return d.FindElement(By.XPath(

   7:                                       "id('ctl00_SampleContent_DropDownList2')/option[text()='" +

   8:                                       car.Model + "']"));

   9:     });

  10:     var modelOptions = new SelectElement(listOfModels);

  11:     modelOptions.SelectByText(car.Model);

  12: }

13:

  14: private void Select_color(Car car, WebDriverWait wait)

  15: {

  16:     var listOfColors = browser.FindElement(By.Id("ctl00_SampleContent_DropDownList3"));

  17:     wait.Until<IWebElement>((d) =>

  18:     {

  19:         return d.FindElement(By.XPath(

  20:                                       "id('ctl00_SampleContent_DropDownList3')/option[text()='" +

  21:                                       car.Color + "']"));

  22:     });

  23:     var colorOptions = new SelectElement(listOfColors);

  24:     colorOptions.SelectByText(car.Color);

  25: }

Now for the validation which checks that the expected message is correctly displayed:

   1: private void Validate_message(Car car, WebDriverWait wait)

   2: {

   3:     var messageActual = wait.Until<IWebElement>((d) =>

   4:     {

   5:         return d.FindElement(By.XPath(

   6:                                       "id('ctl00_SampleContent_Label1')[contains(.,'" +

   7:                                       car.Message + "')]"));

   8:     });

   9:     Assert.IsTrue(messageActual.Text.Contains(car.Message), "Message: " +

  10:                                                             messageActual.Text);

  11: }

The XPath in this method uses the “contains” function to check contents under the element pointed to by id ct100_SampleContent_Label1. I don’t check for an exact match—I only want to check that the message property’s contents for the current car are somewhere in that element.

There you have it: a simple data driven example for your functional test. Key takeaway: Construct your actual data list behind some sort of façade, be it a Factory or some other equivalent. Never let your tests themselves be responsible for constructing the data list. This ensures you’ll always have the easy flexibility to change how the data is built, where it’s built from, what it looks like, etc.

I should also point out that, as with all my examples, I’m not making use of the Page Object pattern. My examples here are all very linear with locators defined right in the tests. Why am I not using Page Objects for you to read? Two reasons. First, the examples are long enough and I’m trying to keep things fairly simple. Secondly, quite frankly I’ve not worked with it enough to be confident in showing you proper examples. Go do your own research on it, get to know it, and decide if it’s a sensible path for you to follow.

If you’re interested, you can find the complete source for this example (and the ones from Day 13) in my GitHub repository.