Coding QA

Episode 67 - Learning From Experience

Mark Berryman and Jim Wang — Thu, 20 Oct 2011 17:23:10 Z

In this episode of Coding QA, Mark and Jim discuss applying the lessons learned from past experiences to produce a better product. First, they discuss improving the quality of the Coding QA podcast itself by leveraging their experiences with professionally produced podcasts. Then, they discuss Microsoft examples of learning from historical failures, using post mortems to prevent future failures and the value of learning from others.

Links from the show:

This Developer's Life

Episode 66 - Testing Strategies for a Toaster Pt.2

Mark Berryman and Jim Wang — Wed, 05 Oct 2011 09:02:29 Z

In this episode of Coding QA, Mark and Jim continue applying the "Test Techniques" defined in James Bach's Heuristic Test Strategy Model. They walk through the categories of Claims, User, Risk and Automatic testing and apply them to a typical household toaster. At the same time, they relate their thought processes and test scenarios back to real world software testing.

Links from the show:

Toaster Instructions used during Claims testing evaluation

James Bach's Heuristic Test Strategy Model

Episode 65 - Testing Strategies for a Toaster

Mark Berryman and Jim Wang — Sun, 25 Sep 2011 20:33:56 Z

In this episode of Coding QA, Mark and Jim embrace your poll feedback (focus on testing tools and techniques) and take a look at the "Test Techniques" defined in James Bach's Heuristic Test Strategy Model. They walk through the categories of Function, Domain, Stress, Flow and Scenario testing and apply them to a typical household toaster. At the same time, they relate their thought processes and test scenarios back to real world software testing.

To help you follow along, here's the toaster they're testing:

Links from the show:

James Bach's Heuristic Test Strategy Model

James Bach's blog

Episode 64 - What Do You Want To Hear?

Mark Berryman and Jim Wang — Sun, 11 Sep 2011 20:08:55 Z

It's been over half a year since Jim and Mark took over the Coding QA podcast and while they're having a lot of fun putting the shows together, they would like to make sure that you, dear listener, are getting to hear what you want to hear.

To help guide their future show topic selection, they've put together this tiny (one question) poll. Please take 15 seconds out of your day to provide them with this value feedback.

The poll is now complete. Thanks for all of your feedback.

Also, if you have any general feedback about the show (format, recording quality, etc.) please don't hesitate to send mail to Mark or Jim.

Episode 63 - Setup Testing

Mark Berryman and Jim Wang — Mon, 29 Aug 2011 21:02:58 Z

In this week's episode of Coding QA, it's just Mark (Jim's on vacation) discussing the topic of Setup Testing. Recently asked to review a setup test plan, Mark takes a step back to draft a mini-framework for defining a comprehensive setup test plan as well as providing guidance for the execution of the test plan and verification of the results.

Listen in as he discusses:

Different categories of variables to help define individual setup test scenarios

Streamlining and prioritizing a matrix of setup test scenarios

The use of external resources to assist with running setup test scenarios

Verification tips to ensure setup has succeeded

Links from the show:

Mark's blog posts on "Setup Testing - Planning" and "Setup Testing - Execution and Verification"

Pairwise testing tools including PICT from Microsoft.

Episode 62 - Team Structure

Mark Berryman and Jim Wang — Sun, 21 Aug 2011 19:59:24 Z

In this week's episode of Coding QA, Mark and Jim discuss the team's recent re-org from an organizational model to a functional model, the changes they think this will bring, and the pros and cons of the switch.

Listen in as they discuss:

What's the difference between an organizational model and a functional model?
Will it be easier to shift test resources where they're needed, and will this cause a loss of test experts for certain feature areas?
Will we lose the feeling of belonging to one team?
Will having multiple test teams report up the same test management chain increase test productivity?
Will work which blurs the discipline lines become harder to justify?

Does the long term career path for a tester become clearer?

What do Mark and Jim actually think about the change?

Episode 61 - Accessibility Testing

Mark Berryman and Jim Wang — Sat, 13 Aug 2011 21:31:09 Z

In this week's episode of Coding QA, Mark and Jim sit down once again with fellow team member Clay Compton to discuss Accessibility testing. Clay's been in charge of this area for years on the ASP.NET team. Listen in as he shares his experience and discusses:

What does "accessible" mean for software?
How did Clay become an expert in accessibility testing?
Why should you care about software accessibility?
Section 508 compliance and the US government
Accessibility testing tools
Interesting accessibility bugs

Links from the show:

Resources

Microsoft Accessibility Portal

MSDN Accessibility Overview

W3C Web Accessibility Initiative Portal

WebAIM

AbleGamers

Tools

Windows 7 Software Development Toolkit

Paciello Group Colour Contrast Analyser

PEAT (Photosensitive Epilepsy Analysis Tool)

AccChecker

Episode 60 - Fede and Atlassian (Part 2)

Mark Berryman and Jim Wang — Wed, 27 Jul 2011 04:54:02 Z

In this week's episode of Coding QA, Mark and Jim continue last week's conversation with Fede and discuss his experiences with the following at Atlassian:

Atlassian's boot camp for new employee on-boarding

Learn company's culture and values
Blur discipline lines from the start
Know each discipline's activities
Everyone presents an idea to help the company

How do projects with no testers succeed?

How did Atlassian get into this situation?
What makes it work?
Testers acting as teachers

Do testers really have special skills that make them good testers?
Are discipline specific engineering roles the right approach?
Team wide bug bashes – it’s not just QA
Where's Fede headed next?

Links from the show:

Fede’s blog
Fede’s email - fsilva.armas [at] gmail.com

Episode 59 - Fede and Atlassian (Part 1)

Mark Berryman and Jim Wang — Thu, 14 Jul 2011 07:41:55 Z

In this episode of Coding QA, Mark and Jim catch up with past Coding QA host and founder Federico (Fede) Silva. Fede left Microsoft in November of 2010 and headed to Australia to work for Atlassian. Listen in as they compare and contrast working at Microsoft and Atlassian and learn about:

Atlassian’s product line and their approach to product marketing
The different disciplines at Atlassian
Atlassian’s support team’s role in product development
Dev/Test/PM ratio at Atlassian
Atlassian’s use of a special cross-product group

Links from the show:

Fede’s blog
Fede’s email - fsilva.armas [at] gmail.com

Episode 58 - Improving Test Team Morale

Mark Berryman and Jim Wang — Thu, 07 Jul 2011 00:00:00 Z

In this episode of Coding QA, Mark and Jim sit down with fellow team member Carl DaCosta to discuss test team morale. Carl has been a member of the ASP.NET QA team for nearly 5 years and has recently become a managing test lead. With this new role, comes the expectation of greater focus on test team morale. In this episode, the guys discuss:

Identifying morale issues
Microsoft’s annual poll for assessing employee happiness
Tactics that have boosted our test team’s morale this past year
Importance of defining clear career paths for non-managing testers

So tune in for a lively discussion on all this and more in this week's episode of Coding QA.

Episode 57 - Optimal Test Automation

Mark Berryman and Jim Wang — Fri, 17 Jun 2011 00:00:00 Z

In this episode of Coding QA, Mark and Jim sit down with fellow team member Anton Piskunov. Anton has been a member of the ASP.NET QA team for about 3 years and during that time, he's played the role of feature tester but has more recently moved into owning the team's test automation effort.

• What are the challenges the team's automated testbed faces today?
• How are we getting developers to share more of the test burden?
• How do we ensure an effective and high quality test bed going forward?
• What role can code coverage play in determining which automated tests to run?

So tune in for a lively discussion on all this and more in this week's episode of Coding QA.

Episode 56 - Localization and Globalization

Mark Berryman and Jim Wang — Wed, 01 Jun 2011 00:00:00 Z

In this episode Mark and Jim discuss localization and globalization testing with resident expert Clay Compton. Clay has been with the ASP.NET test team for over a decade and has a reputation for being an expert in these areas. He covers why this testing is critical to an international product, shares tips for developers dealing with internationalization and describes Microsoft’s test strategies for tackling these challenges. So tune in for all of this and more in another episode of Coding QA.

Links from the show:

Globalization, Internationalization and Localization in ASP.NET MVC 3, JavaScript and jQuery - Part 1

Episode 55 - SDE vs. SDET, Scenario Focused Eng

Mark Berryman and Jim Wang — Fri, 13 May 2011 00:00:00 Z

In this episode Mark and Jim answer a listener question - "What is the difference between the SDE and SDET roles at Microsoft?". Both roles do testing. Both roles do coding. So what's the real difference here and is that difference shrinking? Then, they dive into the concept of Scenario Focused Engineering. What is this and how does it affect the way products are developed and tested? Join them again this week as they provide another look into the world of testing at Microsoft.

Episode 54 - Interviewing at Microsoft

Mark Berryman and Jim Wang — Sat, 23 Apr 2011 00:00:00 Z

In this episode Jim and Mark discuss interviewing for a tester (aka SDET) position at Microsoft. What's it like, what questions might be asked and what are we looking for? Join them as they discuss all of this and more in this week's episode of Coding QA.

What's New?

Mix11

Recorded sessions are available

MVC 3 Tooling Update

Interviewing for an SDET Position at Microsoft

Caveat - These details are related to our team. In other MSFT groups, your mileage may vary.

We're hiring!

Interview process overview - technical screens and then the formal loop.

What's a technical screen?

What skills are assessed?
- Coding skills
  - Why focus on coding skills for a tester?
  - Pay attention to hints and prove you're listening.
- Testing skills
  - How are testing skills assessed?
  - Concrete and abstract testing questions.
  - Where's the creativity?
  - No professional testing background? No problem.
  - Different questions for different levels of SDET positions.
  - Test creativity is critical; day-to-day work can crush it.
  - Be an advocate for the customer.

We want candidates to be comfortable during the technical screen.

Homework based interviewing
- Set of questions you work on at your convenience.
- Time boxed.
- Homework answers will be discussed in person; no cheating!
- No late homework.
- Practice, practice, practice
- Formal interview loop will be a very different format.

Should you prepare for the screen or interview?
- Yes, but to what degree?

Resume inflation - argh?!

Once again - We're hiring!

Episode 53 - Tech Talks and Effective Testing

Mark Berryman and Jim Wang — Sat, 02 Apr 2011 00:00:00 Z

In this episode, Mark and Jim review their past week. Jim was down in Orlando speaking at DevConnections, engaging the community, and missing flights while Mark was hammering away (undisturbed) at an upcoming MVC tooling update and recording it all. Join them as they recount their adventures from the past week and how it all leads to better testing.

This Week’s News

NuGet 1.2 released
Head over to http://www.nuget.org for details

Jim was at DevConnections

Delivering feature talks, the value of preparation, engaging customers and how it all makes you a better tester
The joys of flight

Mark's week as a tester, testing

Focused testing time is critical
Few meetings, no rushing testing, how realistic is this?
Employing testing tools

Screen recording
Cutting bug videos
Use it wisely

Debuggers

Providing a deeper understanding of a bug
Providing more information to the developer

App building rocks again

Balancing test coverage versus producing a real world app
Finding bugs typical feature testing will probably never catch
Broad and up-to-date regression coverage

Episode 52 - Testing - Done Differently

Mark Berryman and Jim Wang — Mon, 14 Mar 2011 00:00:00 Z

In this episode Jim and Mark talk about the recent release of Visual Studio 2010 SP1 and the new web related features included in the SP. They also describe a few new approaches to testing - engaging in feature development early and focusing on using their product like a customer. Learn how they’re able to achieve this, why it’s important and some of the value they’ve already seen in this approach.

This Week’s News:

Visual Studio SP1 released
- IIS Express, Sql Compact Edition (SqlCE) tooling support
- MVC, SqlCE and ASP.NET Web Pages bin deployable assemblies
- HTML5 and CSS3 support
Developer Experience Testing
- Engaging in feature development early. It’s possible if you’re not stuck dealing with test debt from the last release
- Broad product knowledge helps
- Ties in with our application building experience
Application Building and Being the Customer
- Focused application building is better than forced application building
- '”Wishlist” application released as a template for ASP.NET Web Pages
- Highlights design issues that would frustrate users
- Requires significant time investment
- Need to balance test value versus getting an application to “product quality”
- Critical issue discovered involving interaction between three technologies
- If you don’t know how your customers use your product, you need to figure that out.

Episode 51 - New Owners and Test as Dev

Mark Berryman and Jim Wang — Sun, 06 Mar 2011 00:00:00 Z

In this episode Matthew discusses his transition from QA to Developer. CodingQA is officially handed over to two other ASP.NET QA team members - Mark and Jim. We cover the latest product releases since the last episode, and we look at some examples of testers being developers.

This Week's News:

New role
- Matt re-joins the ASP.NET team as a developer
- Why the switch?
New owners
- Mark and Jim take over the show
- Introduction, history and role in the team
Product releases
- WebMatrix, ASP.NET MVC 3, NuGet, and the ASP.NET Sprite and Image Optimization Framework Preview 3

Testers as developers – real world examples:

Mark takes a developer role working on the Sprite and Image Optimization Framework
- How did that happen?
- Jim tests Mark’s code
- What did Mark miss and why?
Jim describes his adventures playing the developer on the on the ASP.NET Web Pages templates NuGet packages
- Mark tests Jim’s code
- What did Jim miss and why?
The value of a second set of eyes
Thoughts on how we can further blur the lines between tester and developer

Episode 50 End of an Era

Mark Berryman and Jim Wang — Mon, 01 Nov 2010 00:00:00 Z

In this episode Matthew and Federico talk about Federico’s departure from Microsoft & the podcast. They talk about the why and where of Fede’s new adventure down under. Along with getting a little sentimental and talking about the evolution of the Coding QA Podcast. Matthew also gives Fede time to ramble on his last thoughts on the team, Microsoft, and life. Join them as they send Federico off in style.

This Week’s News

Contest has ended and prizes handed out
PDC10 and Dev Connections
NuPack/NuGet Package Manager

The End of an Era

Federico is leaving Microsoft & Coding QA
- Why he is leaving
- Where he is headed to
Evolution of the podcast
- Four stages of the podcast
- Funny stories and anecdotes
Federico’s last thoughts
- Three things the team didn't get to improve on
- One thing the team did improve on
Contact Info
- Federico's Email: fsilva_armas@hotmail.com
- Federico’s Blog: http://www.federicosilva.net/

Episode 45 Revisit Developer & Tester Roles

Mark Berryman and Jim Wang — Mon, 11 Oct 2010 00:00:00 Z

In this episode Matthew and Federico revisit the idea of a combined developer and tester role. The discuss some of the benefits and some of the down sides to blurring the line. Providing justification for having a blurred role and also justification for the need to keep the roles separate. Join them as the debate and give their thoughts on the separation of roles or the lack there of.

This Week’s News

Extending the TekPub Give-A-Way deadline to allow for more entries (more information will be post here)
Security Patch for POET exploit

Developer vs. Tester Role

Organizational Structure
- Dev org & Test org
- Orgs don’t combined until several levels up
- Drives organizational goals not product goals
Possible Solutions
- People switch roles between Dev and QA
- Everyone is responsible for both Dev and QA
- Responsible for both with Experts
- Stay separate
A few misc ramblings about it all

Episode 44 Bug Verification Checklist

Mark Berryman and Jim Wang — Wed, 15 Sep 2010 00:00:00 Z

In this episode Matthew sets down with Mark Berryman again and talks bug verification. They discuss a checklist that Mark has created to help ensure proper verification of bug fixes. While not all encompassing it is a great reference point when a tester goes to work verifying a fix.

This Week’s News

Extending the TekPub Give-A-Way deadline to allow for more entries (more information will be post here)

Bug Verification Checklist

Why did mark create the list & the idea behind it
The Checklist:
- Verify the fix works
- Reproduce the original bug
- Access the code fix
- Check the unit tests
- Test around the fix
- Look for similar problems elsewhere
- Think about other ways to hit the bug
- Look for security holes
- Recognize uncertainty
- Run relevant regression automation
- Double check bugs marked as duplicates
Meta points about the list

Mark's Plug

http://www.testingtoasters.com

Episode 43 Black and White Box Testing

Mark Berryman and Jim Wang — Fri, 16 Jul 2010 00:00:00 Z

In this episode Matthew and Federico discuss the differences between black box and white box testing. Based on their experience testing MVC and WebMatrix, they talk about problems that can happen if testers are too focused on code inspections as well as advantages if used properly.

This Week’s News
- Codeplex is hiring
- Coding QA teams up with TekPub for a Give-A-Way [ More Details Here ]

Black box and white box testing.

Definitions
- Black box is testing performed without knowledge of the product code. White box (or glass box) is testing with the knowledge of the product code.
- A black box tester becomes an expert of the relationship between the program and the world on which it runs.
- A white box tester becomes an expert of the structure of the program and the internal representation and management of data.
- White box testing can also be described as “inspection” and is one of the test activities that does not involve code executing by a computer.
Problems with white box testing
- Customer detachment: When a tester starts justifying customer expected behavior based on implementation knowledge.
- Bias based on visual complexity: When a tester bases strategy solely on how hard or simple a piece of code looks.
- Overwhelmed by complexity: When a tester spends too much time understanding every single piece of code instead of testing.
Advantages of white box testing
- Source of information to guide testing.
- Helps to better understand interaction between modules.
- Can help in locating assumptions not easily found by blind inputs.

Episode 42 Introducing WebMatrix

Mark Berryman and Jim Wang — Thu, 08 Jul 2010 00:00:00 Z

In this episode Matthew talks with Scott Hunter about the new released WebMatrix, Microsoft's free tool for building websites on windows. They give you an inside look into how the team comes up with new features and products as they discuss the reasons for creating WebMatrix. They also discuss the short life cycle of the products and how that effected the way the team functioned and how the disciple lines where blurred.

This Week’s News
- WebMatrix Beta is Public!
- Coding QA teams up with TekPub for a Give-A-Way [ More Details Here ]
What are the products and features make up WebMatrix
- SQL CE 4: A New Embedded Database for ASP.NET
- IIS Express: A Lightweight web server
- WebMatrix: A simple easy to use editor
- ASP.NET WebPages: A new framework for creating websites built on ASP.NET Core
- Razor: A new syntax used by both ASP.NET WebPages and a new ASP.NET view Engine
Why we decided to build WebMatrix
- Make it easy to start writing awesome web sites
- Reduce the amount you have to type
WebMatrix’s super short sprints and how it changed the way the feature crews worked
- The QA team was involved in features from the very beginning including prototyping
- The QA team spent more time inside the code
- Members of the feature crews blurred disciple lines
- There was lots of app building to validate designs

Episode 41 iPhone TDD

Mark Berryman and Jim Wang — Tue, 08 Jun 2010 00:00:00 Z

In this episode Matthew talks with Scott Densmore about Test Driven Development for the iPhone. The Objective C world is slowly gaining popularity with platforms like iOS and producing High quality apps is a must. Test Driven Development is just starting to catch on in this world and Scott and Matthew discuss how Scott has been able to maintain a TDD lifestyle while developing for the iPhone.

iPhone TDD

How Test Driven Development (TDD) works on hardware
- What are some of the tools
- What frameworks are available (mocking, unit testing, etc.)
How does the standard TDD pattern change
- Is the red/green pattern still followed
- Can you still have things like Continuous Integration
- Parts of the pattern that just don’t work
Personal Experiences developing iPhone applications
- Benefits to the TDD lifestyle
- Gotchas and downsides to the TDD lifestyle

Information from/about Scott

Episode 40 Continuous Integration

Mark Berryman and Jim Wang — Thu, 27 May 2010 00:00:00 Z

In this episode Matthew talks with Brad Wilson about Continuous Integration. The ASP.NET team has converted almost all of there projects to use some sort of continuous integration pattern. In this episode they discuss what CI is and how it helps the team produce a better product.

Continuous Integration

What is continuous integration
- What is the idea behind it
- What is the difference between an integration build and an incremental build
Details about the CI server
- What is run for each build
- How often builds are run
- What technology is used
How does CI help the team produce a better product
- Provides quick feedback
- Provides new bits to everyone
Tips & Tricks about using a CI server

Episode 39 Product Metrics

Mark Berryman and Jim Wang — Mon, 10 May 2010 00:00:00 Z

In this episode Federico and Matthew talk about product metrics. Metrics are a common technique used by management to measure progress towards some goal, but they are not without risks. This show lays ground to understand what are product metrics and common pitfalls to avoid.

Product Metrics

Definitions (taken mostly from Cem Kaner's article and IEEE standard)
- Attribute: measurable physical or abstract property of an entity.
- Measurement: the assignment of numbers to objects or events according to a rule derived from a model or theory.
- Metric: a measurement function.
- Quality Factor: a type of attribute. A management-oriented attribute of software that contributes to its quality.
- Software quality metric: a function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affect its quality.
Requirement statement
- The most important aspect when designing a metric is to think about the intended goal.
- "To [understand, evaluate, control, predict] the [attribute of the entity] in order to [goal]"
Metric design
- Goal -> Questions -> Metric
Evaluation of a metric
- How much do we need to know about an attribute before it is reasonable considering measuring it?
- How do we know if we have really measured the attribute we wanted to measure?
- Direct and indirect measurements.
- Distortion: metric creates incentives for the employee to allocate his time so as to make the measurements look better rather than to optimize for achieving business goals.
- Dysfunction: if optimizing for a measurement so distorts the employee's behavior that he provides less value to the organization than he would have provided in the absence of measurement.
Examples of bad metrics:
- Bug counts as a metric of the work of testers and programmers.
- Code coverage as a metric of code correctness.
- Test case counts as a metric for quality.
Metrics pitfalls
- Measuring too much, too soon.
- Measuring the wrong things.
- Collecting data that is not used or shared.
- Defining the metric imprecisely.
- Using a metric for individual evaluation.
- Ignoring cultural issues.
- Misinterpreting metric data.
- Expecting the metric design to stay constant.

Episode 38 Surviving the Corporate Jungle

Mark Berryman and Jim Wang — Tue, 04 May 2010 00:00:00 Z

In this episode Matt and Federico have fun talking about tips to survive the corporate jungle. After too many serious episodes, in this show we joke about our secret strategies to avoid getting thrown under the bus.

Surviving the Corporate Jungle:

What to do when somebody asks for volunteers in a meeting?
How to handle feedback from a review meeting?
What to do when your manager tells that “this is a very good opportunity for you” and will “increase your scope of impact”?
Teach, don’t answer.
Avoid offering a suggestion you are not willing to do.
Research who will attend your meetings.
Don’t let your meeting be hijacked.
Have fun!

Episode 37 Installer Testing

Mark Berryman and Jim Wang — Sun, 25 Apr 2010 00:00:00 Z

In this episode Federico and Matthew talk about their experience testing installers. Taking from their experience testing the ASP.NET MVC installer, they share tips and advice for those faced with the challenge of testing the very first experience of many products: the installer.

News:

Installer Testing

Before somebody gets to use a product, they have to get it into their machines. It is the first experience with a product.
When building an installer test plan start by brainstorming 4 general areas:
- Pre-requisites. How does the installer check for requirements? Does it provide enough information if requirements are not met? Does it fail as soon as possible?
- Installation: What files are copies and where? Registry changes? GAC changes? Folders created? IIS changes? Does it need to check the web?
- Uninstall: Is everything rolled back completely? Is anything is supposed to be left behind?
- Machine state: OS, bitness, other applications running, hardware, disk space, network access, permissions.
Based on these variables, start building combinations of scenarios.
Use additional heuristics for identifying and discussing special cases:
- Interruptions: Machine goes into hibernation, unplug network cable, user cancels the installer.
- Full circle: Snapshot the state of the machine before installing, make sure it is the same after uninstalling.
- Side by Side: Previous versions of the software installed, multiple versions installed.
- Reinstall/Repair: Reinstall multiple times, break the application and repair it.
- Temp folder: what is written in the temp folder.
- Unintended changes: Additional files in placed installation directory.
- Partial installs: some of files that are meant to be installed are already present, some of the files that are meant to be removed are not there.

In our experience, installer testing is closer to scripted testing than exploratory testing.

Episode 36 Damian Edwards on Real Testing

Mark Berryman and Jim Wang — Fri, 09 Apr 2010 00:00:00 Z

In this episode Matthew and Federico sit down with Damian Edwards to talk about his experience working for a company that writes functional tests as part of their development cycle. Damian is a new Program Manager in the Visual Web Developer team and used to work for Readify, a consulting company.

Interview with Damian Edwards

Introduction
Types of testing that an agile team can do.
- Unit testing
- Integration testing.
- Functional testing
- Performance/Load testing.
Working on a team that releases to web every week.
- Who is responsible for what type of testing?
- One week is too small of a time to do dev and test work.
- Developers implement features in one week, testers test it the next.
- What happens if a tester finds a bug?
Using functional automated tests.
- Is there enough time for testers to write them?
- What automation framework did you use?
- How did you deal with fragile tests?
- Making markup easier to automate vs. making markup minimal for client.
Links

Episode 35 Code Quality for Automated Tests

Mark Berryman and Jim Wang — Fri, 02 Apr 2010 00:00:00 Z

In this episode Matthew and Federico talk about the quality of test code. They share what to look for when reviewing test code and their experience dealing with automated test cases written by a large team.

News

One year anniversary of the Coding QA podcast!!!

Test Code Quality

A sense of pride on the code you write.
Reviewing tests is more effective when done early.
What to look for when reviewing tests?
- How easy it is to figure out why a test failed from the log?
- How good is the test at catching the intended bug?
- How resilient is the test to other changes?
- Over verifying and under verifying.
- Does it align to .NET coding guidelines?
Common excuses by testers
- I don't have time.
- I don't understand what is the value.
- Test code is not the same as product code.
Test engineers produce code, as professionals we must strive for the best code

Episode 34 Testing Heuristics

Mark Berryman and Jim Wang — Thu, 25 Mar 2010 00:00:00 Z

In this episode Federico and Matthew talk about test heuristics. Starting with what are they and why they are useful, they give a run down several heuristics that the ASP.NET QA team uses while exploring software.

News

MVC 2 Released

Test Heuristics

What is a heuristic?
Why are they useful?
How does the ASP.NET QA team uses them?
Some useful test heuristics
- Configuration: Change variables related to configuration (of program as well of machine)
- Interruptions: Log Off, Hibernate, Kill Process.
- Undermining: Change the state of the program to an invalid state while functions are running.
- Starvation: Progressively lower resources until product collapses or gracefully degrades.
- Dog Piling: Get more processes going concurrently.
- Click Frenzy / Street Fighter: Run through all the keys and click everywhere.
- Multi User: Simultaneous users performing actions on shared resources.
- Multi Instance: Run multiple instances of the application.
- Flood: Multiple simultaneous transactions or requests flooding the queue.
- Dependencies: Identify "has a" relationships and apply CRUD to each.
- Constraints: Violate every constraint.
- Input Methods: Typing, copy/pate, cut/paste, drag/drop
- Sequences: Vary order of operations, undo/redo.
- Position: Begin, middle, end
- Boundaries: Approaching the boundary, at the boundary
- Selection/Count: Some, None, All.
Resources: Most of our heuristics were collected from others across the web. Including, but not limited to James Bach, Michael Bolton and Cem Kaner.

Episode 33 Team Structure

Mark Berryman and Jim Wang — Tue, 02 Mar 2010 00:00:00 Z

In this episode Matthew and Federico sit to talk about some of the changes that are happening in the ASP.NET QA team. We have talked about the evolution of our team, and today we go over the idea of dividing the test team into functional disciplines and how it has worked so far.

News

ASP.NET QA team is hiring
MVP Summit

Organization of the QA team

What was the problem that we identified?
- Lack of specialization.
- Lack of resources to deal with test debt.
- Lack of resources to improve our efficacy.
- 100% of tester's time was dictated by a feature crew.
The 3 pillars of the team
- Product Presence - test presence during product design and prototypes.
- Product Development - core testing of new features as they are incorporated into the product.
- Engineering Services - automation strategy, tools/infrastructure and regression testing.
Work flow of a feature crew
- As a new feature is designed PP is engaged during early prototypes, verifying scenarios with customer expectations, first exploratory sessions.
- When the feature is mature enough, PP hands of testing to PD group. These group of testers are normally assigned to cross discipline feature crews and perform the bulk of exploratory testing and any other test activities.
- As the feature stabilized, ES comes in to help automate regression tests, making sure tests are robust and maintainable. At the end this group continues to run regression tests of the whole product.
Take away
- Still an experimental model.
- So far the gains seem to indicate that the overall capacity of the team is increased.
- A potential next step is to recognize the value of "on-demand" testing and "constant" testing, and to focus a group of testers on each.

Episode 32 Last Minute Changes

Mark Berryman and Jim Wang — Fri, 29 Jan 2010 00:00:00 Z

In this episode Matthew and Federico again take from there experience of working on the MVC project and this time talk about last minute changes.

Why last minute changes happen?
- High priority bug found late in the cycle.
- Design change from customer feedback.
- Stake holders disagree with a design decision.
How to tackle it?
- Extend the date
- Cut corners
- Cut features
- More resources
- Be creative with the resources you have.
How to prevent them?
- Build buffer into your planning.
- Reach out to as many people as you can early and often.
- Figure out the requirements and processes of your dependencies.
- Build trust within your team.

Episode 31 The Vicious Cycle

Mark Berryman and Jim Wang — Fri, 15 Jan 2010 00:00:00 Z

In this episode Federico and Matthew narrate their experience of how having unclear expectations can lead down a path that can eventually create a culture of treating QA as a burden. It is a series of event that we call the vicious cycle of QA. Federico has been on the team for almost eight years and today shares his observations of how the team got itself into an undesired place, what were the symptoms, and what our decisions created. Look for a future podcast to learn what actions we took to break this cycle and move forward.

News

The ASP.NET QA team is hiring.

Vicious cycle of QA

In a nutshell: unclear expectations leads to blame which leads to more process which leads to bottle necks which leads to more unmet expectations.
Step 1: Unclear expectations
- QA, PM and Dev do not have a clear understanding of what is the role of the testers. What can they expect from them, and more importantly, what are the limits of their responsibility.
- Having unclear expectations means that every person on the team has a different idea of what are testers supposed to do. Even higher level managers.
Step 2: Fail and get blamed
- Without having clear conditions of success, what may turn out is that each miss is considered a failure.
- Since the responsibility is not clearly defined, it is easy to have the QA team be the focus of blame.
Step 3: Cover your back with processes.
- The QA team may turn to create processes as a means to cover their backs and transfer responsibility to the rest of the team.
- Many processes are created that gives security that as long as they are followed, the QA team can shield behind them.
Step 4: Push back when process doesn't fit the plan.
- With so many processes and checklist across the board, the QA team may become the bottle neck of the development cycle.
- Since processes have to be guarded, QA team starts to push back on new features due to lack of time to properly do testing.
- Turf wars may begin between the several disciplines.
Repeat:
- Push and pull eventually settles into a set of constraints that QA is unable to meet given the high cost of test process.
- More unmet expectations ensue.
Culture of QA as a burden
- Since the processes where created to cover ones back instead of doing smarter testing, the perceived value of QA is still low.
- Eventually the perception and reputation of the QA team is that of a high cost and low value.
How do we break out of this cycle? How do we turn around the team and perception? This and more questions will be addressed in a future podcast

Episode 30 Pair Testing

Mark Berryman and Jim Wang — Sun, 20 Dec 2009 00:00:00 Z

In the Episode Matthew sets down with Drew Miller, a fellow tester, and they talk about the Pair testing and how they are implementing in on the QA team.

News

Apologies for the lack of episodes on regular basis
ASP.NET MVC 2 RC released
Public release of Visual Studio 2010 RC
QA Team is still hiring

Introduction to Drew Miller

Drew's work history
Fun facts about Drew's interests

Pair Programming/Testing

What is this pairing thing?
How Pair Programming fits into our team's methodologies
Justifying the cost of pair programming
Possible Issues when your new to pairing
Creating a pairing environment to help
Is pairing the end all of coding styles?

Episode 29 Time Not Testing

Mark Berryman and Jim Wang — Fri, 04 Dec 2009 00:00:00 Z

In this episode, Matthew and Federico talk about how easy it is for testers to spend time in activities other than actual testing. They share their experience of being aware of the time spent not testing and how important is to keep a balance.

News

Managing your test time

What is test time?
- Time spent designing tests
- Time spent executing tests
- Time spent evaluating results
- Time spent investigating bugs
Other activities that take time away from testing
- Meetings. Like sync-ups, status reporting, planning, training, triage, etc.
- E-Mail.
- Building relationships.
- Hardware and software issues.
A day in the life of a tester. Matt's and Federico's day.
Some realizations
- Building sofware is a social activity, mantaining relationships is very important.
- There are many support activities that are crucial for the success of the project.
- The importance is to make sure that you still have enough time to sit and test.
- Try to balance test time and non-test time and make sure that you are happy with the ratio.
- One easy way to increase your test results is to identify non-test time and to try to minimize it.
How much percentage of time are you dedicating to actual testing?
What do you think of managers that move away from actual testing?

Episode 28 Chatting with James Bach

Mark Berryman and Jim Wang — Tue, 17 Nov 2009 00:00:00 Z

In this episode Matthew and Federico sit down with James Bach to talk about everything to do with testing. James ideas were very influential in shaping the test methodologies that our team practices, especially around the use of exploratory testing, and he shares his views and passion about becoming a test ninja with us.

How did James got started in the test discipline?
Exploratory testing
- Exploratory testing as an approach instead of a technique.
- Some of the misconceptions about exploratory testing.
- Difference between exploratory testing and ad-hoc
- Teach exploratory testing by creating a test culture around developing mental skills
- How to know when testing is done?
- How to evaluate good exploratory testing notes.
- Using a screen recorder as a supplement to note taking.
Scripted tests
- Strengths and weaknesses of scripted tests.
- Automation as simple product checks.
Testing as a profession
- How to get really good at testing?
- Study and define the test activities until they are clear and no longer intuition.
- Example: how to know if something is a bug
Management
- Work to get rid of bureaucracy that stands in the way of the tester.
- Good leads will sit with testers to guide and evaluate them.
- Use sessions as a way to manage a team of exploratory testers.
Challenges of testing in Microsoft.

Links and Plugs

Episode 27 Unit Testing With Brad Wilson

Mark Berryman and Jim Wang — Sat, 31 Oct 2009 00:00:00 Z

In this episode Matthew and Federico sit with fellow team member Brad Wilson to talk about Unit Testing. Brad is a developer in the ASP.NET team, he has a lot of experience developing using Test Driven Development and is one of the creators of the xUnit testing framework.

What is Unit Testing?
What is the difference between Unit Testing and Test Driven Development (TDD)?
Unit Testing in practice
- How does the MVC team uses unit testing?
- How do you arrange your tests?
- How do you name your tests?
- What about using internal to be able to test something?
- What if I inherited code that didn't had unit tests?
- How can I convince my team to adopt unit tests as a development practice
Addressing common concerns about Unit Testing
- It feels like a lot of work for little value.
- Having to rewrite a lot of tests if something in the design changes is a pain.
- Code ends up being a lot of very small methods and I donâ€™t like that.
Common pitfalls
- Writing very big tests.
- Testing to internal implementation.
Mocking
- What does mocking mean and why is it useful?
- What is your favorite mocking framework?
Resources
- Brad's Blog
- xUnit Framework

Episode 26 Reducing Waste

Mark Berryman and Jim Wang — Fri, 23 Oct 2009 00:00:00 Z

In this episode Federico and Matthew talk about how to apply a concept from lean manufacturing to rethink the processes of software development. Taking the real example of a "Regression Run" they discuss how to dissect the process to identify waste that the team can then drive to eliminate.

News

Visual Studio 2010 Beta 2 released.

The process of a regression run and the waste associated with it

For our team, a regression run is the automated execution of tests that where written for features released in previous releases.
For every process, the question to answer is "what does the customer wants for this process"?
If we think of a regression run as a process, we can define that what the customer wants from it is to be able to install a new version of the product and have his/her applications continue to work as before.
The goal of a regression run is then to find breaking changes for the customer.
Waste is any activity that does not add value to the outcome of the process. In the context of a regression run it could be any activity that did not identify a breaking change for the customer.
Types of waste

Waiting. Time spent waiting for results, for a tool or for fixes.
Unnecessary analysis. Analyzing a failed test that was not caused by a breaking change.
Over analysis. Analyzing a test that ensures higher quality than necessary.
Time of analysis. Time spent between receiving a result and identifying the breaking change.
Unused creativity. Creativity of the people that could be used to improve the process.

Instead of thinking about identifying and enhancing the value-add activities, you could work on identifying and eliminating the non-value-add activities.

Episode 25 Test Priorities

Mark Berryman and Jim Wang — Tue, 20 Oct 2009 00:00:00 Z

On this episode Matthew and Federico talk about how our team partitions the test bed by using priorities. They discuss why having an incremental automation test bed is important and how having a standarized meaning for priorities can help.

News:

Microsoft Ajax Library Preview 6
Welcome to our new team member Drew Miller.

Test case priorities

Grouping tests by a common priority bar is useful when dealing with a big test bed.
Pri 0
- A small number of positive tests under one configuration.
- Constitute 10-20% of a feature's tests.
- Regression amounts to a ship stopper.
Pri 1
- Coverage for 80% of the use cases of a feature under multiple configuration.
- Constitue 80% of a feature's tests.
Pri 2
- Edge or low impact scenarios.
- Constitute 10-20% of a feature's tests.
Why do we need pri 0 functional tests if we already have unit tests?
Your guidelines are ambiguous, how do you apply them?

Episode 24 Process Overhead

Mark Berryman and Jim Wang — Fri, 09 Oct 2009 00:00:00 Z

In this episode Federico and Matthew share some of their personal anecdotes regarding process overhead. By examining concrete examples, they discuss pitfalls and how to remove unnecessary processes.

News:

MVC 2 Preview 2 Releases

Process Overhead

The process of keeping internal specs and documentation up to date.
The process of triaging bug defects for a large project.
The process of checking code into the repository.
The process of maintaining a detailed scheduled of work.
Processes that provide value to the business vs. process that provide value to a person.
Reinvent a process vs. just throw it away.

Episode 23 Port-a-Test a portable test bed

Mark Berryman and Jim Wang — Sat, 26 Sep 2009 00:00:00 Z

In this episode, Federico and Matthew talk about the benefits of having an automated test bed that can be easily transferred and run from any machine. They discuss some of the differences between a remote execution test bed and a portable test bed, and how the latter has worked well for the MVC feature crew.

News

WebSiteSpark was released and is available for all web startups.

The portable test bed.

How the remote execution test bed worked in the ASP.NET QA team.
- Use internal test harness that can distribute tests on remote machines.
- Enforces several design and execution paradigm on test authors.
Problems and limitations that we faced with a remote execution test bed.
- Difficult to share tests with other teams.
- Difficult to react quickly to product changes.
- Tendency for tests to be locked down to a specific execution environment.
The concept of a "portable test bed" as a solution.
- Test bed can be xcopy deployed to any target machine.
- Low dependencies on external factors
A layered automation infrastructure is preferable
- Build automation solution that can move in a spectrum from high flexibility and high manual intervention to low manual intervention and potentially low flexibility.
- If a high automated solution fails, tester can quickly and easily revert back to a more manual approach to achieve goal.

Episode 22 Swatting Bugs

Mark Berryman and Jim Wang — Sun, 20 Sep 2009 00:00:00 Z

In this episode of CodingQA, Federico and Matthew share their experience of what an appropriate response is to critical bugs found during development. Using a real case from MVC, they discuss how the feature crew reacts to high impact bugs and how the problem is addressed for the future.

News:

Microsoft AJAX can now be downloaded via CDN.

The appropriate response

Summary of the MVC build verification process
- Daily checkins
- No continuous integration server.
- Dev is responsible for running unit tests before checkin.
- QA responsible for running functional tests with every build.
- Functional tests are fully automated and run in less than 30 mins.
- Test results are transmitted verbally.
The ship stopper problem
- At some point during the development cycle, QA makes more comprehensive runs.
- During these runs a bug was found where a security exception is thrown when running on Medium trust and MVC is in the GAC.
The response
- First: Verify that you have discovered the extent of the problem. Sometimes a bug could be the tip of the iceberg, are there more bugs around the vicinity that we need to find?
- Second: Investigate and understand how the bug got introduced. Was it there from the beginning? Did a subsequent checking broke the scenario? Is the problem in our code or in a collateral component?
- Third: Place checks to prevent a similar problem from reoccurring.
The learning
- Always explore and app build with Medium trust (a project defaults to Full trust)
- The QA team had been making runs with MVC in the bin instead of the GAC, going forward we will mix bin and GAC from the beginning.
Do not focus on blaming people for a bug not found, focus on improving your process and your safety net.
It is a misconception that the QA team is responsible for missing a bug. It is the whole team's responsibility to produce high quality software

Episode 21 Agile Manifesto

Mark Berryman and Jim Wang — Sat, 12 Sep 2009 00:00:00 Z

In this episode Matthew and Federico give their impressions regarding the agile manifesto. Drawing from experience in the ASP.NET QA team, the meaning of the agile values are discussed, with examples of how the MVC feature crew applies them and some pit falls.

News:

ASP.NET 4.0 Ajax Preview 5
Jim Wang to give a talk at DevConnections

The Agile Manifesto

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

Episode 20 Test Contexts

Mark Berryman and Jim Wang — Fri, 28 Aug 2009 00:00:00 Z

In this episode Matthew and Federico sit down to talk about test Contexts, and how our team uses this concept in automation. Positive and negative points are discussed as well as the challenges that the QA team faces when applying the concept in excess.

News
- The ASP.NET QA team is hiring! If you are interested in working in our team please send us your resume.
- Podcast is now accessible via Zune
Contexts

What are "contexts" or "variations"?
Some of the contexts that we use: TrustLevel, Impersonation, Browsers, ViewState, UNC, RemoteServer, SmartNav, etc.
Pros: breadth coverage, cheap for the tester, confidence of no points missed.
Cons: excessive results, skip evaluation of test value, maintainability cost is often disregarded.
What is the situation in our team? The challenge of maintaining a 10 year old product.

Episode 19 But QA is important too!

Mark Berryman and Jim Wang — Fri, 21 Aug 2009 00:00:00 Z

In this episode Matthew and Federico continue to elaborate on the concept of treating the QA role as a "service" and discuss some of the feedback and discussion that has carried on during the week with their co-workers.

News

Federico has been confirmed as a speaker for Dev Connections!

QA as a "service" revisited

Feedback / Discussion
- The Dev and PM roles also provide a service to the QA role.
- A service is optional, QA is not optional.
- QA provides lots of value why do you say we need to work on providing it?
Continued discussion of QA as a "service"
- What is the problem that treating QA as a "service" solves?
  - Perceptions of the QA team's value.
  - Divisions between disciplines.
- QA as a "Service" is just the beginning and should be part of a bigger picture.

Episode 18 Would you like fries with that?

Mark Berryman and Jim Wang — Fri, 14 Aug 2009 00:00:00 Z

In this episode Matthew and Federico discuss what it means to treat the QA role as a "service" rather than a "requirement".

News

Fede has a new blog about testing.
Fede to give a QA talk at Dev Connections

QA as a service

Signs that development team sees QA as a burden.
Difference in QA cultures:

"some else is responsible, they need to change"
"influence others by changing yourself"

Think of QA as a service provider and the rest of the development team as the consumer. Strive to give as much value.
Perception and reputation and why it's important to provide value that is tangible to the rest of the team.
Change the QA response from "no, we can't do take that work", to "how can we find something that works".
Increase awareness across the development team about what, how and why the QA team does what it does.
First hand anecdotes from our team.

Episode 17 MVC 2 Preview 1

Mark Berryman and Jim Wang — Fri, 07 Aug 2009 00:00:00 Z

In this show Matthew and Federico share their experiences working as testers in the MVC feature crew. They go over how the team works, the development lifecycle from a QA point of view, the principles that the team adopted as well as an honest assessment of what didn't go so well.

News
- MVC 2 Preview 1 released
- What's new in MVC 2(ScottGu)
  - Ability to easily partition and group functionality across an MVC application.
  - Ability to use Data Annotation attributes to validate input.
  - Ability to use Data Annotation attributes to drive the generation of UI in the Views (ala Dynamic Data)
  - Strongly typed UI-Helpers
Inside the MVC team
- Lifecycle of a "feature": Idea -> Prototype -> Exploratory Testing -> Review -> Application Building -> Automation (Rinse & Repeat)
- Challenges of testing an installer.
- Development principles
  - QA is involved during design.
  - All feature check-ins are previously approved by Dev/Test/PM
  - All check-ins contain unit tests
  - QA drives the schedule
  - Weekly test plan.
- What could had gone better
  - QA not working close enough with Devs.
  - A lot of non-MVC work caused a lot of distractions.
  - Big bug debt coming in from MVC 1.0
  - Ramp up time for new QA members took longer than expected.

Episode 16 "The Right Stuff" to be Successful

Mark Berryman and Jim Wang — Fri, 24 Jul 2009 00:00:00 Z

In this episode Matthew and Federico sit down to talk about the personality, skills, and traits of a successful QA member. Join them as they each enumerate what they think makes up a great QA member. Tune in and learn what it takes to become a Mercury 7 astronauts, errrâ€¦, a great QA member.

News

Windows 7 has been released to manufactures.

The Right Stuff

What are the types of testing
- Application
- Framework
What are the types of Testers
- Automater
- Coding QA (yes it's a pun on the name)
Matthew's Wish List
- curiosity
- awareness
- assertiveness
- humbleness (think skin)
Federico's Wish List
- curiosity / creativity
- attention to detail
- awareness
- technical skills
- ability to judge customer value

Episode 15 Bug Review

Mark Berryman and Jim Wang — Fri, 17 Jul 2009 00:00:00 Z

In this episode Matthew and Federico sit down to talk about some interesting bugs and lessons learned that our team has run across while testing ASP.NET. This is a different type of show that we are experimenting where the bugs take center stage.

News

Silverlight 3 is released

Bugs Showcase

Sensitive data stored in ViewState.
How a test abstraction hide a security vulnerability bug.
How testing with slow connection revealed Ajax bugs.
Tale of a breaking change: How a bug that is shipped can be used as a feature by customers.
How we found some accessibility bugs by "being the customer".

Episode 14 Charting with Lakshmi Padala

Mark Berryman and Jim Wang — Fri, 10 Jul 2009 00:00:00 Z

In this episode Matthew and Federico sit down with Lakshmi Padala from the SQL Reporting Services team to talk about how they tested the asp.net chart control and what some of the teams processes are. Join them as they learn about what its like to be a SDET in another business group at Microsoft.

News: The AJAX Control Toolkit has started using the Lightweight Test Automation framework for functional testing.
Introduction to Lakshmi and her choice of QA over DEV.
Overview of what the Chart Control is and what it does.
How did the cross team collaboration work with the ASP.NET team?
Overview of the how the chart control was tested.
- API level testing
- Run time testing
- Design time testing
How do you test a product that only produces an image? How does image comparison work?
Scope of the chart team's test bed. (number, speed, and type of tests)
Successful Model Based Test (MBT) implementation in the Reporting Services team
- Advantages to MBT
- Disadvantage to MBT
- Work a rounds to be successful
Challenges using design time abstractions to test.
What is the life cycle of a feature/product in the Reporting Services team?
Trust level security concerns when testing the chart control.
Lakshmi's favorite part about working at Microsoft.

Links:

Episode 13 Test Maintainability

Mark Berryman and Jim Wang — Fri, 03 Jul 2009 00:00:00 Z

In this episode Matthew and Federico sit down to talk about test maintainability and several factors to keep in mind that can decrease the reaction time once a test fails.

The 3 dimensions of maintainability
- Regressions (test fails due to a product bug). How easy can I identify the cause of failure?
- Product Changes. How easy can I update tests due to new behavior?
- Reusability. How easy can I use a test written in the past to test a new feature?
Some of the factors to keep in mind:
- Unique failure.
- Good logging.
- Reproducibility/Robustness.
- Atomic tests.
- Result history.
- Test recorders.
- Abstractions vs. low level implementations.
- Prediction.
- Framework flexibility and extensibility.

Episode 12 Jim Wang on Microsoft AJAX

Mark Berryman and Jim Wang — Fri, 26 Jun 2009 00:00:00 Z

In the twelfth installment of Coding QA Federico is still enjoying his vacation so Matthew sets down with his officemate Jim Wang to discuss the ins and outs of test the Microsoft AJAX libraries. Tune in as Matthew picks Jimâ€™s brain on what it was like to test a client side framework.

Interviewing Jim Wang

A little about Jim and his history at Microsoft.
What brought Jim to the QA team and his preconceptions about what it means to be QA team member.
Discussion around Jimâ€™s responsibilities on the feature crew.
Do client side libraries have lots of unit tests written?
Browsers and what that means to the feature crew.
Handling hand off of code on a client side library.
Discuss on some of the challenges Jim faces on the Microsoft AJAX framework feature crew.
How the design is affected by the QA team.
What is Jimâ€™s Favorite thing about testing the Microsoft AJAX framework and what is the thing he would most like to get ride of.

Episode 11 Interviewing Hong Li

Mark Berryman and Jim Wang — Fri, 19 Jun 2009 00:00:00 Z

In the eleventh installment of Coding QA Federico is on vacation so Matthew sets down with his mentor, Hong Li, for an interview. Join Matthew as he learns a little about one of his fellow team members, servicing an released product, and working with team members that are offshore in China.

Interview with Hong Li

A little history on Hong and how she came to the ASP.NET QA team.
Discussion on Hongâ€™s role with the servicing team.
What is the difference between a hot fix and a QFE?
What is the life cycle of a servicing request?
What is the servicing team and who is on it?
How the offshore team plays into servicing the product.
How do you handle the two teams being on different sides of the world?
Benefits of having round-the-clock operations.
Is everything handled remotely with the offshore team?
What brought Hong to the QA discipline?
What is the biggest challenge Hong has faced when working at Microsoft.
What Hong likes most about working at Microsoft.

Episode 10 Project Structure

Mark Berryman and Jim Wang — Fri, 12 Jun 2009 00:00:00 Z

In this show Matthew and Federico talk about how the ASP.NET team structures it's sources and the theory behind it all. They talk about how the team handles the organization of all of itâ€™s sources and what seems to work well along with what seems to cause hang ups.

News

Project Structure when developing a product

Keeping Developer and QA source separate
Branching sources and the the types of branches
Bring it all together in â€œMicrospeakâ€ (merging branches)
What seems to work well for the team.
What are some of the issues that the team faces

Disclaimer: LTAF runner will be post soon, sorry for the delay!

Episode 9 Releasing a Product

Mark Berryman and Jim Wang — Fri, 05 Jun 2009 00:00:00 Z

In this show Matthew and Federico talk about what the ASP.NET QA team does to prepare for a release. Taking into account the recent release of ASP.NET 4 Beta 1, theyâ€™ll go over several of the "exit criteria" that the product must meet before delivery to the public.

News

Lightweight Test Automation Framework June Release

Sign off on a release

Difference between Quality Gates and Exit Criteria
Accessibility. Mainly manual tests to validate feature conforms to accessibility standards.
Code Coverage. Metric used to identify problematic development areas.
Stress. Specialized automated long running tests to spot memory leaks and other stress problems.
Localization. Verification that the product can be correctly localized to several languages.
Test Passes. Functional verification of the product across many platform combinations.
Zero Active Bugs. Series of step down goals to drive feature crews to finish development.

Episode 8 Testing, what is that?

Mark Berryman and Jim Wang — Thu, 28 May 2009 00:00:00 Z

This show is an open discussion about what testing means. Federico and Matthew share their opinions around the meaning of "testing", "quality" and the role of a QA team. They don't always agree but each brings something unique to the table on topics so basic yet somewhat mysterious in the testing discipline.

News

Topics of Discussion

What is the role of QA?
What is testing?
What is quality?
Should quality be measurable?
What is risk?
What is the difference between Dev and Test?

Episode 7 A Tour Around Exploratory Testing

Mark Berryman and Jim Wang — Fri, 22 May 2009 00:00:00 Z

In this show Federico and Matthew talk about what exploratory testing (ET) is and how the ASP.NET QA team uses this methodology for testing.

News

Exploratory Testing

Definition
Differences between exploratory testing and scripted testing
Differences between exploratory testing and ad-hoc testing
The 4 principles of exploratory testing
- Creativity
- Goal oriented
- Reproducibility
- Adaptability
How the ASP.NET QA team uses exploratory testing
When is exploratory testing a good fit to test a product
Brief example of an exploratory testing session
What are the outcomes of an exploratory testing session
- Scenarios tried
- Questions
- Assumptions
- Defects
- Future scenarios

Episode 6

Mark Berryman and Jim Wang — Fri, 15 May 2009 00:00:00 Z

In this show Federico and Matthew explore the question of "when too much automation is too much?" Based on their experiences in the ASP.NET QA team, they talk about the danger of automating for the sake of automation, the problems that it can create, and the value of context driven testing.

News

New name and name website: Coding QA.
.NET Micro Framework has been open sourced.

Automation for the sake of automation

What is the goal of automation? To find bugs during development vs. to find regressions after development.
What problems can arise from having too much automation? The tale of the ASP.NET QA "boat anchor".
Context matters when deciding how to ensure a feature does not regress.
What parameters can be used to decide the testing approach
- Automation cost.
- Time of delivery.
- Resources available.
- Risk of a regression.
- Customer impact if a regression is introduced.
- Quickness between when the regression is introduced and when it is discovered.
Real examples from the team
- Testing event handler generation for web controls in VS.
- Testing picker dialogs for web controls in VS.

Episode 5

Mark Berryman and Jim Wang — Fri, 08 May 2009 00:00:00 Z

In this episode Matthew interviews Carl Dacosta, lead tester for the MVC 1.0 project, and talks about what it was like to develop and test a product with so many releases.

MVC crossed the 100,000 downloads mark (currently at 140,000)
MVC released under MS-PL license
Interview with Carl Dacosta
- Some history on Carl and how he got into test.
- Role of Carl as a tester in the MVC crew.
- What was the development and testing process of MVC to enable frequent releases.
- Parts of the testing process that worked well for this kind of project, and parts that didn't work so well.
- How to handle quality of releases when external partners are starting to rely on the framework.
- What did Carl enjoy the most working on the MVC crew.
- Testing the "unit testing" experience for developers working on MVC.

Episode 4

Mark Berryman and Jim Wang — Fri, 24 Apr 2009 00:00:04 Z

In the fourth installment of Coding QA Federico and Matthew discuss "The Good," "The Bad," and "The Ugly" of the history of the ASP.NET QA Team. Discussion ranges from the dark ages, to the renaissance, to the industrial revolution, to the "21st and half century". Join them as they discuss the lessons learned and how the team has improved over time.

The team released the April update to the Lightweight Test Automation Framework.
"The Dark Ages" ASP.NET 1.0/2.0
- During this time the team was very focused on automating everything.
- (disadvantage) This approach is not good with a changing Spec.
- (disadvantage) Bugs were being found to late in the process.
- (advantage) Excellent automation coverage.
"The Renaissance" Atlas/Microsoft Ajax
- The division moved to a feature crew model for development and QA.
- (advantage) The team adopted a heavy unit testing practice. Increase in code coverage.
- (disadvantage) The QA team still worked has it had before. It would automate everything is less time.
"The Industrial Revolution" ASP.NET 3.5
- The QA team began to adopt the Feature crew model and started the process of agile testing with exploratory testing.
- (advantage) Bugs were being found at the beginning of the process.
- (disadvantage) The team still wrote a large amount of automation.
"21st and half century" ASP.NET 3.5 SP1/ASP.NET 4.0/vNext
- The team has really started moving away from automating everything and is spending more time investigating user scenarios.
- The team has formed its own unique blend of agile, scrum, and any other buzz word project management theory.
- (advantage) Creating and automating sample applications provides feedback for real world customer scenarios.
- (disadvantage) The team is young and is still learning the best practices.
- (Struggle) Costing has become less of a clear cut process.
- (Struggle) The is still a barrier between the QA and Dev teams.
- (Struggle) No one wants the blame when a bug is found.

Links from the show :

Episode 3

Mark Berryman and Jim Wang — Fri, 24 Apr 2009 00:00:03 Z

In the third installment of Coding QA Federico and Matthew interview Mark Berryman, a test lead on the ASP.NET QA Team. Join them as they learn the ins and outs of test management through Markâ€™s eyes.

Release of the Mobile Browser Database by Live Dublin.
April refresh of the Lightweight Test Automation Framework.
Interview with Mark Berryman.
- What is the difference between Test Manager and Test Lead.
- A little history of Mark and how he came to join the ASP.NET QA team.
- Responsibilities of a Test Lead / What Mark is working on.
- What Mark looks for in a tester on his team.
- Challenges Mark faced when he was an IC (individual Contributor).
- What Mark would change on items he has tested in the past.
- How Mark Manages risk on his team.
- Who is Marks favorite person in the whole world.

Links from the show :

Episode 2

Mark Berryman and Jim Wang — Fri, 24 Apr 2009 00:00:02 Z

In the second installment of Coding QA Federico and Matthew discuss MIX 09 announcements and the ins and outs of supporting various browser versions on the ASP.NET QA team.

MIX 09 Conference announcements
- MVC RTM 1.0
- Silverlight 3.0
- Web Application Installer
- Web Application Gallery
What browsers the team tests on and how they decided which browsers to test.
What it means when the team says they support a browser.
How the team manages the risk of not testing on every possible browser.
The timeline for testing features on multiple browsers.
The infrastructure the team has to test on multiple browsers.
How the team keeps up with the fast growing browser market, such as the release of four major browsers in just a few months.

Links from the show :

Episode 1

Mark Berryman and Jim Wang — Fri, 24 Apr 2009 00:00:01 Z

In the first ever Coding QA Federico and Matthew discuss the motivation behind the community involvement from the QA team and the release of the QA team's Lightweight Test Automation Framework for ASP.NET and what it is.

Introductions to the podcasters, Federico and Matthew.
What is it that the ASP.NET QA team does.
The recent release of ASP.NET MVC RC2.
The recent release of the source code to the Lightweight Test Automation Framework.
The purpose of the release of the framework to codeplex.com and where it fits in the market.
Why the QA team chose to write their own automation framework.

Links from the show :

Disclaimer : This is our first try at creating, editing, and publishing a podcast. Please bear with us as we iron out all the wrinkles in the process. Your feedback is greatly appreciated.