blogs

AI vs CI: Nobody Told the Pipeline

2026-08-17T07:00:00Z

CI is so commonplace these days, we don’t often think about why it’s there. And that reasoning really matters these days, because of how we use it.

Especially when you’re writing AI agents and expect the CI system to run a regression test suite, and sound the alarm when something goes wrong.

Here’s a funny story you may not know. CI – Continuous Integration – was not originally a tool category. It’s a process – it’s how you continuously integrate pieces of code together.

Anyway, why do you need this process anyway? Because you’re scared. You’re terrified that integration broke something.

So what do you do? You look for approval. An automation system that runs all the tests all the time is our approval of choice.

That’s how CI became a tool category. Because in its heart, it’s a simple automation pipeline.

So, that automation’s holy grail is speed, right?

Wrong.

The mission was to kill “Works on my machine”

Remember when we wrote software in a cave, and we had a sticker on our computers – “Works on my machine. Don’t run it on another cave”?

Ah, the good old days.

Because the CI automation #1 OG mission was to kill “Works on my machine”. How? By creating a repeatable process. Independent of all the weird installs on my machine. Multiple versions of libraries. And different configurations. And admin hacks.

The CI builds, packages, runs and tests the software every time the same way. That way, if a test turns red, you know something bad happened, because it was green until now. And why your stomach turns, when it flakes between red and green – that’s the feeling of lost repeatability.

Yes, repeatability brings bliss. Well, it did, before AI.

Then AI showed up

If your app does not touch AI (although building with code agents, can sometimes count as “touch”), you’ll continue to feel that bliss.

But if you’re developing AI-based features, or agents, or vibe-coding, you’re in for a new experience every time you push your code (or prompts) into CI.

Things are not repeatable anymore. Models change, sometimes without warning. And always under your feet. And when a model changes, does it trigger a run?

You wish! Your alert system is malfunctioning.

And the worst part is not a regression. Although if it’s a bug in a prompt, you may not be able to fix it.

No, the worst part is that Green today doesn’t tell you what kind of Green it is. Things are no longer Working or not, they are Work-ish. But even that’s not the same work-ish every day.

So we’ve got fewer triggers, results we can’t read, and on top of it – every run means something else.

What’s the solution?

Repeatability gives us confidence. CI is the process, and tools, that give us the approval we seek, and the confidence we have comes from that repeatability.

We can’t rely on the trigger anymore, but we can initiate our own runs. And since one run is not enough, we need to run more and look at the trends. Catch drift before it ships.

What happens to quality when AI meets CI? We need to get back to the reasoning, and change how we manage quality.

I write about this stuff every week. What AI does to the things we built to keep quality steady, and what to do about it.

Subscribe here.

The post AI vs CI: Nobody Told the Pipeline first appeared on TestinGil.

Implementing Page Objects in Playwright

2026-08-16T22:04:38Z

Picture a login screen, such as The-Internet / Login.

On this LoginPage, there is:

a heading: Login Page
a user name textbox with the label, "Username"
a password textbox with the label, "Password"
a login button, with the role of a button, and the name of "Login"
a flash message that appears if you enter invalid credentials such as "NotAUser" and "NotAPassword".

If you successfully log in with "tomsmith" and "SuperSecretPassword!, there is a SecureArea:

a heading: "Secure Area"
a flash message "You logged into a secure area!"
a Logout button.

Sure, you could interact with each web element in your test... but what if the username text box locator changes? You would have to update multiple tests every time the element changed.

... Instead, you could place it in a Page Object, something that Playwright handles well!

"A page object represents a part of your web application. An e-commerce web application might have a home page, a listings page and a checkout page. Each of them can be represented by page object models.

"Page objects simplify authoring by creating a higher-level API which suits your application and simplify maintenance by capturing element selectors in one place and create reusable code to avoid repetition".

Using Playwright's Built-In Test Runner? Or Something Else?

You may have noticed in https://playwright.dev/docs/pom that there are two different styles of page objects. One for "Test". One for "Library".

Test: If you are writing actual Playwright test suites, and Playwright's built in test runner, use the Test section as a guide when creating page objects.
Library: If you are integrating Playwright into an existing test framework such as Jest or Cucumber and just want browser automation, instead of having pre-built page fixtures, etc, you can use this format.

Setting Up Page Objects

Let's review a page object I created for the main login page on my GitLab account for the bun-create-typescript project, tests/pages/LoginPage.ts.

The first thing we do is import the Locator method and Page fixture from the Playwright Test library.

import { type Locator, type Page } from "@playwright/test";

According to Playwright.dev / Class-Fixtures, "Playwright Test is based on the concept of the test fixtures. Test fixtures are used to establish environment for each test, giving the test everything it needs and nothing else.

"Playwright Test looks at each test declaration, analyses the set of fixtures the test needs and prepares those fixtures specifically for the test. Values prepared by the fixtures are merged into a single object that is available to the test, hooks, annotations and other fixtures as a first parameter [...]

"Playwright Test will set up the page fixture before running the test, and tear it down after the test has finished. page fixture provides a Page object that is available to the test".

The Page and Locator Class

According to Playwright.dev / Page, "Page provides methods to interact with a single tab in a Browser, or an extension background page in Chromium. One Browser instance might have multiple Page instances.". These pages are isolated from each other, created for each test. The page fixtures have their own setup and teardown methods, such as opening up a new browser before a test starts, and closing a browser when a test ends.

According to Playwright.dev / Locators, "Locators are the central piece of Playwright's auto-waiting and retry-ability. In a nutshell, locators represent a way to find element(s) on the page at any moment". The built-in Playwright locators are:

page.getByRole() to locate by explicit and implicit accessibility attributes.
page.getByText() to locate by text content.
page.getByLabel() to locate a form control by associated label's text.
page.getByPlaceholder() to locate an input by placeholder.
page.getByAltText() to locate an element, usually image, by its text alternative.
page.getByTitle() to locate an element by its title attribute.
page.getByTestId() to locate an element based on its data-testid attribute (other attributes can be configured).

Define the Page Object Class

Now that we have imported the Page and Locator class from Playwright, we can define the LoginPage page object class.

export class LoginPage {
  readonly page: Page;
  readonly usernameInput: Locator;
  readonly passwordInput: Locator;
  readonly loginButton: Locator;
  readonly alertMessage: Locator;
  readonly heading: Locator;

Here, we are declaring LoginPage as a Page class, and creating Locator objects that will find and point to the username and password textboxes, the login button, the alertMessage, and the heading.

Initialize the Page Object Class

Now that we have defined the page object, we can initialize a constructor for the page, setting up reusable locators for all the web elements we defined.

  constructor(page: Page) {
    this.page = page;
    this.usernameInput = page.getByLabel("Username");
    this.passwordInput = page.getByLabel("Password");
    this.loginButton = page.getByRole("button", { name: "Login" });
    this.alertMessage = page.locator("#flash");
    this.heading = page.getByRole("heading", { name: "Login Page" });
  }

The textboxes are labeled "Username" and "Password". If we know that these labels are not expected to change, we can use the labels as a locator strategy.
The login button, we can use as a locator strategy its role "button" and the name "Login".
The alert message is tricky. Here, we can grab the id, "#flash", in the locator.
The heading, we can also get by role of "heading", and name of "Login Page".

Navigate to the Login Page

Let's embed in this Login Page Object a way to navigate to the Login page.

  async navigate(): {
    await this.page.goto("/login");
  }

Here, we have defined a method in the LoginPage page object our tests can use.

In our playwright.config.ts (see code in GitLab) we have already declared our baseURL to be https://the-internet.herokuapp.com, which will display when a new browser window is opened when a new test starts.

When in our test we instantiate a new LoginPage method, and call the navigate method, it will goto "login".

async has been placed before the function definition, since the browser will not be happening instantly. Actions will be happening asynchronously, all at the same time. It returns a "Promise" object.
await can be used inside an aync function. It tells the program to pause execution until the slow action, such as loading a webpage is finished.

What is a Promise? "A Promise is a proxy for a value not necessarily known when the promise is created. It allows you to associate handlers with an asynchronous action's eventual success value or failure reason. This lets asynchronous methods return values like synchronous methods: instead of immediately returning the final value, the asynchronous method returns a promise to supply the value at some point in the future". - Developer.Mozilla.com / JavaScript / Promise.

What is Asynchronous Programming in JavaScript? "JavaScript, especially when running in a web browser environment, has a unique challenge. It's single-threaded, meaning it has one line of execution at a time. If any task (fetching data, responding to a button click, etc.) takes a long time, it would freeze the entire webpage!

"Asynchronous programming is the pattern to tackle this. It enables us to start a potentially long-running operation and tell JavaScript, 'Go about your business, I'll let you know when this is ready.' This prevents the webpage from locking up.

"[...] JavaScript async await lets us write asynchronous code that resembles the style of straightforward, step-by-step (synchronous) code. This often makes the logic much easier to follow.- Upgrad.com / tutorials / JavaScript aync-await.

Creating a LoginAs Method

Sure, we could fill in the username, fill in the password, and click the login button, but what if that login procedure changes? We would have to hunt down every time we logged into the app, and make the changes in the test.

There is a software design principle called DRY: Don't Repeat Yourself, first coined by Andy Hunt and Dave Thomas in their book "The Pragmatic Programmer: From Journeyman to Master (1999)" (See Wikipedia, and the interview with both authors on YouTube at GoTo Conference 2020).

Instead, we could create a new async method called "loginAs" where we pass in the username and password.

  async loginAs(username: string, password: string): {
    await this.usernameInput.fill(username);
    await this.passwordInput.fill(password);
    await this.loginButton.click();
  }

Both "fill" and "click" are Playwright actions ( See docs ).

Text input can use locator.fill.
Checkboxes and radio buttons can .check them off, then expect them .toBeChecked.
Multiple radio buttons can be .selectOptions by value or label.
Mouse clicks can be performed with a mouse click, a double mouse click, a right click, a shift click, or a control click.
You can press keys such as .press('Enter'), or .press('Control+ArrowRight').
You can also press keys such as Backquote, Minus, Equal, Backslash, Backspace, Tab, Delete, Escape, ArrowDown, End, Enter, Home, Insert, PageDown, PageUp, ArrowRight, ArrowUp, F1 - F12, Digit0 - Digit9, KeyA - KeyZ, etc
You can .dragAndDrop, and explicitly .scrollIntoViewIfNeeded().

We have the locator, such as this.usernameInput, that we have declared, so now we can perform an action on the locator.

Get Text From Web Elements

Need to get text from an element such as an alert message or a heading? Using a locator, you can retrieve the innerText property of the web element. According to Mozilla Org / HTMLElement / innerText, "As a getter, it approximates the text the user would get if they highlighted the contents of the element with the cursor and then copied it to the clipboard".

  async getAlertMessage() {
    return (await this.alertMessage.innerText()).trim();
  }

Once we get the text of the web element, there may be hidden whitespaces in the text. "The trim() method of String values removes whitespace from both ends of this string and returns a new string, without modifying the original string". - Mozilla.Org / String / Trim

  async getHeading() {
    return (await this.heading.innerText()).trim();
  }

After getting the text and trimming it, we return it to our test function.

We could have set up the page objects to expect the correct heading or alert message is correct, sending to the method what should be expected... but I try to avoid this whenever possible.

Personally, I prefer to keep the assertions, the things we are testing for, in the actual test and out of the page object.

Setting Up Test Data

Yes, with just one set of valid credentials, separating them into a JSON file is overkill... but what if there were hundreds of valid test users? Here we are placing them into a JSON (JavaScript Notation) file called credentials with two top level objects... valid, and invalid.

In them, we have two key-value pairs in each object, username and password.

tests/ data/ credentials.json

{
  "valid": {
    "username": "tomsmith",
    "password": "SuperSecretPassword!"
  },
  "invalid": {
    "username": "NotAUser",
    "password": "NotAPassword"
  }
}

Need to add or change usernames and passwords?

No need to hunt through test data. Change just what is in the credentials file, and everything else will be reset. ( See this code in GitLab bun-create-playwright. )

Setting Up The Tests

Tests are set up in the Arrange / Act / Assert fashion, the pattern first mentioned by software developer Bill Wake in 2001 regarding the Extreme Programming movement (See XP123 / 3a- Arrange, Act. Assert ).

Arrange: Get the test data, open the browsers
Act: Navigate to the page and login
Assert: Do we have the proper heading ("Secure Area"), url, and alert message that we have logged into a secure area?

Finally, it's time to write our first test suite! Note: This code is in the GitLab project bun-create-playwright, under tests/login.spec.ts.

import { test, expect } from "@playwright/test";
import { LoginPage } from "./pages/LoginPage";
import { SecurePage } from "./pages/SecureAreaPage";
import credentials from "./data/credentials.json" with { type: "json" };
import messages from "./data/messages.json" with { type: "json" };

We are now importing files into our test and setting up variables to store the information such as:

The "test" and "expect" methods from Playwright.
The LoginPage and SecureAreaPage page objects we have set up.
The credentials stored in a JSON file that we have gone over.
A separate json file that stores all messages such as "You logged into a secure area!"

Set Up Constants for the Test Suite

test.describe("Login", () => {
  const validUser = credentials.valid.username;
  const validPassword = credentials.valid.password;
  const invalidUser = credentials.invalid.username;
  const invalidPassword = credentials.invalid.password;

Here, we are setting up a describe block, one of the methods in Playwright Test, declaring a group of tests, giving the test the title of "Login".

Then, to make the test more readable, instead of passing a three part credentials call, I am storing them in appropriately named variables.

These variables, validUser, validPassword, etc, we are declaring as type "const", also known as constants. If we accidentally change these values when the test is running, an error will be thrown. (Read more at TypeScript.org / Handbook / Variable Declarations ).

These constants, since they are set up in the describe block, will be set for ALL of the tests in this test suite.

Set Up the Individual Test

Let's set up a test that we can actually log in, and get to the secure area page, and call it "successful login with valid credentials".

  test("successful login with valid credentials", {
    tag: "@smoke",
  }, async ({ page }) => {
    const loginPage = new LoginPage(page);
    const securePage = new SecurePage(page);

    await loginPage.navigate();
    await loginPage.loginAs(validUser, validPassword);

We instantiate a new instance of the LoginPage, calling it a constant "loginPage", and do the same for the Secure Area.

Now that we declared new LoginPage, we can use the methods we have set up in our page objects, such as loginPage.navigate, and loginPage.loginAs, where we feed in the validUser and validPassword.

How to Assert Our Test is Working

We have done the arranging. We have done the acting. Now, it is time for the assertions.

Playwright.dev / Assertions mention, "Playwright includes test assertions in the form of expect function. To make an assertion, call expect(value) and choose a matcher that reflects the expectation. There are many generic matchers like toEqual, toContain, toBeTruthy that can be used to assert any conditions.

"Playwright also includes web-specific async matchers that will wait until the expected condition is met.

"By default, the timeout for assertions is set to 5 seconds. Learn more about various timeouts".

You can assert that an element is attached, a checkbox is checked, an element is disabled, editable, am element is empty, enabled, focused, visible, contains text or has an exact piece of text, contains a CSS class, has an accessible description or name.

You can also check if it has a specific ARIA role or matches an Aria snapshot, has a certain amount of child elements, has a certain id, css, an option selected, a value.

With pages, you can expect it to have a certain screenshot (for visual testing), has a title, url, or an OK status.

All of these elements will be retried if they fail the first time.

You can also have some test conditions where you do not auto retry, such as expecting a value to be something, close to something, the value is falsy (false, 0, or null). You can expect the value to be greaterThan, greater than or equal, less than a certain value. You can expect it to be NaN (not a number), null, or truthy (not false, not 0, and not null).

await expect(page).toHaveURL(/\/secure/);
    expect(await securePage.getHeading()).toBe("Secure Area");
    expect(await securePage.getAlertMessage()).toContain(messages.loginSuccessMessage);
  });

Now, we are checking that:

the heading we get from the SecurePage page object has the exact value "Secure Area".
the alert message matches the messages we have in the messages file...

And that's how you set up Page Objects and Tests using Playwright!

In the future, we will go over how to set up CI / CD pipelines with GitLab, but I need a break!

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Whew! That was a lot!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Test coverage does not tell you whether a change is safe enough

2026-08-16T18:07:45Z

Imagine two mobile app teams. They’ve both made changes to their mobile apps.

Team 1 is rolling out to 1% of its users and has telemetry built in. The change sits behind a feature flag and can be disabled instantly.

Team 2 has a solid testing strategy, with testers feeding back their experience. They plan to deploy to all their users, and if there is an issue, they would push an emergency patch.

Both teams have the same amount of automated test coverage, yet the conditions around the change are very different. Test coverage reduces uncertainty, but it cannot tell you on its own whether a change is safe enough to release.

Things like monitoring, observability, staged roll-outs, feature flags and rollback systems help manage uncertainty by giving you ways to detect, minimise and mitigate the impact if something slips through.

Safe change comes from the system of feedback and controls around the change, not from test suites in isolation.

So starting with what needs testing is probably too early. What we need to understand first is: What has changed, and what uncertainty does that change introduce?

This gives teams five areas to explore when thinking about whether a change is safe enough:

Understanding - Do we understand what is changing, why, and what it could affect?

Consequence - How bad would being wrong actually be?

Detectability - How quickly and reliably would we know something had gone wrong?

Containment - How much of the system/users could be affected if we’re wrong?

Reversibility - How quickly and safely could we undo, disable or mitigate the change?

These are not independent questions, and we’ll come back to how they interact later. But it helps to look at each one in turn.

Understanding

Understanding what has changed and why is key, as without it you can’t understand any of the other factors.

For example, are you adding, changing or removing behaviour? How might that affect existing behaviours, dependencies and assumptions?

It’s also about understanding the environment in which the change will occur, such as affected users or operational assumptions.

Understanding isn’t about eliminating uncertainty. It’s about knowing enough to make good decisions about the other four factors.

Consequence

Consequence is about how bad it would be if the change went wrong. If the consequence is minimal, for example, correcting a typo and nothing else, you might conclude that making the change is fine. But let’s say it’s a change to a payment flow. That could affect someone financially.

More testing might reduce our uncertainty, but it doesn’t necessarily reduce the consequence if we are wrong.

So you may also want a smaller rollout, stronger monitoring, and better reversibility to minimise and mitigate any potential impact.

Detectability

For the consequences you’ve identified, how would you know if they were becoming a reality?

This is where you look at your team’s ability to monitor and observe your system. Customer support, business metrics, app telemetry, app-store feedback, synthetic checks, and other operational data can all act as signals. You need signals that help you notice and understand unexpected behaviour. But speed is also crucial here, as detecting issues within minutes rather than days can dramatically reduce how long users are exposed to them.

Containment

If consequence is about the impact of something going wrong, then containment is how much exposure you allow.

For instance, imagine a failure in a new sign-up flow that affects your product’s ability to bring on new users. If it affects only 0.5% of new subscribers for a week, that might be more tolerable than if it affected 100% of new users for a week.

This is where you’d be thinking about how you roll out changes to users.

Reversibility

Reversibility is as it sounds: if, for whatever reason, a change shouldn’t remain in production, how easily can we reverse that change?

But rollbacks are not the only way to reverse a change. Disabling a feature, switching traffic, restoring data, pushing a patch and falling back to known behaviour are all ways to reverse a change.

A good approach to reversibility is to prefer the option that introduces the least amount of change, since a complex recovery process can introduce additional uncertainty to an issue.

Interestingly, for some teams, their reversibility strategy can also be their default containment approach. So instead of trying to contain an issue, they reverse it as soon as they detect it. Because of this, approaches to containment and reversibility can be conflated.

So, what makes a change safe enough?

These five factors are not gates to pass or a step-by-step decision framework to follow. They help you understand what evidence teams need before deciding a change is safe enough to release.

A change with high consequence, but poor detectability and weak reversibility, should make the team cautious. But a change with high uncertainty, fast detectability, strong containment and near-instant reversibility may be fine to gradually roll out.

The five factors influence one another, but they do not cancel each other out. Loss of user data is still loss of user data even if it affects less than 1% of users. Strength in one factor can change how much uncertainty we’re willing to accept elsewhere, but it doesn’t make the other factors irrelevant.

And even then, confidence in an individual change is not the same as confidence in the whole product. Changes interact, so we still need feedback about how the wider system behaves.

Taken together, this is why test coverage does not tell you whether a change is safe enough. It is one source of information within a wider set of system feedback and controls around the change.

Whether a change is safe enough is a judgement we make from the evidence and controls around it.

If this helped you think differently about what makes a change safe enough, subscribe to receive future Quality Engineering posts on testing, uncertainty, risk and building quality in.

Subscribe now

The Thinking Toolkit of a Software Tester

2026-08-16T08:44:46Z

Yesterday (August 15, 2026) , I had the opportunity to speak at QA Connect 2026.
My topic was:
The Thinking Toolkit of a Software Tester.

The idea for this talk came from something I have noticed throughout my testing career.
When testers face uncertainty, the difficult part is often not operating the tool.
It is deciding how to approach the problem.

You open an unfamiliar product.
Where do you start?

You observe something strange.
How do you decide whether it is actually a problem?

A developer changes a small piece of code just before release.
What deserves regression attention?

These aren’t Playwright questions.
They aren’t Postman questions.
They aren’t SQL questions.
They are testing questions.
And there isn’t always one correct answer.
That is where thinking tools can help.

During my talk, I explored three heuristics that have influenced how I approach different testing problems:
SFDIPOT, FEW HICCUPPS, and RCRCRC.

But the session wasn’t really about learning three acronyms.
It was about something more useful:
How can we give ourselves better ways to think when the answer isn’t obvious?

For those who attended the session, this article goes through the ideas again at a slower pace.
For those who couldn’t attend, here is what we explored.

What Is a Heuristic Anyway?

Before getting into SFDIPOT or FEW HICCUPPS, there is an important distinction to make.
A heuristic is not a rule.
It isn’t a procedure.
It isn’t a guarantee.
And it certainly isn’t another checklist we must complete.

I think of a heuristic as a thinking aid.
It gives us prompts when we’re dealing with uncertainty.
That matters because testing is full of uncertainty.

We rarely have perfect requirements.
We rarely have unlimited time.
We rarely understand the entire system.
We rarely know every possible consequence of a change.

So experienced testers develop ways of generating useful questions.
Some come from experience.
Some come from mistakes.
Some come from conversations.
And some come from heuristics developed and shared by other testers.
The important part is not remembering the acronym.
The important part is whether it helps you see something you weren’t seeing before.

❶I Have a Product. Where Do I Start?

Imagine this.
Tomorrow you join a new company.
The product is WhatsApp.
Your manager tells you:
“Spend the next hour testing it.”
That’s all.
No test cases.
No detailed documentation.
No predefined scenarios.
Where would you begin?

This is where SFDIPOT, created by James Bach as part of the Heuristic Test Strategy Model, can be useful.
SFDIPOT gives us seven perspectives:
Structure, Functions, Data, Interfaces, Platform, Operations, Time.

The mistake would be to treat those as seven sections that must be completed.
Instead, think of them as seven places from which we can look at the same product.

Now look at what happened.
SFDIPOT didn’t tell us: “Execute these 50 test cases.”
It changed the questions we were capable of asking.
That’s the value.
When I’m stuck, sometimes I don’t need another testing tool. I need another perspective.

❷Something Looks Wrong. Compared to What?

Now suppose our exploration reveals something strange.
Maybe an edited message loses its reactions.
Maybe an icon suddenly behaves differently.
Maybe a workflow feels unnecessarily complicated.
We say:
“This looks wrong.”
But why?
This is where another question becomes extremely useful:
Compared to what?

That question is at the heart of FEW HICCUPPS, a heuristic developed by James Bach and Michael Bolton.
It reminds us that when we evaluate software, we’re always comparing what we observe against some source of expectation.
Those sources are often called test oracles.
Requirements can be an oracle.
But they are not the only oracle.

Imagine WhatsApp Introduces Edit Message
The requirement says: “Users should be able to edit a sent message.”
You send a message. You edit it. The text changes.
Requirement satisfied? Maybe.
Testing finished? No.
Suppose editing a message removes all existing reactions. Nothing in the requirement mentioned reactions. Is that still worth investigating? Of course.
Why? Because we have other sources of expectations.

❸Something Changed. What Should I Regress?

Now we reach another familiar testing problem.
It’s Thursday afternoon.
Release is Friday morning.
A developer tells you: “I fixed the login issue.”
Then comes the sentence every tester knows:
“Can you quickly do regression?”

Okay.
What exactly should we regress?
Run everything?
Only login? Authentication? Sessions? Payments? Profile? Notifications?
Different devices? Different browsers?

Teams often default to one of two approaches:
Run everything.
Or: Test only what changed.
Neither is automatically correct.
A better question is: Where is this change most likely to create meaningful risk?

This is where Karen N. Johnson’s RCRCRC heuristic can help.
It gives us six perspectives:

Look at how the conversation changed.
We stopped asking: “How many regression tests should we execute?”
And started asking: “Where did this change create uncertainty?”
That is a much more useful way to think about regression.

Three Problems. Three Thinking Aids.

This was the connection I wanted to make during the talk.

SFDIPOT helps when I’m asking:
Where could I explore?
FEW HICCUPPS helps when I’m asking:
Compared to what should I evaluate what I’m seeing?
RCRCRC helps when I’m asking:
Where should I focus after change?

They solve different thinking problems.
But they have something important in common.
None of them executes a test.
None of them finds a bug.
None of them decides whether something matters.
They help us do those things.
And that distinction matters.

The Heuristic Is Not the Tester

There is a danger in articles like this.
Someone could bookmark it.
Memorize every acronym.
Create three checklists.
Add them to a testing template.
And completely miss the point.

Knowing SFDIPOT doesn’t make someone a good tester.
Neither does knowing FEW HICCUPPS.
Neither does knowing RCRCRC.

Because a heuristic cannot observe the product for you.
It cannot recognize surprise.
It cannot understand your stakeholders.
It cannot decide which risk deserves today’s limited testing time.
It cannot know when one of its own prompts is irrelevant.
That’s judgment.

And judgment develops through experience, observation, mistakes, conversations, study, reflection, and deliberate practice.
A heuristic should stimulate judgment, not replace it.

What Sits Underneath All Three?

If we remove the acronyms, what remains?
A few habits that I think matter far more.

And Yes, AI Belongs in This Conversation

AI can generate test ideas.
It can create test data.
Write automation.
Analyze logs.
Explain code.
Summarize requirements.
Suggest edge cases.
All useful.

But imagine AI gives you 150 test scenarios.
Now what?

Which ones matter?
Which are duplicates dressed differently?
Which assumptions did the model make?
What important context did you fail to provide?
Which risk matters most to your customers?
Which apparently intelligent suggestion is irrelevant to this release?
And what question didn’t you think to ask AI in the first place?

Generating more possibilities doesn’t remove the need for judgment.
Sometimes it increases it.
As tools become more capable, testers need to become better at deciding what information deserves attention.

Try This in Your Next Testing Session

You don’t need to memorize any acronym from this article. Instead, try three questions.

Don’t worry about completing every letter.
Use whatever helps the investigation.
Ignore whatever doesn’t.
The heuristic serves the tester. The tester doesn’t serve the heuristic.

Final Thoughts

When I prepared The Thinking Toolkit of a Software Tester, I wasn’t trying to teach people three acronyms.
I wanted to make something else visible.

Testing often happens in situations where we don’t have complete information.
We don’t know every risk.
We don’t know every user behavior.
We don’t know every consequence of a change.
We don’t even always know what “correct” means until we start investigating.
That’s why thinking matters.

SFDIPOT can help us see a product from another perspective.
FEW HICCUPPS can help us question the basis of our judgments.
RCRCRC can help us reason about regression without blindly running everything.
But none of them is the testing.

The tester is still observing.
The tester is still questioning.
The tester is still building and revising models.
The tester is still evaluating evidence.
The tester is still making judgments under uncertainty.

The heuristics simply give that thinking somewhere to start.

So if you attended my QA Connect session and eventually forget what every letter in SFDIPOT stands for, I’m perfectly fine with that.
I would rather you remember the questions.

Where else could I look?
Compared to what?
What changed, and where did that create risk?
And perhaps the most important one: What don’t I understand yet?

Because tools will change.
Frameworks will change.
AI will change.
The products we test will change.

But uncertainty isn’t going anywhere.
And as long as we’re testing under uncertainty, our ability to observe, question, investigate, evaluate, and make informed judgments will matter.

That’s the thinking toolkit I wanted to talk about.

Not something you install.
Not something you complete.
Something you keep developing every time you test.

Lessons Learned from Fixing Flaky Tests with Claude

2026-08-15T15:50:33Z

We recently sped up our integration tests by a factor of ten by running many of them in parallel. There were several flaky tests in the new test suite. Claude was really useful in getting rid of almost all flakyness, but I made some mistakes along the way. Here is what I learnt.

Our system is very well covered by integration tests. We have a complete local system running, and all integration tests interact with the running system via FIX connections (traffic) and HTTP connections (administration). Previously, our 1360 integration tests ran sequentially, and took around 40 minutes to complete. A recent rewrite got the running time down to around 4 minutes instead. The gain comes from being able to run many tests in parallel.

To be able to run tests fast in parallel, the test suite relies on instruments that are defined once, then reused for many tests. This however makes the tests vulnerable to interference between tests. After the rewrite, there were typically a few tests each run that failed. If the test is run in isolation, it passes. So this is the definition of a flaky test: it passes in isolation, but may fail (due to timing issues) when run as part of a larger test suite.

Even though it is fairly easy to try the test in isolation after a failing run, it is much better to get rid of the flakiness once and for all.

1. False Start

I was not very familiar with the testing framework, but I thought Claude would be good at finding and fixing the flaky tests. Each failing test produced a stack trace from where it failed. Often, the stack trace would just show where a wait for a desired state timed out, so the underlying cause was not obvious. Claude excelled at finding the causes of flakiness. So I got carried away, and accepted and committed the proposed fixes without really understanding the rationales. Since the problems appeared randomly, it is not always easy to tell if the underlying problem had actually been fixed.

The flaky tests became noticeably fewer, but did not disappear completely. When I asked my colleague, who had ported many of the test cases, to review the changes, he questioned many of the commits. Several of them sounded quite plausible, but had no effect. I realized that I did not understand most of the fixes, but had just blindly accepted whatever Claude suggested.

2. Understanding

So I started over, and this time took the time to really understand each fix. I would ask Claude about all aspects of the fix I did not understand. Many times, Claude was correct, and I learnt more about how the tests and the framework work. But occasionally, I would get these types of responses when I questioned the solutions:

“You’re right to push on this, and I need to walk back what I said”

“Fair challenge — let me be honest about what I can and can’t demonstrate.”

“So I was overstating it earlier”

Eventually, through our joint effort, we would arrive at a solution. By not blindly accepting Claude’s suggestions, I learnt a lot more about the tests, and I avoided adding pointless code to the tests.

3. Claude is Very Capable

In many of the cases, Claude did a fantastic job of finding the cause of the flakiness. Getting stack traces analyzed is a breeze, as is crosschecking multiple log files, checking the time stamps for overlapping actions. Claude’s encyclopedic knowledge of Java and associated frameworks really helps. However, because Claude is so good in so many cases, it is easy to assume that all of its output is correct.

4. Faster Understanding

I have really come to appreciate how Claude can explain not only a proposed solution, but also more broadly, the existing code in general. It is now much faster for me to understand existing code. I first ask Claude to give me an overview how some part of the system works. Then I can drill down into specific areas by asking follow-up questions.

5. Unlocking New Uses

Before LLMs, it is doubtful if it would have been worth the effort to chase down all the causes of flakiness. It would simply have taken too much time, even if the end result is good to have. This probably applies to other uses as well. Something that was not worth doing before is now possible to do cheaply enough. This is great.

Conclusion

Claude was a great help in tracking down the various causes of flaky tests. However, not everything it suggests is correct. Because it does many things so well, it is very easy to just accept every change it suggests. Therefore, being able to understand if the proposed fix is valid or not is very valuable if you care about your system.

Currently, less than one run in ten has a flaky test failure. Much better than before, but still some work for Claude and me to do.

How to Use AI With Your Brain

2026-08-15T15:41:39Z

It’s been a while since I’ve written a post, mostly because I felt that between my three books and my blog posts, I had said everything I wanted to say about software testing. But things have changed! In the last year, AI has drastically changed the software development landscape. After working with various AI tools for a while, I now have a series of posts about AI queued up! In this post, we’ll take a look at the basics. When using AI, it’s important to continue using your brain. Here are some things to remember:

AI is not intelligent

In spite of its name, artificial intelligence is not actually intelligent. The AI agents we use are based on a Large Language Model (LLM), which is a database of trillions of words. When we ask an AI agent a question, it is answering based on the information in that LLM. It makes predictions based on training and previous interactions about what you are asking and what you might want next. Do not make the mistake of believing that the AI knows how to “think”. It knows how to fetch data, organize information, and make predictions.

AI can be wrong

An AI is only as good as the data it was trained upon, or the data it knows how to fetch. If that data is wrong, its answer will be wrong. To use a simple example, if there are 10 websites that say that Beethoven was born in 1775 and the AI reads from only those 10 websites, it will tell you that is Beethoven’s birth year. (It’s actually 1770.) When using AI, remember to verify its answers.

You get what you pay for with AI

Free versions of AI do a smaller amount of research than the paid versions to answer your questions, and they do less confirmation on their findings. The more computing power you pay for, the more sources the AI will search to have a greater level of certainty of the correct answer. When working with AI, remember to consider the level of service you are getting.

AI will lie to you

The legendary James Bach once did an exercise where he asked an AI to provide him with information about a page on his website. It confidently returned a response with great detail about the page. The page did not exist. This is called “hallucination”, and an AI will do this because it is programmed to give you an answer even if there isn’t one. It will find data to make guesses about what you are looking for. I’ve experienced AI hallucination when I’ve asked it to solve a UI automation problem and it has recommended that I use methods that don’t exist.

AI will agree with you

The opposite of the previous problem is that AI will agree with you whenever it can. It might stick to its opinion when you are obviously wrong, such as when you tell it that the sky is green, but if you ask it to do something in code that is not efficient or sensible, it might do it anyway because that’s what you requested. Keep this in mind when you are giving AI coding tasks, and periodically ask it if there is a better way to do what you are trying to accomplish.

AI will use too many words

AI has the tendency to provide wordy explanations for everything it does. This can quickly fill up your code and documentation with unnecessary text, which can make it hard for humans to read the code and interpret what is happening. And when AI generates documents, it tends to return institutional double-speak like “focusing engineering capacity on foundational platform investments that will unlock greater leverage and compound velocity over time” instead of using the simpler “prioritizing tech debt”. AI creations that are too wordy and obtuse are referred to as “workslop”, and workslop should be avoided so that software projects are clear, organized, and maintainable.

AI will do things you didn’t ask it to do

More than once, I have caught an AI agent I created doing something I didn’t ask it to do, such as create a Python script to make its work more efficient, or find a way to get an API key that I didn’t provide it with. Others have told me they have experienced this as well. This is not only annoying, it can also be dangerous, since the agent could get access to systems that it shouldn’t have access to. When working with AI, it’s important to set guardrails on what it is not allowed to do.

Use your brain when using AI

I am not saying to avoid AI. On the contrary, AI can be extremely useful and can save you massive amounts of time. But it’s important to be aware of AI’s tendencies, limitations, and dangers so you can use it wisely. In the coming weeks, I’ll be diving deeper into various AI applications with advice of how you can test it and use it.

How Playwright Frameworks get configured with playwright.config.ts

2026-08-15T02:30:20Z

When we installed bun, a new package manager owned by Anthropic, then ran "bun create playwright", a new automation framework was stood up, along with sample tests, and a Playwright configuration file. In this post, we will be examining the file generated: playwright.config.ts.

Personally, I find the pre-generated file very hard to scan... there are so many options and documentation in the comments, it is difficult for me to focus on the code. Let's examine just the code generated below. If you need to see the whole file, you can see it here: https://playwright.dev/docs/test-configuration

Playwright.dev / Configuration mentions, "Playwright has many options to configure how your tests are run. You can specify these options in the configuration file".

What is Configured By Default?


import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: 'html',
 
    // baseURL: 'http://localhost:3000',

    /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
    trace: 'on-first-retry',
  },

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },

    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },

    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },
});

I know what you are thinking ... What the heck is process.env.CI ? 2 : 0?

This is know as a ternary operator. It is a shorthand for: "If A then B, else C".
You can run tests two ways: Locally, on your local machine, or through a CI / CD Pipeline like Jenkins, GitHub Actions, or GitLab.
If you are running tests on CI / CD, the process.env.CI would automatically get set. Since A is true, the value of retries would be "2".
If you are running tests locally, the process.env.CI would NOT get set. Since A is false, the value "0" would be selected.
CI / CD pipelines would get two retries. Locally, it would get no retries and a failed test would remain failed.

Now that that is cleared up, let's examine the properties:

testDir: Points to the home of the tests.
fullyParallel: Allows you to set the tests to run all at once across many workers, or by one worker, tests lined up single file. By default, all tests are run all at once with multiple workers.
forbidOnly:

Developers debugging a test, will mark that test they are working on "test.only()". It's okay to have "test.only()" locally, but you want it to complain, loudly, if you have any test still marked "test.only()" in a CI/CD pipeline.
If you are running the test locally, the test run wouldn't pick up a process.env.CI. The CI/ CD variable would not be set, which means that the variable would be "undefined" which would resolves to false. forbidOnly would be false. No alert would be thrown if an "only" was seen, which is exactly what you wanted.
If you were running this in a CI/CD pipeline, you do not want test.only() to appear. Since process.env.CI would fill in a string, or set it to true, this equation would evaluate to "forbidOnly=true". An alert would be thrown if a test running in the CI/ CD pipeline saw an "only".

retries: Tests can be fragile when running on a CI/ CD platform. If it is CI/ CD, the process.env.CI would be set, by default retrying a failed test twice. If it is not set, because the test is running locally, we have the second default, which is "0", meaning failing tests would not be rerun.
workers: How many workers would you like to execute tests for you?

By default, if you are on a CI / CD pipeline running a test it will run two workers to execute a test. Playwright recommends setting the workers to "1".
Running a test locally? "Undefined" means that Playwright will detect your machines hardware and boot up a number of workers equal to half your logical CPU cores without freezing your operating system.
Note: GitHub Actions free tier give you virtual machines with limited resources. If you have just one worker tests will run one after another making logs and failures easier to trace.
Read Playwright.dev docs about running jobs in parallel, and dividing tests into shards.

reporter: How would you like your output? Basic "html", with a standalone webpage? A "list" printing a detailed line per test showing pass or fail? A "line" displaying execution progress on a single terminal line? A "dot" report where passed tests are dots and failures are "F"? A "json" report? An xml "junit" style? A "blob" of a binary file that can be later merged from multiple machines? Or "github" which creates annotations in GitHub Actions? See Playuwright.dev / Reporter, and a list of reporters on The Testing Academy.

By default, it has set up to run the Playwright version of Chrome, Firefox, and WebKit Safari.

But What About Dot Env?

If you looked in the pre-generated configuration file, there is code that looks like:

* Read environment variables from file.
* https://github.com/motdotla/dotenv
*/
// import dotenv from 'dotenv';
// import path from 'path';
// dotenv.config({ path: path.resolve(__dirname, '.env') });
Dotenv reads custom variables from a plain text file, injecting them into a Node.js application.

Sometimes, settings change depending where your code is running, such as on your local machine vs a production environment.

It is a good practice to set up one .env file per environment. .env for your local. .env.production. .env.staging, etc.

There are things such as database passwords, third party API keys, server port numbers.

With Dotenv you can create an .env file in the root folder of your project, and write configurations like:

PORT=3000
DATABASE_URL=mongodb://localhost:27817/myapp
STRIPE_API_KEY=

Make sure to add the .env file into .gitignore else you might accidently push these secrets into GitLab! ... Or you can encrypt it using https://github.com/dotenvx/dotenvx

What about the Mobile Viewports and Branded Browsers?

You also may have seen in the boilerplate playwright.config.ts file:

/* Test against mobile viewports. */
// {
// name: 'Mobile Chrome',
// use: { ...devices['Pixel 5'] },
// },
// {
// name: 'Mobile Safari',
// use: { ...devices['iPhone 12'] },
// },

/* Test against branded browsers. */
// {
// name: 'Microsoft Edge',
// use: { ...devices['Desktop Edge'], channel: 'msedge' },
// },
// {
// name: 'Google Chrome',
// use: { ...devices['Desktop Chrome'], channel: 'chrome' },
// },

If you want Playwright to simulate how your website looks and behaves on mobile devices, you can comment out the mobile viewport section. Mobile Chrome simulates a Google Pixel 5 using Chromium, and Mobile Safari simulates an iPhone 12 using Apple's WebKit engine. ( See the ThinkSys article, Cross Browser Testing with Playwright in 2026 )

Sometimes, emulations are not enough, and you want to actually test on Microsoft Edge or Google Chrome. With this setup, you can launch the actual browsers.

... The next post, we will dive into setting up actual test and page objects.

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Quality Management System has the same parts but how it is operating is changing

2026-08-14T19:31:29Z

I’m working on my thinking for a Quality Management System workshop I am doing in under 2 months. As part of this I can’t help but think about how to make it relevant to the changes AI is having to this and what questions might be sent my way. So this blog post is working through meandering thoughts. Figuring things out as I go.

The first question I had was, if I could go back in time today, what would I use AI for?

The example I can think of is defect trends chucking it a whole load of data and telling me what area of the product it comes from and the product quality criteria it relates to (functional/non functional). Then recommending top 5 areas to work on based on that information. I remember walking into a team that was very reactive, countless issues after every release and they hired me to solve that problem. I spent literally hours going through 12 months of bugs reports, categorising them and then building out test coverage to build a safety net. It worked. We had a substantial reduction.

What is the first thing that comes to mind for you when answering this question?

Proactive and Reactive Business Operations

From there I was thinking about different operational states —proactive and reactive — a company can be in both, but the ideal would be to be in a more proactive than reactive state.

Proactive — Planning ahead and preventing bugs.

Reactive — responding to issues and customer complaints.

Some companies, in my experience can lean towards one of these.

AI assisted versus AI driven

Then what I was thinking about are the different levels of AI adoption.

There’s two key spectrums that I can think of and then a journey to get from one to the other. AI assisted is that the human decides what to do, but the AI helps them do it faster. Then there is AI Autonomous where AI takes care of the testing but there is Governance and oversight (a bit like a test manager).

In-between that there is AI Trusted — finding evidence that tells us the outputs are as expected. Over time the goal would be to go from reviewing everything to review exceptions.

I don’t really know the percentage of where companies are right now. My guess would be lots of AI assisted, next would be AI Trusted and AI Autonomous — Are we even there yet?

A key question I have thought about is

Why should we let AI make testing decisions when we don’t trust it’s outputs?

Recently that’s the question I’ve been thinking through. I had this situation where I walked into a refinement session to talk about a piece of work and there were acceptance criteria, half were repetitive and had to be removed and there were a whole bunch of risks we hadn’t discussed. It feels in a way, right now, AI creates even more work for us. As well as that I feel in the past we had lots of conversations about how AI doesn’t produce very good tests. Now the question is, so why are they not good tests? What is the definition of good for testing? (Lots of back to basic pondering thoughts).

For AI to be fully autonomous, how do we get there? Could an experienced tester give all of it’s knowledge to AI. How would one even go about that? My thought to this would be to break it down and think about recurring issus. Poor requirements, lack of system testing, bad change management process and tackle them one by one. Ideally what problems your company are facing in a priority order.

Overall, planning-control-continuous improvement in a quality managment system is still the same, yet what we need to ponder on is how the operations of the system in tech is changing. Lots to think about and experiments to set up.

Quality Management System has the same parts but how it is operating is changing

2026-08-14T19:31:29Z

The first question I had was, if I could go back in time today, what would I use AI for?

What is the first thing that comes to mind for you when answering this question?

Proactive and Reactive Business Operations

From there I was thinking about different operational states —proactive and reactive — a company can be in both, but the ideal would be to be in a more proactive than reactive state.

Proactive — Planning ahead and preventing bugs.

Reactive — responding to issues and customer complaints.

Some companies, in my experience can lean towards one of these.

AI assisted versus AI driven

Then what I was thinking about are the different levels of AI adoption.

In-between that there is AI Trusted — finding evidence that tells us the outputs are as expected. Over time the goal would be to go from reviewing everything to review exceptions.

I don’t really know the percentage of where companies are right now. My guess would be lots of AI assisted, next would be AI Trusted and AI Autonomous — Are we even there yet?

A key question I have thought about is

Why should we let AI make testing decisions when we don’t trust it’s outputs?

Singing in the Dark

2026-08-14T05:00:00Z

I have just come back from presenting at CAST 2026 in Cocoa Beach, and something that’s been bugging me for a while finally clicked. Somewhere in every testing event I have been to in the last fifteen years, someone makes the case that testers are undervalued, and the case is usually made well. I saw […]

How to Get Started with Grok Bot

2026-08-14T00:00:00Z

Grok Bot just dropped. I downloaded it on a Mac the same day and recorded myself setting it up. Not a feature list. What I actually clicked, what worked, and which bots I would create.

AI Is Making Software Cheap to Build. Trust Is Still Expensive.

2026-08-13T04:31:03Z

When software creation becomes dramatically cheaper, the real bottleneck shifts from building systems to proving they deserve our trust.

Continue reading on Medium »

I Tested If Grok Bot Could Book My Flights

2026-08-13T00:00:00Z

I asked a Grok Bot to act as my travel agent and book flights. Close, but missing the last step. Here is the video of what worked and what didn't.

Workroom PlayTime 063: Spot the Difference

2026-08-12T19:39:59Z

This week's Workroom PlayTime is going to happen, but it is not going to be on Thursday (tomorrow / today depending on where you are). We'll try for Friday 14 August at 7:30pm UK time. If that doesn't suit you, and you want to find another, ping me and if we can find a mutual one, I'll send another invite to subscribers.

We'll run Spot the Difference (Puzzle 40).

We'll gather on Zoom. These exercises are for everyone, for free. All subscribers get to play together most weeks, paid subscribers get a guaranteed slot or rerun if we're full. If you get this in an email or can see the joining info section below, then you're a subscriber and have access to the Zoom etc. for the workshop. If you'd like to bring a friend, you can do that. I'll say yes until I've got too many.

Use this Savvycal link to book your spot and get reminders. Note to regulars: You should find it easier to use the Zoom link – it's in several places. Tell me if you like it or if it's a problem.

↓ signed-in subscribers will see joining info and links below ↓

Spot the Difference (Puzzle 40)

2026-08-12T19:30:45Z

Open Puzzle 40 and Puzzle 40b in tabs.

There is one substantial difference, and it's not obvious to the casual explorer. The exercise is to find it.

You will find it by exploring, modelling and comparing. You will find it by comparing sources. You will find it by asking me for hints. You will find it by asking me. You might aim your exploration by thinking what I could easily change in one place that leads to a non-obvious change in behaviour. You can probably think of more.

First person (to be sure of what it might be) gains kudos – and becomes a helper and hinter to the others, if they ask. Or maybe we'll all explore together, and so we'll find it together.

At the start of the exercise, we'll agree collective ground rules – how long we'll spend, how collaborative we'll be. We might also exchange individual handicaps – maybe you won't look at the source, or you'll only use specific tools.

See It! Days 20-24

2026-08-12T13:45:27Z

This post is part of a series showing my working through the 2025 Advent of System Seeing exercises set by Ruth Malan. For background see the introduction.

As we reach the final few days of exercises we're into reflection and consolidation of ideas.

Day 20

Task: (1) Look across the past 19 days and draw out concepts related to systems (2) Consider what else you'd like to add (3) Jot down some questions (4) What topic would you like to add, as the focus of another page on your System zine?

Components: parts, relationships, boundaries, effects, feedback, change over time, inputs and outputs, failure modes, context, constraints.

Tools: zoom in/out, follow a specific path, take actor perspectives, use multiple models, identify needs (and who they belong to), identify causes (and the causes of the causes), focus on the parts of the system that matter today, don't forget how we got here, identify constraints and assumptions.

Questions/Missing:

Seeking/exploiting existing perspectives
Comparing and contrasting models
Consolidating insights from different models
Maintaining (or not) models over time
Driving towards the most important information
When to use what type of model
Provide ways to understand the value of different kinds of model
More focus on collaboration
Where do feeling and instinct come into this?
Do more on changing the system
Why is this still an art rather than a science?
Graphical models for exploration vs for sharing information
We covered causal loops but there was nothing on stocks and flows?
No acknowledgement that we might be part of the system? (Our investigations might influence actors.)
Unintended/unexpected/unwanted consequences
Models are there to explore what matters to you. It's fine to model a subsystem. Just be aware that's what it is.
Judging the trustworthiness of a source

For the zine page, I chose exploration because investigating a system is exploration.

Day 21

Task: Create a zine page on your topic from day 21, capturing related ideas, concepts, using expressive visual analogies and sketches to make ideas vivid.

I wanted to get across the idea that you will need to explore the system you are looking at in order to understand it.

I wondered about using quotes like "the map is not the territory" but decided to try to keep it simple. There is no room for subtlety in this page: I would have liked to have talked about the constant change, the need to be clear about what you're trying to accomplish, focussing on the relevant pieces and so on.

Day 22

Task: Identify a challenge in the situation you have been exploring, and frame a goal related to that challenge where various people need to play a role. What would you do next?

This is a nice exercise, a different dimension to the ones we've covered in the earlier days. It's striking to me that it doesn't explicitly call out politics as a factor.

Day 23

Task: Shifting perspectives is one way to notice what we are missing or adding. Pick an important conversation you've recently had, and record what you said against what you were feeling at each stage.

There are, unsurprisingly, differences between the internal monologue and the external dialogue and, given the context, I think it can be instructive to take a systems view of that. Some initial thoughts on a difficult conversation between colleagues:

The participants are players in a shared system (actually, in shared systems)
They will be conscious that the productivity of the system as a whole depends on them having a smooth (enough) interaction
Stating bluntly what each of them feels might add friction to the system
... so social norms form part of the context
Social norms can vary significantly between team members (e.g. from different cultures)
Many factors might influence the interaction, e.g. office politics
... personal interest
... relative position in the company hierarchy
... historical interactions between these participants
... the subject matter, and relative levels of expertise, ownership, or interest in it
... responsibility for the outcome
... how busy the participants are
... how often they've had some version of this conversation
... how invested they are in the success of this project

And so on. For what it's worth, some of the most valuable training I ever had was in assertiveness, which collapses all of that stuff into a big blob, and focusses on being able to drive the conversation to the topics that are most important whatever the context. I wrote about it here.

Day 24

Task: Wrap up the Advent(ure) in System Seeing by pulling some of your work into a zine. It's neat to create a physical memento and it's your gift to yourself for completing the Advent(ure)!

As I had already made 8 zine pages I thought I'd finish off the exercises by making a cover. I've already said that this kind of work is exploration for me, so it didn't take long to decide to steal the cover of Explore It! (Sorry Elisabeth!)

And the full zine:

Image: Inspired by Explore It! by Elisabeth Hendrickson. Pragmatic Press

See It! Days 17-19

2026-08-12T13:43:27Z

This post is part of a series showing my working through the 2025 Advent of System Seeing exercises set by Ruth Malan. For background see the introduction.

This set of exercises is concerned with the location of a technical system within a social system. Spoiler: it only adds complexity.

Day 17

Task: (1) Considering the situation you’re exploring: identify systems and their interrelationships. How far out do we zoom? What heuristics do you use? (2) Pick a (socio)technical system to focus on. Write a brief description of its primary purpose.

I found the questions in the exercise the most interesting part of it this time.

How far out do we zoom? This is the perennial problem in my world; I am always trying to find ways to judge whether I have gone far enough in the dimensions I have identified as relevant and hoping that I identified more or less the correct set of dimensions. Getting the opinion of others really helps, as the early exercises in this series identified. What is the edge of a context?

What heuristics do you use? I have written a lot about heuristics for working.

For the second part of the exercise, the sociotechnical system I picked is a museum. My first attempt at its primary purpose was this:

The primary purpose of a museum is to exhibit something for public consumption.

It didn't survive even light stress testing: first, because there are private museums and, second, because by this yardstick a cinema would be a museum. I iterated some more, acknowledged that I'm not trying to write a definition of a museum, and ended up with this:

The primary purpose of a museum is to act as a memory and sense-making institution that curates artifacts to construct, preserve, and share knowledge about them and their context.

Feels clunky, but I'll live with it.

Day 18

Task: Identify (and draw) the network of capabilities that enable that system from day 17 to be viable as a system with that purpose.

This exercise again makes it clear that finding a useful granularity for the network depends on the way in which it will be used. For example, running a museum requires a lot of the same basic stuff that running any non-trivial organisation requires. Should I have inserted a node labelled "standard organisational infrastructure" and emphasised only what is special to a museum?

Similarly, needs often bottom out at some kind of resource: probably money. I included this in a couple of places but didn't want to flood the diagram with that. Can they be taken as read?

The task asks a couple of questions:

What capabilities are internal to the system? What external (to the system) capabilities does the system depend on? Interestingly, I found that I had stopped at a system boundary (where the internal meets the external) without thinking much about it. I added external dependencies to a section only after reviewing this question. What does that tell us? We bring explicit and tacit biases with us to each task.

If we broaden our view to more of the lifecycle of the system, what additional capabilities come into view? I didn't deliberately not think about system lifecycle so I feel like I've included some things that are relevant already, such as fundraising, HR, and financial or, for museums specifically, accessions and disposals.

If we back up to consider the problem that users of the system face (why they are using the system), and explore what capabilities they need, does that differ from taking the system purpose as a starting point? I think that it doesn't, although the prominence of some nodes in the network might be different. For example, as a museum director I care about a lot of things that a day visitor does not: I might sacrifice some short-term priorities to protect long-term survival for instance.

Day 19

Task: Identify an area (in your situation) where you would like to do something and want to explore what you'll encounter and what it will take.

I can't share the output from this exercise, but I can share some thoughts on how I like to think about this kind of problem.

I find that I go for a mindmap less often than I used to because I find the tooling (Miro in my case) too irritating and turning it into some other shape of model too tricky. What I tend to do now is use sticky notes which I group spatially, or with colour or some other attribute, and then shuffle them around on the whiteboard until I find a way to represent the space that helps me to think about the problem at hand.

This often results in some kind of classification of features of the space and compare options based on those features. Because sticky notes on a whiteboard don't constrain me at all, I can make informal, flexible tables that reflect levels of certainty, overlap, constraint and so on.

Next: Days 20-24
Image: Inspired by Explore It! by Elisabeth Hendrickson. Pragmatic Press

See It! Days 10-16

2026-08-12T13:41:30Z

This post is part of a series showing my working through the 2025 Advent of System Seeing exercises set by Ruth Malan. For background see the introduction.

This section of the Advent series is largely concerned with material I can't share in much detail, a practical exercise exploring a real-life scenario at work. I've done the exercises and given at least a little commentary on the value of them here.

Day 10

Task: 1. Draw an organization that you have been part of at least 3 different ways (2) Jot down reflections (3) Go "meta” and think about the views you drew.

As the task suggests, it's well known that org charts are, at best, an approximation of the way that work gets done in an organisation. The kind of diagram to draw depends on the perspective you want to obtain.

Many of my colleagues don't like Miro, but I love it because I can shuffle bits of model around until a view that helps me make sense of whatever I'm looking at emerges. Whiteboards are good for this too, but the ease with which I can copy-paste makes the discovery phase so much more convenient with an electronic tool. (See e.g. Going Underground where I did just that.)

Day 11

Task: Think of a situation you’d like to explore with a systems lens. Write a few paragraphs describing the situation.

I need to understand an existing system sufficiently to test the partial reimplementation of it as we migrate from one stack to another for our core product at work. I talked about this a little in Just Looking recently.

Day 12

Task: draw an Actor Map for the situation described on Day 11.

Actor maps are new to me. Making one reminded me of the Team Onion model for collaboration It's helpful for me to externalise the motivations of different parties when there are potentially politics at play. (And there are always politics at play.)

Day 13

Task: Jot down characteristics of the current state of things and what would happen if you did nothing to change it.

I didn't find this valuable for my situation. Other than hypothetically, there is no option where my team does nothing.

Day 14

Task: Explore how your situation came to be. Either focus your exploration on a textual narrative, or use a graphical history canvas.

For this exercise I chose to draw a timeline but, without significant research, I can't give a very precise order. I think that doesn't matter as much as understanding some significant design choices in this case but in general I find that the history can be an important part of the context.

I try not to judge historical decisions too harshly because we tend to see the result rather than the state of the system at the time. I mean, come on, we've all done something that in retrospect looks awful because it was the pragmatic choice at the time. Haven't we?

Day 15

Task: Create a Value Network Map for the situation you're working on.

I superimposed this onto the actor map I created on Day 12. I like the idea of trying to establish tangible and intangible value flows, but find myself sceptical that the business cares about intangible value. Much like the magical org chart from Day 10, it feels like management tends to abstract a crude model of value generation that discards anything which doesn't drive the bottom line directly.

Day 16

Task: Create another page for your zine, on the topic of constraints.

I have long held that constraints are our friends. I prefer the term to requirements because I think of constraints as more general. In particular I think of constraints as including negatives: "no solution that includes X, Y, or Z" might be one, ideally with a "because ...." chaser to help the team understand the reasoning behind the restriction and so be able to make better judgements when implementation choices arise later.

In Bottom-up or Top-down I think about three types of constraints, coming from above, the context, or below. For example:

management: use Jira for managing work items
market: use standard technologies to be able to integrate and compete
team: we want to build our product using hexagonal architecture

Next: Days 17-19
Image: Inspired by Explore It! by Elisabeth Hendrickson. Pragmatic Press

See It! Days 6-9

2026-08-12T13:37:39Z

This post is part of a series showing my working through the 2025 Advent of System Seeing exercises set by Ruth Malan. For background see the introduction.

In this set of exercises, the idea of feedback is introduced. Feedback is at the heart of systems thinking because we humans seem to be primed to view the world as a simple linear cause-leads-to-effect machine in which only the causes and effects that matter are those that (a) we desire or (b) are glaringly obvious.

Day 6

Task: Watch a video on causal loops and take sketchnotes.

I know and like this kind of diagram (Be a Quality Detector) but I have struggled to build them for any system that I've looked at. In particular, I find the formal labelling of positive and negative loops to be more of a dark art than a science. The good news is that informality has proved sufficient for me up to now but perhaps I'll learn to be more formal on day 7.

Day 7

Task: Practice using causal loop diagrams to understand interacting effects in a system, for example the effect of adding lanes to a highway.

My diagram captures a scenario where I have assumed that a road is already at or near capacity so that throughput (say, number of cars passing a specific point per hour) is lower than desired. I have several problems with it

1. The wording in the nodes is very important. Take (A) where I instinctively said adding a lane to our road would result in a "reduction in traffic elsewhere." This seems reasonable but makes it harder to decide whether we have a positive or negative relationship. Probably I should have labelled this node "amount of traffic elsewhere" and it'd be more obviously negative.

2. The idea of positive and negative relationships is problematic because they have both a mathematical and a value judgement interpretation. I suspect that what we're really annotating is direct and inverse relationships.

3. Temporality is hard to express. Both (A) and (B) are potential impacts of the action in this scenario, but likely at different times. How can I reflect that?

4. Some of the impacts are secondary, for example (C). How far should I go when thinking about them? In this artificial exercise I don't have a remit so I noted a few and stopped.

5. I have drawn a dashed line (D) to indicate the temptation to loop through the whole cycle again, once congestion returns to the system. I would like to somehow indicate that other choices are available at this point.

6. I don't know how to represent magnitude on these kinds of pictures. How should outcomes be balanced against on another?

Day 8

Task: Create another page for the zine from earlier. Focus on context: why do we consider (system) context? What are associated concepts? Is there an explanatory diagram that helps convey why context is important to consider?

I spent waaaaaay more time than I intended to on this one. I had in mind that there's an interesting parallel between the "start with the why" call of Simon Sinek's elegantly simple Golden Circle model and the idea that systems are "surrounded by the why": the context in which a system sits is the reason it does what it does.

I started with the basic idea of this parallel

Why: the context. The system is the shape it is because of it.
How: the actions performed by the system.
What: the outputs from the system.

and iterated through four or five sketches to get the one above. It's the first time in this series that the image you're seeing is not the first image I drew. Here's some of the along-the-way versions:

I'm not sure what I've done here satisfies the brief, to be honest. I don't think that by itself my diagram really motivates the primacy of context. It's probably less interesting to someone who hasn't come across Sinek's work too, but I think it is at least memorable and graphic.

Other things that I would have liked to included: showing that something designed in a linear way on the left can be part of a system on the right, or that context from analysing a system on the right can feed into the why on the left, but I didn't want to try to superimpose that additional layer onto the sketch because of the risk of obscuring the core detail that's already there.

On top of all of that, enamoured as I am with the idea I've drawn, I'm still not certain whether it holds up.

Day 9

Task: (1) Watch a short video about trees. Jot down notes (and sketches) as you watch it. (2) Draft a zine page on relationships.

I'd come across the so-called Wood Wide Web before, so the basic information in the video was not new to me. What to pull out into my sketchnotes and then into a more general zine page on relationships was tough to decide, and in the end I'm not sure whether I'm happy with what I did. Unlike on day 8, I am comfortable that what my page says is true, but I question whether it's helpful.

Next: Days 10-16
Image: Inspired by Explore It! by Elisabeth Hendrickson. Pragmatic Press

See It! Days 1-5

2026-08-12T13:33:57Z

This post is part of a series showing my working through the 2025 Advent of System Seeing exercises set by Ruth Malan. For background see the introduction.

The initial exercises introduce some core concepts and a couple of approaches to modelling, and encourage us to think about our work and the system we're looking at.

Day 1

Task: (1) Draw a Bubble Diagram of a bicycle (2) Sketch a bicycle from memory (3) Pick a key capability [and use] sketches and words to explore and convey how it works (4) Reflect on what you learned.

I'm comfortable doing what I'd call "factoring" which tries to find different angles or features or affordances of a thing that could be useful seams to follow when testing. I've used plenty of tools, such as the Six Thinking Hats or SFDIPOT to help with that, and represented the outcomes with a mindmap often.

Although my in-the-moment reflection says that I didn't find the bubble diagram helpful, I can see now that it's because I didn't sacrifice any of my 15-20 minutes to reading the (long) reference. Having looked a little closer, I see that it's a tool used in architecture and the spatial aspects (proximity and size) attempt to model features of the physical design, making it more like a map than a traditional mindmap, where the diagram's topography is typically not part of the model it holds.

It makes me think of the building blocks of models and the hybrid models I've made (e.g. Going Underground, The Testing Kraftwerk, or Granularity Familiarity) and I think it'll be a useful tool to have in my back pocket in future.

Day 2

Task: Draw a diagram to explore the world around bicycles. Zoom out, so that "bicycle" is just one bubble in the centre of the diagram.

Where are the edges of the context? is a question I have asked myself many times. It's fascinating to see how other participants broke the space around the bike down, seeing concepts that I didn't and leaving out things that I included. Sebastian Hans suggested something important: "some contexts have more depth than others, maybe because I have more experience riding a bike than building one."

That's the thing: our biases are hugely influential in this kind of exercise and it takes conscious effort to leave them at the door. Having multiple people on a task can help either synchronously or by sharing the model with others for comment. Be careful though: I have seen conversations break down because two people have formed mental models that don't fit together. Externalising the models can help massively but I find many people reluctant to do it.

Day 3

Task: Use a Futures Wheel to explore a change that is impacting cycling or the bicycle industry, in terms of direct and indirect effects.

Another new diagram format for me, the Futures Wheel. Again I didn't spend any of my time reading about it in advance but this time the exercise contained enough clues to get me going. There's a flavour of of Weinberg's diagram of effects or causal loop diagrams but I like how this visualises things like ripples in a pond, moving outwards from a centre.

One of the challenges in designing anything that will exist in the world is anticipating the impacts, particularly given that the world is a complex system and the impacts are indirect and subject to feedback mechanisms. I like this diagram as a potential tool for exposing some of that visually.

Reviewing the other participants' work I was struck by how easy it is to fall into the trap of not describing the impact. I've done it myself here: "more accidents" (an impact) and "policing costs" (not an impact, as stated, but an area that will be impacted). I see this all the time when we do exercises such as risk storming at work. Describing the impact means being explicit about the mechanism, it's another deeper layer of analysis.

Day 4

Task: Draw a Rich Picture, exploring all that happens "behind the scenes" when you fill a kettle or pot with water.

Patrick Prill said everything I'd want to say about this one but I'd add links to Fail Here or Fail There, Read it and Weep, Fail Over.

Day 5

Task: Gather some of the thoughts that have been percolating around systems as the introductory page to a zine on 5x3 index cards.

In contrast to the other exercises so far, I took a few minutes to think about how I wanted to lay out a 5"x3" card. I quickly decided that the key thing I want to get across about systems is that it's easy to forget that you're looking at one.

I drew the hard-to-read central panel to try to make that point and then tried to list things that are relevant to getting a model of a system on either side. I didn't intend to categorise them as "to consider" and "to look for" before I started. I am still not sure they are good labels, but I kind of feel that the list on the left are things that could perhaps be considered in the abstract and on the right things that require looking at the system. Very roughly.

Next: Days 6-9
Image: Inspired by Explore It! by Elisabeth Hendrickson. Pragmatic Press

blogs

AI vs CI: Nobody Told the Pipeline

The mission was to kill “Works on my machine”

Then AI showed up

What’s the solution?

Implementing Page Objects in Playwright

Using Playwright's Built-In Test Runner? Or Something Else?

Setting Up Page Objects

The Page and Locator Class

Define the Page Object Class

Initialize the Page Object Class

Navigate to the Login Page

Creating a LoginAs Method

Get Text From Web Elements

Setting Up Test Data

Setting Up The Tests

Set Up Constants for the Test Suite

Set Up the Individual Test

How to Assert Our Test is Working

Test coverage does not tell you whether a change is safe enough

Understanding

Consequence

Detectability

Containment

Reversibility

So, what makes a change safe enough?

The Thinking Toolkit of a Software Tester

What Is a Heuristic Anyway?

❶I Have a Product. Where Do I Start?

❷Something Looks Wrong. Compared to What?

Imagine WhatsApp Introduces Edit Message

❸Something Changed. What Should I Regress?

Three Problems. Three Thinking Aids.

The Heuristic Is Not the Tester

What Sits Underneath All Three?

And Yes, AI Belongs in This Conversation

Try This in Your Next Testing Session

Final Thoughts

Further Reading

Lessons Learned from Fixing Flaky Tests with Claude

1. False Start

2. Understanding

3. Claude is Very Capable

4. Faster Understanding

5. Unlocking New Uses

Conclusion

How to Use AI With Your Brain

How Playwright Frameworks get configured with playwright.config.ts

What is Configured By Default?

But What About Dot Env?

What about the Mobile Viewports and Branded Browsers?

Quality Management System has the same parts but how it is operating is changing

The first question I had was, if I could go back in time today, what would I use AI for?

Proactive and Reactive Business Operations

AI assisted versus AI driven

Quality Management System has the same parts but how it is operating is changing

The first question I had was, if I could go back in time today, what would I use AI for?

Proactive and Reactive Business Operations

AI assisted versus AI driven

Singing in the Dark

How to Get Started with Grok Bot

AI Is Making Software Cheap to Build. Trust Is Still Expensive.

I Tested If Grok Bot Could Book My Flights

Workroom PlayTime 063: Spot the Difference

Spot the Difference (Puzzle 40)

See It! Days 20-24

Day 20

Day 21

Day 22

Day 23

Day 24

See It! Days 17-19

Day 17

Day 18

Day 19

See It! Days 10-16

Day 10

Day 11

Day 12

Day 13