blogs

Understand exploratory testing, and unlock its true value

2026-07-28T08:51:20Z

Exploratory testing isn't just a fancy way of saying manual testing. It's a real, important skill that every team building software needs.

The post Understand exploratory testing, and unlock its true value first appeared on The Quality Duck.

You Can’t Fix A Bug With A Prompt

2026-07-28T07:00:00Z

Let’s talk about fixing bugs. My second favorite activity after creating them.

How do I know the bug is fixed? Well, I’ve been taught by wiser and more experienced people than me (way back when), that if you want to make sure the bug never shows its face again – write an automated test for it.

Ok, I admit that there were times I didn’t write a test. Some fixes are so trivial, that sometimes a test is a luxury.

But even then, I was working on a hidden assumption. Just like in Fallout, code never changes. Until it does. But between those times it never changes.

I fix the bug. From this point until it actually changes by a human, or a bot or someone in-between – that code will compile, or transpile, or whatever we call “runs” the same way. A statement will execute, a condition evaluated, an exception caught – every time in the same way.

And now we come to modern programming languages: Prompts. Agents, prompts, workflows – they don’t run the same way every time. What they do is run their interpretation of the request. If they run tools – they run the deterministic parts. If they run sub-agents, they run interpretations based on other interpretations.

Interpretations work mostly the same way, until they don’t. When I was teaching my agent to develop in TDD, I had a couple of requests. One of them was to not create code without a test.

Which I thought was a very normal request from an agent. In fact, when I started out by telling it – we’re working in TDD – I assumed it knows what TDD is. Ha.

Always remember that what LLMs know is exactly what most of the population knows. And usually the “don’t create any code without a test” falls through the cracks.

So I made it official: One of the agent rules was exactly that. This was a legitimate bug fix. At least I thought so.

But it really was a suggestion. Which the agent considered, and depending on its mood, sometimes did, and sometimes didn’t.

I won’t go into the full solution (still in progress, if you want me to elaborate, comment), but part of it was to run a targeted coverage tool – a deterministic one, check it and stop the process if it found extra code. The TDD sequence looks like this:

Per step:
1. Write test(s).
2. `node scripts/tdd.mjs red <test-file>` — all newly added tests must fail. Pre-existing passing tests in the same file are allowed to remain passing.
3. Implement minimum to pass.
4. `node scripts/tdd.mjs targeted <test-file>` — targeted coverage + pass check.
5. `node scripts/tdd.mjs lint` — ESLint.
6. `node scripts/tdd.mjs full` — full suite + coverage.
7. Script says STOP. Human reviews.
8. Human runs `node scripts/tdd.mjs commit "message"`.
9. Human confirms next step. AI runs `/clear`.

But each step here is the suggestion. The real enforcement is done in the tdd.mjs code. Real code.

But this is just an example. The real problem is that more and more “code” is not programmed. It’s interpreted. That means that bugs are a lot more flaky to catch, but also are not permanently fixable.

And don’t get me started on companies switching model capabilities every other Tuesday. In the past, updating versions was a whole ceremony because we were worried something would break.

Now LLM providers do it for us without us knowing.
We can live with that. We should make sure we know.

And remember – bug fixes in prompts are not real bug fixes. They are more like wishes. Which may or may not come true.

Testing features that run on interpretation – and building the checks that don’t – is one of the four things we go through in the Masterclass.

The API Testing Masterclass: The Tactician
Four weeks, live, hands-on. Test data, debugging, fighting flakiness, and testing AI-based features.
Starts October 5. Twenty seats.
Early bird $399 through Jul-31, then $599.
Enroll here

The API Testing Masterclass: The Strategist – API quality strategy at the organizational level.
Join the waitlist

The post You Can’t Fix A Bug With A Prompt first appeared on TestinGil.

Your Test Strategy Was Built for Human Developers. AI Changed the Rules.

2026-07-28T03:46:01Z

Most test strategies were built around one assumption that nobody explicitly wrote down.

Continue reading on Medium »

From a player to a coach - 5 ideas from the Quality Coach`s Handbook

2026-07-28T00:00:00Z

5 ideas from "The Quality Coach`s Handbook" by Anne-Marie Charrett

Launch Your First Vibium CLI Session with Claude Code: A Step-by-Step Guide

2026-07-27T23:14:39Z

Vibium CLI + Claude Code

AI coding agents are changing how developers approach browser automation, and the newest entrant is Vibium — a browser automation tool built by Jason Huggins, one of the original creators of Selenium and Appium. Vibium bills itself as an AI-native successor to those tools: a single ~10 MB binary, built on the WebDriver BiDi standard, that exposes browser control through a CLI, an MCP server, or a client library — with zero driver setup.

In this guide, we’ll install Vibium, hook it up to Claude Code, and drive a real browser using nothing but plain-English instructions.

Why Use Vibium with Claude Code?

Zero configuration. Vibium downloads its own managed copy of Google Chrome for Testing on first run — no separate driver binaries or profile setup.
Semantic references, not brittle selectors. Instead of hand-written CSS selectors, Vibium’s map command returns stable element references like @e1, @e2, designed to be easy for an LLM to reason about.
Two ways to connect. Use Vibium as a plain CLI (agent runs shell commands) or register it as an MCP server (agent calls tools directly, no subprocess spawning).
Lightweight and fast. A small background daemon keeps the browser alive between commands, so each call is quick instead of relaunching a browser every time.
Built on open standards. WebDriver BiDi instead of a vendor-specific protocol, so the same install works across CI, local dev, and agent workflows.

Prerequisites
Step 1: Install Node.js and Verify Your Environment
Step 2: Install the Vibium CLI
Step 3: Install Vibium as a Claude Code Skill
Step 4: (Optional) Register Vibium as an MCP Server
Step 5: Run Your First Automated Session
Troubleshooting
Conclusion

Prerequisites

Before you begin, make sure you have:

Node.js 18 or newer (required for the npm-based installer and the JS client).
Claude Code installed and authenticated. If you don’t have it yet: npm install -g @anthropic-ai/claude-code
A supported platform: Linux x64, macOS (Intel or Apple Silicon), or Windows x64.
A terminal you’re comfortable working in.

You do not need a browser pre-installed — Vibium downloads its own managed build of Chrome for Testing.

Step 1: Verify Your Environment

Check your Node.js version:

node --version

You’ll want v18+. Then confirm Claude Code is installed:

claude --version

If that’s not recognized, install it with npm install -g @anthropic-ai/claude-code and re-check.

Confirming Node 18+ and Claude Code are both installed before starting.

Step 2: Install the Vibium CLI

Install the CLI globally:

npm install -g vibium

This installs the vibium binary. The first command you run that needs a browser triggers a one-time download of Vibium's managed Chrome for Testing build.

If you’d rather not install anything globally, every command also works through npx:

npx -y vibium go https://example.com

To keep the examples below identical either way, alias it in your shell:

alias vibium='npx -y vibium'

Verify the install:

vibium go https://example.com
vibium text

If vibium text prints the page's text content, you're good to go.

Installing Vibium and confirming it can launch and read a page.

Step 3: Install Vibium as a Claude Code Skill

This is the step that makes Claude reliable with Vibium’s exact command syntax instead of guessing at flags:

npx skills add https://github.com/VibiumDev/vibium --skill vibe-check

This installs the vibe-check skill into your project, documenting Vibium's full command set — go, map, click, fill, find, press, text, screenshot, wait, and more — along with the chaining conventions Claude should follow (e.g., using && for back-to-back navigate → interact → verify sequences, and running commands separately when you need to inspect output in between).

The vibe-check skill installed into the project — this is what teaches Claude the exact command syntax.

Step 4 (Optional): Register Vibium as an MCP Server

If you’d rather have Claude call Vibium’s tools directly instead of spawning CLI subprocesses, register it as an MCP server:

claude mcp add vibium -- npx -y vibium mcp

Once registered, Claude gains tools that map 1:1 to the CLI commands — navigation, mapping, finding, clicking, filling, and capture — without shelling out. Either approach (CLI skill or MCP server) works; the CLI skill is the lighter-weight starting point, and MCP is worth adding once you want tighter integration.

Vibium registered as an MCP server — Claude can now call its tools directly instead of shelling out.

Step 5: Run Your First Automated Session

Open a Claude Code session in your project directory:

claude

Now describe what you want in plain language:

Use Vibium to open https://example.com, find the "Learn more" link,
click it, and tell me what the resulting page says.

Behind the scenes, Claude Code will run a sequence along these lines:

vibium go https://example.com
vibium map                        # lists interactive elements: @e1, @e2, ...
vibium find text "Learn more"
vibium click @e1
vibium wait text "IANA"
vibium text                       # reads back the resulting page

A few things worth understanding about that sequence:

map returns a compact list of interactive elements with stable references like @e1, so Claude reads only what it needs rather than the whole page being pushed into context.
find locates elements semantically — by visible text, label, placeholder, or role — instead of relying on brittle CSS selectors.
The small background daemon keeps the browser session alive between each command, so the sequence runs quickly instead of relaunching a browser on every step.

Claude will report back in plain language: what it clicked, what the page said, and whether the session completed successfully.

Claude Code driving Vibium end-to-end from a single natural-language instruction.

Navigate, then map — Vibium turns the page’s interactive elements into stable LLM-friendly references like @e1, ready for Claude to act on.

Clicking a link, waiting for the new page, then reading it back — no selectors, just a reference and a wait condition.

Troubleshooting

“Command not found: vibium” — the global install didn’t add it to your PATH. Use npx -y vibium ... instead, or re-run the global install.
Claude can’t find the right command syntax — confirm the skill installed correctly with npx skills add https://github.com/VibiumDev/vibium --skill vibe-check, and restart your Claude Code session so it picks up the skill.
Browser fails to launch — Vibium downloads Chrome for Testing on first use; check your disk space and network connection, then retry a simple vibium go https://example.com.
MCP server not responding — run npx -y vibium mcp directly in a terminal to see any startup errors, and confirm with your client's MCP list command that the server registered.

Conclusion

You now have Vibium and Claude Code working together — Claude driving a real Chrome browser through plain-English instructions, using semantic element references instead of hand-written selectors. From here, the natural next steps are exploring the fuller command set (vibium record for session capture, vibium diff map for detecting page changes, vibium pdf for exporting pages) and deciding whether the CLI skill or the MCP server fits your workflow better as your automations grow.

Learn More and Stay Connected

If this model of vibe testing resonates with you, here are a few places to explore further:

🌐 Vibium Website
https://vibium.com
Learn how Vibium bridges traditional automation with AI-driven sanity checks.
🧠 Vibium on GitHub
https://github.com/VibiumDev/vibium
Explore the code, architecture, and examples behind Vibium’s traditional API and MCP-based vibe checks.
💼 Vibium on LinkedIn
https://www.linkedin.com/company/vibium-dev
Follow for product updates, architecture deep dives, and discussions on AI in testing.
🐦 Vibium on X (Twitter)
https://x.com/VibiumDev
Short-form thoughts, experiments, and conversations around AI-driven automation.
🐞 Daisy Lady Bug on LinkedIn
https://www.linkedin.com/company/daisy-lady-bug
Thoughtful takes on QA, testing culture, and the human side of quality engineering.

Launch Your First Vibium CLI Session with Claude Code: A Step-by-Step Guide was originally published in Women in Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

AI roulette: A quick win and a time sink

2026-07-27T19:21:11Z

In early March, I was wrapping up the build-out of Mailinator’s Ruby and JavaScript SDKs. I was adding missing endpoints that were available in the API but not the SDKs. It was during this time I experienced what I’ve come to call AI roulette: quick successes followed by sometimes maddening mistakes.

Let me take you back in time because it feels like ages ago in the AI coding days. (If you want more details on my harness back then, read the post on My AI Tooling Stack).

Codex screenshot from the aforementioned AI Tooling Stack article

At the time, I was heavily using OpenAI’s Codex app powered by GPT-5.2-Codex and then later GPT-5.3-Codex. I was doing strict TDD. I had two sessions going at the same time: one for JavaScript and another for Ruby. (If you look closely at the screen above you can see the two projects).

I’d prompt them both to do the same thing: implement a specific endpoint detailed in the OpenAPI spec. My workflow looked like this:

Each session would show me a failing test
I’d review the test and make changes if necessary
The agent would implement the code to get the test working. Then it would run the test and tell me everything was fine
Finally I’d run the tests myself to confirm success and check the responses

It was such a simple operation and it worked most of the time. (I also managed to do all this on Codex’s free plan without hitting limits. Those were the days!)

Saving the hardest for last

Finally it was time to tackle an endpoint called Stream all messages. That endpoint opens a web socket connection and any email that arrives in your inbox gets returned in a near-live stream until the connection closes. Yes, I saved the most complex endpoint for last.

I asked both Ruby and JavaScript sessions to implement the endpoint. Both tests needed a little adjusting but I finished and the agent implemented the logic.

💡

Agent = Model + Harness.

A harness is how you control the model; things like skill files, system instructions, prompts, guardrails, and scripts.

First, the Ruby agent finished. I checked the Ruby code, made sure all ENVs were set and ran the test. The test passed just like the agent said! I was impressed because it was so quick. Maybe this wasn’t the hardest endpoint?!!

Second, the JavaScript agent finished. I made sure all ENVs were set and ran the tests. The tests failed. Weird. As often happens when a failure occurs, I asked the AI agent to debug the code. Then I went back to wrapping up the Ruby SDK work: I committed my code and started documenting the changes. Then I published a new version. Done.

Running in Circles

When I went back to the JavaScript code it thought it had found the problem. It fixed it and then I retested. No good. I told the agent it’s still not working, it looked at the failure in the console, found the problem, fixed the problem and I ran the tests again. I went back and forth with the AI and the code it was producing for the rest of the day. No luck. The next day I picked up where I left off and still no success. I saw the agent running out of options and retrying the same approaches again and again.

When your tool fails

Rather than sit there and make no progress, I started to debug the JavaScript code written by the agent. I made notes of the things previously tried. Then it occurred to me I was essentially running an A/B test by having the two agent sessions, tackling the same problem. Test A passed but B failed.

Knowing the JavaScript agent made a different, incorrect decision that the Ruby version got right I started telling my Ruby chat what I’d done and shared the notes of the JavaScript failures. The Ruby session started to explain it’s decisions which I copied and pasted into the JS session. I went back and forth half a dozen times until the JS implementation worked.

💡

The irony wasn’t lost on me: I, the human engineer, was acting as some sort of carrier pigeon. Delivering notes between the two until they could resolve their differences!

This brought up an interesting problem. Ultimately LLMs like Codex are just tools. What happens when your tools fail? In this case I tried using another tool to help. If this approach hadn’t worked I would have needed to go back and start coding it myself. Or at least made some architectural decisions until the agent could finish.

The Roulette Wheel

If you had asked me how I felt during the quick wins with Ruby, I would have been singing Codex’s praises. For the vast majority of my time building out both SDKs, I followed my workflow and made great progress. In fact, that complex streaming endpoint surprised me with its initial ease.

However, by Day 2 of the JavaScript struggle, all I could think about was how much time I wasted and how I probably could have built the web socket implementation faster myself.

In some ways, this accidental setup became the ultimate A/B test. It was the exact same harness, exact same model and therefore the exact same agents. All tackling the exact same API with the same documentation. Yet it resulted in entirely different choices, code and widely different results.

This is the crazy reality of what a lack of determinism looks like.

While I’ve faced the AI roulette before, this was one of the most extreme cases I’ve experienced. The failure of my tool to help me caused massive delays. Interestingly, there was no breakthrough realization that led to a harness update. In the end, simply moving information between the two sessions was enough to overcome the roadblock. When you hit the AI roulette wheel, the solution isn't always to prompt harder; sometimes, the tool just spins a different number.

Balancing the scales

So, when people celebrate the immediate and magical wins of AI, I get it. I know that feeling of awe when a challenging task finishes in seconds.

Yet I also know how short lived those wins can be and how much unrecorded time you can lose when an agent gets stuck. It’s a constant dance of balancing skepticism and curiosity. Of enjoying the moment when then you win but keeping your hands on the keyboard for when it doesn’t.

Getting back into coding after time away

2026-07-27T18:37:04Z

TL;DR

When people ask me how I got back into coding after years away, my answer is essentially to have a project you can use to either start or add to (either production code or tests), find videos/trainings on the changes since you last coded, and don’t be afraid to just write some code and let Github Copilot give you feedback. Hell, even open up something like CodeAcademy to shake off the cobwebs.

What if you don’t have a project at work you can do this on? Or you want to do it outside of work resources?

Think about what you want to learn or focus on – a specific language, a framework, setting up static analysis or CI/CD pipelines, or just getting back into things.
Find an application you can run tests against (consider that public applications would have DDoS protections that could affect your automation efforts)
- You can use mine (or even the older one) that I built for talks using the basic Todo app tutorials for .NET and then modifying (clone and run locally)
- Or build one yourself (or even have AI build one for you).
Analyze the application like you would any other – what kind of tests/updates does it need that also meets the goals you have for this exercise?
What kind of training/background information do you need to implement the goals you have?
What community do you have that will support you in this endeavor, such as folks at work, LinkedIn, Ministry of Testing, or smaller communities?

Once you get that all squared away, jump in! Start coding/testing/whatevering and share your progress with your community. You’ll be back to confidently writing code in no time

My journey from code to manual and back to code

I spent a good portion of my career “in the code” (mostly C#, some JavaScript) pretty much every day – writing UI and API test automation, doing code reviews of test and production code, setting up and reviewing static analysis, CI/CD pipelines, etc. As well as the manual work of testing, of course.

When I joined Xbox, my job around automation moved to a more PM-type role – what do we need to automate, prioritizing, and working with our tech team to get that implemented (as well as building relationships with the engineers at the studio to get support for our automation from their side). I spent much of my time on the manual work of testing – the test planning and strategy, authoring test collateral, reporting, documentation, support, and the actual hands-on testing. I was getting rusty and itching to get my hands on some code.

As part of our performance reviews, we’d set goals. One goal I set for myself was to spend 1 hour per week on professional development and/or coding. During crunch (and boy did we CRUNCH!), this goal went by the wayside in favor of directly supporting my team. But in the in-between times, I was able to spend a little bit of time here and there helping an internal team with some automation assistance (again in C# and JavaScript, then TypeScript). For a while, I had set my goal to spend 1 day a week coding, and was able to help build an API automation framework for that team, including CI/CD pipelines (C#, RestSharp, and AzureDevOps). I could build with what I knew already, and the small team gave me feedback in PRs about new and fancy features of C# and .NET I could use to improve.

But things got busy again, and again that fell by the wayside.

At the end of 2025, I left direct title support to work in a more experimental place, but in a coding-focused role. The tech team was working on some tooling to make automation easier to implement by more people, and my role was to use the new tools in some experiments and provide feedback. The tools were implemented using C#, and a proprietary framework (part of the experiments). Later, I was tasked with using all of the tools available, including the experiments, integrated in with existing tests for a title. My earlier experiments were solo, but now I’d be working with other engineers to integrate the automation I built into their existing solution.

Here, I needed to skill up a bit. I needed to align with what they were using, their workflows, etc. They were using C# 8 at the time (upgrading to 10 after I started). I looked for some videos to catch me up on the features of C# and .NET since the last one I was super familiar with (C# 3?! lol). The team was also super helpful with suggesting new and fancy ways to do things (like using switch expressions and pattern matching rather than a switch statement). Jumping in and just writing “old style” code, then asking Github Copilot to make it more efficient for me taught me a few things, too. It also helped me to stop the analysis paralysis of “how do I implement this in the best way possible” and just write some damn code!

Another exercise to help me get back into coding was when I was asked to speak at Browserstack Breakpoint this year. I chose a technical talk I had given before (with the short timeline I had), and needed to update the example code for it. The original code used ASP.NET 2.1, last updated almost 4 years ago! So I brought it up to ASP.NET 9 – lots of changes were needed, and that was a fun exercise.

The inevitable question – why didn’t you just let AI write the automation for you, or upgrade the solution for you?

For those experimental tools, and more specifically the framework, AI had nothing to go on. I tried training agents, skills, tools on the framework and existing tests to assist in writing better tests (maybe a skill issue on my end as well). Github Copilot in Visual Studio often suggested code that made no sense and had a ton of errors. Even giving it examples of good code/tests, it often failed. As part of my job at the time, I needed to understand the best ways to write the tests and how the framework was implemented anyway – I needed to provide actionable feedback that would improve the system for everyone. I felt that my hands-on knowledge of the toolset was more important for this task.

As far as not having AI upgrade the solution for my talk – that was fun to do! What better way to learn what has changed in so many years than to do it yourself? And I missed coding for so long, why let AI have all the fun?

The post Getting back into coding after time away appeared first on g33klady.

Sometimes Efficiency Is Just Turning the Suitcase the Other Way

2026-07-27T16:52:06Z

If Medium puts this content behind a paywall, you can also view it here (LinkedIn).

For almost 30 years, I have worked in Engineering Productivity in one form or another. I have spent time at Amazon, Google, and Microsoft, and every large software company has the same underlying problem: at that scale, small inefficiencies compound into developer fatigue, wasted infrastructure, slower delivery, and missed business opportunities.

Yet those inefficiencies often remain unfixed because the pain is distributed across thousands of people, each inconvenienced only slightly. No single person or team experiences enough pain to take ownership. Fixing the problem requires someone to act as an 𝐎𝐰𝐧𝐞𝐫 by quantifying the aggregate cost, defining a credible solution, convincing a senior leader to fund it, and making sure it gets done.

The waste is spread across people, teams, and budgets, which makes it almost invisible. And the irony is that, when you look at the problem holistically, fixing it often costs the company only a fraction of what it is already wasting by allowing the inefficiency to persist.

I wanted to use a real-world example from this week to illustrate that idea. It also highlights three principles I have learned repeatedly:

You do not need to be the domain expert to recognize an inefficient system
You do not need perfect data to decide that a problem is worth investigating
You do not need a perfect solution, be scrappy and iterative and learn

You need enough understanding to form a hypothesis, enough directional data to estimate whether the impact is meaningful, and enough curiosity to test a simple solution.

On Thursday, I landed at Seattle-Tacoma International Airport on Turkish Airlines flight TK203, returning from a visit to my Amazon Berlin team with a stopover in Istanbul. I reached the baggage carousel and waited. And waited. And waited. It took more than 30 minutes for my suitcase to appear. When I finally grabbed it, hundreds of bags were still circling the carousel, while hundreds of increasingly impatient passengers continued waiting.

As I watched the system operate, I realized it was a perfect example of the kind of problem I see in my day job. This is a small inefficiency that inconveniences hundreds of people on one flight and thousands over the course of a day. For each passenger, the impact is annoying but tolerable. In aggregate, and over time, the human cost is massive (SeaTac handled more than 52 million passengers in 2025).

The pain is distributed. The passengers experiencing it do not own the system. The people operating individual parts of the baggage process might not see the end-to-end impact. And the person with the authority to fix it might never experience the problem, much less quantify it. There is no clear owner, no visible measure of the total cost, and little incentive for any individual person or team to act.

As I watched the bags enter the carousel, I noticed that they were arriving in 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐨𝐫𝐢𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧𝐬. Some were lying flat with their long side parallel to the direction of travel, taking up the most space. Others were also lying flat but rotated 90 degrees, so they occupied somewhat less carousel length. A smaller number were positioned on their thin edge, taking up substantially less room.

Near the point where the luggage entered the carousel, there was a sensor that checked whether enough space was available before releasing another bag. When the carousel became too crowded, the feed paused. That safeguard makes sense. You do not want luggage colliding, piling up, or falling off the belt.

The problem was that many of the bags already on the carousel had gone around several times without being claimed. Because they were using much more space than necessary, the system frequently decided that there was not enough room to release another bag. The same bags continued going around and around. There was plenty of activity, but very little throughput.

Meanwhile, hundreds of bags were still waiting behind the entry point, and hundreds of passengers were waiting for those bags.

𝐖𝐡𝐚𝐭 𝐢𝐟 𝐰𝐞 𝐣𝐮𝐬𝐭 𝐭𝐮𝐫𝐧𝐞𝐝 𝐞𝐯𝐞𝐫𝐲 𝐬𝐮𝐢𝐭𝐜𝐚𝐬𝐞 𝐨𝐧𝐭𝐨 𝐢𝐭𝐬 𝐭𝐡𝐢𝐧 𝐞𝐝𝐠𝐞? [Well, not every bag could or should be positioned that way: soft-sided luggage, duffel bags, and unusually shaped or unstable items might require a different orientation. The real goal is to position each bag so that it uses as little carousel as practical while remaining stable, safe, and easy to retrieve.]

Since I was just waiting, I started doing some rough math to keep myself entertained.

Suppose placing a suitcase on its thin edge allows it to occupy 30% less belt length than one flat orientation and 60% less than another flat orientation. If those two inefficient orientations occur equally often, the average reduction would be: (0.5×30%) + (0.5×60%) = 45%.

In this intentionally simplified model, 𝐩𝐫𝐨𝐩𝐞𝐫𝐥𝐲 𝐨𝐫𝐢𝐞𝐧𝐭𝐞𝐝 𝐛𝐚𝐠𝐬 𝐰𝐨𝐮𝐥𝐝 𝐮𝐬𝐞 𝐚𝐛𝐨𝐮𝐭 𝟓𝟓% 𝐨𝐟 𝐭𝐡𝐞 𝐬𝐩𝐚𝐜𝐞, so the 𝐜𝐚𝐫𝐨𝐮𝐬𝐞𝐥 𝐜𝐨𝐮𝐥𝐝 𝐚𝐜𝐜𝐨𝐦𝐦𝐨𝐝𝐚𝐭𝐞 𝟖𝟐% 𝐦𝐨𝐫𝐞 𝐛𝐚𝐠𝐬 (1÷0.55=1.82) before becoming saturated.

Of course, this is only a rough model. Data scientists, please do not bring your pitchforks and torches. It’s not meant to have scientific rigor, just scrappy math. Suitcases come in different shapes and sizes. Bags would not fit together perfectly. Curves in the carousel create gaps, and passengers still need enough room to retrieve their luggage safely. But 𝐝𝐢𝐫𝐞𝐜𝐭𝐢𝐨𝐧𝐚𝐥𝐥𝐲, this back-of-the-napkin calculation hinted that deliberate bag orientation could still make a meaningful difference in throughput.

𝐖𝐚𝐢𝐭𝐢𝐧𝐠 𝟑𝟎 𝐦𝐢𝐧𝐮𝐭𝐞𝐬 𝐟𝐨𝐫 𝐦𝐲 𝐬𝐮𝐢𝐭𝐜𝐚𝐬𝐞 was an annoying but tolerable inconvenience. But that is the individual view. The system-level cost is the delay multiplied across everyone experiencing it.

Could I back-of-the-napkin estimate it? The Airbus A350–900 that brought me home from Istanbul was configured to carry 𝟑𝟐𝟗 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫𝐬. Suppose the flight was 𝟗𝟎% 𝐟𝐮𝐥𝐥 and 𝟗𝟎% 𝐨𝐟 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫𝐬 𝐡𝐚𝐝 𝐜𝐡𝐞𝐜𝐤𝐞𝐝 𝐥𝐮𝐠𝐠𝐚𝐠𝐞. That would mean ~𝟐𝟔𝟔 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫𝐬 𝐰𝐚𝐢𝐭𝐢𝐧𝐠 𝐚𝐭 𝐭𝐡𝐞 𝐜𝐚𝐫𝐨𝐮𝐬𝐞𝐥.

If they waited an average of 30 minutes, that’s 266×30 = 7,980 passenger-minutes, which is 𝟏𝟑𝟑 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫-𝐡𝐨𝐮𝐫𝐬 waiting for the luggage from a single flight. That’s a lot of waste.

My earlier calculation suggested that orienting the bags more efficiently could increase the carousel’s effective capacity by 82%. If carousel saturation were the dominant bottleneck, and that additional capacity translated into a proportional increase in throughput, the average wait could theoretically fall to: 30÷1.82 = 16.5 minutes. That would save 30−16.5 = 13.5 minutes per passenger. Let’s be conservative and hypothesize that 𝐰𝐞 𝐜𝐨𝐮𝐥𝐝 𝐬𝐚𝐯𝐞 𝟏𝟎 𝐦𝐢𝐧𝐮𝐭𝐞𝐬 𝐩𝐞𝐫 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫. Across 266 passengers, that would save: 266×10 = 2,660 passenger-minutes. That equals approximately 𝟒𝟒 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫-𝐡𝐨𝐮𝐫𝐬 on a single flight.

This is why small inefficiencies matter at scale. Ten minutes may not be enough for any one passenger to demand that someone redesign the process. But collectively, it is more than an entire workweek of human time, every time a flight like this arrives.

Of course, simplified efficiency arguments can become misleading. Better orientation would not make other potential bottlenecks faster, such as [1] opening and unloading the aircraft; [2] transporting luggage from the aircraft; [3] sorting and routing the bags; [4] delivering the first bag to the carousel; etc. It would improve only the carousel congestion portion of the process, specifically when it is full and the system is preventing more luggage from entering.

These are obviously just estimates. To calculate the real result, we would need actual data: number of passengers, number of checked bags, passenger arrival times at baggage claim, first-bag and last-bag delivery times, carousel occupancy over time, frequency and duration of feed pauses, bag dimensions and orientations, upstream baggage-handling constraints, …

But this is often how an investigation begins: You observe something, you form a hypothesis, you build a simple model, then you decide whether the potential impact is large enough to justify collecting better data.

So, should we embark on an expensive multi-year redesign of the hardware? No. My immediate solution would be surprisingly scrappy and low-tech. What if one person stood near the entry point and consistently turned each suitcase onto its thinner side?

Swissport, the company that handles luggage at SeaTac, has job listings for baggage handlers for $22/hr. We could add less than a dime to the cost of each ticket to fund the position, and it could potentially save 44 passenger-hours of waiting [the actual employment cost would be somewhat higher once benefits, payroll taxes, training, and overhead were included, but we would still be talking about pennies per ticket, not dollars].

As a customer, I would have gladly paid an extra dime to save ten minutes of my life waiting there.

This illustrates another critical part of Engineering Productivity: always be scrappy and iterative when validating an idea. Start with the simplest experiment that can validate the hypothesis, and learn relentlessly.

We could run an experiment for N months where a baggage handler positions all bags on their thin edge as they come out, and we measure how much time that saves. We could run A/B experiments (with comparable flights) where sometimes we have a baggage handler and sometimes we don’t. That would give you more trustworthy ROI data to help you decide whether or not to fix this long-term, because a manual step is not the most scalable or durable final answer. If the experiment demonstrated meaningful improvement, we could then consider automation, such as a redesigned chute, to orient bags before they enter the carousel.

Maybe the experiment would reveal an 𝐮𝐧𝐢𝐧𝐭𝐞𝐧𝐝𝐞𝐝 𝐜𝐨𝐧𝐬𝐞𝐪𝐮𝐞𝐧𝐜𝐞 of the change. Maybe if the bags are placed on the thin edge it’s harder for passengers to fetch them, so we’ve fixed a problem but introduced a new problem. Sometimes ideas that look good on the surface have side effects you never anticipated. Maybe the entire idea was a bad idea!

But do not begin with the expensive solution. Begin with the cheapest credible experiment.

Engineering Productivity is sometimes misunderstood as “𝐣𝐮𝐬𝐭 𝐛𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐫 𝐭𝐨𝐨𝐥𝐬.” My real job is improving the performance of the entire system, making it easier for thousands of engineers to build features that delight customers and create business value.

Sometimes improving efficiency really is as simple as turning the suitcase.

Transitions are Hard

2026-07-27T15:31:36Z

I’ve been thinking a lot about transitions. I recently had one of those really great weeks where everything clicked, my brain was settled, and everything felt great. Then the week ended, and I couldn’t find my rhythm. Nothing was really wrong. I just couldn’t get back into my regular routine, and I noticed it.

Then I took a long-planned two day backpacking trip on the Olympic Peninsula. Two days on a trail, nobody to talk to but myself, and by the time I got back to the trailhead, something had reset. The thing I couldn’t do sitting in my own living room, I did without trying somewhere around mile six.

I’ve known for a long time that transitions are hard. A lot of the time in life, we need to shift gears, or move into a different context, and sometimes that transition just doesn’t land like we want it. I needed a bridge to get me where I needed to be, and I didn’t build one on purpose. The trail just happened to save me.

The Neutral Zone

There’s a name for this, and it comes from William Bridges’ book Transitions. Bridges argues that every real change has three parts: an ending, a neutral zone, and a new beginning. Most of us plan for the ending and get excited about the new beginning. Almost nobody plans for the neutral zone, the in-between stretch where the old rhythm is gone and the new one hasn’t kicked in yet. Bridges describes it as “a nowhere between two somewheres.” It’s disorienting on purpose. It’s also, he argues, where the actual psychological work of change happens.

A lot of us forget that even events that seem small need transitions. But that’s exactly what happened. I had a good week with a good ending, but I dropped straight into a new beginning (routine) with no neutral zone in between. No wonder it didn’t take.

This isn’t unfamiliar territory for me, either. I am generally good at starting and stopping things. In fact, I’m often intentional about how things start and stop. I know transitions are hard, and I’m often deliberate about building in buffer and time to reconnect or rest when needed. This time, that follow-through didn’t (or couldn’t) happen on my end, and it was noticeable.

Recovery Isn’t Just for the Hard Weeks

I see leaders make this same mistake constantly, and I’ve made it plenty of times myself. We usually only plan recovery for the bad stretches. Nobody schedules a buffer day after the launch that went great, the offsite everyone loved, or the quarter where we hit every metric. We treat good intensity like it’s free.

Jim Loehr and Tony Schwartz’s book The Power of Full Engagement makes the case that this is backwards. Their core argument is that performance runs on oscillation, expending energy and then deliberately recovering it, and that the body doesn’t distinguish between good stress and bad stress the way our calendars do. A win still spends energy. A great time with a friend still spends energy. But when we skip the recovery just because the exertion felt positive, we end up with the same flatness and fog we’d get from skipping recovery after something hard. I didn’t feel bad, I just felt off, and off is what unrecovered energy looks like when nothing is technically wrong.

Build the Bridge

Anyone leading a team, or just leading their own week, should stop treating recovery as a response to difficulty and start treating it as a response to any real intensity, good or bad. After a big win, build in the neutral zone on purpose, a slower day, a walk, an hour with no agenda, before you ask anyone to be sharp again. After the offsite everyone raved about, don’t schedule the hardest meeting of the quarter for the next morning. The team is not being dramatic if they’re off. They just spent energy on something that felt good, and nobody built them a bridge back.

For me, the trail worked, but only by accident. I didn’t plan a neutral zone. I planned it short backpacking trip. The fact that it did double duty as reentry from a good week is luck, not design. Next time, I want to be able to build that bridge on purpose instead of hoping a trailhead shows up right when I need it.

There will be more weeks like that one. That’s the whole point. But I hope I’m done assuming the good ones don’t need a landing strip too.

If you’d like to talk more about this, my calendar is open.

I was an AI sceptic (and I’ve got the old blog posts to prove it)

2026-07-27T10:25:18Z

I was an AI sceptic (and I’ve got the old blog posts to prove it)

TL;DR

I was an AI sceptic in 2023 and I wrote it down, so I can go back and actually answer my old self here
What I got wrong: garbage in, garbage out isn’t a reason to doubt AI, for a tester it’s the best news going
AI is only as good as the thinking you feed it and that thinking is our job, so it rewards testing skill rather than replacing it
For fellow sceptics, the problem was never the tool but how we use it

There’s a certain type of blog post going around at the moment, the one where a tester explains how AI won them over. I’m about to write one, too (so do bear with me) but I think I’ve got something most of them don’t have; I wrote my scepticism down at the time.

Back in 2023 I ran a set of experiments with generative AI and testing, published the fairly lukewarm results and moved on… which means I can go back now and answer the version of me who came away unconvinced. This post pulls those early pieces together and replaces them and really it’s a letter to any tester still standing where I was.

Here’s what I got wrong, back then I kept bumping into garbage in, garbage out where I asked the AI a lazy question and got a bad (AI slop) answer back which gave me reasons to be sceptical. It’s taken me a long time to see it the other way round, that garbage in, garbage out is actually a good outcome for testers; that if the quality of what comes out depends on the quality of what you put in and if putting good thinking into messy problems is literally the job of a tester… then these tools don’t threaten us, they help us to use our innate skills and show our value.

Fig 1. GLaDOS, the original testing AI, quietly judging my early prompts.

What sceptical Callum actually found

Since I’m retiring my original posts I want to keep their findings rather than lose them, so here’s what 2023 me concluded about using AI & LLMs.

I asked whether AI could write a good test approach. I gave it a proper brief, integration between a web front end and a database over REST APIs, an agile team with an embedded tester, Jest for unit tests and Cypress for the API layer. What came back was tidy and decent on the basics, the roles and responsibilities, a “test champion” to advocate for exploratory testing and it sensibly wove automation in to support exploration. But it gave me an average of everything ever written about test strategy online, defaulting almost entirely to end to end automation (saying nothing about unit or integration testing closer to the code) and gave me no real view of shift right or exploration. To get anything more complex and detailed I had to write the approach myself and let it tidy up the words I’d given it.

I asked could AI write good exploratory tests. As a starting point it was okay, a bit like testing flashcards or the basics from a risk storming session; the format of “give me some exploratory test ideas for the risk that…” was a handy thing to put in front of people during a bug bash for them to use. The test ideas were shallow though, focusing on breadth rather than depth, giving risks like data integrity, authorisation, performance and localisation but with no depth. It could tell me to look at things but not give any rationale as to why it mattered or any specificity, for example it’d say localisation but nothing about character sets or regional data standards.

Next was whether it could teach me technical concepts and this was the one that was the best, where even sceptical me came away impressed. I used it to teach me about APIs and it walked me from “what is an API” through the request types and through to how you’d send data with a POST, in clear human understandable language. The bit that I should have paid attention to was what happened when I sharpened how I asked questions; a vague ask gave me a generic answer but naming Postman got me tool specific steps and then asking for an example payload on top got me steps and a worked JSON body. Now we’re cooking with gas, I wrote at the time, completely missing that I’d just discovered the entire trick to AI prompting (that you have to use bigger prompts and be really specific on what you want). I did at least note the danger, when a tool hands you one confident answer and you can’t see the sources behind it so you have to treat what it says with healthy scepticism.

I also wrote a follow up on the risks I’d found about garbage in, garbage out prompting, half braced for the AI to go full HAL 9000 on me, calmly assuring me everything was fine while quietly doing its own wrong thing. Those risks were real and I’ll come back to them properly, but in short they were about maintainability, consistency and a worry that we’d start prizing people who could write a prompt over people who understood engineering.

Fig 2. Hal 9000, the quintessential evil AI that wouldn’t listen to prompts.

My verdict at the time was that using AI was handy for a first draft, it’s shallow without a lot of steering, good for teaching basic well known concepts and a bit risky. Every one of those observations was true, but my mistake was the conclusion I came to from that.

Responding to my old scepticism

I’ve done a lot of learning since I wrote those old posts in 2024 so I want to use this as a way to respond to the objections that I’d raised about AI in the past, because each one has an answer now. This is important because I hope it’ll help other AI sceptics in the testing and software delivery space to go on their journey too.

“It only gives me the average of what’s on the internet, not a considered response“. It did that because I didn’t give it anything else to work from, AI needs to be given more context in order to give a better answer that isn’t just thin and generic. Once you feed it your acceptance criteria, your risks, your guidelines / standards and an example of what good looks like then it stops giving you the average and starts giving you something useful. The thing that I’d missed is: AI wants to please you (it’s a people pleaser) so it won’t push back, critique or point out problems on its own… you have to explicitly tell it to do that.

So my AI prompts now include something like:

Challenge me on my thinking and critique this design, tell me what could go wrong or what won’t work, look for meaningful risks I can use to refine it. Disagree with me where you should rather than just reinforcing my idea and give me your reasoning.

Rather than hallucinate a response, rate your confidence for how to act and respond between 1-10. If you’re not confident on a course of behaviour <6 then stop and ask me for more context or direction in order to provide a better response.

I pair the prompt for critique with an instruction to own up when it doesn’t know what to do by asking it to flag when it hasn’t got enough context to act, give a confidence rating from 1 to 10 with a rule that if it drops below 6 it stops and asks me. That stops it quietly making things up and hallucinating when it doesn’t actually know.

Fig 3. Holly, the AI of Red Dwarf, always needed to be given more context.

On anything big I also get AI to prove it’s actually understood me before it builds anything by having independent agents each summarise the spec and what they think it’s asking for. Then getting a fresh agent to compare those summaries and score how aligned they are, flagging any drift it spots, so I catch problems in uncertainty before anything gets built.

“Its exploratory ideas are shallow and it never says why something’s risky“. That was because I’d given it lazy one line prompts asking for tests. If you give the AI a proper structure for forming exploratory test charters and some trusted heuristics for risk identification then it can go a lot deeper. If you want to read more about how I do this then read my post on exploratory testing with AI.

“It ignores the code and focuses on end to end testing.” Again, this was because I’d never pointed the AI at the code or the testing and engineering standards for types of test I wanted. If you define what good testing looks like and point the AI to that alongside the code repo, then a prompt as simple as “review this code against our unit and integration testing standards and tell me where coverage is missing with fixes” gets it writing the cheaper tests closer to the code (instead of building another costly end to end suite).

“It’ll make us value prompt writers over people with real skills“. This was my biggest worry and it’s the one I’d most want to reassure myself about. Writing a genuinely good AI prompt (or building things with AI) uses so many good testing skills like knowing what to ask, what could go wrong, what to assert on, what heuristics matter and how to define if something is good enough. You won’t be outsourcing or losing your testing skills, even if you end up baking some of them into automated prompts, AI enabled engineering organisations need the exact skills a good tester already has.

“It’ll cause problems, refactoring code a lot will cause regressions“. This risk still stands but it’s not specifically an AI issue, it’s a foundational issue with engineering and a potentially optimistic view of developing at pace. Any refactors teams make can potentially break things, that’s not new to AI, we also know that teams are bad at thinking about testing for looking for regressions so tend to miss this. What’s new is the speed and how customer testing / manual testing at the new pace of change won’t keep up with the new potential rate of failure with faster drift leading to faster AI slop. But we can use AI to retrofit E2E tests to make sure behaviour stays static or add AI skills to report on maintainability and other core code quality areas to give ourselves a safety net that works at pace.

The risks I raised and where they actually stand now

The biggest risk I’d raised was that generative AI builds from everyone else’s code and habits scraped off the internet. The thin answers you see from a basic prompt come from that, a model reaching for the average of everything online in order to people please and give you some kind of a response (rather than a nuanced one based on your actual context). On top of that you then inherit styles and quirks you didn’t choose yourself, which is how you end up with bad results and AI slop.

That risk gets bigger when we want to go faster, or when we fall prey to the idea that AI is a genius that always gives you the right answer (a lot of AI psychosis comes from it people pleasing and confirming your own biases). When we forget to check what it’s given us that’s when we get inconsistent coding patterns, behaviour refactored out from under us, tests rewritten to do the wrong thing and all the anti patterns of code on the internet. I see this constantly with my AI usage… the number of times I’ve had to ask “what does this test actually assert on” and fix for it is wildly high, I’ve also had to keep reprompting to say ASSERT THAT SAVED DATA PERSISTS ACROSS A SAVE VIA THE UI AND API!

None of that is really the AI’s fault specifically, it’s the tool exposing engineering that was already foundationally weak (something that DORA’s research keeps finding), AI is an amplifier that magnifies the strengths and weaknesses a team already has. We need to help teams to fix their quality foundations to get them to a better result consistently.

So yes, the risk is real but it’s not a reason to avoid using AI… it’s a reason to bring quality thinking into how we use and prompt it. Just like when we build features we need to build our prompts using quality engineering thinking to steer AI into the right thing:

Build certainty by asking for everything up front, the more complete the prompt the less AI has to guess at and get wrong.
Don’t assume the output is good, prompt for quality signals and review what comes back.
Be critical of what it’s built, get it to test the risks so you learn about the quality rather than trusting it.
Give it your heuristics and best practices to check its own work against, so “good” means your good and not its people pleasing average.
Prompt in small steps so the feedback loop stays short, rather than one big ask you can’t unpick or understand.

So if you don’t want thin answers, want to meet better standards or are worried about the risk that AI will create slop, derisk it by prompting in a way that tells AI exactly what you want. Don’t do what I’ve seen a lot from people when I’m teaching them to use AI and ASSUME AI KNOWS WHAT YOU WANT, AI isn’t some kind of genius that’ll fill in the gaps and build what was intended from a light one line ask. It’s a dumb number generator that needs telling and properly redirecting to the right answer.

Fig 4. SHODAN had a god complex, but you shouldn’t believe her genius.

The real risk is in how people think about and talk to AI rather than the tool itself… it’s classic garbage in, garbage out!

We’ve been here before

I’m going to be honest, originally I’d come in to looking at AI with a really thin assumption, a confirmation bias that I wanted to prove out: that AI was bad at testing and that it wouldn’t replace me.

I don’t think I’m alone with this, you’ll have seen so many posts and messages saying “AI will never replace manual testing” or “you can’t automate a human tester’s intuition”. Many people are defaulting to “AI can’t fully replicate my judgement so I won’t engage with it at all” rather than looking at what rules we can teach AI to get it part of the way there. It’s a similar pushback we saw from the testing community to dismiss automation testing, something that partially came from a place of imposter syndrome and worrying about not being technical enough or being replaced. The real blocker here isn’t the ability to teach AI to test and use human intuition, it’s far more likely to be (something I felt too, so I know it) feeling like we’re handing our skills away and making ourselves replaceable.

But as I spoke about in You’re Not Ready for QE, our teams want to work with people who adopt new tooling, build things and help them to ship faster. That includes AI too, they’re going to want quality professionals (just like they wanted automation engineers and SDETS) who can help bake their knowledge and intuition into their delivery pipelines. The intuition and testing skills you’re trying to protect is the exact thing that makes you good at steering AI into doing the right things for teams, so it makes sense to apply it where it’s wanted!

If you’re still the sceptic

So if you’re stood where I was after a quick go with AI (thoroughly underwhelmed and thinking about the risks) here’s what I’d say to you; your instincts aren’t wrong, the shallow output is real and so are the risks of AI slop. But maybe you’ve drawn the same conclusions that I did, that this is a tools problem and not worth engaging. Why not push further by trying to give AI the context that you’d give a new starter, tell it what good looks like, let it know that it’s okay to disagree with you and to own up and say when it isn’t sure about something. Then review what comes back the way you’d review a junior’s work, giving guidance where needed for what would be better and keeping the good bits so that you can run them again. You’ll find that AI tends to meet you at whatever level of thinking you bring to it.

And if you’re worried that using AI will rust your skills out or replace you, remember that context changes all the time and you’ll be using your skills to improve the guardrails and test the difficult things. Just like how automated testing didn’t kill the need for manual and exploratory testing… it (supposedly) just freed up our time to do more of it. That’s how I’m seeing using AI, as a tool that I can use to enhance the testing I do and get it into a pipeline where it matters (for speed) but still has to rely on my knowledge for what matters.

Thanks for reading. If this resonated (or if you think that I’ve got it wrong) there’s more to read on the blog… or get in touch via LinkedIn. I’ll talk quality to anyone who wants to listen!

More Blogs

Data testing areas of focus

2026-07-26T20:25:36Z

I’ve just recently come off working with a client in the big data space, where we were looking to verify over 9 billions records of data. This was a big undertaking so we had to narrow down the areas of the data that we looked at for verification.

It’s worth saying that I wouldn’t usually recommend verification of all 9 billion records due to the time and complexity involved and I’ll go in to different techniques to ensure you do not have to do this in a future blog. However this client was very concerned about their data and every individual record meant something very specific so they were very insistent on this.

Structure

The first are we concentrated on is the structure of the data. Our data was in JSON-L format, where each line in a file is a fully formed JSON record. However whatever structure or format your data is in, this will always be a big area to look at.

The main way we did this was by verifying the JSON records against schemas. With JSON format this is easier to do so we would verify the record on the way in to the system and on the way out of the system. By doing this, we knew that every single record that we ingested in to the system had the correct structure and that we did not alter or break that structure before the record left the system.

For instance, we may have wanted to verify the following JSON:

{
   "record":{
      "people":[
         {
            "first_name":"jeff",
            "surname":"peters"
         },
         {
            "first_name":"laura",
            "middle_name":"jane",
            "surname":"smith"
         }
      ],
      "action":"CREATE"
   }
}

In order to do this, we would create a JSON schema similar to the below:

{
  "type": "object",
  "properties": {
    "record": {
      "type": "object",
      "properties": {
        "people": {
          "type": "array",
          "items": [
            {
              "type": "object",
              "properties": {
                "first_name": {
                  "type": "string"
                },
                "middle_name": {
                  "type": "string"
                },
                "surname": {
                  "type": "string"
                }
              },
              "required": [
                "first_name",
                "surname"
              ]
            },
            {
              "type": "object",
              "properties": {
                "first_name": {
                  "type": "string"
                },
                "surname": {
                  "type": "string"
                }
              },
              "required": [
                "first_name",
                "surname"
              ]
            }
          ]
        },
        "action": {
          "type": "string"
        }
      },
      "required": [
        "people",
        "action"
      ]
    }
  },
  "required": [
    "record"
  ]
}

These are just example schemas and your situation and data may differ quite a lot to this JSON example. However structure is the most fundamental thing you can assure with your data. You can also assure data structure very early on in the process which is great as the earlier on you can assure it, then you can build up the confidence levels over time. We had excellent automation at all levels of our components to assure they would cope with the data structures they would be presented with. However nothing can prepare the system for the unexpected nature of real, live data which is where this verification was so useful!

Another thing to bear in mind with data structure is what to do if your data does not meet the required structure. In our case, we were streaming in data so we could send the data to a DLQ if it failed and examine the issue when appropriate.

Content

This is the hardest area to validate in the data because you require context about the data itself. Here we are talking about the actual values in the fields in the data. At this point we are confident on the structure of the data, but not necessarily the values within that structure. There are two parts to validating the content.

1. Basic content validation

Firstly, you can validate the contents of specific fields. This can be very useful and following on our example from above, we chose to put some of this within the schema itself. For instance, given the JSON above you can see the following field:

"type":"CREATE"

In our schema, we could do something similar to the below, which would mean we know when the record gets schema validated that the content of this field must be one of the allowed values.

"action":{
    "type":"string",
    "enum":[
        "CREATE",
        "EDIT",
        "DELETE"
    ]
},

There are many different types of validation that can be performed in this way, including:

Length
Special characters
Any form of regex check on the value
Case checks

This is not an exhaustive list by any means! However this kind of basic data validation will only cover individual fields, not the relationships between the fields and the records.

2. Data meaning validation

To really examine the context, you need to understand what your incoming data means and how the data records all relate to each other. In a lot of data systems, they may be designed to be dumb and not really understand the data they ingest, meaning this data validation is not possible. However, in our system we had good communications to the team who owned the source data and so we had a good understanding of the data itself and how records related to each other.

To give an example of what you can do here, let’s take the example we used earlier and consider the action field again, where you can see there are only three valid options:

"enum":[
    "CREATE",
    "EDIT",
    "DELETE"
]

There is a logical order for these records to come in to the system:

CREATE
EDIT (0 to many)
DELETE

Knowing how these records are created is important, because it might be that records can not have a CREATE (for instance for transient or incomplete records) for example, but let’s assume that the above is the only order that makes sense in this case.

The way we resolved this way to perform incoming validation checks, whereby we would check when a record came in what type it was. If it was a CREATE then we would check that there is no record with that id already. If it was an EDIT we would check that there was one record and it was a CREATE and if it was a DELETE we would simply check if there was a record. This allowed us to validate all the scenarios on every single record.

It’s worth noting that these checks were time consuming on the ingestion and so were actually turned off after a period of time in live running when we had built up enough confidence and ingested enough records that the client was happy with this. It’s another area where I would not recommend performing these kinds of checks on every record but in this case it was very explicitly what the client wanted.

Volume

For big data especially volume testing should form a very important aspect of your data testing. My client wanted to ensure that every single record that was ingested was also exported from the system, which is a big ask when we are dealing with billions of records.

Due to time and cost, it’s only not possible (or certainly not cost effective) to check the entire contents of every record. However we didn’t need to. This is because we are building on top of the structure and content tests that we have already done, meaning we are already confident in the data within the actual records, but we do need to ensure that all the records have been ingested, processed and exported.

To do this, we used import and export manifests. This means as we processed the ingested records both in to and out of the system, we were creating lightweight CSV files as a log of all the records that were successfully ingested. Each line in the CSVs would contain a | delimited line (we used a | because some of the record’s had commas in) that had the following information about a record:

ID -> the id field of the record JSON
Timestamp -> the timestamp within the record (rather than processing time)
Type -> Imported record or exported record

This meant the resulting files were very small and allowed us much more manageable amounts of data to compare.

When you have the two sets of manifest files, there are many ways to compare them and it will depend on your context. We were working in an AWS system so it made sense to use an AWS Glue job in the form of a pyspark script to load all the files and compare the data inside. We then produced an output based on the following things:

Are there any IDs that are in the export but not in the import?
Are there any IDs in the export and the import with different timestamps?

We also used this to retrieve statistics about the number of ids we imported and a few other small things but the two things about were the ones that told us about the quality of the volume of data.

Given the answer to the above two questions is no then we are now at a stage where we are very confident that every single record we have been asked to ingest is ingested correctly and is available to export from the system too.

Conclusion

I’ve looked at structure, content and volume here as the main areas to focus on with data validation and this has served me well in previous data projects. The key is that each area of validation builds on the ones before it and they all work together to give a level of confidence in the data. This is in a similar way to how you might create unit tests for an application to give a certain level of confidence and then build on these, rather than repeat them, with component tests and so on up the layers. None of the layers themselves give us the confidence we need for our validation, but when combined we have that high level of confidence we are looking for.

Musings from a remote pairer

2026-07-26T20:25:36Z

Over the past couple of months I’ve been double a lot of remote pairing for obvious reasons (if you are reading this in the future google “COVID-19”..). I feel like I’ve learnt a lot along the way regarding how to get the most out of remote pairing sessions, both individually and for the pair or team. I wanted to share some of the things to do and things not to do with you.

DO.. remote pair!

I wanted to start off by saying that I’ve found the remote pairing in the main very rewarding and would encourage people to do it. I’ve ended up remote pairing not just on writing code, but peer reviewing design documentation, writing user stories and many other things. It’s a rewarding activity and is a fantastic way to get to know your colleagues better.

DO.. embrace different ways of working

One of the biggest things that I realised early on was that everyone is an individual when it comes to how they code, how they word user stories and how they work in general. Remote pairing gives you the opportunity to see this up close and it should be embraced. There may be many small things that are different to how you work and remote pairing gives you this insight which may help you to understand the person you are pairing with better. It’s very tempting to point out where you believe someone should do something different but I would encourage you not to do this too often. If you genuinely see something they are doing where you think you can help, then by all means talk to them about it, but it’s also important to recognise they might not want to have their ways of working questioned too much. It’s vital to be sensitive to the feelings of the person you are pairing with.

DO.. take turns leading

It can be very tempting to lead the session too much, especially when the subject is something you may have worked on before. A lot more people learn from doing, rather than seeing, so if you are in the position of helping others to learn then allowing them to lead the session will help it sink in easier. This is vital if you are consulting and part of the gig is to help people progress and learn a product that ultimately they will be maintaining in the future. I try to swap the person leading after a set interval as much as possible. Sometimes this feels more disruptive and it’s better to wait for a natural break in the activity, but as long as you are regularly swapping the one in control of the screen, that is the important part.

DONT.. try to take over

This is the one thing I probably still struggle with! When I am pairing with someone I find it hard to not just keep pointing on their screen to direct them where to put the cursor or type something! It can be very demeaning to your partner if you keep doing this because they are not a robot to be controlled, but a real person to bounce ideas off. You have to respect their ability and be sensitive when making suggestions. One of the best things about pairing is having people validate the work as you do it and being able to provide a different viewpoint but when you are the person doing that it’s important to not be too forceful or controlling when putting that viewpoint across!

DONT.. always use a shared IDE

Recently we’ve used shared IDEs on the team via Cloud9 and I’ve used shared IDEs in the part when pairing sat next to someone. They can be very powerful and allow true code collaboration. But they can also lose some of the best bits about pairing as they can split the focus between the pair. When you are pairing through one person’s controls, you have to communicate well with each other and make sure that you talk about the changes you are making, which is a vital part of the learning process. Shared IDEs often don’t encourage that kind of communication and you can end up with two individuals writing code next to each other, rather than together. I do think a shared IDE has its uses, but I think it is best used by people who are experienced already in pairing.

Final thoughts…

I’ve really enjoyed pairing during this lockdown and I feel like I’ve got to know the people I’ve been pairing with more. I’ve always been an advocate of pairing (I even won a flat cap for doing pairing once…) so it’s not hard to convince me to do it. But that isn’t the same for everyone and I’ve done a lot of pairing during recent times with people that haven’t done much of it before and in general they have found it a really useful activity.

I would encourage more people to pair and I hope that we don’t stop doing this when we do eventually end up back in the office.

Why I don’t value test certification

2026-07-26T20:25:36Z

Certification for testers seems to have been a hot topic for a while and many testers seem unsure whether it’s the right thing to do for their career or not.

If you want to work in the USA or India as a software tester then you will almost certainly need an ISTQB qualification. In that case it may be useful for your career but it feels very much like a box ticking exercise at that point.

I’ve recruited software testers for many organisations in different industries and countries and I never value certification as part of the application.

Times, they are a’changing…

The software industry is a fast paced ever changing industry and software testing as a discipline is no exception. You can get some test certification that you never have to retake! Given the pace of change in the industry how can this mean you have learnt knowledge of modern software testing? Given the time and effort put in to creating and arranging the courses many ones I’ve seen are out of date at the time you take them. They simply can’t react fast enough to keep the tests up to date with the industry.

Passing the test

Test certificates will verify that you have a certain amount of knowledge on models and techniques which allow you to pass the tests. They don’t verify that you can apply this knowledge in the right situations or that you understand why these models exist. It’s a similar problem to certification in other areas as you have proved you can pass a test only.

Skills vs behaviours

Test models and techniques are entirely skills based and these are the things certification teaches you. But skills are not a highly valuable asset when I’m looking for new testers. Ultimately technologies, techniques and models can be learned and they change quickly. Behaviours and mindset on the other hand are much harder to learn and change. The behaviours an individual has will determine how well they build relationships, work in a team, lead people, work with clients and so on. This is a much more valuable thing to demonstrate to a potential employer.

I would rather see evidence on an application as to how someone has spent time working on behaviours such as empathy instead of skills learnt.

Forced certification

I’ve heard of and worked with (not worked for, I’m not doing that) companies that force new testers to get ISTQB certification within 6 months of working there if they don’t have it already! Not only does this seem a waste of time and money but as shown above it doesn’t necessarily make them a better tester so no one benefits. Except ISTQB of course.

Should I ever get certified?

Ultimately whether you want to undertake studying for some test certification comes down to your aims. Do you specifically want to work in a country or for a company that requires certification? If you do then it may be worthwhile.

However if you want to become a better tester then reading books about case studies, participating in and learning from the community, and working on your personal skills and behaviours are much better ways of spending your time in my eyes.

If you definitely do want certification, then think about the one that would benefit you most. Is it worthwhile getting something like ISTQB or would you be better learning something like a cloud technology and studying for an AWS certificate or similar?

I’d never get certified to prove you are a great tester, because in my eyes, it doesn’t.

<p class="archive-note">Added later: how this preference for judgement over dogma now shapes the way I hire and lead is in <a href="https://stevenburton.tech/blog/what-testing-taught-me-about-running-technology/">What testing taught me about running technology</a>.</p>

The problem with test coverage

2026-07-26T20:25:36Z

“Test coverage is useless”

Or…

“Test coverage is everything”

Two opposing extremes which I’ve heard recently and many times through my career. As with most things, I think the truth lies somewhere in between.

What is test coverage?

When different people hear test coverage they have there own opinions as to what that means. But it may be entirely different from the measurements that are in place! There are plenty of ways you are measure coverage. You can measure coverage via the requirements – i.e. percentage of acceptance criteria that are covered or feature coverage, but more commonly when it comes to automation, you measure coverage via the the code itself.

Let’s explore some of the common types of code coverage and the issues inherent within them.

Line coverage

You can simply measure the number of lines that are executed in the product code when you run your automated tests (line coverage).

def foo(x, y):
    bar()
    if x == 1 and y > 1:
        bar()

So to get 100% statement coverage you just need one test:

x = 1 and y > 1

What is the issue with line coverage?

If the lines have multiple decisions and multiple possibilities, you do not need to test them to ensure 100% coverage. For example in the above method, do we know what happens when y is less than 2? Maybe there’s a bug where this would still pass but we won’t know because we’ve only made one test. It might actually be that any combination of x and y would produce the same outcome! In this case the coverage statistic might cause us to write less automation than we would of simply because we’ve hit 100%. Can’t beat 100%, right?

Decision Coverage

You could measure the decision branches – i.e. wherever there is a decision made by something like an if statement or a switch statement you ensure that every possible branch of the calculation is covered.

def foo(x, y):
    bar()
    if x == 1 and y > 1:
        bar()
    else:
        baz()

So to cover the above code, you would need two tests:

x = 1 and y > 1
x != 1 and/or y <= 1

What is the issue with decision coverage?

Let’s consider the above code again. Is this scenario it’s a simple if statement, but what if we made the if statement more complicated, like below:

if (x == 1 and y > 1) or z % 2 == 0:

Now we are adding another layer of complexity where we keep the old check but also go in to this branch if “z” is an even number. We would really need to check this with our unit tests as we now have multiple logic possibilities that would cause a TRUE or FALSE result for this if statement. However the above change does not change our coverage percentage. This is because in branch coverage as long as each branch is covered once, then it counts. So our existing two tests will still ensure 100%. Again we are in the situation where we can get top marks while still potentially missing covering some actual logic.

Condition Coverage

This is a type of coverage where you ensure that every condition within your decision statements evaluates to TRUE and FALSE to count. Let’s consider our code again this time with the revised if statement:

def foo(x, y):
    bar()
    if (x == 1 and y > 1) or z % 2 == 0:
        bar()
    else:
        baz()

To get 100% coverage now we need to ensure that both atomic parts of the if statement evaluate to TRUE and FALSE. The atomic parts are the if statement logic broken up by the logical operators of and/or/not. So we have three atomic parts in our statement:

x ==1
y > 1
z % 2 == 0

In order to get 100% coverage we need to have tests that make sure each part evaluates to TRUE and FALSE. However we can cover multiple atomic parts in each test, meaning we can still get 100% coverage with two tests:

x = 1, y = 1, z = 1 (TRUE, FALSE, FALSE)
x = 2, y = 2, z = 2 (FALSE, TRUE, TRUE)

What is the issue with condition coverage?

As you can see we have increased the complexity of the statement but still covered it with two tests. There are many more combinations we could and probably should try. We actually have no guarantee with this type of coverage that the block within the statement has been executed, just that the conditions themselves have been executed.

Common concerns

We’ve seen issues specific to certain types of code coverage, but there are also some common pitfalls with code coverage in general.

Quantity not quality

Test coverage of any form is able to tell you the amount of something, whether that is lines covered, requirements hit, decision branches explored or something else. However it does not tell you the quality of the checks you are performing. You may end up executing every line in your unit tests for example, but missing vital checks anyway.

Let’s say you need to check this method:

def multiply(number_one, number_two):
    return number_one * number_two

Well, that’s simple and we can get code coverage to 100% by doing a test that sends in 1 and 2 and ensures 2 is the response. Awesome! 100%!

But wait… what happens if we pass in null for one or both of the numbers? What happens if we pass in zero? What happens with negative numbers? None of these cases are covered, but going with just test coverage will give us the false sense of security that everything is covered.

Playing the system

Another big issue with enforcing some kind of code coverage metrics the tendency for this to encourage undesirable behaviours in order to satisfy the metric. Unit tests which have no value is the perfect example of this. Let’s say I make a unit test that covers this code:

def foo(x, y):
    if x == 1 and y > 1:
        bar()

Really I’d want to stub the thing that calls bar() and ensure that the method is actually called when I want it to be. But I don’t _need_ to do that in the test to pass the unit test metric.

Consider the following test method:

def test_bar_is_called_given_valid_details():
    foo(1, 2)
    assert(true)

Albeit this is a crude example, but here I couldn’t be bothered creating any kind of stub, so I’ve not bothered. This test would happily pass and will happily hit 100% coverage of the foo() method (well, for line and decision coverage).

This is one example on one method and so it might be unlikely to happen but when you are writing lots of code and you are being forced to hit a measure of 100% or similar coverage this kind of behaviour becomes more and more commonplace when delivery pressures are ramped up.

So don’t bother with code coverage?

Some people would say this, but it isn’t what I would advise. Code coverage can absolutely be a useful metric when used in the right way.

Understand your metric

You need to be aware of what exactly you are measuring with your code coverage. Make a decision as a team what kind of coverage you want to measure and understand what that means and what is and is not included within the coverage.

An indicator, not a target

The bad behaviours and common problems often stem from enforcing a particular amount of coverage on to your code, for example failing the CI build if code coverage is below a specific number. Given the problems that come from this, there is not really any upside from using coverage in this way.

Instead, use it as an indicator that something might be an issue. If one area of code has 80% line coverage and another area of code has 20% then it’s probably worth taking a look at the area with lower coverage. There might be a good reason for this but the coverage metric can give you a handy indicator to look there in the first place.

So…

“Test coverage is useless” -> Rubbish

“Test coverage is everything” -> Rubbish

“Test coverage can be a useful metric when the measurements are well understood and it is used as an indicator of potential risk areas” -> Boring, but true

<p class="archive-note">Added later: judging work by the difference it makes, rather than by a number, is one of the habits I trace from testing into running technology in <a href="https://stevenburton.tech/blog/what-testing-taught-me-about-running-technology/">What testing taught me about running technology</a>.</p>

Unbloating the UI automation

2026-07-26T20:25:36Z

I’m not entirely sure “unbloating” is a word. In fact I’m 95% sure it’s not. Regardless of that, this is about taking those huge, automation packs for the UI and chucking them in the bin as soon as possible.

So, let’s examine the biggest issues with UI automation packs first.

Maintainability

Even with the best CI system set up in the world, the creation of new UI tests can only be pushed so far left. Although the tests themselves can be created as skeletons and even the basic structure and commands, you need to have the components there to finish off the tests and ensure they run as expected. Naturally the UI tests will lag behind the product creation to some extent, which means it’s always a game of catch up. Invariably this means that the UI pack is failing more often than not and failures become commonplace. This leads to not being able to trust the results, whether they pass or fail which means that the pack is ultimately not adding any value.

The ultimate catch all

I think the biggest issue I have with UI automation is the way it is often used as a catch all for covering various risks and issues that should be covered earlier on in the process. The UI pack is often the last to run and the longest pack meaning the feedback loop for any issues is quite large. A lot of scenarios can (and should) be covered further down the stack closer to the code. I’ve seen UI automation used on many occasions as the first place to start adding automation to a product, rather than thinking about a targeted automation strategy and looking to push the automation closer to the code.

Separation of roles

It’s very common to see UI automation separated from the product code in its own repository. While this can have some benefits, the big disadvantage for me is that is makes it seem like a separate task and this does not lend itself to creating the tests as part of a normal feature lifecycle. I also believe that this often leads to developers not getting as involved as possible in the UI test pack.

When automation is part of the product repository and runs when the product builds, it’s easy to encourage people working on the product to create and maintain that automation. If it’s separate and/or in a different language, it’s harder to do this.

My proposal

Remove all the UI automation! Think of that as the crazy goal. Maybe it can’t be achieved, but it can certainly be optimised by reducing the scope and size of UI automation. That’s the aim…now let’s see some ways we can get there…

Push the automation down the stack

This is the biggest one. It’s so common to cover scenarios and functionality at the UI level because it’s not covered elsewhere. Look at the layers of automation you have and examine what your boundaries are at that layer. Have you tested all the enclosed pieces of logic at the unit level? Can you ensure more possible data scenarios and flows through component or integration tests? Are there UI flows that are basically just strings of API calls that could be covered at that layer?

Make a dumb UI

If you are pushing automation down the stack, you can cover most of the functional scenarios and data scenarios. However you will still need to cover the presentation of the data to the user somehow. But rather than a bulky UI automation pack, can you cover some of this in the component testing of the UI? To do this, it helps if you have a simple UI which simply calls endpoints and presents data from those endpoint. Almost all modern languages have excellent automation frameworks allowing you to cover the UI component now. You can write components tests to check the external endpoint connections and contracts and you can write ones to cover how that data is then presented. However if you have a bloated UI with lots of server side logic then it will be very hard to do this.

Introduce continuous testing

An important part of having a good automation strategy is having the automation running as part of a CI system. This way all the automation you introduce at any level will automatically be providing feedback for you. A number of times I have seen CI systems which do not run any automation or ones that some automation but don’t report on it. The result of this is that confidence in the automation is not as high, meaning scenarios get added to the UI automation instead to increase the confidence.

Concentrate on user journeys

A UI pack can easily become bloated by adding defect or edge case scenarios when this logic could be covered elsewhere. One way to ensure you don’t do this is to concentrate on making the scenarios your UI pack covers full user journeys through your system. Take the most used and high risk flows through the system and maybe cover these with a lightweight easy to maintain automation pack. Covering anything else will potentially tip the balance between time and effort to maintain and risks covered the wrong way.

Conclusion

There are absolutely scenarios and products where UI automation makes perfect sense, but it’s hugely overused and misused in my experience.

Removing the UI automation is an aim I always have in any test strategy as this allows faster, higher quality feedback and more time for testing. However this is an aim and it’s not always possible or the right thing to do for the situation.

Pragmatism in Testing

2026-07-26T20:25:36Z

A really important part of being a tester is the ability to be pragmatic when the situation calls for it. It’s important to have principles and values that you are guided by but there are times where you might need to compromise these to get the job done. I don’t think this should be seen as a bad thing, rather an ability to adapt to the situation in front of you. I’m going to examine some of the most common scenarios from my own experience where you may have to compromise and be pragmatic in your thinking.

One caveat first…

In the scenarios below, you should always push back and attempt to influence the situation when it’s possible to do so, but sometimes this isn’t the case. This is very common when working for a client as a contractor or consultant, where you ultimately can suggest and recommend but do not have the final say on the direction the client takes.

Individual/team skills

Every individual and team has a different skillset and this should always factor in to any decisions you make. When you are looking at tools and techniques to use, these need to be maintainable by the team that will maintain them. Of course people can learn new skills and if, in the company you working for, there is an appetite for this you could go down this route. However a lot of companies will not want to spend time re-training individuals or teams and you may have to accept this scenario.

Rules for rules sake

It’s very important in any team to have ways of working and guidelines for doing things but we should be very careful of becoming slaves to these at all costs. We should treat ways of working as guides, but not strict rules. For instance a team might have a way of working that guides the code review and branching process for all code changes committed to the product. But when there is a very small change is this process absolutely necessary to follow? It’s all about risk and whether the team are comfortable will the level of risk is not following a guideline and understand why an exception has been made.

Delivery pressures

Ultimately every team should be looking to deliver things, whether those things are incremental versions of software, documents of some kind, tools or anything else. At times decisions will have to be made about approaches to take and plenty of factors need to be taken in to account, like the time to do something or the cost that approach will take. Every approach will likely change the thing being delivered and the time it takes to deliver that thing. Risk based testing is very important for decisions like this and lends itself to a pragmatic approach when it comes to delivering as soon as possible which the highest quality possible.

Conclusion

In any walk of live I think it’s important to have values you look to follow and this is especially true in testing. A set of principles and guidelines that you look to when you have to make decisions. However being pragmatic is a vital part of a tester’s skills and understanding when and where to stick to your guns and when to push back is not easy but a skill learned over time.

Testing your contracts (5/5)

2026-07-26T20:25:36Z

In parts 1-4 of this series, I’ve looked at what contract testing is, introduced the pact framework and showed how you can use it to create consumer side and provider side code to ensure a specific contract scenario.

Now I’m going to look at building pact in to your pipeline and tips on how to achieve this in an efficient way that doesn’t block or slow product delivery.

The Pact Broker

Pact broker is something I haven’t mentioned before, but it’s an excellent part of the pact framework and it is helps immensely adding pact to a pipeline.

What is it?

It’s a piece of software that sits between the consumers and providers and acts as a store for the contracts themselves. Let’s examine this be re-visiting the pact process:

Consumer runs a unit test which hits a mock and generates a pact json file
This file is passed to the provider
The provider hits the real endpoint and verifies the response against the details in this file

When you have one consumer and one provider, this process could be managed easily enough with some kind of cloud or network based file storage. But what about when you have lots of providers and consumers? Even if you only have one of each, what do you do during development? How does the provider know which are the latest pacts and which are currently in development? You’d either have a complicated file hierarchy or a confusing naming convention or both!

The pact broker provides a simple interface and API that enable your providers and consumers to share pact files, verify the results and display them to the rest of the business. Much more detail on its purpose can be found at https://docs.pact.io/getting_started/sharing_pacts.

How do I run it?

There are many options for this ranging from getting the code and running locally, running in a container or hitting a version hosted elsewhere. Details can be found here https://github.com/pact-foundation/pact_broker.

The best way I have used before is hosting your own broker instance which can then be accessed by all your build systems.

How does it change the process?

If you were to use a pact broker, then the process at its simplest becomes:

Consumer runs a unit test which hits a mock and generates a pact json file
This file is uploaded to the pact broker
Provider builds and downloads pacts from the pact broker where it is a provider
The provider hits the real endpoints and verifies the responses against the details in these files

Adding pact to your pipeline

Now that we know a bit about the pact broker, we can look at integrating this in to our CI pipeline. It can be done without the pact broker, but the broker is easy to use and adds a lot of visibility to the process. We will look at this in a technology agnostic way so that regardless of the languages and technologies you use for your services and CI tools, hopefully you can take something from the process.

Consumer build

The first bit we will look at is the build pipeline for the consumer. At this point we want to ensure that when the consumer builds, it is generating the pact files and uploading them to the broker.

We have seen in the consumer code explanations how to create the pact generation so it runs when the unit tests run so we will assume that this part is already done. So how do we upload the pacts to the broker? This part is called publishing and handily there is a pact command line tool for this. It’s a ruby package called “pact_broker-client” and installation instructions can be found on the GitHub link above.

We can publish the pacts to our broker using the following command:

pact-broker publish "$pactdirectory" -b="https://example-pact-broker.co.uk" -t="$gitbranch" -a=“$VERSION"

This command uses the pact broker utility we installed earlier and publishes the JSON pact files produced by the build to the given broker. Let’s examine the arguments for the command:

Directory -> This is the folder that the pacts are published to. Every language of pact has a default which you can override when creating the pacts, so if you do that, then you need to provide that folder here too
URL -> Simply the url to your pact broker so this must be accessible to your CI system
Version -> This is used to tell the pact broker that this pact is for this version of the consumer

Tag

This is a really important part of the process, so let’s examine it in more details. Tags enable you to group pacts together and allow the provider to only run verified versions of the pacts. This means that during development of a new pact, you can upload it with various tags and not affect any new builds of the provider as it will not run the pacts with those tags. For the consumer CI, the best method we have found is to use your source control branch name as the tag. This means whatever your branching system, if the pact is under development you will have a way of keeping it separate from the live versions of the pacts.

Provider build

Now that we have our pacts publishing automatically to the broker, we will want to ensure that our provider is downloading the right pacts and verifying them during its own build. Brilliantly we don’t need a new command for this. In part 4, when we looked at the provider code, we saw how the provider pact methods run like integration tests, meaning that simply running the integration tests in your CI will run the pacts.

The only thing you want to ensure here is that when you run the integration tests you are downloading the pacts with the right tag. For the CI system, that tag should be the tag you use when deploying to live (see below). This is because we want to ensure at this stage that the provider changes will not break any of the live consumers. We use “prod” for this.

Deploying to live

Now we have the process in the consumer and the provider builds, we need to ensure that when we deploy to live we can also notify to the broker that a specific version of the pact is now live. This pact then becomes the go to pact for that consumer-provider pair to ensure that when a provider changes it is not breaking the live consumer.

How you fit this in to your specific CI system and scripts will of course vary, but either way you can use the pact broker command line utility again using the following command:

pact-broker create-version-tag -b="$pact_broker_url" -t="$environment" -a=“$consumer_name" -e=“$consumer_version"

This call creates a tag on a specific version of the pact. Let’s examine the arguments supplied to the call again:

URL -> Simply the url to your pact broker so this must be accessible to your CI system
Tag -> This is the tag itself – we use the above command in deploying to all environments and the tag is the environment name itself (live is named “prod”)
Name -> This is used to tell the pact broker that this tag is to be applied to this consumer only
Version -> This is used to tell the pact broker that this tag is to be applied to the pact of this version of the consumer only

Visibility

One of the big advantages that the pact broker gives is visibility of the pacts and the results of the pact verifications. The broker hosts a website which allows you to see a number of different things. The best way to investigate is to get a container up and running and have a play around, but the most important area is the home page which lists all the pacts between different consumer and provider pairs that have been uploaded:

Here you can see at a glance all your contracts, the last time they were published and the last time they were verified. This is excellent for any stakeholders who might want to see this kind of information.

Delving further in to each pact you can see the pact itself (the left icon on each line) which shows a physical representation of the latest JSON pact this consumer/provider pair has uploaded.

Clicking the right icon will show you are history of the pact verification:

This screen show every version of the pact that has been uploaded by the consumer and it shows every time that each one has been verified by a provider version. This screen is very useful for the history of a pact and can also show a development history of the pact as you can see executions against branch and in development versions too.

The broker actually works as a REST API too providing you the ability to view all data via API calls and delete if necessary as well. The website contains an API browser where this data can be visualised.

Development process

A big part of getting buy in from the team for contract testing is ensuring that it does not slow down or get in the way of the development process too much. Therefore the process of creating and testing new pacts is important to refine as you go along with the idea of having maximum confidence in the contracts without slowing down development.
This is the process I have used with the most success:

Create the consumer tests on a branch
Publish to broker, tagged to the branch
Create the provider test on a branch
Run against consumer branch tests – repeat 1-4 until passing
Release consumer
Release provider

This process ensures that new pacts are consumer led, but that the consumer won’t break the provider. An important part throughout this process is the communication between the consumer and the provider, which must be good at all times.

Summary

So, this is the end of the ridiculously large five parter on contract testing. We’ve looked at when contract testing is useful, what Pact is, how to write consumer code, how to write provider code and finally how to integrate it in to your CI system.

I think contract testing is an API world is vital and definitely something to look in to if you aren’t currently. It is not right for every scenario but I would definitely encourage you to have a look.

Testing your contracts (4/5)

2026-07-26T20:25:36Z

If you’ve been ready the parts I’ve written so far, you’ll know we’ve gone through what contract testing is, the pact framework and the consumer side code for a specific scenario.

In this part, we’ll be examining the provider side code of the same scenario. I’d therefore recommend checking out the earlier parts if you haven’t yet, or even for a refresher as this part may not have much context otherwise.

As before I’ll be using Java for this example, but there are many other supported languages and Pact provides excellent documentation and examples: https://docs.pact.io/implementation_guides.

Scenario

Let’s remind ourselves of the scenario we are using for this pact:

Consumer

Our consumer is a simple API that orchestrates data from a number of services and consumes from the Employee Service.

Provider

The provider is a REST service that covers Create/Read/Update/Delete (CRUD) operations for the Employee object.

Pact

The pact for this example is:

Given An employee exists with id of 1**
When** I request to view that employeeThen I am returned success and a single employee object

Provider Code

Now that we have refreshed ourselves on the scenario, let’s take a look at the code. The provider side only has the responsibility of running the tests against itself, so the test code is fairly simple compared to the consumer.

Dependancies

You need to add the pact framework for the provider for your chosen technology. For Java, the following package is needed:

<dependency>
   <groupId>au.com.dius</groupId>
   <artifactId>pact-jvm-provider-spring_2.11</artifactId>
   <version>3.5.23</version>
   <scope>test</scope>
</dependency>

This was the latest version at the time of writing, but I’d use the latest stable one when you come to use it.

Provider test class

Let’s create a simple boilerplate class for running the pacts:

@RunWith(SpringRestPactRunner.class)
@Provider("employee-service")
@PactFolder("pacts")
@SpringBootTest(
classes = EmployeeServiceApplication.class,
webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class EmployeeServicePactProviderTestIT {

@TestTarget
public final Target target = new SpringBootHttpTarget();
}

Let’s example the annotations and fields:

We are using SpringRestPactRunner to run this test as the app is a Java Spring Boot app and this works well in this case, other Java runners (like simply PactRunner) are running and what you need will depend on your application under test
Provider – just refers to the provider so we only run pacts for this provider, must match the name used in the pact
PactFolder – tells the test where the pact files are (in this case a local resource folder in the project called “pacts”)
SpringBootTest – part of the spring boot framework and signifies this is an integration test which requires the application to be running when executing
TestTarget – part of the JUnit test runner and tells the test where to hit when running the tests – we give it a SpringBootHttpTarget, which is an addition to the pact framework and tells it to use the running spring boot app as the target

Just having this test class will cause the pact framework to be run as part of our build tests. This means when we build and run tests, pact will look in the given folder and attempt to run those pacts it finds. Currently it will find our pact and if it runs, it will error with the following:

MissingStateMethodException

This error means that the provider has run our pact and the pact is saying it needs a particular state but we haven’t provided a method to create that state.

State Method

So let’s create our method now. All we want to do in this method is use the pact framework to tie in the method to the given state and then set the provider up in that state.

@State("An employee exists with id of 1")
public void anEmployeeExistsWithAnIdOfOne() throws SQLException, IOException {
//mock the service layer
}

There is nothing to it! The reason for this is I haven’t included the mock code because that will be completely different for any application. Although for most applications, mocking a service is extremely simple. The important bit to note is the @State annotation which is the part of the pact framework that ties this method to the given state in the pact from our consumer.

Mocking

Why are we mocking? Well, we are mocking at the layer below the contract because we are not testing this in the contract tests. All we want to test is the contract itself and not the functionality of the provider. Therefore we allow the controller to do it’s work as this is the input point for the consumer and we mock below that as changing this won’t affect the contract. It might affect the returned data values, but we can verify that using unit and integration tests.

Request Validation

There may be occasions when you want to do validation on the incoming request itself – simple data and field checks. If mocking, you have to ensure that this validation is not mocked – this is because it’s a part of the contract. The incoming request structure is a vital part of any consumer contract as it’s the consumer telling you what it will send you.

Summary

So we’ve explored how to create provider code to run your contracts you’ve been given by the consumer and we can now successfully pass our contract tests!

However the approach we have is a bit clunky due to passing files around and not having it in our CI system. In part 5, we’ll explore the pact broker and integrating pact in to a CI system.

Testing your contracts (3/5)

2026-07-26T20:25:36Z

We’ve already looked at contract testing and the PACT framework and in part 3 we’ll be looking at using that framework to create your consumer side contracts.

Language Support

Pact supports many different languages and you can find details and guides for the supported ones at https://docs.pact.io/implementation_guides. I’ve used Pact in Javascript, Java and C# and have chosen to use Java for the examples here as it’s the one I’ve used most recently. The principles are the same, no matter which language you use though.

Scenario

So let’s look at the scenario we will use for this guide:

Consumer

Our consumer here will be a simple API that orchestrates data from a number of services. In this case it will be consuming from the Employee Service.

Provider

The provider will be the Employee Service is a REST service that covers CRUD operations for the Employee object. CRUD is Create/Read/Update/Delete in case it is not a term you have seen before.

Pact

We are going to use the following scenario for our pact:
Given An employee exists with id of 1
When I request to view that employee
Then I am returned success and a single employee object

Consumer Code

So let’s delve in to the code we need for the consumer side.

Dependancies

The first thing we need is to install the pact framework. With Java, this is a matter of adding the following package for the consumer code:

<dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-consumer-junit_2.11</artifactId>
     <version>3.5.23</version>
     <scope>test</scope>
</dependency>

The version is not too important – I’d generally get the latest stable and stick to that unless new features come up you wish to take advantage of.

Initial Class

So, let’s create the boilerplate code. Firstly we’ll create a class and call in EmployeeApiContractTest – it’s good to have a naming convention up front. We’re then going to create two methods, one will be the method that sets up a mock and the request details and expected response and the other will be the one that hits this mock to generate the JSON pact file.

Whatever language you choose, this is the same pattern you will need – something to set the mock up and something to hit that mock and generate the pact file. This pattern is the supported pattern by the framework and in Java is accomplished by the annotations “@Pact” and “@PactVerification”.

Therefore we end up with the following class:

public class EmployeeApiContractTest {

private RestTemplate restTemplate = new RestTemplate();

private final String PROVIDER_NAME = "employee-service";
private final String CONSUMER_NAME = "public-api";

@Rule
public PactProviderRuleMk2 mockProvider = new PactProviderRuleMk2(PROVIDER_NAME, PactSpecVersion.V3, this);

    @Pact(provider = PROVIDER_NAME, consumer = CONSUMER_NAME)
    public RequestResponsePact shouldGeneratePactWhenRetrievingEmployeeWithIdOfOne(PactDslWithProvider builder) {
    }

    @Test
    @PactVerification(fragment = "shouldGeneratePactWhenRetrievingEmployeeWithIdOfOne")
    public void shouldReturnValidResponseWhenRetrievingEmployeeWithIdOfOne() throws URISyntaxException {
    }
}

Pact Mock Setup

Now we need to set up the mock. To do this, we need to set up the request we are going to send, where we are going to send it and the response that we expect back from the provider. We are also going to set a state for the provider so it knows what data needs to exist when we send it this request.

Each supported language has different ways of setting up this information but pact has unless documentation for each.

For Java, you can use a “domain specific language (DSL)” class called “PactDslJsonBody” to create your request and response.

Request

For this particular request, we have no JSON body we are sending in. We will be using the id of the employee we wish to get in the path so we won’t need to use the DSL to create a JSON body for the request. We’ll set the path we are hitting later on, for now we can go straight on to the response.

Response

For the response, will be expecting a JSON body with the following:

A root employee object, containing
- An integer id
- A string first name
- A string surname

We can use the DSL to create this as follows:

PactDslJsonBody bodyResponse = new PactDslJsonBody()
        .object("employee")
        .integerType("id")
        .stringType("firstName")
        .stringType("surname")
        .closeObject()
        .asBody();

As you can see the DSL describes the JSON as if you were writing it line by line. This is a loose description, which means that if there are extra fields we have not specified, these will not break the pact. This is what we want as we have specific the fields in the body we are using, if there is other information it’s not for us to care about. It may matter to a different consumer, but not us.

Another point here is that we do not verify the data in the fields. That’s because it’s not our responsibility to ensure the provider is working correctly, we just need to ensure we are getting the structure of the response we expect or we can’t extract the required data from the response. Therefore we use the “…type” methods, which just check the type of the field and the name of the field. The data could be anything, it’s not going to affect how we use it.

Mock

Now we have our response object which we are expecting, we need to set up our actual mock. This is where we say which endpoint we will hit and the expectations we have when we do that. This is the information that then forms the generated pact JSON:

return builder.given("An employee exists with id of 1")
        .uponReceiving("a request to retrieve the employee")
        .path("/employee/1")
        .body(bodyRequest)
        .method(RequestMethod.POST.name())
        .willRespondWith()
        .headers(headers)
        .status(200)
        .body(bodyResponse)
        .toPact();

As you can see the PACT framework is written in a very readable language so most lines here are pretty self explanatory. The one I want to highlight is the “given”. This is where we describe the state and it is the link with the provider. As we will see in part 4, the provider needs to know this state and understand what it means. I’ll go deeper in to this in part 4, but it’s important to understand that it signifies any state that the provider needs to be in to satisfy this contract.

Pact Verification

Next we need to write the verification code. This is a piece of code which runs like a test – it will hit the mock with the given response and the pact framework the cause a JSON pact to be generated. The “verification” part of this is that it is effectively verifying the code that hits the mock endpoint. I’ll explain a bit more later on but for now, let’s write the verification code.

@Test
@PactVerification
public void shouldReturnValidResponseWhenRetrievingEmployeeWithIdOfOne() throws URISyntaxException {
Integer employeeId = 1;
EmployeeProvider employeeProvider = new EmployeeProvider(restTemplate);

final ResponseEntity<GetEmployeeResponse> response = employeeProvider
.getEmployeeById(new URI(mockProvider.getUrl()), employeeId);

assertEquals(200, response.getStatusCodeValue());
}

Let’s example some aspects of this fairly simple method. The @Test annotation ensures that this test is run along with the unit tests and the @PactVerification annotation is part of the pact framework which ensures that the Pact method we have already written is invoked when we hit the relevant endpoint and the pact file is generated.

Encapsulated Code

An important part of the method above is the EmployeeProvider class. This class is a piece of code which encapsulates the code that calls the actual endpoint we are writing the contract for. This code can then be called by both the pact and the service code. This is very important because if we change the way we call the endpoint, we would want our pacts to fail. However if we don’t use a piece of shared code for the call, this would not happen unless we change the pact too – which we wouldn’t know to do. For context, this is the code within the EmployeeProvider:

@Component
@AllArgsConstructor
public class EmployeeProvider {

private static final String GET_ENDPOINT = "/employee/";

private final RestTemplate restTemplate;

    public ResponseEntity<GetEmployeeResponse> getEmployeeById(final URI urlPrefix, Integer id) {
        return restTemplate.getForEntity(
                urlPrefix + GET_ENDPOINT + id, GetEmployeeResponse.class
        );
    }
}

Asserting the response

Given the point of this test method is to generate the pact which is verified why are we actually performing any assertions on the result? There are two main reasons for this.

Verifying the mock

The first is that we want to ensure we have coded to hit the mock correctly and the easiest way for this test to ensure that is check we have a successful response. If we have, we can be fairly sure we’ve hit the mock and the pact will have been generated.

Unit testing

The second and biggest reason for performing the assertions is that in doing so, we are performing a unit test on the encapsulated code. We are not looking at these tests in place of unit tests on that code as that is not the responsibility of these tests and we’d still want to write quality unit tests for it. However we want to ensure here that if we change the encapsulated code for any reason that our pact test verification also fails. This will ensure that we investigate and change the pact where needed to match the changes we have made if that’s the chosen route. Without any assertion, we could not be aware of a failure and the pact test could stay as is – meaning we’d have a pact file that doesn’t actually represent the contract anymore!

Running the test

With Java, this is nice and simple because the @Test annotation tells JUnit to run the test as it would any other unit test! For the specific test runner you are using, you just need to treat it like a normal test and it will run as a normal unit test and fail if the assertions fail.

Output

The default for Java is to output the pacts to the target folder in a folder called “pacts”. This will change depending on the language (i.e. C# defaults to the bin folder) and can be configured to be a specific folder if needed. Pact has excellent documentation and examples for every supported language.

So, let’s see the output for this pact! Every field should make sense given the code we have written so far, so we won’t go through it in too much details but a few things to note specifically are:

The matchingRules in the response are the matchers we used earlier telling the provider to match on type and not match the data
The generators are what pact uses when hitting the mock in the consumer – because we don’t provide data for the mocked response, it has to generate some that meet the matching rules
The provider state is clearly noted on its own and is the most important part as the provider can’t verify the pact without knowing and coding for this state

{
    "provider": {
        "name": "employee-service"
    },
    "consumer": {
        "name": "public-api"
    },
    "interactions": [
        {
            "description": "a request to retrieve the employee",
            "request": {
                "method": "GET",
                "path": "/employee/1"
            },
            "response": {
                "status": 200,
                "headers": {
                    "Content-Type": "application/json;charset=UTF-8"
                },
                "body": {
                    "employee": {
                        "firstName": "string",
                        "surname": "string",
                        "id": 100
                    }
                },
                "matchingRules": {
                    "body": {
                        "$.employee.id": {
                            "matchers": [
                                {
                                    "match": "integer"
                                }
                            ],
                            "combine": "AND"
                        },
                        "$.employee.firstName": {
                            "matchers": [
                                {
                                    "match": "type"
                                }
                            ],
                            "combine": "AND"
                        },
                        "$.employee.surname": {
                            "matchers": [
                                {
                                    "match": "type"
                                }
                            ],
                            "combine": "AND"
                        }
                    }
                },
                "generators": {
                    "body": {
                        "$.employee.id": {
                            "type": "RandomInt",
                            "min": 0,
                            "max": 2147483647
                        },
                        "$.employee.firstName": {
                            "type": "RandomString",
                            "size": 20
                        },
                        "$.employee.surname": {
                            "type": "RandomString",
                            "size": 20
                        }
                    }
                }
            },
            "providerStates": [
                {
                    "name": "An employee exists with id of 1"
                }
            ]
        }
    ],
    "metadata": {
        "pactSpecification": {
            "version": "3.0.0"
        },
        "pact-jvm": {
            "version": "3.5.23"
        }
    }
}

Summary

We’ve seen how you can write a simple consumer driven pact file using Java and discussed some of the patterns to look out for when we are writing it.

What do we do with this pact file now? We need to give it to the provider so they can verify it. In part 4, we’ll look at the provider side of this pact and creating the code to perform this verification.

Testing your contracts (2/5)

2026-07-26T20:25:36Z

Pact

In part 1, we looked at what contract testing is and the gap it can cover in an automation strategy. In part 2, we’re going to look at Pact, which is the most widely used contract testing technology and how its framework implements contract testing.

Consumer driven

Pact is referred to as a consumer driven framework, which means the consumer is in control of creating the contract and the contract only contains anything that the consumer actually uses.

Pact is no way replaces any communication between a consumer and a provider, in fact it encourages more communication. Consumer driven does not mean consumer controlled. We’ll examine this more when we see about putting pact in to a CI system, but the main point for now is that the consumer is driving the creation of the pact and leading the discussions as they will benefit the most from the pact being in place. It will give them assurances that the provider they rely on will not break their own functionality when it changes.

The Pact framework

I’ve attempted to sum up the pact process using this simple diagram:

Consumer

On the consumer side, the pact framework works by running a mock server where you can add “interactions”. These interactions are mock endpoints which are the endpoints the consumer would be hitting on the provider. The framework has utilities where you set the mock up so that given a particular state and input then as a consumer you are expecting the following output.
This forms the “contract” and when the pact framework mock is hit, the framework automatically produces a JSON file which represents this contract.

JSON Pact

An example of a simple pact is shown below (I’ve had to blank out any sensitive data to the project I’ve taken this from):

This pact is shown within the Pact Broker, which is a utility I’ll go in to in a later pact but for now I’ve used as it shows the JSON in a nice format.
The first part describes the interaction in human readable form – given a state, upon receiving the below request, then the provider will respond with the below response.
It then describes both the request and the response in simple terms.

Request -> The REST method, the endpoint path to hit on the provider and any headers and body that are in the request are shown
Response -> The details here form the checks the provider will perform when it runs this pact and generally consist of a status, one or more headers and a body, but all of these are optional

Provider

The provider uses the pact framework to perform the following actions at build time:

Download or retrieve pacts when it is the provider
For each pact:
- Run itself
- Hit the endpoint in the request part of the pact with details described in the pact
- Check the actual response matches the checks as described in the pact
- Send result (optional)

The key on the provider side is that the pact is checked by hitting the actual endpoint and not a mock, so if the endpoint code is changed then the pact might fail. I say might as the pact only covers the parts of the endpoint that the consumer is using. If the consumer doesn’t use it, it won’t be covered in the pact. I’ll explain this concept further in later parts.

blogs

Understand exploratory testing, and unlock its true value

You Can’t Fix A Bug With A Prompt

Your Test Strategy Was Built for Human Developers. AI Changed the Rules.

From a player to a coach - 5 ideas from the Quality Coach`s Handbook

Launch Your First Vibium CLI Session with Claude Code: A Step-by-Step Guide

Why Use Vibium with Claude Code?

Table of Contents

Prerequisites

Step 1: Verify Your Environment

Step 2: Install the Vibium CLI

Step 3: Install Vibium as a Claude Code Skill

Step 4 (Optional): Register Vibium as an MCP Server

Step 5: Run Your First Automated Session

Troubleshooting

Conclusion

Learn More and Stay Connected

AI roulette: A quick win and a time sink

Saving the hardest for last

Running in Circles

When your tool fails

The Roulette Wheel

Balancing the scales

Getting back into coding after time away

TL;DR

My journey from code to manual and back to code

Sometimes Efficiency Is Just Turning the Suitcase the Other Way

Transitions are Hard

The Neutral Zone

Recovery Isn’t Just for the Hard Weeks

Build the Bridge

I was an AI sceptic (and I’ve got the old blog posts to prove it)

I was an AI sceptic (and I’ve got the old blog posts to prove it)

TL;DR

What sceptical Callum actually found

Responding to my old scepticism

The risks I raised and where they actually stand now

We’ve been here before

If you’re still the sceptic

Data testing areas of focus

Structure

Content

1. Basic content validation

2. Data meaning validation

Volume

Conclusion

Musings from a remote pairer

DO.. remote pair!

DO.. embrace different ways of working

DO.. take turns leading

DONT.. try to take over

DONT.. always use a shared IDE

Final thoughts…

Why I don’t value test certification

Times, they are a’changing…

Passing the test

Skills vs behaviours

Forced certification

Should I ever get certified?

The problem with test coverage

What is test coverage?

Line coverage

What is the issue with line coverage?

Decision Coverage

What is the issue with decision coverage?

Condition Coverage

What is the issue with condition coverage?

Common concerns

Quantity not quality

Playing the system

So don’t bother with code coverage?

Understand your metric

An indicator, not a target

So…

Unbloating the UI automation

Maintainability

The ultimate catch all

Separation of roles

My proposal

Push the automation down the stack