blogs

Coding agents can be evaluated. We just have to evaluate the work.

2026-08-09T15:00:00Z

Stop grading coding agents like chatbots. Here is how to evaluate non-deterministic AI agents using executable contracts and scorecards.

The Invisible Difference: Unicode Normalization in Test Assertions

2026-08-09T13:35:02Z

Every so often a test fails with a diff you cannot see. The expected value and the actual value are printed one above the other, they are character for character the...

Humans Are the New Middleware

2026-08-09T05:41:48Z

A meeting ends. Nobody took notes. The AI did. The transcript lands in someone’s inbox, they glance at the length, forward it to another AI, and ask for a proper document. Four pages come out. They forward that too. The recipient sees four pages, sighs, and asks their AI to summarise it. Three bullets come … Continue reading Humans Are the New Middleware →

Puzzle 40b

2026-08-08T15:59:46Z

Describe how the buttons affect the lamps.

This version has a single meaningful change from Puzzle 40 – you could use diff to find it, you could look at the behaviours, you could think of somewhere I might easily have dropped in a meaningful change that doesn't quite get picked up by typical validation tests. You could ask, of course.

I don't typically put bugs in. This change is intentional, so arguably it isn't a bug. But the behaviour it introduces does, in some sense, break the symmetry of what you might have found in 40. Off we go then.

Currently supported by these lovely people

Julie Gardiner, Joep Schuurkes, Huib Schoots, Ide Koops, Kris Corbus, Alan Richardson, Rob van Steenbergen, Ioana Chiorean, Peter Houghton, Adun Urke, Christine Yen, Pascal Dufour all support me on Patreon.

Help me make more and I'll put your name on this list. There are other perks, too, eventually.

Enjoy this? Support another!

Built by James Lyndsay - @workroomprds

ESP32-C3 supermini

2026-08-08T15:46:06Z

I used one of these recently for a single one-shot timer. For a simple timer with one I/O pin the supermini is even overkill because it has about 8 I/O pins most of them usable for anything and BT + wifi.

I did not use the wifi for my timer and did not disable it, so maybe I should do that now.

Here is a list of pins to not connect to peripherals that might pull them into states the prevent booting

Pin	Affect on the boot up
GPIO2	If low, normal flash boot may fail
GPIO8	If low, flashing/boot may not fail
GPIO9	If low normal flash boot may fail

From below

From above

Simple one-shot timer sketch

void setup() {
  // put your setup code here, to run once:

  // after power up wait a while, turn on output, wait then turn output off and go to sleep
  pinMode(8, INPUT);
  delay(8000);
}
int once = 1;

void loop() {
  // put your main code here, to run repeatedly:

  if (once) {
  delay(1000);
  pinMode(8, OUTPUT);
  digitalWrite(8, LOW);
  delay(300);
  digitalWrite(8, HIGH);
  pinMode(8, INPUT);
  once = 0;
  }
}

Possibly the most overkill one-shot way to use the 3.3V of the micro to drive a 3V coin-cell load.

Simple project that briefly presses the remote on an action cam. 1. wait a determined interval to allow the action cam to boot 2. press the button for abotu 100ms to start recording This works because the ESP32-supermini is powering the camera as passthrough using the 5V pin to just supply USB power. So we get power at the same time and know roughly how long the camera takes to scan the SD card.

Works with 256 and 512GB Sandisk or Kingston class 1 .

“Developers Should Own Quality” Is a Great Way to Ruin Quality

2026-08-08T04:50:46Z

There’s a mantra that’s been floating around tech teams for the past decade or so like it’s some kind of enlightened truth:

“Developers should own quality.”

At first glance, it sounds great. Empowering. Accountable. Forward-thinking.

But peel back the phrasing, and you’ll find a mess of assumptions, mismatched incentives, and semantic fog that leads to worse — not better — quality outcomes.

Let’s talk about why this statement is dangerously vague, and what we should be saying instead.

First: What Does “Quality” Even Mean?

If you ask five people on your team what “quality” is, you’ll get seven answers.

No bugs?
Fast performance?
Beautiful UX?
Secure APIs?
Accessible UI?
Happy customers?
Clean code?

They’re all right. That’s the problem.

“Quality” is a system property. It’s the result of everything — from design, to research, to testing, to user feedback, to operational maturity.

Saying “developers own quality” is like saying “chefs own nutrition”. Sure, they influence it. But so do farmers, grocers, supply chains, and your own self-control.

What This Phrase Usually Means in Practice

Let’s be honest about how this gets used in real teams.

“We don’t have time for QA, so devs should own quality.”
“We let that bug into prod. Who owns quality on that feature?”
“We don’t do post-launch validation. Devs should just build it right.”

This isn’t ownership — it’s outsourcing blame.

It’s a euphemism for “we aren’t investing in feedback loops, testing infrastructure, or cross-functional alignment — so just don’t mess up.”

Developers Do Own QC — Not QA

Let’s draw an important distinction:

Quality Control (QC):
The stuff that happens during and around the act of building. Unit tests. Static analysis. Code review. These are squarely in a developer’s domain.
Quality Assurance (QA):
The broader system: whether the thing does what it’s supposed to do in the real world, for real users, in all the messy ways they’ll use it.

Developers can (and should) own QC.
QA is shared — and pretending otherwise burns teams out and lets important context fall through the cracks.

“Ownership” Without Scope Is a Trap

Another issue: this phrase offers no clarity of scope.

Are developers now responsible for:

Product requirements?
UX testing?
Compliance?
Load testing?
Accessibility?
Analytics?

If everything is your job, then nothing is.

When devs “own quality” without clear boundaries, you end up with:

Slower shipping (because everything needs to be “perfect”)
Fragile confidence (because “perfect” is undefined)
Diminished collaboration (because you’re trying to be the whole QA department and the builder)

It Also Undermines Specialists

Saying “devs own quality” subtly devalues the roles of QA engineers, UX researchers, testers, and product folks who bring vital perspectives into the quality conversation.

Quality isn’t an engineering-only concern. It lives in design tradeoffs, in user behavior, in support tickets, in operational metrics.

True product quality comes from cross-functional ownership, not consolidation.

What To Say Instead

Here are better phrases, with actual clarity:

“Developers are responsible for quality control.”
They write tests. Use static analysis. Ensure correctness at the code level.
“Quality is a shared outcome.”
Everyone has a piece. Designers own usability. QA owns validation. PMs own alignment. Devs own integrity.
“Shipping confidence comes from system feedback, not just test coverage.”
You don’t get real quality until you can see it, measure it, and adapt to it.

TL;DR

“Developers should own quality” sounds nice but is semantically vague and structurally risky.
It often leads to burnout, overreach, and blind spots.
Developers should own QC, not the full spectrum of QA.
Quality is not a hero’s journey — it’s a team sport.
If you want to build high-quality software, build clarity and collaboration, not maxims and mantras.

Add Type Checking and TSConfig to Bun-Create-Playwright

2026-08-08T03:10:43Z

Now that we have installed bun, Anthropic's package manager, and scaffolded a Playwright framework and closely examined the results, we are going to explore with our Bun-Create-Playwright project ways to check if our code is correct.

The first method we will be exploring is typechecking.

Why Typechecking? As the Playwright.dev / Node.js Introduction mentions:

"[...] Playwright does not check the types and will run tests even if there are non-critical TypeScript compilation errors. We recommend you run TypeScript compiler alongside Playwright.

"[...] Note that Playwright only supports the following tsconfig options: allowJs, baseUrl, paths, references and extends.

"[...] By default, Playwright will look up a closest tsconfig for each imported file by going up the directory structure and looking for tsconfig.json or jsconfig.json. This way, you can create a tests/tsconfig.json file that will be used only for your tests and Playwright will pick it up automatically".

Before we go further down this road...

What is Type Checking?

According to Type Checking in TypeScript: A Beginners Guide, every piece of data in TypeScript is given a "type", and this "type" determines what properties the data has, and what methods are available to it. The types can be things like a Number, String, Enum, Boolean, Array, Object, Type assertions, or others.

Since all data in TypeScript is given a type, it means that the TypeScript compiler, tsc, can check every property to make sure it is being used correctly. It makes sure that the expected properties exist and that the methods and functions are all compatible.

And best of all, you can run these checks in your IDE as you are writing the code!

To run typechecking against our code, all we have to do is execute from the Terminal:

tsc --noEmit

Since we are just kicking the tires of our code to see what happens, we don't need to emit "compiler output files like JavaScript source code, source-maps or declarations. This makes room for another tool like Babel, or swc to handle converting the TypeScript file to a file which can run inside a JavaScript environment. You can then use TypeScript as a tool for providing editor integration, and as a source code type-checker". ( TypeScriptLang.org / noEmit )

The only problem? We need a TypeScript configuration file added to our project before we do that, and as you could see when we examined what "bun create playwright" produced, we don't have one.

If you try to run the "tsc --noEmit" command without a tsconfig.json, you get the message:

Version 6.0.3
tsc: The TypeScript Compiler - Version 6.0.3

TS COMMON COMMANDS

tsc: Compiles the current project (tsconfig.json in the working directory.)

tsc app.ts util.ts: Ignoring tsconfig.json, compiles the specified files with default compiler options.

tsc -b: Build a composite project in the working directory.

tsc --init: Creates a tsconfig.json with the recommended settings in the working directory.

tsc -p ./path/to/tsconfig.json: Compiles the TypeScript project located at the specified path.

tsc --help --all: An expanded version of this information, showing all possible compiler options

tsc --noEmit
tsc --target esnext
Compiles the current project, with additional settings.

COMMAND LINE FLAGS

--help, -h Print this message.
...

--outFile Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output.

--outDir Specify an output folder for all emitted files.

You can learn about all of the compiler options at https://aka.ms/tsc

So, how do we create with bun a tsconfig.json file?

bun init

What does Bun Init Do?

According to the Bun.com / Init Templating docs, It creates:

a package.json file with a name that defaults to the current directory name
a tsconfig.json or jsconfig.json file, depending on whether the entry point is a TypeScript file
an entry point, which defaults to index.ts unless any of index.{tsx, jsx, js, mts, mjs} exist or the package.json specifies a module or main field
a README.md file

It also creates AI Agent rules (disable with $BUN_AGENT_RULE_DISABLED=1):

a CLAUDE.md file when Claude CLI is detected (disable with CLAUDE_CODE_AGENT_RULE_DISABLED env var)
a .cursor/rules/*.mdc file when Cursor is detected, which tells Cursor AI to use Bun instead of Node.js and npm

But we already scaffolded a Playwright project! Do we have to start from scratch and do everything all over again?

No, but we do need to make some modifications:

Does your package.json in your root folder of the project have "main" to be "index.js"? Change that JavaScript extension to ".ts".
Do you have a "jsconfig.json" file? Delete it.
Do you have an index.html file? Delete it.
Then run: bun init

TSConfig.json Achieved! First version!

The resulting tsconfig.json file produced by "bun init", according to Google AI, "configures TypeScript to act strictly as a type-checker while offloading the actual bundling, compiling, and running of your JavaScript/TypeScript code to Bun".

{
  "compilerOptions": {
    // Environment setup & latest features
    "lib": ["ESNext"],
    "target": "ESNext",
    "module": "Preserve",
    "moduleDetection": "force",
    "jsx": "react-jsx",
    "allowJs": true,
    "types": ["bun"],

    // Bundler mode
    "moduleResolution": "bundler",
    "allowImportingTsExtensions": true,
    "verbatimModuleSyntax": true,
    "noEmit": true,

    // Best practices
    "strict": true,
    "skipLibCheck": true,
    "noFallthroughCasesInSwitch": true,
    "noUncheckedIndexedAccess": true,
    "noImplicitOverride": true,

    // Some stricter flags (disabled by default)
    "noUnusedLocals": false,
    "noUnusedParameters": false,
    "noPropertyAccessFromIndexSignature": false
  }
}

Environment setup & latest features

These settings ensure TypeScript understands the latest syntax and runtime environment variables.

"lib": ["ESNext"] -- includes type definitions for the newest JavaScript standard library features (Promise, Array methods, etc.).
"target": "ESNext" -- compiles/type-checks against the latest ECMAScript syntax rather than downleveling to an older version.
"module": "Preserve" -- keeps your import/export statements as written (doesn't rewrite ESM to CommonJS or vice versa); left for Bun to handle at runtime.
"moduleDetection": "force" -- treats every file as a module (adds import/export scoping) even if it has no import/export statements itself.
"jsx": "react-jsx" -- compiles JSX using the React 17+ automatic runtime (no need to import React in every file that uses JSX).
"allowJs": true -- lets .js files be included and type-checked alongside .ts files.
"types": ["bun"] -- only loads Bun's global type definitions (e.g. Bun.serve), instead of auto-including every @types/* package found in node_modules.

Bundler mode

These rules match how modern build tools resolve file paths and handle module definitions.

"moduleResolution": "bundler" -- resolves imports the way modern bundlers do, closer to Node's ESM resolution but more permissive (works with paths, package exports, etc.).
"allowImportingTsExtensions": true -- permits writing import './foo.ts' with the explicit extension (normally disallowed).
"verbatimModuleSyntax": true -- requires type-only imports to be explicitly marked (import type { Foo }), so the compiler doesn't have to guess what to strip at build time.
"noEmit": true -- TypeScript only type-checks; it doesn't output compiled .js files (Bun runs the .ts files directly).

Best practices

These options activate strict type-safety rules to catch potential bugs early.

"strict": true -- turns on TypeScript's full strict type-checking mode (strictNullChecks, noImplicitAny, etc. all at once).
"skipLibCheck": true -- skips type-checking inside .d.ts declaration files (faster builds, avoids errors in third-party type definitions you don't control).
"noFallthroughCasesInSwitch": true -- errors if a switch case falls through to the next case without a break/return.
"noUncheckedIndexedAccess": true -- accessing an object/array by index (e.g. arr[i]) returns T | undefined instead of just T, forcing you to handle the "might not exist" case. Useful in test code that reads array/object results.
"noImplicitOverride": true -- requires the override keyword when a subclass method overrides a parent class method (relevant if you're using Page Object classes with inheritance).

Stricter flags (disabled here)

These are options explicitly turned off to keep the development experience smoother and less strict during early coding phases.

"noUnusedLocals": false -- won't error on declared-but-unused local variables.
"noUnusedParameters": false -- won't error on unused function parameters (common in test fixtures/callbacks where you don't use every argument).
"noPropertyAccessFromIndexSignature": false -- allows dot notation (obj.foo) even on objects typed with an index signature, instead of forcing bracket notation (obj['foo']).

... This is what "bun init" produced. Let's investigate with Claude.ai or Google AI to see if we should now be all set with our Bun + Playwright + TypeScript project...

TSConfig.json: Take Two!

After I ran it by GitHub Copilot, it suggested to change the tsconfig like so...

  "compilerOptions": {
    // Environment setup & latest features
    "lib": ["ES2022", "DOM"],
    "target": "ES2022",
    "module": "ESNext",
    "moduleDetection": "force",
    "types": ["node", "bun"],

But Why, Claude?

"These choices are better because they match the runtime and tooling you actually have, rather than using the most generic defaults.target: "ES2022": this targets a modern JavaScript version that is broadly supported and a good default for current Node/Bun environments. It is more practical than ESNext because it gives a stable, explicit baseline.

lib: ["ES2022", "DOM"]: this tells TypeScript about modern JavaScript features plus browser APIs like window, document, and fetch. That matters for Playwright tests, which often interact with browser-like environments.
module: "ESNext": this matches modern ESM-style projects and works well with current Bun/Node tooling. It is a better fit than older module systems when you are using modern package resolution.
types: ["node", "bun"]: this makes the compiler aware of both Node and Bun globals/types. That is useful because your project is using Bun as the package runner while also relying on Node-compatible APIs.

"In short, the new config is more explicit, more compatible with current tooling, and less likely to surprise you later when you add scripts, browser APIs, or ESM-style imports".

Add the TypeScript Checking to a Script

Okay, let's add the "tsc --noEmit" command to our package.json script:

  "scripts": {
    "test": "playwright test",
    "typecheck": "tsc --noEmit"
  },

Now, when you run "bun run typecheck", the TypeScript Compiler will test out your code.

No errors? No output. No problem!

With our next post, we will see other ways to add ways to review our code.

Until then... Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Puzzle 40

2026-08-07T21:32:06Z

Describe how the buttons affect the lamps.

Currently supported by these lovely people

Help me make more and I'll put your name on this list. There are other perks, too, eventually.

Enjoy this? Support another!

Built by James Lyndsay - @workroomprds

Can’t Stop, Won’t Stop

2026-08-07T04:00:00Z

Let's say you have a long-ish Cypress test and you know a place where it might fail. You want to stop / skip the test commands. What options do you have for stopping Cypress tests?

Imagine these two specs in our project

// cypress/e2e/spec-a.cy.js

describe('Suite 1', () => {
  it('works 1', function () {
    cy.wrap('Hello').should('eq', 'Hello').wait(2000)
    // this command fails on purpose,
    // we want to stop the test before we run it
    cy.wrap(42).should('not.eq', 42)
  })

  it('works 2', () => {
    cy.wrap('Bye').should('eq', 'Bye')
  })
})

// cypress/e2e/spec-b.cy.js

describe('Suite 2', () => {
  it('works b', () => {
    cy.wrap('Hello').should('eq', 'Hello')
  })
})

The first spec has the test "works 1" that fails at the end. There is another passing test in the first spec. The second spec has just a single passing test. If we use cypress open and look at the first test, the test runner shows the commands in the two finished tests.

The first test fails, the second one passes.

Option 1: Hard Cypress stop

Let's try using Cypress.stop before the failing command.

it('works 1', function () {
  cy.wrap('Hello').should('eq', 'Hello').wait(2000)
  Cypress.stop()
  // this command fails on purpose,
  // we want to stop the test before we run it
  cy.wrap(42).should('not.eq', 42)
})

it('works 2', () => {
  cy.wrap('Bye').should('eq', 'Bye')
})

Here is what happens when this test runs in Cypress

Notice that Cypress.stop destroys the Command Log and skips the rest of the tests in the spec file. We lose everything. We cannot even see any successful commands inside the first test.

Option 2: Mocha test skip

We can stop the current test without skipping th rest of the tests in the spec file by using the Mocha test skip command. We just need to be careful about pointing this at the test context object, thus I am using it(..., function () { ... }) syntax and not it(..., () => { ... }) version.

it('works 1', function () {
  cy.wrap('Hello')
    .should('eq', 'Hello')
    .wait(2000)
    .then(() => {
      this.skip()
    })
  // this command fails on purpose,
  // we want to stop the test before we run it
  cy.wrap(42).should('not.eq', 42)
})

it('works 2', () => {
  cy.wrap('Bye').should('eq', 'Bye')
})

This is how the spec runs - the first test collapses, but the second test runs and can be viewed in the Command Log.

Tip: you can avoid using it(..., function () { ... }) syntax and simply grab the Mocha test context object using the following internal cy.state command

it('works 1', () => {
  cy.wrap('Hello')
    .should('eq', 'Hello')
    .wait(2000)
    .then(() => {
      const ctx = cy.state('runnable').ctx
      ctx.skip()
    })
  // this command fails on purpose,
  // we want to stop the test before we run it
  cy.wrap(42).should('not.eq', 42)
})

Works the same way: skips and collapses the first test, but lets the rest of the spec proceed

Option 3: skip the rest of the command queue

There is a better way. Instead of hitting "the emergency brake", we can skip the remaining Cypress commands by writing a little bit of custom code. To remind: Cypress runs the test callback function, which adds all cy commands to a queue (which you can watch yourself, read Visualize Cypress Command Queue). So here is what we can do when we decide to skip the rest of the test: simply skip the remaining commands instead of stopping the test.

Cypress.Commands.add('skip', () => {
  const log = Cypress.log({
    message: 'Skipping the rest of commands',
  })
  let next = Cypress.state('current').attributes?.next
  while (next) {
    next.skip()
    next = next.attributes?.next
  }
})

describe('Suite 1', () => {
  it('works 1', () => {
    cy.wrap('Hello').should('eq', 'Hello').wait(2000)
    cy.skip()
    // this command fails on purpose,
    // we want to stop the test before we run it
    cy.wrap(42).should('not.eq', 42)
  })

  it('works 2', () => {
    cy.wrap('Bye').should('eq', 'Bye')
  })
})

The custom command cy.skip() can be added anywhere and it will mark the remaining individual commands using <command>.skip method.

This approach offers an obvious advantage: you can still inspect the current test and see all completed commands. Of course, you can make this cy.skip custom command as complex as needed: add condition / subject support, or run it conditionally using cy.then(callback) syntax:

cy.get(...)
  .then(x => {
    if (!x) {
      // no need to run the rest of the test
      cy.skip()
    }
  })
// the rest of the test

Happy test stopping!

How Antithesis Found and Explained a Bombadil Race Condition

2026-08-06T22:00:00Z

Disclosure: I’m the original author and lead for the Bombadil project at Antithesis.

Carlos Catala, interning at Antithesis and working on testing our webapps, found a strange crash while I was out camping. The tests he works on runs Antithesis’ webapp inside Antithesis and explores it using Bombadil. That surfaced a particular sequence of events leading up to the following crash (logs are simplified for readability):

state machine error: unhandled transition: Running(10) + ActionAccepted(Click("Filters"))

What does this mean? To understand it, we need to back up a little.

The heart of the browser driver is a finite state machine, an abstract model of a tab in Chrome or Chromium, and how we’re interacting with it over CDP. This is necessary due to how you can only invoke certain commands in certain browser states, or it might hang or behave in other confounding ways. In our state machine, some transitions are valid, and those are handled in a big match (state, event) { ... } expression that returns a new state. Invalid transitions are covered by the last branch that returns an Err with a message like the above.

There’s another important component involved which is called the runner. It decides the control flow of the test, and uses the driver to run the system under test. The driver emits a state, and the runner decides what action to take in that state. It hands back the action to the driver so that it can apply it. The browser driver actually pauses the JS runtime when capturing a state, and doesn’t resume execution until the driver has given back the next action.

Hence, the ActionAccepted event (which is when the driver gets a new action from the runner) is not valid in the Running state. It really is only valid in the Paused state. How could this happen? The driver and runner are supposed to operate in lock-step, where the driver is only in control while the JS runtime is paused. This was the nut to crack.

When Carlos told me about this during my PTO, my camping brain was definitely not helpful, so I let it rest until I was back at work. Staring at logs and code also didn’t help much, except for a poorly conceived theory about what could be going on, mislead by the presence of network faults in one test where we saw the bug. We tried but failed to reproduce this locally, so I decided to find a simpler test subject and, crucially, try to reproduce it in Antithesis.

A Simple Reproduction with Causality Analysis

To reduce possible confusion I wanted to first see if I could trigger the same bug in Bombadil with a simpler system under test. I found a Dockerized version of the-internet, an example application by Sauce Labs made for automated browser testing. Based on what we had seen before, I wrote a very small specification that only performs three actions:

export * from "@antithesishq/bombadil/browser/defaults/properties"; import { weighted } from "@antithesishq/bombadil/browser"; import {  back,  forward,  clicks, } from "@antithesishq/bombadil/browser/defaults/actions";  export const navigation = weighted([  [1, clicks],  [1, back],  [1, forward] ]);

In addition to finding various bugs in the Ruby code, it also immediately reproduced the Bombadil bug! Time to dig in. I went through a few instances and started to see a pattern. In addition, this gave me an excellent opportunity to try out Causality Analysis, a mind-bending feature for learning what behavior increases the likelihood of a bug appearing. Here’s the report I got:

As you may see in the chart, there are three points where the probability increases markedly. Correlating those with the logs (which is easy, you see the bumps highlighted in the log too), I could find the chain leading up to the bug:

Click a link that navigates to /frames (probability goes to 20%)
On that page, click a link that navigates to /iframe (probability goes to 47%)
Go back (I was a bit surprised that this didn’t bump probability noticeably)
Go forward (probability goes to 99.98%)

After the last “Go forward” action, it’s a done deal. Digging into the details in between these points, I could then see how going back to a page which immediately throws an exception, after restoring from bfcache, takes a different path than most page loads do.

The Bug

When going back or forward in history, the state machine immediately transitions to Running, waiting for confirmation by the browser to know what actually happened. Some pages prevent bfcache through JavaScript or HTTP headers, so a history navigation can also end up as a full page load with new HTTP requests. When the page is confirmed to have been restored from bfcache, the page and thus the state machine model skips the Loading state entirely, and goes straight to Running.

In this case, that was followed by the exception being thrown immediately (you can see it in the bottom of the screenshot above), which triggers another path: exceptions thrown while in Running aren’t collected and reported at a later state capture, as is the case for the Loading state — they immediately pause the runtime and trigger a new state read. This in turn let the runner pick a new action to perform. But the CDP event that confirms that the page was restored from bfcache hadn’t arrived yet.

This is a race between the asynchronous communication of runner and browser state machine, selecting the next action, and the CDP event sent by Chromium over a WebSocket confirming the bfcache restore. The unfortunate ordering that triggers the bug is:

Back or forward, restoring from bfcache
Exception thrown on restored page, pausing the debugger and capturing a new state
Runner picking a new action and passing it back to the browser state machine
Confirmation of bfcache restoration coming in, transitioning state machine to Running
State machine applying the ActionAccepted event, which, as we noted in the beginning of this excursion, is not valid when in Running

This sequence causes the crash. It only happens with bfcache-enabled history navigation to a page with an exception being thrown before the confirmation from Chromium comes back. Antithesis’ deterministic fuzzing environment messes with process scheduling and action selection and discovers this sequence reliably.

In Conclusion

I’m not going to describe the fix in too much detail, but it involves a more robust use of the already existing Generation value in the browser state machine. It’s a monotonically increasing number that is used to detect stale timeouts and actions, so that the state machine can discard them. The browser state now carries the current generation, and that is threaded through the runner’s action selection process, so that a stale action (as in step 5 above) can be safely ignored. Additionally, some transitions now increment the generation in cases where all existing timeouts or actions are stale.

I’ve had bugs of this kind before, and understanding them from staring at logs, let alone reproducing them, has been painful. This, in contrast, feels a bit like cheating. Moreover, it extends to race conditions and complicated bugs in the webapps themselves, which makes me very excited about continued work on the Bombadil and Antithesis integration.

More about the testing, less about testers

2026-08-06T19:27:35Z

I’m sure shift-left moved old-school testers task to the developers. Twenty odd years ago, when I started as a tester it was all about the test cases, about coverage and test techniques. We spent loads of time covering basic system functionality, but with shift-left, the test automation pyramid and many other advances the basic tester’s task of juggling trivial system functionality was covered closer to the code. Hence, the basic testing job opportunities declined.

It’s part of my ongoing experience that much “business” testing was better done with actual business representatives than a team of testers. Especially in the context of Software-as-a-service or standard system implementations: SAP ERP, Microsoft Dynamics Finance and Operations, ITIL implementations in ServiceNow and the like. I especially remember a Dynamics implementation for a public data services (yeah, it’s a thing here). The development team was configuration specialists and the “testers” actual public service case workers needing to work in the solution afterwards. My job as a test coordinator was primarily to setup a test case tooling and to follow up that tasks where finalized. Was this about testing – YES! But it was never about the testers. There where no formal test team, only actual case workers who had to juggle their “actual” job in the current system on the side.

The Testing, not the Testers

Recent years work experience in the consulting and outsourcing business has also given plenty of examples of testing tasks, that for historic reasons was usually a fringe in the larger testing spaces: usability, accessibility, compliance and security. Each of these are fields of their own, but could as easily been a space for a tester. Similarly I have experienced yet again over the past year, that was is required to lead testing activities is part project management skills, leadership and communication – less about documents and status reports.

Someone else will do it, 2020

So here we are, in 2026. Facing yet another technology shift. When the dust settles in a few years or so, there will be even less testing jobs about curating test cases. And there will be less formal testers in the delivery teams, as some of the menial testing tasks will have been substituted out by technology. In my part of the IT world of consulting and outsourcing, the signs are there that the team-embedded tester is more about automation and AI-test work, than classic hands-on system interaction testing.

Still plenty of testing will happen, and plenty of testing needs coordination and alignment.

Plenty of solution and system requirements needs verification and validation.

The sooner we step up and leave the “testers” label – the more we can grow the testing.

These are my experiences and “reading of the room” – Your milage may vary.

What happens when you scaffold a Playwright framework and run installed tests using bun?

2026-08-06T17:37:06Z

We covered in the last blog post how to install bun, a new package manager.

In this blog post, we will look into using bun to install a new Playwright framework.

Create the Playwright Framework

Once I installed bun on my Windows PC and was up and running I created a new folder, "bun-create-playwright".

I opened up that folder in VS Code, along with a new PowerShell terminal. In that terminal I entered:

bun create playwright

This activated the interactive Playwright installer that Playwright comes with.

I selected I wanted it to create for me a TypeScript project, placing the tests in the default folder, tests, but I decided not to add a GitHub Actions workflow. I wanted to experiment with using GitLab.
I chose it to set up and install all the browsers for me... which it did... using NPX, part of Node.

Wait a second, when it comes to our Playwright framework, doesn't bun replace node?

No. Bun might be a JavaScript runtime, a package manager, and a bundler shipped as a single software tool, but in our case we are simply using it as a package manager in our Playwright framework.

Bun works alongside Node.js in the Playwright project. We still use Playwright's test runner, a Node.js program. Bun handles installations and run tasks.

Alert: Windows Bug!

Initially, that wasn't enough to install Playwright's internal browsers. There is a bug with bun + Playwright Windows where we also need to:

bun add --dev @playwright/test
bunx playwright install

That was enough for the browsers to install.

Create the Bun Lock File

After the browsers are installed, we need to install bun in our project, to create from the package.json a bun installation.

Run in the Terminal: bun install
Review the newly created file bun.lock.file in the root directory of the project.
Delete the package-lock.json file.

Add the test script to the Package.json

If you look in the package.json file, you can see that the scripts code block is currently empty.

"scripts": {}

Inside this code block, we will be adding:

"scripts": {    
    "test": "playwright test"  
  },

Examine the Included Tests

Two tests will automatically be set up for you:

has title: Navigates to playwright.dev and expects the title of the page to have the word "Playwright".
get started: Navigates to playwright.dev and checks the "Get started" link.

tests/ example.spec.ts

import { test, expect } from '@playwright/test';

test('has title', async ({ page }) => {
  await page.goto('https://playwright.dev/');

  // Expect a title "to contain" a substring.
  await expect(page).toHaveTitle(/Playwright/);
});

test('get started link', async ({ page }) => {
  await page.goto('https://playwright.dev/');

  // Click the get started link.
  await page.getByRole('link', { name: 'Get started' }).click();

  // Expects page to have a heading with the name of Installation.
  await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});

There are three stages you want to follow when you are designing an automated test:

Arrange: Setup things like opening up a browser page.
Act: Go to a web site and see if you can get a link and click on it.
Assert: Expect that the conditions of the tests are met, failing the test if the expected results and actual results do not match up.

import pulls code that lives in another file or package so you can use it in your project. We import two specific named functions, test and expect, from the @playwright/test package (the Playwright Test framework's core library).

test is a function used to define an individual test case.
expect is a function used to create assertions (checks that something is true, and fail the test if it isn't). [ See Playwright.dev / Assertions ]

The { test, expect } syntax is called a "named import", getting specific things out of the package rather than the whole package as one object.

test blocks set up two Playwright tests. Strings such as "has title" sets up the name of the test.

async sets up this test function to be asynchronous, with many tests allowed to be executed at the same time as this one. The browser to run the first "has title" test might not be able to navigate to the page right away. The aync keyword allows the test to pause and wait for any slower actions to be completed before moving to the next part of the test. Once we set up async, we can use await later in the test. Meanwhile, it returns a JavaScript Promise, a placeholder container until we determine the success or failure of the action, when it actually will contain a value. This promise will be either pending, fulfilled, or rejected.

page sets up the pre-defined Playwright page fixture [ See Playwright.dev / Fixtures ]. With this call, before the individual test starts, a fresh isolated page is launched just for this test, so the "has title" test does not interfere with the "get started" test. This page fixture has automatic setup and teardown.

=> is a "hash rocket", representing the "arrow function", separating the earlier parameters of the test with the coming body of the test function.

await returns a Promise, a placeholder, so if we can't do something like go to a page right away, the test won't automatically go to the next line of code and possibly fail. Playwright by default waits 30 seconds for a goto to complete, 30 seconds for an action to happen, but only 5 seconds on an assertion like "expect". If it goes over that timeout, the test will fail.

goto is part of the Page fixture, that automatically handles page navigation.

expect is an assertion. By default, the test expects this condition to be met within five minutes, else the test fails.

toHaveTitle is a built in Playwright page assertion. It allows regular expressions such as /Playwright/ to let the test know that having the word "Playwright" in the title is acceptable. Playwright has many assertions, such as the PageAssertions toHaveTitle, toHaveURL, toHaveScrreenshot(name).

getByRole is a Playwright Locator, that helps you find a web element on the page. Are you looking for a certain alert, alertDialog, banner, button, checkbox, heading, img, link, listbox, menu, menubar, list, listbox, progressbar, searchbox, table, or textbox, tooltip, amongst others? You can try to getByRole. With this you are locating elements by their ARIA role, ARIA attributes and accessible name.

... Note that with locators, you can also getByTestId, getByAltText, getByText, getByTitle, etc.

.click is a Playwright action you can perform.

Once you found an element by a locator, you can .click or .dblclick it, .click({ button: 'right' }) or .click({ modifiers: ['Shift']}), and .hover.
You can .check a checkbox or .selectOption on a radio button, and expect it .toBeChecked.
You can drag and drop an item.
You can fill a textbox, input field, or text area.

.toByVisible is a Playwright Locator Assertion.

Once you found an element by a Playwright locator, you can in the test expect it .toBeChecked, .toBeDisabled, .toBeEditable, .toBeEmpty, .toBeEnabled, .toBeFocused, .toBeHidden, .toBeInViewport, .toBeVisible, .toContainClass, .toContainText, /toHavaAccessibleDescription, .toHaveAttribute, .toHaveId, .toHaveRole, .toHaveText, .toHaveValue.
Each of these methods can be negated chaining a "not" value. Expect a text box .not.toBeEmpty.

Execute the Playwright Tests: Bun Run Test

Now, we should be able to run the tests that come with the initial Playwright scaffolding:

bun run test

The Playwright tests, are automatically run in Chromium, Firefox, and WebKit / Safari, and should pass.

You can see the results if you open up the bun-create-playwright folder in an IDE such as VS Code:

Go to the playwright-report folder
Right click on the index.html file that was produced.
Select "Open with Live Server"

You can drill down into the test to see more information about the tests.

What Does Playwright Install?

The Playwright scaffolder installs:

node_modules folder:

Created by a package manager (npm, yarn, bun, pnpm) when you install dependencies. Contains the actual source code of every package your project depends on, plus the dependencies of those dependencies (transitive dependencies), recursively. It's populated by running an install command (npm install, bun install, etc.), and never edited by hand.

Node's module resolution algorithm looks inside node_modules when you import/require a package by name.

Contains .bin, @playwright, @types. bun-types, playwright, playwright-core, typescript, undici-types. According to Claude.ai:

.bin: Executable scripts for installed packages. This is where npm/bun puts symlinks (symbolic links) to CLI (Command Line Interface) tools so you can run them via npx playwright or through package.json scripts, instead of needing global installation of every executable.

@playwright: Scoped npm namespace. Typically contains @playwright/test, the Playwright Test runner package (test runner, assertions, fixtures, config handling — what you import when you write import { test, expect } from '@playwright/test').

@types: Folder that Bun creates to hold TypeScript type declaration packages published under the @types scope on npm. These packages come from DefinitelyTyped.org, a large community-maintained repository on GitHub where volunteers write and publish TypeScript type declarations for JavaScript libraries that don't include their own. When a package doesn't ship built-in types (common for older libraries or runtime built-ins like Node.js itself), DefinitelyTyped contributors write a separate package describing that library's functions, arguments, and return values, then publish it to npm as @types/<library-name>These give you IntelliSense and type-checking for JS APIs.

bun-types: Type declarations for Bun's runtime APIs (Bun.file, Bun.serve, etc.), letting TypeScript understand Bun-specific globals if you're running/building with Bun instead of Node.

playwright: The full Playwright package, including the browser automation driver and CLI. This is distinct from @playwright/test: playwright is the core library (browser launching, contexts, pages), while @playwright/test builds the test runner on top of it.

playwright-core: The stripped-down core of Playwright without the browser-download step. playwright depends on this internally; you'd typically only see it listed as a top-level dependency if something needs Playwright's automation APIs without auto-downloading browser binaries.

typescript: The TypeScript compiler (tsc) itself, used for type-checking and/or transpiling .ts files.

undici-types: type declarations for undici, Node's HTTP client, pulled in automatically by @types/node.

playwright-report folder:

A folder to store HTML Reports produced after a test run.

According to Playwright.dev / HTML Reporter, "HTML reporter produces a self-contained folder that contains report for the test run that can be served as a web page.

"[...] By default, HTML report is opened automatically if some of the tests failed. You can control this behavior via the open property in the Playwright config or the PLAYWRIGHT_HTML_OPEN environmental variable. The possible values for that property are always, never and on-failure (default)".

test-results folder:

After running the test, it contains .last-run.json which contains "status: passed" with an empty array of failedTests.

tests folder:

example.spec.ts contains the pre-installed Playwright tests we have already covered.

.gitignore:

Want something to not be checked in, committed, and merged to an outside source? Place it here.

Playwright automatically has it so node_modules, local test-results, local playwright-reports, and things in the local .cache and .auth do not get committed.

bun.lock

This lockfile is produced by "bun install". Why have a lockfile? According to Code Protection Hub,

"The entire philosophy of lock file programming revolves around a concept called Semantic Versioning (SemVer). In a manifest file (like package.json), developers specify acceptable version ranges using caret (^) or tilde (~) symbols. For example, ^2.4.1 allows the package manager to download minor updates and bug fixes (up to version 3.0.0).

"While this allows projects to automatically receive security patches, it introduces a severe risk: Dependency Drift.

"Imagine Developer A runs npm install on Monday and receives version 2.4.1. On Friday, the package author publishes version 2.5.0 containing a subtle bug. When Developer B joins the team and runs npm install, they receive the buggy 2.5.0 version. The application breaks for Developer B, but works perfectly for Developer A. This is the classic "it works on my machine" nightmare.
The lock file acts as a historical snapshot. It records the exact version, the registry URL, and the cryptographic hash (SHA-512) of the code downloaded on Monday. When Developer B installs the project, the package manager ignores the flexible package.json ranges and strictly enforces the exact tree defined in the lock file".

package.json:

From HeyNode / What is Package.json:

"If you’ve worked with Node.js before, you have likely encountered a package.json file. It is a JSON file that lives in the root directory of your project. Your package.json holds important information about the project. It contains human-readable metadata about the project (like the project name and description) as well as functional metadata like the package version number and a list of dependencies required by the application.

"Your project’s package.json is the central place to configure and describe how to interact with and run your application. It is used by the npm CLI (and yarn) to identify your project and understand how to handle the project’s dependencies. It’s the package.json file that enables npm to start your project, run scripts, install dependencies, publish to the NPM registry, and many other useful tasks. The npm CLI is also the best way to manage your package.json because it helps generate and update your package.json file throughout a project’s life.

"Your package.json fills several roles in the lifecycle of your project, some of which only apply for packages published to NPM. If you’re not publishing your project to the NPM registry or otherwise making it publicly available to others, your package.json is still essential to the development flow.

"The name field defines the name of the package. When publishing to the NPM registry, this is the name the package will be listed under. It must be no more than 214 characters, only lowercase letters, and it must be URL-safe (hyphens and underscores allowed, but no spaces or other characters disallowed in URLs).

"The version field is very important for any published package, and required before publishing. It is the current version of the software that the package.json is describing.

"You are not required to use SemVer, but it is the standard used in the Node.js ecosystem and highly recommended. For an unpublished package, this property isn’t strictly required. Typically, the version number is bumped according to SemVer before publishing new versions to NPM.

"The license field lets us define what license applies to the code the package.json is describing. Again, this is very important when publishing a project to the NPM registry, as the license may limit the use of your software by some developers or organizations. Having a clear license in place helps clearly define what terms the software is able to be used under".


{
  "name": "bun-create-playwright",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "playwright test"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "type": "commonjs",
  "devDependencies": {
    "@playwright/test": "^1.62.1",
    "@types/bun": "latest",
    "@types/node": "^26.1.2"
  },
  "private": true,
  "peerDependencies": {
    "typescript": "^5"
  }
}

playwright.config.ts:

This Playwright Configuration file is the central configuration file for a Playwright test project. It lives at the root of your project and tells Playwright how to run your tests: which browsers to use, where to find test files, how to handle failures, what reports to generate, etc. Instead of passing a dozen command-line flags every time you run tests, you define the settings once here.


import { defineConfig, devices } from '@playwright/test';

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
// import dotenv from 'dotenv';
// import path from 'path';
// dotenv.config({ path: path.resolve(__dirname, '.env') });

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({

Breakdown of the main components

testDir: points Playwright to the folder containing your spec files.
fullyParallel: controls whether tests within a single file run in parallel workers rather than sequentially.
forbidOnly: a safety check, usually enabled only in CI, that fails the build if test.only was left in the code.
retries: how many times a failed test is retried before being marked as failed. Commonly set higher in CI than locally.
workers: how many tests run concurrently. CI environments often limit this to avoid resource contention.
reporter: determines the output format for test results (HTML report, list in the terminal, JUnit XML for CI integration, etc). You can specify multiple reporters at once.
use: shared defaults inherited by every test, unless overridden at the project or test level. Common settings here:

baseURL for relative navigation
trace for capturing a debuggable trace of the test run
screenshot and video for failure artifacts
headless to control whether the browser UI is shown

projects: defines the different browser/device combinations to run the same test suite against. Each project can also override anything in use. This is how you get cross-browser coverage without duplicating test code.

With the next post, we are going to make some changes to the project per Playwright Ambassador Butch Mayhew's recommendation.

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Doing More With Less: Redefining Value in Software Delivery

2026-08-06T14:41:33Z

Software delivery has never been easy. Today, many teams are expected to deliver more features, faster, with fewer people and tighter budgets. Yet expectations haven't reduced—they've increased. The temptation is to respond by working harder: longer hours, bigger backlogs, saying yes to every request. But that approach doesn't scale. It creates burnout, increases technical debt, and ultimately slows delivery. Doing more with less isn't about squeezing more from your teams. It's about making...

The Second Law of Software Delivery

2026-08-06T08:33:56Z

Why the single plan everyone signed off never had a chance — and what actually holds delivery together

The moment a delivery plan is signed off is the most ordered your project will ever be. Everything after that point pulls against it. A requirement gets interpreted. An architecture decision quietly reshapes what is feasible. Three developers read the same ticket and build three subtly different things. An AI assistant generates a plausible function nobody fully reads. A business analyst reconciles two stakeholders who wanted different outcomes. Each of these is someone capable doing their job well — and each nudges the work a fraction away from the tidy single path everyone agreed to.

This drift is not a failure of discipline. It is entropy: the tendency of any ordered system to slide towards disorder unless energy is continuously put back in. Therefore, the quality of your ultimate delivery when viewed against the plan is congruent on the amount of effort you put in to fight that drift. The single agreed path was never going to hold on its own, and the job of everyone downstream of the plan is less to follow it than to keep noticing where reality has pulled away from it.

The plan describes how the work is imagined. The product reflects how the work is actually done. The gap between the two is not a risk you might eliminate with tighter project management – it is a property you have to design around.

A tended garden, left alone, becomes a wilderness

A garden makes the point better than a textbook. Designed, planted and pruned, it is a small triumph of order over its surroundings. Leave it a single season and nature reclaims it – not out of malice, but out of sheer tendency. Weeds do not need a plan. Order does.

This is not a metaphor imported from physics and forced onto software. Software has its own version of it, observed and documented long before the current debate. Manny Lehman’s laws of software evolution, drawn from studying real systems in the 1970s, held that a system’s complexity increases as it evolves unless deliberate work is done to reduce it – a pull Lehman himself described as an analogue of the second law of thermodynamics. A delivery plan is that garden on the day it is finished: at its most manicured, its most complete, its most persuasive. Sign-off is peak order. From there, the only real question is how much energy you are willing to spend keeping the design legible as the season runs on.

Every handoff is a translation, and translation loses

Software is not built at a single point. It is assembled across a chain of hands, and every link in that chain is a translation.

Requirements capture someone’s intent – imperfectly, because intent is slippery and language is lossy. Architecture interprets those requirements into structure. Developers interpret the architecture and the ticket, and where the ticket is silent they fill the gap with reasonable assumptions that may not match anyone else’s. AI now sits inside this chain too, producing fluent, confident code at a speed no reviewer can fully keep pace with. Business analysts reconcile stakeholders who wanted subtly different things. Testers infer the intent behind a specification that was itself an approximation.

Notice where the loss lives. No individual in this chain is careless; each is competent and doing their job well. The degradation is cumulative, and it accrues in the spaces between roles rather than inside any one of them. Quality, distributed across all these hands, drifts not because someone dropped it but because nobody was ever holding all of it.

This is the system, not the people

It is tempting to read all this as a discipline problem – tighter specs, stricter sign-offs, someone to blame when the delivered thing does not match the drawn thing. That instinct misreads what is happening.

Software delivery is a complex system, not merely a complicated one. Dave Snowden’s Cynefin framework draws the distinction well: a complicated system can be fully analysed and planned in advance by experts, whereas a complex one can only be understood by probing it and responding to what emerges. Delivery, full of interacting people and shifting context, sits firmly in the second category. Richard Cook, writing on how complex systems fail, observed that such systems run in a degraded mode as their normal state – always carrying latent faults, always one coincidence away from trouble, and mostly working anyway.

This is why accountability belongs to the system, not the individuals inside it. W. Edwards Deming made the point decades ago and it has aged well: a bad system will beat a good person every time – he reckoned the overwhelming majority of outcomes are produced by the system rather than the people working within it. Sidney Dekker gave the slow accumulation of small, sensible decisions that ends somewhere nobody intended its name: a drift into failure. The gap between plan and product is that drift. Blaming people for it is like blaming the gardener for autumn.

The cost of a drift nobody authorised

Here is a way to make the loss tangible. Treat each handoff as preserving some fraction of the original intent. Be generous and say 90% survives each step – a high bar in practice. Across seven handoffs, from requirement to production, you retain 0.9 to the seventh power: roughly 48%. On flattering assumptions, more than half the original intent has quietly degraded by the time the feature ships.

That figure is an illustration, not a measurement – a way to picture accumulation, not a benchmark to quote. But the mechanism it describes is real, and it has a price. Picture a £750,000 programme built to a business goal that shifted by degrees across the chain, so that what ships technically works but no longer does the job it was funded to do. The rebuild is one cost. The quarter it took to notice, while a competitor moved, is the larger one. None of it surfaces on a status report, because at every stage the work looked correct – against the slightly wrong version of intent that reached that stage.

AI raises the throughput, not the fidelity

The current wave of AI tooling is often sold as a way to remove this variability. It is worth being precise about what it actually does. AI is a new input source in the chain – a fast, fluent one – and it amplifies whatever is already there. Point it at a team with strong feedback and clear intent, and it accelerates good work. Point it at an ambiguous requirement, and it generates plausible, confident output that makes drift harder to spot rather than easier, because polish reads as correctness.

Where you place it decides everything. Wired into the feedback, sensing where output has diverged from intent, it helps hold the system in order. Bolted onto production as a way to generate more, faster, it adds entropy at machine speed and dresses the result in enough polish to pass review. Same tool, opposite effect.

Testing isn’t the gate — it’s the energy you put back in

The plan was never going to be what happened. From the moment it was signed, entropy began pulling the work away from it – not because the project was run badly, but because that is what complex, human systems do. The plan’s real job was never to be obeyed; it was to give you something to measure the drift against. What decides the outcome is whether anyone is spending the energy to notice that drift and pull the work back towards its intent.

This reframes what testing is for. Testing at the end, as a gate that checks the product against the plan, treats the plan as truth and the drift as deviation to be punished. But you cannot assure quality into a product or delivery that way. Leaving it until a gate at the end is far too late.

THE LEADERSHIP QUESTION “How much of your original intent actually reaches production intact — and who is watching the gap?” Ask your delivery leadership to trace one recent feature from the requirement that funded it to the thing that shipped. If they can show you where intent was checked along the way, you have a system that senses drift. If the only checkpoint was at the end, you have a plan and a hope.

Seen through the lens of entropy, information you gain from testing is the energy you feed back into the system to hold it in order. Good testing is a continuous pulse that reveals, throughout delivery, where the work has bent away from intent while the bend is still small and correctable.

Sources

Ashby, D. Writing on quality as a distributed system rather than a single activity, and on testing as the production of information. danashby.co.uk

Lehman, M. M. (1996). “Laws of Software Evolution Revisited.” EWSPT — the laws of Continuing Change, Increasing Complexity (described as an analogue of the second law of thermodynamics) and Feedback System.

Deming, W. E. “A bad system will beat a good person every time,” and the view that the great majority of outcomes are produced by the system. The W. Edwards Deming Institute, deming.org

Cook, R. I. (2000). How Complex Systems Fail. Cognitive Technologies Laboratory, University of Chicago.

Dekker, S. (2011). Drift into Failure: From Hunting Broken Components to Understanding Complex Systems. Ashgate.

Snowden, D. J. & Boone, M. E. (2007). “A Leader’s Framework for Decision Making.” Harvard Business Review (the Cynefin framework).

Hollnagel, E. (2014). Safety-I and Safety-II (work-as-imagined vs work-as-done). Ashgate.

DORA / Google Cloud. Accelerate State of DevOps research programme, on fast feedback and delivery performance. dora.dev

Introducing bun, a new package manager and JavaScript runtime environment

2026-08-06T01:58:25Z

New job? New toolset to explore! In this post I'll be investigating a new JavaScript package manager and runtime environment called bun.sh.

You may have noticed on Playwright.Dev's Getting Started / Installation section there are three different ways to install a ui or api test automation framework with the latest version of Playwright:

npm init playwright@latest
yarn create playwright
pnpm create playwright

For this blog post I'll be investigating a fourth way, the new toolset they just started using at work, bun.sh.

bun create playwright

What is npm, yarn, pnpm, and bun? These are called package managers.

What is a package? Developing a project, you don't have to figure out how to code everything yourself. You can include in your projects outside libraries, or "packages" of code from from the public JavaScript registry npmjs.com.

The problem is that these packages then need to be downloaded, installed, managed, and updated whenever they are updated. JavaScript projects can use a package manager, such as bun, to help this process.

When putting together a Playwright automation framework based around JavaScript and TypeScript, Playwright uses the Node.js JavaScript runtime environment, to run its test runner, execute automation scripts, and manage its browser instances.(See Nodejs.org / About )

There is a division of labor: Node.js runs these internal functions like running tests. NPM, Yarn, PNPM, and now Bun handle the packages.

Why are there so many package managers? Each has its own specialization. The original NPM is bundled already with Node.js. Yarn is a Facebook toolset that was created when NPM was found to work too slow. PNPM was created to be smaller and therefore faster. And bun was built for quick startup and installation speed, good especially if you have a CI/ CD pipeline that has a LOT of packages to install each time the tests run.

NPM, released in 2010, is the Node Package Manager for Node.js, a JavaScript Runtime Environment. According to NodeJS.org's An introduction to the npm package manager, "The npm installs, updates and manages downloads of dependencies of your project. Dependencies are pre-built pieces of code, such as libraries and packages, that your Node.js application needs to work". NPM comes with NPX (Node Package eXecute) is a command-line tool that runs third-party code, known as packages or dependencies, such as Playwright.

Yarn https://yarnpkg.com/, released in 2016, is Facebook's answer to NPM. Facebook needed a lot of JavaScript packages downloaded at the same time. NPM downloaded them only sequentially, one after another, causing it to be very slow the bigger the project became. According to the Engineering at Meta blog article Yarn: A new package manager for JavaScript, "As the size of [Facebook's] codebase and the number of engineers grew, we ran into problems with consistency, security, and performance".

Pnpm (Performant npm) is another JavaScript package manager, released in 2017. It uses symbolic links to organize the node_modules, increasing performance by decreasing the number of files needed.

Bun https://bun.sh released in 2021 and purchased by Anthropic in 2025, isn't just a package manager like NPM. It is a JavaScript runtime environment that was built to replace Node.js.

In spite of the differences, there are many similarities between these package managers:

Package Registry: All tools fetch packages from the official npmjs.com site.
Manifest File: They all rely on package.json to read and write project dependencies and scripts. They show the name, version, package manager, license, and more (See NPM Docs / Package.json or Yarn Package / Manifest )
Dependency Folders: Each tool downloads packages into a local node_modules folder inside your project directory.
Lockfiles: They use lockfiles to ensure reproducible builds across different environments, such as package-lock.json. (See NPM Docs / package-lock.json)

There is also one main difference between the others and bun. NPM, Yarn, and PNMP are all built on top of Node.js. Bun is built using an entirely new language interacting with Apple's JavaScriptCore engine, developed by Apple for Safari.

NPM with V8 vs Apple's JavaScriptCore

NPM is built with V8 and Bun is built on JavaScriptCore... But what is V8 and JavaScriptCore?

According to the Codeverse Chronicle article, JavaScriptCore vs V8 Engine, "JavaScriptCore [...] , is the JavaScript engine developed by Apple. It powers the JavaScript execution within Apple's Safari browser and is also utilized in various macOS and iOS applications. JavaScriptCore is an open-source project, making its codebase accessible to developers and allowing for contributions from the community.

"JavaScriptCore is deeply integrated into the Apple ecosystem, which means it's optimized for Safari and other Apple products. This tight integration ensures excellent performance and compatibility within the Apple ecosystem.

"[The] V8 Engine is developed by Google and is widely known for powering the Chrome browser. It's also used in other projects like Node.js and various server-side applications".

About Bun

Attempting to construct a Minecraft-like game, Jarred Sumner found Next.js painfully slow when trying to save and test his work. Bun initially interacted with the JavaScriptCore engine using the Zig programming language, but converted it to the Rust programming language in 2026.

Installing Bun: https://bun.com/docs/installation
Creating a project: https://bun.com/docs/runtime/templating/create
Running a project: https://bun.com/docs/runtime
GitHub site: https://github.com/oven-sh/bun

According to the GitHub README for Bun, "At its core is the Bun runtime, a fast JavaScript runtime designed as a drop-in replacement for Node.js. It's written in Rust and powered by JavaScriptCore under the hood, dramatically reducing startup times and memory usage.

"The bun command-line tool also implements a test runner, script runner, and Node.js-compatible package manager. Instead of 1,000 node_modules for development, you only need bun. Bun's built-in tools are significantly faster than existing options and usable in existing Node.js projects with little to no changes".

There is one more thing about Bun... it was purchased by Anthropic to be used in Claude Code.

From the press release, Anthropic acquires Bun as Claude Code reaches $1B milestone:

"Bun is redefining speed and performance for modern software engineering and development. Founded by Jarred Sumner in 2021, Bun is dramatically faster than the leading competition. As an all-in-one toolkit—combining runtime, package manager, bundler, and test runner—it's become essential infrastructure for AI-led software engineering, helping developers build and test applications at unprecedented velocity.

"Bun has improved the JavaScript and TypeScript developer experience by optimizing for reliability, speed, and delight. For those using Claude Code, this acquisition means faster performance, improved stability, and new capabilities. Together, we’ll keep making Bun the best JavaScript runtime for all developers, while building even better workflows into Claude Code.

"[...] Bun represents exactly the kind of technical excellence we want to bring into Anthropic," said Mike Krieger, Chief Product Officer of Anthropic. "Jarred and his team rethought the entire JavaScript toolchain from first principles while remaining focused on real use cases. Claude Code reached $1 billion in run-rate revenue in only 6 months, and bringing the Bun team into Anthropic means we can build the infrastructure to compound that momentum and keep pace with the exponential growth in AI adoption."

"As developers increasingly build with AI, the underlying infrastructure matters more than ever—and Bun has emerged as an essential tool. Bun gets more than 7 million monthly downloads, has earned over 82,000 stars on GitHub, and has been adopted by companies like Midjourney and Lovable to increase speed and productivity.

"The decision to acquire Bun is in line with our strategic, disciplined approach to acquisitions: we will continue to pursue opportunities that bolster our technical excellence, reinforce our strength as the leader in enterprise AI, and most importantly, align with our principles and mission".

How to Install Bun

According to the GitHub README for Bun, you can install Bun on your MacBook using HomeBrew, with Curl, with NPM, or pulling down an image from Docker.

On Windows, you can use PowerShell. Since I use a Windows PC at home, I'll install it that way:

powershell -c "irm bun.sh/install.ps1 | iex"

Restarting my PowerShell Terminal, checking that bun was installed correctly:

```
bun -version
```

... And it replied I am using version 1.3.14. Wonderful! It's alive! It's .... alive! Had the installation not gone successfully, it would have errored out.

In the next blog post, we will figure out how to use bun to create a Playwright framework.

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Faker.js for Test Automation: Generate Realistic Test Data That Actually Catches Bugs

2026-08-05T22:00:00Z

Every test suite has a dirty secret: hardcoded test data. You know the pattern. Every test uses john@example.com, the password is always Password123, and the shipping address is 123 Main Street. These comfortable defaults create a dangerous blind spot in your test coverage. Real users have names with apostrophes, emails with plus signs, addresses with...

The post Faker.js for Test Automation: Generate Realistic Test Data That Actually Catches Bugs appeared first on Software Testing & Automation.

Don’t Just Use AI to Build Faster. Use It to Ship Faster.

2026-08-05T20:53:31Z

Coding agents accelerated software creation. The next advantage is using AI to generate the evidence and feedback that let teams deploy with confidence.

This is Part 1 of a series for engineering managers and CTOs about what AI has already changed about QA, engineering ownership, and the path from generated code to production value.

Your team can create software faster than ever. A developer can describe a feature and watch an agent produce code, tests, documentation, and a pull request in minutes. Leadership sees an explosion of activity and expects a matching explosion in production output.

That second explosion is not happening yet because most companies are applying AI to only one part of delivery. The code appears quickly, but reviews still grow, changes still wait, and green test suites still fail to answer the questions that matter. The opportunity now is to apply AI to the work between generated code and a confident production decision.

AI Accelerated Only One Part of Delivery

Most AI coding programs measure what AI makes easy to see: generated code, accepted suggestions, completed tasks, pull requests, and implementation time. Those measures prove that the organization can produce candidate software faster. They do not prove that software reaches customers faster or creates more value.

Generation is only the first part of delivery. The organization still has to understand the change, confirm the product intent, identify broken contracts, collect credible evidence, release safely, observe the result, and recover from failure. AI compressed implementation and exposed everything around it as the next opportunity for acceleration. Generated output becomes valuable when the rest of the system can confidently move it into production.

The New Queue Is a Confidence Queue

AI reduces the queue before implementation, but the queue does not disappear. It moves. Work now waits for someone to establish that it is correct, secure, useful, observable, and recoverable enough to release. The scarce resource is no longer a plausible implementation. It is fast, credible confidence in that implementation.

Generated software often looks finished. The agent wrote the code, added tests, ran the suite, and reported success. The remaining uncertainty is invisible, so teams either move slowly because they distrust the output or move quickly without understanding the risk. A better confidence loop makes that uncertainty visible and gives the team evidence it can act on.

Transform QA to Move at AI Speed

Traditional QA was designed around human-speed changes. Developers produced a manageable number of features, automation engineers expanded regression coverage, and manual testers worked against builds that stayed relevant long enough to find and route defects.

AI-generated volume requires a different architecture. A test pass tied to one build starts expiring as soon as it begins. A separate automation handoff cannot match the context or speed of the coding agent. A large regression suite can show that old checks still pass while missing an unstated business rule, privacy exposure, broken integration, or unrecoverable failure.

The answer is not simply more tests. AI can generate thousands of tests from the same incomplete understanding that generated the code. The builder and its tests can agree perfectly and still be wrong about what the customer or business needs. A green suite is evidence, but it is not confidence. QA leaders can help engineering make that distinction and redesign validation around the speed, context, and risk of AI-generated change.

Use AI Across the Confidence Loop

AI can do far more than generate implementation code. It can inspect a change, identify affected contracts, propose risk-based checks, exercise interfaces, analyze failures, compare production behavior, and help turn incidents into permanent regression evidence. Human experts can then focus on product judgment, customer consequences, privacy, accessibility, business risk, and the uncertainty the organization is willing to accept.

This is not about asking the coding agent that built the feature whether it did a good job. Important changes still need independent challenge. It is about using AI throughout the confidence loop: generating evidence inside development, scaling validation according to risk, controlling production exposure, monitoring real outcomes, and accelerating recovery. Confidence is what allows AI speed to survive contact with production.

Shipping Faster Is the Engineering Opportunity

Engineering managers and CTOs cannot delegate this transition to QA and wait for a new test strategy. It reaches into developer workflows, coding agents, observability, release design, production monitoring, incentives, and the definition of done. Those are engineering-system decisions, but QA leaders can bring the risk judgment and product knowledge needed to make them well.

Start by separating generated output from delivered value. Measure the time from an agent’s claimed completion to a production decision. Identify where work waits, which risks have evidence, and which are covered only by hope. Then use AI to shorten that path: generate functional evidence inside the engineering loop, scale validation according to risk, independently challenge important changes, control production exposure, and focus human judgment on questions the builder cannot answer alone.

You already have the ability to generate software faster. The next return comes from using AI to build confidence faster too. The organizations that connect those two capabilities will not merely produce more code. They will learn faster, release faster, and turn more of their AI investment into customer value.

Next in the series: Your Tokens Aren’t Turning Into Value.

If this transition is now on your plate, IcebergQA helps engineering managers and QA leaders transform QA for AI. We work with your existing people, tools, and engineering workflows to build the confidence to ship faster.

— Jason Arbon

Don’t Just Use AI to Build Faster. Use It to Ship Faster.

2026-08-05T20:53:31Z

Coding agents accelerated software creation. The next advantage is using AI to generate the evidence and feedback that let teams deploy with confidence.

This is Part 1 of a series for engineering managers and CTOs about what AI has already changed about QA, engineering ownership, and the path from generated code to production value.

AI Accelerated Only One Part of Delivery

The New Queue Is a Confidence Queue

Transform QA to Move at AI Speed

Use AI Across the Confidence Loop

Shipping Faster Is the Engineering Opportunity

Next in the series: Your Tokens Aren’t Turning Into Value.

— Jason Arbon

The Integration Seam

2026-08-05T14:35:48Z

Awhile back I posted my call to action posts: Testers: Act Like a Developer. Since that time, I’ve done a few posts on what that looks like specifically and why it matters. This post is another one of those, showing the very early stages of a project.

Previously I talked about the integration pact, which was essentially the idea that problems live at the seams. Something pretty much any tester becomes aware of very quickly. This post is another way of describing the seam.

I spent part of my morning doing something that should have been boring: setting up the scaffolding for a new project called quendor-rs, which is an offshoot of my quendor-py, which is an offshoot of my quendor. I was not writing the software itself yet. Just the plumbing around it. Commit conventions, enforcement hooks, CI checks, merge policy. The kind of thing every project needs and every tutorial makes look like a fifteen-minute task.

It took the better part my morning, and almost none of that time went to the work itself. It went to caveats. “One more thing about this.” “Oh, wait, that reveals this other thing.” By the end I was asking a question I know every developer has asked: why is everything like this? These ecosystems have existed for years. Shouldn’t they be sorted out by now? It’s one thing to know developers ask that question. It’s another to experience why they ask it. And I think testers are well served by knowing exactly that.

I ended my investigations with an actual answer, and it was one I came to long ago since I’ve been developing and testing for a long time. When I first ran into this, way back in the primeval era when dinosaurs still wore polyester, it changed how I look at this kind of friction. But the answer only makes sense against the specifics, so let me show you the specifics first.

The Setup

The project I’m working on, in all of its incarnations, is a Z-Machine interpreter written in Rust. The Z-Machine is the virtual machine that runs Infocom-era interactive fiction. That detail mostly doesn’t matter here. What matters is what I wanted from the project’s process, before the first real line of code:

Commit messages following the Conventional Commits specification.
Local enforcement, so a malformed message is rejected at commit time.
CI enforcement, so nothing malformed can land on main regardless of where it was committed.
A pull-request-only main branch with squash merging, where the PR title becomes the commit message.
Automated dependency updates, so the tooling doesn’t silently rot.

Nothing exotic. Every piece is mainstream: git, GitHub rulesets, GitHub Actions, the gh CLI, a commit-lint tool called cocogitto, its companion GitHub action, Dependabot. Each piece is mature, documented, and widely used.

Here’s what happened at every joint where two of those pieces met.

Five Failures, No Bugs

The tool that changed its meaning. The documented way to install cocogitto’s git hook used to be cog install-hook commit-msg. By version 7, that syntax no longer exists; the tool now installs hooks you declare in a config file. Fair enough. Except that running the new form, cog install-hook --all, with no config file present, exits successfully while installing all zero of the hooks you’ve defined. Green check mark, nothing enforced. I only discovered the hook wasn’t installed because I went looking for what the command had actually done. Which was nothing.

The action that changed its contract. The GitHub action for the same tool went from check: true in version 3 to a required command: check input in version 4. A workflow written from v3-era examples and naively bumped to v4 fails on first run. The CLI and the action are siblings from the same project, and even they don’t drift in sync.

The flag that works, except when it doesn’t. The gh pr merge command has a --delete-branch flag whose help text reads “Delete the local and remote branch after merge.” Combine it with --auto, meaning merge when checks pass … well, it turns out the deletion silently doesn’t happen, because gh merely enables auto-merge server-side and exits. So, when GitHub performs the merge minutes later, there’s no process left to delete anything. No warning tells you this. The truth is in the source code, in an early-return guard the documentation has never heard of. Better still: whether the flag works depends on timing: if the checks happen to already be green when you run the command, it degrades to an immediate merge and the deletion works exactly as documented.

The commit nobody wrote. With everything finally wired up, the very first pull request failed its own commit-message check. Every commit on the branch was valid. The failure was a commit that I didn’t write: on pull-request events, GitHub checks out a synthetic merge commit (effectively: “Merge abc123 into def456”) representing your branch as if merged. The lint tool, doing exactly its job, validated that machine-generated message and rejected it. Two correct behaviors, one broken result. The fix is a config setting that exists precisely because this collision is common enough to deserve
one.

The version nobody chose. The CI action, it turns out, bundles version 6.4 of the lint tool. My machine runs 7.0. Nothing broke. This time. But it means “passes locally” and “passes in CI” are answered by different
software, and someday that gap will likely be a bug report.

Notice what this list does not contain: a single defective tool. Every component did what its maintainers intended. All five failures live in the same place: the joint between two things.

Nobody Owns the Pairs

Here’s the explanation I originally landed on many, many moons ago and it’s the one that holds true today.

Each of these tools is owned. Somebody maintains cocogitto, somebody maintains gh, somebody maintains GitHub’s merge machinery. Within each tool, there’s a test suite, a release process, a place to file bugs.

The pairs of tools are owned by nobody. GitHub doesn’t test its synthetic merge commits against third-party commit linters. The action’s maintainer doesn’t coordinate releases with the CLI it wraps. The gh documentation team doesn’t re-derive every flag interaction when auto-merge semantics change upstream. Each maintainer’s responsibility ends at their own boundary (reasonably so!) which means the interactions between boundaries are maintained by no one.

And the interactions are where I spent my morning. Compose eight tools and you get dozens of pairwise seams, most of which have never been exercised by enough people to have their edges filed down. There is no mechanism by which a bazaar of independent maintainers converges on “sorted,” because sorted-ness is a property of compositions, and a composition belongs to the person doing the composing.

Which means, as a tester who dabbles in development, I recognized my role immediately: I was the integration test. First execution, production environment, no prior runs. Of course it found defects. That’s what integration tests are for.

That “I was the integration test” part is something that I believe is front and center to the relationship between developers and testers and, thus, between development and testing.

The Cathedral Next Door

Here’s the observation that keeps this from curdling into cynicism. While all of this was happening, one part of the stack produced zero caveats: the Rust toolchain. The compiler, the build tool, the formatter, the linter, the toolchain manager: flawless, all morning. Not one surprise. (That also applied to my Python and Node incarnations of this project.)

That’s not luck, and it’s not because Rust’s tools are simpler. It’s structure. The Rust toolchain is a cathedral: one project owns the compiler, the build system, the formatter, and the linter, so the seams between them are internal: somebody’s job to test before every release. The friction in my morning lived almost entirely in the policy-and-automation periphery around GitHub, which is maximally bazaar-shaped: independent tools, no release coordination, composed ad hoc by each user.

So the feeling that ecosystems should be sorted out by now isn’t naive. The sorted ones exist. They’re the ones with owners. What can’t ever be fully sorted is the open composition space around them. And that’s not a failure of engineering discipline; it’s the price of an ecosystem being open. Anyone can build a tool; no one is required to test it against every other tool. You get the bazaar’s variety and the bazaar’s seams together, or neither.

What This Buys You

Three consolations, all practical.

The cost is front-loaded. Setup day is expensive because setup is when you cross every seam for the first time. The loop I’ll actually live in for my project — edit, commit, open a PR, watch checks, squash-merge — now runs entirely on paths that have been walked and pinned down. Seams don’t need re-crossing daily.

Automation converts ambushes into signals. The two robots I installed exist precisely to keep it that way. Dependency updates turn version drift into visible pull requests instead of silent breakage. CI turns “the composition broke” into a red check instead of a corrupted main. The caveats don’t disappear. They stop being ambushes.

Empiricism is the only documentation that never lies. The recurring method that got me through the morning was: check the source, not the docs, and not my memory. The --delete-branch behavior wasn’t in any manual; it was in an if statement. The hook no-op wasn’t announced; it was discovered by looking at what the command actually wrote to disk. In a cathedral you can trust the manual. At the seams of a bazaar, the only reliable spec is the behavior itself. Which is to say: at the seams, everyone is a tester, whether they signed up for it or not.

That last sentence is probably the whole post. The tools were fine. The pairs were untested. (Or, at the very least, circumstantially and situationally tested.) And untested, circumstantially tested, and situationally tested things, when finally executed, behave exactly the way our profession has always said they would.

WhatsApp Ate My Family Dinner

2026-08-05T07:16:11Z

It is 9:47 PM. Someone somewhere reaches for their phone to send a family member a photo of their dinner, their kid, or in my case, their cats fighting over the same cardboard box. Before they find the chat, a red badge catches their eye. Someone in the office WhatsApp group has tagged them. Client … Continue reading WhatsApp Ate My Family Dinner →