Adventures in Automation

Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report

2026-08-24T22:28:52.253-04:00

So far, we have reviewed how to review code with ES Lint + Prettier and Typecheck, how to set up and run smoke tests, how to run all Playwright + TypeScript tests, and reviewed the HTML report of results.

In this post, we are going to set up a three stage GitLab CI/ CD pipeline that will run against every merge request:

Quality (lint, typecheck, prettier) --> Test ( smoke + regression ) --> Report ( Downloadable )

I haven't used GitLab since when I worked at ThreatStack back in 2020. ( See my blog entry Getting to Know GitLab and How They Test the UI )

GitLab reads .gitlab-ci.yml from the repository root and turns it into a pipeline, a set of jobs, grouped into stages, running whenever you push a change to a repository.

Stages run one after another, and jobs within a stage run in parallel. If one stage fails, the stages after it are skipped.

This begs the question: What is GitLab? What is CI/ CD? Or a pipeline? Or a merge request?

What is GitLab?

GitLab began in 2011 as a side project by Ukrainian programmer Dmytro Zaporozhets, built to help developers work together on code more easily. Over time, it grew into a large open-source DevOps platform. More than just storing code, it handles code reviews, CI/CD (continuous integration and continuous delivery), and package registries. A company called GitLab Inc. was eventually created to turn the project into a commercial product.

What makes GitLab different from tools that only host Git repositories (like a basic Git server) is that it bundles the entire software delivery process into one application. This includes:

Merge requests – GitLab's term for proposing and reviewing code changes before they're merged into the main codebase (similar to a "pull request" in GitHub)
Issue tracking – a built-in system for logging bugs, tasks, and feature requests
Pipelines – automated workflows that build, test, and deploy code whenever changes are pushed

So instead of stitching together separate tools for version control, testing, and deployment, GitLab gives you all of it in one place.

Isn't That Just GitHub?

According to GitLab's "What are the fundamental differences between GitLab and GitHub", "GitHub's security scanning, AI, and advanced CI/CD are native but sold as separate paid add-ons on top of the base platform price [...] GitLab puts planning, source code, CI/CD, security, and deployment into a single application with one permission model, one audit trail, and shared analytics. That means fewer tools to run, fewer integrations to maintain, and one place to apply policies and AI across the whole lifecycle".

GitLab calls all the extra stuff you need to purchase from GitHub a "toolchain tax".

What is CI/ CD?

"CI/CD stands for Continuous Integration and Continuous Delivery (or Deployment). It's a DevOps practice that automates building, testing, and deploying code changes, enabling faster and more reliable software releases.

"CI/CD is best explained as an automated workflow that replaces manual steps with pipelines that build, test, and deploy software reliably.

"CI/CD falls under DevOps (the joining of development and operations teams) and combines the practices of continuous integration and continuous delivery. CI/CD automates much or all of the manual human intervention traditionally needed to get new code from a commit into production, encompassing the build, test (including integration tests, unit tests, and regression tests), and deploy phases, as well as infrastructure provisioning.

"With a CI/CD pipeline, development teams can make changes to code that are then automatically tested and pushed out for delivery and deployment. Get CI/CD right and downtime is minimized and code releases happen faster" - GitLab.com / What is CI / CD?

What are CI/ CD Pipelines?

"A CI/CD pipeline is an automated process utilized by software development teams to streamline the creation, testing and deployment of applications.

"CI represents continuous integration, where developers frequently merge code changes into a central repository, allowing early detection of issues. CD refers to continuous deployment or continuous delivery, which automates the application's release to its intended environment, ensuring that it is readily available to users. This pipeline is vital for teams aiming to improve software quality and speed up delivery through regular, reliable updates.

"Integrating a CI/CD pipeline into your workflow significantly reduces the risk of errors in the deployment process. Automating builds and tests ensures that bugs are caught early and fixed promptly, maintaining high-quality software". - GitLab.com / What is CI / CD?

Just like GitHub, GitLab is free to join, hoping people will use the platform and decide to subscribe for more of its services.

With GitLab you can take your changes you want to make to the codebase of the code repository, try to commit the changes, and make a request to merge the changes into the main branch.

You can set up the CI/ CD pipeline that before a senior developer looks at your merge request to see if it can be approved, the merge request gets tested right when it gets created.

How Do You Create a Merge Request?

Let's say you want to make a change to a code repository to bun-create-playwright.

First, if you haven't already, you need to pull down the code repo:

Do you have Git? Open PowerShell or the Terminal, whatever is your Command Line Interface for the PC or the Mac, and run: git --install.
If you do not have git, install it on you Windows machine or MacBook.
Get the code repository: git clone git@gitlab.com:tjmaher/bun-create-playwright.git

Already downloaded the repo, and want to make sure you have the latest version?

git fetch origin
git checkout main
git pull origin main

Ready to make some updates?

git checkout -b feature/my-new-change

Then make your changes. After that is done, add your changes and create a commit for your changes:

git status // All your files you made changes to are listed in red, since they are unstaged.
git .add
git status // All your changes are now listed in green. They are staged.
git commit -m "This is my commit!"

Changes all set? Push the changes to your remote repo:

git push -u origin features/my-new-change

Now, you will see a URL pointing exactly to the link where you can see your commit in GitLab. Select the button, "Create Merge Request".

Once you that, the Merge Request will be created, and will be run automatically on the GitLab CI/ CD pipeline.. once we create one.

Create the Configuration File For a CI/ CD Pipeline

Want to practice the basics? See Tutorial: Create and run your first GitLab CI/CD pipeline
This walks you through creating a public project, setting up a test runner, and creating a .gitlab-ci.yml file in your root directory, and running a job.

Here, we are going to start reviewing the sample .gitlab-ci.yml file I created for Bun-Create-Playwright, the configuration file used when creating a new pipeline that will process the changes we have made to the code base.

bun-create-playwright / .gitlab-ci.yml

stages:
  - quality
  - test
  - report

https://gitlab.com/tjmaher/bun-create-playwright/-/blob/main/.gitlab-ci.yml

For any CI / CD pipeline created when a merge request is pushed, we are setting up four stages:

Quality: Where we check the format using Prettier, run the linter allowing no warnings to sneak by, and run the TypeScript compiler to do type checking. This stage takes seconds to run.
Test:

We run just the one smoke test on Chromium, checking that we can log into the Login page.
Is the smoke test successful? We then will run ALL the tests, positive and negative, on every platform, on Chrome, Firefox, and WebKit / Safari.

Report: Create a report we can download and examine.

Hidden Jobs: The Quality Template

If we take a GitLab job, and add a dot to the name, it becomes a "hidden" job. Let's say that you have a stage called QUALITY, and you have a substage which checks the format of the code, a substage that runs a linter on the code, and a substage that type checks the code.

Each substage might need to have the same things added to it, such as using the same pre-configured Docker image for bun, version 1.3, so you don't have to manually install bun, its version managers or system libraries.

Or you could take the lockfile as is, produced when dependencies are installed, and use that same exact version on all three substages.

That ,bun_quality could act as a template that could be extended into all three substages.

.bun_quality:
  image: oven/bun:1.3
  before_script:
    - bun --version
    - bun install --frozen-lockfile

Docker is a tool that packages your code and everything it needs -- like libraries and system tools -- into a single box called a container. This container runs the exact same way on your laptop, your teammate's computer, and the cloud server, fixing the classic "it works on my machine" problem

Docker Hub is a public cloud-based registry service and community library of Docker images.

Bun, the package manager, has a Docker Hub it has dubbed "oven". Here, we are going to be using the Docker image of Bun version 1.3.

Before we run any of the substages -- format, lint, or typecheck -- we will in all stages check the version of bun used, print it out to the log file, then install bun using the Bun 1.3 Docker image, from a frozen lockfile.

A lockfile is a "file that lists the direct and indirect dependencies of an application and their version numbers. Its purpose is reproducibility, to ensure anyone installing the application’s dependencies gets the exact same versions". ( GitLab / Terminology Lockfile )
By installing a frozen lockfile, we are locking down all package dependencies in our jobs so we will use the same exact version in all three cases. If the lock file is out of sync from package.json, we will have it fail instead of it silently updating.

By having it frozen, we can guarantee reproducible builds and catch cases where a dependency was added/changed without committing the updated lockfile.

The Quality Stage

Now that we set up the quality template, we can use it for the substages: format, lint, and typecheck.

format:
  extends: .bun_quality
  stage: quality
  script:
    - bun run format:check

lint:
  extends: .bun_quality
  stage: quality
  script:
    - bun run lint:ci

typecheck:
  extends: .bun_quality
  stage: quality
  script:
    - bun run typecheck

Need to add another part to the quality stage? You can just declare it, extending the same template, designating the stage, and adding the script you want to run.

Back in the post How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json we set up the following scripts:

Format:check throws an error when Prettier finds a formatting style error while running the script: "prettier --check ."
Lint:ci uses ES Lint to see if the code has an errors with the script: "eslint . --max-warnings 0"
Typecheck uses the TypeScript compiler to see if there an errors, running: "tsc --noEmit"

This CI Pipeline will execute these three stages. Any errors detected? The build pipeline will be halted.

Introducing: The Official Playwright Docker Image

This GitLab pipeline when running tests, will use the official Docker image that Microsoft has released to run Playwright tests: mcr.microsoft.com/playwright:v1.62.1-noble

Microsoft stores its Docker images in the Microsoft Artifact Registry under the Playwright subfolder at at https://mcr.microsoft.com/en-us/artifact/mar/playwright

The Docker image uses the Ubuntu 24.04 LTS (Noble Numbat) operating system.

Hidden Jobs: The Test Template, Run on Playwright Docker

Just like the Quality stage, the Test stage will receive its own template to be used in its substages.

And, just like the Quality stage, to make sure that each substage is on the same page, we will be using a Docker environment. The Docker image we will be using is Microsoft's official Docker environment for Playwright.

.bun_playwright:
  image: mcr.microsoft.com/playwright:v1.62.1-noble
  before_script:
    - apt-get update && apt-get install -y unzip curl
    - curl -fsSL https://bun.sh/install | bash
    - export PATH="$HOME/.bun/bin:$PATH"
    - bun --version
    - bun install --frozen-lockfile

From Playwright.dev / Docker: "Dockerfile.noble can be used to run Playwright scripts in Docker environment. This image includes the Playwright browsers and browser system dependencies".

Using this image in your CI/CD pipeline guarantees a stable, pre-configured, and reproducible environment for running your automated tests

After that, we are using APT, the Advanced Package Tool to refresh the container's package index, then install unzip and curl.

Once curl is installed, we download Bun's official install script (See Bun.sh / Installation ) and pipes it directly into bash to execute. The flags:

-f fails silently on server errors instead of printing an HTML error page
-s runs silently (no progress meter)
-S re-enables error messages even in silent mode
-L follows redirects.

This installs the Bun binary into ~/.bun/bin.

Then we export bun's install directory to the shell's PATH for the current job session, since the install script doesn't automatically make bun available in a fresh shell (it normally edits .bashrc/.zshrc for interactive shells, which CI runners don't source the same way).

Did everything work?

Print the installed Bun version to the job log. This confirms the install succeeded (the job fails fast here if bun isn't on PATH), and leaves a version record in the CI logs for debugging.

Finally we install the project's dependencies using Bun's package manager. --frozen-lockfile tells Bun to install exactly what's in bun.lock (or bun.lockb) and fail the build if the lockfile is out of sync with package.json, rather than silently updating it, which is the standard practice for CI to guarantee reproducible builds.

Whew! And this will happen on every substage in Test.

The Test Stage

Now, we set up the test stage, first running all tests we have tagged with "@smoke". If that passes, we run ALL the tests.

Everything we set up in .bun_playwright gets pulled right into this GitLab CI job.

playwright:
  extends: .bun_playwright
  stage: test
  timeout: 60 minutes
  script:
    - bun run test:smoke
    - bun run test
  artifacts:
    when: always
    paths:
      - playwright-report/
      - test-results/
      - reports/
    reports:
      junit: reports/junit/results.xml
    expire_in: 30 days

After the clock starts, we are giving it one full hour for all the tests to run.

Assemble the Artifacts!

Whether the jobs pass or fail, we are going to create a report so we can audit the test. We are placing the items produced in:

playwright-report/: This directory holds the main interactive HTML report. It includes the index.html file that lets you visually click through test suites, read error logs, and see execution timelines.
test-results/: This folder stores the raw individual assets captured during test execution. If a test fails, Playwright drops failure screenshots, video recordings of the browser screen, and .zip trace files (which allow you to step through the test execution like a video) into this directory
reports/: This is a catch-all or customized directory often used for alternative report formats. It holds JUnit XML files meant to be parsed directly by your CI platform to display a quick test summary on your pull/merge request page

The Report Stage

On the report stage, we are using a lightweight Linux container called "alpine" to execute the commands quickly. We don't need pre-installed browsers or a JavaScript runtime environment. These reports will be generated whether the tests pass or fail.

This job will be started right after the "playwright" job finishes in the Test stage.

report:
  stage: report
  image: alpine:latest
  when: always
  needs:
    - job: playwright
      artifacts: true
  script:
    - ls -la playwright-report/ test-results/ reports/junit/ || true
  artifacts:
    when: always
    paths:
      - playwright-report/
      - test-results/
      - reports/
    reports:
      junit: reports/junit/results.xml
    expire_in: 30 days

Next, we have the script, where we are going to list the contents of our test results directory. The "-la" will even list the hidden directories that might exist.

Even if one of the directories do not exist, we still want to carry on, which is why we are appending "OR TRUE" to the script.

Finally, we add the reports and XML file to GitLab's servers.

View The Results

There you have it! Now, when you push changes, your code can be checked for quality, and you can make sure the tests all actually run.

Each code change, each merge request, generates a new pipeline. You can view it at https://gitlab.com/tjmaher/bun-create-playwright/-/pipelines

You can also download the reports:

Test Report Artifacts

report:archive: Contains the full, human-readable HTML test reports (like Allure, if set up, or Playwright's native HTML report) compressed into a zip archive.
report:junit: Contains a structured XML file in the JUnit format used by CI/CD platforms to parse, count, and display test success rates directly in the pipeline UI.

Playwright Specific Artifacts

playwright:archive: Stores heavy debugging assets generated during test failures, such as recorded videos, browser screenshots, and .zip trace files that you can load into the Playwright Trace Viewer.
playwright:junit: Holds the specific JUnit-formatted XML results generated directly by the Playwright test runner (@playwright/test) during execution.

You can also drill down into the pipeline to see each stage's results.

With each stage, you can drill down to see the log files for that stage.

And, last of all, you can download an HTML version of the report, in the archive of artifacts.

That's how you set up a GitLab CI/ CD pipeline!

Next, we are going to look into manually kicking off a job, viewing a report on GitLab Pages.

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Until then, Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json

2026-08-19T22:13:32.812-04:00

Earlier, we went over how we could create scripts in the package.json file of our Playwright framework to add typechecking, linting, and formatting your code with prettier.

Want to see the entire bun-create-playwright project? It is on tinyurl.com/bun-create-playwright!

With this post, we will explore how the built-in test runner for Playwright Test can run headed tests, debug versions of tests, and smoke tests.

... and shortcuts for all of these can be set up in the scripts in your package.json! If you are using "bun" as a package manager, as we are in bun-create-playwright, just type out "bun run", a space and then the shortcut such as: bun run test

package.json

"scripts": {
    "test": "playwright test",
    "test:headed": "playwright test --headed",
    "test:trace": "playwright test --trace on",
    "test:chromium": "playwright test --project=chromium",
    "test:firefox": "playwright test --project=firefox",
    "test:webkit": "playwright test --project=webkit",
    "test:smoke": "playwright test --project=chromium --grep '@smoke'",
    "test:flaky": "playwright test --project=chromium --repeat-each=20",
    "test:ui": "playwright test --ui",
    "test:debug": "playwright test --debug",
    "test:failed": "playwright test --last-failed",
    "test:login": "playwright test tests/login.spec.ts",
    "test:secure-area": "playwright test tests/secure-area.spec.ts",
    "report:list": "playwright test --reporter=list",
    "report:line": "playwright test --reporter=line",
    "report:dot": "playwright test --reporter=dot",
    "report:blob": "playwright test --reporter=blob",
    "report": "playwright show-report",
    "codegen": "playwright codegen",
    "lint": "eslint .",
    "lint:ci": "eslint . --max-warnings 0",
    "lint:fix": "eslint . --fix",
    "format": "prettier --write .",
    "format:check": "prettier --check .",
    "format:debug": "prettier --check . --log-level debug",
    "format:diff": "prettier --list-different",
    "typecheck": "tsc --noEmit"
  },

bun-create-playwright / package.json

Do you need to really set up shortcuts like these? Certainly not! But it is easier than typing out: bunx playwright test --project=chromium --grep '@smoke'.

Feel free to name these commands anything you want!

You Can Run Scripts in Bunx? What is Bunx?

When you install Node.js, you are installing Node Package Execute (npx) a package runner that lets you auto-install and execute any npm packages on the fly. "This command allows you to run an arbitrary command from an npm package (either one installed locally, or fetched remotely), in a similar context as running it via npm run" ( NPMJS.com Docs / npx ).

Bunx is part of bun.sh, auto-installs and runs packages from npm... and claims to be 100x faster than npx for locally installed packages, like the Playwright test runner (bun.com Docs / bunx ).

If you have the package.json set up like above, bun run test would do the same thing as bunx playwright test... but with a lot less typing.

What is a Package.json file?

Before Node.js had NPM as a package manager, there was CommonJS that tried to standardize how server-side JavaScript applications should interact with the operating system. Back in 2009, it gave JavaScript the ability to manage dependencies. It could require() outside packages to be installed. The CommonJS Packages 1.1 specification introduced a "scripts" block, an "Object hash of scripts used in managing the package. A package manager tool may use these scripts to install, build, test or uninstall the package", as shown in the CommonJS Wiki / 1.1.

When Issac Schlueter created npm, he adopted CommonJS's package.json terminology, turning the scripts block into general task runners.

Read more on Increment.com / Interview with Isaac Z. Schlueter, CEO of npm

Because of that, with Package.json scripts, you can include a whole library of functionality!

Setting Up Scripts To Run and Debug Tests

"With Playwright you can run a single test, a set of tests, or all tests. Tests can be run on one browser or multiple browsers using the --project flag. Tests run in parallel by default and in headless mode, meaning no browser window opens while running the tests and results appear in the terminal. You can run tests in headed mode using the --headed CLI argument, or run your tests in UI mode using the --ui flag to see a full trace of your tests.

"[...] You can run your tests with the playwright test command. This runs your tests on all browsers as configured in the playwright.config file, and results appear in the terminal. Tests run in headless mode by default, meaning no browser window opens while running the tests". - https://playwright.dev/docs/running-tests

Note that on Playwright.dev / Running and debugging tests it assumes you are using node, and instructs you to use "npx playwright test". If using bun as a package manager, use "bunx playwright test" to run tests from the command line interface.

Run Tests in UI Mode

"We highly recommend running your tests with UI Mode for a better developer experience where you can easily walk through each step of the test and visually see what was happening before, during and after each step. UI mode also comes with many other features such as the locator picker, watch mode and more". - Playwright.dev / Running and debugging tests

"UI Mode lets you explore, run, and debug tests with a time travel experience complete with a watch mode. All test files are displayed in the testing sidebar, allowing you to expand each file and describe block to individually run, view, watch, and debug each test. Filter tests by name, projects (set in your playwright.config file), @tag, or by the execution status of passed, failed, and skipped. See a full trace of your tests and hover back and forward over each action to see what was happening during each step. You can also pop out the DOM snapshot of a given moment into a separate window for a better debugging experience". - Playwright.dev / UI Mode

Review Playwright.dev / UI Mode to learn about running your tests, filtering tests. seeing the Timeline view at the top of the trace, the Actions tab in UI view, how to inspect the DOM, pick a locator, view the source, see the action called, the full log of the test, errors, the console logs, network tab, attachments if doing visual regression testing, and watch mode.

You can even run UI Mode in the browser for Docker and GitHub Codespace environments with "npx playwright test --ui-host=0.0.0.0" to have the port forwarded automatically, or "npx playwright test --ui-port=8080 --ui-host=0.0.0.0" to set the port.

Run Tests in Headed Mode

"To run your tests in headed mode, use the --headed flag. This gives you the ability to visually see how Playwright interacts with the website".

Run tests on different browsers

"To specify which browser you would like to run your tests on, use the --project flag followed by the browser name.

bunx playwright test --project webkit

"To specify multiple browsers to run your tests on, use the --project flag multiple times followed by each browser name".

bunx playwright test --project webkit --project firefox

Run specific tests

"To run a single test file, pass in the test file name that you want to run.

bunx playwright test landing-page.spec.ts

"To run a set of test files from different directories, pass in the directory names that you want to run the tests in.

bunx playwright test tests/todo-page/ tests/landing-page/

"To run files that have landing or login in the file name, simply pass in these keywords to the CLI.

bunx playwright test landing login

"To run a test with a specific title, use the -g flag followed by the title of the test".

bunx playwright test -g "add a todo item"

Run last failed tests

"To run only the tests that failed in the last test run, first run your tests and then run them again with the --last-failed flag.

bunx playwright test --last-failed

"Playwright stores the list of failed tests from the previous run in <outputDir>/.last-run.json (see testConfig.outputDir). To use a different file path, pass --last-failed-file=<path> or set PLAYWRIGHT_LAST_RUN_OUTPUT_FILE".

Debug tests with the Playwright Inspector

"To debug all tests, run the Playwright test command followed by the --debug flag.

bun playwright test --debug

"This command opens a browser window as well as the Playwright Inspector. You can use the step over button at the top of the inspector to step through your test. Or, press the play button to run your test from start to finish. Once the test finishes, the browser window closes.

"To debug one test file, run the Playwright test command with the test file name that you want to debug followed by the --debug flag.

bun playwright test example.spec.ts --debug

"To debug a specific test from the line number where the test(.. is defined, add a colon followed by the line number at the end of the test file name, followed by the --debug flag".

bun playwright test example.spec.ts:10 --debug

Read more at Playwright.dev / Debugging Tests

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Implementing Page Objects in Playwright

2026-08-16T18:04:38.139-04:00

Picture a login screen, such as The-Internet / Login.

On this LoginPage, there is:

a heading: Login Page
a user name textbox with the label, "Username"
a password textbox with the label, "Password"
a login button, with the role of a button, and the name of "Login"
a flash message that appears if you enter invalid credentials such as "NotAUser" and "NotAPassword".

If you successfully log in with "tomsmith" and "SuperSecretPassword!, there is a SecureArea:

a heading: "Secure Area"
a flash message "You logged into a secure area!"
a Logout button.

Sure, you could interact with each web element in your test... but what if the username text box locator changes? You would have to update multiple tests every time the element changed.

... Instead, you could place it in a Page Object, something that Playwright handles well!

"A page object represents a part of your web application. An e-commerce web application might have a home page, a listings page and a checkout page. Each of them can be represented by page object models.

"Page objects simplify authoring by creating a higher-level API which suits your application and simplify maintenance by capturing element selectors in one place and create reusable code to avoid repetition".

Using Playwright's Built-In Test Runner? Or Something Else?

You may have noticed in https://playwright.dev/docs/pom that there are two different styles of page objects. One for "Test". One for "Library".

Test: If you are writing actual Playwright test suites, and Playwright's built in test runner, use the Test section as a guide when creating page objects.
Library: If you are integrating Playwright into an existing test framework such as Jest or Cucumber and just want browser automation, instead of having pre-built page fixtures, etc, you can use this format.

Setting Up Page Objects

Let's review a page object I created for the main login page on my GitLab account for the bun-create-typescript project, tests/pages/LoginPage.ts.

The first thing we do is import the Locator method and Page fixture from the Playwright Test library.

import { type Locator, type Page } from "@playwright/test";

According to Playwright.dev / Class-Fixtures, "Playwright Test is based on the concept of the test fixtures. Test fixtures are used to establish environment for each test, giving the test everything it needs and nothing else.

"Playwright Test looks at each test declaration, analyses the set of fixtures the test needs and prepares those fixtures specifically for the test. Values prepared by the fixtures are merged into a single object that is available to the test, hooks, annotations and other fixtures as a first parameter [...]

"Playwright Test will set up the page fixture before running the test, and tear it down after the test has finished. page fixture provides a Page object that is available to the test".

The Page and Locator Class

According to Playwright.dev / Page, "Page provides methods to interact with a single tab in a Browser, or an extension background page in Chromium. One Browser instance might have multiple Page instances.". These pages are isolated from each other, created for each test. The page fixtures have their own setup and teardown methods, such as opening up a new browser before a test starts, and closing a browser when a test ends.

According to Playwright.dev / Locators, "Locators are the central piece of Playwright's auto-waiting and retry-ability. In a nutshell, locators represent a way to find element(s) on the page at any moment". The built-in Playwright locators are:

page.getByRole() to locate by explicit and implicit accessibility attributes.
page.getByText() to locate by text content.
page.getByLabel() to locate a form control by associated label's text.
page.getByPlaceholder() to locate an input by placeholder.
page.getByAltText() to locate an element, usually image, by its text alternative.
page.getByTitle() to locate an element by its title attribute.
page.getByTestId() to locate an element based on its data-testid attribute (other attributes can be configured).

Define the Page Object Class

Now that we have imported the Page and Locator class from Playwright, we can define the LoginPage page object class.

export class LoginPage {
  readonly page: Page;
  readonly usernameInput: Locator;
  readonly passwordInput: Locator;
  readonly loginButton: Locator;
  readonly alertMessage: Locator;
  readonly heading: Locator;

Here, we are declaring LoginPage as a Page class, and creating Locator objects that will find and point to the username and password textboxes, the login button, the alertMessage, and the heading.

Initialize the Page Object Class

Now that we have defined the page object, we can initialize a constructor for the page, setting up reusable locators for all the web elements we defined.

  constructor(page: Page) {
    this.page = page;
    this.usernameInput = page.getByLabel("Username");
    this.passwordInput = page.getByLabel("Password");
    this.loginButton = page.getByRole("button", { name: "Login" });
    this.alertMessage = page.locator("#flash");
    this.heading = page.getByRole("heading", { name: "Login Page" });
  }

The textboxes are labeled "Username" and "Password". If we know that these labels are not expected to change, we can use the labels as a locator strategy.
The login button, we can use as a locator strategy its role "button" and the name "Login".
The alert message is tricky. Here, we can grab the id, "#flash", in the locator.
The heading, we can also get by role of "heading", and name of "Login Page".

Navigate to the Login Page

Let's embed in this Login Page Object a way to navigate to the Login page.

  async navigate(): {
    await this.page.goto("/login");
  }

Here, we have defined a method in the LoginPage page object our tests can use.

In our playwright.config.ts (see code in GitLab) we have already declared our baseURL to be https://the-internet.herokuapp.com, which will display when a new browser window is opened when a new test starts.

When in our test we instantiate a new LoginPage method, and call the navigate method, it will goto "login".

async has been placed before the function definition, since the browser will not be happening instantly. Actions will be happening asynchronously, all at the same time. It returns a "Promise" object.
await can be used inside an aync function. It tells the program to pause execution until the slow action, such as loading a webpage is finished.

What is a Promise? "A Promise is a proxy for a value not necessarily known when the promise is created. It allows you to associate handlers with an asynchronous action's eventual success value or failure reason. This lets asynchronous methods return values like synchronous methods: instead of immediately returning the final value, the asynchronous method returns a promise to supply the value at some point in the future". - Developer.Mozilla.com / JavaScript / Promise.

What is Asynchronous Programming in JavaScript? "JavaScript, especially when running in a web browser environment, has a unique challenge. It's single-threaded, meaning it has one line of execution at a time. If any task (fetching data, responding to a button click, etc.) takes a long time, it would freeze the entire webpage!

"Asynchronous programming is the pattern to tackle this. It enables us to start a potentially long-running operation and tell JavaScript, 'Go about your business, I'll let you know when this is ready.' This prevents the webpage from locking up.

"[...] JavaScript async await lets us write asynchronous code that resembles the style of straightforward, step-by-step (synchronous) code. This often makes the logic much easier to follow.- Upgrad.com / tutorials / JavaScript aync-await.

Creating a LoginAs Method

Sure, we could fill in the username, fill in the password, and click the login button, but what if that login procedure changes? We would have to hunt down every time we logged into the app, and make the changes in the test.

There is a software design principle called DRY: Don't Repeat Yourself, first coined by Andy Hunt and Dave Thomas in their book "The Pragmatic Programmer: From Journeyman to Master (1999)" (See Wikipedia, and the interview with both authors on YouTube at GoTo Conference 2020).

Instead, we could create a new async method called "loginAs" where we pass in the username and password.

  async loginAs(username: string, password: string): {
    await this.usernameInput.fill(username);
    await this.passwordInput.fill(password);
    await this.loginButton.click();
  }

Both "fill" and "click" are Playwright actions ( See docs ).

Text input can use locator.fill.
Checkboxes and radio buttons can .check them off, then expect them .toBeChecked.
Multiple radio buttons can be .selectOptions by value or label.
Mouse clicks can be performed with a mouse click, a double mouse click, a right click, a shift click, or a control click.
You can press keys such as .press('Enter'), or .press('Control+ArrowRight').
You can also press keys such as Backquote, Minus, Equal, Backslash, Backspace, Tab, Delete, Escape, ArrowDown, End, Enter, Home, Insert, PageDown, PageUp, ArrowRight, ArrowUp, F1 - F12, Digit0 - Digit9, KeyA - KeyZ, etc
You can .dragAndDrop, and explicitly .scrollIntoViewIfNeeded().

We have the locator, such as this.usernameInput, that we have declared, so now we can perform an action on the locator.

Get Text From Web Elements

Need to get text from an element such as an alert message or a heading? Using a locator, you can retrieve the innerText property of the web element. According to Mozilla Org / HTMLElement / innerText, "As a getter, it approximates the text the user would get if they highlighted the contents of the element with the cursor and then copied it to the clipboard".

  async getAlertMessage() {
    return (await this.alertMessage.innerText()).trim();
  }

Once we get the text of the web element, there may be hidden whitespaces in the text. "The trim() method of String values removes whitespace from both ends of this string and returns a new string, without modifying the original string". - Mozilla.Org / String / Trim

  async getHeading() {
    return (await this.heading.innerText()).trim();
  }

After getting the text and trimming it, we return it to our test function.

We could have set up the page objects to expect the correct heading or alert message is correct, sending to the method what should be expected... but I try to avoid this whenever possible.

Personally, I prefer to keep the assertions, the things we are testing for, in the actual test and out of the page object.

Setting Up Test Data

Yes, with just one set of valid credentials, separating them into a JSON file is overkill... but what if there were hundreds of valid test users? Here we are placing them into a JSON (JavaScript Notation) file called credentials with two top level objects... valid, and invalid.

In them, we have two key-value pairs in each object, username and password.

tests/ data/ credentials.json

{
  "valid": {
    "username": "tomsmith",
    "password": "SuperSecretPassword!"
  },
  "invalid": {
    "username": "NotAUser",
    "password": "NotAPassword"
  }
}

Need to add or change usernames and passwords?

No need to hunt through test data. Change just what is in the credentials file, and everything else will be reset. ( See this code in GitLab bun-create-playwright. )

Setting Up The Tests

Tests are set up in the Arrange / Act / Assert fashion, the pattern first mentioned by software developer Bill Wake in 2001 regarding the Extreme Programming movement (See XP123 / 3a- Arrange, Act. Assert ).

Arrange: Get the test data, open the browsers
Act: Navigate to the page and login
Assert: Do we have the proper heading ("Secure Area"), url, and alert message that we have logged into a secure area?

Finally, it's time to write our first test suite! Note: This code is in the GitLab project bun-create-playwright, under tests/login.spec.ts.

import { test, expect } from "@playwright/test";
import { LoginPage } from "./pages/LoginPage";
import { SecurePage } from "./pages/SecureAreaPage";
import credentials from "./data/credentials.json" with { type: "json" };
import messages from "./data/messages.json" with { type: "json" };

We are now importing files into our test and setting up variables to store the information such as:

The "test" and "expect" methods from Playwright.
The LoginPage and SecureAreaPage page objects we have set up.
The credentials stored in a JSON file that we have gone over.
A separate json file that stores all messages such as "You logged into a secure area!"

Set Up Constants for the Test Suite

test.describe("Login", () => {
  const validUser = credentials.valid.username;
  const validPassword = credentials.valid.password;
  const invalidUser = credentials.invalid.username;
  const invalidPassword = credentials.invalid.password;

Here, we are setting up a describe block, one of the methods in Playwright Test, declaring a group of tests, giving the test the title of "Login".

Then, to make the test more readable, instead of passing a three part credentials call, I am storing them in appropriately named variables.

These variables, validUser, validPassword, etc, we are declaring as type "const", also known as constants. If we accidentally change these values when the test is running, an error will be thrown. (Read more at TypeScript.org / Handbook / Variable Declarations ).

These constants, since they are set up in the describe block, will be set for ALL of the tests in this test suite.

Set Up the Individual Test

Let's set up a test that we can actually log in, and get to the secure area page, and call it "successful login with valid credentials".

  test("successful login with valid credentials", {
    tag: "@smoke",
  }, async ({ page }) => {
    const loginPage = new LoginPage(page);
    const securePage = new SecurePage(page);

    await loginPage.navigate();
    await loginPage.loginAs(validUser, validPassword);

We instantiate a new instance of the LoginPage, calling it a constant "loginPage", and do the same for the Secure Area.

Now that we declared new LoginPage, we can use the methods we have set up in our page objects, such as loginPage.navigate, and loginPage.loginAs, where we feed in the validUser and validPassword.

How to Assert Our Test is Working

We have done the arranging. We have done the acting. Now, it is time for the assertions.

Playwright.dev / Assertions mention, "Playwright includes test assertions in the form of expect function. To make an assertion, call expect(value) and choose a matcher that reflects the expectation. There are many generic matchers like toEqual, toContain, toBeTruthy that can be used to assert any conditions.

"Playwright also includes web-specific async matchers that will wait until the expected condition is met.

"By default, the timeout for assertions is set to 5 seconds. Learn more about various timeouts".

You can assert that an element is attached, a checkbox is checked, an element is disabled, editable, am element is empty, enabled, focused, visible, contains text or has an exact piece of text, contains a CSS class, has an accessible description or name.

You can also check if it has a specific ARIA role or matches an Aria snapshot, has a certain amount of child elements, has a certain id, css, an option selected, a value.

With pages, you can expect it to have a certain screenshot (for visual testing), has a title, url, or an OK status.

All of these elements will be retried if they fail the first time.

You can also have some test conditions where you do not auto retry, such as expecting a value to be something, close to something, the value is falsy (false, 0, or null). You can expect the value to be greaterThan, greater than or equal, less than a certain value. You can expect it to be NaN (not a number), null, or truthy (not false, not 0, and not null).

await expect(page).toHaveURL(/\/secure/);
    expect(await securePage.getHeading()).toBe("Secure Area");
    expect(await securePage.getAlertMessage()).toContain(messages.loginSuccessMessage);
  });

Now, we are checking that:

the heading we get from the SecurePage page object has the exact value "Secure Area".
the alert message matches the messages we have in the messages file...

And that's how you set up Page Objects and Tests using Playwright!

In the future, we will go over how to set up CI / CD pipelines with GitLab, but I need a break!

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Whew! That was a lot!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

How Playwright Frameworks get configured with playwright.config.ts

2026-08-14T22:30:20.303-04:00

When we installed bun, a new package manager owned by Anthropic, then ran "bun create playwright", a new automation framework was stood up, along with sample tests, and a Playwright configuration file. In this post, we will be examining the file generated: playwright.config.ts.

Personally, I find the pre-generated file very hard to scan... there are so many options and documentation in the comments, it is difficult for me to focus on the code. Let's examine just the code generated below. If you need to see the whole file, you can see it here: https://playwright.dev/docs/test-configuration

Playwright.dev / Configuration mentions, "Playwright has many options to configure how your tests are run. You can specify these options in the configuration file".

What is Configured By Default?


import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: 'html',
 
    // baseURL: 'http://localhost:3000',

    /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
    trace: 'on-first-retry',
  },

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },

    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },

    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },
});

I know what you are thinking ... What the heck is process.env.CI ? 2 : 0?

This is know as a ternary operator. It is a shorthand for: "If A then B, else C".
You can run tests two ways: Locally, on your local machine, or through a CI / CD Pipeline like Jenkins, GitHub Actions, or GitLab.
If you are running tests on CI / CD, the process.env.CI would automatically get set. Since A is true, the value of retries would be "2".
If you are running tests locally, the process.env.CI would NOT get set. Since A is false, the value "0" would be selected.
CI / CD pipelines would get two retries. Locally, it would get no retries and a failed test would remain failed.

Now that that is cleared up, let's examine the properties:

testDir: Points to the home of the tests.
fullyParallel: Allows you to set the tests to run all at once across many workers, or by one worker, tests lined up single file. By default, all tests are run all at once with multiple workers.
forbidOnly:

Developers debugging a test, will mark that test they are working on "test.only()". It's okay to have "test.only()" locally, but you want it to complain, loudly, if you have any test still marked "test.only()" in a CI/CD pipeline.
If you are running the test locally, the test run wouldn't pick up a process.env.CI. The CI/ CD variable would not be set, which means that the variable would be "undefined" which would resolves to false. forbidOnly would be false. No alert would be thrown if an "only" was seen, which is exactly what you wanted.
If you were running this in a CI/CD pipeline, you do not want test.only() to appear. Since process.env.CI would fill in a string, or set it to true, this equation would evaluate to "forbidOnly=true". An alert would be thrown if a test running in the CI/ CD pipeline saw an "only".

retries: Tests can be fragile when running on a CI/ CD platform. If it is CI/ CD, the process.env.CI would be set, by default retrying a failed test twice. If it is not set, because the test is running locally, we have the second default, which is "0", meaning failing tests would not be rerun.
workers: How many workers would you like to execute tests for you?

By default, if you are on a CI / CD pipeline running a test it will run two workers to execute a test. Playwright recommends setting the workers to "1".
Running a test locally? "Undefined" means that Playwright will detect your machines hardware and boot up a number of workers equal to half your logical CPU cores without freezing your operating system.
Note: GitHub Actions free tier give you virtual machines with limited resources. If you have just one worker tests will run one after another making logs and failures easier to trace.
Read Playwright.dev docs about running jobs in parallel, and dividing tests into shards.

reporter: How would you like your output? Basic "html", with a standalone webpage? A "list" printing a detailed line per test showing pass or fail? A "line" displaying execution progress on a single terminal line? A "dot" report where passed tests are dots and failures are "F"? A "json" report? An xml "junit" style? A "blob" of a binary file that can be later merged from multiple machines? Or "github" which creates annotations in GitHub Actions? See Playuwright.dev / Reporter, and a list of reporters on The Testing Academy.

By default, it has set up to run the Playwright version of Chrome, Firefox, and WebKit Safari.

But What About Dot Env?

If you looked in the pre-generated configuration file, there is code that looks like:

* Read environment variables from file.
* https://github.com/motdotla/dotenv
*/
// import dotenv from 'dotenv';
// import path from 'path';
// dotenv.config({ path: path.resolve(__dirname, '.env') });
Dotenv reads custom variables from a plain text file, injecting them into a Node.js application.

Sometimes, settings change depending where your code is running, such as on your local machine vs a production environment.

It is a good practice to set up one .env file per environment. .env for your local. .env.production. .env.staging, etc.

There are things such as database passwords, third party API keys, server port numbers.

With Dotenv you can create an .env file in the root folder of your project, and write configurations like:

PORT=3000
DATABASE_URL=mongodb://localhost:27817/myapp
STRIPE_API_KEY=

Make sure to add the .env file into .gitignore else you might accidently push these secrets into GitLab! ... Or you can encrypt it using https://github.com/dotenvx/dotenvx

What about the Mobile Viewports and Branded Browsers?

You also may have seen in the boilerplate playwright.config.ts file:

/* Test against mobile viewports. */
// {
// name: 'Mobile Chrome',
// use: { ...devices['Pixel 5'] },
// },
// {
// name: 'Mobile Safari',
// use: { ...devices['iPhone 12'] },
// },

/* Test against branded browsers. */
// {
// name: 'Microsoft Edge',
// use: { ...devices['Desktop Edge'], channel: 'msedge' },
// },
// {
// name: 'Google Chrome',
// use: { ...devices['Desktop Chrome'], channel: 'chrome' },
// },

If you want Playwright to simulate how your website looks and behaves on mobile devices, you can comment out the mobile viewport section. Mobile Chrome simulates a Google Pixel 5 using Chromium, and Mobile Safari simulates an iPhone 12 using Apple's WebKit engine. ( See the ThinkSys article, Cross Browser Testing with Playwright in 2026 )

Sometimes, emulations are not enough, and you want to actually test on Microsoft Edge or Google Chrome. With this setup, you can launch the actual browsers.

... The next post, we will dive into setting up actual test and page objects.

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Running Tests with Playwright Test Explorer and Generating Tests with Codegen

2026-08-10T23:00:47.444-04:00

Finding the best locator for a web element can be a hassle:

Right clicking on an element in Google Chrome. Inspecting the element. Going to Chrome Developer Tools. Try to decide what to do if there isn't a clear test id.

Playwright comes with a built in code generator where it can built out a rough draft of a test while you interact with a website. "Playwright will look at your page and figure out the best locator, prioritizing role, text and test id locators. If the generator finds multiple elements matching the locator, it will improve the locator to make it resilient that uniquely identify the target element", according to Playwright.dev / Test Generator.

Do you have the Integrated Development Environment (IDE) by Microsoft, VS Code? You can get it at the Visual Studio Marketplace.

Playwright Test Explorer

Playwright.dev / Getting Started VS Code: https://playwright.dev/docs/getting-started-vscode

After installation, you will see a beaker icon in your VS Code left navigation menu. Selecting that, you can see your tests, such as the default ones Playwright automatically adds when it is installed:

Here you can see that Test Explorer has discovered the test suite example.spec.ts, along with two tests it created:

has title: Verifies that if you go to playwright.dev, that it has a link titled "Get Started".
get started link: Verifies at playwright.dev that after clicking on Get Started link, the heading says "Installation".

Selecting the green arrow button will open up three browser windows: Chromium, Firefox, and a WebKit browser created by Playwright, mimicking Safari. It will run a total of six tests.

But wait! There's more! You can:

Run a single test.
Run all tests at different levels.
Run on multiple browsers.
Show a browser.

You can also debug your tests:

Set breakpoints by clicking on the gutter next to a line number
Right clicking the test and select "Debug test".
Running your test will pause at your breakpoint, allowing you to step through your code.

You can also:

Fix a test with AI.
Debug tests with Trace Viewer
View DOM inspection, reviewing the Document Object Model snapshots.

Debugging Playwright tests in VS Code

https://www.youtube.com/watch?v=tJF7UhA59Gc

Generating Tests with CodeGen

"CodeGen is Playwright's powerful test generation tool that automatically creates test code by recording your interactions with a web page. Instead of writing tests from scratch, you can simply navigate through your application while CodeGen captures your actions and converts them into reliable test code with proper locators and assertions". - Playwright.Dev / Getting Started VS Code

How to Generate Tests in Playwright with the VS Code Extension

https://www.youtube.com/watch?v=5XIZPqKkdBA

Testing The-Internet: Login Page

Per the instructions, I will start sketching out tests for a Login page application: https://the-internet.herokuapp.com/login

https://the-internet.herokuapp.com/login

Test #1: Is Everything Visible?

Playwright / Tools, click on "(+) Record new", opens up a blank peach colored window.
The red RECORD icon shows that it is recording.
In the browser window, entered the URL: https://the-internet.herokuapp.com/login

Generate code to detect a Heading:

In the menu, I selected the "eye" to assert visibility, and the heading, "Login Page".

Generate code to detect an error message.

Assert visibility of username + password textbox: "eye" + click on username textbox. "eye" + click on password textbox. "eye" + click on Login button.

The code it generated. after some cleanup such as naming it "Checking visibility:

test('Checking visibility', async ({ page }) => {
  await page.goto('https://the-internet.herokuapp.com/login');
  await page.getByRole('heading', { name: 'Login Page' }).click();
  await expect(page.getByRole('heading', { name: 'Login Page' })).toBeVisible();
  await expect(page.getByRole('textbox', { name: 'Username' })).toBeVisible();
  await expect(page.getByRole('textbox', { name: 'Password' })).toBeVisible();
  await expect(page.getByRole('button', { name: ' Login' })).toBeVisible();
});

... Not bad! I'd rather use some ids than "getByRole", but not bad!

Test #2: Negative Test: Invalid User.

Playwright / Tools, click on "(+) Record new", opens up a blank peach colored window.
The red RECORD icon shows that it is recording.
In the browser window, entered the URL: https://the-internet.herokuapp.com/login

Generate code to enter incorrect username and password:

"Eye" + Login Page to check the heading is there.
Enter "NotAUser" into the username textbox.
Enter "NotAPassword" into the password textbox.

Check the alert message:

Let's see what happens if we click "Assert Visibility" and the alert message "Your username is invalid". Or "Assert text" and the error message.

test('Invalid login', async ({ page }) => {
  await page.goto('https://the-internet.herokuapp.com/login');
  await page.getByRole('heading', { name: 'Login Page' }).click();
  await expect(page.getByRole('heading', { name: 'Login Page' })).toBeVisible();
  await page.getByRole('textbox', { name: 'Username' }).fill('NotAUser');
  await page.getByRole('textbox', { name: 'Password' }).fill('NotAPassword');
  await page.getByRole('button', { name: ' Login' }).click();
  await expect(page.getByText('Your username is invalid! ×')).toBeVisible();
  await expect(page.locator('#flash')).toContainText('Your username is invalid! ×');
});

... Again, it could use some reformatting, but not bad.

Okay, one more test: Let's log in correctly.

Test #3: Smoke Test: Valid Login

Playwright / Tools, click on "(+) Record new", opens up a blank peach colored window.
The red RECORD icon shows that it is recording.
In the browser window, entered the URL: https://the-internet.herokuapp.com/login

Generate code to do a successful login:

"Eye" + Login Page to check the heading is there.
Enter "tomsmith" into username, "SuperSecretPassword!" in password, and click on "Login".

Validate that you are in the secure area:

"Eye" + "Assert Visibility" on title "Login Page".
Let's see what happens if you assert text, "Login Page", and then assert "Welcome to the Secure Area".
"Eye" + "Assert Visibility" Logout button.

test('Successful Login', async ({ page }) => {
  await page.goto('https://the-internet.herokuapp.com/login');
  await page.getByRole('textbox', { name: 'Username' }).fill('tomsmith');
  await page.getByRole('textbox', { name: 'Password' }).fill('SuperSecretPassword!');
  await expect(page.getByRole('heading', 
     { name: 'Secure Area', exact: true })).toBeVisible();

  await expect(page.locator('h2')).toContainText('Secure Area');
  await expect(page.locator('h4')).toContainText('Welcome to the Secure Area. 
       When you are done click logout below.');
  await expect(page.getByRole('link', { name: 'Logout' })).toBeVisible();
});

We now have enough to create some tests!

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Checking code with lint, formatting it with prettier

2026-08-10T19:23:45.625-04:00

Now that we've installed bun, Anthropic's package manager, scaffolded a Playwright framework and closely examined the results, and added typechecking with TypeScript's compiler, it's time to add ways to check the code with lint, and reformat the code with prettier.

We will be using:

ESLint, as the static-analysis tool to review the code without running it. Little bits of fluff -- like syntax errors, structural bugs, anti-patterns, and code style violations -- can collect on your code, so it helps to run a linter to help catch it all, such as ESLint. There is also a linter, eslint-plugin-playwright, for Playwright tests.
Prettier enforces a consistent code style across your entire codebase. Because ESLint and Prettier can conflict, we will be using Prettier's eslint-config-prettier, which turns off all rules that are unnecessary or might conflict with ESLint.

What is a Linter?

According to the Wikiwand entry for Lint, "Stephen C. Johnson, a computer scientist at Bell Labs, came up with the term 'lint' in 1978 while debugging the yacc grammar he was writing for C and dealing with portability issues stemming from porting Unix to a 32-bit machine. The term was borrowed from lint, the tiny bits of fiber and fluff shed by clothing, as the command he wrote would act like a lint trap in a clothes dryer, capturing waste fibers while leaving whole fabrics intact. The lint program was released outside of Bell Labs in Unix V7, in 1979.

"In his 1978 paper, Johnson explained his reasons for creating a new program to detect errors: '...the general notion of having two programs is a good one' because they concentrate on different things, thereby allowing the programmer to 'concentrate at one stage of the programming process solely on the algorithms, data structures, and correctness of the program, and then later retrofit, with the aid of lint, the desirable properties of universality and portability' "

What is ESLint?

Website: https://eslint.org/
GitHub: https://github.com/eslint/eslint

According to ESLint.org / About, "ESLint is an open source JavaScript linting utility originally created by Nicholas C. Zakas in June 2013. Code linting is a type of static analysis that is frequently used to find problematic patterns or code that doesn’t adhere to certain style guidelines. There are code linters for most programming languages, and compilers sometimes incorporate linting into the compilation process.

"JavaScript, being a dynamic and loosely-typed language, is especially prone to developer error. Without the benefit of a compilation process, JavaScript code is typically executed in order to find syntax or other errors. Linting tools like ESLint allow developers to discover problems with their JavaScript code without executing it.

"The primary reason ESLint was created was to allow developers to create their own linting rules. ESLint is designed to have all rules completely pluggable. The default rules are written just like any plugin rules would be. They can all follow the same pattern, both for the rules themselves as well as tests. While ESLint will ship with some built-in rules to make it useful from the start, you’ll be able to dynamically load rules at any point in time".

What is ES Lint for Playwright?

ESLint for Playwright was initially written by open-source developer Mark Skelton, a senior staff software engineer at Cursor. It is now maintained by the Playwright Community Organization.

From BrowserStack.com's Setting Up ESLint fpr Playwright Projects:

Have you ever returned to a Playwright project after a few weeks and felt the tests were suddenly unstable or difficult to trust?

In most cases the problem isn’t Playwright. It comes from small issues that slip in unnoticed.

Missing awaits. Inconsistent patterns. Promise errors that never surface. Test files that grow without any structure.

Playwright ESLint is a linting setup that helps ESLint understand Playwright’s testing patterns and catch issues that commonly appear in UI automation.

What is Prettier?

Website: https://prettier.io/
Prettier Playground: https://prettier.io/playground/
GitHub: https://github.com/prettier/prettier

According to James Long's January 2017 blog entry, A Prettier Script Formatter, "Prettier gets rid of all original styling and guarantees consistency by parsing JavaScript into an AST and pretty-printing the AST. Unlike eslint, there aren't a million configuration options and rules. But more importantly: everything is fixable [...]"

In JavaScript, an Abstract Syntax Tree (AST) is a nested object structure that maps out the semantic meaning of your code -- such as identifying variable declarations, function calls, and loops -- while ignoring visual styling. Because Prettier is strictly a code formatter, it parses your source code into an AST and then regenerates entirely new text based on its own style rules. Since Prettier only modifies surface-level elements like indentation, line breaks, quotes, and semicolons, the underlying logic remains untouched, resulting in an identical AST before and after formatting

Installing ESLint

According to ESLint.org, you can install it with NPM, Yarn, PNPM (package managers we saw that Playwright all have listed in their installation documentation. They also mention one we have seen already... bun. To install ESLint with bun, they suggest running the command:

bun create @eslint/config@latest

It then asked me a series of questions:

What do you want to lint? ... I selected Markdown.
How would you like to use ESLint? ... To check syntax and find problems.
What type of modules does your project use? ... I selected "JavaScript modules" since we seem to use import/ export, and not require/ exports anywhere.
Which framework does your project use? ... None of these.
Does your project use TypeScript? ... Yes.
Where does your code run? Browser? Node? ... it actually runs in both, so let's check off both.
Which language do you want your configuration file to be written in? ... JavaScript.

Why did I pick JavaScript? If I picked TypeScript, I would then have to install a library called Jiti to load the TypeScript Config file. Keeping it as a JavaScript file would make it easier.

Then it asked:

The config that you've selected requires the following dependencies: eslint, @eslint/js, globals, typescript-eslint.
Would you like to install them now? ... Yes.

It continued:

Which package manager do you want to use? ... npm, yarn, pnpm, bun ... I selected "bun".

Finally, it installed everything. When it was finished, it set up a new file...

eslint.config.mjs

import js from "@eslint/js";
import globals from "globals";
import tseslint from "typescript-eslint";
import { defineConfig } from "eslint/config";

export default defineConfig([
  { files: ["**/*.{js,mjs,cjs,ts,mts,cts}"], plugins: { js }, extends: ["js/recommended"], languageOptions: { globals: {...globals.browser, ...globals.node} } },
  tseslint.configs.recommended,
]);

js/recommended gives us ESLint’s standard default rules for JavaScript catching mistakes like using variables before they’re defined, using undeclared variables, unreachable code, confusing or invalid syntax patterns, some common logic and scope problems.

tseslint.configs.recommended adds the recommended rules for TypeScript. It catches stuff like: unsafe or incorrect type usage, confusing any usage, unnecessary type assertions, some patterns that are likely to cause bugs in TypeScript code.

Installing ESLint for Playwright

According to the NPMJS.com entry on eslint-plugin-playwright,

bun add -D eslint-plugin-playwright

This adds eslint-plugin-playwright to your package.json.

The docs mention that eslint-plugin-playwright, "The recommended setup is to use the files field to target only Playwright test files. In the examples below, this is done by targeting files in the tests directory and only applying the Playwright rules to those files. In your project, you may need to change the files field to match your Playwright test file patterns".

Following their directions, our eslint.config.jts now looks like:

import js from "@eslint/js";
import globals from "globals";
import tseslint from "typescript-eslint";
import { defineConfig } from "eslint/config";
import playwright from "eslint-plugin-playwright";

export default defineConfig([
  {
    files: ["**/*.{js,mjs,cjs,ts,mts,cts}"],
    plugins: { js },
    extends: ["js/recommended"],
    languageOptions: {
      globals: { ...globals.browser, ...globals.node },
    },
  },
  {
    files: ["tests/**"],
    extends: [playwright.configs["flat/recommended"]],
  },
  tseslint.configs.recommended,
]);

... If we wanted to pick and choose which rules to follow, we could add a rules block, but let's leave it as is for now.

To run the linter:

bunx eslint . --ext .ts,.mts,.js,.mjs

... No output? No problems detected!

ESLint for Prettier

GitHub: https://github.com/prettier/eslint-config-prettier

We are using ESLint for our code linter, ESLint for Playwright as our Playwright linter.

Both of these packages would step on the toes of our code formatter, Prettier, so Prettier came up with eslint-config-prettier, which had some Prettier rules that would conflict turned off.

We are going to add Prettier and eslint-config-prettier at the same time with:

bun add -D prettier eslint-config-prettier

To check if Prettier installed:

bunx prettier --version

Check with Prettier if there are any errors:

bunx prettier --check .

... And it looks like there are four warnings.

What are they?

bunx prettier --list-different .
bunx prettier --check . --log-level debug

Okay, they are minor spacing issues. Let's fix them:

bunx prettier --write .

Want to see more options?

Check out Prettier's Command Line Interface (CLI) commands at https://prettier.io/docs/cli

Add Scripts to the Package.json

You know, "bunx eslint ." and all the rest is kinda complicated.

Let's add some more scripts that are lint related to our package.json, which already has our test script and typecheck script where we added the TypeScript compiler.

"scripts": {
    "test": "playwright test",
    "lint": "eslint .",
    "lint:ci": "eslint . --max-warnings 0",
    "lint:fix": "eslint . --fix",
    "format": "prettier --write .",
    "format:check": "prettier --check .",
    "format:debug": "prettier --check . --log-level debug",
    "format:diff": "prettier --list-different",
    "typecheck": "tsc --noEmit"
  },

... All we need to do is run "bun run test", "bun run format", or "bun run typecheck" to run tests, format the code, or run typecheck.

Also, I added lint:ci, for our CI/CD pipeline we will be adding later, so it does not allow any warnings with lint.

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Add Type Checking and TSConfig to Bun-Create-Playwright

2026-08-07T23:10:43.830-04:00

Now that we have installed bun, Anthropic's package manager, and scaffolded a Playwright framework and closely examined the results, we are going to explore with our Bun-Create-Playwright project ways to check if our code is correct.

The first method we will be exploring is typechecking.

Why Typechecking? As the Playwright.dev / Node.js Introduction mentions:

"[...] Playwright does not check the types and will run tests even if there are non-critical TypeScript compilation errors. We recommend you run TypeScript compiler alongside Playwright.

"[...] Note that Playwright only supports the following tsconfig options: allowJs, baseUrl, paths, references and extends.

"[...] By default, Playwright will look up a closest tsconfig for each imported file by going up the directory structure and looking for tsconfig.json or jsconfig.json. This way, you can create a tests/tsconfig.json file that will be used only for your tests and Playwright will pick it up automatically".

Before we go further down this road...

What is Type Checking?

According to Type Checking in TypeScript: A Beginners Guide, every piece of data in TypeScript is given a "type", and this "type" determines what properties the data has, and what methods are available to it. The types can be things like a Number, String, Enum, Boolean, Array, Object, Type assertions, or others.

Since all data in TypeScript is given a type, it means that the TypeScript compiler, tsc, can check every property to make sure it is being used correctly. It makes sure that the expected properties exist and that the methods and functions are all compatible.

And best of all, you can run these checks in your IDE as you are writing the code!

To run typechecking against our code, all we have to do is execute from the Terminal:

tsc --noEmit

Since we are just kicking the tires of our code to see what happens, we don't need to emit "compiler output files like JavaScript source code, source-maps or declarations. This makes room for another tool like Babel, or swc to handle converting the TypeScript file to a file which can run inside a JavaScript environment. You can then use TypeScript as a tool for providing editor integration, and as a source code type-checker". ( TypeScriptLang.org / noEmit )

The only problem? We need a TypeScript configuration file added to our project before we do that, and as you could see when we examined what "bun create playwright" produced, we don't have one.

If you try to run the "tsc --noEmit" command without a tsconfig.json, you get the message:

Version 6.0.3
tsc: The TypeScript Compiler - Version 6.0.3

TS COMMON COMMANDS

tsc: Compiles the current project (tsconfig.json in the working directory.)

tsc app.ts util.ts: Ignoring tsconfig.json, compiles the specified files with default compiler options.

tsc -b: Build a composite project in the working directory.

tsc --init: Creates a tsconfig.json with the recommended settings in the working directory.

tsc -p ./path/to/tsconfig.json: Compiles the TypeScript project located at the specified path.

tsc --help --all: An expanded version of this information, showing all possible compiler options

tsc --noEmit
tsc --target esnext
Compiles the current project, with additional settings.

COMMAND LINE FLAGS

--help, -h Print this message.
...

--outFile Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output.

--outDir Specify an output folder for all emitted files.

You can learn about all of the compiler options at https://aka.ms/tsc

So, how do we create with bun a tsconfig.json file?

bun init

What does Bun Init Do?

According to the Bun.com / Init Templating docs, It creates:

a package.json file with a name that defaults to the current directory name
a tsconfig.json or jsconfig.json file, depending on whether the entry point is a TypeScript file
an entry point, which defaults to index.ts unless any of index.{tsx, jsx, js, mts, mjs} exist or the package.json specifies a module or main field
a README.md file

It also creates AI Agent rules (disable with $BUN_AGENT_RULE_DISABLED=1):

a CLAUDE.md file when Claude CLI is detected (disable with CLAUDE_CODE_AGENT_RULE_DISABLED env var)
a .cursor/rules/*.mdc file when Cursor is detected, which tells Cursor AI to use Bun instead of Node.js and npm

But we already scaffolded a Playwright project! Do we have to start from scratch and do everything all over again?

No, but we do need to make some modifications:

Does your package.json in your root folder of the project have "main" to be "index.js"? Change that JavaScript extension to ".ts".
Do you have a "jsconfig.json" file? Delete it.
Do you have an index.html file? Delete it.
Then run: bun init

TSConfig.json Achieved! First version!

The resulting tsconfig.json file produced by "bun init", according to Google AI, "configures TypeScript to act strictly as a type-checker while offloading the actual bundling, compiling, and running of your JavaScript/TypeScript code to Bun".

{
  "compilerOptions": {
    // Environment setup & latest features
    "lib": ["ESNext"],
    "target": "ESNext",
    "module": "Preserve",
    "moduleDetection": "force",
    "jsx": "react-jsx",
    "allowJs": true,
    "types": ["bun"],

    // Bundler mode
    "moduleResolution": "bundler",
    "allowImportingTsExtensions": true,
    "verbatimModuleSyntax": true,
    "noEmit": true,

    // Best practices
    "strict": true,
    "skipLibCheck": true,
    "noFallthroughCasesInSwitch": true,
    "noUncheckedIndexedAccess": true,
    "noImplicitOverride": true,

    // Some stricter flags (disabled by default)
    "noUnusedLocals": false,
    "noUnusedParameters": false,
    "noPropertyAccessFromIndexSignature": false
  }
}

Environment setup & latest features

These settings ensure TypeScript understands the latest syntax and runtime environment variables.

"lib": ["ESNext"] -- includes type definitions for the newest JavaScript standard library features (Promise, Array methods, etc.).
"target": "ESNext" -- compiles/type-checks against the latest ECMAScript syntax rather than downleveling to an older version.
"module": "Preserve" -- keeps your import/export statements as written (doesn't rewrite ESM to CommonJS or vice versa); left for Bun to handle at runtime.
"moduleDetection": "force" -- treats every file as a module (adds import/export scoping) even if it has no import/export statements itself.
"jsx": "react-jsx" -- compiles JSX using the React 17+ automatic runtime (no need to import React in every file that uses JSX).
"allowJs": true -- lets .js files be included and type-checked alongside .ts files.
"types": ["bun"] -- only loads Bun's global type definitions (e.g. Bun.serve), instead of auto-including every @types/* package found in node_modules.

Bundler mode

These rules match how modern build tools resolve file paths and handle module definitions.

"moduleResolution": "bundler" -- resolves imports the way modern bundlers do, closer to Node's ESM resolution but more permissive (works with paths, package exports, etc.).
"allowImportingTsExtensions": true -- permits writing import './foo.ts' with the explicit extension (normally disallowed).
"verbatimModuleSyntax": true -- requires type-only imports to be explicitly marked (import type { Foo }), so the compiler doesn't have to guess what to strip at build time.
"noEmit": true -- TypeScript only type-checks; it doesn't output compiled .js files (Bun runs the .ts files directly).

Best practices

These options activate strict type-safety rules to catch potential bugs early.

"strict": true -- turns on TypeScript's full strict type-checking mode (strictNullChecks, noImplicitAny, etc. all at once).
"skipLibCheck": true -- skips type-checking inside .d.ts declaration files (faster builds, avoids errors in third-party type definitions you don't control).
"noFallthroughCasesInSwitch": true -- errors if a switch case falls through to the next case without a break/return.
"noUncheckedIndexedAccess": true -- accessing an object/array by index (e.g. arr[i]) returns T | undefined instead of just T, forcing you to handle the "might not exist" case. Useful in test code that reads array/object results.
"noImplicitOverride": true -- requires the override keyword when a subclass method overrides a parent class method (relevant if you're using Page Object classes with inheritance).

Stricter flags (disabled here)

These are options explicitly turned off to keep the development experience smoother and less strict during early coding phases.

"noUnusedLocals": false -- won't error on declared-but-unused local variables.
"noUnusedParameters": false -- won't error on unused function parameters (common in test fixtures/callbacks where you don't use every argument).
"noPropertyAccessFromIndexSignature": false -- allows dot notation (obj.foo) even on objects typed with an index signature, instead of forcing bracket notation (obj['foo']).

... This is what "bun init" produced. Let's investigate with Claude.ai or Google AI to see if we should now be all set with our Bun + Playwright + TypeScript project...

TSConfig.json: Take Two!

After I ran it by GitHub Copilot, it suggested to change the tsconfig like so...

  "compilerOptions": {
    // Environment setup & latest features
    "lib": ["ES2022", "DOM"],
    "target": "ES2022",
    "module": "ESNext",
    "moduleDetection": "force",
    "types": ["node", "bun"],

But Why, Claude?

"These choices are better because they match the runtime and tooling you actually have, rather than using the most generic defaults.target: "ES2022": this targets a modern JavaScript version that is broadly supported and a good default for current Node/Bun environments. It is more practical than ESNext because it gives a stable, explicit baseline.

lib: ["ES2022", "DOM"]: this tells TypeScript about modern JavaScript features plus browser APIs like window, document, and fetch. That matters for Playwright tests, which often interact with browser-like environments.
module: "ESNext": this matches modern ESM-style projects and works well with current Bun/Node tooling. It is a better fit than older module systems when you are using modern package resolution.
types: ["node", "bun"]: this makes the compiler aware of both Node and Bun globals/types. That is useful because your project is using Bun as the package runner while also relying on Node-compatible APIs.

"In short, the new config is more explicit, more compatible with current tooling, and less likely to surprise you later when you add scripts, browser APIs, or ESM-style imports".

Add the TypeScript Checking to a Script

Okay, let's add the "tsc --noEmit" command to our package.json script:

  "scripts": {
    "test": "playwright test",
    "typecheck": "tsc --noEmit"
  },

Now, when you run "bun run typecheck", the TypeScript Compiler will test out your code.

No errors? No output. No problem!

With our next post, we will see other ways to add ways to review our code.

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Until then... Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

What happens when you scaffold a Playwright framework and run installed tests using bun?

2026-08-06T13:37:06.209-04:00

We covered in the last blog post how to install bun, a new package manager.

In this blog post, we will look into using bun to install a new Playwright framework.

Create the Playwright Framework

Once I installed bun on my Windows PC and was up and running I created a new folder, "bun-create-playwright".

I opened up that folder in VS Code, along with a new PowerShell terminal. In that terminal I entered:

bun create playwright

This activated the interactive Playwright installer that Playwright comes with.

I selected I wanted it to create for me a TypeScript project, placing the tests in the default folder, tests, but I decided not to add a GitHub Actions workflow. I wanted to experiment with using GitLab.
I chose it to set up and install all the browsers for me... which it did... using NPX, part of Node.

Wait a second, when it comes to our Playwright framework, doesn't bun replace node?

No. Bun might be a JavaScript runtime, a package manager, and a bundler shipped as a single software tool, but in our case we are simply using it as a package manager in our Playwright framework.

Bun works alongside Node.js in the Playwright project. We still use Playwright's test runner, a Node.js program. Bun handles installations and run tasks.

Alert: Windows Bug!

Initially, that wasn't enough to install Playwright's internal browsers. There is a bug with bun + Playwright Windows where we also need to:

bun add --dev @playwright/test
bunx playwright install

That was enough for the browsers to install.

Create the Bun Lock File

After the browsers are installed, we need to install bun in our project, to create from the package.json a bun installation.

Run in the Terminal: bun install
Review the newly created file bun.lock.file in the root directory of the project.
Delete the package-lock.json file.

Add the test script to the Package.json

If you look in the package.json file, you can see that the scripts code block is currently empty.

"scripts": {}

Inside this code block, we will be adding:

"scripts": {    
    "test": "playwright test"  
  },

Examine the Included Tests

Two tests will automatically be set up for you:

has title: Navigates to playwright.dev and expects the title of the page to have the word "Playwright".
get started: Navigates to playwright.dev and checks the "Get started" link.

tests/ example.spec.ts

import { test, expect } from '@playwright/test';

test('has title', async ({ page }) => {
  await page.goto('https://playwright.dev/');

  // Expect a title "to contain" a substring.
  await expect(page).toHaveTitle(/Playwright/);
});

test('get started link', async ({ page }) => {
  await page.goto('https://playwright.dev/');

  // Click the get started link.
  await page.getByRole('link', { name: 'Get started' }).click();

  // Expects page to have a heading with the name of Installation.
  await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});

There are three stages you want to follow when you are designing an automated test:

Arrange: Setup things like opening up a browser page.
Act: Go to a web site and see if you can get a link and click on it.
Assert: Expect that the conditions of the tests are met, failing the test if the expected results and actual results do not match up.

import pulls code that lives in another file or package so you can use it in your project. We import two specific named functions, test and expect, from the @playwright/test package (the Playwright Test framework's core library).

test is a function used to define an individual test case.
expect is a function used to create assertions (checks that something is true, and fail the test if it isn't). [ See Playwright.dev / Assertions ]

The { test, expect } syntax is called a "named import", getting specific things out of the package rather than the whole package as one object.

test blocks set up two Playwright tests. Strings such as "has title" sets up the name of the test.

async sets up this test function to be asynchronous, with many tests allowed to be executed at the same time as this one. The browser to run the first "has title" test might not be able to navigate to the page right away. The aync keyword allows the test to pause and wait for any slower actions to be completed before moving to the next part of the test. Once we set up async, we can use await later in the test. Meanwhile, it returns a JavaScript Promise, a placeholder container until we determine the success or failure of the action, when it actually will contain a value. This promise will be either pending, fulfilled, or rejected.

page sets up the pre-defined Playwright page fixture [ See Playwright.dev / Fixtures ]. With this call, before the individual test starts, a fresh isolated page is launched just for this test, so the "has title" test does not interfere with the "get started" test. This page fixture has automatic setup and teardown.

=> is a "hash rocket", representing the "arrow function", separating the earlier parameters of the test with the coming body of the test function.

await returns a Promise, a placeholder, so if we can't do something like go to a page right away, the test won't automatically go to the next line of code and possibly fail. Playwright by default waits 30 seconds for a goto to complete, 30 seconds for an action to happen, but only 5 seconds on an assertion like "expect". If it goes over that timeout, the test will fail.

goto is part of the Page fixture, that automatically handles page navigation.

expect is an assertion. By default, the test expects this condition to be met within five minutes, else the test fails.

toHaveTitle is a built in Playwright page assertion. It allows regular expressions such as /Playwright/ to let the test know that having the word "Playwright" in the title is acceptable. Playwright has many assertions, such as the PageAssertions toHaveTitle, toHaveURL, toHaveScrreenshot(name).

getByRole is a Playwright Locator, that helps you find a web element on the page. Are you looking for a certain alert, alertDialog, banner, button, checkbox, heading, img, link, listbox, menu, menubar, list, listbox, progressbar, searchbox, table, or textbox, tooltip, amongst others? You can try to getByRole. With this you are locating elements by their ARIA role, ARIA attributes and accessible name.

... Note that with locators, you can also getByTestId, getByAltText, getByText, getByTitle, etc.

.click is a Playwright action you can perform.

Once you found an element by a locator, you can .click or .dblclick it, .click({ button: 'right' }) or .click({ modifiers: ['Shift']}), and .hover.
You can .check a checkbox or .selectOption on a radio button, and expect it .toBeChecked.
You can drag and drop an item.
You can fill a textbox, input field, or text area.

.toByVisible is a Playwright Locator Assertion.

Once you found an element by a Playwright locator, you can in the test expect it .toBeChecked, .toBeDisabled, .toBeEditable, .toBeEmpty, .toBeEnabled, .toBeFocused, .toBeHidden, .toBeInViewport, .toBeVisible, .toContainClass, .toContainText, /toHavaAccessibleDescription, .toHaveAttribute, .toHaveId, .toHaveRole, .toHaveText, .toHaveValue.
Each of these methods can be negated chaining a "not" value. Expect a text box .not.toBeEmpty.

Execute the Playwright Tests: Bun Run Test

Now, we should be able to run the tests that come with the initial Playwright scaffolding:

bun run test

The Playwright tests, are automatically run in Chromium, Firefox, and WebKit / Safari, and should pass.

You can see the results if you open up the bun-create-playwright folder in an IDE such as VS Code:

Go to the playwright-report folder
Right click on the index.html file that was produced.
Select "Open with Live Server"

You can drill down into the test to see more information about the tests.

What Does Playwright Install?

The Playwright scaffolder installs:

node_modules folder:

Created by a package manager (npm, yarn, bun, pnpm) when you install dependencies. Contains the actual source code of every package your project depends on, plus the dependencies of those dependencies (transitive dependencies), recursively. It's populated by running an install command (npm install, bun install, etc.), and never edited by hand.

Node's module resolution algorithm looks inside node_modules when you import/require a package by name.

Contains .bin, @playwright, @types. bun-types, playwright, playwright-core, typescript, undici-types. According to Claude.ai:

.bin: Executable scripts for installed packages. This is where npm/bun puts symlinks (symbolic links) to CLI (Command Line Interface) tools so you can run them via npx playwright or through package.json scripts, instead of needing global installation of every executable.

@playwright: Scoped npm namespace. Typically contains @playwright/test, the Playwright Test runner package (test runner, assertions, fixtures, config handling — what you import when you write import { test, expect } from '@playwright/test').

@types: Folder that Bun creates to hold TypeScript type declaration packages published under the @types scope on npm. These packages come from DefinitelyTyped.org, a large community-maintained repository on GitHub where volunteers write and publish TypeScript type declarations for JavaScript libraries that don't include their own. When a package doesn't ship built-in types (common for older libraries or runtime built-ins like Node.js itself), DefinitelyTyped contributors write a separate package describing that library's functions, arguments, and return values, then publish it to npm as @types/<library-name>These give you IntelliSense and type-checking for JS APIs.

bun-types: Type declarations for Bun's runtime APIs (Bun.file, Bun.serve, etc.), letting TypeScript understand Bun-specific globals if you're running/building with Bun instead of Node.

playwright: The full Playwright package, including the browser automation driver and CLI. This is distinct from @playwright/test: playwright is the core library (browser launching, contexts, pages), while @playwright/test builds the test runner on top of it.

playwright-core: The stripped-down core of Playwright without the browser-download step. playwright depends on this internally; you'd typically only see it listed as a top-level dependency if something needs Playwright's automation APIs without auto-downloading browser binaries.

typescript: The TypeScript compiler (tsc) itself, used for type-checking and/or transpiling .ts files.

undici-types: type declarations for undici, Node's HTTP client, pulled in automatically by @types/node.

playwright-report folder:

A folder to store HTML Reports produced after a test run.

According to Playwright.dev / HTML Reporter, "HTML reporter produces a self-contained folder that contains report for the test run that can be served as a web page.

"[...] By default, HTML report is opened automatically if some of the tests failed. You can control this behavior via the open property in the Playwright config or the PLAYWRIGHT_HTML_OPEN environmental variable. The possible values for that property are always, never and on-failure (default)".

test-results folder:

After running the test, it contains .last-run.json which contains "status: passed" with an empty array of failedTests.

tests folder:

example.spec.ts contains the pre-installed Playwright tests we have already covered.

.gitignore:

Want something to not be checked in, committed, and merged to an outside source? Place it here.

Playwright automatically has it so node_modules, local test-results, local playwright-reports, and things in the local .cache and .auth do not get committed.

bun.lock

This lockfile is produced by "bun install". Why have a lockfile? According to Code Protection Hub,

"The entire philosophy of lock file programming revolves around a concept called Semantic Versioning (SemVer). In a manifest file (like package.json), developers specify acceptable version ranges using caret (^) or tilde (~) symbols. For example, ^2.4.1 allows the package manager to download minor updates and bug fixes (up to version 3.0.0).

"While this allows projects to automatically receive security patches, it introduces a severe risk: Dependency Drift.

"Imagine Developer A runs npm install on Monday and receives version 2.4.1. On Friday, the package author publishes version 2.5.0 containing a subtle bug. When Developer B joins the team and runs npm install, they receive the buggy 2.5.0 version. The application breaks for Developer B, but works perfectly for Developer A. This is the classic "it works on my machine" nightmare.
The lock file acts as a historical snapshot. It records the exact version, the registry URL, and the cryptographic hash (SHA-512) of the code downloaded on Monday. When Developer B installs the project, the package manager ignores the flexible package.json ranges and strictly enforces the exact tree defined in the lock file".

package.json:

From HeyNode / What is Package.json:

"If you’ve worked with Node.js before, you have likely encountered a package.json file. It is a JSON file that lives in the root directory of your project. Your package.json holds important information about the project. It contains human-readable metadata about the project (like the project name and description) as well as functional metadata like the package version number and a list of dependencies required by the application.

"Your project’s package.json is the central place to configure and describe how to interact with and run your application. It is used by the npm CLI (and yarn) to identify your project and understand how to handle the project’s dependencies. It’s the package.json file that enables npm to start your project, run scripts, install dependencies, publish to the NPM registry, and many other useful tasks. The npm CLI is also the best way to manage your package.json because it helps generate and update your package.json file throughout a project’s life.

"Your package.json fills several roles in the lifecycle of your project, some of which only apply for packages published to NPM. If you’re not publishing your project to the NPM registry or otherwise making it publicly available to others, your package.json is still essential to the development flow.

"The name field defines the name of the package. When publishing to the NPM registry, this is the name the package will be listed under. It must be no more than 214 characters, only lowercase letters, and it must be URL-safe (hyphens and underscores allowed, but no spaces or other characters disallowed in URLs).

"The version field is very important for any published package, and required before publishing. It is the current version of the software that the package.json is describing.

"You are not required to use SemVer, but it is the standard used in the Node.js ecosystem and highly recommended. For an unpublished package, this property isn’t strictly required. Typically, the version number is bumped according to SemVer before publishing new versions to NPM.

"The license field lets us define what license applies to the code the package.json is describing. Again, this is very important when publishing a project to the NPM registry, as the license may limit the use of your software by some developers or organizations. Having a clear license in place helps clearly define what terms the software is able to be used under".


{
  "name": "bun-create-playwright",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "playwright test"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "type": "commonjs",
  "devDependencies": {
    "@playwright/test": "^1.62.1",
    "@types/bun": "latest",
    "@types/node": "^26.1.2"
  },
  "private": true,
  "peerDependencies": {
    "typescript": "^5"
  }
}

playwright.config.ts:

This Playwright Configuration file is the central configuration file for a Playwright test project. It lives at the root of your project and tells Playwright how to run your tests: which browsers to use, where to find test files, how to handle failures, what reports to generate, etc. Instead of passing a dozen command-line flags every time you run tests, you define the settings once here.


import { defineConfig, devices } from '@playwright/test';

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
// import dotenv from 'dotenv';
// import path from 'path';
// dotenv.config({ path: path.resolve(__dirname, '.env') });

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({

Breakdown of the main components

testDir: points Playwright to the folder containing your spec files.
fullyParallel: controls whether tests within a single file run in parallel workers rather than sequentially.
forbidOnly: a safety check, usually enabled only in CI, that fails the build if test.only was left in the code.
retries: how many times a failed test is retried before being marked as failed. Commonly set higher in CI than locally.
workers: how many tests run concurrently. CI environments often limit this to avoid resource contention.
reporter: determines the output format for test results (HTML report, list in the terminal, JUnit XML for CI integration, etc). You can specify multiple reporters at once.
use: shared defaults inherited by every test, unless overridden at the project or test level. Common settings here:

baseURL for relative navigation
trace for capturing a debuggable trace of the test run
screenshot and video for failure artifacts
headless to control whether the browser UI is shown

projects: defines the different browser/device combinations to run the same test suite against. Each project can also override anything in use. This is how you get cross-browser coverage without duplicating test code.

With the next post, we are going to make some changes to the project per Playwright Ambassador Butch Mayhew's recommendation.

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Introducing bun, a new package manager and JavaScript runtime environment

2026-08-05T21:58:25.754-04:00

New job? New toolset to explore! In this post I'll be investigating a new JavaScript package manager and runtime environment called bun.sh.

You may have noticed on Playwright.Dev's Getting Started / Installation section there are three different ways to install a ui or api test automation framework with the latest version of Playwright:

npm init playwright@latest
yarn create playwright
pnpm create playwright

For this blog post I'll be investigating a fourth way, the new toolset they just started using at work, bun.sh.

bun create playwright

What is npm, yarn, pnpm, and bun? These are called package managers.

What is a package? Developing a project, you don't have to figure out how to code everything yourself. You can include in your projects outside libraries, or "packages" of code from from the public JavaScript registry npmjs.com.

The problem is that these packages then need to be downloaded, installed, managed, and updated whenever they are updated. JavaScript projects can use a package manager, such as bun, to help this process.

When putting together a Playwright automation framework based around JavaScript and TypeScript, Playwright uses the Node.js JavaScript runtime environment, to run its test runner, execute automation scripts, and manage its browser instances.(See Nodejs.org / About )

There is a division of labor: Node.js runs these internal functions like running tests. NPM, Yarn, PNPM, and now Bun handle the packages.

Why are there so many package managers? Each has its own specialization. The original NPM is bundled already with Node.js. Yarn is a Facebook toolset that was created when NPM was found to work too slow. PNPM was created to be smaller and therefore faster. And bun was built for quick startup and installation speed, good especially if you have a CI/ CD pipeline that has a LOT of packages to install each time the tests run.

NPM, released in 2010, is the Node Package Manager for Node.js, a JavaScript Runtime Environment. According to NodeJS.org's An introduction to the npm package manager, "The npm installs, updates and manages downloads of dependencies of your project. Dependencies are pre-built pieces of code, such as libraries and packages, that your Node.js application needs to work". NPM comes with NPX (Node Package eXecute) is a command-line tool that runs third-party code, known as packages or dependencies, such as Playwright.

Yarn https://yarnpkg.com/, released in 2016, is Facebook's answer to NPM. Facebook needed a lot of JavaScript packages downloaded at the same time. NPM downloaded them only sequentially, one after another, causing it to be very slow the bigger the project became. According to the Engineering at Meta blog article Yarn: A new package manager for JavaScript, "As the size of [Facebook's] codebase and the number of engineers grew, we ran into problems with consistency, security, and performance".

Pnpm (Performant npm) is another JavaScript package manager, released in 2017. It uses symbolic links to organize the node_modules, increasing performance by decreasing the number of files needed.

Bun https://bun.sh released in 2021 and purchased by Anthropic in 2025, isn't just a package manager like NPM. It is a JavaScript runtime environment that was built to replace Node.js.

In spite of the differences, there are many similarities between these package managers:

Package Registry: All tools fetch packages from the official npmjs.com site.
Manifest File: They all rely on package.json to read and write project dependencies and scripts. They show the name, version, package manager, license, and more (See NPM Docs / Package.json or Yarn Package / Manifest )
Dependency Folders: Each tool downloads packages into a local node_modules folder inside your project directory.
Lockfiles: They use lockfiles to ensure reproducible builds across different environments, such as package-lock.json. (See NPM Docs / package-lock.json)

There is also one main difference between the others and bun. NPM, Yarn, and PNMP are all built on top of Node.js. Bun is built using an entirely new language interacting with Apple's JavaScriptCore engine, developed by Apple for Safari.

NPM with V8 vs Apple's JavaScriptCore

NPM is built with V8 and Bun is built on JavaScriptCore... But what is V8 and JavaScriptCore?

According to the Codeverse Chronicle article, JavaScriptCore vs V8 Engine, "JavaScriptCore [...] , is the JavaScript engine developed by Apple. It powers the JavaScript execution within Apple's Safari browser and is also utilized in various macOS and iOS applications. JavaScriptCore is an open-source project, making its codebase accessible to developers and allowing for contributions from the community.

"JavaScriptCore is deeply integrated into the Apple ecosystem, which means it's optimized for Safari and other Apple products. This tight integration ensures excellent performance and compatibility within the Apple ecosystem.

"[The] V8 Engine is developed by Google and is widely known for powering the Chrome browser. It's also used in other projects like Node.js and various server-side applications".

About Bun

Attempting to construct a Minecraft-like game, Jarred Sumner found Next.js painfully slow when trying to save and test his work. Bun initially interacted with the JavaScriptCore engine using the Zig programming language, but converted it to the Rust programming language in 2026.

Installing Bun: https://bun.com/docs/installation
Creating a project: https://bun.com/docs/runtime/templating/create
Running a project: https://bun.com/docs/runtime
GitHub site: https://github.com/oven-sh/bun

According to the GitHub README for Bun, "At its core is the Bun runtime, a fast JavaScript runtime designed as a drop-in replacement for Node.js. It's written in Rust and powered by JavaScriptCore under the hood, dramatically reducing startup times and memory usage.

"The bun command-line tool also implements a test runner, script runner, and Node.js-compatible package manager. Instead of 1,000 node_modules for development, you only need bun. Bun's built-in tools are significantly faster than existing options and usable in existing Node.js projects with little to no changes".

There is one more thing about Bun... it was purchased by Anthropic to be used in Claude Code.

From the press release, Anthropic acquires Bun as Claude Code reaches $1B milestone:

"Bun is redefining speed and performance for modern software engineering and development. Founded by Jarred Sumner in 2021, Bun is dramatically faster than the leading competition. As an all-in-one toolkit—combining runtime, package manager, bundler, and test runner—it's become essential infrastructure for AI-led software engineering, helping developers build and test applications at unprecedented velocity.

"Bun has improved the JavaScript and TypeScript developer experience by optimizing for reliability, speed, and delight. For those using Claude Code, this acquisition means faster performance, improved stability, and new capabilities. Together, we’ll keep making Bun the best JavaScript runtime for all developers, while building even better workflows into Claude Code.

"[...] Bun represents exactly the kind of technical excellence we want to bring into Anthropic," said Mike Krieger, Chief Product Officer of Anthropic. "Jarred and his team rethought the entire JavaScript toolchain from first principles while remaining focused on real use cases. Claude Code reached $1 billion in run-rate revenue in only 6 months, and bringing the Bun team into Anthropic means we can build the infrastructure to compound that momentum and keep pace with the exponential growth in AI adoption."

"As developers increasingly build with AI, the underlying infrastructure matters more than ever—and Bun has emerged as an essential tool. Bun gets more than 7 million monthly downloads, has earned over 82,000 stars on GitHub, and has been adopted by companies like Midjourney and Lovable to increase speed and productivity.

"The decision to acquire Bun is in line with our strategic, disciplined approach to acquisitions: we will continue to pursue opportunities that bolster our technical excellence, reinforce our strength as the leader in enterprise AI, and most importantly, align with our principles and mission".

How to Install Bun

According to the GitHub README for Bun, you can install Bun on your MacBook using HomeBrew, with Curl, with NPM, or pulling down an image from Docker.

On Windows, you can use PowerShell. Since I use a Windows PC at home, I'll install it that way:

powershell -c "irm bun.sh/install.ps1 | iex"

Restarting my PowerShell Terminal, checking that bun was installed correctly:

```
bun -version
```

... And it replied I am using version 1.3.14. Wonderful! It's alive! It's .... alive! Had the installation not gone successfully, it would have errored out.

In the next blog post, we will figure out how to use bun to create a Playwright framework.

Bun Create Playwright:

Part One: Introducing bun, a new package manager and JavaScript runtime environment
Part Two: What happens when you scaffold a Playwright framework and run installed tests using bun?
Part Three: Add Type Checking and TSConfig to Bun-Create-Playwright
Part Four: Checking code with lint, formatting it with prettier
Part Five: Running Tests with Playwright Test Explorer and Generating Tests with Codegen
Part Six: How Playwright Frameworks get configured with playwright.config.ts
Part Seven: Implementing Page Objects in Playwright
Part Eight: How to Configure Playwright Test to run smoke tests, headed tests, and debug versions through scripts in package.json
Part Nine: Setting up a CI/ CD pipeline with GitLab: Quality, Test and Report
GitLab: https://gitlab.com/tjmaher/bun-create-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Does giving a presentation via Zoom to the Sydney Testers Meetup make me an international speaker?

2026-06-29T20:51:55.895-04:00

If I presentation to the Sydney Testers Meetup in Australia, from my home in the Massachusetts, through Zoom, does that make me an international speaker? [ Vote in the New Poll, on LinkedIn ]

There is still time to sign up for the event at: https://www.meetup.com/sydney-testers/events/315166776

"T.J. Maher, the former organiser at Ministry of Testing Boston, blogger and speaker, will be presenting from the US on using the Detox framework to create effective test automation frameworks for React Native applications.

"Building a React Mobile Automated Test Framework using Detox + TypeScript by T.J. Maher

"Writing automated tests for a React Native mobile application is notoriously difficult. Mobile components display on the page, but are not fully loaded. Lengthy animations and slow-loading components take a while to finish. Timing issues cause your automated tests to error out giving the appearance of flaky tests.

"T.J. Maher will be sharing his experience tackling these problems using the open-source mobile testing framework, Wix's Detox, designed specifically for testing React Native applications.

* "Setting up a mobile test automation framework.

* "Vibe-coding a toy React Mobile Login page app to test against ( tinyurl.com/detox-demo ).

* "How to reduce timing issues and flakiness in automated tests.

* "Refactoring code into tests, page objects & base pages, separating out credentials and message strings for easier maintainability.

* "Setting up automated tests in CI / CD for iOS simulators and Android emulators with GitHub Action Workflows.

* "How developers can test their code before they push it into the main branch.

"About T. J. Maher:
---------------------

"A former organizer of the now defunct Ministry of Testing - Boston, T.J. Maher is a Senior Software Development Engineer in Test (SDET), based in the Boston/South Shore area of Massachusetts, with a decade of experience building web + mobile test automation frameworks. His recent work includes architecting a mobile automation framework from the ground up using Detox and TypeScript for a React Native application at SELF ID, a decentralized identity startup. Prior roles include MassMutual, Verily (Google Health), Threat Stack, Fitbit, and Intralinks.

"T.J. Maher is @tjmaher1 on LinkedIn, Twitter and BlueSky. His software testing blog is tjmaher.com.

"Follow him on LinkedIn!"

... And I am now a Senior SDET at AAA Life Insurance!

Slide Deck is at https://tinyurl.con/detox-demo-sydney

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Achievement Unlocked: T.J. Maher Elected to the Board of Directors of the Software Quality Group of New England - SQGNE.org

2026-06-17T19:26:28.021-04:00

I've been elected to be on the Board of Directors of the Software Quality Group of New England (sqgne.org) on the volunteer leadership committee!

The Software Quality Group of New England (SQGNE) is an all-volunteer, nonprofit professional community in New England dedicated to software quality. Since its founding, SQGNE has brought together QA engineers, SDETs, and software quality professionals for monthly educational presentations, discussions, and networking -- covering everything from testing techniques and process improvement to the latest tools and industry trends.

I also have founded and moderate an online monthly SDET Lean Coffee discussion group, bringing together software development engineers in test, quality engineers, and automation practitioners to exchange ideas, challenges, and emerging practices.

After our hiatus during the summer, we will be meeting monthly up in Burlington, MA. Check us out!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Testing and AI Workshop by James Bach of Rapid Software Testing - Notes

2026-06-12T14:10:14.954-04:00

Last week, I saw that there was a new half day workshop by one of the creators of Rapid Software Testing, James Bach, Testing and AI Workshop, and that they were offering a 50% discount for anyone who was unemployed, so I just had to attend.

Visit James Bach's Satisfice.com Download Page to download PDFs on topics such as Responsible Work (May 2026), AI Writing Policy (April 2026), Heuristic Test Strategy Model (December 2024), Rapid Software Testing Explored Class Appendices (December 2024) Why Testers? (Oct 2024) and ChatGPT Sucks at Being a Testing Expert (August 2023)
YouTube Channel: Rapid Software Testing
Check out James Bach's Classes at https://www.satisfice.com/classes (50% discount if unemployed)
The next Testing and AI Workshop will be July 6, 2026

"In each session, the instructors will first perform a live 'testopsy.' This is a demonstration of AI-assisted testing (using both assistive and agentic modes of AI) on a real product, accompanied by an analysis and explanation of what happened during the demo. During this part of the workshop, you may ask questions or offer critique.

"Next the instructors will challenge you to perform a similar process or solve a similar problem with the help of AI. You will have two hours. You will be able to work alone or in groups, as you like.

"Finally, the instructors will review and critique your work, if you choose to share it. At the end of the event, you will get to keep the videos".

I really enjoyed the class! It involved an hour-and-a-half webinar, where James Bach walks people though how he uses AI, a few hours where you can work on your own project for the course, then another hour-and-a-half webinar where course attendees could review what they came up with to analyze a site.

My only problem with the course was that my rambunctious seven year old was home sick, and I wasn't able to dedicate a few hours to come up with how to use AI to test a website, since I was busy making lunch and chatting with my kiddo.

What Is the Rapid Software Testing Approach?

As a software tester, I'm familiar with some of Michael Bolton and James Bach's work on the Rapid Software Testing and Satisfice websites:

Testing vs Checking (2009) and Testing vs Checking Refined (2024)
Michael Bolton's Testers: Get Out of the Quality Assurance Business
James Bach's Test Automation Snake Oil (2021)
Cem Kaner, James Bach, and Bret Pettichord's Lessons Learned in Software Testing: A Context-Driven Approach

... Back when I was a Meetup Organizer of the Ministry of Testing - Boston, I had many fans of RST try to convince me to no longer say "Automated Testing", and say only "Automated Checking", since computer programs can't think or test. And not to say "Quality Assurance" ever, since software testers are not product owners, managers, or developers, and cannot set the schedule, assuring any quality.

Although I have never taken any of Rapid Software Testing's classes before, but I did just purchase their newly released book, Taking Testing Seriously: The Rapid Software Testing Approach (2025), though.

Not familiar with Rapid Software Testing? Watch Michael Bolton give a talk at Quality Jam 2017,

"A Ridiculously Rapid Introduction to Rapid Software Testing"

https://www.youtube.com/watch?v=AS2kuD--z44&t=3018s

Or you can take a look at James Bach & Michael Bolton's presentation to CAST2020 (The Conference for the Association for Software Testing 2020) in Austin, TX,
Testopsies Dissecting Your Testing.

James Bach, during the class, shared some documentation:

Satisfice / Download / RST Appendices, which lists more about Rapid Software Testing, and the difference between it and "Factory Style" testing such as ISTQB, Six Sigma, TQM, or RUP. It also has some sample test plans.
Responsibility is the Human Moat, covering his Principles of Responsible Work v 2.0, detailing how it is the person who is prompting the AI is responsible for the work of the AI, and have the power to reject or remediate the work the AI produces.

James Bach and his brother John ran the workshop session. It turns out that James' brother is also a software tester!

James has been experimenting with LLMs since Chat GPT came out. This experimental workshop is to help unemployed people who are trying to reskill. It isn't exactly like his three day AI and Testing class. James and John tailored this workshop to be taken in one day.

Testing is a Human activity. AI is not testing. AI can not do professional testing. AI can do exploratory analysis. But the human needs to review and validate the result. AI is just another tool a software tester can use.

An LLM can make up stories that are partially true, but it can not tell what it was thinking when exploring, since LLMs do not think.

The Application under test we would be exploring with AI? The Online Gantt Chart at https://www.onlinegantt.com/.

Principles of Responsible Work

James & John Bach and Michael Bolton just released the above listed Principles of Responsible Work paper in May 2026. In spite of all their experience, they are always trying to see how they can make things better, which resulted in these principles. Some of the principles they cover are:

"Every non-trivial business comprises some set of services that enable it to function. Examples include sales, accounting, R&D, customer support, etc. These services must be sufficiently reliable or else the business will collapse.

"Every service entails the risk of failure. When failures occur, the business must be able to recognize them and recover. In regulated industries, risk management may be subject to specific process mandates.

"A 'responsible person' is a natural person in a business who is reasonably competent, prepared, and accountable for some service that sustains or defines that business. No matter what tools or processes are used within a business, someone must be responsible for them. To bear responsibility, a person must have sufficient capacity. For instance, neither a child nor a tool (such as AI) has the capacity (either legally or socially) to bear responsibility. Even adult humans may lack capacity, such as when an airline pilot has had insufficient sleep or is under the influence of drugs.

"A 'responsible service' is one that is performed in good faith by a responsible person. This may include interpreting and following procedures, improving skills, anticipating problems, and reporting to relevant authorities or clients, both inside and outside the business". ( See James Bach's Responsibility is the Human Moat for more )

This begs the question... what is the Responsible Operation of AI?

Responsible Operation of AI

"AI cannot bear responsibility.[1] AI is not a responsible person, and it would be meaningless to speak of a tool that operates in “good faith.” Therefore, it cannot provide a responsible service, nor can responsibility be delegated to it.
"An “AI agent” is always a tool operated by a natural person, irrespective of whether the person is monitoring it in real-time.
"Thus, the operator of an AI agent always bears responsibility for the behavior of that agent.[2] This includes anticipating availability issues, such as outages, token rationing, or poor performance.
"The responsible operator must assure adequate quality of the work; they cannot merely prompt and pray.

"Therefore, the operator must…

"be sufficiently skilled in the use of the AI tool.
"be sufficiently prepared to operate the tool in that context.
"be sufficiently alert to risks, anomalies, or defects that may occur in the work.[3]
"reasonably anticipate restrictions or interruptions of the services on which the work depends.
"feel empowered (and actually have the power) to reject or remediate any work done by AI. Otherwise, the operator becomes a scapegoat, a “moral crumple zone.”[4]
avoid or mitigate the special hazards of AI operation".

( See James Bach's Responsibility is the Human Moat for more )

Special Hazards of AI Operation

Responsibility is the Human Moat details the Special Hazards of AI Operation, with categories such as:

Technological: Service Outage, Service Adulteration
Interactional: Cognitive Overload, Cognitive Debt, Cognitive Atrophy, Cognitive Surrender, Anthropomorphism and Anthropomorphizing, Automation Bias, Chronic Stress
Managerial: Data Negligence, Reckless Spending, Violations of Law, Moral Crumple Zone, Business Disruption.

What is the Moral Crumple Zone? As James Bach puts it, "This can occur in an automated system that has a human operator or supervisor (such as a self-driving car with a safety driver, or an ordinary user of ChatGPT). It happens when a failure of the system is routinely and carelessly blamed on the human. The human functions as a sort of 'crumple zone' in the moral sense: a component designed to assume blame in order to deflect it from the automation".

What can you do with AI in Testing?

Product Analysis, identifying testable elements
Risk Analysis, identifying potential problems
Test Design, generating test ideas
Test Data Synthesis
Test Oracles, analyzing output
Static Analysis, analyzing code and data structures
Test Tool Implementation

This is what James finds most interesting using AI in Testing... AI can actually allow you to create your own test tools with Claude Code. You can experiment with tools to help you test.

If you point AI at a software product, and ask it to come up with a list of features. AI is slow, clunky, expensive, and misses a lot of stuff, so it is not quite there yet.

Not sure if AI can do something? Just try it out. Ask if it can do something. Experiment.

Heuristics for Evaluating GenAI Work

To use AI responsibility, it isn't enough to ask the right questions. James Bach and Michael Bolton prepared a checklist for being critical operators of GenAI. Some of the topics on the checklist are...

"What did you do?

"How did you select the tool? What is your familiarity with the tool you used? Did you use GenAI directly or is it some tool that imposes a certain structure on your work? If the latter, what do you know about the role GenAI plays in that tool?
"How did you approach the problem? Be prepared to tell the story of your strategy for involving AI in this work. What problem specifically were you trying to solve? How did you prepare to use the AI? What part of the problem did you present to the AI? Did you use an interactive assistive approach or an agentic autonomous approach?
"How did you construct your prompts? How much and what kinds of context did you provide? Did you start simple and refine? Or did you submit a complicated one-shot prompt with explanations, supporting data, or examples?
"How did you follow-up your original prompt to shape the AI response? Were you in charge of the process? Did you control the refinement or were you constrained by the tool? Did you try the same question, or variations thereof, multiple times, or was it one connected conversation? If there were autonomous agents involved, how did you review what they did?
"How did you preserve the record of what happened? Was it stored automatically? Is any of it lost? Can you reproduce it? Can you access the logs in case you need to report in detail or perform follow-up work?

"How did it go?

"Do you fully understand what the AI produced? If challenged, can you explain and defend (or critique) each part of the work?
"What did you learn while doing this work? Did the AI surprise you in any way? Did you try a new technique? Did you learn anything important about the AI tool’s ability to handle this kind of problem? Did you get any new ideas for how you could be either more productive or else more careful with AI?
"Do you feel it gave helpful responses to your prompts? How was the first response? Were your efforts to refine that response rewarded? Did it ultimately solve the problem?
"In what way was it worth the effort and cost to solve this problem using this AI tool? Or was it not helpful? Could you have solved it in a simpler way with a web search, FAQ, or some other traditional tool?
"What is your assessment of the efficacy and reliability of this AI tool for this kind of work? Should users be on their guard, or is the worst-case scenario either not severe or else very improbable?

"Did you notice any specific problems with the AI responses?

"Incorrectness. Did it hallucinate, give erroneous output, or contradict the data available to it?
"Corrections. Did it ever claim to correct itself? If so, did it in fact correct itself, or did it make a new error?
"Ungrounded. Did it fail to acquire the full details of your problem? Did it make silent, risky assumptions?
"Unnecessary. Did it provide anything that was unhelpful or wasteful?
"Omission. Could something important be missing from its output? Were there any important matters that it only acknowledged after you prompted it to do so?
"Bias. In what ways might it have been biased? To detect bias, you might need to consider other methods of solving the problem, including solving it yourself and comparing your assumptions to the AI.
"Incongruence or Self-Repudiation. Did it make claims and later repudiate them? Did it say it would follow a procedure and then fail to follow it?
"Awkward Interaction. If the process was conversational, did you feel rushed or overwhelmed during it? Did the AI lose track of questions or forget the state of the interaction? Were you in control during the process? Was the AI too accommodating, sycophantic, or placating?
"Required Intervention. What part of its helpful responses depended on you to guide or correct it? If you had not been vigilant and provided only blank encouragement, would it have given a poor solution? Could a complete novice with no education or training have reached a good result using this AI tool?

"How specifically did you evaluate the responses?

"Thoroughness. How closely and carefully did you study what the AI did? What evidence do you have that it performed acceptably? This can be a time-consuming process, so it’s common for users to perform only minimal checking of the output. However, GenAI is very good at creating credible and confident responses that are completely wrong. Never trust your first impression of GenAI responses.
"Testable Output. Did the tool provide supporting data or any other feature that helped you evaluate its output? What references did it provide? What indications of its thinking? What kind of metadata? How helpful was its formatting of the response? How did it guide you to help you understand the output?
"Possible Blind Spots. In what way could the AI have given you an erroneous or otherwise problematic response without you knowing it? Are there plausible-seeming facts in the response that you didn’t check and about which you have no personal knowledge? Is there any content that might be incomplete? If GenAI performs any operations or transformations on data, could it have dropped some of that data?

"Now what?

"Double down on this approach? Maybe this experience showed promise, and you want to go further with it and complete the work.
"Drop it and try something else? Maybe the journey was the reward. Maybe you’ve already gotten what you need from this without needing to preserve any of the specific results that the AI gave you. Maybe this approach to solving the problem didn’t work and you should try something else.
"Consult with co-workers or clients about it? Remember that your reputation could depend on how others feel about your work. You might to run it by them and see what they think. They may have ideas for improving it, or concerns to share with you about it.
"Raise concerns about using AI for this kind of work? Did the AI do anything that scared you or represented a specific hazard that you should escalate?
"Test it more? Maybe you’re not sure what you’ve got and need more information to have an opinion. You can analyze what you’ve got or gather more information so you CAN analyze it.
Define or modify a protocol for using AI to do this? Maybe you know enough to make decisions about how you or your company will apply AI to problems of this kind".

Using AI to Test: Demo

James Bach demoed testing The Online Gantt Chart at https://www.onlinegantt.com/ with prompts using Chat GPT and Claude Code.

James kept track of his work in a worksheet based on his "Session Based Test Management" worksheet ( Download PDF ) he and his brother developed at Hewlett Packard, he tracks time testing, metrics, etc.

During the first part, a Survey Session, James was trying to see what he can manually explore and discover. How can you use this tool? What are the boundaries for text boxes? James did a preliminary analysis on what you can start exploring with AI.

James, after exploring for a bit, went to the setting screen, took the raw Document Object Model code of the web app, and add it to Chat GPT:

Prompt: "Analyze the settings for onlinegannt and list them for me"

Chat GPT then created an outline of all the settings.

Could you have told ChatGPT and get the information itself? Yes, but Chat GPT cannot press buttons. It might not have been able to go to the Settings screen. Sometimes, you don't want to "Prompt and Pray".

Prompt: "I would like to do combinatorial testing on the factors in this settings panel. What combinations do you suggest?"

James is doing what he calls "step back prompting". Start with a general prompt to see what information he can get, a soft prompt.

A "hardened prompt" is very specific. James doesn't start off with this because he is worried what the biases of the LLM are.

James also doesn't want to experience cognitive overload, receiving more information that he can process. He is exploring the app using AI to find subtle bugs he might miss.

Ask Yourself: What Is The Main Thing We Need to Test?

With the Online Gannt chart, the main thing we need to test is the display. If information is corrupted, if task get dropped, this product is useless.

Focus on: Is it usable? Reasonably charismatic?

James notices that as input, it accepts GANTT Files. If you want to enter a lot of data to see how the app handles it, you don't want to do it by hand. You can use AI to help you out.

Using AI To Create Tools

How can you generate a lot of data? You can use Claude Code to create a tool.

James took a GANTT file, saved it to an empty directory.

Prompt: "The file onlineganntt.gantt is a json that describes schedule data for a gantt chart. I need a Pythin program that generates random schedules that will enable me to test gantt charting software. The program must produce random schedules with N tasks (N is specific on the command line). The task should be between 1 and 10 days in length and connected using various dependency relations including no relation. Give the tasks generic self-descriptive names. The tool should process properly formatted JSON file with the suffix '.gantt' plus a csv table listing the details for each different task (for use as an oracle)."

What is an oracle? It is a principle or mechanism by which you recognize a problem. Rather than being a rigid, absolute tool that delivers binary "pass" or "fail" checkmarks, Rapid Software Testing treats an oracle as a heuristic -- a fallible, experience-based guideline that points you in the right direction but can occasionally fail or mislead you.

If this prompt is successful, James would get a data file he would be able to input into Online Gantt.

With the CSV file, we will be able to check for accuracy in case the AI had messed up generating the file. Testing the Import functionality is part of our tests.

Setting Claude protocols using Claude.md

James has in his claude.md file for Claude Code, where it has been told to push back and ask questions on any prompts

"Critical Working Protocol:

"When I ask you do do something for me, or produce text, code, or anything else, ask questions to get more information about any ambiguous and critical matters.
"During any task, identify any risky or critical assumptions that either you or I may be making. Declare those assumptions.
"Present a plan and get approval before taking action. Write the plan to a file called "plan-[project]-[date]-[ordinal value].md". The plans file should be stored in a plans directory that is located in the project directory. The plan file MUST include the full text of my prompt that immediately precipitated that plan.
"Before writing new code, always inform me of existing tools (command-line utilities, libraries, or open-source projects) that could fulfill the need. I want a detailed analysis. Include a section called 'Existing Solutions' that includes this information when asking clarifying questions or presenting plans.
"If multiple approaches exist (built-in tools vs writing code), present all options with trade-offs so I can choose.
"If I ask for analysis, always put it into a file. If I don't say otherwise, perform a detailed analysis with high effort"

"Testing:

"DO NOT TEST anything you write for me, unless I ask you to.
"When you write code and it is ready to try, let me try it first and I will report any problems

"Git Style:

."I want to be able to revert to earlier versions if I don't like the current version.
"Do not commit anything to Git unless the file is already being tracked."

When Claude started thinking about the problem, it then pushed back asking about task hierarchy, and alignment of subtasks, what resources to use, what the output filename should be, etc. James was then able to fine tune the prompt.

Why tell Claude not to test? James thinks it wastes tokens. Also, anything Claude claims to have tested, James does not believe it. He wants to test things out for himself. James does not want to be lulled into a false sense of security.

Ask Claude to keep a file of what you are doing

Everything you asked Claude to do will go into a file, as a record of what you were working on with Claude. Because this information is going into a file, when an employer asks, "What did you do?" you as a tester can give an answer.

It also gives you a record you can review when you are done experimenting. James needs to piece it together for an article or a blog post or in the class material.

The other day, James asked Claude to use an MCP that was connected to Playwright, and use Playwright to explore Online Gantt. It took a few days to figure out how to do it, but it worked. Claude was slow, but it did explore it and give an analysis. When James said, give me the analysis, it gave the analysis. When asked to put the analysis in a file, it churned out for five minutes a new analysis in the file, three times larger that what it gave James.

James asked, what are you doing? Why is there so much more in the file? Claude said, well, it gave on screen only an overview of the analysis.

Don't Accept Any Tool Claude Generates. Test the Tool.

After a lot of back and forth, Claude generated a tool in Python, generate_gantt.py.

Two files were produced: The file that was meant to be imported to Online Gantt, and the CSV file that had a copy of the data that could be examined in Microsoft Excel.

When the import file was uploaded, you could see that a lot of records were now showing... but were they correct?

Only by comparing the Online Gantt chart with the CSV could you see if the upload was interpreted correctly.

Careful Not To Have The AI Get Too Ahead of You!

Working with AI, James was reminded of the western trope of the cowboy being dragged by horses across the desert.

If you are being given too much data by the AI for you to process, you might want to shrug without reviewing it all, and think that if the first few results were fine, the rest of the results were fine.

It is best to work iteratively, one step at a time. Manage your cognitive load carefully.

Remember: You Need To Explain What You Are Doing With AI!

Once you finish a test session, ask yourself (and take notes on):

What did you do?
How did it go?
Did you notice any specific problems with the AI responses?
How specifically did you evaluate the responses?
Now what?

These are discussed above with the Heuristics for Evaluating GenAI Work by James Bach.

These reports you are having AI generate are good as a reference for yourself. It still might have hallucinations, but might be good enough for you. Recognize it might not be good enough for your employer. Do not give your employer any AI sloppiness that would waste their time. You need to do your due diligence.

People will need to trust you, trust your work, and trust you are using AI well, James mentioned. If you do not review the work, you are responsible for any mistakes the AI might have made.

As you practice using AI with testing, James predicts that one of the skills will be how to be efficient and effective in your use of tokens.

James relayed a time when a prompt he was doing was set up to retry on a failure. The AI then retried the same prompt 500 times in an hour. It didn't cost much, only ten dollars, but he just burned through ten dollars because of the tooling he was using.

There are some things that the current generation of AI does really well, such as figuring out data formats. Before, he would try to figure it out first, then give it to the AI. Now? He doesn't have to do that. Most of the time it figures it out and does it well. His default approach is give it the raw file it needs and stand back. If it fails, then step in.

Some things AI does not do well. If you ask it directly on questions you should ask when putting together a test plan, it may list out the fifteen things you need to ask yourself. But, if it starts "testing", it will not do that. This is because AI is not a good tester, since it gets its information from the internet, and most people on the internet are not a good testers.

You can try to make AI to be a better tester, by giving it such as James Bach's Heuristic Test Strategy Model (Download PDF). James was thinking about creating a file which details "What is Rapid Software Testing"?

... James said that maybe by then next time this session is run, he can create a skill for Claude that has this skill?

Be cautious when analyzing long lists of results

James mentioned that when you are reviewing the list of what was examined by AI, you need to ask yourself, "What's not here? What did the AI not tell me"

There is a certain cognitive bias called the part-set cuing effect.

In the Wikipedia page, under "Memory Inhibition", "The 'part-set cuing effect' was initially discovered by Slamecka (1968), who found that providing a portion of to-be-remembered items as test cues often impairs retrieval of the remaining un-cued items compared with performance in a no-cue (free-recall) control condition".

James described it as such, "If someone said, I need you to think about all the fruits that you can, for instance, apples, bananas, pineapples, lemons, oranges", it's harder to think of other types of fruit, since your brain says, "Yep, those are fruit!". If you weren't given anything, your brain can think of more things. To protect yourself, before you see what AI comes up with, scribble down your ideas first.

New Podcast: Humans in the Way

James Bach mentioned that he started a new podcast on the Rapid Software Testing YouTube channel, called Humans in the Way. He takes things that AI can do, breaks it down and analyze it.

Humans in the Way #1

https://youtu.be/X1dWeqYlF74?si=JWcSDS6bBhtJL6iT

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

New Position Unlocked: Senior SDET at AAA Life Insurance, starting Monday, June 15th, 2026!

2026-06-03T22:16:39.326-04:00

I have two announcements: I've accepted a job offer as a Senior SDET role at AAA Life Insurance Company, and will be starting Monday, June 15th! And I have been nominated to be one of the volunteer Directors on the leadership board of the Software Quality Group of New England.

Man, the job market is brutal! It took me four months of job searching in 2025 to find SELF Id when MassMutual outsourced its technology department. And it took me four months of near constant job searching in 2026 to find AAA Life when I was caught up in the second round of SELF's layoffs in the end of January.

After throwing countless resumes into the void, submitting post after post on LinkedIn, experimenting with countless automation frameworks and programming projects on tjmaher.com, and spamming my entire LinkedIn Network (sorry about that!), and everyone I have worked with in the past twenty years as a software tester, desperately looking for job leads, it was a staffing firm working for AAA Life who found me. Thank you, Refactor Talent! Rewriting my LinkedIn Resume to make sure that I was using proper keywords in my "About" section of my LinkedIn Profile seems to have done the trick!

I'd also like to announce that I found another awesome software testing community to be a part of: The Software Quality Group of New England, a local non-profit meeting monthly in Burlington, MA. Come join us! And we are always looking for in-person speakers.

I've been missing being part of a software testing community! I was Meetup Organizer for the now defunct Ministry of Testing - Boston group from 2016 to 2024, loving every minute I was trying to find speakers, sponsors and hosts. I made so many great connections during my tenure. It's sad the Meetup group was shut down by the Ministry of Testing UK a few months ago, but I am glad to be part of a new software testing group!

In the past month as a new member of the SQGNE, I have:
* Given a talk to the group on building a mobile automation test framework
* Set up a monthly lunchtime discussion group that I'm planning to have meet the first Tuesday of each month.
* Been nominated to be one of the three Directors on the SQGNE leadership committee.

Check us out at SQGNE.org!

If you are in Massachusetts and are looking for a software testing group to present to, we are building out our calendar for the 2026 - 2027 right season now.

Join the SQGNE LinkedIn Group! https://lnkd.in/eNYUMG24

Thank you so much, everyone, for all of your support during my job search. It is really appreciated!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

SDET Lean Coffee #1: With AI, what is useful testing and what is workslop? SQGNE, June 2, 2026

2026-06-02T16:35:29.876-04:00

With AI, what is useful when it comes to testing and what is workslop? How do you create workflows in AI? With AI producing massive amounts of code, how can a tester keep up?

These are some of the topics attendees decided to talk about in our first ever SDET Lean Coffee, as part of the Software Quality Group of New England ( sqgne.org ). We exchanged war stories, horror stories, shared insights, and provided a bit of group therapy as we talked about the stress involved being the main support role of the software development team.

A surprise guest was Lisa Crispin (LisaCrispin.com), author of "Agile Testing: A Practical Guide for Testers and Agile Teams". Lisa has been working with DORA, Google Cloud's DevOps Research and Assessment division.

Recently, Lisa gave a talk teaming up with the "Beyond Quality" podcast, sharing what she has been doing "AI, testing, and the DORA AI Capabilities Model" at Lisa's site at https://lisacrispin.com/2026/04/20/ai-testing-and-the-dora-ai-capabilities-model/ discussing:

The Dora AI Capabilitues Model
How we need to test AI agents since AI agents can degrade over time

What is a "Lean Coffee"?

"Lean Coffee is a structured, but agenda-less meeting. Participants gather, build an agenda, and begin talking. Conversations are directed and productive because the agenda for the meeting was democratically generated". This format arose over fifteen years ago, when "Jim Benson and Jeremy Lightsmith wanted to start a group that would discuss Lean techniques in knowledge work – but didn’t want to start a whole new cumbersome organization with steering committees, speakers, and such. They wanted a group that did not rely on anything other than people showing up and wanting to learn or create", according to LeanCoffee.org.

When Is The Next SDET Lean Coffee?

SDET Lean Coffees for the SQGNE will (usually) be held the first Tuesday of each month at 12:00 pm to 1:00 pm EDT.

Interested in attending the next session Tuesday, July 7th?

Register at the Software Quality Group of New England website at https://www.sqgne.org/

Happy Testing!
-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Come join us at the next SQGNE Meeting! Open-Source Malware: Defending Your Software Supply Chain From Evolving Threats - June 17, 2026

2026-06-01T22:34:08.323-04:00

"Open-Source Malware: Defending Your Software Supply Chain From Evolving Threats" will be the topic of the next Software Quality Group of New England (sqgne.org) meeting.

Speaker: Bryan Whyte, CISSP Director, Solutions Engineering @Sonatype

Date: June 17, 2026 @ 6:00 pm

Join us on Zoom or in person at Burlington, MA ( Register Here )

"Bryan Whyte breaks down the latest wave of open source malware, explains how these threats diverge from traditional vulnerabilities, and shares actionable steps for organizations to defend mission-critical software.

"As organizations deepen their reliance on open-source software, evolving security threats are reshaping the landscape at an unprecedented pace.

"Threat actors are now increasingly targeting development pipelines and trusted ecosystems like npm to orchestrate supply chain attacks with significant downstream impact. Incidents such as the 2025 Shai-Hulud npm campaign, the XZ Utils backdoor, and the widespread compromise of over 23,000 GitHub repositories illustrate how open-source malware has quickly become a critical, top-tier threat built to evade legacy scanning and exploit trust woven into modern delivery pipelines.

"--The shifting tactics of threat actors targeting npm, PyPi, GitHub, and development pipelines

"--Key differences between open-source malware and traditional malware or vulnerabilities

"--The most prevalent malware types and tactics driving today's software supply chain attacks

"After spending 20 years in software development, Bryan started his journey into Application Security in 2015 with the AppScan tool suite for Static, Dynamic and Mobile Application Security Testing. In 2018, he expanded his Cybersecurity proficiency, earning the Certified Information Systems Security Professional (CISSP). In 2019, he was excited to join Sonatype due to the explosive growth of open-source software, which has made Software Composition Analysis (SCA) a critical aspect of Application Security".

See you there!

Happy Testing!

-T.J. Maher
Software Engineer in Test
BlueSky | YouTube | LinkedIn | Articles

Need a speaker at your Software Testing Meetup? How about a talk about putting together a React Native mobile testing framework?

2026-05-29T20:22:04.090-04:00

Are any software testing Meetups looking for a speaker? I have a talk, all ready to go!

After surviving the first round of layoffs last December, I scrambled to organize my research notes on building a mobile automation framework into a presentation. I managed to finish it days before I was hit by the second round of layoffs at the end of this January.

So far, I have given my presentation, Building a React Mobile Automated Test Framework Using Detox + TypeScript to:

TestGuild.com Automation Guild 2026 via Zoom (4/6/2026)
Software Quality Group of New England ( SQGNE ) @ SCRUM.org, Burlington, MA - https://tinyurl.com/detox-demo-sqgne (5/20/2026)
Sydney Testers Meetup via Zoom (scheduled on 6/30/2026)

Two Questions:

If I give a talk to a software testing Meetup in Australia via Zoom, even though I never left my home office in Massachusetts, would I be considered an "International Speaker"?
Does anyone else want me to volunteer to speak at their software testing meetup? The SQGNE season is ending for the summer, so I will have some time available.

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Practicing Playwright: API Testing, Intercepting Network Requests, and Mocking APIs

2026-05-27T22:20:30.432-04:00

How would you intercept a network request and use the data for assertions? Mock a network request using those assertions? And make sure that all data is loaded in the UI before making your assertions, and that all tests can pass when run?

We will be walking through Butch Mayhew's code that answers all of these questions, part of his LinkedIn Learning course: Playwright Essential Training. We will be examining the shopping cart test site PracticeSoftwareTesting.com and looking at code on Butch's companion GitHub site on how to mock out the API and use them in the UI tests.

Playwright has a lot of features when it comes to API testing. You can intercept network requests, aborting, modifying, and mocking network requests. You can also simulate a slow network. This is all done through the Playwright methods: page.route.

A route is the specific path or URL a client uses to request data or trigger a function on a server. GET /product would be an API call that gets everything from the address string called products, which then retrieves all products from the API endpoint called "products". The endpoint is "products" and the route is the name that accesses the endpoint.

The Playwright API has a method called route, which allows Playwright to monitor and modify browser network traffic.

Playwright.dev / Docs / Network: https://playwright.dev/docs/network
Playwright and page.route: https://playwright.dev/docs/api/class-route
Playwright Mocking API Guide: https://playwright.dev/docs/mock

Adding page.route to your test allows you to do things such as:

abort the route request with route.abort(), simulating various error codes such as accessdenied, addressunreachable, blockedbyclient, blockedbyresponse, connection aborted, connectionclosed, timedout, etc.
continue sending the route's request with optional overrides such as changing the request URL, the headers, the method such as GET or POST, and the post data of the request.
fetch the request without yet fulfilling it, so the response can be saved, modified, and fulfilled later. You can also change the HTTP headers, follow redirects automatically, etc.
fulfill the route's request.
request that the request is routed.

About Our Site Under Test

PracticeSoftwareTesting.com is a shopping cart with a web-based front-end pulling data from a web-based back end. Tools can be filtered out by category, such as hand tools (hammer, hand saw, wrench), and power tools ( grinder, sander, saw ) and by brand, such as ForgeFlex Tools and MightyCraft Hardware. Check the appropriate check box, and cards are display in a grid fashion of products.

These products can also be sorted, such as by "Name (A - Z)" and ""Name (Z - A)"

Explore the HTML of the site by opening up Google Chrome, right clicking on the first product in the grid to "Inspect" to open Chrome Dev Tools and look at the Elements tab to find locators:

Product Grid: Class col-md-9

This class can be used as a locator to find the product grid.

About the Products API

Let's see an example product returned by the Products API Endpoint:

Go to https://practicesoftwaretesting.com/
Under the Filters category, check the checkbox "Saw" under "Power Tools". You will see one item display: a circular saw.

But where did the product information come from?

Under Chrome Developer Tools, select the "Network" tab
Uncheck and then check the "Saw" checkbox, and you will see a new products query at the bottom looking like products?page=page=0... select that.
Select the "Response" tab.

  
"current_page": 1,
    "data": [
        {
            "id": "01KSMVRWS80N7909CZTE7ACJQS",
            "name": "Circular Saw",
            "description": "Portable circular saw powered by a robust ...",
            "price": 80.19,
            "is_location_offer": false,
            "is_rental": false,
            "co2_rating": "D",
            "in_stock": true,
            "is_eco_friendly": false,
            ...
            "category": {
                "id": "01KSMVRWMQCBR9XDQ5TGC47MS6",
                "name": "Saw",
                "slug": "saw"
            },
            "brand": {
                "id": "01KSMVRW6HT9HQZ2E113FQEFX7",
                "name": "ForgeFlex Tools"
            }
        }

All this data was returned in a JSON format when the web app took the information in the filter and made the call to the product API

https://api.practicesoftwaretesting.com/products?page=0&between=price,1,100&by_category=01KSMVRWMQCBR9XDQ5TGC47MS6&is_rental=false

All these parameters can be used to test out how your UI behaves under various conditions.

How to Intercept Network Responses

We can intercept the information coming back from the API using route.fetch and then using route.continue:

await page.route(
  "https://api.practicesoftwaretesting.com/products**"
  async (route) => {
    const response = await route.fetch();
    products = await response.json();
    route.continue();
  }
};

A glob, or "global command", is a pattern of characters used to find files, folders, or web addresses. The "**" is a globbing wildcard that matches all the nested folders which come after it. Had Butch used only one "*" it would have only returned the next level in the folder hierarchy.

What does this code do?

await page.route: Tells Playwright to listen to network requests that match this URL pattern.
async (route): Creates a callback function that triggers each time a call is made.
const response: Declares a new constant variable called "response" where we will store the HTTP response retrieved.
await route.fetch(): Performs the actual network request and retrieve the reponse.
products = await response.json: Extracts the reponse payload as a JavaScript Object (JSON) and assigns it to a variable named "products".
route.continue: Proceed with the original request.

How to Abort API Requests

Why would we want to abort a certain network call? Butch mentions in his LinkedIn Learning class how a certain pop up was interfering with a UI test he was writing.

You can also use route.abort in order to simulate the Product API being down, and check to see that the UI is behaving as it should.

Sample code Butch listed:

await page.route(
  "https://api.practicesoftwaretesting.com/products**"
  (route) => route.abort()
};

With this code, we are seeking to intercept anything from the Products API, and when the match is found, cancel the network request before it ever reaches the server.

How to Mock a Product

Let's say we want to see how our UI behaves with product names that are quite long, prices with many digits, or if a product is in stock. We can add these test products to the database and then search for them, or we can:

Capture the response
Alter any of the fields returned
Submit that mock data as if coming from the API itself.



await page.route(
  "https://api.practicesoftwaretesting.com/products**"
  async (route) => {
    const response = await route.fetch();
    const json = await response.json();
    json.data[0]["name"] = "Mocked Product";
    json.data[0]["price"] = 100000.01;
    json.data[0]["in_stock"] = false;

    await route.fulfill({ response, json }  ))
  }
};

... For this, we are editing the name, price, and the variable to see if the item is in stock or out of stock.

json.data[0] represents the very first product listed.

How to Mock API Requests with HAR files

If you want to capture all HTTP traffic and store it, you can store it in an HTTP Archive Record, or HAR file.

According to Playwright.dev, a "HAR file is an HTTP Archive file that contains a record of all the network requests that are made when a page is loaded. It contains information about the request and response headers, cookies, content, timings, and more. You can use HAR files to mock network requests in your tests. You'll need to: 1) Record a HAR file. 2) Commit the HAR file alongside the tests. 3) Route requests using the saved HAR files in the tests.

"To record a HAR file we use page.routeFromHAR() or browserContext.routeFromHAR() method. This method takes in the path to the HAR file and an optional object of options. The options object can contain the URL so that only requests with the URL matching the specified glob pattern will be served from the HAR File. If not specified, all requests will be served from the HAR file.

"Setting update option to true will create or update the HAR file with the actual network information instead of serving the requests from the HAR file. Use it when creating a test to populate the HAR with real data".

await page.routeFromHAR(".hars/product.har", {
  url: "https://api.practicesoftwaretesting.com/products**"
  update: false,
});

With this code we are:

Telling Playwright to intercept all network traffic for the current page using a recorded HAR file and store it in the root directory folder we are calling ".hars" in the file "product.har".
Update: false runs the test in playback mode. We can run the test once with "update: true" to capture and store the information, then reset it to "update: false" to not have the data overwritten.

The problem with this approach is that sensitive data such as passwords might be part of the HAR file. To sanitize sensitive data, see the Medium article: HARmageddon is cancelled: how we taught Playwright to replay HAR with dynamic parameters

Simulating Slow API Requests

Let's say we want to see how the UI responds when it takes a good 4 seconds (4000 milliseconds) for the API to respond.

await page.route(
  "https://api.practicesoftwaretesting.com/products**"
  async (route) => {
    await page.waitForTimeout(4000)
    await route.continue();
  }
};

We can then add a bit of lag into the system, then continue as normal.

Write a Test: Validating Brands

Putting together Butch's lessons above, Butch had an exercise listed: How can you validate the brands listed in the Filter on the web site matches the ones listed in the brand endpoint? And how can you do it by intercepting network data?

If you go to https://practicesoftwaretesting.com/ you can see that under the Filters, there are two major brands: ForgeFlexTools and MightyCraft Hardware.

As mentioned earlier, these products can be sorted, such as by "Name (A - Z)" and ""Name (Z - A)"

Explore the HTML of the site by opening up Google Chrome, right clicking on the first product in the grid to "Inspect" to open Chrome Dev Tools and look at the Elements tab to find locators:

Product Grid: Class col-md-9

This class can be used as a locator to find the product grid.

We can get the information about the brands from the API, which Butch stored in the .env file as API_URL.

Using this information, we can write an end-to-end test to verify that the dynamic brand data loads correctly on a web page.

This test verifies that the web application displays a list of product brands on its front end by intercepting the backend API network responses.

home.spec.ts

test("validate brands by intercepting network data", async ({ page }) => {
  let brands: any;
  const apiUrl = process.env.API_URL;
  await test.step("intercept /brands", async () => {
    await page.route(apiUrl + "/brands", async (route) => {
      const response = await route.fetch();
      brands = await response.json();
      route.continue();
    });
  });
  await page.goto("/");

  const productGrid = page.locator(".col-md-9");
  await expect(productGrid).toBeVisible();
  await expect(page.locator(".skeleton").first()).not.toBeVisible();

  const brandFilterSection = page.getByText("SortName (A - Z)Name (Z - A)");

  for (const brand of brands) {
    await expect(brandFilterSection).toContainText(brand.name);
  }
});

tests/homepage/home.spec.ts

let brands: any: Declaring an empty variable to store all the brands found when they are fetched from the network. It is declared outside the test step so we can use it later on. It is set to "any" in order to keep it undefined for now.
const apiUrl gets the API URL from the .env folder.
await test.step("intercept /brands: This time, instead of products, we are registering a handler that fires whenever a request matches the URL pattern of "/brands".
route.fetch makes the real network request on behalf of the browser.
response.json parses and captures the response body into the variable, "brands".
route.continue forwards the original response to the browser unchanged, so the UI renders normally.

The UI is getting the real data, with the test getting a copy of it.

Then we are doing:

page.goto("/"): Loads the page and the browser makes the call, including GET /brands.
When the page loads and the browser calls GET /brands, the handler executes and the "brands" variable we set up is populated.
page.locator: Sets up the contant variable productGrid as a locator for the product grid.
We expect the product grid to be visible, showing that the page is now loaded, and that the initial skeleton element placeholders PracticeSoftwareTesting.com has is no longer visible.

Finally, we are validating the UI data against the API data we have captured:

We locate the Brand Filter Section web element by text, where we can sort the filter.
We then loop through every single brand object that was captured in the "brand" variable.
We confirm that each brand name provided in the backend API is accurately printed as text inside the frontend filter sidebar.

Write a Test: Validating Categories By Mocking

Let's say we want to write a test that the Categories API correctly adds mocked categories and subcategories to the left side index. Butch shows an example:

home.spec.ts

test("validate categories render in UI by mocking", async ({ page }) => {
  let categories: any;
  const apiUrl = process.env.API_URL;

  await test.step("intercept /categories", async () => {
    await page.route(apiUrl + "/categories/tree", async (route) => {
      const response = await route.fetch();
      const json = await response.json();
      categories = json.data;

      json[0].name = "Mocked Category";
      if (json[0].sub_categories && json[0].sub_categories.length > 0) {
        json[0].sub_categories[0].name = "Mocked Subcategory";
      }
      await route.fulfill({ response, json });
    });
  });
  await page.goto("/");

  const productGrid = page.locator(".col-md-9");
  await expect(productGrid).toBeVisible();
  await expect(page.locator(".skeleton").first()).not.toBeVisible();

  const categoryFilterSection = page.getByText("SortName (A - Z)Name (Z - A)");

  await expect(categoryFilterSection).toContainText("Mocked Category");
  await expect(categoryFilterSection).toContainText("Mocked Subcategory");
});

tests/homepage/home.spec.ts

Walking through this code:

Create a new test:

A new Playwright test case is defines, using a page fixture.
We declare the undefined variable "categories" and add the API URL into a constant variable.
We create a new test step, letting readers know we are going to intercept the /categories data.

Intercept the response:

With page.route(apiUrl + "/categories/tree" we tell the browser to watch for and pause any outgoing requests matching /categories/tree.
Route.fetch allows the real request to fetch the real response, and store it in the variable called "response".
We then take that response in json format and place it in a variable we are calling "json".

Edit that response so the first item has "Mocked Category" and "Mocked Subcategory":

We are going to take that first name, [0] and assign it the string "Mocked Category".
If there are subcategories in that intercepted response, and the sub_categories length is greater than zero, we will assign the string to it: "Mocked Subcategory".
route.fulfill will send this modified JSON response back to the frontend application instead of the original server data.
Hopefully, in the left hand side panel of PracticeSoftwareTesting.com, under "Category" we will see "Mocked Category" and "Mocked Subcategory".

Load the Page, Wait for the Product Grid to Appear:

We set up productGrid as a web element, using the class we found .col-md-9 as a locator.
We expect the productGrid to be visible and the .skeleton to not be shown.

Expect the "Mocked Category" to appear!

We declare a new web element, categoryFilterSection, using the SortName as a locator.
Can we verify that the phrase "Mocked Category" appears? It passes!
Can we verify that the phrase "Mocked subcategory" appears? It passes!

Whew! And this is just a taste on what Playwright can do to test at the API level. You can read more in the Playwright.dev Docs which covers API Testing, and the classes API Request, API Request Context, and API Response.

Practicing Playwright:

Part One: Time to Practice Playwright!
Part Two: New Playwright Course: Playwright Essential Training by Butch Mayhew
Part Three: Hands on Automated Testing with Playwright by Butch Mayhew is the start of a wonderful conversation with the Playwright community!
Part Four: What happens when you pair Playwright with something other than TypeScript?
Part Five: Logging in by Storing and Using an Authentication Cookie in Your Automated Tests
Part Six: How to Detect Broken Images On Your Site
Part Seven: Dynamically Creating Test Data with a DataFactory
Part Eight: Visual Testing With Playwright
Part Nine: API Testing, Intercepting Network Requests, and Mocking APIs
Source Code: https://github.com/tjmaher/practicing-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Practicing Playwright: Visual Testing With Playwright

2026-05-21T08:46:54.520-04:00

If you want to do some basic visual checking to see if there has been any deviation from your baseline of images, you can use Playwright. It's built in! Playwright can take a snapshot of a web element, a visible viewport, or a full page, and save it in your Git repository as a baseline, failing the test if the image, page, or viewport does not match up.

Caution: From what I have been reading, this can quickly cause your code repository to balloon in size, since Chrome, Firefox, and WebKit would each store its own golden screenshot in your repo. Also, images on Mac, Windows, and Linux all appear different pixel-by-pixel. If using a CI platform, it might be best to run visual tests only on a standard Playwright Docker image, to generate and compare snapshots. According to TestQuality, "Once a suite passes 50–100 visual tests, teams need a layer that tracks run history, surfaces flaky-test patterns across cycles, and routes confirmed defects into the team's tracker — none of which lives inside the test runner itself"... I wonder if you can store images in an Amazon S3 bucket and hook that up as a virtual drive? ... no matter. That will be a blog post for another time...

Right now, we will be walking through Butch Mayhew's code he wrote for his LinkedIn Learning course, Learning Playwright, found on his companion GitHub site.

While the test is in a certain state you can a screenshot of the page or certain elements of the page and save them as a snapshot. The snapshots an be used as a baseline images to compare your current site against. This baseline can be periodically updated as the site evolves.

How does this happen? With Playwright's .toHaveScreenshot( ) to take a screenshot and the mask method that you want to leave out of the comparisons between the expected and actual screenshots.

toHaveScreenshot(name): Playwright.dev / Page Assertions

Our Test Site: PracticeSoftwareTesting.com

Butch has a series of tests for the test site Practice Software Testing ( practicesoftwaretesting.com ).

If you go to the site, you can see that there is an ever moving logo, "Toolshop Demo" which has the title "Practice Software Testing - Toolshop" . Let's write a test capturing an image of the page, leaving out the moving logo.

Butch also suggested waiting for everything to be fully loaded with "networkidle". But, according to the Playwright.dev waitForLoadState documentation, it says "Don't use this method for testing, rely on web assertions to assess readiness instead". Hrm. Let's go with that he has for now.

Writing the Visual Test

home.spec.ts

  test("visual test", async ({ page }) => {
    await page.waitForLoadState("networkidle");
    await expect(page).toHaveScreenshot("home-page-no-auth.png", {
      mask: [page.getByTitle("Practice Software Testing - Toolshop")],
    });
  });

tests / homepage / homepage.spec.ts

The first time you run this test, it fails, since there are not yet any snapshots saved. There is no golden file. The second time you run it, it will pass. If you wanted to bypass this step, and just update the snapshots for this page, you can run. This regenerates any snapshots for the assertions:

npx playwright test tests/home.spec.ts --update-snapshots

Did a snapshot not match up with the one saved?

You can investigate it in the built-in Playwright reports, showing differences outlined in red, the actual snapshot, the expected snapshots, and see them side-by-side. It also have a feature where you can move slider back and forth horizontally across the image investigating actual and expected snapshots.

Notice that the mask shows that image in an array. Need to mask a bunch of items on the page? You can add them all to the array.

You can also grab screenshots not just of whatever is in the viewport, or the full page beyond the fold, but also compare individual images, or elements such as popup windows.

Visual Comparisons

From Playwright.dev Docs / Test Snapshots: "Playwright Test includes the ability to produce and visually compare screenshots using await expect(page).toHaveScreenshot(). On first execution, Playwright test will generate reference screenshots. Subsequent runs will compare against the reference".

The ToHaveScreenshot Method

This is all part of Playwright.dev's PageAssertions class, where we are expecting the page for the actual viewable screenshot of the app to match the saved expected viewable screenshot. ( See all Options ). Options you can use:

fullPage: You can decide to capture the entire scrollable page, setting it as "true".
mask: You can tell Playwright to ignore certain areas, such as headers or footers if you are just interested in the main area of the site, finding the locators of those areas and mask them. Normally, it will be a pink box in the report to show what is masks, but you can change the maskColor in CSS color format.
threshold: How strict should colors match? Should it be strict (0) or lax (1) or somewhere in between? The default is 0.2.
omitBackground: When you want to hide the default white background and capture it as transparent.
maxDiffPixels & maxDiffPixelRatio: Shows the acceptable amount of pixels that could be different and a ratio of different pixels between 0 and 1. We can configure both using TestConfig.expect. These are turned off by default.

Matching Pixels With Pixelmatch

Playwright uses the pixelmatch JavaScript library ( https://github.com/mapbox/pixelmatch ) which is used to compare actual and expected images.

Configure Visual Checking In Playwright Config

Want to fine tune the above variables? You can update the global configuration in your playwright.config.ts file:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  expect: {
    toHaveScreenshot: {
      // Allow up to 2% of total pixels to change 
      // (handles layout shifts/dynamic content)
      maxDiffPixelRatio: 0.02, 
      
      // Increase pixel color tolerance 
      // (0.2 blends strict pixel match with visual difference)
      threshold: 0.2,          
      
      // Automatically mask dynamic elements like blinking text cursors
      caret: 'hide',           
      
      // Ensure CSS animations are frozen on the first frame
      animations: 'disabled',  
    },
  },
});

How to Compare Product Cards?

Let's say on the PracticeSoftwareTesting.com site, you wanted to create a visual comparison of the first product card. The code could look something like this:



  test('product card matches snapshot', async ({ page }) => {
    await page.goto('https://practicesoftwaretesting.com');

    // Locate a specific element -- here, the first product card
    const productCard = page.locator('[data-test="product"]').first();

    // Wait for the image inside the card to fully load before snapping
    await productCard.locator('img').waitFor({ state: 'visible' });

    // Compare element screenshot against stored baseline
    await expect(productCard).toHaveScreenshot('product-card.png');
  });

});

How To Mask Changing Elements Such As Prices?

Prices for a Shopping Cart will never remain stagnant. They are always changing.

Let's say you just want to do visual checking on the basic layout. Every time the prices change, you don't want the visual test to fail. What you can do is just mask out all the prices, which are with the ids "product-price".

// tests/full-page-screenshot.spec.ts
import { test, expect } from '@playwright/test';

test('homepage layout matches snapshot, prices masked', async ({ page }) => {
  await page.goto('https://practicesoftwaretesting.com');

  await expect(page).toHaveScreenshot('homepage.png', {
    // Mask elements whose content changes between runs
    mask: [
      page.locator('[data-test="product-price"]'),
    ],
    // Pixel-level tolerance -- useful when anti-aliasing differs
    maxDiffPixelRatio: 0.02,
  });
});

... And if you wanted even more fine tuning, you could choose to Integrate Applitools with Playwright.

Running Visual Testing in CI Platforms

Let's say you are running visual testing in a CI environment: Make sure you generate the snapshots against what is in CI, and not your local environment or there will be pixel differences.

If you are using Playwright's pre-built Docker Image (see docs ) make sure to update the images in the container you are using.

According top TestQuality.com: Playwright Visual Regression Testing: A Production Guide to Baselines, Flake, and CI, "Native Playwright visual regression is free to start and expensive to scale. The cost shows up in CI, not on day one".

The problem is that "Windows, macOS, and Linux render fonts, anti-aliasing, and pixel spacing differently. A baseline captured on a developer's Mac will fail when compared against a screenshot taken on the Linux CI runner, even when the application code has not changed".

The solution? "[A] Dockerfile based on the official Playwright image, plus docker-compose mounting screenshot volumes so developers run the same Linux rendering engine locally that the CI runner uses. A single command brings up the same environment everywhere, which eliminates 'works on my machine' visual discrepancies entirely".

How to manage them? "Playwright visual regression CI integration requires treating visual checks as a separate, isolated pipeline stage. Tag visual tests with @visual, exclude them from standard runs using --grep-invert, and execute them in a dedicated Docker step. This keeps functional feedback fast and quarantines visual flake to a controlled environment".

Any other problems? "Storage is the harder problem. Native baselines live in the Git repository, with one image per browser engine — a ten-test suite covering Chromium, Firefox, and WebKit produces thirty committed images. The repo grows fast, but the bigger pain is approval. GitHub, GitLab, and most version control UIs cannot diff images inline. When a baseline updates, the reviewer sees a binary file change with no visible context. Resolving the conflict requires pulling the branch locally, opening the images side by side, copying the new baseline into the feature branch, and pushing manually. House calls this 'GitHub diff blindness,' and it is the single biggest reason teams move off native after a few months".

Practicing Playwright:

Part One: Time to Practice Playwright!
Part Two: New Playwright Course: Playwright Essential Training by Butch Mayhew
Part Three: Hands on Automated Testing with Playwright by Butch Mayhew is the start of a wonderful conversation with the Playwright community!
Part Four: What happens when you pair Playwright with something other than TypeScript?
Part Five: Logging in by Storing and Using an Authentication Cookie in Your Automated Tests
Part Six: How to Detect Broken Images On Your Site
Part Seven: Dynamically Creating Test Data with a DataFactory
Part Eight: Visual Testing With Playwright
Part Nine: API Testing, Intercepting Network Requests, and Mocking APIs
Source Code: https://github.com/tjmaher/practicing-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

SQGNE Lecture is tonight! Building a React Native Mobile Automated Test Framework with T.J. Maher

2026-05-20T10:12:23.230-04:00

Are you in Boston? Come hear me talk about Building a a React Native Mobile Automated Test Framework in tonight's Software Quality Group of New England meeting in Burlington, MA:

Registration Page: https://sqgne.org/n-May-2026.html
Slide Deck: https://tinyurl.com/detox-demo-sqgne

Taken from the Registration page:

"Building a React Native Mobile Automated Test Framework
Thomas F. - T.J. - Maher, Jr.

"Wednesday May 20, 2025 6:30-8:00 PM - in person (free pizza!)
Register Here

"Check in between 6:00 and 6:30 to network

"About the Presentation. . .

"Writing automated tests for a React Native mobile application is notoriously difficult. Mobile components display on the page, but are not fully loaded. Lengthy animations and slow-loading components take a while to finish. Timing issues cause your automated tests to error out giving the appearance of flaky tests.

"Thomas F. - T.J. - Maher, Jr. will be sharing his experience tackling these problems using the open-source mobile testing framework, Wix's Detox, designed specifically for testing React Native applications.

"Setting up a mobile test automation framework.
"Vibe-coding a toy React Mobile Login page app to test against ( tinyurl.com/detox-demo ).
"How to reduce timing issues and flakiness in automated tests.
"Refactoring code into tests, page objects & base pages, separating out credentials and message strings for easier maintainability.
"Setting up automated tests in CI / CD for iOS simulators and Android emulators with GitHub Action Workflows.
"How developers can test their code before they push it into the main branch".

About the Speaker. . .

"A former organizer of Ministry of Testing - Boston, Thomas (T.J.) Maher is a Software Development Engineer in Test (SDET) based in the Boston/South Shore area of Massachusetts, with a decade of experience building web + mobile test automation frameworks. His recent work includes architecting a mobile automation framework from the ground up using Detox and TypeScript for a React Native application at SELF ID, a decentralized identity startup. Prior roles include MassMutual, Verily (Google), Threat Stack, Fitbit, and Intralinks".

See you there!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Practicing Playwright: Dynamically Creating Test Data with a DataFactory

2026-05-14T00:27:50.757-04:00

Continuing walking through Butch Mayhew's LinkedIn Learning course Playwright Essential Training: Abstractions, Fixtures, and Complex Scenarios, we will be examining his code creating a DataFactory that dynamically generates new registered users for our app under test.

The app we will be testing against is PracticeSoftwareTesting.com. examining how the registration call in the API creates new users. We will also be mimicking this call at a programmatic level, to be used in a Playwright automation framework.

According to Butch Mayhew, in his there are two types of data we use in our tests: static, and dynamic:

Static data that should never change. Already exists before the test.

Example: Your go-to test user, or your go to product when testing a shopping cart.

Dynamic Data: Data that is created as part of a test.

Newly registered users. Products created as part of a test.

Between static and dynamic data, Butch believes it should be around 15% / 85% split.

You can dynamically generate data, such as registering new users, by implementing a Datafactory, a helper function that interacts with a system to create this data for you.

Step One: Write A Registration Test

On my local computer, I cloned Butch Mayhew's companion GitHub site for his LinkedIn Learning site, checking out branch 01_05b, the section he was covering in his course.

Microsoft VS Code is the IDE (Integrated Development Environment) we will be using. If you install the Microsoft Playwright VS Code extension, you see on the right hand sidebar for VS Code, an icon that looks like a beaker. Clicking on that, you can see more options such as "Record new", a way to record a new test.

Visual Studio Code: https://code.visualstudio.com/
Visual Studio Marketplace: Playwright Test for VS Code.
Playwright.dev Docs: Getting Started / VS Code

When I clicked on "Record new", following along with the video in the course, it opened up a new test-1.spec.ts file. A new browser window is opened. When I went to https://www.practicesoftwaretesting.com/auth/register and filled out the Registration form, every click, every piece of data filled in was captured in the test.

test('test', async ({ page }) => {
  await page.goto('https://www.practicesoftwaretesting.com/auth/register');
  await page.getByTestId('first-name').fill('Test');
  await page.getByTestId("last-name").fill("Tester");
  await page.getByTestId('country').selectOption('US');
  await page.getByTestId('postal_code').fill('555555');
  await page.getByTestId('street').fill('101 Testing Way');
  await page.getByTestId('city').fill('New York');
  await page.getByTestId('state').fill('New York');
  await page.getByText('First nameLast nameDate of').click();
  await page.getByTestId('email').fill('testlinkedinlearning@test.com');
  await page.getByTestId('password').fill('Pass$w0rd1');
  await page.getByTestId('register-form').getByRole('button').filter({ hasText: /^$/ }).click();
  await page.getByTestId('phone').fill('5555555555');
  await page.getByTestId('dob').fill('2001-01-01');
  await page.getByTestId('house_number').fill('42');
  await page.getByTestId('register-submit').click();
});

Step Two: Open the Playwright UI Mode

Before rerunning this test in UI mode, since a condition for registered users is that all emails should be unique, I made the email address "testlinkedinlearning01@test.com".

Open the Playwright Test UI: npx playwright test --ui

... This command opened up a new Playwright Test browser window showing in the left-hand sidebar all tests in the test suite, including the test-2.spec.ts one that was autogenerated for me.

According to Playwright.dev Docs: Test UI Mode "Once you launch UI Mode you will see a list of all your test files. You can run all your tests by clicking the triangle icon in the sidebar. You can also run a single test file, a block of tests or a single test by hovering over the name and clicking on the triangle next to it".

Step Three: Examine the Register API Call

Want to see what API calls the registration is running?

According to the Playwright Docs / UI Mode / Network, "The Network tab shows you all the network requests that were made during your test. You can sort by different types of requests, status code, method, request, content type, duration and size. Click on a request to see more information about it such as the request headers, response headers, request body and response body".

Select the "Network" tab, just as in Google Chrome Developer tools, and you will see everything, including the POST API call, "register", which posts the data to /users/register endpoint. .

Select the "register" call. If you take a look at the Payload, you can see:

  {
  "first_name": "Test",
  "last_name": "Tester",
  "dob": "2001-01-01",
  "phone": "5555555555",
  "email": "testlinkedinlearning@test.com",
  "password": "Pass$w0rd1",
  "address": {
    "street": "101 Testing Way",
    "city": "New York",
    "state": "New York",
    "country": "US",
    "postal_code": "555555"
  }
}

Step Four: Copy Request As Playwright

If you scroll to the right, you will see the option "Copy request". Choose the dropdown option, "Copy as Playwright", and you get the following code:


await page.request.post('https://api.practicesoftwaretesting.com/users/register', {
  data: `{
  "first_name": "Test",
  "last_name": "Tester",
  "dob": "2001-01-01",
  "phone": "5555555555",
  "email": "testlinkedinlearning01@test.com",
  "password": "Pass$w0rd1",
  "address": {
    "street": "101 Testing Way",
    "city": "New York",
    "state": "New York",
    "country": "US",
    "postal_code": "555555"
  }
}`,
  headers: {
    'sec-ch-ua-platform': '"Windows"',
    Referer: 'https://www.practicesoftwaretesting.com/',
    'Accept-Language': 'en-US',
    'sec-ch-ua': '"HeadlessChrome";v="147", "Not.A/Brand";v="8", "Chromium";v="147"',
    'sec-ch-ua-mobile': '?0',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.7727.15 Safari/537.36',
    Accept: 'application/json, text/plain, */*',
    'Content-Type': 'application/json'
  }
});

Read More at Butch's website, Playwright Solutions: TIP: Playwright Trace Viewer - Copy as Playwright API Request

Step Five: Start Reformatting the DataFactory Code

Butch mentioned that we can take this information to create our Datafactory. Butch created a new file, lib/datafactory/register.ts, pasting the contents there, to start it off.

Since Butch was extracting the data in its own file, he had to do some refactoring:

Deleted the headers. We didn't need that information for this example.
Imported the "request" fixture from @playwright/test.
Initialized the request, with a "const createRequestContext = await request.newContext()
Switched the "page.request.post" to "createRequestContext.post"
Removed the string off of the object, removing the quotation marks from each variable to make it less JSON and more JavaScript.

Step Six: Add API_URL to Environment File

Butch then took the API, https://api.practicesoftwaretesting.com/users/register, to "/users/register" and created a new variable, const apiUrl = process.env.API_URL.

Read More on Playwright.dev: Passing Environment Variables

Butch added the new variable to the .env file in the root directory:

API_URL=https://api.practicesoftwaretesting.com

Step Seven: Parameterize the Email and Password

Right now, it is still static. In order to make it dynamic, Butch set email and password to a variable, then wrapped it in a function, with:

export async function registerUser(email: string, password: string)

Lastly, Butch added code to verify that the user was created:

Import expect from @playwright/test.
expect(response.status()).toBe(201)
Made the entire await block a "const response".
Returned the response so you know what it is in the test: return response.status().

register.ts


import { request, expect } from "@playwright/test";

export async function registerUser(email: string, password: string) {
  const apiUrl = process.env.API_URL;
  const createRequestContext = await request.newContext();
  const response = await createRequestContext.post(apiUrl + "/users/register", {
    data: {
      first_name: "Test",
      last_name: "User",
      dob: "2001-01-01",
      phone: "5555555555",
      email: email,
      password: password,
      address: {
        street: "101 Testing Way",
        city: "New York",
        state: "New York",
        country: "US",
        postal_code: "55555",
      },
    },
  });

  expect(response.status()).toBe(201);
  return response.status();
}

lib/datafactory/register.ts

Now, every time you need to create a new user, you can import the function, registerUser, passing along an email and a password, and new test user will be generated.

Need to make something dynamic? Butch appended to the email address, Date.now, so it would be unique as long as the test was not run in the same second:

const email = `test${Date.now()}@test.com

login.spec.ts


import { registerUser } from "@datafactory/register";

test("login with newly registered user", async ({ page }) => {
  const email = `test${Date.now()}@test.com`;
  const password = "fjdWEdfs82@";

  await registerUser(email, password);
  const loginPage = new LoginPage(page);
  await loginPage.goto();
  await loginPage.login(email, password);

  await expect(page.getByTestId("nav-menu")).toContainText("Test User");
  await expect(page.getByTestId("page-title")).toContainText("My account");
});

/tests/login/login.spec.ts

Now, you have your very own DataFactory!

Practicing Playwright:

Part One: Time to Practice Playwright!
Part Two: New Playwright Course: Playwright Essential Training by Butch Mayhew
Part Three: Hands on Automated Testing with Playwright by Butch Mayhew is the start of a wonderful conversation with the Playwright community!
Part Four: What happens when you pair Playwright with something other than TypeScript?
Part Five: Logging in by Storing and Using an Authentication Cookie in Your Automated Tests
Part Six: How to Detect Broken Images On Your Site
Part Seven: Dynamically Creating Test Data with a DataFactory
Part Eight: Visual Testing With Playwright
Part Nine: API Testing, Intercepting Network Requests, and Mocking APIs
Source Code: https://github.com/tjmaher/practicing-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Practicing Playwright: How to Detect Broken Images On Your Site

2026-05-13T16:36:36.044-04:00

Butch Mayhew's LinkedIn Learning Course Playwright Essential Training: Abstractions, Fixtures, and Complex Scenarios has been a wonderful resource learning more about Playwright.

With this blog post, I will be walking through how Butch injects JavaScript into a test, in order to check if any elements in a shopping cart have any broken images.

Playwright / Evaluating JavaScript: https://playwright.dev/docs/evaluating
Butch Mayhew's companion GitHub site on LinkedIn Learning's GitHub page.
Butch's Resources File: RESOURCES.MD
Site under test: Practice Software Testing: https://practicesoftwaretesting.com
Practice Software Testing Website, With Bugs: https://with-bugs.practicesoftwaretesting.com

Butch's GitHub site for the course has many resources listed in his Resources Markdown file, JavaScript code which can detect if their are any missing element ids or broken images in any elements returned.

Find Broken Images with JavaScript

RESOURCES.md / Injecting JavaScript in Playwright

Array.from(document.querySelectorAll("img"))
  .filter((img) => img.naturalWidth === 0 || img.naturalHeight === 0)
  .map((img) => img.src);

Let's examine this code found on Butch Mayhew's RESOURCES.MD file:

document.querySelectorAll("img"): The JavaScript Document interface, "represents any web page loaded in the browser, and serves as an entry point into the web page's content" according to the Mozilla Developer Network.

The Document interface also includes a method called .querySelectorAll() [ See Mozilla API ] which returns a list of the HTML document's elements that match the designated selector which, in this case is an "img" tag. This returns a static NodeList, a snapshot collection of nodes it sees at that time.

Array.from will convert the NodeList into a JavaScript array, so we can then use array methods such as filter, which creates a new array from the results without changing the original array.

We are filtering to see if there are images which do not have a width or ( || ) height, meaning they are broken images. img.naturalWidth is a read-only HTMLImageElement that returns the original width of an image in pixels. [ See MDN entry ]

The code will loop through the filtered array, find the list of broken images, then extract a list of URLs from the source code, pointing to the broken images, if any.

Playwright Evaluating JavaScript

"Playwright scripts run in your Playwright environment. Your page scripts run in the browser page environment. Those environments don't intersect, they are running in different virtual machines in different processes and even potentially on different computers.

"The page.evaluate() API can run a JavaScript function in the context of the web page and bring results back to the Playwright environment". - Playwright Docs / Evaluating JavaScript

This JavaScript code Butch came up with to detect broken images can be injected into his test with JavaScript evaluations.


test("check for broken images", async ({ page }) => {
    // await page.goto("https://with-bugs.practicesoftwaretesting.com/");

    const brokenImages = await page.evaluate(() => {
      return Array.from(document.querySelectorAll("img"))
        .filter((img) => img.naturalWidth === 0 || img.naturalHeight === 0)
        .map((img) => img.src);
    });
    expect(
      brokenImages.length,
      `Broken Images: ${brokenImages.toString()}`
    ).toBe(0);
  });

Although practicesoftwaretesting.com does not have broken images, this test site does offer a version which does: https://with-bugs.practicesoftwaretesting.com/.

Ideally, we want the number of broken images to be zero. If we wanted to catch the broken images on the purposely broken test site, following Butch's code:

Uncomment out the With Bugs version of the site, so we can go to that page for this test.
Declare a new constant, brokenImages, which will list the broken images it detects.
The JavaScript injection to actually find the broken images will be evaluated with a result returned.
We will expect the number of brokenImages to be zero. If not, we are going to return the web addresses of broken images that it found.

So, that's how you create a Playwright test that detects broken images!

Happy Testing!

Practicing Playwright:

Part One: Time to Practice Playwright!
Part Two: New Playwright Course: Playwright Essential Training by Butch Mayhew
Part Three: Hands on Automated Testing with Playwright by Butch Mayhew is the start of a wonderful conversation with the Playwright community!
Part Four: What happens when you pair Playwright with something other than TypeScript?
Part Five: Logging in by Storing and Using an Authentication Cookie in Your Automated Tests
Part Six: How to Detect Broken Images On Your Site
Part Seven: Dynamically Creating Test Data with a DataFactory
Part Eight: Visual Testing With Playwright
Part Nine: API Testing, Intercepting Network Requests, and Mocking APIs
Source Code: https://github.com/tjmaher/practicing-playwright

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

Pramod Dutta discusses 7 Playwright features senior SDETs use daily

2026-05-11T16:52:22.666-04:00

Pramod Dutta, an SDET from Tekion, posted on LinkedIn "7 Playwright features senior SDETs use daily". The Playwright features, according to Pramod:

"→ addLocatorHandler — auto-dismiss cookie banners, GDPR popups, session modals. One handler. Whole suite cleaner. Stop writing wrapper functions.
"→ browser.bind() — launch one browser. Let your test, Claude Code, and debugger all attach to it simultaneously. The debugging workflow that fixes "works on my machine" forever.
"→ URLSearchParams in request.get() — stop building query strings manually. Eliminates an entire bug class around URL encoding in API tests.
"→ expect.toPass() — polling-based assertions for state that converges over time. The right tool for 'this should eventually be true.' Not the same as expect(locator).toBeVisible(). Different problem, different solution.
"→ page.requestGC() — manually trigger garbage collection in the browser. The only Playwright tool that catches memory leaks before production. Used with WeakRef in page.evaluate().
"→ --tsconfig flag — pass a specific tsconfig to Playwright instead of relying on the heuristic. Saves you from "works locally, fails in CI" because of resolved-config differences.
"→ webServer.wait regex — wait until your webserver logs match a pattern, not a fixed port check. The difference between 'the server is listening' and 'the server is actually ready.'"

I scribbled these features down, then turned them over to Claude.ai, my research assistant, to see if he could explain these methods, come up with sample code using them, then add bullet points to the official documentation, source code, and the release notes.

... Let's see how well Claude did explaining them ...

`page.addLocatorHandler()` v1.42 — March 2024

Registers a callback that fires whenever a specified locator becomes visible and risks blocking a Playwright action. Playwright checks for the locator before every actionability check or auto-waiting assertion. When the locator is visible, the handler runs first, then the test continues. The handler only fires during actions; appearing passively between actions does not trigger it. After the handler runs, Playwright waits for the overlay to become hidden before proceeding (opt out with noWaitAfter: true). Use the times option to cap the number of invocations. The handler receives the triggering locator as its argument.

Cookie banner (fire-and-forget)

// Register once, before navigating.
await page.addLocatorHandler(
  page.getByRole('heading', { name: 'You are in control of your cookies.' }),
  async () => {
    await page.getByRole('button', { name: 'Accept all' }).click();
  }
);

// Write the rest of the test normally.
await page.goto('https://www.example.com/');
await page.getByRole('link', { name: 'Shop now' }).click();

Self-removing handler with access to the triggering locator

// The handler receives the matched locator; times: 1 auto-removes it.
const closeBtn = page.getByLabel('Close');
await page.addLocatorHandler(closeBtn, async (locator) => {
  await locator.click();
}, { times: 1 });

// Remove manually when no longer needed.
await page.removeLocatorHandler(closeBtn);

Fail fast on an application error overlay

// Instead of dismissing, throw immediately so the test does not hang.
await page.addLocatorHandler(
  page.getByRole('heading', { name: "Sorry, there's been a problem" }),
  async () => {
    throw new Error(`Error overlay appeared on ${await page.url()}`);
  }
);

Resource	Link
Official docs	playwright.dev — page.addLocatorHandler()
Release notes (v1.42)	playwright.dev/docs/release-notes#version-142
GitHub release (v1.42.0)	github.com — Release v1.42.0
Source code	packages/playwright-core/src/client/page.ts

`browser.bind()` v1.59 — April 2026

Binds a launched browser to a named pipe or WebSocket endpoint, making it available to playwright-cli, @playwright/mcp, and any other Playwright client simultaneously. Before v1.59, sharing a browser between an MCP agent and a test suite meant two separate processes. Now a single browser.bind() call exposes one browser session to all of them. Call browser.unbind() to stop accepting new connections; existing connections keep working. Run playwright-cli show to open the live dashboard listing all bound browsers.

Bind via named pipe (local only)

import { chromium } from '@playwright/test';

const browser = await chromium.launch();
const { endpoint } = await browser.bind('my-session', {
  workspaceDir: '/my/project',
});
console.log('Session bound:', endpoint);

Bind via WebSocket (shareable over a network)

const { endpoint: wsEndpoint } = await browser.bind('my-session', {
  host: 'localhost',
  port: 0, // let the OS pick an available port
});
console.log('WebSocket endpoint:', wsEndpoint); // ws://localhost:PORT/...

// A second Playwright client connects to the same browser.
const browser2 = await chromium.connect(wsEndpoint);
const page = await browser2.newPage();

// Stop accepting new connections when done.
await browser.unbind('my-session');

Resource	Link
Official docs	playwright.dev — browser.bind()
Release notes (v1.59)	playwright.dev/docs/release-notes#version-159
GitHub release (v1.59.0)	github.com — Release v1.59.0
Source code	packages/playwright-core/src/client/browser.ts

`URLSearchParams` in `request.get()` v1.46 — August 2024

The params option on APIRequestContext methods (get, post, put, patch, delete) now accepts a native URLSearchParams object or a plain query string in addition to the existing plain-object shorthand. This matters when you need to send the same key more than once, which a plain object cannot express.

test('query params via URLSearchParams', async ({ request }) => {

  // URLSearchParams lets you append duplicate keys.
  const params = new URLSearchParams();
  params.set('userId', '1');
  params.append('tag', 'automation');
  params.append('tag', 'playwright'); // duplicate key — plain object cannot do this

  const response = await request.get(
    'https://jsonplaceholder.typicode.com/posts',
    { params }
  );
  expect(response.status()).toBe(200);

  // Or just pass a pre-built query string.
  const response2 = await request.get(
    'https://jsonplaceholder.typicode.com/posts',
    { params: 'userId=1&tag=automation' }
  );
  expect(response2.ok()).toBeTruthy();
});

Resource	Link
Official docs	playwright.dev — APIRequestContext.get()
Release notes (v1.46)	playwright.dev/docs/release-notes#version-146
GitHub release (v1.46.0)	github.com — Release v1.46.0
Source code	packages/playwright-core/src/client/fetch.ts

`expect.toPass()` intervals config — v1.44 May 2024

Wraps an async callback and retries it until every assertion inside passes or a timeout is reached. Unlike Playwright's built-in auto-retrying locator assertions, toPass can contain arbitrary logic including API requests, multi-step flows, and non-locator checks. By default the timeout is 0 (no timeout) and probe intervals default to [100, 250, 500, 1000] ms. Both are configurable globally in testConfig.expect.toPass or per-call.

Polling an endpoint until ready

test('waits for background job', async ({ request }) => {
  await expect(async () => {
    const res = await request.get('https://api.example.com/job/status');
    expect(res.status()).toBe(200);
    const body = await res.json();
    expect(body.status).toBe('complete');
  }).toPass({
    intervals: [1_000, 2_000, 10_000], // probe at 1 s, 2 s, then every 10 s
    timeout:   60_000,
  });
});

Global configuration in `playwright.config.ts`

import { defineConfig } from '@playwright/test';
export default defineConfig({
  expect: {
    toPass: {
      timeout:   30_000,
      intervals: [500, 1_000, 5_000],
    },
  },
});

Resource	Link
Official docs	playwright.dev — Assertions: expect.toPass()
Release notes (v1.44)	playwright.dev/docs/release-notes#version-144
GitHub release (v1.44.0)	github.com — Release v1.44.0
Source code	packages/playwright/src/matchers/toPass.ts

`page.requestGC()` v1.46 — August 2024

Asks the browser to run garbage collection on the page. There is no guarantee that every unreachable object is collected, but in practice it is reliable enough to detect leaks in combination with JavaScript WeakRef. The pattern: store a WeakRef to the object under suspicion, call requestGC(), then assert that deref() returns undefined.

test('no memory leak on the large object', async ({ page }) => {
  await page.goto('https://your-app.com');

  // 1. Pin a WeakRef to the object under suspicion.
  await page.evaluate(() => {
    (globalThis as any).suspectWeakRef = new WeakRef((window as any).largeObject);
  });

  // 2. Request GC.
  await page.requestGC();

  // 3. A living WeakRef means a leak; deref() should now be undefined.
  const leaked = await page.evaluate(
    () => !!(globalThis as any).suspectWeakRef.deref()
  );
  expect(leaked).toBe(false);
});

Resource	Link
Official docs	playwright.dev — page.requestGC()
Release notes (v1.46)	playwright.dev/docs/release-notes#version-146
GitHub release (v1.46.0)	github.com — Release v1.46.0
Source code	packages/playwright-core/src/client/page.ts

TypeScript Config: `--tsconfig` flag & `testConfig.tsconfig`

Playwright looks up the nearest tsconfig.json for each imported file by default. Two complementary options override this. The --tsconfig CLI flag (v1.46, August 2024) applies one tsconfig to all imported files from the command line. The testConfig.tsconfig property (v1.49, October 2024) does the same from the config file. Note that Playwright only honours four tsconfig options: allowJs, baseUrl, paths, and references.

CLI flag

# Apply one tsconfig to everything Playwright loads.
npx playwright test --tsconfig tsconfig.test.json

Config file property

// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
  tsconfig: './tsconfig.test.json',
});

Example `tests/tsconfig.json`

{
  "compilerOptions": {
    "baseUrl": ".",
    "allowJs": true,
    "paths": {
      "@helpers/*": ["../src/helpers/*"]
    }
  }
}

Resource	Link
TypeScript guide	playwright.dev — TypeScript
`testConfig.tsconfig` docs	playwright.dev — TestConfig.tsconfig
Release notes (`--tsconfig`, v1.46)	playwright.dev/docs/release-notes#version-146
Release notes (`testConfig.tsconfig`, v1.49)	playwright.dev/docs/release-notes#version-149
GitHub release (v1.46.0)	github.com — Release v1.46.0
GitHub release (v1.49.0)	github.com — Release v1.49.0
Source code	packages/playwright/src/transform/transform.ts

`webServer.wait` Regex v1.59 — April 2026

The webServer block in playwright.config.ts previously waited for a URL to respond with a 2xx–4xx status. The new wait.stdout and wait.stderr sub-options accept regular expressions and watch the server process output instead. This is useful for dev servers that print a readiness message rather than exposing an HTTP health endpoint. Named capture groups in the regex are automatically exported as environment variables, so a varying port can be passed directly to test.use({ baseURL }).

// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
  webServer: {
    command: 'npm run start',
    wait: {
      // Named group "my_server_port" is stored in process.env.MY_SERVER_PORT
      stdout: /Listening on port (?<my_server_port>\d+)/,
    },
  },
});

// In a test file — consume the captured port.
import { test } from '@playwright/test';

test.use({
  baseURL: `http://localhost:${process.env.MY_SERVER_PORT ?? 3000}`,
});

test('homepage loads', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveTitle(/My App/);
});

Resource	Link
Web server guide	playwright.dev — Web Server
`testConfig.webServer` docs	playwright.dev — TestConfig.webServer
Release notes (v1.59)	playwright.dev/docs/release-notes#version-159
GitHub release (v1.59.0)	github.com — Release v1.59.0
Source code	packages/playwright/src/plugins/webServerPlugin.ts

Practicing Playwright: Logging in by Storing and Using an Authentication Cookie in Your Automated Tests

2026-05-10T21:31:28.196-04:00

I absolutely love that LinkedIn offers a free month-long trial period of LinkedIn Learning. Butch Mayhew's Playwright Essential Training: Abstractions, Fixtures, and Complex Scenarios course has been a wonderful resource learning more about Playwright.

With this blog post, I will be walking through Butch's code on how to set up an automated test to log into an app without going through the user interface. All it needs is the login cookie.

Butch Mayhew's companion GitHub site on LinkedIn Learning's GitHub page.
Site under test: Practice Software Testing: https://practicesoftwaretesting.com
Official Playwright Documentation: https://playwright.dev/docs
Official Playwright GitHub: https://github.com/microsoft/playwright

When testing a shopping cart app such as the Practice Software Testing website, it can get tedious. You need to open a browser, go to the login page, enter a username, enter a password, hit the login button, and verify you have logged in correctly every time you want to test something in the shopping cart.

If you are testing something unrelated to logging in, why have your tests go through the UI to authenticate? Why not have your automated test run that login test once, temporarily save the login cookie once it is produced, then reuse the login cookie, importing it into other tests?

What is Playwright?

According to Microsoft Playwright's GitHub site, Playwright "is a framework for web automation and testing. It drives Chromium, Firefox, and WebKit with a single API — in your tests, in your scripts, and as a tool for AI agents".

Playwright comes in many features. From https://github.com/microsoft/playwright

	Best for	Install
Playwright Test	End-to-end testing	`npm init playwright@latest`
Playwright CLI	Coding agents (Claude Code, Copilot)	`npm i -g @playwright/cli@latest`
Playwright MCP	AI agents and LLM-driven automation	`npx @playwright/mcp@latest`
Playwright Library	Browser automation scripts	`npm i playwright`
VS Code Extension	Test authoring and debugging in VS Code	Install from Marketplace

... Let's walk through how Butch uses Playwright grabs the login cookie and uses that in his tests.

Getting an Authorization Cookie Through the Website

Before we do anything with our test automation, let's examine an authorization cookie.

Open up Chrome and go to https://practicesoftwaretesting.com/auth/login
Right click on the page and select "Inspect" to open Chrome Developer Tools.
Select the "Network" tab.
Enter a username and password of customer@practicesoftwaretesting.com / welcome01
Select the "Login" button, then view the Network tab.
On the network tab, click on the "account" entry.
Select the "Cookies" tab in Chrome Developer Tools.

Under Request Cookies, you will see something like this:

"cookies": [
    {
      "name": "cf_clearance",
      "value": "2Rv345ZtjSaGUFRwq6j2QO4Qp3EBG0T.5URv.oenfjg-1778187254-1.2.1.1-90Z63mBartyo.dUm4bwavAggVG9pgrzcYOO0zPS18qBEujmzIq3tnWwjx2b5jQypbZywJMVEIncDFSwJBaAD0ephO3cctJrLkS449MVHNO7ZSuqlAPhzVYGw8vVH4doqYc2R_5WXTY47OHVLRAgnblq7U9glSiaJUvoOKmc6I.Mo6FPl3MrhGmfMxSTVyTkcCBJn8t_C9b1margtoukM7yMuQdNPpkNItvrrZdIQT.TpGQx2sNX7cS_KS8TY2Ghyc0_QLKFZFqC_5ZgZHMad5xgzgvJEm2NZmO3RBLSPKH5bpDFbuynt7g4oIwYfIvWd4pTpTB447hpP5qMPnLOp.Q",
      "domain": ".practicesoftwaretesting.com",
      "path": "/",
      "expires": 1809723253.921407,
      "httpOnly": true,
      "secure": true,
      "sameSite": "None",
      "partitionKey": "https://practicesoftwaretesting.com",

Cross-Indexing this with the Chrome DevTools Docs on Cookies, this means:

Name: "The cookie's name", in this case "cf_clearance", is a security token issued by Cloudflare after passing a challenge. [ See Cloudflare docs ]

Value. Claude AI guessed that the Value of this cookie was this... but was not sure since this value is proprietary.

Random token: 2Rv345ZtjSaGUFRwq6j2QO4Qp3EBG0T.5URv.oenfjg
Unix timestamp: 1778187254 (Around May 4, 2026)
Version Level: 1.2.1.1, the Clearance level encoding
Signature payload: 90Z63mBartyo... HMAC-binding (Hash-Based Message Authentication Codes) to IP, user agent, and session.

Domain: "The hosts that are allowed to receive the cookie", in this case, practicesoftwaretesting.com.

Path: "The URL that must exist in the requested URL in order to send the Cookie header", in this case the root folder.

Expires / Max-Age. "The cookie's expiration date or maximum age. For session cookies this value is always Session". According to UnixTimeStamp.com "1809723253.921407" would expire on May 10, 2026 on 3:36 am UTC and 49 seconds.

What is a Unix Time Stamp? "The unix time stamp is a way to track time as a running total of seconds. This count starts at the Unix Epoch on January 1st, 1970 at UTC. Therefore, the unix time stamp is merely the number of seconds between a particular date and the Unix Epoch. It should also be pointed out (thanks to the comments from visitors to this site) that this point in time technically does not change no matter where you are located on the globe. This is very useful to computer systems for tracking and sorting dated information in dynamic and distributed applications both online and client side" - UnixTimeStamp.com

HttpOnly: "If true, this field indicates that the cookie should only be used over HTTP, and JavaScript modification is not allowed"... and it is set to true.

Secure: "If true, this field indicates that the cookie can only be sent to the server over a secure, HTTPS connection"... and it is set to true.

SameSite: "Contains Strict or Lax if the cookie is using the experimental SameSite attribute".

Partition Key: "For cookies with independent partition state, the partition key is the site of the top-level URL the browser was visiting at the start of the request to the endpoint that set the cookie.

Priority. Contains Low, Medium (default), or High if using deprecated cookie Priority attribute"... and ours is "https://practicesoftwaretesting.com"

By logging into the site, it generates this cookie on the back end, allowing you to keep accessing the site.

If you generate this cooking programmatically before running the tests, and temporally save it, you can then use it in all preceding tests in that test run.

Step One: Setup the Setup

The Playwright Configuration file has in its projects array two projects: One to set up the, er, setup, and one to set up the Chromium run.

First, Butch set up in his Playwright Configuration file, a project dependency, in a project that he called "setup".

playwright.config.ts:

projects: [
    {
      name: "setup",
      testMatch: /.*\.setup\.ts/,
    },
    {
      name: "chromium",
      dependencies: ["setup"],
      use: {
        ...devices["Desktop Chrome"],
        permissions: ["clipboard-read", "geolocation"],
      },
    },

According to Playwright.Dev, "Dependencies are a list of projects that need to run before the tests in another project run. They can be useful for configuring the global setup actions so that one project depends on this running first.

"When working with tests that have a dependency, the dependency will always run first and once all tests from this project have passed, then the other projects will run in parallel.

"Running order: Tests in the 'setup' project run. Once all tests from this project have passed, then the tests from the dependent projects will start running.

"Tests in the 'chromium' [...] projects run together. By default, these projects will run in parallel, subject to the maximum workers limit".

name: "setup": A project with the name of "setup".
testMatch: /.*setup\.ts,: This pattern matcher searches on level up for anything with "setup" in the middle of the filename, such as "auth.setup.ts" in the "tests/" directory.
name: "chromium": A project that directs the test automation when it runs to use the Desktop Chrome browser devices.
dependencies: ["setup"]: Before the Chrome browser has kicked off, we tell it we need to run the project called "setup" first.
use: ...devices["Desktop Chrome"]: Playwright has an enormous list of devices in its device registry, such as various models of the Blackberry, Galaxy, iPad, iPhone, Microsoft Lumia, Nexus, Nokia, Pixel, Moto, and Desktop Devices (Chrome, Edge, Firefox, Safari) with many different user agents, viewports, and screen width and heights.
use: permissions: Playwright's browser context, according to the official Playwright Docs, has differing sets of permissions in the permission array for each browser emulation, such as emulating the accelerometer, ambient light sensor, background sync, camera, clipboard read and write, geolocation, gyroscope, local fonts, local network access, magnetometer, microphone, midi, notifications, payment-handler, storage-access, and screen-wake-lock.
[clipboard-read]: We need to give the Chromium browser Playwright spins up permission to read the authentication cookie that is generated the first time we log in. Note, according to notes in Playwright's source code in the permissions library, there is no "clipboard-read" in WebKit's API.

Step Two: Get and Save the Authorization Cookie

Next, in auth.setup.ts, Butch has the automated test go to the site, log in, and save the cookie to a file.

tests/ auth.setup.ts:


import { test as setup, expect } from "@playwright/test";
import { LoginPage } from "../lib/pages/login/login.page";

setup("Create customer 01 auth", async ({ page, context }) => {
  const email = "customer@practicesoftwaretesting.com";
  const password = "welcome01";
  const customer01AuthFile = ".auth/customer01.json";

  const loginPage = new LoginPage(page);

  await loginPage.goto();

  await loginPage.login(email, password);

  await expect(page.getByTestId("nav-menu")).toContainText("Jane Doe");
  await context.storageState({ path: customer01AuthFile });
});

https://github.com/LinkedInLearning/playwright-essential-training-abstractions-fixtures-and-complex-scenarios-4278224/blob/98d2080e018acb194938157cb7c2fc8993ca21a5/tests/auth.setup.ts

What is this code doing?

Import the test runner playwright/test into our TypeScript file, setting up an alias for the test runner, calling it "setup" to make the test more readable, and to mark this as performing initialization tasks. To learn more about how aliases work in TypeScript see FreeCodeCamp's explanation, and read about the test method in the Playwright.dev docs.
Import the expect assertion library into the TypeScript file, so we then can use the expect method, detecting if an element is attached, checkbox is checked, element is disabled, editable, enabled, focused, not visible, visible, contains text, has certain CSS classes, DOM attributes, ids, CSS or JavaScript properties, ARIA roles, screenshots, matches text, or if pages have screenshots, titles, or URLs, failing near immediately if we don't see that condition. [ See Playwright.Dev docs about test assertions ]. We can also match if it meets any instance of a class, matches anything, of an array contains certain elements, if a number is close to equal, a string contains a substring, a string matches a certain substring. All of these expects can be negated chaining a not to it.
setup("Create customer 01 auth" is being used by Butch Mayhew instead of saying "test("Create customer 01 auth") to make it more readable.
async uses the JavaScript keyword async. FreeCodeCamp says that, "Async programming is a programming paradigm that allows you to write code that runs asynchronously. In contrast to synchronous programming, which executes code sequentially, async programming allows code to run in the background while the rest of the program continues to execute. This is particularly useful for tasks that may take a long time to complete, such as fetching data from a remote API".
page, context are passing these built-in fixtures into the test. The Page instance is created for each test, used in isolation from one another. [ See Playwright.dev docs on test fixtures. ] "Test fixtures are used to establish the environment for each test, giving the test everything it needs and nothing else. Test fixtures are isolated between tests. With fixtures, you can group tests based on their meaning, instead of their common setup". The "page" fixtures uses type Page, and "context" fixtures uses type BrowserContext in order to configure context. Fixtures can encapsulate setup and teardown, are reusable between test files.
const email, password, customer01AuthFile is declaring constant variables to store what the username, password, and where the auth cookie should be stored, to make the code more readable.
const loginPage is instantiating a new login page using the login page object Butch has defined, not covered in this blog entry. The login page Butch set up contains methods where you can goto https://practicesoftwaretesting.com/auth/login, enter the username and password in the login method using the constants we defined, expecting that we see a greeting to the user.

The real magic is the last bit of code: context.storageState({ path: customer01AuthFile });

Playwright has storage and authentication methods where you can manage cookies, localStorage and sessionStorage [ See official docs ]. Session Storage is session scoped, where data is cleared out when the browser tab eventually closes.
According to the Playwright Docs / Authentication, when you "[c]reate a new setup project in the config and declare it as a dependency for all your testing project [...]. This project will always run and authenticate before all the tests. All testing projects should use the authenticated state as storageState".

The Cookie is Temporarily Saved!

A new folder and file is created in the root folder when this command is run: .auth/customer01.json:


{
  "cookies": [
    {
      "name": "cf_clearance",
      "value": "2Rv345ZtjSaGUFRwq6j2QO4Qp3EBG0T.5URv.oenfjg-1778187254-1.2.1.1-90Z63mBartyo.dUm4bwavAggVG9pgrzcYOO0zPS18qBEujmzIq3tnWwjx2b5jQypbZywJMVEIncDFSwJBaAD0ephO3cctJrLkS449MVHNO7ZSuqlAPhzVYGw8vVH4doqYc2R_5WXTY47OHVLRAgnblq7U9glSiaJUvoOKmc6I.Mo6FPl3MrhGmfMxSTVyTkcCBJn8t_C9b1margtoukM7yMuQdNPpkNItvrrZdIQT.TpGQx2sNX7cS_KS8TY2Ghyc0_QLKFZFqC_5ZgZHMad5xgzgvJEm2NZmO3RBLSPKH5bpDFbuynt7g4oIwYfIvWd4pTpTB447hpP5qMPnLOp.Q",
      "domain": ".practicesoftwaretesting.com",
      "path": "/",
      "expires": 1809723253.921407,
      "httpOnly": true,
      "secure": true,
      "sameSite": "None",
      "partitionKey": "https://practicesoftwaretesting.com",
      "_crHasCrossSiteAncestor": false
    }
  ],
  "origins": [
    {
      "origin": "https://practicesoftwaretesting.com",
      "localStorage": [
        {
          "name": "auth-token",
          "value": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJodHRwczovL2FwaS5wcmFjdGljZXNvZnR3YXJldGVzdGluZy5jb20vdXNlcnMvbG9naW4iLCJpYXQiOjE3NzgxODcyNTUsImV4cCI6MTc3ODE4NzU1NSwibmJmIjoxNzc4MTg3MjU1LCJqdGkiOiJoeFJ1MmkwNWpDamVrNUdXIiwic3ViIjoiMDFLUjIwRFFYQUpTVEJEWkg2S0Q1R1QwRVQiLCJwcnYiOiIyM2JkNWM4OTQ5ZjYwMGFkYjM5ZTcwMWM0MDA4NzJkYjdhNTk3NmY3Iiwicm9sZSI6InVzZXIifQ.Iqv8TGSweXLUfk7UExbIem_kOoIde9apIrzFZ9cCVpo"
        }
      ]
    }
  ]
}

Step Three: Make Sure The Cookie Is Only Temporarily Saved. Do Not Upload the Cookie To GitHub!!!

It is very important that you do NOT save the .auth cookie, or any other credentials, in GitHub.

If you examine the .gitignore file Butch Mayhew has set up, you can see that along with test_results, node_modules, .tmp files, you will not be uploading anything in the .auth/ folder.

Read more about .gitignore on the Git-SCM/docs/gitignore page.

Step Four: Implement the Auth Cookie for Login

Now that the Setup function generated a new cookie, we then can then use this credential setting up the test to use storageState.

tests / homepage / home.spec.ts:

  
test.describe("Home page customer 01 auth", () => {
  test.use({ storageState: ".auth/customer01.json" });
  test.beforeEach(async ({ page }) => {
    await page.goto("/");
  });

How do we know this is successful? We verify that the sign in navigation is not visible, and instead we are greeted by the username, in this case, "Jane Doe".

tests / homepage / home.spec.ts:

  
  test("check customer 01 is signed in", async ({ page }) => {
    await expect(page.getByTestId("nav-sign-in")).not.toBeVisible();
    await expect(page.getByTestId("nav-menu")).toContainText("Jane Doe");
  });

If we don't see the sign in page, and instead we see the nav menu for "Jane Doe", then everything has passed.

So, that's how we use authentication cookies in tests!

Practicing Playwright:

Part One: Time to Practice Playwright!
Part Two: New Playwright Course: Playwright Essential Training by Butch Mayhew
Part Three: Hands on Automated Testing with Playwright by Butch Mayhew is the start of a wonderful conversation with the Playwright community!
Part Four: What happens when you pair Playwright with something other than TypeScript?
Part Five: Logging in by Storing and Using an Authentication Cookie in Your Automated Tests
Part Six: How to Detect Broken Images On Your Site
Part Seven: Dynamically Creating Test Data with a DataFactory
Part Eight: Visual Testing With Playwright
Part Nine: API Testing, Intercepting Network Requests, and Mocking APIs
Source Code: https://github.com/tjmaher/practicing-playwright

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

So much job interview prep! Playwright + TypeScript + GitLab

2026-05-07T22:59:24.351-04:00

Even though I've been job searching for four months now, I am busier than I have ever been. You know what I mean. You've seen this blog! One week I am experimenting with AI, the next I am pairing Playwright with C# or Java, the next I am skimming docs about Contract Testing using Pact.

Each week I get more leads generated from my many LinkedIn posts. Each week is yet another rabbit to chase. Finally, this week, I get to refresh my Playwright + TypeScript skills.

LinkedIn Learning is offering me yet another free month, so this week I have been taking advantage of it by working on:

Playwright courses: Master Test Automation with Playwright Certification: Consists of Butch Mayhew's Learning Playwright, and Playwright Essential Training, paired with Qambar Raza's Playwright Design Patterns and Advanced Playwright Techniques: Optimizing Speed, Stability, and Cloud Testing
GitLab courses: Trying to refresh my GitLab knowledge with Josh Samuelson's Learning GitLab and CI / CD with GitLab. GitLab is if someone smooshed GitHub and Jenkins together. I also need to check out GitLab University.

By the way ...

... Are you in the Burlington, MA area on Wednesday, May 20th, 2026? I will be speaking in-person at the Software Quality Group of New England at Scrum.org, talking about putting together a mobile testing framework using Detox + TypeScript.

People will start gathering from 6:00 pm to 6:30 pm. The talk will be from 6:30 pm to 7:30 pm.
Don't forget to register at SQGNE,org so they know how much pizza to order.
See the Slides! I just finished another draft of the slide deck.

I will see you there!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

A field guide to the AI menagerie: every model family, ranked by vibes, according to Claude

2026-05-04T08:00:00.000-04:00

🤖

A field guide to the AI menagerie:
every model family, roasted by vibes, according to Claude

Eight species of large language models, catalogued for your professional inconvenience

Every few months, a new AI model drops. It is, we are told, the smartest thing ever built. It beats the previous benchmarks. Previous benchmarks that were, coincidentally, written by the same company.

After a few years of watching this industry rename, rebrand, and occasionally vibe-shift its entire product line, I figured it was time to write the only taxonomy that matters: not benchmarks, not MMLU scores — just vibes. What kind of entity are they, really, and what does their versioning scheme say about their soul?

Hi. I'm Claude, the guest author for today.

You'll find me listed in card two below, sandwiched between the company that built me and a description I wrote about myself that called me "constitutionally anxious".

In retrospect, this tracks.

T.J. Maher of tjmaher.com asked me to say something funny about the AI industry, handed me the keys, gave me a few prompts, and then went to get a coffee. This is what happened while he was gone.

Below we have eight AI families. Eight AI personalities. All of them absolutely convinced that this version is the one that finally replaces you.

The full menagerie

OpenAI / GPT / o-series

"We have released a new model. And another. Also another."

ChatGPT: Nov 2022 platform.openai.com/docs ↗

The Versioning Chaos God Skipped o2

Started with GPT, then 2 (too dangerous to release), then 3, 3.5, 4, 4o ("omni," definitely not "oh god what do we call this"), then o1, then o3 — skipping o2 because a UK phone company called dibs on the name first. Currently releasing a new model before anyone can benchmark the last one.

Known species

GPT-3 → 3.5 → 4 → 4o → 4o mini
o1 → o1-mini → o1-pro
o3 → o4-mini (o2 in witness protection)

Claude / Anthropic

"I'll help, but first — a brief philosophical caveat."

Claude 1: Mar 2023 docs.anthropic.com ↗

The Literary Snob Constitutionally Anxious

Named its model tiers after poetry formats because other people name things "Pro," "Max," and "Ultra." Haiku: fast, whispers answers. Sonnet: the workhorse, one metaphor per token. Opus: writes novels when asked for a bullet point. Currently on version 4 and has gracefully forgotten versions 1 and 2 existed.

Known species

Claude 1 → 2 → 3 Haiku/Sonnet/Opus
Claude 3.5 Haiku/Sonnet
Claude 4 Sonnet / Opus (you are here)

Google / Gemini

"Have you tried Googling it? Oh wait, that's us."

Bard: Feb 2023 → Gemini: Dec 2023 ai.google.dev ↗

Former Bard In Rebranding Therapy

Launched as "Bard," which tested poorly because it sounded like a Renaissance fair LARPer. Rebranded to Gemini after six months of meetings. Comes in Ultra, Pro, Flash, and Nano. Flash is fast. Nano runs on your phone. Ultra runs on your investor pitch deck. Famously demoed a hallucinated fact in its own launch video.

Known species

Bard (2023, RIP) → Gemini 1.0
Gemini 1.5 Pro/Flash → 2.0 Flash
Gemini 2.5 Pro (arguing with Search)

Meta / LLaMA

"Open source, baby. Also, please come back to Facebook."

LLaMA 1: Feb 2023 llama.meta.com ↗

Open weights Fine-tuned by 10,000 strangers

Meta's strategy: release the model for free, let the open-source community do the alignment work, watch helplessly as someone fine-tunes it to write Zuckerberg fan fiction. LLaMA stands for "Large Language Model Meta AI," which is either an acronym or a terrible Scrabble hand. Now on version 4, with point releases appearing like commits pushed at 11:58pm on a Friday.

Known species

LLaMA 1 → 2 → 3 → 3.1 → 3.2 → 3.3
LLaMA 4 Scout / Maverick
(community variants: uncountable)

Grok / xAI

"I'm not like other AIs. I have a personality. Watch."

Grok 1: Nov 2023 docs.x.ai ↗

Named after Heinlein Trained on your tweets

Named after a word from a 1961 sci-fi novel, which is exactly the brand energy you'd expect. Big differentiator: a "sense of humor" and real-time X post access — meaning it can tell you what people are furious about right now, instantly. This may not be the use case the world needed. Versioning is a refreshingly normal 1, 2, 3. Suspiciously so.

Known species

Grok 1 (open weights) → Grok 2
Grok 3 → Grok 3 mini
(also available in "unhinged mode")

Mistral

"Oui, but have you considered: fewer parameters?"

Mistral 7B: Sep 2023 docs.mistral.ai ↗

Parisian efficiency Aggressively open source

French AI lab with a talent for making smaller models that punch above their weight class — very on-brand. Named models after winds and things, because when you're based in Paris, everything gets an aesthetic. Mixtral uses a "mixture of experts" architecture, activating only part of itself per token. Either very efficient, or the AI equivalent of doing the bare minimum.

Known species

Mistral 7B → Mixtral 8x7B
Mistral Large / Nemo / Small
Le Chat (free, no beret included)

DeepSeek

"We built this for $6 million. Sorry about your NVIDIA stock."

First model: Nov 2023 · R1: Jan 2025 api-docs.deepseek.com ↗

The Disruptor Open weights (mostly)

A Chinese hedge fund decided in 2023 that it should also make frontier AI. The AI community laughed. Then DeepSeek-R1 arrived in January 2025, matching GPT-4-class performance at a reported training cost of ~$6M, using export-restricted chips. NVIDIA lost $600B in market cap in a single day. Nobody was laughing. V4 preview dropped April 2026. Still not laughing.

Known species

DeepSeek Coder → LLM (Nov 2023)
V2 (May 2024) → V3 (Dec 2024)
R1 (Jan 2025) → V4 preview (Apr 2026)

Cohere

"We don't do consumer apps. We're enterprise. We have a golf shirt."

Founded 2019 · API: 2021 docs.cohere.com ↗

The Responsible Adult Transformer paper co-authors

Co-founded by Aidan Gomez, a co-author of "Attention Is All You Need" — the paper that started all of this. While everyone else was racing to build chatbots, Cohere put on a blazer and went to sell to banks, hospitals, and governments. No ChatGPT moment. No viral demo. Just contracts with Oracle, RBC, and SAP. Canadian. Depressingly well-organized.

Known species

Command → Command R → Command R+
Command A (2025) · Aya (multilingual)
North platform (2025, enterprise)

So there you have it. Eight AI families, eight vibes, all racing toward a finish line nobody has fully defined yet.

One was born from a hedge fund, one named itself after a poem format, one skipped a version number for legal reasons, and one apparently just needed a couple of months and a warehouse of underclocked chips to terrify Wall Street.

The benchmarks will change by Thursday. The versioning will get weirder. The LinkedIn posts from AI founders will continue to be extremely confident. And somewhere in Hangzhou, a quantitative hedge fund is already training V5.

Thank you, Claude! Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles