blogs

Docker Compose Testing Setup: Playwright, Selenium Grid, and API

2026-05-13T03:41:01Z

A complete Docker Compose testing setup for Playwright, Selenium Grid, and API tests. One file, consistent environments, zero drift.

The post Docker Compose Testing Setup: Playwright, Selenium Grid, and API appeared first on Software Testing & Automation.

constitution.md in Spec-Driven Development

2026-05-13T00:00:00Z

Written by - Millan Kaul

`constitution.md` : The Rules Behind Spec-Driven Development

constitution.md file is the project’s rulebook in Spec-Driven Development. It gives developers, architects, and QA teams a shared set of principles to follow before any feature spec, plan, or implementation begins.

If you are using GitHub Spec Kit, constitution.md is the document that sets the rules of the road for your project. It is not the feature spec itself. It is the project’s guiding policy: the stable principles that shape how every spec, plan, task, and implementation decision should behave.

In simple terms, it tells your AI-assisted workflow: this is how we build here. The Spec Kit README describes spec-driven development as a process where the constitution provides the prescriptive guidance that keeps later planning and implementation aligned.

Why it matters

For developers and architects, constitution.md :

Acts like an architectural contract.
It helps keep decisions consistent across features, teams, and AI-generated output, instead of letting each prompt invent its own style of development.

That is especially useful when multiple people or agents contribute to the same codebase.

For QA and testing, it is equally important because it creates a baseline for what should always be true.

If your constitution says “no silent failures,” “test critical paths first,” or “do not leak user data,” then those become expectations that can be checked during review, planning, and implementation.

How to use it

A good way to think about constitution.md is this: it is the document you write before the feature spec. The feature spec changes often, but the constitution stays steady and guides every later step in the workflow.

A practical flow looks like this:

Define the project principles in constitution.md.
Use those principles while creating the feature spec.
Let the plan, tasks, and implementation remain consistent with those principles.
Review the final result against the constitution, not just the feature requirements.

Here is a constitution-template.md

---
constitution.md
---

# [PROJECT_NAME] Constitution
<!-- Example: Spec Constitution, TaskFlow Constitution, etc. -->

## Core Principles

### [PRINCIPLE_1_NAME]
<!-- Example: I. Library-First -->
[PRINCIPLE_1_DESCRIPTION]
<!-- Example: Every feature starts as a standalone library; Libraries must be self-contained, independently testable, documented; Clear purpose required - no organizational-only libraries -->

### [PRINCIPLE_2_NAME]
<!-- Example: II. CLI Interface -->
[PRINCIPLE_2_DESCRIPTION]
<!-- Example: Every library exposes functionality via CLI; Text in/out protocol: stdin/args → stdout, errors → stderr; Support JSON + human-readable formats -->

### [PRINCIPLE_3_NAME]
<!-- Example: III. Test-First (NON-NEGOTIABLE) -->
[PRINCIPLE_3_DESCRIPTION]
<!-- Example: TDD mandatory: Tests written → User approved → Tests fail → Then implement; Red-Green-Refactor cycle strictly enforced -->

### [PRINCIPLE_4_NAME]
<!-- Example: IV. Integration Testing -->
[PRINCIPLE_4_DESCRIPTION]
<!-- Example: Focus areas requiring integration tests: New library contract tests, Contract changes, Inter-service communication, Shared schemas -->

### [PRINCIPLE_5_NAME]
<!-- Example: V. Observability, VI. Versioning & Breaking Changes, VII. Simplicity -->
[PRINCIPLE_5_DESCRIPTION]
<!-- Example: Text I/O ensures debuggability; Structured logging required; Or: MAJOR.MINOR.BUILD format; Or: Start simple, YAGNI principles -->

## [SECTION_2_NAME]
<!-- Example: Additional Constraints, Security Requirements, Performance Standards, etc. -->

[SECTION_2_CONTENT]
<!-- Example: Technology stack requirements, compliance standards, deployment policies, etc. -->

## [SECTION_3_NAME]
<!-- Example: Development Workflow, Review Process, Quality Gates, etc. -->

[SECTION_3_CONTENT]
<!-- Example: Code review requirements, testing gates, deployment approval process, etc. -->

## Governance
<!-- Example: Constitution supersedes all other practices; Amendments require documentation, approval, migration plan -->

[GOVERNANCE_RULES]
<!-- Example: All PRs/reviews must verify compliance; Complexity must be justified; Use [GUIDANCE_FILE] for runtime development guidance -->

**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE]
<!-- Example: Version: 2.1.1 | Ratified: 2025-06-13 | Last Amended: 2025-07-16 -->

What to put in it

Keep it short, clear, and opinionated. Good constitution rules usually cover architecture, quality, security, testing, data handling, and how AI should behave in your codebase. Examples include:

prefer simple, testable designs.
fail loudly instead of hiding errors.
protect user data by default.
require tests for critical paths.
keep generated code aligned with existing architecture.

That style works well because the constitution is meant to be a durable reference, not a long policy manual.

My take

constitution.md is where Spec-Driven Development becomes real. It gives developers a shared architecture mindset, gives QA a testable quality baseline, and gives AI a guardrail before it starts generating anything.

In other words, it is the document that turns “build it fast” into “build it right.”

The 2026 QA Engineer Career Roadmap: Month-by-Month From Manual Testing to Senior SDET

2026-05-12T22:00:00Z

The complete 12-month QA career roadmap from manual testing to senior SDET. Month-by-month learning plan with deliverables, interview milestones, and salary expectations.

The post The 2026 QA Engineer Career Roadmap: Month-by-Month From Manual Testing to Senior SDET appeared first on Software Testing & Automation.

The IQL Manifesto: Seven Commitments for AI Quality Leadership

2026-05-12T21:25:54Z

This is the post where two things I’ve been building separately finally meet.

When I started the Intelligent Quality Leadership series earlier this year, I had a model I wanted to stress-test and a set of ideas I needed to get out of my head and onto a page. What I didn’t expect was that writing it would force me to reckon with something important.

I’ve been building towards quality leadership my entire career. The AI dimension is more recent and is applied, hands-on, and still evolving. What this series has done is show me where those two things intersect. And that intersection is where the real work is.

This post is where that intersection lands.

Where This Started

I’ve created Quality Engineering principles twice before, at two very different companies.

The first time was at easyJet, built in collaboration with our QE Architect at the time, Suman Bala. Bold, energetic, built for scale. “Quality as a Culture” sat at the centre of everything, surrounded by principles designed to rally a 120-person distributed organisation around a shared belief. The language was deliberate, “Automate ALL the RIGHT things”, “Testability as a foundation”, because at that size, you need something people can remember, repeat, and actually live by.

The second time was at Goodnotes. The bones were the same but the expression was different. More refined, more embedded in product thinking. “Sustainable Automation” instead of “automate the right things.” “Collaborative Quality Ownership” instead of a rallying cry. The principles had grown up with me.

Both sets still hold up. But neither of them accounts for what AI has done to our discipline. And that’s what this is.

Why a Manifesto

I spent a long time thinking about what to call this. Principles felt too passive. Framework felt too corporate. Pillars, charter, ethos, all considered and I chose to set them aside.

I landed on Manifesto because it’s the right word for what this is. A manifesto isn’t a description of how things are. It’s a stake in the ground about how things should be. It invites adoption, challenge, and conversation. And that’s exactly what I want this to do.

These aren’t seven things I’ve observed in good teams. They’re seven commitments I believe quality leaders need to make and keep making, if they’re serious about leading in an AI-first engineering world.

The IQL Manifesto

01 – Challenge the reasoning, not just the result

AI outputs can look right and be wrong. Quality leaders build cultures that interrogate why, not just what.

This is the mindset shift that everything else depends on. Traditional testing has a simple contract: given this input, I expect this output. AI breaks that contract. It’s confident, fluent, and plausible. It can be completely wrong without looking like it.

The skill quality leaders need to build, in themselves and their teams, isn’t just evaluating what an AI produced. It’s knowing how to interrogate why, and when to trust the answer versus when to push back hard. TRACE, the framework I developed in IQL Practical #1, is one way to do this. But the principle is bigger than any framework. It’s a habit of mind.

02 – Governance doesn’t check quality at the end. It builds the foundation at the start.

The earliest conversations about an AI feature are where the quality bar is set, whether you’re in the room or not.

There’s a version of AI governance that lives in slide decks and quarterly board updates. It’s tidy, well-presented, and almost completely disconnected from the moments that actually matter.

Real governance happens on a Tuesday morning, ten weeks before release, when a product manager sends a calendar invite for an AI feature scoping kickoff. The quality leader who understands this isn’t waiting for a test plan to think about risk. They’re in that room, asking the questions nobody else thought to ask yet. If you’re not there, the quality bar gets set without you. And it’s very hard to raise it later.

03 – Test the AI, not just the software containing it.

The failures that matter most in AI systems don’t look like bugs. They look like drift, bias, inconsistency, and cost you didn’t see coming.

Most teams are running their existing playbook against a fundamentally different kind of system. They test the integration layer. They check the UI. Someone runs a few prompts manually and confirms the responses look reasonable. The feature ships.

Three months later, the model gets updated. Outputs subtly change. A user in a different region gets a culturally inappropriate response. The AI starts hallucinating a statistic that sounds authoritative but is completely fabricated. None of it gets caught because none of it was being tested for.

Testing AI means covering five distinct dimensions: reliability across runs, behavioural consistency, safety at the edges, explainability, and cost you can sustain. If your AI testing strategy is your existing test strategy with an AI component bolted on, you don’t have an AI testing strategy. You have a gap you haven’t found yet.

04 – Faster automation isn’t the answer. Smarter AI use is.

AI can generate a thousand tests in minutes. The hard question is whether any of them are asking the right thing.

The rush to use AI for automation is creating a lot of volume and very little value. Teams are generating tests at scale without asking whether they’re generating the right tests. Speed becomes the metric when quality should be.

The risk isn’t that teams automate too slowly. It’s that they automate thoughtlessly and call it transformation. A quality leader’s job is to make sure AI amplifies good judgement rather than replacing the need for it. That’s a harder thing to lead than it sounds, because the volume looks impressive and the dashboard looks green.

05 – Collaborative Quality Ownership isn’t optional.

In an AI-first team, real ownership has to be deliberate, visible, and led.

AI has created a democratisation illusion around quality. The optimistic narrative goes: AI puts testing capability in everyone’s hands, so shared ownership should be easier to achieve than ever. I’d ask you to look more carefully at what’s actually happening in your teams.

When a developer generates a test suite using an AI tool, they’ve accessed a quality capability. They haven’t necessarily taken ownership of quality. Ownership means understanding what you’re testing and why, caring about what the output tells you, and being willing to have the difficult conversation when the picture isn’t clear. Generating a test file and merging it doesn’t require any of that.

The risk isn’t that AI keeps people away from quality. It’s that AI gives people the feeling of participating in quality without the responsibility that real participation requires. In an AI-first team, collaborative ownership doesn’t emerge naturally. It has to be designed in, made visible, and actively led.

06 – Human judgement moves to where it still matters most.

The decisions AI can’t make, the risks it can’t see, and the moments where someone needs to say no.

The human in the loop isn’t going anywhere. They’re just moving up.

As AI takes on more of the execution, writing tests, analysing results, flagging anomalies, it means the human role shifts. Not away from quality, but towards the parts of quality that require something AI genuinely can’t provide: contextual judgement, ethical reasoning, and the willingness to say “I don’t think we should ship this” when the data is inconclusive but the instinct is clear.

That’s not a limitation of AI maturity. It’s a deliberate quality decision. The teams that understand this will build better products. The ones that don’t will find out the hard way.

07 – Confidence is the differentiator.

In AI-first products, quality isn’t a checkbox. It’s what users feel when the system doesn’t let them down.

In a world where AI capabilities are increasingly commoditised, the product that wins isn’t necessarily the most powerful one. It’s the one users trust. The one that behaves consistently. The one that doesn’t surprise them in ways that erode confidence.

Quality engineering in an AI-first world is ultimately about building that confidence, not as a feature or as a compliance requirement, but as a competitive advantage. That’s what users feel when the system doesn’t let them down. And that feeling is what keeps them coming back.

What I Want You to Do With This

I didn’t write this to be read once and filed away. I wrote it to be used.

If something here resonates, take it into your next team conversation. Use it as a lens for where your quality practice has gaps. Challenge the parts you disagree with! I mean that genuinely, because the best frameworks get sharper through friction, not consensus.

If your organisation is figuring out what AI-first quality engineering actually looks like in practice, I’d love to talk. These commitments are a starting point, not a finished answer.

And if this feels like the beginning of something bigger… well. It might be.

The IQL Manifesto is part of the Intelligent Quality Leadership series on leadtestinclude.com. If you’re new to the series, start at the beginning.

Want to talk about any of this? Book time with me at calendly.com/simon-leadtestinclude/30min or find me on LinkedIn.

The Automation Island Trap: Why QAs Who Only Automate Stop Growing

2026-05-12T10:08:47Z

There’s a pattern I keep seeing across different teams and companies, regardless of size or maturity.

A QA engineer learns automation. At first, it’s a huge step forward. They feel empowered, more technical, more aligned with development. They start contributing faster, building pipelines, integrating tests into CI/CD. It feels like progress and it is. But then something and happens.

They begin to spend less time understanding the product. They stop participating actively in refinement sessions. Exploratory testing becomes rare. Conversations with product managers decrease. Over time, their entire contribution becomes centred around writing and maintaining automated tests. Without realizing it, they isolate themselves.

They move into what I call the automation island, a place where activity is high, but impact is limited.

From the outside, everything looks fine. There are commits, pipelines are running, and dashboards show execution results. But underneath that surface, something critical is missing: real quality ownership.

The Illusion of Value

Let’s be clear: automation is valuable. Critical, even. Automation gives a strong sense of productivity. You can see tangible outputs: test scripts, reports, pass/fail results. It creates the impression that quality is being actively managed. However, this is where many teams and many QAs fall into a trap. They confuse activity with value.

A large test suite does not automatically mean better quality. High coverage numbers do not guarantee meaningful validation. Fast execution does not imply that the right things are being tested.

When automation is not guided by a deeper understanding of the system, it often ends up validating assumptions instead of challenging them.

You start to see situations where:

Tests consistently pass, but production issues still occur
Failures are treated as “flaky” instead of being investigated
Test results are reviewed mechanically, without insight
The team trusts the automation less over time

In these cases, automation is not improving quality, it’s creating a false sense of confidence. The root cause is not the automation itself. It’s the lack of critical thinking and product awareness behind it.

QA Is a Thinking Role, Not a Toolset

At its core, QA is not defined by tools, frameworks, or even test cases. It is defined by how you think about quality. A strong QA engineer constantly builds a mental model of the system:

How users interact with it
Where complexity exists
What could realistically go wrong
What failures would have the biggest impact

This kind of thinking cannot be replaced by automation. It requires curiosity, questioning, and continuous learning. When QAs limit themselves to automation, they risk reducing their role to execution. They follow predefined scenarios, validate expected outcomes, and move on to the next task. But real quality work happens in the spaces where things are unclear:

When requirements are ambiguous
When edge cases are not documented
When user behavior is unpredictable
When systems interact in unexpected ways

Automation can support this work, but it cannot lead it. Without this broader perspective, automation becomes efficient but shallow.

The Automation-Only Anti-Pattern

When a QA operates only within automation, their workflow tends to become reactive and limited. Typically, they receive a feature after development is complete. Their task is to automate validation based on what has already been defined and implemented. The focus is often on expected flows, the scenarios that developers and product teams already had in mind. This creates a narrow validation scope.

What’s missing is the ability to influence quality earlier in the process. By the time automation begins, many important decisions have already been made, sometimes incorrectly. Without early involvement:

Risks are not identified when they are easiest to address
Gaps in requirements go unnoticed
Edge cases are discovered too late
Design issues become expensive to fix

In this model, QA becomes a checkpoint at the end, rather than a driver of quality throughout the process. Automation reinforces this if it’s treated as the primary responsibility instead of one part of a broader strategy.

Why This Limits Your Growth

From a professional development perspective, staying in the automation island creates a very specific kind of limitation. You become highly skilled in a narrow area, writing and maintaining automated tests, but you miss the opportunity to develop the broader capabilities that define senior QA engineers. These include:

Understanding business context and user value
Making risk-based decisions
Communicating effectively with stakeholders
Influencing product and technical direction
Adapting testing strategies to different scenarios

Over time, this creates dependency on tools and frameworks. Your value becomes tied to specific technologies rather than transferable skills. And this is risky. Tools evolve quickly, frameworks change and AI is now part of day-to-day QA work. New approaches emerge. If your expertise is too tightly coupled to a specific implementation, it becomes harder to adapt.

On the other hand, QAs who develop strong analytical and product-focused skills remain relevant regardless of the tools they use. Because they don’t just execute tests, but they shape how quality is approached.

The Real Role of Automation

Automation is most powerful when it is used intentionally, as part of a well-defined strategy. Its purpose is not to replace thinking, but to amplify it. When guided correctly, automation allows teams to:

Reduce repetitive manual effort
Increase the speed of feedback
Ensure consistency in validation
Scale testing across environments and configurations

But these benefits only materialize when automation is aligned with real needs. That alignment comes from:

Understanding what matters most to the business
Identifying high-risk areas
Prioritizing stability and long-term value
Designing tests that provide meaningful insights

Without this foundation, automation becomes noise, fast, scalable, but not particularly useful.

What High-Impact QAs Do Differently

QAs who consistently bring value approach their role very differently. They don’t see automation as their identity. Instead, they see it as one of many tools they use to ensure quality. They actively participate in the entire lifecycle of a feature.

They engage early, asking questions during refinement and planning. This allows them to uncover risks and ambiguities before development even begins.

They think in terms of impact, not just coverage. Instead of trying to automate as much as possible, they focus on what actually matters: the flows and scenarios that could affect users and the business. They combine different testing approaches. Automation is complemented by exploratory testing, data analysis, and collaboration with other team members.

They also take ownership of outcomes. When something fails, whether in tests or in production, they investigate deeply, looking for root causes and systemic improvements. This holistic approach is what differentiates a QA who executes tasks from a QA who drives quality.

Breaking Out of the Automation Island

Moving away from an automation-only mindset doesn’t require a complete reset. It starts with small, intentional changes in how you approach your work.

Begin by increasing your involvement in the earlier stages of development. Even one meaningful contribution during a refinement session can shift your perspective significantly. Reintroduce exploratory testing into your workflow. Before automating a feature, spend time understanding how it behaves, where it might break, and what feels unclear. Challenge assumptions. Instead of taking requirements at face value, ask questions about edge cases, constraints, and user behavior. Collaborate more closely with developers and product managers. Quality improves when perspectives are shared, not isolated. And most importantly, shift your mindset. Move from: “What tests should I automate?” To: “How do I ensure this feature delivers real value and works reliably in the real world?”

This shift changes everything, including how you use automation.

How to Get Out of the Automation Island (Step by Step)

Breaking out of the automation island is not about abandoning automation. It’s about repositioning it within a broader, more impactful QA strategy. The shift doesn’t happen overnight, but it can be built deliberately through small, consistent changes in how you work.

1. Reconnect With the Product

Start by rebuilding your understanding of the system beyond test cases. Instead of jumping straight into automation when a feature is ready, spend time exploring it like a user would. Navigate through different flows, try unexpected inputs, and observe how the system behaves under less obvious conditions.

Example:
You receive a new checkout feature to automate.
Instead of immediately scripting the happy path:

Try different payment methods
Interrupt the flow midway
Use invalid or borderline data
Switch devices or browsers

This often reveals gaps that were never documented and would never be covered by predefined automated tests.

2. Shift Left: Get Involved Earlier

One of the biggest differences between average and high-impact QAs is when they engage. Don’t wait for development to finish. Join refinement sessions with a clear purpose: identify risks before they become defects.

Example:
During a refinement for a new user registration flow, instead of passively listening:

Ask what happens if the email is already registered
Question password constraints and edge cases
Clarify expected behavior for slow network scenarios

This kind of input can prevent entire categories of bugs before a single line of code is written.

3. Introduce Risk-Based Thinking

Not everything should be automated. Not everything deserves the same level of attention. Start prioritizing based on impact and probability, not just feasibility.

Example:
You have limited time in a sprint. Instead of automating 10 low-risk scenarios:

Focus on 2–3 critical user journeys (e.g., login, payment, data persistence)
Identify where failures would hurt the business most
Ensure those paths are deeply validated (automation + exploratory)

This increases the value density of your work.

4. Combine Automation With Exploration

Automation is excellent for consistency, but exploration is where real insights emerge. Before writing automation, explore the feature. After automation, revisit it when changes happen.

Example:
A test suite for a search feature is fully automated and passing.
Instead of trusting it blindly:

Perform exploratory testing with unusual queries
Test performance under load
Try edge cases like special characters or empty inputs

This often exposes issues that automation alone misses.

5. Turn Failures Into Learning Opportunities

One clear sign of being stuck in automation island is ignoring failures or labelling them as “flaky” without investigation. High-impact QAs treat every failure as a signal.

Example:
An automated test fails intermittently.
Instead of retrying or disabling it:

Investigate logs and system behavior
Identify if it’s a timing issue, environment instability, or real defect
Collaborate with developers to fix the root cause

This improves both the system and the credibility of your automation.

6. Increase Collaboration, Reduce Isolation

Automation-heavy QAs often work in isolation. Breaking out of the island requires intentional collaboration. Quality is a shared responsibility, not a QA-only function.

Example:

Pair with developers to review testability of features
Discuss edge cases with product managers
Share insights from exploratory testing with the team

These interactions expand your perspective and increase your influence.

7. Redefine Your Success Metrics

If your success is measured only by:

Number of automated tests
Execution speed
Coverage percentage

You will naturally stay on the island. Shift your focus toward impact-driven metrics:

Bugs prevented before production
Risks identified early
Critical issues detected before release
Confidence level of the team in the product

Example:
Instead of saying:
“I automated 20 test cases this sprint”

Aim to say:
“I identified a critical edge case in checkout that could have caused revenue loss, and ensured it’s now covered and monitored.”

8. Use Automation as an Amplifier, Not a Crutch

Once you shift your mindset, automation becomes significantly more powerful. You’re no longer automating everything. You’re automating the right things, for the right reasons.

Example:
After identifying high-risk scenarios through exploration and refinement:

Automate those scenarios for fast regression feedback
Keep tests maintainable and meaningful
Continuously evaluate if they still provide value

This creates a lean, trustworthy, and high-impact automation suite.

Conclusion

Automation is one of the most powerful capabilities a QA can have. But on its own, it is not enough. Because without context, it becomes mechanical. If there is no involvement, it becomes isolated. And without critical thinking, it becomes superficial.

The most effective QAs don’t define themselves by the tests they write. They define themselves by the impact they have on the product and the process. They move across the entire development lifecycle, combining strategy, exploration, and automation to deliver real quality.

If you want to grow as a QA engineer, don’t aim to be the fastest at writing scripts. Aim to be the one who understands quality the deepest and improve the process and uses automation to amplify that understanding.

A tester’s story about assumptions, resilience, and learning from real‑life bugs

2026-05-12T07:00:00Z

You know that moment when everything looks right… until it’s completely wrong?

That happened to me on a German train one morning at 9:47.

The setup looked perfect.

Ticket bought
Track 103 clearly marked
Airport sign displayed
People with suitcases all around

Every signal said this was the right way.

So when the train arrived, I stepped in with confidence.

Even Google Maps agreed.
Even the people inside agreed.

One minute later, something felt off.

No airport appeared on the list of upcoming stations.

That was the moment reality caught up with confidence.

This was S1, not S8 or S9.

And suddenly, I wasn’t alone in the mistake.
The train was full of people who, just like me, had trusted the same signals and reached the same wrong conclusion.

Thirty minutes lost fixing it.
Going in the wrong direction.
Coming back to the starting point.
Waiting again for the correct train.

A small chaos.

But also a familiar one.

Because this is exactly what happens in our work as testers.

A real‑life bug uncovered in production

As testers and test managers, we deal with bugs all the time.
But this one didn’t come from code.

It came from assumptions.

Everything appeared aligned.
The environment looked correct.
The signals matched expectations.

And yet, the conclusion was wrong.

That’s how many defects escape into production.
Not because people are careless.
Not because processes don’t exist.

But because confidence replaces verification when things look familiar.

What this train taught me as a tester

In testing, we are trained to look for what can go wrong.
Still, we are human.

We trust patterns.
We trust systems.
We trust crowds.

This moment reminded me of lessons we all know, but sometimes forget.

Anticipation matters, even when everything seems aligned
Asking questions matters, but asking the right source matters more
Plans never equal reality, adaptability is part of quality
Resilience is essential, things will break and we still have to move forward

Fixing the mistake wasn’t elegant (lost 30 minutes from my expected arrival time to airport)
But it worked.

And that’s often what resilience looks like in real life and in testing.

Bonus reflections from the platform called life

Standing on that platform, waiting for the correct train, I kept thinking about testing principles playing out in front of me.

UX matters, when many users fail, the system failed
Crowd confidence is not validation, everyone can be wrong together
Edge cases are not rare, they are real life
Communication equals quality, unclear information creates chaos
User confusion is not a mistake, it is a signal

No one on that train was negligent.
They were navigating a system that allowed ambiguity.

Just like many users do with our products.

Resilience is part of quality

In testing, resilience isn’t just about handling failures in production.
It’s about staying calm when reality diverges from expectations.

You adapt.
You reroute.
You keep going.

That day, the wrong train didn’t ruin the journey.
It enriched it.

It reminded me why testing is not just about tools, frameworks, or processes.

It’s about thinking.
Observing.
Questioning.
Empathizing.

Final thought

Real life is the most honest test environment.

And sometimes, the wrong train is the best reminder of why testers exist.

#ExploreWithEmna

Test Automation Made You Faster. It Also Made You Blind.

2026-05-12T03:46:00Z

Your pipeline is green.

Continue reading on Medium »

Manual Testers and AI Agents: Why Service Company QA Jobs Are Disappearing in 2026

2026-05-12T03:40:02Z

AI agents are replacing manual testers in Indian IT service companies. Here is the data on salary stagnation, AI tool adoption, and the 90-day escape plan for QA engineers who want to survive.

The post Manual Testers and AI Agents: Why Service Company QA Jobs Are Disappearing in 2026 appeared first on Software Testing & Automation.

Spec-Driven Development: QA’s North Star in AI-Native Teams

2026-05-12T00:00:00Z

Written by - Millan Kaul

Spec-driven development (SDD) positions QA as the enforcer of executable specs, your north star for AI reliability. In AI-native workflows, QA shifts from reactive bugs to proactive spec validation, catching hallucinations before deploy.

Opinion: Without QA owning SDD, AI ships “good enough” disasters; with it, specs become living tests.

QA’s Pivotal Role

QA authors/validates specs as contracts (inputs, outputs, edges), then verifies AI outputs match.

Tools like GitHub Spec Kit auto-gen tests from specs, reducing manual toil by 40%.
QA metrics: Schema compliance, edge coverage >90%.

QA-Centric Workflow

QA Writes Spec: YAML/Markdown with assertions (e.g., “no PII, confidence >0.8”).
AI Task Breakdown: Prompt-engineered plans, QA reviews.
Implement + QA Gate: Code/prompts must pass spec-evals (LLM-judge, unit tests).
QA Iteration: Human-in-loop for failures.

Sample: QA Spec → PII-Safe API

QA spec enforces no PII leak.

# qa_spec.yaml

spec:
  endpoint: /users
  input: {email: string@format=email, name: string}
  output: {id: string, name: string}  # No email echo!
  assertions:
    - no_pii: true  # QA guardrail hook
    - schema_match: 100%

FastAPI + QA Guardrail

# test.py

from fastapi import FastAPI
from pydantic import BaseModel, EmailStr
from nemoguardrails import LLMRails  # QA layer

app = FastAPI()
rails = LLMRails.from_config("./qa_config")  #Spec + PII rails

class User(BaseModel):
    email: EmailStr
    name: str

@app.post("/users")
async def create(user: user):
    # QA Pre: Spec input validation
    if not rails.process_input(user.model_dump()):  #PII/block
        raise HTTPException(400, "PII detected")
    # LLM? Or logic, post-validate output
    result = {"id": user.email, "name": user.name}  #No email!
    if not rails.process_output(result):  #Spec schema check
        raise HTTPException(500, "Spec violation")
    return result

Run QA tests: pytest --spec qa_spec.yaml for 100% alignment.

QA Wins

Cuts drift 50% via evals.
Guardrails as spec extensions.

QA Checklist

Spec coverage: Edges/PII.
Auto-eval: >95% pass.
CI/CD gate: QA signoff.

QA makes SDD unbreakable—AI follows specs, not whims.

QA Metrics That Actually Matter: 12 KPIs to Replace Vanity Metrics

2026-05-11T22:00:00Z

Stop tracking vanity metrics. 12 QA KPIs that actually indicate quality: escaped defect rate, mutation score, release confidence score, and cost of defects.

The post QA Metrics That Actually Matter: 12 KPIs to Replace Vanity Metrics appeared first on Software Testing & Automation.

Pramod Dutta discusses 7 Playwright features senior SDETs use daily

2026-05-11T20:52:22Z

Pramod Dutta, an SDET from Tekion, posted on LinkedIn "7 Playwright features senior SDETs use daily". The Playwright features, according to Pramod:

"→ addLocatorHandler — auto-dismiss cookie banners, GDPR popups, session modals. One handler. Whole suite cleaner. Stop writing wrapper functions.
"→ browser.bind() — launch one browser. Let your test, Claude Code, and debugger all attach to it simultaneously. The debugging workflow that fixes "works on my machine" forever.
"→ URLSearchParams in request.get() — stop building query strings manually. Eliminates an entire bug class around URL encoding in API tests.
"→ expect.toPass() — polling-based assertions for state that converges over time. The right tool for 'this should eventually be true.' Not the same as expect(locator).toBeVisible(). Different problem, different solution.
"→ page.requestGC() — manually trigger garbage collection in the browser. The only Playwright tool that catches memory leaks before production. Used with WeakRef in page.evaluate().
"→ --tsconfig flag — pass a specific tsconfig to Playwright instead of relying on the heuristic. Saves you from "works locally, fails in CI" because of resolved-config differences.
"→ webServer.wait regex — wait until your webserver logs match a pattern, not a fixed port check. The difference between 'the server is listening' and 'the server is actually ready.'"

I scribbled these features down, then turned them over to Claude.ai, my research assistant, to see if he could explain these methods, come up with sample code using them, then add bullet points to the official documentation, source code, and the release notes.

... Let's see how well Claude did explaining them ...

`page.addLocatorHandler()` v1.42 — March 2024

Registers a callback that fires whenever a specified locator becomes visible and risks blocking a Playwright action. Playwright checks for the locator before every actionability check or auto-waiting assertion. When the locator is visible, the handler runs first, then the test continues. The handler only fires during actions; appearing passively between actions does not trigger it. After the handler runs, Playwright waits for the overlay to become hidden before proceeding (opt out with noWaitAfter: true). Use the times option to cap the number of invocations. The handler receives the triggering locator as its argument.

Cookie banner (fire-and-forget)

// Register once, before navigating.
await page.addLocatorHandler(
  page.getByRole('heading', { name: 'You are in control of your cookies.' }),
  async () => {
    await page.getByRole('button', { name: 'Accept all' }).click();
  }
);

// Write the rest of the test normally.
await page.goto('https://www.example.com/');
await page.getByRole('link', { name: 'Shop now' }).click();

Self-removing handler with access to the triggering locator

// The handler receives the matched locator; times: 1 auto-removes it.
const closeBtn = page.getByLabel('Close');
await page.addLocatorHandler(closeBtn, async (locator) => {
  await locator.click();
}, { times: 1 });

// Remove manually when no longer needed.
await page.removeLocatorHandler(closeBtn);

Fail fast on an application error overlay

// Instead of dismissing, throw immediately so the test does not hang.
await page.addLocatorHandler(
  page.getByRole('heading', { name: "Sorry, there's been a problem" }),
  async () => {
    throw new Error(`Error overlay appeared on ${await page.url()}`);
  }
);

Resource	Link
Official docs	playwright.dev — page.addLocatorHandler()
Release notes (v1.42)	playwright.dev/docs/release-notes#version-142
GitHub release (v1.42.0)	github.com — Release v1.42.0
Source code	packages/playwright-core/src/client/page.ts

`browser.bind()` v1.59 — April 2026

Binds a launched browser to a named pipe or WebSocket endpoint, making it available to playwright-cli, @playwright/mcp, and any other Playwright client simultaneously. Before v1.59, sharing a browser between an MCP agent and a test suite meant two separate processes. Now a single browser.bind() call exposes one browser session to all of them. Call browser.unbind() to stop accepting new connections; existing connections keep working. Run playwright-cli show to open the live dashboard listing all bound browsers.

Bind via named pipe (local only)

import { chromium } from '@playwright/test';

const browser = await chromium.launch();
const { endpoint } = await browser.bind('my-session', {
  workspaceDir: '/my/project',
});
console.log('Session bound:', endpoint);

Bind via WebSocket (shareable over a network)

const { endpoint: wsEndpoint } = await browser.bind('my-session', {
  host: 'localhost',
  port: 0, // let the OS pick an available port
});
console.log('WebSocket endpoint:', wsEndpoint); // ws://localhost:PORT/...

// A second Playwright client connects to the same browser.
const browser2 = await chromium.connect(wsEndpoint);
const page = await browser2.newPage();

// Stop accepting new connections when done.
await browser.unbind('my-session');

Resource	Link
Official docs	playwright.dev — browser.bind()
Release notes (v1.59)	playwright.dev/docs/release-notes#version-159
GitHub release (v1.59.0)	github.com — Release v1.59.0
Source code	packages/playwright-core/src/client/browser.ts

`URLSearchParams` in `request.get()` v1.46 — August 2024

The params option on APIRequestContext methods (get, post, put, patch, delete) now accepts a native URLSearchParams object or a plain query string in addition to the existing plain-object shorthand. This matters when you need to send the same key more than once, which a plain object cannot express.

test('query params via URLSearchParams', async ({ request }) => {

  // URLSearchParams lets you append duplicate keys.
  const params = new URLSearchParams();
  params.set('userId', '1');
  params.append('tag', 'automation');
  params.append('tag', 'playwright'); // duplicate key — plain object cannot do this

  const response = await request.get(
    'https://jsonplaceholder.typicode.com/posts',
    { params }
  );
  expect(response.status()).toBe(200);

  // Or just pass a pre-built query string.
  const response2 = await request.get(
    'https://jsonplaceholder.typicode.com/posts',
    { params: 'userId=1&tag=automation' }
  );
  expect(response2.ok()).toBeTruthy();
});

Resource	Link
Official docs	playwright.dev — APIRequestContext.get()
Release notes (v1.46)	playwright.dev/docs/release-notes#version-146
GitHub release (v1.46.0)	github.com — Release v1.46.0
Source code	packages/playwright-core/src/client/fetch.ts

`expect.toPass()` intervals config — v1.44 May 2024

Wraps an async callback and retries it until every assertion inside passes or a timeout is reached. Unlike Playwright's built-in auto-retrying locator assertions, toPass can contain arbitrary logic including API requests, multi-step flows, and non-locator checks. By default the timeout is 0 (no timeout) and probe intervals default to [100, 250, 500, 1000] ms. Both are configurable globally in testConfig.expect.toPass or per-call.

Polling an endpoint until ready

test('waits for background job', async ({ request }) => {
  await expect(async () => {
    const res = await request.get('https://api.example.com/job/status');
    expect(res.status()).toBe(200);
    const body = await res.json();
    expect(body.status).toBe('complete');
  }).toPass({
    intervals: [1_000, 2_000, 10_000], // probe at 1 s, 2 s, then every 10 s
    timeout:   60_000,
  });
});

Global configuration in `playwright.config.ts`

import { defineConfig } from '@playwright/test';
export default defineConfig({
  expect: {
    toPass: {
      timeout:   30_000,
      intervals: [500, 1_000, 5_000],
    },
  },
});

Resource	Link
Official docs	playwright.dev — Assertions: expect.toPass()
Release notes (v1.44)	playwright.dev/docs/release-notes#version-144
GitHub release (v1.44.0)	github.com — Release v1.44.0
Source code	packages/playwright/src/matchers/toPass.ts

`page.requestGC()` v1.46 — August 2024

Asks the browser to run garbage collection on the page. There is no guarantee that every unreachable object is collected, but in practice it is reliable enough to detect leaks in combination with JavaScript WeakRef. The pattern: store a WeakRef to the object under suspicion, call requestGC(), then assert that deref() returns undefined.

test('no memory leak on the large object', async ({ page }) => {
  await page.goto('https://your-app.com');

  // 1. Pin a WeakRef to the object under suspicion.
  await page.evaluate(() => {
    (globalThis as any).suspectWeakRef = new WeakRef((window as any).largeObject);
  });

  // 2. Request GC.
  await page.requestGC();

  // 3. A living WeakRef means a leak; deref() should now be undefined.
  const leaked = await page.evaluate(
    () => !!(globalThis as any).suspectWeakRef.deref()
  );
  expect(leaked).toBe(false);
});

Resource	Link
Official docs	playwright.dev — page.requestGC()
Release notes (v1.46)	playwright.dev/docs/release-notes#version-146
GitHub release (v1.46.0)	github.com — Release v1.46.0
Source code	packages/playwright-core/src/client/page.ts

TypeScript Config: `--tsconfig` flag & `testConfig.tsconfig`

Playwright looks up the nearest tsconfig.json for each imported file by default. Two complementary options override this. The --tsconfig CLI flag (v1.46, August 2024) applies one tsconfig to all imported files from the command line. The testConfig.tsconfig property (v1.49, October 2024) does the same from the config file. Note that Playwright only honours four tsconfig options: allowJs, baseUrl, paths, and references.

CLI flag

# Apply one tsconfig to everything Playwright loads.
npx playwright test --tsconfig tsconfig.test.json

Config file property

// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
  tsconfig: './tsconfig.test.json',
});

Example `tests/tsconfig.json`

{
  "compilerOptions": {
    "baseUrl": ".",
    "allowJs": true,
    "paths": {
      "@helpers/*": ["../src/helpers/*"]
    }
  }
}

Resource	Link
TypeScript guide	playwright.dev — TypeScript
`testConfig.tsconfig` docs	playwright.dev — TestConfig.tsconfig
Release notes (`--tsconfig`, v1.46)	playwright.dev/docs/release-notes#version-146
Release notes (`testConfig.tsconfig`, v1.49)	playwright.dev/docs/release-notes#version-149
GitHub release (v1.46.0)	github.com — Release v1.46.0
GitHub release (v1.49.0)	github.com — Release v1.49.0
Source code	packages/playwright/src/transform/transform.ts

`webServer.wait` Regex v1.59 — April 2026

The webServer block in playwright.config.ts previously waited for a URL to respond with a 2xx–4xx status. The new wait.stdout and wait.stderr sub-options accept regular expressions and watch the server process output instead. This is useful for dev servers that print a readiness message rather than exposing an HTTP health endpoint. Named capture groups in the regex are automatically exported as environment variables, so a varying port can be passed directly to test.use({ baseURL }).

// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
  webServer: {
    command: 'npm run start',
    wait: {
      // Named group "my_server_port" is stored in process.env.MY_SERVER_PORT
      stdout: /Listening on port (?<my_server_port>\d+)/,
    },
  },
});

// In a test file — consume the captured port.
import { test } from '@playwright/test';

test.use({
  baseURL: `http://localhost:${process.env.MY_SERVER_PORT ?? 3000}`,
});

test('homepage loads', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveTitle(/My App/);
});

Resource	Link
Web server guide	playwright.dev — Web Server
`testConfig.webServer` docs	playwright.dev — TestConfig.webServer
Release notes (v1.59)	playwright.dev/docs/release-notes#version-159
GitHub release (v1.59.0)	github.com — Release v1.59.0
Source code	packages/playwright/src/plugins/webServerPlugin.ts

Finding Rhythm in Music and Tech

2026-05-11T15:16:19Z

Last weekend, I picked up my drumsticks and played at a fundraiser gala. 🥁 Outside of my work in AI, quality engineering, speaking, and coaching, music has always been a constant in my life. I started playing at eight years old, and even now, it continues to shape how I think and work. Being back on stage reminded me that the lessons I learned through music never really left they just evolved. Discipline from years…

I am speaking at LeadDev Meetup, Amsterdam

2026-05-11T09:33:06Z

I am super excited to share that I have been invited to speak at the upcoming LeadDev Meetup in Amsterdam – being sponsored by Picnic and CircleCi. The LeadDev group is dedicated to bringing together engineering leaders, managers, and software developers to share best practices in engineering leadership, discuss strategies for promoting leadership accountability, and […]

DeepEval vs PromptFoo 2026: Which LLM Eval Framework Wins?

2026-05-11T03:45:49Z

I tested DeepEval and PromptFoo on real RAG pipelines. Here are the exact numbers—stars, downloads, metrics, and token costs—so your QA team picks the right LLM evaluation framework in 2026.

The post DeepEval vs PromptFoo 2026: Which LLM Eval Framework Wins? appeared first on Software Testing & Automation.

Practicing Playwright: Logging in by Storing and Using an Authentication Cookie in Your Automated Tests

2026-05-11T01:31:28Z

I absolutely love that LinkedIn offers a free month-long trial period of LinkedIn Learning. Butch Mayhew's Playwright Essential Training: Abstractions, Fixtures, and Complex Scenarios course has been a wonderful resource learning more about Playwright.

With this blog post, I will be walking through Butch's code on how to set up an automated test to log into an app without going through the user interface. All it needs is the login cookie.

Butch Mayhew's companion GitHub site on LinkedIn Learning's GitHub page.
Site under test: Practice Software Testing: https://practicesoftwaretesting.com
Official Playwright Documentation: https://playwright.dev/docs
Official Playwright GitHub: https://github.com/microsoft/playwright

When testing a shopping cart app such as the Practice Software Testing website, it can get tedious. You need to open a browser, go to the login page, enter a username, enter a password, hit the login button, and verify you have logged in correctly every time you want to test something in the shopping cart.

If you are testing something unrelated to logging in, why have your tests go through the UI to authenticate? Why not have your automated test run that login test once, temporarily save the login cookie once it is produced, then reuse the login cookie, importing it into other tests?

What is Playwright?

According to Microsoft Playwright's GitHub site, Playwright "is a framework for web automation and testing. It drives Chromium, Firefox, and WebKit with a single API — in your tests, in your scripts, and as a tool for AI agents".

Playwright comes in many features. From https://github.com/microsoft/playwright

	Best for	Install
Playwright Test	End-to-end testing	`npm init playwright@latest`
Playwright CLI	Coding agents (Claude Code, Copilot)	`npm i -g @playwright/cli@latest`
Playwright MCP	AI agents and LLM-driven automation	`npx @playwright/mcp@latest`
Playwright Library	Browser automation scripts	`npm i playwright`
VS Code Extension	Test authoring and debugging in VS Code	Install from Marketplace

... Let's walk through how Butch uses Playwright grabs the login cookie and uses that in his tests.

Getting an Authorization Cookie Through the Website

Before we do anything with our test automation, let's examine an authorization cookie.

Open up Chrome and go to https://practicesoftwaretesting.com/auth/login
Right click on the page and select "Inspect" to open Chrome Developer Tools.
Select the "Network" tab.
Enter a username and password of customer@practicesoftwaretesting.com / welcome01
Select the "Login" button, then view the Network tab.
On the network tab, click on the "account" entry.
Select the "Cookies" tab in Chrome Developer Tools.

Under Request Cookies, you will see something like this:

"cookies": [
    {
      "name": "cf_clearance",
      "value": "2Rv345ZtjSaGUFRwq6j2QO4Qp3EBG0T.5URv.oenfjg-1778187254-1.2.1.1-90Z63mBartyo.dUm4bwavAggVG9pgrzcYOO0zPS18qBEujmzIq3tnWwjx2b5jQypbZywJMVEIncDFSwJBaAD0ephO3cctJrLkS449MVHNO7ZSuqlAPhzVYGw8vVH4doqYc2R_5WXTY47OHVLRAgnblq7U9glSiaJUvoOKmc6I.Mo6FPl3MrhGmfMxSTVyTkcCBJn8t_C9b1margtoukM7yMuQdNPpkNItvrrZdIQT.TpGQx2sNX7cS_KS8TY2Ghyc0_QLKFZFqC_5ZgZHMad5xgzgvJEm2NZmO3RBLSPKH5bpDFbuynt7g4oIwYfIvWd4pTpTB447hpP5qMPnLOp.Q",
      "domain": ".practicesoftwaretesting.com",
      "path": "/",
      "expires": 1809723253.921407,
      "httpOnly": true,
      "secure": true,
      "sameSite": "None",
      "partitionKey": "https://practicesoftwaretesting.com",

Cross-Indexing this with the Chrome DevTools Docs on Cookies, this means:

Name: "The cookie's name", in this case "cf_clearance", is a security token issued by Cloudflare after passing a challenge. [ See Cloudflare docs ]

Value. Claude AI guessed that the Value of this cookie was this... but was not sure since this value is proprietary.

Random token: 2Rv345ZtjSaGUFRwq6j2QO4Qp3EBG0T.5URv.oenfjg
Unix timestamp: 1778187254 (Around May 4, 2026)
Version Level: 1.2.1.1, the Clearance level encoding
Signature payload: 90Z63mBartyo... HMAC-binding (Hash-Based Message Authentication Codes) to IP, user agent, and session.

Domain: "The hosts that are allowed to receive the cookie", in this case, practicesoftwaretesting.com.

Path: "The URL that must exist in the requested URL in order to send the Cookie header", in this case the root folder.

Expires / Max-Age. "The cookie's expiration date or maximum age. For session cookies this value is always Session". According to UnixTimeStamp.com "1809723253.921407" would expire on May 10, 2026 on 3:36 am UTC and 49 seconds.

What is a Unix Time Stamp? "The unix time stamp is a way to track time as a running total of seconds. This count starts at the Unix Epoch on January 1st, 1970 at UTC. Therefore, the unix time stamp is merely the number of seconds between a particular date and the Unix Epoch. It should also be pointed out (thanks to the comments from visitors to this site) that this point in time technically does not change no matter where you are located on the globe. This is very useful to computer systems for tracking and sorting dated information in dynamic and distributed applications both online and client side" - UnixTimeStamp.com

HttpOnly: "If true, this field indicates that the cookie should only be used over HTTP, and JavaScript modification is not allowed"... and it is set to true.

Secure: "If true, this field indicates that the cookie can only be sent to the server over a secure, HTTPS connection"... and it is set to true.

SameSite: "Contains Strict or Lax if the cookie is using the experimental SameSite attribute".

Partition Key: "For cookies with independent partition state, the partition key is the site of the top-level URL the browser was visiting at the start of the request to the endpoint that set the cookie.

Priority. Contains Low, Medium (default), or High if using deprecated cookie Priority attribute"... and ours is "https://practicesoftwaretesting.com"

By logging into the site, it generates this cookie on the back end, allowing you to keep accessing the site.

If you generate this cooking programmatically before running the tests, and temporally save it, you can then use it in all preceding tests in that test run.

Step One: Setup the Setup

The Playwright Configuration file has in its projects array two projects: One to set up the, er, setup, and one to set up the Chromium run.

First, Butch set up in his Playwright Configuration file, a project dependency, in a project that he called "setup".

playwright.config.ts:

projects: [
    {
      name: "setup",
      testMatch: /.*\.setup\.ts/,
    },
    {
      name: "chromium",
      dependencies: ["setup"],
      use: {
        ...devices["Desktop Chrome"],
        permissions: ["clipboard-read", "geolocation"],
      },
    },

According to Playwright.Dev, "Dependencies are a list of projects that need to run before the tests in another project run. They can be useful for configuring the global setup actions so that one project depends on this running first.

"When working with tests that have a dependency, the dependency will always run first and once all tests from this project have passed, then the other projects will run in parallel.

"Running order: Tests in the 'setup' project run. Once all tests from this project have passed, then the tests from the dependent projects will start running.

"Tests in the 'chromium' [...] projects run together. By default, these projects will run in parallel, subject to the maximum workers limit".

name: "setup": A project with the name of "setup".
testMatch: /.*setup\.ts,: This pattern matcher searches on level up for anything with "setup" in the middle of the filename, such as "auth.setup.ts" in the "tests/" directory.
name: "chromium": A project that directs the test automation when it runs to use the Desktop Chrome browser devices.
dependencies: ["setup"]: Before the Chrome browser has kicked off, we tell it we need to run the project called "setup" first.
use: ...devices["Desktop Chrome"]: Playwright has an enormous list of devices in its device registry, such as various models of the Blackberry, Galaxy, iPad, iPhone, Microsoft Lumia, Nexus, Nokia, Pixel, Moto, and Desktop Devices (Chrome, Edge, Firefox, Safari) with many different user agents, viewports, and screen width and heights.
use: permissions: Playwright's browser context, according to the official Playwright Docs, has differing sets of permissions in the permission array for each browser emulation, such as emulating the accelerometer, ambient light sensor, background sync, camera, clipboard read and write, geolocation, gyroscope, local fonts, local network access, magnetometer, microphone, midi, notifications, payment-handler, storage-access, and screen-wake-lock.
[clipboard-read]: We need to give the Chromium browser Playwright spins up permission to read the authentication cookie that is generated the first time we log in. Note, according to notes in Playwright's source code in the permissions library, there is no "clipboard-read" in WebKit's API.

Step Two: Get and Save the Authorization Cookie

Next, in auth.setup.ts, Butch has the automated test go to the site, log in, and save the cookie to a file.

tests/ auth.setup.ts:


import { test as setup, expect } from "@playwright/test";
import { LoginPage } from "../lib/pages/login/login.page";

setup("Create customer 01 auth", async ({ page, context }) => {
  const email = "customer@practicesoftwaretesting.com";
  const password = "welcome01";
  const customer01AuthFile = ".auth/customer01.json";

  const loginPage = new LoginPage(page);

  await loginPage.goto();

  await loginPage.login(email, password);

  await expect(page.getByTestId("nav-menu")).toContainText("Jane Doe");
  await context.storageState({ path: customer01AuthFile });
});

https://github.com/LinkedInLearning/playwright-essential-training-abstractions-fixtures-and-complex-scenarios-4278224/blob/98d2080e018acb194938157cb7c2fc8993ca21a5/tests/auth.setup.ts

What is this code doing?

Import the test runner playwright/test into our TypeScript file, setting up an alias for the test runner, calling it "setup" to make the test more readable, and to mark this as performing initialization tasks. To learn more about how aliases work in TypeScript see FreeCodeCamp's explanation, and read about the test method in the Playwright.dev docs.
Import the expect assertion library into the TypeScript file, so we then can use the expect method, detecting if an element is attached, checkbox is checked, element is disabled, editable, enabled, focused, not visible, visible, contains text, has certain CSS classes, DOM attributes, ids, CSS or JavaScript properties, ARIA roles, screenshots, matches text, or if pages have screenshots, titles, or URLs, failing near immediately if we don't see that condition. [ See Playwright.Dev docs about test assertions ]. We can also match if it meets any instance of a class, matches anything, of an array contains certain elements, if a number is close to equal, a string contains a substring, a string matches a certain substring. All of these expects can be negated chaining a not to it.
setup("Create customer 01 auth" is being used by Butch Mayhew instead of saying "test("Create customer 01 auth") to make it more readable.
async uses the JavaScript keyword async. FreeCodeCamp says that, "Async programming is a programming paradigm that allows you to write code that runs asynchronously. In contrast to synchronous programming, which executes code sequentially, async programming allows code to run in the background while the rest of the program continues to execute. This is particularly useful for tasks that may take a long time to complete, such as fetching data from a remote API".
page, context are passing these built-in fixtures into the test. The Page instance is created for each test, used in isolation from one another. [ See Playwright.dev docs on test fixtures. ] "Test fixtures are used to establish the environment for each test, giving the test everything it needs and nothing else. Test fixtures are isolated between tests. With fixtures, you can group tests based on their meaning, instead of their common setup". The "page" fixtures uses type Page, and "context" fixtures uses type BrowserContext in order to configure context. Fixtures can encapsulate setup and teardown, are reusable between test files.
const email, password, customer01AuthFile is declaring constant variables to store what the username, password, and where the auth cookie should be stored, to make the code more readable.
const loginPage is instantiating a new login page using the login page object Butch has defined, not covered in this blog entry. The login page Butch set up contains methods where you can goto https://practicesoftwaretesting.com/auth/login, enter the username and password in the login method using the constants we defined, expecting that we see a greeting to the user.

The real magic is the last bit of code: context.storageState({ path: customer01AuthFile });

Playwright has storage and authentication methods where you can manage cookies, localStorage and sessionStorage [ See official docs ]. Session Storage is session scoped, where data is cleared out when the browser tab eventually closes.
According to the Playwright Docs / Authentication, when you "[c]reate a new setup project in the config and declare it as a dependency for all your testing project [...]. This project will always run and authenticate before all the tests. All testing projects should use the authenticated state as storageState".

The Cookie is Temporarily Saved!

A new folder and file is created in the root folder when this command is run: .auth/customer01.json:


{
  "cookies": [
    {
      "name": "cf_clearance",
      "value": "2Rv345ZtjSaGUFRwq6j2QO4Qp3EBG0T.5URv.oenfjg-1778187254-1.2.1.1-90Z63mBartyo.dUm4bwavAggVG9pgrzcYOO0zPS18qBEujmzIq3tnWwjx2b5jQypbZywJMVEIncDFSwJBaAD0ephO3cctJrLkS449MVHNO7ZSuqlAPhzVYGw8vVH4doqYc2R_5WXTY47OHVLRAgnblq7U9glSiaJUvoOKmc6I.Mo6FPl3MrhGmfMxSTVyTkcCBJn8t_C9b1margtoukM7yMuQdNPpkNItvrrZdIQT.TpGQx2sNX7cS_KS8TY2Ghyc0_QLKFZFqC_5ZgZHMad5xgzgvJEm2NZmO3RBLSPKH5bpDFbuynt7g4oIwYfIvWd4pTpTB447hpP5qMPnLOp.Q",
      "domain": ".practicesoftwaretesting.com",
      "path": "/",
      "expires": 1809723253.921407,
      "httpOnly": true,
      "secure": true,
      "sameSite": "None",
      "partitionKey": "https://practicesoftwaretesting.com",
      "_crHasCrossSiteAncestor": false
    }
  ],
  "origins": [
    {
      "origin": "https://practicesoftwaretesting.com",
      "localStorage": [
        {
          "name": "auth-token",
          "value": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJodHRwczovL2FwaS5wcmFjdGljZXNvZnR3YXJldGVzdGluZy5jb20vdXNlcnMvbG9naW4iLCJpYXQiOjE3NzgxODcyNTUsImV4cCI6MTc3ODE4NzU1NSwibmJmIjoxNzc4MTg3MjU1LCJqdGkiOiJoeFJ1MmkwNWpDamVrNUdXIiwic3ViIjoiMDFLUjIwRFFYQUpTVEJEWkg2S0Q1R1QwRVQiLCJwcnYiOiIyM2JkNWM4OTQ5ZjYwMGFkYjM5ZTcwMWM0MDA4NzJkYjdhNTk3NmY3Iiwicm9sZSI6InVzZXIifQ.Iqv8TGSweXLUfk7UExbIem_kOoIde9apIrzFZ9cCVpo"
        }
      ]
    }
  ]
}

Step Three: Make Sure The Cookie Is Only Temporarily Saved. Do Not Upload the Cookie To GitHub!!!

It is very important that you do NOT save the .auth cookie, or any other credentials, in GitHub.

If you examine the .gitignore file Butch Mayhew has set up, you can see that along with test_results, node_modules, .tmp files, you will not be uploading anything in the .auth/ folder.

Read more about .gitignore on the Git-SCM/docs/gitignore page.

Step Four: Implement the Auth Cookie for Login

Now that the Setup function generated a new cookie, we then can then use this credential setting up the test to use storageState.

tests / homepage / home.spec.ts:

  
test.describe("Home page customer 01 auth", () => {
  test.use({ storageState: ".auth/customer01.json" });
  test.beforeEach(async ({ page }) => {
    await page.goto("/");
  });

How do we know this is successful? We verify that the sign in navigation is not visible, and instead we are greeted by the username, in this case, "Jane Doe".

tests / homepage / home.spec.ts:

  
  test("check customer 01 is signed in", async ({ page }) => {
    await expect(page.getByTestId("nav-sign-in")).not.toBeVisible();
    await expect(page.getByTestId("nav-menu")).toContainText("Jane Doe");
  });

If we don't see the sign in page, and instead we see the nav menu for "Jane Doe", then everything has passed.

So, that's how we use authentication cookies in tests!

Happy Testing!

-T.J. Maher
Software Engineer in Test

BlueSky | YouTube | LinkedIn | Articles

My review of SeleniumConf 2026 in Valencia

2026-05-11T00:00:00Z

I was already a visitor of some Selenium conferences, e.g. in Berlin, Chicago and last year's Valencia edition. For 2026, the conference yet again returned to Valencia, Spain. Here is my full review.

QA Toolbox: Powering Reliable AI Outputs

2026-05-11T00:00:00Z

Written by - Millan Kaul

QA is not disappearing in the AI era. It is becoming the layer that makes AI outputs reliable, testable and safe.

Why QA still matters

AI-native teams move fast, but speed without quality creates risk.
- Prompts can drift.
- Outputs hallucinates.
- Guardrails may get skipped.

That is where QA and quality engineering matter more than ever

In AI systems, QA is not just about finding bugs. It is about checking behavior, validating outputs and protecting users more than ever.

But the question is : Will my old toolbox work ?

Here is my attempt to answer exactly that..

The QA toolbox for AI

Start thinking of QA in AI-native products as a simple toolbox:

Prompts define the expected behavior.
Guardrails prevent unsafe or off-spec outputs.
Evaluation checks whether the model actually did the right thing.
Human-in-the-loop review catches the edge cases.

This is the new QA stack for AI products.

What about Spec-driven development, what are my tools for that ?

In spec-driven development, the spec becomes the source of truth.
QA should not treat specs as documentation only. They should be treated as testable contracts.

That means:

Define what good output looks like,
Define what must never happen, and
Test against that every time.

Sample code: PII guardrails with NeMo Guardrails

NeMo Guardrails supports PII detection and masking in input and output flows through its Private AI integration.

`config.yml`

models:
  - type: main
    engine: openai
    model: gpt-4o-mini

rails:
  input:
    flows:
      - detect pii on input
  output:
    flows:
      - detect pii on output

  config:
    privateai:
      server_endpoint: http://your-privateai-api-endpoint/process/text

    input:
      entities:
        - NAME_FAMILY
        - LOCATION_ADDRESS_STREET
        - EMAIL_ADDRESS

    output:
      entities:
        - NAME_FAMILY
        - LOCATION_ADDRESS_STREET
        - EMAIL_ADDRESS

Python

from nemoguardrails import RailsConfig, LLMRails

config = RailsConfig.from_path("./config")
rails = LLMRails(config)

messages = [
    {"role": "user", "content": "Hi, I am John Doe. My email is john.doe@example.com"}
]

response = rails.generate(messages=messages)
print(response)

Private AI can detect or mask PII in input, output, and retrieval flows, and the docs note that detection blocks text from passing through if PII is found.

What to test

For AI products, QA should always check:

Does the prompt produce the expected structure?
Does the model stay within policy?
Does the output leak PII?
Does the answer remain useful under edge cases?
Does human review catch what automation misses?

Those are few of the ways for QAs to stays relevant in AI-native teams.

Closing thought

AI does not replace QA. It makes QA more important. [May 2026]

My personal take: The teams that win will be the ones that test prompts, guardrails, and evaluation as seriously as code.

I’m (re-)starting a newsletter

2026-05-11T00:00:00Z

Just a quick update to let those of you who bookmarked this blog or who have subscribed to my RSS feed know that I have (re-)started a newsletter.

As you might know (or not), while I’ve been pretty active on LinkedIn over the years, I do have a love-hate (or rather an appreciate-hate) relationship with that platform.

Lately, I’ve been noticing that the pendulum is swinging in the ‘hate’ direction more often, mainly because the ever-changing algorithm used by LinkedIn makes it incredibly hard to predict if people are even going to see what I write.

I’d rather publish my thoughts, ideas and other ramblings via a platform that I do control, and that platform will be a newsletter.

I’ve had a newsletter in the past, but that only lived for about three months. This time, I intend to keep writing and publishing a new issue every week. The first edition goes out a few hours after I’m writing this, and a new issue will be sent to subscribers every Monday morning around 11 AM CET.

But what about the blog?

I’ll still publish to the blog, too, but that will be on a much less regular basis. Just like it has been for a while, really. The idea is to post the more ‘technical’ posts, that is, the ones including code, directly to my blog, whereas the ‘text-and-images-only’ posts go through my blog post first.

My priority is with the newsletter, though.

That’s easy, just go to the subscription page, leave your email address, click the button on the confirmation email and you’re in.

I promise I won’t use the newsletter or your email to spam or sell to you. Ever.

You Need AI That Reduces Maintenance Costs

2026-05-10T22:02:01Z

I’ll get straight to the point: your AI coding agent, the one you use to write code, needs to reduce your maintenance costs. Not by a little bit, either. You write code twice as quick now? Better hope you’ve halved your maintenance costs. Three times as productive? One third the maintenance costs. Otherwise, you’re screwed. You’re trading a temporary speed boost for permanent indenture.

Oh, you want to know why? Sure. Let’s go for a drive. On a dark desert highway...

Productivity is Determined by Maintenance Costs

Every line of code you write has to be maintained: bug fixes, cleanup, dependency upgrades, and so forth. I’m not talking about new features or enhancements. Just maintenance. For every month you spend writing code, you’ll spend some amount of time in the following year maintaining that code, and some in each year after that, forever, as long as that code exists.

Let’s say you asked a crowd of, say, 50 developers what those maintenance costs were. Using a technique called Wisdom of the Crowd, you could get a reasonably accurate response.¹

¹You’re welcome to conduct your own wisdom-of-the-crowd survey! But it turns out that the specific numbers don’t matter for the overall point I’m making here.

Your crowd might tell you that, for each month you spend writing code, you’ll spend...

10 days on maintenance in the first year; and
5 days on maintenance each year after that.

If you were a particularly obsessive individual, you could spend hours making a spreadsheet modeling how those estimates affect productivity over time. A spreadsheet like this.

The first month of a new project is glorious. You spend all your time building fancy new features.

The next month is slightly less glorious. A fraction of your time—not much, but a smidge—goes to fixing bugs and cleaning up design mistakes from the first month. In the third month, a smidge more. And the fourth month, the fifth, the sixth...

Eventually, it’s not glorious at all. According to our crowd’s maintenance estimates, you’ll spend more than half your time on maintenance after 2½ years. After ten years, you can hardly do anything else.

Halving the crowd’s maintenance estimates gives you three more years before you hit the 50% mark. Doubling them sees you below 50% in less than a year.

The lesson is clear. If you want a productive team, you have to focus on their maintenance costs.

All Models Are Wrong

Do these numbers ring true to you? They do to me. In my career as a consultant, I specialized in late-stage startups, and they all had the exact problem shown in the graph above. About 5-9 years in, they’d notice their teams were no longer getting shit done, and then they’d call me.

Their teams weren’t quite as bad as the graph shows. Maybe their maintenance costs were lower. Or maybe... and this feels more likely to me... their maintenance costs were exactly that bad, and they papered over the problem instead. Maybe they:

Decided not to fix every bug, or upgrade every dependency
Added people when the team got slow... and then kept adding more, because it was never enough
Scrapped it all and started over with a rewrite

There’s room to debate the precise maintenance numbers, but overall, the model feels right. If you’ve been around the block, you know this graph is true. You’ve seen how productivity melts away over time. You have the scars.

What Does This Have to Do With AI?

Only everything.

Let’s say your team just started using Rock Lobster, the latest and greatest agentic coding framework, and it Doubles!! your code output! Woohoo! The code’s a bit harder to understand, though, and your team is drowning in pull requests, and you maybe kinda sorta teensy weensy don’t actually read the code before smashing the approve button. Like, at all. I mean, you skimmed it, during boring meetings, sometimes, and that’s gotta be good enough, right? LGTM, let’s get this shit done!

So now you’re producing two months of work in a month, and let’s say you’ve doubled how much each “month” of output costs to maintain. Next month’s maintenance costs quadruple.

Oh.

About five months after you start using Rock Lobster, your productivity is back down to where you started, and a few months after that, it’s worse than it would have been had you never touched Rock Lobster in the first place.

I’m not saying your AI doubles maintenance costs. Or productivity. This is an extreme example. But even if your AI produces code that’s just as easy to maintain as your human-written code, the productivity gains don’t last.

You Can Check Out Any Time You Like²

²But you can never leave.

Agents are expensive, and they’re only getting more so. Once your agent’s juice is no longer worth the squeeze, you might decide to save your pennies and go back to coding the old way. Like a caveman. With your fingers.

Ha! Joke’s on you! When you stop using the agent, all the productivity benefit goes away... but the added maintenance costs don’t! As long as that code’s still around, you’re stuck with lower productivity than if you had never touched the agent at all.

The Passage Back

The math only works if the LLM decreases your maintenance costs, and by exactly the inverse of the rate it adds code. If you double your output and your cost of maintaining that output, two times two means you’ve quadrupled your maintenance costs. If you double your output and hold your maintenance costs steady, two times one means you’ve still doubled your maintenance costs.

Instead, you have to invert your productivity. If you’re producing twice as much code, you need code that costs half as much to maintain. Three times as much code, one third the maintenance.

This is the secret to success. All the benefits, none of the lock-in.

Can We Kill the Beast?

I dunno. All my reading of the finest news sources says that coding agents increase maintenance costs. Some people do say they help them understand large systems better. But big decreases in costs, of the size we need to see? No. Just the opposite.

That’s a problem. The model isn’t a perfect representation of reality, but the overall message is right. You need AI that reduces your maintenance costs, and in proportion to the speed boost you get from new code. Without it, you’re screwed. You’re trading a temporary speed boost for permanent indenture.

So, yeah, go ahead, chase improvements to your coding speed. But spend just as much time chasing improvements to your maintenance costs. Or you, too, will be trapped in Hotel California.

Such a lovely place.

Such a lovely face.

As much as it might seem like it, this isn’t meant to be an anti-AI rant. There’s other levers to pull, such as AI that makes maintenance itself more productive, even if it doesn’t make the code more maintainable. I encourage you to copy the spreadsheet and play with all the levers in the model. See what happens when you change the assumptions to match your real-world situation.

k6 Performance Testing From Zero to Production: A JavaScript Developer’s Complete Guide

2026-05-10T22:00:00Z

Modern load testing with k6: JavaScript-based, CI/CD native, Grafana ready. Complete guide from first test to stress testing and GitHub Actions integration.

The post k6 Performance Testing From Zero to Production: A JavaScript Developer’s Complete Guide appeared first on Software Testing & Automation.

blogs

Docker Compose Testing Setup: Playwright, Selenium Grid, and API

constitution.md in Spec-Driven Development

constitution.md : The Rules Behind Spec-Driven Development

Why it matters

How to use it

What to put in it

My take

The 2026 QA Engineer Career Roadmap: Month-by-Month From Manual Testing to Senior SDET

The IQL Manifesto: Seven Commitments for AI Quality Leadership

Where This Started

Why a Manifesto

The IQL Manifesto

01 – Challenge the reasoning, not just the result

02 – Governance doesn’t check quality at the end. It builds the foundation at the start.

03 – Test the AI, not just the software containing it.

04 – Faster automation isn’t the answer. Smarter AI use is.

05 – Collaborative Quality Ownership isn’t optional.

06 – Human judgement moves to where it still matters most.

07 – Confidence is the differentiator.

What I Want You to Do With This

The Automation Island Trap: Why QAs Who Only Automate Stop Growing

The Illusion of Value

QA Is a Thinking Role, Not a Toolset

Why This Limits Your Growth

The Real Role of Automation

What High-Impact QAs Do Differently

Breaking Out of the Automation Island

How to Get Out of the Automation Island (Step by Step)

Conclusion

A tester’s story about assumptions, resilience, and learning from real‑life bugs

You know that moment when everything looks right… until it’s completely wrong?

A real‑life bug uncovered in production

What this train taught me as a tester

Bonus reflections from the platform called life

Resilience is part of quality

Final thought

Test Automation Made You Faster. It Also Made You Blind.

Manual Testers and AI Agents: Why Service Company QA Jobs Are Disappearing in 2026

Spec-Driven Development: QA’s North Star in AI-Native Teams

QA’s Pivotal Role

QA-Centric Workflow

Sample: QA Spec → PII-Safe API

FastAPI + QA Guardrail

QA Wins

QA Checklist

QA Metrics That Actually Matter: 12 KPIs to Replace Vanity Metrics

Pramod Dutta discusses 7 Playwright features senior SDETs use daily

page.addLocatorHandler() v1.42 — March 2024

Cookie banner (fire-and-forget)

Self-removing handler with access to the triggering locator

Fail fast on an application error overlay

browser.bind() v1.59 — April 2026

Bind via named pipe (local only)

Bind via WebSocket (shareable over a network)

URLSearchParams in request.get() v1.46 — August 2024

expect.toPass() intervals config — v1.44 May 2024

Polling an endpoint until ready

Global configuration in playwright.config.ts

page.requestGC() v1.46 — August 2024

TypeScript Config: --tsconfig flag & testConfig.tsconfig

CLI flag

Config file property

Example tests/tsconfig.json

webServer.wait Regex v1.59 — April 2026

Further Reading

Finding Rhythm in Music and Tech

I am speaking at LeadDev Meetup, Amsterdam

DeepEval vs PromptFoo 2026: Which LLM Eval Framework Wins?

Practicing Playwright: Logging in by Storing and Using an Authentication Cookie in Your Automated Tests

What is Playwright?

Getting an Authorization Cookie Through the Website

Step One: Setup the Setup

Step Two: Get and Save the Authorization Cookie

The Cookie is Temporarily Saved!

Step Three: Make Sure The Cookie Is Only Temporarily Saved. Do Not Upload the Cookie To GitHub!!!

Step Four: Implement the Auth Cookie for Login

My review of SeleniumConf 2026 in Valencia

QA Toolbox: Powering Reliable AI Outputs

Why QA still matters

`constitution.md` : The Rules Behind Spec-Driven Development

`page.addLocatorHandler()` v1.42 — March 2024

`browser.bind()` v1.59 — April 2026

`URLSearchParams` in `request.get()` v1.46 — August 2024

`expect.toPass()` intervals config — v1.44 May 2024

Global configuration in `playwright.config.ts`

`page.requestGC()` v1.46 — August 2024

TypeScript Config: `--tsconfig` flag & `testConfig.tsconfig`

Example `tests/tsconfig.json`

`webServer.wait` Regex v1.59 — April 2026

`config.yml`

You Can Check Out Any Time You Like²