blogs

Ideation and oracles - where AI does better than testers

2026-10-19T16:59:50Z

As I was preparing for my PNSQC keynote 'Better together: Humans, AI and the next quality benchmark', I went back on my social media timelines on the public notes I had taken, and two stood out in particular.

The first one was note on research item in 2018 that was making rounds, that I had hard time digesting then.

The second one was few years later, when AI was new and I first time saw it doing better than my colleagues, in ideation for testing.

20.11.2023 "In this particular case, chatGPT did better than colleagues. People are so afraid to say what they think and make decisions."

I have been slowly coming to terms with two things:

Testers are not as good at testing as they think they are. Very few testers are really excellent at finding meaningful (or sufficient) results and targeting the testing they do for risk and architecture.
AI does much better now at testing than we would like to give it credit for.

However, for these two things, testing might not fit your common plan - specify - execute -frame, but would benefit from a different model to how the work is distributed. The frame we commonly recognize as exploratory testing.

Better, you say?

Today I wanted to write about two very specific ways of better I have been observing recently:

AI-enabled identification of properties, and use of properties effectively with extended automation capabilities of AI
AI-enabled ideation of what to consider testing, and use of AI for multi-perspective classification to choose most relevant ideas

Properties

To understand the fundamental shift in testing that AI is now enabling, we need to go back to 2012 and research by Cem Kaner. He had then been teaching BBST courses for a large number of students of testing, and observed that students at large numbers failed at a particular, very relevant exercise: identifying partial oracles that could be used for purposes of automation. The concept he was trying to teach would become essential for assessing systems in scale.

Kaner described the oracle problem as "one of the defining problems of software testing: one of the reasons that skilled testing is a complex cognitive activity rather than a routine activity". He pointed out that a popular notion of heuristic oracles of consistencies don't work: "consistencies don’t guide testers (not me, not the testers I know, not the students I teach) toward oracle ideas that are specific enough to be programmed and used by an automaton.

This is where we left off with BBST. While the course materials were redesigned to teach partial programmatic oracles, testers at large were particularly bad with these. Turns out generative AI is better at this.

The years brought me awareness of developer community solving this problem of designing programmatic oracles. They just used an entirely different word for it: properties. Properties were things that would be true to evaluate against for all or a signicant group of tests, so that you wouldn't have to hand-craft each example and assert - the approach the tester automation community at large adopted. The property-based testing tools, however, stayed in sidelines, and evolved further particularly in the Haskell community as Quickcheck. It has seen mainstream only in recent years, first by applying it in the much larger Python community as Hypothesis, then on APIs as Schemathesis and now language-agnostic for webUIs with Bombadil. The relevant piece of my puzzle however came with Antithesis property research skill, that does what students at Cem Kaner's course never learned to do: identify properties.

The Bombadil tool includes four default properties

No HTTP error codes
No uncaught exceptions
No unhandled promise rejections
No console errors

But the Antithesis research skill is capable of identifying domain-specific properties: statements that should hold true when we use them in our tests, beyond "it does not fail visibly or silently", as the defaults state.

The amount of relevant problems I have been able to identify that I was oblivious to since I caught up with this blows my mind. AI changed testing, for me.

Ideation

The other piece I wanted to offer as essentially different is ideation. At PNSQC last week, I participated a workshop facilitated by Ben Simo on "Collaborative Ideation". While generating ideas on an imagined chocolate wrapping machine might not be my favorite passtime, the steps through the exercise made it painfully obvious that humans are not a match for generating ideas in excess. Humans, however, do really well on asking critical questions about the generated ideas, and modeling dimensions of relevance. In the end, when we were asked to prioritize our ideas for the same target of testing for 10 essentially different project contexts, the overgeneration, classification and filtering approach that AI made straightforward really made sense.

It was not that you would ask AI to give you a perfect list. But you would ask for a list that you would classify towards your critical ideas of better towards perfect, allowing for learning to change the priorities.

You know what they say:

Do your testing so that at any time you run out of time, you have done the best testing you possibly can by that time.

The amount of perspectives to classify and consider that we could work through collaboratively in four hour workshop really solidified the experience: AI changed testing, for me.

Hadestown Promises

2026-10-19T16:18:30Z

Most readers probably don’t know that I’m a pretty big fan of musical theater. I see 4-5 shows a year (and have for the last ~25 years), and have seen a lot of great shows on Broadway as well as local productions.

The Hadestown soundtrack made it onto my regular playing rotation a few years ago, and I had a chance to see it on Broadway last year (where it was even better). I listened to the soundtrack again on a walk this weekend and had some thoughts.

Road to Hell

Hadestown is the story of Eurydice and Orpheus. Early in the show, Orpheus asks Eurydice to marry him, and she responds by asking some practical questions. She wants to know what they’ll use for rings, where they’ll eat, and where they’ll sleep. Orpheus answers in poetry. The river will make the wedding bands, the trees will set the table, and the birds will make the bed.

I’ve always heard those words as symbolic, since he’s a poet describing the world the way he sees it, or even the way he wants it to be.

But later in the story, Orpheus admits he can’t deliver any of it. They were real promises, and under all the poetry, what he was saying was “it’ll all work out.” He was saying it to someone who’d spent most of her life watching things not work out.

I’ve watched a lot of leaders do the same thing.

Many years ago, I worked with a director named Marcus. His org had just been through a round of layoffs, and in the all-hands afterward, people asked whether their projects would continue, who they’d report to in the new org, and whether more cuts were coming. Marcus answered every question with some version of the same thing: we’ll come out of this stronger, and it’s all going to work out. He believed it, too. Two months later there was a second round of cuts, and from then on, nobody believed much of anything Marcus said.

Wedding Song

Eurydice asked concrete questions, and concrete questions deserve concrete answers, even when the best answer is “I don’t know yet.” Vision matters. Orpheus’s song is the reason Eurydice stays in the first place, and teams need leaders who can describe a future worth working toward. The trouble starts when vision shows up as the answer to a practical question. When someone asks where they’ll sleep, a story about birds sounds like either a dodge or a promise they can’t keep.

Leaders reach for poetry most often when the true answer is uncomfortable. “I don’t know whether your project survives the next planning cycle” is hard to say, so it turns into corp-speak like, “we’re investing in the things that matter.” It feels like a safer thing to say in the moment, even though telling the uncomfortable truth is almost always the right thing to do.

Any Way the Wind Blows

Eurydice’s history is important to consider. “It’ll work out” sounds different to someone who’s had a life where things usually haven’t worked out. Orpheus is an optimist who’s never really been cold, while Eurydice has spent years trying to stay warm.

Your team has a history too. Some of them have lived through layoffs, reorgs that went nowhere, or a previous manager who also said reassuring things right before delivering bad news. Our optimism costs us nothing, but trusting it can cost them quite a bit. And when the reassurance turns out to be wrong, people will stop asking.

I wrote about what a quiet team is telling you earlier this year, and vague promises are one of the fastest ways I know to get a team to stop talking.

Promises

Orpheus does learn. By the time he leads Eurydice out of the underworld, he’s stopped promising things that belong to the river and the trees. He tells her that he can’t give her any of that, and offers something that’s entirely in his power instead: just to stay beside her.

As a leader, your promises should be smaller and more specific, and within your control. You can’t promise there won’t be another round of cuts, but you can promise that people will hear it from you before the all-hands. You can’t promise a project survives planning, but you can promise to tell the team the day you know, and to explain why.

Of course, Hades’ condition for letting them leave is that they can’t walk side by side, so Orpheus has to lead while Eurydice follows behind him. In the end, Orpheus breaks his promise, and that’s a story about trust and doubt that’s probably worth a whole other post.

Why We Build the Wall

Almost everything is a system, and this story isn’t any different. The world is cold because Hades and Persephone’s relationship is broken, and the seasons have gone wrong along with it. Eurydice doesn’t go to Hadestown because she’s weak. She goes because she’s hungry, and Hades is offering the only meal around. Even Hades is stuck in the machine he built, and it takes Orpheus’s song to remind him who he used to be. When Orpheus finally turns around, it’s because Hades designed a test where doubt was almost guaranteed to win.

The show doesn’t let anyone completely off the hook, though. Eurydice owns her choice, and Orpheus is the one who turns. The system shapes the choices people make, but they still make them.

Marcus wasn’t a liar either. He was working inside a system that rewarded confident reassurance and treated “I don’t know” as weakness, so he gave the system what it wanted. If you catch yourself answering practical questions with poetry (or corp-speak), it’s worth asking what’s pushing you there, and whether you can change that instead of just coloring inside the lines.

Road to Hell (Reprise)

Before you reassure someone, sort what you’re about to say into three buckets: what you know, what you don’t know, and what you’re hoping for. All three are fine to share, as long as you’re clear about which is which. “I’m hopeful the project survives” is honest, while “the project will be fine” is a promise you may not be able to keep.

Answer the question that was asked. If someone asks where they’ll sleep, tell them where they’ll sleep, or tell them you don’t know yet and when you expect to. Save the vision for when they ask about the future.

Keep a rough count of the promises you make and the ones you keep. When that ratio slips, the people around you will notice long before you do. The fix is usually to make fewer promises rather than to try harder to keep big ones.

If you’ve said “it’ll work out” before and it didn’t, say so. Owning the last broken promise buys more credibility than any amount of new optimism.

Software Testing & QA Jobs – October 19, 2026 | Daily Job Openings

2026-10-19T14:31:20Z

Software Testing & QA Jobs – October 19, 2026 | Daily Job Openings

Position - Automation Tester

Company - DWF

Experience - 4–7 years

Location - Pune, Maharashtra