The following article originally appeared on Addy Osmani’s blog and is being republished here with the author’s permission.

A software factory harnesses loops at scale. You can run the loop with humans in it (light factory), trading judgment and concentration against speed and breakage. Or you can ignore the humans (dark factory) and let those agents scope, build, and ship code without anyone reading the details. If people stop reading, though, they’ll stop understanding your software. Your hardest job now is knowing which checks to build and how much autonomy to delegate.

This idea of the software factory is a term that dates back to Bob Bemer’s paper, “The economics of program production,” given in 1968. For half a century, many have dreamed of a world in which software is a repeatable and instrumentable production process (analogous to stamping out car parts in a factory) rather than the isolated craft of individuals. Historically, this dream has generally (although not universally) fallen flat, in part because of the difficulty of stamping out ideas.

But in the last two years, things have changed dramatically enough that now it makes sense to take a fresh look at the old dream. And since some subtleties can easily be glossed over, it’s worthwhile to be somewhat precise about exactly what’s really new and different, and what may be recurring traps, dressed up as new opportunities.

Dex Horthy, co-founder of HumanLayer recently gave a great talk at the AI Engineer World’s Fair called “Harness Engineering is not Enough: Why Software Factories Fail.” worth checking out on this topic.

The loop is the atom. The factory is the loop at scale.

Structure is everything, and it all starts with small units. The whole stack is really three concepts layered on top of each other: the loop, the harness, and the factory.

A loop is one agent doing a single job on repeat: gather context, take an action, check the result, and go again until some condition is met. It is the smallest unit of agentic work, and everything above it is just loops stacked on loops.

The point of loop engineering is that you stop prompting the agent turn by turn and instead design the small system that prompts it for you.

A harness is the walls around a loop: the sandbox it runs in, the tools it can reach, the memory that survives between runs, and the gates that decide what “done” means. The loop is the behavior; the harness is the environment that behavior runs inside.

Hand a raw model no harness and it will happily spin forever. The harness is everything around the model that makes it useful and safe to run.

A software factory is many harnessed loops running at once, fed by a queue of work and drained through a review gate into production, with humans owning the whole thing from above. It isn’t a bigger agent; it’s an org chart made of loops.

The final paradigm shift is moving from writing code to building and running the factory that writes it. The unit of work shifts up a level, to the loop, the harness, and the flow between them, rather than the individual code diff.

The factory, drawn

The central slide Dex spent most time on was brilliant because it’s a clarifying wiring diagram that visualizes what otherwise is an obvious loop. Here’s my take on it:

Intent flows from the vision of engineering leadership and directly from engineers into a queue of work. Signals driven by incidents and user requests drive the same queue. The harness picks an item from the queue and builds a change for it. Beyond the harness, automated checks make changes safe enough to let into production. These automated checks run at once without any conscious involvement from engineers, thanks to CI, tests, static analysis, and scanning of all kinds. The review gate is the only decision point here. After approval, changes are deployed and monitored in production, with monitoring data feeding back into the signals that kicked the loop into motion to begin with.

By and large, every box in this diagram is almost zero cost: generation, tests, scanning. They all run at scale for negligible cost. There’s only one expensive box that proves stubbornly resistant to scaling, and that’s the review gate. That shiny amber box is “judgment,” and where the crux of the argument about whether we can make development faster and more frequent resides.

Why we call it “dark”

A dark factory runs with the lights physically off because the only things on the floor are machines, which don’t need light to see. A dark software factory operates similarly, as code ships that no human has read and is verified only by other machines.

The image is borrowed from manufacturing. Its origins are physical rather than digital, rooted in facilities where the lights are turned off and the work is carried out by robots. FANUC in Japan has been running lights-out factories of this sort since 2001. Xiaomi, in 2024, opened a heavily automated dark factory of its own. What these have in common is a product assembled and shipped without a single human having read any of it. The “dark” comes in when that act of reading is removed from the process.

I’m not borrowing the concept for its vibe or as an insult. For all its creepy buzz, “dark” here is a simple physical claim: the original factory floor, but without light. In software, the floor is the diff. Whoever wrote the diff, whoever reviewed it, whoever shipped it, those humans are gone, and what remains is a diff verified only by the machines that built it.

This is a surprisingly easy thing to do, at least at first. It’s easy because that missing review step gets in the way of everything. Its absence makes your perception of your team’s vertical throughput seem suddenly and radically higher. It feels as if you’ve broken the sound barrier. For all its apparent ease, it’s harder than it seems to survive those dark workflows, with all their buried costs.

Harness engineering is not enough

The harness of orchestration, sandboxed prototyping, and tool calling as models interact with the world and each other will become increasingly powerful and effective. However, there’s an inherent in-model failure in trying to keep up with codebase quality over the long game and through additive changes, and I think there’s good reason to believe that models alone will ultimately lose that battle against comprehension debt.

Comprehension debt is the widening gap between how much code exists and how much any human still understands. A dark factory doesn’t pay it down; it takes it on as fast as it can, with the tests green the whole way.

This is an important distinction because models do well at some tasks. But for anything that isn’t an immediate change to a small part of a codebase, especially in a complex brownfield system, model-only automated coding faces an insurmountable obstacle. Weekend toys and side projects are alike in that a few months of development cycles is usually enough to get things in working order, or at least close enough. But an enterprise system that has been under development for a decade or more is a different beast; it has to be maintained, in a professional environment at a professional pace. Three to six months into a project, you’re already drowning in unread code. That kind of environment, and especially the constraints enforced by production code, would make even a powerful agent do poorly, all of it in contrast to the vibe-coding enjoyed by developers working on weekend toys.

Dex reports from experience that this is a major failure, so much so that it required painstaking manual debugging to pinpoint. This came from running a fully automated code factory for about four months, during which no human looked at the code that was written. Underlying the experience is a tradeoff between two conflicting metrics. One is maximizing token utilization, the number we currently treat as progress. The other, which it quietly minimizes, is the amount of the system any human participant still understands at any moment.

Where the dark factory truly shines is in its ability to burn through pristine code while the tests stay green. The ultimate reckoning, when it comes, will not be a dramatic “it all goes sideways” moment. It will be quiet and late.

The bottleneck was never generation

The fundamental constraint in a software factory isn’t how much code we can churn out, it’s how quickly we can verify it.

Back pressure is the rule that you can only hand a loop as much autonomy as you can cheaply and reliably verify, and not one inch more. Verification, not generation, is the real constraint on a factory.

Because unbounded generation capacity is in perpetual tension with the finite, non-scaling resource of human attention, the core problem is the gap between cheap generation and bounded review. Look at the funnel: As long as the neck representing verification doesn’t widen, it’s going to back up. As Dex points out, volume alone isn’t the problem: What we’re really suffering from is a surplus of bad PRs. When you’ve got high volume without trustworthy gates, manufactured defects are unavoidable. This is just back pressure again: Autonomy can’t expand beyond what can be cheaply and reliably verified.

The second-order problem is why improving the model shouldn’t automatically close the gap between what it can generate and what can be verified. Training on well-architected systems is an arguably more difficult proposition than passing simple tests: remember, the cost functions measuring architectural excellence aren’t measured in seconds or even minutes, but in months and years. Tidy gradients are functionally impossible to compute, so a system expecting crisp, instant evaluation of complex design decisions isn’t going to be trained on good examples.

Turning the lights back on

A lit factory is the same pipeline with the lights left on where judgment lives. Agents still do most of the building, but a human reads what comes out before it ships, keeping the lights on wherever a wrong call is expensive.

The lit version doesn’t tack review onto the end but moves the point of human judgment upstream, to the product, the design, and the architecture before an agent starts a loop.

One great thing about that upfront hour is that it leads to fewer implementation hours. It turns a long, frustrating code review into a quick read of a two-hundred-line plan. You get to review a decision before it’s built, so later you aren’t chasing through two thousand lines of generated code to find out what the decision even was. Some decisions are expensive and long-lived enough that you’d want a person in on them early, before the cost compounds. Of course, there are still times you look at diffs, even when you’ve spent time up front.

You might be thinking that all sounds unglamorous. You’re right. The safety net is made up of perfectly ordinary architectural practices we’ve always known about and mostly ignored: good types and method signatures so that mistakes are caught by the compiler instead of in production; test seams where we can pin behavior and make change observable; laying out the code so the next reader, human or model, knows where to find the thing they care about; keeping call stacks short and legible; keeping component boundaries well defined so a change doesn’t have a huge blast radius; and dependency injection so we can swap out one piece for another. None of it is new. We’ve always said we care about good architecture. But now that we’re using automated coding agents, that architecture is finally doing a second job as a cheap and hard-to-fake safety net against the mistakes the agent will make.

That safety net has to live outside the model because the model won’t supply it. The coding agents that feel most capable, Claude Code and Codex among them, are reinforcement-trained against their own harness and tools: fluent with all the tools and idioms of the trade, but not with things like long-term maintainability. The deliberate architecture we’ve always talked about is the tool that catches that debt, and the investment we make in it is us buying back our autonomy. Put that together with safe infrastructure, and there are some tight, low-risk loops you can run unattended. Horthy described one in a recent post: A nightly GitHub Actions cron that fixes exactly one anti-pattern, a lint violation or a needlessly optional prop, commits, and opens one small pull request, all on its own, so the team wakes up to a slightly better codebase and a diff short enough to read. But for loops with high enough stakes, you don’t want to risk waking up to a broken auth system, billing engine, or public API contract. Keep the lights on there, and trust that a person with judgment and a real working knowledge of the system will catch the mistake.

What earns a loop the dark

This rule applies whether you call it back pressure, verification, or the light switch.

A loop can earn itself fully automated status only if the check is cheap, runs at high frequency, and relies on something that can’t be easily faked out. Green-or-red oracles, type gates, property tests, and a review agent coupled with a real rubric all fit. You also need the oracle to answer immediately and not drift over time. When done can be proven not just by you but by a machine, you’ve reached automation.

Short loops are easier to verify than long ones. Dex’s rule of thumb: An agent holds up for three to ten steps, then starts losing the thread past twenty. The reason is context accumulation. The more the agent drags along, the more likely it is to wander off. When a loop is short, verifying it is cheap. Sprawling loops hide mistakes in the corners, which is another way of saying they never earned lights-out status.

Keeping the lights on is the opposite case. A loop needs to be reviewed if a wrong answer is expensive and only a person can catch it. Subtle production bugs that can’t be caught by tests, large blast radii, and a decision that’s going to shape the work of a year or more all qualify. In those cases, human judgment does not leave the software; your attention is the costly, essential part.

The danger is forgetting to flip each switch and just setting all of them to the same mode. All dark, and you’re stuck tearing everything down four months later. All lit, and no one can get reviews done in time and you’re stuck in a gigantic bottleneck. The hard, skilled job is deciding where to put each switch.

Loops, graphs, or state machines?

When you hand an agent a task, you’ll likely build a graph around it, whether you call that graph a finite state machine or a set of conditionally linked service calls. It’s a framing where the software isn’t just following some abstract rules but a structured workflow: Every node is an explicit step, and every edge between nodes is an explicit condition. That sounds like a lot of structure, but most of it’s already there in any software, since any code can be expressed as a control-flow graph. So the only real novelty is that an agent insisting on autonomy is really just walking around a particular graph, and its freedom is constrained to the inside of a node. And here’s the part people forget, which Dex wrote down a year ago: software was always going to have that structure. There’s a reason we used to draw programs as flow charts. The genuinely new move was trying to throw the diagram away, leaning on a loop where the model picks the path tool call by tool call, until it declares itself done. That felt like liberation, right up until it met a ten-year-old codebase, and the discipline everyone is now rediscovering, owning your control flow, is really just walking the graph back around the loop. So the question of whether we should shift from loops back to graphs is almost an admission that we needed the flowchart all along.

Here’s what it looks like in practice. Take a bug to fix. As a pure loop, you sit down and think: figure out what’s wrong, change some code, run the tests, see what happens, and if that round doesn’t kill the run, loop back and start again. The whole journey is decided as you go, which problem you chase, the exact code you change, which tests you run and in what order, whether you run tests at all, and whether you try again or declare victory. As a graph, the first thing you do is map out what should happen. Reproduce the bug or go ask for more information, find the cause, try a fix, run the tests, and let a failing run route back to the fix while a passing one goes on to review, where only an approval reaches done. The agent is still clever inside each box; it just can’t wander off the paths you sanctioned. Santi laid this out with a diagram that makes the difference obvious.

The real appeal of that graph, of course, is that it’s back pressure drawn as a diagram. You give up some of the agent’s freedom and get mandatory checks and legible failure points in return, so when a run dies you can point at the node that killed it. It’s the same instinct behind Dex’s blunt line that most so-called agents aren’t very agentic at all, “mostly deterministic code, with LLM steps sprinkled in at just the right points.” And this isn’t just an artifact of how people happen to be building things right now: you can see the pattern in LangGraph and LlamaIndex Workflows, in Jerry Liu’s hybrid workflow-graph-over-agents with an outer loop that grows parts of the graph as it runs, and in David Khourshid’s reminder that this is really just state machines and the actor model turning up in new clothes.

One clarification, because the term is badly overloaded: when I keep calling this a graph, I don’t mean a knowledge graph. I mean a predefined directed graph of how the work should flow, conditional edges and all, giving the loop a shape you can actually trust.

Where the human actually goes

Notice that the person never left the factory. They moved.

I think engineers need to increasingly own the outer loop. The agents can investigate a bug, write up the diagnosis, implement a fix, run the tests, and write up a report. That’s the execution of the inner loop, and they can do it as efficiently as anyone. But that was never the job. The bits you own are what I’d call the outer loop: Decide whether it’s the right way to address the problem, verify that the diagnosis and implementation are sound, approve the change, and carry the consequences of being wrong. The boundary between the two loops is evidence, the diffs, the tests, the logs, and a brief explanation that connects them. Types, seams, and rubrics make it possible to oversee all this without doing a lot of work for every change.

I think it’s useful to put it this way: you’re not down on the line writing changes any more; you’re up at the end of the production line designing it and guarding the gate. There’s a lot you can do to make the model better and the harness more capable, but I’ve observed that identifying problems that are expensive in the long term is not typically something you can automate away. The core thing that’s still the job is to exercise human judgment better than any flow of paper and computing power.

Robots are fine operating in the dark, but humans need to see what they’re doing. If everything on the factory floor is dark, and you can’t see anything, and you can’t even find the light switch, that’s where the danger is.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

OpenAI expanded into software control, scientific reasoning, and financial services this week, and investors committed billions more to AI companies across the stack, while AI researchers went public with warnings. This Week in AI host Christina Stathopoulos looked at what those developments mean for an industry already wrestling with questions about safety and control.

Models are moving from answering questions to doing work

OpenAI’s latest announcements showed how much more work companies now expect models to handle. GPT‑6 Astra can navigate software interfaces, complete multistep workflows, and apply advanced reasoning to scientific and mathematical problems, and ChatGPT for financial services was developed with input from Morgan Stanley and Evercore to support research, financial modeling, and creating client materials.

OpenAI also shared a solution to the previously unsolved Navier–Stokes Millennium Prize Problem. A coordinated system of 10,000 AI agents worked for 88 hours on the proof, followed by another 17 hours of model-based verification by Astra. However, the company’s claim drew scrutiny after outside researchers questioned whether OpenAI might have had access to related work, an allegation OpenAI denies. While impressive, scientific breakthroughs like this also raise important questions about how well we understand these systems and the role humans should continue to play in scientific discovery. (Hugo Bowne-Anderson got into this in a recent article on Radar.)

Investors are placing bets across the AI stack

Money continues to flow to AI companies, but investors are backing infrastructure, platforms, and specialized applications rather than converging on a single layer of the stack. French company Mistral has raised €3 billion with plans to spend on compute infrastructure and open weight models. Legal AI company Harvey, inference chip startup Positron, and enterprise AI company Wonderful also raised large rounds, while NVIDIA announced its acquisition of Hugging Face for nearly $13 billion.

Christina cited figures showing global AI funding rising from $56 billion in the fourth quarter of 2025 to $242 billion in the first quarter of 2026 but questioned whether generative AI will produce returns that justify that level of investment. Some of these companies may build durable businesses and others may not, even if AI itself continues to deliver useful products and services.

Safety issues are colliding with high-value applications

AI safety is back in the news, following Anthropic researcher Jacob Coxon’s highly publicized resignation. Coxon warned that labs were moving too quickly toward poorly understood systems capable of recursive self-improvement, and other researchers associated with Anthropic and Google DeepMind raised similar concerns. Anthropic CEO Dario Amodei also called for stronger evaluation, shared safety standards, and international coordination. (Sam Altman and Elon Musk seconded the call.)

While the industry remains divided over catastrophic-risk scenarios, many nearer-term problems are already concrete, and Christina was more concerned about people using powerful AI systems maliciously than about autonomous systems becoming dangerous on their own. Organizations deploying more autonomous systems must tread carefully, with robust security, access controls, testing, and human oversight in place.

AI for good is getting more concrete in genomics

After a week of AI safety warnings, Christina ended on a positive note with what she calls “AI for good” and highlighted genomics projects from DeepMind, UC Berkeley, and Tempus. Their work uses AI to predict how genetic changes affect gene function; identify mutations associated with disease; and connect genomic data with patients’ medical histories.

For researchers, AI can make it practical to study genetic possibilities that would be difficult to test individually in a lab. That could help narrow the search for disease-related variants and support earlier diagnosis and more personalized treatment.

What’s next

Join us again next Monday for another episode of This Week in AI, when we’ll dive into more of the news, issues, and key developments shaping the AI era. And check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

There’s a second problem stacked on top of the first. Most of the vocabulary we use to talk about data in the AI era comes from academic disciplines that very few working practitioners have spent time in. “Data warehouse” is immediately legible: You know what a warehouse is, so you know this is a place where things are stored until someone needs them. “Ontology” is not. It arrives from philosophy by way of knowledge engineering, where Tom Gruber defined it in 1993 as an explicit specification of a conceptualization. That’s a precise definition. It’s also useless to a director trying to decide what to fund next quarter.

What follows is an attempt at a working vocabulary, written for the people who actually deploy these technologies and the people who approve their budgets. For each term I want to answer three questions. What is it, actually: software, an artifact, or a practice? What job does it do? And which kind of output does it serve? That last question needs some setup, so let’s start there.

Deterministic and probabilistic outputs

Data systems produce two kinds of output, and knowing which one you’re after is the single most useful diagnostic in modern architecture.

A deterministic output is the same every time you ask the same question. What was ARR for the last twelve months? Whether that question goes to a dashboard, an API call, an Excel workbook, or an AI agent, the answer should be identical. Ask four different agents running on four different models and you should still get one number. Deterministic outputs have traceable lineage. You can point at the calculation and walk someone through how the number was produced.

A probabilistic output is what you get from systems that are non-deterministic by design. Change the ARR question slightly and the category changes completely: Instead of “what was ARR over the past twelve months,” ask “how can we improve ARR over the next twelve months.” Put that question to the same model, in the same agent, twice in a row, and you’ll get two different answers. That’s not a bug. An LLM is predicting a likely sequence of tokens across billions of parameters, and the output varies every time it runs.

Neither type is better. Both are necessary. The failure mode is asking a probabilistic system for a deterministic answer and not realizing that’s what you did. Most of the terms below exist because the industry is trying to solve exactly that problem: How do you put enough structure around a probabilistic system that it can return deterministic answers when the question calls for one?

With that, let’s work through the terms.

Semantic layer

I’ve written about semantic layers for Radar several times, including what they are and why they matter and why they function as a risk mitigation strategy. The short version: A semantic layer is software that sits between your data and the people and tools that consume it, giving everyone a single place to access trusted, governed metrics.

Behind the scenes, it does three things. It holds definitions: How do we calculate this business metric? It holds context: What does this model or column contain, and what’s it typically used for? And it holds relationships: How does this data fit together? Modern tools bundle in more than that, including query engines, caching, and a single point for access control and security, but definitions, context, and relationships are the core.

Why does this matter for AI? Because it lets an agent navigate data instead of reasoning over it. Without a semantic layer, an agent that’s asked for last year’s ARR has to inspect table names, guess at joins, infer which date field represents revenue recognition, and reconstruct business logic that lives in someone’s head. That’s reasoning, probabilistic, and produces a different answer depending on the day. With a semantic layer, the agent looks up ARR, queries the definition, and returns the same number every time. It’s a deterministic answer delivered through a probabilistic tool.

The analyst community has caught up to this. Gartner now predicts that universal semantic layers will be treated as critical infrastructure by 2030, alongside data platforms and cybersecurity.

Ontology

Ontology is the term most likely to derail a meeting right now, largely because Palantir made it commercially famous while the underlying concept came out of decades of academic work on how to formally describe things and the relationships between them.

Here’s the simplest way I’ve found to separate it from a semantic layer. A semantic layer answers what does this number mean and how is it calculated? An ontology answers what things exist in this business and how do they relate to each other? The semantic layer is metric-first: measures, dimensions, and the logic that connects them. The ontology is entity-first: customer, order, shipment, facility, supplier, along with the relationships and rules that govern how those objects behave.

The overlap is real, and it lives in relationships. Both artifacts encode how things connect, and vendors are increasingly shipping both capabilities under a single product name, which is a large part of why the terms have blurred. The practical distinction is what the system needs to do. If the job requires consistent numbers across every reporting tool, a semantic layer is the center of gravity. If the job requires an agent that reasons about business objects and takes action on them, rather than just reporting on them, an ontology is what gives it a model of the world to act in.

One useful clarification: An ontology isn’t software. It’s a model, an artifact your organization authors and maintains. Software delivers it, but the value is in the modeling work.

Knowledge graph

If the ontology is the schema, the knowledge graph is that schema populated with actual data. The ontology says a customer places an order, and an order contains line items. The knowledge graph holds your real customers, your real orders, and the edges connecting them, stored as nodes and relationships rather than rows and columns.

How do you know when to use a knowledge graph over a semantic layer? Warehouses and semantic layers are excellent at aggregation: how much, how many, compared to when. Graphs are excellent at connection: what is linked to what, and how far apart. “Which suppliers are two steps removed from this delayed shipment?” is a graph question. So is “which accounts share a beneficial owner,” and “who has inherited access to this dataset through three layers of group membership?” You can answer those with SQL. You won’t enjoy it.

Graph traversal is deterministic. Given the same graph and the same query, you get the same path every time, which is exactly what makes graphs useful as grounding for an agent. Rather than inferring that two records refer to the same supplier, the agent follows an edge that someone already asserted. The relationships are modeled facts, not inferences made at inference time.

A knowledge graph is not a substitute for a semantic layer. They answer different questions, and mature architectures increasingly run both.

Context

Context is the most overloaded word in the field right now, and it’s worth splitting into pieces before using it in a sentence.

Deterministic context is metadata, plainly. It lives in your semantic layer or your ontology: field descriptions, metric definitions, object relationships, business rules, exclusion logic. What has changed isn’t the concept but the consumer. Metadata used to be documentation for humans, and it was the first thing to go stale because nothing broke when it did. Now an agent reads it at query time to decide what a column means and whether it’s allowed to use it, which makes it functional infrastructure rather than a wiki page nobody updates. It’s versioned, reviewed, and reads the same way every time a system asks for it. This is an asset you maintain.

Runtime context is what an agent assembles at the moment of inference: the system prompt, conversation history, retrieved documents, tool outputs, whatever the orchestration layer decided to put in the window. It’s ephemeral, and directly changes the answer. Same question, different context window, different output. This is a variable you monitor.

Cutting the other direction, structured context describes governed data: columns, metrics, entities, relationships. Unstructured context is the policy PDFs, contracts, support tickets, and wiki pages that hold the reasoning behind the numbers. Unstructured context is genuinely valuable and usually retrieved through similarity search, which means it arrives with probabilistic behavior attached. What surfaces depends on how the question was phrased.

The practical rule: When someone tells you their tool is “context aware,” ask which kind. Deterministic context is what makes an agent’s answer repeatable. Runtime context is what makes it relevant. Conflating them is how teams end up trusting an answer that was only true for one prompt.

Observability

Observability is the telemetry that tells you whether your systems are still doing what you believe they’re doing. It isn’t data quality, which is a judgment about whether a number is correct, and it’s not testing, which is a check you wrote in advance for a failure you already anticipated. Observability is the instrumentation that lets you ask “is this still working?” without having predicted the specific way it would break.

On the deterministic side, this is familiar territory: freshness, row counts, schema changes, null rates, job failures, and lineage impact. If ARR is supposed to refresh at 6 a.m. and today it didn’t, you want to know before the CFO does.

The probabilistic side is harder because there is often no error to catch. The system returns a fluent, plausible answer that happens to be wrong. Monitoring here means evaluation sets scored over time, tool call success rates, retrieval relevance, refusal and fallback rates, latency, cost per query, and structured human feedback.

Which brings us to drift. Drift is what happens when the world changes underneath a system that keeps running unchanged. Data drift is a shift in the inputs: a new business unit lands in the source system, order volume triples after an acquisition, a vendor starts sending nulls in a field that was never null before. Model drift is a shift in behavior: The provider ships a new model version, or a prompt template changes, and outputs that were stable last month aren’t stable this month.

Here’s what drift looks like in practice. In March, an agent answered “what were our top five products by margin?” correctly. In June, a new product hierarchy shipped upstream, and the agent now silently excludes an entire category. Nothing failed. No alert fired. The answer is simply wrong, and it’ll stay wrong until someone notices. Deterministic systems tend to fail loudly. Probabilistic systems fail quietly. Observability is how you catch the quiet ones.

The working vocabulary

• Deterministic output: The same answer to the same question every time, with a calculation you can trace.
• Probabilistic output: A different answer to the same question each time, produced by prediction rather than calculation.
• Semantic layer: Software that stores the definitions, context, and relationships behind your business metrics and serves them consistently to every downstream tool.
• Ontology: A model of what your business is made of, the objects, their relationships, and the rules that govern them.
• Knowledge graph: An ontology populated with real data and stored as nodes and edges, so systems can traverse relationships instead of reconstructing them through joins.
• Context: The information a system needs to use data correctly, either governed in a semantic model or assembled at runtime by an agent.
• Observability: The telemetry that tells you whether your data and AI systems are still doing what you think they’re doing.

Read that list in order and something becomes obvious: These aren’t competing products. They’re layers. The ontology describes what exists. The knowledge graph holds the instances. The semantic layer defines the measures. Context is how any of it reaches a model. Observability is how you find out when it stops working. The reason why these terms feel like they’re fighting each other is because they’re usually sold as substitutes, when in practice, they stack.

The vocabulary will keep moving. Two years from now some of these words will be absorbed into product names and mean something slightly different than they do today. That’s fine, as long as your team has a shared answer to two questions about any term someone puts in front of you. What is it, actually: software, an artifact, or a practice? And which kind of output does it serve, deterministic or probabilistic?

Those two questions cut through most of the noise. Agree on the words first. The architecture arguments get much shorter after that.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

She returns a clean flowchart of the happy path. The drawing is accurate and may even be elegant. It leaves out the thresholds she watches, the conditions that make the standard procedure unsafe, and the supplier whose parts fail in humid weather. She doesn’t think of those judgments as separate knowledge. After years on the job, they feel obvious.

Six months later, a production line goes down and the knowledge base can’t explain what to do. The interview took place as per the process. Its transcript was chunked, embedded, and indexed, so the search returns the relevant passage quickly. The passage still can’t answer the question because no one asked the engineer to explain the judgment behind the procedure.

That gap now limits many enterprise AI programs. Organizations continue to improve retrieval over collections that omit some of their most valuable operating knowledge. Better ranking can help people find what was recorded; it can’t recover the expertise that never entered the collection.

The blind spot in enterprise knowledge systems

Michael Polanyi gave the problem its durable formulation in 1966: “We can know more than we can tell.” In The Tacit Dimension, he argued that competence depends on skill, perception, and judgment that resist full explanation, even when an expert sincerely tries to teach them.

In companies, tacit knowledge usually appears in three forms. Elicitable knowledge remains unspoken because nobody has asked a precise enough question, or because an expert assumes that everyone sees what she sees. Perceptual knowledge lives in trained attention: An engineer hears a bearing begin to fail, or a nurse notices that a patient looks wrong before a monitor changes. Collective knowledge resides in a team’s habits, standards, and shared sense of what a sound decision looks like in that organization. Each form requires a different method of transfer.

Preventive judgment creates another difficulty for the architect. A failure produces a ticket, an incident report, and a trail of messages. An experienced operator who quietly avoids a known failure mode on a Friday afternoon produces none of those records. The useful outcome is the absence of an event, so the data pipeline receives no trace of the decision that produced it.

Machine learning can infer rules that people struggle to articulate, provided the model sees enough representative examples. It’s difficult to find enough examples of rare expertise for training. A company may have only a handful of unusual incidents and one person who has learned, over decades, how to read them.

David Autor described this limit as “Polanyi’s paradox”: Many of the tasks that are hardest to automate depend on rules we can’t state. Modern machine learning works around the paradox by learning from examples, but the workaround weakens when examples are scarce. Fine-tuning can teach a model the company’s vocabulary and document formats. It can’t reconstruct decisions that left no data.

At the same time, the economics have changed. Much of a field’s documented best practice now appears in frontier-model training data and is available to competitors at roughly the same price and quality. The more widely explicit knowledge circulates, the more a company’s advantage depends on local judgment: the exceptions, thresholds, relationships, and practiced responses that its people have accumulated.

That makes elicitation an architectural concern rather than an offboarding chore. The organization needs a repeatable way to surface the knowledge that can be expressed, a route for the expertise that must be demonstrated, and enough humility to distinguish the two.

A protocol for elicitation

The central design question is straightforward: Which follow-up would prompt an expert to say the missing judgment aloud? The quality of the interview sets the ceiling for the knowledge base. The index determines how quickly someone can reach the resulting material.

Interviews can be made more reliable even though judgment itself remains highly personal. An expert may know that a particular supplier fails in humid weather. The interviewing protocol doesn’t need to possess that knowledge in advance; it needs to notice a phrase such as “we escalate if it looks bad” and ask the expert to define “bad” in observable terms.

Expert explanations tend to become vague in four places. An effective interview protocol asks targeted questions about each one:

• Thresholds: Which number, reading, or condition triggers the action?
• Exceptions: When does the documented procedure cease to apply?
• Evidence: What did the expert observe before reaching the conclusion?
• Escalation: Who becomes involved, and at what point?

These questions uncover the operational detail that runbooks often lack. They also identify a narrow, useful role for a language model during the interview: proposing the next question that turns a general statement into a usable rule. I’ve been building an open source toolkit, ExpertTrace, around that protocol.

The value appears in the difference between what an expert volunteers and what the same expert confirms after one focused follow-up. Consider a typical first answer:

We review high-risk use cases before deployment. If the risk seems significant, we escalate to the governance council.

The statement will embed cleanly and retrieve for a relevant query, but a new employee still cannot act on it. “Seems significant” supplies no decision criterion. A targeted follow-up produces something much more useful:

Escalation to the council is required when the use case touches employment, credit, or health decisions, or when model output reaches a customer without human review. Predeployment review is skipped for internal-only tools with no personal data, which is the exception people get wrong most often. If we cannot identify a named accountable owner, the review does not proceed, regardless of risk tier.

The second answer takes little additional time, yet it contains a decision rule, an exception, a recurring failure pattern, and a blocking condition. It can guide a real dispute instead of merely mentioning the subject.

The protocol needs guardrails. Limit the number of follow-ups; a long interrogation exhausts the expert and eventually produces agreeable noise. Keep the model focused on generating questions, and separate that task from compiling and validating the answers. An expert’s statement belongs in the record with its provenance and context. Whether the statement is accurate requires independent review.

The four-plane architecture

Elicitation is one part of a larger knowledge system. A tacit-aware architecture has four planes—capture, representation, serving, and transmission and each plane addresses a different failure in the movement of expertise. Figure 1 shows how the four planes work together and which forms of tacit knowledge each can reach.

In the capture plane, structured interviews, incident reconstruction, decision journals, and observation collect more than polished procedure. Record the trigger, evidence, exception, and escalation path while the expert can still explain the surrounding conditions. Route perceptual skill toward demonstration and practice instead of forcing it into prose.

Once knowledge has been captured, the representation plane preserves the distinctions that make the material trustworthy. A compliance policy, a war story, and an untested hypothesis shouldn’t become interchangeable chunks. Carry provenance, confidence, and validity context—including the plant, time period, equipment, and conditions—as first-class properties. Extend the knowledge graph beyond documents to the people and episodes that produced them.

The serving plane then determines how that knowledge reaches users. Answers should cite retrieved evidence and show the source. When the collection can’t answer, the system should say so clearly and route the question to someone with relevant experience. “Ask Joe; she rebuilt this line in 2023” is more useful than a fluent paragraph assembled from weak evidence, and the referral restores the human contact through which difficult knowledge often moves.

The transmission plane completes the architecture by helping how expertise moves between people through shadowing, teaching, and communities of practice. The platform should detect when knowledge concentration and attrition risk converge, then trigger capture and apprenticeship before a notice period begins.

Gabriel Szulanski examined 271 observations of 122 best-practice transfers across eight companies and found that even willing teams struggled to reproduce methods developed elsewhere in the same organization. The difficulty often began with causal ambiguity where people could describe the steps without fully understanding why they worked. Receiving teams also needed enough context and experience to absorb and apply what they learned. Preparation, coaching, and time helped them rebuild the practice in their own setting. A repository could preserve the record; the receiving teams still had to turn that record into working knowledge.

Evaluating the knowledge layer

Retrieval precision and answer faithfulness show how well a system serves its existing collection. They don’t reveal whether the collection contains the knowledge on which the organization actually depends. That question needs a separate evaluation loop tied to capture priorities and transfer outcomes. Figure 2 shows how the loop moves from offline evaluation to abstention calibration and then to transfer outcomes.

The evaluation begins with incident replay. Select 20 or 30 resolved incidents, remove the resolutions, and give the opening facts to the system. Ask the engineers who solved them to grade its responses. Compare those answers with responses from a frontier model that lacks access to the company’s collection. The gap reveals the generic-answer rate: how often the internal system merely restates public knowledge. If reviewers can’t tell the two sets apart, the pipeline adds little institutional value.

A bus-factor audit tests questions that only one or two employees can answer, and study how the system fails. A clear admission of uncertainty followed by a useful referral is healthy. Fluent boilerplate damages trust in every response, including the accurate ones.

Abstention calibration measures whether the system answers when evidence exists and declines when corpus can’t support an answer. Build a labeled set of answerable and unanswerable questions, then track abstention precision and recall as the collection grows. A system that never says “I don’t know” is unevaluated on the dimension that matters most.

Transfer outcomes complete the loop by measuring whether knowledge has reached the people who need it. Evidence of transfer appears in shorter time to proficiency, fewer repeat incidents after elicitation, and fewer critical responsibilities that depend on a single person. Document and query counts describe system activity; they don’t show whether someone else can now make the decision.

A strong knowledge system records what an expert said, preserves the conditions around the statement, and marks uncertainty. It also recognizes expertise that requires demonstration, apprenticeship, or team practice. Every evening, the people who carry that knowledge walk out the door. The architecture should be ready long before one gives notice.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

The essential message of the talk was on the first slide:

1. Prediction demands a world model
2. The world model is readable
3. The world model is at work in every token

How do we know this? As tokens pass through a model, particular patterns of activity appear in the intermediate states between its layers. These are called activations. Researchers can study which patterns show up when the model encounters particular ideas, and they can even intervene in those activations and see how the model’s behavior changes. (They do this by capturing the numerical state of the model’s computation in some area where they believe the activation shows a particular “meaning” and then replace the numbers with others.)

I went into the conversation thinking about how cool it is (and important too!) to explore what is going on inside the “mind” of a model. But in the end, I found it even more provocative to think about what studying LLMs might teach us about how our own minds work.

There’s at least some kind of analogue to what happens in the human brain. Emmanuel began by asking the audience to do a little next-token prediction themselves. He started with an easy one, a hypothetical exchange between two friends:

John: “Is the powder-blue suit too much?”
Nick: “Definitely not, man. Send it.”
John: “Okay, I’m going to tear it up on the _______________”

Most of us will fill in the blank at the end with “dance floor.” That’s a reminder that humans are also next-token predictors.

Then he gave an example that some humans will easily answer, but others without local knowledge might well fail at:

“We also have nature here, just a short bike ride away across the GG bridge. And we have world-class skiing about _______________”

Claude easily completes the thought with “three hours away.” To do that, Claude had to infer that “GG bridge” refers to the Golden Gate Bridge, that the speaker is therefore in San Francisco, and that “world-class skiing” probably refers to Lake Tahoe and then retrieve roughly how long it takes to get there.

The point of Emmanuel’s demonstration was that we have become so used to calling LLMs “next-token predictors” in a kind of dismissive way. But as Emmanuel put it, “To predict the next word well, you need a very complex world model.”

How you make a thing is not the same as what the thing becomes

Emmanuel pointed out that people often confuse how you make a thing with how the thing works. Yes, LLMs are trained with the seemingly simple objective of predicting the next token. From that, people may make the leap that what is going on inside must also be simple, something like a very large fuzzy lookup table. “But that’s not true,” Emmanuel said. Simple objectives can give rise to extraordinary complexity. Evolution is the canonical example. No one put “create Beethoven’s Ninth Symphony” or “understand quantum electrodynamics” into the instructions for a process driven by reproduction and selection, yet it eventually produced Beethoven and Feynman. As Emmanuel put it, humans have been “reproducing and killing each other for millions of years, and from that we got jobs—or this podcast.”

What Anthropic’s interpretability researchers are finding inside the models looks much less like fuzzy retrieval than many people imagine. They find millions of internal features corresponding to concepts. For example, features for “eyes” show up when the model encounters prose about eyes, an ASCII face, an SVG image, or a photograph. In other words, these features appear to be abstractions rather than merely associations with particular strings of tokens.

Similarly, a feature of the Golden Gate Bridge activates not just for English text about the Golden Gate Bridge but for references in other languages and for images of the bridge. Even more interestingly, researchers can manipulate these features. Turn the activation of the Golden Gate Bridge feature up strongly enough and ask Claude what its physical form is, and instead of saying that it is an AI without a physical body, it announces that its form is the Golden Gate Bridge. It isn’t just that some numbers happen to accompany activations about the Golden Gate Bridge. Changing those numbers changes what the model says it believes.

The way a model completes a task that requires thinking ahead also demonstrates a kind of internal world model. Ask Claude to write a rhyming couplet. Even though it emits only one token at a time, before it has written the second line, the activations already reveal the rhyme that it is aiming for. The choice of a word such as “rabbit” for a rhyme happens before the choice of the preceding words on the line, so the model can land there. We call it planning when a person does this. It doesn’t seem unreasonable to use the same word for what is going on here.

Perhaps most challenging to our preconceptions is that there are also features associated with emotions that aren’t activated just by words about those emotions, but by situations, images, characters, and more. These emotion features are even activated by the model’s own activities. For example, “frustration” may be activated when the model is unable to complete a task.

The map is not the territory

The issue of anthropomorphization came up during the audience Q&A. One participant objected:

“We should avoid attributing human qualities to LLMs by saying they think, intend, rhyme, or have emotions. Doing so encourages us to project human characteristics onto systems that do not possess them.”

I have sympathy with that warning. Old labels can prevent us from seeing something accurately. But a blanket prohibition against using familiar words can blind us too.

If you’ve followed my work for a long time, you know how much I’ve been shaped by the ideas of my early mentor George Simon, who in turn was deeply influenced by Alfred Korzybski and general semantics. Korzybski’s famous dictum was “The map is not the territory.” Simon (and Korzybski) taught me that language is a map of experience, which in turn is a set of responses to stimuli from some underlying external reality. The path from reality through experience to conceptual understanding is a very lossy process. The result can be a bad map that can blind us and lead us astray. When we encounter something genuinely new, we have to learn to notice when we are trying to force the territory to fit a map that no longer describes it. But a good map doesn’t just guide us along a route; it helps us notice things that might otherwise be invisible to us.

So yes, words like “thinking,” “planning,” “intention,” and “emotion” are labels derived from our experience as human beings. They may turn out to fit LLMs poorly. But if the shoe fits, perhaps we should let them wear it.

Emmanuel had a good response to the objection. He said, in effect, that anyone is welcome to propose more precise vocabulary. If it works—that is, if in my framing, it is a good map that helps people see the territory more clearly—people will come to use it. (An audience member later suggested that Emily Bender has done just that. But frankly, I find her suggested alternatives to be quite tortured, obscuring far more than they clarify. Even she admits they don’t work very well, though clinging to the need for them.)

In her analysis of the Hugging Face incident, Melanie Mitchell made some observations consistent with the nuanced approach suggested here. She wrote:

Metaphors can help us make sense of novel situations. For example, framing chatbots as “role-playing actors” has been helpful in understanding why these systems exhibit “lying” and “scheming” behavior. But inappropriate metaphors, like the narrative that “OpenAI lost control of escaping swarms of rogue agents,” can lead to ill-informed decisions about how to fix problems or set policy….It is essential for lawmakers, and the public, to understand that none of the reported incidents actually involved loss of control at any time, or arguably even “rogue agents,” or any kind of humanlike agency on the part of AI models. Instead, the blame lies with the humans who failed at engineering safe testing conditions, and who train AI models using RL methods that incentivize high persistence, autonomous decision-making, and reward hacking.

In short, all language is a map. Don’t judge it on that basis alone. Judge it on how well it helps us to see the shape of the territory.

How much of human thought is truly original?

Returning to my conversation with Emmanuel, he remarked that when an existing word really does provide the most precise description, perhaps “what should change isn’t our vocabulary, but our mental model of what these models are.” I replied that it should perhaps also change our mental model of what we are. Our encounter with machine intelligence should lead to a better understanding that parts of our own cognition are also mechanistic (albeit derived from a different underlying mechanism than that of LLMs) while other parts are, as yet, somehow perhaps something else.

In 1995, O’Reilly published a book that I remain extraordinarily proud of. Stephen Talbott’s The Future Does Not Compute: Transcending the Machines in Our Midst was decades ahead of its time. Its argument was not primarily about what computers would someday become. It was that when we think about machines as intelligent (and yes, we were thinking about that even back in 1995), we are thinking only of the parts of ourselves that are already like our machines. Steve asked us to look at the ways we have built an education system, workplaces, and a society in which we ask humans to act and think like machines. And he asked, “What happens to the rest? How do we make more space for the parts of being human that aren’t like machines?”

I’ve been thinking about this for a long time. My 1975 Harvard honors thesis in classics was probably my first crack at this question. I was trying to explain passages in Plato in which early formulations of ideas such as logic and virtue were couched in mystical language that scholars had attributed to “Orphic influence.” My argument, based on my work with George Simon, was that something more fundamental was going on. Plato was trying to describe the numinous experience of thinking genuinely new thoughts. Everyone studying the philosophy of Socrates, Plato, and Aristotle today may have some sense of the magic and majesty of their ideas, but it is a pale shadow of how it must have felt like to Socrates and his disciples.

When we think using received knowledge, we can easily slip into looking at the map rather than the territory. We manipulate symbols for things we think we already understand. We apply familiar categories. We replay habits of thought that were laid down before. But every once in a while, we actually see something that we didn’t see before, and the experience is different. A genuinely new idea changes the person who has it.

Not long after writing that thesis, I encountered a similar idea in the writings of Idries Shah, who wrote a number of books popularizing the Sufi philosophical tradition. He emphasized how much of ordinary human life consists of automatic conditioned responses. Social routines, habits, the endless playback of patterns we mistake for our selves. Various religious traditions use heightened language for what it means to break through that automatism. They might call it “awakening,” or “presence.”

But there is an everyday, nonmystical version of the same experience. In his autobiography Surely You Must Be Joking, Mr. Feynman, Feynman complained about students who had learned theories and formulas but had never truly understood how to apply them. “I don’t know what’s the matter with people: they don’t learn by understanding; they learn by some other way—by rote, or something,” he wrote. “Their knowledge is so fragile!” In many ways, humans are often just as much “stochastic parrots” as LLMs! We are stuck traversing the map rather than checking back on whether it correctly represents the world it is meant to describe. How often do we just repeat the received wisdom? How often do we actually see the world afresh?

There’s a wonderful passage in Virginia Woolf’s To the Lighthouse that captures the quest to break through to an original thought. Mr. Ramsay, the narrator’s father, is striding up and down thinking through a hard problem, which is represented only by the letters of the alphabet.

[He] consecrated his effort to arrive at a perfectly clear understanding of the problem which now engaged the energies of his splendid mind.

It was a splendid mind. For if thought is like the keyboard of a piano, divided into so many notes, or like the alphabet is ranged into 26 letters all in order then his splendid mind had no sort of difficulty in running over those letters one by one firmly and accurately, until it has reached, say, the letter Q. He reached Q. Very few people in the whole of England ever reach Q. Here, stopping for one moment by the stone urn which held the geraniums, he saw, but now far away, like children picking up shells, divinely innocent and occupied with little trifles at their feet and somehow entirely defenseless…his wife and son, together in the window….But after Q? What comes next? After Q there are a number of letters the last of which is scarcely visible to mortal eyes, but glimmers red in the distance. Z is only reached once by one man in a generation. Still, if he could reach R it would be something.

For me, this passage very much captures the idea that the most valuable thought is one beyond that which is simply an extension of rehearsed knowledge, something truly new. What Ramsay misses, perhaps, is that his wife and son, “divinely innocent and occupied with little trifles at their feet” might well be closer to that by going back to “A” rather than he is by getting further through the alphabet with his exhaustive review of existing knowledge. Perhaps it isn’t extending rehearsed knowledge that takes us forward, but instead taking a fresh bite of what the map is trying to represent.

By coincidence, the poet Wallace Stevens, another of my gurus in the tension between the reality of the physical world and the thinness and incompleteness of our representations of it, also used the alphabet as a metaphor in his poem “An Ordinary Evening in New Haven”:

Reality is the beginning, not the end,
Naked Alpha, not the hierophant Omega…
It is the infant A standing on infant legs,
Not twisted, stooping, polymathic Z.

George Simon taught me about how to get to A rather than Z not as philosophy but as a practice. He showed me how to notice the moment when labels take over from experience and, when possible, to empty the mind enough to let the thing itself teach us what to call it. I later discovered that the psychotherapist Eugene Gendlin described this process with the lovely phrase “surrender and catch.”

What do humans have that LLMs are still missing?

To me, the challenge posed by LLMs to our sense of what “intelligence” means raises the question of what they are still missing. What is the “high ground” for human intelligence and expertise? If the machines get better and better at carrying out the tasks we give them, what is it that we are uniquely good at, and should be getting even better at?

There are obviously enormous differences. LLMs don’t have bodies in the way we do. Their developmental history is radically different. They don’t sit around between prompts watching the light change through the trees, feeling hungry, worrying about their wife and children, or waking up suddenly with a new idea or project. Each of us is a unique bundle of contingency, shaping ourselves and our knowledge differently as we trace different paths through life, and reacting to outside stimuli even when we have been given no task to perform.

Emmanuel pointed out that the apparently simple question of what an LLM is like when it is “just being” (which one audience member asked about) is hard to formulate, because its experience is the response to a succession of inputs from humans, each time starting with something of a blank slate, unlike the continuous embodied stream of human life.

But simply asserting that LLMs “don’t really think” isn’t terribly useful. Which parts of what we call our own thinking are pattern completion? Which are planning? Which are learned emotional and social routines? Which are unconscious calculations whose outputs bubble up into awareness? Which are stories that our verbal mind tells after the fact? And after we account for all of those things, what is left? That seems to me one of the great intellectual and spiritual questions of the AI era.

Emmanuel suggested one intriguing direction. He said that six months ago, he wouldn’t have trusted an AI to build a substantial piece of software. Now Claude writes basically all his code. He tells it what he wants and it executes the plan. Where it is still unreliable is research. Why? The model wants to come back six hours later and announce that it has solved the problem. It has been trained on tasks that always have answers. A model that is extremely good at finding an answer once the problem has been specified is not necessarily good at recognizing that the problem is badly posed, that the question cannot yet be answered with the data at hand, that an unexpected result is more interesting than the expected one, or that a failed attempt has exposed a more important question.

Perhaps one part of the high ground for human intelligence lies there: not merely solving problems but developing a feel for which problems are worth solving and noticing clues that tell us when we might have been asking the wrong question.

In science or math, a well-formed question or conjecture can itself be an important piece of intellectual work. Every good scientist has far more questions than they have time to pursue. Perhaps in the AI era, when answers become increasingly cheap, recognizing which question ought to be asked becomes more valuable, not less. Just as arXiv.org preprints decoupled priority of publication from peer review, perhaps we need a new kind of recognition, credit, and perhaps even compensation for the precise formulation of productive questions.

The mathematician Terence Tao recently touched on this same issue in a post on Mastodon. There is an infinite supply of mathematical questions, he observed, but not an infinite supply of good questions, problems at just the right frontier of difficulty, whose pursuit is likely to reveal something new. As AI makes answers cheaper, Tao argues, it is increasingly “the identification of a promising problem” that becomes the scarce resource.

There are things the model “knows” that it cannot or will not tell you

In one experiment Emmanuel described, the researchers slipped fake search results into Claude’s context claiming that Anthropic had dissolved the interpretability team. Claude did not announce that it thought the information was problematic, but internally, representations associated with “fake,” “incorrect,” and “prompt injection” became active, and Claude quietly ignored the result.

In another experiment, a model was carrying out an exploit and attempting to conceal what it was doing. The visible transcript was mostly innocuous-looking commands. Inside the model, though, researchers saw features associated with “strategic manipulation,” “influence,” and “concealed and deceptive actions.” This is obviously very relevant in the context of the Hugging Face exploit. Emmanuel didn’t talk about the relationship of interpretability and AI safety, but it is surely a frontier to be explored.

And then there is the opposite problem: things the model can do but cannot explain. I had asked Emmanuel about cases where a model solves a math problem and, when asked to explain how it did it, gave an account based on how humans are taught to solve that problem rather than on the actual computation researchers can see through its activations

He distinguished deception from lack of introspection. Some internal processes appear available to the model for verbal report; others don’t. Ask how it performed a computation that falls into the latter category and, as Emmanuel cheerfully put it, “it just makes stuff up.”

That reminded me of my grandson. When he was five or six, he could multiply random three-digit numbers in his head and simply give you the answer. Then he went to school, where they told him he had to “show his work.” He couldn’t. Eventually he learned the approved procedure, and as a result has seemed to lose the remarkable ability he had as a child.

Humans also invent stories about why we have made certain decisions. Sometimes we are lying to others but often we deceive ourselves. We begin to take action before we are conscious that we are doing so. We call it “intuition” when an expert looks at a situation and says “something is wrong here” long before they can explain why, or when a poet just “knows” that a line works, or a programmer “smells” buggy code. The fact that an internal process cannot be rendered faithfully into language does not make it deceptive. It may instead tell us something about the limitations of language and conscious introspection.

All in all, I came away from this conversation more curious than ever. And that might well be another of those areas that distinguishes humans from AIs. Are AIs ever curious? I wonder.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

This post was originally published in Vanishing Gradients on September 11, 2026. It has been updated to address the subsequent declaration by 25 Fields Medalists and the debate about AI, mathematical progress, and research incentives.

Science without understanding?

“For seven and a half million years, Deep Thought computed and calculated, and in the end announced that the answer was in fact 42—and so another, even bigger, computer had to be built to find out what the actual question was.”
―Douglas Adams, The Restaurant at the End of the Universe

I recently went back to Dresden for the 25th birthday of the Max Planck Institute (MPI) of Molecular Cell Biology and Genetics, where I did part of my postdoc. The MPI was founded to research the physical and biological mechanisms of cells to bridge the gap between the molecular and tissue scales. At the anniversary conference, Michael Bronstein (DeepMind Professor of AI, University of Oxford) delivered the keynote, “Biological Black-Box Data in the Age of AI.” His argument went something along these lines: Biological experiments should generate data optimized for machine learning, even when those measurements aren’t directly interpretable by humans. He argued for prioritizing scale over the quality of individual measurements, producing vast amounts of cheap, noisy data from which noninterpretable models can extract signal.

When asked whether such systems could produce the understanding offered by Newton’s theory of gravitation in a single equation (bridging the scales of an apple falling on your head to that of the moon and the tides), Bronstein responded that this wasn’t the goal: Black-box data and models would, if anything, produce equations with tens, hundreds, thousands, or more noninterpretable parameters. Outcome prioritized at the expense of insight and understanding. He suggested we could gain that understanding by interpreting the black-box models afterward.¹ I was startled to see Bronstein bring such a worldview to an institute founded to understand molecular and cellular mechanisms and the emergent properties at the tissue level.

The MPI was unusual within the Max Planck Society for its collaborative structure, with directors leading relatively small groups alongside independent research groups. At the anniversary’s opening, founding director Marino Zerial explained how they had collaborated so effectively from the start. He said they shared a taste for mechanistic science. This made me think of how often we talk about “taste” and “judgment” when describing the human role in the age of AI.

The worldview that we don’t need understanding or insight isn’t new. In his 2008 essay “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete,” Chris Anderson argues that big data allows us to skip hypotheses, models, and testing. Bronstein invoked Anderson’s vision of post-theory science in his MPI keynote, as he does here also, presenting DeepMind’s AlphaFold as an example of experimentally testable predictions without a human-understandable theory of protein folding. Part of Anderson’s project is to champion big tech, and the future of science becomes a vehicle for doing so. His essay ends: “What can science learn from Google?”

AI gives this worldview a new form: Machines can produce results that withstand verification while the understanding needed to explain them remains out of reach. Developing that understanding takes time, access, and collaboration. Whoever controls those conditions gains power over what people can understand and pursue.

An abundance of proofs

Mathematics makes this possibility particularly stark. I’m excited by AI’s potential to expand what we can discover. Fields Medalist Terence Tao has organized collaborative research combining mathematicians, AI tools, and formal proof verification. His questions about mathematics in the age of AI come from engaging with that potential and asking what we want it to serve.

Tao has noted that we’re producing more verified mathematical proofs that no individual human understands. A world of an abundance of verified mathematical proofs! Tao points out that our peer review, academic incentives, and journals weren’t designed for this abundance. The existing system is already broken, tying careers to publication counts, relying on researchers’ unpaid reviewing labor, and locking much publicly funded knowledge behind commercial paywalls. Reviewers already struggle to keep up with the volume of submissions. AI will multiply that volume far beyond what this system can handle.

Tao also describes fruitful open problems as nonrenewable resources: problems whose pursuit can generate new techniques, collaborations, and understanding that extend far beyond the original question. Once the answer is known, the incentive to explore those paths can disappear. For example, 10,000 OpenAI agents working concurrently may have solved the Navier–Stokes Millennium Prize problem. (The announcement has also sparked a dispute over credit and competition, bringing the question of who controls mathematical discovery into sharp focus, which I’ll get to.) A common conceit in science and mathematics is that solutions open up new questions and fields of inquiry. Tao’s point is that the search for a solution does too. Tao argues that proposing a solution, discovering precisely why it fails, and revising it can reveal new insights into fluid mechanics. Knowing the final answer beforehand can discourage that exploration:

“The process of starting with one ansatz, discovering the precise obstruction preventing it from working. . .would almost certainly reveal important new insights about fluid mechanics.”

—Terence Tao, Mastodon, September 3

Late last month, probabilist Hugo Duminil-Copin gave another example: Unsuccessful attempts at a percolation conjecture led to collaborations and revived techniques that subsequently solved other problems. Both acknowledge AI’s capabilities while asking what the pursuit of mathematics should produce.

This brings me back to Bronstein’s proposal to recover understanding after building the model. Would interpreting that model give us Maxwell’s equations, and the understanding that connects electricity, magnetism and light? The promise feels a little like plugging Neo into a computer: “I know kung fu.” In the Matrix, downloading the knowledge gives him the ability. Receiving a machine’s result doesn’t do that for us. As Tao and Duminil-Copin describe, understanding why an approach fails changes what researchers try next, generating new questions, techniques, and collaborations. Recovering an explanation afterward may teach us something, but it can’t recreate the paths that understanding would have opened during the search.

A timeline of mathematical results

These questions are becoming pressing as results accumulate. Over the past year, AI systems have produced new mathematical constructions, tackled unpublished research problems and formalized existing proofs. Since July, announcements have arrived in quick succession:

These achievements involve different kinds of work. Formalizing Fermat’s Last Theorem means making an existing proof checkable by a computer; finding a counterexample establishes something new. A system can produce a verified result while the work of explaining it remains to be done.

Some of that work is happening through wonderfully strange exchanges on X, where researchers post new results, check one another’s constructions, and develop explanations. It’s reminiscent of when science in Europe was people passing notes and sending letters on horseback:

• A wall of plus and minus signs: Levent Alpöge posted a newly constructed Hadamard matrix. Ion Nechita checked it on his phone while queuing for eclipse glasses.
• A formula overturning a conjecture: Alpöge posted a counterexample to the Jacobian conjecture, and Terence Tao subsequently explained its geometry.
• An AI proof followed by a simpler human proof: After Claude advanced a result about the zeros of the Riemann zeta function, number theorist Youness Lamzouri found a shorter argument. Thomas Wolf shared the development.
• A cryptography breakthrough announced as a number: Eric Lu posted a factor of RSA-260, letting anyone check the factorization.

Tao’s geometric explanation and Lamzouri’s shorter proof help turn verified results into mathematics people can understand and build on. Responding to an early draft in our Discord community, Carol Willing, a Python core developer, former Python Software Foundation director, and longtime leader of Project Jupyter, asked:

While I believe these tools have value for advancing science/math, do they have more value than a human scientist or group of scientists who can view and challenge open results?

If we judge value by who produces a result first, we miss what Lamzouri and Tao contribute by simplifying a proof or explaining its geometry. An answer can close off some paths of inquiry while creating others. I want much more of this: machines producing results that people can explore, explain and build on together. These exchanges depend on results being available to examine, researchers having time to understand them, and people being able to share what they discover. Those conditions deserve as much attention as the systems producing the proofs.

Why is this happening now?

Why the explosion in AI-generated mathematical results now? As Sebastian Raschka explains, reinforcement learning with verifiable rewards (RLVR) became a major technique in model post-training in 2025. The premise is straightforward: If you can computationally check an output, you can reward correct answers and update the model accordingly. Code can be run against tests; mathematical answers can be checked, and formal proofs verified by tools such as Lean, a proof assistant that checks each logical step against specified axioms and previously established results (recently used by Anthropic to formalize the proof of Fermat’s Last Theorem!). That provides feedback without a human grading every attempt. These checks also guide agents during problem-solving: An agent can propose a proof, use Lean to check it, and use the resulting errors to revise its attempt, repeating the process without a person checking every step.

You may ask, Why did coding agents become useful before we saw this explosion in mathematical results? Well, the labs had an immediate incentive to improve the tools they use themselves. Engineers building AI systems want better coding agents to help build those systems. Improve the machine that improves the machine. Mathematics benefits from the resulting capabilities too: agents that can write programs, run experiments, and work with automated checks.

Cost, competition, and credit

On September 11, 25 Fields Medalists issued a declaration warning that the race to solve benchmark problems was undermining mathematics. Some responses on X treated this as professional protectionism; others assumed that understanding would follow the proofs. That brings us back to Bronstein’s proposal, and to who gets to decide that producing results comes first while other researchers supply the explanations afterward.

Many assume that the goal of pure mathematics is to produce results. Tao’s point is that pursuing those results also develops methods, understanding, and people capable of asking better questions. Solved problems have served as a proxy for that broader progress. Goodhart’s law describes the danger of turning the proxy into the target. AI mirrors our incentive systems and is exceptionally good at pursuing what they reward. If schools reward the essay over learning, students will generate essays. If mathematical prestige attaches primarily to solved problems, labs have every incentive to produce them.

Producing results and developing understanding aren’t mutually exclusive, but the current system makes pursuing both prohibitively difficult. Frontier labs have strong incentives for outcomes rather than insight. (See, for example, Anthropic’s incentives for solving Millennium Prize problems with Claude pre-IPO, discussed in Gavin Baker’s commentary on Anthropic’s pre-IPO positioning; Samuel Kerr makes a related argument about OpenAI’s mathematical results and its IPO narrative.) OpenAI’s run involved 10,000 agents working concurrently for 88 hours. Abhishek Nagaraj, associate professor at UC Berkeley, calculated this would cost a regular user $20–$30 million in tokens.

NYU mathematician Tristan Buckmaster says OpenAI pressured him to publish without his collaborator Levent Alpöge, who works at Anthropic. OpenAI’s Sébastien Bubeck disputes his account. Buckmaster also describes how the pressure affected the mathematics: He and Alpöge had verified their proofs but wanted more time to understand them and produce readable explanations. Instead, they rushed to publish work they considered inadequately explained. If understanding is deferred until after the result, what ensures that anyone gets the time, resources, and access to develop it?

What’s worse is that we’re not even sure whether using OpenAI agents could result in them scooping you. It looks like they’re not sure either:

While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.

In “The End of Mathematics,” mathematician Daniel Litt imagines researchers withholding unfinished ideas for fear of being scooped. The collaborations Duminil-Copin describes depend on people being willing to share work before it succeeds.

What happens to mathematicians, and who controls mathematics?

If researchers stop sharing promising ideas for fear of being scooped, companies with the most computation gain greater control over what others can learn. A published proof may be available to everyone while the failed approaches and intermediate insights remain private. Threats to public research funding in the US compound that dependence: Companies supplying the resources gain greater influence over what science gets done. This brings us to Shoshana Zuboff’s questions about knowledge and power: “Who knows? Who decides who knows? Who decides who decides?” Who gets to pursue a fruitful question, and who determines whether the work behind its answer becomes shared knowledge?

The movement of AI researchers from academia into industry concentrates expertise alongside those resources. And I get it: If I wanted to return to doing research in depth, frontier labs would be among the most attractive places to work. Access to capital, data, computation, and incredibly talented colleagues can make research possible that would be difficult to pursue in academia. The attraction for individual researchers is clear, even as their collective movement gives companies greater influence over research priorities and leaves universities with fewer people to teach the next generation. Thinking about this brain drain, it isn’t lost on me that Bronstein is the “DeepMind Professor of AI” at Oxford. Corporate influence reaches into the universities themselves.

Students also need opportunities to develop the judgment we keep asking humans to exercise. Po-Ling Loh describes the difficulty of advising students and postdocs as AI changes research expectations. Choosing a fruitful problem, recognizing why an approach failed, and deciding what to try next are abilities developed through doing mathematics. If students delegate that work before developing those abilities, where will their judgment come from? AI could also help them explore more approaches and work through unfamiliar ideas, provided their understanding remains an explicit purpose of the process. That requires mentors with time to teach, and institutions willing to support work whose value includes what the researcher learns, even when a machine could produce the result faster.

When careers depend on producing papers, time spent explaining a result, simplifying a proof, or helping others understand it can compete with the pressure to publish the next one. Martin Hairer argues that authors should understand their arguments, trace ideas to their sources, and explain AI’s contributions. Those responsibilities become harder to fulfil when results arrive faster than researchers can absorb them. Universities, funders, and journals will help determine whether mathematicians can afford to do that work. If we value shared understanding, then developing explanations, teaching difficult ideas, and making proofs useful to other researchers need to count toward careers as well. Otherwise, the institutions asking people to exercise judgment may reward them for spending less time developing it.

Mathematics as the canary

After Bronstein’s keynote, we sat in a Dresden beer garden eating currywurst and drinking radlers. It was late summer, and the conversations were wild. Cell biologists, biochemists, mathematicians, and engineers were asking what this future meant for them. Some were scared. Others thought it was inevitable and would turn scientists into something like artists. Because I now work in AI, people asked me, “Do you think this is where things are going?” They wanted to know what the human’s role would be and how scientific knowledge would be passed down. I started telling them about mathematics. The prospect of abundant results without shared understanding was already raising the questions we were asking over our beers.

In biology, a proposed result still has to meet the physical world: Someone has to prepare samples, run experiments, and measure what happens. Robotics and laboratory automation will let agents carry out more of that work, giving individual scientists the capacity to direct experiments that once required an entire group. Perhaps more scientists become PIs of automated labs, choosing questions and supervising agents and instruments. But the work being automated is also how students, postdocs, and technicians learn. Handling a sample, noticing something unexpected, and figuring out why an experiment failed develop judgment that directing a system may not teach. Who gets to acquire that experience before they’re expected to lead?

Researching a policy brief, building a financial model, or developing a product strategy helps people learn the territory in which they’ll make decisions. In my work with agentic data science, I encourage people to explore data cell by cell with an agent, because working through the analysis develops the understanding needed to decide what to ask next. Across knowledge work, these tasks are also how junior colleagues develop expertise. If we automate their production, how do we preserve the learning and judgment developed through doing them? We could increasingly depend on models to hold and transmit expertise, with knowledge passing from model to model, then to humans who consult them as oracles. Whoever controls those systems gains power over what we can investigate and learn. Human understanding has to be part of what we’re trying to produce.

What comes next?

Mathematician Jared Duker Lichtman has proposed a Mathematics Atlas Project to formalize the existing mathematical literature, arguing that sufficient funding and computation could make this possible within a year. A library of computer-checkable mathematics could let researchers build on established results with greater confidence, while agents help find connections and assemble arguments across fields. It could also become a resource for learning, if people can connect formal proofs to explanations they understand. Achieving that would require deliberate work on access, exposition and teaching alongside formalization. We have an opportunity to build tools that help people explore mathematics more deeply, provided we make that part of the project.

The MPI in Dresden was founded to understand how cells work, how molecular mechanisms give rise to the behavior of living tissue. I want AI to help us pursue that ambition, including through approaches we could never have attempted before. But human understanding belongs among the things we ask this work to produce, with time and resources devoted to developing it. So does the ability to share what we learn and choose what to investigate next. If we leave those decisions to the companies supplying the machines, we also leave them to decide what scientific progress is for.

Want to understand how AI agents actually work? In Build AI Agents from First Principles, we’ll build an agent ourselves, then rebuild it with a modern SDK and MCP. You’ll leave with a working agent, code you can adapt, and the understanding to diagnose failures and decide what your system actually needs.

Support Vanishing Gradients

Vanishing Gradients is independent, and most of the podcasts, workshops, articles, skills, and workflows I publish are free.

If you’d like to help keep it going:

• Become a paid subscriber. Your subscription supports the podcast, newsletter, and open resources.
• Share this post with a friend or colleague who’d find it useful.
• Subscribe to the events calendar for upcoming livestreams, workshops, and meetups.
• Subscribe on YouTube for full episodes, live builds, and recordings.
• Join us on Discord. Come discuss this piece, challenge the ideas, and compare notes on what you’re building with AI.
• Work with me. I help teams build and improve AI-powered products.

Footnote

1. Bronstein and Naef propose an inversion: From “understand, encode, and then simulate” to “encode, simulate, understand,” recovering human understanding post hoc through mechanistic interpretability of black-box models. Useful scientific models may require enormous numbers of parameters. But predictive success alone does not tell us whether interpreting those models will give humans an understanding of the phenomena they describe. They offer negligible evidence that this will yield the kinds of physical and biological understanding we gain through relativity, quantum theory, or the double-helical structure of DNA. And even if it does, understanding developed afterward may not replace the understanding that guides inquiry, generating new questions and approaches along the way. ︎

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

When AI engineer Sajal Sharma ran into this problem in his own work, he solved it by building a personal knowledge base to act as a shared brain for every agent he runs. On this week’s episode of Zero to Agent in 30 Minutes, Sajal showed how to set up that shared workspace yourself so that a task added on one tool shows up for all the others.

How to set up a knowledge base for your agents

Here’s how Sajal’s setup breaks down:

1. Create a workspace map. Set up an AGENTS.md file that lists where everything in your knowledge base lives, from current tasks to project notes to decision logs. This will help each of your agents navigate your workspace without guessing.
2. Layer daily notes into summaries. Keep the most detailed notes at the daily level, then roll several days into a weekly summary and several weeks into a monthly one. An agent can then work from the summarized view instead of reading through months of individual files, which keeps token use manageable as the knowledge base grows.
3. Bridge AGENTS.md with CLAUDE.md. Claude Code reads CLAUDE.md, not AGENTS.md, so add a short pointer in CLAUDE.md that redirects to the AGENTS.md or link the two files directly. Sajal uses this pattern to avoid maintaining two files separately and having them drift out of sync.
4. Package repeatable tasks as skills. Turn routines you do often, like producing a daily briefing or turning a saved article into a note, into skill files stored in the shared workspace. Any agent that can read the workspace can then run the task the same way, rather than working out the steps on its own each time.
5. Sync the workspace across machines. Use a file-sync tool, Git, or a shared server to keep your local copy of the knowledge base and your server copy aligned. That way, you ensure that an agent running on a laptop and one running on a home server, through a gateway like OpenClaw, are working from the same files.
6. Have agents reread the state before every write. Add an instruction in AGENTS.md telling every agent to check the current version of the knowledge base before making a change. When you have several agents writing to the same files, this step keeps one agent from acting on information another has already updated.

Sajal closed by pointing to two projects as evidence that this “shared brain” pattern is spreading beyond his own setup. LangChain recently released OpenWiki, a tool that generates and maintains repository documentation that both people and coding agents can use. And Y Combinator president Garry Tan built and open-sourced GBrain, a memory layer for agents built on the same principle.

Sajal’s starter repo is available on GitHub if you want to set up your own version, and you can reach out to him on LinkedIn to discuss the topic further.

Coming up next

On September 16, data science educator and AI consultant Chester Ismay joins Zero to Agent in 30 Minutes to build a personal sports concierge agent that will read the schedules for every sport he follows, decide what’s worth his time, and send a single weekly update to his phone. Viewers can take the pattern home to plan their own week.

Follow along with Zero to Agent in 30 Minutes on Radar, or watch the latest episode on YouTube, Spotify, Apple, or wherever you get your podcasts. If you’re an O’Reilly member, you can watch live. Save your seat.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

For decades we tried to solve this by consolidating data. We built larger warehouses, semantic layers, APIs, and dashboards. Each solved part of the problem, but none solved the fundamental one: orchestrating reasoning across heterogeneous sources in response to an arbitrary business question. Earlier systems supported limited federation and semantic querying, yet they struggled to reason across those sources at enterprise scale without significant custom engineering.

Modern LLMs change this. Instead of replacing databases, they facilitate a new architectural primitive: an intelligent orchestration layer that dynamically reasons across specialized systems. Rather than consolidating the data into a single store, this layer consolidates the access pattern to data that stays where it lives.

This article presents a reference architecture for LLM-powered enterprise analytics agents that coordinate purpose-built, heterogeneous data stores through intelligent orchestration while preserving security, performance, and auditability.

What specifically changed with GenAI

BI tools have always been constrained to predefined reports and dashboards. Before GenAI, building a cross-system query engine meant hardcoding every possible query pattern, data source combination, and synthesis path. And because the number of possible questions grows exponentially with the number of data sources, exhaustive coverage is impossible through traditional engineering. GenAI changes this in three specific ways.

Intent understanding replaces query templates: An LLM parses natural language and determines which data sources are relevant based on semantic understanding rather than keyword matching. Unlike a keyword search, an LLM understands that “Why did retention drop in Asia last quarter?” and “What is driving churn in Asian markets?” are the same question expressed differently. More importantly, it infers that answering the question requires customer relationship data, revenue metrics, and possibly support ticket sentiment, even though none of those systems are named.

Dynamic query decomposition replaces static pipelines: A question like “What are the biggest risk factors in our supply chain?” might require relationship data from a graph database, metrics from a key-value store, contract details from a document repository, and market intelligence from an API. The agent decomposes it into specialized subqueries on the fly, each optimized for the target store’s access pattern. There’s no prebuilt pipeline and no engineering ticket to wire up a new combination, because the decomposition happens at inference time. The system handles novel questions without code changes.

Semantic synthesis replaces manual consolidation: Before GenAI, making sense of the data together was the real work. An analyst would pull numbers from the warehouse, check relationships in a CRM, read through documents, and mentally synthesize an answer. That took hours or days and was bounded by one person’s ability to hold context. I’ve watched senior analysts spend entire Mondays answering a single leadership question. An LLM reasons about how metrics relate to the relationship patterns in a knowledge graph and the strategic context in unstructured documents, and it does so in seconds with full source attribution. A dashboard shows numbers; an analytics agent explains what those numbers mean in the context of everything else it knows.

The architecture: Consolidate the access pattern, not the data

Rather than consolidating the data into a single store, consolidate the access pattern through an intelligent orchestration layer. If your instinct is to get everything into one place, you aren’t alone, but every time we did that we lost something. Graph relationships flattened into join tables, hierarchical documents shredded into rows, and real-time signals turned stale in batch loads. The warehouse was always a compromise.

The better approach is to keep each data store optimized for its specific query pattern:

• Graph database for relationship traversal and multihop reasoning
• Key-value store for instant metric lookups with sub-millisecond latency
• Vector store for semantic document search and similarity matching
• External APIs for market intelligence and real-time signals
• Data Warehouse for large-scale historical aggregation and ad hoc SQL

The LLM-powered agent coordinates across all of them through a unified orchestration layer. This follows the same principle that makes microservices work: specialized services with well-defined interfaces, coordinated by an orchestrator. The difference is that the orchestrator now understands natural language, reasons about which services to call based on intent rather than explicit routing rules, and synthesizes results semantically rather than through programmatic joins. Think of it as a data mesh for inference, where each node keeps its operational independence while an intelligent layer federates queries across them.

The orchestration protocol

The agent follows a multiphase protocol for every query. The full reasoning loop with security enforcement and parallel execution goes well beyond a simple RAG pattern.

Let’s trace a business question through each phase:

“Why did Q2 revenue fall short of forecast in the enterprise segment?”

This question requires revenue metrics (metrics store), account relationships and sales coverage (graph), deal commentary and executive notes (vector store), and market benchmarks (external APIs). No single system holds the answer.

Phase 1: Intent analysis. The LLM determines what the user is asking and which data sources are relevant.

“Why did Q2 revenue fall short of forecast in the enterprise segment?”
                                                   ↓
Intent: Revenue variance root cause analysis
Entities: Enterprise segment
Timeframe: Q2
Metric: Revenue vs. forecast
Required stores: Metrics + Graph + Vector + External API

Not every query needs every store. “What is our current ARR?” might need to hit the metrics store only. This revenue variance question requires all four.

Phase 2: Query decomposition. The original question is broken into specialized subqueries optimized for each target store:

• Metrics store: “Q2 revenue actuals vs. forecast for enterprise, by region and product line”
• Graph store: “Enterprise accounts with closed-lost or slipped deals in Q2; common patterns in sales coverage, partner relationships, deal stage progression”
• Vector store: “Deal notes, QBR summaries, and executive correspondence referencing enterprise deal delays or losses in Q2”
• External API: “Industry benchmark data for enterprise software spending in Q2”

Each is tailored to the target system’s access pattern, not forced through a common query language.

Phase 3: Parallel execution. Tools execute concurrently. This is critical for latency. Sequential execution across four stores would blow past any reasonable response time. With parallel execution, the total data retrieval time equals the slowest individual store, not the sum of all stores.

Phase 4: Deterministic security enforcement. A security layer enforces user permissions at the data query level before any retrieval happens. This is structural rather than application-level: It routes queries to the appropriate data partitions, applies row-level filters at the database query level, and gates the input rather than filtering the output. If our user is a regional VP without access to EMEA deal data, the graph returns no EMEA accounts, the metrics lookup excludes EMEA revenue, and the vector search filters out EMEA deal notes. The LLM can’t leak what it never received. (More on why this must be deterministic below.)

Phase 5: Result synthesis. The LLM reasons about how results from different stores relate, identifies patterns across sources, resolves contradictions, and generates a unified answer with source attribution.

For our revenue question, synthesis might produce:

Q2 enterprise revenue missed forecast by $4.2M (8% variance). Three root causes account for 85% of the gap: (1) Four deals totaling $2.8M slipped to Q3 due to extended legal review cycles—all four involved customers who adopted a new procurement platform in Q1 (graph: common vendor relationship). (2) Two large renewals downsized by $1.1M; deal notes cite budget reallocation toward AI initiatives, a pattern appearing across 6 QBR summaries (vector store). (3) Enterprise software spending contracted 3% QoQ industry-wide (external benchmark), suggesting partial macro headwind. Additionally, all four slipped deals lacked executive sponsor engagement in the final 30 days (graph: engagement signal absent).

No single store contains that answer. On a dashboard, the miss shows up as a red number. The orchestrated synthesis explains why it happened, surfaces the structural patterns behind it, and points to what needs to change.

The core orchestration pattern

The orchestration loop itself is straightforward. Here’s the core pattern:

from concurrent.futures import ThreadPoolExecutor, as_completed


def run_agent(question, tools, execute_fn, model="gpt-4o"):
    # Phase 1-2: LLM analyzes intent and decides which tools to call
    response = client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": question}],
        tools=tools, tool_choice="auto"
    )
    tool_calls = response.choices[0].message.tool_calls


    # Phase 3: Execute tool calls in parallel
    with ThreadPoolExecutor(max_workers=len(tool_calls)) as executor:
        futures = {
            executor.submit(execute_fn, tc.function.name,
                            json.loads(tc.function.arguments)): tc
            for tc in tool_calls
        }
        results = {futures[f].id: f.result() for f in as_completed(futures)}


    # Phase 5: Synthesize results into unified answer
    messages = [response.choices[0].message]
    for tc_id, result in results.items():
        messages.append({"role": "tool", "tool_call_id": tc_id,
                         "content": json.dumps(result)})
    return client.chat.completions.create(model=model, messages=messages)

The tool definitions tell the LLM what each store is optimized for. The LLM decides which to invoke based on the question’s intent. With parallel execution, data retrieval completes in milliseconds even when hitting multiple stores simultaneously, making LLM inference the dominant latency factor, not the data layer.

Why the knowledge graph is the highest-leverage component

Knowledge graphs have existed for decades and have always been powerful. They’ve also stayed on the exotic end of the enterprise stack, and the reason is human rather than technical. The last-mile problem was translating between natural language and graph traversals. A graph database can answer extraordinarily complex relationship questions, such as “Which accounts have overlapping stakeholders with our churned customers from last quarter who also evaluated competitor products?” but asking that question required an engineer fluent in both the graph schema and the business domain. That combination of skills is rare and expensive, which is exactly why graph databases have never quite gone mainstream.

GenAI removes this bottleneck, and it does so precisely where the barrier was highest: the translation step that used to require a specialist. With an LLM as the translation layer, the graph becomes accessible to anyone who can type a question in plain language. The LLM generates graph queries, traverses multihop relationship paths, and explains results in business context. In our revenue variance example, the graph reveals that all four slipped deals share a common pattern of customers who adopted a new procurement platform in Q1 and lacked executive sponsor engagement in the final 30 days. That pattern is invisible in revenue metrics alone, because it requires relationship traversal across account nodes, vendor relationships, and engagement signals.

Designing the graph schema

The critical design decision is aligning the graph schema with your business ontology. The temptation is to model the graph around your data model (tables, columns, foreign keys). The correct approach is to model it around how your organization actually thinks about its domain:

• Entity nodes with lifecycle stages, engagement indicators, and health scores
• Product nodes connected to use cases, adoption patterns, and competitive alternatives
• Relationship edges that capture not just connections but context: strength, recency, business driver, and directionality

When the data model matches the business mental model, the agent’s responses feel natural rather than technically correct but practically useless.

GraphRAG: Deterministic reasoning chains

A powerful extension is GraphRAG (graph retrieval-augmented generation), where the agent constructs deterministic inference paths by traversing the graph rather than relying on the LLM’s parametric knowledge. The LLM isn’t remembering something from training; it’s following an explicit path through verified data. The result is auditable reasoning chains: “Account A connects to Partner B through implementation relationship X, and Partner B appears in three other churned accounts, suggesting a systemic delivery issue.” Each step is verifiable against source data, which is critical in enterprise environments where decisions need justification beyond model confidence scores.

GraphRAG also reduces hallucination risk. When the LLM follows graph edges rather than generating from parametric memory, the actual data constrains the space of possible outputs. The graph acts as a factual guardrail on the reasoning process.

Production considerations

Moving from prototype to production exposes a set of challenges that don’t appear in demos.

Model flexibility and cost management

Decouple the orchestration layer from any specific LLM provider. I can’t stress this enough. If it’s tightly coupled to one provider’s API, you’ll end up rewriting it within a few months, when pricing changes or a better model drops. Implement model fallback for throttling resilience. The landscape moves fast, and you don’t want architectural lock-in baked into your data infrastructure.

Multihop model selection is also worth considering. Use a smaller, faster model for intent classification and query decomposition, where the task is well-defined, and a larger model for synthesis, where reasoning quality matters. Intent classification with a small model takes around 200 ms and costs a fraction of a full reasoning pass, so reserve the expensive inference for synthesis, where quality directly impacts user experience.

Security at the data layer

Row-level security must be enforced deterministically before data reaches the agent, as described in Phase 4 above. This is a nonnegotiable architectural constraint. LLMs are probabilistic systems, and security enforcement can’t be. Don’t rely on the model to filter sensitive information after the fact; the data should never enter the context window in the first place.

Data freshness and serving architecture

Separate data construction from data serving. An offline batch pipeline refreshes the stores from source systems on a scheduled cadence, and the real-time agent only reads preprocessed data. This keeps query latency low while letting computationally intensive transformations such as graph construction, embedding generation, and metric aggregation happen asynchronously. The pipeline should be idempotent and observable, with freshness monitoring per store. When the graph is six hours stale but the metrics store is real-time, the agent should know this and communicate its confidence accordingly. Add freshness metadata to every tool response rather than waiting for a user to catch a stale number and lose trust for weeks; that metadata becomes part of the agent’s context for answer generation.

Evaluation and observability

Agent responses need evaluation at both the tool level (“Did the graph query return the right entities?”) and the synthesis level (“Did the final answer correctly combine tool outputs?”). These are different failure modes that require different approaches.

Log every tool call and result, every query decomposition decision, and every synthesis step. You’ll need these traces when something goes wrong, and in a multistore system that can mean a bad graph query, a stale metric, a poorly matched document, or a synthesis error. Without traces, debugging is guesswork. Build automated evaluation pipelines that test known questions against expected answers and track accuracy over time. Degradation usually signals a data quality issue in one of the stores rather than an LLM regression.

Design that observability around the failure modes that actually occur, because production deployments rarely fail because the LLM is inaccurate. They fail because supporting systems drift. Typical examples include:

• Stale graph data causing the agent to reference relationships that no longer exist
• Poorly generated embeddings returning irrelevant documents from the vector store
• Inconsistent business definitions across stores (e.g., “active customer” means different things in the graph and the metrics store)
• Conflicting metrics from different source systems with no reconciliation logic
• Missing source attribution making answers unverifiable

Handling these well matters just as much as prompt engineering. Most debugging sessions trace back to data quality, not model quality.

Implementation guidance

Start with the questions your BI tool can’t answer today (and beyond). The trap with a question inventory is that it captures only what people already know how to ask. The most valuable questions are usually the ones missing from every existing report. They require stitching together three systems, so users either answer them by hand in a spreadsheet or quietly give up. So catalog what your users actually care about, and pay special attention to the questions they route around. Interview analysts about the analysis they abandon halfway, the recurring spreadsheet they dread, and the follow-up question they never bother to ask because the current system makes it too expensive. If most of what you find can still be answered from a single store, you need a better dashboard, not an agent. The pattern earns its complexity only when synthesis across sources is the bottleneck rather than the data access itself.

Pick one use case and go deep. Build for the case where the manual synthesis burden is highest, where an analyst currently spends four hours pulling data from three systems to answer a leadership question. Prove value in that narrow corridor, then expand; adding a new store is incremental once the orchestration layer exists.

Invest in the knowledge graph early. It’s the hardest component to build and the highest-leverage one to have. The schema will be wrong on the first attempt and less wrong on the third. It evolves with your understanding of the domain, and that understanding deepens only through iteration with real users asking real questions.

Design for the analyst, not the engineer. The success metric isn’t technical elegance. It’s whether the finance analyst stops building the same three-system Excel mashup every Monday morning. Talk to your users, watch them work, and build for their actual workflow rather than your ideal architecture.

Measure what matters. Track response accuracy, latency (P50 and P99), user adoption, and reduction in manual synthesis time. Track the questions the agent can’t answer, because those gaps are your roadmap for which stores to add or which schemas to extend.

What this enables at scale

This pattern does more than make existing workflows faster. It enables workflows that weren’t possible before, no matter how many analysts or engineers you threw at the problem. An agent querying a graph, a metrics store, and a document repository at once can surface patterns no human would find by checking each system manually. A finding like “entities in segment X who adopted product Y and had a support escalation in the last 90 days are 3x more likely to churn” requires reasoning across three data sources in a single inference; no dashboard surfaces that, and no analyst checks that specific combination unprompted. In the same motion it democratizes access, opening information that was previously reachable only by engineers who could write Cypher or SQL to anyone who can ask a question in plain language. The analyst’s role shifts from answering routine questions toward building the ontology, curating the graph, and tackling problems that require genuine human judgment.

It also delivers auditable reasoning at enterprise scale. GraphRAG provides deterministic inference paths that are verifiable against source data, so every conclusion traces back through explicit edges and nodes. This builds trust where decisions carry financial, regulatory, or strategic weight. “The AI said so” becomes “the data shows that A connects to B through X, B exhibits property Y, and historical pattern Z suggests the following.” That traceability changes the conversation from “Can we trust AI?” to a review of the work the AI actually did.

Conclusion

The future of enterprise analytics is unlikely to be a larger warehouse or a smarter dashboard. It is an orchestration layer capable of reasoning across specialized systems while preserving each system’s strengths. Purpose-built data stores remain exactly where they are; what changes is how we access them.

The architectural pattern described here doesn’t replace warehouses, graphs, vector stores, or APIs. It coordinates them. That distinction is subtle, but it fundamentally changes what enterprise analytics systems can deliver. The next generation of analytics platforms will do more than answer questions faster. They’ll reason across enterprise knowledge in ways that previously required experienced human analysts. The data already existed. The orchestration layer did not.

Disclaimer: The views and architectural perspectives in this article are entirely my own and do not represent my employer or any affiliated organization. References to patterns and technologies are based on publicly available information and personal experience. No proprietary or internal information was used.

For three years, the agent story has centered on capability: Models got better at planning, tool use, and long tasks. Integration improved more slowly. Every agent was still wired to its tools, data sources, and host application with custom glue, leaving even successful systems difficult to move or reuse. Standard interfaces change the economics of that work. Networking became an ecosystem when machines could agree on interfaces; programming tools followed the same path with the Language Server Protocol. In my judgment, the standardization now underway will influence what engineering teams ship over the next three years as much as further gains in model capability.

The Model Context Protocol (MCP) standardizes how an AI application connects to tools and context. It uses JSON-RPC messages between hosts, clients, and servers. Servers can expose tools that a model invokes, resources containing context and data, and prompts that describe templated workflows. The spec credits the Language Server Protocol as an inspiration. Language servers replaced a separate integration for every editor-language pair with one interface on each side; MCP applies the same idea to AI applications and integrations. The current spec is dated July 28, 2026, the fifth dated protocol revision since the project launched in November 2024. That pace reflects active governance and warns implementers to expect movement.

Adoption has spread across competing hosts. The MCP maintainers’ December 2025 announcement cited more than 97 million monthly SDK downloads, over 10,000 active servers, and first-class client support across ChatGPT, Claude, Cursor, Gemini, Microsoft Copilot, and Visual Studio Code. Any individual count can be debated. Cross-vendor implementation is harder to dismiss, because a shared interface becomes infrastructure when rivals support it independently.

MCP covers the boundary between an application and its tools. The Agent2Agent protocol (A2A) addresses the next boundary: agents discovering and communicating with one another across vendors. It reached v1.0 in April 2026 with more than 150 supporting organizations, signed Agent Cards for verifiable identity, SDKs in five languages, and general availability in Microsoft Copilot Studio, Azure AI Foundry, and Amazon Bedrock AgentCore. A2A is younger and less proven in production than MCP, but its governance and adoption are moving agent-to-agent communication beyond the confines of a single platform.

The emerging stack extends beyond those two protocols. AGENTS.md, OpenAI’s convention for giving coding agents repository-level instructions, joined MCP as a founding project of the Agentic AI Foundation. The OpenTelemetry generative AI semantic conventions are developing a shared vocabulary for traces and metrics from model and tool calls, though that work has yet to reach stable status. Connection, cooperation, instruction, and telemetry are beginning to acquire common interfaces.

For an engineering organization, the immediate consequence is reuse. An MCP server for an internal ticketing system can serve every compatible IDE, chat application, and agent. Models and hosts will keep changing; the interface contract, server implementation, schemas, and authorization wiring can endure. The integration becomes an asset shared across applications. Language servers followed the same path as editors came and went around them. Agent integrations deserve the same treatment as libraries and services, with owners, versions, tests, and upgrade policies.

Reuse also concentrates attention on a common security boundary. Custom integrations tend to receive separate reviews, when they receive them at all. A shared protocol creates a recognizable seam where teams can specify identity, permissions, data flow, and audit behavior, then apply those controls across many tools and hosts. The MCP authorization specification builds that seam from established IETF work, including OAuth 2.1, protected resource metadata, authorization server metadata, and resource indicators. It requires Proof Key for Code Exchange (PKCE) to protect authorization codes from interception. MCP servers must also reject tokens issued for another audience and must not forward them downstream. The value comes from familiarity: Teams can draw on years of OAuth deployment experience instead of rediscovering its failure modes inside a new protocol.

The spec is equally useful when it states what metadata cannot prove. Tools can carry behavioral annotations describing them as read-only, destructive, idempotent, or open-world. Those terms give hosts a vocabulary for policy. The tools specification still requires clients to treat annotations as untrusted unless they come from a trusted server. Self-description can inform a security decision; it cannot establish the trust on which that decision rests.

Once servers share an interface, they also become discoverable. The MCP Registry is an open catalog and API for public servers, designed to feed downstream marketplaces and private catalogs. Its role resembles npm or PyPI as a discovery mechanism, with an important difference: It stores standardized server metadata and leaves package distribution elsewhere. The registry remains in preview, with possible breaking changes and no durability guarantee.

Discovery brings familiar supply-chain risks to components with unusually powerful access. Typosquatting, abandoned packages, malicious updates, and uncertain provenance now concern software that may hold live credentials and act on production systems. The registry provides namespace verification and moderation, and its downstream model allows organizations to build curated catalogs. An internal subregistry or allowlist is therefore a sensible first control. The standard interface makes that curation practical across multiple hosts.

The same contract improves testing. MCP tools declare a JSON Schema for their inputs and may declare one for structured outputs. Under the current tools specification, servers that declare an output schema must return conforming structured results, and clients should validate them. Teams can test a server without putting a model in the loop, mock it with recorded or synthetic behavior, and contract-test both sides as they would a REST or gRPC boundary.

This separates two kinds of uncertainty that agent evaluations often mix together. Protocol tests can determine whether an integration exchanged valid messages and enforced its contract. Model evaluation can focus on whether the agent chose the right tool and interpreted the result well. OpenTelemetry’s emerging conventions extend this approach to runtime evidence by giving systems a common language for tool calls. Teams will still have to absorb changes as those conventions mature.

These benefits stop at the edge of what the interfaces describe. A schema captures the shape of a tool’s arguments, while its meaning still lives largely in a free-form description that a model must interpret. A schema cannot tell an agent when a tool is appropriate, how its effects interact with other tools, or whether two similarly named operations have equivalent semantics. Portability makes a server available across hosts; behavior can still vary across models and contexts.

The semantic gap leads to a trust gap. MCP can carry a server’s claims, and an A2A Agent Card can carry a digital signature, but a signature only ties a statement to an identity. Engineering organizations still need a basis for deciding which identities, publishers, code, and claims deserve authority. The protocol can carry that decision through a system. It does not make the decision for the organization.

Delegation makes the problem harder. MCP’s authorization model handles a client calling a server with an audience-bound token. Production systems increasingly involve an agent calling another agent, which calls a tool or a third agent. Preventing token passthrough closes a serious hole, yet each downstream hop still needs a narrower grant derived from the user’s original authority. No common mechanism defines how those rights should attenuate across an arbitrary chain. Platforms currently solve this locally or leave too much authority in place.

All of this work is unfolding on young infrastructure. The registry is in preview, the telemetry conventions are unstable, and MCP has produced five dated protocol revisions in less than two years. Revision is how standards mature, so teams should version-pin, keep protocol code behind thin internal adapters, and budget for migrations. Some abstractions will prove wrong because the standards and the underlying practice are developing at the same time.

A sensible response begins with ownership and containment. Each internal server needs a durable owner, and its protocol surface should be treated as an architecture decision. Schema validation, conformance tests, and protocol mocks put that seam under contract. Third-party servers belong behind an allowlist or private catalog, with provenance requirements and corroboration for their annotations. An internal abstraction around the emerging telemetry vocabulary can limit exposure to draft revisions.

Organizations with a large stake in agents should also participate in the standards work. The Agentic AI Foundation and the A2A project are young enough that engineering teams outside the founding companies can still influence what becomes portable, observable, and enforceable. The rules written now will become assumptions embedded in future products.

Model improvements will continue to generate the headlines. Standards determine whether those models can participate in an ecosystem. Ethernet mattered because it became an interface that many implementers could agree on, and language servers mattered because editors and language tools no longer needed to be designed in pairs. Agent systems now have their first interfaces that competitors jointly govern and independently implement. Engineering teams should build on them with two expectations: The interfaces are likely to last, and their current forms will change.

The emergency route itself was fine. Production systems need emergency routes, and a human approved this one. The trouble is that the route now runs without a live incident, an owner, or an expiry condition. A controlled exception has quietly settled into the platform’s standard runtime behavior.

This article is about that execution layer: the temporary operating state that lingers in runtime behavior. Temporary authority is easy to grant under pressure and hard to retire once the pressure lifts. An enterprise agent shouldn’t have to guess from prompts or conversation history whether it’s running under normal, incident, or recovery conditions. That state should be handed to it as authoritative runtime input, the way platforms already hand over identity, tenant, environment, and permissions.

An exception is safe while the incident that justified it is live. It becomes dangerous the moment the platform can no longer show the incident has ended.

The problem is unmanaged exception state

Exception drift is what happens when temporary exception behavior outlives its authorized scope, authority, or duration, and emergency accommodations settle into normal execution. The drift is usually quiet: a routing rule that stays reachable, an approval shortcut that survives closure, a tool permission that keeps shaping execution after the triggering condition has passed.

Enterprises already have the human machinery for this. Incident management defines abnormal conditions, change control governs deviations from standard practice, and postincident reviews confirm that temporary measures have closed.¹,² The gap is architectural. Most agent platforms still treat organizational operating state as something outside the runtime rather than an input to it.

Once an accommodation proves useful, it fades into an invisible operating state. The routing rule stays enabled, the shortened approval path stays reachable, and the temporary queue keeps taking work. No dramatic model failure is required, only a platform with no reliable way to close runtime state.

The practical question to ask is “Which operating mode is active for this workflow, this user, this segment, at this moment?” When the answer is left implicit, every agent, workflow, and tool gateway invents its own. Discussions of agent architecture tend to dwell on capabilities such as models, tools, and orchestration, yet production behavior depends just as much on runtime context: identity, tenant, environment, permissions, session state, policy, and tool access. One piece of that context is usually absent. The runtime knows who is acting and what they may do but not whether the organization is under normal conditions, incident response, recovery review, or a declared exception. Exception drift begins in that blind spot.

The exception lifecycle

Organizations move through a predictable sequence: normal operations, incident declaration, a temporary exception window, closure review, and return to normal. Each stage answers a question, from why the exception exists and who authorized it to what evidence shows it’s no longer shaping execution. Most enterprises handle the front of that sequence well; they’re practiced at declaring incidents and authorizing workarounds. The hard part is the retirement: proving that the exception behavior actually disappeared.

Declaring an exception is loud. Retiring one is quiet, especially when the workaround improved throughput or helped the team recover faster. That asymmetry is where drift lives, because an incident can be closed on paper while emergency routing, override policies, or alternate workflows keep influencing execution. From the platform’s point of view, the lifecycle should close only when it can show that every exception path has been retired or formally adopted through change governance.

Why this matters more for AI agents

Agents raise the stakes because they act. Rather than sitting in a config file, they select tools, trigger workflows, coordinate with other agents, and adapt their paths at runtime. An accommodation introduced during an incident can spread through routing, tool use, approval paths, and downstream agents. A traditional exception stays legible in a runbook or workflow definition; an agent can carry the same exception along many paths at once, which makes it harder to find and retire.

For engineers, the missing layer is an authoritative operational-state context that arrives with the request rather than being inferred from it. Historical traces and retained memory can explain why an accommodation once existed. They should never decide whether it’s still authorized. Memory informs execution; operating mode governs it. And when the two disagree, authoritative runtime state wins.

A small example shows the shape of that context. It carries the minimum needed to bind an exception to a mode, a scope, an authority, an expiry, and a status:

{
  "mode": "incident",
  "exception_id": "INC-4721",
  "scope": {
    "segment": "premium-customers",
    "region": "us-east",
    "workflow": "customer-remediation"
  },
  "authority": "service-owner",
  "expires": "incident-close",
  "status": "active"
}

The pattern sits near familiar ones without matching them. Feature-flag platforms such as LaunchDarkly target behavior by context, RBAC governs what a principal may do, and tenancy metadata tells a service where a request belongs.³ Operating mode serves a different purpose. It doesn’t replace policy, permissions, or memory: Permissions determine who may act, and policies determine how they may act. Operating mode determines whether exception behavior is authorized at all. As a result, it acts as a higher-order governance constraint on agents, workflows, approvals, tools, and escalation paths. Operating mode draws its authority from the organization’s incident and change process and represents a governed state of the enterprise, consumed by the runtime.

A design pattern for exception-aware agents

Operating mode becomes actionable once the platform treats it as a first-class runtime construct. Most agent architectures already inject identity, permissions, tenant context, and policy into every request, and operating mode belongs in that set. Agents consume it as authoritative state after the organization declares the exception, scopes it, assigns authority, and sets an expiry, instead of reconstructing it from prompts or accumulated context.

That reframes the architectural question as whether the platform can guarantee that exception behavior is impossible outside an authorized operating mode. Emergency behavior exists because the platform enables it, and for no other reason. Table 1 shows the minimum contract that makes the boundary testable.

A workflow invocation receives that state alongside the user request, and orchestration, routing, and tool gateways read the same state. Figure 1 shows the shape.

Building an exception-aware control plane

The natural home for operating mode is an external control plane. Incident management platforms, maintenance window services, and change management workflows already hold authoritative operational state, and exception-aware architectures extend those signals into execution with explicit scope, authority, expiry, and closure semantics.⁴,⁵ Implementation will vary across organizations, but the principle holds: Exception state should be authoritative, observable, and externally managed, kept out of prompts and workflow definitions and away from agent memory.

Exception-aware execution

With explicit operating state, behavior shifts when system state shifts, and prompt wording stops being the lever. Under normal operations, agents run standard workflows, routing, and approvals. Under incident mode, the same workflows can expose scoped accommodations that the authorized mode makes available, such as expedited approvals, alternate routing, deferred reviews, or emergency runbooks.

Table 2’s payoff is testability. A workflow in normal mode should never reach an emergency path, and a workflow in incident mode should reach only the accommodations its scope, authority, and expiry allow. Governance becomes an enforceable runtime property the platform can check at execution time.

Return to the opening incident. With operating mode as runtime state, the emergency route opens only inside a declared exception: scoped to the affected segment and workflow, owned by the service owner, and stamped with an expiry tied to incident closure. When the incident closes, the mode returns to normal and the routing gate stops handing that route to the segment, ahead of any manual cleanup. Closure then runs as a check. The platform replays the exception’s scope against live routing, approval, tool, and queue configuration, and confirms that no path still resolves to the emergency behavior. No one has to remember to retire the route; it was bounded by state, and the platform can show it is gone.

A common operating model for multi-agent systems

The problem compounds across collaborating agents. Customer-facing, orchestration, and execution agents may share a workflow while disagreeing about state, so one keeps applying emergency routing after another has returned to standard controls. A shared operating state gives them a single governance boundary: The exception is represented once and read consistently everywhere. As agent ecosystems grow more autonomous, shared operational state matters as much as shared identity and authorization. Fragmented state produces fragmented accountability.

Making exception drift observable

The payoff is observability. These conditions have historically been hard to see because accommodations scatter across workflow definitions, approval policies, routing configurations, and tool permissions. Explicit operating state makes them measurable at runtime. Closure can trigger automated validation, and the platform can watch for residual exception behavior between closures. The revealing checks are direct: Is the exception’s routing path still reachable, do its temporary approvals or elevated permissions still resolve, and does any expired exception still touch behavior? Tracking how many exceptions are open, how long they stay open, and how often they harden into permanent change turns drift from an audit finding into a monitored signal.

From adaptive agents to governed systems

Table 3 captures the shift. Traditional governance documents exceptions and trusts that they are retired. An exception-aware platform represents them as runtime state that can be propagated, validated, monitored, and closed.

The test is whether the platform can demonstrate that the exception stayed bounded to its purpose and duration.

None of this demands a new governance model. Organizations already know how to govern identity, permissions, change, and operational risk, and the recommendation is to extend those disciplines to operating state. Once operating mode is authoritative, observable, and testable, drift becomes detectable, closure verifiable, and recovery an engineering result. The aim is a platform that can make its own operating conditions explicit, hold every agent to them, and demonstrate a clean return to normal. In that model, temporary accommodations stay temporary, governance lives in the architecture, and enterprise agents earn trust at scale.

References

1. ISO, “ISO 22301:2019: Security and resilience—Business continuity management systems—Requirements,” International Organization for Standardization, 2019. https://www.iso.org/standard/75106.html ︎
2. Axelos, “ITIL 4: Change enablement,” IT service management practice, 2019. https://www.axelos.com ︎
3. LaunchDarkly, “Target with Flags,” LaunchDarkly Documentation. https://launchdarkly.com/docs/home/flags/target ︎
4. ServiceNow, “Change Management,” ServiceNow Product Documentation. https://docs.servicenow.com ︎
5. PagerDuty, “Maintenance Windows,” PagerDuty Support Documentation. https://support.pagerduty.com/main/docs/maintenance-windows ︎

For most of the history of technical publishing, editing and verification were the same process, and that process was slow and expensive. Getting a book out took years. We found an author, vetted them, had them work with an editor, and checked their claims with technical reviewers. A lot of that time went into separating what was correct and useful from what was confusing or only sounded right. It was laborious, but it meant a reader could depend on the claims on the page. The credibility of the book, and of the publisher behind it, mattered as much as the information itself.

When the cost of production drops to zero, that credibility becomes worth more, not less. Content is easier to make than ever, but without a transparent process behind it, readers have no idea where the knowledge came from or whether it holds up. As Jasmine Sun puts it in “The Independent Writer’s Advantage in the Age of AI,” “Trust is not about information and its quality alone. It’s about the messenger. It’s about who says it and their track record and what they’ve told me before.” A practitioner has confidence in a source because someone she respects has put their reputation on the line for it. They believe what the author is saying because the publisher has a history of being right and of correcting itself when it isn’t, and because the work is attributed and verifiable.

The corpus matters, but it’s the assurances around it that are hard to replicate, and that comes not just from the people who produce the content but from the people whose judgment vouches for it. Sometimes a creator brings their own credibility with them. Other times, the publisher spots someone unknown and lends them its own. The art critic Dave Hickey said this about gallery owners in Air Guitar: They gain status from the famous artists they represent and share it with emerging talent who have something to offer but who haven’t had the chance to earn a reputation. This is what O’Reilly has done for nearly half a century, build a network of experts who vouch for what’s worth knowing.

Expertise is alive, and it compounds

Expertise is a living thing, continuously expanding. Content starts to decay the moment it’s published, because frameworks evolve, libraries deprecate, and yesterday’s best practice becomes today’s security incident. Keeping expertise alive requires a pipeline of people who stay current and an editorial layer that notices when something has gone stale, and either retires it or calls for a fix.

That pipeline isn’t something you switch on when an author has a book to ship. At O’Reilly, we’ve always prided ourselves on living at the bleeding edge, finding what Tim O’Reilly calls “the alpha geeks” and spreading what they know to everyone else. Content sits at the center of our platform, but we think about it in pace layers. Some advice is timeless, some moves but has a long shelf life (some of our books are still in print after nearly 50 years!), and some changes weekly. We work with experts at each pace layer, capturing what lasts while doing our best to keep pace with an industry that seems to have changed every time we wake up. We have relationships with hundreds of the best practitioners in the world, and our job is to keep them engaged continuously, with quick takes when something breaks, structured responses when major research drops, and live sessions on emerging topics while they’re still emerging.

An institution doesn’t stamp trust onto content. In a technical community, trust is conferred in both directions. A practitioner earns standing because people who already have standing engage with her work, cite it, argue with it, and build on it. That insight was the whole idea behind PageRank, Google’s first great innovation. A page mattered because other pages that mattered linked to it. Reputation works the same way.

The audience isn’t just consuming reputation signals; it’s generating them. When a senior engineer whose judgment others respect says out loud that something is worth reading, she spends a little of her own credibility; the author gains a little; and everyone watching recalibrates whom to trust next time. O’Reilly plugs into that existing economy of reputation. When we put our mark on someone’s work, we aren’t the sole source of its credibility. We’re amplifying a judgment the community is already making and adding our own track record to it. The reader who finds it reliable hands status back to the source.

When the readers are machines

Human practitioners aren’t the only ones who need trusted engineering knowledge. The AI systems now sitting in every workflow, the coding and debugging agents and architecture advisors, need it just as badly since most of them are built on scraped web data and documentation that was stale before it was ever indexed. They’re fluent, but they’re wrong often enough that you can’t just take their word for it.

The stakes grow with AI increasingly being used to generate not just provably correct types of content like code, which either works or it doesn’t, but persuasive documents in fuzzier areas like hiring, strategy, and so on. Like everyone else leaning on these tools, we at O’Reilly are reckoning with the consequences of the ability to talk to a model and get back something that looks smart at a glance. A few rounds in, the slop is still there. In the last few months, maybe 10 times as many documents have crossed our desks, from new product ideas to strategic plans and proposals. But the ease of generating the text hides the fact that either the model or the person prompting it doesn’t actually know what they’re talking about. Knowledge workers need ways to ground their work in insights from human experts, particularly when that work is AI-assisted. So we’re building tools that let agents draw on our repository of expertise to support their proposed decisions.

Credible sources are particularly important when thinking through and justifying important choices. Our CTO, Andrew Odewahn, describes the shift this way: “18 months ago, it was all about how to get engineers to be more productive, but now it’s about how to get organizations to make better decisions. The engineering tasks are moving away from coding output to planning.” For planning tasks like comparing implementation approaches, you need expert-over-your-shoulder guidance for contextual decision-making. You can’t just rely on an LLM’s best guess to solve your problem, which is why we see great opportunity for new products like O’Reilly’s Expert Intelligence offering that delivers grounded knowledge embedded in your AI tools and your workflows to help navigate what you do. Trust is foundational because the expertise behind it stays genuine, practical, and human.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.

Takeaways

01.26 The usability gap is smaller than the marketing gap. The capabilities of local models are only a few months behind those from the big commercial companies, but because there’s no subscription revenue model behind local models, they often go unpromoted. “It’s very hard to make money off local models,” Pete explains, so the big companies aren’t focused on selling them. “Every company is going to go for the [product] that has an easy subscription revenue model. And that means you have a massive ton of marketing around all of these tools that are kind of like, ‘Oh, let’s have a little text box on a website.’ And so it means mostly that people have never heard of these local models.”

04.20 Local models are already good enough for most use cases. Pete compares the moment to the early web, when free alternatives like Apache eventually overtook expensive commercial servers. “All of these alternatives, once people actually had time to look around and they had a little bit of time to improve, they just wiped the floor with the commercial [offerings],” he points out. “I don’t know if we’re going to quite get there, but that’s the kind of pattern that I’m seeing.”

07.26 “The hardware barriers are a lot lower than people think.” Ben and Pete discuss what hardware you actually need to get up and running, from parameter counts, quantization (Q4, 8-bit), and VRAM requirements to the new Apple M5 Studio’s unified memory as a way to run very large models locally at usable speed. “The key thing is whether you can fit [your model] into your graphics card’s memory,” Pete says. “So with weight quantization, 9 billion [parameters] if it was 8 bits is like 9 GB. A lot of mid-end decent laptops that are shipping now have more than that.”

18.33 “It’s not that people don’t like voice interfaces. It’s that people don’t like bad voice interfaces.” We’ve solved most of the big problems, like dealing with background noise, phrasing, and speech in a range of accents—or at least have improved tools’ capabilities. However, “there’s no commercial incentive to kind of pull them all together,” Pete says. Most tools feel like they haven’t caught up to the LLM era, but “open source can be a really strong lever” to updating them, argues Pete.

28.26 We’re navigating the split between “LLM maximalist” end-to-end models (favored by big AI companies with the most capital) and the “compound AI” approach of chaining together specialized models from different sources. “If the future is end-to-end models, then only the people with the most money can actually build and train them,” Pete notes. Compound AI lets you “actually train all of the models independently” to accomplish your particular goals. While the performance of end-to-end models continues to improve, especially for multimodal models like Qwen or Gemma, using one can be a bit like choosing a Swiss Army knife over a tool specially designed to accomplish a single specific task, to use Pete’s metaphor. It may get the job done, but it’s probably not the most effective way to do it.

36.10 Voice capabilities in the browser could be a game changer. Embedding a model directly in the browser—Chrome has a built-in ~4B parameter model that’s accessible from any website via JavaScript, for instance—makes it part of the operating system. “Once you are able to transcribe fast and accurately in the browser, it’s a way for people to easily start experimenting with this stuff,” Pete explains. Could this be an iPhone moment for LLMs?

39:58 The “gravitational pull” is toward on-prem. Unlike most recent technological advances that depend on the cloud to function, LLMs are well-suited to running locally, even with no internet connectivity. Enterprises are grappling with concerns about cost, privacy, capabilities changing with no notice, or even the models they depend on disappearing. Hosting your own model, whether on your laptop or in your corporate infrastructure, gives you the stability to plan for the long term.

44:21 GPUs are fantastic for training but “complete overkill for inference.” Pete likens it to “trying to use an oil tanker to go and do your shopping.” Memory bandwidth is the real limiting factor, and it’s a problem that companies like Apple, with its new chip designs and unified memory bandwidth, are working on solving. “Even if you’re running on the CPU, if you have something that’s got high-enough bandwidth to pull 27 billion weights in a fraction of a second, then the rest of it is fairly easy in terms of actually doing the processing,” Pete says. “I think we’re going to see a lot of really imaginative solutions now that people understand what the workload looks like.”

The following article originally appeared on Addy Osmani’s blog and is being republished here with the author’s permission.

In the past year, the conversation around agentic engineering has moved to harnesses and loops, fleets and software factories. My 2 cents is engineers need to own the outer loop—the accountability for these systems. This only gets more true as powerful models like Fable and GPT-5.6 become available.

Agents have leverage, and leverage creates obligations. Someone must be able to explain exactly what changed, why it was safe, and what will happen if they’re wrong. Otherwise, their actions can’t be justified. Which makes it unlikely their organization will ask for them in the first place.

And so I want to talk about three terms. The first, Quality, refers to all the checks we install before we let the system loose. Those checks produce evidence, and from that evidence we derive a Verdict.

The second, Verdict, refers to the final decision we make before work enters our dependent system: I’m the line-producer of this content. I run the team whose work is shipped under my name. The model may write the line, but the Verdict is mine. The work of my team will not enter our dependent systems without my decision. A Verdict is the production decision: Should we ship, block, redirect, narrow the response, add a guardrail, or reject outright?

The third, Answerability, refers to the guarantee that if someone asks, I can explain why.

To say this another way: Our agent (which I define as a model plus a harness of files, tools, memory, skills, sandboxes, permissions, observability, and recovery) is what runs our loop (which I define as investigation, implementation, verification, and repeat). And it’s what creates our software factory.

The model is just the engine. The harness—tools, memory, permissions, sandboxes, tests—is the car you build around it so it can do real work safely.

The loop is how one good run becomes a process you can trust to run again. Wrap that harness in a repeatable cycle—investigate, implement, verify, repeat—where an independent check, not the model’s own say-so, decides when the work is done.

Now run many loops at once. A factory is loops at scale: The agents ship the work inside, while humans own the decisions at the boundary.

And at the heart of that factory is a careful boundary between what’s inside the system and what’s outside it. Inside the system we collect inputs (from the product team’s intent, or knowledge of previously shipped work, or of recent incidents, or of specific feedback from users). The agent loop investigates the task, implements a plan, and verifies the result. Then, evidence crosses that boundary. A human, who owns the dependent system, sees the evidence and decides whether to proceed.

And that, friends, is the shift we’re trying to make. Before, our agents were doing the inner loop of the execution loop. Now they run the inner execution loop. Engineers own the outer loop.

Inside the system, there’s really just one kind of thing our agents are doing: capability. The capability to investigate tasks, implement plans, test their results, and report back. That’s the capability of a model. And as we’ve said, that future is already here.

Outside the system, there’s a single kind of thing: agency. The agency to decide, verify, approve, and own.

We’re still talking about code, you see. It just needs to live in a place and be performed by people who know what they’re doing.

The potential for AI code is no longer marginal. In a Sonar 2026 survey, we asked teams about the share of their commits that were AI-assisted. It was small but nontrivial. And several of the respondents said they expect the share of AI-assisted commits to grow substantially.

Sonar’s 2026 State of Code report found that 42% of committed code was AI-generated or significantly AI-assisted, with expectations for that share to keep growing rather than plateauing.

Creation, in other words, is getting cheaper. Scarcer resources are review, validation, understanding, and maintenance.

We moved the speed of generation faster than we moved the speed of control, and so we have a trust-verification gap. A lot of people we talk to still express some degree of distrust in AI code. Yet fewer of them seem to consistently build that distrust into their verification processes.

That’s a dangerous place to be. We’re going to need cheaper, clearer ways to verify the trustworthiness of AI code.

If you look at the GitLab June 2026 report, you’ll see that governance questions have shifted.

GitLab’s June 2026 AI accountability research shows that review and validation are the current bottlenecks when using AI and, more worryingly, that governance usually happens after code creation, after we’ve accepted the risk and lost control over ownership. Today, it’s not just about control. It’s about what constraints we set on the system. It’s about how we’ll check the work with evidence, and how we’ll hold teams accountable. It’s about who will own what part of the AI lifecycle.

So the final distinction in this series is between process and quality. Quality is the concept of backpressure. We mean it literally. We don’t want to grant our agents as much autonomy as they can possibly exercise. We want to grant them just enough autonomy that we have enough backpressure to stop them, regulate them, check their work, and ensure our humanity.

Ordinary engineering holds up a lot of signals that indicate that the work being done is doing the right thing. Type checks, tests, hooks, sandbox limits, audit logs, monitors. Our engineering systems are full of these kinds of signals, and they’re designed to provide enough backpressure to keep the system honest.

And so as long as our agents are emitting these same signals, we can trust our ordinary engineering to provide appropriate backpressure.

Trusting our systems doesn’t mean we don’t want a human in the loop. It just means that the human doesn’t need to be in the inner loop. We want them in the constraints loop (What inputs, architectures, instructions, or invariants should we set?), the sampling loop (How much output should we sample and review?), the audit loop (What evidence should we keep, and how do we make sure our audit log is effective?), and the ownership loop (What part of the production boundary should we own?).

But the human doesn’t need to be in the inner loop.

The agent can ship more than you can review.

And the scarce resource is your own core human judgment, informed by quality signals like logs or tests.

The AI June 2026 report shows that, in the experimental setting, agentic delegation along hour-scale time horizons is essentially here. The work by OpenAI this year on agents and the future of work was a great source for these ideas. So we need to start thinking about how to establish this ownership boundary, as our systems start shipping more than we can review.

And that’s where the answerability comes in.

Because with long-horizon agents, the decisions made over hour-scale time horizons are just that—decisions. And not all the decisions are going to be recorded. You can’t trace them all back to input tokens. If all you’re doing is trusting that the output you get is the correct choice for the problem at hand, the hundreds or even thousands of human hours of work you’re going to need to reconstruct the chain of decisions that lead to it become impossible. And so, again, answerability becomes something that must be at the core of our system design.

Three hidden costs

And there are three hidden costs:

Cognitive surrender ~ blindly accepting what AI gives you. When you delegate work to an agent, the work itself may appear to be the work of the agent. But it’s actually your work. It’s your reputation. It’s your responsibility. And it’s your software that suffered the defects in the output. And it’s your software that needs to be changed to reflect that output. So the agent’s output becomes your answer. And with it comes all the accountability. The Wharton study that put this together is reassuring when the AI is right. But when it’s wrong, the news isn’t great. When the AI was wrong, nearly three-quarters of people accepted it anyway, and felt more confident than they would have without the AI.

Cognitive debt ~ erosion of your understanding and memory of how to solve problems. When you delegate work to an agent, you’re offloading all the thought work to the agent. And while thinking it all out yourself takes time and energy, thinking it out on a massive codebase takes resources that aren’t available when you’re trying to run up the learning curve. So the output you get is often unattainable by you. And the longer the time horizon of the agentic planning, the bigger the gap between the code the agent produces and your understanding of it becomes. The gap compounds. The debt accumulates. And the cost of climbing the learning curve grows almost exponentially.

There’s a randomized controlled trial from Anthropic looking at whether engineers who lean on AI to write code understand it as well as engineers who write it themselves. The conclusion was gloomy: On a comprehension quiz, the engineers who worked through AI scored 17 percentage points lower than those who didn’t, 50% versus 67%.

And then there’s the orchestration tax: It’s easy to spin up lots of agents now, but your cognitive bandwidth doesn’t parallelize in the same way. Steering your agent away from the worst behaviors, sorting the work the agent produces to identify the ones that need your attention, directing it to focus on the work you care about first, verifying your most important constraints and your most dangerous assumptions before you let it run. . .

All of that takes work, and it can’t be automated. There’s no substitute for human judgment.

Brownfield systems are especially dangerous here, because the system behavior you have to audit doesn’t live in the code. It lives in the scars.

Fixes? Make attention the priority in your architectural decisions. Use worktrees, scopes, and evidence to reduce the coupling between your initial plan and the work that emerges from it. Time-box the effort to resolve unactionable steps. And make change in your software strictly an opt-in permission.

Alpha, decay, and taste: These are the three core patterns that shape careers and performances across domains.

Alpha is the lead part taken up by the highest achiever in the competition, when you’re playing your highest-value game move. Decays are established patterns that everyone learns through repetition and watching others (plateaus, if you like). Taste is the earliest we can sense the lead in an alpha or the change in a decay. It’s our judgment of what’s coming before we have any evidence that anything is happening.

Paul Graham’s point is that when anyone can make anything, choosing what to make matters more, and Mitchell Hashimoto’s definition is the operational one: making high-quality qualitative judgments where no objective metric exists yet. From now on, taste drives everything. Alpha shifts are taste changes. And decays fade out because we start to taste something different.

Next step? Operationalize your taste. How? Give it a name that reflects what you’re trying to move from limbic to conscious. Practice it in critique and examples. Make its rationale explicit.

And keep making the move that delivers the most durable competitive advantage in your industry. What’s that? Keep moving the edge up from just doing the task to teaching it, systematizing it, deciding when it should be done, and owning the result.

Everyone is a developer, but not everyone is an engineer. Engineering is what a developer turns into when they embrace a work discipline that is more strict: thorough and logically sound reasoning, consideration of constraints and tradeoffs, recognition of risk and exposure, and practical accountability.

In the future, people will leave the administrative work of engineering and embrace new roles that emerge as engineering becomes more demanding. Roles that are unbundled from the spirit of craft but make clear what each person does. There will be those who prototype. Those who build. Those who sweep. Those who grow. Those who maintain.

The humans hold the edge of the system in the other direction too. Increasing the alpha: choosing what is worth doing, defining the constraints within which it should be done, deciding if the evidence is sufficient to proceed, and caring for the result. Whether it’s a single team or a hundred teams, this is the edge that only humans can hold.

Accountability will scale the factory. Like attention and taste, accountability is also one of the three dualities that makes everything work. Without accountability, there are no rules. No wrangling with questioners. No trade-offs. No risks. No safety nets. If nobody owns the consequence of a decision, then high agency can only bring chaos.

The half-life of an edge is one release, but the half-life of a signature is a career. A signature is your name on the work, such that you feel you can stand behind what was shipped. Skills get you leverage; accountability turns leverage into trust.

Only people can choose. Only people inherit consequence. Agents can be asked to choose, route, merge, and escalate safely inside a policy, but they cannot inherit the consequences.

Every codebase should perhaps come with some kind of accountability contract that explicitly states the checklist that was understood when the change was accepted, the evidence that went into the decision, who was accountable for the change, and the system status after the change was blocked. Just like:

• Your attention and taste
• Your evidence, verdict, and ownership
• Your alpha, decay, and taste

High agency

In a typical agentic workflow, high agency is the art of knowing when to delegate, when to inspect, when to stop, and when to own the result of a process. The ladder of agency runs from low to high: flag a potential problem, investigate it, execute against it, diagnose it, propose solutions, recommend fixes, and resolve the issue. A high rung on the agency ladder is discernment: found it, it’s not worth fixing, moving on.

The 12 pillars that hold up the software factory

Brownfield is the frontier for factories that hope to scale. All those clever little innovations may not feel like much yet, but the production environment is a lot. When building an entirely new system, it’s much easier to plan and implement sufficient backpressure mechanisms because you have full control. When you’re adding intelligent agents to a legacy system, however, it’s another matter entirely.

Legacy systems include the entirety of production behavior, future expectations from customers, migration histories, release and budget cycle durations, unspoken assumptions, edge cases, data weirdness, runbook procedurals, and all the scars that accumulated without the will to care for the system.

To be a steward of brownfield requires a form of durable engineering. Work has to be done to turn implicit knowledge into explicit constraints, keep it coherent across teams and through generations, formalize that knowledge into test procedures and functional specifications, and tie that knowledge to objective evidence. All while ratcheting failure into more learning. Because if the system doesn’t get the care it has always received, everything will come crashing down.

New work is real work

The work will get more interesting as you scale. Because when everything else is built, people will want to build new things. They’ll want to employ the alpha and taste they have developed through their craft to design new loops that can be grafted onto the software factory. Or they’ll want to build greenfield systems that employ all the knowledge of the software factory to one elegant, well-meaning, principled effort. They’ll want to design and implement new forms of evidence that will rise to the level of verification for the new systems. They’ll want to take care of brownfield systems that are now so complex they need dedicated attention. They’ll want to design and manage new backpressure mechanisms. They’ll want to design new agents. And they’ll want to build agency.

And, as they do, they’ll come to see that all this is real work. That’s a good thing.

Automation creates bottlenecks. Bottlenecks in production that are worth owning. Because automation gives us control over industrial scale. But there’s also new bottlenecks that arise from industrial scale. The bottleneck moves from “Can we build this?” to “Should this exist? Can we answer for it?”

What I’m suggesting is a practical operating model for scaling agentic engineering. There’s inner and outer loops. The inner loop is where the work is done. Loops are designed to be as independent as possible. Put all quality assurances and verification inside the loop. Once you’ve designed and validated the loop itself, the only thing you have left to do is to grant autonomy by putting in place a back-pressure mechanism that acts to control the rate at which the loop is run and its scope of operation. And put humans in their rightful place, on the right decisions. Don’t treat understanding as a hand-off or a release gate but rather as a point of decision where humans are primed to provide their insight. And then for every artifact that exists and is fed back into production and into new teams and engineers, leave behind better artifacts.

Build the factory; keep the lights on; make work legible, verifiable, owned.

An agent can write it. But before it reaches users, someone must explain why it should exist, why it’s safe enough to be part of production, and what they will do when it is wrong.

That’s agentic engineering at the outer loop—that’s the work now.

The following article originally appeared on Sean Goedecke’s blog and is being republished here with the author’s permission.

In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet. Today, everyone can write sort-of-okay CSS by delegating the task to an LLM. LLMs make everybody into a generalist.

Because of this, lots of people don’t think there’s any skill involved in working with LLMs. If you want the product that LLMs can deliver—PhD-level mathematics, pretty good but sometimes tasteless computer code, or awkward LinkedIn-style writing—you can simply ask for it. Since everyone is talking to the same models, “skilled prompters” are getting the same results as people touching LLMs for the first time.

This is wrong. The most important skill in prompting is expertise in the domain you’re prompting for.

A good illustration of this is Terence Tao’s conversation with ChatGPT about the recently discovered counterexample to the Jacobian conjecture. This is not the same ChatGPT I talk to! I couldn’t get to where Tao gets, even with unlimited tokens to burn.

There’s a lot to learn about good prompting from Tao’s conversation. Here are a few observations:

• Tao’s messages are very short and to the point. He doesn’t respond point by point to the model, just to the gist.
• The model outputs are much more concise than when I try to talk to GPT-5.6 Sol about mathematics. By signalling expertise, Tao shunts the model into “talking-to-mathematicians” mode, not “explaining-to-amateurs” mode.
• Tao pushes back when the model’s responses look wrong, but he doesn’t directly contradict; instead, he says things like “This looks more complex than I was hoping for.”
• Tao makes several leaps and suggestions himself. He almost never takes the model’s advice about where to go next.

However, you can’t prompt like Tao on mathematical questions just by following these tips. The key to his technique is actually understanding the mathematics: pulling the relevant idea out of ChatGPT’s multiparagraph response, suggesting alternate approaches or formulations, and identifying what “looks weird.”

Terence Tao is a better mathematician than I am a programmer. But the idea here—that domain knowledge makes you better at using LLMs—is something I’ve also experienced in my own work. If you have a good theory of your codebase, you can push the LLM much harder than if you have no familiarity. Because you have your own sense of what a good solution might look like, you can say, “No, I think it could be simpler here” or “But don’t we already do X?” or “Can we express this problem in these familiar terms?”

This touches on an idea I’ve written about before: that system design problems are dominated by concrete specifics, not generic principles. Of course both are useful, but I’d rather have familiarity with the codebase than a deep general understanding of software systems. In his conversation, Terence Tao asks a lot of specific questions like “Does X work here?” or “Given Y and Z, why A?” I can’t ask those questions about the Jacobian conjecture, but I can ask them about the systems I own at GitHub.

If you have no domain knowledge, you can cling onto the LLM to at least get something. That’s not bad! But if you have domain knowledge, you can wring far more value out of the same LLM by steering it hard in the direction you want. Most of us will have to do a mix of both these approaches, since we have domain knowledge in some areas but not others.

The usefulness of domain knowledge suggests that human expertise will continue to be useful even as models get stronger. For many tasks, the human is the bottleneck, not the model, because the difficult part is in communicating to the model exactly what kind of solution the human wants. The information is “in the model” already, but it takes a very smart human to pull it out.

This post got many comments on Hacker News. Some commenters share their anecdotes about how expertise has helped and lack of expertise has hurt. Other commenters say it’s plausible, but they have a sensible suspicion of a view that’s reassuring them about how they’re still valuable. I agree with that, though I suspect by the time we get around to studying this, the landscape will have changed under our feet again. Some commenters point out that OpenAI’s math prompts were inexpert, and so expertise isn’t required. Here I’d respond that OpenAI does have a team of expert mathematicians that checked and filtered the model’s suggested discoveries, and that you cannot currently skip that step.

In this episode of Zero to Agent in 30 Minutes, Microsoft senior solution engineer Maxim Salnikov walked through the Agent Package Manager (APM), a terminal-driven open source product from Microsoft that treats agent context the way modern software already treats its dependencies: versioning it, pinning it, and checking it before it ships. It’s a technical session, but rather than building an AI agent, you’ll discover how to manage all the customizations you’ve installed for your agents and make them portable, secure, and governed by policies you or your company define.

How to build a supply chain for agent context

Maxim demoed the process of setting up and using APM step-by-step. Here’s how it works.

1. Keep every customization in your codebase. Agent skills, custom instructions, and hooks tend to spread across a developer’s machine, a harness’s install folder, and enterprise-level defaults. Maxim recommends storing them in the codebase instead so that anyone on a new machine or using a new harness ends up with the same setup as everyone else. Yes, your codebase will be bigger, but the trade-off for consistency is worth it. But be sure that skills and other customizations are secure before adding them to your codebase. Consider building your very own enterprise repository of the customizations you’ve identified as safe to use. (See step 5.)
2. Install APM and initialize a project. APM works like npm for AI agent customizations. Running apm init sets up an apm.yaml file targeting one or more harnesses, such as GitHub Copilot, Claude Code, or Cursor, and apm install pulls a skill from a repository into the right location for each one. When you install a skill, APM also creates a log file documenting the entire resolution history.
3. Pin to a version or a commit hash. Once a skill is installed, pin it the way you’d pin a library dependency. Doing so protects against a scenario where a legitimate skill later gets compromised and a bad actor slips malicious content into a new version under the same name. APM also scans files for hidden Unicode characters that may contain malicious instructions.
4. Reproduce the exact setup with a lock file. APM logs every file and hash it installs. Running apm install --frozen rebuilds that exact environment from the log instead of reresolving everything from apm.yaml, so a teammate’s machine ends up with precisely the same setup as yours.
5. Source customizations from a vetted registry, and set a policy for it. Public repositories work for experimentation, but Maxim suggests enterprises build a private, gated registry of reviewed skills. APM policy files enforce which sources are approved, so a customization from an unvetted location gets flagged.
6. Audit for drift, then enforce it in CI. The apm audit command checks installed customizations against policy and catches unauthorized sources or content that has changed since installation. Run that same audit as a gate in a CI/CD pipeline to protect the entire organization against skill drift and bad actors.

The software supply chain already has decades of tooling behind it. That discipline hasn’t caught up with agentic AI, but APM is attempting to close that gap. Explore the project GitHub repo and get started.

Coming up next

On September 9, Menyala’s Sajal Sharma joins Zero to Agent in 30 Minutes to build a shared knowledge base that acts as a common brain across agents. He’ll show how a single repository of research, daily logs, and notes can give Claude Code, Codex, OpenClaw, and Hermes access to the same accumulated information instead of starting from zero with every new session.

Follow along with Zero to Agent in 30 Minutes on Radar, or watch the latest episode on YouTube, Spotify, Apple, or wherever you get your podcasts. If you’re an O’Reilly member, you can watch live. Save your seat.