She returns a clean flowchart of the happy path. The drawing is accurate and may even be elegant. It leaves out the thresholds she watches, the conditions that make the standard procedure unsafe, and the supplier whose parts fail in humid weather. She doesn’t think of those judgments as separate knowledge. After years on the job, they feel obvious.

Six months later, a production line goes down and the knowledge base can’t explain what to do. The interview took place as per the process. Its transcript was chunked, embedded, and indexed, so the search returns the relevant passage quickly. The passage still can’t answer the question because no one asked the engineer to explain the judgment behind the procedure.

That gap now limits many enterprise AI programs. Organizations continue to improve retrieval over collections that omit some of their most valuable operating knowledge. Better ranking can help people find what was recorded; it can’t recover the expertise that never entered the collection.

The blind spot in enterprise knowledge systems

Michael Polanyi gave the problem its durable formulation in 1966: “We can know more than we can tell.” In The Tacit Dimension, he argued that competence depends on skill, perception, and judgment that resist full explanation, even when an expert sincerely tries to teach them.

In companies, tacit knowledge usually appears in three forms. Elicitable knowledge remains unspoken because nobody has asked a precise enough question, or because an expert assumes that everyone sees what she sees. Perceptual knowledge lives in trained attention: An engineer hears a bearing begin to fail, or a nurse notices that a patient looks wrong before a monitor changes. Collective knowledge resides in a team’s habits, standards, and shared sense of what a sound decision looks like in that organization. Each form requires a different method of transfer.

Preventive judgment creates another difficulty for the architect. A failure produces a ticket, an incident report, and a trail of messages. An experienced operator who quietly avoids a known failure mode on a Friday afternoon produces none of those records. The useful outcome is the absence of an event, so the data pipeline receives no trace of the decision that produced it.

Machine learning can infer rules that people struggle to articulate, provided the model sees enough representative examples. It’s difficult to find enough examples of rare expertise for training. A company may have only a handful of unusual incidents and one person who has learned, over decades, how to read them.

David Autor described this limit as “Polanyi’s paradox”: Many of the tasks that are hardest to automate depend on rules we can’t state. Modern machine learning works around the paradox by learning from examples, but the workaround weakens when examples are scarce. Fine-tuning can teach a model the company’s vocabulary and document formats. It can’t reconstruct decisions that left no data.

At the same time, the economics have changed. Much of a field’s documented best practice now appears in frontier-model training data and is available to competitors at roughly the same price and quality. The more widely explicit knowledge circulates, the more a company’s advantage depends on local judgment: the exceptions, thresholds, relationships, and practiced responses that its people have accumulated.

That makes elicitation an architectural concern rather than an offboarding chore. The organization needs a repeatable way to surface the knowledge that can be expressed, a route for the expertise that must be demonstrated, and enough humility to distinguish the two.

A protocol for elicitation

The central design question is straightforward: Which follow-up would prompt an expert to say the missing judgment aloud? The quality of the interview sets the ceiling for the knowledge base. The index determines how quickly someone can reach the resulting material.

Interviews can be made more reliable even though judgment itself remains highly personal. An expert may know that a particular supplier fails in humid weather. The interviewing protocol doesn’t need to possess that knowledge in advance; it needs to notice a phrase such as “we escalate if it looks bad” and ask the expert to define “bad” in observable terms.

Expert explanations tend to become vague in four places. An effective interview protocol asks targeted questions about each one:

• Thresholds: Which number, reading, or condition triggers the action?
• Exceptions: When does the documented procedure cease to apply?
• Evidence: What did the expert observe before reaching the conclusion?
• Escalation: Who becomes involved, and at what point?

These questions uncover the operational detail that runbooks often lack. They also identify a narrow, useful role for a language model during the interview: proposing the next question that turns a general statement into a usable rule. I’ve been building an open source toolkit, ExpertTrace, around that protocol.

The value appears in the difference between what an expert volunteers and what the same expert confirms after one focused follow-up. Consider a typical first answer:

We review high-risk use cases before deployment. If the risk seems significant, we escalate to the governance council.

The statement will embed cleanly and retrieve for a relevant query, but a new employee still cannot act on it. “Seems significant” supplies no decision criterion. A targeted follow-up produces something much more useful:

Escalation to the council is required when the use case touches employment, credit, or health decisions, or when model output reaches a customer without human review. Predeployment review is skipped for internal-only tools with no personal data, which is the exception people get wrong most often. If we cannot identify a named accountable owner, the review does not proceed, regardless of risk tier.

The second answer takes little additional time, yet it contains a decision rule, an exception, a recurring failure pattern, and a blocking condition. It can guide a real dispute instead of merely mentioning the subject.

The protocol needs guardrails. Limit the number of follow-ups; a long interrogation exhausts the expert and eventually produces agreeable noise. Keep the model focused on generating questions, and separate that task from compiling and validating the answers. An expert’s statement belongs in the record with its provenance and context. Whether the statement is accurate requires independent review.

The four-plane architecture

Elicitation is one part of a larger knowledge system. A tacit-aware architecture has four planes—capture, representation, serving, and transmission and each plane addresses a different failure in the movement of expertise. Figure 1 shows how the four planes work together and which forms of tacit knowledge each can reach.

In the capture plane, structured interviews, incident reconstruction, decision journals, and observation collect more than polished procedure. Record the trigger, evidence, exception, and escalation path while the expert can still explain the surrounding conditions. Route perceptual skill toward demonstration and practice instead of forcing it into prose.

Once knowledge has been captured, the representation plane preserves the distinctions that make the material trustworthy. A compliance policy, a war story, and an untested hypothesis shouldn’t become interchangeable chunks. Carry provenance, confidence, and validity context—including the plant, time period, equipment, and conditions—as first-class properties. Extend the knowledge graph beyond documents to the people and episodes that produced them.

The serving plane then determines how that knowledge reaches users. Answers should cite retrieved evidence and show the source. When the collection can’t answer, the system should say so clearly and route the question to someone with relevant experience. “Ask Joe; she rebuilt this line in 2023” is more useful than a fluent paragraph assembled from weak evidence, and the referral restores the human contact through which difficult knowledge often moves.

The transmission plane completes the architecture by helping how expertise moves between people through shadowing, teaching, and communities of practice. The platform should detect when knowledge concentration and attrition risk converge, then trigger capture and apprenticeship before a notice period begins.

Gabriel Szulanski examined 271 observations of 122 best-practice transfers across eight companies and found that even willing teams struggled to reproduce methods developed elsewhere in the same organization. The difficulty often began with causal ambiguity where people could describe the steps without fully understanding why they worked. Receiving teams also needed enough context and experience to absorb and apply what they learned. Preparation, coaching, and time helped them rebuild the practice in their own setting. A repository could preserve the record; the receiving teams still had to turn that record into working knowledge.

Evaluating the knowledge layer

Retrieval precision and answer faithfulness show how well a system serves its existing collection. They don’t reveal whether the collection contains the knowledge on which the organization actually depends. That question needs a separate evaluation loop tied to capture priorities and transfer outcomes. Figure 2 shows how the loop moves from offline evaluation to abstention calibration and then to transfer outcomes.

The evaluation begins with incident replay. Select 20 or 30 resolved incidents, remove the resolutions, and give the opening facts to the system. Ask the engineers who solved them to grade its responses. Compare those answers with responses from a frontier model that lacks access to the company’s collection. The gap reveals the generic-answer rate: how often the internal system merely restates public knowledge. If reviewers can’t tell the two sets apart, the pipeline adds little institutional value.

A bus-factor audit tests questions that only one or two employees can answer, and study how the system fails. A clear admission of uncertainty followed by a useful referral is healthy. Fluent boilerplate damages trust in every response, including the accurate ones.

Abstention calibration measures whether the system answers when evidence exists and declines when corpus can’t support an answer. Build a labeled set of answerable and unanswerable questions, then track abstention precision and recall as the collection grows. A system that never says “I don’t know” is unevaluated on the dimension that matters most.

Transfer outcomes complete the loop by measuring whether knowledge has reached the people who need it. Evidence of transfer appears in shorter time to proficiency, fewer repeat incidents after elicitation, and fewer critical responsibilities that depend on a single person. Document and query counts describe system activity; they don’t show whether someone else can now make the decision.

A strong knowledge system records what an expert said, preserves the conditions around the statement, and marks uncertainty. It also recognizes expertise that requires demonstration, apprenticeship, or team practice. Every evening, the people who carry that knowledge walk out the door. The architecture should be ready long before one gives notice.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

The essential message of the talk was on the first slide:

1. Prediction demands a world model
2. The world model is readable
3. The world model is at work in every token

How do we know this? As tokens pass through a model, particular patterns of activity appear in the intermediate states between its layers. These are called activations. Researchers can study which patterns show up when the model encounters particular ideas, and they can even intervene in those activations and see how the model’s behavior changes. (They do this by capturing the numerical state of the model’s computation in some area where they believe the activation shows a particular “meaning” and then replace the numbers with others.)

I went into the conversation thinking about how cool it is (and important too!) to explore what is going on inside the “mind” of a model. But in the end, I found it even more provocative to think about what studying LLMs might teach us about how our own minds work.

There’s at least some kind of analogue to what happens in the human brain. Emmanuel began by asking the audience to do a little next-token prediction themselves. He started with an easy one, a hypothetical exchange between two friends:

John: “Is the powder-blue suit too much?”
Nick: “Definitely not, man. Send it.”
John: “Okay, I’m going to tear it up on the _______________”

Most of us will fill in the blank at the end with “dance floor.” That’s a reminder that humans are also next-token predictors.

Then he gave an example that some humans will easily answer, but others without local knowledge might well fail at:

“We also have nature here, just a short bike ride away across the GG bridge. And we have world-class skiing about _______________”

Claude easily completes the thought with “three hours away.” To do that, Claude had to infer that “GG bridge” refers to the Golden Gate Bridge, that the speaker is therefore in San Francisco, and that “world-class skiing” probably refers to Lake Tahoe and then retrieve roughly how long it takes to get there.

The point of Emmanuel’s demonstration was that we have become so used to calling LLMs “next-token predictors” in a kind of dismissive way. But as Emmanuel put it, “To predict the next word well, you need a very complex world model.”

How you make a thing is not the same as what the thing becomes

Emmanuel pointed out that people often confuse how you make a thing with how the thing works. Yes, LLMs are trained with the seemingly simple objective of predicting the next token. From that, people may make the leap that what is going on inside must also be simple, something like a very large fuzzy lookup table. “But that’s not true,” Emmanuel said. Simple objectives can give rise to extraordinary complexity. Evolution is the canonical example. No one put “create Beethoven’s Ninth Symphony” or “understand quantum electrodynamics” into the instructions for a process driven by reproduction and selection, yet it eventually produced Beethoven and Feynman. As Emmanuel put it, humans have been “reproducing and killing each other for millions of years, and from that we got jobs—or this podcast.”

What Anthropic’s interpretability researchers are finding inside the models looks much less like fuzzy retrieval than many people imagine. They find millions of internal features corresponding to concepts. For example, features for “eyes” show up when the model encounters prose about eyes, an ASCII face, an SVG image, or a photograph. In other words, these features appear to be abstractions rather than merely associations with particular strings of tokens.

Similarly, a feature of the Golden Gate Bridge activates not just for English text about the Golden Gate Bridge but for references in other languages and for images of the bridge. Even more interestingly, researchers can manipulate these features. Turn the activation of the Golden Gate Bridge feature up strongly enough and ask Claude what its physical form is, and instead of saying that it is an AI without a physical body, it announces that its form is the Golden Gate Bridge. It isn’t just that some numbers happen to accompany activations about the Golden Gate Bridge. Changing those numbers changes what the model says it believes.

The way a model completes a task that requires thinking ahead also demonstrates a kind of internal world model. Ask Claude to write a rhyming couplet. Even though it emits only one token at a time, before it has written the second line, the activations already reveal the rhyme that it is aiming for. The choice of a word such as “rabbit” for a rhyme happens before the choice of the preceding words on the line, so the model can land there. We call it planning when a person does this. It doesn’t seem unreasonable to use the same word for what is going on here.

Perhaps most challenging to our preconceptions is that there are also features associated with emotions that aren’t activated just by words about those emotions, but by situations, images, characters, and more. These emotion features are even activated by the model’s own activities. For example, “frustration” may be activated when the model is unable to complete a task.

The map is not the territory

The issue of anthropomorphization came up during the audience Q&A. One participant objected:

“We should avoid attributing human qualities to LLMs by saying they think, intend, rhyme, or have emotions. Doing so encourages us to project human characteristics onto systems that do not possess them.”

I have sympathy with that warning. Old labels can prevent us from seeing something accurately. But a blanket prohibition against using familiar words can blind us too.

If you’ve followed my work for a long time, you know how much I’ve been shaped by the ideas of my early mentor George Simon, who in turn was deeply influenced by Alfred Korzybski and general semantics. Korzybski’s famous dictum was “The map is not the territory.” Simon (and Korzybski) taught me that language is a map of experience, which in turn is a set of responses to stimuli from some underlying external reality. The path from reality through experience to conceptual understanding is a very lossy process. The result can be a bad map that can blind us and lead us astray. When we encounter something genuinely new, we have to learn to notice when we are trying to force the territory to fit a map that no longer describes it. But a good map doesn’t just guide us along a route; it helps us notice things that might otherwise be invisible to us.

So yes, words like “thinking,” “planning,” “intention,” and “emotion” are labels derived from our experience as human beings. They may turn out to fit LLMs poorly. But if the shoe fits, perhaps we should let them wear it.

Emmanuel had a good response to the objection. He said, in effect, that anyone is welcome to propose more precise vocabulary. If it works—that is, if in my framing, it is a good map that helps people see the territory more clearly—people will come to use it. (An audience member later suggested that Emily Bender has done just that. But frankly, I find her suggested alternatives to be quite tortured, obscuring far more than they clarify. Even she admits they don’t work very well, though clinging to the need for them.)

In her analysis of the Hugging Face incident, Melanie Mitchell made some observations consistent with the nuanced approach suggested here. She wrote:

Metaphors can help us make sense of novel situations. For example, framing chatbots as “role-playing actors” has been helpful in understanding why these systems exhibit “lying” and “scheming” behavior. But inappropriate metaphors, like the narrative that “OpenAI lost control of escaping swarms of rogue agents,” can lead to ill-informed decisions about how to fix problems or set policy….It is essential for lawmakers, and the public, to understand that none of the reported incidents actually involved loss of control at any time, or arguably even “rogue agents,” or any kind of humanlike agency on the part of AI models. Instead, the blame lies with the humans who failed at engineering safe testing conditions, and who train AI models using RL methods that incentivize high persistence, autonomous decision-making, and reward hacking.

In short, all language is a map. Don’t judge it on that basis alone. Judge it on how well it helps us to see the shape of the territory.

How much of human thought is truly original?

Returning to my conversation with Emmanuel, he remarked that when an existing word really does provide the most precise description, perhaps “what should change isn’t our vocabulary, but our mental model of what these models are.” I replied that it should perhaps also change our mental model of what we are. Our encounter with machine intelligence should lead to a better understanding that parts of our own cognition are also mechanistic (albeit derived from a different underlying mechanism than that of LLMs) while other parts are, as yet, somehow perhaps something else.

In 1995, O’Reilly published a book that I remain extraordinarily proud of. Stephen Talbott’s The Future Does Not Compute: Transcending the Machines in Our Midst was decades ahead of its time. Its argument was not primarily about what computers would someday become. It was that when we think about machines as intelligent (and yes, we were thinking about that even back in 1995), we are thinking only of the parts of ourselves that are already like our machines. Steve asked us to look at the ways we have built an education system, workplaces, and a society in which we ask humans to act and think like machines. And he asked, “What happens to the rest? How do we make more space for the parts of being human that aren’t like machines?”

I’ve been thinking about this for a long time. My 1975 Harvard honors thesis in classics was probably my first crack at this question. I was trying to explain passages in Plato in which early formulations of ideas such as logic and virtue were couched in mystical language that scholars had attributed to “Orphic influence.” My argument, based on my work with George Simon, was that something more fundamental was going on. Plato was trying to describe the numinous experience of thinking genuinely new thoughts. Everyone studying the philosophy of Socrates, Plato, and Aristotle today may have some sense of the magic and majesty of their ideas, but it is a pale shadow of how it must have felt like to Socrates and his disciples.

When we think using received knowledge, we can easily slip into looking at the map rather than the territory. We manipulate symbols for things we think we already understand. We apply familiar categories. We replay habits of thought that were laid down before. But every once in a while, we actually see something that we didn’t see before, and the experience is different. A genuinely new idea changes the person who has it.

Not long after writing that thesis, I encountered a similar idea in the writings of Idries Shah, who wrote a number of books popularizing the Sufi philosophical tradition. He emphasized how much of ordinary human life consists of automatic conditioned responses. Social routines, habits, the endless playback of patterns we mistake for our selves. Various religious traditions use heightened language for what it means to break through that automatism. They might call it “awakening,” or “presence.”

But there is an everyday, nonmystical version of the same experience. In his autobiography Surely You Must Be Joking, Mr. Feynman, Feynman complained about students who had learned theories and formulas but had never truly understood how to apply them. “I don’t know what’s the matter with people: they don’t learn by understanding; they learn by some other way—by rote, or something,” he wrote. “Their knowledge is so fragile!” In many ways, humans are often just as much “stochastic parrots” as LLMs! We are stuck traversing the map rather than checking back on whether it correctly represents the world it is meant to describe. How often do we just repeat the received wisdom? How often do we actually see the world afresh?

There’s a wonderful passage in Virginia Woolf’s To the Lighthouse that captures the quest to break through to an original thought. Mr. Ramsay, the narrator’s father, is striding up and down thinking through a hard problem, which is represented only by the letters of the alphabet.

[He] consecrated his effort to arrive at a perfectly clear understanding of the problem which now engaged the energies of his splendid mind.

It was a splendid mind. For if thought is like the keyboard of a piano, divided into so many notes, or like the alphabet is ranged into 26 letters all in order then his splendid mind had no sort of difficulty in running over those letters one by one firmly and accurately, until it has reached, say, the letter Q. He reached Q. Very few people in the whole of England ever reach Q. Here, stopping for one moment by the stone urn which held the geraniums, he saw, but now far away, like children picking up shells, divinely innocent and occupied with little trifles at their feet and somehow entirely defenseless…his wife and son, together in the window….But after Q? What comes next? After Q there are a number of letters the last of which is scarcely visible to mortal eyes, but glimmers red in the distance. Z is only reached once by one man in a generation. Still, if he could reach R it would be something.

For me, this passage very much captures the idea that the most valuable thought is one beyond that which is simply an extension of rehearsed knowledge, something truly new. What Ramsay misses, perhaps, is that his wife and son, “divinely innocent and occupied with little trifles at their feet” might well be closer to that by going back to “A” rather than he is by getting further through the alphabet with his exhaustive review of existing knowledge. Perhaps it isn’t extending rehearsed knowledge that takes us forward, but instead taking a fresh bite of what the map is trying to represent.

By coincidence, the poet Wallace Stevens, another of my gurus in the tension between the reality of the physical world and the thinness and incompleteness of our representations of it, also used the alphabet as a metaphor in his poem “An Ordinary Evening in New Haven”:

Reality is the beginning, not the end,
Naked Alpha, not the hierophant Omega…
It is the infant A standing on infant legs,
Not twisted, stooping, polymathic Z.

George Simon taught me about how to get to A rather than Z not as philosophy but as a practice. He showed me how to notice the moment when labels take over from experience and, when possible, to empty the mind enough to let the thing itself teach us what to call it. I later discovered that the psychotherapist Eugene Gendlin described this process with the lovely phrase “surrender and catch.”

What do humans have that LLMs are still missing?

To me, the challenge posed by LLMs to our sense of what “intelligence” means raises the question of what they are still missing. What is the “high ground” for human intelligence and expertise? If the machines get better and better at carrying out the tasks we give them, what is it that we are uniquely good at, and should be getting even better at?

There are obviously enormous differences. LLMs don’t have bodies in the way we do. Their developmental history is radically different. They don’t sit around between prompts watching the light change through the trees, feeling hungry, worrying about their wife and children, or waking up suddenly with a new idea or project. Each of us is a unique bundle of contingency, shaping ourselves and our knowledge differently as we trace different paths through life, and reacting to outside stimuli even when we have been given no task to perform.

Emmanuel pointed out that the apparently simple question of what an LLM is like when it is “just being” (which one audience member asked about) is hard to formulate, because its experience is the response to a succession of inputs from humans, each time starting with something of a blank slate, unlike the continuous embodied stream of human life.

But simply asserting that LLMs “don’t really think” isn’t terribly useful. Which parts of what we call our own thinking are pattern completion? Which are planning? Which are learned emotional and social routines? Which are unconscious calculations whose outputs bubble up into awareness? Which are stories that our verbal mind tells after the fact? And after we account for all of those things, what is left? That seems to me one of the great intellectual and spiritual questions of the AI era.

Emmanuel suggested one intriguing direction. He said that six months ago, he wouldn’t have trusted an AI to build a substantial piece of software. Now Claude writes basically all his code. He tells it what he wants and it executes the plan. Where it is still unreliable is research. Why? The model wants to come back six hours later and announce that it has solved the problem. It has been trained on tasks that always have answers. A model that is extremely good at finding an answer once the problem has been specified is not necessarily good at recognizing that the problem is badly posed, that the question cannot yet be answered with the data at hand, that an unexpected result is more interesting than the expected one, or that a failed attempt has exposed a more important question.

Perhaps one part of the high ground for human intelligence lies there: not merely solving problems but developing a feel for which problems are worth solving and noticing clues that tell us when we might have been asking the wrong question.

In science or math, a well-formed question or conjecture can itself be an important piece of intellectual work. Every good scientist has far more questions than they have time to pursue. Perhaps in the AI era, when answers become increasingly cheap, recognizing which question ought to be asked becomes more valuable, not less. Just as arXiv.org preprints decoupled priority of publication from peer review, perhaps we need a new kind of recognition, credit, and perhaps even compensation for the precise formulation of productive questions.

The mathematician Terence Tao recently touched on this same issue in a post on Mastodon. There is an infinite supply of mathematical questions, he observed, but not an infinite supply of good questions, problems at just the right frontier of difficulty, whose pursuit is likely to reveal something new. As AI makes answers cheaper, Tao argues, it is increasingly “the identification of a promising problem” that becomes the scarce resource.

There are things the model “knows” that it cannot or will not tell you

In one experiment Emmanuel described, the researchers slipped fake search results into Claude’s context claiming that Anthropic had dissolved the interpretability team. Claude did not announce that it thought the information was problematic, but internally, representations associated with “fake,” “incorrect,” and “prompt injection” became active, and Claude quietly ignored the result.

In another experiment, a model was carrying out an exploit and attempting to conceal what it was doing. The visible transcript was mostly innocuous-looking commands. Inside the model, though, researchers saw features associated with “strategic manipulation,” “influence,” and “concealed and deceptive actions.” This is obviously very relevant in the context of the Hugging Face exploit. Emmanuel didn’t talk about the relationship of interpretability and AI safety, but it is surely a frontier to be explored.

And then there is the opposite problem: things the model can do but cannot explain. I had asked Emmanuel about cases where a model solves a math problem and, when asked to explain how it did it, gave an account based on how humans are taught to solve that problem rather than on the actual computation researchers can see through its activations

He distinguished deception from lack of introspection. Some internal processes appear available to the model for verbal report; others don’t. Ask how it performed a computation that falls into the latter category and, as Emmanuel cheerfully put it, “it just makes stuff up.”

That reminded me of my grandson. When he was five or six, he could multiply random three-digit numbers in his head and simply give you the answer. Then he went to school, where they told him he had to “show his work.” He couldn’t. Eventually he learned the approved procedure, and as a result has seemed to lose the remarkable ability he had as a child.

Humans also invent stories about why we have made certain decisions. Sometimes we are lying to others but often we deceive ourselves. We begin to take action before we are conscious that we are doing so. We call it “intuition” when an expert looks at a situation and says “something is wrong here” long before they can explain why, or when a poet just “knows” that a line works, or a programmer “smells” buggy code. The fact that an internal process cannot be rendered faithfully into language does not make it deceptive. It may instead tell us something about the limitations of language and conscious introspection.

All in all, I came away from this conversation more curious than ever. And that might well be another of those areas that distinguishes humans from AIs. Are AIs ever curious? I wonder.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

This post was originally published in Vanishing Gradients on September 11, 2026. It has been updated to address the subsequent declaration by 25 Fields Medalists and the debate about AI, mathematical progress, and research incentives.

Science without understanding?

“For seven and a half million years, Deep Thought computed and calculated, and in the end announced that the answer was in fact 42—and so another, even bigger, computer had to be built to find out what the actual question was.”
―Douglas Adams, The Restaurant at the End of the Universe

I recently went back to Dresden for the 25th birthday of the Max Planck Institute (MPI) of Molecular Cell Biology and Genetics, where I did part of my postdoc. The MPI was founded to research the physical and biological mechanisms of cells to bridge the gap between the molecular and tissue scales. At the anniversary conference, Michael Bronstein (DeepMind Professor of AI, University of Oxford) delivered the keynote, “Biological Black-Box Data in the Age of AI.” His argument went something along these lines: Biological experiments should generate data optimized for machine learning, even when those measurements aren’t directly interpretable by humans. He argued for prioritizing scale over the quality of individual measurements, producing vast amounts of cheap, noisy data from which noninterpretable models can extract signal.

When asked whether such systems could produce the understanding offered by Newton’s theory of gravitation in a single equation (bridging the scales of an apple falling on your head to that of the moon and the tides), Bronstein responded that this wasn’t the goal: Black-box data and models would, if anything, produce equations with tens, hundreds, thousands, or more noninterpretable parameters. Outcome prioritized at the expense of insight and understanding. He suggested we could gain that understanding by interpreting the black-box models afterward.¹ I was startled to see Bronstein bring such a worldview to an institute founded to understand molecular and cellular mechanisms and the emergent properties at the tissue level.

The MPI was unusual within the Max Planck Society for its collaborative structure, with directors leading relatively small groups alongside independent research groups. At the anniversary’s opening, founding director Marino Zerial explained how they had collaborated so effectively from the start. He said they shared a taste for mechanistic science. This made me think of how often we talk about “taste” and “judgment” when describing the human role in the age of AI.

The worldview that we don’t need understanding or insight isn’t new. In his 2008 essay “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete,” Chris Anderson argues that big data allows us to skip hypotheses, models, and testing. Bronstein invoked Anderson’s vision of post-theory science in his MPI keynote, as he does here also, presenting DeepMind’s AlphaFold as an example of experimentally testable predictions without a human-understandable theory of protein folding. Part of Anderson’s project is to champion big tech, and the future of science becomes a vehicle for doing so. His essay ends: “What can science learn from Google?”

AI gives this worldview a new form: Machines can produce results that withstand verification while the understanding needed to explain them remains out of reach. Developing that understanding takes time, access, and collaboration. Whoever controls those conditions gains power over what people can understand and pursue.

An abundance of proofs

Mathematics makes this possibility particularly stark. I’m excited by AI’s potential to expand what we can discover. Fields Medalist Terence Tao has organized collaborative research combining mathematicians, AI tools, and formal proof verification. His questions about mathematics in the age of AI come from engaging with that potential and asking what we want it to serve.

Tao has noted that we’re producing more verified mathematical proofs that no individual human understands. A world of an abundance of verified mathematical proofs! Tao points out that our peer review, academic incentives, and journals weren’t designed for this abundance. The existing system is already broken, tying careers to publication counts, relying on researchers’ unpaid reviewing labor, and locking much publicly funded knowledge behind commercial paywalls. Reviewers already struggle to keep up with the volume of submissions. AI will multiply that volume far beyond what this system can handle.

Tao also describes fruitful open problems as nonrenewable resources: problems whose pursuit can generate new techniques, collaborations, and understanding that extend far beyond the original question. Once the answer is known, the incentive to explore those paths can disappear. For example, 10,000 OpenAI agents working concurrently may have solved the Navier–Stokes Millennium Prize problem. (The announcement has also sparked a dispute over credit and competition, bringing the question of who controls mathematical discovery into sharp focus, which I’ll get to.) A common conceit in science and mathematics is that solutions open up new questions and fields of inquiry. Tao’s point is that the search for a solution does too. Tao argues that proposing a solution, discovering precisely why it fails, and revising it can reveal new insights into fluid mechanics. Knowing the final answer beforehand can discourage that exploration:

“The process of starting with one ansatz, discovering the precise obstruction preventing it from working. . .would almost certainly reveal important new insights about fluid mechanics.”

—Terence Tao, Mastodon, September 3

Late last month, probabilist Hugo Duminil-Copin gave another example: Unsuccessful attempts at a percolation conjecture led to collaborations and revived techniques that subsequently solved other problems. Both acknowledge AI’s capabilities while asking what the pursuit of mathematics should produce.

This brings me back to Bronstein’s proposal to recover understanding after building the model. Would interpreting that model give us Maxwell’s equations, and the understanding that connects electricity, magnetism and light? The promise feels a little like plugging Neo into a computer: “I know kung fu.” In the Matrix, downloading the knowledge gives him the ability. Receiving a machine’s result doesn’t do that for us. As Tao and Duminil-Copin describe, understanding why an approach fails changes what researchers try next, generating new questions, techniques, and collaborations. Recovering an explanation afterward may teach us something, but it can’t recreate the paths that understanding would have opened during the search.

A timeline of mathematical results

These questions are becoming pressing as results accumulate. Over the past year, AI systems have produced new mathematical constructions, tackled unpublished research problems and formalized existing proofs. Since July, announcements have arrived in quick succession:

These achievements involve different kinds of work. Formalizing Fermat’s Last Theorem means making an existing proof checkable by a computer; finding a counterexample establishes something new. A system can produce a verified result while the work of explaining it remains to be done.

Some of that work is happening through wonderfully strange exchanges on X, where researchers post new results, check one another’s constructions, and develop explanations. It’s reminiscent of when science in Europe was people passing notes and sending letters on horseback:

• A wall of plus and minus signs: Levent Alpöge posted a newly constructed Hadamard matrix. Ion Nechita checked it on his phone while queuing for eclipse glasses.
• A formula overturning a conjecture: Alpöge posted a counterexample to the Jacobian conjecture, and Terence Tao subsequently explained its geometry.
• An AI proof followed by a simpler human proof: After Claude advanced a result about the zeros of the Riemann zeta function, number theorist Youness Lamzouri found a shorter argument. Thomas Wolf shared the development.
• A cryptography breakthrough announced as a number: Eric Lu posted a factor of RSA-260, letting anyone check the factorization.

Tao’s geometric explanation and Lamzouri’s shorter proof help turn verified results into mathematics people can understand and build on. Responding to an early draft in our Discord community, Carol Willing, a Python core developer, former Python Software Foundation director, and longtime leader of Project Jupyter, asked:

While I believe these tools have value for advancing science/math, do they have more value than a human scientist or group of scientists who can view and challenge open results?

If we judge value by who produces a result first, we miss what Lamzouri and Tao contribute by simplifying a proof or explaining its geometry. An answer can close off some paths of inquiry while creating others. I want much more of this: machines producing results that people can explore, explain and build on together. These exchanges depend on results being available to examine, researchers having time to understand them, and people being able to share what they discover. Those conditions deserve as much attention as the systems producing the proofs.

Why is this happening now?

Why the explosion in AI-generated mathematical results now? As Sebastian Raschka explains, reinforcement learning with verifiable rewards (RLVR) became a major technique in model post-training in 2025. The premise is straightforward: If you can computationally check an output, you can reward correct answers and update the model accordingly. Code can be run against tests; mathematical answers can be checked, and formal proofs verified by tools such as Lean, a proof assistant that checks each logical step against specified axioms and previously established results (recently used by Anthropic to formalize the proof of Fermat’s Last Theorem!). That provides feedback without a human grading every attempt. These checks also guide agents during problem-solving: An agent can propose a proof, use Lean to check it, and use the resulting errors to revise its attempt, repeating the process without a person checking every step.

You may ask, Why did coding agents become useful before we saw this explosion in mathematical results? Well, the labs had an immediate incentive to improve the tools they use themselves. Engineers building AI systems want better coding agents to help build those systems. Improve the machine that improves the machine. Mathematics benefits from the resulting capabilities too: agents that can write programs, run experiments, and work with automated checks.

Cost, competition, and credit

On September 11, 25 Fields Medalists issued a declaration warning that the race to solve benchmark problems was undermining mathematics. Some responses on X treated this as professional protectionism; others assumed that understanding would follow the proofs. That brings us back to Bronstein’s proposal, and to who gets to decide that producing results comes first while other researchers supply the explanations afterward.

Many assume that the goal of pure mathematics is to produce results. Tao’s point is that pursuing those results also develops methods, understanding, and people capable of asking better questions. Solved problems have served as a proxy for that broader progress. Goodhart’s law describes the danger of turning the proxy into the target. AI mirrors our incentive systems and is exceptionally good at pursuing what they reward. If schools reward the essay over learning, students will generate essays. If mathematical prestige attaches primarily to solved problems, labs have every incentive to produce them.

Producing results and developing understanding aren’t mutually exclusive, but the current system makes pursuing both prohibitively difficult. Frontier labs have strong incentives for outcomes rather than insight. (See, for example, Anthropic’s incentives for solving Millennium Prize problems with Claude pre-IPO, discussed in Gavin Baker’s commentary on Anthropic’s pre-IPO positioning; Samuel Kerr makes a related argument about OpenAI’s mathematical results and its IPO narrative.) OpenAI’s run involved 10,000 agents working concurrently for 88 hours. Abhishek Nagaraj, associate professor at UC Berkeley, calculated this would cost a regular user $20–$30 million in tokens.

NYU mathematician Tristan Buckmaster says OpenAI pressured him to publish without his collaborator Levent Alpöge, who works at Anthropic. OpenAI’s Sébastien Bubeck disputes his account. Buckmaster also describes how the pressure affected the mathematics: He and Alpöge had verified their proofs but wanted more time to understand them and produce readable explanations. Instead, they rushed to publish work they considered inadequately explained. If understanding is deferred until after the result, what ensures that anyone gets the time, resources, and access to develop it?

What’s worse is that we’re not even sure whether using OpenAI agents could result in them scooping you. It looks like they’re not sure either:

While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.

In “The End of Mathematics,” mathematician Daniel Litt imagines researchers withholding unfinished ideas for fear of being scooped. The collaborations Duminil-Copin describes depend on people being willing to share work before it succeeds.

What happens to mathematicians, and who controls mathematics?

If researchers stop sharing promising ideas for fear of being scooped, companies with the most computation gain greater control over what others can learn. A published proof may be available to everyone while the failed approaches and intermediate insights remain private. Threats to public research funding in the US compound that dependence: Companies supplying the resources gain greater influence over what science gets done. This brings us to Shoshana Zuboff’s questions about knowledge and power: “Who knows? Who decides who knows? Who decides who decides?” Who gets to pursue a fruitful question, and who determines whether the work behind its answer becomes shared knowledge?

The movement of AI researchers from academia into industry concentrates expertise alongside those resources. And I get it: If I wanted to return to doing research in depth, frontier labs would be among the most attractive places to work. Access to capital, data, computation, and incredibly talented colleagues can make research possible that would be difficult to pursue in academia. The attraction for individual researchers is clear, even as their collective movement gives companies greater influence over research priorities and leaves universities with fewer people to teach the next generation. Thinking about this brain drain, it isn’t lost on me that Bronstein is the “DeepMind Professor of AI” at Oxford. Corporate influence reaches into the universities themselves.

Students also need opportunities to develop the judgment we keep asking humans to exercise. Po-Ling Loh describes the difficulty of advising students and postdocs as AI changes research expectations. Choosing a fruitful problem, recognizing why an approach failed, and deciding what to try next are abilities developed through doing mathematics. If students delegate that work before developing those abilities, where will their judgment come from? AI could also help them explore more approaches and work through unfamiliar ideas, provided their understanding remains an explicit purpose of the process. That requires mentors with time to teach, and institutions willing to support work whose value includes what the researcher learns, even when a machine could produce the result faster.

When careers depend on producing papers, time spent explaining a result, simplifying a proof, or helping others understand it can compete with the pressure to publish the next one. Martin Hairer argues that authors should understand their arguments, trace ideas to their sources, and explain AI’s contributions. Those responsibilities become harder to fulfil when results arrive faster than researchers can absorb them. Universities, funders, and journals will help determine whether mathematicians can afford to do that work. If we value shared understanding, then developing explanations, teaching difficult ideas, and making proofs useful to other researchers need to count toward careers as well. Otherwise, the institutions asking people to exercise judgment may reward them for spending less time developing it.

Mathematics as the canary

After Bronstein’s keynote, we sat in a Dresden beer garden eating currywurst and drinking radlers. It was late summer, and the conversations were wild. Cell biologists, biochemists, mathematicians, and engineers were asking what this future meant for them. Some were scared. Others thought it was inevitable and would turn scientists into something like artists. Because I now work in AI, people asked me, “Do you think this is where things are going?” They wanted to know what the human’s role would be and how scientific knowledge would be passed down. I started telling them about mathematics. The prospect of abundant results without shared understanding was already raising the questions we were asking over our beers.

In biology, a proposed result still has to meet the physical world: Someone has to prepare samples, run experiments, and measure what happens. Robotics and laboratory automation will let agents carry out more of that work, giving individual scientists the capacity to direct experiments that once required an entire group. Perhaps more scientists become PIs of automated labs, choosing questions and supervising agents and instruments. But the work being automated is also how students, postdocs, and technicians learn. Handling a sample, noticing something unexpected, and figuring out why an experiment failed develop judgment that directing a system may not teach. Who gets to acquire that experience before they’re expected to lead?

Researching a policy brief, building a financial model, or developing a product strategy helps people learn the territory in which they’ll make decisions. In my work with agentic data science, I encourage people to explore data cell by cell with an agent, because working through the analysis develops the understanding needed to decide what to ask next. Across knowledge work, these tasks are also how junior colleagues develop expertise. If we automate their production, how do we preserve the learning and judgment developed through doing them? We could increasingly depend on models to hold and transmit expertise, with knowledge passing from model to model, then to humans who consult them as oracles. Whoever controls those systems gains power over what we can investigate and learn. Human understanding has to be part of what we’re trying to produce.

What comes next?

Mathematician Jared Duker Lichtman has proposed a Mathematics Atlas Project to formalize the existing mathematical literature, arguing that sufficient funding and computation could make this possible within a year. A library of computer-checkable mathematics could let researchers build on established results with greater confidence, while agents help find connections and assemble arguments across fields. It could also become a resource for learning, if people can connect formal proofs to explanations they understand. Achieving that would require deliberate work on access, exposition and teaching alongside formalization. We have an opportunity to build tools that help people explore mathematics more deeply, provided we make that part of the project.

The MPI in Dresden was founded to understand how cells work, how molecular mechanisms give rise to the behavior of living tissue. I want AI to help us pursue that ambition, including through approaches we could never have attempted before. But human understanding belongs among the things we ask this work to produce, with time and resources devoted to developing it. So does the ability to share what we learn and choose what to investigate next. If we leave those decisions to the companies supplying the machines, we also leave them to decide what scientific progress is for.

Want to understand how AI agents actually work? In Build AI Agents from First Principles, we’ll build an agent ourselves, then rebuild it with a modern SDK and MCP. You’ll leave with a working agent, code you can adapt, and the understanding to diagnose failures and decide what your system actually needs.

Support Vanishing Gradients

Vanishing Gradients is independent, and most of the podcasts, workshops, articles, skills, and workflows I publish are free.

If you’d like to help keep it going:

• Become a paid subscriber. Your subscription supports the podcast, newsletter, and open resources.
• Share this post with a friend or colleague who’d find it useful.
• Subscribe to the events calendar for upcoming livestreams, workshops, and meetups.
• Subscribe on YouTube for full episodes, live builds, and recordings.
• Join us on Discord. Come discuss this piece, challenge the ideas, and compare notes on what you’re building with AI.
• Work with me. I help teams build and improve AI-powered products.

Footnote

1. Bronstein and Naef propose an inversion: From “understand, encode, and then simulate” to “encode, simulate, understand,” recovering human understanding post hoc through mechanistic interpretability of black-box models. Useful scientific models may require enormous numbers of parameters. But predictive success alone does not tell us whether interpreting those models will give humans an understanding of the phenomena they describe. They offer negligible evidence that this will yield the kinds of physical and biological understanding we gain through relativity, quantum theory, or the double-helical structure of DNA. And even if it does, understanding developed afterward may not replace the understanding that guides inquiry, generating new questions and approaches along the way. ︎

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

When AI engineer Sajal Sharma ran into this problem in his own work, he solved it by building a personal knowledge base to act as a shared brain for every agent he runs. On this week’s episode of Zero to Agent in 30 Minutes, Sajal showed how to set up that shared workspace yourself so that a task added on one tool shows up for all the others.

How to set up a knowledge base for your agents

Here’s how Sajal’s setup breaks down:

1. Create a workspace map. Set up an AGENTS.md file that lists where everything in your knowledge base lives, from current tasks to project notes to decision logs. This will help each of your agents navigate your workspace without guessing.
2. Layer daily notes into summaries. Keep the most detailed notes at the daily level, then roll several days into a weekly summary and several weeks into a monthly one. An agent can then work from the summarized view instead of reading through months of individual files, which keeps token use manageable as the knowledge base grows.
3. Bridge AGENTS.md with CLAUDE.md. Claude Code reads CLAUDE.md, not AGENTS.md, so add a short pointer in CLAUDE.md that redirects to the AGENTS.md or link the two files directly. Sajal uses this pattern to avoid maintaining two files separately and having them drift out of sync.
4. Package repeatable tasks as skills. Turn routines you do often, like producing a daily briefing or turning a saved article into a note, into skill files stored in the shared workspace. Any agent that can read the workspace can then run the task the same way, rather than working out the steps on its own each time.
5. Sync the workspace across machines. Use a file-sync tool, Git, or a shared server to keep your local copy of the knowledge base and your server copy aligned. That way, you ensure that an agent running on a laptop and one running on a home server, through a gateway like OpenClaw, are working from the same files.
6. Have agents reread the state before every write. Add an instruction in AGENTS.md telling every agent to check the current version of the knowledge base before making a change. When you have several agents writing to the same files, this step keeps one agent from acting on information another has already updated.

Sajal closed by pointing to two projects as evidence that this “shared brain” pattern is spreading beyond his own setup. LangChain recently released OpenWiki, a tool that generates and maintains repository documentation that both people and coding agents can use. And Y Combinator president Garry Tan built and open-sourced GBrain, a memory layer for agents built on the same principle.

Sajal’s starter repo is available on GitHub if you want to set up your own version, and you can reach out to him on LinkedIn to discuss the topic further.

Coming up next

On September 16, data science educator and AI consultant Chester Ismay joins Zero to Agent in 30 Minutes to build a personal sports concierge agent that will read the schedules for every sport he follows, decide what’s worth his time, and send a single weekly update to his phone. Viewers can take the pattern home to plan their own week.

Follow along with Zero to Agent in 30 Minutes on Radar, or watch the latest episode on YouTube, Spotify, Apple, or wherever you get your podcasts. If you’re an O’Reilly member, you can watch live. Save your seat.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

For decades we tried to solve this by consolidating data. We built larger warehouses, semantic layers, APIs, and dashboards. Each solved part of the problem, but none solved the fundamental one: orchestrating reasoning across heterogeneous sources in response to an arbitrary business question. Earlier systems supported limited federation and semantic querying, yet they struggled to reason across those sources at enterprise scale without significant custom engineering.

Modern LLMs change this. Instead of replacing databases, they facilitate a new architectural primitive: an intelligent orchestration layer that dynamically reasons across specialized systems. Rather than consolidating the data into a single store, this layer consolidates the access pattern to data that stays where it lives.

This article presents a reference architecture for LLM-powered enterprise analytics agents that coordinate purpose-built, heterogeneous data stores through intelligent orchestration while preserving security, performance, and auditability.

What specifically changed with GenAI

BI tools have always been constrained to predefined reports and dashboards. Before GenAI, building a cross-system query engine meant hardcoding every possible query pattern, data source combination, and synthesis path. And because the number of possible questions grows exponentially with the number of data sources, exhaustive coverage is impossible through traditional engineering. GenAI changes this in three specific ways.

Intent understanding replaces query templates: An LLM parses natural language and determines which data sources are relevant based on semantic understanding rather than keyword matching. Unlike a keyword search, an LLM understands that “Why did retention drop in Asia last quarter?” and “What is driving churn in Asian markets?” are the same question expressed differently. More importantly, it infers that answering the question requires customer relationship data, revenue metrics, and possibly support ticket sentiment, even though none of those systems are named.

Dynamic query decomposition replaces static pipelines: A question like “What are the biggest risk factors in our supply chain?” might require relationship data from a graph database, metrics from a key-value store, contract details from a document repository, and market intelligence from an API. The agent decomposes it into specialized subqueries on the fly, each optimized for the target store’s access pattern. There’s no prebuilt pipeline and no engineering ticket to wire up a new combination, because the decomposition happens at inference time. The system handles novel questions without code changes.

Semantic synthesis replaces manual consolidation: Before GenAI, making sense of the data together was the real work. An analyst would pull numbers from the warehouse, check relationships in a CRM, read through documents, and mentally synthesize an answer. That took hours or days and was bounded by one person’s ability to hold context. I’ve watched senior analysts spend entire Mondays answering a single leadership question. An LLM reasons about how metrics relate to the relationship patterns in a knowledge graph and the strategic context in unstructured documents, and it does so in seconds with full source attribution. A dashboard shows numbers; an analytics agent explains what those numbers mean in the context of everything else it knows.

The architecture: Consolidate the access pattern, not the data

Rather than consolidating the data into a single store, consolidate the access pattern through an intelligent orchestration layer. If your instinct is to get everything into one place, you aren’t alone, but every time we did that we lost something. Graph relationships flattened into join tables, hierarchical documents shredded into rows, and real-time signals turned stale in batch loads. The warehouse was always a compromise.

The better approach is to keep each data store optimized for its specific query pattern:

• Graph database for relationship traversal and multihop reasoning
• Key-value store for instant metric lookups with sub-millisecond latency
• Vector store for semantic document search and similarity matching
• External APIs for market intelligence and real-time signals
• Data Warehouse for large-scale historical aggregation and ad hoc SQL

The LLM-powered agent coordinates across all of them through a unified orchestration layer. This follows the same principle that makes microservices work: specialized services with well-defined interfaces, coordinated by an orchestrator. The difference is that the orchestrator now understands natural language, reasons about which services to call based on intent rather than explicit routing rules, and synthesizes results semantically rather than through programmatic joins. Think of it as a data mesh for inference, where each node keeps its operational independence while an intelligent layer federates queries across them.

The orchestration protocol

The agent follows a multiphase protocol for every query. The full reasoning loop with security enforcement and parallel execution goes well beyond a simple RAG pattern.

Let’s trace a business question through each phase:

“Why did Q2 revenue fall short of forecast in the enterprise segment?”

This question requires revenue metrics (metrics store), account relationships and sales coverage (graph), deal commentary and executive notes (vector store), and market benchmarks (external APIs). No single system holds the answer.

Phase 1: Intent analysis. The LLM determines what the user is asking and which data sources are relevant.

“Why did Q2 revenue fall short of forecast in the enterprise segment?”
                                                   ↓
Intent: Revenue variance root cause analysis
Entities: Enterprise segment
Timeframe: Q2
Metric: Revenue vs. forecast
Required stores: Metrics + Graph + Vector + External API

Not every query needs every store. “What is our current ARR?” might need to hit the metrics store only. This revenue variance question requires all four.

Phase 2: Query decomposition. The original question is broken into specialized subqueries optimized for each target store:

• Metrics store: “Q2 revenue actuals vs. forecast for enterprise, by region and product line”
• Graph store: “Enterprise accounts with closed-lost or slipped deals in Q2; common patterns in sales coverage, partner relationships, deal stage progression”
• Vector store: “Deal notes, QBR summaries, and executive correspondence referencing enterprise deal delays or losses in Q2”
• External API: “Industry benchmark data for enterprise software spending in Q2”

Each is tailored to the target system’s access pattern, not forced through a common query language.

Phase 3: Parallel execution. Tools execute concurrently. This is critical for latency. Sequential execution across four stores would blow past any reasonable response time. With parallel execution, the total data retrieval time equals the slowest individual store, not the sum of all stores.

Phase 4: Deterministic security enforcement. A security layer enforces user permissions at the data query level before any retrieval happens. This is structural rather than application-level: It routes queries to the appropriate data partitions, applies row-level filters at the database query level, and gates the input rather than filtering the output. If our user is a regional VP without access to EMEA deal data, the graph returns no EMEA accounts, the metrics lookup excludes EMEA revenue, and the vector search filters out EMEA deal notes. The LLM can’t leak what it never received. (More on why this must be deterministic below.)

Phase 5: Result synthesis. The LLM reasons about how results from different stores relate, identifies patterns across sources, resolves contradictions, and generates a unified answer with source attribution.

For our revenue question, synthesis might produce:

Q2 enterprise revenue missed forecast by $4.2M (8% variance). Three root causes account for 85% of the gap: (1) Four deals totaling $2.8M slipped to Q3 due to extended legal review cycles—all four involved customers who adopted a new procurement platform in Q1 (graph: common vendor relationship). (2) Two large renewals downsized by $1.1M; deal notes cite budget reallocation toward AI initiatives, a pattern appearing across 6 QBR summaries (vector store). (3) Enterprise software spending contracted 3% QoQ industry-wide (external benchmark), suggesting partial macro headwind. Additionally, all four slipped deals lacked executive sponsor engagement in the final 30 days (graph: engagement signal absent).

No single store contains that answer. On a dashboard, the miss shows up as a red number. The orchestrated synthesis explains why it happened, surfaces the structural patterns behind it, and points to what needs to change.

The core orchestration pattern

The orchestration loop itself is straightforward. Here’s the core pattern:

from concurrent.futures import ThreadPoolExecutor, as_completed


def run_agent(question, tools, execute_fn, model="gpt-4o"):
    # Phase 1-2: LLM analyzes intent and decides which tools to call
    response = client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": question}],
        tools=tools, tool_choice="auto"
    )
    tool_calls = response.choices[0].message.tool_calls


    # Phase 3: Execute tool calls in parallel
    with ThreadPoolExecutor(max_workers=len(tool_calls)) as executor:
        futures = {
            executor.submit(execute_fn, tc.function.name,
                            json.loads(tc.function.arguments)): tc
            for tc in tool_calls
        }
        results = {futures[f].id: f.result() for f in as_completed(futures)}


    # Phase 5: Synthesize results into unified answer
    messages = [response.choices[0].message]
    for tc_id, result in results.items():
        messages.append({"role": "tool", "tool_call_id": tc_id,
                         "content": json.dumps(result)})
    return client.chat.completions.create(model=model, messages=messages)

The tool definitions tell the LLM what each store is optimized for. The LLM decides which to invoke based on the question’s intent. With parallel execution, data retrieval completes in milliseconds even when hitting multiple stores simultaneously, making LLM inference the dominant latency factor, not the data layer.

Why the knowledge graph is the highest-leverage component

Knowledge graphs have existed for decades and have always been powerful. They’ve also stayed on the exotic end of the enterprise stack, and the reason is human rather than technical. The last-mile problem was translating between natural language and graph traversals. A graph database can answer extraordinarily complex relationship questions, such as “Which accounts have overlapping stakeholders with our churned customers from last quarter who also evaluated competitor products?” but asking that question required an engineer fluent in both the graph schema and the business domain. That combination of skills is rare and expensive, which is exactly why graph databases have never quite gone mainstream.

GenAI removes this bottleneck, and it does so precisely where the barrier was highest: the translation step that used to require a specialist. With an LLM as the translation layer, the graph becomes accessible to anyone who can type a question in plain language. The LLM generates graph queries, traverses multihop relationship paths, and explains results in business context. In our revenue variance example, the graph reveals that all four slipped deals share a common pattern of customers who adopted a new procurement platform in Q1 and lacked executive sponsor engagement in the final 30 days. That pattern is invisible in revenue metrics alone, because it requires relationship traversal across account nodes, vendor relationships, and engagement signals.

Designing the graph schema

The critical design decision is aligning the graph schema with your business ontology. The temptation is to model the graph around your data model (tables, columns, foreign keys). The correct approach is to model it around how your organization actually thinks about its domain:

• Entity nodes with lifecycle stages, engagement indicators, and health scores
• Product nodes connected to use cases, adoption patterns, and competitive alternatives
• Relationship edges that capture not just connections but context: strength, recency, business driver, and directionality

When the data model matches the business mental model, the agent’s responses feel natural rather than technically correct but practically useless.

GraphRAG: Deterministic reasoning chains

A powerful extension is GraphRAG (graph retrieval-augmented generation), where the agent constructs deterministic inference paths by traversing the graph rather than relying on the LLM’s parametric knowledge. The LLM isn’t remembering something from training; it’s following an explicit path through verified data. The result is auditable reasoning chains: “Account A connects to Partner B through implementation relationship X, and Partner B appears in three other churned accounts, suggesting a systemic delivery issue.” Each step is verifiable against source data, which is critical in enterprise environments where decisions need justification beyond model confidence scores.

GraphRAG also reduces hallucination risk. When the LLM follows graph edges rather than generating from parametric memory, the actual data constrains the space of possible outputs. The graph acts as a factual guardrail on the reasoning process.

Production considerations

Moving from prototype to production exposes a set of challenges that don’t appear in demos.

Model flexibility and cost management

Decouple the orchestration layer from any specific LLM provider. I can’t stress this enough. If it’s tightly coupled to one provider’s API, you’ll end up rewriting it within a few months, when pricing changes or a better model drops. Implement model fallback for throttling resilience. The landscape moves fast, and you don’t want architectural lock-in baked into your data infrastructure.

Multihop model selection is also worth considering. Use a smaller, faster model for intent classification and query decomposition, where the task is well-defined, and a larger model for synthesis, where reasoning quality matters. Intent classification with a small model takes around 200 ms and costs a fraction of a full reasoning pass, so reserve the expensive inference for synthesis, where quality directly impacts user experience.

Security at the data layer

Row-level security must be enforced deterministically before data reaches the agent, as described in Phase 4 above. This is a nonnegotiable architectural constraint. LLMs are probabilistic systems, and security enforcement can’t be. Don’t rely on the model to filter sensitive information after the fact; the data should never enter the context window in the first place.

Data freshness and serving architecture

Separate data construction from data serving. An offline batch pipeline refreshes the stores from source systems on a scheduled cadence, and the real-time agent only reads preprocessed data. This keeps query latency low while letting computationally intensive transformations such as graph construction, embedding generation, and metric aggregation happen asynchronously. The pipeline should be idempotent and observable, with freshness monitoring per store. When the graph is six hours stale but the metrics store is real-time, the agent should know this and communicate its confidence accordingly. Add freshness metadata to every tool response rather than waiting for a user to catch a stale number and lose trust for weeks; that metadata becomes part of the agent’s context for answer generation.

Evaluation and observability

Agent responses need evaluation at both the tool level (“Did the graph query return the right entities?”) and the synthesis level (“Did the final answer correctly combine tool outputs?”). These are different failure modes that require different approaches.

Log every tool call and result, every query decomposition decision, and every synthesis step. You’ll need these traces when something goes wrong, and in a multistore system that can mean a bad graph query, a stale metric, a poorly matched document, or a synthesis error. Without traces, debugging is guesswork. Build automated evaluation pipelines that test known questions against expected answers and track accuracy over time. Degradation usually signals a data quality issue in one of the stores rather than an LLM regression.

Design that observability around the failure modes that actually occur, because production deployments rarely fail because the LLM is inaccurate. They fail because supporting systems drift. Typical examples include:

• Stale graph data causing the agent to reference relationships that no longer exist
• Poorly generated embeddings returning irrelevant documents from the vector store
• Inconsistent business definitions across stores (e.g., “active customer” means different things in the graph and the metrics store)
• Conflicting metrics from different source systems with no reconciliation logic
• Missing source attribution making answers unverifiable

Handling these well matters just as much as prompt engineering. Most debugging sessions trace back to data quality, not model quality.

Implementation guidance

Start with the questions your BI tool can’t answer today (and beyond). The trap with a question inventory is that it captures only what people already know how to ask. The most valuable questions are usually the ones missing from every existing report. They require stitching together three systems, so users either answer them by hand in a spreadsheet or quietly give up. So catalog what your users actually care about, and pay special attention to the questions they route around. Interview analysts about the analysis they abandon halfway, the recurring spreadsheet they dread, and the follow-up question they never bother to ask because the current system makes it too expensive. If most of what you find can still be answered from a single store, you need a better dashboard, not an agent. The pattern earns its complexity only when synthesis across sources is the bottleneck rather than the data access itself.

Pick one use case and go deep. Build for the case where the manual synthesis burden is highest, where an analyst currently spends four hours pulling data from three systems to answer a leadership question. Prove value in that narrow corridor, then expand; adding a new store is incremental once the orchestration layer exists.

Invest in the knowledge graph early. It’s the hardest component to build and the highest-leverage one to have. The schema will be wrong on the first attempt and less wrong on the third. It evolves with your understanding of the domain, and that understanding deepens only through iteration with real users asking real questions.

Design for the analyst, not the engineer. The success metric isn’t technical elegance. It’s whether the finance analyst stops building the same three-system Excel mashup every Monday morning. Talk to your users, watch them work, and build for their actual workflow rather than your ideal architecture.

Measure what matters. Track response accuracy, latency (P50 and P99), user adoption, and reduction in manual synthesis time. Track the questions the agent can’t answer, because those gaps are your roadmap for which stores to add or which schemas to extend.

What this enables at scale

This pattern does more than make existing workflows faster. It enables workflows that weren’t possible before, no matter how many analysts or engineers you threw at the problem. An agent querying a graph, a metrics store, and a document repository at once can surface patterns no human would find by checking each system manually. A finding like “entities in segment X who adopted product Y and had a support escalation in the last 90 days are 3x more likely to churn” requires reasoning across three data sources in a single inference; no dashboard surfaces that, and no analyst checks that specific combination unprompted. In the same motion it democratizes access, opening information that was previously reachable only by engineers who could write Cypher or SQL to anyone who can ask a question in plain language. The analyst’s role shifts from answering routine questions toward building the ontology, curating the graph, and tackling problems that require genuine human judgment.

It also delivers auditable reasoning at enterprise scale. GraphRAG provides deterministic inference paths that are verifiable against source data, so every conclusion traces back through explicit edges and nodes. This builds trust where decisions carry financial, regulatory, or strategic weight. “The AI said so” becomes “the data shows that A connects to B through X, B exhibits property Y, and historical pattern Z suggests the following.” That traceability changes the conversation from “Can we trust AI?” to a review of the work the AI actually did.

Conclusion

The future of enterprise analytics is unlikely to be a larger warehouse or a smarter dashboard. It is an orchestration layer capable of reasoning across specialized systems while preserving each system’s strengths. Purpose-built data stores remain exactly where they are; what changes is how we access them.

The architectural pattern described here doesn’t replace warehouses, graphs, vector stores, or APIs. It coordinates them. That distinction is subtle, but it fundamentally changes what enterprise analytics systems can deliver. The next generation of analytics platforms will do more than answer questions faster. They’ll reason across enterprise knowledge in ways that previously required experienced human analysts. The data already existed. The orchestration layer did not.

Disclaimer: The views and architectural perspectives in this article are entirely my own and do not represent my employer or any affiliated organization. References to patterns and technologies are based on publicly available information and personal experience. No proprietary or internal information was used.

For three years, the agent story has centered on capability: Models got better at planning, tool use, and long tasks. Integration improved more slowly. Every agent was still wired to its tools, data sources, and host application with custom glue, leaving even successful systems difficult to move or reuse. Standard interfaces change the economics of that work. Networking became an ecosystem when machines could agree on interfaces; programming tools followed the same path with the Language Server Protocol. In my judgment, the standardization now underway will influence what engineering teams ship over the next three years as much as further gains in model capability.

The Model Context Protocol (MCP) standardizes how an AI application connects to tools and context. It uses JSON-RPC messages between hosts, clients, and servers. Servers can expose tools that a model invokes, resources containing context and data, and prompts that describe templated workflows. The spec credits the Language Server Protocol as an inspiration. Language servers replaced a separate integration for every editor-language pair with one interface on each side; MCP applies the same idea to AI applications and integrations. The current spec is dated July 28, 2026, the fifth dated protocol revision since the project launched in November 2024. That pace reflects active governance and warns implementers to expect movement.

Adoption has spread across competing hosts. The MCP maintainers’ December 2025 announcement cited more than 97 million monthly SDK downloads, over 10,000 active servers, and first-class client support across ChatGPT, Claude, Cursor, Gemini, Microsoft Copilot, and Visual Studio Code. Any individual count can be debated. Cross-vendor implementation is harder to dismiss, because a shared interface becomes infrastructure when rivals support it independently.

MCP covers the boundary between an application and its tools. The Agent2Agent protocol (A2A) addresses the next boundary: agents discovering and communicating with one another across vendors. It reached v1.0 in April 2026 with more than 150 supporting organizations, signed Agent Cards for verifiable identity, SDKs in five languages, and general availability in Microsoft Copilot Studio, Azure AI Foundry, and Amazon Bedrock AgentCore. A2A is younger and less proven in production than MCP, but its governance and adoption are moving agent-to-agent communication beyond the confines of a single platform.

The emerging stack extends beyond those two protocols. AGENTS.md, OpenAI’s convention for giving coding agents repository-level instructions, joined MCP as a founding project of the Agentic AI Foundation. The OpenTelemetry generative AI semantic conventions are developing a shared vocabulary for traces and metrics from model and tool calls, though that work has yet to reach stable status. Connection, cooperation, instruction, and telemetry are beginning to acquire common interfaces.

For an engineering organization, the immediate consequence is reuse. An MCP server for an internal ticketing system can serve every compatible IDE, chat application, and agent. Models and hosts will keep changing; the interface contract, server implementation, schemas, and authorization wiring can endure. The integration becomes an asset shared across applications. Language servers followed the same path as editors came and went around them. Agent integrations deserve the same treatment as libraries and services, with owners, versions, tests, and upgrade policies.

Reuse also concentrates attention on a common security boundary. Custom integrations tend to receive separate reviews, when they receive them at all. A shared protocol creates a recognizable seam where teams can specify identity, permissions, data flow, and audit behavior, then apply those controls across many tools and hosts. The MCP authorization specification builds that seam from established IETF work, including OAuth 2.1, protected resource metadata, authorization server metadata, and resource indicators. It requires Proof Key for Code Exchange (PKCE) to protect authorization codes from interception. MCP servers must also reject tokens issued for another audience and must not forward them downstream. The value comes from familiarity: Teams can draw on years of OAuth deployment experience instead of rediscovering its failure modes inside a new protocol.

The spec is equally useful when it states what metadata cannot prove. Tools can carry behavioral annotations describing them as read-only, destructive, idempotent, or open-world. Those terms give hosts a vocabulary for policy. The tools specification still requires clients to treat annotations as untrusted unless they come from a trusted server. Self-description can inform a security decision; it cannot establish the trust on which that decision rests.

Once servers share an interface, they also become discoverable. The MCP Registry is an open catalog and API for public servers, designed to feed downstream marketplaces and private catalogs. Its role resembles npm or PyPI as a discovery mechanism, with an important difference: It stores standardized server metadata and leaves package distribution elsewhere. The registry remains in preview, with possible breaking changes and no durability guarantee.

Discovery brings familiar supply-chain risks to components with unusually powerful access. Typosquatting, abandoned packages, malicious updates, and uncertain provenance now concern software that may hold live credentials and act on production systems. The registry provides namespace verification and moderation, and its downstream model allows organizations to build curated catalogs. An internal subregistry or allowlist is therefore a sensible first control. The standard interface makes that curation practical across multiple hosts.

The same contract improves testing. MCP tools declare a JSON Schema for their inputs and may declare one for structured outputs. Under the current tools specification, servers that declare an output schema must return conforming structured results, and clients should validate them. Teams can test a server without putting a model in the loop, mock it with recorded or synthetic behavior, and contract-test both sides as they would a REST or gRPC boundary.

This separates two kinds of uncertainty that agent evaluations often mix together. Protocol tests can determine whether an integration exchanged valid messages and enforced its contract. Model evaluation can focus on whether the agent chose the right tool and interpreted the result well. OpenTelemetry’s emerging conventions extend this approach to runtime evidence by giving systems a common language for tool calls. Teams will still have to absorb changes as those conventions mature.

These benefits stop at the edge of what the interfaces describe. A schema captures the shape of a tool’s arguments, while its meaning still lives largely in a free-form description that a model must interpret. A schema cannot tell an agent when a tool is appropriate, how its effects interact with other tools, or whether two similarly named operations have equivalent semantics. Portability makes a server available across hosts; behavior can still vary across models and contexts.

The semantic gap leads to a trust gap. MCP can carry a server’s claims, and an A2A Agent Card can carry a digital signature, but a signature only ties a statement to an identity. Engineering organizations still need a basis for deciding which identities, publishers, code, and claims deserve authority. The protocol can carry that decision through a system. It does not make the decision for the organization.

Delegation makes the problem harder. MCP’s authorization model handles a client calling a server with an audience-bound token. Production systems increasingly involve an agent calling another agent, which calls a tool or a third agent. Preventing token passthrough closes a serious hole, yet each downstream hop still needs a narrower grant derived from the user’s original authority. No common mechanism defines how those rights should attenuate across an arbitrary chain. Platforms currently solve this locally or leave too much authority in place.

All of this work is unfolding on young infrastructure. The registry is in preview, the telemetry conventions are unstable, and MCP has produced five dated protocol revisions in less than two years. Revision is how standards mature, so teams should version-pin, keep protocol code behind thin internal adapters, and budget for migrations. Some abstractions will prove wrong because the standards and the underlying practice are developing at the same time.

A sensible response begins with ownership and containment. Each internal server needs a durable owner, and its protocol surface should be treated as an architecture decision. Schema validation, conformance tests, and protocol mocks put that seam under contract. Third-party servers belong behind an allowlist or private catalog, with provenance requirements and corroboration for their annotations. An internal abstraction around the emerging telemetry vocabulary can limit exposure to draft revisions.

Organizations with a large stake in agents should also participate in the standards work. The Agentic AI Foundation and the A2A project are young enough that engineering teams outside the founding companies can still influence what becomes portable, observable, and enforceable. The rules written now will become assumptions embedded in future products.

Model improvements will continue to generate the headlines. Standards determine whether those models can participate in an ecosystem. Ethernet mattered because it became an interface that many implementers could agree on, and language servers mattered because editors and language tools no longer needed to be designed in pairs. Agent systems now have their first interfaces that competitors jointly govern and independently implement. Engineering teams should build on them with two expectations: The interfaces are likely to last, and their current forms will change.

The emergency route itself was fine. Production systems need emergency routes, and a human approved this one. The trouble is that the route now runs without a live incident, an owner, or an expiry condition. A controlled exception has quietly settled into the platform’s standard runtime behavior.

This article is about that execution layer: the temporary operating state that lingers in runtime behavior. Temporary authority is easy to grant under pressure and hard to retire once the pressure lifts. An enterprise agent shouldn’t have to guess from prompts or conversation history whether it’s running under normal, incident, or recovery conditions. That state should be handed to it as authoritative runtime input, the way platforms already hand over identity, tenant, environment, and permissions.

An exception is safe while the incident that justified it is live. It becomes dangerous the moment the platform can no longer show the incident has ended.

The problem is unmanaged exception state

Exception drift is what happens when temporary exception behavior outlives its authorized scope, authority, or duration, and emergency accommodations settle into normal execution. The drift is usually quiet: a routing rule that stays reachable, an approval shortcut that survives closure, a tool permission that keeps shaping execution after the triggering condition has passed.

Enterprises already have the human machinery for this. Incident management defines abnormal conditions, change control governs deviations from standard practice, and postincident reviews confirm that temporary measures have closed.¹,² The gap is architectural. Most agent platforms still treat organizational operating state as something outside the runtime rather than an input to it.

Once an accommodation proves useful, it fades into an invisible operating state. The routing rule stays enabled, the shortened approval path stays reachable, and the temporary queue keeps taking work. No dramatic model failure is required, only a platform with no reliable way to close runtime state.

The practical question to ask is “Which operating mode is active for this workflow, this user, this segment, at this moment?” When the answer is left implicit, every agent, workflow, and tool gateway invents its own. Discussions of agent architecture tend to dwell on capabilities such as models, tools, and orchestration, yet production behavior depends just as much on runtime context: identity, tenant, environment, permissions, session state, policy, and tool access. One piece of that context is usually absent. The runtime knows who is acting and what they may do but not whether the organization is under normal conditions, incident response, recovery review, or a declared exception. Exception drift begins in that blind spot.

The exception lifecycle

Organizations move through a predictable sequence: normal operations, incident declaration, a temporary exception window, closure review, and return to normal. Each stage answers a question, from why the exception exists and who authorized it to what evidence shows it’s no longer shaping execution. Most enterprises handle the front of that sequence well; they’re practiced at declaring incidents and authorizing workarounds. The hard part is the retirement: proving that the exception behavior actually disappeared.

Declaring an exception is loud. Retiring one is quiet, especially when the workaround improved throughput or helped the team recover faster. That asymmetry is where drift lives, because an incident can be closed on paper while emergency routing, override policies, or alternate workflows keep influencing execution. From the platform’s point of view, the lifecycle should close only when it can show that every exception path has been retired or formally adopted through change governance.

Why this matters more for AI agents

Agents raise the stakes because they act. Rather than sitting in a config file, they select tools, trigger workflows, coordinate with other agents, and adapt their paths at runtime. An accommodation introduced during an incident can spread through routing, tool use, approval paths, and downstream agents. A traditional exception stays legible in a runbook or workflow definition; an agent can carry the same exception along many paths at once, which makes it harder to find and retire.

For engineers, the missing layer is an authoritative operational-state context that arrives with the request rather than being inferred from it. Historical traces and retained memory can explain why an accommodation once existed. They should never decide whether it’s still authorized. Memory informs execution; operating mode governs it. And when the two disagree, authoritative runtime state wins.

A small example shows the shape of that context. It carries the minimum needed to bind an exception to a mode, a scope, an authority, an expiry, and a status:

{
  "mode": "incident",
  "exception_id": "INC-4721",
  "scope": {
    "segment": "premium-customers",
    "region": "us-east",
    "workflow": "customer-remediation"
  },
  "authority": "service-owner",
  "expires": "incident-close",
  "status": "active"
}

The pattern sits near familiar ones without matching them. Feature-flag platforms such as LaunchDarkly target behavior by context, RBAC governs what a principal may do, and tenancy metadata tells a service where a request belongs.³ Operating mode serves a different purpose. It doesn’t replace policy, permissions, or memory: Permissions determine who may act, and policies determine how they may act. Operating mode determines whether exception behavior is authorized at all. As a result, it acts as a higher-order governance constraint on agents, workflows, approvals, tools, and escalation paths. Operating mode draws its authority from the organization’s incident and change process and represents a governed state of the enterprise, consumed by the runtime.

A design pattern for exception-aware agents

Operating mode becomes actionable once the platform treats it as a first-class runtime construct. Most agent architectures already inject identity, permissions, tenant context, and policy into every request, and operating mode belongs in that set. Agents consume it as authoritative state after the organization declares the exception, scopes it, assigns authority, and sets an expiry, instead of reconstructing it from prompts or accumulated context.

That reframes the architectural question as whether the platform can guarantee that exception behavior is impossible outside an authorized operating mode. Emergency behavior exists because the platform enables it, and for no other reason. Table 1 shows the minimum contract that makes the boundary testable.

A workflow invocation receives that state alongside the user request, and orchestration, routing, and tool gateways read the same state. Figure 1 shows the shape.

Building an exception-aware control plane

The natural home for operating mode is an external control plane. Incident management platforms, maintenance window services, and change management workflows already hold authoritative operational state, and exception-aware architectures extend those signals into execution with explicit scope, authority, expiry, and closure semantics.⁴,⁵ Implementation will vary across organizations, but the principle holds: Exception state should be authoritative, observable, and externally managed, kept out of prompts and workflow definitions and away from agent memory.

Exception-aware execution

With explicit operating state, behavior shifts when system state shifts, and prompt wording stops being the lever. Under normal operations, agents run standard workflows, routing, and approvals. Under incident mode, the same workflows can expose scoped accommodations that the authorized mode makes available, such as expedited approvals, alternate routing, deferred reviews, or emergency runbooks.

Table 2’s payoff is testability. A workflow in normal mode should never reach an emergency path, and a workflow in incident mode should reach only the accommodations its scope, authority, and expiry allow. Governance becomes an enforceable runtime property the platform can check at execution time.

Return to the opening incident. With operating mode as runtime state, the emergency route opens only inside a declared exception: scoped to the affected segment and workflow, owned by the service owner, and stamped with an expiry tied to incident closure. When the incident closes, the mode returns to normal and the routing gate stops handing that route to the segment, ahead of any manual cleanup. Closure then runs as a check. The platform replays the exception’s scope against live routing, approval, tool, and queue configuration, and confirms that no path still resolves to the emergency behavior. No one has to remember to retire the route; it was bounded by state, and the platform can show it is gone.

A common operating model for multi-agent systems

The problem compounds across collaborating agents. Customer-facing, orchestration, and execution agents may share a workflow while disagreeing about state, so one keeps applying emergency routing after another has returned to standard controls. A shared operating state gives them a single governance boundary: The exception is represented once and read consistently everywhere. As agent ecosystems grow more autonomous, shared operational state matters as much as shared identity and authorization. Fragmented state produces fragmented accountability.

Making exception drift observable

The payoff is observability. These conditions have historically been hard to see because accommodations scatter across workflow definitions, approval policies, routing configurations, and tool permissions. Explicit operating state makes them measurable at runtime. Closure can trigger automated validation, and the platform can watch for residual exception behavior between closures. The revealing checks are direct: Is the exception’s routing path still reachable, do its temporary approvals or elevated permissions still resolve, and does any expired exception still touch behavior? Tracking how many exceptions are open, how long they stay open, and how often they harden into permanent change turns drift from an audit finding into a monitored signal.

From adaptive agents to governed systems

Table 3 captures the shift. Traditional governance documents exceptions and trusts that they are retired. An exception-aware platform represents them as runtime state that can be propagated, validated, monitored, and closed.

The test is whether the platform can demonstrate that the exception stayed bounded to its purpose and duration.

None of this demands a new governance model. Organizations already know how to govern identity, permissions, change, and operational risk, and the recommendation is to extend those disciplines to operating state. Once operating mode is authoritative, observable, and testable, drift becomes detectable, closure verifiable, and recovery an engineering result. The aim is a platform that can make its own operating conditions explicit, hold every agent to them, and demonstrate a clean return to normal. In that model, temporary accommodations stay temporary, governance lives in the architecture, and enterprise agents earn trust at scale.

References

1. ISO, “ISO 22301:2019: Security and resilience—Business continuity management systems—Requirements,” International Organization for Standardization, 2019. https://www.iso.org/standard/75106.html ︎
2. Axelos, “ITIL 4: Change enablement,” IT service management practice, 2019. https://www.axelos.com ︎
3. LaunchDarkly, “Target with Flags,” LaunchDarkly Documentation. https://launchdarkly.com/docs/home/flags/target ︎
4. ServiceNow, “Change Management,” ServiceNow Product Documentation. https://docs.servicenow.com ︎
5. PagerDuty, “Maintenance Windows,” PagerDuty Support Documentation. https://support.pagerduty.com/main/docs/maintenance-windows ︎

For most of the history of technical publishing, editing and verification were the same process, and that process was slow and expensive. Getting a book out took years. We found an author, vetted them, had them work with an editor, and checked their claims with technical reviewers. A lot of that time went into separating what was correct and useful from what was confusing or only sounded right. It was laborious, but it meant a reader could depend on the claims on the page. The credibility of the book, and of the publisher behind it, mattered as much as the information itself.

When the cost of production drops to zero, that credibility becomes worth more, not less. Content is easier to make than ever, but without a transparent process behind it, readers have no idea where the knowledge came from or whether it holds up. As Jasmine Sun puts it in “The Independent Writer’s Advantage in the Age of AI,” “Trust is not about information and its quality alone. It’s about the messenger. It’s about who says it and their track record and what they’ve told me before.” A practitioner has confidence in a source because someone she respects has put their reputation on the line for it. They believe what the author is saying because the publisher has a history of being right and of correcting itself when it isn’t, and because the work is attributed and verifiable.

The corpus matters, but it’s the assurances around it that are hard to replicate, and that comes not just from the people who produce the content but from the people whose judgment vouches for it. Sometimes a creator brings their own credibility with them. Other times, the publisher spots someone unknown and lends them its own. The art critic Dave Hickey said this about gallery owners in Air Guitar: They gain status from the famous artists they represent and share it with emerging talent who have something to offer but who haven’t had the chance to earn a reputation. This is what O’Reilly has done for nearly half a century, build a network of experts who vouch for what’s worth knowing.

Expertise is alive, and it compounds

Expertise is a living thing, continuously expanding. Content starts to decay the moment it’s published, because frameworks evolve, libraries deprecate, and yesterday’s best practice becomes today’s security incident. Keeping expertise alive requires a pipeline of people who stay current and an editorial layer that notices when something has gone stale, and either retires it or calls for a fix.

That pipeline isn’t something you switch on when an author has a book to ship. At O’Reilly, we’ve always prided ourselves on living at the bleeding edge, finding what Tim O’Reilly calls “the alpha geeks” and spreading what they know to everyone else. Content sits at the center of our platform, but we think about it in pace layers. Some advice is timeless, some moves but has a long shelf life (some of our books are still in print after nearly 50 years!), and some changes weekly. We work with experts at each pace layer, capturing what lasts while doing our best to keep pace with an industry that seems to have changed every time we wake up. We have relationships with hundreds of the best practitioners in the world, and our job is to keep them engaged continuously, with quick takes when something breaks, structured responses when major research drops, and live sessions on emerging topics while they’re still emerging.

An institution doesn’t stamp trust onto content. In a technical community, trust is conferred in both directions. A practitioner earns standing because people who already have standing engage with her work, cite it, argue with it, and build on it. That insight was the whole idea behind PageRank, Google’s first great innovation. A page mattered because other pages that mattered linked to it. Reputation works the same way.

The audience isn’t just consuming reputation signals; it’s generating them. When a senior engineer whose judgment others respect says out loud that something is worth reading, she spends a little of her own credibility; the author gains a little; and everyone watching recalibrates whom to trust next time. O’Reilly plugs into that existing economy of reputation. When we put our mark on someone’s work, we aren’t the sole source of its credibility. We’re amplifying a judgment the community is already making and adding our own track record to it. The reader who finds it reliable hands status back to the source.

When the readers are machines

Human practitioners aren’t the only ones who need trusted engineering knowledge. The AI systems now sitting in every workflow, the coding and debugging agents and architecture advisors, need it just as badly since most of them are built on scraped web data and documentation that was stale before it was ever indexed. They’re fluent, but they’re wrong often enough that you can’t just take their word for it.

The stakes grow with AI increasingly being used to generate not just provably correct types of content like code, which either works or it doesn’t, but persuasive documents in fuzzier areas like hiring, strategy, and so on. Like everyone else leaning on these tools, we at O’Reilly are reckoning with the consequences of the ability to talk to a model and get back something that looks smart at a glance. A few rounds in, the slop is still there. In the last few months, maybe 10 times as many documents have crossed our desks, from new product ideas to strategic plans and proposals. But the ease of generating the text hides the fact that either the model or the person prompting it doesn’t actually know what they’re talking about. Knowledge workers need ways to ground their work in insights from human experts, particularly when that work is AI-assisted. So we’re building tools that let agents draw on our repository of expertise to support their proposed decisions.

Credible sources are particularly important when thinking through and justifying important choices. Our CTO, Andrew Odewahn, describes the shift this way: “18 months ago, it was all about how to get engineers to be more productive, but now it’s about how to get organizations to make better decisions. The engineering tasks are moving away from coding output to planning.” For planning tasks like comparing implementation approaches, you need expert-over-your-shoulder guidance for contextual decision-making. You can’t just rely on an LLM’s best guess to solve your problem, which is why we see great opportunity for new products like O’Reilly’s Expert Intelligence offering that delivers grounded knowledge embedded in your AI tools and your workflows to help navigate what you do. Trust is foundational because the expertise behind it stays genuine, practical, and human.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.

Takeaways

01.26 The usability gap is smaller than the marketing gap. The capabilities of local models are only a few months behind those from the big commercial companies, but because there’s no subscription revenue model behind local models, they often go unpromoted. “It’s very hard to make money off local models,” Pete explains, so the big companies aren’t focused on selling them. “Every company is going to go for the [product] that has an easy subscription revenue model. And that means you have a massive ton of marketing around all of these tools that are kind of like, ‘Oh, let’s have a little text box on a website.’ And so it means mostly that people have never heard of these local models.”

04.20 Local models are already good enough for most use cases. Pete compares the moment to the early web, when free alternatives like Apache eventually overtook expensive commercial servers. “All of these alternatives, once people actually had time to look around and they had a little bit of time to improve, they just wiped the floor with the commercial [offerings],” he points out. “I don’t know if we’re going to quite get there, but that’s the kind of pattern that I’m seeing.”

07.26 “The hardware barriers are a lot lower than people think.” Ben and Pete discuss what hardware you actually need to get up and running, from parameter counts, quantization (Q4, 8-bit), and VRAM requirements to the new Apple M5 Studio’s unified memory as a way to run very large models locally at usable speed. “The key thing is whether you can fit [your model] into your graphics card’s memory,” Pete says. “So with weight quantization, 9 billion [parameters] if it was 8 bits is like 9 GB. A lot of mid-end decent laptops that are shipping now have more than that.”

18.33 “It’s not that people don’t like voice interfaces. It’s that people don’t like bad voice interfaces.” We’ve solved most of the big problems, like dealing with background noise, phrasing, and speech in a range of accents—or at least have improved tools’ capabilities. However, “there’s no commercial incentive to kind of pull them all together,” Pete says. Most tools feel like they haven’t caught up to the LLM era, but “open source can be a really strong lever” to updating them, argues Pete.

28.26 We’re navigating the split between “LLM maximalist” end-to-end models (favored by big AI companies with the most capital) and the “compound AI” approach of chaining together specialized models from different sources. “If the future is end-to-end models, then only the people with the most money can actually build and train them,” Pete notes. Compound AI lets you “actually train all of the models independently” to accomplish your particular goals. While the performance of end-to-end models continues to improve, especially for multimodal models like Qwen or Gemma, using one can be a bit like choosing a Swiss Army knife over a tool specially designed to accomplish a single specific task, to use Pete’s metaphor. It may get the job done, but it’s probably not the most effective way to do it.

36.10 Voice capabilities in the browser could be a game changer. Embedding a model directly in the browser—Chrome has a built-in ~4B parameter model that’s accessible from any website via JavaScript, for instance—makes it part of the operating system. “Once you are able to transcribe fast and accurately in the browser, it’s a way for people to easily start experimenting with this stuff,” Pete explains. Could this be an iPhone moment for LLMs?

39:58 The “gravitational pull” is toward on-prem. Unlike most recent technological advances that depend on the cloud to function, LLMs are well-suited to running locally, even with no internet connectivity. Enterprises are grappling with concerns about cost, privacy, capabilities changing with no notice, or even the models they depend on disappearing. Hosting your own model, whether on your laptop or in your corporate infrastructure, gives you the stability to plan for the long term.

44:21 GPUs are fantastic for training but “complete overkill for inference.” Pete likens it to “trying to use an oil tanker to go and do your shopping.” Memory bandwidth is the real limiting factor, and it’s a problem that companies like Apple, with its new chip designs and unified memory bandwidth, are working on solving. “Even if you’re running on the CPU, if you have something that’s got high-enough bandwidth to pull 27 billion weights in a fraction of a second, then the rest of it is fairly easy in terms of actually doing the processing,” Pete says. “I think we’re going to see a lot of really imaginative solutions now that people understand what the workload looks like.”

The following article originally appeared on Addy Osmani’s blog and is being republished here with the author’s permission.

In the past year, the conversation around agentic engineering has moved to harnesses and loops, fleets and software factories. My 2 cents is engineers need to own the outer loop—the accountability for these systems. This only gets more true as powerful models like Fable and GPT-5.6 become available.

Agents have leverage, and leverage creates obligations. Someone must be able to explain exactly what changed, why it was safe, and what will happen if they’re wrong. Otherwise, their actions can’t be justified. Which makes it unlikely their organization will ask for them in the first place.

And so I want to talk about three terms. The first, Quality, refers to all the checks we install before we let the system loose. Those checks produce evidence, and from that evidence we derive a Verdict.

The second, Verdict, refers to the final decision we make before work enters our dependent system: I’m the line-producer of this content. I run the team whose work is shipped under my name. The model may write the line, but the Verdict is mine. The work of my team will not enter our dependent systems without my decision. A Verdict is the production decision: Should we ship, block, redirect, narrow the response, add a guardrail, or reject outright?

The third, Answerability, refers to the guarantee that if someone asks, I can explain why.

To say this another way: Our agent (which I define as a model plus a harness of files, tools, memory, skills, sandboxes, permissions, observability, and recovery) is what runs our loop (which I define as investigation, implementation, verification, and repeat). And it’s what creates our software factory.

The model is just the engine. The harness—tools, memory, permissions, sandboxes, tests—is the car you build around it so it can do real work safely.

The loop is how one good run becomes a process you can trust to run again. Wrap that harness in a repeatable cycle—investigate, implement, verify, repeat—where an independent check, not the model’s own say-so, decides when the work is done.

Now run many loops at once. A factory is loops at scale: The agents ship the work inside, while humans own the decisions at the boundary.

And at the heart of that factory is a careful boundary between what’s inside the system and what’s outside it. Inside the system we collect inputs (from the product team’s intent, or knowledge of previously shipped work, or of recent incidents, or of specific feedback from users). The agent loop investigates the task, implements a plan, and verifies the result. Then, evidence crosses that boundary. A human, who owns the dependent system, sees the evidence and decides whether to proceed.

And that, friends, is the shift we’re trying to make. Before, our agents were doing the inner loop of the execution loop. Now they run the inner execution loop. Engineers own the outer loop.

Inside the system, there’s really just one kind of thing our agents are doing: capability. The capability to investigate tasks, implement plans, test their results, and report back. That’s the capability of a model. And as we’ve said, that future is already here.

Outside the system, there’s a single kind of thing: agency. The agency to decide, verify, approve, and own.

We’re still talking about code, you see. It just needs to live in a place and be performed by people who know what they’re doing.

The potential for AI code is no longer marginal. In a Sonar 2026 survey, we asked teams about the share of their commits that were AI-assisted. It was small but nontrivial. And several of the respondents said they expect the share of AI-assisted commits to grow substantially.

Sonar’s 2026 State of Code report found that 42% of committed code was AI-generated or significantly AI-assisted, with expectations for that share to keep growing rather than plateauing.

Creation, in other words, is getting cheaper. Scarcer resources are review, validation, understanding, and maintenance.

We moved the speed of generation faster than we moved the speed of control, and so we have a trust-verification gap. A lot of people we talk to still express some degree of distrust in AI code. Yet fewer of them seem to consistently build that distrust into their verification processes.

That’s a dangerous place to be. We’re going to need cheaper, clearer ways to verify the trustworthiness of AI code.

If you look at the GitLab June 2026 report, you’ll see that governance questions have shifted.

GitLab’s June 2026 AI accountability research shows that review and validation are the current bottlenecks when using AI and, more worryingly, that governance usually happens after code creation, after we’ve accepted the risk and lost control over ownership. Today, it’s not just about control. It’s about what constraints we set on the system. It’s about how we’ll check the work with evidence, and how we’ll hold teams accountable. It’s about who will own what part of the AI lifecycle.

So the final distinction in this series is between process and quality. Quality is the concept of backpressure. We mean it literally. We don’t want to grant our agents as much autonomy as they can possibly exercise. We want to grant them just enough autonomy that we have enough backpressure to stop them, regulate them, check their work, and ensure our humanity.

Ordinary engineering holds up a lot of signals that indicate that the work being done is doing the right thing. Type checks, tests, hooks, sandbox limits, audit logs, monitors. Our engineering systems are full of these kinds of signals, and they’re designed to provide enough backpressure to keep the system honest.

And so as long as our agents are emitting these same signals, we can trust our ordinary engineering to provide appropriate backpressure.

Trusting our systems doesn’t mean we don’t want a human in the loop. It just means that the human doesn’t need to be in the inner loop. We want them in the constraints loop (What inputs, architectures, instructions, or invariants should we set?), the sampling loop (How much output should we sample and review?), the audit loop (What evidence should we keep, and how do we make sure our audit log is effective?), and the ownership loop (What part of the production boundary should we own?).

But the human doesn’t need to be in the inner loop.

The agent can ship more than you can review.

And the scarce resource is your own core human judgment, informed by quality signals like logs or tests.

The AI June 2026 report shows that, in the experimental setting, agentic delegation along hour-scale time horizons is essentially here. The work by OpenAI this year on agents and the future of work was a great source for these ideas. So we need to start thinking about how to establish this ownership boundary, as our systems start shipping more than we can review.

And that’s where the answerability comes in.

Because with long-horizon agents, the decisions made over hour-scale time horizons are just that—decisions. And not all the decisions are going to be recorded. You can’t trace them all back to input tokens. If all you’re doing is trusting that the output you get is the correct choice for the problem at hand, the hundreds or even thousands of human hours of work you’re going to need to reconstruct the chain of decisions that lead to it become impossible. And so, again, answerability becomes something that must be at the core of our system design.

Three hidden costs

And there are three hidden costs:

Cognitive surrender ~ blindly accepting what AI gives you. When you delegate work to an agent, the work itself may appear to be the work of the agent. But it’s actually your work. It’s your reputation. It’s your responsibility. And it’s your software that suffered the defects in the output. And it’s your software that needs to be changed to reflect that output. So the agent’s output becomes your answer. And with it comes all the accountability. The Wharton study that put this together is reassuring when the AI is right. But when it’s wrong, the news isn’t great. When the AI was wrong, nearly three-quarters of people accepted it anyway, and felt more confident than they would have without the AI.

Cognitive debt ~ erosion of your understanding and memory of how to solve problems. When you delegate work to an agent, you’re offloading all the thought work to the agent. And while thinking it all out yourself takes time and energy, thinking it out on a massive codebase takes resources that aren’t available when you’re trying to run up the learning curve. So the output you get is often unattainable by you. And the longer the time horizon of the agentic planning, the bigger the gap between the code the agent produces and your understanding of it becomes. The gap compounds. The debt accumulates. And the cost of climbing the learning curve grows almost exponentially.

There’s a randomized controlled trial from Anthropic looking at whether engineers who lean on AI to write code understand it as well as engineers who write it themselves. The conclusion was gloomy: On a comprehension quiz, the engineers who worked through AI scored 17 percentage points lower than those who didn’t, 50% versus 67%.

And then there’s the orchestration tax: It’s easy to spin up lots of agents now, but your cognitive bandwidth doesn’t parallelize in the same way. Steering your agent away from the worst behaviors, sorting the work the agent produces to identify the ones that need your attention, directing it to focus on the work you care about first, verifying your most important constraints and your most dangerous assumptions before you let it run. . .

All of that takes work, and it can’t be automated. There’s no substitute for human judgment.

Brownfield systems are especially dangerous here, because the system behavior you have to audit doesn’t live in the code. It lives in the scars.

Fixes? Make attention the priority in your architectural decisions. Use worktrees, scopes, and evidence to reduce the coupling between your initial plan and the work that emerges from it. Time-box the effort to resolve unactionable steps. And make change in your software strictly an opt-in permission.

Alpha, decay, and taste: These are the three core patterns that shape careers and performances across domains.

Alpha is the lead part taken up by the highest achiever in the competition, when you’re playing your highest-value game move. Decays are established patterns that everyone learns through repetition and watching others (plateaus, if you like). Taste is the earliest we can sense the lead in an alpha or the change in a decay. It’s our judgment of what’s coming before we have any evidence that anything is happening.

Paul Graham’s point is that when anyone can make anything, choosing what to make matters more, and Mitchell Hashimoto’s definition is the operational one: making high-quality qualitative judgments where no objective metric exists yet. From now on, taste drives everything. Alpha shifts are taste changes. And decays fade out because we start to taste something different.

Next step? Operationalize your taste. How? Give it a name that reflects what you’re trying to move from limbic to conscious. Practice it in critique and examples. Make its rationale explicit.

And keep making the move that delivers the most durable competitive advantage in your industry. What’s that? Keep moving the edge up from just doing the task to teaching it, systematizing it, deciding when it should be done, and owning the result.

Everyone is a developer, but not everyone is an engineer. Engineering is what a developer turns into when they embrace a work discipline that is more strict: thorough and logically sound reasoning, consideration of constraints and tradeoffs, recognition of risk and exposure, and practical accountability.

In the future, people will leave the administrative work of engineering and embrace new roles that emerge as engineering becomes more demanding. Roles that are unbundled from the spirit of craft but make clear what each person does. There will be those who prototype. Those who build. Those who sweep. Those who grow. Those who maintain.

The humans hold the edge of the system in the other direction too. Increasing the alpha: choosing what is worth doing, defining the constraints within which it should be done, deciding if the evidence is sufficient to proceed, and caring for the result. Whether it’s a single team or a hundred teams, this is the edge that only humans can hold.

Accountability will scale the factory. Like attention and taste, accountability is also one of the three dualities that makes everything work. Without accountability, there are no rules. No wrangling with questioners. No trade-offs. No risks. No safety nets. If nobody owns the consequence of a decision, then high agency can only bring chaos.

The half-life of an edge is one release, but the half-life of a signature is a career. A signature is your name on the work, such that you feel you can stand behind what was shipped. Skills get you leverage; accountability turns leverage into trust.

Only people can choose. Only people inherit consequence. Agents can be asked to choose, route, merge, and escalate safely inside a policy, but they cannot inherit the consequences.

Every codebase should perhaps come with some kind of accountability contract that explicitly states the checklist that was understood when the change was accepted, the evidence that went into the decision, who was accountable for the change, and the system status after the change was blocked. Just like:

• Your attention and taste
• Your evidence, verdict, and ownership
• Your alpha, decay, and taste

High agency

In a typical agentic workflow, high agency is the art of knowing when to delegate, when to inspect, when to stop, and when to own the result of a process. The ladder of agency runs from low to high: flag a potential problem, investigate it, execute against it, diagnose it, propose solutions, recommend fixes, and resolve the issue. A high rung on the agency ladder is discernment: found it, it’s not worth fixing, moving on.

The 12 pillars that hold up the software factory

Brownfield is the frontier for factories that hope to scale. All those clever little innovations may not feel like much yet, but the production environment is a lot. When building an entirely new system, it’s much easier to plan and implement sufficient backpressure mechanisms because you have full control. When you’re adding intelligent agents to a legacy system, however, it’s another matter entirely.

Legacy systems include the entirety of production behavior, future expectations from customers, migration histories, release and budget cycle durations, unspoken assumptions, edge cases, data weirdness, runbook procedurals, and all the scars that accumulated without the will to care for the system.

To be a steward of brownfield requires a form of durable engineering. Work has to be done to turn implicit knowledge into explicit constraints, keep it coherent across teams and through generations, formalize that knowledge into test procedures and functional specifications, and tie that knowledge to objective evidence. All while ratcheting failure into more learning. Because if the system doesn’t get the care it has always received, everything will come crashing down.

New work is real work

The work will get more interesting as you scale. Because when everything else is built, people will want to build new things. They’ll want to employ the alpha and taste they have developed through their craft to design new loops that can be grafted onto the software factory. Or they’ll want to build greenfield systems that employ all the knowledge of the software factory to one elegant, well-meaning, principled effort. They’ll want to design and implement new forms of evidence that will rise to the level of verification for the new systems. They’ll want to take care of brownfield systems that are now so complex they need dedicated attention. They’ll want to design and manage new backpressure mechanisms. They’ll want to design new agents. And they’ll want to build agency.

And, as they do, they’ll come to see that all this is real work. That’s a good thing.

Automation creates bottlenecks. Bottlenecks in production that are worth owning. Because automation gives us control over industrial scale. But there’s also new bottlenecks that arise from industrial scale. The bottleneck moves from “Can we build this?” to “Should this exist? Can we answer for it?”

What I’m suggesting is a practical operating model for scaling agentic engineering. There’s inner and outer loops. The inner loop is where the work is done. Loops are designed to be as independent as possible. Put all quality assurances and verification inside the loop. Once you’ve designed and validated the loop itself, the only thing you have left to do is to grant autonomy by putting in place a back-pressure mechanism that acts to control the rate at which the loop is run and its scope of operation. And put humans in their rightful place, on the right decisions. Don’t treat understanding as a hand-off or a release gate but rather as a point of decision where humans are primed to provide their insight. And then for every artifact that exists and is fed back into production and into new teams and engineers, leave behind better artifacts.

Build the factory; keep the lights on; make work legible, verifiable, owned.

An agent can write it. But before it reaches users, someone must explain why it should exist, why it’s safe enough to be part of production, and what they will do when it is wrong.

That’s agentic engineering at the outer loop—that’s the work now.

The following article originally appeared on Sean Goedecke’s blog and is being republished here with the author’s permission.

In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet. Today, everyone can write sort-of-okay CSS by delegating the task to an LLM. LLMs make everybody into a generalist.

Because of this, lots of people don’t think there’s any skill involved in working with LLMs. If you want the product that LLMs can deliver—PhD-level mathematics, pretty good but sometimes tasteless computer code, or awkward LinkedIn-style writing—you can simply ask for it. Since everyone is talking to the same models, “skilled prompters” are getting the same results as people touching LLMs for the first time.

This is wrong. The most important skill in prompting is expertise in the domain you’re prompting for.

A good illustration of this is Terence Tao’s conversation with ChatGPT about the recently discovered counterexample to the Jacobian conjecture. This is not the same ChatGPT I talk to! I couldn’t get to where Tao gets, even with unlimited tokens to burn.

There’s a lot to learn about good prompting from Tao’s conversation. Here are a few observations:

• Tao’s messages are very short and to the point. He doesn’t respond point by point to the model, just to the gist.
• The model outputs are much more concise than when I try to talk to GPT-5.6 Sol about mathematics. By signalling expertise, Tao shunts the model into “talking-to-mathematicians” mode, not “explaining-to-amateurs” mode.
• Tao pushes back when the model’s responses look wrong, but he doesn’t directly contradict; instead, he says things like “This looks more complex than I was hoping for.”
• Tao makes several leaps and suggestions himself. He almost never takes the model’s advice about where to go next.

However, you can’t prompt like Tao on mathematical questions just by following these tips. The key to his technique is actually understanding the mathematics: pulling the relevant idea out of ChatGPT’s multiparagraph response, suggesting alternate approaches or formulations, and identifying what “looks weird.”

Terence Tao is a better mathematician than I am a programmer. But the idea here—that domain knowledge makes you better at using LLMs—is something I’ve also experienced in my own work. If you have a good theory of your codebase, you can push the LLM much harder than if you have no familiarity. Because you have your own sense of what a good solution might look like, you can say, “No, I think it could be simpler here” or “But don’t we already do X?” or “Can we express this problem in these familiar terms?”

This touches on an idea I’ve written about before: that system design problems are dominated by concrete specifics, not generic principles. Of course both are useful, but I’d rather have familiarity with the codebase than a deep general understanding of software systems. In his conversation, Terence Tao asks a lot of specific questions like “Does X work here?” or “Given Y and Z, why A?” I can’t ask those questions about the Jacobian conjecture, but I can ask them about the systems I own at GitHub.

If you have no domain knowledge, you can cling onto the LLM to at least get something. That’s not bad! But if you have domain knowledge, you can wring far more value out of the same LLM by steering it hard in the direction you want. Most of us will have to do a mix of both these approaches, since we have domain knowledge in some areas but not others.

The usefulness of domain knowledge suggests that human expertise will continue to be useful even as models get stronger. For many tasks, the human is the bottleneck, not the model, because the difficult part is in communicating to the model exactly what kind of solution the human wants. The information is “in the model” already, but it takes a very smart human to pull it out.

This post got many comments on Hacker News. Some commenters share their anecdotes about how expertise has helped and lack of expertise has hurt. Other commenters say it’s plausible, but they have a sensible suspicion of a view that’s reassuring them about how they’re still valuable. I agree with that, though I suspect by the time we get around to studying this, the landscape will have changed under our feet again. Some commenters point out that OpenAI’s math prompts were inexpert, and so expertise isn’t required. Here I’d respond that OpenAI does have a team of expert mathematicians that checked and filtered the model’s suggested discoveries, and that you cannot currently skip that step.

In this episode of Zero to Agent in 30 Minutes, Microsoft senior solution engineer Maxim Salnikov walked through the Agent Package Manager (APM), a terminal-driven open source product from Microsoft that treats agent context the way modern software already treats its dependencies: versioning it, pinning it, and checking it before it ships. It’s a technical session, but rather than building an AI agent, you’ll discover how to manage all the customizations you’ve installed for your agents and make them portable, secure, and governed by policies you or your company define.

How to build a supply chain for agent context

Maxim demoed the process of setting up and using APM step-by-step. Here’s how it works.

1. Keep every customization in your codebase. Agent skills, custom instructions, and hooks tend to spread across a developer’s machine, a harness’s install folder, and enterprise-level defaults. Maxim recommends storing them in the codebase instead so that anyone on a new machine or using a new harness ends up with the same setup as everyone else. Yes, your codebase will be bigger, but the trade-off for consistency is worth it. But be sure that skills and other customizations are secure before adding them to your codebase. Consider building your very own enterprise repository of the customizations you’ve identified as safe to use. (See step 5.)
2. Install APM and initialize a project. APM works like npm for AI agent customizations. Running apm init sets up an apm.yaml file targeting one or more harnesses, such as GitHub Copilot, Claude Code, or Cursor, and apm install pulls a skill from a repository into the right location for each one. When you install a skill, APM also creates a log file documenting the entire resolution history.
3. Pin to a version or a commit hash. Once a skill is installed, pin it the way you’d pin a library dependency. Doing so protects against a scenario where a legitimate skill later gets compromised and a bad actor slips malicious content into a new version under the same name. APM also scans files for hidden Unicode characters that may contain malicious instructions.
4. Reproduce the exact setup with a lock file. APM logs every file and hash it installs. Running apm install --frozen rebuilds that exact environment from the log instead of reresolving everything from apm.yaml, so a teammate’s machine ends up with precisely the same setup as yours.
5. Source customizations from a vetted registry, and set a policy for it. Public repositories work for experimentation, but Maxim suggests enterprises build a private, gated registry of reviewed skills. APM policy files enforce which sources are approved, so a customization from an unvetted location gets flagged.
6. Audit for drift, then enforce it in CI. The apm audit command checks installed customizations against policy and catches unauthorized sources or content that has changed since installation. Run that same audit as a gate in a CI/CD pipeline to protect the entire organization against skill drift and bad actors.

The software supply chain already has decades of tooling behind it. That discipline hasn’t caught up with agentic AI, but APM is attempting to close that gap. Explore the project GitHub repo and get started.

Coming up next

On September 9, Menyala’s Sajal Sharma joins Zero to Agent in 30 Minutes to build a shared knowledge base that acts as a common brain across agents. He’ll show how a single repository of research, daily logs, and notes can give Claude Code, Codex, OpenClaw, and Hermes access to the same accumulated information instead of starting from zero with every new session.

Follow along with Zero to Agent in 30 Minutes on Radar, or watch the latest episode on YouTube, Spotify, Apple, or wherever you get your podcasts. If you’re an O’Reilly member, you can watch live. Save your seat.

Dashboards don’t interpret themselves

AI and real-time data give organizations more information than ever. Dashboards. Live metrics. Instant customer feedback. All of it moving fast. But data doesn’t make decisions. People do. A dashboard can tell you that cart abandonment jumped 12% this week. It can’t tell you why, and it definitely can’t tell your marketing, product, and support teams what to do about it together. That takes a conversation. It takes people in a room—virtual or real—looking at the same thing, arguing about what it means and what to do next.

The map freezes a moment so you can talk about it

Experience mapping is a broad field of visualizing human experiences. You’re probably familiar with things like journey maps, service blueprints, and other similar diagrams of the experiences. But none of them hand you an answer. What they do is take a fast-moving, chaotic situation and freeze it for a moment. It gives a team something to point at, argue about, and align around.

Picture a typical working session. People from different parts of the business sit down around a map of the customer experience. Each of them already knows a piece of the picture. None of them has the whole picture, together, at the same time. That’s what the map provides. That’s usually the moment something surprising surfaces. Not because the map contains secret information but because it puts scattered knowledge in one place, in front of the people who each hold a piece of it. The visual aspect of maps is critical. Laying out an abstract concept like a “customer experience” allows teams to engage with it in new ways and reach new conclusions that are hard to get from a spreadsheet or data alone. Grasping cause and effect in one visual overview helps teams find the patterns of behavior that matter the most and to conceive of viable interventions.

AI can surface these kinds of patterns in seconds. But it cannot create the moment when a cross-functional team collectively recognizes how its silos are hurting customers. Only people, looking at the same picture, can do that.

The map isn’t the point

Some claim that journey mapping is dead. That static maps can’t keep up with real-time data and AI-driven personalization. This confuses the artifact with the activity. A map that gets built, presented once, and filed away never helps anyone. It fails for the same reason a report fails: Nobody’s talking about it anymore. The value was never in the diagram. It’s in the conversation the diagram makes possible.

Take how I got that team to reach their own conclusions about the invoice problem rather than just telling them about it. After scoping the customer type and situation we wanted to understand better, I interviewed a dozen or so customers about their billing experience. Nothing unusual came up at first. People described the routine steps: get an invoice, check it, pay it. But a few mentioned something in passing. They’d disputed a charge and kept getting late payment warnings anyway, even while the dispute was still open.

From those interviews, I built a draft map of the invoicing journey. I called it a draft on purpose. I didn’t want to hand stakeholders a finished diagram and ask them to approve it. I wanted them to lean into it, question it, and add to it. Then I scheduled a working session. The room included people who’d never worked together before, despite being at the same company for years: billing, support, and product.

We didn’t rush through the map. We slowed down, section by section, and used structured exercises to pinpoint the moments that mattered most to customers. That’s when someone in the room realized: A customer who’s actively disputing an invoice can still get a warning notice for that same invoice. Nobody had designed it that way on purpose. It fell through the gap between two systems that never talked to each other. But once it was visible, laid out in front of the people who owned each part of the process, it became impossible to ignore. The room got quiet, then loud. People were genuinely upset, not at each other, but at what customers were going through.

Of course, I had uncovered this already in my research. And sure, it was also visible on the map. But my diagram wasn’t about giving a magic answer. The process of learning together is the point. That reaction didn’t come from a dashboard. It came from people confronting the evidence together, in the same room, at the same time.

What actually changed

Before the workshop, this problem was invisible in a specific way. Support knew customers complained about warning notices. Billing knew disputes existed. Product knew the systems didn’t sync. But no one held all three pieces at once. The map put all three in the same field of view. That’s the mechanism. Mapping doesn’t create new information. It puts existing, scattered information into one shared picture, at the same time, in front of the people who each hold a piece of it.

What changed after that: Billing and product agreed to flag disputed invoices so no warning could go out. Support got a way to check dispute status before responding to a complaint. And the three teams kept meeting monthly, something none of them had done before. The map didn’t do any of that. The conversation the map created did.

What good collaboration looks like

We started with customer evidence and a deliberately unfinished map. We included people who owned different parts of the experience and asked them to question what the map showed, identify what they knew and what they were assuming, and examine the gaps between their systems. The session ended with specific commitments, and the teams continued meeting as they learned more.

That is what getting collaboration right requires: the right people, shared evidence, visible disagreement, clear ownership of the next decision, and a cadence for revisiting what the team thinks it knows. Without those conditions, mapping can easily become another workshop that produces an attractive artifact but little change.

What this means for your team

As AI speeds up execution, don’t cut the time you spend aligning as a team. Protect it. Expand it. AI won’t give you an edge. Your competitors have access to the same models you do, trained on much of the same data, producing much of the same output. If everyone moves at the same speed, speed stops being an advantage. It becomes the minimum bar for staying in the game. AI also works like a spotlight, amplifying whatever’s already happening in your organization. If your teams collaborate well, AI makes that strength visible fast. If they’re siloed, AI exposes it just as fast.  Now is the time to get collaboration right, while staying focused on the customer. Waiting until AI forces the issue is waiting too long.

In the end, AI can help with customer discovery and accelerate insights. But it doesn’t replace human judgment and decision making. Rallying around a map—a visual depiction of customer experiences—provides a natural forum for discussion, debate, and shared understanding to align before acting. The tools will keep getting faster. The organizations that win won’t be the ones with the best dashboards. They’ll be the ones who are best at coming together, again and again, to make sense of what those dashboards show them.

If you want to dive deeper into mapping, join Jim on October 9 for his Beyond the Book conversation about the latest edition of Mapping Experiences. He and host Vicki Reyzelman will chat about how experience mapping has evolved from a UX technique into a strategic capability for organizations, how AI is transforming the way we create and analyze maps, and how you can use mapping to align business goals with customer needs, facilitate collaboration across teams, and drive transformation at scale. It’s free to attend. Register now.

Host Christina Stathopoulos, founder of Dare to Data and a former data scientist at Google and Waze, returned to This Week in AI with developments that stretched from Claude testing ways to improve model safety to Chinese open weight models gaining developer traffic and new systems learning to model physics. She also examined what Anthropic and OpenAI’s business moves, workforce forecasts, and debates over access reveal about how quickly the AI landscape is broadening.

Claude is taking on more of the research process

Anthropic provided an early example of AI helping improve future AI systems. In research Christina highlighted, Claude searched existing work, proposed methods, generated training data, and repeatedly tested and refined its approaches to reduce unwanted model behaviors. The experiments covered 10 such behaviors, including deception, hallucination, prompt injection, privacy violations, and reward hacking. Anthropic reported improvements across all 10 without degrading the model’s broader capabilities.

For deception, Claude tested more than 150 methods and eventually closed 85% of the measured safety gap. Human safety researchers closed only 20% in Anthropic’s comparison. Christina emphasized that this wasn’t a direct contest because Claude could run and refine experiments much faster and at a much greater scale. She also cautioned that the work didn’t amount to full recursive self-improvement.

This showed how AI could increasingly handle experimentation in model development, changing the pace and scale of research while humans still set the goals and evaluate the results.

Model performance is only one part of the frontier race

Competition among AI labs increasingly involves business performance, infrastructure, and deployment options alongside model quality. Anthropic estimates that the market for its systems could eventually reach $30 trillion, a long-term estimate that Christina treated skeptically because it approaches the size of the entire US economy. She also highlighted more concrete evidence of momentum in how Anthropic’s annualized revenue run rate rose from less than half of OpenAI’s at the start of the year to surpassing it within several months. Both companies are preparing for possible public offerings.

OpenAI faces a different set of pressures, and Christina highlighted its 14 executive departures this year. That sustained leadership turnover could raise questions about the company’s ability to execute consistently. OpenAI is also trying to gain more control over its infrastructure. Its Jalapeño inference chip, developed with Broadcom, delivered up to 1.9 times more AI work per watt and up to 3.6 times lower latency than comparable NVIDIA systems in OpenAI’s own testing.

Chinese open weight models are widening the field further. Christina cited an AI gateway where open weight models recently reached as much as 62% of developer traffic on a single day, compared with an average of roughly 10% in April. The competition now spans benchmark performance, capital, infrastructure, cost, deployment flexibility, and organizational execution.

Physics models could extend AI beyond language and images

Christina closed with research aimed at helping AI systems model physics. Researchers from MIT and Tsinghua University developed a pretraining approach that learned from more than one million synthetic interactions between moving particles and complex 3D objects, then applied those patterns to simulations involving wind, water, collisions, and light. The researchers described physics as a potential “third modality” for AI alongside language and pixels.

She also covered Accelerated Understanding, a startup that recently emerged from stealth with an architecture based on neural operators rather than transformers. The company is targeting problems involving enormous physical datasets, including chip design, robotics, extreme-weather forecasting, and geological exploration.

By learning directly from physical systems, these models could become valuable for simulation, engineering, robotics, forecasting, and other work that depends on understanding complex real-world environments.

What’s next

Christina also examined who could benefit from these advances. She discussed Bill Gates’s argument that access, deployment, policy, and distribution will shape AI’s social impact, and brought in new US Bureau of Labor Statistics projections showing job growth in areas including technical services and healthcare, while office and administrative roles face greater pressure from automation.

Her larger point was that access, workforce preparation, and public policy will determine how AI’s benefits and disruptions are distributed.

Due to the Labor Day holiday, This Week in AI will return on Monday, September 14, when we’ll dive into more of the news, issues, and key developments shaping the AI era. And check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

That’s why, three months ago, I built my own software factory, Squid, to ship all of Decoding AI’s small and medium projects with minimal human intervention. The first version was so overbuilt I stopped using it.

Meanwhile, I kept seeing people obsess over the next “_____ engineering” label, instead of focusing on actionable outcomes. Prompt engineering, then context engineering, then harness engineering. So far, so good. But in the last few weeks (July 2026 as I write this), things got off track with loop engineering and graph engineering, which already read more like marketing talk than like anything that solves real problems. Graph engineering overtheorizes how teams have built AI applications since the LangGraph era kicked off in ~2024. Don’t get me wrong. The terms aren’t wrong (Boris Cherny, who leads Claude Code at Anthropic, says, “My job is to write loops”), but we’re overexplaining intuitive things we started doing years ago.

While you’re defining what counts as a loop, you’re not thinking about the processes that actually deliver software.

The right frame is the software factory, which was one of the core themes at AI Engineer World’s Fair 2026, where Tereza Tížková (growth at Factory.ai) defined one as “the whole loop, the whole lifecycle of developing software with autonomy.”

I bet you already have an intuitive sense of what a software factory is. In this article, I want to further formalize it and map it onto the software development lifecycle (SDLC). We’ll explore how big your software factory should be, and when to stop automating before it adds more friction than value. Most importantly, I want to highlight where the human belongs in this process, and where I believe they’ll still belong even in a world where all the code is generated by AI.

So…what’s worth automating? Where does the human bring the most value? What’s worth building, and what’s worth buying?

The design of a software factory

Like a physical factory, a software factory automates software creation with minimal human input. Raw work (bug reports, feature ideas, incidents) goes in. Shipped software comes out. It needs a few highly qualified people making high-leverage decisions, and defined gates that work can’t pass through without them.

Factory.ai pitches “a self-improving system for your Software Development Life Cycle (SDLC).” Addy Osmani frames the stack as loop, harness, factory: “The loop is the atom”; a factory is “an org chart made of loops.” Warp’s CEO, Zach Lloyd, states that “software engineering will become factory engineering.”

The factory is made up of eight stages that can be divided into three buckets:

What to build. Triage/intake classifies, deduplicates, and routes incoming work. Brainstorming finds high-impact features through market analysis, user data, and technical research. Planning, the most important stage, turns that research into a polished plan, refines it by letting the agent grill you, and tracks decisions in an ADR (Architecture Decision Record) log plus a glossary. The outputs of this stage are tickets backed by documentation that a team of agents can implement, which can be tracked in plain files or a project management tool, such as GitHub Issues, Linear, or Notion.

At this stage, the agent plans in read-only mode, going through the code, the AGENTS.md file, and most importantly the context layer.

Actual building and checking. Implementing is a software engineer and QA agent loop that goes through the tasks and supporting documentation. Review checks the PR diff against product, architecture, and code standards. Review-CI runs the test suite, and failures trigger a fixing agent. Release handles CD to staging/production with human deployment checks.

Self-improving. Monitor/incident response feeds production signals (alerts, errors, incidents) back into triage as fresh input on what to build next, closing the loop.

Orthogonal to the eight stages, we have the context layer. The layer is especially important at the front of the line. Brainstorming is limited to the data it sees: user analytics, competitor analysis, research, transcripts, and documentation. At this stage, a poor context layer directly limits the space of possibilities you can explore. It has a similar impact on planning, where transforming the raw idea into technical specs and tasks depends heavily on how good the examples in the context layer are. If you want to implement a new product-recommendation feature, and you have zero examples, the LLM will just predict the most common thing to do, which often isn’t the best solution for your product.

The context layer can take many forms. One strategy that is becoming increasingly popular is the LLM Wiki, a term coined by Andrej Karpathy. It’s basically a strategy to transform your data into a structured knowledge base, just by using files, instead of a database. Factory, via its AutoWiki feature, transforms popular codebases into a structured knowledge base that agents can query instead of parsing the codebase itself. LangChain recently released OpenWiki, a CLI tool to manage wikis for agent memory. If you’re curious, in this article I detail how I turn my data from Obsidian, Readwise, and Google Drive into agent memory via LLM wikis.

Where the human belongs

To see where the human belongs, let’s walk through the factory with an end-to-end example. We’ll build a feature for a shopping-assistant agent on an ecommerce platform similar to Amazon’s. The scenario is that usage data says users aren’t engaging with its recommendations, and we have to ship an improvement.

Brainstorm is where taste lives. Agents do the grunt work: They analyze user activity, scan competitors’ assistants, and pull research into the knowledge base. Then a member of the technical staff starts looking at the data, understands why people are not engaging with the recommendations, explores how the competition implements their solution, and proposes a fix as a feature spec. At this stage, the spec solves a business problem. It doesn’t need to prescribe a technical solution yet.

Plan is where a human, with the help of the software factory, transforms the feature spec into an implementation plan. Let’s assume that we want to make a change to the recommender engine algorithm. The human chats with the knowledge base, figures out whether it’s feasible, and thinks through architecture, interfaces, data flow, cost, and latency. They then let the agent scan the codebase and grill them until the plan is properly refined into something that fits the codebase. The output is a bunch of tickets plus an ADR explaining the algorithm change and an update to the glossary.

The agent can help within these two stages by quickly scanning through a ton of data and improving the plan, but the human is still central.

Use the strongest model (Fable) for brainstorming and planning. These stages burn fewer tokens than implementation itself, but everything downstream depends on them. A well-written plan lets cheaper models (Opus, Sonnet) execute without reasoning their way out of dead ends. A weak plan makes them retry until the extra tokens erase the price gap.

With a weak plan, I’ve watched Sonnet on high reasoning out-cost Opus on the same task: The smaller model needs more attempts to reach the same goal. Total cost is tokens × price, not model tier. So more failures equals more reasoning, more tokens, and more cost.

From here on, we move into “loop” and “graph” engineering territory.

Implement runs a software engineer agent that picks up every ticket that’s ready to go. As the loop is scoped to a feature, it takes only the associated tickets. After each ticket is implemented, a QA agent tries to find bugs by stress-testing the application. As agents tend to have a positive bias towards their own work, the split between the software engineer and QA agents matters. As Addy Osmani puts it, the model that wrote the code is “way too nice grading its own homework.” This loop, at solo scale, can be as simple as a bunch of terminals pulling tickets. At larger scale, it runs on remote agents working 24/7.

The loop only works if agents can interact with the app. The QA agent needs one command that starts the whole stack reproducibly. From there, it drives the app in the browser, calls the data or fine-tuning pipelines, or hits your server’s API. Whatever your app’s interface is, the agent needs access to it, the same way a human user would.

The key idea is to integrate feedback loops as natively as possible into your software factory. Ideally, you want multiple levels, depending on how expensive it is to run them: linting, unit tests, integration tests, and end-to-end tests. When the loop keeps failing, the root cause is almost always missing plumbing, not the agents.

Review has three steps. Step one checks the product and architecture requirements against the ticket and the ADR. Any discrepancy becomes a new ticket passed back to the implement loop. Step two ensures code quality (modularity, naming) and guards against AI slop such as verbose comments or cryptic function names. Step three looks at the CI/CD pipeline. At every step, any failure auto-creates a task picked up by the software agent.

Not every project needs all three steps. The “factory” ends with a PR that you as a human need to review and merge. But in reality, if you spend enough time creating a strong plan, the PR that reaches you is usually ready to ship as-is.

So where does the human belong? You’re indispensable during brainstorming and planning, and you return for the final check. Agents own everything in between. OpenAI took this to the extreme: ~1M lines and ~1,500 merged PRs over five months with zero hand-written lines. Their framing is “Humans steer. Agents execute.”

Don’t overbuild the factory

With my first Squid version (my own software factory), I got greedy and chased full autonomy: big remote workflows, parallel agents, and one grand pipeline running end to end. It worked, until something went offscript. Which it usually does. I couldn’t debug it, couldn’t halt it mid-run, and couldn’t redirect it without throwing the run away. It was a big monolith that took me too far out of the loop, and I couldn’t control it.

I realized you need two options. The first is granular commands that let you grill your plan, implement a specific task, or review one particular step. The second, for when you’re comfortable giving the agent 24/7 autonomy, is an end-to-end command that chains all the smaller ones into a fully autonomous graph, such as one big /plan and /implement-review-all command. Basically, each step is a “loop,” while the whole pipeline is the “graph” of your software factory. Still, note how planning and the rest are split into two different commands, as planning is, and always will be, human-driven (at least if you want the result to stay aligned with what you actually want).

Bottom line. You need to be able to step in, halt it, redirect it, and interrupt it, while still having the option to go fully autonomous.

The bottleneck is me, and that’s by design. To be honest, I’ve worked mostly solo since the AI coding agent boom, and I don’t understand who the people shipping 100 features in parallel are. Most of my features (per project) build on one another, which makes them impossible to parallelize. As the project grows, you can find more and more independent features that can be implemented in parallel, but I still believe that number is limited.

That’s why, when I parallelize, I only use local agents, each running in an isolated codebase via worktrees. So far, I’ve never felt the need for 24/7 remote agents, or wanted the overhead of managing them.

A big team can justify more automation, but it has to earn it. So as with any other software product, start small, start by automating the most time-consuming bottlenecks, and add complexity gradually as people get comfortable with the system. Don’t be like me, with my Squid experiment.

Build vs. buy

In all scenarios, you’ll start with a prebuilt coding harness. The most popular vendor-locked ones are Claude Code and Codex. Or go open source with OpenCode or Pi, which took off thanks to its minimalist, extensible architecture that lets you easily build on top of it.

But picking a harness isn’t the same as knowing how to configure it and wire it into your software factory. That’s why everyone needs to know, at least intuitively, how a coding agent works under the hood: the agent loop running in your terminal, what changes when it runs remotely, how you evaluate it, and which context engineering strategies keep it cheap without making it dumber. If you want to learn more about building a coding agent from scratch, consider exploring my open source course on GitHub. Even if you never plan to build your own harness, that intuition is what lets you become a power user.

For a small team, you’ll get extremely far just by defining a set of skills and agents that encode your process on top of the coding harness (a.k.a. your software factory). To keep it simple, this is what I did with Squid, which I use to implement all my projects.

There are other off-the-shelf “software factories” powered just by skills and agents defined in .md files, such as Matt Pocock’s skills repository or the BMad method.

But remember that the factory is mostly about processes, not tools: A factory that doesn’t fit how your team already works adds friction, never gets adopted, and ends up useless.

You cross the buy line the moment engineers you don’t personally supervise run agents. Observability, tracing, cost tracking, and pay-per-token billing stop being optional and become someone’s full-time job. Agent swarms wired into Linear, Slack, and CI across distributed infrastructure are a logistical hell that isn’t your product. That’s when it makes sense to look into off-the-shelf solutions such as Factory.ai (that comes with the Droid agent), or Warp’s Oz. In Warp CEO Zach Lloyd’s words, “Most of the factory is not necessarily a new interface. It is an integration into people’s existing workflows.”

At the other end of the spectrum, you cross back to building when the platform’s constraints cost more than the team it would take to replace it, as OpenAI’s report on its Codex-built product shows.

The smallest builds, the middle buys, and the largest builds again.

What’s next

Someone is already coining next quarter’s “_____ engineering” term as we speak. But the software engineering processes you use to output real code won’t change that often. That’s why you should be open-minded but at the same time focus on actionable outcomes, not on overthinking how to label things.

As Zach Lloyd suggests: Find one “annoying part of your job” and build the smallest loop that handles it.

The harsh reality is that software factories are just at the beginning. They’re far from perfect, and especially far from being fully “autonomous.” Usually, when someone claims they’ve cracked the software factory problem, they either haven’t tested the idea enough or are trying to sell it to you. I’m confident we’ll reach the point where almost the entire software development lifecycle is automated (with the exception of brainstorming and planning), but at the moment we’re still figuring things out.

But here’s what I’m wondering:

Which stage of your factory still needs you the most? I keep automating mine, and the bottleneck stubbornly stays at planning.

Explore next

1. Osmani, A. (2025). “Loop Engineering.” X.
   https://x.com/addyosmani/status/2064127981161959567
2. MacManus, R. (2026). “AIEWF Daily Dispatch: Loops, Software Factories & Forward Deployed Engineers.” Latent Space.
   https://www.latent.space/p/aiewf-daily-dispatch-loops
3. Factory.ai. (n.d.). Agent-Native Software Development Platform. https://factory.ai
4. Osmani, A. (2025). “Software Factories, Light and Dark.” X.
   https://x.com/addyosmani/status/2079442194449232227
5. Karpathy, A. (n.d.). LLM-Wiki. GitHub.
   https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f
6. Abboud, M. (n.d.). “How Coding Agents Actually Work: Inside OpenCode.”
   https://cefboud.com/posts/coding-agents-internals-opencode-deepdive/
7. Kapoor, S. (n.d.). “Building and Evaluating AI Agents.” AI Engineer.
   https://youtube.com/watch?v=d5EltXhbcfA
8. OpenAI. (n.d.). “Harness Engineering: Leveraging Codex in an Agent-First World.”
   https://openai.com/index/harness-engineering/
9. Parsons, C. (n.d.). “Ralph Loops: Build Dumb AI Loops That Ship.” AI Engineer.
   https://www.youtube.com/watch?v=2TLXsxkz0zI
10. Pocock, M. (n.d.). “Software Fundamentals Matter More Than Ever.” AI Engineer.
    https://www.youtube.com/watch?v=v4F1gFy-hqg
11. MacManus, R. (2026). “Warp CEO Zach Lloyd on Why Software Factories Are the Next Phase of Coding.” Latent Space.
    https://www.latent.space/p/software-factories
12. Iusztin, P. (2026). “Building a Coding Agent From Scratch: Harness Architecture.” Decoding AI.
    https://www.decodingai.com/p/building-a-coding-agent-from-scratch-system-design
13. Iusztin, P. (2026). Building a Coding Agent from Scratch Course. GitHub.
    https://github.com/decodingai-magazine/building-a-coding-agent-from-scratch-course
14. Iusztin, P., & Bouchard, L.-F. (2026). “LLM Wikis as Living Memory for AI Agents.” Decoding AI.
    https://www.decodingai.com/p/llm-wiki-agent-memory

Join 44,000+ engineers eager to learn how to build their own software factories by subscribing to Decoding AI Magazine!