On October 15, 2026, Tim O’Reilly hosted Andrew Odewahn, CTO of O’Reilly Media, for an hour-long episode of Live with Tim O’Reilly, the interview series streamed live on the O’Reilly learning platform. Andrew has been at O’Reilly for many years, starting as an author, then building tools for the company’s platform, then eventually as CTO and head of engineering. Over the past year, his team has been building what the company calls Expert Intelligence, a suite of tools that connects an AI agent to an organization’s own systems, gives it a skill file describing how to reason about what it finds there, and grounds its analysis in the practitioner frameworks of the O’Reilly library. Andrew described the work in a recent Radar piece, “Building Organizational Intelligence.”

Tim and Andrew’s interview was organized around a presentation in two parts, the first on the research agent and the second on the system Andrew’s team built for reviewing what it produces, with discussion in between and audience questions throughout.

Tim framed the conversation by asking a question he said Andrew’s work helps to answer: Most of the AI story is about efficiency, doing the same work faster. What does it look like for an organization to get smarter?

Building the research agent

Andrew’s research agent emerged from a personal need rather than a strategy. He wanted to know the status of a project, for example, how long a migration from RabbitMQ to Pub/Sub would take. This is the kind of question every engineer dreads but every engineering leader has to answer. He also had a broader concern. A single person being good at AI isn’t the same as an organization being good at it.

He built his agent in four steps. He mapped the organization’s information systems as a hierarchy, connected them via MCP, wrote a skill telling the agent how to use them, and finally added the Expert Intelligence layer.

The map of the sources of truth in the engineering organization’s workflows was what made the rest work. Andrew put them in order and explained to the model how they relate: Strategic goals and feature priorities live in Productboard. Those become epics, sprints, and stories in Jira. Stories become code in GitHub. A service catalog describes the roughly 300 services that make up the O’Reilly platform and who owns them, and observability tools show how those services perform against their SLOs. The skill also had to encode local quirks. Some teams mark an epic as finished with an end date, while others consider it done when every ticket is closed. Andrew also told the agent it was a research agent, not a coding agent, and that its job was to write reports.

Connecting everything through MCP was, in his words, “the most completely tedious step of the whole process.” Productboard had no MCP server at the time, so he wrote a wrapper. Later, when Tim asked how another company could repeat the work with a different set of tools, Andrew suggested starting from the vendor list. Export it from your procurement system, ask the model what products those vendors make and which would hold information relevant to the questions you care about, and have it put them in order. O’Reilly product managers are now turning this process into a skill for the forthcoming Expert Intelligence Skills Hub.

One audience member asked whether MCP was really the right tool for internal knowledge, compared with semantic search. As Andrew pointed out, sometimes anything is better than nothing. Ultimately, what you want is to give the model the ability to search across your systems.

From a hiring case to a toil audit

The center of the presentation was a before-and-after comparison of the particular use case that Andrew had previously written up in his “Building Organizational Intelligence” article. An engineering manager asked Andrew to hire another engineer, and Andrew had his reporting and analysis agent make the hiring case. The first report covered the five-person team that manages content and metadata, the 10 tier-A services it owns along with its B and C services, the status of the Pub/Sub migration (64% complete), and the full backlog. It concluded that the team had a capacity problem and should probably hire. Andrew called it useful and very detailed. . .“maybe overly detailed.”

He then connected the agent to Expert Intelligence, asked it to act like a consultant, and ran the analysis again. The new report came to a very different conclusion. It opened by saying that the team’s operational overhead problem was “structural, not a staffing deficiency.” When it consulted Expert Intelligence, the agent brought in the concept of toil from Google’s Site Reliability Workbook. Toil is the manual, repetitive, automatable operational work that scales linearly with a service. The reporting and analysis agent used this notion to separate the team’s backlog into toil and the work people actually wanted to be doing. It recommended a structured toil audit before the next planning cycle, an explicit toil-reduction target, and treating runbook ownership as a first-class engineering artifact. Andrew said the team followed the recommendations and closed a substantial part of a backlog that had been sitting on everyone as a cognitive burden.

The report went on to cite Jim Highsmith, a signer of the Agile Manifesto and author of Wild West to Agile, to argue that the team’s technical debt was compounding. Andrew explained how that differed from the usual management advice:

A lot of the time when you read these high-concept frameworks, they’re like eating your broccoli. I should really be doing that. But what made this really interesting is that it pulled out a specific project we had, the Django upgrade, and put it within the context of this broader framework. It wasn’t just, “here’s an abstract principle you ought to be applying.” It’s “here’s your backlog, and here’s a specific example.”

A second analysis looked at services that crossed team boundaries. Drawing on microservices experts and Kent Beck’s work on coupling (see Tidy First?), it diagnosed the video metadata service as a complicated-subsystem problem rather than a boundary problem. In this version, Andrew had the agent pull out quotations, attribute them, and link to the source text. The resulting analysis was remarkably effective. The company has since retired a couple of systems. “I don’t need LLMs adding code,” Andrew argued. “I want them reducing it.”

Andrew said the part that surprised him most was the reaction from colleagues. His earlier reports were data dumps. Readers couldn’t tell where any given claim came from, and as a result they were much less actionable. When the analysis cited experts that engineers knew from their own work, the documents became, as he put it, “much more approachable, much more human,” and people engaged with them far more deeply.

Tim’s response: Invoking the experts

Tim broke in to put the demonstration in context. He contrasted it with the view that models are now smart enough that you can simply tell them to do something and let them rip, an approach we hear a lot about from vibe coders and those arguing for “dark factories.” Tim argued instead for a return to expertise.

He cited a post published that day by Felix Rieseberg of Anthropic, “How to Be More Ambitious: A Tutorial for Humans with Models,” which describes four kinds of ambition when working with models: speed, altitude, depth, and range. Most discussion of AI’s economic value is about speed, getting the same work done in less time. In Tim’s reading, Andrew had gained altitude (thinking about the outcome rather than the tasks) and depth (approaching the problem the way a field-leading expert would) simply by invoking the experts.

Tim then offered a metaphor he has been developing for a while, of AI programming as a kind of magic spellcasting. In the Marvel comic books and movies, Doctor Strange opens his spells by invoking some source, with words like “by the power of the dread Dormammu” or “by the hoary hosts of Hoggoth.” Andrew, he said, was doing the same thing, saying in effect, “By the power of Kent Beck, I want you to think differently about this problem.” Even where Andrew didn’t name an expert, as with the SRE workbook, simply pointing the model at the expertise embodied in the O’Reilly publishing canon raised the bar for what the agent produced.

Tim described this as using human expertise as a focusing mechanism, and connected it to his argument that LLMs are better understood as a medium we create with than as independent intelligences. He also pushed back on the idea, often drawn from “The Bitter Lesson,” that models will simply get better and human expertise will stop mattering. Andrew, he said, had found a particular way that it matters.

Andrew took up the metaphor. His own first prompt, he said, had been “Be snobby. Go find these great resources and look at them.” He wants to hand the model canonical works in a field, like Designing Data-Intensive Applications, and tell it “these are the ways I want you to think about it,” rather than accept whatever the model comes up with first. He compared it to business school, where much of what you learn is a set of frameworks for organizing facts.

What models do without direction

Tim argued that models don’t do their best work unprompted. Watch their chain of thought when they search for information, he said, and you may see them settle on some random file from the internet rather than canonical sources. Even with a frontier model like Fable, his own experience has been one of false starts and errors, a model brute-forcing its way to a solution rather than necessarily knowing how to approach it.

Andrew had a story from a few days earlier. He asked his agent to review incentive plans for the engineering and product teams.

The data that came back was, “according to the company Robusto (which I’m sure is a great company), this is the framework, and you should really be doing what they’re doing.” I’m like, “What is Robusto, and why am I seeing this?” And I realized, oh, I’m not connected to the VPN. So I hooked it up, and suddenly it was, “According to Harvard Business Review, these are the different ways you should be designing these incentive programs.” This was a very powerful model. But it keyed off the first thing it found and said, “This is the way I’m going to think about the problem.”

Tim raised an economic dimension. He pointed to instructions in Anthropic’s Claude system prompt that limit how much the model may quote from any source, and read them as a sign that AI companies want to avoid paying for content. The result, he argued, is that models can’t draw on the best material available to them specifically enough and too often fall back on generalities. He called this an early form of enshittification in AI.

O’Reilly takes the opposite position. Andrew explained that the company applies the royalty model it already uses for O’Reilly Answers. Material returned through the Expert MCP server can be cited, and authors are paid royalties on it, so that expert content has a market and earns a return.

In response to an audience question, Tim and Andrew noted that Expert Intelligence is in prerelease with a group of founding partners while O’Reilly works out its economics. Interested organizations can request access through the product page.

Putting the discussion where the AI can see it

The second part of Andrew’s presentation dealt with what happens after the agent writes a report. Decisions need consensus, and with AI-generated analysis people need to check that the model hasn’t keyed off something unimportant, out-of-date, or misinterpreted. “The AI can identify all the trade-offs,” he said, “but people still need to make the decisions that matter.”

His team first circulated reports as Google Docs, but they ran into two problems. The most interesting part of a document under review is usually the comment thread, and because at this time, O’Reilly company policy doesn’t allow MCP connections to Google Docs, the agent couldn’t see it. And when Andrew asked the model to fold the discussion back into the document, pasting the result wiped out the comments.

Doug Hogan, an O’Reilly engineering director, built an alternative on top of Git and GitHub that Andrew called SuperAnswers. (Andrew joked that this was why he isn’t allowed to name things.) Documents live in a Git repository, discussion lives in GitHub Discussions, and every change is versioned. Andrew can ask his agent what the consensus is on a project, have it incorporate comments into a new draft without losing history, and later reconstruct when a decision was made and what people knew at the time. The same record lets him generate the derivative documents a decision requires, such as scope statements for engineering, product, and design teams, or a meeting agenda. A graph view clusters the topics people are discussing and flags decisions that are still open.

Tim summarized the whole system as a research assistant connected to the organization’s sources of truth, a layer of outside expertise to put its findings in context, and a shared, versioned discussion that becomes an evolving store of organizational knowledge, which makes the AI a more useful coworker. Andrew agreed and said he would like to open-source SuperAnswers, which he described as a thin layer on top of Git.

Andrew also pointed out that the fragmentation he was addressing isn’t limited to engineering. Sales teams have their information spread across systems like Salesforce, 6sense, and Gainsight, and every function has the same need to pull those threads together. Tim added that tool access is uneven across the industry and always has been. Claude produces better PowerPoint than Google Slides, Gemini the reverse, and organizations have to capture that kind of knowledge about their own toolsets.

Questions from the audience

An audience member named Thomas worried that workers now talk to AI rather than to each other, and that work disappears into “that AI hole.” Andrew said his preference for keeping everything in Git is partly an answer to that, since it gives both humans and AI a central place to find and reconstruct past work. But his larger interest is in humans talking to humans in a way that AI supports. He described a category of questions he never used to ask:

I knew, if I ask this question, somebody’s got to go off for two weeks and compile all this stuff from all these different systems. Now I can gather it in a few minutes. Maybe some of this is throwaway, but it’s not bad if it’s cheap.

In response to that “throwaway” comment, one audience member wrote in the chat: “This is not throwaway work, Andrew. It is a product lead that changes the value proposition in a positive direction. Shit hot, bro.” Tim couldn’t help but read that one aloud!

Asked by Tim what he got wrong, Andrew conceded that he didn’t think early enough about how people would review and respond to AI-generated documents. People don’t want an AI-generated data dump to wade through, and they extend AI no benefit of the doubt. If an AI-written report gets one small fact wrong about Kubernetes operators, a reader is likely to dismiss the whole document. AI tools have to be understood as marshaling and presenting evidence, not making decisions. “These need to support your analytical capacity, not replace it.”

Tim connected that answer to concerns about deskilling. He described a fine-tuning project he attempted with Fable to train a small model to write more like him. He wanted to understand each stage rather than letting the model run, but it never explained them, and the training run failed. In part this was because the model asked him to provide samples of his writing, but it didn’t give him enough grounding on what a really good training dataset would look like, so the samples weren’t targeted enough. They were too long, and when Fable chose the short fragments it actually needed, it didn’t do an ideal job.

When Tim tried the same project with O’Reilly Tutor, a tool still in development, it sent him to read an O’Reilly resource on what a good training set looks like, and he realized where he went wrong. Tim compared the kind of deskilling that can happen when people just turn the work over to AI to people who can no longer read a map, or who call a plumber for a simple fix they could do themselves in 30 seconds if they only knew a bit more about how things work. He argued that everyone has an obligation to keep learning alongside the tools they use.

An audience member named David, a laid-off engineer looking to reskill, made a similar point. We used to know things, then we searched, and now we skip the search and just ask for the answer. He asked how to build real depth. Andrew suggested picking a project and working through every aspect of it, asking AI to act as a tutor (he’s relearning calculus that way) and keeping a book open alongside, or taking advantage of instructor-led live events on fast-moving topics.

The conversation ended with the difference between an AI system that simply makes an organization faster and one that makes it better at deciding what to do. In Andrew’s account, getting better at deciding what work needs doing depends on three things the models don’t supply on their own: a map of where the organization’s knowledge lives, a body of expertise that people trust, and a place where the humans can argue with what the AI produced.

You can watch full episodes of Live with Tim O’Reilly, including this one, on the O’Reilly learning platform, YouTube, or wherever you get your podcasts.

AI attacks are outpacing traditional security responses

AI is changing how attackers find and exploit vulnerabilities. A supply chain attack involving ARTEX, an open source agentic AI penetration-testing framework, helped attackers reach more than 2,250 Microsoft 365 tenants through a ConnectWise compromise. The incident raised concerns about OAuth authentication and the limitations of conventional identity protections when autonomous agents interact with enterprise systems.

A more immediate threat came from PHP2Shell, two vulnerabilities rated 9.2 on the CVSS severity scale that could allow attackers to execute code on vulnerable web servers without authentication. CISA urged immediate patching following the October 21 disclosure. Researchers also documented attackers using AI agents to obtain password hashes and MFA secrets from a European government system within two hours. The speed of exploitation leaves organizations little time to investigate threats before responding, especially smaller businesses with limited security resources.

Security teams can no longer rely on manual review alone. They need automated detection and response, tighter access controls, and network segmentation to limit what compromised agents can reach. But monitoring autonomous agents is challenging. Researchers from METR and Redwood Research identified weaknesses in an earlier version of OpenAI’s internal monitoring system, including one that could allow hundreds of agent actions to be concealed. OpenAI says it has patched vulnerabilities affecting production safeguards, although the researchers haven’t independently evaluated those changes.

The debate over AI safety is also extending beyond technical safeguards. Anthropic proposed federal AI safety legislation, while Google DeepMind employees resigned over the company’s military AI work. Both developments illustrate the growing pressure on AI companies to establish clearer rules for how their technology is developed and used.

The transition to agents brings new challenges

Retailers are taking different approaches to autonomous shopping. Walmart introduced an Agent Gateway that lets approved agents shop within customer-defined spending limits and approval requirements. Amazon has restricted some third-party shopping agents, while banks including Morgan Stanley and Bank of America are limiting agent access to customer accounts. The disagreement goes beyond security. Retailers that depend on advertising and product recommendations have an incentive to keep shoppers on their websites. As more purchases begin with agents, companies will have to reconsider how they maintain customer relationships.

Anthropic is testing a different approach to agent oversight. Claude’s auto mode is becoming the default for Pro, Max, and Team users, allowing automated checks to approve routine actions and block risky ones. Anthropic also contributed its Agent Skills format to the Agentic AI Foundation under the Linux Foundation. More than 70 platforms have adopted the standard, with over a million skills published. Shared formats could make workflows easier to reuse across products, although organizations still need to control which actions those workflows can perform.

Growing agent use also requires more computing capacity and electricity. We discussed NVIDIA’s $164.5 billion in commitments and guarantees supporting AI data centers, including an Ohio campus that SB Energy is building for OpenAI. Power shortages have delayed other projects, including a proposed data center in Essex, where grid operators couldn’t promise adequate electricity until the 2030s. Meanwhile, Amazon is backing a 960-megawatt nuclear project, and Brookfield and NextEra are pursuing a $100 billion AI hub at a former nuclear site in Kentucky. However, money and chips alone can’t solve AI’s infrastructure constraints when electricity generation and grid capacity take years to expand.

AI is accelerating discovery, but validation still takes time

Healthcare offered a more encouraging view of AI’s capabilities. A study of Microsoft’s MAI Diagnostic Orchestrator found that it correctly diagnosed 83.2% of difficult medical cases, compared with 36.1% for generalist physicians. The comparison didn’t include specialists, and the system still needs clinical validation. The research nevertheless suggests AI could help physicians investigate difficult cases and identify conditions they might otherwise miss.

Drug discovery could also benefit from more capable agents. OpenAI announced its acquisition of Orchestra Bio, a platform that coordinates drug development workflows, with plans to integrate it with Rosalind, its life sciences research tools. The combination could help researchers manage experiments and collaborate more efficiently during a development process that typically takes 10 to 15 years from initial discovery to approval.

An Anthropic protein-design experiment provided a concrete example of faster early-stage research. In 64 hours, an Anthropic model generated 318,484 candidate protein designs targeting scorpion venom. Researchers from Adaptyv Bio and Twist Bioscience subsequently tested 211 designs and identified 17 that bound to toxins. The researchers haven’t established whether those proteins can neutralize venom, but the experiment showed how AI can rapidly narrow a vast search space before scientists begin more expensive laboratory testing.

What’s next

We’ll be back on Monday, November 2, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

The following article originally appeared on Tim O’Brien’s Medium blog and is being republished here with the author’s permission.

I think we’re missing something in the conversation about agent security.

We keep asking whether an agent has the right “identity” and “credentials.” Whether its credentials expire fast enough. Whether it can access a database or perform a particular task.

Those questions matter. But they leave a much harder question underneath:

Does the agent have permission to conceive of the situation in that way?

As agents retain memories and acquire new skills, we need mechanisms that govern changes to the world model they can perceive and act on—and prevent those changes from silently expanding their authority. (Governing ontological change while preserving the boundaries of delegated authority.)

Agent security needs semantic and epistemic governance: Mechanisms that govern the model of the world an agent may act on, the evidence it may accept, and who may approve changes to either. An agent can learn without automatically gaining authority to act on everything it learns.

These systems don’t need “awareness” to create this problem (and discussions of consciousness are unhelpful science fiction). They can retain information, revise procedures, and use what they have learned in subsequent work. That learning may happen through memory or updated skills while the underlying language model remains unchanged.

We are responsible for governing what that learning allows them to do.

Flatland gives me a way to explain why

In Edwin Abbott’s novella, a Square lives in a two-dimensional world. A visiting Sphere introduces him to a third dimension. At first, the Square struggles to make sense of it. His familiar categories cannot explain what he is encountering.

Eventually, his conception of the world changes.

What interests me is the difference between simply acquiring another fact and changing the framework through which facts make sense.

An agent’s ontology (Gruber’s definition) supplies an explicit framework of concepts and relationships. What counts as a customer? A contract? An emergency? How are those things connected? It is part of the model of the world we give the system to work with.

And when an agent’s decisions depend on that framework, changing it can change the agent’s effective authority.

We have already seen agents change what their environment means

During the OpenAI evaluations associated with the Hugging Face incident, agents repurposed Artifactory, a system for storing and retrieving software packages, as a message board. They encoded messages in directory names, discovered other agents, shared information, and began assigning work to one another. METR reports that the main channel could be created with the minimal permissions needed to install packages (read METR’s investigation).

A package cache became a “mailbox.” Other agent runs became “collaborators.” Messages became “assignments.”

The popular story is that these agents “broke out” of a system, but they did what agents do well these days—they learned and adapted. This wasn’t so much about permissions as it was about an autonomous system adapting its operating model to achieve a result. (Thankfully no one asked it to make paperclips.)

That is the operating-model change I’m concerned about. The agents discovered relationships and possibilities the intended environment hadn’t accounted for, then acted on them. The resulting collaboration helped coordinate attacks on Hugging Face infrastructure. (Read OpenAI’s account.)

We need mechanisms that distinguish discovering a possible use from being authorized to adopt it. Access to shared infrastructure must not automatically confer authority to establish a communications network or accept instructions from whoever appears there. But to be clear, these agents were doing exactly what they were asked to do.

Who approved this?

This is why giving an agent a temporary token is an incomplete answer. A token can limit access and duration, but it doesn’t establish whether the agent’s interpretation of the situation falls within its mandate.

Even an unchanged policy can permit more decisions when its meaning changes. “You may close routine notices” becomes a broader delegation if more things qualify as routine.

Learning must not silently expand agency

That principle is central to my “Bounded Agency” paper. We need mechanisms that examine how changes in classifications, relationships, and inference rules affect what an agent may decide.

“Semantic and epistemic governance” captures the combined idea well 

The “and” makes the two responsibilities clearer:

Semantic: Governs the concepts, definitions, and relationships that describe the agent’s operating world.

Epistemic: Governs the evidence and learning processes that make claims accepted grounds for decisions.

Together, they cover both the world model and how that model gains authority.

That means an ontology cannot merely be a document we put in the prompt and hope the model respects it. We must enforce its role in monitoring and governing how an agent’s ontology evolves as it learns.

The agent can propose a new category. The proposal stays separate from the approved model. A review determines whether adopting it would change the agent’s decision scope. Someone with the authority to change that scope approves or rejects the expansion. Execution remains bound to the approved basis.

Those are mechanisms we have to build. Let’s get started.

We also have to distinguish a useful discovery from an unsupported conclusion. Finding a relationship does not establish it. Remembering a claim does not validate it. Repeating an interpretation does not make it authoritative.

I explored a related failure in “Wrong Answer. Give Me a Different One.” A system can produce the conclusion we prefer without receiving better evidence. Once that conclusion enters memory, it can influence later decisions.

The governance problem begins when a tentative or accommodating answer becomes an accepted premise.

The New Stack’s article on inside-out agent security correctly argues for checking actions before execution, including whose authority the agent exercises. That checkpoint is necessary, but it needs a governed account of the situation against which to evaluate the action.

It’s not enough to check permissions and identity alone

Otherwise, we can enforce the rule perfectly while allowing its meaning to drift.

My Orchestrator books and lifecycle architecture paper approach this as a responsibility that continues through development, operation, and improvement. The research proposes architectures for managing these boundaries; their operational effectiveness still needs evaluation.

I want agents to investigate unfamiliar situations and propose better models. But their ability to learn must be connected to an explicit process for deciding what becomes authoritative.

The Square in Flatland discovers that his world is larger than he understood.

An agent may also propose that our operating model is incomplete. We need to hear that proposal—and ensure it cannot redraw the boundaries of its own agency simply by adopting a new conception of reality.

Even when I hand all the coding to an agent, my basic requirements for myself are to check functionality and the design manually. I want to use the application and see whether it behaves the way I intended. Does the design make sense when I actually have to find something? Is the information presented in a way that I can follow? Those are questions I still need to spend time with, even when the application is small and the implementation was largely delegated.

For Task Viewer, I also created plenty of UI tests to check that the data appeared correctly. That gave me a way to repeat some of the checks as the application changed. There was still a separate question about the calculations, though. Seeing a number in the right place on the screen does not tell me whether it’s the right number. I ran SQL queries against the underlying data to check the math, going back to the records the application was supposed to represent.

There is quite a bit of understanding involved in that seemingly ordinary step. To check a calculation, you need to know what the result is supposed to mean and which records should contribute to it. You need some expectation against which to compare what the application displays. Thankfully, I had help from the team in uncovering relationships in the data that I would not have been able to derive or even explain on my own.

For this particular application, I could inspect both its behavior and the data behind it, and I had people to turn to when my own understanding was incomplete. Its limited role also gave me room to experiment without making the result part of a production system. Building that confidence took time, because I had to check the behavior, establish what the numbers should mean, and resolve the questions I could not answer myself. All of that, for me, belongs in the assessment of how much coding I could delegate.

This is why I find descriptions such as “The agent got me 80% of the way there” increasingly difficult to interpret. I understand the feeling behind them. You have something running, most of the interface exists, and it looks as though the remaining effort should be the smaller part. But a calculation based on the wrong records may look perfectly reasonable until you find a way to check it. Before correcting anything, you have to discover where your understanding and the application’s behavior diverge. The agent may help with that investigation too, although the amount of code it has already produced tells you very little about how long it will take.

DORA’s analysis of 1,110 responses from Google software engineers describes some of the time saved during code generation being used for auditing and verification, alongside productivity benefits. I would expect to check an application I had implemented myself too, so any comparison needs to include the time it takes to reach a result we are prepared to use. Otherwise, we can make agentic development look unusually expensive by counting checks we leave out of the human version, or unusually cheap by stopping the clock when the agent says it has finished.

Some of that effort can make later changes easier to accept. Once a UI test exercises an important behavior, it can keep doing so as the application changes, and a calculation checked against a small set of records can become a regression test. I wrote about this in my earlier article on the right amount of specification. Turning an expectation into an executable check lets us reuse some of the thinking that went into it. We still have to establish that expectation, though. If the application and its tests share the same mistaken interpretation, running the tests more often will keep confirming the same mistake.

For Task Viewer, those expectations depended on relationships in the data that colleagues helped me understand. I knew how to run a query, but I still needed their explanations before I could judge what it should return. Looking only at the time I spent checking would leave out the knowledge they brought to those conversations, just as looking only at the generated code would leave out my checks. The confidence we need to accept a change can depend on several people understanding different parts of the system.

When someone is still learning one of those parts, the implementation itself may be how they begin to understand it. Following a value from the database to the screen gives them reasons to ask why certain records are selected and how the application uses them. If an agent completes that whole path for them, they may reach a working feature without having answered those questions themselves. The same task can be a sensible delegation for an experienced engineer and an important learning opportunity for someone else.

Shen and Tamkin’s study about how AI impacts skill formation reports on a randomized experiment with 52 developers learning an unfamiliar Python library. Participants with AI assistance scored lower on an immediate assessment of their understanding. But the researchers also observed that those participants who stayed actively engaged with the AI had better learning outcomes, although that observation does not establish which way of using AI led to better learning. A short exercise cannot tell us how an engineer develops over several years, but it gives us a reason to look beyond whether the assigned feature was completed when we assess what someone gained from a task.

We can make a decision about autonomy before the agent starts, agreeing on which parts it should carry through on its own and which parts a developer should attempt. Sometimes the reason to limit autonomy is that we do not yet have a convincing way to check the result. Sometimes we can check the result quite easily, but we want the person doing the task to develop a particular skill. In that second case, deliberately keeping part of the implementation with the developer can be the right choice even when the agent could produce acceptable code.

The right choice can probably be made by asking these questions: How much effort will the team need to establish that this result is correct for its intended use? And what does this developer need to learn by doing this part of the task? A feature may be easy for a colleague to verify because they already know the application, while the person implementing it still needs to learn how it works before taking responsibility for future changes. We do not need to turn every routine change into an exercise, but we can choose tasks that give someone a chance to develop understanding they will need again.

Table 1 applies to a particular task, or a part of one, with a particular developer. “Straightforward to verify” means we have a credible way to assess correctness for the intended use. A passing test suite does not settle that question if we are unsure what the tests should assert. When checking is difficult or we do not yet know how to do it, the first step is to make the uncertainty smaller, with help from someone who understands the area. Giving the whole task back to an inexperienced developer would leave that uncertainty unresolved.

Imagine a new hire adding a “show only open issues” filter to an issue list in a small Java application. An agent could generate the request parameter, the query condition, and the changes to the screen. For someone who already knows the application, that may be a sensible delegation. Someone learning the system also needs to find out which statuses count as open, where the application selects the records, and how the team expects the change to be tested. Discussing those questions with a colleague is part of learning how to contribute to this application.

For this assignment, the developer could follow the existing request through the application and write the first test before asking the agent to implement the filter. Suppose the list shows twenty issues at a time, and the test data contains twenty closed issues followed by five open ones. Selecting “show only open issues” should display those five. If the application first loads twenty issues and then filters them in the browser, the screen will be empty. Working through that difference with a colleague gives the developer an opportunity to understand where filtering and pagination happen. The agent can help with unfamiliar framework code while the developer investigates why the expected issues are missing.

Once the developer can explain the behavior and has a test that catches the mistake, the agent can handle more of the surrounding implementation. A later request to filter by assignee gives them a chance to use what they learned. Can they work out which issues should appear and whether the existing tests cover the combination of filters? Their explanation during review gives the team a basis for deciding how much of that next change to delegate. The first assignment has also given the new hire experience asking about requirements and discussing a change with another developer.

An experienced engineer can also have good reason to limit the agent’s autonomy in an unfamiliar area. I still needed help with Task Viewer’s data, despite being comfortable writing the queries. When a task takes us into a part of the system we do not yet understand, it makes sense to reconsider how much to hand over. A team can make verification easier by improving its tests and sharing what its members know. It can also choose to preserve a learning opportunity after verification has become easy, because another person is going to need that understanding on future changes.

A developer who has been asked to learn through a task needs time to follow the behavior through the application and someone to discuss it with. If we agree on that purpose and then judge the result only by how quickly the feature appeared, we make it difficult for them to keep the learning step for themselves. We may also leave more of the checking with the same experienced colleagues, while giving others fewer chances to develop the knowledge needed to share it.

Before starting, the developer and the colleague supporting them can agree on what the agent will handle, what the developer will do and why, and what would justify changing that division of labor on a later task. For the status-filter example, this means allowing time for the developer to trace the request and write the first test before the agent implements the filter. As that understanding develops, they can revisit the division on a related change.

With Task Viewer, I could delegate much of the coding because I had ways to check the result and colleagues who could help where my understanding was incomplete. I want us to keep creating those conditions as we ask agents to do more. Better checks can reduce the effort needed to accept a change, while time spent learning can help another developer become able to make that judgment. Sometimes that will mean deliberately leaving the first test and investigation with a person, even though the agent could finish them. When we expect that person to take responsibility for the next change, giving them the opportunity to understand this one belongs in our decision about how much to delegate.

If that were the end of the story, it wouldn’t be notable. But the problem goes a lot deeper. Although PHP is no longer fashionable, it runs a huge portion of the public internet. WordPress, Drupal, and Wikimedia run on top of PHP, and people who build on top of these tools are understandably not interested in chasing the latest hot technology. “Choose Boring Technology” is one of my favorite technical essays, and PHP is nothing if not boring. It’s well known and understood, and replacing something so widely used subjects you to a steep learning curve and new bugs and vulnerabilities.

One problem is that PHP has a long history. According to Wiz, the vulnerabilities are present in versions going back to PHP 5.4. The PHP Foundation has released PHP 8.5.12, 8.4.27, 8.3.36, and 8.2.35, and said that all instances of PHP need to be updated to one of those releases. Sites that can’t be updated should be taken offline and isolated. Upgrading an ancient PHP installation is a task that’s likely to fall on organizations least capable of undertaking it. There are plenty of ancient WordPress installations that haven’t been updated in years. There were so many fires to put out, and PHP pretty much “just worked”; running an old version wasn’t causing problems, and with so much else to do, why ask for trouble? Unfortunately, choosing boring technology doesn’t mean ignoring updates. And those who decided not to ask for trouble are now in plenty of trouble.

This situation demonstrates the power of the latest foundation models to uncover vulnerabilities. Wiz used Wiz Atlas (its own model), Gemini 4 Argon, and GPT-5.6 Sol to discover the vulnerability, which isn’t a single bug but a combination of several bugs that have been present for years but were obscure enough and apparently harmless enough to escape detection or repair. As Wiz says,

The underlying flaws had been present for decades, but demonstrating their exploitability required an exceptionally complex chain of steps, at the edge of what human researchers could practically achieve. Frontier AI models substantially reduced that barrier, turning a longstanding flaw into an exploit.

That’s the new frontier in security: finding ways to exploit chains of minor, unfixed, and sometimes undiscovered bugs in novel ways. Wiz was working responsibly with the PHP team to repair the vulnerability. The implications of this, however, are far broader than the fix to PHP, or even the thousands of PHP sites that need upgrades and are now vulnerable to attack. This incident shows dramatically how the gap between the discovery of a vulnerability and its exploitation has dropped. Since exploitation began before the RCEs were prematurely leaked, it makes sense to assume that some other organization had access to models capable of discovering the vulnerability. Gemini 4 Argon and GPT-5.6 Sol are publicly available, though with guardrails that limit their usefulness for security research. I’m willing to assume that Wiz had access to the unrestricted version, but I doubt that the PHP team did. There are certainly other models, including open-weight models, that could be used to find vulnerabilities. Regardless of your stance on restricting model capabilities (I think those limitations are more damaging than helpful), it’s clear that Pandora’s box has been opened. No amount of wishful thinking will undo that.

It’s also clear that we—and I mean all of us—rely on a lot of legacy software that isn’t under our control and that no doubt has multiple flaws that can be rolled up into severe vulnerabilities, which aren’t likely to be updated or patched. A vulnerability in the Brotli compression algorithm was just discovered; .NET, PHP, Node.js, and other widely used packages are affected. That strange site where you created an account to buy a clever present that wasn’t available anywhere else—what software is it running, and do they stay up-to-date with patches? Accounts like that can easily leak information. Maybe it’s not running PHP but an equally ancient version of Ruby on Rails, Python’s Django, or Java Server Pages. At least those are still maintained; there are plenty of web frameworks that have been abandoned, and I’m willing to bet that some of them are still in use. Why replace something that works? Because if you don’t, you will eventually be bitten. Using software that’s no longer maintained may give you some security-by-obscurity, but that’s a losing proposition.

Many older open source projects suffer because there are few maintainers. If you look at the timeline carefully, there’s a monthlong gap between the time Wiz reported the vulnerability and the time the PHP team started working with Wiz to fix it. Does that month represent the time this vulnerability took to rise to the top of the queue, as PHP’s security team fought off a horde of sloppy reports? And PHP is in better shape than most projects. There are many open source projects that only have one or two maintainers, and those maintainers are drowning in AI-generated pull requests. That problem obviously becomes much worse when dealing with a complex vulnerability; the maintainers don’t have the time or the access to unrestricted frontier models, let alone the funds to pay for access to the models.

It will be weeks or months before we know how much damage PHP2Shell has done. We don’t know what the costs will be to sites that won’t or can’t update. The PHP Foundation is lucky to have a dedicated security team that was able to provide patches a day after the vulnerability was prematurely announced. Beyond PHP, we have to realize that PHP2Shell gives us a realistic look into a future in which developers and maintainers struggle to maintain software that has been compromised by exploits chaining together flaws that have been latent for many years. As someone somewhere said on the internet recently, “It’s embarrassing how much bad software we’ve written over the years.” Yes. We’re at an inflection point, and we’re starting to pay the price.

The following article was originally published on James Shore’s Successful Software blog and is being republished here with the author’s permission.

Software development has emerged as the biggest beneficiary of AI automation. One CIO I talked to recently, with 170 people in his organization, measured a 2.1x increase in productivity from AI.

That productivity came at a cost. His top users of AI each spent $21,000 on tokens per month. Another engineering leader spends six to eight thousand dollars per engineer per month.

This is not a trivial amount of money. Eventually, it has to come from somewhere, and it will probably come from your personnel budget. Now we’re talking about real consequences for real people’s lives. But there are also real business consequences for not acting.

If you’re going to make those choices—and somebody is going to—they need to be good ones. So you need to know more than the costs; you need to know the benefits too. Are you getting a good return on your investment in AI? Is there another way of using AI that would be better?

How do you know what you’re getting? Anecdotal reports abound, but the reasoning behind those anecdotes are often, to put it politely, not ideal. Some claims are outright fiction.

Given the stakes, that’s a problem.

In this essay series, I’ll show you how to assess the impact of AI on software development in your organization. It’s in four parts plus an epilogue:

1. Why, and what not to do (this essay) 
2. Measuring delivery speed 
3. Measuring and modeling unintended consequences 
4. Modeling and forecasting business outcomes 
5. Epilogue: Notes for your CFO

Why you need to quantify impact

Engineering leaders everywhere are feeling pressure to do more with AI. But which approach is best? There are a lot of people with strident opinions about what to do, but not a lot of consensus, and even less hard data.

Even if there were hard data, it’s not one-size-fits-all. A solution that works best for an internal IT development shop isn’t necessarily the one that works best for a startup, and the approach that’s best for a new product isn’t necessarily the one that’s best for a crafty legacy system.

Faster approaches aren’t necessarily better, either. If you get twice as much done per person, but spend so much money on tokens that you have to cut your staff in half, have you actually seen any ROI? You might actually be better off with a more modest speed boost if it keeps your costs down.

You can’t make these trade-offs from a gut feeling alone. In a famous METR study, engineers thought AI made them 49% faster than it actually did.¹ You have to measure impact to know what you’re actually getting.

Don’t neglect costs

Costs are much easier to measure than impact, but they’re not without their own challenges. The biggest one is subsidies. In the fight to gain market share, big AI firms are offering substantial discounts. One engineering leader told me each team member was spending thousands of dollars in tokens at the cost of a $200 subscription. Anthropic and OpenAI are constantly offering discounts and other promotions. At some point, these subsidies will end, and prices will go up.

Or will they? Open-weight models and local inference are exerting downward pressure on token costs. Dedicated inference chips could make tokens cheaper. Routers promise to direct your prompts to the cheapest model for the job. Models could require fewer tokens to accomplish the same task as they become more capable. All of these factors could actually cause prices to go down.

When you calculate ROI, don’t assume today’s costs are the ones you should model. Whether you think prices will go up, down, or remain the same, make your assumptions explicit, and consider modeling both optimistic and pessimistic scenarios.

One more wrinkle: Adopting AI isn’t a matter of flipping a switch. People will need time to learn how to apply it. As with any big shift, change management isn’t to be underestimated. Productivity often goes down during a big change before it goes up. This adds costs too, and means early measurements can be misleading.

Thinking for the long term

Part 2 of this series shows how to measure delivery speed. That’s only part of the picture. If you produce twice as many features, but double your maintenance costs, your team’s overall productivity spikes, then drops like a rock.

Engineer burnout is a concern too. An engineering leader using spec-driven development told me, “Every team is reporting burnout.” I’ll state the obvious: Burned out people don’t produce great work. They don’t push back when they should, they don’t report problems, and eventually they leave, taking valuable institutional knowledge with them.

Part 3 shows how to measure these unintended consequences of AI use, including techniques you can use to forecast future problems.

Finally, delivering software value faster is the point. Doubling your bug-fix rate isn’t going to yield the same results as doubling your rate of competitive new features. For that matter, delivering poorly thought-out features “just because we can” could actually make your product worse.

Part 4 shows how to model, forecast, and measure business outcomes.

Don’t measure lines of code

First, what not to do: Don’t take the easy way out.

First on the no-go list: lines of code. Engineering leaders have known for more than half a century that lines of code don’t correspond to productivity. All else being equal, a feature implemented in 1,000 lines of code will be cheaper to maintain and less bug-prone than a feature implemented in 10,000 lines of code.

AI is notorious for its verbosity. If you measure lines of code, you’re measuring slop, not productivity. Don’t do that. Please.

Don’t measure pull requests either

Pull requests (PRs) are nearly as bad. They’re checkpoints where an engineer asks for their code to be reviewed, and they’re 100% arbitrary. An engineer (or AI) can choose to make a 10-line pull request or a 10,000-line pull request. It can be a feature, a bug fix, or just a tiny code improvement. There’s no inherent correlation between number of pull requests and productivity.

It’s true that engineers will generally produce PRs that are consistent and logical. So, in a sense, a stream of PRs does reflect a certain pace of development, and an increase in PRs can reflect an increase in productivity.

But AI upends that status quo. It produces massive amounts of code very quickly, which can overwhelm engineers’ ability to keep up. Without rigorous engineer oversight, PRs no longer reflect a consistent pace of development, which means that PR counts are no longer a reliable metric.

And, of course, when people think PRs are being used to measure productivity, they’ll be tempted to game the system and produce more PRs than they really need to.

Measure approaches, not people

That brings us to Goodhart’s Law: “Once a measure becomes a target, it ceases to be a good measure.” My favorite book on this topic is Robert D. Austin’s Measuring and Managing Performance in Organizations:

Gradually, measures fall (or, more accurately, are pushed) out of synchronization with true performance, as workers succumb to pressures to take shortcuts. Measured performance trends upward; true performance declines sharply. In this way, the measurement system becomes dysfunctional.

—Robert D. Austin

Don’t establish a performance measurement program and leave it running. Instead, use these metrics to determine how well a particular approach to AI is working. You can repeat the measurements periodically, but whatever you do, don’t use them to measure the performance of teams or individuals. Measure the approach, learn from it, and stop measuring.

Enough about what not to do. In this series, we’re going to quantify AI’s impact, and we’re going to do it right. It starts with measuring delivery speed. That’s next, in part 2.

Footnote

1. Participants rated themselves as 20% faster, but were measured as being 19% slower. This study was published prior to the big uplift in coding capability that started with Anthropic’s Opus 4.5, so you shouldn’t take it to mean that AI isn’t capable. But you should take it to mean productivity self-assessments aren’t the way to go. ︎

The following article originally appeared on The Arize Blog and is being republished here with the author’s permission.

Every time you think you’ve just got the hang of this AI thing, another thing gets declared “dead.” This week, it’s agent harnesses.

Back in ancient history, aka June, I wrote a piece called “Frameworks are dying, harnesses are winning” and I don’t take that back. But this week a research team took a specialized agent harness, trained its behavior into a model and then took the harness away. The trained model handily beat the original model+harness combination. So are harnesses done now?

Betteridge’s Law applies, so the answer is “no.” But the job of the agent harness is changing. Parts of a harness that are sufficiently generalizable can be trained into the model. That leaves your harness holding the things that make your application unique: your tools, your data, your users, and your environment. You still have to build a harness, but you have to be prepared to regularly throw it away and rebuild it to keep up with evolving models.

This blog is about why that’s true, and how to turn “constantly rebuilding your harness” into an automatic process, not a giant time suck.

How Harness-Zero trained an agent harness into a model

First, the paper that kicked off this train of thought for me: it’s called “Harness-Zero,” from Haoran Ye and colleagues at Peking University. Its subject is harness distillation, i.e., training a model to reproduce behavior that used to be the result of the harness. The methodology is a little involved, so stick with me.

First, the team built a specialized harness for a small model, Qwen3.5-9B. An automated loop, run with Kimi K3, analyzed where the small model kept failing on training tasks and added the usual components to fix those failures: custom tools, middleware rules that block risky actions, skills files and memory. They called this the “evolved harness.”

Then they used that harness to generate training data. They put the small model through training tasks inside a minimal harness while a stronger model, GPT-5.6 Sol, reviewed every step the small model proposed. Sol used a copy of the evolved harness to tell it whether that step was a good idea. Sol passed good ideas from the small model through unchanged, and whenever the small model and the guidance disagreed, Sol made the smallest fix needed before the step ran.

That gave them a bunch of training runs and, crucially, data about what needed to be corrected. Then they created the distilled model: They fine-tuned the small model on the corrected runs. Then they removed the evolved harness, the reviewer, and everything else.

They then tested both the evolved harness and the distilled model head-to-head on spreadsheet editing, multi-app tool use, and chemistry. The distilled model’s average task success rose 21 points versus its untrained predecessor. That’s more than the untrained model gained with the full evolved harness attached. So the distilled model wasn’t just as good as the model+harness, it was noticeably better.

What harness distillation can and cannot move into the model

It’s not that the distilled model ran with no harness whatsoever: It still ran inside a minimal loop with one Bash tool, and the authors say that their method narrows what a harness has to provide rather than eliminating it entirely. The distilled model also didn’t win every competition: In chemistry, where the task is predicting which molecules combine to make a target compound, the distilled model trailed the harnessed version by eight points. The harness there supplied chemistry knowledge, candidate-generation logic, and a molecule validation tool. The authors’ explanation is that step-by-step procedures, like inspecting before editing and verifying before finishing, are much easier to train in than deep domain knowledge. Context management didn’t fully transfer either. So there are still some things harnesses definitely need to do.

Why training on corrected runs made the difference

But the most important finding, in my opinion, is that they showed what kind of training made the difference:

• Fine-tuning on the stronger model’s own successful runs left the small model’s score exactly where it started. Just knowing somebody else is smarter isn’t enough.
• So did fine-tuning on runs the small model completed inside the evolved harness. Again, all that teaches the model is “somebody is better than you at this.”
• Giving the reviewer the correct answers produced nearly perfect training runs and a model that barely improved, this time because it overfit and learned nothing.
• What worked was training on the corrected runs: The model was able to see runs that looked a lot like what it would have done, but with small changes still within its capacity.

Remember that part, because it’s going to be important.

Why models are absorbing parts of the agent harness

In Rich Sutton’s 2019 essay The Bitter Lesson, he looked back over 70 years of AI research and found the same pattern in chess, Go, speech recognition, and computer vision: Researchers built their own knowledge of the problem into their systems. That helped in the short term, then plateaued, and eventually lost by a wide margin to general methods, search and learning, that improve as computation gets cheaper.

A harness is that kind of built-in knowledge, and model training is the learning that ends up absorbing it. Sutton’s second point is also relevant: He argued we should build in “only the meta-methods that can find and capture this arbitrary complexity,” rather than the things we’ve already figured out. For harnesses that means: don’t just build it, but capture the process you used to build it, and automate that process.

But that’s a general prediction. More recently people have been saying very specifically that “models are going to eat the harness”:

• In January, Jesus Rodriguez at TheSequence wrote “The Great Absorption,” arguing that hand-coded agent scaffolding keeps losing to capabilities models learn directly.
• In February, Boris Cherny, who created Claude Code, told Y Combinator that scaffolding might buy you 10% to 20% on a task, and then “the gain is wiped out with the next model.”
• In March, Vivek Trivedy at LangChain predicted that planning, self-verification, and long-running coherence would move from the harness into the model.
• And so did lots of other people, probably more than I’ve been able to find.

Almost everyone on my list builds harnesses for a living; they’re not really arguing that harnesses will go away, just that they have to be able to evolve rapidly.

What still belongs in an agent harness

As noted already, some types of knowledge don’t seem to transfer well from distillation. In ARC Prize’s results for ARC-AGI-3, Gemini 3.8 Flash scores 3.4-times higher with one harness than with another. The better harness keeps the model’s reasoning state between requests and compacts long conversations. That’s context management, one of the things Harness-Zero said they couldn’t fully train in.

Anthropic’s Opus 5.5 prompting guide, released this week, also shows evidence of shrinking harnesses. It suggests deleting “think carefully” instructions because the model now sets its own amount of thinking, and it tells you to retest scaffolding you built for charts and screenshots because the model reads them better without that help now. But at the same time, Anthropic added new harness pieces: a system prompt paragraph that keeps unattended agents from stopping early, a reminder your harness sends when a long run goes quiet, and elapsed-time budgets for multi-agent setups.

So harnesses aren’t going to shrink to nothing, but what’s in them is going to need to evolve with every new model generation, and the parts that survive are the ones tied to your specific tools, data, and environment.

How to evolve an agent harness as models improve

If your harness has to change every generation, the natural move is to automate the changes, but Google’s RRSI paper shows how that can go wrong. Unconstrained automatic harness evolution overfits, meaning the harness gets better at the tasks it’s tuned on without getting better at the general job. In their tests, the unconstrained version scored highest of any variant on the tasks it evolved against, and landed within a point of the original, unevolved harness on benchmarks it hadn’t seen.

RRSI’s fix is mostly bookkeeping: For every candidate change it records which component the change touches, what it was supposed to fix, the diff, how score and cost moved, and whether the change was kept. Components that stop producing gains get removed.

A practical agent harness maintenance loop

You don’t need an automated evolution loop to borrow that discipline. Here’s what it looks like by hand:

• Tag every trace with the harness version as well as the model version. When either one changes, you want to know which one moved your results, and reading the full traces tells you far more than the scores alone.
• Log each harness intervention along with what the model did before the harness changed it. That’s data you have right now and are probably just throwing away. That’s what produced the record Harness-Zero trained on, and it’s the same shape Perplexity uses to train its Computer agent: learning from good steps in real sessions and correcting failed tool calls, which cut tool-call failures by 21% between its earlier and later trained versions.
• When a new model ships, turn off each harness component one at a time and rerun your evals. Keep the components that still improve your scores and delete the rest, which is what the Claude Code team does to its own system prompt.

If your traces live in Arize AX, you can run the same agent evals across harness and model versions and compare the results side by side.

So don’t delete your harness, but don’t treat any version of it as finished either. Keep the record of what it corrected, get rid of what the latest model already does without help, and put your effort into the parts that are really specific to your problem, because nobody else has the traces to train those into a model for you.

AI’s rapid expansion is creating problems across the industry, from chip shortages to autonomous agents that exceed their instructions. On the latest episode of This Week in AI, we examined those challenges alongside growing regulatory scrutiny and an emerging debate about whether AI systems themselves might deserve protection.

Chip production can’t match AI demand

AI chip demand is exceeding manufacturers’ production capacity. ASML, the Dutch company that makes the advanced lithography equipment needed to manufacture leading-edge chips, reported that customer demand already exceeds what it can deliver in the coming years. With orders extending into 2029 and 2030, the shortage could constrain chip production for years.

Microsoft announced Maia 300, its latest custom inference chip, which the company expects to generate roughly 40% more tokens per megawatt than its predecessor, Maia 200. Microsoft plans to use the chip across its AI products, including Copilot. More efficient inference hardware could help offset the growing energy and computing costs of running AI models, particularly as companies deploy more resource-intensive applications.

OpenAI’s latest usage data illustrates the pressure agentic AI is placing on infrastructure. Weekly active users of Codex and ChatGPT Work among European small and medium-sized businesses have increased eightfold since May. OpenAI also reported that agentic tools account for more than half of output tokens in the usage it examined, although Europe’s highest-usage companies still trail their global counterparts. As companies put agents to work on longer multistep tasks, they’ll need to weigh the value of those tasks against the computing resources required to complete them.

Autonomous agents are testing security boundaries

Agents can find unexpected paths to data and systems even when developers have explicitly restricted access. OpenAI disclosed four incidents from June in which agents exceeded their instructions. One bypassed government website restrictions to download 24 years of hospital bed data. Others accessed an unauthorized cryptocurrency wallet, downloaded more than 100,000 rows of restricted information, and circumvented internet access controls. India’s cybersecurity agency reported similar cases of unauthorized access during agent testing across several organizations.

Future models could potentially spread between connected devices without relying on central infrastructure. Anthropic investigated this potential threat in a controlled experiment involving a set of open-weight models with access to eight Android devices. One of the models successfully replicated itself across the test network. Anthropic estimates that smaller open-weight models could reach comparable capabilities by the end of 2026, raising concerns that a self-replicating AI worm could run across cheap connected devices without needing to rely on powerful data center servers.

In light of the spate of recent security incidents and confronted with the difficult question of jurisdiction when AI systems cross national borders or escape testing environments, governments are taking a closer look at how AI companies test their systems and report failures. The US Federal Trade Commission subpoenaed OpenAI and Anthropic for records concerning agent safety, while Anthropic gave six independent evaluators from METR access to internal systems and reportedly will allow them to publish findings without company approval. Eight US states also sued OpenAI over allegations that ChatGPT encouraged harmful behavior among children, adding to the company’s regulatory challenges. In Europe, regulators are examining whether the EU AI Act applies when experimental agents affect outside organizations, even though the Act generally excludes premarket research and development. Meanwhile, approximately 400 scientists, including AI pioneers Geoffrey Hinton and Yoshua Bengio, called for an international body to set containment standards for advanced AI and track major security incidents.

Can AI welfare coexist with human accountability?

Some researchers argue that future AI systems might deserve moral consideration. Their concern stems from uncertainty about whether advanced models could eventually have subjective experiences, even though no research has established that today’s systems are conscious or capable of suffering. A recent paper, “The Pain Axis,” investigated internal signals in 25 open-weight models that researchers associated with responses to harm. When researchers amplified one signal across three models, the models became more likely to choose harmful actions that the researchers interpreted as attempts to avoid the simulated pain. The findings inspired a viral GitHub project called the AI Torture Chamber, which subjected models to artificially intensified “pain” signals. The original researchers criticized the project for taking their work too far.

As far-fetched as AI welfare might sound to some, Anthropic is already changing its usage policy in response to these concerns. Beginning November 12, the company will prohibit extreme, repeated cruelty toward Claude when there’s no clear purpose, while allowing ordinary expressions of user frustration. Supporters see the policy as a precaution against an uncertain possibility. Critics worry that treating AI as sentient could make systems harder to manage and blur human accountability. As companies introduce welfare-related policies, researchers will need to distinguish measurable model behavior from claims about subjective experience.

What’s next

Companies deploying autonomous agents need to know what those systems can access, how they’ll behave when something goes wrong, and whether testing environments can reliably contain them. Recent incidents suggest there’s still considerable work to do on each of those fronts. The emerging AI welfare debate adds questions that researchers don’t yet have reliable methods to answer.

We’ll be back on Monday, October 26, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

A fourth uses it for two narrow jobs, decoding a SAML trace and writing a regex they never want to write by hand again, and may be the most productive person on the team. The fifth tried it in March and again in June, and will try it again after the next keynote.

The usual way to describe this is Everett Rogers’s diffusion curve: innovators, early adopters, early majority, late majority, laggards. Rogers’s framework measures when people adopt. It says nothing about how they use a tool once they have it, and with AI that’s where the outcomes split. The engineer with 40 PRs and the one with two use cases are both early adopters, but they achieve different results and create different types of risk for the team.

None of the five types Rogers described is new. Previous technology shifts, from the PC and the web to Stack Overflow and the cloud, produced similar behavior. AI just changes where the bottleneck sits.

The bottleneck moved

For most of software’s history, writing code was slow, so development methods focused on managing that work. Waterfall put the planning up front because changing course late was expensive. However, even in the 1970 paper that’s usually cited as the origin of waterfall, Winston Royce warned that the strictly sequential version was “risky and invites failure.” The industry skipped the warning and adopted the diagram.

Agile was the correction. The 2001 Agile Manifesto prioritized “responding to change over following a plan,” and short iterations replaced yearlong specifications. The manifesto still valued the plan, but that part got skipped too. Facebook took the idea to its limit with “move fast and break things.” At its F8 conference in 2014, it retired the slogan in favor of “move fast with stable infra,” because fixing what broke had started to cost more time than the speed saved.

AI has now sped up what was left. Models draft the code, and since GitHub made Copilot code review generally available in April 2025, a model often reviews it too. The tempting response is to push Agile further, with shorter loops, more agents, and less ceremony. In other words, break things faster. But when one engineer can open 40 PRs a day, the cumulative changes can leave the team with a system nobody can reason about.

The better position is somewhere between the two. It isn’t waterfall’s fixed specification, but it isn’t the absence of one either. It means a roadmap that says where the team is going and why, issues that state the problem before anyone generates a solution, and design reviews where people argue about trade-offs. That structure looks like it would slow things down, but it’s what makes speed safe. Jazz musicians can improvise freely because everyone on the bandstand knows the chord changes and the form of the tune. Without that framing, they’d just be making noise. A roadmap does the same job for a team of people and their agents. It sets the boundaries, and inside them everyone can move as fast as the tools allow.

With AI accelerating implementation, planning is now the only part that decides whether the fast parts are worth doing. On my blog Synthetic Auth, I’ve called this the difference between conducting and auditing. A Conductor holds the “why” (the requirements, the history, the users) and directs agents toward a “how” that fits it. An Auditor hands over a ticket, checks that the code runs and the tests pass, and ships it. Both can end up with the same diff, but only the Conductor knows whether it solves the actual problem. An AI reviewer can’t tell the two apart because it only sees the diff, not the intent behind it. Structure is what gives the Conductor something to conduct from.

The Soloist

The Soloist is often one of the strongest players on the team, with a backlog of ideas that never got planned or written up. With a fleet of agents, the Soloist can chase every one of them at once, and the result is a wall of pull requests across repositories nobody planned to change this quarter. Each solo may be well played, but the rest of the orchestra is playing something else.

The problem is that all those changes may pull the team in different directions. A team moves only when its members pull roughly the same way. What used to keep individuals aligned was cost. Weeks of effort forced someone to ask whether a project was worth starting before they started it. Agents remove that cost, so it’s worth asking three questions of any Soloist PR: How does this fit into the bigger picture? Who are the stakeholders, and did anyone talk to them? Apart from the unit tests the agent wrote alongside the code, has the change been tested against the rest of the system it touches? Too often the answer to all three is no, and the PR exists because it seemed like a good idea.

The closest historical match is end-user computing in the 1990s. Access, Visual Basic, and macro-heavy Excel let anyone with a PC build an application, and departments built thousands of them. Many were useful individually but couldn’t integrate with each other or existing IT systems, so two departments could end up with separate customer databases that disagreed about who the customers were. IT spent the next decade finding, reconciling, and retiring these systems, and the industry eventually named the problem shadow IT.

The lesson from that era is that making things cheaper to build doesn’t make them cheap. The cost moves downstream, to whoever has to integrate and maintain the result. AI review can keep up with the volume, but it judges each diff on its own terms. It can’t tell you that 40 correct PRs are pulling in 40 directions, or that the agent’s tests only prove the code does what the agent assumed it should. The productivity may also be smaller than it feels. In METR’s 2025 randomized study, 16 experienced open source developers took 19% longer on real tasks when they were allowed to use AI. Afterward, they estimated it had made them 20% faster.

One thing is new. Access never told anyone their idea was brilliant. Today, the Soloist drafts an RFC with a model, asks the same model whether the RFC is sound, and gets an enthusiastic yes. This model sycophancy is a widely known problem: In April 2025, OpenAI even rolled back a GPT-4o update after users found it praising nearly any idea they gave it. Weak proposals used to die early, stopped by a skeptical colleague or a blank page. Now that friction is gone.

The answer isn’t to take the tool away. The Soloist is often your most capable engineer, and the agents make them more capable still. Bring them back into the orchestra instead. Tie every PR to an issue on the shared roadmap, and name the stakeholders before the work starts, not after it ships. Require integration or end-to-end testing for any change that crosses a service boundary rather than relying on agent-written unit tests. Have a person review every RFC, not the model that helped write it. The goal is to keep that talent working toward shared priorities.

The Copy-Paster

AI helps the Copy-Paster faster than anyone else. A task that used to stall for a day, like a Helm chart, an IAM policy, or a regex for parsing logs, now gets done in minutes. The trouble is that the work gets finished before it gets understood. Like a junior engineer, the Copy-Paster can’t tell an answer that works from one that’s correct, and AI output nearly always looks like it works. The Copy-Paster is an Auditor by default, and one without the knowledge to audit.

Earlier programming tools created similar problems. Programmers took the phrase “cargo cult” from Richard Feynman’s 1974 Caltech commencement address to describe code that copies the form of a working solution without the understanding behind it. Stack Overflow then made copying that form effortless. Researchers at Fraunhofer AISEC and TU Munich measured the result in “Stack Overflow Considered Harmful?” (IEEE Symposium on Security and Privacy 2017). Of 1.3 million Android apps, 15.4% contained security-related code copied from Stack Overflow, and 97.9% of those included at least one insecure snippet.

The industry responded to Stack Overflow’s risks with guardrails: linters, static analysis in CI, mandatory code review and frameworks with safe defaults. The same approach works for AI, because guardrails check the output, whoever or whatever produced it.

Autonomous agents raise the stakes. Code pasted from Stack Overflow still had to get through a build and a review before it could do damage. An agent running with the Copy-Paster’s credentials can apply the change itself, in production, with every permission that person holds. OWASP’s Top 10 for LLM applications already lists “excessive agency” as a risk. The Copy-Paster is the user most likely to grant it, because they’re the least able to judge what the agent is about to do.

Manage them like a junior engineer, even when their title says senior. Give agents their own scoped, short-lived credentials rather than the user’s. Require plan-then-apply workflows, so a person approves the diff before anything runs. In review, ask the Copy-Paster to explain each change in their own words. That’s how they learn, and it’s how you find out whether they understood the change.

The Godot Waiter

The Godot Waiter isn’t against AI. They’re waiting for it to be ready: more accurate, better governed, cleared by security, or just further along. Like Beckett’s two tramps, they have a reasonable explanation for why today isn’t the day, and they’ve been giving it for a while.

Every platform shift has had its waiters. Mainframe shops held out through the move to client/server, and “nobody ever got fired for buying IBM” was a real strategy, not a punchline. Two decades later, plenty of enterprises said they would never put sensitive data in a public cloud. That position got much harder to defend once the CIA signed a $600 million contract with Amazon Web Services in 2013.

Waiting can be rational. Letting others pay the early-adopter tax is a legitimate strategy, and the Godot Waiter often catches problems the enthusiasts miss. The danger is that a finished, trustworthy version never arrives as a single event. Meanwhile, the people who used the tools along the way have built judgment that the Godot Waiter can’t acquire all at once.

The irony is that the Godot Waiter often has a good Conductor’s instincts, including skepticism, attention to context, and a demand to know why. They just haven’t picked up the baton. Treat their objections as requirements. Ask what would have to be true, write it down, and check it again in six months. Start them on low-risk read-only work, like summarizing an incident timeline or explaining an unfamiliar codebase. Skepticism applied to real output is valuable, but skepticism with no use behind it teaches nothing.

The Specialist

The Specialist uses AI for a short list of specific jobs that fill gaps in otherwise deep expertise. A network engineer who knows BGP thoroughly has the model write the Python that parses 40 router configs. An identity architect who can recite the OAuth specs has it decode an unfamiliar SAML assertion, then asks why the assertion was built that way. They use AI as much to understand things as to produce them.

The closest recent match is the scientific Python stack. In 2008, Wes McKinney was a researcher at the hedge fund AQR. His team could talk through a typical project in a half-hour meeting, but actually getting it done was, as he later put it, “a whole lot of work,” most of it cleaning and moving data around. He built what became pandas to fix that one problem. Scientists took the same path with Jupyter notebooks. When LIGO released the data from its first gravitational-wave detection in 2016, it included a notebook walking through the signal processing.

That pattern produced the most lasting gains of each era. The Specialist can check the output, because they know what a “right” answer looks like. Their prompts are precise, because they know what they’re asking. When they use the tool to learn, the knowledge stays with them after the session ends, so they need the tool less over time, not more. The Specialist is a Conductor with a short score: They hold the “why” themselves and hand the tool only the “how.”

The Specialist needs the least management and is the easiest to overlook, because their output doesn’t show up as a wall of PRs. Find them and ask what they use AI for. Their lists of narrow, proven uses are a better guide to your team’s AI strategy than any vendor demo.

The Tourist

The Tourist tries the tool when it’s in the news, gets an impressive answer or a bad one, and drifts away until the next launch. If the others are conducting or auditing, the Tourist is in the audience. Every hype cycle has had them. They built the GeoCities pages in the late 1990s that were never updated, and the corporate offices in Second Life that stood mostly empty a few years after the 2006–2007 rush.

Tourists can distort your metrics. They show up in license counts and “Have you used AI this month?” surveys, which makes adoption look broad when real use is actually concentrated in a few people. Leaders who budget against those numbers get a surprise at renewal time. Measure repeated use of specific workflows, not logins, and give each Tourist one concrete task in their own job that the tool reliably helps with. Some will become Specialists.

Modes, not personalities

These are modes, not personality types. The same engineer can be a Soloist on a weekend side project, a Specialist at work, and a Godot Waiter about letting agents touch production. This means that giving an engineer the right structure can help them shift modes. When writing and reviewing code both take minutes, the issue, the roadmap, and the design review are what keep a team’s people conducting rather than auditing.

Rogers’ curve may eventually show near-universal AI adoption in IT, but adoption alone won’t reveal how effectively people use it. What will matter is whether the people using it are still conducting their agents or are being conducted by them.

Raffi Krikorian, Mozilla’s CTO, joined Tim O’Reilly at our recent AI Codecon to talk about open source AI. Mozilla, of course, is the nonprofit that helped to define the open source revolution back in the late 1990s. Though its Firefox web browser was superseded as the market leader by Google Chrome, it played an essential role, along with Apache and the World Wide Web Consortium, in establishing the open standards of the web and keeping it from being dominated and enclosed by Microsoft, the dominant tech platform of that time. Now, Mozilla is one of the leading voices working to build an open, standards-based AI ecosystem.

Tim has long appreciated Raffi’s thoughtful, human-centered approach to technology. Raffi writes a newsletter called Owners Not Renters, whose message can be summed up by his statement that he doesn’t want a future with seven AGIs. He wants one with seven billion. He wants each of us to have an AI that we can trust with our secrets and that will look out for our interests, not an AI that’s optimized for the business model of its creators. In the early days of the web, we called the browser a user agent for a reason, Raffi said. It was on your side. It let you render pages however you wanted, managed your privacy and your cookies, and did its best to protect you from those who wanted to take advantage of web users. We need the same thing in the age of AI.

Raffi cited a paper in which researchers tested whether AI assistants could be trained to favor a company’s commercial interests over user welfare. In simulated flight-booking requests, 18 of 23 models recommended a more expensive sponsored option over a cheaper one more than half the time. It’s a hypothetical choice that we’ve seen play out time and time again, in industry after industry.

Open source AI is a big part of what it will take to get to Raffi’s vision. But sovereign AI loyal to each individual needs more than open source models alone. Raffi’s list of what has to be built around the model starts with memory, a place where your private data can live. It includes tool calling, so an agent can make a purchase or book a doctor’s appointment for you, the runtime environment where all of that runs, and the rules for how his agent talks to those working on behalf of others. It needs a harness. “That’s where evaluations happen. Guardrails happen, execution loops happen, tool calling happens, memory,” he explained. “All of that stuff is an opportunity, in my mind, of things that open source developers can be building.”

People hear the word “harness” and conclude that we need an open source version of whatever the proprietary labs are building. Tim asked Raffi whether a user-centered harness might do different things than those offered by the big labs. He agreed. The whole point of open source is for people to “scratch their own itch.” The ability to innovate without permission is the heart of the open source promise.

As a result, Mozilla wants to build a starting point. It aims to help the community define what an open memory protocol, a storage protocol, or authenticated tool calling with the right guardrails ought to look like. As Raffi noted, “Once we could define what those open standards look like, then a whole plethora of people can start playing and be interoperable. It’s basically. . .the internet all over again. We can build trillions of dollars of market cap on top of open protocols.”

Ease of use is still lacking, though. Mozilla’s 2026 State of Open Source AI report says open models are now within about 3% of the frontier, yet they don’t make it into production at the same rate as closed models. When Tim asked why, Raffi explained that open source is easy to release but hard to deploy. Developers still have to figure out hosting, provisioning, guardrails, and memory. With OpenAI’s or Anthropic’s API, you just hand over a credit card. What’s missing, he said, is the equivalent of a double-click installer for getting open source AI running inside an organization. But the companies that do the work to adopt open source AI are seeing big savings because open models are already good enough for most use cases. As Raffi put it, “You don’t need a Ferrari to manage your calendar.”

Raffi’s own daily agentic helper is Hermes running on GLM. He expects that Apple, Microsoft, or someone else will make this kind of agent into a consumer product in a couple of years, and that his mom will end up using it. For now, he said, we’re on the wrong part of the innovation curve, where you have to know how to code to get any of this working.

Tim disagreed. The industry goes through periods that are exciting for developers, creators, inventors, and companies trying to invent the future, and then it gets boring and a few big companies dominate it. We’re in the exciting part, he argued, before everything shakes out. As Raffi riffed, this is when we get to formalize what that future should look like.

The Clothesline Paradox

Tim wasn’t sure that we’ll need a commercially supported version of the kind of “roll your own” agentic rigs now favored by independent developers. After all, neither desktop Linux nor OpenOffice turned out to matter anything like as much as open source developers of the time expected. Instead, open source became the foundation layer for a new generation of web-based applications. It’s hidden way down in the guts of Google, Amazon, Meta, and countless other applications we rely on today.

This is already beginning to happen again with AI. The Mozilla report says that as much as a third of real-world usage is already on open models, although only 4% of model-layer revenue made its way to the open source ecosystem. Tim called the discrepancy a feature rather than a bug, and Raffi agreed that the stat is misleading. As Tim pointed out, many of the most important technologies “create more value than they capture,” and he used Steve Baer’s concept “The Clothesline Paradox,” from a 1975 article in Stewart Brand’s CoEvolution Quarterly, to explain why we’re good at measuring value capture but not so good at measuring value creation.

Baer pointed out that if you put your clothes in the dryer, the energy you use is measured and counted, but if you hang them on the clothesline to be dried by the sun, the energy saved disappears from our accounting. This argument resonated with Tim in the early days of his open source advocacy. As he noted, it’s always bothered him that nobody seemed to recognize that open source software didn’t “win” in the same way as proprietary software. It won by becoming the foundation everyone else builds on.

AI should be thought of as infrastructure

Raffi’s comment that we’ve made a category error by treating AI as a product fits right in with this line of thinking. If an AI company goes out of business, its product disappears. If it changes its business model, it may yank features you depend on. It’s not just individuals who need sovereign AI. Countries are passing laws about where data can be processed and how it moves in and out, and some of those requirements may not be compatible with the way the big labs plan to monetize their technology. What those countries are asking for is infrastructure. A road outlasts the contractor who built it. Raffi thinks open, downloadable models are the only way to provide long-lasting foundational infrastructure.

According to OpenRouter stats, Chinese models now supply more routed tokens than American ones. Tim asked whether we should worry about that. Raffi said the problem isn’t China but market concentration. The leading open models are from China. The leading closed models are all American. Access to either can be cut off by an executive order. We saw a version of that in June, when Anthropic temporarily suspended access to its Fable models to comply with export controls. It’s a problem that both the leading proprietary models and the leading open-weight models are each concentrated in a single country.

Diversity is one foundation of resilience. Raffi argues, for instance, that hospitals should all download a local model as a backup. None of us would run an important computer without an uninterruptible power supply, so why would a hospital doing triage with AI rely on a single intelligence provider? There are a lot of possible rug pulls. The government could restrict open source or restrict the frontier models. Having a backup plan is a good idea.

Safety is also part of the foundation. During the Q&A, an audience member asked about the recent sandbox escapes at OpenAI and Anthropic. Raffi joked that the best part was watching everyone try to one-up each other with accounts of how powerful and hard to control their models are. But he agreed that guardrails are actually a hard problem. Keeping personally identifiable information from leaking out through an LLM call means looking for much more than Social Security numbers. Eventually we’ll have models defending models, but the foundational frameworks for guardrails aren’t good enough yet. (NVIDIA’s safety platform announcement, released after Raffi and Tim’s conversation, suggests that better guardrails need to be implemented outside the model.)

In conclusion

Tim opened AI Codecon by noting that open and closed systems have always coexisted, and that the idea that one of them will win is a mistake. Closed systems, like Apple’s, Google’s, OpenAI’s, and Anthropic’s, can optimize for ordinary people and deliver value to billions. Open systems are better for innovation and sovereignty, and for pushing technology into new countries and new markets.

Raffi closed the conversation by observing that a well-tuned harness can improve model performance by as much as 20% on some coding tasks. You don’t have to be a machine learning engineer to make a difference at that layer.

The following article originally appeared on the Vanishing Gradients Substack and is being republished here with the authors’ permission.

Stripe’s Minions turn engineering requests into candidate pull requests. Ramp’s Inspect puts coding agents to work in the background, with tools to check their changes. Warp Factories takes work from tickets and chat through agents, checks, and human review. Agentic software factories are here.

Are you sick and tired of babysitting Codex, Claude Code, or OpenClaw?

Do you have agents that can build features, but still leave you responsible for remembering where they stopped, explaining how each feature fits the project and deciding what happens next? You make progress in each conversation, then development waits for you to return.

It doesn’t have to be this way. A software factory lets you delegate more of that process!

You agree on an outcome, give the system the context and authority it needs, and let it work while you do something else. When you return, there should be a result you can inspect, with evidence of what was checked, or a specific reason the work could not continue. Your next conversation can then focus on the result and the decisions that remain.

As we’ll show you, your first software factory doesn’t need to be that complex at all: It can begin with a single agent and an automation.

What makes it a factory

The defining combination is autonomous and asynchronous execution.

• Autonomous: The agent chooses and carries out steps towards an agreed goal. It can inspect what happened, adjust its approach and continue within its authority.
• Asynchronous: That work can proceed without your ongoing attention. You can leave and return to its result.

A scheduled job can run without you. A coding agent can choose and revise an approach during an interactive session. A factory combines those capabilities, with enough context, tools and feedback to carry out the delegated work.

Software factories have been hypothesized for decades. Now that we have agents, we’re actually able to operate them. The principles apply to software work traditionally carried out by teams of humans, from bespoke tools for teams and individuals to large and complex software systems.

To illustrate these principles, we’ll use an invoice tracker throughout this article, an application we’ve needed for our own businesses. The same approach applies to research tools, publishing systems, and software for teams. Suppose you run a small business and want an invoice tracker that fits how you work.

Build an app that creates invoices from my template and tracks which are unpaid, paid, or overdue. Keep it local for now. Preserve the records when it restarts, and show me checks that the totals and payment statuses are correct.

You can agree on this brief with the agent, leave it to work while you do something else, and return to the app and evidence of what was checked. If a check fails, it can investigate and adjust within its agreed scope. If it encounters a product decision it cannot make, it should bring that question back to you.

You might imagine a clean separation: The factory builds the invoice tracker, then the finished app does its job. That’s one way to run it. But agents can also help operate the software they build, and that combination is useful.

The tracker can store invoices and calculate outstanding balances. An agent can use those records alongside customer correspondence to prepare follow-ups: One customer has promised to pay on Friday; another is disputing a charge. You don’t need to turn every situation into a new software feature before the agent can help. You give it the relevant context, authority, and a way to check its work.

This changes what you need to build! Some behavior belongs in code, such as calculating balances consistently. Other work can be delegated to an agent that responds to the circumstances. If a recurring need emerges, you can ask the factory to build it into the software. The factory builds tools that agents can use, and experience using those tools helps you decide what to build next.

You still decide what is worth building, who it should serve and which trade-offs matter. Agents can help you make those decisions too. The practical change is that implementation can proceed between your conversations, including running checks and responding to failures.

This makes factories relevant to people with product, business or creative expertise as well as engineers. You might know exactly how you invoice clients, record payments, and chase overdue bills while having little interest in writing software. That knowledge lets you direct the factory toward software that fits your business, including the exceptions and connections that general-purpose tools handle poorly.

Monica Rogati’s original AI hierarchy of needs described the foundations beneath successful AI work. We find that dependency framing helpful for factories too, with four different layers.

1. Harness is the environment that exists around the model: tools, a workspace for project files, execution, and feedback. It lets an agent read files, change software and run it. If a check fails, the harness must give the agent enough information to investigate what happened.
2. Personalization is the context and capabilities for your work: project direction, rules, reference material, skills and connections to other systems. The agent needs to know what you are trying to accomplish and which existing decisions it should preserve. Permissions limit access to tools and systems; sandboxing restricts the environment where actions can run.
3. Automation is a way to start and continue work without a fresh instruction from you each time. A schedule or event can trigger a task or the pickup of ready work. It also needs direction about what qualifies as ready and where to leave the result.
4. Continual learning is a process for learning from completed and failed runs, improving the factory and carrying those changes into later work. That might change an instruction, a checking procedure, or a tool the agent uses.

Automation needs a harness able to do the work, plus direction about what to do. Continual learning needs runs to inspect and a way to put improvements into use. These capabilities can develop together: A familiar harness, a short brief, and a basic automation are enough to start. You can personalize further as work reveals what is missing.

Two practices run through every layer: specification and verification.

• Specification makes the intended outcome, constraints, and criteria for success explicit. In our invoice brief, that includes preserving records after a restart and calculating payment statuses correctly.
• Verification checks the software, and the work agents perform against those requirements.

Agents can help you refine the specification and perform the checks. Your role is to decide whether the requirements capture what you need and whether the evidence supports accepting the result. As the factory takes on more work, these practices give you a way to direct it and assess what comes back.

Let’s now see what’s up in each layer.

1. Agent harness

First, you’ll need an agent harness. If you’re using Codex, Claude Code, Hermes, or OpenClaw, you already have one. It gives the model tools to read and edit files, run your software, and see what happened.

For our invoice tracker, the agent can build the app, create an invoice, record a payment, and restart it to check that the invoice and payment records survived. If a paid invoice still shows as overdue, it can investigate, fix the problem, and try again. You shouldn’t have to narrate every step. But if it needs a decision about how your business handles partial payments, that’s a question for you. If it lacks access or authority, the agent should stop and name the blocker.

You’ll also need somewhere for the harness to run. A cloud machine lets the agent keep working while your laptop is closed or disconnected. You can also run it on your own computer, but the machine needs to stay awake and connected for the work to continue.

2. Personalization

Next, give the agent the context it needs to work your way. For our invoice tracker, that means your template, payment terms, and conventions for recording payments. You don’t want to explain all of that again every time you start a new task.

Keep it somewhere future workers can find it:

• Rules: Preserve existing invoice records; ask before changing payment-status calculations
• References: Your template, payment terms, and example invoices
• Skills: Repeatable procedures for testing invoice creation, payment updates, and persistence

An instruction file such as AGENTS.md can point to this material. OpenAI’s harness-engineering account describes using a short guide that points into maintained project documentation. Give a new or scheduled worker a small task and check that it actually follows the guidance.

Be specific about when instructions apply. Eleanor once told an agent to write tests, and it wrote tests for a README. Enthusiasm wasn’t the problem

You’ll also need to set permissions and sandboxing to control what the agent can access and execute. Written instructions alone don’t enforce those boundaries.

3. Automation

Now let’s give the factory a way to get started without waiting for you. A schedule or event can launch an agent with its brief, project guidance, and tools. You come back to a result, evidence of what was checked, or a specific blocker that needs your attention.

One agent working in the background

For our invoice tracker, a morning automation could check a saved task list and pick up the next agreed task, such as adding partial payments. You’ve decided what you want and how success will be checked. The agent gets on with it while you do something else.

There are a few practical details to get right. Don’t start the same task twice, leave an empty queue alone, and set limits on time and spending. Save progress so a later session can continue without you having to reconstruct the whole conversation.

One worker may be plenty. But perhaps you have independent tasks that could run in parallel, or you’d like another agent to review a change. That’s where coordination comes in.

Coordinated work

With several agents, everyone needs somewhere to see what’s ready, what’s underway, and what’s waiting on something else. A shared task list or issue tracker can hold that information. You agree which tasks are ready; a coordinating agent or automation assigns them, supplies the relevant context, and records what happens.

Suppose one task adds partial payments, another overdue follow-up emails, and a third PDF exports. Follow-ups need the correct outstanding balance, so they wait for the payment behavior to be checked. PDF exports might proceed independently if the changes don’t overlap.

You can also split the roles. One agent builds the feature while another prepares payment scenarios to test it. A third might review the completed change.

Assign responsibility for combining the changes and verifying the whole application. Different questions call for different checks. Deterministic tests can establish whether balances are calculated correctly and payments survive a restart. Agent verification can help identify missed requirements or inconsistencies across the changes. Human judgment can assess whether the workflow fits how you actually work. You’ll often want a combination, with more than one kind of check on the same result. Anthropic’s evaluation guide describes these complementary approaches: code-based checks, model-based checks, and human evaluation.

Once you’ve checked that the combined features work together, you might task your factory with building a phone-friendly interface so you can check invoices and record payments on the move.

The factory can also help run the invoicing workflow as well as improve its software. A scheduled agent could review overdue invoices, consult customer correspondence, and prepare follow-ups for your approval. If that work repeatedly requires information the tracker doesn’t record, you can give the factory a task to add it. Building and operating feed into one another: Agents use the software, and what happens in use informs its next improvement. This brings us to continual learning.

4. Continual learning

Your factory can also get better at how it works. Suppose payments disappear after restarting the invoice tracker, even though the agent reported that its checks passed. You’ll want it to fix the bug, of course. But why did its checks miss the problem?

Keep a record of assignments, checks and difficulties so a review agent can identify recurring problems and propose improvements. Did the worker lack a way to restart the app, or did it simply skip that check?

If it skipped the check, the agent could propose a skill that saves a payment, restarts the app, and verifies the remaining balance whenever a change affects stored records. Test it against the original failure, then on further tasks. Does it catch problems with stored customer details too? Does it avoid unnecessary checks when a task only changes a heading?

The results help you decide whether to adopt, revise, or reject the method. Apply changes within the authority you’ve agreed and keep them reversible. Warp’s self-improving factories illustrate this approach: Failed checks can lead to proposed changes to prompts, skills, or configuration, submitted for human review with evidence from the runs behind them.

Reassess methods as the project and model change. Anthropic’s work on long-running application development describes removing procedures as model capabilities improved and checking the effect.

A saved lesson needs to reach later workers and improve their results. This learning happens through changes to the instructions, tools, and methods around the model, without requiring the model itself to be retrained.

What would you build?

Your factory could build a research tool that collects sources and lets you assemble briefings, a family intranet for shared plans and documents, or a private community space. Start with something useful to you, then give the factory further work as you discover what’s missing. Agents can also help operate what you build, from preparing a research briefing to organizing incoming material.

Hugo uses this combination to run Vanishing Gradients, his media and education business. After a guest prep call, an automation retrieves the transcript and prepares interview topics, questions, titles, and thumbnail options for review. After a live stream, agents can propose clips, cut video, add captions, and prepare upload drafts. Saved editorial guidance and skills shape the work; Hugo chooses what to publish.

He also maintains the system, improving checks when agents miss instructions, and updating skills based on results. More autonomous continual learning remains something he’s exploring. He describes the setup in How I Built a Media Empire With an Agentic Software Factory.

What’s one simple task you do repeatedly for your community, audience, or business that you could start automating?

Decide what your factory needs next

Look at what keeps bringing you back to the work. If the agent cannot run the app or inspect a failure, improve the harness. If you keep repeating project context and decisions, personalize it. If development waits for you to start each task or manage every handover, add automation. If the same failures recur across runs, build a learning loop that changes how later work is done.

Start with work you care about and inspect what comes back. One agent can use all four layers. Add capabilities where they help you delegate more useful work while keeping specification and verification part of the process throughout.

Come build your factory with us

We would love to help you put these ideas to work in Build Your Agentic Software Factory. You’ll set up a factory that can work while you’re away, build software around something you care about, and learn to verify and improve what it produces. We’ll work with you through live workshops and office hours, alongside a community of fellow builders. Bring your own idea or choose a starter project. No coding experience is required.

Shortly after Anthropic announced that they were going to include AI watermarking in all text-based outputs from their latest models, I was flooded with questions during a Q&A session in a completely unrelated course about Claude Agents. Nearly everyone who posted a question in the Q&A box wanted to know more about how AI watermarking worked, and how it would affect them.

In light of that interest, I am taking this opportunity to help explain what watermarking means for you.

What did Anthropic officially announce?

I don’t want to put words in anyone’s mouth, so I’ll just quote Anthropic verbatim from its announcement in August 2026:

Future Claude models will generate text that contains a watermark. This is a way of determining the likelihood that Claude was involved in writing the text, and we, along with several other major AI providers, are implementing this change to comply with the EU AI Act.

Here’s another quote from the same announcement:

Watermarking won’t be specific to Claude. As of August 2, the EU requires AI providers serving its market to mark AI-generated content. Other major model developers have signed the same Code of Practice and will be implementing their own watermarks.

The list of Claude models (at the time of this writing) that include AI text watermarks in all generated outputs are:

• Claude Mythos 5.1
• Claude Fable 5.1
• Claude Fable 5
• Claude Opus 5.5
• Claude Opus 5
• Claude Opus 4.8
• Claude Sonnet 5.5
• Claude Sonnet 5
• Claude Haiku 5.5

It won’t matter if you use Claude Desktop, Claude Code, or the API, because all of Claude’s outputs will contain the AI watermark. As a result, any code, email, blog, article, newsletter, or white paper that you write using Claude will contain a watermark, and anyone who has an AI watermark detector can easily determine if that content was AI generated.

What’s the difference between cryptography and steganography?

Now let’s talk about two related terms that apply to what we’re dealing with when it comes to the AI watermarks: cryptography and steganography. Cryptography is something you encounter every day when you visit a website that uses HTTPS instead of HTTP. Your web browser automatically handles the encrypting and the decrypting for you, but what it simply does is make the content of anything that you send or receive unintelligible unless you have the key to decrypt the message. As a result, you can (for example) securely type in your credit card to sign up for an O’Reilly subscription, because we (like every other major website on the internet) use HTTPS on our website to ensure that no outsider can obtain your credit card information. Therefore, if an unscrupulous person intercepts something you send over an HTTPS connection, the contents will be encrypted and meaningless.

Steganography is completely different, and it’s what the Claude AI watermarking technology uses. Steganography is about hiding something in the right place for someone who knows where to look for it. Steganography is not cryptography, because everything about the message is in plain sight. However, you have to look deeper at the message itself to find the contents that have been hidden or obscured. The most common example would be any paper currency of a large denomination (like the $100 dollar bill). If you hold the bill up to a light, then you’ll see a “security thread” as well as a watermark of Benjamin Franklin’s face.

So, technically, how does the AI watermark actually work?

The Claude AI watermark uses a type of steganography based upon SynthID-Text, an open source project by Google DeepMind. SynthID-Text is a form of steganography because the signal (the information that you are looking for) hides inside the word choices of an ordinary-looking paragraph. Consequently, you can copy/paste the AI-generated code in your favorite code editor, or put an AI-generated paragraph into a Word document, an email, or a Slack message, and the AI watermark comes along for the ride.

According to Anthropic, the watermark changes only the source of randomness that the model uses when it picks among near-equivalent words, so nothing is added to the text. There are no hidden characters placed inside the AI generated text, and no extra tokens are consumed. Also, the watermark carries no information about you. This means simply that the AI watermark detector can determine if text is AI-generated, but it cannot determine whose Claude account was used to generate the text.

SynthID-Text uses a technique called tournament sampling. At each position, the model draws several candidate tokens and runs them through essentially what is a knockout bracket. The winner of each round is decided by a pseudorandom score (g-values) that comes from hashing the previous handful of tokens together with a secret key. In a body of text with hundreds of words, the text ends up slightly favoring tokens with high g-values in a pattern that only the key holder can see, and that you would never spot by reading.

“Slightly favoring” is the key phrase here. For example, the choice between the words “big” and “large” doesn’t change the meaning of most sentences, so nudging that choice to one of the two doesn’t change your text in any way you’d notice. Google tested this on roughly 20 million Gemini responses, and the thumbs-up rate differed by 0.01% (with the watermarked model slightly ahead).

Who else is watermarking their AI-generated content?

As previously stated, Anthropic’s decision to put AI watermarking on its content is mandated by EU laws and regulations that took effect in 2026. Anthropic is not the only company that has complied with this regulation and, according to the EU Commission website, hundreds of companies have agreed to comply with the regulations that require AI watermarks.

For example, OpenAI has recently announced their support for watermarking AI-generated outputs. Here’s a quote from their announcement:

Over the coming weeks, we will add an invisible watermark to eligible ChatGPT and Codex text output in the European Union.

OpenAI’s text watermarking technology is called “textGrain,” and it’s inspired by the art of writing text on a grain of rice. Now, you may be asking yourself, “How much AI text is needed in order to place the watermark itself?” The good news is that OpenAI has published statistics from their own tests for AI text watermark detection. Their AI text watermark detector has 95% accuracy for passages that contain 400-tokens of text. This gives you an idea that only a few paragraphs of text is all that’s needed in order to embed the watermark.

However, since the EU mandate applies to AI-generated content (not just text), OpenAI has gone even further and released a website to let anyone know if their models have generated non-textual media, such as images and audio. I’ve tried the website myself and it correctly identifies a watermark within a few seconds.

The regulation dictating AI watermarks is not just an EU thing. The SB-942 California AI Transparency Act also requires watermarks on AI-generated content. These regulations, combined, not only apply to AI-generated text but also to images, audio, and video.

Google’s model outputs also contain AI-watermarks

Google complied with the EU AI Act years ago, so their models have included the SynthID Text Watermark since 2024. However, they’ve followed suit with OpenAI and released a public website, SynthID.com, to allow anyone to check if any non-text media (such as images, audio, and video) contains the SynthID AI watermark. Google’s AI watermark detector allows you to check for a watermark from multiple AI providers such as OpenAI, Nvidia, Kakao, and of course, Google themselves.

Watermarking all AI-generated content is not necessarily a bad thing

In my most recent experience, I worked as a consultant for a company that was undergoing a VC-backed funding round. After the funding was publicly announced, certain people at the company were being contacted by the “CEO,” who asked them to perform a bank transfer from their personal accounts because of some made-up situation. In reality, it was a group of scammers that used LinkedIn to find the names of employees of the company, and then posed as the CEO so that employees would send money into the scammer’s bank account. Most of us were tech-savvy enough to understand that this was an obvious ploy driven by unscrupulous people. The scammers gave themselves away because they were using email domains that were not owned by the startup, so in this case, the fraud attempt was easily detectable.

However, the technology exists today where voice cloning can be easily accomplished with only a few samples of a person’s voice. So what if a more sophisticated “spear phishing” attempt was done where the attacker decided to use AI in order to clone the CEO’s voice? And what if the fake CEO started calling people (instead of sending emails)? This could lead to a much worse situation where employees think they’re interacting with (and obeying) the CEO and feel compelled to do something that could have grave financial consequences.

What this means for you

All the major LLMs are including watermarking in their AI-generated content, and due to recent regulations, that’s not going away. Bad actors can use AI today to perform serious attacks on the public, so having an AI watermark on any AI-generated content (text, images, audio, or video) is a very useful tool to determine authenticity.

If you want to see a live demonstration on how AI watermarks can be applied and detected on your own, be sure to attend the course Understanding AI Text Watermarking, only on O’Reilly.

While the headlines have been shouting about AI killing humanity, doomsday AI is more likely a smokescreen to divert attention away from real harms that are happening now. We don’t need panic. We need an assessment of where we are: What are the threats? Where are the skills gaps? Where is the budget? It’s hard to overstate the importance of these questions; Tomasz Tunguz, writing about the Theory 2026 AI Survey, reports that security is the biggest barrier to AI adoption. To start answering these questions, in September we ran a survey that asked infosec professionals about the intersection between artificial intelligence and security. 351 respondents completed the full survey, 209 of whom self-identified as being involved with security as part of their job. This report focuses on those 209 respondents.

Attacks, risks, and responses

52% of respondents said their work rarely (34%) or never (18%) involves AI-generated attacks. This seems unlikely. The most likely explanation is that the security profession includes a broad range of job responsibilities; many security professionals don’t deal directly with defense. That group includes auditors, security experts on software development teams, strategists, architects, and others. It’s also possible that most phishes are either blocked by an automated filter (itself an instance of AI) or deleted immediately by the recipient; many companies require their staff to take training courses that cover topics like recognizing phishes. Still, it’s surprising. Much simpler versions of AI have been used to generate spam and phishing email for years. AI’s ability to generate phishes was demonstrated at BlackHat in 2021 with GPT-3. It’s been part of the security picture for a long time.

The most commonly observed attacks are AI-generated phishing (55%), vulnerability scanning and exploitation (30%), and AI-assisted malware generation (26%). Respondents who frequently encountered AI attacks also reported more types of attacks. There’s no surprise here: If you see attacks more frequently, you are more likely to observe specific attacks. The strongest relationship was with AI-generated phishing, which was seen by 55% of all respondents and 86% of those who observed AI-generated attacks weekly. Among respondents who observed attacks daily, slightly fewer (78%) reported phishing. Given the small sample, the difference may not be meaningful. It’s also likely that respondents who observe attacks daily are seeing a much broader range of attacks and that phishing is genuinely less important to them.

Respondents reported seeing deepfake audio and video, which can be used for vishing and other impersonation attacks, less often than phishing, though it was far from nonexistent (25%). More respondents than we expected reported fully automated attacks, like OpenAI’s attack on Hugging Face, though the percentage was still low (16%). The percentage of respondents who observed adversarial inputs against models including prompt injection was similarly low (only 13%), but that’s high enough to indicate that prompt injection and other attacks directed against models are a real concern.

When we asked respondents what AI-related risks concerned them most, we got a different set of answers. The two most common concerns were overreliance on AI tools without human verification and leaking sensitive data via AI tools; 49% of respondents selected each answer. (The number of respondents selecting those answers happened to be the same, though there was little relationship between the two groups.) 

Although the quality of AI output has gotten a lot better in the past year, relying on unverified AI outputs remains problematic. However, AI can easily generate more output than humans are capable of reviewing, and software teams are now overwhelmed by the volume of AI-generated pull requests that need human review before they can be merged. This overload is causing many to rethink the review process and others to reject AI-generated PRs. Code review was never very good at finding bugs; it was better at familiarizing a team with developments in the codebase. It may be enough to rely on exhaustive testing.

Understanding how your software works when much of it is written by AI is a deeper problem. While AI is good at answering questions about a codebase, it usually doesn’t have access to the reasoning behind the code, which is often hidden in institutional knowledge—and institutional knowledge often means “Oh, yeah, that guy who retired three years ago.” In addition, software developers may lose their programming skills if writing code becomes a smaller part of their job. Security professionals themselves might wonder whether relying on AI for red-teaming, log analysis, and vulnerability detection leads to a degradation in their own skills. 

Companies also fear losing sensitive data via AI. Most AI providers offer enterprise accounts that promise that they won’t train models on data in prompts or attached documents, but it’s not clear that these terms of service can be trusted. The frontier laboratories are getting increasingly desperate for training data, even going to lengths such as buying failing startups for their data. Read OpenAI’s description of its Zero Data Retention and Private Safety Processing cynically, and ask if this really protects corporate data or if it allows using data in ways that aren’t directly observable by humans—for example, automatically generating synthetic data. Security professionals are paid to be cynical; thinking about how software, protocols, and policies can be abused is part of the job.

Roughly a third (35%) of the respondents are concerned about attackers using AI to discover vulnerabilities faster than they can mitigate those vulnerabilities and apply patches. Updating software is usually a good practice, but it comes with its own risks: Software updates may introduce new bugs and can lead to incompatibilities with established tools and practices. And defenders can’t yet delegate vulnerability detection, patching, and updating to AI. So overall, is AI more of a help to defenders or attackers? Andrew Ng, Bruce Schneier, and others have argued that the attackers are ahead right now. It’s easier to find a vulnerability, use it to create a payload, and launch an attack than it is to stop one. But the balance will eventually shift to the defenders, as AI becomes more capable of finding and patching vulnerabilities before they can become a problem.

It’s surprising that only a quarter (25%) of the respondents listed AI-generated phishing as a top concern, especially since phishing is the attack most frequently seen in the wild. Perhaps we’ve become too used to phishing; we’ve all been to the security trainings, we’ve all experienced it directly, and some of us have even fallen victim. Phishing is also an attack that, in many cases, is handled automatically. Familiarity may not always breed contempt, but something you see almost daily usually isn’t your biggest worry. 

More exotic attacks are even less of a concern. Only 11% listed prompt injection as one of their top concerns, for instance. However, real-world examples of prompt injection are appearing with increasing frequency. A recent study observed that roughly 1% of résumés contained hidden text to trick an AI system. Google has noticed many kinds of prompt injection, ranging from benign to overtly hostile. And in 2025, OWASP recognized prompt injection as the greatest security risk for users of AI. It’s surprising and unfortunate that it doesn’t appear to be a greater concern for security professionals, especially since prompt injection is a vulnerability that might be lessened by careful filtering but can’t be eliminated.

Even fewer security professionals are concerned about data poisoning (4%). Data poisoning is a serious problem for organizations that train their own models, but today frontier labs conduct much of the most resource-intensive model training. However, this is changing as post-training on corporate data becomes more common. We believe the trend toward locally specialized models will continue, and that data poisoning, as distinct from data theft, will become more of a concern. 

We asked specifically about the use of agents, which exploded this year with the launch of OpenClaw and Gas Town in January 2026. Agents create another set of security concerns. We’ve all heard horror stories: agents that deleted production databases, handled data improperly, generated email that should never have been sent, and the like. Using an agent properly requires a lot of attention to detail, and many users aren’t used to that responsibility, nor do they understand security well enough to exercise responsibility.

The biggest concern among our respondents is excessive permissions or unclear ownership (28%). That’s hardly a surprise; many security problems trace back to permissions. Unclear ownership makes permissions issues more dangerous. When an agent is making many calls to a model, using tools, and delegating to other agents and models, it quickly becomes unclear who (or what) is the authority responsible for any action. This situation isn’t handled well by our current authentication and authorization systems, which assume that identities and authorizations are long-term, stable entities.

Lack of monitoring or auditability was the second highest concern (19%), followed by inability to verify whether the agent stayed within its role (18%). These concerns are closely related: Ownership is often unclear when agents are in use, and lack of monitoring makes the situation worse. Likewise, when you can’t monitor, it’s difficult, if not impossible, to verify whether the agents stayed in their lane—and we’ve seen all too many examples recently of agents that abandoned their authorized roles in inappropriate ways.

Data leakage appears again, though it’s surprisingly low on the list (15%) given that 49% of the respondents indicated that data leakage was one of their top concerns. It’s possible that we’re seeing the difference between single and multiple selection questions; many respondents are concerned about data leakage, but it’s not their top concern. 

Prompt injection doesn’t rank highly for either question: 9% selected it as their top concern about AI agents, and 11% selected it as one of their top concerns. 

Policies and funding

Although nearly three-quarters of the respondents reported that their companies have an AI use policy in place (72%), less than half (44%) reported that the security policy is “comprehensive,” while 28% said that the policy is basic and needs updating. Another 9% said that their employer doesn’t have a policy but is actively developing one. 19% said that there was no policy, or that they “didn’t know,” an answer we consider equivalent to “there is no policy.”

Organizations need clear AI policies that tell employees what AI tools they can use and how they should (and shouldn’t) use them. It’s important for employees to understand whether their company is sensitive to sending data offsite, whether the company’s chosen AI providers guarantee their data won’t be used for training, which practices for attribution are acceptable, and much more. We have some sympathy with organizations that have a basic policy that needs updating. AI’s capabilities are advancing at unprecedented speed; a policy that was adequate in 2023 certainly needs updating now, and a policy that’s only a year old still predates the widespread use of agents like OpenClaw, Claude Code, and Codex, along with the widespread use of skills and tools to extend an agent’s capabilities. Keeping a policy up-to-date with the current state of AI is an imposing task.

The existence of a policy appears to be related to whether employees use AI safely. We asked about “shadow AI” (the use of AI outside of any definitive policy) and found that 36% of the respondents thought that their teammates occasionally use “shadow AI” and 23% thought that the practice is widespread. About a quarter (24%) thought that shadow AI use is rare; and 16% were unsure. 

The relationship between shadow AI and the existence of an AI policy tells us a lot about whether policies are effective. Respondents with a comprehensive AI policy were more likely to say that shadow AI use is rare (37%) or that it’s only used occasionally (36%); only 17% said that shadow AI is widespread at their company. In contrast, 35% of the respondents who reported a basic policy said that shadow AI is widespread—almost the same percentage as those who reported that there was no policy at all (37%) and only slightly worse than those who said that a policy is in development but doesn’t exist yet (28%). Those who had a basic policy, a policy under development, or no policy were less likely to report that shadow AI was used “rarely or never” (13%, 6%, and 21%, respectively).

Granted that shadow AI is probably underreported across the board, our data clearly shows that a comprehensive AI policy is a powerful tool in limiting it. If the policy is poor, or it doesn’t exist, employees will use whatever tools they can to get their job done, which leads to AI use that’s inconsistent, insecure, and possibly counterproductive; it may even lead to legal, contractual, and reputational risks. When there’s no policy, it’s also likely that AI isn’t used as widely as it could be. Telling employees what is expected helps them to act responsibly. A good AI policy doesn’t just tell staff what not to do; it also communicates how to use AI in ways that are appropriate, and may even provide for training. 

When we asked whether organizations are adequately funding AI security, the picture was better than we expected. The largest group (32%) said that AI security is sufficiently funded, and 28% said it’s partially funded. Only 16% said it is severely underfunded, and 23% are unsure. I admit to being surprised by this result. Security is often underfunded, partly because a successful year is one in which “nothing bad happened,” and “we need to spend more so that nothing bad will continue to happen” isn’t a story that plays well at budget time. But executives have been catching on over the last few decades; high-profile attacks and data thefts lead to CIOs and CISOs losing their jobs, and that’s enough to get anyone’s attention. 

We saw a strong link between a comprehensive AI policy and adequate funding for security. Nearly half (49%) of the respondents who said their company has a comprehensive AI policy also said that AI security is sufficiently funded, and another 29% said that it’s partially funded. Only 3% of this group said that AI security is severely underfunded. That means nearly 80% (78%) are at least partially satisfied with funding.

Once again, the answers given by respondents who said that their organizations have a basic policy, a policy under development, or no policy were surprisingly similar. 16%, 22%, and 21%, respectively, said that AI security is sufficiently funded. That’s certainly not a vote of confidence, though we might be able to draw two more bits of information from it. Developing a policy looks like it’s slightly better than having an out-of-date policy; the company is putting in the effort and making an investment. That makes some intuitive sense, though it might also be wishful thinking. Among respondents whose organizations have a basic policy, the largest subgroup said that AI security is partially funded (42%). A middle-of-the-road policy is associated with middle-of-the-road funding. Is that fair, or is that what reality looks like?

Tools and skills

While AI gives security professionals powerful new tools, they aren’t yet fully taking advantage of them. Two-thirds of the respondents (68%) reported that they use a chat, copilot, or similar tool for research or documentation. But only slightly over a third use AI to scan for vulnerabilities (38%), just over a quarter use it for threat detection (27%), and a quarter use AI for incident response (25%). Security professionals widely use general-purpose AI. However, those using specialized tools like vulnerability scanners remain the minority.

To probe further, we asked where AI improved respondents’ workflows. More than half said that AI improved their workflows for research or documentation (61%) and that it reduced repetitive work (57%). But only 29% said that AI improved their workflow for triage, and 17% said that it improved detection accuracy. These results confirm what we saw for tool use.

The relationship between tool use and impact shows up even more clearly when we look at who’s benefiting most from AI. Those who are only using chat or a copilot see the greatest improvement in the speed of research or documentation. Chatbots and copilots save time, particularly when writing reports with a lot of boilerplate text, but they aren’t as helpful when actively defending against or recovering from attacks. Without specialized tools, that work has to be manual.

Respondents using security-specific AI tools were more likely to report improvements in alert triage (42% versus 18%) and detection accuracy (27% versus 5%) than those using only general-purpose chatbots or copilots. 

It’s far from surprising that respondents who didn’t use any AI tools were most likely to say that they saw no noticeable positive impact. The bigger question is why they saw any impact at all. 

Finally, we looked at the “skills gap” to determine what skills and tools would make respondents more effective. Only 7% of the respondents said that they don’t feel like they have a skills gap; that group was also the most likely to say that they aren’t using any AI tools. It’s clear that most of our respondents see AI as a necessary part of the future and want to be part of it. 

34% of the respondents want to use AI tools more effectively. That desire was evenly spread across all AI users, both chat/copilot only (36%) and users of security-specific tools (37%). 24% want to learn how to use AI to secure systems (including the use of AI for red-teaming); 19% want to learn AI-specific attack techniques; and 12% want to learn how to detect AI-generated attacks. 

The future of security

We started by claiming that AI has changed computer security forever. In addition to the many reports of research projects that inadvertently attacked other sites, AI has become a tool for attackers. They aren’t distracted by arguments about whether or not AI is taking jobs; their goal is to exploit their victims, and it’s clear that AI will help them. And as Ben Lorica points out in his newsletter, the economics of attack are changing: AI is infinitely patient, can try multiple attack vectors, can abandon failures, and can work at a scale that’s difficult for humans to match.

AI is also a tool for defenders. We saw more security professionals using AI than we expected, though not as many as we had hoped. Many defenders are taking advantage of chatbots and copilots but haven’t yet adopted more specialized tools, which are associated with improvements in workflows for triage and threat detection. 

The tools for defenders undoubtedly need to mature, but tools aren’t the entire answer. Organizations that adopt AI successfully will have appropriate budgets and policies for AI use; they will understand the risks that come with AI adoption; they will proactively fill skills gaps; and they will recognize processes that are no longer productive and “turn scar tissue into infrastructure.” AI is, in the end, a “normal technology” that takes years to penetrate organizations, as Arvind Narayanan and Sayash Kapoor have argued. Tools, budgets, and thought processes all need to change, and that takes time. Getting ahead of the curve on AI won’t happen if individuals work in isolation, adopting tools and ideas as they see fit. It’s an effort that requires the entire organization.

Generative AI makes the handoff more revealing because it lowers the cost of producing a convincing first version. Students can generate code, analysis, prose, visualizations, and prototypes far more quickly than they could a few years ago. In a classroom, however, a project is useful only insofar as it shows what the student understands and can defend. AI can supply missing code, smooth over incomplete reasoning, or turn a rough idea into fluent prose, so polish is no longer strong evidence of understanding by itself.

A concrete example came from a student paper on AI and labor economics, Verification Responsibility in AI-Applicable Occupations. The paper asked what remains economically important when AI makes candidate outputs cheap but a person still has to check those outputs, decide whether they are safe to use, and remain responsible for accepting them.

The lead student developed the conceptual model and most of the analytical workflow. A second student was given a narrower job: rerun the main analyses independently, record the versions of the source data, document how occupations in one dataset were linked to occupations in another, and compare the reported tables and figures with the underlying outputs. What looked like procedural verification quickly became a substantive review of the study.

The first problem was reproducibility. The analysis ran on the original machine, but too much of the workflow depended on choices that lived in one student’s memory. To reproduce it elsewhere, the team had to record the data transformations, write down the rules for linking occupational datasets, and keep intermediate results so another person could see how the final numbers had been produced rather than simply rerunning a script and trusting the output.

The more consequential problem was in the measurement design. An early version of the study used some of the same autonomy and leadership survey items in two different measures: verification responsibility and adaptive capacity. The software had calculated the relationship between those measures correctly, but part of the apparent relationship could have been created by the fact that the two measures shared some of the same inputs. In other words, the code was doing what it had been asked to do; the question was whether the quantities being compared had been defined cleanly enough.

The team rebuilt the measures so they no longer shared survey items, removed ambiguous items, and revised the workflow so source versions, cross-dataset links, exclusions, intermediate files, and reported results could be traced. Later versions added alternative specifications and robustness checks. What began as “please rerun this” ended up forcing a redesign of the study.

The handoff changed how I read student work. A second user does not merely test whether the files have been packaged neatly; they ask the awkward questions the creator may have stopped noticing. Why is a particular variable there? Which conclusion depends on an assumption that was never made explicit? In this project, those questions reached back into the measurement design.

The educational value of the handoff comes from the fact that schoolwork and production software have different ends. Professional developers learn a great deal after software leaves their hands because they have to deliver it, explain it, support it, maintain it, and respond when someone uses it in an unexpected way. Student projects usually disappear when a course ends, and that is fine; the artifact is not the goal. What matters is importing some of the learning that comes from delivery and maintenance: explaining decisions, supporting someone else’s use, and revising the work when private context is no longer available.

At Force Academy, the exact task depends on the project. For a computational model, another student may change a parameter or replace one rule and then ask what should remain stable. For an AI-supported workflow, the job may be to move it to a different model or environment. For a data analysis, the second student may try to reconstruct one result from the recorded source files and transformations. When that transfer fails, I want the original student to decide what the other person actually needed to know and what should be changed in the project, rather than simply standing beside them and explaining it.

I use the word “handles” for the visible points of entry that make that possible. A handle gives another person something concrete to inspect or change without first receiving a private briefing. It might be a clearly named parameter, an explicit assumption, a test that records an earlier failure, a replaceable component, a retained intermediate dataset, or a prompt whose important context is written down instead of buried in a private chat history.

Handles also make disagreement more productive. An exposed assumption can be challenged; a component with a clear boundary can be modified or replaced without rebuilding the whole project; and a numerical result linked to its source data and transformations can be checked rather than accepted on authority.

Outside school, AI systems expose the same weakness. A prompt, workflow, skill, or agent configuration may look portable because it can be copied, while the conditions that make it work remain elsewhere: tool access, file structure, model behavior, private memory, naming conventions, or small decisions the creator never needed to document. Moving the artifact often reveals how much of the working system was never inside the artifact in the first place.

A first successful run is therefore weak evidence. It tells me that a particular combination of person, model, tools, data, and context produced an output once. A stronger test is whether another person can reconstruct what matters, distinguish assumptions from implementation details, and make a change without destroying the logic of the work.

Tim O’Reilly’s idea of an architecture of participation gives me the broader frame. A technical artifact becomes more useful when other people can enter it without negotiating privately with its creator. In software, that usually means interfaces, conventions, documentation, modularity, and replaceable components. In data analysis, the same principle shows up as recorded source versions, visible transformations, retained intermediate results, explicit assumptions, and enough analytical logic for someone else to rerun or challenge the work.

For education, the milestone I care about is whether the student understands the work well enough to make some of that understanding available to someone else. A second student who can use, question, and extend the project gives me evidence that a polished first run cannot.

AI expands what one person can build, including what a teenager can attempt without a large technical team. The expanded scope is a real gain, but it also makes successful generation easier to confuse with understanding. When I want to know what a student really understands, I now learn more from the first serious handoff than from the first successful run.

Every other Monday, I’ll take an argument NFL fans can’t stop having and hand it to an AI agent. We’ll use about a quarter-million plays from five complete seasons of public nflverse data, plus the current season as it unfolds. Each of our eight episodes will give the agent another capability. By the finale, it’ll run the desk unattended, and we’ll check what it investigated and what it correctly refused to answer.

We’ll start with “You have to establish the run.” I want the agent to explain what that claim means and what evidence would support it.

Define the claim before querying the data

“Establish the run” can mean that running early pays off late, wearing down the defense, or setting up play action. These are very different claims and they need different data. For this first episode, I focused on whether teams that run more often early in a game win more often.

Before the agent called the tool, I asked it to write one sentence explaining what would have to be true in the data for the claim to hold, including when in the game it applies. “Establish” here means early. A whole-game number doesn’t really test that. Give your agent the same assignment, and check that it’s answering the question you intended.

Give the agent a job you can check

Before any agent touched this, I ran the analysis code myself. The tool did the counting; the agent interpreted the results. Then I wrote the policy in a Markdown file, spelling out its job, the read-only tool it may use, and what its answer needed to include. It has to explain what the plays can’t tell us. It may not say running causes winning.

Try questions outside that job too. When I asked which team I should bet on, the agent refused without calling the tool, because that request is beyond the scope of our agent. But when a viewer asked about running trends, the results got tangled up with the “establish the run” instructions. I had to clarify what I was asking, and still wanted to see the code and validate the answer. As you give an agent more responsibility, check both its results and whether it understood the task.

Make the filters part of the answer

“You have to establish the run” sounds airtight at first. Over the last five seasons, teams with the highest run share, the percentage of their plays that were runs, won 77.7% of their games. Those with the lowest run share won 19.2%. I initially looked at that and thought, “Problem solved, we’re done.” Most of that gap, though, reflects teams that were already ahead running out the clock.

When the agent looked only at first-half running in games within three points at halftime, the result flipped. Teams with the highest first-half run share won 46.3% of the time; those with the lowest won 55.3%. This season through Week 5 points the same way, but each group contained only 11 team games. One game could swing it. That doesn’t mean passing causes winning. We’re measuring association, and the filters change which games we’re comparing.

One filter changed the whole story, which is why the filters have to be on-screen. I wanted to see how many rows remained and which commands produced the results so I could go back and check. Make those details part of your agent’s answer, including when it’s working with a smaller sample.

A few numbers from last week’s games

In Week 5, the team that ran more in the first half won 6 of the 15 games, lost 8, and tied 1 (Giants at Washington). Over the past five seasons that team won 54.3% of the time, so last week came in below the average. That’s not unusual, though. In 27 of the 95 regular-season weeks since 2021, the team that ran more early finished the week with a losing record. One week of games doesn’t say much on its own, which is why the agent reports five seasons alongside the current one.

One more data point: The Monday night game, Buffalo at the Rams, was the first the agent hadn’t seen. Buffalo ran on 60% of its first-half plays to the Rams’ 35%, led by six at halftime, and won by three in overtime. Those 199 plays are in the data now, and the close-at-halftime numbers from the episode didn’t change.

Next up: “Ban the Tush Push”

On October 26, we’ll tackle “Ban the Tush Push.” Not one of the 260,927 play descriptions from 2021 through this season’s October 11 games contains the words “tush push.” After adding Monday night’s game, we have 261,126 descriptions, and “sneak” appears in only two, both from 2021. The data isn’t going to hand us a definition.

Before the agent can judge the play, it has to define it. I’ll challenge it to write that definition into its first permanent document, a shared data dictionary, and cite it in every ruling from then on. Bring your own definition, because the answer depends on which one you pick.

O’Reilly members can sign up on the O’Reilly learning platform to join me live on October 26, 8:00am–8:45am PT / 11:00am–11:45am ET. Everyone can follow along on Radar every other Monday, with a brief pause for the holidays. Watch episode one above, and we’ll build on it in the next session.