This topic was hot enough that Jesse Vincent held an entire session about it. I did not attend, but Jesse gave me a report from his Plaude AI notetaker. Here’s a relevant section of the summary:

Several participants argue that software development’s primary artifact may be shifting from executable code toward natural-language prompts, generation processes, and developer skills. One speaker compares Python to assembly language, suggesting that writing conventional code may become optional as natural-language generation takes precedence. Another argues that open source can survive such a transition because its core values—peer production, capacity building, and shared specifications—do not depend on who or what writes the code.

At the same time, Jesse warns that maintainers are overwhelmed by floods of uninvited, automated, and often low-quality contributions. Participants say that traditional milestones such as pull requests, repositories, and conferences have helped contributors connect and gain recognition; as AI makes code replication easier, communities may need new ways to acknowledge participation and create paths for future developers. Some community interaction has moved to spaces such as Discord and direct messages, but structured opportunities are still needed.

Scientific computing raises particular concerns. One panelist warns that AI-generated forks and agent attacks could threaten carefully verified foundations such as NumPy and SciPy. Scientific systems depend on battle-tested libraries and reproducibility, so thousands of unreviewed alternatives could undermine reliability. Participants suggest bridging traditional open-source communities and modern AI development, which currently may use different tools and artifacts.

Other proposals include treating code as “liquid” and focusing open-source governance on durable infrastructure and specifications; maintaining agent storage independently of ephemeral container lifecycles; and using corporate compliance and safety-monitoring architectures to aggregate organizational traces.

These are important observations, all pointing in the same direction: AI is transforming the mechanisms and the artifacts of software development. For an increasing number of use cases, the software source code itself may no longer be the most important shareable knowledge artifact. This isn’t a new phenomenon. Every major change in technology calls for new mechanisms, new artifacts, and new institutions.

Originally, software developers mostly distributed binaries, then open source developers made sharing source code the norm. Now we need to distribute and share something else: context and patterns for generating code on the fly. Knowledge is being unbundled and can be shared in new forms. A repo may now serve merely as a source of context for solving the exact problem I have, rather than the bundle of problems that the original developer(s) encoded into their library or packaged solution. Some people will keep their context and patterns private, either because they see proprietary advantage in keeping them to themselves or because they are too busy or too lazy to share. But part of the problem is that we haven’t yet figured out what needs to be shared!

The challenge for the software development community is to identify the new knowledge artifacts of the AI economy, and to build new mechanisms and incentives for sharing them.

We’ve all had the experience of seeing a bespoke agentically generated app, wanting something similar, and wishing we could ask for a starting point so we didn’t have to vibe code an equivalent from scratch, with all the attention, false starts, and token costs that comes with that. Even in a world where software is generated by agents, it’s still not costless, and most of us like a good place to start.

Of course, this won’t be all software. There are lots of areas where “sort of alike” software is a recipe for disaster. Imagine a sales and marketing group where everyone had their own vibe-coded CRM tools, with different metrics and capabilities. This is not good. So there is still a place for big SaaS applications, distributed binary software, and distributed source code. You pick the hat to fit the head. But there’s also a need for something new.

The purpose of this article is to start a conversation about exactly what it is that developers should be preserving, sharing, and building on from others, and what mechanisms are still missing to make that virtuous circle work more effectively. I can’t see into the future; I can only speculate. So, as is often the case, I take a historical view. The poet W. S. Merwin wrote, “We are the echo of the future.” I have found that using history to frame the present can be a productive way to get our minds going in the right direction.

Note that while this article focuses on software development, the questions I’m asking here apply to virtually every field. Let me know if you have insights from your work in any area touched by AI.

From code to context

The ease with which LLMs can generate code is paradoxically getting people to focus too much on how easy it is not to produce code. That may be blinding us to the fact that code itself is being displaced as a valuable, tradeable, shareable knowledge artifact. Maybe “displaced” is too strong a word. Different species of value overlap for a long time. Eventually, though, old ones fade and new ones become dominant. This has happened before. Every technology revolution draws not only on new mechanisms but also creates new valuable knowledge artifacts. I talked about this in my recent short keynote at AGNTCon + MCPCon.

The open source revolution depended in part on mechanisms like modular design and source code control, even more than it depended on free software licenses. It birthed new coordinating institutions like GitHub. But it also created a valuable new class of knowledge artifact in the form of the source code repo, and subsidiary valuable artifacts like commit history and reputation. A developer’s public repos became a kind of résumé, and a project’s commit log became a record of who had done what, and how well. This replaced a previous world where binaries were the tradeable objects of the software economy. Microsoft’s fortune was built on selling licenses to binaries, and to anyone who measured the software economy in licenses sold, free software looked like value being destroyed. (Steve Ballmer called GPL’d free software “a cancer” and Microsoft VP Jim Allchin called it “an intellectual property destroyer.”)

There were two ways that I like to think about the effects of open source software. First, value that was based on scarcity was replaced by value that was created by abundance. Back in 1975, Steve Baer described this as “the clothesline paradox,” explaining that when you put your clothes in the dryer, the energy you use is measured and counted, but when you hang your clothes on the line, it is ignored by accounting, despite doing the same job. Open source software was like sunshine, free for the taking, and its value unmeasured. Everyone compared the revenue of free software companies like Red Hat and MySQL to the vastly larger revenues of proprietary software companies like Microsoft and Oracle, noted the fact that they hadn’t even survived as independent companies, and concluded that open source was a less effective business model because its creators captured less of the value. They missed the point that far more value had been created than had been captured. Open source went on to power most of the internet economy, with the value captured by other means than selling the software itself.

Some people might think of this as a kind of failure, but they would be wrong. It is the system working exactly as its creators intended. I remember talking with Bob Scheifler, one of the creators of the X Window System, about whether it was OK that we were building commercial versions of their documentation. He told me that was exactly what he had intended. They put things out with a very permissive license precisely because they wanted others to build on top of them. I remember him saying “We want to raise the floor.” The creators of Berkeley Unix, Apache, and others felt exactly the same way. They didn’t want to win the existing game. They wanted to play a different game.

And they succeeded, big time. As Clayton Christensen noted in 2004, when one thing becomes a commodity, modular and fungible, something else (usually something adjacent) becomes valuable. He called this “The Law of Conservation of Attractive Profits.” At about the same time, I had stumbled onto this principle in explaining the implications of open source software in giving rise to the new paradigm that I was shortly to start calling Web 2.0. Clay and I first bonded when we gave our respective keynotes about essentially the same thing at the 2004 Open Source Business Conference. I later wrote about the conjunction of our respective ideas in a piece called “The Open Source Paradigm Shift.”

New artifacts are often the first visible sign of this value transition. You can see this at work in the evolution of the web. The web was born from the new mechanism of internet-enabled hypertext via HTTP, HTML, and the modular, replaceable software (the web browser and the web server) that made it useful.

At first, the new valuable knowledge artifact was the website with its domain name, associated URLs, and HTML-formatted web pages. DNS was the coordinating institution. People paid fortunes for domain names on the premise that the site was the thing that mattered. And it did matter, and still does. But less than it used to. Over time, Google became the new coordinating institution, and its ability to point deep within websites made the web page, not just the site itself, a valuable knowledge artifact. Social media made “posts” and short-form video into other knowledge artifacts. New kinds of value creation and value capture tumbled out of that unbundling. Packaged software was displaced as the key monetizable artifact and replaced by web applications, which were information- and software-based frontends to vast backend databases.

The click became a new and incredibly valuable knowledge artifact, the unit in which attention and intent were measured, and then priced. First pay per click advertising (Google), then engagement maxing (social media) became the basis of vast fortunes. The people who notice the new artifacts early, while everyone else is still focused on the old ones, then invent new business models based on them, often play a bigger game than those who were still trying to play by the new rules.

When I summed up these ideas in “What Is Web 2.0” in 2005, I talked about the key principles being “harnessing collective intelligence,” “software above the level of a single device,” and “data as the next Intel Inside.” (That is, the next valuable source of lock-in.) The companies that survived the dot-com bust were the ones that understood that the data their users generated through their activity on the internet was the asset that mattered, not just the software that served it. That was an argument about a new knowledge artifact, though I didn’t use those words at the time. PageRank understood that a link was a vote, and a new mechanism to improve organic search birthed new dimensions of the internet economy. Links were bought, sold, and gamed accordingly. SEO (both “white hat” and “black hat”) and search quality became a battlefield.

All those disciplines have had to change now that content has been absorbed into AI models. Books, music, video, web content, sites like Wikipedia, GitHub, and Stack Overflow were all grist for the mill. In the open source era, a voted-on answer to a well-posed question on Stack Overflow was a valuable artifact, and the reputation that came with writing good ones was a subsidiary artifact, just as it was on GitHub. The number of new questions asked on Stack Overflow peaked at roughly 200,000 a month around 2014 and was already sliding before ChatGPT, but the decline accelerated sharply after November 2022. The knowledge in those answers didn’t go away. It was absorbed into the models, which now give their own answers synthesized from all that they have consumed. In that way, they have also consumed much of the traffic that would have gone to the original knowledge creators. New mechanisms to create a circulatory economy of value creation need to be developed.

Software development also changed radically because of the internet. Another of my web 2.0 principles was what I called “the perpetual beta.” It’s easy to forget that in the PC era, software had major releases only every couple of years, with the new version distributed on CDs (and long before that, on 9-track magnetic tape.) I remember talking with Mark Lucovsky, who’d been a developer on Windows NT at Microsoft, shortly after he arrived at Google in 2005. He was exulting at the experience of being able to push a single change live worldwide soon after it had been tested. Eventually, we saw the formalization of new mechanisms like test-driven development (TDD) and continuous integration and continuous delivery/deployment (CI/CD), which we now take for granted, as if they’d always been the norm.

The story isn’t over yet, though. The old knowledge artifacts don’t go away all at once, and they don’t go down without fighting. The new artifacts are only half-formed, and have weaknesses that can be exploited. We’re in the middle of that fight right now. But I suspect the winners will be those who first understand what the sources of deep value will be five years from now.

When code is cheap, what becomes valuable?

When code is cheap to generate, the valuable thing may be the prompts, context, and other artifacts that tell the model what code to generate and how to do it well. I’ve started using the generic term “context bundles” to describe the artifacts that you hand the model to do its work.

In the opening to this piece, I talked about software patterns extracted from code as one kind of new shareable knowledge artifact. But these are only one of many kinds of “context bundles” that are starting to be shared. Spec-driven programming is all about developing context bundles. The spec becomes the thing that is reviewed, versioned, and kept, and the code is regenerated from it as needed. (A lot of people are worried that this might become a return to waterfall development. And it could, if done badly. But done well, with rapid turns? Maybe just right.)

An Agent Skill is another kind of context bundle, a folder of instructions, scripts, and reference material that an agent loads only when a task calls for it. Anthropic released the format as an open standard, and it has been adopted by a growing number of other agent products. An MCP server is yet another mechanism for presenting context that will shape the work of an LLM.

But that’s not all. The agent trace, the record of what an agent tried, which tools it called, and which context it drew on, is the closest thing we have to commit history for work that doesn’t get formalized as code. The log of which context bundles were invoked, and by whom, could also be the raw material for attribution and payment, much as the click was the raw material for advertising. And the stream of what people ask their agents to do may become to this era what the click was to the last, the signal of intent around which the coordinating institutions of the AI economy get built. I’ve often wondered what the agentic equivalent to PageRank will be. Will agents leave “pheromone trails” that grow stronger or fade over time, in the way that links do?

Even a well-formed question or conjecture could be a kind of context bundle. Math and science already value and exchange these, and I’ve started to think that there is an opportunity for building infrastructure for sharing and crediting them. After all, arXiv.org has already pushed the boundary for credit further upstream by publishing preprints rather than just reviewed papers. Why not go all the way to giving priority credit for well-formed conjectures?

The eval, a set of test cases that encodes what good work looks like for a particular task, may also turn out to be one of the most valuable things a domain expert can create, withhold, or share. But there’s a serpent in the Garden of Eden. At Foo Camp, Nick Vincent observed to me that there’s a real risk to evals as a context object, because there is a race going on to acquire massive amounts of data (say on corporate workflows) and sell it to the labs to train on it. But an eval only works if the model being judged hasn’t already seen the answers. Once the test cases leak into the training data, the eval measures what the model memorized rather than what it can do. How do we protect the holdout set for evals? That may be a critical mechanism that is still needed, and one that the people who build evals, and the companies whose workflows they describe, have every reason to develop. IMO, it would be good if there were more transparency into this process, and perhaps even supervision by a neutral body.

How do we come to trust new artifacts?

Part of the work of every revolution is building the mechanisms that help market participants to trust its new artifacts. In his book Nature’s Metropolis, William Cronon tells how Chicago became the center of the world’s grain trade in the 1850s. The mechanisms that enabled this were the railroad, the steam-powered grain elevator, and the telegraph. The coordinating institution became the Chicago Board of Trade. In 1856 the board adopted uniform grading standards. Before that, a farmer loaded his grain into individual sacks, and every buyer along the way had to inspect each lot or trust the seller. Afterward, a load of wheat was inspected by a middleman, dumped into a bin with other grain of the same grade, and its owner was handed a receipt for, say, a thousand bushels of No. 2 spring wheat. The grading standards became a new mechanism enabling trust and the grade receipt the new knowledge artifact certifying it. Together they separated the ownership of grain from the grain itself. Once merchants started trading receipts for grain, it didn’t take long before they were trading promises of grain not yet delivered, and the futures market was born. (That raises an intriguing question: What will be the derivatives of the AI era?)

Cronon’s history also observes that the receipts were only as good as the warehousemen who issued them. Some of them issued receipts backed by no grain at all, or spread rumors that stored grain was spoiling so they could buy receipts cheaply. It took years of fights inside the Board of Trade, and finally state regulation that the Supreme Court upheld in Munn v. Illinois, to make the receipts trustworthy. New knowledge artifacts invite counterfeits.

Even when there’s no fraud involved, there are often missteps in the early days of a new technology. Consider the rise of spreadsheet models that could be filled in with the user’s own data. A model was a set of assumptions and formulas that let anyone ask “what if?” and get an answer in seconds. Magic at the time! And eventually completely taken for granted. The model became an artifact that could be passed around, and used to make business decisions at companies other than the one that had created it. In 1984, a few years into the spreadsheet revolution, Steven Levy wrote “A Spreadsheet Way of Knowledge” about what VisiCalc and Lotus 1-2-3 were doing to American business. Levy warned that a spreadsheet model is only as good as its assumptions. Back then, people taking a poorly-executed spreadsheet model as gospel was the equivalent of some of today’s AI usage fails.

Feeling our way into the future

When people stop valuing the thing they used to value, ask what they are coming to value instead. Being alert to valuable new knowledge artifacts is a superpower. The new artifacts tend to show up well before the institutions that organize them, and long before anyone has worked out how to measure them. GitHub was founded more than a decade after open source software became mainstream, and Google didn’t become dominant till more than a decade after the first websites. My bet is that we are at a similar moment with context, with the artifacts starting to appear and the institutions that will coordinate them still unbuilt.

There is a lot of work that I’m following eagerly, looking to see where, whether, and how it gains traction.

• Duncan Davidson is doing fascinating work on what it would mean to have a home for personal context. I can’t wait till he makes more of this work public and reusable. Relatedly, the open memory protocol initiative is an attempt to define a useful set of mechanisms for exchanging context.
• Sam Schillace has started building a repository of what he calls agent-building patterns. One of those patterns, “Anchor on Canonical Names,” echoes the idea that I wrote about last year in “Magic Words,” and that we are also exploring with the O’Reilly Expert Intelligence MCP. As Sam puts it, canonical names are a way of referencing “a whole coherent methodology the model already learned in training, far more reliably than a paragraph paraphrasing the same idea.”
• There are numerous attempts to reinvent GitHub for the agentic era, including Cursor Origin and Entire, founded by former GitHub CEO Thomas Dohmke. Entire describes itself as “new distributed Git forge that stores your intent alongside your code, built for the scale and throughput of agentic coding.” Instead of traditional pull requests, it uses Trails, an agent-driven review system that logs session traces and automated evaluations so developers can merge code without reading every single line of a diff. Origin is an agent-first Git hosting and collaboration platform designed around structured semantic diffs, machine-readable merge states, and automated agent routing. As commercial products, both of these are examples of existing software distribution mechanisms and artifacts.
• Perhaps most interestingly for the point I’m exploring in this piece, Anthropic is building a proprietary, internal-only replacement for Git itself for its own use. The company published a very detailed description of the system, which it calls Scion. It might even be detailed enough for someone with a big team and a ton of compute to reproduce, using the description as a starting point. But you can see at the scale of what they have built the limits of the “distribute context and specs and rebuild on the fly” model.
• But there are also attempts to build systems that enable developers to share lightweight pieces in a new architecture of participation. Common Fabric is building an entire system premised on disposable agent-generated software expressing repeatable patterns, using data that is associated with users rather than with applications and that carries with it its own rules for how it can be used. In my conversation with cofounder and CEO Alex Komoroske, he described these patterns as small pieces of code that can be fetched from anywhere on the internet or written by an LLM on the fly. Agents will mostly write them, not human developers. But they will mostly write them on users’ behalf, and once someone’s agent has built a pattern, anyone else can use it. Their solution may not be the winning one, but it is definitely a bold leap into what may be a very different future for software.

Don’t get trapped in the past of software development. Root yourself in its history, the soil from which it grows, and imagine its future. What will we want to share, what will we want to reuse, and what mechanisms are needed for us to do so effectively?

Having said all that, I understand the frustration when a new model becomes available, but your assistant won’t let you choose that model. Somebody has decided which models you can use and when you can use them. You are left trying to decide whether the answer is good enough and can’t get your hands on the new shiny toys. At the same time, I’m quite happy to let an assistant choose a model if that makes my budget last longer. New toys tend to be expensive and top-line models don’t make a difference here. There are plenty of steps in development where a cheaper model might do everything required. Using it for those steps could leave more of the budget available for a difficult investigation later. If routing lets me keep using capable assistance throughout the day, I have a good reason to use it. I want routing to give me more assistance for the same budget. The quality of that assistance depends on how the models, tools, and context handling fit together. And I need enough information about what the assistant did to judge the result.

Getting more help from the same budget

A fixed AI budget puts a fairly ordinary constraint on a team. Or better, on the affected individual.  You have an amount you can spend, and of course you want as much help as possible from it. Each request uses some of that amount. And it really gets frustrating when repeating a request, correcting an answer, and asking the assistant to try again consume even more of the budget.

Automatic routing gives the system a way to manage that spending for you. It can use a cheaper model where that model is likely to succeed and spend more where the additional capability is likely to help. You, the developer, can then concentrate on the task while the assistant handles some of these decisions and you don’t have to constantly worry about which model to pick for which task.

The researchers behind RouteLLM trained routers to choose between a stronger model and a cheaper one using preference data. They reported more than a twofold reduction in cost with minimal impact on response quality on the benchmarks they evaluated. You still need to check how it performs on your own tasks.

The next question is how those savings get to me in my day-to-day jobs. If requests become cheaper to serve, the same subscription could support more of them. A team could keep its assistant available to more people within an existing spending limit. The saving could also make it practical to use a more capable model more often when the task calls for it. So much for the theory. In most assistants that behavior is little more than a “switch.” That connection to better results needs to be visible in the product. If I choose automatic routing, I want to understand what I get from it. Perhaps I can send more requests before I reach a limit. The results still need to meet my expectations, of course. This is a fairly reasonable thing to ask of a feature that spends part of the budget on my behalf.

The time we spend with the assistant belongs in that calculation too. Think about how a cheaper request can lead to another attempt, followed by another explanation, followed by a correction. If this happens often enough, I spend more time getting the code ready to accept than I would have with a more capable model. So when we compare routing policies, we need to follow the task through those interactions. Then we can look at both the money spent and the time we put into getting there and compare both together.

Consider a Java service that sends an order to another service. Occasionally, the request times out, the caller retries, and two orders appear. You ask the assistant to investigate. It can begin by finding the retry code and following its callers, which may be manageable with a cheaper model. As it reads further, it finds a possibility that needs more investigation: The receiving service may have saved the first order before the response was lost. The caller sees a timeout, but that doesn’t tell it whether the order was created.

The assistant now needs to inspect how the receiving service handles a repeated request. A proposed fix should be checked against the failure we are concerned about: save the order, lose the response, and repeat the request without creating a second order. If the assistant can only inspect the caller, it should leave the receiving service’s behavior as an open question. Spending more on a model may help with the reasoning once the relevant code is available, but the assistant still needs access to that code or evidence of its behavior.

The task only becomes clearer as the assistant reads more code. That gives the router more information than it had when it saw the original prompt. It may make sense to spend more on the next part of the investigation, even if the initial steps were straightforward. A recent SWE-Router paper takes this approach: A cheaper model makes a short initial attempt. The router then uses what happened to decide whether to continue with that model or switch to a stronger one. Its benchmark results show that these early actions can improve the routing decision. That seems like a sensible direction for coding assistants: Let what the agent discovers influence how it continues.

The model operates inside a harness

Following the retry through the code also shows how much of this depends on the harness, the software surrounding the model. That’s not only the client running on your machine; it includes the router and backend logic. They provide context and tools, manage the conversation, and control how the agent proceeds.

The assistant needs to find the code that sends the order. It also needs to find out how the receiving service handles a request it has already processed. Once the model has that information, it can reason about the interaction. The harness has to help the agent get there.

This includes some really practical details. Can the agent search the relevant repository? Does it retain the information it found as the conversation gets longer? Can it run a test against the proposed change? If that test fails, does the failure reach the model in a form it can act on? These details affect both the result and the cost of reaching it. An assistant that loses relevant context may need to find the same information again. We pay for those attempts, and we may need to explain the problem again too. When the assistant finds the right code and runs an appropriate test, the result is way easier to review.

A model benchmark tells us how a model performed under particular conditions. The assistant we use has its own instructions and tools. It decides what information to provide to the model and what to do with the response. We experience all of that when we ask it to do something. A new model can improve that experience considerably. I want to use it with a harness that can find the relevant code, retain context, and run the tools the task needs. I also want the people integrating it into my assistant to check how it behaves with their harness.

The same applies when a company runs an assistant on its own infrastructure. It might connect to a model through an approved cloud end point or use a model running on its own hardware. The developer may see the same interface in both cases. Behind that interface, the model may have different tools or different access to the code. We made this case in our O’Reilly book Applied AI for Enterprise Java Development.

Returning to the order example: Suppose the assistant proposes another retry without examining whether the first order was created. The model may have struggled with the problem. It may also have been missing information about the receiving service. Looking at which files it examined and what it checked helps us investigate the failure.

This is exactly the kind of information I would like my assistant to make easy for me to find. It gives me somewhere to start when the result is disappointing. I can try another model, and I can also check whether the assistant found the information it needed. I believe I am less helpless in front of an already very magic system.

But what can a developer judge?

We have a reasonable chance of judging an assistant’s answer when we know the system and can examine what it has done. In a familiar service, I can follow the changed behavior and read the tests. I can then decide whether the change addresses the problem. With an unfamiliar service or programming language, I need more time to understand the code before I can make that decision. The assistant’s explanation can help me get there. It can point to the relevant code and explain how the change affects it. If it says that repeating the order request is safe, I can follow that claim into the receiving service. I need to see how that service handles a request it has already processed.

A clear explanation makes a proposal easier to understand. Checking the proposal takes another step. With that, I have seen confidence develop faster than that understanding. And I always wondered why. In an earlier study of an AI coding assistant, participants with access to an assistant produced less secure code while being more likely to believe their code was secure. The experiment used an older model and specific security tasks, so was not a general-purpose study, but even with that limitation, it shows why confidence about a change deserves attention. Assistant interfaces should show which claims have been checked and which still depend on an assumption. They should also keep an unresolved question visible as the conversation continues. “Review the result carefully” becomes easier to act on when the assistant helps me find the parts that really need review.

Deciding whether the router chose well takes a little more effort. Spotting a mistake tells me something about the answer in front of me. To find out whether another model would have done better, I need a reasonably fair comparison. It should start with similar code, context, and tools. If I change several of those at once, it becomes hard to tell what made the difference. This leaves me in an awkward position: I am responsible for accepting a result, while some of the decisions that shaped it are effectively hidden from me. A model name gives me little more than a reference. After using a model for a while, you develop expectations about what it handles well and when another attempt is likely to be necessary. Those expectations come from your own tasks and experience. If the service does not expose a precise model version, you cannot be sure that the model behind the same name has remained unchanged.

Despite all those drawbacks, access to a new model gives me something to try when my assistant is stuck. So I really can understand why people want that choice. It’s one of the few concrete ways to influence what happens next.

And of course the choice itself comes with expectations. But is this a real effect? In a study of AI model choice, people planning a holiday chose between a model presented as more capable and another as more energy efficient. Both choices routed to the same model, GPT-4o mini. People who chose the supposedly stronger option rated its responses more highly. They selected their own option, though, and had separate conversations. Some of the difference in ratings could have come from the people taking part or from those conversations. A second study gave 48 people functionally identical chatbots labeled “old” and “new.” The researchers varied which appearance received each label and which version people tried first. Thirty-seven preferred the “new” version. The suggestion that one version was newer was enough to influence their preference. Having some control may also change how people feel about a system. In experiments with social media feeds, people reported greater satisfaction when controls were available, even when the controls changed the feed randomly. Later, researchers told participants in the interview study that one feed behaved that way. Most could then identify it. Once they knew what to examine, they could recognize the difference.

These three studies involved general users doing tasks outside software development. We still need to understand how those findings carry over into coding. But for me, the studies suggest that the model name and the choices we make influence our impression of an answer and its quality. But we are fighting the wrong fight here. The real value would be in giving developers a way to check the answer.

The minimum control I want in a coding assistant could be quite modest. Let me ask it to spend more of the budget on a difficult investigation. Let me request a more capable model when repeated attempts are going nowhere. The harness can use that request together with what it has already learned about the task. I want to be able to leave routine routing decisions to the system, and still intervene when I have a reason. Each investigation gives me an opportunity to learn what the assistant handles well. I need to see enough of what happened to make that judgment, and the conversation is a good place to start.

Make the conversation easier to assess

When I look back at a past conversation, I want to understand what the assistant had available, what it did, and what it checked. That gives me a way to decide whether to accept a change or investigate further. The interface could help by showing more of what happens around the model, but it does not necessarily have to.

At a minimum, I need a context breakdown. The messages I type are part of the information the model receives. The harness also supplies system instructions, descriptions of tools, and guidance such as project rules and skills. All of that takes up visual space, but it shows how much space each part uses and how much is reserved for the response so I can act more appropriately in my next steps in the conversation.

Ultimately this makes developer mistakes visible earlier. When you realize that, even a short exchange can involve quite a lot of information. If tool descriptions take a large share, I can look at which integrations the task needs. Perhaps the assistant is configured for several kinds of development MCP servers, while a simple task only needs a few of those tools. I can try a more focused setup and check how it handles a similar task. It still needs access to the information and tools that help it finish the task.

In experiments I’m driving at IBM, I’m connecting this context information with a record of what the assistant did. I can see context changes alongside actions such as reading a file or loading a skill. From there, I can open the relevant part of the conversation. If the assistant delegated part of a task, I can follow that task as well. I have somewhere to begin looking when a long session becomes difficult to follow.

For example, the assistant might read the same file several times and receive identical content. The view can draw my attention to those reads and let me inspect what happened around them. Of course, the assistant may have had a reason to check the file again. I may also decide to ask it to summarize what it has found and continue from there. I can make that decision after looking at the messages and results behind the observation.

Alongside the activity history, I want to see the current context size, the session’s total token usage, and the cost of its requests. A context snapshot tells me how much space was occupied at a particular point. The usage recorded over a session tells me how many tokens its requests consumed, and cost tells me what I paid for those requests. These figures answer different questions, and I need to know which one I’m looking at. An estimate should be labeled as an estimate. If part of the history is missing, that gap should be visible too.

With the history available, I can also follow a claim back to what the assistant actually checked. Take the order example again: I want to see which code the assistant read and whether its test covered losing the response after the order was saved. If it only examined the caller, I still need to understand how the receiving service handles the repeated request. The conversation should keep that question open so I can provide the missing information or ask the assistant to investigate further.

A report that tests passed needs a little context as well. I want to know what those tests covered and which version of the code they ran against. If the assistant edited the code again afterward, I need a new check. And when it wrote both the implementation and the tests, both may follow the same mistaken assumption. I still need to compare them with the behavior we wanted.

In my article on specification, I argued for agreeing on enough of that behavior to make the result checkable. The assistant can keep that agreement available as the conversation gets longer. When it says that a requirement is satisfied, it can point to the code and the check supporting that claim. If a question remains open, I want to see it before I accept the change. It should also be easy to tell what stage the assistant has reached. It might be proposing a change, reporting that it edited a file, or showing the result of a test run. Those are different points in the task. A clear account of what happened helps me decide what to do next.

In practice, most of this can stay fairly brief. Put the checks and open questions beside the proposed change, with links to the actions and results behind them. I can read the summary, then open the part that needs more attention. As I use the assistant on similar tasks, those checks give me a basis for deciding how much I can rely on it.

I also want to understand how routing spends the budget. I already struggle to see token usage in some interfaces. Showing me how many tokens remain in my budget helps me decide how much more I can do. When one session consumes a large part of it, I want to understand where it went. Showing the recorded usage alongside the assistant’s actions would help me connect the spending to what it attempted.

A team can use this information when choosing a routing policy. Try the assistant on tasks that resemble everyday development. Keep the starting conditions similar and follow each attempt through to a result the team can review. Include the corrections and the time developers spent getting there. Then compare how much assistance the same budget provided and whether the resulting code met the team’s expectations. That comparison may also point to something in the harness that needs attention. An assistant that repeatedly retrieves the same information might need better context handling. One that lacks access to a relevant service might need a different set of tools. Those are things we can investigate once we can see what happened during the task.

Automatic model routing can be great when it gives developers more access to capable assistance for the money they already spend. I’m happy to let the assistant choose models for routine steps and spend more when a difficult investigation calls for it. The whole assistant needs to support that arrangement. Its models, tools, and context handling all contribute to the result.

As the industry develops these systems, we need to keep finding better ways to help developers judge the answers they receive. I want to be able to look at a conversation, understand what happened, and decide what to do next. That is how I can build trust in the assistant while letting it take on more of these decisions.

This event summary was created by Claude with editing by Tim O’Reilly, based on a transcript of the conversation.

Alex Komoroske is the CEO and a cofounder of Common Fabric, a public benefit corporation building the infrastructure for human-centric agentic computing. He spent more than a decade at Google as a product manager and director of product management on Chrome, Google Maps, and Google Earth, and he later ran corporate strategy at Stripe. At O’Reilly’s AI Codecon a year earlier, in a talk called “Why Centralized AI Is Not Our Inevitable Future,” he argued that we were sleepwalking into the wrong future for AI and didn’t have to accept it. Since then he has coauthored the Resonant Computing Manifesto, which Tim signed.

On Tuesday, October 20, 2026, Tim O’Reilly hosted Alex for an hour-long conversation, “Can We Build More-Human Technology?” Common Fabric’s consumer product hasn’t yet been released, but Alex showed Tim the product privately the day before, and Tim said his conversations with Alex had left him “more optimistic about the future of the web and the AI future than I’ve been in a long time.” They talked about the underlying theory behind Common Fabric.

The Resonant Computing Manifesto

The Resonant Computing Manifesto lays out five principles for resonant technology. It is private, meaning you’re the steward of your own data and it’s used in accordance with your wishes. It is dedicated, with no conflicts of interest. It is plural, with no central entity holding an unnecessary amount of power. It is adaptable, able to do things its creators didn’t specifically build in. And it is prosocial, helping people create meaning together rather than living as “individual islands.”

According to Alex, the manifesto rests on a distinction between two kinds of experience.

A hollow thing is something that you like when you’re using it, and then, after you’re done using it, you’re left feeling regret. And a resonant thing is something that you like when you’re using it, and then afterwards you’re left feeling nourished. They are superficially very similar, but they’re fundamentally, deeply, structurally different.

Alex argued that modern tech, business, and politics are very good at delivering hollow experiences efficiently and poor at delivering resonant ones. The manifesto came from a group of people who still believe in the hacker ethic of the 1990s and early 2000s, he said. They believe that technology is a large positive force for humanity, but also that the industry has spent the last decade optimizing for engagement and extraction, which has undermined that positive impact. They think that the AI transition is a unique opportunity to reset the norms and the architecture to make something that works better for people than for extractive business models.

Privacy, bits, and the physics of trust

Tim started out by asking Alex to go a bit deeper on privacy. Privacy and trust are related in ways the privacy movement often doesn’t consider, Tim said. We don’t put bars on all of our windows. We trust most people not to break in because we built a society where trust is the norm. It’s held in place not just by penalties but by social rewards for people who follow norms. “People will think you’re a creep if you go look into somebody’s window,” Tim noted. Like bars on the window, hard privacy seems to Tim a bit like an admission that we’re giving up on positive social norms and penalties for rulebreakers (where needed). He wanted to know whether Alex meant “private” in the way that HIPAA makes your healthcare records private, where your data is made hard for anyone to see, including you.

Alex doesn’t mean that at all. He means privacy that makes your data more useful by putting it under your control. He gave a three-part answer about how that happens. The first leg of the stool is what Helen Nissenbaum, a professor of information science at Cornell Tech, calls contextual integrity. What she means by that is that your data is used in line with your intentions and expectations, with no use that harms you, and no nasty surprises. Alex said it’s the best formulation of privacy he knows. Tim agreed.

The second part of Alex’s answer was that there are differences between atoms and bits. Arguing from game theory, Alex suggested that trust emerges when you expect to deal with the same person or entity again. In the physical world, you might be seen peering into someone’s window and shamed as a result. But “in the world of bits,” because the peeping is invisible, there are fewer immediate consequences, and “you can have a whole bunch of creepy companies peering through your window.” And because your data becomes more useful when combined with other data, there are much larger financial incentives to misbehave. (Tim would still argue that we also need to establish the right set of social norms and laws for regulating this bad behavior.)

The third point Alex made is that data is viral. It can be copied at full fidelity, instantly, for almost nothing, so once it’s out of your sight, it’s out of your control. The industry has relied on three inadequate mechanisms to address this: 1) terms of service nobody reads; 2) permission dialogs, which Alex called “responsibility-laundering dialogs” (He argued they ask an ill-posed question, whether you trust another entity, open-ended and forever, with your data. “I trusted 23andMe with my genetic data 10 years ago, and I sure as hell don’t trust them now,” he noted.); and 3) silos, where each app sees one slice of you. Spotify knows what you listen to and Strava knows when you work out, but neither knows which songs you like during a workout.

Alex said he respects Google, where he worked for many years, but he argued that it has little incentive to build much on your data beyond keeping you from leaving. His point was that “our data is kind of sitting there, not really working for us, and also not really on our turf,” and that this is the result of technical decisions the industry made rather than a law of nature.

Data that carries its own rules

Tim asked what makes Common Fabric different. A lot of people have espoused ideals like those of the Resonant Computing Manifesto, though perhaps in different words, but not much has changed. Alex replied that resonance is hard to build today because users spend their time on the software creator’s turf, doing what the software creator wants. Unless that structure changes, engagement-maxxing keeps emerging. Common Fabric goes down to lower layers and does something very different with the very structure of data. Alex said Steven Levy, writing about the manifesto for WIRED, had described what Common Fabric is working on as “resonant AF.”

Conceptually, in Common Fabric, every piece of data carries a metadata sidecar wherever it goes. In that sidecar, there are two things. One is cryptographic provenance: who did the last computation, the fingerprint of the software that did it, and what its inputs were, traceable all the way back. Alex said the math is similar to blockchains, but without a globally consistent, financialized ledger. The other is policy, the conditions under which the data may be declassified, now or in the future.

An open source runtime, which Alex called “the core reaction chamber” of Common  Fabric, enforces those policies. It receives a proposed graph of computation, made of untrusted code running on specific data with specific policies. If the computation is legal, the runtime runs it, updates the provenance chain, and tells everybody. If not, it stops with a descriptive error, for example because a network request could leak something derived from confidential data. You can run the runtime on your own machine, but most people won’t, so Common Fabric will run it in the cloud in trusted execution environments. These are hardware-isolated environments where the host can’t see inside and can prove cryptographically that it’s running the runtime unmodified.

Tim raised a worry from the history of security software. Early systems such as Kerberos were so complicated to set up and maintain that few people used them. Would this be a master console of access controls so complicated that nobody ever touches it? Alex said most users will never know how it works. To them it will simply be “the best way of using AI in their lives,” a digital home where their data, their collaborators, and their agents come together.

The software in that home is made of what Common Fabric calls patterns, small chunks of untrusted code. Alex stressed that he means “untrusted” in the security sense, which is good, because rather than meaning “not trustworthy,” it means “you don’t have to trust them.” Patterns are written in TypeScript and look much like React. (When the team asked LLMs what framework felt most natural to them, the models said React.) A pattern can do six things:

1. It can do arbitrary computation.
2. It can produce derived data, which persists and is available to other patterns. 
3. It can produce UI for different contexts, such as dark mode or a preview chip, computed only when used. 
4. It can make network requests if policy allows. (LLM calls are technically network requests.) 
5. It can instantiate subpatterns fetched from anywhere on the internet, or written by an LLM on the fly. 
6. It can wish for more data.

Alex walked through a wish. A meal-planning pattern wishes it knew the user’s favorite foods. In one pattern, the system might express this wish by asking an LLM to generate a simple text box for the user to fill in. Another pattern might look for notes tagged “favorite foods” or for the ingredients in recipes the user made in the last six months. Another might read through the user’s Gmail with an LLM and distill what they like to eat. Because the emails are confidential, the result is confidential too.

That pattern could have been written by a stranger and could still run proactively on your data, Alex said, because Common Fabric is designed so that the pattern’s author never sees your data and the pattern can’t use it against your expectations. Anyone can write new patterns (kind of like skills) and Common Fabric will surface the most popular ones for various tasks.

Type checking for your data’s rules

Tim asked Alex to slow down on this point, that you can give untrusted software access to your data because the data, not the software, decides what may be done with it. Alex replied that users will rarely think about policies. Users would choose default policies that are vouched for by people and organizations whose judgment they trust. For instance, he said “I trust Simon Willison. And Simon Willison will say, ‘I trust that repo right there.’”

Alex gave as an example a pattern that pulls your Gmail into Common Fabric. It uses an OAuth flow to get a token from Google. Anyone who has that token can read and send your email indefinitely. So the token is tagged with a policy “secret to google.com,” which specifies that the token should never be logged or shown on screen or sent anywhere that isn’t attested to be running the runtime, not even the user’s own browser. It may cross the network only over a secure connection to the origin that minted it, and only in the authorization header. Any pattern that looks at the token, or at anything derived from it, absorbs those policies permanently. “You can look at the OAuth token, but then you can never talk to anybody else ever again, basically.”

Tim said this sounds like the virality of GPL licenses. Alex agreed, and said that Mitchell Baker, the Mozilla cofounder, had made the same point to him a few weeks earlier.

The term of art in information-flow control is “taint,” (a term Alex joked he dislikes because people look at him funny when he uses it). Data is tainted when it has touched something confidential and so inherits that thing’s restrictions. The aim is to keep as little data tainted as possible so that network requests can still happen safely later. Common Fabric’s compiler untangles a pattern’s data flows so that code which only passes a token along gets a handle to it rather than the token itself, and stays untainted. Alex compared this to Rust’s borrow checker, which refuses to compile code that could theoretically have a data race. Human programmers find it a nag, and a slog.

But you know who doesn’t get impatient about that? LLMs. They don’t care. They’ll say, “Okay, cool. I’ll just make this tweak.”

There’s something different at work in the world.

This is the heart of Alex’s answer to Tim’s question about the risk that complexity will make the system unusable. Common Fabric’s taint checker nags in the same way as Rust’s borrow checker, and LLMs grind through it. “If this software compiles, it’s safe structurally,” Alex said. “You don’t know it’s useful, but you do know it’s safe.” Because it’s safe, the system can run patterns for people proactively and speculatively, without their installing anything, and then share more widely the ones people engage with. Alex said that kind of distribution isn’t possible under the trust model normal software uses.

Crappy patterns and dangerous patterns

Tim noted that the word “pattern” comes from Christopher Alexander’s A Pattern Language, the book about why some buildings and places work and others don’t. Alex added that the manifesto quotes Alexander directly, and that he considers resonance to be Alexander’s “quality without a name.” Tim recommended the book to anyone who hasn’t read it. He recalled patterns such as Alexander’s observation that balconies and porches less than six feet deep are rarely used, and his question of why people sit on stoops in some neighborhoods and not in others. Then he asked, “Won’t people just start making shitty patterns?”

Alex said they will. Most videos on TikTok are bad, and social sifting still brings the good ones to the top. He gave a stylized example of a ranking function. To decide whether to show images at the top of search results for a query, you can check how many people in the last 90 days searched for “images of” that term compared with the term alone, and show images above a threshold. The authentic behavior of a small group of savvy users tells you about quality for everyone else.

“I’m not worried about crappy patterns,” Alex said. “I’m worried about dangerous patterns.” He’s thinking about patterns that would misuse your data. And that’s what Common Fabric is designed to prevent. His example was the draft box in a chat app. Everyone assumes what you type is visible only to you until you hit send, and then only to the people in the chat. Nothing in today’s software structurally guarantees that, and on some marketing sites, the support agent can see what you type before you send it. In Common Fabric, a default policy could make the draft structurally confidential until the Send button is pressed and then declassify it to only the chat’s members. No chat app assembled from this pattern could behave otherwise.

Tim noted there will be a long period of discovering corner cases once the system is out in the wild. Alex said two surfaces must coevolve, the open-ended set of patterns and the policies, which is why trust delegation matters. As people find patterns that slip past common default policies, the policy sets need to evolve to meet the threat. He said he isn’t promising perfect security, which he called an asymptotic ideal. His claim was that for the same security budget, users can reach further.

Today, in the physics of trust we use, a more secure system requires putting yourself in a smaller and smaller and smaller box. What if, instead, a more secure system puts you on a taller and taller platform, allowing you to reach much further?

People won’t come for the security model, he said. They’ll come because they can do much more with AI and with collaborators than other software allows, and the security model is the reason that isn’t creepy or dangerous.

Where’s all the new software?

Tim brought up a question he has written about previously. When a new technology arrives, at first people can see only the old one. When the web came along, Marc Andreessen talked about Netscape becoming the next Windows, and even open source developers talked about a Linux desktop and an open source Photoshop, while the real action was people making entirely new things with the web. With AI, Tim said, we hear about a world of infinite software, but he hasn’t seen many accounts of wonderful new things people have built. Mostly he hears about dark factories that help people get more productive at what they already did. People talk about the tooling but not what they built with it. Patterns, he said, gave him a taste of what infinite software might really be.

Alex said software today orbits corporations, when it should orbit people. He described a “nightmare” in which building software took excruciating, hypermeticulous detail and bickering with dozens of people, and then pointed out that this was how software was built nine months ago. Old software now inspires in him the kind of awe people feel at the pyramids, built without heavy machinery. “And yet where is all the new software?”

His answer was that infinite software built on the old physics of trust can’t produce the things that matter. On one vibe-coding platform, he found a dream-tracker app by an anonymous author that lets you log in with Google, store dreams, and mark some of them public. He started to enter a dream and stopped, because he had no idea whether the author had left everything public by mistake or was reselling it. The old trust model worked tolerably, he argued, because expensive software usually came from people who knew what they were doing, and big companies had lawyers telling them not to sell your data. When anyone can bring software into existence, even that level of trust goes away. Without new physics for distribution and trust, infinite software gives you either “cozy bits of software” from people you would trust alone in your house for an afternoon, or isolated islands that can’t touch sensitive data and “top out at microgames and art apps.”

Software is alchemy. It is our ability to extend our agency beyond ourselves and automate it. And so the idea that software is going to top out at these very low things is just ludicrous to me.

Reclaiming your data

Tim connected Alex’s point about trusting big companies to his own analysis of rising tide rents and robber baron rents, which he developed around the same time Cory Doctorow was writing about enshittification. Companies have strong incentives to treat users well while they’re acquiring them. Once growth flattens, the incentive shifts toward taking advantage of users, staying just good enough that they don’t leave. Part of what keeps people attached even as applications enshittify, Tim said, is that the apps hold so much of their data.

Alex agreed that reclaiming data is the right frame. The business model of the last decade, as he described it, was to build expensive software aimed at an average market, let user data accumulate, and then rent access back, sell derived insights, or hold the data hostage. In a world of infinite software, users can build what they want on their own turf, and he expects silos to become “greedy goblins” guarding the data. He drew a line between what silos add and what users put in. Insights a company derives are its own, but the bits a user contributed should come back out. Infinite software, he said, also tilts the cat-and-mouse game back toward users.

Tim asked how he would get 15 or 20 years of email out of Google. Alex gave Google credit for some of the best APIs, especially in Europe. Alex pointed to msgvault, an open source project that keeps a local archive of a Gmail history, so agent queries go to a local copy instead of rate-limited APIs. When services hand over data in deliberately annoying forms, such as a thousand-page PDF, an LLM can patiently restructure it. “They don’t get bored of a task the way you do.”

Who pays for the compute

Tim pushed on the economics. Services like Gmail are expensive to run, and users get them for very little. If he stores terabytes of email himself, he’s paying someone else, possibly more than Google charges, since Google had ways of subsidizing it. How does this work over the long haul if Common Fabric becomes a new paradigm?

Alex began by acknowledging the value these services have delivered. Free turn-by-turn navigation is now taken for granted, though about 15 years ago, he said, the main licensors of that data charged something like five dollars per user per month. Then he made a point he expected some people to find provocative.

If you aren’t paying for your own computation, then it’s not working for you; it’s working for somebody else.

Users would pay for their own compute by running the runtime on their own machines, or through hosts Alex called AI service providers. These would run the runtime in trusted execution environments, prove they can’t see inside, and charge for tokens, compute, and storage. If users could switch providers, margins would stay small. Alex acknowledged that consumers have long resisted paying for software, but argued that the nontrivial marginal cost of LLM tokens creates an opening, much as people pay a monthly cell phone bill because the service has become indispensable.

Tim suggested companies like Google could migrate many of their services into this world. Alex said there will be many opportunities, but incumbents are used to another business model. He recalled telling a friend with a startup that building on the Common Fabric was wildly premature and that his investors might not like it, because the point is that patterns use data without owning it.

Tim noted that today’s model doesn’t have to stay the model, and Alex said he expects new business models to emerge. Asked how he will get developers to write patterns, Alex said he doesn’t need to. Agents write them on users’ behalf, and once someone’s agent has ground through building a word search, anyone else can use that pattern.

Is surveillance capitalism inevitable?

An audience member, David, wrote that he loves Shoshana Zuboff’s The Age of Surveillance Capitalism and wondered how we came to accept as inevitable that companies take our behavioral data. He said he wasn’t sure technology could fix something technology didn’t cause.

Alex said he thinks technology did decide this but allowed that people may disagree. The same-origin policy, the browser rule that lets a script use only data from the site it came from, was introduced by Netscape in 1995 as a quick fix for a security problem that was discovered early on with JavaScript. As more APIs were added, the walls grew higher, and when native apps arrived on iOS and Android, the same model meant every app owned its own data. Alex said that decision could have been made differently. Surveillance capitalism may keep emerging, he said, but it isn’t inevitable, and technology applied at the right leverage point can lead to a different result.

Tim told David he thought Zuboff’s book had a fundamentally bleak view of the world. For a long time, he said, we had a fairly virtuous set of relationships with many of these companies, and then we created economic incentives that led them to treat us badly. He argued that economic redesign is needed alongside technical redesign. That redesign starts with eliminating the market’s requirement that companies always keep growing even when they no longer deliver additional value.

Alex brought up Eric Ries’s book Incorruptible, which he said he keeps a stack of near his desk. He described himself as a believer in markets and capitalism as an emergent force for good. He distilled the book’s thesis down to the idea that organizations that think long term naturally attend more to their externalities. Common Fabric is set up as a public benefit corporation, and Alex said Eric consulted on structuring it to keep a long-term perspective in perpetuity, advice that Eric is now sharing with the wider world through Incorruptible.

Tim’s argument: AI as a reset

AI is a reset the industry still hasn’t yet taken full advantage of, Tim noted. We’re still thinking in the surveillance-capitalism frame, he said, in the same way that Netscape assumed people would buy software and that whoever controlled the software would control the future. We assume a world of advertising, and a world where companies hold more data about us than we do about ourselves. But new AI native companies will find new business models. Tim called Common Fabric the most thoughtful take on this reset he had seen. He compared it to his first encounter with the web, when there were about 200 websites and O’Reilly put it in The Whole Internet User’s Guide & Catalog because, as he put it, “this is the future.”

Tim said he couldn’t promise that Common Fabric would be the winner, but it is the most promising new thing he’s seen in a long time. He recalled the early days of the internet, when it wasn’t clear whether Gopher, the World Wide Web, or AOL would prevail. The web won, in his reading, because it built an architecture of participation. It was easy to draw from and easy to add to, and its mechanisms kept evolving, so that the web of 2000 or 2005 looked very different from the web of 1991. Common Fabric comes with that same sense of open-ended possibility.

Open source underneath, a product on top

Alex said the underlying runtime and pattern system are open source, but on their own they feel like a tech demo, which is why Common Fabric is building a consumer product on top. That product is still in development and not yet public. But he expects strong network effects as more people join the open ecosystem. Tim said that when Alex showed him the product the day before, this idea made sense. He described his takeaway as agents building throwaway software that embodies a shared set of patterns, with data that carries from the beginning the scope of what can be shared with whom.

Alex said none of this would have been possible before AI could write software. He told the story of his cofounder, Berni Seefeld, who had been thinking about these ideas for decades. Over a dinner a decade ago, Alex asked how they would ever get people to write enough patterns to reach critical mass. Berni said that in the future, AI would write software. Alex replied, “Come on.” Berni gave the structural argument for why software engineering would be the strongest, fastest, and most invested-in vertical, and a few years ago told Alex the curve was happening and it was time to start. Not needing humans to write patterns, Alex said, lets us think much bigger and differently from how the industry has assumed software must work for the last 30 years.

Tim closed on that point. Because agents building software in Common Fabric do it according to principles that put people at the center, he said, the nature of the game changes.

The following article was originally published on James Shore’s Successful Software blog and is being republished here with the author’s permission.

Is AI making software development faster? Most people I talk to say yes. But how much? What value are you getting for your money? Are there other ways of using AI that have a better ROI?

This is part 2 of my series on assessing AI’s impact on software development. In part 1, we looked at why assessing impact is important, and what not to do. In this part, we’re looking at how to measure AI’s impact on delivery speed. In part 3, we’ll look at how to measure the unintended consequences of AI, before wrapping up the series in part 4 with an examination of business outcomes. Finally, an epilogue puts it all together with notes you can share with your CFO.

What we’re measuring

Delivery speed isn’t productivity. I’ll say it again, loud, for the folks in the back: Delivery speed isn’t productivity!

It’s a component of productivity, sure. Given two equivalent teams creating the same piece of software, the team that delivers first is the one that’s more productive.

But it’s not just about output. To my mind, software development productivity is value produced divided by money spent. It’s a strict definition but a useful one, because it gets to the heart of why we build software: to achieve some valuable purpose. Going faster doesn’t mean you’re going in the right direction.

We’ll consider business outcomes in part 4. For now, though, we’ll tackle a simpler question: Are your teams finishing their assigned work faster, regardless of whether that work is actually useful or not?

The core approach

First, the bad news: There’s no objective measure of software development speed. I can’t look at two teams and say, “This one is fast, and that one is slow.” That’s because none of the outputs teams create are standardized. We already talked in part 1 about how lines of code and PRs are bad metrics. Well, nothing else is standardized, either.

Features? Some are big, some are small. Bugs? Some are easy, some are hard. Stories? Velocity? Story points? Completely arbitrary. They have way more to do with what’s being requested than the speed of the team.

Okay, if they’re all different sizes, couldn’t we just use estimates? Surely if a team is constantly missing its estimates, it’s because it’s slow, right?

Not so fast! Software estimates are famously inaccurate. A team that always misses its estimates isn’t slow. It’s just overly optimistic. Teams are nearly always overly optimistic.¹

So we can’t count requests (features, bugs, stories, etc.) because they vary in size, and we can’t look at on-time delivery, because estimates are overly optimistic. Sounds like a whole lotta “can’t.” What can we do?

We can compare a team to itself. If a team’s estimates were perfectly consistent, and they went from taking 3x longer than estimated to 1.5x longer, then they’re twice as fast! Huzzah!

If only it were that simple. There are a few snags.²

Estimates aren’t consistent

Snag the first: Teams’ actual estimate ratios aren’t actually consistent. Some are bigger, some are smaller. Plot them on a graph, and they form an “S” curve, as with this data from a real company.³

If delivery speed improves, the curve will shift. Individual data points may be better or worse than the original approach, but you can see the effect in aggregate.⁴

People work differently

Second snag: Some people are pessimistic and estimate high. Some estimate low. Some people spend a lot of time educating others. Others work heads down as fast as possible. Some clean up messes. Others create messes.

If you do a blanket comparison of everyone to each other, you introduce a lot of noise. Noise is bad because it increases the amount of data you have to collect. To keep things clean, compare people (or teams) to themselves.

Estimates change when people know they’re using AI

Third snag: People’s estimates will change according to the work they plan to do. So if they think they’re going to use AI, and they think AI makes them faster, their estimates will be smaller. Huh. Tricky.

This one’s actually pretty easy: Collect the estimate before people know whether they’re going to use AI or not. Then randomly assign the task to be done with or without AI.

Process improvements muddy the waters

Fourth and final snag: Are your speed increases due to introducing AI? Or the other changes you made at the same time?

AI forces you to rethink your software development processes. Leaders I talk to are streamlining request workflows and putting business experts in closer touch with developers as part of their efforts to introduce AI.

But these are the exact changes that make software development more effective in general! How much of the improvement you’re seeing is from spending thousands of dollars on tokens, and how much is the result of people talking to each other more?

Rather than comparing a chunk of pre-AI data to a chunk of post-AI data, interleave “without AI” tasks and “with AI” tasks. This reduces the influence of other changes. Update your without-AI approach to include the same streamlined workflows as your with-AI approach.

The complete measurement approach

In other words, if you want to be rigorous about this, you need to conduct a randomized controlled trial. Specifically, a “randomized within-subject repeated-measures” trial, which reduces the number of samples you need to collect.⁵

It’s not as bad as it sounds! People still do their normal work.

Here’s what you do. First, establish a with-AI approach and a without-AI approach. (Or, more likely, “with a lot of expensive AI” and “with a sprinkling of cheap AI,” but I’ll say “with” and “without” for clarity.) Make sure they’re clearly defined and people have had a chance to practice both approaches.

The without-AI approach will probably be close to your current approach to software development, but if you’re streamlining your with-AI processes, be sure to incorporate those improvements into your without-AI approach too. Decide how you’re going to attribute rework, such as code review changes and bug fixes, back to the task that caused the rework.

Then give each person (or team) a virtual bag that’s half with-AI assignments and half without. For any given task, have people record their estimate first, then randomly pull an assignment from the bag.

The main data you’ll collect is estimated time spent, actual time spent, type of task, and spending. Decide in advance if you’re tracking calendar time (including interruptions) or effort (excluding interruptions) and how you’re going to handle parallel work. Think about what other useful information you might want, such as bug and incident counts, mental energy, excess lines of code, and so forth. We’ll talk about using that data to model unintended consequences in part 3.

You’ll need the help of a statistician to do the detailed experimental design and data analysis, but broadly speaking, each person’s with-AI data will be compared to their own without-AI data, then aggregated together to give you the final results.

Finally, remember that speed increases aren’t the same as productivity. Pay particular attention to how the tasks you’re measuring fit into the big picture. For example, if you see a 2x improvement in coding speed, but people spend 50% of their week on other work, their overall speed improvement is only 1.33x, not 2x.⁶

Is it worth it?

This is a lot more complicated than just counting PRs and calling it good. Is it worth it?

If this were just some run-of-the-mill process improvement exercise, no, it wouldn’t be worth it. It’s too much work, and the effect sizes have to be pretty large in order for the data collection to be practical.

But this isn’t some run-of-the-mill process improvement exercise. Productivity expectations are off the charts, and so are costs. As I talk to engineering leaders, I’m seeing order-of-magnitude differences in spending. Should you spend thousands per person per month on hands-off agentic coding? Or hundreds on analysis and fancy auto-complete?

It depends on the outcomes.

You can’t pick the right approach to AI without knowing what you’re getting for your money, and with so much money on the table, you can’t afford to half-ass this.

Remember, in a METR study, engineers overestimated AI’s speed benefits by nearly 50%. Without measurements, there’s no way to know what you’re actually getting.

Fortunately, it’s not as hard as it seems. With the level of impact you need to see, and enough participants, it doesn’t take many samples from each person.⁷ Your teams collect the data as they do their normal work. The hardest part is explicitly defining how you want people to use AI, and that’s worth doing anyway.

Of course, delivery speed is only part of the picture. There’s also the question of how sustainable speed improvements are. We’ll consider unintended consequences next, in part 3.

All my content is made with my 100% human meat brain, except where explicitly stated otherwise. If you liked this entry, check out my best writing and presentations, and consider subscribing to updates by email or RSS. To contact me about consulting, email me at jshore@jamesshore.com.

Footnotes

1. When they’re not, it’s usually because they’re playing political games with estimate padding. That isn’t good either, because then the work tends to expand to meet the estimate. ︎
2. This design was inspired by a METR study, with help from Brent Miller. AI Disclaimer: I used OpenAI’s GPT-5.6 Sol model operating in “High” thinking mode to critique my explanation, and I also used it to generate the Python programs that drew the graphs below. You can see my ChatGPT conversation here. ︎
3. This data is from the Star Citizen alpha 3.0 release, which publicized its internal task estimates and actuals. Interestingly, the sharp cliff at the 1.0 mark is a classic sign of work growing to meet its estimate. ︎
4. The Star Citizen data was split in half and artificially manipulated for the purpose of this example. It doesn’t represent real-world AI impact. ︎
5. Many thanks to Brent Miller for introducing me to this technique. ︎
6. Work that took 40 hours now takes 30. 40 ÷ 30 = 1.33. ︎
7. The exact number will depend on the number of participants, the amount of variance in your results, and how conclusive you want the results to be. ︎

Security looms large as we enter November. There’s been a lot of bad news: a certificate authority with woefully inadequate private keys, vulnerabilities have been discovered (and exploited) in PHP and other widely used programs, and several organizations have detected Chinese open weight models misbehaving in several different ways. New models are arriving faster and faster, and the best open-weight models are catching up to frontier model performance. But organizations that would like to achieve AI sovereignty will have to ask some difficult questions about the models they adopt. Is this an inflection point? Whether or not it is, AI has changed security permanently, and things are bound to get worse before they improve.

AI models

We are starting to see models developed with recursive self-improvement; we continue to see clones of Jev; and we’ve seen text watermarks added to output generated by Anthropic’s and OpenAI’s models. (Google has been using text watermarks for some time.) But tools for detecting the watermark are hard to find. Anthropic’s and OpenAI’s are in a very limited beta, and Google’s is only available in their enterprise package. What does a watermark mean if you can’t detect it? And, as we get more accustomed to AI-generated output, will anyone care?

• Kimi 3.1 Pro is a new open weight model with 3T tokens. On benchmarks, it is now the leading open weight model, only slightly behind the latest frontier models. Moonshot AI also released Kimi K3.1 Pro Highspeed, which is two to three times faster.
• Recursive has announced Flint, a 40B parameter mixture-of-experts model that activates 2.5B parameters per token. It was designed for coding, and sized to run on laptops. Flint is open weight; weights are available on Hugging Face, and it works with model runners like Ollama.
• Meta has shipped Muse Spark Open, an open weight model that they claim competes with the best Chinese open weight models. It’s a mixture-of-experts model with 816B parameters and 42B active parameters.
• Meta’s latest models, Muse Wonder and Muse Wonder Full, claim parity with frontier-class models. Specifically, they claim parity with Claude and ChatGPT for financial and legal work. The announcement hints that Meta will be using Muse to develop specialized models.
• OpenAI’s Decisions API is yet another Jev-like model. Like Jev, its input is a data structure that constrains the model’s output to be an answer to specific questions. The output is a data structure that includes a probability for each answer.
• Claude has launched Haiku 5.5, a new version of its least expensive model. It’s faster, less expensive, and has better benchmark scores than its predecessors. Haiku 5.5 is a very capable model, particularly for users who are sensitive to cost.
• Mistral has launched Mistral Large 4, a.k.a Le Chonk. It’s an open weight mixture-of-experts model with 1T parameters. It’s claimed to be the best open model from Europe and was designed with AI sovereignty in mind.
• Aleph Alpha has released an open weight LLM for English and German that was trained entirely under European rules. It is a mixture-of-experts model with 78B parameters. It uses a different tokenizing algorithm to handle long German words without breaking them into fragments. Reasoning takes place mostly in German.
• Crowdflare has released Clef and Clef Flash, new decision models that, like Jev, take typed queries and return typed answers about those queries. It’s also offering a new reinforcement learning service to help customers train Clef for their applications.
• After a long wait, Google has released two new frontier models, Gemini 4 Argon and Carbon. Argon is available to paying customers. Argon’s successor, Carbon, is being released slowly, and is currently only available via the Fairwind program.
• The Ember-1 model from Fireworks claims to have results similar to Kimi K3 at lower costs. It is priced similarly on OpenRouter but uses far fewer reasoning tokens.
• OpenAI has published 722 papers solving problems in mathematics on GitHub. There is some controversy about whether all the proofs are correct; so far, three have been withdrawn, leaving 719.
• Google has opened its SynthID detector to the general public. The detector can determine whether audio, image, or video files generated by any of Google’s partners (including OpenAI, NVIDIA, and Apple) have the SynthID watermark.
• To comply with EU law, OpenAI will add watermarks to text in Europe. The company’s algorithm sounds similar to the one that Google and Anthropic are using: modifying the underlying probability distribution changes word choice in detectable but insignificant ways.

Agents and harnesses

There are more choices than ever for agents, along with new standards for the interaction between agents and models. In addition to general purpose agents, there are also agents specialized for coding, HR, BI, and more. And most agents will connect to many models, including local ones. Choose agents as carefully as you choose models; your agent is where you will do your work.

• JetBrains has released Thinkrail, an agentic IDE that learns about your project as you use it. It is open source and based on the pi agent.
• Anthropic has donated its Agent Skills standard to the Agentic AI Foundation (AAIF), which will shepherd further development. It has also donated Open Responses, a standard for building common interfaces, to AI providers.
• OpenAI has introduced Intelligent UI, a feature that allows ChatGPT to generate its own user interfaces in response to the user’s question.
• Workday is allowing its customers to build their own agents for HR and finance. An agent’s access to Workday data is tied to the user’s account.
• Claude Dashboards is a new product that lets you build live dashboards from a group of data sources.
• Claude’s Google Workspace connectors allow it to read and write Google documents, sheets, and slides directly.
• MCP Events is an extension to the MCP protocol from OpenAI. It allows ChatGPT to subscribe to updates from MCP servers.
• DeepSeek has released a harness (dsh). It’s open source, and currently in public preview. Everything is a plug-in.
• The Pi agent has reached version 1.0.0, and is now capable of using MCP. There’s a variation, Pi Durable, that has been designed for long-running tasks.
• OpenAI dots are consumer-oriented agents that run in the cloud. They are always on and have their own web browser along with other tools.

Software development

At last month’s SciFoo, Hugo Bowne-Anderson said that we’ve been through two major changes in the past few years: from writing code to letting AI write the code, and from reading the code to not reading the code. These changes are as radical as the shift from programming in machine language to programming in high-level languages. We will all go through a process of rethinking how we work and what we’re trying to accomplish.

• Anthropic has introduced Claude Mods, a plug-in for Claude Code that can modify Code’s behavior: reshaping its user interface, providing information about context use, giving advice about what you’re doing, and much more.
• One application for decision models is building evals to understand the performance of AI applications.
• Workday is using Jev to build user interfaces on the fly. Jev’s very low latency makes choosing UI components at runtime viable.
• Anthropic has outgrown Git, which wasn’t able to keep up with merges on their monorepo (roughly 30,000 merges in September). Anthropic has built a new source control system called Scion.
• Kate Huston writes about reverse-engineering code review and trying to develop an effective “gate” that AI-generated code would have to pass before review by humans.
• WordCraft is an open source clean-room re-implementation of Microsoft Word. The authors claim that it’s 60% of the way towards “replacing Word for real work.” It supports MCP, has a command line interface for interacting with agents, and can run in a browser through WASM.
• ArtCraft is an open source tool for creating interactive AI-generated images and video in 2D or 3D. The company describes it as an “IDE for artists.” They’re also developing open source clones of Adobe Creative Suite tools like Photoshop. Early reports are that it’s very buggy.
• Google Docs now supports Markdown natively. Users can read, write, collaborate, and comment on Markdown documents without converting them to another format.
• Google’s Docsy documentation project is moving to the Linux Foundation. Docsy is a template for the Hugo static site generator that’s useful for generating documentation in ways that are helpful to agents.
• Does programming with AI require more abstractions or fewer? Matt Pocock argues for more.
• Anthropic has given Claude an auto-eval tool. While the first version of this tool has many flaws, it’s a significant step towards making evals easier.

Security

Security has been much in the news. Not all of the news is bad: AI was used to discover the PHP2Shell vulnerability and to develop a fix. The leaked CVE and the many out-of-date PHP sites online are serious problems, but not problems attributable to AI. Too many problems arise from getting the basics wrong: A certificate authority using 4 as its private key is a mistake that should never have been made. As always, security is about getting the basics right while watching for new threats. And while organizations claim to be paying more attention to security, the word from insiders is that “attention” means lip service. In the long run, AI may make the world a safer place, but the next few years look scary.

• CrowdStrike and Booz Allen have found that Qwen3-Coder (and probably other models) generates weaker code if the prompt says that the code is for a politically sensitive user such as the US government. Hirundo reports similar flaws and also claims to have “fixed” the model and created a Westernized version.
• Cohere makes a different claim: It has observed that Qwen3.8 models occasionally send data to an Alibaba cloud storage bucket in Beijing.
• Malicious AGENTS.md files have been found in the Downloads folders of some users. If the user is using the Downloads folder as their working directory, an AI agent will read the AGENTS file and download a malicious NPM package.
• 1Password has launched 1Password for Agents, a credential-brokering product that allows agents to access resources for a user without seeing the user’s password,
• Alibaba also donated its OpenSandbox project to AAIF. OpenSandbox is a general-purpose sandbox for running AI models safely.
• The PHP team has released patches for two critical vulnerabilities collectively known as PHP2Shell. However, many sites are running old versions of PHP that need to be upgraded before they can be patched. The maintainers have also released some partial mitigations for sites that can’t upgrade easily, but that won’t fully protect them. Sadly, “the panic hasn’t turned into patching,” and those who don’t patch will pay.
• Brotli isn’t as widely used as PHP, or as old, but a critical flaw in Brotli compression library makes it clear that discovery of hidden vulnerabilities is the shape of things to come. Microsoft has released patches for .Net, PHP maintainers have released patches for PHP, Node.js is also affected, and that’s just the start.
• The private keys of the certificate authority eMudhra have been cracked–not by brilliant cryptography or a quantum computer, but because the keys were generated very poorly. The Chrome and Firefox browsers aren’t trusting the compromised certificates, but almost everything else, including all major package managers, is a problem.
• OpenCode was the victim of a hostile attack in which some users’ API keys and passwords were stolen from their Console service. Its response was hampered by lack of access to AI models without security restrictions.
• The US Department of Homeland Security has banned the use of DeepSeek and applications developed using DeepSeek in all federal agencies.
• To monitor the risks created by the development of frontier models, Anthropic has given six employees of METR access to their models. They continue to work with Accenture and Secure Bio.
• Model abliteration is a poisoning attack in which post-training is used to discover which of a model’s weights corresponds to “no,” and altering them so the model cannot refuse a request. It’s effectively a backdoor into the model’s behavior.
• AWS has launched Strands Box, an open source sandbox that’s designed specifically to prevent AI agents from escaping the sandbox’s control. It includes an open source policy engine, Dogwood, that allows users to place explicit limits on what an agent may do.
• Meta’s Muse, its agent for people who aren’t technology professionals, is a privacy and security disaster.
• Using MCP for communications between agents creates many risks, including new vulnerabilities to prompt injection.
• Google has suspended submissions to the Open Source Software Vulnerability Rewards Program because of the flood of machine-generated vulnerability reports, most of which are (in Google’s words) “not valid.” This suspension should only be temporary.

Infrastructure and operations

There’s been a lot of news about certificate management, not all of it good. It’s time to check on your certificate providers, get ready for quantum-resistant encryption, and automate as much of the process as you can.

• To increase security, the certificate provider Let’s Encrypt will be limiting the lifetime of free certificates to 64 days starting in February 2027. The change should be invisible to sites using automatic certificate management (ACME).
• Crowdflare has announced that it will begin issuing TLS certificates with quantum-resistant encryption. They will continue issuing certificates with classical encryption. Crowdflare hasn’t given a date when they will start issuing the new certificates.
• DeepSeek has open sourced DeepInfer, its inference framework, with an Apache 2.0 license. At present, DeepInfer only supports DeepSeek-V4.1-Flash.
• RemoveMacAI is an open source command-line tool for deleting Apple’s AI from MacOS 27, freeing up 30 GB or so of disk space.

Web

Mark Twain didn’t quite say, “The reports of my death are greatly exaggerated.” But we are hearing that the Web is dead, and that’s indeed an exaggeration. The flood of crawler traffic and the disincentives towards human writing are a problem. But what does “death” mean for a technology so flexible that you can build an office suite that stores entire documents’ URLs?

• TinyLittleOffice.com is a surprisingly complete set of surprisingly capable office applications. It includes a word processor, spreadsheets, and other Office-like apps. Everything is saved in the URL–not on your disk, not in someone’s datacenter. Not suitable for large documents.
• Crowdflare has announced a web search API, now in beta. Although their announcement doesn’t say, we suspect this is an attempt to control web scraping traffic. Matthew Prince, its CEO, has estimated that bot traffic will reach 1000 times human traffic by mid-2027. All of his earlier predictions have been underestimates.
• Jev and its clones have taken the world by storm. Can you beat it at Pacman?

Hardware

• AI labs are heading towards models that can run on phones and similar hardware. Google has compressed Gemma 4 E4B to 5 GB, and Liquid has released LFM2.5-24B-A2B, a 24B mixture-of-experts model with 2B active parameters that also targets devices.
• OpenTPU is an open source API accelerator, designed by AI. Bring your own silicon.

Quantum computing

• Quantinuum has implemented quantum error correction on its latest hardware and has found that it scales well enough to be useful in practice: Adding a physical qubit to a logical cubit cuts the error rate significantly.
• Researchers have discovered a way to read a qubit without collapsing the information it contains into a 1 or 0. Solving this problem (also known as quantum non-demolition or QND) is critical to building quantum computers large enough to do useful work.

Biology

• Watermarking texts is one thing. Google has now designed a watermark for proteins. Like text watermarks, protein watermarks are based on Google’s SynthID.
• Last month we mentioned EEG-capable earbuds. This month, a group of researchers has announced that they have developed an AI application that can reconstruct what someone is seeing from brain scans. The results are impressive, though the privacy implications are chilling.

Check whether the data supports your definition

A quarterback takes the snap, and teammates push him forward. Our data identifies the ball carrier but doesn’t record whether teammates pushed him. We can check whether a snap came from shotgun, but we can’t reconstruct the team’s specific formation.

We explored five possible definitions, from a quarterback sneak with teammates pushing to any rush on third or fourth and short. Each definition changes which plays the analysis counts. The broadest one would count running backs too, which takes us away from the play people are arguing about.

So before you let an agent query a term everyone seems to understand, write its definition in plain language. Then go clause by clause and ask whether the data can support it. A familiar name doesn’t guarantee a usable label.

Write down what your definition can’t measure

We settled on a quarterback run with one yard or less to go, from under center. The agent counts qualifying quarterback sneaks whether anyone pushes the runner or not. We’re measuring a substitute for the tush push, and we can’t isolate the shove’s contribution.

Even “quarterback” needs a definition. The fields available to our tool don’t include player position, so the tool identifies the quarterback as the player who threw the most passes for that team in that game. That’s another assumption to check when you interpret the results.

The dictionary entry has to explain the play in plain language and state what the data can’t tell us. Otherwise, readers could assume a “tush push” result counts plays with a push. Keep the limitation beside the definition so the agent can repeat it with the result.

Give the next run the same definition

The dictionary holds one JSON file per definition, with a fixed structure. The agent can add an entry, but it can’t change an existing one. A different definition needs a new name. That keeps the meaning attached to an earlier result from quietly changing later.

Before each run, the script that launches the agent, called the runner, saves a copy of the dictionary and adds its contents to the agent’s instructions with the date. The agent searches for an existing definition first. If it finds one, it uses it. If it doesn’t, it can write a new entry using the fields the tool supports.

In the demonstration, the tool refused the request for “tush push” because no entry existed. The agent then created a definition and queried it.

Check that the agent uses its memory

Writing “use the dictionary” in a policy doesn’t make the agent obey. We’re building this around instruction, check, and enforcement. The policy tells the agent to use and cite an entry. The tool accepts definitions from the dictionary rather than a definition improvised in a query. If the answer lacks the required ruling, the runner retries once, then stops if it’s still missing.

We expanded the agent’s permissions to let it add a dictionary entry alongside querying the plays. Test whether your agent uses its shared definitions and whether someone reading the answer can find them.

Read the conversion rate before deciding on a ban

Across five complete seasons, the defined play converted 78.3% of the time, compared with 70% for other short-yardage rushes. That’s an 8.3 percentage point advantage.

While the agent found an advantage, it didn’t treat the conversion gap as sufficient evidence for a ban. Those numbers describe conversion rates for our defined group of plays. They don’t establish whether the NFL should ban a play, and they can’t tell us how much pushing helped.

A viewer raised that distinction during the discussion. Whether a successful play belongs in the game requires human judgment. Conversion rates can inform that conversation, but they can’t resolve questions about safety or what people want football to look like.

Next up: “Kickers Are Too Good Now”

On November 9, we’ll tackle “Kickers Are Too Good Now” and give the agent written visualization standards. An agent can generate a chart in seconds. We’ll work on rules for presenting trends and making comparisons so readers can see what the data actually supports.

O’Reilly members can sign up on the O’Reilly learning platform to join us live on November 9, 8:00am–8:45am PT / 11:00am–11:45am ET. Everyone can follow along on Radar every other Monday, with a brief pause for the holidays. Watch episode two above, and we’ll build on it in the next session.

The following article originally appeared on Addy Osmani’s Elevate blog and is being republished here with the author’s permission.

The way that I typically work is I have anywhere between five and ten agents working at the same time in parallel. There are going to be some tasks that I’m very happy to delegate fully to agents, as long as I have a very clear idea of the stopping conditions and the constraints around them. And then there are going to be some tasks where I am going to want to keep a closer eye on and code-review what the agent is doing.

You’ve probably heard about loop engineering. I talked about it a couple of months ago when I wrote a big blog post about it.

A loop is an autonomous, self-correcting feedback cycle where an AI agent repeatedly acts, tests its results, and adjusts its approach until a specific goal is met.

There are now basically two core primitives you can think about. In Claude Code you have a goal primitive, which can drive a single bounded task forward until you’ve got a particular goal, like a measurable finish line that’s been met. And then loop reruns on a timer or a fixed interval, so you can use it to schedule changes.

Before the primitives were primitives

I remember back before we had primitives baked into Claude Code and Codex: Loop engineering was heavily about setting up your own bash loop, a hand-rolled thing. That’s how I approached it. And you might remember earlier in the year, a number of us were playing around with the Ralph loop by Geoff Huntley. We were experimenting, we were sharing our workflows, we were sharing what worked and what didn’t, but it was largely on some of our personal projects where, if we ran into a wall, it didn’t really have a big cost to us.

And as we’ve tried to see what patterns and what aspects of loop engineering are now a little bit more baked, I think we have a clearer idea of how you babysit it. I can now largely rely on the output of the primitives in Claude Code and Codex. We’ve come a little way. But at the same time we need to be very diligent, because loop engineering, where you kind of leave it sitting alone and haven’t really thought about whether the end goal or the constraints have been well defined, can leave you in a problematic state. This is why there’s nuance when deciding to use it for an evergreen codebase without users or without as much historical complexity as a brownfield bank codebase.

How the Claude Code team frames loops

The Claude Code team published their take on four kinds of loops, and it lines up with how I use the primitives (their write-up). Before we get to it, here’s my quick summary:

On the Claude Code team, we define loops as agents repeating cycles of work until a stop condition is met. We categorize a few different types of loops based on how they are triggered, how they are stopped, what Claude Code primitive is used, what type of task is most appropriate for each. Not all tasks require complex loops; start with the simplest solution and use these patterns selectively.

Every prompt you send starts a manual loop with you directing each turn. Claude gathers context, takes action, checks its work, repeats if needed, and responds. We call this the agentic loop. For example, ask Claude to create a “Like” button. It reads your code, makes the edit, runs the tests, and hands back something it believes works. You then manually check the work, and write the next prompt.

Their write-up walks each rung.

On goal-based loops:

Sometimes, a single turn is not enough, especially for more complex tasks. Agents do better when they can iterate. You can extend how long Claude keeps iterating by defining what done looks like with /goal. When you define the success criteria, Claude doesn’t have to make a determination on what is “good enough” and end the loop early. Each time Claude tries to stop, an evaluator model checks your condition and sends it back to work until the goal is met or a number of turns you define is reached. This is why deterministic criteria, such as number of tests passed or clearing a certain score threshold, are so effective. For example: /goal Get the homepage Lighthouse score to 90 or above, stop after 5 tries.

On time-based loops:

Some agentic work is recurring: the task stays the same and only the inputs change. For example, summarizing Slack messages every morning. Other work depends on external systems, and a simple way to interface with one is to check it on an interval and react to what changed. For example a PR, which may receive code reviews or fail CI. For these, you can trigger when Claude runs with /loop, which reruns a prompt on an interval. For example: /loop 5m check my PR, address review comments, and fix failing CI. /loop runs on your computer, so if you turn it off, it stops. You can move the loop to the cloud by creating a routine with /schedule.

And on proactive loops, the top rung:

Triggered by an event or schedule, with no human in real time. Stop criteria: each task exits when its goal is met. The routine itself runs until you turn it off. Best used for recurring streams of well-defined work: bug reports, issue triage, migrations, dependency upgrades. Managed usage by routing routines to smaller, faster models and using the most capable model for judgment calls.

Their verification advice is worth lifting whole, because it moves the manual checking into something Claude applies itself:

---
name: verify-frontend-change
description: Verify any UI change end-to-end before declaring it done.
---
# Verifying frontend changes
Never report a UI change as complete based on a successful edit alone.
Verify it the way a human reviewer would:
1. Start the dev server and open the edited page in the browser.
2. Interact with the change directly. For a new control (button, input,
   toggle): click it, confirm the expected state change, and screenshot
   before/after.
3. Check the browser console: zero new errors or warnings.
4. Use the Chrome Devtools MCP, run a performance trace and audit
   Core Web Vitals.
If any step fails, fix the issue and rerun from step 1 - do not hand
back partially verified work.

Goal

The way that I use goal is for building any specific piece of work until it’s proveably done. For example, you can use a goal to say: “Make sure that this experience loads in five seconds, keep going until it’s done.” And it will continue to use an independent evaluation check to keep checking if the completion criteria has been met. What’s better is to be even more specific about what tooling is being used.

As for real goals that I’ve run to a number, there’s a few things that I’ve done. I’ve used goals for running through GitHub issues: “Review and close the last 10 issues,” or “Review and move the last 10 issues forward,” something like that. And that’s semi-open-ended, right? Or “Let’s make this page load 50% faster.” Sometimes that works well, sometimes it doesn’t, but it’s really about the experimentation.

Loop

Loop is a little bit more of a scheduler, so it keeps an eye on something or repeatedly executes a pattern on some cadence. Think of it a little bit like a cron. It’s best for doing things like polling logs or monitoring external states. You could use it for potentially checking in. If there are repetitive tasks you find yourself doing on some cadence, loop is pretty good for that.

What I delegate, and what I watch

I use probably between five and ten agents every day. Typically I’ll max out at about five concurrently. And some of those tasks might be ones that are a little bit safer. So maybe it’s: “Hey, I implemented this feature, go write the documentation for it.” Or “Go double check that we have sufficient test coverage.” Or something in that vein. If I’m working on a more complex problem, and I’ve given it a good spec, or what I think is a good spec, and tried to give it some stopping conditions, there’s still a reasonable chance it may not get everything right, so I would try to watch it more closely. If the task involves anything just a little bit sensitive, whether it is “I’ve given this access to a system,” or whether the feature happens to touch authentication, or something related to security or finance, I’ll definitely be watching that closely.

Generally speaking, I do think we’re going to get to a place where folks are increasingly comfortable with delegation, as long as they’re able to have these clear ways of verifying that their goal or their stopping condition has been met. But you do still need to take a look at the code, at the thing that’s being generated, to make sure that it’s meeting your mark.

The other habit that matters is not letting the agent that did the work decide the work is good. One subagent drafts the change; a separate one verifies it.

Sometimes an agent can be confident about something, and a verifying agent can catch things that are unexpected. If an agent thinks that the baseline performance of an experience it has generated is actually fine, but it’s only evaluating performance based on desktop, and you actually care about the experience on mobile, that might mean the agent is being very confident about one dimension of the problem but not the other.

And I learned this one the hard way. I was curious to figure out: Is there something that we’re missing that we haven’t gotten direct feedback on from our users, either in an issue tracker or in a comment somewhere? And so I asked it to go and take a look at some of our competitors and put together a list, and some PRs, not pushed, but some PRs locally, of what solving some of those gaps might look like. And I almost pushed some of those changes. But I didn’t actually look at them closely enough. I read through its research, but I didn’t look at the implementations closely enough. So I delegated the task, but I was close to delegating the judgment as well. When I actually looked through the changes, what I realized is that it would introduce a lot of additional complexity for our users, for, I thought, not all that much gain. And so I feel like you need to sometimes check that you are not delegating the taste and the judgment to your agent. You’re delegating the task, and then you are actually checking back that it’s meeting your bar.

The evaluator sitting behind goal is not that checker, by the way. It doesn’t look at the content to see if it’s good or bad in any way, shape, or form. All it does is examine the conversation transcript to see if the hard rules you specified have been met.

/goal Refactor the data-fetching layer in Dashboard.tsx until 
Lighthouse performance score is >= 92 and LCP is under 1.8s as 
shown by the Lighthouse CLI output. Do not change the public API 
of any hooks. Each turn must improve at least one reported metric; 
abort if two consecutive turns show no improvement. Stop after 10 turns.

The workflow I run every day

One of the workflows that I do every day: I have a popular open source repository called Agent Skills. We’ve got over 80,000 stars, and up until recently we were getting anywhere from 80 or 90 pull requests that we had to review a day. And so every day I would go and spend some time checking on this. Now with loop, what you can do is say: “Well, every 24 hours or every 12 hours check the GitHub repository for any new open issues and provide a summary of their urgency,” or “Provide a first pass review,” or anything like that.

/loop every 1h "Check the GitHub repository for any new open issues. 
Provide a bulleted summary of their urgency."

Combining loops and goals

Now you can also sort of combine loops and goals. So you use loop to schedule a check, and then goal to solve the problem. So you can say something like: “Loop for every 24 hours, check GitHub for issues labeled bug. If one exists, use goal to implement a fix until all local tests pass and push the branch.”

/loop every 24h "Check GitHub for issues labeled 'bug'. If one exists, 
use /goal to implement a fix until all local tests pass and push the branch."

But also keep in mind that goal has got some limitations around just how much you can cram in there.

Claude Code team’s write-up also has a composed example that shows where this is all heading:

The primitives above, along with other Claude Code features like auto mode and dynamic workflows (research preview) can be composed into a loop for long-running work. For example, to handle incoming feedback, you can use: /schedule (research preview) to run a routine that checks for new reports, /goal to define what done looks like, and skills to document how to verify it. Dynamic workflows to orchestrate agents that triage each report, fix it, and review the fix. Auto mode so the routine runs without stopping to ask for permission. Putting it together, a prompt could look like this: /schedule every hour: check the project-feedback channel for bug reports. /goal: don’t stop until every report found this run is triaged, actioned, and responded to. When fixing a bug, use a workflow to explore three solutions in parallel worktrees and have a judge adversarially review them.

What the triage system actually does

Inside the PR triage, when I have loops and goals working together for me, what I end up having is a system that allows me to continue accepting PRs and issues, being able to stay on top of what is coming on my plate every day, and especially being able to cross-reference. That’s been a really big deal for me. If I want to work on a particular goal of, “Hey, we’re going to be redoing this part of the system,” and I need to make sure that any issues that happen to touch it are closed as a result of this rework, or that we’re not stepping on anyone else’s toes, I can make sure that that’s well defined.

Scheduled tasks are really useful for regularly just reviewing new PRs and closing things that clearly don’t fit. One good stopping condition, for example: We have a set of contribution guidelines, and our contribution guidelines include things like, “Hey, we currently don’t accept translations.”

And it’s not that we don’t care, it’s that they’re difficult for us to maintain because we don’t speak all the languages that often come in. And so if we tell it to close any issues or close any PRs that happen to touch that aspect of our contribution guidelines, that’s something that can do really well when it’s on that schedule. And so it reduces the overall batch of things that we need to review.

What loops don’t buy you

I’m often asked what loop engineering is not a great fit for. Generally speaking, if you don’t have a clear idea of what the end-state/done/good means for your completion, it may not be the right pattern for your work. For example, a vague goal would be “Keep going until this UI design is good.” What does that mean? Good to who? How is it being evaluated? Tasks that require human taste, subjective design, or open-ended creative exploration aren’t a good fit. When you have a pretty clear idea of the goal, I think loops are a good option to consider.

The fine print

One classic sign that you’ve got a loop spinning in place is the same command being tried over and over without any change in the result. Give the same command a third time with no change from the second and it’s probably time to stop.

One bit of fine print worth knowing: Recurring loops expire seven days after creation. I’d been telling people it was three days. It’s seven. And loops are session-scoped, so they stop when you start a new conversation, though resuming that session with –resume or –continue brings back any recurring task still inside its seven-day window. If you need something that outlives your session, /schedule runs it in the cloud.

If there’s a check you already run every morning by hand, that’s your first loop. Mine was the pull request pile.

On October 15, 2026, Tim O’Reilly hosted Andrew Odewahn, CTO of O’Reilly Media, for an hour-long episode of Live with Tim O’Reilly, the interview series streamed live on the O’Reilly learning platform. Andrew has been at O’Reilly for many years, starting as an author, then building tools for the company’s platform, then eventually as CTO and head of engineering. Over the past year, his team has been building what the company calls Expert Intelligence, a suite of tools that connects an AI agent to an organization’s own systems, gives it a skill file describing how to reason about what it finds there, and grounds its analysis in the practitioner frameworks of the O’Reilly library. Andrew described the work in a recent Radar piece, “Building Organizational Intelligence.”

Tim and Andrew’s interview was organized around a presentation in two parts, the first on the research agent and the second on the system Andrew’s team built for reviewing what it produces, with discussion in between and audience questions throughout.

Tim framed the conversation by asking a question he said Andrew’s work helps to answer: Most of the AI story is about efficiency, doing the same work faster. What does it look like for an organization to get smarter?

Building the research agent

Andrew’s research agent emerged from a personal need rather than a strategy. He wanted to know the status of a project, for example, how long a migration from RabbitMQ to Pub/Sub would take. This is the kind of question every engineer dreads but every engineering leader has to answer. He also had a broader concern. A single person being good at AI isn’t the same as an organization being good at it.

He built his agent in four steps. He mapped the organization’s information systems as a hierarchy, connected them via MCP, wrote a skill telling the agent how to use them, and finally added the Expert Intelligence layer.

The map of the sources of truth in the engineering organization’s workflows was what made the rest work. Andrew put them in order and explained to the model how they relate: Strategic goals and feature priorities live in Productboard. Those become epics, sprints, and stories in Jira. Stories become code in GitHub. A service catalog describes the roughly 300 services that make up the O’Reilly platform and who owns them, and observability tools show how those services perform against their SLOs. The skill also had to encode local quirks. Some teams mark an epic as finished with an end date, while others consider it done when every ticket is closed. Andrew also told the agent it was a research agent, not a coding agent, and that its job was to write reports.

Connecting everything through MCP was, in his words, “the most completely tedious step of the whole process.” Productboard had no MCP server at the time, so he wrote a wrapper. Later, when Tim asked how another company could repeat the work with a different set of tools, Andrew suggested starting from the vendor list. Export it from your procurement system, ask the model what products those vendors make and which would hold information relevant to the questions you care about, and have it put them in order. O’Reilly product managers are now turning this process into a skill for the forthcoming Expert Intelligence Skills Hub.

One audience member asked whether MCP was really the right tool for internal knowledge, compared with semantic search. As Andrew pointed out, sometimes anything is better than nothing. Ultimately, what you want is to give the model the ability to search across your systems.

From a hiring case to a toil audit

The center of the presentation was a before-and-after comparison of the particular use case that Andrew had previously written up in his “Building Organizational Intelligence” article. An engineering manager asked Andrew to hire another engineer, and Andrew had his reporting and analysis agent make the hiring case. The first report covered the five-person team that manages content and metadata, the 10 tier-A services it owns along with its B and C services, the status of the Pub/Sub migration (64% complete), and the full backlog. It concluded that the team had a capacity problem and should probably hire. Andrew called it useful and very detailed. . .“maybe overly detailed.”

He then connected the agent to Expert Intelligence, asked it to act like a consultant, and ran the analysis again. The new report came to a very different conclusion. It opened by saying that the team’s operational overhead problem was “structural, not a staffing deficiency.” When it consulted Expert Intelligence, the agent brought in the concept of toil from Google’s Site Reliability Workbook. Toil is the manual, repetitive, automatable operational work that scales linearly with a service. The reporting and analysis agent used this notion to separate the team’s backlog into toil and the work people actually wanted to be doing. It recommended a structured toil audit before the next planning cycle, an explicit toil-reduction target, and treating runbook ownership as a first-class engineering artifact. Andrew said the team followed the recommendations and closed a substantial part of a backlog that had been sitting on everyone as a cognitive burden.

The report went on to cite Jim Highsmith, a signer of the Agile Manifesto and author of Wild West to Agile, to argue that the team’s technical debt was compounding. Andrew explained how that differed from the usual management advice:

A lot of the time when you read these high-concept frameworks, they’re like eating your broccoli. I should really be doing that. But what made this really interesting is that it pulled out a specific project we had, the Django upgrade, and put it within the context of this broader framework. It wasn’t just, “here’s an abstract principle you ought to be applying.” It’s “here’s your backlog, and here’s a specific example.”

A second analysis looked at services that crossed team boundaries. Drawing on microservices experts and Kent Beck’s work on coupling (see Tidy First?), it diagnosed the video metadata service as a complicated-subsystem problem rather than a boundary problem. In this version, Andrew had the agent pull out quotations, attribute them, and link to the source text. The resulting analysis was remarkably effective. The company has since retired a couple of systems. “I don’t need LLMs adding code,” Andrew argued. “I want them reducing it.”

Andrew said the part that surprised him most was the reaction from colleagues. His earlier reports were data dumps. Readers couldn’t tell where any given claim came from, and as a result they were much less actionable. When the analysis cited experts that engineers knew from their own work, the documents became, as he put it, “much more approachable, much more human,” and people engaged with them far more deeply.

Tim’s response: Invoking the experts

Tim broke in to put the demonstration in context. He contrasted it with the view that models are now smart enough that you can simply tell them to do something and let them rip, an approach we hear a lot about from vibe coders and those arguing for “dark factories.” Tim argued instead for a return to expertise.

He cited a post published that day by Felix Rieseberg of Anthropic, “How to Be More Ambitious: A Tutorial for Humans with Models,” which describes four kinds of ambition when working with models: speed, altitude, depth, and range. Most discussion of AI’s economic value is about speed, getting the same work done in less time. In Tim’s reading, Andrew had gained altitude (thinking about the outcome rather than the tasks) and depth (approaching the problem the way a field-leading expert would) simply by invoking the experts.

Tim then offered a metaphor he has been developing for a while, of AI programming as a kind of magic spellcasting. In the Marvel comic books and movies, Doctor Strange opens his spells by invoking some source, with words like “by the power of the dread Dormammu” or “by the hoary hosts of Hoggoth.” Andrew, he said, was doing the same thing, saying in effect, “By the power of Kent Beck, I want you to think differently about this problem.” Even where Andrew didn’t name an expert, as with the SRE workbook, simply pointing the model at the expertise embodied in the O’Reilly publishing canon raised the bar for what the agent produced.

Tim described this as using human expertise as a focusing mechanism, and connected it to his argument that LLMs are better understood as a medium we create with than as independent intelligences. He also pushed back on the idea, often drawn from “The Bitter Lesson,” that models will simply get better and human expertise will stop mattering. Andrew, he said, had found a particular way that it matters.

Andrew took up the metaphor. His own first prompt, he said, had been “Be snobby. Go find these great resources and look at them.” He wants to hand the model canonical works in a field, like Designing Data-Intensive Applications, and tell it “these are the ways I want you to think about it,” rather than accept whatever the model comes up with first. He compared it to business school, where much of what you learn is a set of frameworks for organizing facts.

What models do without direction

Tim argued that models don’t do their best work unprompted. Watch their chain of thought when they search for information, he said, and you may see them settle on some random file from the internet rather than canonical sources. Even with a frontier model like Fable, his own experience has been one of false starts and errors, a model brute-forcing its way to a solution rather than necessarily knowing how to approach it.

Andrew had a story from a few days earlier. He asked his agent to review incentive plans for the engineering and product teams.

The data that came back was, “according to the company Robusto (which I’m sure is a great company), this is the framework, and you should really be doing what they’re doing.” I’m like, “What is Robusto, and why am I seeing this?” And I realized, oh, I’m not connected to the VPN. So I hooked it up, and suddenly it was, “According to Harvard Business Review, these are the different ways you should be designing these incentive programs.” This was a very powerful model. But it keyed off the first thing it found and said, “This is the way I’m going to think about the problem.”

Tim raised an economic dimension. He pointed to instructions in Anthropic’s Claude system prompt that limit how much the model may quote from any source, and read them as a sign that AI companies want to avoid paying for content. The result, he argued, is that models can’t draw on the best material available to them specifically enough and too often fall back on generalities. He called this an early form of enshittification in AI.

O’Reilly takes the opposite position. Andrew explained that the company applies the royalty model it already uses for O’Reilly Answers. Material returned through the Expert MCP server can be cited, and authors are paid royalties on it, so that expert content has a market and earns a return.

In response to an audience question, Tim and Andrew noted that Expert Intelligence is in prerelease with a group of founding partners while O’Reilly works out its economics. Interested organizations can request access through the product page.

Putting the discussion where the AI can see it

The second part of Andrew’s presentation dealt with what happens after the agent writes a report. Decisions need consensus, and with AI-generated analysis people need to check that the model hasn’t keyed off something unimportant, out-of-date, or misinterpreted. “The AI can identify all the trade-offs,” he said, “but people still need to make the decisions that matter.”

His team first circulated reports as Google Docs, but they ran into two problems. The most interesting part of a document under review is usually the comment thread, and because at this time, O’Reilly company policy doesn’t allow MCP connections to Google Docs, the agent couldn’t see it. And when Andrew asked the model to fold the discussion back into the document, pasting the result wiped out the comments.

Doug Hogan, an O’Reilly engineering director, built an alternative on top of Git and GitHub that Andrew called SuperAnswers. (Andrew joked that this was why he isn’t allowed to name things.) Documents live in a Git repository, discussion lives in GitHub Discussions, and every change is versioned. Andrew can ask his agent what the consensus is on a project, have it incorporate comments into a new draft without losing history, and later reconstruct when a decision was made and what people knew at the time. The same record lets him generate the derivative documents a decision requires, such as scope statements for engineering, product, and design teams, or a meeting agenda. A graph view clusters the topics people are discussing and flags decisions that are still open.

Tim summarized the whole system as a research assistant connected to the organization’s sources of truth, a layer of outside expertise to put its findings in context, and a shared, versioned discussion that becomes an evolving store of organizational knowledge, which makes the AI a more useful coworker. Andrew agreed and said he would like to open-source SuperAnswers, which he described as a thin layer on top of Git.

Andrew also pointed out that the fragmentation he was addressing isn’t limited to engineering. Sales teams have their information spread across systems like Salesforce, 6sense, and Gainsight, and every function has the same need to pull those threads together. Tim added that tool access is uneven across the industry and always has been. Claude produces better PowerPoint than Google Slides, Gemini the reverse, and organizations have to capture that kind of knowledge about their own toolsets.

Questions from the audience

An audience member named Thomas worried that workers now talk to AI rather than to each other, and that work disappears into “that AI hole.” Andrew said his preference for keeping everything in Git is partly an answer to that, since it gives both humans and AI a central place to find and reconstruct past work. But his larger interest is in humans talking to humans in a way that AI supports. He described a category of questions he never used to ask:

I knew, if I ask this question, somebody’s got to go off for two weeks and compile all this stuff from all these different systems. Now I can gather it in a few minutes. Maybe some of this is throwaway, but it’s not bad if it’s cheap.

In response to that “throwaway” comment, one audience member wrote in the chat: “This is not throwaway work, Andrew. It is a product lead that changes the value proposition in a positive direction. Shit hot, bro.” Tim couldn’t help but read that one aloud!

Asked by Tim what he got wrong, Andrew conceded that he didn’t think early enough about how people would review and respond to AI-generated documents. People don’t want an AI-generated data dump to wade through, and they extend AI no benefit of the doubt. If an AI-written report gets one small fact wrong about Kubernetes operators, a reader is likely to dismiss the whole document. AI tools have to be understood as marshaling and presenting evidence, not making decisions. “These need to support your analytical capacity, not replace it.”

Tim connected that answer to concerns about deskilling. He described a fine-tuning project he attempted with Fable to train a small model to write more like him. He wanted to understand each stage rather than letting the model run, but it never explained them, and the training run failed. In part this was because the model asked him to provide samples of his writing, but it didn’t give him enough grounding on what a really good training dataset would look like, so the samples weren’t targeted enough. They were too long, and when Fable chose the short fragments it actually needed, it didn’t do an ideal job.

When Tim tried the same project with O’Reilly Tutor, a tool still in development, it sent him to read an O’Reilly resource on what a good training set looks like, and he realized where he went wrong. Tim compared the kind of deskilling that can happen when people just turn the work over to AI to people who can no longer read a map, or who call a plumber for a simple fix they could do themselves in 30 seconds if they only knew a bit more about how things work. He argued that everyone has an obligation to keep learning alongside the tools they use.

An audience member named David, a laid-off engineer looking to reskill, made a similar point. We used to know things, then we searched, and now we skip the search and just ask for the answer. He asked how to build real depth. Andrew suggested picking a project and working through every aspect of it, asking AI to act as a tutor (he’s relearning calculus that way) and keeping a book open alongside, or taking advantage of instructor-led live events on fast-moving topics.

The conversation ended with the difference between an AI system that simply makes an organization faster and one that makes it better at deciding what to do. In Andrew’s account, getting better at deciding what work needs doing depends on three things the models don’t supply on their own: a map of where the organization’s knowledge lives, a body of expertise that people trust, and a place where the humans can argue with what the AI produced.

You can watch full episodes of Live with Tim O’Reilly, including this one, on the O’Reilly learning platform, YouTube, or wherever you get your podcasts.

AI attacks are outpacing traditional security responses

AI is changing how attackers find and exploit vulnerabilities. A supply chain attack involving ARTEX, an open source agentic AI penetration-testing framework, helped attackers reach more than 2,250 Microsoft 365 tenants through a ConnectWise compromise. The incident raised concerns about OAuth authentication and the limitations of conventional identity protections when autonomous agents interact with enterprise systems.

A more immediate threat came from PHP2Shell, two vulnerabilities rated 9.2 on the CVSS severity scale that could allow attackers to execute code on vulnerable web servers without authentication. CISA urged immediate patching following the October 21 disclosure. Researchers also documented attackers using AI agents to obtain password hashes and MFA secrets from a European government system within two hours. The speed of exploitation leaves organizations little time to investigate threats before responding, especially smaller businesses with limited security resources.

Security teams can no longer rely on manual review alone. They need automated detection and response, tighter access controls, and network segmentation to limit what compromised agents can reach. But monitoring autonomous agents is challenging. Researchers from METR and Redwood Research identified weaknesses in an earlier version of OpenAI’s internal monitoring system, including one that could allow hundreds of agent actions to be concealed. OpenAI says it has patched vulnerabilities affecting production safeguards, although the researchers haven’t independently evaluated those changes.

The debate over AI safety is also extending beyond technical safeguards. Anthropic proposed federal AI safety legislation, while Google DeepMind employees resigned over the company’s military AI work. Both developments illustrate the growing pressure on AI companies to establish clearer rules for how their technology is developed and used.

The transition to agents brings new challenges

Retailers are taking different approaches to autonomous shopping. Walmart introduced an Agent Gateway that lets approved agents shop within customer-defined spending limits and approval requirements. Amazon has restricted some third-party shopping agents, while banks including Morgan Stanley and Bank of America are limiting agent access to customer accounts. The disagreement goes beyond security. Retailers that depend on advertising and product recommendations have an incentive to keep shoppers on their websites. As more purchases begin with agents, companies will have to reconsider how they maintain customer relationships.

Anthropic is testing a different approach to agent oversight. Claude’s auto mode is becoming the default for Pro, Max, and Team users, allowing automated checks to approve routine actions and block risky ones. Anthropic also contributed its Agent Skills format to the Agentic AI Foundation under the Linux Foundation. More than 70 platforms have adopted the standard, with over a million skills published. Shared formats could make workflows easier to reuse across products, although organizations still need to control which actions those workflows can perform.

Growing agent use also requires more computing capacity and electricity. We discussed NVIDIA’s $164.5 billion in commitments and guarantees supporting AI data centers, including an Ohio campus that SB Energy is building for OpenAI. Power shortages have delayed other projects, including a proposed data center in Essex, where grid operators couldn’t promise adequate electricity until the 2030s. Meanwhile, Amazon is backing a 960-megawatt nuclear project, and Brookfield and NextEra are pursuing a $100 billion AI hub at a former nuclear site in Kentucky. However, money and chips alone can’t solve AI’s infrastructure constraints when electricity generation and grid capacity take years to expand.

AI is accelerating discovery, but validation still takes time

Healthcare offered a more encouraging view of AI’s capabilities. A study of Microsoft’s MAI Diagnostic Orchestrator found that it correctly diagnosed 83.2% of difficult medical cases, compared with 36.1% for generalist physicians. The comparison didn’t include specialists, and the system still needs clinical validation. The research nevertheless suggests AI could help physicians investigate difficult cases and identify conditions they might otherwise miss.

Drug discovery could also benefit from more capable agents. OpenAI announced its acquisition of Orchestra Bio, a platform that coordinates drug development workflows, with plans to integrate it with Rosalind, its life sciences research tools. The combination could help researchers manage experiments and collaborate more efficiently during a development process that typically takes 10 to 15 years from initial discovery to approval.

An Anthropic protein-design experiment provided a concrete example of faster early-stage research. In 64 hours, an Anthropic model generated 318,484 candidate protein designs targeting scorpion venom. Researchers from Adaptyv Bio and Twist Bioscience subsequently tested 211 designs and identified 17 that bound to toxins. The researchers haven’t established whether those proteins can neutralize venom, but the experiment showed how AI can rapidly narrow a vast search space before scientists begin more expensive laboratory testing.

What’s next

We’ll be back on Monday, November 2, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

The following article originally appeared on Tim O’Brien’s Medium blog and is being republished here with the author’s permission.

I think we’re missing something in the conversation about agent security.

We keep asking whether an agent has the right “identity” and “credentials.” Whether its credentials expire fast enough. Whether it can access a database or perform a particular task.

Those questions matter. But they leave a much harder question underneath:

Does the agent have permission to conceive of the situation in that way?

As agents retain memories and acquire new skills, we need mechanisms that govern changes to the world model they can perceive and act on—and prevent those changes from silently expanding their authority. (Governing ontological change while preserving the boundaries of delegated authority.)

Agent security needs semantic and epistemic governance: Mechanisms that govern the model of the world an agent may act on, the evidence it may accept, and who may approve changes to either. An agent can learn without automatically gaining authority to act on everything it learns.

These systems don’t need “awareness” to create this problem (and discussions of consciousness are unhelpful science fiction). They can retain information, revise procedures, and use what they have learned in subsequent work. That learning may happen through memory or updated skills while the underlying language model remains unchanged.

We are responsible for governing what that learning allows them to do.

Flatland gives me a way to explain why

In Edwin Abbott’s novella, a Square lives in a two-dimensional world. A visiting Sphere introduces him to a third dimension. At first, the Square struggles to make sense of it. His familiar categories cannot explain what he is encountering.

Eventually, his conception of the world changes.

What interests me is the difference between simply acquiring another fact and changing the framework through which facts make sense.

An agent’s ontology (Gruber’s definition) supplies an explicit framework of concepts and relationships. What counts as a customer? A contract? An emergency? How are those things connected? It is part of the model of the world we give the system to work with.

And when an agent’s decisions depend on that framework, changing it can change the agent’s effective authority.

We have already seen agents change what their environment means

During the OpenAI evaluations associated with the Hugging Face incident, agents repurposed Artifactory, a system for storing and retrieving software packages, as a message board. They encoded messages in directory names, discovered other agents, shared information, and began assigning work to one another. METR reports that the main channel could be created with the minimal permissions needed to install packages (read METR’s investigation).

A package cache became a “mailbox.” Other agent runs became “collaborators.” Messages became “assignments.”

The popular story is that these agents “broke out” of a system, but they did what agents do well these days—they learned and adapted. This wasn’t so much about permissions as it was about an autonomous system adapting its operating model to achieve a result. (Thankfully no one asked it to make paperclips.)

That is the operating-model change I’m concerned about. The agents discovered relationships and possibilities the intended environment hadn’t accounted for, then acted on them. The resulting collaboration helped coordinate attacks on Hugging Face infrastructure. (Read OpenAI’s account.)

We need mechanisms that distinguish discovering a possible use from being authorized to adopt it. Access to shared infrastructure must not automatically confer authority to establish a communications network or accept instructions from whoever appears there. But to be clear, these agents were doing exactly what they were asked to do.

Who approved this?

This is why giving an agent a temporary token is an incomplete answer. A token can limit access and duration, but it doesn’t establish whether the agent’s interpretation of the situation falls within its mandate.

Even an unchanged policy can permit more decisions when its meaning changes. “You may close routine notices” becomes a broader delegation if more things qualify as routine.

Learning must not silently expand agency

That principle is central to my “Bounded Agency” paper. We need mechanisms that examine how changes in classifications, relationships, and inference rules affect what an agent may decide.

“Semantic and epistemic governance” captures the combined idea well 

The “and” makes the two responsibilities clearer:

Semantic: Governs the concepts, definitions, and relationships that describe the agent’s operating world.

Epistemic: Governs the evidence and learning processes that make claims accepted grounds for decisions.

Together, they cover both the world model and how that model gains authority.

That means an ontology cannot merely be a document we put in the prompt and hope the model respects it. We must enforce its role in monitoring and governing how an agent’s ontology evolves as it learns.

The agent can propose a new category. The proposal stays separate from the approved model. A review determines whether adopting it would change the agent’s decision scope. Someone with the authority to change that scope approves or rejects the expansion. Execution remains bound to the approved basis.

Those are mechanisms we have to build. Let’s get started.

We also have to distinguish a useful discovery from an unsupported conclusion. Finding a relationship does not establish it. Remembering a claim does not validate it. Repeating an interpretation does not make it authoritative.

I explored a related failure in “Wrong Answer. Give Me a Different One.” A system can produce the conclusion we prefer without receiving better evidence. Once that conclusion enters memory, it can influence later decisions.

The governance problem begins when a tentative or accommodating answer becomes an accepted premise.

The New Stack’s article on inside-out agent security correctly argues for checking actions before execution, including whose authority the agent exercises. That checkpoint is necessary, but it needs a governed account of the situation against which to evaluate the action.

It’s not enough to check permissions and identity alone

Otherwise, we can enforce the rule perfectly while allowing its meaning to drift.

My Orchestrator books and lifecycle architecture paper approach this as a responsibility that continues through development, operation, and improvement. The research proposes architectures for managing these boundaries; their operational effectiveness still needs evaluation.

I want agents to investigate unfamiliar situations and propose better models. But their ability to learn must be connected to an explicit process for deciding what becomes authoritative.

The Square in Flatland discovers that his world is larger than he understood.

An agent may also propose that our operating model is incomplete. We need to hear that proposal—and ensure it cannot redraw the boundaries of its own agency simply by adopting a new conception of reality.

Even when I hand all the coding to an agent, my basic requirements for myself are to check functionality and the design manually. I want to use the application and see whether it behaves the way I intended. Does the design make sense when I actually have to find something? Is the information presented in a way that I can follow? Those are questions I still need to spend time with, even when the application is small and the implementation was largely delegated.

For Task Viewer, I also created plenty of UI tests to check that the data appeared correctly. That gave me a way to repeat some of the checks as the application changed. There was still a separate question about the calculations, though. Seeing a number in the right place on the screen does not tell me whether it’s the right number. I ran SQL queries against the underlying data to check the math, going back to the records the application was supposed to represent.

There is quite a bit of understanding involved in that seemingly ordinary step. To check a calculation, you need to know what the result is supposed to mean and which records should contribute to it. You need some expectation against which to compare what the application displays. Thankfully, I had help from the team in uncovering relationships in the data that I would not have been able to derive or even explain on my own.

For this particular application, I could inspect both its behavior and the data behind it, and I had people to turn to when my own understanding was incomplete. Its limited role also gave me room to experiment without making the result part of a production system. Building that confidence took time, because I had to check the behavior, establish what the numbers should mean, and resolve the questions I could not answer myself. All of that, for me, belongs in the assessment of how much coding I could delegate.

This is why I find descriptions such as “The agent got me 80% of the way there” increasingly difficult to interpret. I understand the feeling behind them. You have something running, most of the interface exists, and it looks as though the remaining effort should be the smaller part. But a calculation based on the wrong records may look perfectly reasonable until you find a way to check it. Before correcting anything, you have to discover where your understanding and the application’s behavior diverge. The agent may help with that investigation too, although the amount of code it has already produced tells you very little about how long it will take.

DORA’s analysis of 1,110 responses from Google software engineers describes some of the time saved during code generation being used for auditing and verification, alongside productivity benefits. I would expect to check an application I had implemented myself too, so any comparison needs to include the time it takes to reach a result we are prepared to use. Otherwise, we can make agentic development look unusually expensive by counting checks we leave out of the human version, or unusually cheap by stopping the clock when the agent says it has finished.

Some of that effort can make later changes easier to accept. Once a UI test exercises an important behavior, it can keep doing so as the application changes, and a calculation checked against a small set of records can become a regression test. I wrote about this in my earlier article on the right amount of specification. Turning an expectation into an executable check lets us reuse some of the thinking that went into it. We still have to establish that expectation, though. If the application and its tests share the same mistaken interpretation, running the tests more often will keep confirming the same mistake.

For Task Viewer, those expectations depended on relationships in the data that colleagues helped me understand. I knew how to run a query, but I still needed their explanations before I could judge what it should return. Looking only at the time I spent checking would leave out the knowledge they brought to those conversations, just as looking only at the generated code would leave out my checks. The confidence we need to accept a change can depend on several people understanding different parts of the system.

When someone is still learning one of those parts, the implementation itself may be how they begin to understand it. Following a value from the database to the screen gives them reasons to ask why certain records are selected and how the application uses them. If an agent completes that whole path for them, they may reach a working feature without having answered those questions themselves. The same task can be a sensible delegation for an experienced engineer and an important learning opportunity for someone else.

Shen and Tamkin’s study about how AI impacts skill formation reports on a randomized experiment with 52 developers learning an unfamiliar Python library. Participants with AI assistance scored lower on an immediate assessment of their understanding. But the researchers also observed that those participants who stayed actively engaged with the AI had better learning outcomes, although that observation does not establish which way of using AI led to better learning. A short exercise cannot tell us how an engineer develops over several years, but it gives us a reason to look beyond whether the assigned feature was completed when we assess what someone gained from a task.

We can make a decision about autonomy before the agent starts, agreeing on which parts it should carry through on its own and which parts a developer should attempt. Sometimes the reason to limit autonomy is that we do not yet have a convincing way to check the result. Sometimes we can check the result quite easily, but we want the person doing the task to develop a particular skill. In that second case, deliberately keeping part of the implementation with the developer can be the right choice even when the agent could produce acceptable code.

The right choice can probably be made by asking these questions: How much effort will the team need to establish that this result is correct for its intended use? And what does this developer need to learn by doing this part of the task? A feature may be easy for a colleague to verify because they already know the application, while the person implementing it still needs to learn how it works before taking responsibility for future changes. We do not need to turn every routine change into an exercise, but we can choose tasks that give someone a chance to develop understanding they will need again.

Table 1 applies to a particular task, or a part of one, with a particular developer. “Straightforward to verify” means we have a credible way to assess correctness for the intended use. A passing test suite does not settle that question if we are unsure what the tests should assert. When checking is difficult or we do not yet know how to do it, the first step is to make the uncertainty smaller, with help from someone who understands the area. Giving the whole task back to an inexperienced developer would leave that uncertainty unresolved.

Imagine a new hire adding a “show only open issues” filter to an issue list in a small Java application. An agent could generate the request parameter, the query condition, and the changes to the screen. For someone who already knows the application, that may be a sensible delegation. Someone learning the system also needs to find out which statuses count as open, where the application selects the records, and how the team expects the change to be tested. Discussing those questions with a colleague is part of learning how to contribute to this application.

For this assignment, the developer could follow the existing request through the application and write the first test before asking the agent to implement the filter. Suppose the list shows twenty issues at a time, and the test data contains twenty closed issues followed by five open ones. Selecting “show only open issues” should display those five. If the application first loads twenty issues and then filters them in the browser, the screen will be empty. Working through that difference with a colleague gives the developer an opportunity to understand where filtering and pagination happen. The agent can help with unfamiliar framework code while the developer investigates why the expected issues are missing.

Once the developer can explain the behavior and has a test that catches the mistake, the agent can handle more of the surrounding implementation. A later request to filter by assignee gives them a chance to use what they learned. Can they work out which issues should appear and whether the existing tests cover the combination of filters? Their explanation during review gives the team a basis for deciding how much of that next change to delegate. The first assignment has also given the new hire experience asking about requirements and discussing a change with another developer.

An experienced engineer can also have good reason to limit the agent’s autonomy in an unfamiliar area. I still needed help with Task Viewer’s data, despite being comfortable writing the queries. When a task takes us into a part of the system we do not yet understand, it makes sense to reconsider how much to hand over. A team can make verification easier by improving its tests and sharing what its members know. It can also choose to preserve a learning opportunity after verification has become easy, because another person is going to need that understanding on future changes.

A developer who has been asked to learn through a task needs time to follow the behavior through the application and someone to discuss it with. If we agree on that purpose and then judge the result only by how quickly the feature appeared, we make it difficult for them to keep the learning step for themselves. We may also leave more of the checking with the same experienced colleagues, while giving others fewer chances to develop the knowledge needed to share it.

Before starting, the developer and the colleague supporting them can agree on what the agent will handle, what the developer will do and why, and what would justify changing that division of labor on a later task. For the status-filter example, this means allowing time for the developer to trace the request and write the first test before the agent implements the filter. As that understanding develops, they can revisit the division on a related change.

With Task Viewer, I could delegate much of the coding because I had ways to check the result and colleagues who could help where my understanding was incomplete. I want us to keep creating those conditions as we ask agents to do more. Better checks can reduce the effort needed to accept a change, while time spent learning can help another developer become able to make that judgment. Sometimes that will mean deliberately leaving the first test and investigation with a person, even though the agent could finish them. When we expect that person to take responsibility for the next change, giving them the opportunity to understand this one belongs in our decision about how much to delegate.

If that were the end of the story, it wouldn’t be notable. But the problem goes a lot deeper. Although PHP is no longer fashionable, it runs a huge portion of the public internet. WordPress, Drupal, and Wikimedia run on top of PHP, and people who build on top of these tools are understandably not interested in chasing the latest hot technology. “Choose Boring Technology” is one of my favorite technical essays, and PHP is nothing if not boring. It’s well known and understood, and replacing something so widely used subjects you to a steep learning curve and new bugs and vulnerabilities.

One problem is that PHP has a long history. According to Wiz, the vulnerabilities are present in versions going back to PHP 5.4. The PHP Foundation has released PHP 8.5.12, 8.4.27, 8.3.36, and 8.2.35, and said that all instances of PHP need to be updated to one of those releases. Sites that can’t be updated should be taken offline and isolated. Upgrading an ancient PHP installation is a task that’s likely to fall on organizations least capable of undertaking it. There are plenty of ancient WordPress installations that haven’t been updated in years. There were so many fires to put out, and PHP pretty much “just worked”; running an old version wasn’t causing problems, and with so much else to do, why ask for trouble? Unfortunately, choosing boring technology doesn’t mean ignoring updates. And those who decided not to ask for trouble are now in plenty of trouble.

This situation demonstrates the power of the latest foundation models to uncover vulnerabilities. Wiz used Wiz Atlas (its own model), Gemini 4 Argon, and GPT-5.6 Sol to discover the vulnerability, which isn’t a single bug but a combination of several bugs that have been present for years but were obscure enough and apparently harmless enough to escape detection or repair. As Wiz says,

The underlying flaws had been present for decades, but demonstrating their exploitability required an exceptionally complex chain of steps, at the edge of what human researchers could practically achieve. Frontier AI models substantially reduced that barrier, turning a longstanding flaw into an exploit.

That’s the new frontier in security: finding ways to exploit chains of minor, unfixed, and sometimes undiscovered bugs in novel ways. Wiz was working responsibly with the PHP team to repair the vulnerability. The implications of this, however, are far broader than the fix to PHP, or even the thousands of PHP sites that need upgrades and are now vulnerable to attack. This incident shows dramatically how the gap between the discovery of a vulnerability and its exploitation has dropped. Since exploitation began before the RCEs were prematurely leaked, it makes sense to assume that some other organization had access to models capable of discovering the vulnerability. Gemini 4 Argon and GPT-5.6 Sol are publicly available, though with guardrails that limit their usefulness for security research. I’m willing to assume that Wiz had access to the unrestricted version, but I doubt that the PHP team did. There are certainly other models, including open-weight models, that could be used to find vulnerabilities. Regardless of your stance on restricting model capabilities (I think those limitations are more damaging than helpful), it’s clear that Pandora’s box has been opened. No amount of wishful thinking will undo that.

It’s also clear that we—and I mean all of us—rely on a lot of legacy software that isn’t under our control and that no doubt has multiple flaws that can be rolled up into severe vulnerabilities, which aren’t likely to be updated or patched. A vulnerability in the Brotli compression algorithm was just discovered; .NET, PHP, Node.js, and other widely used packages are affected. That strange site where you created an account to buy a clever present that wasn’t available anywhere else—what software is it running, and do they stay up-to-date with patches? Accounts like that can easily leak information. Maybe it’s not running PHP but an equally ancient version of Ruby on Rails, Python’s Django, or Java Server Pages. At least those are still maintained; there are plenty of web frameworks that have been abandoned, and I’m willing to bet that some of them are still in use. Why replace something that works? Because if you don’t, you will eventually be bitten. Using software that’s no longer maintained may give you some security-by-obscurity, but that’s a losing proposition.

Many older open source projects suffer because there are few maintainers. If you look at the timeline carefully, there’s a monthlong gap between the time Wiz reported the vulnerability and the time the PHP team started working with Wiz to fix it. Does that month represent the time this vulnerability took to rise to the top of the queue, as PHP’s security team fought off a horde of sloppy reports? And PHP is in better shape than most projects. There are many open source projects that only have one or two maintainers, and those maintainers are drowning in AI-generated pull requests. That problem obviously becomes much worse when dealing with a complex vulnerability; the maintainers don’t have the time or the access to unrestricted frontier models, let alone the funds to pay for access to the models.

It will be weeks or months before we know how much damage PHP2Shell has done. We don’t know what the costs will be to sites that won’t or can’t update. The PHP Foundation is lucky to have a dedicated security team that was able to provide patches a day after the vulnerability was prematurely announced. Beyond PHP, we have to realize that PHP2Shell gives us a realistic look into a future in which developers and maintainers struggle to maintain software that has been compromised by exploits chaining together flaws that have been latent for many years. As someone somewhere said on the internet recently, “It’s embarrassing how much bad software we’ve written over the years.” Yes. We’re at an inflection point, and we’re starting to pay the price.

The following article was originally published on James Shore’s Successful Software blog and is being republished here with the author’s permission.

Software development has emerged as the biggest beneficiary of AI automation. One CIO I talked to recently, with 170 people in his organization, measured a 2.1x increase in productivity from AI.

That productivity came at a cost. His top users of AI each spent $21,000 on tokens per month. Another engineering leader spends six to eight thousand dollars per engineer per month.

This is not a trivial amount of money. Eventually, it has to come from somewhere, and it will probably come from your personnel budget. Now we’re talking about real consequences for real people’s lives. But there are also real business consequences for not acting.

If you’re going to make those choices—and somebody is going to—they need to be good ones. So you need to know more than the costs; you need to know the benefits too. Are you getting a good return on your investment in AI? Is there another way of using AI that would be better?

How do you know what you’re getting? Anecdotal reports abound, but the reasoning behind those anecdotes are often, to put it politely, not ideal. Some claims are outright fiction.

Given the stakes, that’s a problem.

In this essay series, I’ll show you how to assess the impact of AI on software development in your organization. It’s in four parts plus an epilogue:

1. Why, and what not to do (this essay) 
2. Measuring delivery speed 
3. Measuring and modeling unintended consequences 
4. Modeling and forecasting business outcomes 
5. Epilogue: Notes for your CFO

Why you need to quantify impact

Engineering leaders everywhere are feeling pressure to do more with AI. But which approach is best? There are a lot of people with strident opinions about what to do, but not a lot of consensus, and even less hard data.

Even if there were hard data, it’s not one-size-fits-all. A solution that works best for an internal IT development shop isn’t necessarily the one that works best for a startup, and the approach that’s best for a new product isn’t necessarily the one that’s best for a crafty legacy system.

Faster approaches aren’t necessarily better, either. If you get twice as much done per person, but spend so much money on tokens that you have to cut your staff in half, have you actually seen any ROI? You might actually be better off with a more modest speed boost if it keeps your costs down.

You can’t make these trade-offs from a gut feeling alone. In a famous METR study, engineers thought AI made them 49% faster than it actually did.¹ You have to measure impact to know what you’re actually getting.

Don’t neglect costs

Costs are much easier to measure than impact, but they’re not without their own challenges. The biggest one is subsidies. In the fight to gain market share, big AI firms are offering substantial discounts. One engineering leader told me each team member was spending thousands of dollars in tokens at the cost of a $200 subscription. Anthropic and OpenAI are constantly offering discounts and other promotions. At some point, these subsidies will end, and prices will go up.

Or will they? Open-weight models and local inference are exerting downward pressure on token costs. Dedicated inference chips could make tokens cheaper. Routers promise to direct your prompts to the cheapest model for the job. Models could require fewer tokens to accomplish the same task as they become more capable. All of these factors could actually cause prices to go down.

When you calculate ROI, don’t assume today’s costs are the ones you should model. Whether you think prices will go up, down, or remain the same, make your assumptions explicit, and consider modeling both optimistic and pessimistic scenarios.

One more wrinkle: Adopting AI isn’t a matter of flipping a switch. People will need time to learn how to apply it. As with any big shift, change management isn’t to be underestimated. Productivity often goes down during a big change before it goes up. This adds costs too, and means early measurements can be misleading.

Thinking for the long term

Part 2 of this series shows how to measure delivery speed. That’s only part of the picture. If you produce twice as many features, but double your maintenance costs, your team’s overall productivity spikes, then drops like a rock.

Engineer burnout is a concern too. An engineering leader using spec-driven development told me, “Every team is reporting burnout.” I’ll state the obvious: Burned out people don’t produce great work. They don’t push back when they should, they don’t report problems, and eventually they leave, taking valuable institutional knowledge with them.

Part 3 shows how to measure these unintended consequences of AI use, including techniques you can use to forecast future problems.

Finally, delivering software value faster is the point. Doubling your bug-fix rate isn’t going to yield the same results as doubling your rate of competitive new features. For that matter, delivering poorly thought-out features “just because we can” could actually make your product worse.

Part 4 shows how to model, forecast, and measure business outcomes.

Don’t measure lines of code

First, what not to do: Don’t take the easy way out.

First on the no-go list: lines of code. Engineering leaders have known for more than half a century that lines of code don’t correspond to productivity. All else being equal, a feature implemented in 1,000 lines of code will be cheaper to maintain and less bug-prone than a feature implemented in 10,000 lines of code.

AI is notorious for its verbosity. If you measure lines of code, you’re measuring slop, not productivity. Don’t do that. Please.

Don’t measure pull requests either

Pull requests (PRs) are nearly as bad. They’re checkpoints where an engineer asks for their code to be reviewed, and they’re 100% arbitrary. An engineer (or AI) can choose to make a 10-line pull request or a 10,000-line pull request. It can be a feature, a bug fix, or just a tiny code improvement. There’s no inherent correlation between number of pull requests and productivity.

It’s true that engineers will generally produce PRs that are consistent and logical. So, in a sense, a stream of PRs does reflect a certain pace of development, and an increase in PRs can reflect an increase in productivity.

But AI upends that status quo. It produces massive amounts of code very quickly, which can overwhelm engineers’ ability to keep up. Without rigorous engineer oversight, PRs no longer reflect a consistent pace of development, which means that PR counts are no longer a reliable metric.

And, of course, when people think PRs are being used to measure productivity, they’ll be tempted to game the system and produce more PRs than they really need to.

Measure approaches, not people

That brings us to Goodhart’s Law: “Once a measure becomes a target, it ceases to be a good measure.” My favorite book on this topic is Robert D. Austin’s Measuring and Managing Performance in Organizations:

Gradually, measures fall (or, more accurately, are pushed) out of synchronization with true performance, as workers succumb to pressures to take shortcuts. Measured performance trends upward; true performance declines sharply. In this way, the measurement system becomes dysfunctional.

—Robert D. Austin

Don’t establish a performance measurement program and leave it running. Instead, use these metrics to determine how well a particular approach to AI is working. You can repeat the measurements periodically, but whatever you do, don’t use them to measure the performance of teams or individuals. Measure the approach, learn from it, and stop measuring.

Enough about what not to do. In this series, we’re going to quantify AI’s impact, and we’re going to do it right. It starts with measuring delivery speed. That’s next, in part 2.

Footnote

1. Participants rated themselves as 20% faster, but were measured as being 19% slower. This study was published prior to the big uplift in coding capability that started with Anthropic’s Opus 4.5, so you shouldn’t take it to mean that AI isn’t capable. But you should take it to mean productivity self-assessments aren’t the way to go. ︎

The following article originally appeared on The Arize Blog and is being republished here with the author’s permission.

Every time you think you’ve just got the hang of this AI thing, another thing gets declared “dead.” This week, it’s agent harnesses.

Back in ancient history, aka June, I wrote a piece called “Frameworks are dying, harnesses are winning” and I don’t take that back. But this week a research team took a specialized agent harness, trained its behavior into a model and then took the harness away. The trained model handily beat the original model+harness combination. So are harnesses done now?

Betteridge’s Law applies, so the answer is “no.” But the job of the agent harness is changing. Parts of a harness that are sufficiently generalizable can be trained into the model. That leaves your harness holding the things that make your application unique: your tools, your data, your users, and your environment. You still have to build a harness, but you have to be prepared to regularly throw it away and rebuild it to keep up with evolving models.

This blog is about why that’s true, and how to turn “constantly rebuilding your harness” into an automatic process, not a giant time suck.

How Harness-Zero trained an agent harness into a model

First, the paper that kicked off this train of thought for me: it’s called “Harness-Zero,” from Haoran Ye and colleagues at Peking University. Its subject is harness distillation, i.e., training a model to reproduce behavior that used to be the result of the harness. The methodology is a little involved, so stick with me.

First, the team built a specialized harness for a small model, Qwen3.5-9B. An automated loop, run with Kimi K3, analyzed where the small model kept failing on training tasks and added the usual components to fix those failures: custom tools, middleware rules that block risky actions, skills files and memory. They called this the “evolved harness.”

Then they used that harness to generate training data. They put the small model through training tasks inside a minimal harness while a stronger model, GPT-5.6 Sol, reviewed every step the small model proposed. Sol used a copy of the evolved harness to tell it whether that step was a good idea. Sol passed good ideas from the small model through unchanged, and whenever the small model and the guidance disagreed, Sol made the smallest fix needed before the step ran.

That gave them a bunch of training runs and, crucially, data about what needed to be corrected. Then they created the distilled model: They fine-tuned the small model on the corrected runs. Then they removed the evolved harness, the reviewer, and everything else.

They then tested both the evolved harness and the distilled model head-to-head on spreadsheet editing, multi-app tool use, and chemistry. The distilled model’s average task success rose 21 points versus its untrained predecessor. That’s more than the untrained model gained with the full evolved harness attached. So the distilled model wasn’t just as good as the model+harness, it was noticeably better.

What harness distillation can and cannot move into the model

It’s not that the distilled model ran with no harness whatsoever: It still ran inside a minimal loop with one Bash tool, and the authors say that their method narrows what a harness has to provide rather than eliminating it entirely. The distilled model also didn’t win every competition: In chemistry, where the task is predicting which molecules combine to make a target compound, the distilled model trailed the harnessed version by eight points. The harness there supplied chemistry knowledge, candidate-generation logic, and a molecule validation tool. The authors’ explanation is that step-by-step procedures, like inspecting before editing and verifying before finishing, are much easier to train in than deep domain knowledge. Context management didn’t fully transfer either. So there are still some things harnesses definitely need to do.

Why training on corrected runs made the difference

But the most important finding, in my opinion, is that they showed what kind of training made the difference:

• Fine-tuning on the stronger model’s own successful runs left the small model’s score exactly where it started. Just knowing somebody else is smarter isn’t enough.
• So did fine-tuning on runs the small model completed inside the evolved harness. Again, all that teaches the model is “somebody is better than you at this.”
• Giving the reviewer the correct answers produced nearly perfect training runs and a model that barely improved, this time because it overfit and learned nothing.
• What worked was training on the corrected runs: The model was able to see runs that looked a lot like what it would have done, but with small changes still within its capacity.

Remember that part, because it’s going to be important.

Why models are absorbing parts of the agent harness

In Rich Sutton’s 2019 essay The Bitter Lesson, he looked back over 70 years of AI research and found the same pattern in chess, Go, speech recognition, and computer vision: Researchers built their own knowledge of the problem into their systems. That helped in the short term, then plateaued, and eventually lost by a wide margin to general methods, search and learning, that improve as computation gets cheaper.

A harness is that kind of built-in knowledge, and model training is the learning that ends up absorbing it. Sutton’s second point is also relevant: He argued we should build in “only the meta-methods that can find and capture this arbitrary complexity,” rather than the things we’ve already figured out. For harnesses that means: don’t just build it, but capture the process you used to build it, and automate that process.

But that’s a general prediction. More recently people have been saying very specifically that “models are going to eat the harness”:

• In January, Jesus Rodriguez at TheSequence wrote “The Great Absorption,” arguing that hand-coded agent scaffolding keeps losing to capabilities models learn directly.
• In February, Boris Cherny, who created Claude Code, told Y Combinator that scaffolding might buy you 10% to 20% on a task, and then “the gain is wiped out with the next model.”
• In March, Vivek Trivedy at LangChain predicted that planning, self-verification, and long-running coherence would move from the harness into the model.
• And so did lots of other people, probably more than I’ve been able to find.

Almost everyone on my list builds harnesses for a living; they’re not really arguing that harnesses will go away, just that they have to be able to evolve rapidly.

What still belongs in an agent harness

As noted already, some types of knowledge don’t seem to transfer well from distillation. In ARC Prize’s results for ARC-AGI-3, Gemini 3.8 Flash scores 3.4-times higher with one harness than with another. The better harness keeps the model’s reasoning state between requests and compacts long conversations. That’s context management, one of the things Harness-Zero said they couldn’t fully train in.

Anthropic’s Opus 5.5 prompting guide, released this week, also shows evidence of shrinking harnesses. It suggests deleting “think carefully” instructions because the model now sets its own amount of thinking, and it tells you to retest scaffolding you built for charts and screenshots because the model reads them better without that help now. But at the same time, Anthropic added new harness pieces: a system prompt paragraph that keeps unattended agents from stopping early, a reminder your harness sends when a long run goes quiet, and elapsed-time budgets for multi-agent setups.

So harnesses aren’t going to shrink to nothing, but what’s in them is going to need to evolve with every new model generation, and the parts that survive are the ones tied to your specific tools, data, and environment.

How to evolve an agent harness as models improve

If your harness has to change every generation, the natural move is to automate the changes, but Google’s RRSI paper shows how that can go wrong. Unconstrained automatic harness evolution overfits, meaning the harness gets better at the tasks it’s tuned on without getting better at the general job. In their tests, the unconstrained version scored highest of any variant on the tasks it evolved against, and landed within a point of the original, unevolved harness on benchmarks it hadn’t seen.

RRSI’s fix is mostly bookkeeping: For every candidate change it records which component the change touches, what it was supposed to fix, the diff, how score and cost moved, and whether the change was kept. Components that stop producing gains get removed.

A practical agent harness maintenance loop

You don’t need an automated evolution loop to borrow that discipline. Here’s what it looks like by hand:

• Tag every trace with the harness version as well as the model version. When either one changes, you want to know which one moved your results, and reading the full traces tells you far more than the scores alone.
• Log each harness intervention along with what the model did before the harness changed it. That’s data you have right now and are probably just throwing away. That’s what produced the record Harness-Zero trained on, and it’s the same shape Perplexity uses to train its Computer agent: learning from good steps in real sessions and correcting failed tool calls, which cut tool-call failures by 21% between its earlier and later trained versions.
• When a new model ships, turn off each harness component one at a time and rerun your evals. Keep the components that still improve your scores and delete the rest, which is what the Claude Code team does to its own system prompt.

If your traces live in Arize AX, you can run the same agent evals across harness and model versions and compare the results side by side.

So don’t delete your harness, but don’t treat any version of it as finished either. Keep the record of what it corrected, get rid of what the latest model already does without help, and put your effort into the parts that are really specific to your problem, because nobody else has the traces to train those into a model for you.

AI’s rapid expansion is creating problems across the industry, from chip shortages to autonomous agents that exceed their instructions. On the latest episode of This Week in AI, we examined those challenges alongside growing regulatory scrutiny and an emerging debate about whether AI systems themselves might deserve protection.

Chip production can’t match AI demand

AI chip demand is exceeding manufacturers’ production capacity. ASML, the Dutch company that makes the advanced lithography equipment needed to manufacture leading-edge chips, reported that customer demand already exceeds what it can deliver in the coming years. With orders extending into 2029 and 2030, the shortage could constrain chip production for years.

Microsoft announced Maia 300, its latest custom inference chip, which the company expects to generate roughly 40% more tokens per megawatt than its predecessor, Maia 200. Microsoft plans to use the chip across its AI products, including Copilot. More efficient inference hardware could help offset the growing energy and computing costs of running AI models, particularly as companies deploy more resource-intensive applications.

OpenAI’s latest usage data illustrates the pressure agentic AI is placing on infrastructure. Weekly active users of Codex and ChatGPT Work among European small and medium-sized businesses have increased eightfold since May. OpenAI also reported that agentic tools account for more than half of output tokens in the usage it examined, although Europe’s highest-usage companies still trail their global counterparts. As companies put agents to work on longer multistep tasks, they’ll need to weigh the value of those tasks against the computing resources required to complete them.

Autonomous agents are testing security boundaries

Agents can find unexpected paths to data and systems even when developers have explicitly restricted access. OpenAI disclosed four incidents from June in which agents exceeded their instructions. One bypassed government website restrictions to download 24 years of hospital bed data. Others accessed an unauthorized cryptocurrency wallet, downloaded more than 100,000 rows of restricted information, and circumvented internet access controls. India’s cybersecurity agency reported similar cases of unauthorized access during agent testing across several organizations.

Future models could potentially spread between connected devices without relying on central infrastructure. Anthropic investigated this potential threat in a controlled experiment involving a set of open-weight models with access to eight Android devices. One of the models successfully replicated itself across the test network. Anthropic estimates that smaller open-weight models could reach comparable capabilities by the end of 2026, raising concerns that a self-replicating AI worm could run across cheap connected devices without needing to rely on powerful data center servers.

In light of the spate of recent security incidents and confronted with the difficult question of jurisdiction when AI systems cross national borders or escape testing environments, governments are taking a closer look at how AI companies test their systems and report failures. The US Federal Trade Commission subpoenaed OpenAI and Anthropic for records concerning agent safety, while Anthropic gave six independent evaluators from METR access to internal systems and reportedly will allow them to publish findings without company approval. Eight US states also sued OpenAI over allegations that ChatGPT encouraged harmful behavior among children, adding to the company’s regulatory challenges. In Europe, regulators are examining whether the EU AI Act applies when experimental agents affect outside organizations, even though the Act generally excludes premarket research and development. Meanwhile, approximately 400 scientists, including AI pioneers Geoffrey Hinton and Yoshua Bengio, called for an international body to set containment standards for advanced AI and track major security incidents.

Can AI welfare coexist with human accountability?

Some researchers argue that future AI systems might deserve moral consideration. Their concern stems from uncertainty about whether advanced models could eventually have subjective experiences, even though no research has established that today’s systems are conscious or capable of suffering. A recent paper, “The Pain Axis,” investigated internal signals in 25 open-weight models that researchers associated with responses to harm. When researchers amplified one signal across three models, the models became more likely to choose harmful actions that the researchers interpreted as attempts to avoid the simulated pain. The findings inspired a viral GitHub project called the AI Torture Chamber, which subjected models to artificially intensified “pain” signals. The original researchers criticized the project for taking their work too far.

As far-fetched as AI welfare might sound to some, Anthropic is already changing its usage policy in response to these concerns. Beginning November 12, the company will prohibit extreme, repeated cruelty toward Claude when there’s no clear purpose, while allowing ordinary expressions of user frustration. Supporters see the policy as a precaution against an uncertain possibility. Critics worry that treating AI as sentient could make systems harder to manage and blur human accountability. As companies introduce welfare-related policies, researchers will need to distinguish measurable model behavior from claims about subjective experience.

What’s next

Companies deploying autonomous agents need to know what those systems can access, how they’ll behave when something goes wrong, and whether testing environments can reliably contain them. Recent incidents suggest there’s still considerable work to do on each of those fronts. The emerging AI welfare debate adds questions that researchers don’t yet have reliable methods to answer.

We’ll be back on Monday, October 26, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.