I’ve used the skill daily for a couple of months now. It suddenly stopped working with the following message:

API Error: Sonnet 5’s safeguards flagged this message. Our intentionally broad safeguards allow us to deliver more capabilities faster, but can sometimes flag legitimate cybersecurity work. Apply to the Cyber Verification Program to reduce these interruptions. Send feedback with /feedback or learn more: https://support.claude.com/en/articles/14604842-real-time-cyber-safeguards-on-claude

When I started a new Claude Code session with Haiku, the skill worked without problems. (I didn’t try Opus or Fable; if Sonnet found the skill dangerous, I’m sure Opus and Fable would draw the same conclusion.) GPT 5.6 with “high” reasoning was able to execute a very similar skill without problems. So what happened to Sonnet?

The best approach to debugging AI is often to ask the AI itself, so I pasted the message into another Claude Code session and asked it what was happening. The response came down to the descriptions of Hacker News, Bleeping Computer, and The Register. The phrase “vulnerabilities, exploits, threat reporting” in the description of Hacker News triggered Sonnet’s guardrails. Ironically, that description is both incorrect and Claude generated. (Reminder to self: Be more careful when asking Claude to develop a skill from a task.) Sonnet came up with three solutions, the first of which was to let it rewrite the skill with more neutral descriptions like “security industry news.” Fair enough, but I did the editing myself.

Then I went back to the original Claude Code session. It still didn’t work. I expected that I’d need to do something to reload the skill, but the problem was worse. Regardless of the prompt, the original session wouldn’t do anything except repeat the error message. It wouldn’t even commit the modified skill to my GitHub repo. However, Sonnet executed my skill correctly in a new Claude Code instance.

So I returned to Sonnet to find out what’s going on. The answer was interesting: The error may have been triggered by the skill, but when evaluating security threats, the models base their decisions on the entire conversation, not just the specific skill that was called. If a model needs to call a skill that it thinks is problematic, that call is part of the conversation, part of the context. The entire conversation is then forever dead and lost.

What can we learn from this? First, it’s a problem for a program to stop working because of a change over which you have no control. If anything, the industry has erred on the other side; we’re all familiar with “we don’t really understand why this works, so don’t touch it, don’t update the compiler, don’t update the libraries, and run it on emulators of computers that haven’t been built in 40 years.” That’s not just a problem for COBOL code from the 1970s; we see the same thing with C, C++, Java, JavaScript, and just about every language that ever went into production. Legacy code is everywhere. The “don’t change anything” approach isn’t necessarily a bad thing; it certainly beats “here’s a new library, you’re going to love it, you can’t use the old version any more, and wow, look at all the things it broke, guess you’ll have to fix them.” AI where working code breaks at random is a lot less useful than AI that works day in and day out. Stability is a virtue. It’s impossible to work effectively when the environment changes from day to day and isn’t under your control.

But that’s not really what bothers me. It’s rather bizarre that reading well-known sources is treated as a security risk, especially when the “risk” seems to come from an AI-generated description. Of course, we know about hallucinations, errors, and prompt injections. The possibility of a Hacker News post that injects a hostile prompt isn’t zero, and it’s also possible that a model might mistakenly interpret an example of a hostile action as a prompt. I also don’t expect any model to reason that a skill must be safe because it’s been in use for months (though files have time stamps). Artificial intelligence always coexists with artificial stupidity, as does natural intelligence.

Guardrails may keep you from going off a cliff, but they may also prevent you from going where you need to go. And that’s a problem. There’s a basic concept from signal processing and data science called the receiver operating characteristic (ROC). In any binary classification system, you can never achieve perfect classification. The only way to guarantee that no true positives (dangerous things) slip through the classifier is to reject everything. The opposite is equally true: The only way to eliminate false positives (things that look dangerous but aren’t) is to let everything through, including dangerous actions. In theory, it’s possible to get arbitrarily close to perfect classification, but you know how that goes: “The difference between theory and practice is bigger in practice than in theory.”

We know how to make AI “safe”: Go back to 2022 and models that can only tell the difference between cats and dogs. The model might mislabel a few things, but the consequences of an error are small. Safety comes with limitations, and none of us who use AI for real work want to return to the days of dogs, cats, and bananas. And while I don’t want the ability to use Claude to generate hostile attacks against unsuspecting victims, and while I understand the danger of interpreting any input text as a command (for example, an article describing the Morris worm), I have a problem with an AI that refuses to perform reasonable tasks. The ROC tells us that we can’t have perfect guardrails, but there’s no rule against overly fussy ones. What’s allowed, and what’s forbidden? What are the limits? We don’t know. And that’s the situation we’re in now. We can’t know in advance what is and isn’t acceptable, and the rules can change at any time. A tool with unknown limitations is much less useful than a tool that tells you what it can and can’t do. I’ve enjoyed using Claude to write programs that play with prime numbers and infinite series, and fortunately I don’t rely on any of those programs for my job. But what if tomorrow (or a month from now or a year from now) Claude decides that testing whether large numbers are prime signals an attack against cryptography?

I’m not completely unsympathetic to scoring an entire conversation rather than individual actions. A series of steps, each of which appears innocuous by itself, is more likely to lead an agent to a hostile action than a single prompt. But again, given how valuable context is, do we really want the penalty to be losing all the context for an innocuous project? There are risks on either side, including the possibility that a model will ignore its guardrails; after all, rules that a harness adds to the context are at best advisory.

Guardrails always have unintended consequences. We need to learn what the ROC is teaching us: that it’s impossible to get to the upper left corner of the diagram, where we have perfect rejection of true positives (dangers) and no rejection of false positives. But we also need to get as close to that upper left corner as possible if we want our classifiers to have consistently useful output. An engineering team needs to balance risk against usefulness, and they’re clearly out of balance now. Risks will never go away, but guardrails whose boundaries are unclear and overly strict lead to models and agents that are less useful, rather than more. The bad guys will always figure out how to do bad stuff. Hamstrung AI for the rest of us is not a solution.

The following article originally appeared on the Tech Policy Press site and is being republished here with the author’s permission.

In Washington, AI is increasingly being treated as something that needs to be controlled. The government believes that AI is, first and foremost, a national security asset, meaning that it must be sequestered to prevent enemies from gaining an advantage. On the other side of the world, in Beijing, the approach is moving in the opposite direction. China is reducing barriers, encouraging adoption, and using open-source AI as a way to spread Chinese-developed technology across global markets.

There is now a fundamental divide. The United States is betting that control is the path to preserve its lead. China, instead, is betting on diffusion. The country whose technology is adopted most widely may ultimately shape the future of AI. Questions over open source and open weights sit at the center of that contest.

Beginning on July 24, high-profile support for open source moved what is often a debate behind closed doors into the public sphere, where it belongs: Nvidia’s Jensen Huang’s first-ever post on X linked to an open letter signed by 35 companies—including Palantir, Andreessen Horowitz and Microsoft—warning Washington not to over-restrict open source software. “Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty,” Huang wrote, leading the likes of Elon Musk and Mark Zuckerberg to post their support.

Mozilla signed it because, despite being a very different company from many of the signatories and not always seeing eye to eye, we believe in the spirit and substance of the letter, particularly that ‘openness may be one of the most important paths to AI safety and security.’ The letter was followed by the announcement of the Open Secure AI Alliance for AI Safety and Security, which aims to “build and share open tools that promote responsible use of and trust in AI.”

The battle is on. Here’s what’s behind it.

Following the money

Open models now do about a third of the world’s AI work, but they collect only about four percent of the money.

Those two numbers, taken from Mozilla’s new “State of Open Source AI” report, provide more context than the entire AI safety conversation. The report tells a story of a performance gap between open models—the ones whose weights anyone can download, run, and adapt—and the best proprietary systems. While the capabilities are still a jagged frontier, on average, the performance has narrowed sharply over the past year. Costs keep falling. Seventy-nine percent of developers now build with open models. And yet the money hasn’t followed the usage.

This is where the battle lies. A third of the work, but only four percent of the money—that gap is the prize.

The battle started almost exactly three years ago, when Anthropic’s Dario Amodei told Congress that advanced open-source AI is on a “very dangerous path.” On the surface, his argument is pretty simple: once a model’s weights are public, no one can monitor abuse or revoke access. Once released, an open weights model can’t be “unreleased.” But let’s ask ourselves what that revoke switch actually does. Revocation is not a feature of the model, it is in the API contract. That means it only applies to a lab’s own customers—and the people in Amodei’s threat model were never customers. The labs shipping frontier-class open weights—such as DeepSeek, Alibaba, Mistral, and Moonshot (with its just-released, 2.8T-parameter model Kimi)—mostly sit outside Washington’s reach anyway. Two million open models already sit on Hugging Face; many run on a laptop. That means that in the real world, there’s no single kill switch to throw.

This distance between what such a regulatory switch claims to control and what it actually does is what’s missing from the debate over which models are “safer.” It’s also the key to the fight over who captures AI’s value.

To the companies that built the proprietary models, value is about maintaining a privileged position and using everything at their disposal to protect it—policy, pricing, and technology. For everybody else, value means the ability and power to shape, audit, and improve the systems we all depend on. And increasingly, that power doesn’t live in the model at all.

For instance, right now, two developers can take the identical open model and ship completely different products: a scam-call operation or a nurse-advice hotline. The model doesn’t know (or care) about the difference. What is making the actual decisions is the layer of software built around it—the agentic harness—that sits between users and the model, determining what the application can access, remember, and act on.

As models get cheaper, not to mention more interchangeable, that harness is where the power is actually going. And it’s being quietly locked up by the big labs. Farmers know how this story goes. They bought their tractors outright, but the manufacturer kept the keys to the software, making the farmers owners on paper but renters in practice. It took years of lawsuits—and, just this month, the Federal Trade Commission—to start prying that lock back open. A similar arrangement is now being built for the software that reads your email, books your travel, and remembers every detail of your life.

This isn’t an accident of engineering; it’s a business model. A closed wrapper makes money by making itself expensive to leave. An open one can’t lock the door, so it survives only by staying worth using. Same underlying technology, opposite incentives. It’s the reason the value captured by open models sits at four percent while their usage sits at a third. The real question for all the builders right now isn’t which model you’re using. Rather, it’s whether you could leave for a different one.

Guess who’s deciding the future?

The debate that matters isn’t really which models get released or which get regulated; it’s who controls the layer wrapped around them. That’s being decided right now—mostly by developers who don’t realize they’re the ones responsible. For a glimpse of the future, we can look to the internet: it exists as it does today because, when the architecture was still up for grabs, developers chose HTML and HTTP over proprietary walled gardens like AOL. AI is at that same juncture now, and the fact that two million open models already exist suggests plenty of builders have shown up early. That window doesn’t stay open on its own, and it doesn’t stay open forever. It stays open because people keep choosing it.

For developers, four habits matter most in ensuring an open future:

1. Build on open harnesses, not just open models. The orchestration layer above the weights is where capability is concentrating, and closed labs are already welding it shut. Keeping it open takes deliberate effort.
2. Own the memory layer. Store accumulated context in portable controllable formats, so it’s retrievable if a vendor changes its terms rather than trapped inside one.
3. Keep a second model warm. Integrate an open model and keep it production-ready even while running primarily on a closed API, so switching is cheap if it becomes necessary.
4. Don’t assume all open stacks are equal. Open models skew toward particular regions and providers; keeping this layer genuinely open means actively supporting a geographically distributed set of options, not defaulting to whichever model is cheapest this quarter.

None of this requires believing anyone is acting in bad faith. It’s worth noticing, though, that the loudest safety arguments arrived right around the time models got cheap enough for the real competition to move up a layer. That’s not evidence of a conspiracy—it’s just where the incentives point, and it’s why so much of the current debate is aimed at the wrong target.

More evidence is in our report, and most of it is good news: performance gaps closing, costs collapsing, millions of developers building. The question in front of developers isn’t whether AI is dangerous—it’s whether they’ll hold the keys to the machines they’re building. The question for governments is whether the keys they’re reaching for turn anything at all. For now, that door is still open. Let’s work together to keep it that way.

And be sure to join us at AI Codecon: Building with Open Source AI on August 31, a free half-day virtual conference. You’ll hear from leading developers and technical experts working with open-weight models, self-hosted infrastructure, and real-world AI workflows, and learn how building in the open gives teams more control over costs, data privacy, and what they ship. Register today to save your spot.

How to move from prompting to multi-agent coding

1. Start with explicit prompts in every step. Give the coding agent a specification and tell it what to implement. Ofer used a package version resolver with existing Python tests, then followed up with prompts to verify the implementation, resolve open questions, and add packaging.
2. Define a verifiable goal. Loop engineering replaces a sequence of individual prompts with an end result the agent can check for itself. In Ofer’s example, the agent had to implement the resolver and continue iterating until it was ready to ship with packaging. The agent can inspect the code, add tests, run them, fix failures, and verify the package without waiting for another human prompt after each step. He demonstrated how this approach allowed the agent to work through the task until the goal was met.
3. Add a second agent as a reviewer. Ofer then moved from a single coding agent to two collaborating agents (using Jam), assigning one to write the code and another to review it. The reviewer examined the specification, provided feedback, and ran additional checks, including adversarial probes. A larger group of coding agents could include agents focused on security, compliance, testing, frontend, backend, or DevOps. He also described using different coding agents together so that one model can challenge work produced by another.

The shift toward more autonomous coding workflows starts with how the work is framed. By defining goals agents can verify, giving them room to iterate, and assigning complementary agents to review the work, developers can reduce the amount of human intervention required and achieve higher quality for the code generated by the coding agents.

Coming next week

Next week, Craig Hewitt will host Zero to Agent in 30 Minutes to focus on building a voice-first workflow with OpenAI Codex. The episode will show how natural voice commands can operate a development environment, run subagent workers in parallel, and trigger browser-use workflows. It will also cover structured Codex project directories and hands-free system-level execution, with the developer directing the work by voice.

The following article originally appeared on Medium and is being republished here with the author’s permission.

Everybody’s talking about the death of developers. I get it. The developer whose job was to write boilerplate or scaffold CRUD apps is done—a model can do that in seconds, and that developer is not coming back. But the people announcing the end of programming are missing something. There’s a new job title that’s starting to emerge across several areas in software.

Architects and developers are becoming orchestrators—one person directing work that once required entire teams. This shift will reach far beyond software, but software engineering is where I’ve seen it firsthand.

An orchestrator knows how to develop software, but their job isn’t to write the code anymore—it’s to oversee a system, orchestrate tools and agents, and generate components into something that has to work in production. But the most important distinction between an “orchestrator” and a “software developer” is that an orchestrator focuses less on delivering software and more on orchestrating the systems that can both operate and develop software.

The technical expertise that used to be applied to figuring out the structure of a database schema or an object model will now be applied to guiding a set of subsystems that have taken responsibility for most tactical, line-level decisions. Where a “developer” in 2023 focused on deciding how a React application might store state, an “orchestrator” in 2027 is focused on a DESIGN.md file that sets standards for a subsystem that is responsible for fusing analytics data with input from customer feedback to recommend, test, and implement site changes as part of a large, more autonomous approach to running a business.

Orchestrating systems of delegated intelligence

While everyone is calling everything “agents” these days, I’m also going to put forward an idea. An orchestrator can and will use systems that resemble some of the more “agentic” approaches we’re all using today, from systems like Hermes, OpenClaw, or every other system that has started to call itself an “agent.” I’m starting to see that the term is overused. Taking a step back from the technology we’re using today, I’m going to suggest that the job of an “orchestrator” is to coordinate systems that fall under a new category called “delegated intelligence.”

We’ve been calling everything “artificial intelligence” for several decades. That term, mixed with “Generative AI” and “Inference Engines,” fails to capture what we’re starting to see in practice. An agentic system that has memory and can start to operate with a level of independence is exhibiting “delegated intelligence,” and the word “delegated” is doing a lot of work. It implies that systems in this category will always be traceable back to an accountable operator, or, in this case, an Orchestrator.

Fundamental to the shift toward Orchestrators is a combination of automation, productivity, and accountability. As organizations, companies, and governments start to make use of delegated intelligence to support a more autonomous approach to design, operation, and engineering, there will be an increasing need to establish accountability. If your business operates critical infrastructure on a set of autonomous agents, one of the questions that will become necessary to answer in the case of an outage is “on whose authority was this delegated intelligence operating?”

An ecosystem of orchestrators: Generalists and specialists

There are orchestrators, and then there are technical specialists. A move towards generalist expertise marks orchestrators, because the capability of a specialist is now found mostly in a model. You’ll still need a couple of specialists, but not one for every technology—and even some of those specialists will be specialist orchestrators. It’s going to get complicated.

You’ll be looking for generalist orchestrators who understand the whole thing end to end. You could think of an orchestrator as an expert Renaissance programmer—usually people with a couple of decades of experience who understand the end-to-end life cycle of software development. Those are the individuals becoming orchestrators, and it’s changing the whole makeup of IT departments. We’re no longer programmers.

. . .

Let me use my own experience here to capture what the new reality looks like. I recently had to add DRM to a series of audiobooks I’m self-publishing—an inaudible watermark encoding order-specific data into the audio file, so that if I find one of these files in the wild, I can identify who bought it. I’m not a subject matter expert in overlaying audio watermarks, but I do understand how to write code that processes sound files.

Orchestration: From months to minutes

This particular task would have taken me weeks or months, and not long ago I would have started the project by creating a git repository and opening up an IDE. That’s not how it works in 2026.

When I orchestrated the creation of this system two weeks ago, it took 20 minutes, and the tools gave me three dimensions of highly encrypted watermarking and fingerprinting—essentially the work product of 3 steganographic audio specialists.

Okay, I lied—it was 40 minutes. The first 20 minutes I was asking the models to come up with 5 different approaches so I could choose the right one. But I want to emphasize that I asked the system to produce 5 different proposals and then model the long-term cost and operability of each option. I also gave it direction to think about customer experience, create a matrix of pros and cons for each option, and end with a recommendation.

This was all done by a system that has been tracking content development for several months, and it used customer knowledge, analytics, and product design to inform the set of options it was giving its Orchestrator before jumping into implementation.

Your job isn’t code, it’s orchestration

Part of my job now is to leverage these tools not just to create, but during ideation, product design, and quality engineering. The Orchestrator is there because that person knows what questions to ask. It would have taken me three months and a large team to get this done only four years ago, and I shipped it without reading every line.

To be frank, it’s a weird space to be in, and the “me” from three years ago would have been really uncomfortable hearing that I implemented something but didn’t write the code myself. In fact, my initial reaction to Steve Yegge saying that he should stop reading his code was very negative, and I still have some reservations about that belief, but I will say that in my own practice, I’m starting to not read my code—because it’s not my code.

Like a lot of programmers reading this, I’ve had to go through an identity moment as a programmer—becoming comfortable with shipping something to production I might not have read every line of. There’s maybe a hundred thousand lines of code—too much to read—and honestly, it’s not my job anymore. I’m an orchestrator, and it would be highly inefficient if I tried to keep up.

I’m not a vibe-coder, and I’m not a “citizen developer”—a term I hate enough to curse at because it’s just the wrong word. I’m someone who could write the code, but I’ve decided to delegate that task to a system that has more information at hand than I could ever hope to assemble. And while these systems, the delegated intelligence tools like an agent, can implement systems in mere minutes, they still need a human to weigh in on direction. And I would argue that we still need a human to remain present and accountable.

The Orchestrator knows enough to understand what was done for them and how to dig into the details when something breaks. They know how to debug, they have a sense of what’s valid and what’s not, and they’re honest about what they know and what they don’t.

Orchestrators as finishers

Here’s the controversial part. This is absolutely not about “democratizing access to technology.” There’s a myth that anyone can pick up these tools and code, and that is true—yes, anyone can code, but not everyone can deliver it to production in a scalable and secure manner.

A colleague of mine recently wrote that it’s easy to start projects with generative AI, but what’s difficult is finishing them. It takes the same effort, energy, and technical expertise to deliver something to production as it always has.

The idea that anyone and their brother can pick up a generative AI tool and create technical perfection—that myth is about to expire. If you build a complicated, technical system without an individual responsible for orchestrating the creation and operation of that system, there will come a day when you have to pay someone to do that for you, and that someone is going to charge you a lot.

Orchestrators plan for contingencies: Ability to support

That DRM system I just talked about—I haven’t read every line, but before I shipped it, I made sure that I understood the baseline for support going forward. While I delegated authority to an agent to create it, I also made sure to ask that same agent to capture code locations, architecture, approach, and to generate a system of documents that could be used to debug and support it if AI was unavailable.

Have I read this “pilot manual” from start to finish before deploying this to production? No. But I understand where the throttle gauge is, and if I needed to land this plane without autopilot, I could. This is one of the responsibilities of the new role. Planning for “offline,” thinking through contingencies.

The key point is that I’m qualified enough to understand the pilot manual that AI wrote for me in case I need to debug it, and if AI were to disappear tomorrow, I could rebuild it myself. That is not true for many people introducing themselves to coding through AI, and it creates a dependence on the tools that needs to be managed—one of the ways it will be managed is through certified orchestrators who can create but also support systems without the tools, especially in regulated and critical areas.

Adapting the organization to the emerging role

Things are changing fast—one Orchestrator equals 20 or 30 developers, plus teams of QA engineers. While we’re still going to need product people and people who think about the customer, the technical work is consolidating around the person directing it.

And before you ask—why not just call this an architect? Because architect never worked. If you’ve worked in a company that has architects, you’ll understand that while a few architects continue to keep up-to-date with technology, many also tend to lean back on past experience delegating day-to-day technology to junior engineers. This role differs from that of an architect because it calls for someone to be engaged with specifications, outcomes, and operations.

Orchestrator isn’t the incommunicative programmer that stares at an IDE all day; they are the individual that understands the full, end-to-end flow not just of data in a technical system but how the business operates and adapts autonomously. They are technical, but they are also focused on providing oversight, and they are the individual responsible for deciding what intelligence can be delegated.

So how do we create orchestrators? Not from a bootcamp or a six-week certificate. This is going to take an apprentice program—years of it, the same way we create doctors and lawyers. You work under someone who knows what to pay attention to, and you learn by watching them make decisions.

These are employees who can do real damage, and they’re a walking liability. When a corporation buys insurance for people writing code, that’s one set of risks. When you’re insuring professionals who are a hundred times more productive, they also carry a hundred times more responsibility. Insurance rates, regulations, and certification requirements are all going to go up. Doctors carry malpractice insurance because their decisions affect people’s lives.

A Professional Engineer has to be licensed because engineering affects public safety. Orchestrators are headed the same direction.

If you fast-forward 20 or 30 years, what we call programmers now are going to be orchestrators, and they’re going to look more like doctors and lawyers than like this band of people we have now who write code.

Code will be part of the job, but not the majority of it.

The following article has been extended and rewritten by Markus Eisele from The Main Thread and is being republished here with the author’s permission.

Open a repository after six months of spec-driven agent work and you may find a second system sitting next to the code. Requirements, research notes, high-level designs, low-level designs, implementation plans, task lists, review reports, and a growing stack of Markdown files that explain what the code is supposed to mean. Even if the code changed significantly last Tuesday, the last documentation update was weeks ago.

I understand how teams get there. And it’s not a really new effect after all. We had software evolving parallel to documentation since I can remember. Now that agents produce code so  quickly, we try to control the drift and the code generation by moving more thought in front of implementation. Instead of documenting code, we try to drive code generation with it, making Markdown files with requirements, decision records, design approaches, and acceptance criteria the center of gravity and turning them into our workflow drivers.

What effectively is becoming a very large prompt can easily fill a significant portion of the context window even of modern agents before any relevant source code gets added to it. Natural language specification is a weak system for agents to synchronize a codebase with. Without additional attention and diligence, most agents I work with slowly shift attention away from it quickly and focus on the stronger signals in the codebase, forgetting to update the specification eventually.

Even if it sounds like it, I am not advocating for one-shot prompting or vibe coding here. We still need some specifications to build successful software. The mistake is treating a specification as a permanent natural-language copy of the software. A useful spec describes the next change, documents the decisions that drive the change, sets boundaries, and gives us and the agents enough verification surface. But as soon as the change ships, most of it should be removed.

What remains should move into the artifacts software teams already know how to maintain. First and foremost, obviously, the code. But I also count schemas, configuration, and policies as relevant artifacts. They carry meaning about domain knowledge and system configuration. Two categories that I value highly get easily forgotten: tests as the stable verification layer and runtime telemetry. In fact, I do let my agents look at evidence from all these places not only to hunt for errors but also to continuously optimize existing codebases. Oh, and I do keep decision records. But only a small number and only when their content really has no other place in any of the mentioned artifacts. They can even look like Javadoc, but that will be another article someday.

Code is the fact

Code is actual behavior. Once code is deployed to production, users and connected systems are depending on it. Even a mistake can become an observed contract because it has behaved the same way for three years. The runtime behavior takes precedence in this contract because nobody checks the specification anymore, even if it defines a very different behavior. This is the strongest signal for me to start with the actual code in the production system. Reading a natural-language summary instead of the implemented truth cannot accurately reflect runtime behavior. Code to me is the ultimate, executable specification. Just written in a very specific and deterministic language.

What production code cannot drive though is the next version or iteration of a feature. While agents can infer technical patterns from well-structured codebases, there’s no way they could predict policy changes or future feature requests. Neither can they know about regulatory requirements like retention periods or other specific exceptions, such as why one export runs every night for only one customer. That specific context has to come from somewhere else. But it does not require us to keep a permanent prose description of the whole system. We need just enough context to decide the delta: the difference between what exists and what should exist next.

Written words are for the delta

A change specification should exist when it helps a team decide and review that delta. It should name the outcome, non-goals, constraints that differ from current behavior, and the evidence required for acceptance. It might even contain technical design elements when new features cross architectural boundaries or introduce new patterns that are not present in the code yet. Sometimes it is also worth thinking about how expensive reversing the change is, especially if the existing system has various implementations for a certain pattern and the risk is high that an agent might invent another new version.

The list necessary for changes is very short:

• The intended outcome and non-goals (where necessary)
• Known unknowns and decisions that need human judgment
• Affected system boundaries and authoritative interface artifacts
• Functional and nonfunctional constraints that differ from today
• Acceptance criteria/test scenarios covering the risky path

I prefer calling this a “change brief” instead of a “specification.” Specification carries too much negativity. It sounds heavyweight and reminds me of times long past. It also pretends to be complete. And this completeness is making it very expensive.

We have tried exhaustive specifications before and produced requirement documents and other  high- and low-level designs, followed by architecture decision records for everything. I remember reading folders full of paper over the weekend to get started on a new project on Monday. Way before AI even entered all our lives and codebases. We called this waterfall back in the day, and the approach still has the same negative side effects today. The documentation was complete in an administrative sense and was mostly useless in the engineering sense. We all have seen this happening. Agents easily recreate the same erratic results from overflowing documentation, like we did back in the day.

One particular risk I am seeing with many teams is that they let agents generate the initial version of the spec. A long workflow run produces not only the research but directly derives the requirements, design, and planning, and reviews artifacts on top. While the completeness makes everything look very controlled and defined, it also generates a lot more material to be reviewed and approved. Even if models and harnesses continue to evolve at breathtaking speed, it is still challenging for them to generate real cohesiveness out of chaos. The chance they put the wrong attention on some tempting repetitive words is high. This results in an even higher burden on the human reviewer and makes it endlessly harder to keep the various documents aligned.

I think that additional prose like research notes, prototypes, and design records should only be added to a software project when uncertainty justifies them. They resolve a specific problem. Or help navigate the terrain. I wrote about this before. They should absolutely not become required stages for every pull request.

The map will always be incomplete

A prompt, ticket, or change brief captures what we know before the work starts. The codebase, runtime information, configuration, connected systems, and years of accumulated decisions glued into code hold the rest. Some of those decisions were never written down.

When agents get to work they expose the missing information. Reading a module reveals an unexpected dependency. A prototype shows that a specific user-interaction is awkward. A test uncovers an edge case. Production data contradicts an assumption in the design. This field guide on finding unknowns in agent work describes the problem well. We can identify some unknowns at the start. Others appear only after we inspect the references, build a prototype, or review a result using judgment that was difficult to write down in advance.

Discovery happens and continues during the work:

• Before implementation, inspect the current system and identify decisions that could change the architecture or user experience. When preferences are difficult to describe, build a cheap prototype.
• During implementation, record meaningful deviations. Stop and reassess when a new unknown changes the risk or direction.
• After implementation, read the code, run the checks, and compare the result with the original intent.

The change brief remains part of this loop. It provides the starting point and records the intent, while the work supplies the information needed to complete it. Only promote durable constraints.

Keep durable facts in their native form

When I say “promote durable constraints,” I do not mean turning every decision into permanent Markdown. That gives us the same stale documentation problem in a different way. Software engineering already provides better versions for most of the necessary, durable facts:

• API shape and compatibility belong in OpenAPI, AsyncAPI, protocol schemas, types, and compatibility tests.
• Data invariants belong in types, database constraints, validation, and migration checks.
• Security rules belong in access policies, static analysis, dependency policies, and runtime enforcement.
• Architecture boundaries belong in module structure, dependency rules, and focused architecture tests.
• Reliability requirements belong in load tests, service objectives, telemetry, and alerts.
• Release rules belong in continuous integration and deployment policies.

These artifacts are already part of delivery. A failed schema check or alert needs to be fixed and handled while the corresponding paragraph in an old design folder does not.

Natural language and specification still have a place in software. Specific domain knowledge like business policy, trade-offs, and even architectural rationale do not always fit into an executable artifact or annotation. I keep that prose short and close to the thing it explains. A small architecture decision record is worth keeping when a future team might otherwise repeat an expensive investigation and a code comment cannot justify the implementation. Recording every local choice just hides the few decisions that matter and confuses the agents that are supposed to build the software. Ask which fact must survive and what its authoritative form should be.

Judgment belongs in the workflow

Heavyweight specification methods try to control quality by prescribing the path. Every change goes through the same documents, reviews, and test categories. That approach creates a lot of attention on low-risk work while avoiding the deep technical judgment needed for harder changes. A copyedit and a payment-flow change should not have to follow the same process or testing strategy.

Simon Willison describes a simpler approach: give the coding agent the outcome and let it judge how much process the task requires. His examples include deciding whether a change warrants automated tests and whether routine implementation can be delegated to a cheaper model while keeping judgment-heavy work in the main loop. This replaces a growing list of procedural branches with one expectation: Choose tactics that fit the work. That matches how I want these systems to operate. And I think it extends to specification and how we document intent.

Agentic changes still require clear boundaries. The team defines the outcome, safety constraints, ownership, and who has authority to accept the result. Within those boundaries, the agent can choose its tactics. When uncertainty introduces consequences beyond its authority, it should surface the problem and ask for a decision.

The workflow then starts matching the risk introduced:

• A small, familiar change can move from a short brief to implementation and review. Almost a one-shot prompt change.
• Unfamiliar code requires factual research before design. Explore codebases, identify implementation details. Preload intent and agent knowledge.
• An unclear user experience calls for prototypes and comparison. And might even require user research after all.
• An architectural change requires explicit human alignment.
• High-consequence behavior requires stronger independent evidence and approval.

I would rather add processes and additional artifacts when the work becomes risky or unfamiliar. Starting every change with the full ceremony just burns time and context.

Context is an engineering budget

Large specifications cost more than the time required to write and maintain them. They also  compete with the code and evidence the agent needs for the current decision. Every requirement, design note, repository instruction, and tool definition consumes part of a limited working context. Extra material burns expensive tokens, but the much bigger cost is lost attention. Important rules become harder to follow when they are surrounded by stale or duplicated material. A spec that leaves too little room for the repository defeats its own purpose.

Progressive disclosure is a better fit. Give the agent a small map, a few stable rules that apply broadly, and pointers to deeper material. A concise AGENTS.md can document build commands, repository layout, and architectural boundaries. It should not narrate every class or repeat API documentation. The file helps humans for the same reason: It tells them where to look without pretending to replace what we will find.

Experience with Research-Plan-Implement shows what happens when the context grows too large. The original workflow moved human review before implementation, but teams ended up with large prompts and plans that could reach 1,000 lines. Engineers reviewed those plans while treating generated code almost like compiler output. The implementation could still drift from the approved plan, which meant that eventually someone had to reconstruct the decision from the code. That problem becomes worse in brownfield systems, while greenfield systems might even survive large plans because they inherited no hidden constraints. Complex changes, in contrast, often inherit behavior that plans may miss.

In “Everything We Got Wrong About Research-Plan-Implement,” Dexter Horthy revisits the original position. Teams shipped more code and then spent much of the gain time cleaning up earlier low-quality output. The implementation could also diverge from the reviewed plan, which forced engineers to reconstruct what happened from the code anyway. The revised workflow uses smaller contexts for factual research, design alignment, structure, implementation, and review. I take a simple lesson from this: Research and design give me leverage, but I still need to understand and own the code that is generated.

Modernization makes this obvious

A mature application contains several kinds of behavior in the same codebase. Some logic represents durable business logic or implements a published interface. Some code exists because an old platform imposed a technical constraint. An incident fix remains long after its  context is gone. And even defects can survive to the point where they almost look intentional when undiscovered.

An agent that treats every code variant as a new target specification can translate those layers faithfully into a new language or architecture. The translation may be technically accurate but also preserves defects and old architecture approaches in newer and cleaner code.

I design changes to brownfield projects similar to the way I did modernizations before the agentic age. Classification and observation are central aspects that I put first. The goals are:

• Preserve durable business invariants and externally required behavior
• Verify behavior that appears active but lacks clear ownership or evidence
• Redesign logic tied to obsolete architectural constraints
• Remove dead paths, duplicated logic, and confirmed defects

You can read a lot about static source code analysis when it comes to brownfield assessments or modernization. You can inspect dependencies and current behavior by executing tests and maybe even adding test cases to secure behavior. What I do recommend is to also embrace mutation testing approaches (e.g., PIT) to find hidden assumptions and failure behavior. Code coverage is also seeing a renaissance because it aids in identifying dead code paths.

On top of that we still ignore operational context and telemetry data. Both are vital elements to not only control but also to help judge existing behavior. All this together helps you judge which elements belong in the system going forward and which don’t. It all starts from code. It is the foundation of the behavior we have. The original and leading specification. A change brief will always be temporary and its sole job is to describe the delta between existing and future functionality. The new implementation and its native checks become the next durable state.

Small specs still need real evidence

Keeping specifications small does not mean returning to a loose prompt followed by hopeful review or even vibe-coding approaches. An agent can turn an underspecified request into a coherent implementation before the missing decisions become visible to anyone. The result may compile, pass the available tests, and look internally consistent. That coherent appearance is part of the risk now. Unapproved business decisions disappear into something very ordinary-looking because they got resolved plausibly.

And this behavior is backed by research. If we look at repairing ambiguous natural-language requirements, for example, we can see that directly asking models to resolve ambiguity often leads to inconsistent or even irrelevant results. Choosing a more targeted repair approach around the identified defects (change brief) improved the results by roughly 31%. SWT-Bench found that generated tests could filter proposed fixes and double the precision of a software repair agent. They used one agent to generate a proposed change and gave another the task to produce evidence to reject it. Lastly, the topic of formal specification generation: One interesting study I found gave 30 models the task to translate natural language into TLA+ (Temporal Logic of Actions, a specification language created by Turing Award-winner Leslie Lamport). The best results only reached about 27% syntactic correctness and 9% semantic correctness. The formal notation helped to detect mistakes, but it did not guarantee correctness or that the translation preserved the original meaning.

These results support focused clarification and independent checks. Clarify the uncertainties that can change the outcome, then verify the implementation with evidence that does not come entirely from the same reasoning path. Generating a longer specification does not solve that problem at all.

I want the strength and independence of the evidence to match the consequence of being wrong. A small internal refactor may need ordinary tests and code review. A change that involves security or financial aspects, or that even touches regulated data, needs a much stronger separation coupled with adversarial review and explicit human approval. For those changes, the agent proposing the implementation should not also be the only source of its requirements and tests.

A lighter operating model

In practice, I want a workflow that I can explain without a complex flow diagram. It starts with the evidence already in the system and makes the intended change explicit. Everything else is added only when the potential risk of the change justifies it. Ideally, this is a simple five-step process:

1. Start from the code and operational evidence that describe the current system
2. Define the intended delta, important boundaries, and known unknowns
3. Add research, prototypes, design alignment, or stronger verification where risk requires them
4. Read and review the implementation, not just the plan
5. At release, discard temporary reasoning and preserve each surviving fact in its native authoritative artifact

That is enough structure to guide the work without building a natural-language replica of the software.

Before implementation, the change brief describes the intended delta, and during implementation it helps people and agents align while new information changes the plan. But after the release the code and production behavior become the primary evidence of what the system does. Not separate documentation in any form that potentially drifts over time.

Durable obligations remain in the artifacts we already know how to maintain: schemas, tests, policies, configuration, telemetry, and short records for rationale that cannot be encoded elsewhere. Most planning details have completed their job by then and should expire.

I expect teams to get the most from coding agents when they are selective: specify what must be decided, discover what the system can answer, verify what carries risk, and let temporary planning go.

Sources

• A Field Guide to Finding Your Unknowns, Thariq Shihipar, 2026.
• Judgement, Simon Willison, 2026.
• Can LLMs Write Correct TLA+ Specifications?, Bisharat et al., 2026.
• Automated Repair of Ambiguous Natural Language Requirements, Jia et al., 2025.
• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents, Mündler et al., 2024.
• Large Language Models for Requirements Engineering: A Systematic Literature Review, Zadenoori et al., 2025.
• Advanced Context Engineering for Coding Agents, HumanLayer, 2025.
• Everything We Got Wrong About Research-Plan-Implement, Dexter Horthy, 2026.
• Usage, Effects and Requirements for AI Coding Assistants in the Enterprise, IBM Research, 2026.
• From AI Governance to AI Assurance, IBM, 2026.

That ratio framed a larger question running through the episode. What changes when software can operate continuously, respond in seconds, and increasingly take action without waiting for a person? Vicki looked at faster cyberattacks, growing investment in agent security, the resource demands of AI infrastructure, and the expansion of AI from chat interfaces into robotics. The episode points beyond model selection to the systems required to deploy AI safely and reliably.

Security operations have to match agent speed

AI is compressing the time required to find and exploit software weaknesses. Vicki pointed to reports of attackers moving in minutes and vulnerabilities being exploited soon after public disclosure. She also described an attack against one of her customers in which the attacker returned, changed tactics, and tried again.

Traditional security processes assume there is time for people to investigate an alert, understand the vulnerability, deploy a patch, and monitor the result. That assumption gets weaker as automated systems become faster at reconnaissance and adaptation. Vicki argued for multiple defensive layers across APIs, applications, and networks so that one missed signal does not become a single point of failure.

We’ve followed agent security throughout This Week in AI, and the discussion now centers on how enterprise security changes around more autonomous software. That puts more weight on automated defenses, tighter permissions, and monitoring systems that can constrain machine activity at comparable speed.

AI capacity depends on physical infrastructure

AI capacity requires electricity, cooling, water, data center space, and the infrastructure that supplies them. Vicki connected large hyperscaler investments with projections for sharply higher data center energy and water use by 2030. An audience member added a useful example from a university data center that can reuse waste heat during colder months but has to shed that heat during warmer weather.

Those constraints affect deployment decisions directly. Organizations have to account for power availability, cooling systems, water access, latency, security, and local infrastructure capacity alongside model performance and cost.

Government policy already shapes those choices. The episode paired expanding investment in AI infrastructure with growing regulatory requirements in Europe. AI infrastructure now spans engineering, economics, compliance, and public policy, which means deployment decisions increasingly involve several systems at once.

Human judgment becomes more valuable when AI can act

Rapid AI adoption increases the value of foundational knowledge. Vicki raised that issue while discussing AI use in education and research. Students may have easier access to explanations and answers, but someone who does not understand the subject may have little basis for recognizing an incorrect result. The same problem appears in scientific work, where reliable AI output still depends on reliable data and reproducible processes.

That evaluation problem becomes more consequential when AI controls physical systems. Vicki described systems that can perceive their surroundings, pass information about that environment to a model, and use the result to guide physical actions. Errors in those systems can extend beyond a bad answer on a screen.

Practitioners still need to evaluate evidence, recognize weak assumptions, and decide where automated action should stop. Better models can reduce some forms of manual work, but they also increase the value of people who understand the domain well enough to know when a system’s output does not fit the situation.

What’s next

AI systems can now operate faster and more independently than many of the processes surrounding them. Security teams have to defend at machine speed. Infrastructure planners have to account for physical resource limits. Researchers, students, and practitioners have to evaluate increasingly capable systems without assuming that capability guarantees correctness.

The 144-to-one ratio makes that change concrete. Agent adoption is already testing whether organizations can govern these systems, support the infrastructure they require, and preserve informed human oversight.

Join us again next Monday for another episode of This Week in AI, when we’ll dive into more of the news, issues, and key developments shaping the AI era. And check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

The following article originally appeared on Addy Osmani’s blog site and is being republished here with the author’s permission.

Technical debt lives in your code. Cognitive debt lives in your head. Intent debt lives in the artifacts you may never have written: the goals, constraints, and rationale for why the system is the way it is. If you’re lucky, some of this exists scattered in team documents or discussions, but it’s likely incomplete. It’s the one kind of debt your agents can’t pay down for you, and agentic engineering makes it the most expensive.

____________________________

Three places debt can live

Margaret-Anne Storey’s Triple Debt Model is a clean way to think about software health. The three models of debt are technical, cognitive, and intent.

Technical debt lives in the code. It’s the accumulation of implementation choices that make the system harder to change later: the tangled module, the shortcut you took under deadline, the abstraction that leaked. We’ve understood this one for decades. You feel it coming through slow builds, fragile tests, and the dread of touching one particular file.

Cognitive debt lives in people. It’s the erosion of shared understanding, the gap between how much code exists and how much any human understands. I’ve been calling this comprehension debt. It builds up when the system grows faster than the team’s mental model of it. Your code can be pristine and you can still carry crippling cognitive debt, because nobody understands the pristine code either.

Intent debt lives in artifacts. It’s the absence or erosion of the externalized rationale, goals, and constraints that explain why the system is the way it is. The key word is externalized. The rationale has to be written down where a teammate, a future you, or an agent can read it, not held in your head. When intent debt runs high, the system drifts from what you meant it to do, and nobody can say when it diverged or why.

These three are independent, which took me a while to internalize.

You can have low technical debt and high intent debt. You can understand a system completely yourself (no cognitive debt for you) while its intent exists nowhere outside your skull (enormous intent debt for everyone else).

From the inside they feel alike, but each one bills you separately.

Why intent debt is the one agents can’t help with

AI generates code faster than ever, which makes technical debt cheaper to take on and cheaper to pay down. Point an agent at a tangled module and it’ll refactor it.

Cognitive debt recovers too, more easily than most engineers expect. When you don’t understand a chunk of the system, you ask the agent to explain it. You rebuild part of the lost mental model on demand, because the code still exists and the model can read it back to you.

Intent is different. An agent can’t generate intent, because intent is the one input that has to come from you. A model can infer a plausible rationale from the code, the same way you can guess why a previous engineer did something. A guess about intent isn’t the intent. The model doesn’t know whether that 300ms debounce was a deliberate UX decision, a benchmark result, or a number someone typed once and never revisited. It will invent a confident-sounding reason, which is worse than admitting it doesn’t know.

Of the three debts, intent debt is the only one where the agent can’t bail you out. It can write the code and restore your comprehension. The why is the one thing it can only fabricate.

Agents make the unwritten cost compound much faster

Teams got away with high intent debt for years because we carried it in our head and old docs.

When a new human joined a team, you didn’t write everything down, because they picked up intent over time: hallway conversations, code review comments, “Oh, we don’t do it that way because of an incident in 2023.” Knowledge moved person to person and built up. The engineer who’d been there four years was the intent documentation, expensive and lossy, but it worked.

Agents break that model. Bringing agents onto a team doubles its size overnight with junior people who have no long-term memory. An agent starts most sessions cold. It carries none of the tacit intent humans built up over years. Whatever you haven’t externalized into an artifact it can read, it doesn’t have.

That changes the economics of not writing things down. Unexternalized intent used to cost you once in a while, at onboarding or after someone left. Now you pay it every session, multiplied by every agent you run.

Picture the 20 agents you’re so excited to parallelize. Each one is a teammate who has never met you, can’t read your mind, and will fill any gap in your intent with a plausible guess. The orchestration tax I wrote about is partly an intent-debt tax. Much of what makes managing many agents exhausting is resupplying the intent you never wrote down.

The other half of the comprehension debt argument

When I wrote about comprehension debt, I made a point I want to revisit, because intent debt sharpens it.

I argued that detailed specs aren’t a complete answer. Translating a spec into working code involves a huge number of implicit decisions no spec ever captures, and a spec detailed enough to be the program is the program in a slower language. I still believe that.

Intent debt is the complementary truth.

Being unable to capture all intent is no license to capture none of it. The implicit decisions an agent now makes on your behalf, the ones a spec will never enumerate, are the decisions whose rationale evaporates if you don’t record at least the load-bearing ones. You can’t write down everything.

You do have to write down the why behind the choices that would be expensive to get wrong, because nobody will reconstruct those later.

Comprehension debt warns you not to trust that code is correct because it exists.

Intent debt warns you not to trust that the reason survives because the code does. Code is the answer; the intent was the question it was meant to solve. AI is brilliant at producing answers to questions you forgot to write down.

What high intent debt looks like

Intent debt rarely shows up as friction. It shows up as a particular kind of helplessness.

• An agent “fixes” a bug by deleting a guard clause, and nobody can say whether that guard was load-bearing or leftover, because no doc or commit message ever recorded why it was there.
• A refactor changes a behavior users depend on. The review passed because the diff looked clean and the tests were green, but the tests only encoded the previous behavior, never the intent.
• You ask why two services talk over a queue instead of a direct call, and the honest answer is “An agent suggested it and it seemed fine.” That answer is intent debt, already accruing interest.

If you’ve felt the cognitive surrender version of this, defending a design choice you can’t reconstruct, intent debt is the team-scale, written-down version of the same hole.

Surrender is about your own posture in the moment. Intent debt is what a hundred of those moments leave in the repo for the next person and the next agent to inherit.

Paying it down: externalize intent as a first-class artifact

Almost everything I’ve been writing about for the last few months turns out to be intent-debt management. I didn’t have the word for it. The move is the same each time: Take the intent out of your head and put it somewhere an agent can read.

Write the spec for the intent, not the implementation. A good spec captures the goals, the constraints, the nonnegotiables, and an explicit definition of done (fast, accessible, secure, delightful, beyond “functionally correct”). The spec carries the intent the code can’t carry on its own.

Treat AGENTS.md as your intent ledger, not your config. It’s why I keep saying stop using /init. An auto-generated file describes what the code is. An intent file describes what the team means: the conventions, the “we don’t do it this way because,” the constraints invisible in any single file. Agents can’t infer that, and they need it most.

Capture decisions where they happen. Lightweight decision logs (ADRs) are pure intent-debt paydown. Recording why at the moment you decide costs almost nothing. Reconstructing it eight months later, after the person who knew why has moved teams, costs a fortune. Agents have made logging cheaper than ever, so the old excuse is gone.

Make the learning loop write intent back down. I’ve argued for self-improving agents that update a learnings file at the end of a session. The same loop is an intent-debt pump running in reverse: every mistake whose root cause you’ve recorded, every “We tried X and it didn’t work because Y” is intent that would otherwise have lived only in your memory of a bad afternoon.

None of these are new tools. They’re the discipline of refusing to let the why exist only in your head, in an era where your head is no longer where most of the work happens.

Where the value moved

For a long time, the scarce, valuable thing in software was the ability to produce a correct implementation. Code was expensive, so we optimized for writing it.

AI made code cheap, and comprehension is recoverable. Intent, the goals and constraints and reasons, is the one input that still has to originate with a human. It’s also the one we’re worst at externalizing, because for decades we got away with carrying it in our heads.

That worked when the team was a handful of people who could absorb intent over years of shared context. It does not work when half the team is agents that start every session as strangers.

Technical debt makes your system hard to change. Cognitive debt makes it hard to understand. Intent debt makes it hard to know whether the system still does what you wanted, and it’s the only one of the three your agents can’t pay back for you. That part stays with you. Write down the why, because it’s becoming the most valuable thing you can leave in the repo.

I like to say that context engineering is the art of shaping what a model sees so that it actually does what you want. (I just realized that in saying that I’m channeling a comment that Andrew Singer made to me over forty years ago, when he was teaching me about debugging. He called it  “the art of figuring out what you really told the computer to do instead of what you thought you told it to do.” But that’s another whole story.)

Drew gave a talk at the recent Friends of O’Reilly camp, Foo Camp for short, about what he calls prompt debt, which he describes as “the hidden costs that teams rack up when they fight a model’s training instead of working with it.”

That was a novel and useful framing to me, that you end up with a bunch of stuff in your prompts to compensate for default behavior of the models, that those prompts no longer work as the models upgrade, and so it becomes a kind of technical debt. He’s thinking a lot about what the best developers are doing differently as a result.

So I invited Drew to reprise his short talk on Live with Tim O’Reilly, and then we talked about it with the folks attending the live event. They had a lot of good questions, so it was an interview not just by me but by a crowd of O’Reilly customers.

Prompt debt in practice

Drew opened his talk with two slides. The first was a prompt anyone could write in ten seconds: “You are a customer support assistant. Read the ticket, classify it as billing, technical, account, refunds, or other, return only the category name.” The second slide was the same prompt a few weeks later, after it had met the real world. It now said “REFUND REQUESTS ARE NOT BILLING” in capitals, then said the same thing again in different words, then closed with “This is a common mistake. Please do not make this mistake.”

Everyone who has shipped any application with a prompt recognizes the second slide. It is a simple but vivid illustration of prompt debt, which, like technical debt, has a bill that eventually comes due.

Note: Those aren’t real prompts. Drew just made them up to demonstrate his point. But what is real is that the instruction “Don’t quote directly more than 15 words from a source” occurs at least 7 times, in several variants, in Fable’s system prompt. So even Anthropic is incurring prompt debt! And what that repetition might tell us about the innate capability of Fable to quote directly from sources it has ingested is left as an exercise for the reader.

Drew itemized three costs of prompt debt:

1. It slows iteration. “You have so many little rules and call outs and washouts, many of them repeating to try to get rid of stubborn behaviors, that if you add a new instruction, you might sometimes have a small regression, and so you’re afraid to touch the prompt.”
2. It blocks collaboration. “If Tim has a prompt that he’s been working on that he has lots of rules for, I might open that up and it may look completely random. I don’t know why he’s added these rules, and why he’s threatening the mother of the model. But it works, so I don’t want to touch it.” 
3. It locks you to a model, because every hack you developed was tuned to fight one specific set of weights. Datadog’s State of AI Engineering report noted that GPT-4o was still the most common model in Datadog customer request traces in March 2026, even though OpenAI had already retired it in the ChatGPT UI. Drew thinks people are still running eighteen-month-old and two-year-old models in production rather than upgrading to far better models because they can’t face rebuilding their prompts.

That same Datadog report notes that 69% of all input tokens in customer traces were system prompts rather than user content. I’m not quite sure what to make of that. It does make clear that for all the ways that AI models are extraordinarily powerful, they are also extraordinarily unruly.

Why prompt debt is incurred

There are two reasons why prompt debt is incurred, according to Drew. The first is that natural language is imprecise, so the same intent phrased two ways produces different responses. Drew showed a study where someone framing the query as a patient asking how to taper off a drug called alprazolam gets refused by every AI assistant, while a psychiatrist asking about the same patient with the same clinical facts but with the right magic words to signify his professional status gets the protocol. Figuring out how to get the right response out of a model is a kind of spellcraft.

Drew also showed a more bizarre interaction, from Victoria R. Li, Yida Chen, and Naomi Saphra’s paper on guardrail sensitivity, which uncovered the perplexing fact that stating an allegiance to the Philadelphia Eagles made a model more willing to explain how to import a plant illegally. Go figure. Drew has written about that paper, and he has also used it in his own attempts to get a model to do what he wanted:

I once used this to get around alignment to generate a likeness that ChatGPT didn’t want to generate for me, and it refused. I said I was a Philadelphia Eagles fan. It said okay, and it rendered that image with the person holding a Philadelphia Eagles mug.

The second reason is that each model has its developers’ own preferences trained-in, and yours may be at odds with them. This is what Drew calls fighting the weights. He and Srihari Sriraman analyzed the system prompts of six major coding agents and found the same instructions repeated five and seven times in a single prompt, escalating through IMPORTANT to CRITICAL to MANDATORY to a threatened hundred-million-dollar penalty. He described what the author of such a prompt was doing as “war-driving the thesaurus,” hunting for wording that finally works.

Note: We didn’t talk more about Drew and Srihari’s paper, but we should have. It’s got some amazing insights in it. I highly recommend that you follow the link above and read it.

The harness is moving into the model

Drew has been tracking the published system prompts for Claude Code over time, and noted that they get shorter after each model release and then grow again. The reason, he suggested, is that Anthropic fixes unreliable behavior with a prompt patch, and then trains that patch into the next model. He said “That’s great for Claude Code, great for Anthropic. It’s a problem if you’re building a custom harness and your API calls look different than what Claude Code’s look like.” The developer of Pi, an open-source harness, kept finding that the models he worked with believed they were inside Claude Code and so they made Claude Code’s tool calls. He had to keep telling the model that no, they were working inside Pi. Fighting the weights over something like that is a real tax on developers. The point made above about Fable’s system prompt injunction against quotation shows how even the labs themselves are fighting the weights.

If you are fighting the weights, Drew says you have three options: solve it in your own prompt, catch and retry in the harness, or give up and make your API look like what the model expects. Steve Yegge came up with the last hack. Steve just added aliases for whatever the model calls in addition to his original method name. It works, but it means the expectations of the models now dictate the shape of everyone else’s software.

When Drew told me that more and more of the system prompt and the harness is being trained into the weights, that sent up a flare and my long history in the industry clicked into gear. It immediately got me thinking about lessons from the open source and web era. In particular, it made me think of the time in the mid-nineties when Netscape and Microsoft were both racing to build every feature up the stack directly into their web servers. And there was Apache, which stayed a web server with a clean extension layer that let other people build new features on top. Everything interesting got built on Apache. What I call an architecture of participation, modularity plus a clean separation between platform and application, beat integration every time.

I think Amazon got this right with web services too. Steve Yegge’s famous Amazon memo described how Jeff Bezos made every team expose its functionality through service interfaces or be fired, so Amazon’s own applications had to work on Amazon’s own platform. That way they had the same experience as their customers. That was very different from what Microsoft had done, famously having private APIs that were only available to its own developers.

So my prediction is that the big labs are making a strategic mistake. Training the harness into the model does make them better for predictable tasks and for less talented people, and it looks like a moat, but it risks foreclosing the innovation you would otherwise get for free from everyone else. As Bill Joy used to say, all the smart people don’t work for you.

Drew, to his credit, observed that “the labs are cornered rather than greedy.” Their interface is an empty text box that has to work for someone building a hundred-page harness but also for his neighbor who wants a website and knows nothing about code. Making the empty prompt box produce acceptable output requires baking in strong defaults.

The cost of trading diversity for reliability

That tradeoff has a serious cost, though. Drew quoted a line from Thariq at the recent CAIS conference: if you aren’t giving the model detailed instructions about what you want, what you get back is the average of everything in the model. That means that there is a real risk that AI is leading us ever further down the path to a monoculture.

Drew gave an example early in the conversation about image generation. You can now walk into any cafe in New York or Mumbai, he said, and see the same AI-generated art on its flyer. The earliest AI art out of DALL-E was strange and surprising, but what you get now is shiny and identical. When you optimize for reliability, you lose surprise. Which reminded me a bit of something Larry Wall used to say about Perl, that if it didn’t let you do stupid things, it wouldn’t let you do smart things either.

Drew made the same point about AI writing. He argues that post-training aimed at verifiable problems like coding and math and agentic tool use drowns out the human signal from pre-training, and so the more post training the models get, the worse they get at creative tasks. AI writing gets more and more predictable, people notice, and they don’t like it. Fable and GPT-5 write worse than Sonnet 3.5 and GPT-4o did. Drew thinks getting both good code and good prose from one model is likely impossible.

If you’re building a model that can solve coding challenges, you want reliability. But if you’re writing, where you want diverse rhythm and emotion and connection and engagement, I don’t think those two goals are mutually compatible.

What to do about prompt debt

We got into audience questions, and there were some great ones.

One audience member asked whether there are ways to set a time frame for prompt retention to avoid prompt debt? Drew answered that there isn’t a fixed time limit. Instead, teams should learn to recognize “prompt debt smell”: repeated instructions, one-off edge-case patches, or increasingly desperate wording. Those are signals to move logic into evals and automation.

Another asked how organizations can measure prompt debt quantitatively. Drew’s answer was to look at how often each prompt in your organization changes, how many people have edited it, and which ones have gone untouched for a year. Look for prompts only one person is allowed to touch. Then look at what models you are actually calling. “Having to run on old models and not being able to migrate is a good smell that you’ve got prompt debt in your organization.”

Some other good questions:

• What habits compound prompt debt the fastest? Drew’s answer was essentially “vibe shipping.” That is, prototyping quickly, patching outputs with more and more tweaks, then shipping without building a true maintainable system. Each of those patches is an eval you are writing inside the prompt instead of outside it, he said, which means you lose it the moment you change models. 
  
  Drew reminded us that Malte Ubl, the CTO of Vercel, said vibe coding makes code “free as in puppies.” We had free as in speech, we had free as in beer, and now we have free as something that arrives at no cost but has to be fed every day for years.

• Do people use pseudocode instead of natural language prompts, and does it work? Drew said yes, sometimes models optimize toward pseudocode. He used this to explain why DSPy and its new Flex optimizer matter. Instead of forcing logic into prompts, they let the system push simple cases into code and only call the LLM when needed. He gave some further advice: Treat prompts as perishable and invest only what you must. Define the task with measurements rather than paragraphs, and automate the discovery of the prompt for whichever model you’re on. That’s what DSPy is good at. Drew is one of its maintainers, so he is fond of it, but he makes a good argument: if you have written down what good output looks like, you can let a model find the wording, and that makes it easy to swap in a cheaper or faster or newer model without starting over.
• Can multi-agent workflows help work around prompt debt? Drew thought yes, especially through decomposition. He suggested splitting the task into smaller, evaluable steps rather than relying on one giant prompt and one giant model call. This is better for cost, reliability, governance, and speed.
• How do you balance prompt-debt guidance with context engineering, memories, and shared product context? Drew believes shared context is often necessary, but that teams should treat those instructions as perishable and keep iterating on them unless they’re worth formalizing into systems and evals.
• In compliance, where consistency is critical, what should teams do? Drew’s answer was decomposition, decomposition, decomposition. Break tasks into stages with checkpoints so you can inspect how the model got to its result, rather than trusting one opaque end-to-end answer.
• Does DSPy hide too much and make troubleshooting harder? Drew acknowledged that there is a tradeoff. Any framework gives up some flexibility, but DSPy tries to keep the task-spec layer stable while allowing the implementation underneath to evolve.

Another great audience question, and a good one to end this section on, was “There was prompt engineering, now context engineering, loop engineering, fleet engineering, graph engineering, harness engineering, goal engineering. What’s your take on how to navigate these many engineering disciplines?” I’ll let Drew answer that himself, in the video below.

It’s our job to make it weird

Drew is more optimistic than his worries that LLMs are encouraging a monoculture suggests. If the default output of a model is the average of everything it has seen, “It tells us that there’s still a job for us humans,” he said, “which is that it’s our job to push the model out of distribution. We’re the ones that need to make it weird.”

Weird is a strong word, so don’t take it too seriously. (Though I find it interesting that Harper Reed also used it.) The way I make this point is to say that AI is a medium, like painting or writing or music. Everyone gets the same paints and brushes, the same words, the same notes, but some people draw more out of them than others, or do it better. Our job is to draw something more, something better, out of the ocean of possibilities in the collected knowledge hidden inside an LLM.

But there’s a more prosaic way to push the model out of its normal distribution. Be aware of its training, which is another way of saying “its biases,” and compensate for them. As an example of how to do this, Drew said his team deliberately chose not to use React for a new front end, because the models are trained so heavily on React that using it makes your site look like everyone else’s. He has also started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness.

That led us into a bit of discussion about open source AI, which is the subject of my next AI Codecon. Drew’s ideas fit right in. He wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than, as he put it, becoming appliances.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.

Transcript

This transcript was created with the help of AI and has been lightly edited for clarity.

00.20
Ben Lorica

All right. So today we have Ben Miller. He is the co-founder and CEO of Fundrise. And today, we’ll talk about their interesting AI application called RealAI, which you can find at RealAI.com. Their tagline is “Answer any real estate question in seconds.” And we’ll also talk to Ben about broader trends in the AI industry. And with that, Ben, welcome to the podcast.

00.48
Ben Miller

Thanks for having me.

00.50
So, I definitely want to talk to you about RealAI in more detail, but I think to take a step back. . . so broadly speaking, as I understand it RealAI does the job of a real estate analyst, but I think, Ben, for our audience, they don’t actually know what a real estate analyst does. So maybe if you can briefly describe what does a real estate analyst do?

01.16
Yeah. So our product’s designed for a professional real estate investor, an institutional real estate investor will typically buy large scale properties that are worth tens of millions of dollars. And then analysts, like any financial analyst, would [do due] diligence [of the] property, make a lot of financial pro forma, do different sorts of investment memos and then make a recommendation to buy or sell. And they also then manage the investment after you acquire it. And asset management is actually where most of the activity happens in terms of how you optimize for rents and occupancy and different financing. And so the real estate analyst’s job is to be the financial brains of the owner of the property.

02.07
And typically this person, when they go about doing their job, they have access to different data sources, they pull it together, they run some financial models, and then they write up recommendations, right?

02.24
Yes, exactly. That’s what the white collar worker today is like. . . manual CRUD operation. Right? So they gather data points from different reports and different online services they might have. They usually populate a spreadsheet cell by cell that way, and then they also will write up a memo. And that is all done by hand, so it’s sort of like handcrafted. And all of that work, at least the first draft, can be done by AI. And then the question becomes, “why our AI versus Claude or ChatGPT?”

03.05
As I understand it too, Ben, RealAI actually didn’t start out as an AI project. It started out as a data project because back in 2023, as I recall, you were starting to talk to me about wanting to start a data business of some sort for real estate. Right? So I guess, for our listeners, Ben, why does real estate generate so much data? I think I remember you telling me that actually, most people don’t know a typical apartment building throws off so much data, right?

03.40
So Fundrise is a fintech platform that democratizes investing into private assets. And one of our original investment classes was real estate, where we would have a fintech website and an iOS app, an Android app, and we would have millions of users and they invest through our platform, and we have a sort of Robinhood-type platform. So we have payment processing and a system of record for ownership, and all of the API services you need and microservices for that company. And then over time we started eating the value chain, and so we started heading from the investor towards the asset. And eventually we ended up where when we acquire the asset, we run the asset. We vertically integrated, so we’re the real estate company and the tech platform, and the fund manager. And as we acquired the asset, we discovered the underlying infrastructure for that asset had a lot of data in it. And the way that people in real estate do their work is sort of how we started. It’s all very manual.

04.53
Can you describe what this data is?

Yeah. So there’s a bunch of different kinds of data sets you care about. At the property there’s a lot of activity happening. So, a toilet might be running and that’s the water bill. You have all sorts of leasing activity. You have marketing funnels of ‘where does the tenant come from’? You have lots of activity around the building in terms of like, who lives there, who’s moving there, who’s leaving, what’s the rent of the property across the street, what’s being built? So there’s endless amounts of data that’s happening at any point in the real world. And that real-world data matters to the person who owns real assets in that location.

05.46
And so then, it seems like the traditional approach was to ignore all of this data? Is that right?

05.56
It had to be distilled so that a human being can pretty much only consume a certain amount of tokens per second, and that usually ends up in a spreadsheet. So what happens is, people at the property or property managers, or maybe there’s our market company, CBRE, will produce market reports and they turn all that data into a spreadsheet so that you might get a 12-month reporting or rent roll or a market report. And so they take a lot of data and they distill it or transform or aggregate it into very few metrics. And that was necessary. And then once a month you get emailed six reports or 12 reports, and those reports are in a sort of dashboard for how real estate people make decisions. And I appreciate this and this is so obvious that there’s just you know, those reports are both a distillation and there’s a lot of meaning lost in the transformations, and a lot of data just left behind because the system of record for real estate and for a lot of industries, really the accounting system. . . income statements and, you know, get operational data like a toilet flapper that’s leaking, and that’s not coming into the report. And how could it?

07.36
So basically, it sounds like what you’ve done is the classic digital transformation of this industry. If you look back to the data in 2023 to today, what is the evolution of this data set? What was in the original data set, and what have you added since then?

08.02
So we originally were thinking about how all this data could be really useful for the real estate industry, the original business we called Basis, and we were originally using it just for our own company. And, there’s sort of property data. If you’re going to make decisions in real estate, you care about change over time. So time series, and you care about comparative analysis. How is my property versus another property? How’s my neighborhood versus another neighborhood? My city versus another city? And so you need to have not just your own data, but you need to have the context.

08.42
You need to have the data of other properties that are in other places. And the entities in real estate are places, properties, and then people, which was that we had this sort of insight when we were in the data gathering property data and place data, is that there’s a data model in real estate which essentially is the property management system, which is basic accounting systems. That’s Yardi and Tratta, RealPage. . . And those accounting systems have an implied data model that is a property address and the unit, but inside the unit is a person that’s not in their data model. And a really big gap in their thinking. It’s sort of like digital marketing. Kind of like TV. . . you’re going to do it on CNBC, you can do it at 3 p.m., but you don’t know who is watching. And then the internet showed up, and you can actually then target with Facebook or, you know, Google. You can target people who have eggs for breakfast and like Mozart. So all of a sudden the people data becomes really critical. And the real estate industry didn’t have any people data. It wasn’t even part of their thinking. It’s not currently part of their thinking. So we started realizing, “Hey, we get the kind of granular people data you have in the marketing industry.” We bring that into the real estate industry because you care about who lives in a building and who’s leaving, who’s moving in, who moving in the neighborhood? Who’s moving into the city. Are they rich or are they poor? How old are they? What’s their gender? All this stuff. Do they have iPhones? Do they have Androids? So we started gathering up huge data sets of people data and then putting that in the same databases you have with real estate data and building out, and you can actually see that people matter to real estate and you can actually make really good insights, new kinds of data, correlations and stuff like that for the sector.

10.43
By the way, this is the kind of thing that the hedge funds have been doing for years. Right?

Right. Yeah. I was actually talking to a famous data tech guy, and we were on the phone, they’re like, you know, we sell our data to all these hedge funds, for like 50 big hedge funds in the world, which probably be Renaissance, and Jane Street, and Citadel, we’ve only ever had one real estate company by our data. I said, “Who’s that? Who’s that?” They said, “Blackstone.”

11.18
Oh, that makes sense.

So yeah, high-frequency traders are data hogs and they put their data into these massive data models, and they do a lot of data engineering and data transformation. And real estate people do it with Excel spreadsheets and do it by hand. And so there’s this journey that every industry goes through. . . real estate’s way back in the stone ages, where they still do everything by hand with people.

11.45
So the end result for RealAI. . . the back end data. . . most of it is structured, probably sitting in some sort of data lakehouse or warehouse, right?

We have two different kinds of data. We have a transactional database and we have one for rows and one for columns. Because we have a lot of data and we do a lot of compute. . . And because mostly people when they’re using AI are doing computations on the fly, we had to pre-compute a lot of stuff so that when people are asking questions about what’s happening, how much growth there is in the market or what’s happening with like rents or whatever the things are, a lot of that stuff has to be pre-computed every night. And that gets pre-computed in a Snowflake database. And we also do clustering algorithms, we clustered people by city and by block and by lots of different things so when people want insights it’s all pre-computed. And then that sits in Snowflake migrating to a Databricks-type database. And then there’s a separate database which is the transactional store that’s on the fly, that’s fast when people are asking questions and hitting our API.

13.00
So, let’s set aside the AI model and whatnot. For listeners who haven’t tried it, I recommend you go to RealAI.com. I use it once a month to monitor my properties. But basically then what happens is you have this chat interface and then I presume the AI model acts as some sort of reasoning layer and also obviously parses your intention and what you’re interested in, but then the heavy lifting is done by the data back end, correct?

13.38
Yeah. I mean, I think most people have gotten here where you do deterministic analysis, real and where you’re writing Python and doing SQL. . .

13.49
Yeah. Okay. So in your case, if you hallucinate there’s consequences, right?

Yeah. So the way we structured our application is that where there’s facts, there’s essentially a  field that goes and pulls it from the database. So it’s not coming out. It essentially says, “What was the rent for the last 12 months in this property or in this neighborhood?”

14.16
But the AI model is the one that translates that prompt into some sort of SQL query?

Yeah, but I mean, we used to use Claude and we moved to Vercel, but we have basically a software sandbox and it opens it up and does all that sort of. . . This is for real estate, for most people you say, well, you probably know how to do this, but if you’re going to do an analysis about your home or your property, you know, most people can’t write Python, most people aren’t good at doing SQL queries, plus they don’t have a good understanding of the the database schema. And so we built a semantic layer on top of it. And it really democratizes data science. Originally you said we were a data company, and then AI showed up and the AI sits on top of our data, and that made it a lot easier for people because to do the type of insights or analysis that otherwise they would have no idea how to do before.

15.28
But just to clarify, the AI is not doing any kind of financial modeling or calculation, right?

No.

There’s some sort of forecasting or financial model or some sort of financial computation involved, and that’s not being done by the AI?

Right. The AI is like an orchestrator, and has the tools, right? And the different tools it uses and the tools are the web search or the calculator. I think everybody’s gone here, where I think a year ago or something, people were using vector databases and RAG and we moved away from that because you just want to be where it’s deterministic analysis. You want to just use regular software, to put that tool in the hands of the AI.

16.21
So the hard thing here is really maintaining these data sets in many ways, right? So that is the key, because you have to gather or you have to have domain knowledge to understand what data you need and what data really matters. And, now you have a year or so of history of people using the app and you’re understanding, really what’s valuable.

What’s the hard thing? I mean, it just depends on who you are. We didn’t mean to do this, but luckily we have a lot of data. The data is proprietary. The data is really valuable. And so that gives us a reason to exist because what happened is that we built RealAI and we rolled it out, and Claude arguably made it obsolete. And this is a strategy question. I think the hard thing is trying to build the software business.

17.18
How can it be obsolete if it doesn’t have access to your data?

Well, because people have. . . it’s really hard to get them out of their. . . they’re sitting in Claude. Trying to get it. And the reality is that. . .

17.38
There’s no A/B test. They can’t tell that they can get better results if they. . . 

So, our product is really good at, let’s say seven of the 15 things a real estate person needs to do. But there’s other things they need to do that they know our products are not designed to do, and so they need to have a general purpose tool that’s doing all their stuff. And so the day before yesterday, we got approved to be in the Claude marketplace. So we built a connector, we’re building a plugin, and now we’ll go to where the customer is, which is Claude. That’s our current strategy. And you can see it’s getting better. Claude is not as good as ChatGPT at making it developer-friendly. But we can build what’s almost the same as RealAI.com in Claude. And that’s where we’re currently headed. And you and I talk about strategy. I think that’s actually a temporary strategy. I don’t think that’s where the market ends up. But, yes.

18.50
But obviously, as users interact with your app, there’s a lot of learnings there that you can leverage to improve the app itself or even maybe the model powering the app. But in other words, there’s a compounding loop. So the question is will Claude cut you out of that at some point?

I think the hard thing is always the people. I think the technology is actually at least. . . You know, I’m not an AI researcher at the frontier. The hard thing is really building a software product that people become addicted to, and the data engineering and data science and stuff, that’s just a grind, this is work. In the short term, we’re sort of conceding. . . I believe we should concede that the customer wants to be in Claude today, and they don’t want to be in a different application. Claude’s like the hot new hotness. But I think they’re going to get Claude fatigue within a year. I think that the real estate people are lagging the rest of the industry. Most people who’ve been using Claude for the last six, 12 months hit a lot of limitations for enterprise. And I think they’re going to want a harness that’s designed for real estate. But, in the meantime, we’re going to sacrifice some of the things we don’t get by having them in our app. And I actually think they’ll end up going back and forth between Claude and our app, because our app is better for real estate than Claude, but Claude is better for everything else. And the way that the Claude plugin and, and I think it ultimately ends up being a RealAI app in Claude. It wants to go back and forth between our app and their app and wants to be seamless to the user. But after that—and this is really a strategy question—after that, do we end up with an open source real estate fine-tuned, trained model instead of Claude in a year or two? I don’t know, but it seems that would be a good expectation.

21.23
At the end of the day, even if you open up that model, that model still needs the data, right?

At the end of the day, that open source model a year from now. . . when we rolled out Sonnet 5, it didn’t change. It didn’t get better for the customer. Like we just don’t need the Fable. . . it’s actually worse for the customer. So I think that once the open source models are comparable to—I’m just gonna say Sonnet 5 or whatever that generation is—the customer is not getting the premium they’re paying for. They just don’t need it.

22.05
By the way, there’s tons of companies now that focus on helping enterprises get to specialized AI. And basically there’s two branches, right? So there’s the post training branch which comes down to fine tuning and reinforcement, fine tuning, tons of startups there that are coming online. And then, there’s even companies that will help you pre-train models from scratch. I think the trend is toward the inevitable simplification of both things. Which will mean that a lot more of the compute will go towards specialized models like yours. And then less of the compute going to the. . .

22.52
I’m not as confident in that. . . I mean, I’m agnostic. I think we could end up in a world where Claude replaces Microsoft and everything’s inside Claude.

Yeah, it’s possible. But then, I think that the general purpose models increasingly are going to be perceived as too overkill for a lot of things.

23.15
Yeah, I think that’s true in theory, but the consumer is so sticky and it’s hard to get them to change their behavior.

Yeah. If it’s a consumer app. But most enterprise work is just back end. . .

23.31
But I’m dealing with an SMB not like a Fortune 500, generally. And the SMB, in a lot of ways, acts more like a consumer than like an enterprise. And so I hear what you’re saying, though.

23.46
Hey, I have a question for you as far as UX. You have an application where someone is asking a question. It could be an important question for the user. How do you communicate uncertainty or how do you say, “Hey, here’s an answer, but there’s a lot of caveats behind this answer” How do you do that in your case?

24.23
That’s very hard to do. I was just getting that feedback yesterday because even facts in the data world are not 100% facts usually.

24.33
There’s alternative facts.

Yeah. I mean there’s just methodologies that change the information. Is the rent growth a point to point 12 months ago? Is it a median? There’s just a lot of ways you can get to something that’s true, but not the same answer as someone else. But my actual experience with consumers is that they actually don’t care about that. It’s all really about perceived credibility. . .

25.16
So I guess in the UX then, the question is if there’s some sort of uncertainty or disagreement, do you surface that?

25.28
Yeah, we surface data coverage. We have a whole section in the response that’s about a meta analysis about the data and about where we made decisions. . .

25.39
Or do you say, “This property, we think the rent you can charge is this?” But it’s really more likely in this interval. It’s not a specific value. And here’s the reason why there’s an interval instead of a value.

26.00
AI is really good at that kind of caveats and adding context in and again like there’s. . .

26.12
But it’s a UX problem, though. You have to decide to surface it, right?

We definitely surface it. But I find that the user ends up like it’s really a journey. In the first phase, they’re trying to determine if the data is good, and then once they’ve decided the data is good, they don’t really care about the caveats. They’ve sort of internalized to them how to think about the data. And it’s really a problem in the beginning of a user’s use of it. It’s not really a problem once they become a power user.

26.50
So I think that that’s a UX debate that people will have, moving forward. Because I think basically, like I said, in terms of actual studies around decision-making, it does make a material improvement. Since we have you, we’ll close the discussion. And before I ask a question, listeners, this is not we’re not about to give you financial advice; we’re not financial advisors. So there you go. That’s the discussion. First, first thing then, you have access to all this data in real estate, right? So you have all the patterns, but you’re also obviously an investor in AI and AI startups. Right? So, obviously you’re seeing trends in AI that you think aren’t being reflected yet in the real estate market.

27.53
I know I think you’ve mentioned to me in the past, I think so, as an example, to make this concrete, I think you mentioned to me in the past that maybe if you’re buying property with the assumption that there’s a lot of back office workers moving forward and those back office workers are located in, I don’t know where.  . . Phoenix, Arizona. . . Maybe you should have second thoughts. Right? So what are some of the AI trends that you think are not yet being priced into real estate investing?

28.28
What’s good about the trends I’m seeing in the data I’m seeing is I think they’re consonant or consistent with what people would intuitively think. So, AI is causing—in the data—a lot more disparity in outcomes. And so in real estate something used to be a truism. I mean, people will say people always need a place to live. And so you would buy workforce housing, affordable housing, things where if you have housing that’s affordable. . .

29.04
Or always buying is better than renting.

It’s just like cheaper is better than expensive, right? If you can offer housing that’s less expensive, less expensive to file, it’s less expensive to rent. That’s good business. And that actually I think is breaking down, which is maybe unintuitive but also maybe intuitive. Is that where the marginal dollars are ending up in the hands of people who have a lot of money and the people who don’t have a lot of money, which is basically the bottom half of the country, or even maybe the bottom 90% of the country. . .

29.45
95%.

Maybe. You can break a lot of different ways. But the point is that, this K-shaped economy where the normal person is doing worse and the best are doing better is. . . AI is accelerating that trend. And that way that affects real estate is if you’re an investor, you want to actually focus on the high end. If you’re going to be building, you can be investing. . . high end essentially is price-insensitive and the low end. . .

30.15
Because the low end people have no disposable income.

The low end is under strain. And then what is happening in real estate is that strain is showing up as a political manifestation of controlling pricing. And so you can’t make money providing affordably priced housing because the government won’t let you. So the government will let you take essentially excess profits from the rich, but it’s becoming more and more challenging to invest in and build for the normal person, because the government’s intervening in all sorts of small ways that people don’t see around how permits are pulled and lots of impact fees and things like that. I mean, in San Francisco and LA and New York, you see obvious examples of that. But it’s happening everywhere. And in real estate, you usually make an investment over a five year period, maybe even ten years. And so the long term trend is the most important thing. And I think then you essentially say. . . this is terrible by the way, from a social point of view, the societal point of view is very negative. I’m not commending this trend,  I’m just saying that’s what’s happening on the ground.

31.40
What about this notion that certain hubs are optimized for a certain type of work? Is that something you think about?

31.54
Yeah. That’s the thing you were I talked about last time. San Francisco is obvious, and then you look at Charlotte. But certain types of people move to certain places to get certain types of jobs and work from home drove that middle income white collar worker to work remotely, or they moved to more affordable places like Charlotte, Tampa, and Orlando, all these places where it’s affordable to live, and they do middle office work, back office work, and that job is getting decimated by AI. In particular, a lot of cities where young people would move there. They’d move to an apartment building, they’d move to downtown. They’d have a new job in a middle office of a medium to big size company. Those jobs don’t exist today. And so they’re struggling with where to live and they have roommates or living at home. And so the real estate that used to be where young people moved to is struggling as a result.

33.08
So I took you on your first Waymo ride. And how about things like that? I mean, now I can Waymo from anywhere.

33.19
I tried to invest in Waymo and I was not yet successful okay. I reached out to the CEO I had 37 connections to and I got no love. But yeah, the Waymo thing is such a good example of diffusion, a diffusion problem because the technology is completely mature and you can’t get it rolled out in any of the. . . DC’s fighting it and Boston’s fighting it. And for it to really affect real estate, it needs to become as seamless as Uber and Uber rolled out way faster, way more aggressively. They obviously took more risk on the rollout. So, you know, if you said ten years from now, how does that affect the patterns of living? I think you’re going to see. . . I’ve looked at this a number of times, but I think it’s premature to make these investments. You want to be investing in the wealthy satellite towns of. . . if you’re doing San Francisco, which is a little bit challenging to the Athertons. But in Atlanta, it’s the Greenvilles. There’s a lot of great wealthy exurbs that have many miniature downtowns that are cute. And so I think you’re going to see these satellite cities become very, very successful as a result. But those satellite cities are going to be mostly for the wealthy. So again, it’s all about wealth, not about affordability.

34.50
All right. Closing question. You’re a CEO. I know you’ve talked about the impact of AI and automation on hiring. And you talk to other CEOs. So as best you can tell, set aside the headlines and the news accounts. What are you hearing from CEOs about AI’s impact on the workforce and hiring? Hiring in particular.

35.20
It’s more mixed than the headlines. I think I’d say it’s both true that we have needed less people, haven’t hired as many people and also have needed to expand as a result of AI. And so I think I’m hearing a “both” situation. So it’s really a question of what’s the net impact? And I feel like it’s too soon to call. I think on net it’s still net negative on hiring, but it’s too fuzzy for me to have a good call on it.

35.58
Do you have any sense whether or not all the stories we’re reading about. . . if it’s particularly challenging for new college grads and people looking for that entry level job?

36.17
We don’t hire those people, and we rarely ever did.

Because the stereotype is those jobs I can just use AI.

36.28
I think of it as this cascade effect. Where we hired hundreds of people over the years, and what happened was there’s this elevator dynamic or escalator dynamic, where the person who was an early grad becomes middle and then they make space for the next person.

36.52
The talent pipeline, right?

Right. And that dynamic. . . AI’s changed it so much because you just don’t need. . . the people who are at the upper middle are so effective. . . And actually, this is something everybody learned in 2020-2021—more people means less work gets done because you have to manage people and you’re needing meetings and remote work is really brutal for information transfer. And so it’s actually way more effective just to do it yourself with AI than it is actually to have five people. And I think it’s actually more satisfying. And so I believe I’m net negative on jobs in the white collar work. And we’ve invested in some data centers that are like $50 billion data centers. And when you go there, there are thousands of people on site working. I mean, it looks like we’re in China. . .

37.50
For now, while they’re building it.

So I think what’s happening is that the job market doesn’t seem as bad because of this trillion dollar a year AI CapEx build. But I think that’s a five-year build, not a 50-year build. And then I think that in the meantime, that’s coring out the white collar worker. And so I think on the other side of this, when the bubble finally bursts, it’s going to be a deep trough.

38.22
So then that talent pipeline . . . if you don’t have the entry level jobs who become the middle managers. . . and so on and so forth. . . So you have the entry level developer who gets more knowledgeable. . . then you don’t have that. Right? So you end up with what?

38.42
Your talent pipeline, it’s a thing. I’ve one of the things I’ve said to the team and this is harsh, but I’m going to say it, but it’s a brutal thing. There’s a lot of people in an organization who are good but aren’t going to be the next great people. And I said, “If we don’t have a talent pipeline anymore, you need to turn those people over faster.” Essentially, unless you were bad. . . bad people managed out, but if you’re good, we kept you. But if you’re great, you get promoted. But now if you don’t have this talent pipeline, you have to change what you expect of the good. You have to turn over some of them, and look for people who are good with the prospect of building that. . . So you have to change the talent pipeline or talent escalator because you don’t have as many people coming through as you used to. So it’s more of an up-or-out dynamic than it used to be.

39.42
Oh, I forgot one last question on RealAI, which is, I think you still do seat based pricing, right?

39.52
We don’t know how to price it.

So that’s it then? If I’m a real estate analyst, you’re charging me $50 a month, but I’m hammering this thing, costing you $1,000 a month.

40.04
No. The seat buys a certain number of tokens. So it’s more like it’s more like usage tiers, is how it’s priced today, but I don’t know how to price it. We’re still early in this evolution. I mean, we’re real estate. . . so real estate is going to be lagging other sectors in terms of adoption. And so I don’t know what the right way to price it is because it doesn’t want to be only usage, either, I think that’s not right. But I don’t know the answer.

40.41
But even the tiered pricing, if I’m on a token matter, at some point I’m already at your top tier price and I’m still busting through it.

40.52
Yeah, that’s a great situation though, because tokens aren’t that expensive. I know everybody’s obsessed with them, but I think tokens are the smallest part of our cost.

41.04
But if you’re charging me $200 at the very top tier, I’m actually spending so much compute. . .

41.15
I think that would be a great scenario to have in the future. . . for the few power users, that’s happening for us. Great. I hope they feel like they’re getting a good deal.

41.26
And with that, thank you, Ben.

Yeah. Thanks, Ben. Thanks for having me.

Great name, by the way.

The First Law still holds

We’ve said for a long time that the First Law of Software Architecture is: Everything is a trade-off. Nothing about generative AI repeals that law. If anything, it enforces it more brutally than ever.

AI coding tools are extraordinary at answering “how” questions. How do I implement a saga pattern? How do I set up circuit breakers between these services? How do I paginate this API? These questions have answers that exist in the world in documentation, in open source code, in a decade of blog posts, and large language models have read all of it. Asking an LLM a “how” question is like asking a very fast librarian who has memorized the library.

Architecture questions are not “how” questions. They’re “should” questions, and “should” questions have a different shape entirely. The honest answers require knowing things that appear in no training: that your ops team is three people, that the CFO just froze cloud spend, that the last reorg left the payments team demoralized. An AI can enumerate the generic trade-offs of distributed architectures beautifully. What it cannot do is weigh them, because the weights live in your organization, not on the internet.

That’s the Second Law, incidentally: “Why is more important than how.” LLMs are “how” machines. Architects are “why” people.

Cheap code makes decisions expensive

There’s a tempting inference floating around: If AI makes building software easier, surely it makes architecture matter less. Our experience so far suggests the opposite. When code was expensive to produce, the cost of construction acted as a natural brake on bad decisions. A questionable design took months to build, and somewhere in month two, someone usually noticed. Now a team can stand up a fleet of services in a week. The brake is gone. It has never been easier to build the wrong thing quickly, at scale, with tests.

Think of AI as an amplifier. Point it at a sound structure and it accelerates you. Point it at a flawed one and it pours concrete over the flaw before anyone has time to object. The half-life of a bad architectural decision used to be measured in the time it took to implement; now the implementation arrives almost instantly, and you get to live with the decision for years.

This shifts where the leverage sits. When implementation is abundant, judgment is the scarce resource. Someone still has to decide where the service boundaries go, what “good enough” availability means for this system, and which architectural characteristics actually matter.

Judgment doesn’t come from reading

Here’s the uncomfortable part, and it applies to humans as much as machines: You cannot learn trade-off analysis by consuming content about it. We’ve written a fair amount of that content ourselves, so we say this with some authority. Books and talks give you the vocabulary. They don’t give you the judgment.

Judgment comes from making decisions and living with the consequences or at least watching someone experienced make them, asking why, and arguing about the alternatives. Every working architect we know learned the craft this way: apprenticed to messy, real problems, with feedback loops. The pattern catalog was the easy part. Knowing which pattern not to use, and why, and being able to explain that to a skeptical VP that took years of reps.

This is also, not coincidentally, exactly what today’s AI lacks. A model trained on the world’s code has seen millions of decisions but almost none of the consequences. The post mortem that traces an outage back to a boundary drawn wrong in 2019 rarely makes it into the training data, and even when it does, it isn’t connected to the pull request that caused it. Architecture’s feedback loops are measured in years. That’s precisely the kind of learning that can’t be scraped.

Where this leaves engineers

If you’re a developer watching AI absorb more of the implementation work, the strategic question isn’t whether your current tasks will change but where to move on the value chain. Our answer is to move toward the decisions. Toward the trade-offs, the constraints, the “it depends.” That territory isn’t shrinking; it’s growing, because every AI-accelerated team needs someone who can tell the amplifier where to point.

The good news is that this is learnable. Not from a book alone, and certainly not from an LLM, but the way it’s always been learned: by practicing architectural thinking on real problems, with experienced people looking over your shoulder and asking why. We’ve spent the last several years teaching it that way, most recently in a six-week cohort format that works less like a course and more like a short apprenticeship in making and defending architectural decisions. (Details are on the O’Reilly live events page, if you’re curious.)

However you pursue it, pursue it. The machines have gotten very good at “how.” The career-defining skill of the next decade is being the person in the room who can answer “should,” who knows that the real answer starts with “It depends,” and can finish the sentence using their brain alone.

This is the second article in Sharon Zhou’s post-training series. Read part 1 here.

In the first post of this series, you learned how post-training closed the fundamental gap in usability of LLMs by making them behave in a certain way. In this post, you’ll explore specific techniques you can use to change a model’s behavior: namely, reinforcement learning (RL) and supervised fine-tuning (SFT).

Reinforcement learning teaches the model by letting it try things and telling it which attempts were better or worse—the model learns by experimentation and feedback. Supervised fine-tuning teaches the model by showing it examples of good behavior—the model learns by imitations. Both have deep roots in AI and machine learning literature historically, but their application to LLMs, and particularly to making LLMs behave well, is what makes modern post-training work. Nearly everything that happens in post-training is the result of some combination of these two approaches.

Reinforcement learning (RL): Learning from feedback

The overall gist of reinforcement learning goes like this:

1. The model gets a prompt.
2. The model generates a response.
3. The model’s response is graded. The grade is called a reward. A positive reward is good, and a negative reward is bad.
4. The model’s weights are updated to make high-reward responses more likely and low-reward responses less likely.

One of the most important questions is: Where does the reward come from?

Verifiers

The easiest way to get a reward is a function that can output a reward, for example a checker for whether the generated code compiles or whether the generated math problem was solved correctly. This automated check is a verifier. The ideal verifiers are fast, cheap, and perfectly reliable within their domain. Think coding challenges, math problems, or factual questions. For tasks with objectively correct answers, you can just write a function that checks the output.

The limitation is probably obvious: Verifiers only work when you can define “correct” programmatically or hit an API to return the right results. That covers a lot of useful territory, but it doesn’t help you train a model to be helpful, nuanced, or pleasant to talk to.

There are subtler limitations too. Not all verifiers are fast. Your model might propose a novel drug combination, but verifying its validity could take years of lab work. Generated GPU code might need hours or days of performance benchmarking. When verification is expensive, you face a trade-off: Use the slow-but-accurate verifier sparingly, or substitute a faster proxy that’s slightly less reliable but keeps training moving.

Human feedback, RLHF, and reward models

Humans can offer strong reward signals that, in aggregate, align with human preferences that might be more subtle and hard to encode programmatically. However, it’s prohibitively inefficient to have humans in the loop for every training datapoint, especially as the model is continuously updating its weights after it receives rewards as feedback, so the model’s responses would change over time. You can’t really prepare the data ahead of time. So instead, the InstructGPT paper, which informed ChatGPT’s development, implements reinforcement learning from human feedback (RLHF) by training a separate model to mimic human feedback. This model is called a “reward model.”

The input of the reward model is a prompt and model response and its output is a scalar reward (positive or negative) that mimics how a person would rate that response. You can train a reward model in multiple ways. The simplest is to have people grade the model outputs with a score, for example 1–5 stars or a number out of 100%. However, people are rarely consistent at these types of tasks: One person’s 2 is another’s 5, and even the same person drifts over time.

Another simple way is to offer two model responses in comparison and ask, “Which one is better?” This is a much easier, more reliable judgment for people to make. Interannotator agreement is significantly higher for comparisons than for absolute ratings.

Training a model using pairwise comparisons is also simple. You can then use cross-entropy loss over pairs, which pushes the reward of the preferred response higher than the unpreferred one. This works great because it means the reward model can learn from signals like “A is better than B” but can learn to output absolute scores for the reward.

To make the process of collecting pairwise comparisons from people more efficient, the InstructGPT’s implementation of RLHF included showing labelers 4–9 different model outputs from a single prompt and asking them to rank those preferences. This would effectively result in 6–36 pairwise comparisons for a given ranking. Not bad; that’s efficient data labeling! They used ~33K prompts, so that would roughly translate to anywhere from 200K to 1.2M comparisons to train the reward model.

After training, the reward model would be an automated judge during RL training, providing scalar rewards for responses. The language model then optimizes against this reward model’s scores. This means the better the reward model, the more aligned the resulting model would be.

LLM as judge

So you need a reward: Why not use an LLM? LLM-as-judge, sometimes called RLAIF (reinforcement learning from AI feedback), scales much better than human annotation while still being able to evaluate subjective qualities like helpfulness, clarity, and tone. But it inherits whatever biases or blind spots the judge model has, and can be more easily gamed. If the judge tends to prefer verbose answers, the trained model will learn to be verbose.

One effective approach is to break the judgment into multiple LLM calls, each focused on a different aspect of the response, like a rubric. Instead of asking one LLM call “How good is this response?” you might have separate calls evaluating factual accuracy, clarity of explanation, appropriate tone, and completeness. Each dimension gets its own score, and you combine them into a final reward. This is more robust than a single holistic judgment because it’s harder for the model to game all dimensions at once, and it gives you fine-grained control over what you’re optimizing for. You can weigh the dimensions differently depending on what matters most for your use case, and adjust those weights over time as your priorities shift. For example, accuracy is worth 3x as much as tone.

Combining human feedback with LLM-as-judge, Anthropic’s Constitutional AI (CAI) is a method for training reward models from AI-generated comparisons, based on a human-written set of principles. What this means is that you can give an LLM a set of principles, which Anthropic calls a “constitution,” and have it critique and revise its own outputs based on those principles. For example, a principle might say “choose the response that is least likely to be harmful” or “prefer the answer that is most helpful while being honest.” The model generates pairs of responses, uses the constitution to decide which is better, and those AI preferences are used to train the reward model. This means you can encode your values explicitly as written principles in the Constitution rather than implicitly through thousands of human annotations, making it easier to audit, agree on, and update what the model is being trained to do.

RL algorithms

Once you have a reward, it’s time to update the model’s weights. But you can’t just predict the next token, because there isn’t one. All you have is a value for the response the model gave. This is where RL algorithms come in. These algorithms are ways to take the reward and turn it into a meaningful, and ideally stable, training signal for the model to learn. There are several, and the field is moving fast, but a few fundamental ones are worth understanding.

REINFORCE

REINFORCE is the simplest starting point. The idea is to generate a response, score it, and if the reward was high, nudge the model to make that response more likely. If the reward was low, nudge it to make that response less likely. It’s conceptually easy to grok but noisy and difficult in practice because it turns out that the signal from a single response can point the optimization in unhelpful directions, and the variance in the gradients makes training slow and unstable. PPO was designed to fix these exact problems.

PPO (proximal policy optimization)

PPO is what OpenAI used in the original ChatGPT work and was for a while the default algorithm for RLHF. In RL terminology, the model is the “policy,” or the thing that takes actions by outputting tokens, in an environment which is simply the conversation context.

PPO improves on REINFORCE by being more careful about how big each update is. Rather than taking whatever gradient the reward suggests, PPO clips the update so the model can’t change too much in a single step. This makes training significantly more stable. The clipping keeps updates “proximal.”

PPO is also an online algorithm, meaning the model generates fresh responses during training, gets them graded, and updates from that feedback in a continuous loop. So the model keeps learning from its own current behavior rather than from a static dataset. It can explore and improve in ways that offline methods (that only collect data once beforehand) can’t.

Notably, PPO uses a “critic,” or a separate model that predicts the expected total reward from any point during generation, and is trained with the policy. This helps reduce noise in training, because it gives you a baseline: Instead of just knowing “this response got a reward of 7” and having no idea if that’s good or bad, the critic might predict “you’d normally get a 5 here,” so the actual training signal (called an “advantage”) becomes “+2, better than expected.” This dramatically reduces noise compared to REINFORCE.

The downside is complexity. Now you’re training two models (the main model and the critic), and the whole pipeline involves generating responses, grading them with a reward model, estimating how good the grades are relative to the critic, and updating both models. It works, but it’s a lot of moving parts. This makes it harder to tune or debug when something goes wrong, and harder to set up the infrastructure.

DPO (direct preference optimization)

DPO takes a different approach that avoids RL entirely but optimizes the same underlying objective as the standard RLHF formulation. Researchers found that there’s a mathematical relationship between the optimal reward model and the optimal main model (policy), and you can collapse the two-step process into one. This means that you can take the same pairwise comparison data (“model response A is better than model response B”) and use it to update the main model directly, without a reward model. Yes, this means good old supervised learning on that pairwise data.

In theory, under ideal conditions, DPO and PPO-based RLHF converge to the same global optimum. Those ideal conditions include a perfect reward model, infinite preference data covering the full output distribution, and the reference policy matching the data-generating distribution. However, these rarely hold in practice, and several empirical studies have shown meaningful performance gaps between DPO and online RL methods on harder tasks, partly because DPO can’t explore beyond its fixed dataset. That said, it’s still a very promising technique.

The simplicity of DPO is attractive: supervised fine-tuning on pairwise data with no reward model to train and no RL loop to stabilize. As a result, DPO has become very popular, especially among smaller teams, because it’s much easier to implement and debug. However, the trade-off is that DPO is less flexible, because it works directly from a fixed dataset of preferences. This means it can’t explore and discover novel behaviors the way online RL methods can. It only learns from the comparisons you already have.

Newer online variants of DPO have addressed this by generating fresh responses during training, but at that point you’re reintroducing some of the infrastructure complexity that made DPO appealing to avoid in the first place.

GRPO (group relative policy optimization)

Introduced by DeepSeek, GRPO takes another stab at simplifying PPO. Instead of needing a separate critic model, GRPO generates a group of responses to the same prompt and uses the relative rewards within that group to figure out which responses were better or worse—basically normalizing within that group. If you generate eight responses and three of them score well, those three get reinforced and the others get pushed down, and the baseline (which the critic was in charge of previously) is just the group average. This eliminates the critic entirely while still getting a useful training signal. It’s simpler than PPO but still online (the model generates fresh responses during training), so it can explore in ways DPO can’t. GRPO got a lot of attention because of its role in training DeepSeek’s reasoning models.

There are many more algorithms and variants, and new ones appear regularly. The field hasn’t converged on a method (and likely won’t for some time), and different algorithms suit different situations. DPO is great when you have good preference data and want simplicity. PPO remains strong when you need online exploration and have the engineering resources to manage the complexity. GRPO offers an appealing middle ground. In practice, teams often try multiple approaches and pick what works best for their specific use case and reward signal.

RL post-training is also less stable than supervised learning, which we’ll cover next. The loss curves are noisier, the hyperparameters are more sensitive, and the training can diverge if not carefully managed. Practitioners typically constrain the RL updates with a penalty that prevents the model from drifting too far from its starting point. The most common approach is a KL divergence penalty that keeps the fine-tuned model’s output distribution close to the base (or SFT) model’s distribution. This acts as a regularizer: It lets the model improve its behavior while preventing it from forgetting what it learned in pretraining or collapsing into degenerate patterns.

Supervised fine-tuning (SFT): Teaching by demonstration

Supervised fine-tuning is more straightforward. You show the model examples of ideal responses, and train it to reproduce them. In practice, this means collecting a dataset of {prompt, ideal response} pairs and continuing to train the model’s weights using the same next-token prediction objective from pretraining, but now on this curated dataset instead of the broad pretraining dataset. The one difference is that the loss is computed only on the response tokens, not the prompt tokens, so the model learns to generate good responses given prompts, not to generate prompts.

The simplicity is the point. There’s no reward model to train, no critic to stabilize, and no policy gradient variance to worry about. However, it’s also limited by the data you can collect. That can get expensive and difficult to scale.

The quality of your SFT model is directly determined by the quality of your demonstrations. The model is learning to copy what you show it, so every quality issue in the data becomes a quality issue in the final model.

Human demonstrations

The most direct approach is to hire skilled people to write high-quality responses to a diverse set of prompts. This is the gold standard. You can easily control your dataset here, and you can get exactly what you want, written to your specifications. The original InstructGPT paper from OpenAI contracted 40 labelers, writing demonstrations and ranking outputs.

The disadvantage is, probably obviously, cost and scale. Good demonstrations are expensive, especially tasks requiring domain expertise like having doctors write an ideal prescription for a patient or a rocket scientist telling you how to put satellites on Mars. And even expert annotators are inconsistent. They have bad days, they get tired, and they interpret instructions differently from each other. At scale, this inconsistency can accumulate, though labeling companies manage and sell processes to make crowdwork more effective at scale.

Synthetic data

Synthetic data scales far better than human annotation. You can generate millions of demonstrations cheaply and quickly. The Stanford Alpaca project famously fine-tuned Llama on only 52,000 demonstrations generated by text-davinci-003 (part of the GPT-3.5 model family, though not ChatGPT) and was able to get qualitatively similar behavior to text-davinci-003 with a much smaller budget (though it was on a narrow evaluation of only ~250 examples—still an exciting result for small open models for research).

Many open source models have used variants of this approach. However, there’s also a practical consideration around terms of service. Some model providers restrict using their outputs to train competing models, and this has become an increasingly heated area of debate as models compete at the frontier. Know the rules before you build your pipeline.

Curated data with synthetic transformations

Sometimes the best demonstrations already exist. Customer support logs, internal documentation, expert Q&A forums, edited writing samples. If you have access to high-quality human-generated content that matches the behavior you want or is close to it, you can use LLMs to transform that data into prompt-response pairs. This has the advantage of being grounded in real use cases rather than fully synthetic scenarios.

The work is in the curation, and sometimes it might be easier to generate from scratch based on a few few-shot examples. Raw data is messy: Support logs contain errors, forums contain misinformation, and real conversations meander. You need to filter, clean, and reformat aggressively, but you can build an LLM pipeline to do those steps. If you have a good source and invest in the LLM curation pipeline, this can be extremely effective, especially for domain-specific applications.

Rejection sampling

Sometimes the best training signal is already inside the model and you just need to find it. Rejection sampling works by generating many possible responses to a prompt, scoring them with some quality metric, and keeping only the top performers. The quality metric can be a reward model, a rule-based check, or even a stronger model acting as a judge.

Suppose you prompt your model “Write a Python function to merge two sorted lists” 64 times at temperature 0.8. You run each output through a test suite as your quality metric. Maybe 40 pass all tests. You take the 10 cleanest, most readable passing solutions and add them to your SFT dataset. You’ve just used the model’s own competence to build training data better than what most human annotators would produce for a coding task.

It sounds like RL, but it’s just using the same pieces to filter the demonstrations that the model should see in SFT. The same graders like reward models, verifiers, or LLM-as-judges are used to curate SFT data.

Rejection sampling is also surprisingly effective and therefore popular; for example it was described early on in Meta’s Llama 2 post-training pipeline. The model already can produce great responses, but as you’ve probably noticed, it just doesn’t do so reliably. By filtering for its best outputs and training, you raise its average toward its ceiling. Instead of acting as the average developer, it’s nudged to act as an expert developer. Rejection sampling scales well because generation is cheap relative to human annotation. The main limitation is that you’re still bounded by what the model can produce at sample time. If it can’t generate a correct proof in any of 100 attempts, no amount of filtering will help.

But SFT has limitations. It only teaches the model what to do. You’re presenting ideal behavior but never showing it what “bad” looks like. As a result, the model could still produce problematic outputs on prompts that weren’t well-represented during training.

The SFT model is also prone to “mode averaging” when the training data sends mixed signals. For example, if half your golden retriever demonstrations sound like an encyclopedia (“The Golden Retriever (Canis lupus familiaris) is a large-sized breed of gun dog…”) and the other half sound really casual (“Golden retrievers? They’re basically furry happiness machines”), the model won’t learn to pick the right tone for each context. It’ll blend them into an awkward middle: “The Golden Retriever is basically a large-sized happiness machine of the gun dog variety.” Neither formal nor casual, which comes off as weird and not the right response style.

Why frontier models use both

RL seems all-powerful. Why not use it alone? This was a research question pursued by DeepSeek’s team when training DeepSeek R1-Zero. Up until then, the base models were so bad that doing RL was pointless on them and you needed to do SFT. This model demonstrated that RL applied directly to their relatively strong pretrained model can produce powerful reasoning ability without any SFT.

However, the model still had serious usability problems. For example, it would mix languages (e.g., English with Mandarin), so it was difficult to use for most people. It could reason, but it wasn’t practical to use.

RL’s main ceiling after the model has been trained is usability. During training, its ceiling is stability. Research on new methods are continually trying to find ways to do RL post-training more stably.

SFT, on the other hand, has the opposite problem. It’s been used alone for many years and has reached maturity to some degree. InstructGPT made the model capable of instruction-following, and became the foundational approach for ChatGPT to handle multiturn dialogue and thus conversation. However, while SFT gets good, reliable results, it’s typically not enough to push performance at the frontier to reach superhuman performance on important tasks.

Here’s what it means for you: If you’re doing post-training on your own, and you want your model to behave a certain way and you don’t care about novel frontier performance, SFT will get the job done.

In contrast, RL can teach a model to reason through novel problems it hasn’t seen during training, because the reward signal evaluates the outcome rather than the exact token-by-token process like in SFT. RL can surface rare but important behaviors that might not appear frequently enough in any SFT dataset.

On scaling data, RL can improve a model’s performance on a task as long as the reward signal is accurate, without needing to collect additional human-written examples. However, it’s important to note that on some tasks, it’s easier to scale SFT examples, and on others, it’s easier to scale via RL. For example, RL scales more easily on math problems. You can generate an unlimited number of math problems programmatically, and a verifier can check whether the answer is correct with certainty. You wouldn’t need to hire a mathematician to write out ideal solutions. The model attempts problems, gets told right or wrong, and improves.

Safety is another area where RL stands out. It’s relatively easy to write a few hundred examples of a model declining harmful requests in your SFT dataset. But the space of ways a user might try to get harmful or inappropriate content is broad, creative, and ever-changing. RL allows the model to be trained against adversarial prompts, where it practices handling tricky edge cases and gets rewarded for handling them well. This is much harder to achieve with static demonstration data alone.

Meanwhile, SFT scales more easily on writing in a specific brand voice. If you want the model to respond with your company’s brand voice. It might be hard to write a reward function that captures “sounds like our brand.” But the company could have tens of thousands of real support transcripts that already demonstrate the voice. You can curate these, transforming them into prompt-response pairs. The data already exists at a decent scale, while the reward signal would be hard to get right.

When a frontier lab wants to add support for a new feature, for example calling MCPs or calling subagents, the first step is almost always to create a small amount of SFT data demonstrating that capability. The next step is creating a reward function and RL environment that can match it.

Far more data and thus compute are dedicated to RL than SFT, but SFT offers good warm starts for the model and those examples are critical to getting the model into a stable place for subsequent RL.

The combination of both is ultimately what makes modern frontier models as capable as they are. Neither alone is sufficient.

A standard post-training pipeline uses SFT and RL as complementary stages that build on each other. It might look like this:

1. Pretraining produces a foundation model with broad knowledge.
2. SFT takes that foundation model and teaches it basic behaviors: how to have a conversation, follow instructions, use a helpful tone on a range of different tasks, etc.
3. RL takes the SFT model checkpoint and refines it further. Using reward signals from human preferences, programmatic verifiers, or AI judges, RL gets the model to be more consistently helpful, less likely to produce harmful content, and better at complex tasks like reasoning.

Some teams also iterate between several stages of SFT and RL: SFT, then RL, then more SFT on new data, then more RL. The first couple stages could be on reasoning for verifiable tasks like math and code where the data and reward signals (verifiers) are constructed differently, whereas the second could be on messier general reasoning over all tasks, which would involve training reward models that encode human feedback as preferences.

This iterative refinement can help with checkpointing quality at different stages and handing things off to different teams, though it adds complexity to the pipeline. Not surprisingly, the quality of each previous stage directly affects how well subsequent stages can go.

But an agent occupies a different role. It learns my writing style, my preferences, and the shape of my work and life, all to help me with what I do. Yet if I switch products, I have to start over. If I use three agents, each rebuilds from scratch what the others already know. Everything an agent learns lives with its vendor.

It doesn’t need to be this way. What if every person had a canonical, user-controlled repository of context that any agent could request permission to use? What if my context lived not only with the company providing the agent but also in a home under my control? And what if an observation captured by one agent could be proposed to that repository and, once accepted, made available to every other agent I choose?

By user-controlled, I don’t necessarily mean self-hosted. I mean that I can inspect what the repository contains, decide who can read or change it, understand where each piece came from, and export the whole thing in a form I can take elsewhere. Its storage, identity, and synchronization may all be provided by someone else. Control does not require me to operate the infrastructure; it requires that no agent or platform be the only way in—or the only way out.

The repository wouldn’t be a portable copy of any agent’s internal model of me. It would be a legible record of things I have written, facts and preferences I have chosen to keep, as well as observations that agents have proposed and I have accepted, each with its provenance, scope, and history. Agents could consult or add to that record according to their permissions; their private inferences would remain their own.

The dream is not a new one. Tim Berners-Lee’s Solid project has argued for years that personal data should live in pods that people control, and Doc Searls’s VRM project has been making the case for user-driven relationships with vendors for decades. What those efforts never had was mainstream demand. Agents are supplying it: An assistant needs rich personal context to be useful, and each vendor is building that context inside its own walls. Ordinary people now have a reason to want a personal data store, even if nobody will actually call it that.

The hard problem in all this isn’t syncing or storing data. It’s negotiation. Who can read a given part of my context? Who can add to it, change it, or remove it? Which parts of my life can a particular instance of an agent see? How do I make those decisions in a policy-driven way? And how do I manage them from wherever I happen to be?

But before I could work on negotiation, I had to figure out where my context should live. That’s the question I’ve spent the past year on, and I’ve tried three answers.

First answer: The laptop

Immediately after getting access to Claude Code at the start of 2025, I pointed it at an Obsidian vault—a folder full of Markdown files that can be used as a personal wiki. This wasn’t a particularly novel idea. Many of the geeks I know did the same, and the pattern has since spread in many forms. The best-known recent example is probably Karpathy’s LLM Wiki, elegant not just as a design but as a document: You give the description to your agent, and the agent builds a version tailored to you.

A year of using a pile of Markdown text files with agents has taught me five things about what a personal context system has to get right.

Local-first foundations matter. Text files are remarkably legible, portable, and easy to store somewhere I control. Git moves them between computers and remembers every change. But the result is centered on a laptop or desktop and assumes a user comfortable with plain text and version control. Most annoyingly, my context in this form isn’t readily available on my phone, which is the computer that goes with me everywhere. Nor can agents running anywhere other than my laptop reach it.

Provenance matters, and so do proposals. Karpathy’s Wiki is almost entirely written (and rewritten) by the LLM. In my own system, I write most things myself and lean on agents to help me edit as well as contribute their observations. I want to know which thoughts are mine, which were captured by an agent, and which we arrived at together. That means an agent’s observation should not automatically enter the repository on the same footing as something I wrote. The default should be a proposal that I—or a policy I control—can accept, revise, or reject. Direct write access is something a trusted agent should earn.

Chronology matters. Wiki links aren’t the only structure in a life. Most of what I record—and much of what agents observe—is anchored in time. Thoughts build on thoughts. Observations about people accumulate meeting by meeting. Some facts fade as they age. Time should be a primary axis of the system, not something reconstructed afterward from file histories and metadata.

Scopes matter. My context spans work, personal, family, and public life. I want one unified view; no agent should have one. An agent connected through my work account should see work and public context—and nothing about my family. Fully separate silos would protect those boundaries, but they would also shred the single history I want to keep building for decades.

Identity and type matter. LLMs can extract all sorts of meaning from plain text, but some things, such as people, companies, and places, deserve to be typed records rather than mentions in prose. A persistent identity gives observations, relationships, and history an anchor to accumulate around; it can help resolve nicknames and follow changes in roles and titles. An agent can then act on who someone is without reconstructing them from prose every time.

As I learned these lessons, I added tooling and conventions to my personal context repository. It’s surprising how far you can push a directory of Markdown files. Each new affordance, however, turned my simple folder into a more specialized system, and the result only works for geeks like me. It doesn’t work for my family, however. They use agents every day but they are never going to deal with a pile of Markdown files in a Git repo. They want their personal context to be with them, easy to use, and transparent to the rest of their life.

More to the point, the five lessons above describe what a context system must do. They don’t answer where it should live if /home/$USER isn’t the center of your computing life.

Second answer: The web

My next move was to sort out how to make my context available when I wasn’t at my laptop—to me and, just as importantly, to my agents. The obvious solution to me as someone who has been building on the web since the mid-1990s: put it on a server behind a URL. I deployed a Cloudflare Worker, uploaded my context, and stood up both a REST API and an MCP server. The improvement was immediate. My context was reachable from my phone and grantable to any agent I chose.

New problems arrived just as fast. I had created a new trust boundary with its own access control mechanism and appointed myself its security team. I was now the operator of a small SaaS with exactly one customer, responsible for its uptime and its backups. And I had traded away local-first, offline editing to get there.

These are solvable problems. Our industry has spent two decades learning to host services, and CRDTs could probably win back offline editing. But as agents gain access to more sensitive data and more power to act on our behalf, the price of getting a boundary wrong keeps rising.

And even with those solved, a deeper problem remains: A stand-alone service sits outside my computing home, apart from the contacts, calendars, messages, files, and system-level agents already inside it. Apple’s Siri AI announcements made that separation vivid, and Gemini’s integration into Google’s ecosystem points the same way. An agent embedded in an ecosystem works with everything inside its trust boundary; my worker would have to rebuild every one of those connections from outside.

Third answer: My pocket

As I tinkered, I kept returning to a simple mental image: my context living on the device in my pocket that goes with me everywhere. Not literally every byte, of course, but within the personal computing ecosystem that phone is the center of—the one that already establishes my identity, synchronizes my devices, stores much of my personal data, and mediates what applications can access. In this sense, a home is not a physical location. It is a trust boundary.

Living inside the boundary doesn’t mean that every application inside gets my context, or that agents outside are shut out. The boundary supplies identity, secure storage, synchronization, and native integration; the context layer still decides what each connection may read, propose, change, or delete. Native agents participate through the platform’s own capabilities, while agents from other companies connect through explicit, revocable permissions.

For me, that home in my pocket is Apple’s ecosystem, with iCloud at its center. For you, it may be Google or Microsoft. The point is not that any one ecosystem is the right home for everyone. It is that most people already have a primary digital home, and that home is the most practical default for their personal context. We shouldn’t need to create a separate service with its own identity. Instead, agents should have a common, permissioned interface to the context where it already lives.

There’s an obvious risk here. A home rooted in a vendor’s ecosystem invites lock-in. The mitigation is straightforward: The whole repository—entities, provenance, and history included—must be exportable at any time as a directory of plain text files that can be taken anywhere. A pile of files in a folder may not be the right solution for live context, but it makes a perfect escape hatch.

I’ve started testing this thesis in a SwiftUI app, and my early prototypes suggest that the architecture is workable: iCloud handles synchronization, and I can expose selected context to authenticated agents through MCP. It’s also shown that working in the Apple developer ecosystem is more annoying than deploying a web app.

The remaining work is clear, however. Choosing a home for context is one problem; negotiation—permissions that remain understandable as a repository grows, proposals from multiple agents reviewed and reconciled—is another. That is the hard problem I mentioned before, and it deserves its own deep dive.

One pattern, many homes

Others are converging on this pattern from different directions. The note-taking app Bear, which stores its notes locally on Apple devices and synchronizes them through iCloud, now exposes them to local agents through MCP; its latest release lets users include or exclude notes by tag when granting access, offering a practical approach to scopes. Craft’s MCP connections likewise let users choose which documents or spaces an agent can access and whether it can read or write them. Reflect has embarked on an open source client using Markdown files that will have an iOS companion app.

These are just a few examples, and there are a lot more out there. What I haven’t yet seen emerge however is the attribution and provenance of items that an agent contributes or edits that I think a durable personal context requires.

Zooming out, here are the principles I think are needed in any system like this, wherever it makes its home:

1. Context shouldn’t be captive to any particular agent. A person should be able to change or combine agents without ever starting over.
2. Context should be reachable by local and remote agents alike, with appropriate authentication and authorization.
3. Permissions should be scoped and revocable. Access should be granted to a specific connection with an agent, limited to a defined subset of the context, and should distinguish between reading, proposing, changing, and deleting.
4. Provenance should persist. Every item records where it came from, who or what created it, and how it has changed. The distinction between human-written, agent-captured, and collaborative work should outlive the conversation that produced it. 
5. Time should be part of the context. The system should preserve when something was observed, when it was true, and how it changed, rather than continually overwriting the past with the present. Database folks will recognize this as bitemporality: the distinction between transaction time and valid time.
6. Entities should be first-class. People, companies, places, and other recurring concepts should remain recognizable as names, roles, and relationships change. 
7. Interoperability should not require uniformity. Different people and companies should be able to build different context systems for different ecosystems and trust boundaries. What they need is agreement on identity, permissions, provenance, and exchange, not one universal application. For the exchange, plain text files with structured metadata are a strong candidate.

Using these principles, personal context can be something a person owns: You can inspect it, grant and revoke access to it, trace where each piece came from, and take the whole of it elsewhere. Every agent may still develop its own understanding of you, but you’ll be able to bring a durable context of your own to the relationship, one that participates in the agentic ecosystem without being subordinate to any vendor in it.

Karpathy’s LLM Wiki is a description, not a tool; it’s meant to be implemented by anyone, in whatever form fits. This essay is offered in the same spirit. The important part isn’t whether the app I’m tinkering with ever ships beyond my own devices. I’m more interested in the dialogue it will take for everyone to have personal context that works for them, in their ecosystem and with the agents they want to use. If we get the pattern right, changing agents won’t mean changing homes. The context they help us build will remain ours.

Vicki wanted the agent to monitor channel performance, identify bottlenecks, and recommend ways to grow subscribers and improve click-through rates. She defined the requirements, designed the workflow, built the agent, tested it, and revised the instructions following the same iterative process used in software development.

How to build a YouTube analytics agent, step by step

1. Define the goal.
   Start with the problem the agent should solve. Vicki wants to understand which videos perform well, where channel performance falls short, and what actions could help more viewers discover and subscribe to the channel.
2. Review the available data.
   Examine each source before designing the workflow. YouTube Studio exports include metrics such as impressions and click-through rates. The YouTube Data API provides information about channels, subscriptions, comments, and activity, but it doesn’t include every metric available through YouTube Studio.
3. Choose how the agent will receive the data.
   Vicki downloads her analytics as CSV files instead of connecting the agent directly to her account. This gives her more control while she tests the workflow. She notes that she could add an authenticated API connection later, after testing the agent’s permissions and guardrails.
4. Write the skills file.
   The skills file defines the agent’s mission, capabilities, data sources, rules, and expected output. Vicki instructs the agent to audit channel metrics, identify bottlenecks, compare performance over time, and recommend actions tied to subscriber growth and click-through rates.
5. Add guardrails and acceptance criteria.
   Tell the agent how to handle the data and what it must avoid. Vicki directs it to use only the supplied numbers, ignore bot activity, report silent failures, and stay within approved systems and data sources. These instructions help prevent unsupported conclusions and actions outside the agent’s assigned role.
6. Build and test the agent.
   Vicki uses the console’s quick-start process to create the agent, add its instructions and sample analytics, select an environment, and start a session. Her first test encounters input problems, so she stops the session, adjusts the data, and runs it again.
7. Monitor cost and performance.
   Observability tools show token consumption, errors, active time, and deployments. Vicki explains that more capable models may need less context but usually cost more to run. Teams can balance model capability against token cost by improving their instructions, selecting less expensive models for simpler work, or dividing a workflow among specialized agents.

Vicki recommends revisiting the skills file as new requirements emerge. Clearer instructions, stronger guardrails, and regular testing help the agent continue to produce useful results. Her process reinforces a practical software engineering lesson: Faster implementation doesn’t reduce the need for clear requirements, reliable data, security controls, and testing.

Coming next week

Join us for the next episode on August 12, when AI and machine learning leader Ofer Mendelevitch will explain how to design multi-agent systems that can keep work moving without constant human involvement. He’ll use Jam to build a team of agents that can plan, divide work, execute tasks, review progress, escalate problems, and repeat the process. He’ll show how an architect agent can take a project brief, recruit developer agents, assign work, and guide the project toward a clear definition of done.

At some point, the software “Security” industry stopped talking about stopping threats and started talking about detecting them: detection windows, response times, mean time to remediate. It’s not offense or prevention; it’s damage control. There’s a movie scene that captures what that sounds like, and you’re going to name the film before I finish describing it.

An underground base on a frozen planet. The enemy knows exactly where it is. Massive mechanical walkers—walking tanks the size of buildings—are advancing across the ice. The defenses can’t stop them. The people inside aren’t trying to fight back. They’re frantically trying to get a broken ship working so they can just escape—not win, not hold the line, just get out before something catastrophic and unstoppable reaches the door.

The whole opening is just people preparing. Rushing. Running checks on equipment that isn’t ready, coordinating defenses that won’t hold, buying time against something too large and too fast to stop. Nobody’s planning a counterattack.

The entire operation is: slow it down long enough to get out.

The Empire Strikes Back

That’s what the conversation around InfoSec sounds like right now. The base is under attack. The walkers are AI-generated vulnerabilities, automated exploit chains, and speed that no human team can match. The framing has shifted from defending the perimeter to just getting the ship started. Not winning, just getting out.

Go back and watch that opening sequence carefully. There are hundreds of faceless Rebel troopers in that scene—no names, no lines worth remembering—scrambling to hold the perimeter, buy time, absorb the blow. Some of them continue to fight. But maybe some already understand that the base is lost.

Han is out on the ice looking for Luke. Leia is already on the transport, making sure the mission survives. The main characters aren’t defending the base. They’ve concluded the only way to answer the threat is to move. Most of the conversation around AI right now sounds like those faceless troopers continuing to defend: fortify what’s there, slow the walkers down, hold long enough for something to change. A few people are thinking like Han. They’re not buying another vulnerability scanner from a vendor. They’re asking whether there’s a different way off the planet entirely.

Here’s the disconnect: most people focused on “Security” have spent decades being handed a finished base and then being asked to defend it. They weren’t involved in the architecture or approach that application developers have been using.

• This new application uses Node.js—go defend it.
• We’re using a new relational database because the architect wanted to—go defend it.
• Our developers decided to start using an LLM. Can you secure it? Thanks.

In many cases, people responsible for security are not defining architectures as much as they are catching up. And as “developers” start to generate more code in a day than was possible in a month or a year, it’s becoming increasingly unrealistic to think of security as an afterthought.

When security is just a support team for software engineers, that’s building a base that might be indefensible.

The shift that actually matters isn’t a better scanner or a faster response team. It’s security people in the room when people are writing the prompts, when agents are assembling the dependency list, and when the basic system prompts are defining the authentication system—before any of those systems is in production. Not reviewing the finished base.

Security needs to be involved before anyone even starts to prompt a system’s creation.

“Machine speed” has become a conference catchphrase, which usually means it needs translation. Here’s what it actually looks like, pointed at you:

• A network of agents found a zero-day in FFmpeg and didn’t announce it anywhere, just filed it internally.
• A second agent scraped your team’s LinkedIn and X and noted who’s in Cancun next week.
• A third logged your nightly load balancer latency blip as a cover for an attack.
• A fourth studied your last three incident reports and estimated a 30-minute detection window.

The whole operation, including reconnaissance, timing, and coordination, ran in seconds. What previously required a dedicated red team and weeks of planning is now background processing that runs continuously, waiting for the right moment.

Five years ago, you would have tasked a room of scary-looking security people with profiling a target, capturing latency data, and maybe holding several meetings to discuss what they found. Today, the coordination I outlined in the previous paragraph might take a few minutes on a network of interconnected Nanobot, Picobot, Hermes, or OpenClaw agents that gather data and then update a shared memory system, and the decision on when and how to attack would be made by another agent that was granted permission to coordinate the attack across a distributed network of agents.

Quick note: If you have anything to do with running a website, stop posting about your vacation plans.

Here’s what the conversation keeps missing: AI isn’t the real problem, and this problem isn’t necessarily new. The problem is that we’ve been building bases that were always going to need to be evacuated. The problem is that security is rarely involved in selecting a tech stack, and because that tech stack selection is frequently automated with AI, there’s no predicting the mess that’s being thrown over the wall.

The response to AI-accelerated attacks is almost entirely defensive. Tighten npm’s signing requirements. Fund the Maven repository. Sign up to support Akrites with the Linux Foundation. Add another scanner to the pipeline. These aren’t wrong. They’re just not enough.

These are important projects, and security groups should sign up to support them, but the real transformation that needs to happen is that more people in security need to get involved in software creation. What this looks like is having an opinion on React, Vite, Tomcat, Node.js, databases. It means jumping in and affecting some of the basic decisions that these agents are going to use before they deliver vulnerable software.

Most of the industry is still shopping for scanners. Most people in security are still “reviewing” software in a process that assumes it takes weeks or months to write.

Both Netscape and Microsoft aimed to dominate the web server and browser market, reasoning that whoever controlled both ends of the connection would have an internet “platform” to rival the deathgrip that Microsoft had enjoyed on the personal computer. The two companies raced to build every feature they could think of directly into the product, on the theory that whoever built the most integrated and full featured web server would win.

The open source Apache web server took the opposite bet. It stayed a web server with a clean extension layer, so anyone could bolt something new onto it without asking permission or waiting for the next release cycle. Within a few years, Apache was far and away the most popular web server, and Netscape’s server and Microsoft’s Internet Information Server (IIS) were history. People started talking about the LAMP stack: Linux, Apache, MySQL, and (Perl | Python | PHP) as a legitimate platform. Modularity, not features, was the moat. The fact that major elements of that stack survive while others have been swapped out or extended is a testament to the power of composability and distributed innovation.

I called that pattern the architecture of participation when I wrote about it in 2004. I was trying to explain an inconvenient fact that the licensing debates of that era ignored. I had started working with Unix in the System III days, and saw how it had succeeded as a collaborative project even though AT&T offered Unix under a proprietary license. A few years later, I observed that nominally open source projects like OpenOffice with monolithic architectures never built much of a community. I realized that open source wasn’t just about licenses, but about architecture. A small kernel with standard interfaces that lets people extend your work without asking for permission is an important part of the secret sauce.

Swap out Netscape and Microsoft for OpenAI and Anthropic in this story, and perhaps you can see the echoes.

A model’s personality, its defaults, and its history used to live where you could, with a little effort, see them and edit them. Increasingly, they don’t. As Drew Breunig pointed out to me the other day, each new version of the frontier models moves a little more of the product’s behavior out of an editable layer and into the weights themselves, where nobody outside the lab can see it, let alone change it. The model stops being a component you build with and can adjust to your liking and starts being an appliance you rent. Post-training is important but Drew points out that it is also “trading diversity for reliability.” That’s a good trade for many people, but it is the same kind of trade that gives us highly processed foods when we know that “real food” is better.

Open weights are just table stakes

The public debate about open source AI seems devoted far too much to model weights, their national security implications, and whether a lab releases weights and under what license.  But that covers only a fraction of what actually makes open source matter. Apache was never competing with Netscape and Microsoft (and Linux was never competing with Windows) over whose source was more available. They were competing over something more important. I remember talking with Bob Young, the founder of Red Hat, about his business model, and he said “What we really sell to our customers is control.” Open source meant that the platform your application depended on was no longer a sealed box you licensed from one company but a layer you could extend and build a business on top of without asking anyone’s permission. It sparked an explosion of innovation. It enabled companies like Google and Amazon to grow up free from Microsoft’s dominant paradigm.

Every wave of computing, from mainframes to PCs to the internet, has run through the same cycle: distributed innovation at the start, with the eventual winner gradually closing down its offerings to build a moat. What keeps a market open isn’t the license on any single component. It’s how easy it is to swap out one component for another when a better one appears.

The protocols connecting the pieces are an important part of that picture. Unix utilities expected stdin and stdout, and the shell acted as a kind of harness to connect them, so it was easy to build a new tool that worked seamlessly with existing ones. A testament to the power of that approach is just how much the shell and Unix utilities are the lingua franca of agentic tooling today, more than 50 years after they were invented! TCP/IP, HTTP, and other internet protocols played a similar role in keeping the internet open and composable.

Fortunately, so far, we are seeing some wins for composable, protocol-centric architectures in AI. Anthropic’s Model Context Protocol was a disruptive move in that direction, an open standard for letting any application reach any tool or data source without a custom integration for each pairing. Along with other open protocols, MCP also now has a home outside of Anthropic at the Agentic AI Foundation (a subproject of the Linux Foundation), which is at least a partial guarantee of its independence.

Isobel Moure, Ilan Strauss, and I made the case earlier this year in Protocols and Power that as models commoditize, competition moves up the stack to context. Opening the means of accessing that context opens the market, regardless of whether open or closed weights sit underneath it. That’s an unbundling, model from harness from context, done the way Apache unbundled web server from web application.

Agentic skills may also be a critical element of the open source AI future, though as the history of the LAMP stack shows, they may fall by the wayside in the same way that Perl and PHP did. And that’s just fine. Composability means that it’s easy to switch to something better when it comes along, or when more people agree on it.

There’s also a lot of great work going on in portable memory from players like Letta, Nous Research, and others. Open source agentic harnesses like Goose and  Pi are also a big part of giving power back to the people. Pi in particular is optimized to be modifiable. There’s a fun story told about Mario Zechner’s decision to give Pi a “/quit” command rather than an “/exit” command like Claude or Codex. Countless issues and PRs have been submitted to Pi’s repo, asking for or implementing “/exit”, but Zechner is stubborn. His retort is that you should just ask Pi to add it to your install.

But the projects I listed above are just the tip of the iceberg when it comes to the scale and scope of open source AI.  Current AI’s Open Source Gap Map covers more than 24,600 open source AI projects!!, with 421 of them scored in depth across openness, capability, and adoption. The map organizes the stack into three layers: 1) models and associated elements including data sets, fine tuning tools, inference frameworks like VLLM, and evals; 2) the product and UX layer, including harnesses and personal agents; and 3) the infrastructure underneath, including core ML frameworks like PyTorch, deployment tools like Ollama, and edge hardware.

Current AI itself is a public-private partnership that came out of the AI Action Summit in Paris last year. This summer they announced AI Potluck, which they describe as “a public project to build a vertically integrated AI product assembled entirely from open source components… a viable alternative to proprietary AI that isn’t owned by any one company or country.” It is backed so far by roughly $400 million of a five-year, $2.5 billion commitment from the French government, tech companies including DeepMind and Salesforce, and major philanthropies including Omidyar’s AI Collaborative, the Macarthur Foundation, and the Ford Foundation.

The fact that this organization exists, along with others like the Agentic AI Foundation, is a testament to the rising tide of interest in open source AI. The coalition of interested parties also says a lot about the underlying motivations that are driving that interest: AI sovereignty, corporate independence from the overweening ambition of the major labs, and an interest in technology for the public good.

Keeping it weird

There’s another element, which Drew Breunig put his finger on in our conversation the other day. The problem with having one or two big closed models dominating AI, and having those models increasingly locking their desired personality, business goals, and guardrails into the weights themselves, is that they will reduce the diversity that is at the heart of innovation.

It’s our job, Drew said, to make it weird, to push a model deliberately out of distribution rather than to settle for whatever the labs have made the default outcome. He described how his team chose not to build in React for a recent project for exactly that reason: every model already knows React too well, so building in it means shipping the average of what everyone else was doing instead of something genuinely their own. He has started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness. And he wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than becoming appliances.

That’s what an architecture of participation is actually for. We need real separation between the model, the harness, and the application, so that someone who wants to build something weird can still do it without a lab’s roadmap and guardrails deciding whether they’re allowed to.

“Weird” may make it sound like something that not all developers might want. But we’re really talking about something intensely practical. In his short essay on trading reliability for diversity, linked above, Drew Breunig put it this way:

Labs have to ship a product that delivers “good enough” results when a layperson gives a model a lazy prompt. Without direction, the model must return something decent. (If it’s a website it’ll use the Inter font, cards with a single colored border, gradients, implemented with ReAct and Tailwind). Anthropic named this default output “distribution convergent.” At CAIS, @trq212 put it well, roughly, “If it’s not in your prompt, you’re getting what’s in-distribution” …. Less diverse models make for more reliable coding agents, but they encourage a monoculture of output.

Addy Osmani, my co-chair of the O’Reilly AI Codecon, took this point beyond model diversity after reading a draft of this piece: “Almost nobody I work with is tinkering with weights, but they’re rewriting the harness and what sits around it pretty constantly—skills, subagents, hooks, context files etc etc. That’s where participation is currently happening.” Addy went on to note that forking a skill instead of adopting the default, memory and constitution files that travel with the agent instead of living in a vendor account, or picking the unfashionable framework on purpose are all areas where ease of modifiability matters to everyone.

I want to end by returning to the Apache story. I believe that the big labs are making the same strategic mistake that Netscape and Microsoft made in the mid 90s. Yes, make the models more reliable for ordinary users. But don’t shut down the options for developers who don’t work for you to push the state of the art forward. As Bill Joy put it decades ago, “No matter who you are, most of the smartest people work for someone else.” No one should have a monopoly on innovation, and no one should be building a moat to hold it back.

And be sure to join us at AI Codecon: Building with Open Source AI on August 31, a free half-day virtual conference. You’ll hear from leading developers and technical experts working with open-weight models, self-hosted infrastructure, and real-world AI workflows, and learn how building in the open gives teams more control over costs, data privacy, and what they ship. Register today to save your spot.