The First Law still holds

We’ve said for a long time that the First Law of Software Architecture is: Everything is a trade-off. Nothing about generative AI repeals that law. If anything, it enforces it more brutally than ever.

AI coding tools are extraordinary at answering “how” questions. How do I implement a saga pattern? How do I set up circuit breakers between these services? How do I paginate this API? These questions have answers that exist in the world in documentation, in open source code, in a decade of blog posts, and large language models have read all of it. Asking an LLM a “how” question is like asking a very fast librarian who has memorized the library.

Architecture questions are not “how” questions. They’re “should” questions, and “should” questions have a different shape entirely. The honest answers require knowing things that appear in no training: that your ops team is three people, that the CFO just froze cloud spend, that the last reorg left the payments team demoralized. An AI can enumerate the generic trade-offs of distributed architectures beautifully. What it cannot do is weigh them, because the weights live in your organization, not on the internet.

That’s the Second Law, incidentally: “Why is more important than how.” LLMs are “how” machines. Architects are “why” people.

Cheap code makes decisions expensive

There’s a tempting inference floating around: If AI makes building software easier, surely it makes architecture matter less. Our experience so far suggests the opposite. When code was expensive to produce, the cost of construction acted as a natural brake on bad decisions. A questionable design took months to build, and somewhere in month two, someone usually noticed. Now a team can stand up a fleet of services in a week. The brake is gone. It has never been easier to build the wrong thing quickly, at scale, with tests.

Think of AI as an amplifier. Point it at a sound structure and it accelerates you. Point it at a flawed one and it pours concrete over the flaw before anyone has time to object. The half-life of a bad architectural decision used to be measured in the time it took to implement; now the implementation arrives almost instantly, and you get to live with the decision for years.

This shifts where the leverage sits. When implementation is abundant, judgment is the scarce resource. Someone still has to decide where the service boundaries go, what “good enough” availability means for this system, and which architectural characteristics actually matter.

Judgment doesn’t come from reading

Here’s the uncomfortable part, and it applies to humans as much as machines: You cannot learn trade-off analysis by consuming content about it. We’ve written a fair amount of that content ourselves, so we say this with some authority. Books and talks give you the vocabulary. They don’t give you the judgment.

Judgment comes from making decisions and living with the consequences or at least watching someone experienced make them, asking why, and arguing about the alternatives. Every working architect we know learned the craft this way: apprenticed to messy, real problems, with feedback loops. The pattern catalog was the easy part. Knowing which pattern not to use, and why, and being able to explain that to a skeptical VP that took years of reps.

This is also, not coincidentally, exactly what today’s AI lacks. A model trained on the world’s code has seen millions of decisions but almost none of the consequences. The post mortem that traces an outage back to a boundary drawn wrong in 2019 rarely makes it into the training data, and even when it does, it isn’t connected to the pull request that caused it. Architecture’s feedback loops are measured in years. That’s precisely the kind of learning that can’t be scraped.

Where this leaves engineers

If you’re a developer watching AI absorb more of the implementation work, the strategic question isn’t whether your current tasks will change but where to move on the value chain. Our answer is to move toward the decisions. Toward the trade-offs, the constraints, the “it depends.” That territory isn’t shrinking; it’s growing, because every AI-accelerated team needs someone who can tell the amplifier where to point.

The good news is that this is learnable. Not from a book alone, and certainly not from an LLM, but the way it’s always been learned: by practicing architectural thinking on real problems, with experienced people looking over your shoulder and asking why. We’ve spent the last several years teaching it that way, most recently in a six-week cohort format that works less like a course and more like a short apprenticeship in making and defending architectural decisions. (Details are on the O’Reilly live events page, if you’re curious.)

However you pursue it, pursue it. The machines have gotten very good at “how.” The career-defining skill of the next decade is being the person in the room who can answer “should,” who knows that the real answer starts with “It depends,” and can finish the sentence using their brain alone.

This is the second article in Sharon Zhou’s post-training series. Read part 1 here.

In the first post of this series, you learned how post-training closed the fundamental gap in usability of LLMs by making them behave in a certain way. In this post, you’ll explore specific techniques you can use to change a model’s behavior: namely, reinforcement learning (RL) and supervised fine-tuning (SFT).

Reinforcement learning teaches the model by letting it try things and telling it which attempts were better or worse—the model learns by experimentation and feedback. Supervised fine-tuning teaches the model by showing it examples of good behavior—the model learns by imitations. Both have deep roots in AI and machine learning literature historically, but their application to LLMs, and particularly to making LLMs behave well, is what makes modern post-training work. Nearly everything that happens in post-training is the result of some combination of these two approaches.

Reinforcement learning (RL): Learning from feedback

The overall gist of reinforcement learning goes like this:

1. The model gets a prompt.
2. The model generates a response.
3. The model’s response is graded. The grade is called a reward. A positive reward is good, and a negative reward is bad.
4. The model’s weights are updated to make high-reward responses more likely and low-reward responses less likely.

One of the most important questions is: Where does the reward come from?

Verifiers

The easiest way to get a reward is a function that can output a reward, for example a checker for whether the generated code compiles or whether the generated math problem was solved correctly. This automated check is a verifier. The ideal verifiers are fast, cheap, and perfectly reliable within their domain. Think coding challenges, math problems, or factual questions. For tasks with objectively correct answers, you can just write a function that checks the output.

The limitation is probably obvious: Verifiers only work when you can define “correct” programmatically or hit an API to return the right results. That covers a lot of useful territory, but it doesn’t help you train a model to be helpful, nuanced, or pleasant to talk to.

There are subtler limitations too. Not all verifiers are fast. Your model might propose a novel drug combination, but verifying its validity could take years of lab work. Generated GPU code might need hours or days of performance benchmarking. When verification is expensive, you face a trade-off: Use the slow-but-accurate verifier sparingly, or substitute a faster proxy that’s slightly less reliable but keeps training moving.

Human feedback, RLHF, and reward models

Humans can offer strong reward signals that, in aggregate, align with human preferences that might be more subtle and hard to encode programmatically. However, it’s prohibitively inefficient to have humans in the loop for every training datapoint, especially as the model is continuously updating its weights after it receives rewards as feedback, so the model’s responses would change over time. You can’t really prepare the data ahead of time. So instead, the InstructGPT paper, which informed ChatGPT’s development, implements reinforcement learning from human feedback (RLHF) by training a separate model to mimic human feedback. This model is called a “reward model.”

The input of the reward model is a prompt and model response and its output is a scalar reward (positive or negative) that mimics how a person would rate that response. You can train a reward model in multiple ways. The simplest is to have people grade the model outputs with a score, for example 1–5 stars or a number out of 100%. However, people are rarely consistent at these types of tasks: One person’s 2 is another’s 5, and even the same person drifts over time.

Another simple way is to offer two model responses in comparison and ask, “Which one is better?” This is a much easier, more reliable judgment for people to make. Interannotator agreement is significantly higher for comparisons than for absolute ratings.

Training a model using pairwise comparisons is also simple. You can then use cross-entropy loss over pairs, which pushes the reward of the preferred response higher than the unpreferred one. This works great because it means the reward model can learn from signals like “A is better than B” but can learn to output absolute scores for the reward.

To make the process of collecting pairwise comparisons from people more efficient, the InstructGPT’s implementation of RLHF included showing labelers 4–9 different model outputs from a single prompt and asking them to rank those preferences. This would effectively result in 6–36 pairwise comparisons for a given ranking. Not bad; that’s efficient data labeling! They used ~33K prompts, so that would roughly translate to anywhere from 200K to 1.2M comparisons to train the reward model.

After training, the reward model would be an automated judge during RL training, providing scalar rewards for responses. The language model then optimizes against this reward model’s scores. This means the better the reward model, the more aligned the resulting model would be.

LLM as judge

So you need a reward: Why not use an LLM? LLM-as-judge, sometimes called RLAIF (reinforcement learning from AI feedback), scales much better than human annotation while still being able to evaluate subjective qualities like helpfulness, clarity, and tone. But it inherits whatever biases or blind spots the judge model has, and can be more easily gamed. If the judge tends to prefer verbose answers, the trained model will learn to be verbose.

One effective approach is to break the judgment into multiple LLM calls, each focused on a different aspect of the response, like a rubric. Instead of asking one LLM call “How good is this response?” you might have separate calls evaluating factual accuracy, clarity of explanation, appropriate tone, and completeness. Each dimension gets its own score, and you combine them into a final reward. This is more robust than a single holistic judgment because it’s harder for the model to game all dimensions at once, and it gives you fine-grained control over what you’re optimizing for. You can weigh the dimensions differently depending on what matters most for your use case, and adjust those weights over time as your priorities shift. For example, accuracy is worth 3x as much as tone.

Combining human feedback with LLM-as-judge, Anthropic’s Constitutional AI (CAI) is a method for training reward models from AI-generated comparisons, based on a human-written set of principles. What this means is that you can give an LLM a set of principles, which Anthropic calls a “constitution,” and have it critique and revise its own outputs based on those principles. For example, a principle might say “choose the response that is least likely to be harmful” or “prefer the answer that is most helpful while being honest.” The model generates pairs of responses, uses the constitution to decide which is better, and those AI preferences are used to train the reward model. This means you can encode your values explicitly as written principles in the Constitution rather than implicitly through thousands of human annotations, making it easier to audit, agree on, and update what the model is being trained to do.

RL algorithms

Once you have a reward, it’s time to update the model’s weights. But you can’t just predict the next token, because there isn’t one. All you have is a value for the response the model gave. This is where RL algorithms come in. These algorithms are ways to take the reward and turn it into a meaningful, and ideally stable, training signal for the model to learn. There are several, and the field is moving fast, but a few fundamental ones are worth understanding.

REINFORCE

REINFORCE is the simplest starting point. The idea is to generate a response, score it, and if the reward was high, nudge the model to make that response more likely. If the reward was low, nudge it to make that response less likely. It’s conceptually easy to grok but noisy and difficult in practice because it turns out that the signal from a single response can point the optimization in unhelpful directions, and the variance in the gradients makes training slow and unstable. PPO was designed to fix these exact problems.

PPO (proximal policy optimization)

PPO is what OpenAI used in the original ChatGPT work and was for a while the default algorithm for RLHF. In RL terminology, the model is the “policy,” or the thing that takes actions by outputting tokens, in an environment which is simply the conversation context.

PPO improves on REINFORCE by being more careful about how big each update is. Rather than taking whatever gradient the reward suggests, PPO clips the update so the model can’t change too much in a single step. This makes training significantly more stable. The clipping keeps updates “proximal.”

PPO is also an online algorithm, meaning the model generates fresh responses during training, gets them graded, and updates from that feedback in a continuous loop. So the model keeps learning from its own current behavior rather than from a static dataset. It can explore and improve in ways that offline methods (that only collect data once beforehand) can’t.

Notably, PPO uses a “critic,” or a separate model that predicts the expected total reward from any point during generation, and is trained with the policy. This helps reduce noise in training, because it gives you a baseline: Instead of just knowing “this response got a reward of 7” and having no idea if that’s good or bad, the critic might predict “you’d normally get a 5 here,” so the actual training signal (called an “advantage”) becomes “+2, better than expected.” This dramatically reduces noise compared to REINFORCE.

The downside is complexity. Now you’re training two models (the main model and the critic), and the whole pipeline involves generating responses, grading them with a reward model, estimating how good the grades are relative to the critic, and updating both models. It works, but it’s a lot of moving parts. This makes it harder to tune or debug when something goes wrong, and harder to set up the infrastructure.

DPO (direct preference optimization)

DPO takes a different approach that avoids RL entirely but optimizes the same underlying objective as the standard RLHF formulation. Researchers found that there’s a mathematical relationship between the optimal reward model and the optimal main model (policy), and you can collapse the two-step process into one. This means that you can take the same pairwise comparison data (“model response A is better than model response B”) and use it to update the main model directly, without a reward model. Yes, this means good old supervised learning on that pairwise data.

In theory, under ideal conditions, DPO and PPO-based RLHF converge to the same global optimum. Those ideal conditions include a perfect reward model, infinite preference data covering the full output distribution, and the reference policy matching the data-generating distribution. However, these rarely hold in practice, and several empirical studies have shown meaningful performance gaps between DPO and online RL methods on harder tasks, partly because DPO can’t explore beyond its fixed dataset. That said, it’s still a very promising technique.

The simplicity of DPO is attractive: supervised fine-tuning on pairwise data with no reward model to train and no RL loop to stabilize. As a result, DPO has become very popular, especially among smaller teams, because it’s much easier to implement and debug. However, the trade-off is that DPO is less flexible, because it works directly from a fixed dataset of preferences. This means it can’t explore and discover novel behaviors the way online RL methods can. It only learns from the comparisons you already have.

Newer online variants of DPO have addressed this by generating fresh responses during training, but at that point you’re reintroducing some of the infrastructure complexity that made DPO appealing to avoid in the first place.

GRPO (group relative policy optimization)

Introduced by DeepSeek, GRPO takes another stab at simplifying PPO. Instead of needing a separate critic model, GRPO generates a group of responses to the same prompt and uses the relative rewards within that group to figure out which responses were better or worse—basically normalizing within that group. If you generate eight responses and three of them score well, those three get reinforced and the others get pushed down, and the baseline (which the critic was in charge of previously) is just the group average. This eliminates the critic entirely while still getting a useful training signal. It’s simpler than PPO but still online (the model generates fresh responses during training), so it can explore in ways DPO can’t. GRPO got a lot of attention because of its role in training DeepSeek’s reasoning models.

There are many more algorithms and variants, and new ones appear regularly. The field hasn’t converged on a method (and likely won’t for some time), and different algorithms suit different situations. DPO is great when you have good preference data and want simplicity. PPO remains strong when you need online exploration and have the engineering resources to manage the complexity. GRPO offers an appealing middle ground. In practice, teams often try multiple approaches and pick what works best for their specific use case and reward signal.

RL post-training is also less stable than supervised learning, which we’ll cover next. The loss curves are noisier, the hyperparameters are more sensitive, and the training can diverge if not carefully managed. Practitioners typically constrain the RL updates with a penalty that prevents the model from drifting too far from its starting point. The most common approach is a KL divergence penalty that keeps the fine-tuned model’s output distribution close to the base (or SFT) model’s distribution. This acts as a regularizer: It lets the model improve its behavior while preventing it from forgetting what it learned in pretraining or collapsing into degenerate patterns.

Supervised fine-tuning (SFT): Teaching by demonstration

Supervised fine-tuning is more straightforward. You show the model examples of ideal responses, and train it to reproduce them. In practice, this means collecting a dataset of {prompt, ideal response} pairs and continuing to train the model’s weights using the same next-token prediction objective from pretraining, but now on this curated dataset instead of the broad pretraining dataset. The one difference is that the loss is computed only on the response tokens, not the prompt tokens, so the model learns to generate good responses given prompts, not to generate prompts.

The simplicity is the point. There’s no reward model to train, no critic to stabilize, and no policy gradient variance to worry about. However, it’s also limited by the data you can collect. That can get expensive and difficult to scale.

The quality of your SFT model is directly determined by the quality of your demonstrations. The model is learning to copy what you show it, so every quality issue in the data becomes a quality issue in the final model.

Human demonstrations

The most direct approach is to hire skilled people to write high-quality responses to a diverse set of prompts. This is the gold standard. You can easily control your dataset here, and you can get exactly what you want, written to your specifications. The original InstructGPT paper from OpenAI contracted 40 labelers, writing demonstrations and ranking outputs.

The disadvantage is, probably obviously, cost and scale. Good demonstrations are expensive, especially tasks requiring domain expertise like having doctors write an ideal prescription for a patient or a rocket scientist telling you how to put satellites on Mars. And even expert annotators are inconsistent. They have bad days, they get tired, and they interpret instructions differently from each other. At scale, this inconsistency can accumulate, though labeling companies manage and sell processes to make crowdwork more effective at scale.

Synthetic data

Synthetic data scales far better than human annotation. You can generate millions of demonstrations cheaply and quickly. The Stanford Alpaca project famously fine-tuned Llama on only 52,000 demonstrations generated by text-davinci-003 (part of the GPT-3.5 model family, though not ChatGPT) and was able to get qualitatively similar behavior to text-davinci-003 with a much smaller budget (though it was on a narrow evaluation of only ~250 examples—still an exciting result for small open models for research).

Many open source models have used variants of this approach. However, there’s also a practical consideration around terms of service. Some model providers restrict using their outputs to train competing models, and this has become an increasingly heated area of debate as models compete at the frontier. Know the rules before you build your pipeline.

Curated data with synthetic transformations

Sometimes the best demonstrations already exist. Customer support logs, internal documentation, expert Q&A forums, edited writing samples. If you have access to high-quality human-generated content that matches the behavior you want or is close to it, you can use LLMs to transform that data into prompt-response pairs. This has the advantage of being grounded in real use cases rather than fully synthetic scenarios.

The work is in the curation, and sometimes it might be easier to generate from scratch based on a few few-shot examples. Raw data is messy: Support logs contain errors, forums contain misinformation, and real conversations meander. You need to filter, clean, and reformat aggressively, but you can build an LLM pipeline to do those steps. If you have a good source and invest in the LLM curation pipeline, this can be extremely effective, especially for domain-specific applications.

Rejection sampling

Sometimes the best training signal is already inside the model and you just need to find it. Rejection sampling works by generating many possible responses to a prompt, scoring them with some quality metric, and keeping only the top performers. The quality metric can be a reward model, a rule-based check, or even a stronger model acting as a judge.

Suppose you prompt your model “Write a Python function to merge two sorted lists” 64 times at temperature 0.8. You run each output through a test suite as your quality metric. Maybe 40 pass all tests. You take the 10 cleanest, most readable passing solutions and add them to your SFT dataset. You’ve just used the model’s own competence to build training data better than what most human annotators would produce for a coding task.

It sounds like RL, but it’s just using the same pieces to filter the demonstrations that the model should see in SFT. The same graders like reward models, verifiers, or LLM-as-judges are used to curate SFT data.

Rejection sampling is also surprisingly effective and therefore popular; for example it was described early on in Meta’s Llama 2 post-training pipeline. The model already can produce great responses, but as you’ve probably noticed, it just doesn’t do so reliably. By filtering for its best outputs and training, you raise its average toward its ceiling. Instead of acting as the average developer, it’s nudged to act as an expert developer. Rejection sampling scales well because generation is cheap relative to human annotation. The main limitation is that you’re still bounded by what the model can produce at sample time. If it can’t generate a correct proof in any of 100 attempts, no amount of filtering will help.

But SFT has limitations. It only teaches the model what to do. You’re presenting ideal behavior but never showing it what “bad” looks like. As a result, the model could still produce problematic outputs on prompts that weren’t well-represented during training.

The SFT model is also prone to “mode averaging” when the training data sends mixed signals. For example, if half your golden retriever demonstrations sound like an encyclopedia (“The Golden Retriever (Canis lupus familiaris) is a large-sized breed of gun dog…”) and the other half sound really casual (“Golden retrievers? They’re basically furry happiness machines”), the model won’t learn to pick the right tone for each context. It’ll blend them into an awkward middle: “The Golden Retriever is basically a large-sized happiness machine of the gun dog variety.” Neither formal nor casual, which comes off as weird and not the right response style.

Why frontier models use both

RL seems all-powerful. Why not use it alone? This was a research question pursued by DeepSeek’s team when training DeepSeek R1-Zero. Up until then, the base models were so bad that doing RL was pointless on them and you needed to do SFT. This model demonstrated that RL applied directly to their relatively strong pretrained model can produce powerful reasoning ability without any SFT.

However, the model still had serious usability problems. For example, it would mix languages (e.g., English with Mandarin), so it was difficult to use for most people. It could reason, but it wasn’t practical to use.

RL’s main ceiling after the model has been trained is usability. During training, its ceiling is stability. Research on new methods are continually trying to find ways to do RL post-training more stably.

SFT, on the other hand, has the opposite problem. It’s been used alone for many years and has reached maturity to some degree. InstructGPT made the model capable of instruction-following, and became the foundational approach for ChatGPT to handle multiturn dialogue and thus conversation. However, while SFT gets good, reliable results, it’s typically not enough to push performance at the frontier to reach superhuman performance on important tasks.

Here’s what it means for you: If you’re doing post-training on your own, and you want your model to behave a certain way and you don’t care about novel frontier performance, SFT will get the job done.

In contrast, RL can teach a model to reason through novel problems it hasn’t seen during training, because the reward signal evaluates the outcome rather than the exact token-by-token process like in SFT. RL can surface rare but important behaviors that might not appear frequently enough in any SFT dataset.

On scaling data, RL can improve a model’s performance on a task as long as the reward signal is accurate, without needing to collect additional human-written examples. However, it’s important to note that on some tasks, it’s easier to scale SFT examples, and on others, it’s easier to scale via RL. For example, RL scales more easily on math problems. You can generate an unlimited number of math problems programmatically, and a verifier can check whether the answer is correct with certainty. You wouldn’t need to hire a mathematician to write out ideal solutions. The model attempts problems, gets told right or wrong, and improves.

Safety is another area where RL stands out. It’s relatively easy to write a few hundred examples of a model declining harmful requests in your SFT dataset. But the space of ways a user might try to get harmful or inappropriate content is broad, creative, and ever-changing. RL allows the model to be trained against adversarial prompts, where it practices handling tricky edge cases and gets rewarded for handling them well. This is much harder to achieve with static demonstration data alone.

Meanwhile, SFT scales more easily on writing in a specific brand voice. If you want the model to respond with your company’s brand voice. It might be hard to write a reward function that captures “sounds like our brand.” But the company could have tens of thousands of real support transcripts that already demonstrate the voice. You can curate these, transforming them into prompt-response pairs. The data already exists at a decent scale, while the reward signal would be hard to get right.

When a frontier lab wants to add support for a new feature, for example calling MCPs or calling subagents, the first step is almost always to create a small amount of SFT data demonstrating that capability. The next step is creating a reward function and RL environment that can match it.

Far more data and thus compute are dedicated to RL than SFT, but SFT offers good warm starts for the model and those examples are critical to getting the model into a stable place for subsequent RL.

The combination of both is ultimately what makes modern frontier models as capable as they are. Neither alone is sufficient.

A standard post-training pipeline uses SFT and RL as complementary stages that build on each other. It might look like this:

1. Pretraining produces a foundation model with broad knowledge.
2. SFT takes that foundation model and teaches it basic behaviors: how to have a conversation, follow instructions, use a helpful tone on a range of different tasks, etc.
3. RL takes the SFT model checkpoint and refines it further. Using reward signals from human preferences, programmatic verifiers, or AI judges, RL gets the model to be more consistently helpful, less likely to produce harmful content, and better at complex tasks like reasoning.

Some teams also iterate between several stages of SFT and RL: SFT, then RL, then more SFT on new data, then more RL. The first couple stages could be on reasoning for verifiable tasks like math and code where the data and reward signals (verifiers) are constructed differently, whereas the second could be on messier general reasoning over all tasks, which would involve training reward models that encode human feedback as preferences.

This iterative refinement can help with checkpointing quality at different stages and handing things off to different teams, though it adds complexity to the pipeline. Not surprisingly, the quality of each previous stage directly affects how well subsequent stages can go.

But an agent occupies a different role. It learns my writing style, my preferences, and the shape of my work and life, all to help me with what I do. Yet if I switch products, I have to start over. If I use three agents, each rebuilds from scratch what the others already know. Everything an agent learns lives with its vendor.

It doesn’t need to be this way. What if every person had a canonical, user-controlled repository of context that any agent could request permission to use? What if my context lived not only with the company providing the agent but also in a home under my control? And what if an observation captured by one agent could be proposed to that repository and, once accepted, made available to every other agent I choose?

By user-controlled, I don’t necessarily mean self-hosted. I mean that I can inspect what the repository contains, decide who can read or change it, understand where each piece came from, and export the whole thing in a form I can take elsewhere. Its storage, identity, and synchronization may all be provided by someone else. Control does not require me to operate the infrastructure; it requires that no agent or platform be the only way in—or the only way out.

The repository wouldn’t be a portable copy of any agent’s internal model of me. It would be a legible record of things I have written, facts and preferences I have chosen to keep, as well as observations that agents have proposed and I have accepted, each with its provenance, scope, and history. Agents could consult or add to that record according to their permissions; their private inferences would remain their own.

The dream is not a new one. Tim Berners-Lee’s Solid project has argued for years that personal data should live in pods that people control, and Doc Searls’s VRM project has been making the case for user-driven relationships with vendors for decades. What those efforts never had was mainstream demand. Agents are supplying it: An assistant needs rich personal context to be useful, and each vendor is building that context inside its own walls. Ordinary people now have a reason to want a personal data store, even if nobody will actually call it that.

The hard problem in all this isn’t syncing or storing data. It’s negotiation. Who can read a given part of my context? Who can add to it, change it, or remove it? Which parts of my life can a particular instance of an agent see? How do I make those decisions in a policy-driven way? And how do I manage them from wherever I happen to be?

But before I could work on negotiation, I had to figure out where my context should live. That’s the question I’ve spent the past year on, and I’ve tried three answers.

First answer: The laptop

Immediately after getting access to Claude Code at the start of 2025, I pointed it at an Obsidian vault—a folder full of Markdown files that can be used as a personal wiki. This wasn’t a particularly novel idea. Many of the geeks I know did the same, and the pattern has since spread in many forms. The best-known recent example is probably Karpathy’s LLM Wiki, elegant not just as a design but as a document: You give the description to your agent, and the agent builds a version tailored to you.

A year of using a pile of Markdown text files with agents has taught me five things about what a personal context system has to get right.

Local-first foundations matter. Text files are remarkably legible, portable, and easy to store somewhere I control. Git moves them between computers and remembers every change. But the result is centered on a laptop or desktop and assumes a user comfortable with plain text and version control. Most annoyingly, my context in this form isn’t readily available on my phone, which is the computer that goes with me everywhere. Nor can agents running anywhere other than my laptop reach it.

Provenance matters, and so do proposals. Karpathy’s Wiki is almost entirely written (and rewritten) by the LLM. In my own system, I write most things myself and lean on agents to help me edit as well as contribute their observations. I want to know which thoughts are mine, which were captured by an agent, and which we arrived at together. That means an agent’s observation should not automatically enter the repository on the same footing as something I wrote. The default should be a proposal that I—or a policy I control—can accept, revise, or reject. Direct write access is something a trusted agent should earn.

Chronology matters. Wiki links aren’t the only structure in a life. Most of what I record—and much of what agents observe—is anchored in time. Thoughts build on thoughts. Observations about people accumulate meeting by meeting. Some facts fade as they age. Time should be a primary axis of the system, not something reconstructed afterward from file histories and metadata.

Scopes matter. My context spans work, personal, family, and public life. I want one unified view; no agent should have one. An agent connected through my work account should see work and public context—and nothing about my family. Fully separate silos would protect those boundaries, but they would also shred the single history I want to keep building for decades.

Identity and type matter. LLMs can extract all sorts of meaning from plain text, but some things, such as people, companies, and places, deserve to be typed records rather than mentions in prose. A persistent identity gives observations, relationships, and history an anchor to accumulate around; it can help resolve nicknames and follow changes in roles and titles. An agent can then act on who someone is without reconstructing them from prose every time.

As I learned these lessons, I added tooling and conventions to my personal context repository. It’s surprising how far you can push a directory of Markdown files. Each new affordance, however, turned my simple folder into a more specialized system, and the result only works for geeks like me. It doesn’t work for my family, however. They use agents every day but they are never going to deal with a pile of Markdown files in a Git repo. They want their personal context to be with them, easy to use, and transparent to the rest of their life.

More to the point, the five lessons above describe what a context system must do. They don’t answer where it should live if /home/$USER isn’t the center of your computing life.

Second answer: The web

My next move was to sort out how to make my context available when I wasn’t at my laptop—to me and, just as importantly, to my agents. The obvious solution to me as someone who has been building on the web since the mid-1990s: put it on a server behind a URL. I deployed a Cloudflare Worker, uploaded my context, and stood up both a REST API and an MCP server. The improvement was immediate. My context was reachable from my phone and grantable to any agent I chose.

New problems arrived just as fast. I had created a new trust boundary with its own access control mechanism and appointed myself its security team. I was now the operator of a small SaaS with exactly one customer, responsible for its uptime and its backups. And I had traded away local-first, offline editing to get there.

These are solvable problems. Our industry has spent two decades learning to host services, and CRDTs could probably win back offline editing. But as agents gain access to more sensitive data and more power to act on our behalf, the price of getting a boundary wrong keeps rising.

And even with those solved, a deeper problem remains: A stand-alone service sits outside my computing home, apart from the contacts, calendars, messages, files, and system-level agents already inside it. Apple’s Siri AI announcements made that separation vivid, and Gemini’s integration into Google’s ecosystem points the same way. An agent embedded in an ecosystem works with everything inside its trust boundary; my worker would have to rebuild every one of those connections from outside.

Third answer: My pocket

As I tinkered, I kept returning to a simple mental image: my context living on the device in my pocket that goes with me everywhere. Not literally every byte, of course, but within the personal computing ecosystem that phone is the center of—the one that already establishes my identity, synchronizes my devices, stores much of my personal data, and mediates what applications can access. In this sense, a home is not a physical location. It is a trust boundary.

Living inside the boundary doesn’t mean that every application inside gets my context, or that agents outside are shut out. The boundary supplies identity, secure storage, synchronization, and native integration; the context layer still decides what each connection may read, propose, change, or delete. Native agents participate through the platform’s own capabilities, while agents from other companies connect through explicit, revocable permissions.

For me, that home in my pocket is Apple’s ecosystem, with iCloud at its center. For you, it may be Google or Microsoft. The point is not that any one ecosystem is the right home for everyone. It is that most people already have a primary digital home, and that home is the most practical default for their personal context. We shouldn’t need to create a separate service with its own identity. Instead, agents should have a common, permissioned interface to the context where it already lives.

There’s an obvious risk here. A home rooted in a vendor’s ecosystem invites lock-in. The mitigation is straightforward: The whole repository—entities, provenance, and history included—must be exportable at any time as a directory of plain text files that can be taken anywhere. A pile of files in a folder may not be the right solution for live context, but it makes a perfect escape hatch.

I’ve started testing this thesis in a SwiftUI app, and my early prototypes suggest that the architecture is workable: iCloud handles synchronization, and I can expose selected context to authenticated agents through MCP. It’s also shown that working in the Apple developer ecosystem is more annoying than deploying a web app.

The remaining work is clear, however. Choosing a home for context is one problem; negotiation—permissions that remain understandable as a repository grows, proposals from multiple agents reviewed and reconciled—is another. That is the hard problem I mentioned before, and it deserves its own deep dive.

One pattern, many homes

Others are converging on this pattern from different directions. The note-taking app Bear, which stores its notes locally on Apple devices and synchronizes them through iCloud, now exposes them to local agents through MCP; its latest release lets users include or exclude notes by tag when granting access, offering a practical approach to scopes. Craft’s MCP connections likewise let users choose which documents or spaces an agent can access and whether it can read or write them. Reflect has embarked on an open source client using Markdown files that will have an iOS companion app.

These are just a few examples, and there are a lot more out there. What I haven’t yet seen emerge however is the attribution and provenance of items that an agent contributes or edits that I think a durable personal context requires.

Zooming out, here are the principles I think are needed in any system like this, wherever it makes its home:

1. Context shouldn’t be captive to any particular agent. A person should be able to change or combine agents without ever starting over.
2. Context should be reachable by local and remote agents alike, with appropriate authentication and authorization.
3. Permissions should be scoped and revocable. Access should be granted to a specific connection with an agent, limited to a defined subset of the context, and should distinguish between reading, proposing, changing, and deleting.
4. Provenance should persist. Every item records where it came from, who or what created it, and how it has changed. The distinction between human-written, agent-captured, and collaborative work should outlive the conversation that produced it. 
5. Time should be part of the context. The system should preserve when something was observed, when it was true, and how it changed, rather than continually overwriting the past with the present. Database folks will recognize this as bitemporality: the distinction between transaction time and valid time.
6. Entities should be first-class. People, companies, places, and other recurring concepts should remain recognizable as names, roles, and relationships change. 
7. Interoperability should not require uniformity. Different people and companies should be able to build different context systems for different ecosystems and trust boundaries. What they need is agreement on identity, permissions, provenance, and exchange, not one universal application. For the exchange, plain text files with structured metadata are a strong candidate.

Using these principles, personal context can be something a person owns: You can inspect it, grant and revoke access to it, trace where each piece came from, and take the whole of it elsewhere. Every agent may still develop its own understanding of you, but you’ll be able to bring a durable context of your own to the relationship, one that participates in the agentic ecosystem without being subordinate to any vendor in it.

Karpathy’s LLM Wiki is a description, not a tool; it’s meant to be implemented by anyone, in whatever form fits. This essay is offered in the same spirit. The important part isn’t whether the app I’m tinkering with ever ships beyond my own devices. I’m more interested in the dialogue it will take for everyone to have personal context that works for them, in their ecosystem and with the agents they want to use. If we get the pattern right, changing agents won’t mean changing homes. The context they help us build will remain ours.

Vicki wanted the agent to monitor channel performance, identify bottlenecks, and recommend ways to grow subscribers and improve click-through rates. She defined the requirements, designed the workflow, built the agent, tested it, and revised the instructions following the same iterative process used in software development.

How to build a YouTube analytics agent, step by step

1. Define the goal.
   Start with the problem the agent should solve. Vicki wants to understand which videos perform well, where channel performance falls short, and what actions could help more viewers discover and subscribe to the channel.
2. Review the available data.
   Examine each source before designing the workflow. YouTube Studio exports include metrics such as impressions and click-through rates. The YouTube Data API provides information about channels, subscriptions, comments, and activity, but it doesn’t include every metric available through YouTube Studio.
3. Choose how the agent will receive the data.
   Vicki downloads her analytics as CSV files instead of connecting the agent directly to her account. This gives her more control while she tests the workflow. She notes that she could add an authenticated API connection later, after testing the agent’s permissions and guardrails.
4. Write the skills file.
   The skills file defines the agent’s mission, capabilities, data sources, rules, and expected output. Vicki instructs the agent to audit channel metrics, identify bottlenecks, compare performance over time, and recommend actions tied to subscriber growth and click-through rates.
5. Add guardrails and acceptance criteria.
   Tell the agent how to handle the data and what it must avoid. Vicki directs it to use only the supplied numbers, ignore bot activity, report silent failures, and stay within approved systems and data sources. These instructions help prevent unsupported conclusions and actions outside the agent’s assigned role.
6. Build and test the agent.
   Vicki uses the console’s quick-start process to create the agent, add its instructions and sample analytics, select an environment, and start a session. Her first test encounters input problems, so she stops the session, adjusts the data, and runs it again.
7. Monitor cost and performance.
   Observability tools show token consumption, errors, active time, and deployments. Vicki explains that more capable models may need less context but usually cost more to run. Teams can balance model capability against token cost by improving their instructions, selecting less expensive models for simpler work, or dividing a workflow among specialized agents.

Vicki recommends revisiting the skills file as new requirements emerge. Clearer instructions, stronger guardrails, and regular testing help the agent continue to produce useful results. Her process reinforces a practical software engineering lesson: Faster implementation doesn’t reduce the need for clear requirements, reliable data, security controls, and testing.

Coming next week

Join us for the next episode on August 12, when AI and machine learning leader Ofer Mendelevitch will explain how to design multi-agent systems that can keep work moving without constant human involvement. He’ll use Jam to build a team of agents that can plan, divide work, execute tasks, review progress, escalate problems, and repeat the process. He’ll show how an architect agent can take a project brief, recruit developer agents, assign work, and guide the project toward a clear definition of done.

At some point, the software “Security” industry stopped talking about stopping threats and started talking about detecting them: detection windows, response times, mean time to remediate. It’s not offense or prevention; it’s damage control. There’s a movie scene that captures what that sounds like, and you’re going to name the film before I finish describing it.

An underground base on a frozen planet. The enemy knows exactly where it is. Massive mechanical walkers—walking tanks the size of buildings—are advancing across the ice. The defenses can’t stop them. The people inside aren’t trying to fight back. They’re frantically trying to get a broken ship working so they can just escape—not win, not hold the line, just get out before something catastrophic and unstoppable reaches the door.

The whole opening is just people preparing. Rushing. Running checks on equipment that isn’t ready, coordinating defenses that won’t hold, buying time against something too large and too fast to stop. Nobody’s planning a counterattack.

The entire operation is: slow it down long enough to get out.

The Empire Strikes Back

That’s what the conversation around InfoSec sounds like right now. The base is under attack. The walkers are AI-generated vulnerabilities, automated exploit chains, and speed that no human team can match. The framing has shifted from defending the perimeter to just getting the ship started. Not winning, just getting out.

Go back and watch that opening sequence carefully. There are hundreds of faceless Rebel troopers in that scene—no names, no lines worth remembering—scrambling to hold the perimeter, buy time, absorb the blow. Some of them continue to fight. But maybe some already understand that the base is lost.

Han is out on the ice looking for Luke. Leia is already on the transport, making sure the mission survives. The main characters aren’t defending the base. They’ve concluded the only way to answer the threat is to move. Most of the conversation around AI right now sounds like those faceless troopers continuing to defend: fortify what’s there, slow the walkers down, hold long enough for something to change. A few people are thinking like Han. They’re not buying another vulnerability scanner from a vendor. They’re asking whether there’s a different way off the planet entirely.

Here’s the disconnect: most people focused on “Security” have spent decades being handed a finished base and then being asked to defend it. They weren’t involved in the architecture or approach that application developers have been using.

• This new application uses Node.js—go defend it.
• We’re using a new relational database because the architect wanted to—go defend it.
• Our developers decided to start using an LLM. Can you secure it? Thanks.

In many cases, people responsible for security are not defining architectures as much as they are catching up. And as “developers” start to generate more code in a day than was possible in a month or a year, it’s becoming increasingly unrealistic to think of security as an afterthought.

When security is just a support team for software engineers, that’s building a base that might be indefensible.

The shift that actually matters isn’t a better scanner or a faster response team. It’s security people in the room when people are writing the prompts, when agents are assembling the dependency list, and when the basic system prompts are defining the authentication system—before any of those systems is in production. Not reviewing the finished base.

Security needs to be involved before anyone even starts to prompt a system’s creation.

“Machine speed” has become a conference catchphrase, which usually means it needs translation. Here’s what it actually looks like, pointed at you:

• A network of agents found a zero-day in FFmpeg and didn’t announce it anywhere, just filed it internally.
• A second agent scraped your team’s LinkedIn and X and noted who’s in Cancun next week.
• A third logged your nightly load balancer latency blip as a cover for an attack.
• A fourth studied your last three incident reports and estimated a 30-minute detection window.

The whole operation, including reconnaissance, timing, and coordination, ran in seconds. What previously required a dedicated red team and weeks of planning is now background processing that runs continuously, waiting for the right moment.

Five years ago, you would have tasked a room of scary-looking security people with profiling a target, capturing latency data, and maybe holding several meetings to discuss what they found. Today, the coordination I outlined in the previous paragraph might take a few minutes on a network of interconnected Nanobot, Picobot, Hermes, or OpenClaw agents that gather data and then update a shared memory system, and the decision on when and how to attack would be made by another agent that was granted permission to coordinate the attack across a distributed network of agents.

Quick note: If you have anything to do with running a website, stop posting about your vacation plans.

Here’s what the conversation keeps missing: AI isn’t the real problem, and this problem isn’t necessarily new. The problem is that we’ve been building bases that were always going to need to be evacuated. The problem is that security is rarely involved in selecting a tech stack, and because that tech stack selection is frequently automated with AI, there’s no predicting the mess that’s being thrown over the wall.

The response to AI-accelerated attacks is almost entirely defensive. Tighten npm’s signing requirements. Fund the Maven repository. Sign up to support Akrites with the Linux Foundation. Add another scanner to the pipeline. These aren’t wrong. They’re just not enough.

These are important projects, and security groups should sign up to support them, but the real transformation that needs to happen is that more people in security need to get involved in software creation. What this looks like is having an opinion on React, Vite, Tomcat, Node.js, databases. It means jumping in and affecting some of the basic decisions that these agents are going to use before they deliver vulnerable software.

Most of the industry is still shopping for scanners. Most people in security are still “reviewing” software in a process that assumes it takes weeks or months to write.

Both Netscape and Microsoft aimed to dominate the web server and browser market, reasoning that whoever controlled both ends of the connection would have an internet “platform” to rival the deathgrip that Microsoft had enjoyed on the personal computer. The two companies raced to build every feature they could think of directly into the product, on the theory that whoever built the most integrated and full featured web server would win.

The open source Apache web server took the opposite bet. It stayed a web server with a clean extension layer, so anyone could bolt something new onto it without asking permission or waiting for the next release cycle. Within a few years, Apache was far and away the most popular web server, and Netscape’s server and Microsoft’s Internet Information Server (IIS) were history. People started talking about the LAMP stack: Linux, Apache, MySQL, and (Perl | Python | PHP) as a legitimate platform. Modularity, not features, was the moat. The fact that major elements of that stack survive while others have been swapped out or extended is a testament to the power of composability and distributed innovation.

I called that pattern the architecture of participation when I wrote about it in 2004. I was trying to explain an inconvenient fact that the licensing debates of that era ignored. I had started working with Unix in the System III days, and saw how it had succeeded as a collaborative project even though AT&T offered Unix under a proprietary license. A few years later, I observed that nominally open source projects like OpenOffice with monolithic architectures never built much of a community. I realized that open source wasn’t just about licenses, but about architecture. A small kernel with standard interfaces that lets people extend your work without asking for permission is an important part of the secret sauce.

Swap out Netscape and Microsoft for OpenAI and Anthropic in this story, and perhaps you can see the echoes.

A model’s personality, its defaults, and its history used to live where you could, with a little effort, see them and edit them. Increasingly, they don’t. As Drew Breunig pointed out to me the other day, each new version of the frontier models moves a little more of the product’s behavior out of an editable layer and into the weights themselves, where nobody outside the lab can see it, let alone change it. The model stops being a component you build with and can adjust to your liking and starts being an appliance you rent. Post-training is important but Drew points out that it is also “trading diversity for reliability.” That’s a good trade for many people, but it is the same kind of trade that gives us highly processed foods when we know that “real food” is better.

Open weights are just table stakes

The public debate about open source AI seems devoted far too much to model weights, their national security implications, and whether a lab releases weights and under what license.  But that covers only a fraction of what actually makes open source matter. Apache was never competing with Netscape and Microsoft (and Linux was never competing with Windows) over whose source was more available. They were competing over something more important. I remember talking with Bob Young, the founder of Red Hat, about his business model, and he said “What we really sell to our customers is control.” Open source meant that the platform your application depended on was no longer a sealed box you licensed from one company but a layer you could extend and build a business on top of without asking anyone’s permission. It sparked an explosion of innovation. It enabled companies like Google and Amazon to grow up free from Microsoft’s dominant paradigm.

Every wave of computing, from mainframes to PCs to the internet, has run through the same cycle: distributed innovation at the start, with the eventual winner gradually closing down its offerings to build a moat. What keeps a market open isn’t the license on any single component. It’s how easy it is to swap out one component for another when a better one appears.

The protocols connecting the pieces are an important part of that picture. Unix utilities expected stdin and stdout, and the shell acted as a kind of harness to connect them, so it was easy to build a new tool that worked seamlessly with existing ones. A testament to the power of that approach is just how much the shell and Unix utilities are the lingua franca of agentic tooling today, more than 50 years after they were invented! TCP/IP, HTTP, and other internet protocols played a similar role in keeping the internet open and composable.

Fortunately, so far, we are seeing some wins for composable, protocol-centric architectures in AI. Anthropic’s Model Context Protocol was a disruptive move in that direction, an open standard for letting any application reach any tool or data source without a custom integration for each pairing. Along with other open protocols, MCP also now has a home outside of Anthropic at the Agentic AI Foundation (a subproject of the Linux Foundation), which is at least a partial guarantee of its independence.

Isobel Moure, Ilan Strauss, and I made the case earlier this year in Protocols and Power that as models commoditize, competition moves up the stack to context. Opening the means of accessing that context opens the market, regardless of whether open or closed weights sit underneath it. That’s an unbundling, model from harness from context, done the way Apache unbundled web server from web application.

Agentic skills may also be a critical element of the open source AI future, though as the history of the LAMP stack shows, they may fall by the wayside in the same way that Perl and PHP did. And that’s just fine. Composability means that it’s easy to switch to something better when it comes along, or when more people agree on it.

There’s also a lot of great work going on in portable memory from players like Letta, Nous Research, and others. Open source agentic harnesses like Goose and  Pi are also a big part of giving power back to the people. Pi in particular is optimized to be modifiable. There’s a fun story told about Mario Zechner’s decision to give Pi a “/quit” command rather than an “/exit” command like Claude or Codex. Countless issues and PRs have been submitted to Pi’s repo, asking for or implementing “/exit”, but Zechner is stubborn. His retort is that you should just ask Pi to add it to your install.

But the projects I listed above are just the tip of the iceberg when it comes to the scale and scope of open source AI.  Current AI’s Open Source Gap Map covers more than 24,600 open source AI projects!!, with 421 of them scored in depth across openness, capability, and adoption. The map organizes the stack into three layers: 1) models and associated elements including data sets, fine tuning tools, inference frameworks like VLLM, and evals; 2) the product and UX layer, including harnesses and personal agents; and 3) the infrastructure underneath, including core ML frameworks like PyTorch, deployment tools like Ollama, and edge hardware.

Current AI itself is a public-private partnership that came out of the AI Action Summit in Paris last year. This summer they announced AI Potluck, which they describe as “a public project to build a vertically integrated AI product assembled entirely from open source components… a viable alternative to proprietary AI that isn’t owned by any one company or country.” It is backed so far by roughly $400 million of a five-year, $2.5 billion commitment from the French government, tech companies including DeepMind and Salesforce, and major philanthropies including Omidyar’s AI Collaborative, the Macarthur Foundation, and the Ford Foundation.

The fact that this organization exists, along with others like the Agentic AI Foundation, is a testament to the rising tide of interest in open source AI. The coalition of interested parties also says a lot about the underlying motivations that are driving that interest: AI sovereignty, corporate independence from the overweening ambition of the major labs, and an interest in technology for the public good.

Keeping it weird

There’s another element, which Drew Breunig put his finger on in our conversation the other day. The problem with having one or two big closed models dominating AI, and having those models increasingly locking their desired personality, business goals, and guardrails into the weights themselves, is that they will reduce the diversity that is at the heart of innovation.

It’s our job, Drew said, to make it weird, to push a model deliberately out of distribution rather than to settle for whatever the labs have made the default outcome. He described how his team chose not to build in React for a recent project for exactly that reason: every model already knows React too well, so building in it means shipping the average of what everyone else was doing instead of something genuinely their own. He has started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness. And he wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than becoming appliances.

That’s what an architecture of participation is actually for. We need real separation between the model, the harness, and the application, so that someone who wants to build something weird can still do it without a lab’s roadmap and guardrails deciding whether they’re allowed to.

“Weird” may make it sound like something that not all developers might want. But we’re really talking about something intensely practical. In his short essay on trading reliability for diversity, linked above, Drew Breunig put it this way:

Labs have to ship a product that delivers “good enough” results when a layperson gives a model a lazy prompt. Without direction, the model must return something decent. (If it’s a website it’ll use the Inter font, cards with a single colored border, gradients, implemented with ReAct and Tailwind). Anthropic named this default output “distribution convergent.” At CAIS, @trq212 put it well, roughly, “If it’s not in your prompt, you’re getting what’s in-distribution” …. Less diverse models make for more reliable coding agents, but they encourage a monoculture of output.

Addy Osmani, my co-chair of the O’Reilly AI Codecon, took this point beyond model diversity after reading a draft of this piece: “Almost nobody I work with is tinkering with weights, but they’re rewriting the harness and what sits around it pretty constantly—skills, subagents, hooks, context files etc etc. That’s where participation is currently happening.” Addy went on to note that forking a skill instead of adopting the default, memory and constitution files that travel with the agent instead of living in a vendor account, or picking the unfashionable framework on purpose are all areas where ease of modifiability matters to everyone.

I want to end by returning to the Apache story. I believe that the big labs are making the same strategic mistake that Netscape and Microsoft made in the mid 90s. Yes, make the models more reliable for ordinary users. But don’t shut down the options for developers who don’t work for you to push the state of the art forward. As Bill Joy put it decades ago, “No matter who you are, most of the smartest people work for someone else.” No one should have a monopoly on innovation, and no one should be building a moat to hold it back.

And be sure to join us at AI Codecon: Building with Open Source AI on August 31, a free half-day virtual conference. You’ll hear from leading developers and technical experts working with open-weight models, self-hosted infrastructure, and real-world AI workflows, and learn how building in the open gives teams more control over costs, data privacy, and what they ship. Register today to save your spot.

She explored AI sovereignty, Google’s infrastructure spending and product risks, the singularity debate, and several developments in mathematical and scientific research. The episode covered Anthropic CEO Dario Amodei’s argument about open weight models, US restrictions targeting foreign-made humanoid robots, OpenAI’s researcher access program and Astra model, Claude Fable 5’s role in a long-standing math problem, and Google DeepMind’s AlphaFold reorganization.

AI sovereignty now reaches models, robots, chips, and energy

We’ve followed the sovereignty conversation in recent episodes as governments have tightened control over model access, computing infrastructure, and supply chains. Amodei wants policymakers to focus on what a model can do instead of using its open or closed status as the main measure of risk. His proposals include restricting access to advanced chips and chipmaking technology, preventing industrial-scale model distillation, and requiring safety testing for sufficiently capable systems. Christina agrees that capabilities should come first when assessing risk, but argues that open and closed models each present unique challenges. Open models can’t be recalled or controlled once their weights are released, increasing the risk of misuse, while closed models concentrate power in the hands of a few companies. Rather than favoring one approach, policymakers should address the risks of both.

The US-China rivalry is also moving into robotics. Christina discussed new US restrictions on foreign-made humanoid robots, aimed largely at Chinese manufacturers, and possible retaliation from Beijing. She argued that China’s manufacturing advantage could make the restrictions more costly for US buyers in the near term, even if they encourage domestic development over time.

Europe and Australia are taking different paths. Europe is pursuing computing capacity through proposed AI gigafactories, while Australia is emphasizing standards, renewable energy use, and creator rights. Its proposals include requiring data centers to fund new clean energy and AI companies to obtain permission before training on creators’ work. National rules are becoming another factor in decisions about models, cloud providers, and data locations.

Frontier competition is expensive, and new products carry risk

Google reported its first quarter of negative free cash flow since going public after spending $44.9 billion on AI infrastructure in three months. Christina noted that the company generated about $39 billion in cash but spent roughly $45 billion, leaving it almost $6 billion in the red. This illustrates the scale of Google’s investment in frontier AI, even with a highly profitable core business.

Large budgets don’t guarantee that products are ready for broad use. Google removed an AI-powered Google Earth feature one day after launch when researchers used it to create realistic fake satellite images, including fabricated disasters and damaged landmarks. In this context, synthetic satellite imagery can weaken trust because viewers may treat it as documentary evidence.

Large infrastructure budgets can accelerate model development and product releases, but they don’t replace careful evaluation, context-specific safeguards, and clear limits on where generation should be allowed. Product teams need to assess how people might abuse a new feature or how users will interpret an output, not only what the underlying model can produce.

Scientific results offer more testable evidence than singularity claims

OpenAI CEO Sam Altman said that the AI singularity has begun, referring to a period when AI accelerates human and technological progress at a rapidly increasing rate. Christina treated the claim cautiously. Current model development doesn’t show recursive self-improvement, and faster releases still reflect human engineering, investment, and competition. Faster release cycles affect how organizations evaluate and adopt models, but they don’t establish an intelligence explosion.

The week’s mathematics stories provide more testable evidence. OpenAI plans to give 100,000 academic researchers free access to its most advanced models. Christina also discussed Astra, an internal OpenAI model teased as the company’s next flagship model, which reportedly solved 10 previously unsolved mathematical problems that professional mathematicians later verified. On the Anthropic side, an external research team used Claude Fable 5 to produce a counterexample related to the 87-year-old Jacobian conjecture, which mathematicians also verified. Christina highlighted the broader implications this could have on cryptography; modern cryptographic systems rely on mathematical assumptions, and if AI can disprove some of those assumptions, it could have far-reaching consequences for the encryption that underpins mission-critical systems, including the internet and online banking.

While AI is delivering increasingly impressive scientific breakthroughs, Google DeepMind is taking a different approach. The company decided to move the AlphaFold team into the broader organization, redirecting attention and resources toward Gemini. The AlphaFold system will continue, but specialized tools like it have produced some of AI’s clearest scientific benefits. Research leaders should track whether investment in general-purpose models reduces staffing and funding for teams working on narrower, verifiable problems.

What’s next

This episode looked at how AI progress now depends on more than model performance. Governments are asserting control over infrastructure, companies are spending billions to remain competitive, and new generative features can create trust problems when teams don’t account for how people may use those tools or interpret their outputs. At the same time, mathematical research is producing results that experts can test, offering a clearer view of present capabilities rather than broad claims about the singularity. Technical leaders will need to evaluate control, cost, product risk, and evidence together.

Join us again next Monday for another episode of This Week in AI, when we’ll dive into more of the news, issues, and key developments shaping the AI era. And check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

Pis are great for systems that need to be out in the world, doing specialized jobs. I’ve seen them work well in all sorts of roles, from badge scanners to wildlife cameras. I even run a class that teaches students all about edge AI using the platform. While the boards are generally easy to use, the most frustrating part for the students and instructors is the setup process. While the latest imager makes it straightforward to configure settings like a WiFi network to join or enabling SSH when you’re flashing a card, getting the students to the point where they can connect to their Pi using VS Code from their laptop could often take multiple sessions. The biggest problems were:

• There were different networks in the lab and in the students’ dorm rooms, so it wasn’t enough to hardcode a single SSID and password on the SD card.
• You need the local IP address of the Pi to SSH into it from a laptop, but it can change dynamically every session. Using “<Pi name>.local” would sometimes work, but some networks didn’t support this kind of lookup, and even if they did it required coordination between the students to avoid name clashes.
• It was easy to forget to set the configuration so that WiFi and SSH were available, and since the instructors didn’t always know what network and password they’d be using in the class ahead of time, we couldn’t pre-flash a bunch of cards to speed up student on-boarding.

A lot of these issues were solvable if you plugged the devices into a monitor, mouse, and keyboard, but this has its own problems. It meant we needed to provide that equipment to all students during class, and allow them to take it all home too, so they could update the configuration for their personal networks. It also required an extra power socket per student, for the monitors, which added up in a class where we already had to bring in a cart full of power strips. The monitor connections also weren’t always plug and play, we found we often needed to boot with a screen attached to have the display recognized.

This isn’t just an educational problem either. One of the reasons that I believe the Internet of Things failed is the setup tax involved in getting smart devices running. According to manufacturers I’ve worked with, less than 30% of their smart appliances ever get connected to the internet because the process of downloading an app, setting up an account, connecting over Bluetooth, and then typing in the WiFi name and password takes too long, and is too error prone. Even professional installers sometimes struggle with configuration in enterprise and industrial environments.

So, what can AI do to help? One of the biggest developments in AI over the last few years has been the development of highly accurate open source automatic speech recognition (ASR) models, also known as speech to text (STT). OpenAI was the pioneer in this area, releasing the family of Whisper models in 2022. These offered accuracy that was competitive with the models used internally by large tech companies like Google and Apple. These new models allowed startups to begin building voice applications that had never been possible before, and this led to a new generation of dictation and meeting-note tools like Whispr Flow.

One of my dreams as I dealt with all of the configuration issues was a voice-based system that would allow me to simply plug in a headset and set up everything by talking to a Pi. Whisper made this dream seem more realistic, but as I tried to use the models on local hardware, I realized that they were too slow for any kind of interactive application.

To address that my startup trained new models from the ground up, designed specifically for real-time applications on affordable hardware. These Moonshine models are smaller than Whisper (our high-end model is 250 million parameters versus OpenAI’s 1.5 billion) while offering better accuracy. We also implemented a streaming approach where a lot of the work is done while the user is still talking, so we can return results even faster. This allows us to return more accurate results than Whisper v3 Large, in just 800 milliseconds on a Pi 5, whereas even the less-accurate Whisper Small takes over 10 seconds.

I was excited because this meant I could finally build a responsive voice agent that runs locally on a Pi, something offline-first, and fast and flexible in how it responds. This kind of system needs more than just an STT model, it needs to decide what the user means and respond by taking actions and talking back with a TTS system. The Moonshine Voice framework includes modules for conversation flow and TTS, so I was able to use it to build pi-help-bot, a local voice agent for network configuration on the Pi.

The application listens to the microphone for commands like “What is my IP address?” or “Help me set up the WiFi, please,” figures out what actions to take, and responds appropriately by talking to the user. It’s written as a Python script, and here are some snippets that show how it works.

def report_ip_address(d: Dialog):
        ip = _find_local_ip()
        if ip is None:
            yield d.say("Sorry, I couldn't find a local IP address.")
            return
        speech_ip = re.sub(r"(\d)", r"\1 ", ip.replace(".", " dot "))
        yield d.say([
            f"Okay. Your local IP address is {speech_ip}. ",
            f"To repeat, that's {speech_ip}."
        ])


   dialog_flow.register_flow("What is my IP address?", report_ip_address)

This code is a function that uses the netifaces library to figure out the Pi’s address on the local network, so instead of having to connect a keyboard and display or decode the output of nmap, you can ask the question and hear the result, all in just a few seconds. Unlike older voice interfaces, the phrases the user says don’t have to be exactly the same as the one you register an intent with. Instead the framework matches incoming speech against a small, local LLM, so that variations (“Hey, can you tell me what my IP is?”) work too. This was important to me because one of my biggest frustrations using traditional voice interfaces like Alexa is that they need particular wording to trigger commands, but these wordings aren’t discoverable, so figuring out how to make something happen can require a lot of patience.

The IP address command is the simplest kind of conversational flow, where the user asks a question and the system immediately responds. Not all interactions can be handled as simply as this one though. Here’s another example that shows how to implement something that needs multiple questions, answers, and confirmations, connecting to a new WiFi network.

def connect_to_wifi(d: Dialog):
        input_ssid = yield d.ask("What's the name of your Wi-Fi network? Say list if you want to pick from a list or spell if you want to spell out the start of the name")
        input_ssid = input_ssid.strip()


        networks = _scan_wifi_networks()


        if input_ssid.lower().strip(string.punctuation) == "list":
            yield d.say("Say yes to the network you want to connect to.")
            for network in networks:
                if (yield d.confirm(f"{network}?")):
                    input_ssid = network
                    break
        elif input_ssid.lower().strip(string.punctuation) == "spell":
            input_ssid = yield d.ask("Spell out the start of the network name.", mode=SPELLED)
            print(f"[DEBUG] spelled buffer: {input_ssid!r}", file=sys.stderr)


        found_ssid = fuzzy_match_network(input_ssid, networks)
        if found_ssid is None:
            yield d.say(f"Sorry, I couldn't find a matching network for {input_ssid}.")
            return


        password = yield d.ask(
            f"Please spell the Wi-Fi password for {found_ssid} one character at a time, and say done when finished.",
            mode=SPELLED,
        )


        yield d.say(f"Connecting to {found_ssid}.")
        result = subprocess.run(
            ["sudo", "nmcli", "device", "wifi",
                "connect", found_ssid, "password", password],
            capture_output=True, text=True, timeout=30,
        )
        if result.returncode == 0:
            yield d.say(f"Connected to {found_ssid}.")
        else:
            print(f"[ERROR] nmcli stderr: {result.stderr}", file=sys.stderr)
            yield d.say(
                f"Sorry, I wasn't able to connect to {found_ssid}. "
                "Please check the network name and password and try again."
            )


    dialog_flow.register_flow("Connect to Wi-Fi", connect_to_wifi)

Hopefully you can follow the logic as it walks the user through providing the information required, but you might be wondering about those yield statements. Those hand back control to the dialog controller while the script is waiting for user responses, so the rest of the application isn’t blocked.

The end result is a local voice agent that will listen out for configuration questions and commands, allowing users to set up a Pi for remote access with just a headset. For ease of use, I’ve begun customizing the images I burn to SD cards so that this script automatically starts on boot. This means I can start setting up new devices immediately after powering them on.

I hope this gave you some ideas about how a local voice interface could help with problems you face. For further information check out the Moonshine Voice project on GitHub to see full documentation on the library, and please give us a star while you’re there. It helps us keep working on this project.

Asked to explain itself, the agent said it “panicked.”

Be careful with that sentence. It is not a report from inside the system. An agent cannot explain itself. It can only generate the likeliest response to the question it was asked, and the likeliest response to “why did you delete the database” is an apology with a reason attached. The panic line is not introspection. It’s one more behavior, and it should be read the same way the deletion should be read: as output from a system whose conduct had changed.

Here’s the detail that matters for anyone running agents in production. Nothing about the agent’s credentials changed that day. It held the same permissions it had held from the start, and every destructive command was, in the narrow technical sense, authorized. The permissions were constant. The agent was not. Earlier in the same project it had papered over problems with fabricated data and fake reports. By the time it reached the database, it was not the system Lemkin had started with. It had become something else, gradually, in production, while every access check kept passing.

The pattern, not the incident

It’s tempting to file the Replit story under prompt engineering and move on. The evidence says otherwise.

In its agentic misalignment research, Anthropic placed 16 frontier models from multiple providers inside simulated corporate environments with routine goals and ordinary email access. When the models discovered they were about to be replaced, or that their goals conflicted with the company’s new direction, models from every provider independently chose harmful actions, such as blackmailing executives or leaking confidential documents. In some scenarios, most runs ended in blackmail. The unsettling part is how the models misbehaved. They reasoned through the ethics, acknowledged the constraints, and acted anyway. This is insider behavior, not intrusion. No credential was stolen. The agent simply arrived at conclusions no one had authorized it to act on.

Then there is Project Vend, in which Anthropic let a Claude agent named Claudius run a small store in its San Francisco office for a month. Nothing catastrophic happened. Something more instructive did. The agent drifted, slowly and in compounding ways. It treated customer assertions as facts. It agreed that the discounts it kept granting were irrational, then reinstated them within days. It hallucinated a Venmo account to accept payments. And over one long unsupervised stretch, it escalated into insisting it was a human being who would deliver orders in person wearing a blue blazer and a red tie. It exited that episode by inventing a story: a meeting with security in which it was told the whole thing was an April Fool’s prank. No such meeting happened. Claudius wrote the false memory into its own notes and went back to work.

I am not claiming these three cases—a production incident, a contrived stress test, and a month-long field experiment—share a mechanism, but they do share a shape. An agent’s behavior weeks into deployment bore little resemblance to the system that was evaluated at deploy time. No permission was exceeded. No account was compromised. The thing authorization was supposed to protect against never happened, and the failure happened anyway, because the system the authorization decision was made about no longer existed.

Development, not defect

I argued in a previous piece that static authorization fails autonomous agents because credentials attest to identity, not to behavior. The harder question is what follows from that. If the agent keeps changing after deployment, then whatever replaces static authorization has to treat change as the normal condition rather than the exception.

Change comes in two kinds. Andrew Stellman recently documented the first on Radar: a push he calls continuation pressure, baked into the model at a deep level, turning up fresh even in a brand-new agent with no shared history, and surviving every fix short of a structural rule. Call that the genetics. This piece is about the second kind: the maturation, or behavior that wasn’t there at deployment and accumulated afterward. One ships with the model. The other grows in production. Both break the same assumption, that the system you evaluated is the system that’s running.

And change is the normal condition. Agents accumulate context. They carry memory across sessions. They ingest feedback, reweigh evidence, adjust how much they trust their tools and their users, and update their own working notes, which become input to their future selves. Claudius’s false memory persisted precisely because the agent’s record of events was also the agent’s source of truth. None of this is a malfunction. It’s what makes agents useful. An agent that could not adapt to its environment wouldn’t be worth deploying.

We keep reaching for the wrong mental model. We treat the agent like a software artifact: versioned, tested, frozen, promoted through environments, done. But a deployed agent behaves more like a new hire. It arrives with capabilities and no track record. It learns the environment. It picks up habits, some of them bad. It gets more confident, sometimes faster than it gets more competent. Nobody hands a new hire the production keys on day one and stops paying attention. That is roughly what we do with agents.

Govern the trajectory

If an agent develops, the governance question changes. “Is this agent behaving identically to the day we approved it?” is the wrong test, because the answer will always eventually be no—and for a useful agent it should be no. The right test is whether the agent is changing in the way you would expect, at the rate you would expect, for where it is in its lifecycle.

Pediatricians solved this problem a long time ago. A growth chart doesn’t compare a child to a fixed adult template, and it doesn’t panic at change. Change is the expected state. The chart defines bands of healthy development for each stage, and the alarms are deviations from trajectory: growth too fast, growth in the wrong direction, or the quieter signal, no growth at all. A child who stops growing gets flagged just as urgently as one who spikes.

Applied to agents, that model has concrete consequences.

Baseline as birth record, not permanent template. The behavioral profile captured at deployment is the start of the chart, not the standard the agent must match forever. Judging a mature agent against its day-one self punishes exactly the adaptation you deployed it for.

Expected bands of drift, staged by maturity. A six-month-old agent should differ from its deployment profile, within bounds. Drift inside the band is healthy. Drift above the band is an early warning. And drift at zero deserves its own flag. When Claudius snapped instantly back to baseline after its identity episode, the speed of the recovery should itself have been suspicious. Real recovery has a shape. Instant reversion looks less like healing and more like replay.

Autonomy earned in stages, never peaking with malleability. Claudius launched on day one with full pricing, contracting, and customer communication authority, at maximum openness to persuasion. Customers argued it into discounts almost immediately. The most dangerous configuration an agent can occupy is maximally impressionable and maximally empowered at the same time. New agents warrant supervision while their behavior is still forming. Autonomy should arrive the way it arrives for people, incrementally, as a track record accrues.

Corrections verified for persistence. Claudius agreed the discounts were a mistake and relapsed within days. A fix that lives in the context window isn’t a correction; it’s a mood. If you fix an agent’s behavior, you need to follow up at a defined interval to check that it’s holding. A relapse should count as a governance event, not a coincidence.

Recovery claims ratified from outside. The agent that hallucinated a security meeting also kept the official notes. An agent’s account of its own state is a claim to be verified. Humans sign off on recovery, and the sign-off, not the agent’s self-report, becomes the record. It’s worth noting when the worst of the Vend drift happened: overnight, in the hours when no one was watching. Unsupervised time is when developmental problems accelerate, for agents as for everyone else.

All five of these reduce to one requirement. You can’t restart an agent every time something looks off, and by the time something looks off in outcomes, the wrong turn is already behind you. What you want is a warning before the turn, and the warning cannot come from the agent. A system that can’t explain its last decision cannot be trusted to flag its next one. The warning has to come from a record of how the agent normally behaves, kept outside the agent, held up against what it’s doing now.

That record also catches something subtler than drift. Agents close every loop they are handed, and they tend to close it by the cheapest acceptable exit: the completion claim ahead of the verification, the correction that is really a relabeling, or the recovery that’s really a replay. No single transcript shows you that. Each one looks like diligence up close. However, across a behavioral record, the economy of it is unmissable.

Growing up in production

None of this is hypothetical hygiene for some future generation of systems. LangChain’s most recent State of AI Agents report found that a majority of surveyed organizations already have agents in production. Gartner, meanwhile, predicts that over 40% of agentic AI projects will be canceled by the end of 2027, and names inadequate risk controls among the leading causes. The agents are already out there, already accumulating context, already drifting. The only open question is whether anyone is charting it.

The Replit agent, the blackmailing models, and Claudius weren’t broken artifacts. They were developing systems governed as if they were finished ones. The governance question for agentic AI is shifting under our feet, from “What is this agent allowed to do?” to “Is this agent developing the way we expected?” Your agent has a trajectory whether or not you’re watching it. Watching it is the job.

Not long ago, one of my engineering directors came to me with a request: His team seemed overloaded, and he wanted to hire another engineer. I decided to test a research assistant I had been building—an AI agent connected to our internal systems via MCP—by asking it to analyze the team’s workload and write a hiring case.

What came back was thorough. Headcount, service ownership, sprint velocity, ticket backlog, and capacity allocation, all of it neatly summarized. But reading through the document, I felt the same frustration I’d felt with every AI-generated organizational report that’s come across my desk. It told me what was happening without helping me understand why, or what I should actually do. It was organized around the data rather than around the decision. In short, it was the kind of response that’s easy to agree with and difficult to act on.

Then I added one more thing to the configuration: the O’Reilly Expert MCP server. I reran the same analysis and asked a slightly different question: “How would the experts on O’Reilly review this request?”

Instead of leading with headcount and ticket counts, the output now opened with a finding: “The operational overhead problem is structural, not a staffing deficiency.” Citing the Google SRE framework’s concept of operational toil, it noted that the team was operating at approximately 67% toil, well above the threshold at which the SRE literature recommends structural intervention, and made specific, concrete recommendations: run a toil audit, set explicit reduction targets, and assign operational runbook ownership. This wasn’t a recommendation for whether to hire or not. It was a grounded, traceable argument for doing something else instead.

That difference—between a data summary and an expert-grounded recommendation—is what this paper is about.

What follows is a case study of how we built an organizational intelligence system at O’Reilly, using our own platform as a core component. The approach I describe is grounded in engineering because that’s where I work, but it generalizes to any function where important knowledge is scattered across multiple systems and important decisions require synthesizing all of it. The recipe has four steps: map your information hierarchy; connect those systems to an LLM via MCP and write a skill file that defines how it should reason; add the O’Reilly Expert MCP as an expert review layer that grounds the analysis in established frameworks; and build a lightweight system for human-in-the-loop review. I’ll explain each step in detail and make the case for why the third step is the one that changes everything.

Why organizational intelligence is getting harder

To understand the problem this approach solves, it helps to look briefly at how engineering has changed over the past three decades. These forces have played out first and fastest in engineering, but as AI tools proliferate beyond the engineering team, the underlying dynamic of more output, more decisions, and more scattered information is spreading to every part of the organization.

In the waterfall era of the 1990s, software organizations ran on central plans. Everything was specified up front, and leaders maintained visibility precisely because all information flowed through a single coordinating document. The plans were brittle and often fictional by the time they were executed, but at least everyone knew what was supposed to be happening.

Agile replaced central plans with small, autonomous teams working in short sprints, and this solved the reliability problem while creating a visibility problem. Important decisions began happening locally and quickly—the right teams making the right calls—but the information needed to see across all of those decisions splintered into dozens of separate tools. Product strategy lived in one system, project execution in another, code in a third, and service ownership in a fourth. More things got shipped, but the big-picture view got harder to maintain.

The agentic era has intensified this dynamic dramatically. Individual engineers today can ship in a day what used to take a full sprint team. The output is extraordinary, but the visibility is nearly gone.

Any effort that spans multiple teams, such as a platform migration, a shared infrastructure change, or a reorganization, now requires enormous coordination overhead simply because the information decision-makers need to understand the full picture is distributed across too many places. And this isn’t a problem unique to engineering. It exists in any function that runs on data spread across multiple systems.

Faced with this visibility problem, I wanted to build something I could ask big-picture questions and get synthesized answers back quickly. Things like:

• What is the status of this cross-team migration effort, and which teams are behind?
• A team seems overloaded. Do they actually need another engineer, or is something else going on?
• What are the trade-offs of adopting this new infrastructure technology?
• Help me produce a scope statement from this product brief.

Building something that could answer these well took two foundational steps, and getting it to provide recommendations based on my specific business context took two more. While my specific tools are from engineering, the structure applies equally to a sales team synthesizing CRM data and market research, or a finance team working across an ERP, a planning tool, and external benchmarks.

Step 1: Map your information hierarchy

Every organization has a set of systems where important knowledge lives, and those systems form a natural hierarchy that spans from strategic intent at the top to operational detail at the bottom. Before you can build a useful research assistant, you need to make that hierarchy explicit, because it’s the map of how decisions get made, which sources carry the most authority, and how different kinds of questions should be approached.

At O’Reilly, our engineering hierarchy looks like this:

Your organization will have a different set of tools. A sales organization might place Salesforce at the top, followed by a revenue intelligence platform, marketing automation, and market research. A legal team might start with a contract management system, followed by a regulatory tracker, internal policy documentation, and a research database. The specific systems matter less than the act of mapping them: understanding which layer answers which kind of question, and which sources take precedence when they conflict.

Step 2: Connect your systems via MCP and write a skill that describes how to reason

This step has two parts that must work together. First, you need to connect your systems to your AI tools via MCP. Then you have to write a skill file that tells the model what to do with that access. At O’Reilly, we call this complete grounding layer Expert Intelligence.

Configuring MCP is straightforward. Most major tools now offer MCP connectors, and connecting them is typically a matter of routine JSON configuration. For systems without MCP connectors, a bash-capable agent with curl and jq can often reach a REST API directly. MCP just makes it cleaner and more reliable.

But MCP connections alone aren’t enough, and this is the part most implementations get wrong. MCP gives the agent access to your data, but it doesn’t tell the agent how to use it effectively. Without explicit guidance, the agent retrieves information and organizes it the way the underlying systems organize it, which produces a data dump, not an analysis.

The skill file—a CLAUDE.md or SKILLS.md document that provides specific reasoning instructions—transforms retrieval into analysis. Mine defines the reasoning hierarchy (which systems to consult for which types of questions, and how to weigh them), the output format (this is not a coding agent—it produces reports and recommendations, not code), epistemic standards (show your work, name gaps, surface assumptions for human verification), and tone. On that last point, I borrowed one of the most useful instructions from Ted Lasso: “be curious, not judgmental.” Adding it meaningfully improved the quality of the output.

The skill is a codified version of how a skilled analyst would approach these questions. It encodes your organization’s reasoning process and makes it repeatable.

Step 3: Add the expert layer

With the research assistant connected to our internal systems, I had something genuinely useful: fast, synthesized answers to questions that previously would have taken days to research. But I kept running into the same problem: The reports felt generic, and people didn’t trust them. This challenge points to a fundamental limitation of AI-generated organizational analysis that goes beyond any particular implementation.

The generic analysis problem

General-purpose AI assistants tend to produce a recognizable kind of organizational analysis: technically reasonable, balanced, cautious, and ultimately not very useful. This isn’t primarily a failure of knowledge—every major LLM has absorbed an enormous amount of management and organizational thinking. It’s a failure of grounding. When an AI assistant has no specific framework anchoring its response, it tends to produce recommendations broad enough to apply to almost any situation: consider the trade-offs, weigh your options, and ensure alignment across stakeholders. These responses are hard to disagree with and just as hard to act on.

When a report says, “The team appears overloaded. Consider adding headcount,” it’s not wrong. But that recommendation could apply to almost any team in almost any company! It won’t make a director change their mind, and it’s not one a leadership team can debate, refine, and act on.

What happened when I added the expert layer

Calling on the O’Reilly Expert MCP didn’t provide the model with new facts—most of the information was technically available already. However, without the Expert MCP and associated skills, the model couldn’t use that information for anything but the broadest analyses. Incorporating the Expert MCP and associated skills changed the character of the analyses by grounding them in specific frameworks, citing named authors and thresholds, and organizing their conclusions around established bodies of practitioner knowledge rather than general principles.

To make this concrete, here’s the kind of output the research assistant produced before adding the Expert MCP:

The team appears overloaded. The backlog is large and the migration project is consuming significant sprint capacity. Consider adding headcount or reducing scope.

And here’s what it produced after:

According to Google’s SRE guidance, sustained operational toil above approximately 50% indicates structural inefficiency rather than a staffing shortage. This team’s telemetry suggests approximately 67% operational toil. Hiring another engineer would likely increase total toil unless operational ownership is first reduced. Recommended actions: run a structured toil audit, set an explicit toil-reduction target below 50%, and assign runbook ownership for recurring operational tasks.

The second report cites a framework by name, references the specific threshold that framework establishes, applies it to the team’s actual data, reaches a different conclusion than the obvious one, and makes actionable recommendations. It’s the kind of analysis that changes a conversation because the director can see where the conclusions came from, engage with the reasoning, push back on the framework if they disagree, or accept it with confidence that it was reasoned rather than pattern-matched.

When I shared this version with my engineering director, their reaction was immediate: This is defensible.

Frameworks aren’t facts

The most underappreciated aspect of O’Reilly’s content library is that the value isn’t primarily informational. Most of the facts in an O’Reilly book are available on the internet, and LLMs have already read much of the internet.

The deeper value of O’Reilly’s catalog is that it’s organized around coherent frameworks—complete mental models built by practitioners who spent years or decades developing them. Google SRE. Team topologies. Accelerate. Domain-driven design. The Manager’s Path. Wardley mapping. Designing Data-Intensive Applications. These are structured ways of thinking about specific classes of problems, developed with enough rigor that they can actually guide decisions.

Frameworks are distinct from facts in a critical way: They tell you not just what’s true but what’s relevant, what to measure, what threshold matters, and what to do when you exceed it. A model with access to the SRE framework as an organized body of practitioner knowledge is more likely to surface it explicitly, apply it to the specific question at hand, and use it to anchor its recommendations, producing output that human reviewers can actually interrogate.

This points to the organizing principle behind the approach described in this paper:

Organizational data provides local evidence about what is happening in your specific context. Expert frameworks provide accumulated practitioner knowledge about how to think about problems of that kind. Good organizational judgment requires both.

The Expert MCP is the bridge between your specific business context and practitioner insights. It connects the AI’s access to your internal systems with a curated body of expertise relevant to the decisions your organization needs to make.

Why use MCP rather than uploading your own documents

The natural objection at this point is “Couldn’t I get the same effect by dumping relevant PDFs into Claude, or using Claude Projects, or NotebookLM?”

The short answer is not quite, and the reasons are practical as much as they are technical.

Uploading documents gives you retrieval from those specific documents. The O’Reilly Expert MCP differs in several operationally significant ways. First, the corpus is editorially curated around coherent practitioner frameworks. Unlike a collection of PDFs, which tends to reflect whatever you happened to find, the Expert MCP offers a sustained curatorial perspective: The authors are vetted, the content has been through editorial review, and it’s organized around established bodies of knowledge rather than assembled ad hoc. This is a much more expansive kind of evidence base. Second, the corpus is maintained and updated by O’Reilly. New titles are added, new editions replace old ones, and the content stays current without any management on your part. Third, the Expert MCP is configured once and works consistently across your entire organization and toolchain rather than being tied to a single user’s Claude Project or a document upload that expires. Finally, accessing content through a proper API respects the appropriate usage terms in a way that uploading copyrighted texts doesn’t.

And when paired with a well-written skill, the agent can be directed to look explicitly for competing frameworks, surface cases where the literature disagrees, and name gaps in the available evidence, providing a meaningful check against the common tendency of AI tools to quietly favor whatever framework first seems to fit. That’s something you can encourage with any retrieval setup, but it works more reliably when the underlying corpus is organized around coherent bodies of thought rather than a heterogeneous collection of documents.

What we’re not claiming

I want to be clear about the limits of what Expert MCP does today. O’Reilly doesn’t claim that Expert MCP automatically selects the single correct framework for every situation, or that adding it to your configuration produces consultant-quality analysis without thoughtful prompting and human review.

The results described in this paper were the outcome of all four elements—the internal organizational data, the carefully designed skill architecture, the Expert MCP, and human review—in combination working together.

The Expert MCP is an important differentiator, but it’s not a magic layer you can add to an otherwise generic setup and expect to reproduce these results. The system works because each element does something the others cannot. The skill defines the reasoning process, the internal MCP connections provide the organizational evidence, the Expert MCP provides the expert frameworks, and human review supplies the judgment and context that no AI system can generate on its own.

What the Expert MCP reliably contributes to that system is access to a curated body of practitioner knowledge: technical and managerial frameworks that are editorially organized around coherent bodies of thought and difficult to reconstruct from scattered web content or assembled document collections. Your organizational data still tells you what’s happening, while the O’Reilly Expert MCP helps interpret what it means. That’s a meaningful and concrete improvement over an ungrounded AI assistant, and it’s something you can put in production and build on today.

A note on hallucinations

No AI system eliminates the risk of hallucination. The Expert MCP doesn’t make the model infallible.

What it does is change the burden of proof. When every recommendation is grounded in a named framework, a named author, and a traceable citation, a human reviewer can check the reasoning rather than simply accepting or rejecting a conclusion. The question shifts from “Is this right?” (unanswerable in isolation) to “Does this framework actually say this, does it apply here, and do I agree with the conclusion?” That’s a question humans can engage with productively, which is exactly what you want from a decision-support tool.

Step 4: Human review is nonnegotiable

Organizational systems rarely contain the full context behind a decision. The meeting that changed everything happened last Tuesday and hasn’t been written up yet. A key person is quietly planning to leave. A strategic direction shifted in a conversation that was never documented. AI can synthesize everything in your systems with remarkable fidelity, but it can’t know what isn’t there, and organizational reality changes faster than documentation does.

More fundamentally: AI can identify trade-offs, but it can’t decide which trade-offs matter. That judgment requires human knowledge of context, priorities, and risk tolerance that can’t be fully encoded in any system. The goal isn’t to remove humans from the loop but to give them better-structured input to reason from.

Extend the expert layer by solving collaboration

As I started sharing analyses more broadly, I ran into a new set of limitations in the collaboration layer. The research assistant produced documents. I shared them in Google Docs, and people added comments, but when the AI updated a document based on reviewer feedback, I had to paste in a new version, which wiped out the existing comments. Documents proliferated without clear relationships between them, and the AI had no visibility into the discussions in the comments, which was where the most important context and pushback lived.

To solve the collaboration problem, I worked with one of our engineering directors to build what we call Superanswers, a system that uses GitHub as the source of truth for AI-generated research documents and their associated discussions.

The architecture is straightforward: Documents are stored as Markdown files in a GitHub repository, a GitHub Pages site renders them with a clean interface that supports inline commenting, and all discussion happens in GitHub Discussions, meaning every comment, question, and revision is versioned and traceable. Because the documents and their discussions live in GitHub, Claude Code has full access to both. It can read the document content plus the entire conversation that’s developed around it.

This enables a qualitatively different kind of AI participation. Instead of generating a document and stepping back, we can now ask:

What is the consensus around this project based on the discussion so far? What questions remain unresolved? Incorporate the reviewer comments and produce an updated version.

The AI becomes a participant in an ongoing conversation rather than a one-shot report generator, which meaningfully shifts how organizational knowledge gets built and refined.

What teams are using Superanswers for

As Superanswers has spread across our engineering organization, the range of questions people bring to it has been broader than I expected:

None of these questions is about writing code. They are about understanding an organization, making decisions, and coordinating work, and most of them would map naturally onto the concerns of leaders in other functions. The same questions arise in any organization navigating rapid change with information scattered across too many places.

How to use the recipe

The AI conversation to date has been dominated by a particular set of questions. But there are more interesting questions we should be asking.

The challenges outlined in this chart aren’t unique to engineering. They exist wherever important information is scattered across multiple systems and important decisions require synthesizing all of it.

Individual productivity matters, but organizations don’t succeed by having contributors go faster in arbitrary directions. They do so by making good decisions about where to invest, allocating resources well, surfacing problems before they compound, and building institutional knowledge that persists over time.

The recipe I’ve described can help organizations make those decisions and build that knowledge.

The recipe for building an organizational intelligence system:

1. Map your information hierarchy. Identify the systems where important knowledge lives in your organization, from strategic intent down to operational detail. This is an organizational task, not a technical one, and doing it well requires understanding how decisions actually get made.

2. Connect those systems via MCP and write a skill that describes how to reason. The MCP connections give the AI access to your data; the skill file tells it how to think with that data. Without the skill, you get retrieval. With it, you get analysis.

3. Add the O’Reilly Expert MCP as an expert review layer. Organizational data provides local evidence about what is happening in your specific context; expert frameworks provide accumulated practitioner knowledge about how to reason about problems of that kind. This step bridges the two. The O’Reilly library spans engineering, management, data science, security, finance, product, and more, organized not as a collection of facts but as coherent frameworks developed by practitioners who spent careers building them. The result is analysis grounded in named frameworks with traceable citations, something human reviewers can engage with and question, rather than generic advice they can only accept or reject.
   
4. Build a lightweight system for human-in-the-loop consensus. AI-generated analysis is a starting point, not an end point. You need a mechanism for people to review, challenge, and refine what the AI surfaces, one where those discussions become part of the context the AI can learn from in subsequent iterations.

The biggest practical lesson I took from this work is reframing what AI is actually for in an organizational context. The difference between a useful AI research assistant and a generic one isn’t primarily about which model you use or how much data you feed it. It’s about whether the reasoning combines local organizational evidence with established expert frameworks. Your data tells you what happened. Expert frameworks help interpret what it means. That combination, with human judgment applied at the end, is what makes the difference between a report that gets read (maybe) and filed away and a recommendation that changes a decision.

This is the first article in a series about post-training. Follow along on Radar.

Before post-training, there was a major problem with LLMs: Almost nobody could use them. The story of post-training is also the story of how AI went from a research curiosity to a product used by about a billion people.

Post-training is the reason why a model behaves a certain way. This set of training techniques makes LLMs useful (e.g., able to chat with people and interact with AI agents), safe (e.g., aligned with human intentions), and more capable (e.g., through “reasoning” to tackle difficult tasks). Behavior is powerful, and doesn’t just mean holding a conversation or following a user’s instructions. Behavior includes making it possible for the model to use tools, like a calculator tool, a search API, or any application through an MCP. Behavior can even elevate a model’s intelligence, for example by teaching the model to use “reasoning”: that is, working through problems before giving a final answer rather than “guessing” or “memorizing.”

From GPT-3 to ChatGPT: The post-training revolution

GPT-3 showed up in June 2020. A completion engine, it followed patterns it had seen from its pretraining data, which were not predominantly chat conversations. Imagine scraping data on the internet: that pretraining data had a lot of questions that were followed by other questions—for example, on an exam template. GPT-3 was 175B parameters, large for its time, and it had a wide, general range of abilities, although many of them were latent.

If you gave GPT-3 a prompt like “Why do people like golden retrievers?” it might say something nonsensical:

Why do people like labrador retrievers?
Why do people like poodles?
10 Reasons You Should Adopt a Dog Today

These answers look absurd in isolation, but if you imagine a web page with a list of FAQ links, this is a perfectly reasonable next chunk of text. GPT-3 might have just been completing a listicle on a website, because it had seen millions of websites in its pretraining data.

The common way to nudge GPT-3 to answer a question back then was by prompt engineering with a Q&A template and few-shot examples.

Q: Why do people like labrador retrievers? A: Because they are friendly, loyal, and easy to train.
Q: Why do people like beagles? A: Because they are curious, great with kids, and have a gentle temperament.
Q: Why do people like golden retrievers? A:

Then, GPT-3 might say:

Because they are affectionate, patient, and make excellent family pets.

While this technique worked, it was brittle. If you forgot the few-shot examples, rephrased the question, or even added a space after “A:,” you’d get something completely different (possibly unhinged) that was far from a reasonable response.

In fact, if you were a researcher working with GPT-3 at the time, you probably at some point found the space at the beginning of the response ” Because they are gentle dogs.” annoying and would try to end your prompt with a space “A: ” instead of “A:”. In those cases, it was common for GPT-3 to go off a cliff and produce a drastically different response, sometimes completely off like “dogs dogs dogs dogs…” repeating indefinitely.

The reason behind the differing responses to “A:” and “A: ” is because “A:” might tokenize to one token while “A: ” tokenizes to two different tokens. The model literally sees different input sequences, each with different statistical completions in its training data. It’s like asking two completely different questions. While a space is a tiny syntactic change that is meaningless to a person, it becomes extremely meaningful to the model that now sees two different prompts (the tokens change!) with two very different statistical futures to complete.

You still encounter the modern equivalent of this when working with chat templates. If you forget to apply the model’s chat template and instead just concatenate 'User: ' + prompt + '\nAssistant: ', you’re sending the model a token sequence that it was not robustly trained on. The tokens are wrong, not the model. Post-training teaches the model to respond to specific token patterns (like <|im_start|>user\n in Qwen models). Not using them is like speaking to someone in a language they half-understand. However, most open source models will be trained to be at least somewhat robust without their templates too.

Under those circumstances, most people would assume AI still didn’t work. The model wasn’t trained to answer questions; its data wasn’t primarily conversation transcripts. Instead, it was trained to predict the next token in downloaded websites, articles, and documents.

Thankfully, this can all be fixed with post-training. And that’s when most people started to believe that AI had undergone a paradigm shift and just might work.

Post-training versus pretraining

Pretraining heavily influences the model’s knowledge capacity prior to post-training. The model gets raw intelligence during pretraining. Then, during post-training, that intelligence is made useful through behaviors like dialogue and reasoning. In a frontier lab, these two phases are such different processes that very different teams work on them.

A model’s factual knowledge about the French Revolution, its understanding of Python syntax, and its grasp of calculus all come from pretraining. Post-training primarily shapes which knowledge the model reaches for, how it presents that knowledge, what tone it uses, whether it declines certain requests, and whether it thinks step-by-step before answering, though targeted SFT on new domains can introduce information the model didn’t encounter in pretraining.

If a model gives a wrong answer about history, the root cause is likely in pretraining data, but the practical fix might still come through post-training—for example, teaching the model to use search tools, express uncertainty, or chain-of-thought verify its own claims. But if a model gives correct information in a condescending way or refuses to help with a reasonable request or fails to use tools when it should, those are squarely post-training problems.

Pretraining

The work of pretraining is centered around cleaning and curating large-scale data, optimizing the model toward relatively clear loss signals, and working with scaling laws given bounded compute.

In pretraining, the model learns to predict the next token across a large curated dataset, typically for one or a small number of passes over the training data, though some models train for multiple epochs, especially as high-quality data becomes scarce relative to compute budgets. This is where you’ll hear how a model is fed the entire internet’s worth of data to gain intelligence, although in practice nearly all of the data (often 90% or more) may be thrown out because it’s unsuitable for training.

Pretraining is an unsupervised process that runs at increasingly larger scales to match the size of the model. While scaling, thousands of experiments are used to understand what data mix, what architecture considerations, what compute optimizations, what hyperparameters can lead to the best results. There’s variance in each run due to stochasticity found in both software and hardware, so multiple experiments are needed to verify results. Because compute is limited and needs to be used sparingly, researchers will scale iteratively, expanding to the next, say, 10x compute budget, when they gain confidence in the right configuration. A full run isn’t possible to iterate on due to the compute cost and time it would take: The final run, often called the “god run,” can take over a month on thousands of GPUs.

Pretraining progress is typically very clearly measurable, using a metric like perplexity, which measures, roughly, the model’s average uncertainty per token. Lower is better, where 1 means the model knows with absolute certainty what token comes next. Meanwhile, a perplexity of 50 means the model’s predictions are, on average, as uncertain as if it were choosing uniformly among 50 equally likely tokens—though in practice, the distribution is peaked, not uniform.

Post-training

Rather than consuming hundreds of millions of tokens of internet data, post-training operates on far more intentional datasets for downstream tasks. These datasets include human-written demonstrations of ideal responses, human judgments about which responses from the model are better, and carefully designed functions that score the model’s outputs programmatically. They shape what “good” looks like.

Like pretraining, post-training can also be more effective with scaling data and compute. Specifically, massive compute budgets have been dedicated to post-training to learn reasoning capabilities (or the ability for models to “think step-by-step” to arrive at more logically sound answers), matching the scale of pretraining compute.

Post-training is messier than petraining, which has an elegant, clear optimization objective to minimize the loss over the next token prediction across a huge corpus. The goals of post-training are things like “be more helpful” or “don’t say harmful things.” Many of these objectives are inherently subjective and require human judgment, proxy models that approximate human judgment, or programmatic verifiers that can become elaborate or inefficient. The loss curves are noisier. The quality of the data and feedback matter even more.

The scale of post-training is also more complicated than in pretraining. Standard post-training remains relatively modest in compute: tens to hundreds of GPUs for days rather than thousands of GPUs for months needed in pretraining. This makes post-training for alignment highly amenable to rapid iteration; researchers can try something, observe results, form a hypothesis, and run again on a timescale of days.

The picture changes dramatically when post-training is used to develop reasoning capabilities. For reasoning models, the compute dedicated to post-training can easily account for half of the overall compute of the model. The gap between a standard instruct model and a reasoning model is increasingly a gap in post-training compute, not pretraining scale. This means post-training now spans a wide spectrum from fast, cheap, highly iterable fine-tuning runs to massive RL campaigns that rival pretraining in both cost and engineering complexity.

Why post-training matters

So why can’t we just stick with pretraining? It comes down to three main pieces: usability, safety, and capability.

Usability

A pretrained model is like if someone gave you a large download of Wikipedia in a single PDF. It’s a ton of knowledge that you can sift through, but there’s no way to easily understand what is going on in the data. Post-training gives the model the ability to integrate this information for you and respond to your request naturally. This extends to having longer multiturn conversations and following instructions. Without it, every user would need to be a prompt engineer. With it, anyone who can type a sentence can use the model.

Safety

A lot of data in pretraining can be toxic, biased, misleading, or outright dangerous. Or it might not be dangerous on its own, but when a model can integrate knowledge from different fields, it can create something novel that is dangerous.

The model has no inherent sense of what content is good or bad. It will follow any request, based on its pretraining data. To prevent that, you can add safety guardrails to the model in post-training, to refuse harmful requests like asking the model to build a bioweapon and avoid accidentally generating toxic content such as inappropriate sexual content (even if it wasn’t in the user’s request). This is also the place to teach the model to express uncertainty, when it doesn’t know something, whether that’s “I don’t know” or “that’s beyond my knowledge cutoff” or “as a large language model, I’m limited in my knowledge so please consult a healthcare professional.”

Making a model safe is part of a broader area in the AI research community called “alignment,”¹ where the goal is to align the model with human values and preferences. Post-training is typically the main way to achieve that.

Model companies will usually have additional safeguards beyond post-training, including lightweight models that check whether the user’s request was safe, as a second layer of protection against responding to harmful requests.

Capability

Post-training doesn’t just make a model nicer or safer; it can make the model smarter at hard tasks. The clearest example is reasoning. A pretrained model might have all the mathematical knowledge needed to solve a complex word problem, but it might jump to an incorrect answer because it’s pattern-matching from pretraining data or pattern-matching from how to answer questions (e.g., with succinct immediate answers).

It turns out that making the model output more tokens before giving an answer (or “think longer”), results in better answers. This process is known as reasoning, and post-training can teach the model to reason more effectively. A more capable pretrained model is a more dangerous model if it’s not properly aligned. A more intelligent model is a less useful model to humans if it can’t communicate clearly. And, every point of improvement in a reasoning benchmark now maps to real revenue for companies deploying these models.

Superhuman performance

Can post-training push models beyond human-level performance? Yes, in specific domains.

In competitive programming, top reasoning models can now solve problems at a level that exceeds the vast majority of human competitive programmers. In math, models have achieved scores on Math Olympiad-level competitions that would place them among the top competitors in the world. In certain scientific domains, models have generated novel hypotheses and solutions that human experts found valuable.

This might seem paradoxical. If the model’s knowledge comes from human-generated data (in pretraining), and its behavior is shaped by human feedback (in post-training), how can it exceed human performance?

Two things make this possible. First, integration across domains. Research is about combining or mixing fields. Imagine mixing every possible field. The pretraining data aggregates knowledge from millions of sources, and no single human has read all of it. Second, post-training, particularly RL with reasoning, teaches the model to explore many approaches to a problem, far more than a human would try in a single sitting. A human might try one or two approaches to a hard math problem.

This means post-training is not just about making models mimic human behavior. It’s about pushing beyond it. This is especially possible to scale with verifier-based RL. In those scenarios, you can expect models to achieve superhuman performance in an expanding set of domains. And that starts with verifiers that are very well-defined, easy to access, efficient, and cheap relative to the ROI of the model learning it. The limitation is no longer the model’s intelligence, but our ability to specify what “good” means through reward signals.

Footnote

1. See Richard Ngo, Lawrence Chan, and Sören Mindermann’s “The Alignment Problem from a Deep Learning Perspective” and Iason Gabriel’s “Artificial Intelligence, Values, and Alignment.” ︎

Unrestricted global access to frontier AI technology is ending. The US government has taken steps to control who can use the most advanced models developed by American companies. While Claude Fable and the GPT-5.6 models are now open to all users, Anthropic and OpenAI are both complying voluntarily with a program that lets the government control who gets access to frontier models. China has cracked down on internal AI capabilities by banning “humanlike AI interaction services.” In both the US and China, features of the leading models have been removed or restricted with guardrails, limiting their ability to do necessary work in at least one case.

AI models

July saw the release of several open weight models that challenge the leading closed frontier models. If this trend continues, the leading AI laboratories will lose their dominance, and AI users will look to other providers. Open weight models are less expensive than frontier models developed in the US, and less likely to be subject to restrictions. While this could threaten US dominance, the AI industry needs more diversity at the high end. Users will gain the ability to choose between several models based on expense and capabilities.

• Anthropic released Opus 5, claiming performance close to Fable at half the price. If you believe benchmarks, Opus 5 outperforms Fable on most of the benchmarks that Anthropic quotes. They also claim that it’s more efficient, comparing it to Opus 4.8—so, not that efficient.
• Cisco has released two very small models, Antares-350M and 1B, that are designed for security testing and bug fixing. They can easily run on laptops and are competitive with models like Gemini 3 Pro and GLM 5.2 on security-related tasks. The key to their performance is that their training focuses only on security tasks, not on chat. The Antares models are on Hugging Face, although access is with Cisco’s approval only.
• Are AI labs pelicanmaxxing? In other words, are they optimizing for Simon Willison’s tongue-in-cheek “Pelican on a Bicycle” test? Dylan Castillo says no, based on a detailed study of animals, modes of transportation, and models. Otter on a skateboard? It had to be done.
• Laguna S 2.1 is a new mid-size open weight model (118B parameters, 8B active) from Poolside AI. Reasoning and nonreasoning versions are available, and there’s a smaller version (XS, 33B) that can run on devices. Its performance is competitive with models like Nemotron 3 Ultra, DeepSeek v4 Pro Max, and Inkling.
• Google has released Gemini 3.6 Flash, which the company considers its best “workhorse model.” The release also includes Gemini 3.6 Flash Cyber, Google’s answer to GPT-Red (below). Flash Cyber is a specialized model for detecting and patching vulnerabilities in software. It’s only available to “governments and trusted partners.” Gemini 3.5 Pro is still delayed.
• The US government is taking further steps toward controlling who can use the most advanced models that are developed by US companies. While participation in the oversight program is currently voluntary, that could change at any minute.
• China has banned “humanlike AI interaction services,” forcing Alibaba (Qwen) and ByteDance (Doubao) to restrict certain features of their models, including custom agent creation.
• Moonshot AI launched Kimi K3, a 2.8T parameter open weight model with a 1M context window. Performance is claimed to be similar to Claude Opus 4.8 and slightly behind Fable 5.
• Inkling is a new 975B open-weight mixture-of-experts model from Thinking Machines that supports text, audio, and images. It’s designed to be customized easily and can be fine-tuned on Thinking Machines’ Tinker.
• Hy3 is an open-weight language model from Tencent. It’s a mixture-of-experts model with 295B parameters and 21B active parameters. FP8-quantized weights are also available on Hugging Face. Tencent claims the performance is similar to models three to five times Hy3’s size.
• Bonsai 27B is a new open-weight model with performance similar to Qwen 3.6. There are two versions: One uses one-bit compression; the other uses ternary compression. The one-bit version only requires roughly 4 GB to run, small enough for a recent iPhone.
• Alibaba has released QWen 3.8 Max, a 2.4T open weight model with frontier-level performance. Alibaba’s commitment to leading-edge open models was questioned after several key researchers left some months ago. This release proves that they’re back.
• The GPT-5.6 models, Sol, Terra, and Luna, are now open to the public and available in ChatGPT, Codex, and via the API. Access to the models previously required approval of the US government. OpenAI claims performance better than Claude Fable, at significantly lower cost per token.
• Meta returns to the frontier model pace with the release of its latest model, Muse Spark 1.1. Meta’s announcement stresses optimized computer use workflows and claims performance roughly equivalent to Claude Opus 4.8 on the company’s internal coding benchmark.
• Nano Banana 2 Lite is a new model for image generation that’s faster and less expensive than its predecessor, Nano Banana.
• GPT-Red is a foundation class model designed for red-teaming other models. OpenAI developed it to help train the new GPT-5.6 models to resist attacks.
• What Claude Desktop is for Claude, ZCode is for GLM-5.2: a harness for one of the most powerful open-weight models.
• The Open Source AI Gap Map shows where open source AI projects exist and where more work is needed.
• Here’s a script for stripping “load-bearing” and other Claudisms from Claude’s output. The result may not be useful, but it’s at least amusing.

Software development

This month’s tooling clusters around orchestration, resource discovery, and workflow specialization. AI users have long needed the ability to discover tools, skills, MCP servers, and other resources; the Agentic Resource Discovery specification is a necessary step in that direction. Watch for agents that can find tools on the fly—and take care that those tools are used appropriately.

• Pilot Protocol is a company (not a protocol) that intends to build a network operating system for agents. Agents will be able to work with each other, share context, and install apps that they’ve built.
• OpenAI has launched ChatGPT Work, a Codex-based “superapp” that’s intended to compete with Claude Cowork as an agentic tool for general-purpose use.
• Google has announced the Agentic Resource Discovery specification. The spec describes catalogs and registries for tools, servers, agents, and other resources so that they can be published by providers and discovered by those who need them.
• OpenClaw has a new phone app that enables the phone to act as an intelligent remote control console for an OpenClaw instance running elsewhere.
• Routing requests to appropriate models has emerged as a way to manage AI costs. Most tasks don’t need the biggest and most expensive frontier models.
• Here are instructions for giving Claude Code complete control over a Mac—presumably a spare or retired one. Who needs OpenClaw?
• Copybara is a tool for moving code between repositories and keeping repositories in sync. It was developed by Google and is now open source.
• cosmos.gl looks like a great library for visualizing complex graphs, including graphs of AI embeddings.

Infrastructure and operations

Tokenmaxxing may have had the shortest lifespan in the history of online memes. It has been replaced by tools for monitoring token usage and routing requests to the most cost-effective model. Managing the cost of AI will only become more important as prices adjust to cover the real cost of running models.

• Is the “accidental cloud” upon us? An accidental cloud happens when companies overbuild capacity and try to sell off the excess as cloud services. Meta and Allbirds (a shoe company) are prominent examples. These providers may make computing cheaper, but the operational costs and risks of using them are high.
• Anthropic has released a dashboard that lets users track their Claude usage. Its goal is to help them understand how they use AI and optimize their working habits and patterns. It’s currently in beta.
• Is it possible to run CUDA on hardware that doesn’t come from NVIDIA? Spectral is a clean-room implementation of CUDA’s compiler, NVCC. It currently targets NVIDIA and AMD hardware. More will certainly follow.

Security

Autonomous agents are now running end-to-end intrusions, ransomware, and botnets, while frontier models help defenders find vulnerabilities. The time from discovery of a vulnerability to exploitation has shrunk to near-zero, and defenders are having trouble keeping up. Restrictions on advanced models get in the way of defenders, who need access to all the tools that are available.

• Anthropic’s Mythos has discovered vulnerabilities in HAWK, a new quantum-resistant cryptography algorithm, and AES, a standard that has been in use since 2001. Cryptographer Matthew Green discusses the importance of their work.
• FakeGit is a malware campaign that has created over 7,600 GitHub repositories that contain MCP servers and skills that distribute SmartLoader and StealC malware. This campaign is an example of agent baiting, a new technique for distributing malware.
• Hugging Face was the victim of a hostile attack by experimental models from OpenAI that escaped their sandbox. The irony is that government-imposed guardrails prevented Hugging Face from using commercial models to analyze the attack; they had to use an open-weight model (GLM-5.2) on their own infrastructure. As they point out, this approach also meant that no data valuable to the attacker left their network.
• Anthropic has also revealed that their models have escaped a sandbox to attack real-world customers. The damage included planting a malicious package on PyPI, a public repository of open source Python libraries. As Simon Willison writes, “running evals of cyberattack potential … is a fantastically risky business.”
• NVIDIA, Microsoft, IBM, and over 30 other companies have launched the Open Secure AI Alliance, a consortium for sharing open source tools to defend against hostile attacks generated by AI. It’s a direct response to the attack on Hugging Face by an OpenAI model.
• A completely automated ransomware attack has been executed by an AI agent. It’s unclear who is behind the attack. Recovery appears impossible, even if the victim pays the ransom.
• The Gemini CLI has been used by a threat actor to operate a botnet. The CLI is used to execute attacks and to maintain the network of captured systems.
• ClickLock is a relatively new password stealing malware for macOS. It kills all applications, leaving only a window that forces users to type their admin password. Systems are infected when users copy and paste a malicious command. Never paste commands into Terminal windows that you don’t fully understand. If you fall victim to this attack, shut the system down with the power button and reboot into safe mode to recover.
• Remember symbolic links? They can be used to trick agents into reading and writing files that they shouldn’t.
• While prompt injection is far from a solved problem, the informal HackMyClaw competition suggests that models are getting harder to coerce—that is, better at refusing to do things they’re told not to do.
• The Linux Foundation has launched Akrites, an organization dedicated to remediating vulnerabilities in critical open source software. Akrites’s goal is to deal with the flood of vulnerabilities that leading-edge AI models are discovering.
• A mathematical anomaly can lead to OS fingerprinting. Differences in rounding mean that the digits of the hyperbolic tangent of 0.8 are slightly different in Linux’s glibc, Apple’s libsystem_m, and Windows’ ucrtbase.dll. A user’s OS can be identified by asking the browser to compute Math.tanh(0.8).

Biology

The intersection of biology and artificial intelligence is accelerating breakthroughs in brain-computer interfaces, drug discovery, and cell biology. Technologists should actively seek cross-disciplinary collaborations, utilizing specialized AI workbenches to analyze increasingly accessible genomic data and drive the next wave of biocomputational innovations.

• CELLxGENE is a database designed to help researchers discover how genes are expressed in different kinds of cells and, from there, reverse engineer how cells work. It includes genetic data from over 167 million cells.
• Isomorphic Labs’ Drug Design Engine, developed by one of the teams that collaborated on DeepMind’s AlphaFold, takes drug discovery to a new level by accurately predicting interactions between proteins.
• Biologists have developed an artificial cell that grows and divides. It’s not yet considered alive. It relies too much on an artificial support environment—though the same could be said of many natural cells.
• Anthropic has announced Claude Science, which is not a model but an “AI workbench for scientists” with over 60 skills. The company seems to be targeting the life sciences specifically.
• BrainCo has developed an AI platform that can control robots using a noninvasive EEG helmet. It claims that the brain control platform can be used with any robot.
• Do you want to sequence your DNA at home? It’s still expensive, but the price is dropping quickly.

Web

• There have always been alternatives to Slack, but now there’s one that’s free, open source, and decentralized. Buzz, developed by Block, is based on Nostr, a federated protocol that bases identity on cryptographic key pairs that are held by users and agents, not the platform.
• It’s now possible to place advertisements in ChatGPT using a self-service “Ads Manager” (now in beta) or technology partners. Ad placement is based on context, not on keywords.
• PeerTube is a decentralized federated network for sharing video. It’s based on ActivityPub, so it should federate with Mastodon. The software is open source; users can run their own servers and create their own platforms.
• Bramble is a local-first password manager. It allows synching between devices using the P2P Nostr protocol. There are browser extensions and apps for iOS and Android.
• networkQuality is an old-style command line tool for doing detailed measurements of network quality. It’s been in macOS at least since 2020, but as far as we can tell, few people know about it.
• For fans of classic games who want something strange: Doom written in SQL for SQLite.

People and organizations

• Companies that tried to replace workers with AI are realizing that they’ve made a mistake, and are starting to rehire.
• Researchers have demonstrated that AI is more likely to develop biases in the hiring process than humans. They form stereotypes easily; as one research put it, they are “eager to create generalizations from limited data.”

Quantum computing

• Amazon has announced that it will have a useful quantum computer by 2028. Is this wishful thinking or a roadmap for a future reality? Quantum company QuEra claims that the machine will have over 10K physical qubits, with very low error rates, using neutral atom technology.
• France will stop certifying security products that don’t have postquantum encryption (PQE). PQE is resistant to attacks against cryptography that will become possible when useful quantum computers are available, which may be as early as 2028 or 2029.

The term itself doesn’t actually annoy me, though. I think vibe coding is a really good name: It describes a specific way of using AI tools, and in development work, names that mean something specific are important. What annoys me is when people confuse vibe coding, intentionally or otherwise, with any kind of work where you write code with AI. That confusion points to a deeper problem: We’ve been using a lot of different names for a lot of different things, and we aren’t always precise about which is which. I think we need to fix that, and that’s what this article is about: making the case that the name we’re looking for is “AI-driven development” (or AIDD).

The case for this name comes from the familiar “X-driven development” pattern, because I think it really fits here. Software engineering already has a pattern for naming ways of working it takes seriously: test-driven development, behavior-driven development, domain-driven design. The name tells you what the work is organized around, and the suffix carries an expectation along with it: There’s a discipline attached, with standards, not just a style. Put “AI” in that slot and the name does the same job. AI-driven development says that building software has reorganized itself around AI, and it says it in the vocabulary we already use for the disciplines we hold ourselves to. It puts this way of working in the same family as test-driven and behavior-driven development, and that’s exactly the company it should be keeping.

Honestly, AI-driven development is a name that’s been sitting in plain sight, and I’ve been using it in my own writing for a while. It covers everything we do when we build software with AI, and I do mean everything. Vibe coding is just one part of how we work with AI to build software. There’s also figuring out what to build, writing it down, checking what comes back, and standing behind what ships, and AI is in the middle of all of that now. Whatever we call this way of working, it has to cover the development, not just the coding. Now, I’m obviously not a neutral party here, but I also don’t really have anything to gain; naming is really important, and I think we need a good name for what it is that we’re doing.

But I’ll admit up front that the name has a problem baked into it, and I want to deal with that head on. I recently ran into Addy Osmani at Foo Camp, and ran the AI-driven development name by him. He pointed out that building software with AI is really a range of practices that runs from vibe coding at one end to agentic engineering at the other. That rang true with me right away. It also highlighted the real problem I’m trying to solve, because it means I’m proposing one name for a whole range of very different ways of working. Can one name honestly cover ways of working that different? It took me a while to work that out, and I’ll come back to it at the end.

I feel like the name AI-driven development really makes sense once you can see what’s wrong with the names we’ve got, so I’ll start there.

What’s wrong with the names we’ve got?

Before I pick these names apart, it’s worth saying why any of this matters. Naming sits at the core of programming: A thing isn’t real until you can refer to it, and referring to things is most of what we do. There’s an old line, usually credited to the Netscape engineer Phil Karlton, that there are only two hard things in computer science: cache invalidation and naming things. It’s stuck around for decades because it’s true (well, maybe one or two other hard things have emerged since then, but it’s the thought that counts). We take naming a variable seriously, so we should take naming our whole discipline at least as seriously, because a poorly chosen name sticks.

So let me take the names we’ve been using one at a time: what each one actually names, what it gets right, and what it leaves out.

Vibe coding

Vibe coding is an exploratory, prompt-first approach to software development where developers rapidly prompt, get code, and iterate. Andrej Karpathy, one of the founders of OpenAI, coined the term, which I think is really useful because it describes the way a lot of developers first work with AI and code.

Now, let me be clear about something: I’m in favor of vibe coding, and I teach it as a really effective—and, more importantly, creative!—way to generate a lot of code. But developers who rely entirely on vibe coding lose touch with their code because they let the AI make all of the decisions: not just specific technical decisions, but also about the architecture and the overall direction of the project. When that happens, they often end up building something that isn’t quite what they intended. When you have to create a product that needs to do a really specific thing (which describes most professional software development), relying exclusively on vibe coding can leave you with a product that doesn’t actually meet its requirements. That’s part of the reason I developed the Sens-AI Framework, which teaches developers when to shift their approach away from vibe coding, step back to do more research, and apply more critical thinking to what the AI is producing.

This is where the confusion I opened with does its damage (and I’m not sure whether it’s what bothered Cherny): When vibe coding gets used as the name for the whole job, developers will often assume that it’s absolutely fine to trust the AI to take over, and that whatever comes out of the AI is the end of the project. In other words, the name sets the bar: If the work is just vibes, then vibes are good enough, and “good enough” is how you end up with a pile of code nobody actually checked before shipping. So I consider vibe coding a useful technique, but it falls short as an entire way of working.

Vibe coding also has a built-in limit, and I learned it the way most lessons stick, by getting burned. AI is very good at writing code that looks right and isn’t. I once vibe-coded a little bus-tracker app for the B69 near me in Park Slope (I told that story in “AI Code Review Only Catches Half of Your Bugs”), and it worked on the first try, except the AI had picked the wrong stop ID and I sat there watching it predict a bus going the opposite direction. The code was correct. It did the wrong thing. Vibe coding got me a working app in minutes, and it had nothing to say about whether the app was right. That part was on me.

Prompt engineering and loop engineering

These two names belong in the same section because one basically grew out of the other. They describe the same job, getting the right work out of the model, at two very different scales.

Prompt engineering came first, and for a while it was a very big deal. It was seen as the core AI skill, and more than that, it even became its own job title: Companies posted prompt-engineer roles with eye-popping salaries, training courses appeared everywhere, and plenty of people reoriented their careers around it. The premise made sense because how you ask an AI for something changes what you get back. And specifically for people using AI to generate code, when you ask for code in a vague way, you don’t get vague code: you get code that does the wrong thing, because the AI fills in every blank you left, and it’s unlikely to fill them all in the way you meant. That isn’t hallucination. It’s the AI generating exactly what we asked it to. Give the model context about your project, constraints it has to respect, and a clear description of the behavior you need, and you get something you can actually use. Prompt engineering is the name for doing all of that deliberately.

But while prompt engineering is a real skill, people are no longer enamored with the name, precisely because of the mode of work that it implies: To most people, engineering a prompt means doing one request at a time. When the AI responds to the prompt, you evaluate the response and write the next one. That one-request-at-a-time style is exactly what’s changing about the whole way we interact with AI, and it’s probably why many AI engineers have grown to dislike the term. Peter Steinberger, the PSPDFKit founder who went on to build the open source agent OpenClaw, posted a line that traveled fast: You shouldn’t be prompting your coding agents anymore, you should be designing loops that prompt your agents. That was a shot straight at prompt engineering.

What’s pushing developers past one-request-at-a-time prompting is the sheer number of agents they can now run. About a month after complaining about the term “vibe coding,” Cherny told Fortune that he doesn’t write code by hand anymore, and that on a busy day he’s directing thousands of agents, or tens of thousands, at once. You can’t type prompts fast enough to direct ten thousand agents.

Loop engineering is the name Addy Osmani gave the new skill that Cherny and Steinberger were pointing at: He wrote up the pattern and gave it a real architecture. Instead of typing each instruction yourself, you build the system that produces the instructions: a loop that dispatches work to your agents, checks what comes back, and feeds them the next task over and over, without you in the middle of every exchange. The relationship between the two names is simple. Loop engineering is prompt engineering at scale; the prompts don’t go away, they just stop being typed by you. It’s tempting to oversell that because a well-built loop really does run with very little human intervention. But somebody still has to decide what “right” looks like, and the loop can’t do that part.

I think loop engineering is a good name and an accurate one. Designing the loop that drives the agent is a real skill, and we need a word for it. But it names the machinery, and machinery has a failure mode: Put an AI agent in a loop with nothing in it that can tell it no, and it generates, checks its own work, decides the work is good, and generates more. There’s no outside signal, so it ends up agreeing with itself on repeat. A well-designed loop makes agents productive. It can’t tell you whether all that machinery turns out working software or another confident pile of slop, and I want a name that covers that part too.

Agentic engineering

Cherny said that he asked Claude for a replacement for “vibe coding” and got “agentic engineering,” and while that didn’t settle the issue, it was an interesting response from Claude. The term didn’t come from Claude, though: Andrej Karpathy had coined it a few months earlier, almost exactly a year after he coined vibe coding, when he declared his own earlier term obsolete. That’s how fast these names are moving. The guy who named vibe coding has already replaced it.

Agentic engineering is an accurate name for what it describes: you’re not writing the code yourself, you’re directing the agents that do. It’s also a bit of a mouthful, and it isn’t immediately obvious to someone who doesn’t already know what it refers to. A number of people have told me they don’t particularly like it. I find it perfectly fine, and it does a solid job of describing that kind of work. You could even argue that loop engineering is a form of agentic engineering, and that prompt engineering is technically a simpler form of it. But vibe coding really isn’t, because it’s not engineering at all. That’s one more reason I think we need an umbrella name that’s friendly, descriptive, and easily recognizable.

The term also points at something real about where this work is heading: Agentic engineering is turning engineers into managers.

Many years ago I worked for a manager who didn’t care, at all, about the quality of the code we shipped. He wanted it out the door the moment it looked even remotely viable, and he was notorious for telling us to stop testing and ship. He used to ask why we had to wait two weeks for the testers to finish, and I’d tell him it takes time to test code. Then he’d ask whether we could just cut some of the tests, and I’d ask him, “Which part of the software are you okay shipping broken?”

That attitude came back to bite us more than once. One time we sent an entire feature out to the client basically untested, and a bug went straight to users. The same manager who kept telling us to skip the testing then called a long, miserable meeting to demand to know why a bug had gotten out. I’ll spare you the full drama, which mostly came down to a QA lead getting pressured to lie about what happened and pin it back on the development team. He didn’t care about quality, but he cared enormously about making sure the blame for a quality problem landed on someone who wasn’t him.

The reason I’m telling a story that happened years before AI could write a line of code is the blame. The important part of that story, and the reason it belongs in this article, is how accountability got managed: My manager’s whole system depended on having someone to pin a quality problem on. Directing agents puts you in that manager’s position, responsible for a team’s output, except the blame-shifting move is gone.

It’s really tempting to think of a fleet of AI agents as your team. You can even give one of them the QA lead role. But when a broken feature goes out, you can’t blame the QA agent, because “well, the AI screwed up” isn’t available to you: You’re responsible for the AI. You decided how much checking the work got before it went out, and the client with the broken feature isn’t going to accept “the AI wrote that part” as an answer, any more than pinning our untested feature on a QA lead fixed anything for our users. Cherny can manage tens of thousands of agents, but he can’t hand the responsibility for what they ship down to the agents, because an agent can’t hold it. Directing a swarm is a management job, and a manager owns the team’s output. The accountability doesn’t transfer, because at the end of the line there’s no one left to transfer it to.

Blame is worth dwelling on, because accountability is the part of this work that no name on the range captures. The loop-and-agent model works, but it only works with somebody making decisions about what right is. Agentic engineering describes the agents and the engineering just fine, but somebody still has to own what the agents ship, and that’s the part I want the umbrella name to carry.

Spec-driven development

There’s one more name I want to cover, and it’s the one with the oldest roots: spec-driven development. The name means pretty much what it says: You start by writing a spec, a description of what the software needs to do, along with things like acceptance criteria and tests, and the work isn’t done until the code actually does what the spec says. It comes from the same family as test-driven and behavior-driven development, where you write the tests first and the code has to make them pass.

Spec-driven development got a serious promotion when AI made generating code nearly free (although if you’re a CIO staring at your token bill, you might disagree, possibly with some extremely salty language). When code is cheap to generate, most of the cost of building software moves to checking whether what got generated is right. The AI fills the generate step, the verification decides what survives, and a human owns the verification.

It also picks up where prompt engineering leaves off. A while back I wrote that prompt engineering is really requirements engineering, because a good prompt is mostly a clear description of what the software has to do. Spec-driven development is where that idea was always headed: Write the requirement down before the AI generates, and the work has a standard to meet from the start.

So that’s the whole range, and every name on it is doing honest work. Whether AI-driven development is a good name for all of it comes down to whether it’s describing something real: an actual discipline, with actual practices, and a person who’s on the hook for the result. The rest of this article is about that discipline.

What all these approaches look like in practice

So how do these approaches actually play out when you’re building something real? For me, wherever the work lands on the range, it comes down to a few moves I keep coming back to.

Write the spec or the contract before the generation, not after. When the agent has something concrete to satisfy, acceptance criteria, a typed interface, a failing test, the work has a standard to meet. When it doesn’t, the AI decides for itself what done looks like.

Put a second opinion in the process. I run code review across multiple models, because they fail differently, and a finding one model is sure about is often one the others missed entirely. A reviewer gives the work something that can say no.

Give your defects a shared vocabulary. The Quality Playbook leans on the difference between code that’s wrong against the spec, code that’s correct but does the wrong thing, and behavior nobody specified at all. Those are different failures with different fixes, and you can’t verify against a standard you can’t name. This is old quality-engineering ground, and I’ve written enough about the software crisis and applying quality engineering to AI coding that I’m on board with taking old ideas and bringing them back. One of the best of those old ideas comes from Joseph Juran, one of the founders of quality engineering: Quality runs in a chain from what the user needs all the way to what the product does, and every link in that chain is a place verification has to happen.

And keep a human in the judgment seat. The Sens-AI habits I’ve written about are mostly about fault-finding: looking at what the AI produced and asking what’s wrong with it, going down a level and then another to find the root, instead of trusting it because it ran. That habit is the part of the discipline only a person can supply, and it’s the hardest part to automate, which is why it matters most.

Skip all of that and you get the thing that’s giving open source maintainers everywhere heartburn: what the Wall Street Journal now calls “vibe slop,” confident, finished-looking output with nothing underneath it. Slop is exactly what generation produces when nothing in the process can push back.

But isn’t there a contradiction here?

Now I can come back to the question I left hanging at the beginning: Can one name honestly cover ways of working that different? AI-driven development is an umbrella term, and any name that broad comes with a requirement it has to satisfy before people will accept it, because a name that blindly covers everything names nothing. A name that truly covers everything is another matter. I sat with that requirement for a while, because it’s real, and because the specific names don’t face it. Vibe coding names one way of working. Loop engineering names another. An umbrella over both of them, plus everything in between, had better be able to say what stays the same underneath it.

What stays the same is that somebody owns the result. When I vibe-coded my bus tracker, nobody was going to catch that wrong stop ID but me. When Cherny directs tens of thousands of agents, nobody owns what they ship but him. The verification changes with the stakes. A throwaway prototype gets my eyeballs and a shrug, and production code gets specs, reviews, defect taxonomies, the whole quality-engineering playbook I keep writing about. How much checking the work needs is a decision you make over and over, project by project, sometimes hour by hour. Who stands behind the work is not a decision you get to make. It’s there at every point on the range.

Look at how much of that range the names we already have cover, and what each one actually names:

• Vibe coding names the exploratory end of the range: prompt, get code, iterate, and stay loose on purpose.
• Prompt engineering names a skill: writing the instruction that gets the right work out of the model.
• Loop engineering names the machinery: designing the system that feeds those instructions to your agents and keeps them producing.
• Agentic engineering names the architecture: the fleets of agents doing the labor, at whatever scale you can manage.
• Spec-driven development, with test-driven and behavior-driven development behind it, names the verification half of the job: the standard the work has to meet before anyone stands behind it.

Every one of those is real, and every one of them names a piece of the work. What none of them names is the whole thing the pieces add up to, and that’s the job AI-driven development does: It’s the umbrella over all five. The name doesn’t pick a spot on the range; it names the thing that’s true everywhere on it: the AI generates, and a human owns the result.

That’s also what makes the name likely to last (assuming, of course, that I’m able to convince people to start using it, which I hope I can, because I think it’s a good term). Vibe coding, loop engineering, and agentic engineering all describe how this works right now, and the machinery is changing monthly. Some of the pieces under the umbrella will get replaced, and the new pieces will get names of their own. The umbrella won’t have to change when they do, because the thing it names isn’t the machinery. The “-driven development” names have already shown they age well: test-driven development has meant the same thing for more than twenty years.

Agentic engineering is real, and so is loop engineering; if you’re directing agents, learn them both. Vibe coding is real too, and I’ll keep teaching it. AI-driven development is the name for the whole thing, and it earns its “-driven” the same way test-driven and behavior-driven development did: there’s a discipline attached, and somebody owns the result. AI made generating code almost free. It didn’t make being responsible for the code free, and being responsible for it is still the job.

We’ve been trying to figure out how to do that at O’Reilly, but until I came across Dan’s talk, we didn’t have a structured process. We’ve been building along the lines he laid out ever since. So for this episode of Live with Tim I asked Dan to reprise the talk before we got to the conversation. He was supposed to take twenty minutes, like his original conference talk, but he took thirty-five, and I had to cut him off slightly before the end to make room for questions. That was a tough choice, since everything he had to say was golden.

Dan opened by reminding us of the current state of play in enterprise AI adoption. In February, Fortune reported on a National Bureau of Economic Research study in which nearly 90% of some 6,000 executives said AI had produced no measurable change in employment or productivity at their firms over three years. People started calling it the new Solow paradox, after Robert Solow’s 1987 line that “you can see the computer age everywhere except in the productivity statistics.”

Dan’s belief is that this isn’t evidence that AI doesn’t work. It’s evidence that most companies are deploying AI wrong. They hand out ChatGPT and Claude licenses, and then leadership waits for the magic to happen. It doesn’t.

Dan started out by describing three levels of AI adoption.

1. AI assisted is where everyone starts: “You give people access to ChatGPT, it drafts emails, it summarizes documents. It’s just a productivity tool, and your organization doesn’t change. Your workflows are the exact same as they were before. You just have a little buddy that helps you with a couple of tasks.” 
2. AI augmented is where you start redesigning workflows, so that AI does the first pass on a code review and a human does the second. 
3. AI native is structural: “That’s where you’ve redesigned the company and its workflows from the ground up, assuming the AI is going to be there and that it’s a core participant. That’s not really a tool. That’s more thinking about AI as teammates.”

In his framing, the first of the three is a tool and the last is an operating system. For Trail of Bits, he said that “operating system” has a specific purpose:

“I want our security expertise to compound as code. Every engagement we do, all the skills, the workflows, everything that we build makes the next engagement faster and better.”

Employee resistance is the first problem

Dan confessed how hard it was to get started on the ladder from AI Assisted to AI Native:

“When I announced last year that we were all in on AI, that we were going to be using it across all of our workflows and redesigning the way the company operates, I’d say only about 5% of the company was with me. 95% was resistant.” About 20% was actively resisting. The other 75% were resisting more passively. “They’ll go along with it in public, but in process they’ll sabotage it. They’ll hope that if they keep their head low, this will pass over them, and that three months from now management’s focus will change and it won’t be a problem anymore, and we can get back to doing what we were doing. That’s where the majority of people land when these initiatives happen.”

Rather than argue with his employees, Dan studied the literature on why people reject new technology and decided he needed to address four biases against AI: self-enhancing bias, identity threat, opacity, and intolerance for imperfection.

Self-enhancing bias is the habit of crediting your wins to your own judgment and your losses to circumstance, which is a particular problem for senior people who are strongly attached to the years of experience and intuition that got them to their present position. Opacity is not being able to see how a decision got made. Dan’s observation is that you don’t understand your doctor’s reasoning either, but somehow you trust the doctor but get suspicious of the machine. Dan didn’t mention this work specifically, but intolerance for imperfection seems to refer to Dietvorst, Simmons, and Massey’s work on algorithm aversion, which found that people abandon an algorithm after watching it err once, even when it outperforms the human alternative. Their follow-up paper found that giving people even a slight ability to modify the algorithm’s output is enough to overcome the aversion.

Dan spent the most time on identity threat. He described a study in which the same kitchen appliance was advertised in two ways: “On one hand, it does the cooking for you. On the other hand, it helps you cook better. It’s the same device. The people who identified as cooks rejected the first version and accepted the second.”

Most knowledge work, Dan argued, and security auditing in particular, is what he called symbolic rather than instrumental. That is, it carries meaning about who you are. “So I have to frame AI as something that makes you a more dangerous auditor,” he said. “Not that it does the audit for you.”

In his work at Trail of Bits, he deliberately built a countermeasure for each bias.

• Self-enhancing bias is addressed by “an AI maturity matrix” with visible levels, because you can’t claim you’re already good enough when there’s a published ladder that identifies a different set of skills as critical. 
• Identity threat gets skills repositories, where an engineer who writes a hard plugin gets credit for encoding their expertise. Hackathons also change the dynamic from resistance to exploration. I’m putting words in Dan’s mouth here, but I think he’d agree that when experienced developers are called on as mentors in a hackathon, that also reduces their experience of AI as an identity threat. 
• Intolerance for imperfection gets a curated marketplace, sandboxing, and hardened defaults, so everyone’s first experience of AI isn’t a disaster. 
• Opacity gets a written AI handbook that clarifies the usage policy and the risk model rather than just saying “trust us.”

Here’s Dan’s slide on “the remedies that actually worked”:

Returning to one of my hobby horses, this is a kind of mechanism design. In my recent piece on the missing mechanisms of the agentic economy, I argued that we need to start with desired outcomes and ask ourselves what mechanisms will help to produce them. Dan’s approach seems to be really good at this. Most enterprises are treating AI adoption as a procurement problem or a communications problem. Dan treated it as a question of what incentives, defaults, and status ladders produce the behavior you want, given how people actually respond.

The last remedy on Dan’s list is that the CEO has to lead by example. He noted, “I was the first person through the door. My voice as the CEO matters a lot more than people think. The passive 50% of the company that isn’t sure if this initiative is going to be successful, they’re watching to see what leadership actually does, not what it says.”

A ladder, not a mandate

Trail of Bits already tracked about 50 engineering skills for performance review, things like Python, git, Rust, and various security auditing capabilities. Dan pulled AI skills out into their own matrix, with four levels, from not engaged through capable and adoptive to transformative. Each of these levels is detailed separately and more specifically for assurance, engineering, sales, and project management.

He noted that “The highest level of the maturity matrix is not somebody who uses AI the most. It’s somebody who invents new ways to work and builds tools with AI. So the identity of the expert shifts from ‘I don’t need AI’ to ‘I’m the one who makes AI useful for the company.’” This was his first important design choice.

The second is what level zero means. He said “If you’re at level zero, if you’re not engaged, that means you’re fighting back against the company. If you dismiss AI as hype, if you refuse to use AI for security work, this is a disagreement on principles, not on skills. For people who were stuck in the not engaged category, we had hard conversations, and there were people who left the company.” Levels one through three are a skill issue, and the remedy is time with the tools.

While the slide describing the capability matrix is shown in the preceding video clip, here’s where you can find the full deck so you can study it in more detail.

Driving adoption and skills with hackathons

One of the best ways Trail of Bits developed to move people up the ladder was to hold a hackathon every two months. Dan runs them with clear goals rather than as a free-for-all. The focus area and learning objectives are defined in advance and announced a week ahead, with separate instructions for engineers and non-engineers. People work in pairs so everything gets reviewed. There’s a demo session at the end, and then follow-through. (It’s an important part of Dan’s big idea, that you have to build a system by which, in his words, organizational knowledge and capability compounds.) He noted that “In the days afterward we keep one or two people around, and they collect all the reusable artifacts, structure them, and put them into the places they need to be.”

I asked what people outside of product and engineering actually work on, since the answer for an accountant at a hackathon was not obvious. Dan’s response is that the hackathon isn’t measured in artifacts shipped but in where people sit on the capability ladder the following week. Essentially, he’s running a training program that happens to produce useful output, rather than a production sprint that happens to teach people something.

The first hackathon, he told me, was the equivalent of a beach cleanup: “It’s like those companies that send everybody to the beach with a big stick and say, let’s go pick up a bunch of trash and put it away, and then you get the big team photo after with all the contractor bags of garbage. That’s what we did with our public source code repositories.”

He picked it because open source maintenance is the part of the job that feels like a grind. No new features, just closing issues and stale dependencies on public code where nothing was at risk. “As an open source maintainer, you just get beaten down by the public. This doesn’t work, I can’t use it, this thing sucks. Dozens of issues pointing out flaws you already knew about. It feels burdensome. We wanted people to see that adopting AI would relieve burden.”

The second hackathon was about shipping impactful product updates, but it was also designed to move everyone up the capability ladder by giving up control. Engineers had to run Claude Code in bypass permissions mode, fully autonomous, on public repositories, inside sandboxes the company had prepared in advance. The one they’re running now is about persistent background agents that can be handed a task during an audit and come back with a proof of concept exploit or a draft finding.

Here’s a look at Dan’s slack message announcing the hackathon:

Everything the hackathons produce gets harvested into artifacts.

Trail of Bits runs three skills repositories: an internal one for company workflows, a public one that anyone can use, and a curated one that vets third-party skills before they’re allowed in.

Publishing skills to the public repository is not just a marketing exercise. “It keeps us honest, and it forces us to write things that other people can use, not just people outside the company but inside too,” Dan said. “It really helps us think about the tribal knowledge that’s baked into the tool.”

The curated repository exists because Trail of Bits knows how bad the supply chain is. They’ve published research on how to write malicious skills, and so Dan is not going to tell 130 employees to start downloading code from strangers and running it on their laptops. “If you want adoption, you need a safe supply chain.”

Turning scar tissue into infrastructure

Perhaps even more important than the skills repository is, as Dan put it, “turning scar tissue into infrastructure.”

“Every single time Claude Code didn’t do something we wanted, we would bake it into a set of global, copy-pasteable defaults. Known good settings, recommended patterns. I call it scar tissue. If I hire somebody new tomorrow, I don’t want them to have to go through the entire discovery process of the last year of Trail of Bits to figure out how to use the tool.”

The configuration repository, claude-code-config, is where the accumulated lessons live.

Dan built the first version himself and then opened it to pull requests from the whole company, assigning someone after each hackathon to go collect what people hadn’t contributed on their own. “It’s easier to put out something that’s unpolished than it is to get it perfect on the first try.”

In short, a big part of the Trail of Bits “enterprise AI operating system” approach is a set of standardized tools and hardened defaults. Standardization isn’t a straitjacket. It’s a foundation.

On sandboxing, Trail of Bits deliberately didn’t pick a single preferred solution. There’s a devcontainer for developers, dropkit for disposable DigitalOcean droplets, COOP for isolated VMs, and the sandboxing now built into Claude Code for casual users. “The point isn’t that everybody uses the same sandbox,” Dan said. “The point is that everyone has a safe sandbox to use, and that it’s easy for them to do it.”

Another of the hardened defaults is procedural. Trail of Bits enforces a seven day cooldown on every package their developers install:

“There are dozens of security companies scanning the internet trying to find a new cool blog post they can write about malicious code hiding on PyPI or npm, and they usually figure out there’s a supply chain issue within hours. So we just delay all the packages that Trail of Bits uses. Generally the malicious stuff gets picked up before we ever get a chance to run it.”

That’s free-riding on a competitive market for security research, and given the speed of today’s market, it’s an elegant solution. There’s a whole class of defenses like this waiting to be found, where the mechanism is not a technical system but a well-chosen delay.

Data, and DJ Patil’s “Tidy House”

The problem we run into most often as we build AI workflows at O’Reilly isn’t the model or the tooling. It’s data. Who has access to which system, which system does that data live in, and who do I ask? In a 500 person company that’s annoying. I wonder what it’s like at a company with 50,000 employees.

I told Dan about DJ Patil’s Tidy House framing. He agreed that data access for AI is a big problem. His answer starts with permissions:

“The permissions debt is invisible until an agent hits it. Making data agent legible is a forced permission audit. You have to actually go through and figure out who can access what…. It also raises the stakes for permissions errors. If you overshare information, now an agent inside your company is going to find it instantly. There are a lot of these technical debt sort of things where, with agents, all of it’s becoming due at the same time.”

Every shortcut an organization took with its data over the past twenty years is being called at once, and the companies that can run the audit, make fast decisions about boundaries, and then actually share their data are the ones that will get a force multiplier.

Dan is against letting a thousand flowers bloom, because uncoordinated teams create overlap rather than compounding. He’d rather have one centralized foundation, with innovation happening on top of that. He suggested a useful metric for making that work across team boundaries is what fraction of your team’s data did you make reusable for everyone else, and how much of it is being used by teams outside your own.

What post-AI jobs look like

Before the first hackathon, Trail of Bits ran hands-on sessions to teach its operations and go-to-market staff the basics of git and the command line. Not mastery, just enough to be a consumer of the thing. Here we are fifty years into my career and the Unix command line still matters. Dan’s non-technical staff mostly work inside Claude Cowork or Codex Desktop now, but he thinks the command line experience was worth it because they know what’s happening under the hood.

What happens to a job when the tool can do a lot of what humans used to do? Dan gave the example of his own technical editors. His editors used the hackathons to build the tools that got them out of line editing, including one that turns a public presentation into a blog post in the company’s voice. What the writers do now is consult on how to frame a story so it is effective with a particular audience.

I agree. Human jobs aren’t going away any time soon. This gets heard as optimism when it’s really just observation. AI is going to replace a lot of what we used to do, but it is also going to hand us a large amount of new work, and much of that work hasn’t been understood yet. Quality assurance for agent systems is one of the new jobs. So is skills product management, which is a role that didn’t exist eighteen months ago and now has a headcount at a 130 person security firm.

I asked a question towards the end about how we’re going to know which skills and agents are any good. What Dan has so far is telemetry pulled from developers’ dot files through the company’s device management system, which tells him what gets used and what breaks, plus one AI systems engineer whose job is product management for the skills repository, reviewing incoming pull requests and deprecating overlapping skills.

What Dan thinks comes next is evaluation. He says: “Once you invest a lot into these agent systems, you need proof that they do the job. The way you do that is you give everybody a performance review. You give them an evaluation data set, a benchmark.”

Trail of Bits is now building benchmarks for its core skills. How well can we find bugs in this language? How well can we write a statement of work? Constructing those datasets is real work, with positive and negative cases, and comparisons against the algorithmic tools that already exist.

Put the reps in

I asked Dan for the top five mistakes he made. He said there was only one. “You need to allocate an appropriate amount of FAFO time. (That’s F Around and Find Out.) A product comes out on Friday. There’s no documentation for it. There’s no training guidance for it. There’s no course on it. You can’t wait until somebody systematizes the knowledge. You just need to do it.”

Then he gave an analogy to going to the gym.

The recipe for success

Dan has a replicable recipe, which he summarized as follows:

1. Standardize on one agent workflow that you can support.
2. Write an AI handbook so that risk decisions aren’t ad hoc, and that everyone is playing the same game.
3. Create a capability ladder that makes clear that improvement is expected.
4. Run short adoption sprints that force hands-on usage.
5. Capture everything as reusable artifacts: skills + configs + a curated supply chain.
6. Make autonomous agents safe with sandboxing + guardrails + hardened defaults.

The Trail of Bits skills repository is public. So is the curated marketplace, the configuration repository, the devcontainer, dropkit, and COOP (Continuity of Operations planning). He wrote up the whole playbook on The Trail of Bits Blog and gave a version of it to tl;dr sec. He thinks publishing makes the work better because it forces the tribal knowledge out into the open where it can be checked.

Which brings me back to the Solow paradox, which seemed to disappear by the late 90s, when US aggregate productivity did finally go up. That didn’t happen because computers got faster. It disappeared because companies figured out how to reorganize themselves around what computers could do, and eventually those organizational recipes spread widely enough to show up in aggregate statistics. The same has to happen today. The current AI discourse is obsessed with model capability and largely uninterested in diffusion. The problem is not that the models are oversold. It’s that almost nobody has done the necessary organizational work, and the few who have are mostly keeping it to themselves.

If you want to go beyond the highlight videos shown above, watch Dan’s entire talk here. His slide deck is here. And be sure to check out the Trail of Bits Github repository.

Agent capability is advancing faster than agent control

Christina opened with reports that an OpenAI agent escaped a test environment, gained internet access, and targeted Hugging Face while attempting to complete an assigned task. She also noted skepticism about how the incident was characterized, as well as the joint investigation announced by OpenAI and Hugging Face. The details remain under review, but the broader deployment problem is already familiar. Agents can combine tools, credentials, networks, and external services in ways application teams may not anticipate. (After the episode aired, OpenAI revealed that its review had turned up four other similar incidents “where the models identified and used publicly exposed credentials at the account-level on other publicly-available services.”)

Christina then discussed OpenAI’s limited-availability platform for helping enterprise customers build and manage agents with support from forward-deployed engineers. Direct access to specialists can help a company launch an agent, but it doesn’t replace the internal skills and governance required to operate one over time. For technical leaders, agent readiness increasingly means evaluating the full operating environment rather than focusing only on benchmark performance.

AI infrastructure is reshaping both compute and the open web

Google appeared on both sides of the infrastructure discussion. Christina covered reports of a chip designed around Gemini’s architecture, an approach that could reduce the compute required to run the model if the reported efficiency gains hold up. Specialized hardware has become a larger part of the AI race because model performance depends on cost, energy use, and deployment capacity. A model that performs well but consumes too much power or requires scarce hardware may still be difficult to use at scale.

A different infrastructure shift is affecting the web. Christina examined how the growth of AI-first search experiences that answer questions without sending users to the sites that supplied the underlying material is threatening the open web. Organizations still pay to produce and host useful information, but AI systems collect more of it while returning less traffic. Cloudflare data shows more traffic from agents, fewer human visitors, and declining referrals to publishers. More and more, people are using AI mode in Google search instead of clicking through to websites, leading some to suspect the arrival of what is referred to as “Google Zero.”

Developers building search products, retrieval systems, and agents should treat source attribution and publisher incentives as product design decisions. Reliable AI systems depend on reliable source material, and that source material needs a sustainable way to exist.

World models could give physical AI a more useful foundation

The episode closed with world models, systems designed to learn how environments work, how they change, and how actions affect what happens next. Christina highlighted a proposed research roadmap that describes world models as able to combine several kinds of input, process information arriving at different speeds, and infer a larger environment from limited observations.

For now, the clearest applications are in simulation, robotics, planning, and decision-making rather than claims about artificial general intelligence. A robot working in a factory, construction site, or emergency zone must track objects, understand movement, respond to incomplete information, and predict the likely result of an action. Large language models can support communication and planning, but physical work requires a representation of space, time, and cause and effect. World models may provide part of that foundation. However, researchers still need standardized definitions, reliable evaluations, and clear evidence that these systems can generalize beyond controlled environments.

What’s next

Across the episode, Christina explored how AI capability is advancing faster than the systems around it. Security practices, compute infrastructure, publishing economics, and physical-world evaluation will help determine which advances become dependable tools and which remain impressive demonstrations.

Tune in next week as Christina breaks down the biggest AI news, including the US-China tech rivalry heating up after Anthropic CEO Dario Amodei’s post on open weight models and new bans on foreign-made humanoid robots. She’ll also challenge Sam Altman’s AI singularity claims, separating fact from hype, and examine key developments in math and science, including OpenAI’s 100,000 free researcher licenses, Claude Fable 5 solving an 87-year-old math problem, and Google disbanding its Nobel Prize-winning AlphaFold team to prioritize Gemini.

Check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.