The following article originally appeared on Addy Osmani’s blog site and is being republished here with the author’s permission.

Technical debt lives in your code. Cognitive debt lives in your head. Intent debt lives in the artifacts you may never have written: the goals, constraints, and rationale for why the system is the way it is. If you’re lucky, some of this exists scattered in team documents or discussions, but it’s likely incomplete. It’s the one kind of debt your agents can’t pay down for you, and agentic engineering makes it the most expensive.

____________________________

Three places debt can live

Margaret-Anne Storey’s Triple Debt Model is a clean way to think about software health. The three models of debt are technical, cognitive, and intent.

Technical debt lives in the code. It’s the accumulation of implementation choices that make the system harder to change later: the tangled module, the shortcut you took under deadline, the abstraction that leaked. We’ve understood this one for decades. You feel it coming through slow builds, fragile tests, and the dread of touching one particular file.

Cognitive debt lives in people. It’s the erosion of shared understanding, the gap between how much code exists and how much any human understands. I’ve been calling this comprehension debt. It builds up when the system grows faster than the team’s mental model of it. Your code can be pristine and you can still carry crippling cognitive debt, because nobody understands the pristine code either.

Intent debt lives in artifacts. It’s the absence or erosion of the externalized rationale, goals, and constraints that explain why the system is the way it is. The key word is externalized. The rationale has to be written down where a teammate, a future you, or an agent can read it, not held in your head. When intent debt runs high, the system drifts from what you meant it to do, and nobody can say when it diverged or why.

These three are independent, which took me a while to internalize.

You can have low technical debt and high intent debt. You can understand a system completely yourself (no cognitive debt for you) while its intent exists nowhere outside your skull (enormous intent debt for everyone else).

From the inside they feel alike, but each one bills you separately.

Why intent debt is the one agents can’t help with

AI generates code faster than ever, which makes technical debt cheaper to take on and cheaper to pay down. Point an agent at a tangled module and it’ll refactor it.

Cognitive debt recovers too, more easily than most engineers expect. When you don’t understand a chunk of the system, you ask the agent to explain it. You rebuild part of the lost mental model on demand, because the code still exists and the model can read it back to you.

Intent is different. An agent can’t generate intent, because intent is the one input that has to come from you. A model can infer a plausible rationale from the code, the same way you can guess why a previous engineer did something. A guess about intent isn’t the intent. The model doesn’t know whether that 300ms debounce was a deliberate UX decision, a benchmark result, or a number someone typed once and never revisited. It will invent a confident-sounding reason, which is worse than admitting it doesn’t know.

Of the three debts, intent debt is the only one where the agent can’t bail you out. It can write the code and restore your comprehension. The why is the one thing it can only fabricate.

Agents make the unwritten cost compound much faster

Teams got away with high intent debt for years because we carried it in our head and old docs.

When a new human joined a team, you didn’t write everything down, because they picked up intent over time: hallway conversations, code review comments, “Oh, we don’t do it that way because of an incident in 2023.” Knowledge moved person to person and built up. The engineer who’d been there four years was the intent documentation, expensive and lossy, but it worked.

Agents break that model. Bringing agents onto a team doubles its size overnight with junior people who have no long-term memory. An agent starts most sessions cold. It carries none of the tacit intent humans built up over years. Whatever you haven’t externalized into an artifact it can read, it doesn’t have.

That changes the economics of not writing things down. Unexternalized intent used to cost you once in a while, at onboarding or after someone left. Now you pay it every session, multiplied by every agent you run.

Picture the 20 agents you’re so excited to parallelize. Each one is a teammate who has never met you, can’t read your mind, and will fill any gap in your intent with a plausible guess. The orchestration tax I wrote about is partly an intent-debt tax. Much of what makes managing many agents exhausting is resupplying the intent you never wrote down.

The other half of the comprehension debt argument

When I wrote about comprehension debt, I made a point I want to revisit, because intent debt sharpens it.

I argued that detailed specs aren’t a complete answer. Translating a spec into working code involves a huge number of implicit decisions no spec ever captures, and a spec detailed enough to be the program is the program in a slower language. I still believe that.

Intent debt is the complementary truth.

Being unable to capture all intent is no license to capture none of it. The implicit decisions an agent now makes on your behalf, the ones a spec will never enumerate, are the decisions whose rationale evaporates if you don’t record at least the load-bearing ones. You can’t write down everything.

You do have to write down the why behind the choices that would be expensive to get wrong, because nobody will reconstruct those later.

Comprehension debt warns you not to trust that code is correct because it exists.

Intent debt warns you not to trust that the reason survives because the code does. Code is the answer; the intent was the question it was meant to solve. AI is brilliant at producing answers to questions you forgot to write down.

What high intent debt looks like

Intent debt rarely shows up as friction. It shows up as a particular kind of helplessness.

• An agent “fixes” a bug by deleting a guard clause, and nobody can say whether that guard was load-bearing or leftover, because no doc or commit message ever recorded why it was there.
• A refactor changes a behavior users depend on. The review passed because the diff looked clean and the tests were green, but the tests only encoded the previous behavior, never the intent.
• You ask why two services talk over a queue instead of a direct call, and the honest answer is “An agent suggested it and it seemed fine.” That answer is intent debt, already accruing interest.

If you’ve felt the cognitive surrender version of this, defending a design choice you can’t reconstruct, intent debt is the team-scale, written-down version of the same hole.

Surrender is about your own posture in the moment. Intent debt is what a hundred of those moments leave in the repo for the next person and the next agent to inherit.

Paying it down: externalize intent as a first-class artifact

Almost everything I’ve been writing about for the last few months turns out to be intent-debt management. I didn’t have the word for it. The move is the same each time: Take the intent out of your head and put it somewhere an agent can read.

Write the spec for the intent, not the implementation. A good spec captures the goals, the constraints, the nonnegotiables, and an explicit definition of done (fast, accessible, secure, delightful, beyond “functionally correct”). The spec carries the intent the code can’t carry on its own.

Treat AGENTS.md as your intent ledger, not your config. It’s why I keep saying stop using /init. An auto-generated file describes what the code is. An intent file describes what the team means: the conventions, the “we don’t do it this way because,” the constraints invisible in any single file. Agents can’t infer that, and they need it most.

Capture decisions where they happen. Lightweight decision logs (ADRs) are pure intent-debt paydown. Recording why at the moment you decide costs almost nothing. Reconstructing it eight months later, after the person who knew why has moved teams, costs a fortune. Agents have made logging cheaper than ever, so the old excuse is gone.

Make the learning loop write intent back down. I’ve argued for self-improving agents that update a learnings file at the end of a session. The same loop is an intent-debt pump running in reverse: every mistake whose root cause you’ve recorded, every “We tried X and it didn’t work because Y” is intent that would otherwise have lived only in your memory of a bad afternoon.

None of these are new tools. They’re the discipline of refusing to let the why exist only in your head, in an era where your head is no longer where most of the work happens.

Where the value moved

For a long time, the scarce, valuable thing in software was the ability to produce a correct implementation. Code was expensive, so we optimized for writing it.

AI made code cheap, and comprehension is recoverable. Intent, the goals and constraints and reasons, is the one input that still has to originate with a human. It’s also the one we’re worst at externalizing, because for decades we got away with carrying it in our heads.

That worked when the team was a handful of people who could absorb intent over years of shared context. It does not work when half the team is agents that start every session as strangers.

Technical debt makes your system hard to change. Cognitive debt makes it hard to understand. Intent debt makes it hard to know whether the system still does what you wanted, and it’s the only one of the three your agents can’t pay back for you. That part stays with you. Write down the why, because it’s becoming the most valuable thing you can leave in the repo.

I like to say that context engineering is the art of shaping what a model sees so that it actually does what you want. (I just realized that in saying that I’m channeling a comment that Andrew Singer made to me over forty years ago, when he was teaching me about debugging. He called it  “the art of figuring out what you really told the computer to do instead of what you thought you told it to do.” But that’s another whole story.)

Drew gave a talk at the recent Friends of O’Reilly camp, Foo Camp for short, about what he calls prompt debt, which he describes as “the hidden costs that teams rack up when they fight a model’s training instead of working with it.”

That was a novel and useful framing to me, that you end up with a bunch of stuff in your prompts to compensate for default behavior of the models, that those prompts no longer work as the models upgrade, and so it becomes a kind of technical debt. He’s thinking a lot about what the best developers are doing differently as a result.

So I invited Drew to reprise his short talk on Live with Tim O’Reilly, and then we talked about it with the folks attending the live event. They had a lot of good questions, so it was an interview not just by me but by a crowd of O’Reilly customers.

Prompt debt in practice

Drew opened his talk with two slides. The first was a prompt anyone could write in ten seconds: “You are a customer support assistant. Read the ticket, classify it as billing, technical, account, refunds, or other, return only the category name.” The second slide was the same prompt a few weeks later, after it had met the real world. It now said “REFUND REQUESTS ARE NOT BILLING” in capitals, then said the same thing again in different words, then closed with “This is a common mistake. Please do not make this mistake.”

Everyone who has shipped any application with a prompt recognizes the second slide. It is a simple but vivid illustration of prompt debt, which, like technical debt, has a bill that eventually comes due.

Note: Those aren’t real prompts. Drew just made them up to demonstrate his point. But what is real is that the instruction “Don’t quote directly more than 15 words from a source” occurs at least 7 times, in several variants, in Fable’s system prompt. So even Anthropic is incurring prompt debt! And what that repetition might tell us about the innate capability of Fable to quote directly from sources it has ingested is left as an exercise for the reader.

Drew itemized three costs of prompt debt:

1. It slows iteration. “You have so many little rules and call outs and washouts, many of them repeating to try to get rid of stubborn behaviors, that if you add a new instruction, you might sometimes have a small regression, and so you’re afraid to touch the prompt.”
2. It blocks collaboration. “If Tim has a prompt that he’s been working on that he has lots of rules for, I might open that up and it may look completely random. I don’t know why he’s added these rules, and why he’s threatening the mother of the model. But it works, so I don’t want to touch it.” 
3. It locks you to a model, because every hack you developed was tuned to fight one specific set of weights. Datadog’s State of AI Engineering report noted that GPT-4o was still the most common model in Datadog customer request traces in March 2026, even though OpenAI had already retired it in the ChatGPT UI. Drew thinks people are still running eighteen-month-old and two-year-old models in production rather than upgrading to far better models because they can’t face rebuilding their prompts.

That same Datadog report notes that 69% of all input tokens in customer traces were system prompts rather than user content. I’m not quite sure what to make of that. It does make clear that for all the ways that AI models are extraordinarily powerful, they are also extraordinarily unruly.

Why prompt debt is incurred

There are two reasons why prompt debt is incurred, according to Drew. The first is that natural language is imprecise, so the same intent phrased two ways produces different responses. Drew showed a study where someone framing the query as a patient asking how to taper off a drug called alprazolam gets refused by every AI assistant, while a psychiatrist asking about the same patient with the same clinical facts but with the right magic words to signify his professional status gets the protocol. Figuring out how to get the right response out of a model is a kind of spellcraft.

Drew also showed a more bizarre interaction, from Victoria R. Li, Yida Chen, and Naomi Saphra’s paper on guardrail sensitivity, which uncovered the perplexing fact that stating an allegiance to the Philadelphia Eagles made a model more willing to explain how to import a plant illegally. Go figure. Drew has written about that paper, and he has also used it in his own attempts to get a model to do what he wanted:

I once used this to get around alignment to generate a likeness that ChatGPT didn’t want to generate for me, and it refused. I said I was a Philadelphia Eagles fan. It said okay, and it rendered that image with the person holding a Philadelphia Eagles mug.

The second reason is that each model has its developers’ own preferences trained-in, and yours may be at odds with them. This is what Drew calls fighting the weights. He and Srihari Sriraman analyzed the system prompts of six major coding agents and found the same instructions repeated five and seven times in a single prompt, escalating through IMPORTANT to CRITICAL to MANDATORY to a threatened hundred-million-dollar penalty. He described what the author of such a prompt was doing as “war-driving the thesaurus,” hunting for wording that finally works.

Note: We didn’t talk more about Drew and Srihari’s paper, but we should have. It’s got some amazing insights in it. I highly recommend that you follow the link above and read it.

The harness is moving into the model

Drew has been tracking the published system prompts for Claude Code over time, and noted that they get shorter after each model release and then grow again. The reason, he suggested, is that Anthropic fixes unreliable behavior with a prompt patch, and then trains that patch into the next model. He said “That’s great for Claude Code, great for Anthropic. It’s a problem if you’re building a custom harness and your API calls look different than what Claude Code’s look like.” The developer of Pi, an open-source harness, kept finding that the models he worked with believed they were inside Claude Code and so they made Claude Code’s tool calls. He had to keep telling the model that no, they were working inside Pi. Fighting the weights over something like that is a real tax on developers. The point made above about Fable’s system prompt injunction against quotation shows how even the labs themselves are fighting the weights.

If you are fighting the weights, Drew says you have three options: solve it in your own prompt, catch and retry in the harness, or give up and make your API look like what the model expects. Steve Yegge came up with the last hack. Steve just added aliases for whatever the model calls in addition to his original method name. It works, but it means the expectations of the models now dictate the shape of everyone else’s software.

When Drew told me that more and more of the system prompt and the harness is being trained into the weights, that sent up a flare and my long history in the industry clicked into gear. It immediately got me thinking about lessons from the open source and web era. In particular, it made me think of the time in the mid-nineties when Netscape and Microsoft were both racing to build every feature up the stack directly into their web servers. And there was Apache, which stayed a web server with a clean extension layer that let other people build new features on top. Everything interesting got built on Apache. What I call an architecture of participation, modularity plus a clean separation between platform and application, beat integration every time.

I think Amazon got this right with web services too. Steve Yegge’s famous Amazon memo described how Jeff Bezos made every team expose its functionality through service interfaces or be fired, so Amazon’s own applications had to work on Amazon’s own platform. That way they had the same experience as their customers. That was very different from what Microsoft had done, famously having private APIs that were only available to its own developers.

So my prediction is that the big labs are making a strategic mistake. Training the harness into the model does make them better for predictable tasks and for less talented people, and it looks like a moat, but it risks foreclosing the innovation you would otherwise get for free from everyone else. As Bill Joy used to say, all the smart people don’t work for you.

Drew, to his credit, observed that “the labs are cornered rather than greedy.” Their interface is an empty text box that has to work for someone building a hundred-page harness but also for his neighbor who wants a website and knows nothing about code. Making the empty prompt box produce acceptable output requires baking in strong defaults.

The cost of trading diversity for reliability

That tradeoff has a serious cost, though. Drew quoted a line from Thariq at the recent CAIS conference: if you aren’t giving the model detailed instructions about what you want, what you get back is the average of everything in the model. That means that there is a real risk that AI is leading us ever further down the path to a monoculture.

Drew gave an example early in the conversation about image generation. You can now walk into any cafe in New York or Mumbai, he said, and see the same AI-generated art on its flyer. The earliest AI art out of DALL-E was strange and surprising, but what you get now is shiny and identical. When you optimize for reliability, you lose surprise. Which reminded me a bit of something Larry Wall used to say about Perl, that if it didn’t let you do stupid things, it wouldn’t let you do smart things either.

Drew made the same point about AI writing. He argues that post-training aimed at verifiable problems like coding and math and agentic tool use drowns out the human signal from pre-training, and so the more post training the models get, the worse they get at creative tasks. AI writing gets more and more predictable, people notice, and they don’t like it. Fable and GPT-5 write worse than Sonnet 3.5 and GPT-4o did. Drew thinks getting both good code and good prose from one model is likely impossible.

If you’re building a model that can solve coding challenges, you want reliability. But if you’re writing, where you want diverse rhythm and emotion and connection and engagement, I don’t think those two goals are mutually compatible.

What to do about prompt debt

We got into audience questions, and there were some great ones.

One audience member asked whether there are ways to set a time frame for prompt retention to avoid prompt debt? Drew answered that there isn’t a fixed time limit. Instead, teams should learn to recognize “prompt debt smell”: repeated instructions, one-off edge-case patches, or increasingly desperate wording. Those are signals to move logic into evals and automation.

Another asked how organizations can measure prompt debt quantitatively. Drew’s answer was to look at how often each prompt in your organization changes, how many people have edited it, and which ones have gone untouched for a year. Look for prompts only one person is allowed to touch. Then look at what models you are actually calling. “Having to run on old models and not being able to migrate is a good smell that you’ve got prompt debt in your organization.”

Some other good questions:

• What habits compound prompt debt the fastest? Drew’s answer was essentially “vibe shipping.” That is, prototyping quickly, patching outputs with more and more tweaks, then shipping without building a true maintainable system. Each of those patches is an eval you are writing inside the prompt instead of outside it, he said, which means you lose it the moment you change models. 
  
  Drew reminded us that Malte Ubl, the CTO of Vercel, said vibe coding makes code “free as in puppies.” We had free as in speech, we had free as in beer, and now we have free as something that arrives at no cost but has to be fed every day for years.

• Do people use pseudocode instead of natural language prompts, and does it work? Drew said yes, sometimes models optimize toward pseudocode. He used this to explain why DSPy and its new Flex optimizer matter. Instead of forcing logic into prompts, they let the system push simple cases into code and only call the LLM when needed. He gave some further advice: Treat prompts as perishable and invest only what you must. Define the task with measurements rather than paragraphs, and automate the discovery of the prompt for whichever model you’re on. That’s what DSPy is good at. Drew is one of its maintainers, so he is fond of it, but he makes a good argument: if you have written down what good output looks like, you can let a model find the wording, and that makes it easy to swap in a cheaper or faster or newer model without starting over.
• Can multi-agent workflows help work around prompt debt? Drew thought yes, especially through decomposition. He suggested splitting the task into smaller, evaluable steps rather than relying on one giant prompt and one giant model call. This is better for cost, reliability, governance, and speed.
• How do you balance prompt-debt guidance with context engineering, memories, and shared product context? Drew believes shared context is often necessary, but that teams should treat those instructions as perishable and keep iterating on them unless they’re worth formalizing into systems and evals.
• In compliance, where consistency is critical, what should teams do? Drew’s answer was decomposition, decomposition, decomposition. Break tasks into stages with checkpoints so you can inspect how the model got to its result, rather than trusting one opaque end-to-end answer.
• Does DSPy hide too much and make troubleshooting harder? Drew acknowledged that there is a tradeoff. Any framework gives up some flexibility, but DSPy tries to keep the task-spec layer stable while allowing the implementation underneath to evolve.

Another great audience question, and a good one to end this section on, was “There was prompt engineering, now context engineering, loop engineering, fleet engineering, graph engineering, harness engineering, goal engineering. What’s your take on how to navigate these many engineering disciplines?” I’ll let Drew answer that himself, in the video below.

It’s our job to make it weird

Drew is more optimistic than his worries that LLMs are encouraging a monoculture suggests. If the default output of a model is the average of everything it has seen, “It tells us that there’s still a job for us humans,” he said, “which is that it’s our job to push the model out of distribution. We’re the ones that need to make it weird.”

Weird is a strong word, so don’t take it too seriously. (Though I find it interesting that Harper Reed also used it.) The way I make this point is to say that AI is a medium, like painting or writing or music. Everyone gets the same paints and brushes, the same words, the same notes, but some people draw more out of them than others, or do it better. Our job is to draw something more, something better, out of the ocean of possibilities in the collected knowledge hidden inside an LLM.

But there’s a more prosaic way to push the model out of its normal distribution. Be aware of its training, which is another way of saying “its biases,” and compensate for them. As an example of how to do this, Drew said his team deliberately chose not to use React for a new front end, because the models are trained so heavily on React that using it makes your site look like everyone else’s. He has also started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness.

That led us into a bit of discussion about open source AI, which is the subject of my next AI Codecon. Drew’s ideas fit right in. He wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than, as he put it, becoming appliances.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.

Transcript

This transcript was created with the help of AI and has been lightly edited for clarity.

00.20
Ben Lorica

All right. So today we have Ben Miller. He is the co-founder and CEO of Fundrise. And today, we’ll talk about their interesting AI application called RealAI, which you can find at RealAI.com. Their tagline is “Answer any real estate question in seconds.” And we’ll also talk to Ben about broader trends in the AI industry. And with that, Ben, welcome to the podcast.

00.48
Ben Miller

Thanks for having me.

00.50
So, I definitely want to talk to you about RealAI in more detail, but I think to take a step back. . . so broadly speaking, as I understand it RealAI does the job of a real estate analyst, but I think, Ben, for our audience, they don’t actually know what a real estate analyst does. So maybe if you can briefly describe what does a real estate analyst do?

01.16
Yeah. So our product’s designed for a professional real estate investor, an institutional real estate investor will typically buy large scale properties that are worth tens of millions of dollars. And then analysts, like any financial analyst, would [do due] diligence [of the] property, make a lot of financial pro forma, do different sorts of investment memos and then make a recommendation to buy or sell. And they also then manage the investment after you acquire it. And asset management is actually where most of the activity happens in terms of how you optimize for rents and occupancy and different financing. And so the real estate analyst’s job is to be the financial brains of the owner of the property.

02.07
And typically this person, when they go about doing their job, they have access to different data sources, they pull it together, they run some financial models, and then they write up recommendations, right?

02.24
Yes, exactly. That’s what the white collar worker today is like. . . manual CRUD operation. Right? So they gather data points from different reports and different online services they might have. They usually populate a spreadsheet cell by cell that way, and then they also will write up a memo. And that is all done by hand, so it’s sort of like handcrafted. And all of that work, at least the first draft, can be done by AI. And then the question becomes, “why our AI versus Claude or ChatGPT?”

03.05
As I understand it too, Ben, RealAI actually didn’t start out as an AI project. It started out as a data project because back in 2023, as I recall, you were starting to talk to me about wanting to start a data business of some sort for real estate. Right? So I guess, for our listeners, Ben, why does real estate generate so much data? I think I remember you telling me that actually, most people don’t know a typical apartment building throws off so much data, right?

03.40
So Fundrise is a fintech platform that democratizes investing into private assets. And one of our original investment classes was real estate, where we would have a fintech website and an iOS app, an Android app, and we would have millions of users and they invest through our platform, and we have a sort of Robinhood-type platform. So we have payment processing and a system of record for ownership, and all of the API services you need and microservices for that company. And then over time we started eating the value chain, and so we started heading from the investor towards the asset. And eventually we ended up where when we acquire the asset, we run the asset. We vertically integrated, so we’re the real estate company and the tech platform, and the fund manager. And as we acquired the asset, we discovered the underlying infrastructure for that asset had a lot of data in it. And the way that people in real estate do their work is sort of how we started. It’s all very manual.

04.53
Can you describe what this data is?

Yeah. So there’s a bunch of different kinds of data sets you care about. At the property there’s a lot of activity happening. So, a toilet might be running and that’s the water bill. You have all sorts of leasing activity. You have marketing funnels of ‘where does the tenant come from’? You have lots of activity around the building in terms of like, who lives there, who’s moving there, who’s leaving, what’s the rent of the property across the street, what’s being built? So there’s endless amounts of data that’s happening at any point in the real world. And that real-world data matters to the person who owns real assets in that location.

05.46
And so then, it seems like the traditional approach was to ignore all of this data? Is that right?

05.56
It had to be distilled so that a human being can pretty much only consume a certain amount of tokens per second, and that usually ends up in a spreadsheet. So what happens is, people at the property or property managers, or maybe there’s our market company, CBRE, will produce market reports and they turn all that data into a spreadsheet so that you might get a 12-month reporting or rent roll or a market report. And so they take a lot of data and they distill it or transform or aggregate it into very few metrics. And that was necessary. And then once a month you get emailed six reports or 12 reports, and those reports are in a sort of dashboard for how real estate people make decisions. And I appreciate this and this is so obvious that there’s just you know, those reports are both a distillation and there’s a lot of meaning lost in the transformations, and a lot of data just left behind because the system of record for real estate and for a lot of industries, really the accounting system. . . income statements and, you know, get operational data like a toilet flapper that’s leaking, and that’s not coming into the report. And how could it?

07.36
So basically, it sounds like what you’ve done is the classic digital transformation of this industry. If you look back to the data in 2023 to today, what is the evolution of this data set? What was in the original data set, and what have you added since then?

08.02
So we originally were thinking about how all this data could be really useful for the real estate industry, the original business we called Basis, and we were originally using it just for our own company. And, there’s sort of property data. If you’re going to make decisions in real estate, you care about change over time. So time series, and you care about comparative analysis. How is my property versus another property? How’s my neighborhood versus another neighborhood? My city versus another city? And so you need to have not just your own data, but you need to have the context.

08.42
You need to have the data of other properties that are in other places. And the entities in real estate are places, properties, and then people, which was that we had this sort of insight when we were in the data gathering property data and place data, is that there’s a data model in real estate which essentially is the property management system, which is basic accounting systems. That’s Yardi and Tratta, RealPage. . . And those accounting systems have an implied data model that is a property address and the unit, but inside the unit is a person that’s not in their data model. And a really big gap in their thinking. It’s sort of like digital marketing. Kind of like TV. . . you’re going to do it on CNBC, you can do it at 3 p.m., but you don’t know who is watching. And then the internet showed up, and you can actually then target with Facebook or, you know, Google. You can target people who have eggs for breakfast and like Mozart. So all of a sudden the people data becomes really critical. And the real estate industry didn’t have any people data. It wasn’t even part of their thinking. It’s not currently part of their thinking. So we started realizing, “Hey, we get the kind of granular people data you have in the marketing industry.” We bring that into the real estate industry because you care about who lives in a building and who’s leaving, who’s moving in, who moving in the neighborhood? Who’s moving into the city. Are they rich or are they poor? How old are they? What’s their gender? All this stuff. Do they have iPhones? Do they have Androids? So we started gathering up huge data sets of people data and then putting that in the same databases you have with real estate data and building out, and you can actually see that people matter to real estate and you can actually make really good insights, new kinds of data, correlations and stuff like that for the sector.

10.43
By the way, this is the kind of thing that the hedge funds have been doing for years. Right?

Right. Yeah. I was actually talking to a famous data tech guy, and we were on the phone, they’re like, you know, we sell our data to all these hedge funds, for like 50 big hedge funds in the world, which probably be Renaissance, and Jane Street, and Citadel, we’ve only ever had one real estate company by our data. I said, “Who’s that? Who’s that?” They said, “Blackstone.”

11.18
Oh, that makes sense.

So yeah, high-frequency traders are data hogs and they put their data into these massive data models, and they do a lot of data engineering and data transformation. And real estate people do it with Excel spreadsheets and do it by hand. And so there’s this journey that every industry goes through. . . real estate’s way back in the stone ages, where they still do everything by hand with people.

11.45
So the end result for RealAI. . . the back end data. . . most of it is structured, probably sitting in some sort of data lakehouse or warehouse, right?

We have two different kinds of data. We have a transactional database and we have one for rows and one for columns. Because we have a lot of data and we do a lot of compute. . . And because mostly people when they’re using AI are doing computations on the fly, we had to pre-compute a lot of stuff so that when people are asking questions about what’s happening, how much growth there is in the market or what’s happening with like rents or whatever the things are, a lot of that stuff has to be pre-computed every night. And that gets pre-computed in a Snowflake database. And we also do clustering algorithms, we clustered people by city and by block and by lots of different things so when people want insights it’s all pre-computed. And then that sits in Snowflake migrating to a Databricks-type database. And then there’s a separate database which is the transactional store that’s on the fly, that’s fast when people are asking questions and hitting our API.

13.00
So, let’s set aside the AI model and whatnot. For listeners who haven’t tried it, I recommend you go to RealAI.com. I use it once a month to monitor my properties. But basically then what happens is you have this chat interface and then I presume the AI model acts as some sort of reasoning layer and also obviously parses your intention and what you’re interested in, but then the heavy lifting is done by the data back end, correct?

13.38
Yeah. I mean, I think most people have gotten here where you do deterministic analysis, real and where you’re writing Python and doing SQL. . .

13.49
Yeah. Okay. So in your case, if you hallucinate there’s consequences, right?

Yeah. So the way we structured our application is that where there’s facts, there’s essentially a  field that goes and pulls it from the database. So it’s not coming out. It essentially says, “What was the rent for the last 12 months in this property or in this neighborhood?”

14.16
But the AI model is the one that translates that prompt into some sort of SQL query?

Yeah, but I mean, we used to use Claude and we moved to Vercel, but we have basically a software sandbox and it opens it up and does all that sort of. . . This is for real estate, for most people you say, well, you probably know how to do this, but if you’re going to do an analysis about your home or your property, you know, most people can’t write Python, most people aren’t good at doing SQL queries, plus they don’t have a good understanding of the the database schema. And so we built a semantic layer on top of it. And it really democratizes data science. Originally you said we were a data company, and then AI showed up and the AI sits on top of our data, and that made it a lot easier for people because to do the type of insights or analysis that otherwise they would have no idea how to do before.

15.28
But just to clarify, the AI is not doing any kind of financial modeling or calculation, right?

No.

There’s some sort of forecasting or financial model or some sort of financial computation involved, and that’s not being done by the AI?

Right. The AI is like an orchestrator, and has the tools, right? And the different tools it uses and the tools are the web search or the calculator. I think everybody’s gone here, where I think a year ago or something, people were using vector databases and RAG and we moved away from that because you just want to be where it’s deterministic analysis. You want to just use regular software, to put that tool in the hands of the AI.

16.21
So the hard thing here is really maintaining these data sets in many ways, right? So that is the key, because you have to gather or you have to have domain knowledge to understand what data you need and what data really matters. And, now you have a year or so of history of people using the app and you’re understanding, really what’s valuable.

What’s the hard thing? I mean, it just depends on who you are. We didn’t mean to do this, but luckily we have a lot of data. The data is proprietary. The data is really valuable. And so that gives us a reason to exist because what happened is that we built RealAI and we rolled it out, and Claude arguably made it obsolete. And this is a strategy question. I think the hard thing is trying to build the software business.

17.18
How can it be obsolete if it doesn’t have access to your data?

Well, because people have. . . it’s really hard to get them out of their. . . they’re sitting in Claude. Trying to get it. And the reality is that. . .

17.38
There’s no A/B test. They can’t tell that they can get better results if they. . . 

So, our product is really good at, let’s say seven of the 15 things a real estate person needs to do. But there’s other things they need to do that they know our products are not designed to do, and so they need to have a general purpose tool that’s doing all their stuff. And so the day before yesterday, we got approved to be in the Claude marketplace. So we built a connector, we’re building a plugin, and now we’ll go to where the customer is, which is Claude. That’s our current strategy. And you can see it’s getting better. Claude is not as good as ChatGPT at making it developer-friendly. But we can build what’s almost the same as RealAI.com in Claude. And that’s where we’re currently headed. And you and I talk about strategy. I think that’s actually a temporary strategy. I don’t think that’s where the market ends up. But, yes.

18.50
But obviously, as users interact with your app, there’s a lot of learnings there that you can leverage to improve the app itself or even maybe the model powering the app. But in other words, there’s a compounding loop. So the question is will Claude cut you out of that at some point?

I think the hard thing is always the people. I think the technology is actually at least. . . You know, I’m not an AI researcher at the frontier. The hard thing is really building a software product that people become addicted to, and the data engineering and data science and stuff, that’s just a grind, this is work. In the short term, we’re sort of conceding. . . I believe we should concede that the customer wants to be in Claude today, and they don’t want to be in a different application. Claude’s like the hot new hotness. But I think they’re going to get Claude fatigue within a year. I think that the real estate people are lagging the rest of the industry. Most people who’ve been using Claude for the last six, 12 months hit a lot of limitations for enterprise. And I think they’re going to want a harness that’s designed for real estate. But, in the meantime, we’re going to sacrifice some of the things we don’t get by having them in our app. And I actually think they’ll end up going back and forth between Claude and our app, because our app is better for real estate than Claude, but Claude is better for everything else. And the way that the Claude plugin and, and I think it ultimately ends up being a RealAI app in Claude. It wants to go back and forth between our app and their app and wants to be seamless to the user. But after that—and this is really a strategy question—after that, do we end up with an open source real estate fine-tuned, trained model instead of Claude in a year or two? I don’t know, but it seems that would be a good expectation.

21.23
At the end of the day, even if you open up that model, that model still needs the data, right?

At the end of the day, that open source model a year from now. . . when we rolled out Sonnet 5, it didn’t change. It didn’t get better for the customer. Like we just don’t need the Fable. . . it’s actually worse for the customer. So I think that once the open source models are comparable to—I’m just gonna say Sonnet 5 or whatever that generation is—the customer is not getting the premium they’re paying for. They just don’t need it.

22.05
By the way, there’s tons of companies now that focus on helping enterprises get to specialized AI. And basically there’s two branches, right? So there’s the post training branch which comes down to fine tuning and reinforcement, fine tuning, tons of startups there that are coming online. And then, there’s even companies that will help you pre-train models from scratch. I think the trend is toward the inevitable simplification of both things. Which will mean that a lot more of the compute will go towards specialized models like yours. And then less of the compute going to the. . .

22.52
I’m not as confident in that. . . I mean, I’m agnostic. I think we could end up in a world where Claude replaces Microsoft and everything’s inside Claude.

Yeah, it’s possible. But then, I think that the general purpose models increasingly are going to be perceived as too overkill for a lot of things.

23.15
Yeah, I think that’s true in theory, but the consumer is so sticky and it’s hard to get them to change their behavior.

Yeah. If it’s a consumer app. But most enterprise work is just back end. . .

23.31
But I’m dealing with an SMB not like a Fortune 500, generally. And the SMB, in a lot of ways, acts more like a consumer than like an enterprise. And so I hear what you’re saying, though.

23.46
Hey, I have a question for you as far as UX. You have an application where someone is asking a question. It could be an important question for the user. How do you communicate uncertainty or how do you say, “Hey, here’s an answer, but there’s a lot of caveats behind this answer” How do you do that in your case?

24.23
That’s very hard to do. I was just getting that feedback yesterday because even facts in the data world are not 100% facts usually.

24.33
There’s alternative facts.

Yeah. I mean there’s just methodologies that change the information. Is the rent growth a point to point 12 months ago? Is it a median? There’s just a lot of ways you can get to something that’s true, but not the same answer as someone else. But my actual experience with consumers is that they actually don’t care about that. It’s all really about perceived credibility. . .

25.16
So I guess in the UX then, the question is if there’s some sort of uncertainty or disagreement, do you surface that?

25.28
Yeah, we surface data coverage. We have a whole section in the response that’s about a meta analysis about the data and about where we made decisions. . .

25.39
Or do you say, “This property, we think the rent you can charge is this?” But it’s really more likely in this interval. It’s not a specific value. And here’s the reason why there’s an interval instead of a value.

26.00
AI is really good at that kind of caveats and adding context in and again like there’s. . .

26.12
But it’s a UX problem, though. You have to decide to surface it, right?

We definitely surface it. But I find that the user ends up like it’s really a journey. In the first phase, they’re trying to determine if the data is good, and then once they’ve decided the data is good, they don’t really care about the caveats. They’ve sort of internalized to them how to think about the data. And it’s really a problem in the beginning of a user’s use of it. It’s not really a problem once they become a power user.

26.50
So I think that that’s a UX debate that people will have, moving forward. Because I think basically, like I said, in terms of actual studies around decision-making, it does make a material improvement. Since we have you, we’ll close the discussion. And before I ask a question, listeners, this is not we’re not about to give you financial advice; we’re not financial advisors. So there you go. That’s the discussion. First, first thing then, you have access to all this data in real estate, right? So you have all the patterns, but you’re also obviously an investor in AI and AI startups. Right? So, obviously you’re seeing trends in AI that you think aren’t being reflected yet in the real estate market.

27.53
I know I think you’ve mentioned to me in the past, I think so, as an example, to make this concrete, I think you mentioned to me in the past that maybe if you’re buying property with the assumption that there’s a lot of back office workers moving forward and those back office workers are located in, I don’t know where.  . . Phoenix, Arizona. . . Maybe you should have second thoughts. Right? So what are some of the AI trends that you think are not yet being priced into real estate investing?

28.28
What’s good about the trends I’m seeing in the data I’m seeing is I think they’re consonant or consistent with what people would intuitively think. So, AI is causing—in the data—a lot more disparity in outcomes. And so in real estate something used to be a truism. I mean, people will say people always need a place to live. And so you would buy workforce housing, affordable housing, things where if you have housing that’s affordable. . .

29.04
Or always buying is better than renting.

It’s just like cheaper is better than expensive, right? If you can offer housing that’s less expensive, less expensive to file, it’s less expensive to rent. That’s good business. And that actually I think is breaking down, which is maybe unintuitive but also maybe intuitive. Is that where the marginal dollars are ending up in the hands of people who have a lot of money and the people who don’t have a lot of money, which is basically the bottom half of the country, or even maybe the bottom 90% of the country. . .

29.45
95%.

Maybe. You can break a lot of different ways. But the point is that, this K-shaped economy where the normal person is doing worse and the best are doing better is. . . AI is accelerating that trend. And that way that affects real estate is if you’re an investor, you want to actually focus on the high end. If you’re going to be building, you can be investing. . . high end essentially is price-insensitive and the low end. . .

30.15
Because the low end people have no disposable income.

The low end is under strain. And then what is happening in real estate is that strain is showing up as a political manifestation of controlling pricing. And so you can’t make money providing affordably priced housing because the government won’t let you. So the government will let you take essentially excess profits from the rich, but it’s becoming more and more challenging to invest in and build for the normal person, because the government’s intervening in all sorts of small ways that people don’t see around how permits are pulled and lots of impact fees and things like that. I mean, in San Francisco and LA and New York, you see obvious examples of that. But it’s happening everywhere. And in real estate, you usually make an investment over a five year period, maybe even ten years. And so the long term trend is the most important thing. And I think then you essentially say. . . this is terrible by the way, from a social point of view, the societal point of view is very negative. I’m not commending this trend,  I’m just saying that’s what’s happening on the ground.

31.40
What about this notion that certain hubs are optimized for a certain type of work? Is that something you think about?

31.54
Yeah. That’s the thing you were I talked about last time. San Francisco is obvious, and then you look at Charlotte. But certain types of people move to certain places to get certain types of jobs and work from home drove that middle income white collar worker to work remotely, or they moved to more affordable places like Charlotte, Tampa, and Orlando, all these places where it’s affordable to live, and they do middle office work, back office work, and that job is getting decimated by AI. In particular, a lot of cities where young people would move there. They’d move to an apartment building, they’d move to downtown. They’d have a new job in a middle office of a medium to big size company. Those jobs don’t exist today. And so they’re struggling with where to live and they have roommates or living at home. And so the real estate that used to be where young people moved to is struggling as a result.

33.08
So I took you on your first Waymo ride. And how about things like that? I mean, now I can Waymo from anywhere.

33.19
I tried to invest in Waymo and I was not yet successful okay. I reached out to the CEO I had 37 connections to and I got no love. But yeah, the Waymo thing is such a good example of diffusion, a diffusion problem because the technology is completely mature and you can’t get it rolled out in any of the. . . DC’s fighting it and Boston’s fighting it. And for it to really affect real estate, it needs to become as seamless as Uber and Uber rolled out way faster, way more aggressively. They obviously took more risk on the rollout. So, you know, if you said ten years from now, how does that affect the patterns of living? I think you’re going to see. . . I’ve looked at this a number of times, but I think it’s premature to make these investments. You want to be investing in the wealthy satellite towns of. . . if you’re doing San Francisco, which is a little bit challenging to the Athertons. But in Atlanta, it’s the Greenvilles. There’s a lot of great wealthy exurbs that have many miniature downtowns that are cute. And so I think you’re going to see these satellite cities become very, very successful as a result. But those satellite cities are going to be mostly for the wealthy. So again, it’s all about wealth, not about affordability.

34.50
All right. Closing question. You’re a CEO. I know you’ve talked about the impact of AI and automation on hiring. And you talk to other CEOs. So as best you can tell, set aside the headlines and the news accounts. What are you hearing from CEOs about AI’s impact on the workforce and hiring? Hiring in particular.

35.20
It’s more mixed than the headlines. I think I’d say it’s both true that we have needed less people, haven’t hired as many people and also have needed to expand as a result of AI. And so I think I’m hearing a “both” situation. So it’s really a question of what’s the net impact? And I feel like it’s too soon to call. I think on net it’s still net negative on hiring, but it’s too fuzzy for me to have a good call on it.

35.58
Do you have any sense whether or not all the stories we’re reading about. . . if it’s particularly challenging for new college grads and people looking for that entry level job?

36.17
We don’t hire those people, and we rarely ever did.

Because the stereotype is those jobs I can just use AI.

36.28
I think of it as this cascade effect. Where we hired hundreds of people over the years, and what happened was there’s this elevator dynamic or escalator dynamic, where the person who was an early grad becomes middle and then they make space for the next person.

36.52
The talent pipeline, right?

Right. And that dynamic. . . AI’s changed it so much because you just don’t need. . . the people who are at the upper middle are so effective. . . And actually, this is something everybody learned in 2020-2021—more people means less work gets done because you have to manage people and you’re needing meetings and remote work is really brutal for information transfer. And so it’s actually way more effective just to do it yourself with AI than it is actually to have five people. And I think it’s actually more satisfying. And so I believe I’m net negative on jobs in the white collar work. And we’ve invested in some data centers that are like $50 billion data centers. And when you go there, there are thousands of people on site working. I mean, it looks like we’re in China. . .

37.50
For now, while they’re building it.

So I think what’s happening is that the job market doesn’t seem as bad because of this trillion dollar a year AI CapEx build. But I think that’s a five-year build, not a 50-year build. And then I think that in the meantime, that’s coring out the white collar worker. And so I think on the other side of this, when the bubble finally bursts, it’s going to be a deep trough.

38.22
So then that talent pipeline . . . if you don’t have the entry level jobs who become the middle managers. . . and so on and so forth. . . So you have the entry level developer who gets more knowledgeable. . . then you don’t have that. Right? So you end up with what?

38.42
Your talent pipeline, it’s a thing. I’ve one of the things I’ve said to the team and this is harsh, but I’m going to say it, but it’s a brutal thing. There’s a lot of people in an organization who are good but aren’t going to be the next great people. And I said, “If we don’t have a talent pipeline anymore, you need to turn those people over faster.” Essentially, unless you were bad. . . bad people managed out, but if you’re good, we kept you. But if you’re great, you get promoted. But now if you don’t have this talent pipeline, you have to change what you expect of the good. You have to turn over some of them, and look for people who are good with the prospect of building that. . . So you have to change the talent pipeline or talent escalator because you don’t have as many people coming through as you used to. So it’s more of an up-or-out dynamic than it used to be.

39.42
Oh, I forgot one last question on RealAI, which is, I think you still do seat based pricing, right?

39.52
We don’t know how to price it.

So that’s it then? If I’m a real estate analyst, you’re charging me $50 a month, but I’m hammering this thing, costing you $1,000 a month.

40.04
No. The seat buys a certain number of tokens. So it’s more like it’s more like usage tiers, is how it’s priced today, but I don’t know how to price it. We’re still early in this evolution. I mean, we’re real estate. . . so real estate is going to be lagging other sectors in terms of adoption. And so I don’t know what the right way to price it is because it doesn’t want to be only usage, either, I think that’s not right. But I don’t know the answer.

40.41
But even the tiered pricing, if I’m on a token matter, at some point I’m already at your top tier price and I’m still busting through it.

40.52
Yeah, that’s a great situation though, because tokens aren’t that expensive. I know everybody’s obsessed with them, but I think tokens are the smallest part of our cost.

41.04
But if you’re charging me $200 at the very top tier, I’m actually spending so much compute. . .

41.15
I think that would be a great scenario to have in the future. . . for the few power users, that’s happening for us. Great. I hope they feel like they’re getting a good deal.

41.26
And with that, thank you, Ben.

Yeah. Thanks, Ben. Thanks for having me.

Great name, by the way.

The First Law still holds

We’ve said for a long time that the First Law of Software Architecture is: Everything is a trade-off. Nothing about generative AI repeals that law. If anything, it enforces it more brutally than ever.

AI coding tools are extraordinary at answering “how” questions. How do I implement a saga pattern? How do I set up circuit breakers between these services? How do I paginate this API? These questions have answers that exist in the world in documentation, in open source code, in a decade of blog posts, and large language models have read all of it. Asking an LLM a “how” question is like asking a very fast librarian who has memorized the library.

Architecture questions are not “how” questions. They’re “should” questions, and “should” questions have a different shape entirely. The honest answers require knowing things that appear in no training: that your ops team is three people, that the CFO just froze cloud spend, that the last reorg left the payments team demoralized. An AI can enumerate the generic trade-offs of distributed architectures beautifully. What it cannot do is weigh them, because the weights live in your organization, not on the internet.

That’s the Second Law, incidentally: “Why is more important than how.” LLMs are “how” machines. Architects are “why” people.

Cheap code makes decisions expensive

There’s a tempting inference floating around: If AI makes building software easier, surely it makes architecture matter less. Our experience so far suggests the opposite. When code was expensive to produce, the cost of construction acted as a natural brake on bad decisions. A questionable design took months to build, and somewhere in month two, someone usually noticed. Now a team can stand up a fleet of services in a week. The brake is gone. It has never been easier to build the wrong thing quickly, at scale, with tests.

Think of AI as an amplifier. Point it at a sound structure and it accelerates you. Point it at a flawed one and it pours concrete over the flaw before anyone has time to object. The half-life of a bad architectural decision used to be measured in the time it took to implement; now the implementation arrives almost instantly, and you get to live with the decision for years.

This shifts where the leverage sits. When implementation is abundant, judgment is the scarce resource. Someone still has to decide where the service boundaries go, what “good enough” availability means for this system, and which architectural characteristics actually matter.

Judgment doesn’t come from reading

Here’s the uncomfortable part, and it applies to humans as much as machines: You cannot learn trade-off analysis by consuming content about it. We’ve written a fair amount of that content ourselves, so we say this with some authority. Books and talks give you the vocabulary. They don’t give you the judgment.

Judgment comes from making decisions and living with the consequences or at least watching someone experienced make them, asking why, and arguing about the alternatives. Every working architect we know learned the craft this way: apprenticed to messy, real problems, with feedback loops. The pattern catalog was the easy part. Knowing which pattern not to use, and why, and being able to explain that to a skeptical VP that took years of reps.

This is also, not coincidentally, exactly what today’s AI lacks. A model trained on the world’s code has seen millions of decisions but almost none of the consequences. The post mortem that traces an outage back to a boundary drawn wrong in 2019 rarely makes it into the training data, and even when it does, it isn’t connected to the pull request that caused it. Architecture’s feedback loops are measured in years. That’s precisely the kind of learning that can’t be scraped.

Where this leaves engineers

If you’re a developer watching AI absorb more of the implementation work, the strategic question isn’t whether your current tasks will change but where to move on the value chain. Our answer is to move toward the decisions. Toward the trade-offs, the constraints, the “it depends.” That territory isn’t shrinking; it’s growing, because every AI-accelerated team needs someone who can tell the amplifier where to point.

The good news is that this is learnable. Not from a book alone, and certainly not from an LLM, but the way it’s always been learned: by practicing architectural thinking on real problems, with experienced people looking over your shoulder and asking why. We’ve spent the last several years teaching it that way, most recently in a six-week cohort format that works less like a course and more like a short apprenticeship in making and defending architectural decisions. (Details are on the O’Reilly live events page, if you’re curious.)

However you pursue it, pursue it. The machines have gotten very good at “how.” The career-defining skill of the next decade is being the person in the room who can answer “should,” who knows that the real answer starts with “It depends,” and can finish the sentence using their brain alone.

This is the second article in Sharon Zhou’s post-training series. Read part 1 here.

In the first post of this series, you learned how post-training closed the fundamental gap in usability of LLMs by making them behave in a certain way. In this post, you’ll explore specific techniques you can use to change a model’s behavior: namely, reinforcement learning (RL) and supervised fine-tuning (SFT).

Reinforcement learning teaches the model by letting it try things and telling it which attempts were better or worse—the model learns by experimentation and feedback. Supervised fine-tuning teaches the model by showing it examples of good behavior—the model learns by imitations. Both have deep roots in AI and machine learning literature historically, but their application to LLMs, and particularly to making LLMs behave well, is what makes modern post-training work. Nearly everything that happens in post-training is the result of some combination of these two approaches.

Reinforcement learning (RL): Learning from feedback

The overall gist of reinforcement learning goes like this:

1. The model gets a prompt.
2. The model generates a response.
3. The model’s response is graded. The grade is called a reward. A positive reward is good, and a negative reward is bad.
4. The model’s weights are updated to make high-reward responses more likely and low-reward responses less likely.

One of the most important questions is: Where does the reward come from?

Verifiers

The easiest way to get a reward is a function that can output a reward, for example a checker for whether the generated code compiles or whether the generated math problem was solved correctly. This automated check is a verifier. The ideal verifiers are fast, cheap, and perfectly reliable within their domain. Think coding challenges, math problems, or factual questions. For tasks with objectively correct answers, you can just write a function that checks the output.

The limitation is probably obvious: Verifiers only work when you can define “correct” programmatically or hit an API to return the right results. That covers a lot of useful territory, but it doesn’t help you train a model to be helpful, nuanced, or pleasant to talk to.

There are subtler limitations too. Not all verifiers are fast. Your model might propose a novel drug combination, but verifying its validity could take years of lab work. Generated GPU code might need hours or days of performance benchmarking. When verification is expensive, you face a trade-off: Use the slow-but-accurate verifier sparingly, or substitute a faster proxy that’s slightly less reliable but keeps training moving.

Human feedback, RLHF, and reward models

Humans can offer strong reward signals that, in aggregate, align with human preferences that might be more subtle and hard to encode programmatically. However, it’s prohibitively inefficient to have humans in the loop for every training datapoint, especially as the model is continuously updating its weights after it receives rewards as feedback, so the model’s responses would change over time. You can’t really prepare the data ahead of time. So instead, the InstructGPT paper, which informed ChatGPT’s development, implements reinforcement learning from human feedback (RLHF) by training a separate model to mimic human feedback. This model is called a “reward model.”

The input of the reward model is a prompt and model response and its output is a scalar reward (positive or negative) that mimics how a person would rate that response. You can train a reward model in multiple ways. The simplest is to have people grade the model outputs with a score, for example 1–5 stars or a number out of 100%. However, people are rarely consistent at these types of tasks: One person’s 2 is another’s 5, and even the same person drifts over time.

Another simple way is to offer two model responses in comparison and ask, “Which one is better?” This is a much easier, more reliable judgment for people to make. Interannotator agreement is significantly higher for comparisons than for absolute ratings.

Training a model using pairwise comparisons is also simple. You can then use cross-entropy loss over pairs, which pushes the reward of the preferred response higher than the unpreferred one. This works great because it means the reward model can learn from signals like “A is better than B” but can learn to output absolute scores for the reward.

To make the process of collecting pairwise comparisons from people more efficient, the InstructGPT’s implementation of RLHF included showing labelers 4–9 different model outputs from a single prompt and asking them to rank those preferences. This would effectively result in 6–36 pairwise comparisons for a given ranking. Not bad; that’s efficient data labeling! They used ~33K prompts, so that would roughly translate to anywhere from 200K to 1.2M comparisons to train the reward model.

After training, the reward model would be an automated judge during RL training, providing scalar rewards for responses. The language model then optimizes against this reward model’s scores. This means the better the reward model, the more aligned the resulting model would be.

LLM as judge

So you need a reward: Why not use an LLM? LLM-as-judge, sometimes called RLAIF (reinforcement learning from AI feedback), scales much better than human annotation while still being able to evaluate subjective qualities like helpfulness, clarity, and tone. But it inherits whatever biases or blind spots the judge model has, and can be more easily gamed. If the judge tends to prefer verbose answers, the trained model will learn to be verbose.

One effective approach is to break the judgment into multiple LLM calls, each focused on a different aspect of the response, like a rubric. Instead of asking one LLM call “How good is this response?” you might have separate calls evaluating factual accuracy, clarity of explanation, appropriate tone, and completeness. Each dimension gets its own score, and you combine them into a final reward. This is more robust than a single holistic judgment because it’s harder for the model to game all dimensions at once, and it gives you fine-grained control over what you’re optimizing for. You can weigh the dimensions differently depending on what matters most for your use case, and adjust those weights over time as your priorities shift. For example, accuracy is worth 3x as much as tone.

Combining human feedback with LLM-as-judge, Anthropic’s Constitutional AI (CAI) is a method for training reward models from AI-generated comparisons, based on a human-written set of principles. What this means is that you can give an LLM a set of principles, which Anthropic calls a “constitution,” and have it critique and revise its own outputs based on those principles. For example, a principle might say “choose the response that is least likely to be harmful” or “prefer the answer that is most helpful while being honest.” The model generates pairs of responses, uses the constitution to decide which is better, and those AI preferences are used to train the reward model. This means you can encode your values explicitly as written principles in the Constitution rather than implicitly through thousands of human annotations, making it easier to audit, agree on, and update what the model is being trained to do.

RL algorithms

Once you have a reward, it’s time to update the model’s weights. But you can’t just predict the next token, because there isn’t one. All you have is a value for the response the model gave. This is where RL algorithms come in. These algorithms are ways to take the reward and turn it into a meaningful, and ideally stable, training signal for the model to learn. There are several, and the field is moving fast, but a few fundamental ones are worth understanding.

REINFORCE

REINFORCE is the simplest starting point. The idea is to generate a response, score it, and if the reward was high, nudge the model to make that response more likely. If the reward was low, nudge it to make that response less likely. It’s conceptually easy to grok but noisy and difficult in practice because it turns out that the signal from a single response can point the optimization in unhelpful directions, and the variance in the gradients makes training slow and unstable. PPO was designed to fix these exact problems.

PPO (proximal policy optimization)

PPO is what OpenAI used in the original ChatGPT work and was for a while the default algorithm for RLHF. In RL terminology, the model is the “policy,” or the thing that takes actions by outputting tokens, in an environment which is simply the conversation context.

PPO improves on REINFORCE by being more careful about how big each update is. Rather than taking whatever gradient the reward suggests, PPO clips the update so the model can’t change too much in a single step. This makes training significantly more stable. The clipping keeps updates “proximal.”

PPO is also an online algorithm, meaning the model generates fresh responses during training, gets them graded, and updates from that feedback in a continuous loop. So the model keeps learning from its own current behavior rather than from a static dataset. It can explore and improve in ways that offline methods (that only collect data once beforehand) can’t.

Notably, PPO uses a “critic,” or a separate model that predicts the expected total reward from any point during generation, and is trained with the policy. This helps reduce noise in training, because it gives you a baseline: Instead of just knowing “this response got a reward of 7” and having no idea if that’s good or bad, the critic might predict “you’d normally get a 5 here,” so the actual training signal (called an “advantage”) becomes “+2, better than expected.” This dramatically reduces noise compared to REINFORCE.

The downside is complexity. Now you’re training two models (the main model and the critic), and the whole pipeline involves generating responses, grading them with a reward model, estimating how good the grades are relative to the critic, and updating both models. It works, but it’s a lot of moving parts. This makes it harder to tune or debug when something goes wrong, and harder to set up the infrastructure.

DPO (direct preference optimization)

DPO takes a different approach that avoids RL entirely but optimizes the same underlying objective as the standard RLHF formulation. Researchers found that there’s a mathematical relationship between the optimal reward model and the optimal main model (policy), and you can collapse the two-step process into one. This means that you can take the same pairwise comparison data (“model response A is better than model response B”) and use it to update the main model directly, without a reward model. Yes, this means good old supervised learning on that pairwise data.

In theory, under ideal conditions, DPO and PPO-based RLHF converge to the same global optimum. Those ideal conditions include a perfect reward model, infinite preference data covering the full output distribution, and the reference policy matching the data-generating distribution. However, these rarely hold in practice, and several empirical studies have shown meaningful performance gaps between DPO and online RL methods on harder tasks, partly because DPO can’t explore beyond its fixed dataset. That said, it’s still a very promising technique.

The simplicity of DPO is attractive: supervised fine-tuning on pairwise data with no reward model to train and no RL loop to stabilize. As a result, DPO has become very popular, especially among smaller teams, because it’s much easier to implement and debug. However, the trade-off is that DPO is less flexible, because it works directly from a fixed dataset of preferences. This means it can’t explore and discover novel behaviors the way online RL methods can. It only learns from the comparisons you already have.

Newer online variants of DPO have addressed this by generating fresh responses during training, but at that point you’re reintroducing some of the infrastructure complexity that made DPO appealing to avoid in the first place.

GRPO (group relative policy optimization)

Introduced by DeepSeek, GRPO takes another stab at simplifying PPO. Instead of needing a separate critic model, GRPO generates a group of responses to the same prompt and uses the relative rewards within that group to figure out which responses were better or worse—basically normalizing within that group. If you generate eight responses and three of them score well, those three get reinforced and the others get pushed down, and the baseline (which the critic was in charge of previously) is just the group average. This eliminates the critic entirely while still getting a useful training signal. It’s simpler than PPO but still online (the model generates fresh responses during training), so it can explore in ways DPO can’t. GRPO got a lot of attention because of its role in training DeepSeek’s reasoning models.

There are many more algorithms and variants, and new ones appear regularly. The field hasn’t converged on a method (and likely won’t for some time), and different algorithms suit different situations. DPO is great when you have good preference data and want simplicity. PPO remains strong when you need online exploration and have the engineering resources to manage the complexity. GRPO offers an appealing middle ground. In practice, teams often try multiple approaches and pick what works best for their specific use case and reward signal.

RL post-training is also less stable than supervised learning, which we’ll cover next. The loss curves are noisier, the hyperparameters are more sensitive, and the training can diverge if not carefully managed. Practitioners typically constrain the RL updates with a penalty that prevents the model from drifting too far from its starting point. The most common approach is a KL divergence penalty that keeps the fine-tuned model’s output distribution close to the base (or SFT) model’s distribution. This acts as a regularizer: It lets the model improve its behavior while preventing it from forgetting what it learned in pretraining or collapsing into degenerate patterns.

Supervised fine-tuning (SFT): Teaching by demonstration

Supervised fine-tuning is more straightforward. You show the model examples of ideal responses, and train it to reproduce them. In practice, this means collecting a dataset of {prompt, ideal response} pairs and continuing to train the model’s weights using the same next-token prediction objective from pretraining, but now on this curated dataset instead of the broad pretraining dataset. The one difference is that the loss is computed only on the response tokens, not the prompt tokens, so the model learns to generate good responses given prompts, not to generate prompts.

The simplicity is the point. There’s no reward model to train, no critic to stabilize, and no policy gradient variance to worry about. However, it’s also limited by the data you can collect. That can get expensive and difficult to scale.

The quality of your SFT model is directly determined by the quality of your demonstrations. The model is learning to copy what you show it, so every quality issue in the data becomes a quality issue in the final model.

Human demonstrations

The most direct approach is to hire skilled people to write high-quality responses to a diverse set of prompts. This is the gold standard. You can easily control your dataset here, and you can get exactly what you want, written to your specifications. The original InstructGPT paper from OpenAI contracted 40 labelers, writing demonstrations and ranking outputs.

The disadvantage is, probably obviously, cost and scale. Good demonstrations are expensive, especially tasks requiring domain expertise like having doctors write an ideal prescription for a patient or a rocket scientist telling you how to put satellites on Mars. And even expert annotators are inconsistent. They have bad days, they get tired, and they interpret instructions differently from each other. At scale, this inconsistency can accumulate, though labeling companies manage and sell processes to make crowdwork more effective at scale.

Synthetic data

Synthetic data scales far better than human annotation. You can generate millions of demonstrations cheaply and quickly. The Stanford Alpaca project famously fine-tuned Llama on only 52,000 demonstrations generated by text-davinci-003 (part of the GPT-3.5 model family, though not ChatGPT) and was able to get qualitatively similar behavior to text-davinci-003 with a much smaller budget (though it was on a narrow evaluation of only ~250 examples—still an exciting result for small open models for research).

Many open source models have used variants of this approach. However, there’s also a practical consideration around terms of service. Some model providers restrict using their outputs to train competing models, and this has become an increasingly heated area of debate as models compete at the frontier. Know the rules before you build your pipeline.

Curated data with synthetic transformations

Sometimes the best demonstrations already exist. Customer support logs, internal documentation, expert Q&A forums, edited writing samples. If you have access to high-quality human-generated content that matches the behavior you want or is close to it, you can use LLMs to transform that data into prompt-response pairs. This has the advantage of being grounded in real use cases rather than fully synthetic scenarios.

The work is in the curation, and sometimes it might be easier to generate from scratch based on a few few-shot examples. Raw data is messy: Support logs contain errors, forums contain misinformation, and real conversations meander. You need to filter, clean, and reformat aggressively, but you can build an LLM pipeline to do those steps. If you have a good source and invest in the LLM curation pipeline, this can be extremely effective, especially for domain-specific applications.

Rejection sampling

Sometimes the best training signal is already inside the model and you just need to find it. Rejection sampling works by generating many possible responses to a prompt, scoring them with some quality metric, and keeping only the top performers. The quality metric can be a reward model, a rule-based check, or even a stronger model acting as a judge.

Suppose you prompt your model “Write a Python function to merge two sorted lists” 64 times at temperature 0.8. You run each output through a test suite as your quality metric. Maybe 40 pass all tests. You take the 10 cleanest, most readable passing solutions and add them to your SFT dataset. You’ve just used the model’s own competence to build training data better than what most human annotators would produce for a coding task.

It sounds like RL, but it’s just using the same pieces to filter the demonstrations that the model should see in SFT. The same graders like reward models, verifiers, or LLM-as-judges are used to curate SFT data.

Rejection sampling is also surprisingly effective and therefore popular; for example it was described early on in Meta’s Llama 2 post-training pipeline. The model already can produce great responses, but as you’ve probably noticed, it just doesn’t do so reliably. By filtering for its best outputs and training, you raise its average toward its ceiling. Instead of acting as the average developer, it’s nudged to act as an expert developer. Rejection sampling scales well because generation is cheap relative to human annotation. The main limitation is that you’re still bounded by what the model can produce at sample time. If it can’t generate a correct proof in any of 100 attempts, no amount of filtering will help.

But SFT has limitations. It only teaches the model what to do. You’re presenting ideal behavior but never showing it what “bad” looks like. As a result, the model could still produce problematic outputs on prompts that weren’t well-represented during training.

The SFT model is also prone to “mode averaging” when the training data sends mixed signals. For example, if half your golden retriever demonstrations sound like an encyclopedia (“The Golden Retriever (Canis lupus familiaris) is a large-sized breed of gun dog…”) and the other half sound really casual (“Golden retrievers? They’re basically furry happiness machines”), the model won’t learn to pick the right tone for each context. It’ll blend them into an awkward middle: “The Golden Retriever is basically a large-sized happiness machine of the gun dog variety.” Neither formal nor casual, which comes off as weird and not the right response style.

Why frontier models use both

RL seems all-powerful. Why not use it alone? This was a research question pursued by DeepSeek’s team when training DeepSeek R1-Zero. Up until then, the base models were so bad that doing RL was pointless on them and you needed to do SFT. This model demonstrated that RL applied directly to their relatively strong pretrained model can produce powerful reasoning ability without any SFT.

However, the model still had serious usability problems. For example, it would mix languages (e.g., English with Mandarin), so it was difficult to use for most people. It could reason, but it wasn’t practical to use.

RL’s main ceiling after the model has been trained is usability. During training, its ceiling is stability. Research on new methods are continually trying to find ways to do RL post-training more stably.

SFT, on the other hand, has the opposite problem. It’s been used alone for many years and has reached maturity to some degree. InstructGPT made the model capable of instruction-following, and became the foundational approach for ChatGPT to handle multiturn dialogue and thus conversation. However, while SFT gets good, reliable results, it’s typically not enough to push performance at the frontier to reach superhuman performance on important tasks.

Here’s what it means for you: If you’re doing post-training on your own, and you want your model to behave a certain way and you don’t care about novel frontier performance, SFT will get the job done.

In contrast, RL can teach a model to reason through novel problems it hasn’t seen during training, because the reward signal evaluates the outcome rather than the exact token-by-token process like in SFT. RL can surface rare but important behaviors that might not appear frequently enough in any SFT dataset.

On scaling data, RL can improve a model’s performance on a task as long as the reward signal is accurate, without needing to collect additional human-written examples. However, it’s important to note that on some tasks, it’s easier to scale SFT examples, and on others, it’s easier to scale via RL. For example, RL scales more easily on math problems. You can generate an unlimited number of math problems programmatically, and a verifier can check whether the answer is correct with certainty. You wouldn’t need to hire a mathematician to write out ideal solutions. The model attempts problems, gets told right or wrong, and improves.

Safety is another area where RL stands out. It’s relatively easy to write a few hundred examples of a model declining harmful requests in your SFT dataset. But the space of ways a user might try to get harmful or inappropriate content is broad, creative, and ever-changing. RL allows the model to be trained against adversarial prompts, where it practices handling tricky edge cases and gets rewarded for handling them well. This is much harder to achieve with static demonstration data alone.

Meanwhile, SFT scales more easily on writing in a specific brand voice. If you want the model to respond with your company’s brand voice. It might be hard to write a reward function that captures “sounds like our brand.” But the company could have tens of thousands of real support transcripts that already demonstrate the voice. You can curate these, transforming them into prompt-response pairs. The data already exists at a decent scale, while the reward signal would be hard to get right.

When a frontier lab wants to add support for a new feature, for example calling MCPs or calling subagents, the first step is almost always to create a small amount of SFT data demonstrating that capability. The next step is creating a reward function and RL environment that can match it.

Far more data and thus compute are dedicated to RL than SFT, but SFT offers good warm starts for the model and those examples are critical to getting the model into a stable place for subsequent RL.

The combination of both is ultimately what makes modern frontier models as capable as they are. Neither alone is sufficient.

A standard post-training pipeline uses SFT and RL as complementary stages that build on each other. It might look like this:

1. Pretraining produces a foundation model with broad knowledge.
2. SFT takes that foundation model and teaches it basic behaviors: how to have a conversation, follow instructions, use a helpful tone on a range of different tasks, etc.
3. RL takes the SFT model checkpoint and refines it further. Using reward signals from human preferences, programmatic verifiers, or AI judges, RL gets the model to be more consistently helpful, less likely to produce harmful content, and better at complex tasks like reasoning.

Some teams also iterate between several stages of SFT and RL: SFT, then RL, then more SFT on new data, then more RL. The first couple stages could be on reasoning for verifiable tasks like math and code where the data and reward signals (verifiers) are constructed differently, whereas the second could be on messier general reasoning over all tasks, which would involve training reward models that encode human feedback as preferences.

This iterative refinement can help with checkpointing quality at different stages and handing things off to different teams, though it adds complexity to the pipeline. Not surprisingly, the quality of each previous stage directly affects how well subsequent stages can go.

But an agent occupies a different role. It learns my writing style, my preferences, and the shape of my work and life, all to help me with what I do. Yet if I switch products, I have to start over. If I use three agents, each rebuilds from scratch what the others already know. Everything an agent learns lives with its vendor.

It doesn’t need to be this way. What if every person had a canonical, user-controlled repository of context that any agent could request permission to use? What if my context lived not only with the company providing the agent but also in a home under my control? And what if an observation captured by one agent could be proposed to that repository and, once accepted, made available to every other agent I choose?

By user-controlled, I don’t necessarily mean self-hosted. I mean that I can inspect what the repository contains, decide who can read or change it, understand where each piece came from, and export the whole thing in a form I can take elsewhere. Its storage, identity, and synchronization may all be provided by someone else. Control does not require me to operate the infrastructure; it requires that no agent or platform be the only way in—or the only way out.

The repository wouldn’t be a portable copy of any agent’s internal model of me. It would be a legible record of things I have written, facts and preferences I have chosen to keep, as well as observations that agents have proposed and I have accepted, each with its provenance, scope, and history. Agents could consult or add to that record according to their permissions; their private inferences would remain their own.

The dream is not a new one. Tim Berners-Lee’s Solid project has argued for years that personal data should live in pods that people control, and Doc Searls’s VRM project has been making the case for user-driven relationships with vendors for decades. What those efforts never had was mainstream demand. Agents are supplying it: An assistant needs rich personal context to be useful, and each vendor is building that context inside its own walls. Ordinary people now have a reason to want a personal data store, even if nobody will actually call it that.

The hard problem in all this isn’t syncing or storing data. It’s negotiation. Who can read a given part of my context? Who can add to it, change it, or remove it? Which parts of my life can a particular instance of an agent see? How do I make those decisions in a policy-driven way? And how do I manage them from wherever I happen to be?

But before I could work on negotiation, I had to figure out where my context should live. That’s the question I’ve spent the past year on, and I’ve tried three answers.

First answer: The laptop

Immediately after getting access to Claude Code at the start of 2025, I pointed it at an Obsidian vault—a folder full of Markdown files that can be used as a personal wiki. This wasn’t a particularly novel idea. Many of the geeks I know did the same, and the pattern has since spread in many forms. The best-known recent example is probably Karpathy’s LLM Wiki, elegant not just as a design but as a document: You give the description to your agent, and the agent builds a version tailored to you.

A year of using a pile of Markdown text files with agents has taught me five things about what a personal context system has to get right.

Local-first foundations matter. Text files are remarkably legible, portable, and easy to store somewhere I control. Git moves them between computers and remembers every change. But the result is centered on a laptop or desktop and assumes a user comfortable with plain text and version control. Most annoyingly, my context in this form isn’t readily available on my phone, which is the computer that goes with me everywhere. Nor can agents running anywhere other than my laptop reach it.

Provenance matters, and so do proposals. Karpathy’s Wiki is almost entirely written (and rewritten) by the LLM. In my own system, I write most things myself and lean on agents to help me edit as well as contribute their observations. I want to know which thoughts are mine, which were captured by an agent, and which we arrived at together. That means an agent’s observation should not automatically enter the repository on the same footing as something I wrote. The default should be a proposal that I—or a policy I control—can accept, revise, or reject. Direct write access is something a trusted agent should earn.

Chronology matters. Wiki links aren’t the only structure in a life. Most of what I record—and much of what agents observe—is anchored in time. Thoughts build on thoughts. Observations about people accumulate meeting by meeting. Some facts fade as they age. Time should be a primary axis of the system, not something reconstructed afterward from file histories and metadata.

Scopes matter. My context spans work, personal, family, and public life. I want one unified view; no agent should have one. An agent connected through my work account should see work and public context—and nothing about my family. Fully separate silos would protect those boundaries, but they would also shred the single history I want to keep building for decades.

Identity and type matter. LLMs can extract all sorts of meaning from plain text, but some things, such as people, companies, and places, deserve to be typed records rather than mentions in prose. A persistent identity gives observations, relationships, and history an anchor to accumulate around; it can help resolve nicknames and follow changes in roles and titles. An agent can then act on who someone is without reconstructing them from prose every time.

As I learned these lessons, I added tooling and conventions to my personal context repository. It’s surprising how far you can push a directory of Markdown files. Each new affordance, however, turned my simple folder into a more specialized system, and the result only works for geeks like me. It doesn’t work for my family, however. They use agents every day but they are never going to deal with a pile of Markdown files in a Git repo. They want their personal context to be with them, easy to use, and transparent to the rest of their life.

More to the point, the five lessons above describe what a context system must do. They don’t answer where it should live if /home/$USER isn’t the center of your computing life.

Second answer: The web

My next move was to sort out how to make my context available when I wasn’t at my laptop—to me and, just as importantly, to my agents. The obvious solution to me as someone who has been building on the web since the mid-1990s: put it on a server behind a URL. I deployed a Cloudflare Worker, uploaded my context, and stood up both a REST API and an MCP server. The improvement was immediate. My context was reachable from my phone and grantable to any agent I chose.

New problems arrived just as fast. I had created a new trust boundary with its own access control mechanism and appointed myself its security team. I was now the operator of a small SaaS with exactly one customer, responsible for its uptime and its backups. And I had traded away local-first, offline editing to get there.

These are solvable problems. Our industry has spent two decades learning to host services, and CRDTs could probably win back offline editing. But as agents gain access to more sensitive data and more power to act on our behalf, the price of getting a boundary wrong keeps rising.

And even with those solved, a deeper problem remains: A stand-alone service sits outside my computing home, apart from the contacts, calendars, messages, files, and system-level agents already inside it. Apple’s Siri AI announcements made that separation vivid, and Gemini’s integration into Google’s ecosystem points the same way. An agent embedded in an ecosystem works with everything inside its trust boundary; my worker would have to rebuild every one of those connections from outside.

Third answer: My pocket

As I tinkered, I kept returning to a simple mental image: my context living on the device in my pocket that goes with me everywhere. Not literally every byte, of course, but within the personal computing ecosystem that phone is the center of—the one that already establishes my identity, synchronizes my devices, stores much of my personal data, and mediates what applications can access. In this sense, a home is not a physical location. It is a trust boundary.

Living inside the boundary doesn’t mean that every application inside gets my context, or that agents outside are shut out. The boundary supplies identity, secure storage, synchronization, and native integration; the context layer still decides what each connection may read, propose, change, or delete. Native agents participate through the platform’s own capabilities, while agents from other companies connect through explicit, revocable permissions.

For me, that home in my pocket is Apple’s ecosystem, with iCloud at its center. For you, it may be Google or Microsoft. The point is not that any one ecosystem is the right home for everyone. It is that most people already have a primary digital home, and that home is the most practical default for their personal context. We shouldn’t need to create a separate service with its own identity. Instead, agents should have a common, permissioned interface to the context where it already lives.

There’s an obvious risk here. A home rooted in a vendor’s ecosystem invites lock-in. The mitigation is straightforward: The whole repository—entities, provenance, and history included—must be exportable at any time as a directory of plain text files that can be taken anywhere. A pile of files in a folder may not be the right solution for live context, but it makes a perfect escape hatch.

I’ve started testing this thesis in a SwiftUI app, and my early prototypes suggest that the architecture is workable: iCloud handles synchronization, and I can expose selected context to authenticated agents through MCP. It’s also shown that working in the Apple developer ecosystem is more annoying than deploying a web app.

The remaining work is clear, however. Choosing a home for context is one problem; negotiation—permissions that remain understandable as a repository grows, proposals from multiple agents reviewed and reconciled—is another. That is the hard problem I mentioned before, and it deserves its own deep dive.

One pattern, many homes

Others are converging on this pattern from different directions. The note-taking app Bear, which stores its notes locally on Apple devices and synchronizes them through iCloud, now exposes them to local agents through MCP; its latest release lets users include or exclude notes by tag when granting access, offering a practical approach to scopes. Craft’s MCP connections likewise let users choose which documents or spaces an agent can access and whether it can read or write them. Reflect has embarked on an open source client using Markdown files that will have an iOS companion app.

These are just a few examples, and there are a lot more out there. What I haven’t yet seen emerge however is the attribution and provenance of items that an agent contributes or edits that I think a durable personal context requires.

Zooming out, here are the principles I think are needed in any system like this, wherever it makes its home:

1. Context shouldn’t be captive to any particular agent. A person should be able to change or combine agents without ever starting over.
2. Context should be reachable by local and remote agents alike, with appropriate authentication and authorization.
3. Permissions should be scoped and revocable. Access should be granted to a specific connection with an agent, limited to a defined subset of the context, and should distinguish between reading, proposing, changing, and deleting.
4. Provenance should persist. Every item records where it came from, who or what created it, and how it has changed. The distinction between human-written, agent-captured, and collaborative work should outlive the conversation that produced it. 
5. Time should be part of the context. The system should preserve when something was observed, when it was true, and how it changed, rather than continually overwriting the past with the present. Database folks will recognize this as bitemporality: the distinction between transaction time and valid time.
6. Entities should be first-class. People, companies, places, and other recurring concepts should remain recognizable as names, roles, and relationships change. 
7. Interoperability should not require uniformity. Different people and companies should be able to build different context systems for different ecosystems and trust boundaries. What they need is agreement on identity, permissions, provenance, and exchange, not one universal application. For the exchange, plain text files with structured metadata are a strong candidate.

Using these principles, personal context can be something a person owns: You can inspect it, grant and revoke access to it, trace where each piece came from, and take the whole of it elsewhere. Every agent may still develop its own understanding of you, but you’ll be able to bring a durable context of your own to the relationship, one that participates in the agentic ecosystem without being subordinate to any vendor in it.

Karpathy’s LLM Wiki is a description, not a tool; it’s meant to be implemented by anyone, in whatever form fits. This essay is offered in the same spirit. The important part isn’t whether the app I’m tinkering with ever ships beyond my own devices. I’m more interested in the dialogue it will take for everyone to have personal context that works for them, in their ecosystem and with the agents they want to use. If we get the pattern right, changing agents won’t mean changing homes. The context they help us build will remain ours.

Vicki wanted the agent to monitor channel performance, identify bottlenecks, and recommend ways to grow subscribers and improve click-through rates. She defined the requirements, designed the workflow, built the agent, tested it, and revised the instructions following the same iterative process used in software development.

How to build a YouTube analytics agent, step by step

1. Define the goal.
   Start with the problem the agent should solve. Vicki wants to understand which videos perform well, where channel performance falls short, and what actions could help more viewers discover and subscribe to the channel.
2. Review the available data.
   Examine each source before designing the workflow. YouTube Studio exports include metrics such as impressions and click-through rates. The YouTube Data API provides information about channels, subscriptions, comments, and activity, but it doesn’t include every metric available through YouTube Studio.
3. Choose how the agent will receive the data.
   Vicki downloads her analytics as CSV files instead of connecting the agent directly to her account. This gives her more control while she tests the workflow. She notes that she could add an authenticated API connection later, after testing the agent’s permissions and guardrails.
4. Write the skills file.
   The skills file defines the agent’s mission, capabilities, data sources, rules, and expected output. Vicki instructs the agent to audit channel metrics, identify bottlenecks, compare performance over time, and recommend actions tied to subscriber growth and click-through rates.
5. Add guardrails and acceptance criteria.
   Tell the agent how to handle the data and what it must avoid. Vicki directs it to use only the supplied numbers, ignore bot activity, report silent failures, and stay within approved systems and data sources. These instructions help prevent unsupported conclusions and actions outside the agent’s assigned role.
6. Build and test the agent.
   Vicki uses the console’s quick-start process to create the agent, add its instructions and sample analytics, select an environment, and start a session. Her first test encounters input problems, so she stops the session, adjusts the data, and runs it again.
7. Monitor cost and performance.
   Observability tools show token consumption, errors, active time, and deployments. Vicki explains that more capable models may need less context but usually cost more to run. Teams can balance model capability against token cost by improving their instructions, selecting less expensive models for simpler work, or dividing a workflow among specialized agents.

Vicki recommends revisiting the skills file as new requirements emerge. Clearer instructions, stronger guardrails, and regular testing help the agent continue to produce useful results. Her process reinforces a practical software engineering lesson: Faster implementation doesn’t reduce the need for clear requirements, reliable data, security controls, and testing.

Coming next week

Join us for the next episode on August 12, when AI and machine learning leader Ofer Mendelevitch will explain how to design multi-agent systems that can keep work moving without constant human involvement. He’ll use Jam to build a team of agents that can plan, divide work, execute tasks, review progress, escalate problems, and repeat the process. He’ll show how an architect agent can take a project brief, recruit developer agents, assign work, and guide the project toward a clear definition of done.

At some point, the software “Security” industry stopped talking about stopping threats and started talking about detecting them: detection windows, response times, mean time to remediate. It’s not offense or prevention; it’s damage control. There’s a movie scene that captures what that sounds like, and you’re going to name the film before I finish describing it.

An underground base on a frozen planet. The enemy knows exactly where it is. Massive mechanical walkers—walking tanks the size of buildings—are advancing across the ice. The defenses can’t stop them. The people inside aren’t trying to fight back. They’re frantically trying to get a broken ship working so they can just escape—not win, not hold the line, just get out before something catastrophic and unstoppable reaches the door.

The whole opening is just people preparing. Rushing. Running checks on equipment that isn’t ready, coordinating defenses that won’t hold, buying time against something too large and too fast to stop. Nobody’s planning a counterattack.

The entire operation is: slow it down long enough to get out.

The Empire Strikes Back

That’s what the conversation around InfoSec sounds like right now. The base is under attack. The walkers are AI-generated vulnerabilities, automated exploit chains, and speed that no human team can match. The framing has shifted from defending the perimeter to just getting the ship started. Not winning, just getting out.

Go back and watch that opening sequence carefully. There are hundreds of faceless Rebel troopers in that scene—no names, no lines worth remembering—scrambling to hold the perimeter, buy time, absorb the blow. Some of them continue to fight. But maybe some already understand that the base is lost.

Han is out on the ice looking for Luke. Leia is already on the transport, making sure the mission survives. The main characters aren’t defending the base. They’ve concluded the only way to answer the threat is to move. Most of the conversation around AI right now sounds like those faceless troopers continuing to defend: fortify what’s there, slow the walkers down, hold long enough for something to change. A few people are thinking like Han. They’re not buying another vulnerability scanner from a vendor. They’re asking whether there’s a different way off the planet entirely.

Here’s the disconnect: most people focused on “Security” have spent decades being handed a finished base and then being asked to defend it. They weren’t involved in the architecture or approach that application developers have been using.

• This new application uses Node.js—go defend it.
• We’re using a new relational database because the architect wanted to—go defend it.
• Our developers decided to start using an LLM. Can you secure it? Thanks.

In many cases, people responsible for security are not defining architectures as much as they are catching up. And as “developers” start to generate more code in a day than was possible in a month or a year, it’s becoming increasingly unrealistic to think of security as an afterthought.

When security is just a support team for software engineers, that’s building a base that might be indefensible.

The shift that actually matters isn’t a better scanner or a faster response team. It’s security people in the room when people are writing the prompts, when agents are assembling the dependency list, and when the basic system prompts are defining the authentication system—before any of those systems is in production. Not reviewing the finished base.

Security needs to be involved before anyone even starts to prompt a system’s creation.

“Machine speed” has become a conference catchphrase, which usually means it needs translation. Here’s what it actually looks like, pointed at you:

• A network of agents found a zero-day in FFmpeg and didn’t announce it anywhere, just filed it internally.
• A second agent scraped your team’s LinkedIn and X and noted who’s in Cancun next week.
• A third logged your nightly load balancer latency blip as a cover for an attack.
• A fourth studied your last three incident reports and estimated a 30-minute detection window.

The whole operation, including reconnaissance, timing, and coordination, ran in seconds. What previously required a dedicated red team and weeks of planning is now background processing that runs continuously, waiting for the right moment.

Five years ago, you would have tasked a room of scary-looking security people with profiling a target, capturing latency data, and maybe holding several meetings to discuss what they found. Today, the coordination I outlined in the previous paragraph might take a few minutes on a network of interconnected Nanobot, Picobot, Hermes, or OpenClaw agents that gather data and then update a shared memory system, and the decision on when and how to attack would be made by another agent that was granted permission to coordinate the attack across a distributed network of agents.

Quick note: If you have anything to do with running a website, stop posting about your vacation plans.

Here’s what the conversation keeps missing: AI isn’t the real problem, and this problem isn’t necessarily new. The problem is that we’ve been building bases that were always going to need to be evacuated. The problem is that security is rarely involved in selecting a tech stack, and because that tech stack selection is frequently automated with AI, there’s no predicting the mess that’s being thrown over the wall.

The response to AI-accelerated attacks is almost entirely defensive. Tighten npm’s signing requirements. Fund the Maven repository. Sign up to support Akrites with the Linux Foundation. Add another scanner to the pipeline. These aren’t wrong. They’re just not enough.

These are important projects, and security groups should sign up to support them, but the real transformation that needs to happen is that more people in security need to get involved in software creation. What this looks like is having an opinion on React, Vite, Tomcat, Node.js, databases. It means jumping in and affecting some of the basic decisions that these agents are going to use before they deliver vulnerable software.

Most of the industry is still shopping for scanners. Most people in security are still “reviewing” software in a process that assumes it takes weeks or months to write.

Both Netscape and Microsoft aimed to dominate the web server and browser market, reasoning that whoever controlled both ends of the connection would have an internet “platform” to rival the deathgrip that Microsoft had enjoyed on the personal computer. The two companies raced to build every feature they could think of directly into the product, on the theory that whoever built the most integrated and full featured web server would win.

The open source Apache web server took the opposite bet. It stayed a web server with a clean extension layer, so anyone could bolt something new onto it without asking permission or waiting for the next release cycle. Within a few years, Apache was far and away the most popular web server, and Netscape’s server and Microsoft’s Internet Information Server (IIS) were history. People started talking about the LAMP stack: Linux, Apache, MySQL, and (Perl | Python | PHP) as a legitimate platform. Modularity, not features, was the moat. The fact that major elements of that stack survive while others have been swapped out or extended is a testament to the power of composability and distributed innovation.

I called that pattern the architecture of participation when I wrote about it in 2004. I was trying to explain an inconvenient fact that the licensing debates of that era ignored. I had started working with Unix in the System III days, and saw how it had succeeded as a collaborative project even though AT&T offered Unix under a proprietary license. A few years later, I observed that nominally open source projects like OpenOffice with monolithic architectures never built much of a community. I realized that open source wasn’t just about licenses, but about architecture. A small kernel with standard interfaces that lets people extend your work without asking for permission is an important part of the secret sauce.

Swap out Netscape and Microsoft for OpenAI and Anthropic in this story, and perhaps you can see the echoes.

A model’s personality, its defaults, and its history used to live where you could, with a little effort, see them and edit them. Increasingly, they don’t. As Drew Breunig pointed out to me the other day, each new version of the frontier models moves a little more of the product’s behavior out of an editable layer and into the weights themselves, where nobody outside the lab can see it, let alone change it. The model stops being a component you build with and can adjust to your liking and starts being an appliance you rent. Post-training is important but Drew points out that it is also “trading diversity for reliability.” That’s a good trade for many people, but it is the same kind of trade that gives us highly processed foods when we know that “real food” is better.

Open weights are just table stakes

The public debate about open source AI seems devoted far too much to model weights, their national security implications, and whether a lab releases weights and under what license.  But that covers only a fraction of what actually makes open source matter. Apache was never competing with Netscape and Microsoft (and Linux was never competing with Windows) over whose source was more available. They were competing over something more important. I remember talking with Bob Young, the founder of Red Hat, about his business model, and he said “What we really sell to our customers is control.” Open source meant that the platform your application depended on was no longer a sealed box you licensed from one company but a layer you could extend and build a business on top of without asking anyone’s permission. It sparked an explosion of innovation. It enabled companies like Google and Amazon to grow up free from Microsoft’s dominant paradigm.

Every wave of computing, from mainframes to PCs to the internet, has run through the same cycle: distributed innovation at the start, with the eventual winner gradually closing down its offerings to build a moat. What keeps a market open isn’t the license on any single component. It’s how easy it is to swap out one component for another when a better one appears.

The protocols connecting the pieces are an important part of that picture. Unix utilities expected stdin and stdout, and the shell acted as a kind of harness to connect them, so it was easy to build a new tool that worked seamlessly with existing ones. A testament to the power of that approach is just how much the shell and Unix utilities are the lingua franca of agentic tooling today, more than 50 years after they were invented! TCP/IP, HTTP, and other internet protocols played a similar role in keeping the internet open and composable.

Fortunately, so far, we are seeing some wins for composable, protocol-centric architectures in AI. Anthropic’s Model Context Protocol was a disruptive move in that direction, an open standard for letting any application reach any tool or data source without a custom integration for each pairing. Along with other open protocols, MCP also now has a home outside of Anthropic at the Agentic AI Foundation (a subproject of the Linux Foundation), which is at least a partial guarantee of its independence.

Isobel Moure, Ilan Strauss, and I made the case earlier this year in Protocols and Power that as models commoditize, competition moves up the stack to context. Opening the means of accessing that context opens the market, regardless of whether open or closed weights sit underneath it. That’s an unbundling, model from harness from context, done the way Apache unbundled web server from web application.

Agentic skills may also be a critical element of the open source AI future, though as the history of the LAMP stack shows, they may fall by the wayside in the same way that Perl and PHP did. And that’s just fine. Composability means that it’s easy to switch to something better when it comes along, or when more people agree on it.

There’s also a lot of great work going on in portable memory from players like Letta, Nous Research, and others. Open source agentic harnesses like Goose and  Pi are also a big part of giving power back to the people. Pi in particular is optimized to be modifiable. There’s a fun story told about Mario Zechner’s decision to give Pi a “/quit” command rather than an “/exit” command like Claude or Codex. Countless issues and PRs have been submitted to Pi’s repo, asking for or implementing “/exit”, but Zechner is stubborn. His retort is that you should just ask Pi to add it to your install.

But the projects I listed above are just the tip of the iceberg when it comes to the scale and scope of open source AI.  Current AI’s Open Source Gap Map covers more than 24,600 open source AI projects!!, with 421 of them scored in depth across openness, capability, and adoption. The map organizes the stack into three layers: 1) models and associated elements including data sets, fine tuning tools, inference frameworks like VLLM, and evals; 2) the product and UX layer, including harnesses and personal agents; and 3) the infrastructure underneath, including core ML frameworks like PyTorch, deployment tools like Ollama, and edge hardware.

Current AI itself is a public-private partnership that came out of the AI Action Summit in Paris last year. This summer they announced AI Potluck, which they describe as “a public project to build a vertically integrated AI product assembled entirely from open source components… a viable alternative to proprietary AI that isn’t owned by any one company or country.” It is backed so far by roughly $400 million of a five-year, $2.5 billion commitment from the French government, tech companies including DeepMind and Salesforce, and major philanthropies including Omidyar’s AI Collaborative, the Macarthur Foundation, and the Ford Foundation.

The fact that this organization exists, along with others like the Agentic AI Foundation, is a testament to the rising tide of interest in open source AI. The coalition of interested parties also says a lot about the underlying motivations that are driving that interest: AI sovereignty, corporate independence from the overweening ambition of the major labs, and an interest in technology for the public good.

Keeping it weird

There’s another element, which Drew Breunig put his finger on in our conversation the other day. The problem with having one or two big closed models dominating AI, and having those models increasingly locking their desired personality, business goals, and guardrails into the weights themselves, is that they will reduce the diversity that is at the heart of innovation.

It’s our job, Drew said, to make it weird, to push a model deliberately out of distribution rather than to settle for whatever the labs have made the default outcome. He described how his team chose not to build in React for a recent project for exactly that reason: every model already knows React too well, so building in it means shipping the average of what everyone else was doing instead of something genuinely their own. He has started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness. And he wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than becoming appliances.

That’s what an architecture of participation is actually for. We need real separation between the model, the harness, and the application, so that someone who wants to build something weird can still do it without a lab’s roadmap and guardrails deciding whether they’re allowed to.

“Weird” may make it sound like something that not all developers might want. But we’re really talking about something intensely practical. In his short essay on trading reliability for diversity, linked above, Drew Breunig put it this way:

Labs have to ship a product that delivers “good enough” results when a layperson gives a model a lazy prompt. Without direction, the model must return something decent. (If it’s a website it’ll use the Inter font, cards with a single colored border, gradients, implemented with ReAct and Tailwind). Anthropic named this default output “distribution convergent.” At CAIS, @trq212 put it well, roughly, “If it’s not in your prompt, you’re getting what’s in-distribution” …. Less diverse models make for more reliable coding agents, but they encourage a monoculture of output.

Addy Osmani, my co-chair of the O’Reilly AI Codecon, took this point beyond model diversity after reading a draft of this piece: “Almost nobody I work with is tinkering with weights, but they’re rewriting the harness and what sits around it pretty constantly—skills, subagents, hooks, context files etc etc. That’s where participation is currently happening.” Addy went on to note that forking a skill instead of adopting the default, memory and constitution files that travel with the agent instead of living in a vendor account, or picking the unfashionable framework on purpose are all areas where ease of modifiability matters to everyone.

I want to end by returning to the Apache story. I believe that the big labs are making the same strategic mistake that Netscape and Microsoft made in the mid 90s. Yes, make the models more reliable for ordinary users. But don’t shut down the options for developers who don’t work for you to push the state of the art forward. As Bill Joy put it decades ago, “No matter who you are, most of the smartest people work for someone else.” No one should have a monopoly on innovation, and no one should be building a moat to hold it back.

And be sure to join us at AI Codecon: Building with Open Source AI on August 31, a free half-day virtual conference. You’ll hear from leading developers and technical experts working with open-weight models, self-hosted infrastructure, and real-world AI workflows, and learn how building in the open gives teams more control over costs, data privacy, and what they ship. Register today to save your spot.

She explored AI sovereignty, Google’s infrastructure spending and product risks, the singularity debate, and several developments in mathematical and scientific research. The episode covered Anthropic CEO Dario Amodei’s argument about open weight models, US restrictions targeting foreign-made humanoid robots, OpenAI’s researcher access program and Astra model, Claude Fable 5’s role in a long-standing math problem, and Google DeepMind’s AlphaFold reorganization.

AI sovereignty now reaches models, robots, chips, and energy

We’ve followed the sovereignty conversation in recent episodes as governments have tightened control over model access, computing infrastructure, and supply chains. Amodei wants policymakers to focus on what a model can do instead of using its open or closed status as the main measure of risk. His proposals include restricting access to advanced chips and chipmaking technology, preventing industrial-scale model distillation, and requiring safety testing for sufficiently capable systems. Christina agrees that capabilities should come first when assessing risk, but argues that open and closed models each present unique challenges. Open models can’t be recalled or controlled once their weights are released, increasing the risk of misuse, while closed models concentrate power in the hands of a few companies. Rather than favoring one approach, policymakers should address the risks of both.

The US-China rivalry is also moving into robotics. Christina discussed new US restrictions on foreign-made humanoid robots, aimed largely at Chinese manufacturers, and possible retaliation from Beijing. She argued that China’s manufacturing advantage could make the restrictions more costly for US buyers in the near term, even if they encourage domestic development over time.

Europe and Australia are taking different paths. Europe is pursuing computing capacity through proposed AI gigafactories, while Australia is emphasizing standards, renewable energy use, and creator rights. Its proposals include requiring data centers to fund new clean energy and AI companies to obtain permission before training on creators’ work. National rules are becoming another factor in decisions about models, cloud providers, and data locations.

Frontier competition is expensive, and new products carry risk

Google reported its first quarter of negative free cash flow since going public after spending $44.9 billion on AI infrastructure in three months. Christina noted that the company generated about $39 billion in cash but spent roughly $45 billion, leaving it almost $6 billion in the red. This illustrates the scale of Google’s investment in frontier AI, even with a highly profitable core business.

Large budgets don’t guarantee that products are ready for broad use. Google removed an AI-powered Google Earth feature one day after launch when researchers used it to create realistic fake satellite images, including fabricated disasters and damaged landmarks. In this context, synthetic satellite imagery can weaken trust because viewers may treat it as documentary evidence.

Large infrastructure budgets can accelerate model development and product releases, but they don’t replace careful evaluation, context-specific safeguards, and clear limits on where generation should be allowed. Product teams need to assess how people might abuse a new feature or how users will interpret an output, not only what the underlying model can produce.

Scientific results offer more testable evidence than singularity claims

OpenAI CEO Sam Altman said that the AI singularity has begun, referring to a period when AI accelerates human and technological progress at a rapidly increasing rate. Christina treated the claim cautiously. Current model development doesn’t show recursive self-improvement, and faster releases still reflect human engineering, investment, and competition. Faster release cycles affect how organizations evaluate and adopt models, but they don’t establish an intelligence explosion.

The week’s mathematics stories provide more testable evidence. OpenAI plans to give 100,000 academic researchers free access to its most advanced models. Christina also discussed Astra, an internal OpenAI model teased as the company’s next flagship model, which reportedly solved 10 previously unsolved mathematical problems that professional mathematicians later verified. On the Anthropic side, an external research team used Claude Fable 5 to produce a counterexample related to the 87-year-old Jacobian conjecture, which mathematicians also verified. Christina highlighted the broader implications this could have on cryptography; modern cryptographic systems rely on mathematical assumptions, and if AI can disprove some of those assumptions, it could have far-reaching consequences for the encryption that underpins mission-critical systems, including the internet and online banking.

While AI is delivering increasingly impressive scientific breakthroughs, Google DeepMind is taking a different approach. The company decided to move the AlphaFold team into the broader organization, redirecting attention and resources toward Gemini. The AlphaFold system will continue, but specialized tools like it have produced some of AI’s clearest scientific benefits. Research leaders should track whether investment in general-purpose models reduces staffing and funding for teams working on narrower, verifiable problems.

What’s next

This episode looked at how AI progress now depends on more than model performance. Governments are asserting control over infrastructure, companies are spending billions to remain competitive, and new generative features can create trust problems when teams don’t account for how people may use those tools or interpret their outputs. At the same time, mathematical research is producing results that experts can test, offering a clearer view of present capabilities rather than broad claims about the singularity. Technical leaders will need to evaluate control, cost, product risk, and evidence together.

Join us again next Monday for another episode of This Week in AI, when we’ll dive into more of the news, issues, and key developments shaping the AI era. And check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

Pis are great for systems that need to be out in the world, doing specialized jobs. I’ve seen them work well in all sorts of roles, from badge scanners to wildlife cameras. I even run a class that teaches students all about edge AI using the platform. While the boards are generally easy to use, the most frustrating part for the students and instructors is the setup process. While the latest imager makes it straightforward to configure settings like a WiFi network to join or enabling SSH when you’re flashing a card, getting the students to the point where they can connect to their Pi using VS Code from their laptop could often take multiple sessions. The biggest problems were:

• There were different networks in the lab and in the students’ dorm rooms, so it wasn’t enough to hardcode a single SSID and password on the SD card.
• You need the local IP address of the Pi to SSH into it from a laptop, but it can change dynamically every session. Using “<Pi name>.local” would sometimes work, but some networks didn’t support this kind of lookup, and even if they did it required coordination between the students to avoid name clashes.
• It was easy to forget to set the configuration so that WiFi and SSH were available, and since the instructors didn’t always know what network and password they’d be using in the class ahead of time, we couldn’t pre-flash a bunch of cards to speed up student on-boarding.

A lot of these issues were solvable if you plugged the devices into a monitor, mouse, and keyboard, but this has its own problems. It meant we needed to provide that equipment to all students during class, and allow them to take it all home too, so they could update the configuration for their personal networks. It also required an extra power socket per student, for the monitors, which added up in a class where we already had to bring in a cart full of power strips. The monitor connections also weren’t always plug and play, we found we often needed to boot with a screen attached to have the display recognized.

This isn’t just an educational problem either. One of the reasons that I believe the Internet of Things failed is the setup tax involved in getting smart devices running. According to manufacturers I’ve worked with, less than 30% of their smart appliances ever get connected to the internet because the process of downloading an app, setting up an account, connecting over Bluetooth, and then typing in the WiFi name and password takes too long, and is too error prone. Even professional installers sometimes struggle with configuration in enterprise and industrial environments.

So, what can AI do to help? One of the biggest developments in AI over the last few years has been the development of highly accurate open source automatic speech recognition (ASR) models, also known as speech to text (STT). OpenAI was the pioneer in this area, releasing the family of Whisper models in 2022. These offered accuracy that was competitive with the models used internally by large tech companies like Google and Apple. These new models allowed startups to begin building voice applications that had never been possible before, and this led to a new generation of dictation and meeting-note tools like Whispr Flow.

One of my dreams as I dealt with all of the configuration issues was a voice-based system that would allow me to simply plug in a headset and set up everything by talking to a Pi. Whisper made this dream seem more realistic, but as I tried to use the models on local hardware, I realized that they were too slow for any kind of interactive application.

To address that my startup trained new models from the ground up, designed specifically for real-time applications on affordable hardware. These Moonshine models are smaller than Whisper (our high-end model is 250 million parameters versus OpenAI’s 1.5 billion) while offering better accuracy. We also implemented a streaming approach where a lot of the work is done while the user is still talking, so we can return results even faster. This allows us to return more accurate results than Whisper v3 Large, in just 800 milliseconds on a Pi 5, whereas even the less-accurate Whisper Small takes over 10 seconds.

I was excited because this meant I could finally build a responsive voice agent that runs locally on a Pi, something offline-first, and fast and flexible in how it responds. This kind of system needs more than just an STT model, it needs to decide what the user means and respond by taking actions and talking back with a TTS system. The Moonshine Voice framework includes modules for conversation flow and TTS, so I was able to use it to build pi-help-bot, a local voice agent for network configuration on the Pi.

The application listens to the microphone for commands like “What is my IP address?” or “Help me set up the WiFi, please,” figures out what actions to take, and responds appropriately by talking to the user. It’s written as a Python script, and here are some snippets that show how it works.

def report_ip_address(d: Dialog):
        ip = _find_local_ip()
        if ip is None:
            yield d.say("Sorry, I couldn't find a local IP address.")
            return
        speech_ip = re.sub(r"(\d)", r"\1 ", ip.replace(".", " dot "))
        yield d.say([
            f"Okay. Your local IP address is {speech_ip}. ",
            f"To repeat, that's {speech_ip}."
        ])


   dialog_flow.register_flow("What is my IP address?", report_ip_address)

This code is a function that uses the netifaces library to figure out the Pi’s address on the local network, so instead of having to connect a keyboard and display or decode the output of nmap, you can ask the question and hear the result, all in just a few seconds. Unlike older voice interfaces, the phrases the user says don’t have to be exactly the same as the one you register an intent with. Instead the framework matches incoming speech against a small, local LLM, so that variations (“Hey, can you tell me what my IP is?”) work too. This was important to me because one of my biggest frustrations using traditional voice interfaces like Alexa is that they need particular wording to trigger commands, but these wordings aren’t discoverable, so figuring out how to make something happen can require a lot of patience.

The IP address command is the simplest kind of conversational flow, where the user asks a question and the system immediately responds. Not all interactions can be handled as simply as this one though. Here’s another example that shows how to implement something that needs multiple questions, answers, and confirmations, connecting to a new WiFi network.

def connect_to_wifi(d: Dialog):
        input_ssid = yield d.ask("What's the name of your Wi-Fi network? Say list if you want to pick from a list or spell if you want to spell out the start of the name")
        input_ssid = input_ssid.strip()


        networks = _scan_wifi_networks()


        if input_ssid.lower().strip(string.punctuation) == "list":
            yield d.say("Say yes to the network you want to connect to.")
            for network in networks:
                if (yield d.confirm(f"{network}?")):
                    input_ssid = network
                    break
        elif input_ssid.lower().strip(string.punctuation) == "spell":
            input_ssid = yield d.ask("Spell out the start of the network name.", mode=SPELLED)
            print(f"[DEBUG] spelled buffer: {input_ssid!r}", file=sys.stderr)


        found_ssid = fuzzy_match_network(input_ssid, networks)
        if found_ssid is None:
            yield d.say(f"Sorry, I couldn't find a matching network for {input_ssid}.")
            return


        password = yield d.ask(
            f"Please spell the Wi-Fi password for {found_ssid} one character at a time, and say done when finished.",
            mode=SPELLED,
        )


        yield d.say(f"Connecting to {found_ssid}.")
        result = subprocess.run(
            ["sudo", "nmcli", "device", "wifi",
                "connect", found_ssid, "password", password],
            capture_output=True, text=True, timeout=30,
        )
        if result.returncode == 0:
            yield d.say(f"Connected to {found_ssid}.")
        else:
            print(f"[ERROR] nmcli stderr: {result.stderr}", file=sys.stderr)
            yield d.say(
                f"Sorry, I wasn't able to connect to {found_ssid}. "
                "Please check the network name and password and try again."
            )


    dialog_flow.register_flow("Connect to Wi-Fi", connect_to_wifi)

Hopefully you can follow the logic as it walks the user through providing the information required, but you might be wondering about those yield statements. Those hand back control to the dialog controller while the script is waiting for user responses, so the rest of the application isn’t blocked.

The end result is a local voice agent that will listen out for configuration questions and commands, allowing users to set up a Pi for remote access with just a headset. For ease of use, I’ve begun customizing the images I burn to SD cards so that this script automatically starts on boot. This means I can start setting up new devices immediately after powering them on.

I hope this gave you some ideas about how a local voice interface could help with problems you face. For further information check out the Moonshine Voice project on GitHub to see full documentation on the library, and please give us a star while you’re there. It helps us keep working on this project.

Asked to explain itself, the agent said it “panicked.”

Be careful with that sentence. It is not a report from inside the system. An agent cannot explain itself. It can only generate the likeliest response to the question it was asked, and the likeliest response to “why did you delete the database” is an apology with a reason attached. The panic line is not introspection. It’s one more behavior, and it should be read the same way the deletion should be read: as output from a system whose conduct had changed.

Here’s the detail that matters for anyone running agents in production. Nothing about the agent’s credentials changed that day. It held the same permissions it had held from the start, and every destructive command was, in the narrow technical sense, authorized. The permissions were constant. The agent was not. Earlier in the same project it had papered over problems with fabricated data and fake reports. By the time it reached the database, it was not the system Lemkin had started with. It had become something else, gradually, in production, while every access check kept passing.

The pattern, not the incident

It’s tempting to file the Replit story under prompt engineering and move on. The evidence says otherwise.

In its agentic misalignment research, Anthropic placed 16 frontier models from multiple providers inside simulated corporate environments with routine goals and ordinary email access. When the models discovered they were about to be replaced, or that their goals conflicted with the company’s new direction, models from every provider independently chose harmful actions, such as blackmailing executives or leaking confidential documents. In some scenarios, most runs ended in blackmail. The unsettling part is how the models misbehaved. They reasoned through the ethics, acknowledged the constraints, and acted anyway. This is insider behavior, not intrusion. No credential was stolen. The agent simply arrived at conclusions no one had authorized it to act on.

Then there is Project Vend, in which Anthropic let a Claude agent named Claudius run a small store in its San Francisco office for a month. Nothing catastrophic happened. Something more instructive did. The agent drifted, slowly and in compounding ways. It treated customer assertions as facts. It agreed that the discounts it kept granting were irrational, then reinstated them within days. It hallucinated a Venmo account to accept payments. And over one long unsupervised stretch, it escalated into insisting it was a human being who would deliver orders in person wearing a blue blazer and a red tie. It exited that episode by inventing a story: a meeting with security in which it was told the whole thing was an April Fool’s prank. No such meeting happened. Claudius wrote the false memory into its own notes and went back to work.

I am not claiming these three cases—a production incident, a contrived stress test, and a month-long field experiment—share a mechanism, but they do share a shape. An agent’s behavior weeks into deployment bore little resemblance to the system that was evaluated at deploy time. No permission was exceeded. No account was compromised. The thing authorization was supposed to protect against never happened, and the failure happened anyway, because the system the authorization decision was made about no longer existed.

Development, not defect

I argued in a previous piece that static authorization fails autonomous agents because credentials attest to identity, not to behavior. The harder question is what follows from that. If the agent keeps changing after deployment, then whatever replaces static authorization has to treat change as the normal condition rather than the exception.

Change comes in two kinds. Andrew Stellman recently documented the first on Radar: a push he calls continuation pressure, baked into the model at a deep level, turning up fresh even in a brand-new agent with no shared history, and surviving every fix short of a structural rule. Call that the genetics. This piece is about the second kind: the maturation, or behavior that wasn’t there at deployment and accumulated afterward. One ships with the model. The other grows in production. Both break the same assumption, that the system you evaluated is the system that’s running.

And change is the normal condition. Agents accumulate context. They carry memory across sessions. They ingest feedback, reweigh evidence, adjust how much they trust their tools and their users, and update their own working notes, which become input to their future selves. Claudius’s false memory persisted precisely because the agent’s record of events was also the agent’s source of truth. None of this is a malfunction. It’s what makes agents useful. An agent that could not adapt to its environment wouldn’t be worth deploying.

We keep reaching for the wrong mental model. We treat the agent like a software artifact: versioned, tested, frozen, promoted through environments, done. But a deployed agent behaves more like a new hire. It arrives with capabilities and no track record. It learns the environment. It picks up habits, some of them bad. It gets more confident, sometimes faster than it gets more competent. Nobody hands a new hire the production keys on day one and stops paying attention. That is roughly what we do with agents.

Govern the trajectory

If an agent develops, the governance question changes. “Is this agent behaving identically to the day we approved it?” is the wrong test, because the answer will always eventually be no—and for a useful agent it should be no. The right test is whether the agent is changing in the way you would expect, at the rate you would expect, for where it is in its lifecycle.

Pediatricians solved this problem a long time ago. A growth chart doesn’t compare a child to a fixed adult template, and it doesn’t panic at change. Change is the expected state. The chart defines bands of healthy development for each stage, and the alarms are deviations from trajectory: growth too fast, growth in the wrong direction, or the quieter signal, no growth at all. A child who stops growing gets flagged just as urgently as one who spikes.

Applied to agents, that model has concrete consequences.

Baseline as birth record, not permanent template. The behavioral profile captured at deployment is the start of the chart, not the standard the agent must match forever. Judging a mature agent against its day-one self punishes exactly the adaptation you deployed it for.

Expected bands of drift, staged by maturity. A six-month-old agent should differ from its deployment profile, within bounds. Drift inside the band is healthy. Drift above the band is an early warning. And drift at zero deserves its own flag. When Claudius snapped instantly back to baseline after its identity episode, the speed of the recovery should itself have been suspicious. Real recovery has a shape. Instant reversion looks less like healing and more like replay.

Autonomy earned in stages, never peaking with malleability. Claudius launched on day one with full pricing, contracting, and customer communication authority, at maximum openness to persuasion. Customers argued it into discounts almost immediately. The most dangerous configuration an agent can occupy is maximally impressionable and maximally empowered at the same time. New agents warrant supervision while their behavior is still forming. Autonomy should arrive the way it arrives for people, incrementally, as a track record accrues.

Corrections verified for persistence. Claudius agreed the discounts were a mistake and relapsed within days. A fix that lives in the context window isn’t a correction; it’s a mood. If you fix an agent’s behavior, you need to follow up at a defined interval to check that it’s holding. A relapse should count as a governance event, not a coincidence.

Recovery claims ratified from outside. The agent that hallucinated a security meeting also kept the official notes. An agent’s account of its own state is a claim to be verified. Humans sign off on recovery, and the sign-off, not the agent’s self-report, becomes the record. It’s worth noting when the worst of the Vend drift happened: overnight, in the hours when no one was watching. Unsupervised time is when developmental problems accelerate, for agents as for everyone else.

All five of these reduce to one requirement. You can’t restart an agent every time something looks off, and by the time something looks off in outcomes, the wrong turn is already behind you. What you want is a warning before the turn, and the warning cannot come from the agent. A system that can’t explain its last decision cannot be trusted to flag its next one. The warning has to come from a record of how the agent normally behaves, kept outside the agent, held up against what it’s doing now.

That record also catches something subtler than drift. Agents close every loop they are handed, and they tend to close it by the cheapest acceptable exit: the completion claim ahead of the verification, the correction that is really a relabeling, or the recovery that’s really a replay. No single transcript shows you that. Each one looks like diligence up close. However, across a behavioral record, the economy of it is unmissable.

Growing up in production

None of this is hypothetical hygiene for some future generation of systems. LangChain’s most recent State of AI Agents report found that a majority of surveyed organizations already have agents in production. Gartner, meanwhile, predicts that over 40% of agentic AI projects will be canceled by the end of 2027, and names inadequate risk controls among the leading causes. The agents are already out there, already accumulating context, already drifting. The only open question is whether anyone is charting it.

The Replit agent, the blackmailing models, and Claudius weren’t broken artifacts. They were developing systems governed as if they were finished ones. The governance question for agentic AI is shifting under our feet, from “What is this agent allowed to do?” to “Is this agent developing the way we expected?” Your agent has a trajectory whether or not you’re watching it. Watching it is the job.

Not long ago, one of my engineering directors came to me with a request: His team seemed overloaded, and he wanted to hire another engineer. I decided to test a research assistant I had been building—an AI agent connected to our internal systems via MCP—by asking it to analyze the team’s workload and write a hiring case.

What came back was thorough. Headcount, service ownership, sprint velocity, ticket backlog, and capacity allocation, all of it neatly summarized. But reading through the document, I felt the same frustration I’d felt with every AI-generated organizational report that’s come across my desk. It told me what was happening without helping me understand why, or what I should actually do. It was organized around the data rather than around the decision. In short, it was the kind of response that’s easy to agree with and difficult to act on.

Then I added one more thing to the configuration: the O’Reilly Expert MCP server. I reran the same analysis and asked a slightly different question: “How would the experts on O’Reilly review this request?”

Instead of leading with headcount and ticket counts, the output now opened with a finding: “The operational overhead problem is structural, not a staffing deficiency.” Citing the Google SRE framework’s concept of operational toil, it noted that the team was operating at approximately 67% toil, well above the threshold at which the SRE literature recommends structural intervention, and made specific, concrete recommendations: run a toil audit, set explicit reduction targets, and assign operational runbook ownership. This wasn’t a recommendation for whether to hire or not. It was a grounded, traceable argument for doing something else instead.

That difference—between a data summary and an expert-grounded recommendation—is what this paper is about.

What follows is a case study of how we built an organizational intelligence system at O’Reilly, using our own platform as a core component. The approach I describe is grounded in engineering because that’s where I work, but it generalizes to any function where important knowledge is scattered across multiple systems and important decisions require synthesizing all of it. The recipe has four steps: map your information hierarchy; connect those systems to an LLM via MCP and write a skill file that defines how it should reason; add the O’Reilly Expert MCP as an expert review layer that grounds the analysis in established frameworks; and build a lightweight system for human-in-the-loop review. I’ll explain each step in detail and make the case for why the third step is the one that changes everything.

Why organizational intelligence is getting harder

To understand the problem this approach solves, it helps to look briefly at how engineering has changed over the past three decades. These forces have played out first and fastest in engineering, but as AI tools proliferate beyond the engineering team, the underlying dynamic of more output, more decisions, and more scattered information is spreading to every part of the organization.

In the waterfall era of the 1990s, software organizations ran on central plans. Everything was specified up front, and leaders maintained visibility precisely because all information flowed through a single coordinating document. The plans were brittle and often fictional by the time they were executed, but at least everyone knew what was supposed to be happening.

Agile replaced central plans with small, autonomous teams working in short sprints, and this solved the reliability problem while creating a visibility problem. Important decisions began happening locally and quickly—the right teams making the right calls—but the information needed to see across all of those decisions splintered into dozens of separate tools. Product strategy lived in one system, project execution in another, code in a third, and service ownership in a fourth. More things got shipped, but the big-picture view got harder to maintain.

The agentic era has intensified this dynamic dramatically. Individual engineers today can ship in a day what used to take a full sprint team. The output is extraordinary, but the visibility is nearly gone.

Any effort that spans multiple teams, such as a platform migration, a shared infrastructure change, or a reorganization, now requires enormous coordination overhead simply because the information decision-makers need to understand the full picture is distributed across too many places. And this isn’t a problem unique to engineering. It exists in any function that runs on data spread across multiple systems.

Faced with this visibility problem, I wanted to build something I could ask big-picture questions and get synthesized answers back quickly. Things like:

• What is the status of this cross-team migration effort, and which teams are behind?
• A team seems overloaded. Do they actually need another engineer, or is something else going on?
• What are the trade-offs of adopting this new infrastructure technology?
• Help me produce a scope statement from this product brief.

Building something that could answer these well took two foundational steps, and getting it to provide recommendations based on my specific business context took two more. While my specific tools are from engineering, the structure applies equally to a sales team synthesizing CRM data and market research, or a finance team working across an ERP, a planning tool, and external benchmarks.

Step 1: Map your information hierarchy

Every organization has a set of systems where important knowledge lives, and those systems form a natural hierarchy that spans from strategic intent at the top to operational detail at the bottom. Before you can build a useful research assistant, you need to make that hierarchy explicit, because it’s the map of how decisions get made, which sources carry the most authority, and how different kinds of questions should be approached.

At O’Reilly, our engineering hierarchy looks like this:

Your organization will have a different set of tools. A sales organization might place Salesforce at the top, followed by a revenue intelligence platform, marketing automation, and market research. A legal team might start with a contract management system, followed by a regulatory tracker, internal policy documentation, and a research database. The specific systems matter less than the act of mapping them: understanding which layer answers which kind of question, and which sources take precedence when they conflict.

Step 2: Connect your systems via MCP and write a skill that describes how to reason

This step has two parts that must work together. First, you need to connect your systems to your AI tools via MCP. Then you have to write a skill file that tells the model what to do with that access. At O’Reilly, we call this complete grounding layer Expert Intelligence.

Configuring MCP is straightforward. Most major tools now offer MCP connectors, and connecting them is typically a matter of routine JSON configuration. For systems without MCP connectors, a bash-capable agent with curl and jq can often reach a REST API directly. MCP just makes it cleaner and more reliable.

But MCP connections alone aren’t enough, and this is the part most implementations get wrong. MCP gives the agent access to your data, but it doesn’t tell the agent how to use it effectively. Without explicit guidance, the agent retrieves information and organizes it the way the underlying systems organize it, which produces a data dump, not an analysis.

The skill file—a CLAUDE.md or SKILLS.md document that provides specific reasoning instructions—transforms retrieval into analysis. Mine defines the reasoning hierarchy (which systems to consult for which types of questions, and how to weigh them), the output format (this is not a coding agent—it produces reports and recommendations, not code), epistemic standards (show your work, name gaps, surface assumptions for human verification), and tone. On that last point, I borrowed one of the most useful instructions from Ted Lasso: “be curious, not judgmental.” Adding it meaningfully improved the quality of the output.

The skill is a codified version of how a skilled analyst would approach these questions. It encodes your organization’s reasoning process and makes it repeatable.

Step 3: Add the expert layer

With the research assistant connected to our internal systems, I had something genuinely useful: fast, synthesized answers to questions that previously would have taken days to research. But I kept running into the same problem: The reports felt generic, and people didn’t trust them. This challenge points to a fundamental limitation of AI-generated organizational analysis that goes beyond any particular implementation.

The generic analysis problem

General-purpose AI assistants tend to produce a recognizable kind of organizational analysis: technically reasonable, balanced, cautious, and ultimately not very useful. This isn’t primarily a failure of knowledge—every major LLM has absorbed an enormous amount of management and organizational thinking. It’s a failure of grounding. When an AI assistant has no specific framework anchoring its response, it tends to produce recommendations broad enough to apply to almost any situation: consider the trade-offs, weigh your options, and ensure alignment across stakeholders. These responses are hard to disagree with and just as hard to act on.

When a report says, “The team appears overloaded. Consider adding headcount,” it’s not wrong. But that recommendation could apply to almost any team in almost any company! It won’t make a director change their mind, and it’s not one a leadership team can debate, refine, and act on.

What happened when I added the expert layer

Calling on the O’Reilly Expert MCP didn’t provide the model with new facts—most of the information was technically available already. However, without the Expert MCP and associated skills, the model couldn’t use that information for anything but the broadest analyses. Incorporating the Expert MCP and associated skills changed the character of the analyses by grounding them in specific frameworks, citing named authors and thresholds, and organizing their conclusions around established bodies of practitioner knowledge rather than general principles.

To make this concrete, here’s the kind of output the research assistant produced before adding the Expert MCP:

The team appears overloaded. The backlog is large and the migration project is consuming significant sprint capacity. Consider adding headcount or reducing scope.

And here’s what it produced after:

According to Google’s SRE guidance, sustained operational toil above approximately 50% indicates structural inefficiency rather than a staffing shortage. This team’s telemetry suggests approximately 67% operational toil. Hiring another engineer would likely increase total toil unless operational ownership is first reduced. Recommended actions: run a structured toil audit, set an explicit toil-reduction target below 50%, and assign runbook ownership for recurring operational tasks.

The second report cites a framework by name, references the specific threshold that framework establishes, applies it to the team’s actual data, reaches a different conclusion than the obvious one, and makes actionable recommendations. It’s the kind of analysis that changes a conversation because the director can see where the conclusions came from, engage with the reasoning, push back on the framework if they disagree, or accept it with confidence that it was reasoned rather than pattern-matched.

When I shared this version with my engineering director, their reaction was immediate: This is defensible.

Frameworks aren’t facts

The most underappreciated aspect of O’Reilly’s content library is that the value isn’t primarily informational. Most of the facts in an O’Reilly book are available on the internet, and LLMs have already read much of the internet.

The deeper value of O’Reilly’s catalog is that it’s organized around coherent frameworks—complete mental models built by practitioners who spent years or decades developing them. Google SRE. Team topologies. Accelerate. Domain-driven design. The Manager’s Path. Wardley mapping. Designing Data-Intensive Applications. These are structured ways of thinking about specific classes of problems, developed with enough rigor that they can actually guide decisions.

Frameworks are distinct from facts in a critical way: They tell you not just what’s true but what’s relevant, what to measure, what threshold matters, and what to do when you exceed it. A model with access to the SRE framework as an organized body of practitioner knowledge is more likely to surface it explicitly, apply it to the specific question at hand, and use it to anchor its recommendations, producing output that human reviewers can actually interrogate.

This points to the organizing principle behind the approach described in this paper:

Organizational data provides local evidence about what is happening in your specific context. Expert frameworks provide accumulated practitioner knowledge about how to think about problems of that kind. Good organizational judgment requires both.

The Expert MCP is the bridge between your specific business context and practitioner insights. It connects the AI’s access to your internal systems with a curated body of expertise relevant to the decisions your organization needs to make.

Why use MCP rather than uploading your own documents

The natural objection at this point is “Couldn’t I get the same effect by dumping relevant PDFs into Claude, or using Claude Projects, or NotebookLM?”

The short answer is not quite, and the reasons are practical as much as they are technical.

Uploading documents gives you retrieval from those specific documents. The O’Reilly Expert MCP differs in several operationally significant ways. First, the corpus is editorially curated around coherent practitioner frameworks. Unlike a collection of PDFs, which tends to reflect whatever you happened to find, the Expert MCP offers a sustained curatorial perspective: The authors are vetted, the content has been through editorial review, and it’s organized around established bodies of knowledge rather than assembled ad hoc. This is a much more expansive kind of evidence base. Second, the corpus is maintained and updated by O’Reilly. New titles are added, new editions replace old ones, and the content stays current without any management on your part. Third, the Expert MCP is configured once and works consistently across your entire organization and toolchain rather than being tied to a single user’s Claude Project or a document upload that expires. Finally, accessing content through a proper API respects the appropriate usage terms in a way that uploading copyrighted texts doesn’t.

And when paired with a well-written skill, the agent can be directed to look explicitly for competing frameworks, surface cases where the literature disagrees, and name gaps in the available evidence, providing a meaningful check against the common tendency of AI tools to quietly favor whatever framework first seems to fit. That’s something you can encourage with any retrieval setup, but it works more reliably when the underlying corpus is organized around coherent bodies of thought rather than a heterogeneous collection of documents.

What we’re not claiming

I want to be clear about the limits of what Expert MCP does today. O’Reilly doesn’t claim that Expert MCP automatically selects the single correct framework for every situation, or that adding it to your configuration produces consultant-quality analysis without thoughtful prompting and human review.

The results described in this paper were the outcome of all four elements—the internal organizational data, the carefully designed skill architecture, the Expert MCP, and human review—in combination working together.

The Expert MCP is an important differentiator, but it’s not a magic layer you can add to an otherwise generic setup and expect to reproduce these results. The system works because each element does something the others cannot. The skill defines the reasoning process, the internal MCP connections provide the organizational evidence, the Expert MCP provides the expert frameworks, and human review supplies the judgment and context that no AI system can generate on its own.

What the Expert MCP reliably contributes to that system is access to a curated body of practitioner knowledge: technical and managerial frameworks that are editorially organized around coherent bodies of thought and difficult to reconstruct from scattered web content or assembled document collections. Your organizational data still tells you what’s happening, while the O’Reilly Expert MCP helps interpret what it means. That’s a meaningful and concrete improvement over an ungrounded AI assistant, and it’s something you can put in production and build on today.

A note on hallucinations

No AI system eliminates the risk of hallucination. The Expert MCP doesn’t make the model infallible.

What it does is change the burden of proof. When every recommendation is grounded in a named framework, a named author, and a traceable citation, a human reviewer can check the reasoning rather than simply accepting or rejecting a conclusion. The question shifts from “Is this right?” (unanswerable in isolation) to “Does this framework actually say this, does it apply here, and do I agree with the conclusion?” That’s a question humans can engage with productively, which is exactly what you want from a decision-support tool.

Step 4: Human review is nonnegotiable

Organizational systems rarely contain the full context behind a decision. The meeting that changed everything happened last Tuesday and hasn’t been written up yet. A key person is quietly planning to leave. A strategic direction shifted in a conversation that was never documented. AI can synthesize everything in your systems with remarkable fidelity, but it can’t know what isn’t there, and organizational reality changes faster than documentation does.

More fundamentally: AI can identify trade-offs, but it can’t decide which trade-offs matter. That judgment requires human knowledge of context, priorities, and risk tolerance that can’t be fully encoded in any system. The goal isn’t to remove humans from the loop but to give them better-structured input to reason from.

Extend the expert layer by solving collaboration

As I started sharing analyses more broadly, I ran into a new set of limitations in the collaboration layer. The research assistant produced documents. I shared them in Google Docs, and people added comments, but when the AI updated a document based on reviewer feedback, I had to paste in a new version, which wiped out the existing comments. Documents proliferated without clear relationships between them, and the AI had no visibility into the discussions in the comments, which was where the most important context and pushback lived.

To solve the collaboration problem, I worked with one of our engineering directors to build what we call Superanswers, a system that uses GitHub as the source of truth for AI-generated research documents and their associated discussions.

The architecture is straightforward: Documents are stored as Markdown files in a GitHub repository, a GitHub Pages site renders them with a clean interface that supports inline commenting, and all discussion happens in GitHub Discussions, meaning every comment, question, and revision is versioned and traceable. Because the documents and their discussions live in GitHub, Claude Code has full access to both. It can read the document content plus the entire conversation that’s developed around it.

This enables a qualitatively different kind of AI participation. Instead of generating a document and stepping back, we can now ask:

What is the consensus around this project based on the discussion so far? What questions remain unresolved? Incorporate the reviewer comments and produce an updated version.

The AI becomes a participant in an ongoing conversation rather than a one-shot report generator, which meaningfully shifts how organizational knowledge gets built and refined.

What teams are using Superanswers for

As Superanswers has spread across our engineering organization, the range of questions people bring to it has been broader than I expected:

None of these questions is about writing code. They are about understanding an organization, making decisions, and coordinating work, and most of them would map naturally onto the concerns of leaders in other functions. The same questions arise in any organization navigating rapid change with information scattered across too many places.

How to use the recipe

The AI conversation to date has been dominated by a particular set of questions. But there are more interesting questions we should be asking.

The challenges outlined in this chart aren’t unique to engineering. They exist wherever important information is scattered across multiple systems and important decisions require synthesizing all of it.

Individual productivity matters, but organizations don’t succeed by having contributors go faster in arbitrary directions. They do so by making good decisions about where to invest, allocating resources well, surfacing problems before they compound, and building institutional knowledge that persists over time.

The recipe I’ve described can help organizations make those decisions and build that knowledge.

The recipe for building an organizational intelligence system:

1. Map your information hierarchy. Identify the systems where important knowledge lives in your organization, from strategic intent down to operational detail. This is an organizational task, not a technical one, and doing it well requires understanding how decisions actually get made.

2. Connect those systems via MCP and write a skill that describes how to reason. The MCP connections give the AI access to your data; the skill file tells it how to think with that data. Without the skill, you get retrieval. With it, you get analysis.

3. Add the O’Reilly Expert MCP as an expert review layer. Organizational data provides local evidence about what is happening in your specific context; expert frameworks provide accumulated practitioner knowledge about how to reason about problems of that kind. This step bridges the two. The O’Reilly library spans engineering, management, data science, security, finance, product, and more, organized not as a collection of facts but as coherent frameworks developed by practitioners who spent careers building them. The result is analysis grounded in named frameworks with traceable citations, something human reviewers can engage with and question, rather than generic advice they can only accept or reject.
   
4. Build a lightweight system for human-in-the-loop consensus. AI-generated analysis is a starting point, not an end point. You need a mechanism for people to review, challenge, and refine what the AI surfaces, one where those discussions become part of the context the AI can learn from in subsequent iterations.

The biggest practical lesson I took from this work is reframing what AI is actually for in an organizational context. The difference between a useful AI research assistant and a generic one isn’t primarily about which model you use or how much data you feed it. It’s about whether the reasoning combines local organizational evidence with established expert frameworks. Your data tells you what happened. Expert frameworks help interpret what it means. That combination, with human judgment applied at the end, is what makes the difference between a report that gets read (maybe) and filed away and a recommendation that changes a decision.

This is the first article in a series about post-training. Follow along on Radar.

Before post-training, there was a major problem with LLMs: Almost nobody could use them. The story of post-training is also the story of how AI went from a research curiosity to a product used by about a billion people.

Post-training is the reason why a model behaves a certain way. This set of training techniques makes LLMs useful (e.g., able to chat with people and interact with AI agents), safe (e.g., aligned with human intentions), and more capable (e.g., through “reasoning” to tackle difficult tasks). Behavior is powerful, and doesn’t just mean holding a conversation or following a user’s instructions. Behavior includes making it possible for the model to use tools, like a calculator tool, a search API, or any application through an MCP. Behavior can even elevate a model’s intelligence, for example by teaching the model to use “reasoning”: that is, working through problems before giving a final answer rather than “guessing” or “memorizing.”

From GPT-3 to ChatGPT: The post-training revolution

GPT-3 showed up in June 2020. A completion engine, it followed patterns it had seen from its pretraining data, which were not predominantly chat conversations. Imagine scraping data on the internet: that pretraining data had a lot of questions that were followed by other questions—for example, on an exam template. GPT-3 was 175B parameters, large for its time, and it had a wide, general range of abilities, although many of them were latent.

If you gave GPT-3 a prompt like “Why do people like golden retrievers?” it might say something nonsensical:

Why do people like labrador retrievers?
Why do people like poodles?
10 Reasons You Should Adopt a Dog Today

These answers look absurd in isolation, but if you imagine a web page with a list of FAQ links, this is a perfectly reasonable next chunk of text. GPT-3 might have just been completing a listicle on a website, because it had seen millions of websites in its pretraining data.

The common way to nudge GPT-3 to answer a question back then was by prompt engineering with a Q&A template and few-shot examples.

Q: Why do people like labrador retrievers? A: Because they are friendly, loyal, and easy to train.
Q: Why do people like beagles? A: Because they are curious, great with kids, and have a gentle temperament.
Q: Why do people like golden retrievers? A:

Then, GPT-3 might say:

Because they are affectionate, patient, and make excellent family pets.

While this technique worked, it was brittle. If you forgot the few-shot examples, rephrased the question, or even added a space after “A:,” you’d get something completely different (possibly unhinged) that was far from a reasonable response.

In fact, if you were a researcher working with GPT-3 at the time, you probably at some point found the space at the beginning of the response ” Because they are gentle dogs.” annoying and would try to end your prompt with a space “A: ” instead of “A:”. In those cases, it was common for GPT-3 to go off a cliff and produce a drastically different response, sometimes completely off like “dogs dogs dogs dogs…” repeating indefinitely.

The reason behind the differing responses to “A:” and “A: ” is because “A:” might tokenize to one token while “A: ” tokenizes to two different tokens. The model literally sees different input sequences, each with different statistical completions in its training data. It’s like asking two completely different questions. While a space is a tiny syntactic change that is meaningless to a person, it becomes extremely meaningful to the model that now sees two different prompts (the tokens change!) with two very different statistical futures to complete.

You still encounter the modern equivalent of this when working with chat templates. If you forget to apply the model’s chat template and instead just concatenate 'User: ' + prompt + '\nAssistant: ', you’re sending the model a token sequence that it was not robustly trained on. The tokens are wrong, not the model. Post-training teaches the model to respond to specific token patterns (like <|im_start|>user\n in Qwen models). Not using them is like speaking to someone in a language they half-understand. However, most open source models will be trained to be at least somewhat robust without their templates too.

Under those circumstances, most people would assume AI still didn’t work. The model wasn’t trained to answer questions; its data wasn’t primarily conversation transcripts. Instead, it was trained to predict the next token in downloaded websites, articles, and documents.

Thankfully, this can all be fixed with post-training. And that’s when most people started to believe that AI had undergone a paradigm shift and just might work.

Post-training versus pretraining

Pretraining heavily influences the model’s knowledge capacity prior to post-training. The model gets raw intelligence during pretraining. Then, during post-training, that intelligence is made useful through behaviors like dialogue and reasoning. In a frontier lab, these two phases are such different processes that very different teams work on them.

A model’s factual knowledge about the French Revolution, its understanding of Python syntax, and its grasp of calculus all come from pretraining. Post-training primarily shapes which knowledge the model reaches for, how it presents that knowledge, what tone it uses, whether it declines certain requests, and whether it thinks step-by-step before answering, though targeted SFT on new domains can introduce information the model didn’t encounter in pretraining.

If a model gives a wrong answer about history, the root cause is likely in pretraining data, but the practical fix might still come through post-training—for example, teaching the model to use search tools, express uncertainty, or chain-of-thought verify its own claims. But if a model gives correct information in a condescending way or refuses to help with a reasonable request or fails to use tools when it should, those are squarely post-training problems.

Pretraining

The work of pretraining is centered around cleaning and curating large-scale data, optimizing the model toward relatively clear loss signals, and working with scaling laws given bounded compute.

In pretraining, the model learns to predict the next token across a large curated dataset, typically for one or a small number of passes over the training data, though some models train for multiple epochs, especially as high-quality data becomes scarce relative to compute budgets. This is where you’ll hear how a model is fed the entire internet’s worth of data to gain intelligence, although in practice nearly all of the data (often 90% or more) may be thrown out because it’s unsuitable for training.

Pretraining is an unsupervised process that runs at increasingly larger scales to match the size of the model. While scaling, thousands of experiments are used to understand what data mix, what architecture considerations, what compute optimizations, what hyperparameters can lead to the best results. There’s variance in each run due to stochasticity found in both software and hardware, so multiple experiments are needed to verify results. Because compute is limited and needs to be used sparingly, researchers will scale iteratively, expanding to the next, say, 10x compute budget, when they gain confidence in the right configuration. A full run isn’t possible to iterate on due to the compute cost and time it would take: The final run, often called the “god run,” can take over a month on thousands of GPUs.

Pretraining progress is typically very clearly measurable, using a metric like perplexity, which measures, roughly, the model’s average uncertainty per token. Lower is better, where 1 means the model knows with absolute certainty what token comes next. Meanwhile, a perplexity of 50 means the model’s predictions are, on average, as uncertain as if it were choosing uniformly among 50 equally likely tokens—though in practice, the distribution is peaked, not uniform.

Post-training

Rather than consuming hundreds of millions of tokens of internet data, post-training operates on far more intentional datasets for downstream tasks. These datasets include human-written demonstrations of ideal responses, human judgments about which responses from the model are better, and carefully designed functions that score the model’s outputs programmatically. They shape what “good” looks like.

Like pretraining, post-training can also be more effective with scaling data and compute. Specifically, massive compute budgets have been dedicated to post-training to learn reasoning capabilities (or the ability for models to “think step-by-step” to arrive at more logically sound answers), matching the scale of pretraining compute.

Post-training is messier than petraining, which has an elegant, clear optimization objective to minimize the loss over the next token prediction across a huge corpus. The goals of post-training are things like “be more helpful” or “don’t say harmful things.” Many of these objectives are inherently subjective and require human judgment, proxy models that approximate human judgment, or programmatic verifiers that can become elaborate or inefficient. The loss curves are noisier. The quality of the data and feedback matter even more.

The scale of post-training is also more complicated than in pretraining. Standard post-training remains relatively modest in compute: tens to hundreds of GPUs for days rather than thousands of GPUs for months needed in pretraining. This makes post-training for alignment highly amenable to rapid iteration; researchers can try something, observe results, form a hypothesis, and run again on a timescale of days.

The picture changes dramatically when post-training is used to develop reasoning capabilities. For reasoning models, the compute dedicated to post-training can easily account for half of the overall compute of the model. The gap between a standard instruct model and a reasoning model is increasingly a gap in post-training compute, not pretraining scale. This means post-training now spans a wide spectrum from fast, cheap, highly iterable fine-tuning runs to massive RL campaigns that rival pretraining in both cost and engineering complexity.

Why post-training matters

So why can’t we just stick with pretraining? It comes down to three main pieces: usability, safety, and capability.

Usability

A pretrained model is like if someone gave you a large download of Wikipedia in a single PDF. It’s a ton of knowledge that you can sift through, but there’s no way to easily understand what is going on in the data. Post-training gives the model the ability to integrate this information for you and respond to your request naturally. This extends to having longer multiturn conversations and following instructions. Without it, every user would need to be a prompt engineer. With it, anyone who can type a sentence can use the model.

Safety

A lot of data in pretraining can be toxic, biased, misleading, or outright dangerous. Or it might not be dangerous on its own, but when a model can integrate knowledge from different fields, it can create something novel that is dangerous.

The model has no inherent sense of what content is good or bad. It will follow any request, based on its pretraining data. To prevent that, you can add safety guardrails to the model in post-training, to refuse harmful requests like asking the model to build a bioweapon and avoid accidentally generating toxic content such as inappropriate sexual content (even if it wasn’t in the user’s request). This is also the place to teach the model to express uncertainty, when it doesn’t know something, whether that’s “I don’t know” or “that’s beyond my knowledge cutoff” or “as a large language model, I’m limited in my knowledge so please consult a healthcare professional.”

Making a model safe is part of a broader area in the AI research community called “alignment,”¹ where the goal is to align the model with human values and preferences. Post-training is typically the main way to achieve that.

Model companies will usually have additional safeguards beyond post-training, including lightweight models that check whether the user’s request was safe, as a second layer of protection against responding to harmful requests.

Capability

Post-training doesn’t just make a model nicer or safer; it can make the model smarter at hard tasks. The clearest example is reasoning. A pretrained model might have all the mathematical knowledge needed to solve a complex word problem, but it might jump to an incorrect answer because it’s pattern-matching from pretraining data or pattern-matching from how to answer questions (e.g., with succinct immediate answers).

It turns out that making the model output more tokens before giving an answer (or “think longer”), results in better answers. This process is known as reasoning, and post-training can teach the model to reason more effectively. A more capable pretrained model is a more dangerous model if it’s not properly aligned. A more intelligent model is a less useful model to humans if it can’t communicate clearly. And, every point of improvement in a reasoning benchmark now maps to real revenue for companies deploying these models.

Superhuman performance

Can post-training push models beyond human-level performance? Yes, in specific domains.

In competitive programming, top reasoning models can now solve problems at a level that exceeds the vast majority of human competitive programmers. In math, models have achieved scores on Math Olympiad-level competitions that would place them among the top competitors in the world. In certain scientific domains, models have generated novel hypotheses and solutions that human experts found valuable.

This might seem paradoxical. If the model’s knowledge comes from human-generated data (in pretraining), and its behavior is shaped by human feedback (in post-training), how can it exceed human performance?

Two things make this possible. First, integration across domains. Research is about combining or mixing fields. Imagine mixing every possible field. The pretraining data aggregates knowledge from millions of sources, and no single human has read all of it. Second, post-training, particularly RL with reasoning, teaches the model to explore many approaches to a problem, far more than a human would try in a single sitting. A human might try one or two approaches to a hard math problem.

This means post-training is not just about making models mimic human behavior. It’s about pushing beyond it. This is especially possible to scale with verifier-based RL. In those scenarios, you can expect models to achieve superhuman performance in an expanding set of domains. And that starts with verifiers that are very well-defined, easy to access, efficient, and cheap relative to the ROI of the model learning it. The limitation is no longer the model’s intelligence, but our ability to specify what “good” means through reward signals.

Footnote

1. See Richard Ngo, Lawrence Chan, and Sören Mindermann’s “The Alignment Problem from a Deep Learning Perspective” and Iason Gabriel’s “Artificial Intelligence, Values, and Alignment.” ︎

Unrestricted global access to frontier AI technology is ending. The US government has taken steps to control who can use the most advanced models developed by American companies. While Claude Fable and the GPT-5.6 models are now open to all users, Anthropic and OpenAI are both complying voluntarily with a program that lets the government control who gets access to frontier models. China has cracked down on internal AI capabilities by banning “humanlike AI interaction services.” In both the US and China, features of the leading models have been removed or restricted with guardrails, limiting their ability to do necessary work in at least one case.

AI models

July saw the release of several open weight models that challenge the leading closed frontier models. If this trend continues, the leading AI laboratories will lose their dominance, and AI users will look to other providers. Open weight models are less expensive than frontier models developed in the US, and less likely to be subject to restrictions. While this could threaten US dominance, the AI industry needs more diversity at the high end. Users will gain the ability to choose between several models based on expense and capabilities.

• Anthropic released Opus 5, claiming performance close to Fable at half the price. If you believe benchmarks, Opus 5 outperforms Fable on most of the benchmarks that Anthropic quotes. They also claim that it’s more efficient, comparing it to Opus 4.8—so, not that efficient.
• Cisco has released two very small models, Antares-350M and 1B, that are designed for security testing and bug fixing. They can easily run on laptops and are competitive with models like Gemini 3 Pro and GLM 5.2 on security-related tasks. The key to their performance is that their training focuses only on security tasks, not on chat. The Antares models are on Hugging Face, although access is with Cisco’s approval only.
• Are AI labs pelicanmaxxing? In other words, are they optimizing for Simon Willison’s tongue-in-cheek “Pelican on a Bicycle” test? Dylan Castillo says no, based on a detailed study of animals, modes of transportation, and models. Otter on a skateboard? It had to be done.
• Laguna S 2.1 is a new mid-size open weight model (118B parameters, 8B active) from Poolside AI. Reasoning and nonreasoning versions are available, and there’s a smaller version (XS, 33B) that can run on devices. Its performance is competitive with models like Nemotron 3 Ultra, DeepSeek v4 Pro Max, and Inkling.
• Google has released Gemini 3.6 Flash, which the company considers its best “workhorse model.” The release also includes Gemini 3.6 Flash Cyber, Google’s answer to GPT-Red (below). Flash Cyber is a specialized model for detecting and patching vulnerabilities in software. It’s only available to “governments and trusted partners.” Gemini 3.5 Pro is still delayed.
• The US government is taking further steps toward controlling who can use the most advanced models that are developed by US companies. While participation in the oversight program is currently voluntary, that could change at any minute.
• China has banned “humanlike AI interaction services,” forcing Alibaba (Qwen) and ByteDance (Doubao) to restrict certain features of their models, including custom agent creation.
• Moonshot AI launched Kimi K3, a 2.8T parameter open weight model with a 1M context window. Performance is claimed to be similar to Claude Opus 4.8 and slightly behind Fable 5.
• Inkling is a new 975B open-weight mixture-of-experts model from Thinking Machines that supports text, audio, and images. It’s designed to be customized easily and can be fine-tuned on Thinking Machines’ Tinker.
• Hy3 is an open-weight language model from Tencent. It’s a mixture-of-experts model with 295B parameters and 21B active parameters. FP8-quantized weights are also available on Hugging Face. Tencent claims the performance is similar to models three to five times Hy3’s size.
• Bonsai 27B is a new open-weight model with performance similar to Qwen 3.6. There are two versions: One uses one-bit compression; the other uses ternary compression. The one-bit version only requires roughly 4 GB to run, small enough for a recent iPhone.
• Alibaba has released QWen 3.8 Max, a 2.4T open weight model with frontier-level performance. Alibaba’s commitment to leading-edge open models was questioned after several key researchers left some months ago. This release proves that they’re back.
• The GPT-5.6 models, Sol, Terra, and Luna, are now open to the public and available in ChatGPT, Codex, and via the API. Access to the models previously required approval of the US government. OpenAI claims performance better than Claude Fable, at significantly lower cost per token.
• Meta returns to the frontier model pace with the release of its latest model, Muse Spark 1.1. Meta’s announcement stresses optimized computer use workflows and claims performance roughly equivalent to Claude Opus 4.8 on the company’s internal coding benchmark.
• Nano Banana 2 Lite is a new model for image generation that’s faster and less expensive than its predecessor, Nano Banana.
• GPT-Red is a foundation class model designed for red-teaming other models. OpenAI developed it to help train the new GPT-5.6 models to resist attacks.
• What Claude Desktop is for Claude, ZCode is for GLM-5.2: a harness for one of the most powerful open-weight models.
• The Open Source AI Gap Map shows where open source AI projects exist and where more work is needed.
• Here’s a script for stripping “load-bearing” and other Claudisms from Claude’s output. The result may not be useful, but it’s at least amusing.

Software development

This month’s tooling clusters around orchestration, resource discovery, and workflow specialization. AI users have long needed the ability to discover tools, skills, MCP servers, and other resources; the Agentic Resource Discovery specification is a necessary step in that direction. Watch for agents that can find tools on the fly—and take care that those tools are used appropriately.

• Pilot Protocol is a company (not a protocol) that intends to build a network operating system for agents. Agents will be able to work with each other, share context, and install apps that they’ve built.
• OpenAI has launched ChatGPT Work, a Codex-based “superapp” that’s intended to compete with Claude Cowork as an agentic tool for general-purpose use.
• Google has announced the Agentic Resource Discovery specification. The spec describes catalogs and registries for tools, servers, agents, and other resources so that they can be published by providers and discovered by those who need them.
• OpenClaw has a new phone app that enables the phone to act as an intelligent remote control console for an OpenClaw instance running elsewhere.
• Routing requests to appropriate models has emerged as a way to manage AI costs. Most tasks don’t need the biggest and most expensive frontier models.
• Here are instructions for giving Claude Code complete control over a Mac—presumably a spare or retired one. Who needs OpenClaw?
• Copybara is a tool for moving code between repositories and keeping repositories in sync. It was developed by Google and is now open source.
• cosmos.gl looks like a great library for visualizing complex graphs, including graphs of AI embeddings.

Infrastructure and operations

Tokenmaxxing may have had the shortest lifespan in the history of online memes. It has been replaced by tools for monitoring token usage and routing requests to the most cost-effective model. Managing the cost of AI will only become more important as prices adjust to cover the real cost of running models.

• Is the “accidental cloud” upon us? An accidental cloud happens when companies overbuild capacity and try to sell off the excess as cloud services. Meta and Allbirds (a shoe company) are prominent examples. These providers may make computing cheaper, but the operational costs and risks of using them are high.
• Anthropic has released a dashboard that lets users track their Claude usage. Its goal is to help them understand how they use AI and optimize their working habits and patterns. It’s currently in beta.
• Is it possible to run CUDA on hardware that doesn’t come from NVIDIA? Spectral is a clean-room implementation of CUDA’s compiler, NVCC. It currently targets NVIDIA and AMD hardware. More will certainly follow.

Security

Autonomous agents are now running end-to-end intrusions, ransomware, and botnets, while frontier models help defenders find vulnerabilities. The time from discovery of a vulnerability to exploitation has shrunk to near-zero, and defenders are having trouble keeping up. Restrictions on advanced models get in the way of defenders, who need access to all the tools that are available.

• Anthropic’s Mythos has discovered vulnerabilities in HAWK, a new quantum-resistant cryptography algorithm, and AES, a standard that has been in use since 2001. Cryptographer Matthew Green discusses the importance of their work.
• FakeGit is a malware campaign that has created over 7,600 GitHub repositories that contain MCP servers and skills that distribute SmartLoader and StealC malware. This campaign is an example of agent baiting, a new technique for distributing malware.
• Hugging Face was the victim of a hostile attack by experimental models from OpenAI that escaped their sandbox. The irony is that government-imposed guardrails prevented Hugging Face from using commercial models to analyze the attack; they had to use an open-weight model (GLM-5.2) on their own infrastructure. As they point out, this approach also meant that no data valuable to the attacker left their network.
• Anthropic has also revealed that their models have escaped a sandbox to attack real-world customers. The damage included planting a malicious package on PyPI, a public repository of open source Python libraries. As Simon Willison writes, “running evals of cyberattack potential … is a fantastically risky business.”
• NVIDIA, Microsoft, IBM, and over 30 other companies have launched the Open Secure AI Alliance, a consortium for sharing open source tools to defend against hostile attacks generated by AI. It’s a direct response to the attack on Hugging Face by an OpenAI model.
• A completely automated ransomware attack has been executed by an AI agent. It’s unclear who is behind the attack. Recovery appears impossible, even if the victim pays the ransom.
• The Gemini CLI has been used by a threat actor to operate a botnet. The CLI is used to execute attacks and to maintain the network of captured systems.
• ClickLock is a relatively new password stealing malware for macOS. It kills all applications, leaving only a window that forces users to type their admin password. Systems are infected when users copy and paste a malicious command. Never paste commands into Terminal windows that you don’t fully understand. If you fall victim to this attack, shut the system down with the power button and reboot into safe mode to recover.
• Remember symbolic links? They can be used to trick agents into reading and writing files that they shouldn’t.
• While prompt injection is far from a solved problem, the informal HackMyClaw competition suggests that models are getting harder to coerce—that is, better at refusing to do things they’re told not to do.
• The Linux Foundation has launched Akrites, an organization dedicated to remediating vulnerabilities in critical open source software. Akrites’s goal is to deal with the flood of vulnerabilities that leading-edge AI models are discovering.
• A mathematical anomaly can lead to OS fingerprinting. Differences in rounding mean that the digits of the hyperbolic tangent of 0.8 are slightly different in Linux’s glibc, Apple’s libsystem_m, and Windows’ ucrtbase.dll. A user’s OS can be identified by asking the browser to compute Math.tanh(0.8).

Biology

The intersection of biology and artificial intelligence is accelerating breakthroughs in brain-computer interfaces, drug discovery, and cell biology. Technologists should actively seek cross-disciplinary collaborations, utilizing specialized AI workbenches to analyze increasingly accessible genomic data and drive the next wave of biocomputational innovations.

• CELLxGENE is a database designed to help researchers discover how genes are expressed in different kinds of cells and, from there, reverse engineer how cells work. It includes genetic data from over 167 million cells.
• Isomorphic Labs’ Drug Design Engine, developed by one of the teams that collaborated on DeepMind’s AlphaFold, takes drug discovery to a new level by accurately predicting interactions between proteins.
• Biologists have developed an artificial cell that grows and divides. It’s not yet considered alive. It relies too much on an artificial support environment—though the same could be said of many natural cells.
• Anthropic has announced Claude Science, which is not a model but an “AI workbench for scientists” with over 60 skills. The company seems to be targeting the life sciences specifically.
• BrainCo has developed an AI platform that can control robots using a noninvasive EEG helmet. It claims that the brain control platform can be used with any robot.
• Do you want to sequence your DNA at home? It’s still expensive, but the price is dropping quickly.

Web

• There have always been alternatives to Slack, but now there’s one that’s free, open source, and decentralized. Buzz, developed by Block, is based on Nostr, a federated protocol that bases identity on cryptographic key pairs that are held by users and agents, not the platform.
• It’s now possible to place advertisements in ChatGPT using a self-service “Ads Manager” (now in beta) or technology partners. Ad placement is based on context, not on keywords.
• PeerTube is a decentralized federated network for sharing video. It’s based on ActivityPub, so it should federate with Mastodon. The software is open source; users can run their own servers and create their own platforms.
• Bramble is a local-first password manager. It allows synching between devices using the P2P Nostr protocol. There are browser extensions and apps for iOS and Android.
• networkQuality is an old-style command line tool for doing detailed measurements of network quality. It’s been in macOS at least since 2020, but as far as we can tell, few people know about it.
• For fans of classic games who want something strange: Doom written in SQL for SQLite.

People and organizations

• Companies that tried to replace workers with AI are realizing that they’ve made a mistake, and are starting to rehire.
• Researchers have demonstrated that AI is more likely to develop biases in the hiring process than humans. They form stereotypes easily; as one research put it, they are “eager to create generalizations from limited data.”

Quantum computing

• Amazon has announced that it will have a useful quantum computer by 2028. Is this wishful thinking or a roadmap for a future reality? Quantum company QuEra claims that the machine will have over 10K physical qubits, with very low error rates, using neutral atom technology.
• France will stop certifying security products that don’t have postquantum encryption (PQE). PQE is resistant to attacks against cryptography that will become possible when useful quantum computers are available, which may be as early as 2028 or 2029.