Paul Hammant's blog

BTRON, OLE, OpenDoc and the Web: Why Dumb Won

2026-09-27T00:00:00+00:00

Note: By “dumb” I don’t mean badly designed. I mean something more specific: the architecture with the weakest internal semantics had the strongest boundaries.

Japan’s TRON project bubbled up again recently - an XDA piece by Adam Conway and an OSNews write-up by Thom Holwerda with a good comment thread. BTRON, the desktop branch, is the interesting one. It tried to replace “files owned by applications” with a document model: typed parts, handlers per part type, and persistent system-managed links between parts. A document holding writing, a table and a figure would invoke a handler for each, with each handler drawing inside the parent document’s window. Its filesystem was an arbitrary directed graph, not a tree. That was late 1980s (it only began in 1984).

Meanwhile ITRON, the embedded sibling, quietly became one of the most deployed operating systems in the world - in cameras, engine control units and phones - because it was small, deterministic and royalty-free. That’s the “dumb won” story in miniature, inside TRON itself. Maybe dumb-won is too pithy and the architecture with the weakest internal semantics had the strongest boundaries is more accurate.

Thom’s pet theory is that the application-first model won because it is better at wealth extraction. Commenter ssokolow disagreed: compound documents were a fine idea while documents mostly left the building as paper or PDF, but once the Internet arrived, making the recipient acquire every handler for every embedded part became prohibitive. Another commenter, metaning, remembered OpenDoc files sent to a service bureau, which then had to own the same OpenDoc components you did before they could RIP your job to an imagesetter.

I think ssokolow is closer, and I went back and forth with ChatGPT to put it in a table.

Four answers to “what comes after files + applications?”

Between roughly 1984 and 1997 the industry tried repeatedly to answer that question. Four answers stand out:

BTRON - hypermedia + compound documents at the OS level
OLE - compound documents retrofitted onto the conventional Windows desktop
OpenDoc - documents as federations of small, interchangeable parts (Apple/IBM et al)
The Web - hypermedia stripped down enough to become universal

Aspect	BTRON	Microsoft OLE	OpenDoc	Web
Peak innovation period	1985-1990	1990-1996	1992-1997	1989-1996
Core vision	Computer as a structured hypermedia/document environment	Desktop as compound documents assembled from application components	Documents assembled from independent interchangeable parts	A universal distributed hypertext information space
Fundamental unit	Typed document part	COM/OLE object	OpenDoc part	Resource / document
Applications are …	Handlers for part types	COM servers / containers	Part editors	Browser + servers + helper apps
Linking	System-managed typed links	OLE links, monikers	Part relationships	Simple one-way hyperlinks
Links survive move/rename?	Designed to	Potentially, via monikers	Within its framework	No - link rot
Requires a new OS worldview?	Largely yes	No - retrofit onto Windows	No - over existing OSes	No - runs above almost anything
Network-native?	Not BTRON’s killer property	No (DCOM later)	Not fundamentally	Absolutely
Ease of copying content from the canvas	High in principle, within BTRON tooling	High for ordinary representations; lower for preserving live-object semantics	High by design	High — browser provides common selection/copy for ordinary document content
Ease of pasting elsewhere, incl. lesser technologies	Medium - TAD interchange, but semantics lost leaving the ecosystem	High to medium - clipboard formats degrade, but rich objects need their server	Medium-high - depends on the receiver	Extremely high - HTML, text, image or just the URL
Navigation master control	Medium - navigation integral, but no universal user-owned stack	Low - belongs to each container/app	Low-medium - container coordinates parts	Extremely high - Back / Forward / Reload / Stop + location bar sit above the page
Historical outcome	Desktop branch stayed niche (ITRON thrived embedded)	Deeply embedded in Windows/Office	Terminated 1997	Ate the world

The dates are approximate, and mean when each was introducing its distinctive ideas rather than when it peaked commercially.

The first eight rows are the conventional comparison, and on those the Web looks primitive. It’s the dumbest of the four. The three rows I asked to be added just before “historical outcome” are where it wins, and I think they are the rows that matter.

Then also “high - browser supplies selection/copy for every page” re copy/paste wasn’t there from the outset of the web, had a peak, and has declined quite a bit in the last decade. Canvas-rendered applications, images of text, custom controls, DRM-ish interfaces, CSS selection suppression, etc. can defeat normal selection. And I have to say I really hate that defeating of one of the true genius aspects of the web.

Graceful degradation outward

The richer systems optimized for movement of content within themselves: from a BTRON part to another BTRON part, from an OLE server to an OLE container, from one OpenDoc part editor to another. That’s interoperability between equally sophisticated peers.

To be fair, BTRON did think about degradation inside the system: applications could skip TAD data types they didn’t support. But Conway’s article has a telling detail: if no handler exists for a part, or the handler fails to draw, the system strikes a diagonal line through the region where the content should have been. That’s the compound-document failure mode drawn literally - a box you can’t see into, which is exactly what ssokolow’s recipient without the right handlers gets.

The Web’s artifacts instead degrade gracefully when they leave:

CONTENT:
interactive page
  -> HTML / structured fragment
    -> formatted text + images
      -> plain text

REFERENCE:
interactive page
  -> URL

^ The Web has both graceful representation degradation and cheap referential escape

At each step you throw away semantics and still have something useful. The destination needn’t understand the source. An email client, a Slack message, a Word doc, a text file, or a sticky note on a monitor will each take some level of that. That’s precisely ssokolow’s point about handlers: the compound-document systems required the receiver to be as capable as the sender, and on the open Internet that’s rarely the case.

A better name for the row might be “graceful degradation when copied/pasted outward”. The question is how easily can information escape the system without the destination knowing anything about it?

This is the one I find most under-appreciated. OLE says the container/application controls your experience. OpenDoc says the compound document coordinates your parts. BTRON says the information environment manages your graph. The browser says:

<- Back   -> Forward   Reload   Stop   [ Location ]

… and the page doesn’t get a vote (fundamentally the page doesn’t own those controls). A terrible page and a brilliant page get the same navigation machinery, owned by the user, sitting outside the document. Back is a generic undo-like operation over navigation that requires no cooperation from the content author. The location bar gives Web resources - and well-designed application states - a name the user can copy independently of their contents.

Single-page apps eventually had to learn to cooperate with that model via the History API: if application state changes aren’t reflected in browser history, Back stops meaning what users expect.

Green-screen techs as represented via IBM’s “common user interface” definitions from the late 1980s, had an F12 for back-one-screen (not page) and F3 for back-out-completely (exit) didn’t make the jump to the web, even if F5 for refresh did. Well, less and less today.

What about Lotus Notes?

Notes was a love-it-or-hate-it thing, and it’s a fifth answer from the same window. Ray Ozzie’s Iris Associates started in late 1984 and Notes shipped in late 1989. The fundamental unit was the document - rich text with embedded objects and attachments - and an “application” was really just a database design (forms, views, some scripting) wrapped around documents.

Its links had one important BTRON-like property: they were based on persistent identities rather than merely filesystem paths. Doclinks keyed on a replica ID plus the document’s unique ID (UNID), so they could survive replication and server moves where the Web’s rot. And it had a strength none of the four above had: replication. Offline-first documents that synced between servers and laptops, years before anyone said “local-first”.

But on the three rows that matter, it looked like OLE and OpenDoc. Rich Notes content degraded unevenly outside the Notes ecosystem; preserving Notes-specific behavior required Notes-aware software, and navigation belonged to the Notes client - until Release 5 added browser-like Back/Forward and bookmarks, which was Notes imitating the browser. The “Domino” version (1996 or so) then served Notes databases over HTTP as HTML. Domino eventually gave Notes databases a Web escape route: HTTP/HTML could expose their contents without requiring the Notes client.

Ozzie kept pushing the replication idea after Notes - Groove (2000-2005) for peer-to-peer shared workspaces, then Live Mesh (2008) at Microsoft. It was the browser-based, URL-addressed takes on the same idea - Google Docs, then Microsoft’s own cloud Office - that won.

Notes’ data model did survive, though: Damien Katz described CouchDB (JSON docs, map/reduce views, replication) as “Lotus Notes built from the ground up for the web,” and PouchDB (~2012-13 (?)) put the offline-first replicating client in the browser. The easy built-in UI didn’t survive - as I noted in Lotus Notes vs CouchDB back in 2013 - the Web’s own UI stack took that job.

The linked-parts half of the idea is back too. Conway notes that Roam, Logseq and Obsidian have spent the last decade rediscovering BTRON’s model of linked, transcludable blocks - mostly as local-first files on disk.

Denny called this in 1990 - in reverse

Back in 2013 I wrote about Interface Builder’s alternative Lisp timeline, and Dennison “Denny” Bollay told me about “Dynamic Documents”, built for the CIA after Xerox PARC’s NoteCards. It had unidirectional, bidirectional, contextual and searchable links, stored out-of-band in an object database, and linking arbitrary portions of documents from word processors, drawing programs and databases. When the CIA handed them an HTML spec, they and PARC concluded it was “dumb from almost every angle”: every document had to be rewritten in HTML, links couldn’t be contextual, links were one-way with no dead-link checking, it was completely static, and “even HTTP was stupid”.

“Obviously we misjudged…”, he added.

Most of those criticisms accurately identify capabilities the early Web omitted compared with richer hypertext systems, and in the same post I noted that Action! itself had no forward/back button and no URL concept. Denny’s list is really a list of the Web’s trade-offs, and the three rows above are what those trade-offs bought. Converging on a small set of simple, openly specified representations made graceful degradation much easier. Links being the same for all users is what makes a URL something you can paste into an email. Static documents with a browser in charge are what let the user, not the author, own back.

So, wealth extraction?

I don’t think Thom is wrong that application-first suits vendors - look at how many SaaS products break copy/paste, deep-linking or Back on purpose today. But BTRON, OLE and OpenDoc didn’t lose to greed. They lost to something simpler that was easier to leave. If someone wants to try document-centric computing again, ssokolow’s advice holds: solve the dependency problem first. And I’d add: keep the Back button outside the document, and make sure every part can collapse to plain text and a URL.

Where do phones fit?

That’s where the burning hot race is, so iOS and Android deserve a look against the three rows that matter. I’ve kept them out of the big table as their peak years (roughly 2007-2014) would break the 1984-1997 story:

Concept	iOS	Android
Fundamental unit	Sandboxed app	App, but composed of Activities wired together by Intents
Content extractability	Low-medium - many apps block text selection or draw their own UI	Same
Graceful degradation outside the ecosystem	Via the share sheet - transfers typed representations—URLs, text, images, files, etc .. to another app	Via share Intents - typed representations including text, URLs, images and files.
User-owned navigation layer	Weak - no system Back; a per-app nav bar, the edge-swipe convention, and the “◀ Previous App” breadcrumb	Medium - a system Back exists, apps can intercept it, and “predictive back” is slowly pulling it back under OS control

A few things stand out:

BTRON’s bet paid off - upside down.

Phones hid the filesystem (iOS until the Files app in 2017) and imposed a whole new platform worldview, on a non-x86 CPU no less. Nobody had to be talked out of the desktop, because the phone was a new device category. But what won inside that new worldview was the app silo - the opposite of documents composed of parts. This is where Thom’s wealth extraction theory has its strongest evidence: a 30% store cut gives the platform an economic incentive to keep content inside apps.

Android Intents are the closest mainstream thing to BTRON’s handlers.

Well, maybe closest: “I have an image/jpeg, who can EDIT it?” and the OS offers a chooser. The handler model survived, just at whole-app granularity rather than parts-within-a-document.

Android popularized the verb × typed-data model.

ACTION_VIEW, ACTION_EDIT and ACTION_SEND against a MIME type, with an “Open with … Always / Just once” chooser, made handler selection everyday UI for the first time. Most flows are VIEW or SEND though - EDIT rarely surfaces.

The idea was old. Prior art:

Unix mailcap (metamail, Bellcore, ~1991 (?); RFC 1524, 1993 (?)) - a view command per MIME type, plus optional edit=, compose= and print= fields. Possibly the first view/edit split per MIME type (?). Mutt and pine users configured it; edit= was rarely filled in.
OLE 1.0 (1990-91 (?)) - per-class verbs for embedded objects. A sound object’s primary verb was “Play” and “Edit” was secondary. The one place desktop users visibly met the idea.
Windows registry (3.1’s REG.DAT 1992 (?), routine by Windows 95) - shell\<verb>\command per file type, keyed on extension rather than MIME. Right-click “Edit” on a .bat or .reg opened Notepad instead of running or importing it; on XP, “Edit” on an image opened Paint.
Mac OS X Launch Services - apps declare themselves Viewer or Editor per document type. Architecturally explicit, but largely invisible in ordinary UI.

In ordinary desktop use, users overwhelmingly interacted with one cell of the matrix: Open. And that cell became a battleground - media players and browsers hijacking associations on install, “make X your default?” nags, and eventually Windows 10 hash-protecting defaults so apps couldn’t grab them silently. Nobody fought over the Edit slot. Vendors fought fiercely over the one slot that decides whose app you live in. Whatever the motivation, that’s at least compatible with Thom’s wealth-extraction argument: the economically valuable position wasn’t “editor for this type”; it was “default place where this type opens.”

The Web nevertheless supplies the most universal escape hatch: when an app has a public Web representation, sharing a URL exports a reference that virtually any destination can preserve, even if it understands none of the app’s internal data model. Deep links and universal links are apps adopting URLs after the fact. The graceful-degradation row still holds on phones - it’s just the Web providing it, not the platform.

And in closing, the hot race today is a fight over navigation master control. Apple’s App Intents (since iOS 16) and Google’s equivalents on Android let apps expose typed actions that an OS-level assistant or agent can invoke across apps. That’s a layer above the apps that the user - or their agent - drives: browser chrome again. Architecturally, it rhymes with BTRON/OpenDoc composition: applications expose typed capabilities that a higher-level orchestrator can combine, with apps as verb-handlers stitched together by an orchestrator. The open question is the same one as 1993: who owns Back, and who owns the layer above the content? Platform vendors are increasingly putting an OS-level assistant/orchestration layer above apps. The Web put that layer in the user agent.

Files + applications again

They don’t go away just yet. I enjoy using my Chromebook and Google is an almost entirely online company, but they still have dirs and files and apps that would operate on them for ChromeOS. I’m 70% in that world day to day. Google are rolling out AluminiumOS presently so I get to find out how much my preferred OS changes in the transition, I guess.

Charting AI Today: What It Leaves Behind

2026-07-26T00:00:00+00:00

A few days ago I wrote about build-time AI vs run-time AI, after the Thoughtworks Future of Software Development Retreat in Engelberg - three days in June, convened by Martin Fowler and Thoughtworks, following an earlier gathering in Utah in February. Two write-ups are out now: Martin’s closing fragment and the official Thoughtworks report. Five headline findings, of which the first is the one that matters here: code generation is no longer the bottleneck - verification is. Agents can produce code, specs, tests and infrastructure faster than any team can trust it. The trick is - how to match the generated code’s tests to the generated code’s prod source.

My post was a 3×2 table. I’ve been thinking about it since, and the table wasn’t really getting accross what I wanted it to. Too many things sat awkwardly across the line, and the interesting distinctions were happening inside the cells rather than between them.

So Claude and I made a chart instead. It’s here, and it has toggles so you can play with it. 15 years ago I was using AngularJS for interactivty within SVG, and today I don’t ask Claude what it picked. Indeed Claude would have said yes to any of 50 JS/TS techs.

The axis that took the longest

I started with “artefact surface”: versioned artifacts on the left, operational state on the right. Two halves, four quadrants, done. My biases toward source control systems were clear - which I what I wanted.

It wasn’t two halves though. Pure chat has no versioned artifact and doesn’t touch operational state. Media synthesis has a library - durable, real, yours - that nothing downstream reads. And agents like OpenClaw don’t have a surface so much as an absence of one: whatever the machine can reach by whatever means … which is why I’ve not tried one of the many claws.

The version I settled on runs through six waypoints:

Chat Log → Versioned Artifacts → Media Library → Change Specific → Change Anything → Become

The ordering isn’t reach, and it isn’t persistence. It’s what’s left behind that someone can inspect. A transcript you can read. A commit that gets reviewed before it counts. Assets in a library. Records other biz systems depend on. Traces wherever the thing happened to reach. And then ultimately “Become” rightmost.

Persistence rises, then does something strange

If you follow the waypoints expecting durability to climb steadily, you’ll be wrong twice.

A chat log is less durable than a generated video file. And a media library is arguably more durable than a commit while being far less connected - nothing downstream reads it, which is why it sits where it does rather than further right. Durability and consequence aren’t the same axis, and conflating them is how you end up thinking Sora and alike are more consequential than Dependabot.

What “Become” means

The last waypoint is the one I keep coming back to. Someone built an operating system that was all AI - no code for the file editor, no code for the browser, no code for the spreadsheet. Just a model behaving editor-ly on demand. I’ll add the link when I can find it again.

My first instinct was that nothing survives there. That’s wrong. If the AI-generated word processor saves a .docx file, the file is on a filesystem, as real as one written by actual code.

Two artifacts are in play and “Become” splits them:

The output: your .docx file. Survives fine.
The producer: the word processor itself. Doesn’t exist as reviewable code, and may not be the same word processor next time you open the document.

And the sharper edge: with real code, save either worked or threw an exception. Here, “Saved!” is generated text. The file might be there. Might be elsewhere. Might not exist. You can’t read the save path to check, because there is no save path. You’re verifying from outside, every time.

Sam Ruby, on what survived contact: “rigor doesn’t vanish when the agent writes the code - it migrates. Upstream into specifications, down into test suites treated as first-class artifacts.” His sharper claim is that the oracle - the objective, and how you evaluate whether you met it - is the asset that can’t be delegated. Implementation can be generated and thrown away; the oracle can’t.

Which gives me a cleaner way to say what Become is. Not the absence of code - the absence of an oracle. Nothing independent left to check the output against.

That’s not the absence of an artifact. It’s the absence of any guarantee about one, which is worse. Perhaos thats absence you’d notice. See “side story” at the bottom about how the industry does historicaly like things that work but bobody can explain how precesiley - with the risk being “if it breaks, can it be fixed and how long will that take and what guarantees.”

The overlays

The chart has three toggle rows, and the reason they’re overlays rather than axes is that none of them are positional. They cut across.

Containment - Augmented, Harnessed, Unharnessed. Augmented is the bottom of the chart, and it’s mostly classic products with a model bolted into an existing deterministic frame. Zapier didn’t get displaced by LLMs; it became a distribution channel for them. The model fills a slot; the workflow around it stays developer-defined. That’s the deployment pattern most organisations actually reach for, and it’s the least discussed.

I’m using “harness” in roughly Thoughtworks’ sense - the scaffolding round an agent - though I care less about how well it’s built than whether it exists at all, and whether anything checks the output before it counts. That’s a looser test than theirs, which is why the Harnessed lasso picks up commercial CRM and support agents where the vendor built the rig rather than you. The rig is still there. You just didn’t assemble it, and can’t see much of it.

When it goes wrong - the four MTTD/MTTR regimes. Mean time to detect and mean time to repair are independent, and people conflate them constantly. Fast/fast is CI linters and autocomplete: wrong is visible immediately and costs seconds. Fast/slow is a Zapier job that mangled 10,000 records overnight - you know by morning, and unwinding takes a week.

Slow/slow is where it gets uncomfortable. Data enrichment writing a plausible summary into a CRM field. Nobody notices for months. By then humans have read it and trusted it, it’s been exported into reports, and there’s no git revert for “this record has been quietly wrong since March.”

Note that MTTR correlates with the x-axis - versioned artifacts are revertable by construction - but MTTD doesn’t correlate with anything on the chart. Detection time is a property of how the output gets consumed, not of what the system is. That’s why it needed its own overlay.

This is the retreat’s “verification is the bottleneck” finding wearing different clothes. If verification is what’s scarce, then MTTD is just the measure of how long unverified output sits around being trusted. The chart’s slow-detect region is where verification isn’t happening at all - not because it’s hard, but because nobody scheduled it.

The Thoughtworks report gets to a recommendation I’d endorse from a different direction:

Adopt a risk-tiered autonomy model per system or component (cobot-style human oversight vs. dark-factory-style full automation), explicitly based on risk, reversibility and blast radius - do not apply a single autonomy policy uniformly across a portfolio.

Reversibility and blast radius are precisely what the x-axis and the MTTx overlay are measuring. If you want a risk-tiered autonomy model, you need a way to see which tier a given system is actually in - and “it’s agentic” doesn’t tell you. Dependabot and OpenClaw are both agents. They’re nowhere near each other on any dimension that matters for setting a policy.

Human veto - before it counts, after the fact, or not at all.

I got this one wrong first time. I had the whole chat column as “human-free,” reasoning that there’s no PR, no approval workflow, no review artifact. But chat has the tightest veto on the chart: you read every token before anything happens, and rejection costs one prompt. I’d confused the absence of a formal review process with the absence of oversight. They’re not the same thing, and the informal one is often stronger.

Kief Morris, writing up the same retreat, reduces the whole thing to: “how much do we let an agent decide, and how do we stay confident in what it does?” His ops teams keep a narrow remit - “diagnose, yes; decide, no.” That’s a veto boundary drawn inside a single system, which my chart can’t show; one dot per product doesn’t have the resolution. He also found line-by-line code review had stopped working as a guardrail, with rigor migrating upstream to acceptance criteria and downstream to automated checks. Worth holding against my “before it counts” lasso: the gate is there, but it may not be carrying the weight we assume it is.

Ian Cooper’s gears make the same point from another angle: he drives in high gear for boilerplate and drops to low gear when certainty falls or the blast radius grows - auth, payments. So it isn’t just that the veto boundary sits inside a system rather than around it. It moves, hour to hour, with the same tool in the same hands. My chart gives each dot one veto state and holds it there.

Kief and I were at the same three-day event. Showing him the open source thing I’ve been building in exactly his space was in my top five must-dos for the trip. I flew home without having done it. Forty sessions (albeit multi-track) and plenty of hallway time, and the one conversation I’d specifically planned was the one I missed.

Only two dots are genuinely veto-free, both at the right-hand end, and they fail differently. OpenClaw’s defects get found late and can’t be cleanly undone. The AI-native applications one is worse in an odd way - there’s no defect to point at, because there’s no code to diff against intent. You just sense that something is off - “uncanny valley” in the modern age.

Abby Bangser makes a point I hadn’t considered: “removing humans from the loop also removes organisational and product learning.” The veto isn’t only a correctness mechanism. It’s where people find out what the system actually does - which is the apprenticeship problem arriving by a side door.

Caveats

Dot placement is editorial, not measured. I’d defend the clusters and the general diagonal; I would not defend any individual coordinate to two decimal places. The chart is a thinking tool, not a dataset.

The Media Library island property is a fact about today’s integrations, not about the technology. The moment someone wires generated assets into a publishing pipeline, that dot slides right. Integrations by another name.

And the whole right-hand end is thinly evidenced - one shipping product and one experiment. That’s the nature of drawing a chart about where things are going rather than where they’ve been.

Have a play with it. The overlays combine - Augmented plus slow/slow detect is the combination I find most interesting, and it isn’t the one anyone is writing threads about.

Others who wrote it up

Worth your time, and I’ve only borrowed from some of them above: Ivett Ördög on whether agents should maintain codebases like engineers or regenerate them like compilers - the question my x-axis is really about; David Whitney on why “almost always working isn’t enough”, and code mattering more rather than less; Bartosz Ocytko on software factories and the handoff points where humans still sit; Giles Edwards-Alexander on “Optimisers vs Learners” as two organisational shapes, and on learning being the constraint now rather than delivery; Andrew Harmel Law asking which of our practices were rafts we can put down.

Footnote: a reversibility I’m not measuring

Mathias Verraes argues for keeping the ability to “revert from agentic code generation to human engineering.” That’s a reversibility my chart doesn’t measure - not undoing a change, but undoing a way of working. The x-axis says nothing about whether you can walk back from a waypoint once you’re standing on it.

Side Story

In the mid-1990s, at the University of Sussex, researcher Adrian Thompson used a genetic algorithm to repeatedly reconfigure a Xilinx FPGA until it evolved a circuit that could distinguish between two audio tones. Instead of producing a neat digital design, evolution exploited subtle analogue properties of the physical silicon timing delays, capacitance and other quirks to create a solution that worked but was largely incomprehensible to its creators and often depended on the characteristics of that specific chip. The result was a striking demonstration that evolutionary search can discover highly effective solutions beyond human intuition, but it was never adopted for telecoms or other safety-critical industries because engineers could neither fully explain nor reliably verify its behaviour across different devices, temperatures and operating conditions. In the end, its greatest strength finding unconventional solutions unconstrained by human assumptions was also the reason it could not satisfy industries that require predictable, reproducible and certifiable designs. Are we back in that risk place now?

Build-time AI vs Run-time AI

2026-07-20T00:00:00+00:00

I was at a ThoughtWorks “The Future of Software Engineering” event in Switzerland some weeks ago. What a braintrust. Lots of conversations about AI generally, agentic AI in particular and harness engineering.

Here’s a distinction I’ve been mulling for a while and keep reaching for in conversations about AI systems in the enterprise: build-time AI versus run-time AI. One is artifact-oriented, the other is execution-oriented.

Mode	Primary artifacts	Versioned?	Typical outputs
Build-time AI	Source files, configs, prompts, AGENTS.md, tests, policies, IaC	Yes (Git)	Commits, PRs, CI/CD, deployments
Run-time AI	Live state, requests, conversations, workflows	Usually no	Database rows, tickets, CRM updates, emails, transactions, generated media

Run-time AI isn’t only transactional, either. Generation of movies, songs, images, and slide-shows is run-time too. Those outputs are files, which makes them look artifact-like, but they’re products of execution, not engineering artifacts. Say a generated slide-show doesn’t specify the behavior of any system, and nobody’s putting it through code review as it might be going straight into GoogleDocs or Office 365. That’s the real test for which side of the line something sits on: build-time artifacts specify (later) behavior; run-time outputs are the consequences of wished-for behaviors.

Cross that with the conversational -> workflow -> agentic ladder that has become the common way of classifying AI systems, and you get a tidy 3×2:

AI type	Build-time	Run-time
Conversational	Prompt engineering: templates, system prompts	A single chat session
Workflow	Workflow engineering: workflow definitions, tool configuration	Executing the predefined process
Agentic	Harness engineering: code, prompts, tools, memory policies, guardrails	The agent reasoning, planning, acting, updating systems

Harness engineering is a build-time discipline

The interesting consequence: harness engineering is predominantly a build-time discipline, even though it exists entirely to influence run-time behavior.

The build-time harness looks like source code, because it is source code:

AGENTS.md
tools.yaml
memory_policy.py
planner.py
evals/
guardrails/
prompts/

Well, it is today. It gets committed to Git, reviewed in pull requests, and deployed via CI/CD - the full software delivery treatment.

Run-time is a different world entirely. Steps in a workflow:

Customer asks for a refund
Agent plans
Looks up CRM
Issues refund
(Audit log written)
(Database updated)

The last two are in parentheses because the agent isn’t doing them directly - the existing application or service (AI made or not) it invoked does those as a matter of course.

No Git commit occurs during that interaction. The agent changed the state of several systems of record, but it didn’t change itself.

An orthogonal axis, not a maturity stage

It’s tempting to present build-time vs run-time as another rung on the maturity ladder, after conversational, workflow, and agentic. I’d resist that. It’s an orthogonal axis. There are two independent questions to ask of any AI system:

What kind of AI system is this? - Conversational → workflow → agentic.
Where does the “engineering” happen? - Build-time artifacts ↔ run-time execution.

Every cell in the 3×2 above is a legitimate place to be, with the non-agentic ones included in that claim. A prompt-engineered conversational system with its templates under version control is not “less mature” than an agentic one. Indeed, it may be exactly the right amount of machinery for the job.

What a “harness” actually is

This framing also gives “harness” a satisfyingly software-engineering flavor. Edward Mangini’s recent Harness Engineering: The Devil is in Your Details defines a harness very broadly - essentially “everything except the model.”. He credits others (Trivedy & Birgitta Böckeler) for that. I think there’s a cleaner separation: the harness is the version-controlled, deployable specification of agent behavior. The live conversation, the database state, and the external side effects are not the harness - they are the consequences of the harness executing.

That gives harnesses a lifecycle human software engineers will recognize:

**author -> review -> commit -> deploy -> execute -> observe -> revise

Everything before “execute” is build-time. “Observe” is where run-time telemetry feeds back into the next build-time iteration. Which is to say: harness engineering is just engineering, and all the disciplines we already have for source code - code review, trunk-based development, continuous delivery, configuration as code - apply to it directly.

Well of course, build-time vs run-time isn’t common language in the AI field, so I’ll keep on trying to update my mental model.

Google-style DAG build systems (with Aether Build)

2026-07-02T00:00:00+00:00

For years I’ve contrasted depth-first recursive build technologies (Maven, Gradle, MSBuild) with directed acyclic graph (DAG) ones (Blaze/Bazel, Buck, Nx). I keep a small simulation repo, google-monorepo-sim, to make the difference concrete rather than hand-wavy. This entry walks through it.

The sim used to be a pile of .compile.sh / .tests.sh / .dist.sh bash scripts with a shared-build-scripts/ folder of common logic and a .buildStepsDoneLastExecution file for tracking what had already run. That’s all gone now. The DAG side is built with Aether Build (aeb), a small polyglot build system written in Aether. It’s still crude and teachable in a way that compares to Bazel/Buck/Nx.

Multi-module build systems per language

Language	Depth-First Recursive	Directed Acyclic Graphs (leaf-first)	Other / Community Tools
Python	setuptools, Poetry, SCons	Pants	Hatch, PDM
C++		CMake, Ninja, Bazel, Buck	Meson, Tup
Java	Maven, Gradle, Ant	Bazel, Buck
C		Make, CMake, Bazel	Meson, Ninja, Autotools
C#	MSBuild, Cake, NAnt		Fake (F#), dotnet CLI
JavaScript	npm/yarn	Webpack, Rollup, Nx	Vite, Gulp, Grunt, esbuild
Go		go build, Make, Task, Bazel	Mage
PHP	Composer, Phing		Robo, Deployer
Ruby	Rake, Bundler		Thor, Hoe
Swift	SwiftPM, Xcode	Bazel	Tuist
Kotlin	Gradle, Maven	Bazel
TypeScript		tsc, Webpack, Rollup	esbuild, Vite
R	R CMD build, devtools	Make
Rust	Cargo	Buck, Bazel, Make
Scala	sbt, Gradle, Maven	Pants, Bazel	Mill
Elixir	Mix		Rebar (from Erlang), Bake
Haskell	Stack, Cabal	Bazel	Shake
Dart	pub	Bazel (via rules_dart)	build_runner
Erlang	Rebar3		erlang.mk
Zig	zig build system		CMake (rare, via wrappers)

Two anemic applications

We’re going to build two contrived applications: DirectedGraphBuildSystemsAreCool and MonoreposRule. All they do is print to STDOUT then exit. Each only needs the letters in its own class name, which is what gives us a nice non-trivial dependency graph without any actual business logic to get in the way.

Components that the applications use

The components are categorized into modules, each representing a type of phonetic sound:

Vowels: A, E, I, O, U.
Nasal: M and N.
Voiceless: P and T.
Sonorants: L and R.
Fricatives: S.
Labiodental: V and W.
Glides: H, J, and Y.
Sibilants: Q, X, and Z.
Velar: G and K.
Voiced: B and D.

Since the last time I wrote this up, the sim went polyglot. It’s now five languages, and some of the components crossed a language boundary:

vowelbase bridges Java and Rust via JNI. The VowelBase Java class loads a native library libvowelbase.so, implemented in Rust, whose printString wraps a string in parentheses. A gratuitous use of Java-invoking-Rust, but it demonstrates a cross-language edge in the graph.
nasal now bridges Java and Go. The Go module compiles c-shared to libgonasal.so, and the Java nasal component loads it via JNI.
sonorants (L, R) is now a Kotlin module rather than Java.
There are also TypeScript, C#, Python and even an Aether-language component in the tree for other demos, but the two apps above don’t depend on them.

The dependency graph

Here’s the class-dependency graph for the two apps. Each app (top) depends only on the letter classes in its own name; the letter classes live in phonetic-category component modules (middle); and two of those modules cross a language boundary via JNI - vowels through the Rust libvowelbase.so, nasal through the Go libgonasal.so - while sonorants is Kotlin. sibilants and labiodental are present in the tree but neither app needs them, so a DAG build never touches them.

The DAG side: Aether Build

I’m not going to use Bazel (bazel.build, Google’s open-sourcing of Blaze) or Nx (nx.dev). I’m using aeb, which illustrates the two aspects of Blaze I most remember from my nearly two years in Google’s Test Mercenaries team:

a leaf-first directed (acyclic) graph approach to building in a monorepo, and
a changed-or-not check of inputs to skip individual steps.

Build files

Each module directory carries up to three declarative build files:

File	Purpose
`.build.ae`	Compile - declares deps, invokes the language compiler
`.tests.ae`	Test - declares deps + test libs, compiles and runs tests
`.dist.ae`	Package - builds a fat jar or other distributable

Here’s java/applications/monorepos_rule/.build.ae:

import build
import build (prereq)
import java

aeb(cap) {
    b = build.start()
    prereq(b, "jdk:21")
    build.dep(b, "java/components/fricatives/.build.ae")
    build.dep(b, "java/components/nasal/.build.ae")
    build.dep(b, "kotlin/components/sonorants/.build.ae")
    build.dep(b, "java/components/voiceless/.build.ae")
    build.dep(b, "java/components/vowels/.build.ae")
    java.javac(b)
}

Three things worth pointing at:

aeb(cap) is the entrypoint - not main(). The build receives a capability handle cap from the trusted aeb host (the same handle that backs runtime sandbox containment). A build file never constructs its own authority; it only receives it. (The legacy main() spelling still lowers to the same context-receiving entrypoint, but aeb(cap) is the convention the repo demonstrates.)
prereq(b, "jdk:21") declares the toolchain this node needs, OS-agnostically. A remote agent’s requester can read aeb --prereqs and pick, say, an aeb-tc:rust or aeb-tc:go image per dispatch - the bare host need not have every toolchain installed.
Dependencies are one build.dep(b, "path") per line - greppable, so the DAG can be extracted without compiling anything.

The cross-language edges are just more build.dep lines. java/components/vowelbase/.build.ae depends on rust/components/vowelbase/.build.ae; java/components/nasal/.build.ae depends on go/components/nasal/.build.ae. The Rust one is a cargo_project producing a cdylib; the Go one is a go_build in c-shared mode.

A single aeb invocation scans the relevant .build.ae / .tests.ae / .dist.ae files, topologically sorts the dependency graph, and runs everything in one process with an in-memory visited-module map. No per-step shell fork storm, no .buildStepsDoneLastExecution file.

First build

Picking the MonoreposRule app and asking for its distribution jar. First, a genuinely cold build — aeb keeps a content-addressed cache in ~/.aeb/cache, so to see everything actually compile I clear both that and target/:

rm -rf target ~/.aeb/cache
aeb java/applications/monorepos_rule/.dist.ae

The telemetry summary:

[telemetry]
  dist:    java/applications/monorepos_rule 0.00s [n/a]
  build:   java/applications/monorepos_rule 0.00s [miss]
  build:   java/components/fricatives       0.00s [miss]
  build:   java/components/nasal            0.00s [miss]
  build:   kotlin/components/sonorants      0.00s [miss]
  build:   java/components/voiceless        0.00s [miss]
  build:   java/components/vowels           0.00s [miss]
  build:   go/components/nasal              0.00s [miss]
  build:   java/components/vowelbase        0.00s [miss]
  build:   rust/components/vowelbase        0.00s [miss]
  jni.crate: libs/rust/registry/vendor/jni    0.00s [n/a]
total: 15.80s wall
aeb: 9 compile + 1 dist + 0 test
  compile: java/components/fricatives
  compile: go/components/nasal
  compile: java/components/nasal
  compile: kotlin/components/sonorants
  compile: java/components/voiceless
  compile: rust/components/vowelbase
  compile: java/components/vowelbase
  compile: java/components/vowels
  compile: java/applications/monorepos_rule
  dist:    java/applications/monorepos_rule

Note what is not there. The MonoreposRule app only needs the letters M O N O R E P O S R U L E, so aeb built exactly the nine modules feeding those letters - Java fricatives/nasal/voiceless/ vowels/vowelbase, Kotlin sonorants, Go nasal, Rust vowelbase (plus the vendored jni crate)

and the app itself. sibilants, labiodental, consonants, glides, velar, voiced are all present in the filesystem but irrelevant to this target, so they never compiled. That skipping of present-but-not-pertinent source is the defining trait of a DAG build system. Five languages, one command, one linked binary orchestrating the lot.

Running it again

Run the same command a second time and every node is a cache hit:

[telemetry]
  dist:    java/applications/monorepos_rule 0.00s [n/a]
  build:   java/applications/monorepos_rule 0.00s [hit]
  build:   java/components/fricatives       0.00s [hit]
  build:   java/components/nasal            0.00s [hit]
  build:   kotlin/components/sonorants      0.00s [hit]
  build:   java/components/voiceless        0.00s [hit]
  build:   java/components/vowels           0.00s [hit]
  build:   go/components/nasal              0.00s [hit]
  build:   java/components/vowelbase        0.00s [hit]
  build:   rust/components/vowelbase        0.00s [hit]
  jni.crate: libs/rust/registry/vendor/jni    0.00s [n/a]
total: 3.20s wall

15.8s cold, ~3.2s warm. The cache is keyed on the digest of the inputs, not file timestamps - a meaningful upgrade over the old bash sim, which used modification times. Because the cache lives in ~/.aeb/cache and not under target/, blowing away target/ alone still yields all-hits; you have to clear the cache to force real recompilation. [hit] / [miss] / [n/a] in the telemetry is the whole incrementality story, right there.

Running the tests

aeb javatests/applications/monorepos_rule/.tests.ae

[telemetry]
  tests:   javatests/applications/monorepos_rule 0.00s [miss] 1/1 PASS
  build:   java/applications/monorepos_rule 0.00s [hit]
  junit.jar: libs/java/junit                  0.00s [n/a]
  hamcrest.jar: libs/java/hamcrest               0.00s [n/a]
  ...
total: 4.16s wall

The .tests.ae file depends on the app’s .build.ae plus the vendored JUnit and Hamcrest jars, then runs java.javac_test and java.junit. Because the app’s compile is already cached, only the test compile-and-run actually happens.

Running the app

The .dist.ae step told us how to launch it. The fat jar carries all 1000-odd classes plus both native libraries (libvowelbase.so, libgonasal.so) at the jar root, so:

java -Djava.library.path=. -jar monorepos-rule.jar

main() .. MonoreposRule instance created:
<M>(O)<N>(O){R}(E)P(O)S{R}(U){L}(E)
Key: (vowels via Rust), <nasal via Go>, {sonorants via Kotlin}, all others pure Java

(...) are vowels routed through the Rust JNI library, <...> are the nasal letters routed through the Go shared library, {...} are sonorants from the Kotlin module, and the bare letters are plain Java. That single line of stdout is exercising four languages linked into one process. Then it exits. That is all these apps do.

Building the other application

aeb java/applications/directed_graph_build_systems_are_cool/.dist.ae

DirectedGraphBuildSystemsAreCool needs a bigger alphabet (consonants, glides, velar, voiced come into play), so its graph pulls in more modules - 13 compile targets versus 9. But everything it shares with MonoreposRule - fricatives, nasal, sonorants, voiceless, vowels, vowelbase and its Rust/Go natives - is already a cache hit from the previous build. Only the genuinely new modules compile. The fat jar it produces contains that app’s deps and none of the MonoreposRule-only ones. It runs the same way:

main() .. DirectedGraphBuildSystemsAreCool instance created:
D(I){R}(E)CT(E)DG{R}(A)PHB(U)(I){L}DSYST(E)<M>S(A){R}(E)C(O)(O){L}
Key: (vowels via Rust), <nasal via Go>, {sonorants via Kotlin}, all others pure Java

The depth-first recursive side: Maven

The sim’s depth-first_recursive_modular_monorepo branch has the same Java and Rust sources, in a classic modular layout, built by Maven.

In a typical Maven build, modules are built from the root, resolving and building dependencies in the right sequence. Maven encodes its build instructions in XML, one pom.xml per module. It traverses each module to its full depth of submodules before moving to the next sibling, in order to understand the entire implicit graph. Intelligence from that graph lets Maven reorder modules so the most-depended-on ones compile and test first. That’s a depth-first visitation.

Note: This branch is significantly out of date now. I had hope to make it a solid peer of the aeb-using one but that has not happened.

A full clean build

./quieter-mvn clean package

(that shell script is just mvn clean package with fewer lines of output.) Maven visits every module, running clean, compile, test-compile, test and jar for each - dozens of steps - because package from the root touches the whole reactor. On my middle-of-the-pack AMD desktop that was about 11 seconds.

An incremental build (no `clean`)

./quieter-mvn package

Maven skips recompilation where sources haven’t changed (it compares source timestamps against compiled classes - Rust included). But it still enters the compile and Surefire plugins for every module and still executes tests regardless of whether that module’s sources changed, because a linked dependency may have. The skip of the actual do-compile / run-tests happens inside the plugin, so the modules still get listed. About 5.5 seconds - roughly half the clean run.

Focusing with `-pl` and `-am`

./quieter-mvn package -pl applications/monorepos_rule -am

-pl (project list) names the module to build; -am (also-make) ensures its dependencies build too. Maven still visits every module however deep to build understanding, then subsets the reactor to the target and its deps - so sibilants and friends are skipped. This is the closest Maven gets to the DAG tools’ behaviour, and its build time here matches the equivalent focused aeb invocation.

Maven’s Reactor is the part that schedules module builds by dependency order and runs the lifecycle phases - compile, test, package - within each. The key contrast with aeb: Maven’s reactor reads the whole source base up front to plan, whereas the DAG side has no scheduler surveying everything first - it walks out from the requested target through its declared deps.

Comparing features

In a larger company’s monorepo, where the intent is to share code at the source level and minimize reliance on a binary repository like Nexus or Artifactory, you’d reach for a DAG build system — Buck, Bazel, or Nx - for:

Incremental builds - track input changes, rebuild only what’s affected.
Remote caching and execution - share artifacts across machines / teammates.
Hermetic builds - same inputs, same outputs, regardless of environment.
Parallel execution - independent modules across cores.
Fine-grained dependency management - deps declared at file, not just module, granularity.
Cross-language support - one graph spanning many languages.
Reproducibility - the DAG makes builds repeatable.
Scalability - handle huge numbers of modules and deps.
Custom build rules - teams define their own build logic.
CI/CD integration.
Community and ecosystem.

My aeb-based sim, scored 0–5 against those:

4 - real content-addressed caching now (digest of inputs, not timestamps), though it still doesn’t do fine-grained sub-file dependency analysis.
1 - a shared local cache exists (~/.aeb/cache); no remote cache/execution.
2 - prereq() declarations move it toward hermeticity (toolchain selection per node), but it’s not sealed.
3 - single-process topo-sorted execution; independent nodes can run without ceremony.
3 - module-level deps; Bazel can depend on individually-listed sources, which I don’t.
5 - genuinely polyglot now: Java, Kotlin, Go, Rust and TypeScript in one graph, with JNI/FFI edges across language boundaries. This is the big change since the bash version.
3.
3 - the .build.ae files are far less repetitive than the old bash scripts thanks to the SDK’s javac() / kotlinc() / go_build() / cargo_build() / tsc() / shade() helpers.
2 - doable via the SDK; I haven’t shown a bespoke rule.
4 - nothing about this would trouble CI; the reporting could be prettier.
1 - it’s a teaching toy, not an ecosystem.

Circular references

Neither class of build system allows circular references at the module level - both fail early on a compile/test/package attempt. Some languages allow circular references between files (Laurel.java ↔ Hardy.java), but that lives inside a single compiler invocation (one module), which is a different thing.

Subsetting the checkout (expand/contract)

Google’s secret sauce.

The sim also demonstrates Git sparse-checkout, driven by Aether. A gcheckout tool recursively walks .build.ae / .tests.ae, extracts every transitive dep() / lib() / npm_dep() / cargo_dep(), and adds just those directories to the sparse checkout - so your working tree can hold one app and its deps and nothing else, exactly as Google’s in-house Piper-based monorepo lets you expand and contract. Earlier writing on this:

Googlers Subset their Trunk (2014)
Turning Bazel back into Blaze for monorepo nirvana (2015)
Further Experiments With Expanding/Contracting Monorepos (2017)

Interpreted config vs a compiled build program

It’s worth being precise about how the two DAG worlds turn build files into work, because Aeb and Bazel sit on opposite sides of a line here.

Bazel’s build files are written in Starlark - a deliberately-constrained Python dialect (no unbounded loops, no recursion, deterministic by construction). Each BUILD file is parsed to an AST, and that AST is then interpreted - tree-walked - by Bazel’s embedded Starlark interpreter. The output of that evaluation is data: the action graph Bazel’s engine subsequently schedules and executes. There is no native-code compilation of your build logic, and no static type checking of it; Starlark is dynamically typed and evaluated afresh on each invocation.

Aeb is a different pipeline. The .build.ae / .tests.ae / .dist.ae files are Aether source, and a single aeb invocation hands them (plus the build SDK - javac(), kotlinc(), go_build(), cargo_build(), tsc(), shade() and friends) to the Aether compiler (ae), which compiles and links the whole thing into one native binary and then executes it. The build logic itself becomes native code, not a graph fed to an interpreter. Because it goes through a real compiler, the build files are statically type-checked on the way through - that’s the Type checking completed with N warning(s) chatter you see scroll past during a build, catching (for instance) an unused variable in a build file before anything runs. The whole dependency graph is topologically sorted and driven from that single process with its in-memory visited-module map.

	Starlark (Bazel)	Aeb (Aether)
Pipeline	source → AST → interpret (tree-walk)	source → compile + link to native binary → execute
Build logic is	data producing an action graph	a compiled native program
Type checking	none (dynamically typed)	static, at compile time
Conditionals	`if`/`elif`/`else` - but only inside functions, and no way to run at “load” time beyond simple selects	full `if` (and `if`-expressions), `match` dispatch, guarded function clauses, and a compile-time `when` static-if that only type-checks the taken arm
Loops	`for` over a sequence, plus comprehensions - no `while`, no recursion (bounded by design)	unrestricted `while` (Turing-complete), head/tail list destructuring, higher-order closures - it’s a real systems language
Execution	interpreter re-evaluates `BUILD` files	one linked binary, single process

Neither is “the right answer” - and it’s the sort of distinction that gets hand-waved as “they’re both just DAG build tools.” They parse and evaluate your intent very differently. The place people assume Bazel wins outright is containment: Starlark’s interpretation and Bazel’s sandboxed action execution buy hermeticity guarantees, and you’d expect a “little compiled binary” to make no attempt at that. But that assumption is where Aeb is actually most interesting.

Where this is heading: containment

Look again at that entrypoint - aeb(cap), not main(). The build receives a capability handle from the trusted host and never constructs its own authority. That isn’t cosmetic. Aether ships a whole containment model: a deny-by-default grant DSL (grant_fs_read, grant_tcp, grant_exec, grant_env), enforced at three layers - the compiler refusing capability imports under --emit=lib, hide / seal except at lexical-scope level, and an LD_PRELOAD shim (libaether_sandbox.so) that intercepts libc against the grant list, with a seccomp-bpf fence closing the raw-clone3/vfork process-creation bypass. The containment principle is Avalon-style Inversion of Control: the container wires the grants, the contained code merely receives them and can’t tell it’s boxed.

Point that at a build system and the picture almost writes itself. A build action is exactly a piece of code that should only touch its declared inputs, write only its declared outputs, and - for most targets - reach no network at all. The prereq(b, "rust:1.75") line already declares the toolchain a node needs OS-agnostically; the obvious next move is that each node runs under a sandbox scope derived from its dep() / prereq declarations: read access to exactly its sources and dependency artifacts, write access to exactly its target/ slot, exec of exactly its compiler, and nothing else - a build step that tries to phone home or read /etc/shadow mid-compile is denied by construction, not by policy review. That’s Bazel’s hermeticity, but arrived at from the language’s capability model rather than from a syscall-emulating action runner.

And the repo already leans this way. There’s a vision doc describing the large version - ingesting 141 strangers’ Todo-Backend implementations across 30+ languages into one DAG with hermetic per-implementation containers, cloud-agent fan-out, and supply-chain veto on untrusted code - and the scaffolding for it is already in the source: lib/sandbox/, lib/agent/, lib/provision/ and a veto/ tree. The toolchain-selection story (aeb --prereqs picking a per-dispatch aeb-tc:rust image) is the same capability handle flowing outward to a remote build agent. So my prediction is straightforward: the aeb(cap) handle grows from “the thing that runs your build files” into “the thing that contains them” - per-node grants, containerised remote execution, and a veto layer over untrusted third-party build files - and the compiled-binary model turns out to be an advantage here, because the sandbox is the same one the language already enforces at compile time, not a separate runtime wrapper bolted on.

The honest caveat: today none of that is wired into the build path I ran above - aeb(cap) receives the handle but the per-node grant plumbing is vision, not shipped. Bazel’s sandboxing is real and running now; Aeb’s is a credible trajectory with the primitives already in the tree. But “they’re both just DAG build tools” misses that one of them is a config language that produces an action graph, and the other is a capability-secure systems language that could end up being the sandbox.

A note on scale

The modules here are not representative of enterprise sizes - each is a handful of tiny classes with a single placeholder test. In reality there could be hundreds of sources per module and tens of seconds of compile and test each. For Google’s thousands of applications and libraries with high test coverage, a from-root build of everything could take days on one workstation - and is never done, even with Blaze, given the trunk checkout was 90 GB back in 2012 and no dev workstation ever pulled all of it. That’s exactly why expand/contract and a leaf-first DAG matter.

This repo goes hand in hand with my book, Trunk-Based Development and Branch by Abstraction, and a short video about it … though that last is the older shell-script version thats now deleted even if the experience was the same.

Cherry-pick Asymmetry

2026-06-22T00:00:00+00:00

Context is Trunnk-Based Development with Branch for Relase, and cherry picking. Trunk is main or master for may Git repos, of course.

A friend put a scenario to me and I gave him a strong answer: I said it was statistically impossible for the bug to still be on trunk once formal QA had signed it off on the release-branch-coupled CD environment. He raised an eyebrow. This post is me making good on that claim - and being honest about the tiny sliver where “impossible” is really “one-in-a-few-thousand,” and what that sliver actually is.

The scenario. You’re doing trunk-based development with a branch for release. Cherry-picks go one way only - trunk to release branch, never back. A bug is found. The fix goes onto trunk, gets a desk-check sign-off (someone stands the site up on localhost, clicks around, says “yep, fixed”), and then it’s cherry-picked to the release branch. From there Jenkins (or similar) continuous-deploys it into a coupled “release-candidate” environment, and a QA colleague does a formal sign-off against the bug in that environment. All that in addition to test automation that the same Jenkins performed for the commit(s).

So the same fix now lives on two branches, but with two very different levels of scrutiny:

Trunk: desk-check only. One developer, localhost, eyeballs, and automated tests appreciated.
Release branch: desk-check (inherited via the cherry-pick) plus a formal QA pass in a real deployed environment.

My friend’s question: what’s the statistical chance the bug is still present in trunk, given it’s been formally signed off as gone on the release branch? My answer: effectively zero, and not by luck - by construction.

Why it’s (almost) impossible, not merely unlikely

This is not a “did the fix get applied?” question. The fix is the same commit on both branches. A clean cherry-pick puts the identical diff on the release branch - and crucially, if the cherry-pick landed without a clash, trunk’s copy and release’s copy of the fix are the same bytes against compatible context. If those identical bytes pass formal QA on release, the very same bytes are sitting in trunk. There is no mechanism by which “this exact change works there but not here” can be true when there and here hold the same change. That’s the “impossible” - and it’s a structural fact about cherry-picking from trunk, not a probability. At least, it is when using a modern VCS with merge point tracking.

So the residual risk is not “the fix doesn’t work on trunk.” It’s two things, both of which are not the bug you fixed:

The formal check didn’t catch it. The sign-off said “fixed” but it wasn’t - a flaky repro, the environment happened to mask it, the wrong path got exercised. Then it’s not fixed anywhere, release included, and the premise “formal QA signed it off” carried less information than we assumed. Worth being careful here about why: this is rarely anyone slacking. The honest cause is usually scale. A manual gate that was statistically airtight for a small app and a small team gets asked to span more surface per release as the portfolio of change grows with the company - the same gate, now covering more. Holding the old bar at the new volume would have meant doubling or tripling QA headcount, which orgs almost never do; so the gap widens as an under-investment decision made above the team, not a failing within it. It’s a property of how the verification model was funded against growth, not of the branching model. (The CRT’s “started with manual QA for a smaller application and dev team” node is exactly this.)
Trunk has since become a different codebase - a later commit re-broke it, or the surrounding code the fix relies on drifted. That’s a new defect on trunk, not the one you fixed surviving.

Neither contradicts the claim. The specific bug, once its cherry-pick clears QA on release, cannot be the thing still broken on trunk. Put a number on the residual anyway, to give you a feel: for a well-resourced QA gate and a tight cherry-pick loop, you’re in 1-in-10,000 to 1-in-100,000 territory; for a gate stretched thin by a grown portfolio of change, or a long-lived branch where trunk has drifted hard, it climbs toward 1-in-1,000. Those are the odds of one of the two other failures above - never of the fixed bug itself reappearing on trunk.

The divergence mechanisms - i.e. the only ways to lose the bet

These are the only routes by which the formal release sign-off fails to cover trunk. Read them as ways trunk became a different codebase or QA was wrong - not as ways the fix failed to transfer.

1. Trunk moved after the desk-check, in the same region. The fix was desk-checked on trunk at commit X. By the time you cherry-picked, trunk was at X+n. If one of those n commits touched the same code the fix depends on - refactored the function, changed a caller, altered a shared helper - then trunk’s current behaviour is no longer what was desk-checked. The release branch is frozen-ish and got the clean cherry-pick; trunk kept moving. The release branch can be more correct than trunk precisely because it stopped changing. This is the big one, and it scales with your cherry-pick latency and your trunk commit rate.

2. The same hunks landed against different surrounding chunks. Covered in depth in the limits-of-merging experiment. A clean cherry-pick isn’t really a merge - it’s a two-way apply: the fix’s changed hunks go in, and the surrounding chunks are taken as-is, unchallenged. Nothing reconciles them. So the very same hunks can land against release’s surrounding code and against trunk’s surrounding code, and if those untouched chunks differ - because trunk drifted (mechanism #1) - the effective behaviour differs even though the patch “applied cleanly” on both. The fix’s lines are identical; the code they execute against is not. Clean apply is necessary, not sufficient. Low probability, nasty when it hits.

And here is the tell. When the surrounding code has drifted enough, the cherry-pick doesn’t apply cleanly - you get a merge clash. That clash is not an annoyance to be force-resolved at 4pm on release day; it is the system telling you the two branches have diverged in the fix’s blast radius. A clash is the loud, honest version of mechanism #2 - the same divergence that, when it’s just a hair smaller, applies “cleanly” and lies to you. So the clash is a gift: it converts a silent risk into a visible one. The mistake is treating a clash as friction rather than as intelligence. Which is the whole argument for a dry-run, below.

3. Trunk took a second commit that the release branch never got. Here’s the version of this that sounds paradoxical until you look at it on a timeline. Suppose trunk has two commits in play: the fix (cherry-picked to release) and a separate, later commit that re-breaks the same thing - a regression, or a new second instance of the same defect. The obvious objection: if QA passed in the release-coupled CD env, how can the bug still be live in trunk? And the answer is clean - because the two commits are not both on the release branch. Only the fix was cherry-picked. The release branch is fix-and-nothing-else: QA stands it up, drives the repro, signs off correctly. Trunk is fix-plus-regressor.

So how did the desk-check on trunk miss it? Because of when it ran. The desk-check happened at the fix commit - and at that moment trunk really was fixed, and the desk-checker was telling the truth. The regressor landed afterwards. The desk-check didn’t fail; it expired. Trunk is broken, release is green, both “passed” their respective checks, and there’s no contradiction - the checks ran at different points on a moving trunk, and only trunk kept moving.

Notice this is not a half-fix arriving on the release branch. The fix arrived whole and works there. The trouble is entirely on trunk, and it is just mechanism #1 (drift) with the clock made explicit: a human sign-off is true at one commit; trunk is a moving target; the release branch stopped moving and the desk-check didn’t get a do-over. That’s the whole answer to “can it be green on release and still broken in trunk” - yes, and only this way.

4. Environment-specific masking, in the other direction. The release-candidate environment is “coupled” and real; localhost is not. There are bugs that only manifest with the real environment’s data, config, or integrations. The formal QA pass in the RC environment can therefore prove the fix more thoroughly than the desk-check ever could - which means the desk-check on trunk may have signed off on a fix that the desk-checker was a little optimistic on. Trunk inherits a weaker proof. The bug could still be “present” on trunk in the sense that nobody actually verified it there under conditions that could expose it.

So what’s the number?

Remember what the number is the probability of: not “the fixed bug survived on trunk” (that’s the impossible part), but “the formal check didn’t catch it, or trunk quietly became a different codebase in the fix’s blast radius.” With that framing:

Tight loop, ‘on it’ QA team, thin slices (cherry-pick minutes-to-hours after the desk-check, short-lived release branch): you’re at roughly 1-in-100,000. The fix is the same bytes against near-identical context; QA passed; trunk hasn’t moved meaningfully. About as close to “impossible” as software gets.
Ordinary shop (cherry-pick same-day, busy trunk, decent-but-human QA): call it 1-in-10,000. Drift is small but nonzero; quality assurance is good but not perfect.
Long-lived release branch, hard-drifted trunk, a verification gate stretched past what it was sized for: it climbs toward 1-in-1,000, dominated by mechanism #1 (drift) - trunk became a different codebase while the branch sat still. Even here it’s trunk drift or a missed check doing the work, never the fixed bug reappearing - and the “missed check” end is a scale-and-staffing story (the portfolio grew, the gate didn’t), not a worse team.

There’s a fourth axis cutting across all three: how good is your test automation, and what shape is it? A healthy test pyramid - broad fast unit coverage, a solid integration band, a thin full-stack testing cap/peak - attacks both residual terms at once. It shrinks P(QA wrong) because the formal sign-off isn’t a lone manual gesture; the same behaviour is asserted by a unit test and an integration test that run on every commit, so a false negative has to fool all three. And it shrinks P(drift) because the moment a later trunk commit re-breaks the thing (mechanism #3) or shifts the surrounding code (mechanism #2), trunk’s own CI goes red - you find out in minutes, not in next quarter’s release. An inverted pyramid does the opposite: it pushes you toward the 1-in-1,000 end and makes the dry-run and the trunk regression test below far more valuable, because they’re compensating for coverage the pyramid should have provided. So read the three rows above as “× your pyramid”: a fat-pyramid ordinary shop behaves like the tight-loop row; an full stack heavy tight-loop dev shop behaves like the ordinary row.

The headline I’d give my friend, sharpened: a formal QA pass on the release branch proves the fix bytes work; because those exact bytes (and solidly indicated commit) are also in trunk, the only ways trunk can still be broken are a false QA pass or a later change to trunk - and neither is the bug you fixed. That’s why I called it statistically impossible and stand by it. The residual is a rounding error with three dials - QA bar, drift, and test-automation shape and a deep pyramid turns all three down at once.

Cadence is the master variable

Trunk and the release branch are strongly related - they share the entire history up to the branch point and only diverge afterwards. Every divergence mechanism above is really one quantity wearing different hats: how far has trunk drifted from the branch point by the time this cherry-pick happens? And that distance is governed almost entirely by release cadence. At least, planned release cadende more so than unplanned.

Compare two shops shipping the same software:

One release branch a year, ~500 cherry-picks (feeding, say, ~100 unplanned point releases off that one long-lived branch). By cherry-pick #400, trunk is months ahead of the branch point. The surrounding chunks (mechanism #2) have been rewritten, callers have moved (mechanism #1), the bug may have sprouted a second home (mechanism #3) that didn’t exist when the branch was cut. Every cherry-pick is fighting a trunk that’s drifted enormously. The release branch and trunk are technically still related, but the relationship is distant, and the release sign-off’s evidential value for trunk is correspondingly weak. This is the troublesome situation. The first time the merge engine says can’t merge without arbitration, all bets are off - the quantum link bwteeen the two testing places (two branches) is lost..
Twelve release branches a year, a few cherry-picks each. Each branch is only ever cut a few weeks before its cherry-picks land, so trunk has barely moved relative to the branch point. The same fix lands against near-identical surrounding code on both branches. Drift is small, so all four mechanisms shrink toward zero, and the formal sign-off on release transfers to trunk almost perfectly.

Same total release count, same total cherry-picks-ish - wildly different risk. The number of branches per year is a proxy for short distance between trunk and each branch, which is the thing that actually keeps the cherry-pick honest. This is the quantitative version of the advice in the limits-of-merging post: long-lived release branches that absorb selective fixes are a structural risk, and the structure is drift. Branch more often, keep each branch close to trunk, delete it fast.

But “branch more often” is easy to say and hard to do, and the reasons it’s hard are exactly the constraints I mapped in the software-development Current Reality Tree starter pack. A team can’t dial cadence up - can’t cut twelve short-lived branches a year instead of one long-lived one - until it has cleared the bottlenecks upstream of release frequency. Four nodes in that tree map straight onto this post’s dials:

Build times are too long (“we build too much per build”, “the full build compiles too much code”, “integration tests are too slow”). This one I didn’t call out above, but it’s load-bearing. Put a number on “too long”: say it’s two hours from dev-done to the change being stood up in a QA environment and tested - compile, full test suite, deploy to the coupled env, then QA can start. When that round trip is two hours, you can’t afford to cut and verify a fresh branch per release, so you keep one alive and let trunk drift away from it; and every dry-run and every re-test of a cherry-pick costs another two hours, so people do fewer of them. Slow builds force the long-lived-branch shape that maximises mechanism #1, and they tax the very mitigations (dry-run, re-test against trunk) that drain the residual. Get that round trip down to minutes and the whole calculus flips. This is the reduction-of-cycle-times argument wearing a different hat: those cycles nest like russian dolls, and your effective build time is the innermost one that sets a floor under all the others - cherry-pick latency and release cadence included. Shrink the build and every outer cycle can shrink; leave it at two hours and none of them can.
Not enough QA is automated (“over-reliance on manual testing”, “not enough QA automators”, “QA test suites not run that frequently”). This is the same fourth-axis point as the test pyramid above, seen from the constraint side: a thin or inverted pyramid is why the formal QA pass is a lone manual gesture, which is why P(QA wrong) doesn’t shrink.
The release process itself is error-prone / “we don’t practice releases safely” - no rehearsals. Rehearsal is a whole family: dress-rehearsal deploys to a prod-like environment, blue-green and canary cutovers practised before they’re needed, rollback and roll-forward drills, smoke-test runbooks, game-days. The dry-run cherry-pick below is just the cheapest, narrowest member of that family - it rehearses one question (does this fix apply cleanly against the branch?), not the deploy. But the muscle is the same: a team that never rehearses anything is a team that discovers the clash at 4pm on release day.
CI isn’t actually continuous (per commit), and the branching model is crappy. If trunk’s CI doesn’t run on every commit, mechanism #3 (a later regressor) goes undetected for longer; and a weak branching model is the soil all of this grows in.

So the residual probability in this post and the slow-cadence at the top of that CRT are the same phenomenon viewed at two altitudes. The cherry-pick asymmetry is what you measure once a fix is in flight; the CRT is the map of why the fix is in flight on a branch that’s drifted too far in the first place.

The genuinely counterintuitive bit

Why do we cherry-pick trunk → release exclusively in the first place? Not because we distrust the release branch - it’s about forgetting. If you let people fix on the release branch and rely on merging the fix back to trunk afterwards, you will eventually forget to do it. The fix ships, QA signs off, the client pops the champagne - and three months later the next release is cut from a trunk that never got the fix, and the bug walks straight back in. A regression, manufactured by a missed merge-back. Cherry-picking from trunk forces the fix to exist in trunk by construction: there’s no merge-back step to forget, so that class of regression is structurally impossible. The direction of the arrow is the safety mechanism.

But look at what that buys you on this one bug, at this one moment: the causality inverts. The release branch is now the better-verified artifact. It got the formal pass; trunk got an eyeball. The fix is definitely in trunk - that’s the impossible-to-be-broken part - but the formal evidence that it works lives on the release branch, not on trunk. Trunk has the cure and a weaker certificate; release has the same cure and the strong certificate. They share the fix; they don’t share the proof.

This is not an argument to start merging release back to trunk - that’s exactly the missed-merge-back regression trap the one-way rule exists to prevent (plus every merge-tracking hazard on top). The fix is in trunk already. What’s missing from trunk is the verification, and the cure for that is cheap:

Dry-run the cherry-pick early - well ahead of the moment of nerves. First, the honest caveat: most cherry-picks just apply, cleanly, every time - if that’s your experience, you don’t need this as standing ceremony, and a same-day cherry-pick is already most of a dry-run. This earns its keep only for teams with a track record of cherry-picks not being seamless - the long-lived-branch, hard-drifted-trunk shops at the 1-in-1,000 end, where a clash is a regular event rather than a rarity. Those teams should: long before you need it, do a throwaway cherry-pick (git cherry-pick --no-commit and throw it away, or a scratch branch) just to ask one question: does it clash? A clean apply is reassurance; a clash is intelligence - it’s mechanism #2 announcing that trunk and the release branch have diverged in the fix’s blast radius, which is precisely when a “clean” apply would have lied to you. Rehearsing the cherry-pick ahead of time converts a 4pm-on-Friday panic into a calm Tuesday decision. The clash is information you want, as early as you can get it.
If quick and cheap, run the same formal check against trunk, or at least the automated portion of it. If the bug is worth a formal QA sign-off on release, it’s worth an automated regression test that runs in trunk’s CI forever. Then trunk’s coverage of this bug stops decaying - it’s pinned by a test, and the 1-in-10,000 drops further.
Cherry-pick fast. Every hour between desk-check and sign-off is more trunk drift (mechanism #1). Short loops shrink the dominant residual term. Perhaps only a problem for Google-sized companies with select teams doing branch-for-release.
Keep the slices thin. A single-commit fix that can’t be partially-applied removes mechanism #3 entirely.

When is the second pass actually redundant?

This is the part my friend and I went around on. Frame it as sets: the desk-check covers a region D (localhost, one moment); the formal QA pass covers a region Q (coupled environment, one moment). The bug is a point. If Q ⊆ D - the release pass touches nothing the desk-check didn’t - then the second test adds zero bits and is pure ceremony. But Q and D only partially overlap: the coupled environment reaches behaviours localhost can’t (mechanism #4), so Q \ D is usually non-empty, and a manual pass is also a snapshot that decays the instant trunk moves (the desk-check “expires”).

What decides the size of Q \ D and whether it stays small isn’t the manual pass at all - it’s what automated coverage exists for this exact feature/bug, and what shape it is. Automation is what turns a coverage snapshot into a coverage invariant (re-asserted every commit, so b ∉ D keeps holding as trunk drifts). Fast, elegant, component-ish tests pin the behaviour cheaply and run every commit; slow full-stack tests pin it expensively and tend to get relegated to a non-continuous build, so they decay almost like the manual check does. So the real question - “do I need a second manual QA pass on this fix?” - splits six ways:

Automated coverage for this fix	Shape	Second manual QA pass?	Why
None	full-stack / slow (or n/a)	Yes - it’s the only check	Nothing pins the behaviour. Manual is your sole signal, on both branches. Maximum exposure to drift.
None	fast / component-ish	n/a	If there were fast component tests, you’d have automation - this row doesn’t exist.
Some	full-stack / slow	Yes, but scoped	A slow suite that may not run per-commit; manual still earns its keep for the `Q \ D` crescent (environment-specific behaviour).
Some	fast / component-ish	Rarely	The fast tests already re-assert the core behaviour every commit; manual only justified for genuinely environment-only effects.
Good	full-stack / slow	Redundant-ish	Behaviour is pinned, but if the suite isn’t continuous you still inherit some decay - a thin manual smoke is defensible, not required.
Good	fast / component-ish	Redundant	`b ∉ D` is re-proven on every commit and the cheap test is the invariant. The second manual pass adds no bits. This is the case where my “not needed” is flatly true.

Read top-to-bottom, the manual second pass goes from load-bearing to theatre exactly as automation deepens and gets faster/component-ish. The diagonal is the point: it’s not enough to have “good” tests - slow good tests that don’t run continuously still let coverage decay between the desk-check and release day. Fast, elegant, component-level tests are what actually collapse Q \ D toward empty and keep it there, which is the only regime where double-testing the fix is genuinely wasted motion. So when I say the double testing isn’t needed, the honest scope is the bottom row - and the engineering move for everyone above it is to climb toward that row, not to keep paying for the manual pass.

Closing

So, was I right to tell my friend it’s statistically impossible? Yes - with the precision that makes it true rather than glib. The fixed bug cannot be the thing still broken on trunk after the release-branch QA pass, because trunk holds the same fix bytes and the same bytes can’t pass quality assurance there and fail here. What remains is a sub-thousandth-to-sub-hundred-thousandth chance, and it isn’t the bug: it’s P(QA was wrong) + P(trunk became a different codebase since the fix). Drift and bad QA - not resurrection.

The cheap moves that drain even that residual: leave a regression test behind in trunk’s CI (this one always pays, for everyone), and - if your cherry-picks aren’t reliably seamless - dry-run the cherry-pick early so a clash informs you instead of ambushing you. The dry-run is for the hard-drift shops; for a tight-loop team whose cherry-picks just apply, the regression test is the move that matters and the fast same-day cherry-pick already does the rest. The dry-run, where it’s warranted, turns silent divergence into a visible signal before nerves are involved. And the regression test pins the fix for good: a human desk-check is true at one commit; a new unit test is true at every commit after it. Branch-for-release with one-way cherry-picks is a fine model precisely because the fix is already in trunk - the work left over is verification, and verification is cheap to automate.

Final question for the reader to answer: Change the full QA team sign off to the trunk (CI into “shared dev” say), and don’t do even the desk check in the release environment after the very procedural cherry pick … what’s the 1 in N failure rate?

The limits of merging experiment

2026-04-28T00:00:00+00:00

A colleague suggested the cherry-picking way for trunk-based-development with branch for release needs worked examples cos there are foot-guns.

Cherry-pick to a release branch isn’t crystal clear as a workflow

The failure modes are more interesting than they look. To pick at it, I built a small playground - a single-file Sinatra CRUD app, an end-to-end Playwright test, and a series of trunk commits that would let me run the same scenario two different ways and see what git actually does.

The repo is here: paul-hammant/limits-of-merging-experiment. Clone it, run ./start.sh to make solution/ folder the git folder not the one you cloned.

The run bundle install to go get deps.

The setup

Folder solution/ contains the ruby/sinatra app and a playwright test for it. It is three tier - html with JS, a ruby middle tier, and a sqlite base tier. reset.sh will keep getting us back to a starting point. See “C1” below.

Five hypothetical trunk commits, all reachable as patches in patches/ and re-applicable on demand, are key to this experiment.

	Change	Touches
C1	Initial Person CRUD app, seeded Flintstones, Playwright happy-path test	everything
C2	Add `hair_color` (string) - dropdown, JS validation, DB `CHECK` constraint	`app.rb`, `happy_path_test.rb`
C3	Button text change to UPPERCASE: NEW PERSON / EDIT / DELETE / SAVE / CANCEL	`app.rb`, `happy_path_test.rb`
C4	`hair_color` becomes `INTEGER` (1..6); dropdown values are now ints	`app.rb`, `happy_path_test.rb`
C5	Add maintainer comment in header (cosmetic, previously untouched region)	`app.rb`

The shape that matters for this experiment: C3 is cosmetic and unrelated to hair colour. C4 builds on C2. C5 is in an untouched corner. This is normal trunk life - small unrelated changes interleaving in the same files.

The hypothetical release branch is cut from C2. The team wishes that was it for the release, and continues on an unfrozen trunk as normal. Later there’s something that agreed as a bug fix (definately not feature creep) and it should be cherry picked to the release branch by the responsible “merge meister” or release engineer. We want to ship bugfix C4 (the int conversion) but not C3 feature change (the uppercase buttons).

Scenario 1: `git am` is honest, and that’s why it fails

First release engineer instinct: apply the C4 patch directly to “stable” release branch.

$ git checkout -b release c2
$ git am patches/c4-hair-color-int.patch
Applying: C4: hair_color stored as INTEGER (1..6), dropdown values become ints
error: patch failed: app.rb:133
error: app.rb: patch does not apply

Loud failure. Why? Look at the failing hunk:

       <td><%= h p['dob'] %></td>
-      <td><%= h p['hair_color'] %></td>
+      <td><%= h(HAIR_COLORS[p['hair_color']]) %></td>
       <td class="row-actions">
         <a class="button" href="/people/<%= p['id'] %>/edit">EDIT</a>

The patch’s context lines (the unchanged EDIT reference around the change) include C3’s uppercase text. On the release branch the buttons still say Edit. Context doesn’t match. git am is strict about context - it refuses.

This is git am correctly doing its job. It’s not the failure mode I’m interested in.

Scenario 2: `git cherry-pick` is helpful, and that’s why it lies

(we do a ./reset.sh to go back to the starting position)

Second instinct, and what most developers actually type:

$ git checkout -b release c2
$ git cherry-pick c4
Auto-merging app.rb
[release ...] C4: hair_color stored as INTEGER (1..6), dropdown values become ints

Clean. No conflict. Test passes.

Why? Because git cherry-pick doesn’t apply patches by context - it does a three-way merge with the parent of C4 as the merge base. The merger sees:

merge base (C3 on trunk) had EDIT
the cherry-pick target (release branch) has Edit
C4 didn’t change either of those lines

So three-way merge correctly concludes “leave the case alone, just apply the hair-colour change.” Release ends up with C4’s diff cleanly applied on top of Edit.

That’s the textbook good outcome. And it’s exactly the failure mode the colleague was warning about - not because this cherry-pick was wrong, but because the success was contingent on a property of the diffs that nobody checked. C3 happened to touch only button text. If C3 had ever-so-slightly tidied up the dropdown the cherry pick may have conflicted - forcing a human to arbitrate on it.

The lies are little lies, from good intentions, perhaps.

What about merge-point tracking?

Here’s where Subversion fans get nostalgic. SVN’s svn:mergeinfo tries to record on the release branch “I have integrated revisions r3, r4 from trunk.” A subsequent sweep merge of trunk into release knows to skip those. Not just Subversion, but the “bigger” VCS technologies Perforce and Microsoft’s TFVC.

Git has none of that. A cherry-pick produces a commit with a different SHA from the original - git cat-file -p on the two reveals different parents, different trees, different hashes. The only audit trail is the optional (cherry picked from commit ...) line that git cherry-pick -x leaves in the commit message, and git itself does not consult that line for anything. It’s a comment.

So if you later merge trunk wholly into release, git’s merge base is the last common ancestor - which is C2, not C4. Git will re-apply C3, C4, and C5 from trunk on top of release. If the cherry-picked C4 on release is byte-identical to trunk’s C4, the merger deduplicates silently. If they differ even slightly (a hotfix on release, a typo correction), you get spurious conflicts that look very real, with no way for git to say “you already integrated this, just take trunk’s copy.” Coming from Svn, Perforce or TFVC merge-point-tracking is not as you remember it. In a TBD + release branches workflow you would never do a sweeping commit from trunk to the release branch - you would only do cherry picks and less and less so over time to that release branch. At some point the release branch has been superceded and eleigible for deletion.

SVN’s mergeinfo aimed at this and got bitten by edge cases - subtree mergeinfo creep, properties getting out of sync if you bypass svn merge. The “slightly broken” reputation is earned. But the intent - cross-branch awareness of what’s been integrated - is something git deliberately doesn’t have. Linus rejected it on simplicity grounds. The price is paid by anyone running long-lived release branches.

The order of cherry-picks question

If C3 and C4 are eventually both wanted on the release branch, does the order matter? Two scenarios:

Scenario A - out of order. Cherry-pick C4 first (the urgent fix), then C3 later (because someone decided uppercase buttons should ship after all):

release branch cherry-picks: C2 then C4' then C3'

Scenario B - in order. Cherry-pick in trunk order:

release branch cherry-picks: C2 then C3' then  C4'

In both cases, trunk has C2, C3, and C4. We could then sweep-merge c5 from trunk into the release branch.

Resulting tree hashes (with author/date/identity pinned so SHAs are deterministic):

	Tree hash after sweep merge	Merge commit SHA
Scenario A (out of order)	`088b679...`	`e96a8e0...`
Scenario B (in order)	`088b679...`	`4ad3efc...` (fast-forward!)

Tree hashes match. Content is byte-identical. Empty git diff between the two release branches.

But the commit graphs are different shapes. Scenario A produced a real merge commit with two parents - the cherry-picked C4’/C3’ on release and the trunk C5 - because the histories diverged. Scenario B’s cherry-picks produced commits byte-identical to trunk’s (same diffs, same pinned timestamps), so the sweep merge fast-forwarded; release’s tip is main’s tip.

So the answer to “does order matter” is layered:

Content: no, both converge to the same source tree.
History shape: yes, you get a merge node in one and a flat history in the other.
SHA equivalence: no, never - parent chains differ, so commit SHAs cascade differently. SHA equality was never the right test for “did this work.”

Blocking a commit (Scenario C)

The order question above is about C3 and C4 both eventually shipping. What about C3 not shipping at all — a hard “no, this isn’t for this release”? Cut release from C1 (so C2 also becomes a cherry-pick), cherry-pick C2 and C4, then:

$ git merge -s ours --no-edit \
    -m "block C3: merge -s ours (record without applying diff)" c3
$ git merge --no-edit main                      # sweep

git merge -s ours c3 makes c3 a parent of release via a merge commit, but the resulting tree is exactly “ours” — none of c3’s diff is applied. The merge-base machinery on subsequent merges then sees c3 as already integrated and skips it. It’s the structural equivalent of SVN’s --record-only and Perforce’s resolve -ay.

The scenario script probes “is each trunk tag reachable from release?” after every step, since reachability is git’s audit trail:

after Cherry-pick C2:       c2 yes, c3 no,  c4 no,  c5 no
after Cherry-pick C4:       c2 yes, c3 no,  c4 no,  c5 no
after -s ours block of C3:  c2 yes, c3 YES, c4 no,  c5 no   ← merge node makes c3 an ancestor
after sweep:                c2 yes, c3 yes, c4 yes, c5 yes  ← main is now an ancestor

The c3 flip on the third row is the block being recorded. After the sweep, all of main’s tags are reachable through the merge edge — but git diff release main reports only C3’s button-text changes (UPPERCASE on main, mixed-case on release). The block held.

(Aside: c2 says “yes” right after its cherry-pick because the experiment pins author/date/identity, so the cherry-pick reproduced c2’s original SHA byte-for-byte. In a real workflow timestamps differ on every cherry-pick, so c2 would show “no” too — cherry-picks leave no DAG fingerprint, only -s ours does.)

The thing to notice: git can block a commit durably, but the audit shape is different from SVN/P4. Where SVN updates a property string and P4 writes an ignored integration record, git records the decision as a graph fact — a merge node with the blocked commit as a parent. Reading the audit trail later means inspecting the DAG and the commit message at the block step. There’s no git mergeinfo, no git integrated. The data lives in git log --merges, and the meaning lives in the message you typed.

What the SHA actually proves

This is the bit I had to stop and think about. I’d been comparing commit SHAs and getting confused by the differences. The right framing:

Commit SHA equality means “byte-identical commit object including parents.” Tree hash equality means “byte-identical content.” Cherry-picks change parents. They cannot preserve commit SHAs. They can preserve tree hashes - and that’s the only equivalence that matters for correctness.

So when judging whether a cherry-pick + sweep merge produced the right result: diff the trees, not the commits. A clean git diff main release after the sweep is the only proof you need.

What git can’t tell you

Putting it together, here is what git silently cannot answer for a release branch built from cherry-picks:

“Have I already integrated this trunk commit?” Git’s answer is “no” - even if you cherry-picked it. The DAG has no link after the even (ignoring formatted comments).
“When I sweep merge runk to release, will it be a no-op?” Only if every cherry-picked patch is byte-identical to its trunk twin. There’s no machine check.
“Are these two release branches functionally equivalent?” Only git diff of trees can tell you. Commit history is misleading.
“Did this cherry-pick land safely?” Only your automated tests can tell you. Three-way merge succeeding is necessary, not sufficient.

The real risk in real corporate codebases

A common pushback: “in a million-line corporate codebase, two unrelated commits almost never touch the same file region - cherry-picks land cleanly the vast majority of the time.” That’s empirically true. The base rate of textual collision is low.

But the question isn’t frequency, it’s severity when it does happen. The classic bad outcome isn’t a noisy git am rejection - it’s a quiet git cherry-pick that three-way-merges into wrong-but-plausible code, ships to a release branch, passes your tests because the tests don’t cover the exact corner that broke, and surfaces in production a week later. Git gives you no warning. There’s no git fsck --semantic.

The mitigations I keep coming back to:

Work in thin vertical slices. This is a good idea generally, but it especially pays off when cherry-picks are in your future. A single commit/PR that changes the DB schema, the middle tier, and the UI together is one cherry-pick - either it all lands on the release branch or none of it does. Split the same bug fix across three commits (one per tier) and you now have three cherry-picks that each have to be remembered, ordered, and applied. Miss one and you ship a half-fix; the release branch compiles, the smoke test passes, and the bug is “fixed” everywhere except the layer you forgot. Your pre-commit automated tests on the release branch should catch the omission - but “should” is doing heavy lifting there, and the failure mode is exactly the kind of partial-state subtlety tests are weakest at.
Test the release branch like it’s a fresh codebase, not “trunk minus a few commits.” End-to-end, not just the change you cherry-picked.
Cherry-picks of schema/data changes need extra scrutiny. Migration logic written assuming a trunk DB state may break against a release DB state.
Prefer release-from-trunk over release-with-cherry-picks when your cadence allows it - roughly weekly or faster. If you ship every few days, a release branch buys you very little and the cherry-pick overhead isn’t worth it; just tag trunk. Cherry-picked release branches earn their keep at monthly/quarterly cadences where the stabilization window is long enough that trunk has moved on substantially. Long-lived release branches that absorb selective fixes are a structural risk, not a tooling problem git is going to grow out of.
For high-cost cherry-picks, run the full test suite on the cherry-picked branch before merge. This is what the playground demonstrates: a happy-path Selenium/Cypress/Playwright test that hits the UI and asserts on the DB shows up regressions that git diff won’t.

Reproducing this

git clone https://github.com/paul-hammant/limits-of-merging-experiment
cd limits-of-merging-experiment
./start.sh                          # set up solution/ as a fresh playground
./scenario-a-out-of-order.sh        # release: C2, C4, C3, then merge main
./rollback.sh
./scenario-b-in-order.sh            # release: C2, C3, C4, then merge main
./rollback.sh
./scenario-c-block-c3.sh            # release@C1: cherry-pick C2 + C4, BLOCK C3, sweep main

Each script prints the resulting graph, tree hash, and diff. Compare the two.

The patches in patches/ are real git format-patch output - readable, replayable, and the source of truth for the trunk timeline. The scenario scripts pin author identity and timestamps so anyone running them gets the same SHAs I did. That’s the only way to make a cherry-pick experiment reproducible.

Repeating in SVN: what `svn:mergeinfo` actually looks like

Earlier I waved at svn:mergeinfo as the thing git deliberately doesn’t have. Worth showing the property string itself, because the shape is the whole point.

The same C2–C5 timeline replayed in a fresh local SVN repo gives this revision map:

Repo rev	Meaning
r1	layout (`mkdir trunk + branches + tags`)
r2	C1 — initial Person CRUD app
r3	C2 — `hair_color` (string)
r4	C3 — UPPERCASE buttons
r5	C4 — `hair_color` INTEGER
r6	C5 — maintainer comment
r7	`svn copy /trunk@r3 /branches/release` (cut at C2)

Scenario A — cherry-pick out of order (C4 then C3, then sweep)

$ cd svn-wc/branches/release

$ svn merge -c5 ^/trunk .            # cherry-pick C4
$ svn commit -m "cherry-pick C4 from trunk@r5"
$ svn propget svn:mergeinfo .
  /trunk:5

$ svn merge -c4 ^/trunk .            # cherry-pick C3
$ svn commit -m "cherry-pick C3 from trunk@r4"
$ svn propget svn:mergeinfo .
  /trunk:4-5

$ svn merge ^/trunk .                # sweep
--- Merging r6 through r9 into '.':
U    app.rb
$ svn commit -m "sweep merge ^/trunk into release"
$ svn propget svn:mergeinfo .
  /trunk:4-9

Scenario B — cherry-pick in trunk order (C3 then C4, then sweep)

$ svn merge -c4 ^/trunk .            # cherry-pick C3
$ svn commit -m "cherry-pick C3 from trunk@r4"
$ svn propget svn:mergeinfo .
  /trunk:4

$ svn merge -c5 ^/trunk .            # cherry-pick C4
$ svn commit -m "cherry-pick C4 from trunk@r5"
$ svn propget svn:mergeinfo .
  /trunk:4-5

$ svn merge ^/trunk .                # sweep
$ svn commit -m "sweep merge ^/trunk into release"
$ svn propget svn:mergeinfo .
  /trunk:4-9

What the property is telling you

Both scenarios converge to /trunk:4-9, and svn diff ^/trunk ^/branches/release reports no file content difference — only this property exists on the branch root. The intermediate path differs (/trunk:5 → /trunk:4-5 versus /trunk:4 → /trunk:4-5), but order doesn’t matter to the end state, just like in git.

What is different from git: the sweep svn merge ^/trunk reads the property and refuses to re-apply revisions named in it. Only r6 (C5) actually produced edits in the sweep — r4 and r5 were already accounted for. SVN can answer the question “have I integrated this trunk revision yet?” because it wrote down the answer the first time. Git cannot, because git deliberately wrote nothing down.

There’s a quirk visible in the final string: /trunk:4-9, not /trunk:4-6. SVN records the closed range it considered during the sweep, including repo revisions that touched neither trunk nor any merge source — r7 was the branch copy, r8 and r9 were the cherry-pick commits themselves. This is exactly the kind of “mergeinfo creep” SVN earned its slightly-broken reputation for. It’s harmless here; it can become noisy across years of long-lived branches, particularly if anyone bypasses svn merge and edits properties by hand.

Scenario C — blocking C3 with `--record-only`

The third question worth asking: what about the change you don’t want to ship? Suppose the team’s verdict on C3 (UPPERCASE buttons) is “not for this release” — not “we’ll cherry-pick it later” but a hard no. Re-cut the release branch from C1 (so C2 also becomes a cherry-pick), then:

$ svn merge -c3 ^/trunk .                # cherry-pick C2 (the wanted feature)
$ svn merge -c5 ^/trunk .                # cherry-pick C4 (bugfix on top of C2)
$ svn merge --record-only -c4 ^/trunk .  # block C3: record but do not apply
$ svn merge ^/trunk .                    # sweep — should only pick up C5

The --record-only flag adds the revision to svn:mergeinfo without applying its diff. The property evolution:

after cherry-pick C2 (-c3):       /trunk:3
after cherry-pick C4 (-c5):       /trunk:3,5      ← non-contiguous, r4 absent
after record-only block (-c4):    /trunk:3-5      ← r4 fills in, no diff applied
after sweep:                      /trunk:3-10

The sweep consults the property, sees r4 is accounted for, and skips it. The final release tree differs from trunk only by C3’s button-text change — the block held.

The thing to notice: the post-block property string is /trunk:3-5. That is exactly what the string would say if C3 had been merged normally. It cannot tell you whether r4 was applied or blocked — only that it was considered. SVN’s mergeinfo records “we accounted for this revision,” not the intent behind that accounting. Future-you reading the property in a year has to fall back on the commit message at the block step. The data structure remembers the bookkeeping but not the decision.

Reproducing the SVN side

The scripts are on the svn-version branch of the same repo:

git checkout svn-version
sudo apt install subversion       # or your platform's equivalent
./start.sh                        # build trunk r2..r6 + release@r3
./scenario-a-out-of-order.sh      # cherry-pick C4 then C3, then sweep
./rollback.sh                     # wipe svn-repo/ and svn-wc/
./scenario-b-in-order.sh          # cherry-pick C3 then C4, then sweep
./rollback.sh
./scenario-c-block-c3.sh          # release@r2; cherry-pick C2 + C4, BLOCK C3, sweep

Each scenario script wipes and rebuilds the repo (SVN is append-only, so “reset to a past revision” means start over), prints the svn:mergeinfo value after every step, and ends with a svn diff of trunk against release. The same patches/ directory feeds both the git and SVN flows — the patches are applied with patch -p1 rather than git apply, because the SVN working copy lives inside the outer git worktree and git apply would treat it as the parent repo’s index.

So: SVN does have the audit trail, the property is human-readable, and the sweep merge is genuinely aware of it. The cost is the property’s tendency to grow ranges that include revisions it had no business including, plus the institutional discipline of never touching svn:mergeinfo directly. Whether that’s a better trade than git’s “we keep no record at all” is a judgement call about what failure mode you’d rather face — false reassurance from a slightly-wrong record, or no record and a test suite doing all the work.

Repeating in Perforce: integration records, not properties

Perforce solves the same problem SVN does — what’s been integrated where — but it stores the answer per-file in the integration database rather than as a string-property on the branch root. Run p4 integrated and the depot tells you, for every file revision on the branch, which trunk revision it came from and whether it arrived as a clean copy or a three-way merge.

Same C2–C5 timeline, replayed against a local p4d:

Changelist	Meaning
CL1	C1 — initial Person CRUD app
CL2	C2 — `hair_color` (string)
CL3	C3 — UPPERCASE buttons
CL4	C4 — `hair_color` INTEGER
CL5	C5 — maintainer comment
CL6	`p4 populate //depot/trunk/...@2 //depot/branches/release/...` (cut at C2)

Scenario A — cherry-pick out of order (C4 then C3, then sweep)

$ p4 integrate //depot/trunk/...@4,@4 //depot/branches/release/...
$ p4 resolve -am //depot/branches/release/...
$ p4 submit -d "cherry-pick C4 from trunk@CL4"
$ p4 integrated //depot/branches/release/app.rb
//depot/branches/release/app.rb#1 - branch from //depot/trunk/app.rb#1,#2
//depot/branches/release/app.rb#2 - merge from //depot/trunk/app.rb#4

$ p4 integrate //depot/trunk/...@3,@3 //depot/branches/release/...
$ p4 resolve -am //depot/branches/release/...
$ p4 submit -d "cherry-pick C3 from trunk@CL3"
$ p4 integrated //depot/branches/release/app.rb
//depot/branches/release/app.rb#1 - branch from //depot/trunk/app.rb#1,#2
//depot/branches/release/app.rb#3 - merge from //depot/trunk/app.rb#3
//depot/branches/release/app.rb#2 - merge from //depot/trunk/app.rb#4

$ p4 integrate //depot/trunk/... //depot/branches/release/...
$ p4 resolve -am //depot/branches/release/...
$ p4 submit -d "sweep merge //depot/trunk into //depot/branches/release"
$ p4 integrated //depot/branches/release/app.rb
//depot/branches/release/app.rb#1 - branch from //depot/trunk/app.rb#1,#2
//depot/branches/release/app.rb#3 - merge from //depot/trunk/app.rb#3
//depot/branches/release/app.rb#2 - merge from //depot/trunk/app.rb#4
//depot/branches/release/app.rb#4 - copy from //depot/trunk/app.rb#5

Scenario B — cherry-pick in trunk order (C3 then C4, then sweep)

$ p4 integrate //depot/trunk/...@3,@3 //depot/branches/release/...
$ p4 resolve -am //depot/branches/release/... ; p4 submit -d "cherry-pick C3"
$ p4 integrated //depot/branches/release/app.rb
//depot/branches/release/app.rb#1 - branch from //depot/trunk/app.rb#1,#2
//depot/branches/release/app.rb#2 - copy from //depot/trunk/app.rb#3

$ p4 integrate //depot/trunk/...@4,@4 //depot/branches/release/...
$ p4 resolve -am //depot/branches/release/... ; p4 submit -d "cherry-pick C4"
$ p4 integrated //depot/branches/release/app.rb
//depot/branches/release/app.rb#1 - branch from //depot/trunk/app.rb#1,#2
//depot/branches/release/app.rb#2 - copy from //depot/trunk/app.rb#3
//depot/branches/release/app.rb#3 - copy from //depot/trunk/app.rb#4

$ p4 integrate //depot/trunk/... //depot/branches/release/...
$ p4 resolve -am //depot/branches/release/... ; p4 submit -d "sweep merge"
$ p4 integrated //depot/branches/release/app.rb
//depot/branches/release/app.rb#1 - branch from //depot/trunk/app.rb#1,#2
//depot/branches/release/app.rb#2 - copy from //depot/trunk/app.rb#3
//depot/branches/release/app.rb#3 - copy from //depot/trunk/app.rb#4
//depot/branches/release/app.rb#4 - copy from //depot/trunk/app.rb#5

What the records are telling you

Same content in both scenarios — p4 diff2 //depot/trunk/... //depot/branches/release/... reports identical for every file. The two depots converged.

But the integration verbs differ in a way SVN’s mergeinfo and git’s history don’t expose at all:

Scenario A: every cherry-pick is recorded as merge from. Cherry-picking C4 onto a branch that doesn’t yet have C3 forced a three-way resolve under the hood — P4 noticed and labelled it.
Scenario B: every cherry-pick is recorded as copy from. C3 then C4 in trunk order produced clean takes on each step.

The “verb that landed me here” is part of the audit trail. If you ever investigate why a release branch file diverges from trunk, knowing whether it got there via a merge or a copy (and from which exact source revision) is the question P4 answers and the question git can’t.

The sweep p4 integrate //depot/trunk/... //depot/branches/release/... with no rev range consults the integration database and only re-applies revisions that haven’t been credited yet — r5 (C5) in our run. That’s what P4 marketing meant by “merge tracking” decades before SVN tried to bolt the same idea on with svn:mergeinfo. The cost is that it’s all server-side state, locked behind the depot — there’s no offline, no pull-request workflow, and an Unloaded depot for archival is its own ceremony. The benefit is the integration history is structured, queryable per-file, and never gets out of sync with what was actually integrated.

Scenario C — blocking C3 with `resolve -ay`

Same workflow as the SVN scenario C, branched from C1: cherry-pick C2, cherry-pick C4, block C3, sweep.

$ p4 integrate //depot/trunk/...@3,@3 //depot/branches/release/...
$ p4 resolve -ay //depot/branches/release/...     # accept yours = keep target, ignore source
$ p4 submit -d "block C3: integrate + accept-yours of trunk@CL3 (no diff)"
$ p4 integrated //depot/branches/release/app.rb
//depot/branches/release/app.rb#1 - branch  from //depot/trunk/app.rb#1
//depot/branches/release/app.rb#2 - copy    from //depot/trunk/app.rb#2     (C2)
//depot/branches/release/app.rb#3 - merge   from //depot/trunk/app.rb#4     (C4)
//depot/branches/release/app.rb#4 - ignored //depot/trunk/app.rb#3          (C3 — blocked)

The integration verb is ignored. P4’s per-file integration database has three states for any source revision: branch/copy/merge (it landed) or ignored (it was considered and rejected). After the sweep:

//depot/branches/release/app.rb#5 - merge   from //depot/trunk/app.rb#5     (C5)

C3 stays ignored. The block held, and the audit trail says, in machine-readable form, what happened.

The thing to notice: this is the cleanest case where P4 carries strictly more information than SVN. SVN’s svn:mergeinfo records that a revision was accounted for; P4’s integration database records whether the bytes were taken or rejected. Asked “did anyone consider C3 for this release?” the SVN string says yes; the P4 record says yes and tells you the answer was “no, ignored.”

Reproducing the Perforce side

Scripts are on the perforce-version branch:

git checkout perforce-version
# install p4 + p4d — see https://www.perforce.com/downloads/helix-core
# (or the primer at github.com/paul-hammant/fast_perforce_setup)
./start.sh                       # build trunk CL1..CL5 + release@CL2 on a sandbox p4d
./scenario-a-out-of-order.sh     # cherry-pick C4 then C3, then sweep
./rollback.sh                    # stop p4d, wipe p4-server/ and p4-wc/
./scenario-b-in-order.sh         # cherry-pick C3 then C4, then sweep
./rollback.sh
./scenario-c-block-c3.sh         # release@CL1; cherry-pick C2 + C4, BLOCK C3, sweep

The sandbox runs p4d on localhost:1667 (not the conventional 1666), with no SSL and no security level set, so no passwords. Everything lives under p4-server and p4-wc; both are wiped on every run. Patches are applied with patch -p1 for the same reason as the SVN scripts — git apply would notice the outer git worktree and refuse to write.

Closing

So which VCS “knows what’s been integrated”?

	Records integration history?	Where it lives	What it records	Distinguishes “applied” from “blocked”?
Git	No	Nowhere (optional `(cherry picked from …)` comment, never read)	Nothing machine-checkable	No
SVN	Yes	`svn:mergeinfo` property on branch root	A revision-range string, e.g. `/trunk:4-9`	No — both look the same in mergeinfo
Perforce	Yes	Per-file integration database	Source path, source rev, integration verb (`branch`, `copy`, `merge`, `ignored`)	Yes — the `ignored` verb

All three converge on the same source tree when the underlying patches don’t conflict. The difference is what the tool can tell you afterwards about how that tree got built — and therefore what kinds of “did the cherry-pick land safely?” questions you can ask the tool versus answer with tests.

Cherry-pick is not infallible. It also isn’t usually wrong. The hazard is the gap between those two facts: git’s vocabulary for “this is the same change” is “this is byte-identical bytes,” and outside of that narrow case, it has no opinion. SVN tried to fill that gap and is imperfect in it implementation. Git decided the mess wasn’t worth it.

What this means in practice for trunk-based development with release branches: cherry-pick is a tool that requires you to bring your own audit. The audit is your test suite, your code review, and your CI. If those are weak, cherry-pick is a foot-gun. If they’re strong, cherry-pick is fine - and the SHA divergence between trunk and release is just bookkeeping noise.

So per my colleagues nudge - cherry picks are great, but be careful and know the limits.

One last thing: I’ve been in engineering leadership previously for a 12 planned releases a year team, and sometimes late features were merged to the release branch using cherry-pick. Don’t do that - wait for the next release. I lost the argument cos the business really really wanted those late features - more than once from the for the same component of the system. Chery pick is for release stabilizing things only: toggle off your work that isn’t ready to go live before the branch cut moment.

Updates:

May 7, 2026: add scenario-c. Also svn and perforce branches.

Live Verify

2026-03-17T00:00:00+00:00

Nobody cares about a thing in the tech world until it is successful, and I have something here that could be, but isn’t yet. Its a brand new idea and has an immense chasm to cross. Most of it is an adoption/patronage chasm, but there are technical challenges too.

I’ve blogged before on this enough to have timed out any change of filing a patent on it. So there’s no way I could make a SaaS that would have some exclusivity to operate based on patent. Not that patents really protect business ideas these days anyway.

Live verify links

GitHub home page: live-verify.github.io/live-verify/
GitHub Repo: github.com/live-verify/live-verify

This is me and ClaudeCode mostly. Me reminding Claude that I really really like automated tests.

The home page has screenshots of live-verify in action (before, after verification, trust statement), and some real videos of tests running. Some of those videos are me showing camera apps, and some are voiceless screen recordings of a larger multi-tier simulation of all the tech pieces. The home page also links to 480 or so use cases (and has a search feature).

What is it?

A system to verify a document or part of a document in your hands that might be way away from the system that produced it. And verify means a claim within - could be just a rectangle of text mid way through the document. Could be 20 rectangles and 20 verifications. The could also be from something that’s printed and in front of you, using a camera-using phone app. Perhaps only for small rectangles of text. Most of the time you’ve the document as a digital document. Say a PDF or a web page. In that case the verification is via a Chrome extension. In time the same tech is built into Adobe, Outlook, MsWord etc. At some point it is in the OS and the need for a chrome extension disappears.

There’s 480 use cases that are listed. Maybe that’s only 280 after consolidation. Uses cases are anti-fraud, safety, and accountability. To say more, use cases cover preventing document fraud (fake certificates, forged receipts), verifying safety credentials (building inspectors, healthcare workers, equipment compliance), and enforcing accountability (audit trails, regulatory compliance, chain of custody). “Accountability” is a catch-all though. It captures the compliance, ethics, audit, and delegated-authority themes that don’t fit neatly under fraud or safety.

Most of the time it is about reducing costs of things by rolling back fraud, because fraudulent claims would now be easier and quicker to identify and avoid the consequences of. Yes, “claim” in an insurance thing, but I’m talking about more general use like this claim “I, Jimmy Cricket, worked at Microsoft in Seattle from 2010 to 2013 in the DevOps team”, or “Mr Alex E Spooner earned his millions from his family’s corner shop and not from selling drugs at all. He would make a great investor”. Other uses, could be ID systems with a camera app confirming the person with the eInk badge-ID is actually a police officer. That’d need rules of conduct for those verification moments.

The system rests on Human eyes being quickly able to scan a claim, and wonder “I would trust this if only there was a way of verifying it as true”, then a button press and some maths suggesting it is true (is “verified”) and a chain of trust that human eyes can also quickly scan toward another trust or not trust decision. And a system that’s much more trustworthy than presentation and use of QR code.

No personal data ever leaves your device. The text is captured, normalized, and hashed entirely on-device. Only the hash — a one-way fingerprint that reveals nothing about the document — is sent over the network. The verification endpoint never sees your degree certificate, salary receipt, or passport. Tens of thousands of cryptography PhDs could testify in court that hashing is irreversible.

The maths is hashing - SHA256 by default, but could be stronger or weaker. Some education needed for ordinary people to broadly understand why it is one-way. The hashes are all in public and not indexed. You know the hash, you can see the payload (default would be { "status":"verified" }). You don’t know the hash, you can’t discover it, subject to the plain-text’s entropy.

Revocation is built in. “Verified” isn’t permanent. An issuer can change the status to revoked, suspended, or expired at any time — just update what the endpoint returns. A doctor loses their license, a certificate is superseded, an employee is terminated: the next person who verifies sees the current status, not a stale “OK.” This is what static digital signatures can’t do.

Trust rests on the domain, not the app. There’s no central authority or certificate registry. You decide whether ed.ac.uk is an authority for Edinburgh degrees, or whether coned.com is an authority for utility worker badges. The verify: line on the document tells you whose domain is making the claim — and that domain can declare who authorized it. For example, in the automated tests there’s a James Whitfield bank statement from Meridian National Bank with verify:meridian-national.bank.us/statements. The extension verifies the hash, then walks the authority chain: Meridian National Bank says it’s authorized by the FDIC (fdic.gov), which in turn is authorized by the US Treasury (treasury.gov). The verifier sees the full chain and decides whether to trust it.

So we have two camera-using phone apps, and a Chrome extension. Ideally we’d go on to make some plugin for Acrobat and Outlook and more. The Chrome extension is working well, but the camera-apps have upper limits. One is size of “document” to be verified which is understandable as OCR though a camera lens isn’t perfect. The other is tabular data - read on for more on that.

I wish there was more shared code, between the implementations:

Platform	Normalization	Hashing
Web app (`public/normalize.js`)	JavaScript (canonical)	Web Crypto API
Chrome extension (`shared/normalize.js`)	JavaScript (auto-generated copy via `scripts/sync-shared.js`)	Web Crypto API
iOS app	JavaScript via JSBridge (runs normalize.js directly)	CryptoKit (native Swift)
Android app	JavaScript via Rhino (runs transpiled normalize.js)	Native `MessageDigest`

The web-app version was were I started with this some months ago. It used Tesseract OCR and would run online just asking for the camera. Tesseract (via WASM) was really problematic so I shifted to proper apps ahead of schedule.

I also have a reference backend that we use in the built-in automated tests. The possibly hundreds of SaaS companies that supply into this space may make different choices. The value is the open standard here.

Post-Verification Actions

Verification doesn’t have to stop at “verified.” There are two sources of follow-up actions.

The first is the app or phone itself. If you’re scanning a coffee shop receipt and you have Expensify installed, the app can offer “Send to Expensify?” — that’s a client-side decision based on context, and a choice for you to accept or ignore. The receipt issuer doesn’t know or control what’s offered. The second is the verification endpoint deliberately returning actions in its response. A building inspector’s badge could offer a form for the homeowner to record visit details (areas accessed, duration, any concerns). A lawyer’s credentials could link to the bar association’s public disciplinary record. These are issuer choices — the endpoint decides what to offer.

The endpoint pattern scales from light (a link) to strong (a POST form for reporting). The strong version matters where there’s a power dynamic — an inspector at your door, a healthcare worker with a vulnerable patient. The verification response tells the verifier “you may record details of this interaction” and explicitly says they will never be told not to.

Current Tech Problems

Live Verify’s camera mode works beautifully for prose documents — certificates, references, claims where text flows continuously line by line. Point your phone, tap verify, done.

But tabular data — receipts, invoices, anything with left-aligned descriptions and right-aligned prices — breaks the pipeline. Many of the use cases involve tabular data.

What Works: Prose

An employment reference like this verifies perfectly:

I, Paul Hammant, worked for Kevin Behr in
his role as CIO of HedgeServ in New York City
in 2015 and 2016
verify:paulhammant.com/refs

The text flows left-to-right with no gaps. The OCR engine (Apple Vision on iOS, Google ML Kit on Android) sees one contiguous block of text. Verification succeeds.

What Breaks: Tabular Receipts

A coffee shop receipt like this fails:

Flat White                £3.40
Almond Croissant          £3.25
SUBTOTAL:                 £6.65

Where a human sees one line, the OCR engine sees two separate text blocks — “Flat White” and “£3.40” — because the visual gap signals “these are separate regions.” A receipt that should be one block becomes 10+ fragments. Both iOS and Android camera apps I have made attempting to use Live-Text features of the OS show up to 10 fragments. And that’s the best case sometimes there’s 7 and some crucial numbers on the right hand side are omitted completely.

Where This Leaves Us

Document Type	Clip Mode (Browser)	Camera (Android)	Camera (iOS)
Short prose (peer references, badge claims)	Works	Works	Works
Longer prose (certificates, full letters)	Works	Works (OCR errors creep in)	Works (OCR errors creep in)
Tabular (receipts, invoices)	Works	Works (stitching)	Broken (single-rectangle)

Clip mode — the browser extension — handles everything perfectly because it operates on digital text, not pixels. No OCR, no rectangle fragmentation.

The Real Fix: Registration Marks for Tabular Data

Our stitching on Android is a workaround. It handles the common case but it’s fragile — font size changes, multi-column layouts, and unusual spacing can all defeat Y-coordinate grouping.

The proper solution is for Apple and Google to support registration marks for tabular data in their OCR APIs. Two marks at diagonal corners define a bounding rectangle. The OCR engine sees the marks, treats everything inside as one text block, and strips the marks from the output — like a QR finder pattern that the camera uses for orientation but doesn’t include in the payload.

We already have one mark: the vfy: line at the bottom-left of every verifiable region. A Unicode corner character — ⌝ (U+231D, upper right corner) — on the first line at the right margin provides the opposing corner:

8 Market Square                 ⌝
Henley-on-Thames RG9 2AA
Flat White                  £3.40
Almond Croissant            £3.25
SUBTOTAL:                   £6.65
vfy:r.the-daily-grind.co.uk

It’s already in the pic I took above.

The ⌝ is a control mark, not content. The OCR engine consumes it for bounds detection and omits it from the text output, just as it would omit a QR finder square. The vfy: line stays — it’s already part of the verification protocol and already in the OCR output.

Why a Unicode character rather than an image or a special printed mark? Because it works everywhere text works: HTML, PDF, LaTeX, Word, thermal receipt printers. Any system that can render U+231D can print the mark. No image embedding, no special font, no binary format dependency.

This is future work — it requires Apple and Google to recognize the ⌝ + vfy: pair as a single region boundary in their Vision and ML Kit frameworks. But the convention is simple, the marks are unobtrusive on the printed document, and the benefit is large: receipts, invoices, lab results, bank statements, and every other tabular document becomes verifiable by camera.

Until then, camera apps work better for prose, and clip mode works for everything - including most of the anti-fraud cases.

Did You Send This - for module phone SMS/Voice

2025-11-30T00:00:00+00:00

This article is a comprehensive 2025 update to the original 2006 post: “Did you send this - another weapon against spam?”

Applying DYST to Mobile Communication

The original DYST idea could theoretically be adapted by Google and Apple, who could implement it unilaterally for mobile calls and messages. Rather than relying on telecommunication carriers, they could use their existing TCP/IP infrastructure for verification. An app on the sending device would get a temporary token from an Apple/Google server, and the receiving device would check that token with the same central server before alerting the user.

DYST supporting phone or (iOS or Android)

Here is a sequence diagram illustrating the back-channel exchange for a genuine call

sequenceDiagram
    participant Alice as Alice's Handset<br>(Originator)
    participant AliceTeleco as Alice's<br>Teleco
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Alice->>AliceTeleco: 1. Alice actually<br>Initiates call
    AliceTeleco->>Bob: Call Signalling
    activate Bob
    Note over Bob: Receives signalling,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is fully DYST-enabled.
    Alliance->>Alice: 3. DYST Challenge
    activate Alice
    Note over Alice: Alice does not sees<br/>"Are you placing a call<br>to <bobs phone numb>"<br>question. And handset<br>silently confirms she is
    Alice-->>Alliance: 4. Challenge Response (Confirmed)
    deactivate Alice
    Note over Alliance: Alice's auto<br>confirmation received.
    Alliance->>Bob: 5. Verification Success
    deactivate Alliance
    activate Bob
    Note over Bob: Now rings or displays<br>message. Call/Text<br>connection established.<br/>Phone buzzes or rings
    deactivate Bob

If Bob is in the address book, then <bobs phone numb> is replaced with Bob’s name

Here is a sequence diagram illustrating the back-channel exchange for a genuine Apple/Google “Messages” (not SMS)

sequenceDiagram
    participant Alice as Alice's Handset<br>(Originator)
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Alice->>Bob: 1. Alice actually<br>sends text msg
    activate Bob
    Note over Bob: Receives signalling,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is fully DYST-enabled.
    Alliance->>Alice: 3. DYST Challenge
    activate Alice
    Note over Alice: Alice does not see<br/>"Did you send this<br>to <bob phone num>"<br>question. And handset<br>silently confirms she did
    Alice-->>Alliance: 4. Challenge Response (Confirmed)
    deactivate Alice
    Note over Alliance: Alice's auto<br>confirmation received.
    Alliance->>Bob: 5. Verification Success
    deactivate Alliance
    activate Bob
    Note over Bob: Now rings or displays<br>message. Call/Text<br>connection established.<br/>Phone buzzes
    deactivate Bob

Apple launched the iPhone with iMessage (for other iPhone users). Google did the same for Android uses, and these days Rich Communication Services (RCS) allows both to interoperate. In the diagram above the teleco’s systems are not shown because they are not involved for the routing of “smart” (non SMS) messages.

Here is a sequence diagram illustrating the back-channel exchange for a fake call

sequenceDiagram
    participant Eve as Eve's Handset<br>(Originator)
    participant EveVOIP as Eve's VoIP<br>Gateway
    participant Alice as Alice's Handset<br>(Originator)
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Eve->>EveVOIP: 1. Eve actually<br>Initiates call<br>caller-id = Alice
    EveVOIP->>Bob: Call Signalling
    activate Bob
    Note over Bob: Receives signalling,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is fully DYST-enabled.
    Alliance->>Alice: 3. DYST Challenge
    activate Alice
    Note over Alice: Alice does not see<br/>"Are you placing a call<br>to <bobs phone num>"<br>message. And handset<br>silently DENIES she is
    Alice-->>Alliance: 4. Challenge Response (Denied)
    deactivate Alice
    Note over Alliance: Alice's auto<br>denial received.
    Alliance->>Bob: 5. Verification Failure
    deactivate Alliance
    activate Bob
    Note over Bob: Drops the call without<br>notifying Bob and doesnt<br>place it in recent calls
    deactivate Bob

The sequence diagram for messages would look very similar.

Smartphone supporting RCS but not yet supporting DYST (older iOS or Android version perhaps)

Legit call via RCS (no DYST)

sequenceDiagram
    participant Alice as Alice's Handset<br>(Originator)
    participant AliceTeleco as Alice's<br>Teleco
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Alice->>AliceTeleco: 1. Alice actually<br>Initiates call
    AliceTeleco->>Bob: Call Signalling
    activate Bob
    Note over Bob: Receives signalling,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is RCS-enabled but not<br/>DYST-enabled.
    Alliance->>Alice: 3. DYST Challenge (RCS)
    activate Alice
    Note over Alice: Alice sees interactive<br/>"Are you placing a call<br>to <bobs phone numb>"<br>question. And she<br>manually confirms.
    Alice-->>Alliance: 4. Challenge Response (Confirmed)
    deactivate Alice
    Note over Alliance: Alice's manual<br>confirmation received.
    Alliance->>Bob: 5. Verification Success
    deactivate Alliance
    activate Bob
    Note over Bob: Now rings or displays<br>message. Call/Text<br>connection established.<br/>Phone buzzes or rings
    deactivate Bob

Legit message via RCS (no DYST)

sequenceDiagram
    participant Alice as Alice's Handset<br>(Originator)
    participant AliceTeleco as Alice's<br>Teleco
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Alice->>AliceTeleco: 1. Alice actually<br>sends text msg
    AliceTeleco->>Bob: Message signalling
    activate Bob
    Note over Bob: Receives signalling,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is RCS-enabled but not<br/>DYST-enabled.
    Alliance->>Alice: 3. DYST Challenge (RCS)
    activate Alice
    Note over Alice: Alice sees interactive<br/>"Did you send this<br>to <bob phone num>"<br>question. And she<br>manually confirms.
    Alice-->>Alliance: 4. Challenge Response (Confirmed)
    deactivate Alice
    Note over Alliance: Alice's manual<br>confirmation received.
    Alliance->>Bob: 5. Verification Success
    deactivate Alliance
    activate Bob
    Note over Bob: Now rings or displays<br>message. Call/Text<br>connection established.<br/>Phone buzzes
    deactivate Bob

Fake caller ID call via RCS (no DYST)

sequenceDiagram
    participant Eve as Eve's Handset<br>(Originator)
    participant EveVOIP as Eve's VoIP<br>Gateway
    participant Alice as Alice's Handset<br>(Originator)
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Eve->>EveVOIP: 1. Eve actually<br>Initiates call<br>caller-id = Alice
    EveVOIP->>Bob: Call Signalling
    activate Bob
    Note over Bob: Receives signalling,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is RCS-enabled but not<br/>DYST-enabled.
    Alliance->>Alice: 3. DYST Challenge (RCS)
    activate Alice
    Note over Alice: Alice sees interactive<br/>"Are you placing a call<br>to <bobs phone num>"<br>message. And she<br>manually DENIES.
    Alice-->>Alliance: 4. Challenge Response (Denied)
    deactivate Alice
    Note over Alliance: Alice's manual<br>denial received.
    Alliance->>Bob: 5. Verification Failure
    deactivate Alliance
    activate Bob
    Note over Bob: Drops the call without<br>notifying Bob and doesnt<br>place it in recent calls
    deactivate Bob

Phone not supporting RCS nor DYST (much older iOS or Android version perhaps)

Legit call via SMS (no RCS, no DYST)

sequenceDiagram
    participant Alice as Alice's Handset<br>(Originator)
    participant AliceTeleco as Alice's<br>Teleco
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Alice->>AliceTeleco: 1. Alice actually<br>Initiates call
    AliceTeleco->>Bob: Call Signalling
    activate Bob
    Note over Bob: Receives signalling,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is not RCS or DYST enabled.<br>Will use SMS.
    Alliance->>Alice: 3. DYST Challenge (SMS)
    activate Alice
    Note over Alice: Alice sees SMS asking if<br/>she's calling<br><bobs phone num>.<br/>She replies 'YES'.
    Alice-->>Alliance: 4. Challenge Response (Confirmed)
    deactivate Alice
    Note over Alliance: Alice's SMS reply<br>of 'YES' received.
    Alliance->>Bob: 5. Verification Success
    deactivate Alliance
    activate Bob
    Note over Bob: Now rings or displays<br>message. Call/Text<br>connection established.<br/>Phone buzzes or rings
    deactivate Bob

Legit message via SMS (no RCS, no DYST)

sequenceDiagram
    participant Alice as Alice's Handset<br>(Originator)
    participant AliceTeleco as Alice's<br>Teleco
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Alice->>AliceTeleco: 1. Alice actually<br>sends SMS msg
    AliceTeleco->>Bob: SMS received
    activate Bob
    Note over Bob: Receives SMS,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is not RCS or DYST enabled.<br>Will use SMS.
    Alliance->>Alice: 3. DYST Challenge (SMS)
    activate Alice
    Note over Alice: Alice sees SMS asking if<br/>she sent message to<br><bobs phone num>.<br/>She replies 'YES'.
    Alice-->>Alliance: 4. Challenge Response (Confirmed)
    deactivate Alice
    Note over Alliance: Alice's SMS reply<br>of 'YES' received.
    Alliance->>Bob: 5. Verification Success
    deactivate Alliance
    activate Bob
    Note over Bob: Now displays message.<br/>Phone buzzes
    deactivate Bob

Fake caller ID call via SMS (no RCS, no DYST)

sequenceDiagram
    participant Eve as Eve's Handset<br>(Originator)
    participant EveVOIP as Eve's VoIP<br>Gateway
    participant Alice as Alice's Handset<br>(Originator)
    participant Bob as Bob's Handset<br>(Recipient)
    participant Alliance as Spam Alliance<br>Backend

    Eve->>EveVOIP: 1. Eve actually<br>Initiates call<br>caller-id = Alice
    EveVOIP->>Bob: Call Signalling
    activate Bob
    Note over Bob: Receives signalling,<br/>waits for verification.

    Bob->>Alliance: 2. Verification Request
    deactivate Bob
    activate Alliance

    Note over Alliance: Determines Alice's handset<br/>is not RCS or DYST enabled.<br>Will use SMS.
    Alliance->>Alice: 3. DYST Challenge (SMS)
    activate Alice
    Note over Alice: Alice sees SMS asking if<br/>she is calling <bobs phone num>.<br/>She ignores it or replies 'NO'.
    Alice-->>Alliance: 4. Challenge Response (Denied)
    deactivate Alice
    Note over Alliance: Alice's SMS reply<br>of 'NO' received (or timeout).
    Alliance->>Bob: 5. Verification Failure
    deactivate Alliance
    activate Bob
    Note over Bob: Drops the call without<br>notifying Bob and doesnt<br>place it in recent calls
    deactivate Bob

Of course a text reply of “No” would be inferred for anyting other that ‘Y’, y, YES, 是, 对, हाँ (haan), sí, oui, نعم, হ্যাঁ , da, sim, ہاں, ya, ja, はい, ndiyo, हो , అవును , evet, vâng / dạ

This system could be enhanced with network intelligence. For IP-based messages, the receiving service’s backend (e.g., Apple’s servers) could perform a traceroute back to the sender’s IP. This wouldn’t reveal the user’s precise location but would identify the network path. A path from a reputable mobile carrier like O2 in the UK would contribute to a “low-probability spammer” score, whereas a path from a data center known for fraudulent activity would raise suspicion, all without exposing the sender’s private location.

A user could configure their device to “block all unverified calls”, “warn”, or perform no checks. However, limitations of this solution would including the risk of blocking legitimate calls if the verification service has an outage and the small delay the check adds to every communication. To counter denial-of-service attacks where bad actors flood users with fake verification messages to create fatigue and distrust, the system would rely on server-side intelligence from the central servers to detect and rate-limit anomalous floods of verification requests.

A challenge remains communication with “legacy users” on older devices. While most phones would handle the fallback SMS correctly, older feature phones (“dumbphones”) in particular might display the verification message as multiple confusing parts or silently drop it due to full inboxes, undermining the system. To build trust for this fallback, a logical conclusion would be for participating companies to form a “Spam Alliance”, complete with a website listing members, and brand the message with this neutral alliance name.

2026 footnote: The DYST challenge-response pattern — “did you send this?” queried back to the originator’s domain — is the same model underpinning Live Verify (source), but for documents instead of calls. A bank statement, police ID card, or sanctions attestation is normalized, SHA-256 hashed, and looked up at the issuer’s domain: GET /v/{hash} — effectively asking the issuer “did you issue this?” The domain-as-authority principle is identical: the originating server is the source of truth, no central registry needed. Where DYST proposes a Spam Alliance coordinating telcos, Live Verify proposes authority chains — issuer endorsed by regulator endorsed by sovereign root — to establish trust without a single coordinating body.

The telcos’ resistance would likely be because their large customers, such as call centers using VoIP, would face significant hurdles. Much of their software may be old and difficult to update to support a new verification protocol. A call center vendor might even configure their system to automatically answer “yes” to all challenges, legitimate or not. This would necessitate another layer of defense, led by the Spam Alliance, to apply reputation scoring to the responders themselves. If an endpoint blindly says “yes” to everything, its attestations would eventually be deemed untrustworthy.

Finally, it is instructive to look at the history of DMARC, SPF, and DKIM for email. These technologies did not eliminate spam, but they largely solved the problem of direct domain spoofing. Similarly, a DYST-like system for voice and SMS would not be a silver bullet for all unwanted communication, but it could effectively end caller-ID spoofing of legitimate numbers, forcing bad actors onto more easily traceable and blockable channels.

Example DYST Messages

As RCS (Rich Communication Services) supports rich cards and suggested actions, the user experience for a DYST-style verification could be significantly improved over a plain SMS fallback.

Challenge (JSON payload) to claimed originator who has a DYST-enabled Smartphone

When the claimed originator’s Smartphone is DYST-enabled, the verification happens silently in the background between the originating service’s server and the handset. This interaction uses a JSON payload, and the handset automatically answers the challenge without user intervention.

Recieving handset sends this JSON payload to the DYST-enabled handset of the claimed originator:

{
  "dyst_challenge": {
    "protocol_version": "1.0",
    "challenge_id": "dyst-challenge-12345",
    "timestamp": "2025-11-29T10:30:00Z",
    "originator_id": "+14155550110",
    "recipient_id": "+12025550148",
    "communication_type": "voice_call",
    "communication_id": "call-abc-789",
    "message_digest": "sha256:abcdef12345...",
    "ttl_seconds": 30
  }
}

Claimed originators DYST-enabled handset responds silently:

{
  "dyst_response": {
    "challenge_id": "dyst-challenge-12345",
    "status": "confirmed",
  }
}

Or if they had not placed the voice call (someone else faked caller ID):

{
  "dyst_response": {
    "challenge_id": "dyst-challenge-12345",
    "status": "denied",
  }
}

Challenge RCS message to claimed originator who has a Smartphone that’s not DYST enabled (a fallback)

This is what the person making the call would receive on their device if the recipient doesn’t recognize them. It’s designed to be simple and quick.

Spam Alliance Verification ✓
---------------------------------------
🛡️ Did you just try to contact `+1-202-555-0148`?

To connect your call, please confirm it was you.

[ ✅ Yes, that was me ]  [ ❌ No, not me ]
---------------------------------------
<small>Sent by the Spam Alliance. [Learn More]</small>

Challenge SMS message to claimed originator with a non-RCS phone

If the originator’s phone does not support RCS (e.g., a “dumbphone”), the system must fall back to plain SMS. This experience is the most basic and relies on the user to manually reply.

Spam Alliance: Did you just try to contact +1-202-555-0148? To connect, reply YES to this message.

Language Selection for a Global DYST System

For a global anti-spam system to be effective, it must communicate with users in their native language. The DYST system would handle this differently depending on the originator’s device capabilities.

1. First-Class (DYST-enabled Handset)

This is the simplest scenario. The silent JSON payload exchanged between the services is machine-readable and language-agnostic. If the originator’s handset needs to display any notification related to the verification, it uses its own local OS language setting to do so. No language information needs to be transmitted in the challenge itself.

2. RCS Fallback (non-DYST Smartphone)

When falling back to a user-facing RCS message, the system must send the challenge in the correct language. This would be solved by having the originator’s device report its language preference (e.g., ‘es-MX’ for Mexican Spanish) as part of its standard RCS capabilities. The Spam Alliance’s service would then deliver a pre-translated, interactive message in that specific language.

3. SMS Fallback (non-RCS phone)

This is the most challenging scenario. Like the RCS fallback, the system would attempt to determine the device’s language. However, without the rich capabilities of an RCS client, this may not be possible. As a last resort, the system would have to make an educated guess based on the phone number’s country code, which is less reliable but better than defaulting to a single language like English.

Modern CV Technology: JSON Resume embedded in HTML

2025-10-12T00:00:00+00:00

Problem: uploading your resume/CV to a job portal should yield a perfectly parsed CV but often does not. This is true even if your template .docx is a claimed good starting point for later ingesting into such systems.

Building the Future of Digital Resumes: A Technical Deep Dive

In an era where Applicant Tracking Systems (ATS) and job portals increasingly dominate the recruitment landscape, the traditional PDF resume is showing its age. What if we could create a resume format that’s simultaneously machine-readable, human-friendly, and completely self-contained? Well, that ended up being a side project - read on.

The Problem with Traditional Resumes

Traditional resume formats present a fundamental challenge:

PDFs look great but are difficult for ATS systems to parse accurately, even in the nascent AI era.
Word documents are editable but inconsistent across platforms
Plain text is machine-readable but lacks visual appeal
Linked In would like to own this space, but they’re way too much lock in, and spend too much trying to keep your engagement in their pages, vs get you a job.

The Solution: JSON Resume Schema + Interactive HTML

This repository showcases a evolutionary approach that combines the best of two words - mnachine parsable and appealing to human eyes. The raw storage of the CV/resume data:

Every resume uses the official JSON Resume standard:

{
  "$schema": "https://raw.githubusercontent.com/jsonresume/resume-schema/v1.0.0/schema.json",
  "basics": { "contact": "info and summary elemnts" },
  "work": [ { "employment": "history blah blah blah" } ],
  "education": [ { "degrees": "and certifications" } ],
  "skills": [ { "technical": "abilities yada yada"} ],
  "and": "ten more standardized sections"
}

Each resume/CV is a single HTML file containing:

Embedded JSON data above in a <script> tag for ATS extraction
Inlined CSS (~1000 lines) for complete visual control
Inlined JavaScript (~1200 lines) for interactive features
Zero external dependencies for viewing (optional CDN for PDF generation)

The same file serves two masters:

Machines: Extract structured JSON data for database import. It could have as easily been XML or YAML, but JSON parsing in web pages is built in.
Humans: Styled and responsive HTML with some interactive features

Some interactivity to control verbosity

You can click a pen to go into edit mode. Editing isn’t text, it is contract (-) and expand (+) affordances to reduce the verbosity of sections. While I may gush about the origin story of Selenium an in-firm recruiter, then agent who’s taken my CV to them, the downstream interviewers are spectacularly uninterested in that so they’ll hit (-) to collapse that section. This will persist if you go on to print the resume/CV, but not if you close the tab.

Collapse affordance shown:

That clicked, expand affordance shown:

This is only useful for someone wanting customize the document for a purpose. For example, and interview stage with the candidate or a discussion with a colleague about take forward in the process or decline.

Responsive Design with Print Optimization

The CSS includes specialized media queries to ensures URLs are visible in printed versions—crucial for ATS systems and recruiters. Versus just hyperlinks when that’s in a browser. No big deal perhaps.

PDF Generation Pipeline

For enhanced PDF output, the system dynamically loads:

html2canvas: Renders HTML to canvas with pixel-perfect accuracy
jsPDF: Converts canvas to professional PDF format

Ignoring this lazy load of JavaScript from CDNs, this was otherwise a zero dependency tech.

Sample CVs in the repository

Current collection includes 14 example resumes featuring fictional characters:

Technical roles: Tony Stark (Genius/Inventor), Harold Finch (Software Engineer)
Leadership positions: T’Challa (Head of State), Princess Leia (Rebel Leader)
Diverse backgrounds: Hermione Granger (Academic), Mulan (Military Officer)

Take a look at Sam “Root” Groves from “Person of Interest” TV series; paul-hammant.github.io/better-cv-tech/Samantha_Groves_Resume.html

Implementation Highlights

Markdown Support Within JSON

The system supports basic markdown in key text fields:

**bold text** and *italic text*
[link text](https://example.com) for clickable links
Paragraph breaks with \n\n

This enhances human readability while maintaining ATS compatibility. Well, maybe.

ATS Integration Strategy

For ATS systems to adopt this format, they need minimal changes:

// Extract resume data from HTML
const resumeScript = document.getElementById('cv-data-json');
const resumeData = JSON.parse(resumeScript.textContent);
// Now import structured data directly into database

Likely they’d just be snipping out of the unparsed source file though. That would go into their databases, but also maybe a pipeline that uses a JSON Resume to HTML pipeline. Either way, this is orders of magnitude more reliable than PDF or .docx text extraction or HTML scraping. Even with claimed AI on their side in 2025

Performance Characteristics

File size: ~80KB per resume (including all assets)
Load time: Near instant (no external requests for viewing)
Browser support: Modern browsers with JavaScript turned on (ES6+ required)
Mobile responsive: Breakpoints at 768px and 480px

Security Considerations

For recruiters receiving HTML resumes:

✅ No file system access - Pure DOM manipulation
✅ No external requests - Self-contained execution
✅ Standard JavaScript - No eval() or dangerous APIs
✅ Data transparency - JSON visible in source

Likely there will be some over-cautiousness. Bigger companies could verify the JavaScript within each uploaded CV/resume if they really wanted to. A whitelist of sorts (extract > lint > pretty-print > SHA256 > check against whitelist). Or just

Getting Started

To create your own resume:

Copy a template: Use Lorem_Ipsum_Resume.html as your starting point
Replace JSON data: Update the embedded resume data with your information
Inline css and javascript: They are separate in the repo, as there is fourteen or so sample CV/resumes.
Test thoroughly: Verify print output and mobile responsiveness
Name appropriately: Use FirstName_LastName_Resume.html format

Or get AI to take the constituent pieces and make the page for you. It did so for mine in a couple of mins. Prompt is in the repo.

Links:

Repository: paul-hammant/better-cv-tech
Live Demo: GitHub Pages Gallery - 14 resume/CVs
Schema: JSON Resume v1.0.0

Building a Secure Container Sandbox on ChromeOS for Testing Untrusted Code

2025-09-18T00:00:00+00:00

The Problem: Running Random GitHub Code Safely

As developers, we frequently encounter interesting GitHub repositories, development tools, or scripts that we want to test. However, running` untrusted code directly on our development machines poses significant security risks:

Supply chain attacks: Malicious code that modifies system binaries or installs backdoors (like the 2025 Chalk npm package compromise that affected millions of downloads)
Data theft: Scripts that scan for SSH keys (passphrase protected or not), API tokens, or sensitive files
System compromise: Privilege escalation attacks that gain persistent access
Resource abuse: Cryptocurrency miners or botnet participation

Traditional solutions like virtual machines are heavyweight and slow to reset. Docker provides isolation but shares the kernel and can be escaped. ChromeOS’s unique architecture offers a compelling alternative through its layered container system.

ChromeOS Container Architecture: Defense in Depth

ChromeOS provides multiple layers of isolation that make it ideal for secure sandboxing:

┌────────────────────────────────────────-─────┐
│              ChromeOS (Host)                 │
│  ┌─────────────────────────────────────────┐ │
│  │           Termina VM                    │ │
│  │  ┌─────────────┐  ┌─────────────────────┤ │
│  │  │   Penguin   │  │        OSS          │ │
│  │  │  (Trusted)  │  │   (Untrusted)       │ │
│  │  │             │  │                     │ │
│  │  │ - Your work │  │ - Random GitHub     │ │
│  │  │ - SSH keys  │  │   repositories      │ │
│  │  │ - Configs   │  │ - Untested tools    │ │
│  │  │             │  │ - No sensitive      │ │
│  │  │             │  │   data              │ │
│  │  └─────────────┘  └─────────────────────┤ │
│  └─────────────────────────────────────────┘ │
└─────────────────────────────────────────-────┘

Or a second ascii-art way of outlining the same situation, with more detail ()

ChromeOS (host)
 └─ crosvm (VM with KVM acceleration if hardware supports it)
     └─ Termina (tiny VM OS, runs LXD daemon and socket)
         │    └- uses LXD API to query kernel level cgroups/namespaces
         │    └- avoids trusting oss's /bin/ps /bin/find etc
         │         
         ├─ oss       (Debian container, untrusted)
         │    └─ [supply-chain attacker could replace ps/find/ls]
         │
         └─ penguin   (main Debian container, trusted code)

Complete Setup Process

Prerequisites

This setup must be run in ChromeOS’s Termina VM, not inside a container. You can access this by pressing ctrl-alt-t and the resulting terminal should say “Welcome to crosh, the ChromeOS developer shell.” You should see a prompt like crosh>

Quick Start Script

There’s no vi, emacs or nano in chrosh. You’ll prepare scripts elsewhere and email them to yourself. You’ll use ChromeOS’ regular mail app to read those, as you logged in with your google account after all. The ChromeOS text editor is a bit weak for my liking, and I like to keep a copy of things that may get refined and are not canonically in source control.

cat > /tmp/filename_as_instructed.sh << 'SETUP_SCRIPT_EOF' | bash
The bash code below 
SETUP_SCRIPT_EOF

Save this as /tmp/setup-secure-containers.sh and run it with bash /tmp/setup-secure-containers.sh as you can’t make it executable:

#!/bin/bash
# ChromeOS Secure Container Setup
# This creates two containers:
# - penguin (default/trusted) - your main work environment  
# - oss (untrusted) - for running cloned repos and untrusted code

# IMPORTANT: This script must be run in Termina (the ChromeOS VM)
# because lxc commands don't work from inside containers

set -e

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

echo "================================================="
echo "ChromeOS Secure Container Setup"
echo "================================================="
echo ""
echo "This script will create a two-container security setup:"
echo "- penguin: Your trusted container (already exists)"
echo "- oss: Untrusted container for running random code"
echo ""

# Verify we're in Termina
if ! command -v lxc &> /dev/null; then
    echo "ERROR: lxc command not found. This script must be run in Termina"
    echo "Open the Terminal app in ChromeOS and run this script there."
    exit 1
fi

echo "Environment check passed - lxc command found"

# Step 1: Show current containers
echo ""
echo "Step 1: Current container status"
echo "================================"
lxc list

# Step 2: Get available images
echo ""
echo "Step 2: Finding Debian image..."
echo "==============================="
IMAGE_FINGERPRINT=$(lxc image list --format csv | grep -i debian | cut -d',' -f2 | head -1)

if [ -z "$IMAGE_FINGERPRINT" ]; then
    echo "No Debian image found. Available images:"
    lxc image list
    exit 1
fi

# Step 3: Create the oss container
echo ""
echo "Step 3: Creating oss container..."
echo "=================================="

# Check if oss container already exists
if lxc info oss &>/dev/null; then
    echo "Container 'oss' already exists, skipping..."
else
    echo "Creating 'oss' container..."
    lxc launch $IMAGE_FINGERPRINT oss
    echo "Waiting for container to start..."
    sleep 5
fi

# Show updated container list
echo ""
lxc list

echo ""
echo "Step 4: Setting resource limits for oss container..."
echo "===================================================="

# Set resource limits and security restrictions
lxc config set oss limits.cpu 2
lxc config set oss limits.memory 2GB
lxc config set oss security.nesting false
lxc config set oss security.privileged false

echo "Resource limits and security restrictions applied to oss container"

echo ""
echo "Step 5: Setting up oss container for untrusted code..."
echo "====================================================="

# Install packages in oss container
lxc exec oss -- apt-get update
lxc exec oss -- apt-get install -y git build-essential python3 python3-pip curl wget nodejs npm
lxc exec oss -- mkdir -p /workspace
lxc exec oss -- bash -c "echo 'Untrusted code workspace - Only run random GitHub repos here!' > /workspace/README.txt"

echo "OSS container setup complete"

# Step 6: Set up baseline in trusted container (penguin)
echo ""
echo "Step 6: Creating baseline in penguin (trusted container)..."
echo "=========================================================="

# Create baseline hashes for binary integrity monitoring
lxc exec penguin -- bash -c "find /bin /usr/bin -type f -executable -exec sha256sum {} \; > /root/baseline_hashes.txt"
lxc exec penguin -- bash -c "echo 'Baseline created at $(date)' >> /root/baseline_hashes.txt"
lxc exec penguin -- bash -c "wc -l < /root/baseline_hashes.txt | xargs -I {} echo 'Baseline hash count: {}'"

# Ensure penguin has essential tools
lxc exec penguin -- apt-get update
lxc exec penguin -- apt-get install -y vim git

echo "Baseline created in penguin container"

# Step 7: Create monitoring script in Termina
echo ""
echo "Step 7: Creating Termina-based monitoring script..."
echo "================================================"

cat > /tmp/monitor-containers.sh << 'MONITOR_EOF'
#!/bin/bash
# Container Security Monitor - Runs in Termina
# Monitors the oss (untrusted) and penguin (trusted) containers for supply chain attacks

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

function setup_baseline() {
    local baseline_file="/tmp/.container_baseline"
    
    if [ ! -f "$baseline_file" ]; then
        echo "Creating baseline for binary integrity monitoring..."
        echo "# Container Security Baseline - Created $(date)" > "$baseline_file"
        
        # Create baseline for both containers
        for container in oss penguin; do
            if lxc info "$container" &>/dev/null; then
                echo "# $container container baseline" >> "$baseline_file"
                for binary in /bin/bash /bin/sh /usr/bin/python3 /usr/bin/node /usr/bin/git; do
                    hash=$(lxc exec $container -- sha256sum $binary 2>/dev/null | awk '{print $1}')
                    if [ -n "$hash" ]; then
                        echo "$container:$binary:$hash" >> "$baseline_file"
                    fi
                done
            fi
        done
        echo "Baseline created at $baseline_file"
    fi
}

function check_binary_integrity() {
    local container=$1
    local found_issues=0
   
    echo -e "${YELLOW}Binary Integrity Check - $container${NC}"
    
    # Check key binaries against baseline
    while IFS=: read -r base_container base_binary base_hash; do
        if [[ "$base_container" == "$container" ]] && [[ "$base_binary" =~ ^/.*$ ]]; then
            current_hash=$(lxc exec $container -- sha256sum $base_binary 2>/dev/null | awk '{print $1}')
            if [ -n "$current_hash" ] && [ "$current_hash" != "$base_hash" ]; then
                echo -e "  ${RED}!!!  Modified: $base_binary${NC}"
                echo "    Expected: $base_hash"
                echo "    Current:  $current_hash"
                found_issues=1
            fi
        fi
    done < "/tmp/.container_baseline" 2>/dev/null
    
    if [ $found_issues -eq 0 ]; then
        echo "  All monitored binaries match baseline"
    fi
    
    return $found_issues
}

function check_processes() {
    local container=$1
    echo -e "${YELLOW}Process Check - $container${NC}"
    echo "  Top processes:"
    
    # Show top CPU-consuming processes
    lxc exec $container -- ps aux --sort=-%cpu 2>/dev/null | head -6 | while IFS= read -r line; do
        echo "    $line"
    done
}

function check_network() {
    local container=$1
    echo -e "${YELLOW}Network Connections - $container${NC}"
   
    # Count listening ports and established connections
    listening=$(lxc exec $container -- ss -tlnp 2>/dev/null | grep LISTEN | wc -l)
    connections=$(lxc exec $container -- ss -tnp 2>/dev/null | grep ESTAB | wc -l)
    
    echo "  Listening ports: $listening"
    echo "  Active connections: $connections"
    
    # Alert on any external connections from untrusted container
    if [ "$container" = "oss" ] && [ "$connections" -gt 0 ]; then
        echo -e "  ${RED}!!!  External connections detected in untrusted container:${NC}"
        lxc exec $container -- ss -tnp 2>/dev/null | grep ESTAB | while IFS= read -r line; do
            echo "    $line"
        done
    fi
}

function check_recent_files() {
    local container=$1
    echo -e "${YELLOW}Recently Modified Files - $container${NC}"
    echo "  Files modified in last 10 minutes:"
   
    # Focus on system directories for supply chain attacks
    lxc exec $container -- find /bin /usr/bin /lib -xdev -type f -mmin -10 2>/dev/null | while IFS= read -r line; do
        echo "    $line"
    done
    
    # Also check workspace for oss
    if [ "$container" = "oss" ]; then
        echo "  Workspace files:"
        lxc exec $container -- find /workspace -xdev -type f -mmin -10 2>/dev/null | head -5 | while IFS= read -r line; do
            echo "    $line"
        done
    fi
}

function check_suspicious_files() {
    local container=$1
    echo -e "${YELLOW}Suspicious Files Check - $container${NC}"
    
    # Look for hidden files in tmp directories
    suspicious_count=$(lxc exec $container -- find /tmp /var/tmp /dev/shm -name ".*" -type f 2>/dev/null | wc -l)
    
    if [ "$suspicious_count" -gt 0 ]; then
        echo -e "  ${RED}!!!  Found $suspicious_count hidden files in temp directories${NC}"
    else
        echo "  No suspicious hidden files found"
    fi
}

function check_suid_changes() {
    local container=$1
    
    # Check for new SUID binaries
    lxc exec $container -- find / -xdev -perm -4000 -type f 2>/dev/null | while read suid_file; do
        echo "    SUID: $suid_file"
    done
}

function setup_suid_baseline() {
    for container in oss penguin; do
        if lxc info "$container" &>/dev/null; then
            if [ ! -f "/tmp/.known_suid_$container" ]; then
                lxc exec $container -- find / -xdev -perm -4000 -type f 2>/dev/null > "/tmp/.known_suid_$container"
            fi
        fi
    done
}

# Main monitoring loop
echo "================================================="
echo "Container Security Monitor"
echo "================================================="
echo "Monitoring for supply chain attacks in:"
echo "  - oss (untrusted) - where you run random code"
echo "  - penguin (trusted) - your main environment"
echo ""

# Setup baselines on first run
setup_baseline
setup_suid_baseline

while true; do
    clear
    echo "Container Security Monitor - $(date)"
    echo "====================================="
    echo ""
   
    # Monitor the untrusted container more closely
    echo -e "${RED}=== OSS Container (UNTRUSTED) ===${NC}"
    if ! check_binary_integrity "oss"; then
        echo -e "\n${RED}!!!  ALERT: Binary modification detected in OSS container!${NC}"
        echo -e "${RED}This could indicate a supply chain attack from recently run code${NC}\n"
    fi
    echo ""
    check_processes "oss"
    echo ""
    check_network "oss"
    echo ""
    check_recent_files "oss"
    echo ""
    check_suspicious_files "oss"
   
    echo ""
    echo -e "${GREEN}=== Penguin Container (TRUSTED) ===${NC}"
    if ! check_binary_integrity "penguin"; then
        echo -e "\n${RED}!!!  CRITICAL: Binary modification in TRUSTED container!${NC}\n"
    fi
    echo ""
    check_network "penguin"
   
    echo ""
    echo "Next scan in 30 seconds... (Press Ctrl+C to exit)"
    sleep 30
done
MONITOR_EOF

chmod +x /tmp/monitor-containers.sh
echo "Monitoring script created at /tmp/monitor-containers.sh"

# Step 8: Create helper scripts
echo ""
echo "Step 8: Creating helper scripts..."
echo "=================================="

# Create a script to enter each container
cat > /tmp/enter-oss.sh << 'ENTER_OSS_EOF'
#!/bin/bash
echo "========================================="
echo "Entering OSS (UNTRUSTED) Container"
echo "========================================="
echo "!!!  SECURITY WARNING:"
echo "- Only run untrusted code here"
echo "- No sensitive files or keys"
echo "- Monitor with: bash /tmp/monitor-containers.sh"
echo "========================================="
lxc exec oss -- bash
ENTER_OSS_EOF

chmod +x /tmp/enter-oss.sh

cat > /tmp/enter-penguin.sh << 'ENTER_PENGUIN_EOF'
#!/bin/bash
echo "========================================="
echo "Entering Penguin (TRUSTED) Container"
echo "========================================="
echo "This is your trusted development environment"
echo "========================================="
lxc exec penguin -- bash
ENTER_PENGUIN_EOF

chmod +x /tmp/enter-penguin.sh

# Create status script
cat > /tmp/status.sh << 'STATUS_EOF'
#!/bin/bash
echo "Container Security Setup Status"
echo "==============================="
lxc list
echo ""
echo "OSS Container Security Settings:"
echo "  CPU limit: $(lxc config get oss limits.cpu)"
echo "  Memory limit: $(lxc config get oss limits.memory)"
echo "  Privileged: $(lxc config get oss security.privileged)"
echo "  Nesting: $(lxc config get oss security.nesting)"
echo ""
echo "Quick security check:"
for container in oss penguin; do
    echo -n "  $container binary integrity: "
    ps_hash=$(lxc exec $container -- sha256sum /bin/ps 2>/dev/null | awk '{print substr($1,1,8)}')
    echo "$ps_hash"
done
STATUS_EOF

chmod +x /tmp/status.sh

# Create a reset script for the oss container
cat > /tmp/reset-oss.sh << 'RESET_EOF'
#!/bin/bash
echo "This will destroy and recreate the OSS container"
echo "Any data in the OSS container will be lost!"
read -p "Are you sure? (y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
    echo "Resetting OSS container..."
    lxc stop oss
    lxc delete oss
    
    # Recreate with same settings
    IMAGE_FP=$(lxc image list --format csv | grep -i debian | cut -d',' -f2 | head -1)
    lxc launch $IMAGE_FP oss
    
    # Reapply security settings
    lxc config set oss limits.cpu 2
    lxc config set oss limits.memory 2GB
    lxc config set oss security.nesting false
    lxc config set oss security.privileged false
    
    # Reinstall packages
    lxc exec oss -- apt-get update
    lxc exec oss -- apt-get install -y git build-essential python3 python3-pip curl wget nodejs npm
    lxc exec oss -- mkdir -p /workspace
    
    echo "OSS container reset complete"
    
    # Update baseline
    rm -f /tmp/.container_baseline
    echo "Run: bash /tmp/monitor-containers.sh to recreate baseline"
fi
RESET_EOF

chmod +x /tmp/reset-oss.sh

# Step 9: Optional SSH Setup for OSS Container
echo ""
echo "Step 9: Setting up SSH access to OSS (optional)..."
echo "=================================================="

# Install and configure SSH
lxc exec oss -- apt-get install -y openssh-server
lxc exec oss -- rm -f /etc/ssh/sshd_not_to_be_run
lxc exec oss -- ssh-keygen -A
lxc exec oss -- systemctl restart ssh
lxc exec oss -- systemctl enable ssh

# Create non-root user
lxc exec oss -- useradd -m -s /bin/bash dev 2>/dev/null || true
lxc exec oss -- bash -c "echo 'dev:changeme' | chpasswd"
lxc exec oss -- usermod -aG sudo dev

# Get IP address
OSS_IP=$(lxc list oss -f csv -c 4 | cut -d' ' -f1)
echo ""
echo "SSH access configured:"
echo "  ssh dev@$OSS_IP"
echo "  Default password: changeme (change immediately!)"
echo ""
echo "!!!  WARNING: Never use SSH agent forwarding (-A) with untrusted containers!"

# Final summary
echo ""
echo "Setup Complete!"
echo "==============="
lxc list
echo ""
echo "Available commands:"
echo "  bash /tmp/monitor-containers.sh - Start security monitoring"
echo "  bash /tmp/enter-oss.sh         - Enter untrusted container"
echo "  bash /tmp/enter-penguin.sh     - Enter trusted container"
echo "  bash /tmp/status.sh            - Check security status"
echo "  bash /tmp/reset-oss.sh         - Reset OSS container (if compromised)"
echo ""
echo "Usage pattern:"
echo "1. Run random code ONLY in oss container (bash /tmp/enter-oss.sh)"
echo "2. Keep monitoring running in another terminal (bash /tmp/monitor-containers.sh)"
echo "3. If monitor detects modified binaries, consider using reset script"
echo ""
echo "The monitor will detect:"
echo "- Modified system binaries (supply chain attacks)"
echo "- Suspicious processes and network connections"
echo "- Recently modified files in system directories"
echo "- Hidden files in temporary directories"
echo ""
echo "🔒 Your trusted work remains safe in the penguin container!"

Manual Setup Steps

If you prefer to understand each step, here’s the manual process:

1. Create the Untrusted Container

# Get available image
IMAGE_FP=$(lxc image list --format csv | grep -i debian | cut -d',' -f2 | head -1)

# Create OSS container
lxc launch $IMAGE_FP oss

# Set security restrictions
lxc config set oss limits.cpu 2
lxc config set oss limits.memory 2GB
lxc config set oss security.nesting false
lxc config set oss security.privileged false

2. Install Development Tools

# Update and install common development tools
lxc exec oss -- apt-get update
lxc exec oss -- apt-get install -y git build-essential python3 python3-pip curl wget nodejs npm

# Create workspace directory
lxc exec oss -- mkdir -p /workspace

3. Create Monitoring Script

The monitoring script runs from Termina and watches both containers for signs of compromise:

# Create the monitoring script
cat > /tmp/monitor-containers.sh << 'EOF'
#!/bin/bash
# [Full monitoring script from above]
EOF

chmod +x /tmp/monitor-containers.sh

Troubleshooting Common Issues

Termina Filesystem Issues

Issue: Scripts fail with “Read-only file system” error. Termina’s home directory (~/) is read-only:

Solution: Use /tmp for all scripts which is not read only.

Issue: no editors

Solution pipe to file trick as mentioned.

# Correct - use /tmp
cat > /tmp/script.sh << 'EOF'
...
EOF
bash /tmp/script.sh

SSH Setup

SSH service needs manual setup in containers:

You get into the container using lxc exec container_name -- bash from Termina (crosh):

# In the target container (run via lxc exec)
apt-get install -y openssh-server
rm -f /etc/ssh/sshd_not_to_be_run  # Remove startup blocker
ssh-keygen -A                       # Generate host keys
systemctl restart ssh
systemctl enable ssh
echo 'root:changeme' | chpasswd    # Set password

LXC Command Not Found

Do NOT run from inside penguin or any other container - you need to be in crosh: ctrl-alt-t

Secure Git Access Patterns

You don’t want you SSH private key on the container that may be taken over by ‘chalk’ style actions. At least you don’t want it without a lengthy passphrase, but there are traditional solutions:

The SSH Agent Forwarding Security Risk

Critical Warning: SSH agent forwarding allows untrusted code to use your keys!

While your SSH session with -A flag is active:

Malicious code can push to ANY repo you have write access to
Can clone ANY private repo you have access to
Cannot steal your key, but can USE it

Safer Alternatives for Git Access

Option 1: Read-Only Deploy Keys (RECOMMENDED)

# Create separate key for untrusted work
ssh-keygen -t ed25519 -f ~/.ssh/oss_readonly_key -N ""

# Add to GitHub as deploy key (READ-ONLY) for specific repos
# Copy ONLY this key to oss container
lxc exec penguin -- cat /home/USER/.ssh/oss_readonly_key | \
  lxc exec oss -- bash -c "cat > /home/dev/.ssh/id_ed25519 && chmod 600 /home/dev/.ssh/id_ed25519"

Option 2: Fine-Grained Personal Access Tokens

# Create token with minimal permissions (public_repo only)
# Use in oss container:
git clone https://TOKEN@github.com/user/repo.git

Option 3: Time-Limited Agent Forwarding (USE SPARINGLY)

# Only when absolutely necessary for push operations
ssh -A dev@[oss-ip]
# Do your git operation
# EXIT IMMEDIATELY
exit

# Or use confirmation-required keys
ssh-add -c ~/.ssh/id_ed25519  # Requires confirmation for each use

Security Best Practices

What to NEVER Do

Don’t store sensitive data in the OSS container:

# NEVER DO THIS
cp ~/.ssh/id_rsa /path/to/oss/container

Don’t run trusted code in OSS container:

# NEVER DO THIS
lxc exec oss -- git clone git@github.com:yourcompany/private-repo.git

Don’t disable monitoring:

# NEVER DO THIS - Always keep monitoring running
pkill monitor-containers

What to ALWAYS Do

Reset compromised containers:
```
# If monitor detects issues
bash /tmp/reset-oss.sh
```
Don’t attempt to repair them. Heck, maybe reset with some regularity anyway.

Regularly update baselines:

# After legitimate updates
rm /tmp/.container_baseline
bash /tmp/monitor-containers.sh  # Recreates baseline

Container Isolation Rules

Container	Purpose	SSH Keys	Git Configs	Sensitive Data
penguin	Trusted development	Safe	Safe	Safe
oss	Untrusted testing	Never	Never	Never

Git Security Matrix

Operation	Penguin (Trusted)	OSS (Untrusted)	Method
Clone public repos	y	y	HTTPS
Clone private repos	y	!	Deploy keys only
Push to repos	y	N	Never from OSS
Store SSH keys	y	N	Never
Store PATs	y	!	Limited scope only
Agent forwarding	N/A	!️	Brief sessions only

Container Access Patterns

Daily workflow

Terminal App -> penguin # Normal development
Ctrl+Alt+T -> vsh termina -> bash /tmp/enter-oss.sh # Testing
Or ssh from penguin to oss

Security monitoring

Ctrl+Alt+T -> vsh termina -> bash /tmp/monitor-containers.sh

Never create shortcuts that bypass security
Don’t alias direct access to OSS in penguin
Don’t auto-start monitoring (review alerts manually)

Lessons from Production Use

Key Insight: The separation between Termina (VM host) and containers is crucial. Many security solutions try to work entirely within containers, but the real power comes from leveraging the host-level view.

What we learned:

Container isolation is only as good as your monitoring for breaches
Host-level monitoring provides better security visibility
ChromeOS’s architecture is designed for this type of security model

Understanding the Monitoring System

The monitoring system watches for several types of compromise:

1. Binary Integrity Monitoring

I grant you this is underdeveloped at the point of this blog entry.

# Creates baseline hashes of critical binaries
for binary in /bin/bash /bin/sh /usr/bin/python3 /usr/bin/node /usr/bin/git; do
    hash=$(lxc exec $container -- sha256sum $binary 2>/dev/null | awk '{print $1}')
    echo "$container:$binary:$hash" >> ~/.container_baseline
done

What it detects: Supply chain attacks that modify system binaries

Real-world example: The 2024 Chalk npm package compromise replaced legitimate packages with malicious versions that could modify Node.js binaries or install backdoors. Our monitoring would detect such changes immediately.

Example alert:

!!!  Modified: /usr/bin/python3
Expected: a1b2c3d4e5f6...
Current:  x9y8z7w6v5u4...

2. Process Monitoring

# Shows CPU-intensive processes
lxc exec $container -- ps aux --sort=-%cpu | head -6

What it detects: Cryptocurrency miners, botnet activity, unexpected daemons

3. Network Connection Analysis

# Counts active connections
connections=$(lxc exec $container -- ss -tnp | grep ESTAB | wc -l)

What it detects: Data exfiltration, command & control communication, unexpected servers

4. File System Changes

# Finds recently modified system files
lxc exec $container -- find /bin /usr/bin /lib -xdev -type f -mmin -10

What it detects: System file tampering, backdoor installation

5. Hidden File Detection

# Searches for hidden files in temp directories
lxc exec $container -- find /tmp /var/tmp /dev/shm -name ".*" -type f

What it detects: Malware staging areas, credential harvesting tools

Advanced Usage Patterns

Testing Suspicious GitHub Repositories

# 1. Start monitoring (separate terminal)
bash /tmp/monitor-containers.sh

# 2. Enter untrusted container
bash /tmp/enter-oss.sh

# 3. In OSS container, test the repo
cd /workspace
git clone https://github.com/suspicious/repo.git
cd repo
./setup.sh  # This runs in isolation

# 4. Monitor terminal will alert on any suspicious changes
# 5. If compromised, reset the container
exit  # Leave OSS container
bash /tmp/reset-oss.sh

Development Tool Testing

# Test a new development tool safely
bash /tmp/enter-oss.sh

# In OSS container
curl -fsSL https://some-tool.com/install.sh | bash
some-new-tool --help

# Monitor for:
# - Modified binaries
# - Network connections
# - Hidden files
# - Unexpected processes

Multi-Container Workflow

# Terminal 1: Monitoring
bash /tmp/monitor-containers.sh

# Terminal 2: Trusted work
bash /tmp/enter-penguin.sh
# Do your normal development here

# Terminal 3: Untrusted testing
bash /tmp/enter-oss.sh
# Test random GitHub projects here

# Terminal 4: Status checking
bash /tmp/status.sh

Performance Considerations

Resource Limits

The OSS container is intentionally limited to prevent resource abuse:

# Current limits (modify as needed)
lxc config set oss limits.cpu 2        # 2 CPU cores max
lxc config set oss limits.memory 2GB   # 2GB RAM max

Monitoring Overhead

Monitoring script uses minimal resources
Scans every 30 seconds (configurable)
Focuses on security-critical changes only
Can run continuously without impact

Container Reset Speed

# Full OSS container reset takes ~2-3 minutes
time bash /tmp/reset-oss.sh
# Includes: stop, delete, recreate, configure, install packages

Integration with Development Workflow

IDE Integration

You can configure your IDE to work with the container setup:

# VS Code with Remote-Containers
# Point to penguin container for trusted development
code --folder-uri vscode-remote://attached-container+penguin/path/to/project

# For untrusted code testing, always use terminal access
bash /tmp/enter-oss.sh

Git Configuration

# In penguin (trusted) - normal git config
git config --global user.name "Your Name"
git config --global user.email "your.email@domain.com"

# In OSS (untrusted) - minimal or fake config only
git config --global user.name "Test User"
git config --global user.email "test@example.com"
# Never configure real credentials

# Share files from trusted to untrusted (one-way only)
# Copy from penguin to OSS for testing
lxc file push /path/in/penguin/container/file.txt oss/workspace/

# NEVER copy from OSS to penguin without verification
# Instead, manually recreate verified files in penguin

Extending the Security Model

Custom Monitoring Rules

Add your own detection rules to the monitoring script:

# Example: Monitor for specific file types
function check_crypto_miners() {
    local container=$1
    miners=$(lxc exec $container -- find /tmp -name "*mine*" -o -name "*crypto*" 2>/dev/null | wc -l)
    if [ "$miners" -gt 0 ]; then
        echo -e "${RED}!!!  Potential cryptocurrency miner detected${NC}"
    fi
}

Log Integration

# Enhanced logging in monitoring script
LOG_FILE="$HOME/container-security.log"

function log_alert() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') ALERT: $1" | tee -a "$LOG_FILE"
}

Conclusion

ChromeOS’s layered container architecture provides an excellent foundation for safely testing untrusted code. This setup gives you:

True isolation: Each container is properly sandboxed
Real-time monitoring: Immediate detection of compromise attempts
Easy recovery: Quick container reset when needed
Maintained productivity: Your trusted environment stays clean

The key insight is using ChromeOS’s existing security model rather than fighting it. By running the monitoring from Termina and isolating untrusted code in its own container, you get enterprise-grade security with developer-friendly workflows.

All that said, I’m unlikely to use it without the Terminal integration (see below). I’d like to open Terminal then click oss a line below penguin. Instead, I’m probably use a container in podman inside penguin - that seems hardened as a solution and workable today.

The Terminal App Limitation

CromeOS’ Terminal app’s inability to add custom container entries is a significant UX gap. The fact that only penguin appears as a clickable row means:

No visual distinction between trusted/untrusted environments
Can’t theme the OSS terminal differently (red background would be perfect!)
Extra friction to access the untrusted container
No way to know at a glance which container you’re in

The SSH workaround (penguin -> ssh dev@oss-ip) adds complexity to what should be a single click.

Security Implications

These UX limitations create real security risks:

Command confusion: Without visual distinction, you might accidentally run trusted commands (like git push with your real credentials) in the untrusted container
Increased attack surface: The SSH workaround opens network ports and adds authentication complexity that could be exploited
Remote access temptation: Lack of remote access means you might be tempted to run untrusted code directly on your primary machine when traveling, defeating the entire security model

I would really want to be able to make a second container from within the Terminal app. The UX hints that it should be possible but the feature is not there. A huge shame given how incredibly strong the dev experience on ChromeBooks (that have enough RAM and SSD).

The Chrome Remote Desktop Problem

This is even more frustrating. A ChromeOS Flex machine effectively becomes an island because, only Windows and Mac are first-class choices for destination for Chrome Remote Desktop. HOST-OS Linux is a second class choice because one seems to need PhD-level understanding of X11/Wayland/DISPLAY. Multi-user possibilities might be one of complexities, and systemd could be on the “complicating” mix too.

Chrome Remote Desktop TO ChromeOS/Linux (the very platform Google controls!) is not supported at all.

Yet, Chrome Remote Desktop FROM ChromeOS to Windows or Mac is supported.

This means you can’t remotely access your secure container (in ChromeOS Flex on a spare PC) from your Chromebook when traveling, defeating much of the purpose of having a dedicated security testing environment. Potential Workarounds (all imperfect). Or from your Mac or Windows laptop which support Chrome Remoting as an origin just fine.

For Terminal access I could create a web-based terminal (ttyd or similar) in each container, accessible via different ports, but I’d rather be in my terminal of choice: Terminal

I have other VNC server in penguin or Guacamole, too, but I wish this were a mainstream feature of ChromeOS once you’ve enabled developer features.

The irony is that Google has all the pieces (Crostini, Chrome Remote Desktop, Terminal app) but hasn’t connected them properly. A simple “Add Container to Terminal” button and proper Chrome Remote Desktop support would solve everything. It’s particularly galling because ChromeOS is supposed to be the “simple, secure” option, yet these limitations push us toward complex workarounds that probably decrease security.

Googlers

And if any Googlers have got this far: can you have an explicit “Disable trackpad while typing” setting as macOS Sierra had. It was removed after sierra. Chromebooks have plastic chassis and mild weight adjacent to the trackpad while typing can cause a click at current pointer position.

Updates

Jan 2026: The “Multi-container” ChromeOS flag #crostini-multi-container has moved from a hidden experiment to a core feature then deprecated. Now there’s a neg baugutte direction that is containerless - wee VMs instead. Its early days with that, and I’m playing with nixos-crostini toward the same goals.

Paul Hammant's blog

BTRON, OLE, OpenDoc and the Web: Why Dumb Won

Four answers to “what comes after files + applications?”

Graceful degradation outward

Navigation master control

What about Lotus Notes?

Denny called this in 1990 - in reverse

So, wealth extraction?

Where do phones fit?

Files + applications again

Charting AI Today: What It Leaves Behind

The axis that took the longest

Persistence rises, then does something strange

What “Become” means

The overlays

Caveats

Others who wrote it up

Footnote: a reversibility I’m not measuring

Side Story

Build-time AI vs Run-time AI

Harness engineering is a build-time discipline

An orthogonal axis, not a maturity stage

What a “harness” actually is

Google-style DAG build systems (with Aether Build)

Multi-module build systems per language

Two anemic applications

Components that the applications use

The dependency graph

The DAG side: Aether Build

Build files

First build

Running it again

Running the tests

Running the app

Building the other application

The depth-first recursive side: Maven

A full clean build

An incremental build (no clean)

Focusing with -pl and -am

Comparing features

Circular references

Subsetting the checkout (expand/contract)

Interpreted config vs a compiled build program

Where this is heading: containment

A note on scale

Cherry-pick Asymmetry

Why it’s (almost) impossible, not merely unlikely

The divergence mechanisms - i.e. the only ways to lose the bet

So what’s the number?

Cadence is the master variable

The genuinely counterintuitive bit

When is the second pass actually redundant?

Closing

The limits of merging experiment

The setup

Scenario 1: git am is honest, and that’s why it fails

Scenario 2: git cherry-pick is helpful, and that’s why it lies

What about merge-point tracking?

The order of cherry-picks question

Blocking a commit (Scenario C)

What the SHA actually proves

What git can’t tell you

The real risk in real corporate codebases

Reproducing this

Repeating in SVN: what svn:mergeinfo actually looks like

Scenario A — cherry-pick out of order (C4 then C3, then sweep)

Scenario B — cherry-pick in trunk order (C3 then C4, then sweep)

What the property is telling you

Scenario C — blocking C3 with --record-only

Reproducing the SVN side

Repeating in Perforce: integration records, not properties

Scenario A — cherry-pick out of order (C4 then C3, then sweep)

Scenario B — cherry-pick in trunk order (C3 then C4, then sweep)

What the records are telling you

Scenario C — blocking C3 with resolve -ay

Reproducing the Perforce side

Closing

So which VCS “knows what’s been integrated”?

Live Verify

Live verify links

An incremental build (no `clean`)

Focusing with `-pl` and `-am`

Scenario 1: `git am` is honest, and that’s why it fails

Scenario 2: `git cherry-pick` is helpful, and that’s why it lies

Repeating in SVN: what `svn:mergeinfo` actually looks like

Scenario C — blocking C3 with `--record-only`

Scenario C — blocking C3 with `resolve -ay`