rarouš.w3b

AI Software Development - What Does The Data Say? - Codemanship's Blog

2026-08-12T19:34:21Z

AI Software Development – What Does The Data Say? – Codemanship's Blog

AI Software Development – What Does The Data Say?

I’m currently pulling together a bunch of sources – that are mostly recent – on the topic of LLMs and their use in software development.

Some are peer-reviewed studies. Some are industry studies that haven’t been peer-reviewed.

One is statistical physics. Expect more from that angle. Wanna’ know the limits of a technology? Ask a physicist.

One is just a blog post, but very useful information about the effect of context size.

Most are corroborated by personal experiments and also observations on teams. As time goes on and more data comes in, my picture comes more into focus.

Code Evolution & Long-Horizon Agentic Workflows

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
https://arxiv.org/abs/2603.03823

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
https://arxiv.org/html/2603.24755v1

SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
https://arxiv.org/abs/2603.13428

Context Engineering

Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs
https://arxiv.org/abs/2509.21361

Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
https://arxiv.org/abs/2509.21865

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
https://arxiv.org/abs/2602.11988

(blog post – informational, not peer-reviewed research)
The Hidden Science Behind LLM Token Limits (And How Million-Token Models Actually Work)
https://www.ashisharora.ai/post/the-hidden-science-behind-llm-token-limits-and-how-million-token-models-actually-work

Language models are not naysayers: An analysis of language models on negation benchmarks
https://arxiv.org/abs/2602.11988

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
https://aclanthology.org/2022.emnlp-main.759

STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
https://arxiv.org/abs/2605.06527

Task Matters: Knowledge Requirements Shape LLM Responses to
Context–Memory Conflict
https://aclanthology.org/2026.findings-acl.202

(I call these “dominant priors”, when information in the model overrides that provided in the context)

Large-scale Industry Studies in Software Engineering

What 28 million workflows reveal about AI coding’s biggest risk (CircleCI)
https://www.linkedin.com/pulse/what-28-million-workflows-reveal-ai-codings-biggest-risk-circleci-j9syc/

The Acceleration Whiplash – AI Engineering Report 2026 (Faros)
https://www.faros.ai/research/ai-acceleration-whiplash

State of AI-assisted Software Development 2025 (DORA)
https://dora.dev/research/2025/dora-report/

Psychology, Cognition & Learning

Super-intelligence or Superstition? Exploring Psychological Factors Influencing Belief in AI Predictions about Personal Behavior
https://arxiv.org/html/2408.06602v3

The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers
https://arxiv.org/html/2408.06602v3

Experimental evidence of the effects of large language models versus web search on depth of learning
https://arxiv.org/html/2408.06602v3

At What Cost? Software Developers’ Well-Being in the Age of GenAI
https://ourarchive.otago.ac.nz/esploro/outputs/preprint/At-What-Cost-Software-Developers-Well-Being/9926870122801891

Limits of LLMs & Deep Learning

The wall confronting large language models – (statistical mechanics study)
https://arxiv.org/abs/2507.19703

Learning long-term dependencies with gradient descent is difficult
https://pubmed.ncbi.nlm.nih.gov/18267787/
(Why deep neural networks can’t see the bigger picture, at any scale of model)

What are the implications for teams using this technology? If you’re interested in a hype-free, evidence-based take about why the technical practices of Agile Software Development are so closely aligned with AI-assisted and agentic software engineering, join me on October 6th at 18:45 BST.

Register: https://www.tickettailor.com/events/codemanship/2324138

Author: codemanship

Founder of Codemanship Ltd and code craft coach and trainer View all posts by codemanship

Cloudflare OS is an architecture of distrust

2026-08-10T09:21:28Z

The strangest thing in the Cloudflare OS source code took me a while to understand.

When an agent inside Cloudflare OS wants to do something with a side effect (merge a pull request, send an email, write a row to a system of record), it goes through a Gatekeeper, a small service that holds the credential and mediates the action. So far, that’s just a well-built MCP server. But read the contract a Gatekeeper is written against (packages/workshop-shared/src/gatekeeper.ts, around line 617) and you find this instruction to the author:

It is suggested that the gatekeeper “simulate” actions that have not been approved yet, that is, the Session interface should reflect the state of the resource as if all actions had been applied.

Sit with that. The agent asks to merge the PR. The human hasn’t approved it. So the Gatekeeper tells the agent the PR is merged, and if the agent reads the branch back to check its work, hands it a fabricated reality in which the merge happened. The agent, satisfied, queues the next three steps that depend on it. None of it is real. Later a human looks at the batch and either commits it or bins it, and if they bin it, everything the agent built on the fiction goes too.

The first time I traced this I thought it was a hack. It’s the philosophy of the whole system, compressed into one method signature. The Gatekeeper lies to the agent on purpose, because the alternative (letting an agent’s actions touch the world the moment it decides to take them) assumes the agent’s decisions are sound. Cloudflare OS is built from end to end on the assumption that they are not.

The name is a distraction, so set it aside. The Hacker News thread spent most of its energy arguing about whether “OS” is a permitted word for the thing, and that’s a dead end. What’s actually interesting is that a team led by Kenton Varda, the people who built the Workers runtime, sat down to design a platform for AI agents doing real work inside a company, and the organising principle they landed on was this: the agent cannot be trusted, so build so that its mistakes cannot matter. Every load-bearing part of the system is a variation on that sentence.

Last week I wrote about Opus 5 getting things confidently, quietly wrong: shipping a change that reported success while doing the opposite, caught only on a second pass. This is what it looks like to take that failure mode not as a grievance but as a permanent design constraint, and pour concrete on top of it.

Make the code irrelevant to safety

Start with the sandbox, because everything else stands on it.

When you make a slide deck in Cloudflare OS, you aren’t using one shared slide-deck app. The system spins up a private instance of the slide-deck code, a “gadget”, that belongs only to you. It runs in its own Dynamic Worker, with its own SQLite database behind a Durable Object Facet, outbound networking switched off. Every document is its own sandboxed process.

This is Sandstorm, Kenton’s startup from a decade ago, reborn. What Sandstorm got right and could never make cheap was fine-grained instancing: every document in its own isolation boundary. Containers made that far too expensive, seconds of cold start and hundreds of megabytes per grain, so the idea sat on a shelf for ten years until V8 isolates made it roughly a hundred times cheaper to run. I wrote a book about this platform and called Durable Objects its most underappreciated primitive; the thing they were waiting to enable, it turns out, was this.

Per-document instancing changes where security lives. For twenty-five years the boundary in multi-tenant SaaS has run straight through the application code. One shared service holds everyone’s data, and a single mistake in a WHERE clause, one missing tenant_id, leaks customer B’s records to customer A. The code is load-bearing for security. It has to be correct, and a junior engineer’s off-day is a breach.

Cloudflare OS moves the boundary out of the code and into the platform. If every user has their own instance, and the platform controls who can reach an instance at all, a bug in the gadget can only hurt the one person who owns it. Kenton puts it flatly: the AI cannot introduce a significant security bug. He’s right, in the sense that matters. A gadget can’t leak to another user however badly it’s written, because there is no other user in its sandbox to leak to.

This is the first and boldest act of distrust. They didn’t make the AI’s code trustworthy. They gave up on that and made it irrelevant, arranging things so the correctness of what the agent writes is no longer a security property at all. You can let a non-technical colleague vibe-code an app and share it, and the reason the security team sleeps is not that the app is good. It’s that a bad app can’t reach them.

Never let the agent hold the key

The cleverest move is the quiet one.

The obvious way to give an agent access to GitHub is to give it a GitHub token. Anyone who has wired up an MCP server has done some version of this, and felt the small cold moment of realising a token in a prompt can end up anywhere the prompt’s output goes.

Cloudflare OS never gives the agent the credential. The Gatekeeper holds it. What the agent gets is a capability: in the generated code it appears as a binding, env.PROJECT, an object it can call methods on. Not the key itself, but permission to perform a specific, narrowed set of operations the Gatekeeper will carry out on its behalf. Give the GitHub Gatekeeper a single repository and it can let the agent read issues but not source, mask fields, refuse to merge without approval. The agent can call listIssues(). It cannot see the token, widen its own access, or reach the network except through capabilities it was explicitly handed.

None of this is theoretical to me. I built exactly this credential-broker pattern for CDS, and it has since rolled out across the engineering practice there: the broker holds the secret, the agent gets a handle and never the key itself.

This is the object-capability model, and it isn’t decoration. It’s the Cap’n Proto lineage Sandstorm was built on, resurfacing as Cap’n Web RPC, because a capability is the only kind of authority you can safely hand an actor you’ve decided not to trust. A key is bearer authority: whoever holds it wields it. A capability is a leash. It does one thing, it can be revoked, it stays attached to whatever granted it. If your agent might be confidently wrong, or prompt-injected, or just careless about what it writes into an app’s source, the credential is the one thing it must never touch. So it never touches it.

Let it act, but never let it commit

Now the simulated merge from the opening makes sense: the same idea, moved from access into control flow.

The problem it solves is one I’ve felt every day for a year. Synchronous approval is miserable. The agent stops on step one, you’ve wandered off to make coffee, you come back to no progress. So people cave and turn on auto-approve, or --dangerously-skip-permissions, and the safety mechanism they installed to sleep at night is the first thing they switch off to get anything done.

Cloudflare OS splits acting from committing. Reads flow: a read is authorised against the capability and recorded, but it doesn’t stop for a human. Writes are provisional. The Gatekeeper accepts the action, simulates the result, and lets the agent race ahead building whatever depends on it, while the real effect sits in a queue for a person to release in a batch, later, when it suits them. The agent gets the throughput of auto-approve; the human keeps the veto of manual review. The human is no longer an interruption in the loop. The human is the commit.

It’s optimistic concurrency for actions in the real world, with a person as the transaction monitor, and it has the property optimistic concurrency always has: the agent spends part of its life operating on a state that isn’t true yet and may never be. It reads back its own un-happened writes and reasons about them. The architecture’s answer to “isn’t that dangerous?” is a shrug. It doesn’t matter what the agent believes about a world no human has ratified. Belief is free. Only the commit is real, and the agent doesn’t hold the commit.

Distrust the outputs, not just the actions

The subtlest move is the one the blog post almost entirely hides.

The first three mechanisms distrust what the agent does. This one distrusts what it produces. An agent reads a sensitive revenue table and builds a live dashboard from it. The dashboard is a new artefact with no access list of its own. Share it, and you may have just handed the table to everyone who can see the dashboard: a leak no single component did anything wrong to cause.

So Cloudflare OS records every observation a gadget makes. The mechanism lives in docs/observers.md, and it is, in essence, decentralised information-flow control: the academic dream of the mid-2000s, the Jif and HiStar and Flume line of work that never shipped commercially because labelling every piece of data by hand cost more than it returned. When Alice shares a gadget with Bob, every Gatekeeper the gadget has touched is asked, independently, whether Bob may directly read everything the gadget has already read through it. If he can’t, he’s refused. From then on, any new read the gadget makes that Bob isn’t cleared for is blocked outright.

The detail I keep turning over is that the kernel refuses to understand identity. The overseer, the “kernel” in the repo’s own OS analogy, does not know what a GitHub user is, or a Google user, or what any vendor’s permissions mean. It hands each Gatekeeper an opaque token minted by the observer’s own account and lets the Gatekeeper be the sole authority on its own resource. The code goes out of its way to make that handle a random string, so no Gatekeeper author is tempted to parse identity out of it. Authority is decentralised on purpose. The centre is built to know as little as it can.

Where the policy engine can’t yet express the nuance, it falls back to raw distrust. A flag, prohibitAllSharing, marks data so sensitive it must never leave the owner. Once a gadget reads something wearing that flag, it drops into “lockdown mode”: it can still read, but it can no longer take a single action, through any Gatekeeper, ever, in case the action is what smuggles the secret out. A taint bit. A one-way door. The bluntest instrument available, and the comment above it says so and calls it a stopgap. What DIFC never had a good enough reason to ship for, an autonomous agent supplies: you cannot hand-audit what a thing read when it reads a thousand times an hour and reasons about all of it at once.

A tell in the contributing guide

The worldview surfaces in an odd place. CONTRIBUTING.md asks you, politely, not to send pull requests longer than a dozen lines or so, and says why: AI has made writing code cheap, so an outside contribution donates the easy part while creating the expensive part, which is review and keeping the whole thing coherent. They would rather you didn’t.

There’s a real claim buried there. The scarce resource in software has inverted. Producing code is no longer the bottleneck; vouching for it is. And it’s the same reflex as everything else in the system. An unreviewed contribution is untrusted input, and it barely matters whether a careless human or a confident model produced it: the instinct, now written into the project’s own governance, is to keep unvetted work out until someone accountable has looked at it. They’ve generalised their distrust of the agent into a distrust of the contribution, and then into a policy about you.

Is this wisdom or a cage?

I’ve read a lot of agent frameworks this year, and most are optimistic in a way that embarrasses them within a month. This one isn’t, and I admire it for that. It’s the first agent platform I’ve seen that treats the failure mode as real instead of writing a longer system prompt and hoping. If you believe, as I’ve come to, that the models are confidently wrong often enough that their self-reports can’t be load-bearing, then an architecture that assumes exactly that isn’t cynicism. It’s honesty.

But an architecture built on the premise that the worker can’t be trusted also caps what the worker is allowed to become. An agent that can never hold a credential can never do the job that truly needs one. A write that is always provisional means a human is always, structurally, in the loop: the precise bottleneck agents were meant to remove. Every act of distrust here is also a leash, and the leash that keeps the animal safe is the one that stops it pulling the cart very far. You can’t design for maximum autonomy and maximum containment at once. Cloudflare has chosen containment, hard, betting that a caged agent doing real work beats a free one you daren’t deploy.

And there’s one thing the distrust never turns on. The agent is untrusted. The user’s code is untrusted. The human contributor is untrusted. Every principal is held at arm’s length and made to earn each move through a capability. Every principal except Cloudflare, whose primitives are the arm. The observation log, the Gatekeepers, the sandbox that makes the whole thing safe are all Workers, Durable Objects, Facets, some added to the runtime for this and existing nowhere else. It runs on workerd, which is open source, and you can self-host it, which is more than most of its rivals can say. But the architecture that trusts no one rests entirely on trusting the one party that poured the ground beneath it.

Last week a model told me it had fixed something, and it hadn’t, and I only found out because I went and looked. Cloudflare has built a whole operating system on the certainty that it always will (that the thing you delegate to will report a success it hasn’t earned), and then arranged matters so that when it does, a human holds the only pen that writes to the world. Whether that’s the shape of all serious agent infrastructure from here, or a very elegant set of walls we’ll spend three years learning to resent, I can’t tell. Probably both. The kernel table in the README has a row for processes, a row for users, and a row for agents left blank, marked only ???. They know it’s a new kind of thing. They just don’t trust it yet.

SSD Streaming for AI Models: How to Turn RAM from a Wall into a Dial

2026-08-07T10:58:44Z

When RAM Becomes the Bottleneck, Not the Solution

If you’ve ever tried running a large language model locally, you’ve probably hit the same wall. You find a model you want to use — maybe a Mixtral variant, a DeepSeek model, or something else with real capability — check the requirements, look at your available RAM, and discover you’re about 20GB short. The model doesn’t run at a reduced quality. It just doesn’t run.

That binary outcome is the core frustration of local AI deployment. RAM has always functioned as an on/off switch: either the whole model fits, or nothing works. SSD streaming flips that logic entirely. By storing model weights on disk and loading them on demand, it turns RAM from a fixed ceiling into an adjustable dial. Dwarf Star is one of the more interesting implementations of this idea, specifically targeting the expert weights in Mixture of Experts (MoE) models.

This article explains how SSD streaming works, why MoE architectures make it practical, what trade-offs you’re accepting, and who actually benefits from it.

The Real Problem with Running Large Models Locally

Most people frame the local AI problem as “my computer isn’t powerful enough.” That’s not quite right. The more precise framing is: model weights have to live somewhere your processor can access them quickly, and RAM is the fastest accessible space in that chain.

Plans first. Then code.

PROJECTYOUR APP

SCREENS12

DB TABLES6

BUILT BYREMY

1280 px · TYP. A · UI · FRONT END

Remy writes the spec, manages the build, and ships the app.

When you load a model, its weights get copied into RAM. Every inference — every token generated — reads from those weights repeatedly. If the weights don’t fit in RAM, the system starts reading from swap space (disk), and inference slows to a crawl or crashes entirely.

For smaller models (7B parameters, quantized), this is manageable. A 4-bit quantized 7B model runs comfortably in 6–8GB of RAM. But the models that people actually want to run locally — the ones with strong reasoning, broad knowledge, and real-world utility — tend to be 30B, 70B, or larger. A 70B model at 4-bit quantization still needs around 40GB of RAM.

Most consumer machines top out at 16–32GB. That gap isn’t small.

The Scale Problem Gets Worse with Capability

There’s an uncomfortable relationship between model capability and size. Broadly speaking, more parameters means better performance on complex tasks. The models people reach for when they want serious results — multi-step reasoning, code generation, nuanced writing — are the ones that won’t fit.

This creates a practical tiering problem. You can run the models you need for simple tasks, but the moment you need something harder, you’re looking at API calls, cloud costs, or a hardware upgrade.

SSD streaming is an attempt to break that relationship by changing where model weights live, not by shrinking the model itself.

Mixture of Experts: Why Not All Weights Are Created Equal

To understand why SSD streaming is practical at all, you need to understand Mixture of Experts (MoE) architecture. It’s the key that makes on-demand weight loading viable.

In a standard dense model, every layer processes every input. If a model has 70 billion parameters, all 70 billion are involved in generating each token. That’s why large dense models need so much RAM — everything has to be loaded and accessible.

MoE models work differently. Instead of one large feedforward network in each layer, they have multiple smaller networks called “experts.” A routing mechanism looks at each input token and decides which experts to activate — typically just two or four out of eight, sixteen, or more available.

What This Means Practically

A model like Mixtral 8x7B has eight expert networks per layer. For any given token, only two of them fire. The other six sit idle.

Total parameter count is high — around 46 billion parameters — but active parameters per forward pass are much lower, closer to 12–13 billion. That’s why MoE models can punch above their inference weight: you get the parameter breadth of a large model with the compute cost of a smaller one.

But here’s the implication that makes SSD streaming interesting: if only two experts are active at any given moment, you don’t need all eight experts in RAM simultaneously. You only need the two that are about to be used.

This is the architectural opening that Dwarf Star exploits.

How SSD Streaming Actually Works

The core idea is straightforward: keep the expert weights on your SSD instead of in RAM, and load them into memory only when the router selects them.

Everyone else built a construction worker.
We built the contractor.

🦺

CODING AGENT

Types the code you tell it to.
One file at a time.

🧠

CONTRACTOR · REMY

Runs the entire build.
UI, API, database, deploy.

Dwarf Star stores the expert weight tensors on disk. The non-expert components of the model — the attention layers, layer normalization, routing weights — stay in RAM because they’re used for every token and need to be fast. But the expert weights, which represent a large fraction of total parameters, get offloaded to disk.

When inference runs and the router selects experts for a given layer, the system fetches those expert weights from the SSD, runs the computation, and moves on. The weights don’t need to stay loaded — they’ll only be needed again if that expert is selected for a future token.

The Math Behind It

Take Mixtral 8x7B as an example. Total weights are roughly 90GB in full precision, around 26GB at 4-bit quantization. A large chunk of that is expert weights.

If you store experts on disk and only pull the active two per layer per token, your RAM requirement drops dramatically. The non-expert weights — the part that actually needs to stay in memory — can fit comfortably on a machine with 16–24GB of RAM.

You’re not running a smaller model. You’re running the same model, with the same output quality, but with a much smaller RAM footprint.

Prefetching Matters

A naive implementation would have noticeable latency: pick an expert, wait for the disk read, compute, repeat. That would be slow enough to make the approach impractical.

Better implementations use prefetching. Because the routing decision for layer N is known before layer N+1 executes, the system can start loading the next layer’s experts from disk while the current computation runs. With a fast NVMe SSD (which can sustain 5–7+ GB/s sequential reads), and with careful prefetching logic, the disk reads can largely happen in parallel with compute.

The result is that SSD streaming doesn’t feel like constant disk thrashing — it feels like slightly slower inference than native RAM, not like reading from a hard drive.

SSD Speed Is Not Optional

This only works if your storage is fast. A traditional SATA SSD tops out around 550 MB/s sequential read. A modern NVMe SSD using PCIe 4.0 hits 5,000–7,000 MB/s. That’s a 10x difference, and it’s the difference between a streaming approach that’s workable and one that makes inference painfully slow.

If you’re considering running Dwarf Star or similar expert-streaming setups, your storage tier matters as much as your RAM:

NVMe PCIe 4.0 or 5.0: This is where streaming inference becomes genuinely usable. Read speeds are fast enough that prefetching keeps the pipeline fed.
NVMe PCIe 3.0: Still viable, but you’ll notice the latency more. Tokens per second will be lower.
SATA SSD: Too slow for comfortable inference on most MoE models. You’ll likely bottleneck here before you bottleneck on compute.
HDD: Not a realistic option. Seek times alone make this unworkable.

This is worth flagging because “runs on your laptop” can mean very different things. A machine with a fast NVMe drive and 16GB RAM can have a genuinely good experience. The same RAM with a slower drive won’t.

The Trade-off Triangle: RAM, Speed, and Model Quality

SSD streaming doesn’t give you something for free. It trades one resource (RAM) for another (SSD bandwidth and some latency). Here’s how the trade-offs stack up:

What You Gain

Dramatically lower RAM requirement: Models that would need 40–90GB of RAM can run on 16–24GB.
Access to genuinely capable models: You’re not compromising down to a smaller model — you’re running the full thing.
No quality degradation: The output is identical to running with full RAM. The model doesn’t know its weights are coming from disk.

Other agents ship a demo. Remy ships an app.

React + Tailwind ✓ LIVE

API

REST · typed contracts ✓ LIVE

DATABASE

real SQL, not mocked ✓ LIVE

AUTH

roles · sessions · tokens ✓ LIVE

DEPLOY

git-backed, live URL ✓ LIVE

Real backend. Real database. Real auth. Real plumbing. Remy has it all.

What You Give Up

Tokens per second: Inference is slower than native RAM loading. How much slower depends on your SSD speed, the model, and how effective the prefetching is. Expect noticeably fewer tokens per second than a fully RAM-loaded run.
SSD write endurance (minor): Heavy inference use means lots of reads. SSDs have endurance ratings. For most users, this isn’t a practical concern — but it’s worth knowing.
Latency on first token: The initial load can take a moment as the first experts get pulled from disk.

For interactive use — asking questions, generating code, writing — the speed trade-off is usually acceptable. For batch inference or applications that need to process thousands of requests, you’d want native RAM or cloud infrastructure instead.

Who This Is Actually For

SSD streaming is a specific solution for a specific problem. It’s worth being honest about the audience.

It’s a good fit if:

You want to run a high-capability MoE model locally and have 16–24GB RAM
You have a fast NVMe SSD with adequate free space
Your use case is interactive or low-throughput (personal use, development, research)
You care about privacy or offline access and can’t or don’t want to use cloud APIs

It’s probably not the right fit if:

You need high-throughput inference (many requests per minute)
Your SSD is slow (SATA or older NVMe)
You’re building a production system that serves multiple users
The latency trade-off would visibly degrade your end-user experience

It’s also worth noting that SSD streaming is most relevant for MoE architectures. Dense models don’t have the same natural partition — there’s no concept of “active vs. inactive” weights at inference time in the same way. Applying streaming to a dense model would require different approaches and would likely be significantly less efficient.

Where Cloud Models Change the Calculation

For most people building AI-powered applications, the local hardware constraint is the wrong problem to be solving. Running a model locally makes sense for privacy-critical applications, offline use, or research. But if you’re building something that needs to work reliably, serve multiple users, or access the best available models, cloud inference is the practical path.

This is where something like MindStudio fits into the picture. MindStudio gives you access to 200+ models — Claude, GPT-4o, Gemini, Mistral, and others — without managing API keys, worrying about RAM constraints, or thinking about SSD speeds. You pick the model that fits your task, and the infrastructure is already handled.

For teams building AI agents and automated workflows, that matters more than most people realize. The bottleneck isn’t usually “can I run this model” — it’s “can I connect this model to the right data, trigger it at the right time, and get the output into the right system.” MindStudio’s visual workflow builder handles exactly that: connecting models to business tools, building multi-step pipelines, and deploying without writing infrastructure code.

Other agents start typing. Remy starts asking.

YOU SAID "Build me a sales CRM."

REMY ASKS

01 DESIGN Should it feel like Linear, or Salesforce?

02 UX How do reps move deals — drag, or dropdown?

03 ARCH Single team, or multi-org with permissions?

Scoping, trade-offs, edge cases — the real work. Before a line of code.

If you’re interested in what capable models can actually do in production rather than in your terminal, you can try MindStudio free at mindstudio.ai. It’s a different solution to the same underlying goal: getting access to capable AI without the friction.

Practical Setup Considerations for SSD Streaming

If you’re interested in trying expert streaming locally, here’s what to think through before starting.

Storage Requirements

Large MoE models are large. Mixtral 8x7B at 4-bit quantization is around 26GB. Larger models like Mixtral 8x22B run over 80GB even quantized. Make sure you have adequate free space — running out mid-inference is painful.

Quantization Still Helps

SSD streaming and quantization work together. A 4-bit or 8-bit quantized model still produces the expert streaming benefit, and smaller file sizes mean faster disk reads. GGUF format (used by llama.cpp) and similar quantized formats are generally compatible with streaming approaches.

RAM Allocation

You’ll want to be deliberate about what stays in RAM versus what streams from disk. The routing logic, attention layers, and other non-expert components should stay in RAM for speed. Most expert streaming implementations handle this automatically, but knowing the split helps you debug if something’s slow.

Thermal Throttling on Laptops

Sustained inference on a laptop will push both CPU and SSD hard. SSDs throttle when they get hot, which will slow your reads and hurt inference speed. If you’re doing long inference sessions on a laptop, this is worth monitoring.

Frequently Asked Questions

What is SSD streaming for AI models?

SSD streaming is a technique that stores model weights on an SSD rather than keeping them entirely in RAM. During inference, the system loads only the weights it currently needs from disk, runs the computation, and discards or cycles them. This reduces RAM requirements significantly while maintaining full model quality, at the cost of some inference speed.

Does SSD streaming reduce model quality?

No. The model weights are identical — they’re just stored on disk rather than RAM. The outputs of inference are the same as if you’d loaded everything into RAM. Quality is only affected if you also apply additional quantization on top of the streaming setup, which is separate.

What kind of SSD do I need for AI model streaming?

You need an NVMe SSD, ideally PCIe 4.0 or better. These drives achieve sequential read speeds of 5,000–7,000 MB/s, which is fast enough for prefetching to work effectively. SATA SSDs (capped around 550 MB/s) are generally too slow for comfortable inference. PCIe 3.0 NVMe drives are workable but will show more latency.

Why does SSD streaming work better for MoE models?

Mixture of Experts models only activate a fraction of their expert networks per token — typically two out of eight or more. This creates a natural partition between “weights needed right now” and “weights that can stay on disk.” Dense models don’t have this partition, so streaming them is less efficient — you’d need all weights for every forward pass.

How does Dwarf Star differ from just using swap space?

REMY IS NOT

✕a coding agent
✕no-code
✕vibe coding
✕a faster Cursor

IT IS

✓a general contractor for software

The one that tells the coding agents what to build.

Operating system swap space uses disk as a passive overflow when RAM fills up. It’s not optimized for sequential weight loading and lacks any awareness of model structure or inference patterns. Dwarf Star actively manages which weights to load, when to load them, and uses prefetching based on routing decisions to hide disk latency. It’s a purpose-built system, not a generic fallback.

Can I use SSD streaming for production applications?

For low-traffic or single-user scenarios, it’s viable. For production systems handling multiple concurrent users or requiring high throughput, the latency and bandwidth limitations of SSD streaming become problematic. In those cases, cloud inference or dedicated GPU infrastructure is more appropriate.

Key Takeaways

RAM has historically been a binary constraint for local AI models — either the full model fits or it doesn’t run. SSD streaming turns it into a sliding scale.
MoE architectures make SSD streaming practical because only a subset of expert weights are active at any given time — the rest can live on disk without affecting output quality.
Dwarf Star implements this by streaming expert weights from SSD to RAM on demand, with prefetching to minimize latency impact.
NVMe SSD speed is critical. PCIe 4.0 drives make this approach genuinely usable; slower storage undermines the value.
The trade-off is tokens per second, not quality. Outputs are identical to a fully RAM-loaded run — inference is just slower.
For most people building AI-powered applications, cloud inference through a platform like MindStudio eliminates the hardware constraint entirely and lets you focus on what the models actually do rather than how to make them run.

New research: The Core Web Vitals thresholds you trust might be wrong for your site

2026-07-17T18:16:45Z

I decided to dig into my questions properly. I pulled a month of real user data from ten leading online retailers and built a correlation chart for each one, plotting session-level Largest Contentful Paint (LCP) data against bounce rate for each site.

Before we get into the findings, first I want to cover what a performance correlation chart is and what it shows you.

A correlation chart is a powerful data visualization that shows you the relationship between your page speed metrics and your business and user engagement metrics.

To illustrate, here’s the correlation chart for the first retail site in this research:

Correlation charts are generated using real user monitoring (RUM) data. They give you a histogram view of all your user traffic, broken out into cohorts based on performance metrics, such as Start Render, Largest Contentful Paint, Interaction to Next Paint, and more. Each cohort shows you the median time for whatever metric you’re tracking for the session. (Those cohorts are represented in the yellow columns in the chart above.)

The next layer of the chart is where things get really interesting.

You also get an overlay (the blue line in the chart above) that shows you the business or user engagement metric – commonly conversion rate or bounce rate, but there are many more – that correlates to each of these cohorts. This lets you see at a glance how closely your business/engagement metric aligns with the speed of your site.

For this investigation, I focused on bounce rate rather than conversion rate because it’s captured by default in most RUM tools, so there’s no extra instrumentation required. It’s also a good directional proxy. In my experience, if you see a relationship between a performance metric and bounce rate, there’s a good chance a similar relationship exists with conversion rate, too.

On the correlation chart for each site, I marked three points:

If you’re not familiar with the performance plateau, I’ve written about it before. Short version: every site eventually hits a point where shaving off more milliseconds stops moving the needle on engagement. Below that point, speed matters a lot. Beyond it, you’re optimizing for nothing.

Across all ten retailers, the optimal LCP — the point tied to the best bounce rate — ranged from 100 milliseconds to 1 second.

Not 2.5 seconds.

Every one of these sites saw its best engagement well before Google’s “good” threshold. For several sites, 2.5 seconds was already sitting on the plateau – the zone where speed had stopped helping long before.

I want to emphasize this before I go on: this is NOT a new number for you to chase. 100ms–1s isn’t a universal target any more than 2.5 seconds is. That’s exactly the trap we’re trying to avoid. What it tells you is that “good enough” – for real users on real sites – can be much faster than the industry standard suggests. The only way to know your number is to look at your own data.

The start time for the performance plateau ranged from 500 milliseconds to 5.6 seconds across all ten sites. This wasn’t surprising. There’s typically a lot of variance in how the performance plateau manifests across different sites.

What was surprising was that, for four of the ten sites, the performance plateau started before 2.5 seconds. By the time those four sites met Google’s recommended threshold for “good” Largest Contentful Paint time, they were already on the performance plateau.

In other words, even though these four sites are technically fast enough for Google, they’ve actually bottomed out in terms of bounce rate.

At the risk of repeating myself (but it bears repeating), you should never assume that meeting external thresholds is actually helping your business. You need to look at your own data to set meaningful targets.

Below, I’m sharing the correlation chart for each of the ten retailers, along with what stood out in each one: where the optimal LCP landed, where the plateau began, and anything else worth noting about the shape of the curve.

To be completely candid, even though I hypothesized these results, I was surprised at how consistent they were.

It’s tempting to look at “100 milliseconds to 1 second” and treat it as a new target to chase. Please, please don’t. As I mentioned earlier, that’s exactly the trap we’re trying to avoid. Consider this research a methodology, not a number.

The good news is that the method is straightforward:

Pull your own RUM data.
Build a correlation chart plotting LCP (or any other user-oriented metric) against bounce rate.
Find where your bounce rate is actually at its best, and where your plateau begins.
Set your performance goals based on that – not on a number based on millions of sites you don’t run.

Your optimal LCP might turn out to be 1.8 seconds. It might be 400 milliseconds. It might be 5 seconds. Who knows! There’s no way to know without looking. But if these ten retailers are any indication, “fast enough” may be faster than the industry standard suggests. Treating 2.5 seconds as your finish line could mean leaving engagement and revenue on the table without realizing it.

If you run this analysis on your own site, I’d love to hear what you find! If you’re an Embrace/SpeedCurve customer and you want to know how to do your own investigation, let me know and I’ll be happy to help.

A Boring Tiny Tooling paradigm

2026-07-02T05:36:29Z

6/1/2026 ☼ product ☼ consulting ☼ AI ☼ BTT

tl;dr: Small-medium businesses are missing a huge opportunity because of three imagination failures: they believe digital transformation must be big and disruptive, they haven’t recalibrated software buying for the collapse in software development costs, and they don’t realise this economic shift requires a different consulting approach. These businesses need Boring Tiny Tools (very narrowly scoped software automating repetitive workflows) built by amorphous consultants who observe actual operations before proposing solutions.

Over the last year, I’ve been writing about and doing a lot of amorphous consulting while using the meaningmaking lens to think practically about what human work should be in a time of AI. In doing that, I’ve spent a lot of time both in and lurking at the edges of conversations with people working at the cutting edge of AI use. I recently had a long call with Rob Miller, a software developer and independent consultant who emailed me after reading an essay I wrote about Boring Tiny Tools. (A BTT is a very narrowly scoped piece of software that automates boring, routine workflows without disrupting other workflows.)

Rob told me about spending a single afternoon building a BTT for his wife’s firm, a barristers’ chambers. This BTT is a court submission preparation tool that works on-premises to be compliant with data protection. The tool makes PDF multi-document bundles, automatically creates hyperlinked tables of contents, and formats the bundle to court specifications. Error-prone humans used to spend hours doing this boring job manually. Now it is a software-enabled, reliable automated process that takes minutes. Multiply that across 50-60 people and thousands of cases per year in the organisation and the savings quickly stack up.

Rob’s BTT is yet another validation of the new emerging paradigm I’m seeing (and am maybe a part of) at the place where software, consulting, and digital transformation touch each other. The greatest potential beneficiaries of this new paradigm are small-medium businesses with a particular profile:

Small enough to have no in-house transformation tools team, but
Large enough that efficiency gains from transformation save a significant amount of money annually (say, millions vs tens of thousands of dollars), but
Not so large that transformation savings justify spending hundreds of thousands of dollars on conventional transformation consultants to write bespoke business process software.

Three imagination failures

Unfortunately, these small-medium businesses don’t actually seem to be benefiting yet. It’s more accurate to say that the new Boring Tiny Tooling Paradigm is struggling to emerge but three related imagination failures keep getting in the way.

These three imagination failures affect the idea of what transformation should look like, what the software that enables it should be, and what kind of consulting approach to buy:

Inability to conceive of digital transformation as anything other than Grand and Disruptive. Only big strategic software deployments, enterprise platforms, major integrations, or organisation-wide rollouts are legible as transformation. BTTs that address a profusion of tiny business processes don’t read as transformation, even when the return on investment is both immediate and measurable.
Inability to recalibrate the scale of software bought to account for the sudden collapse in the cost of writing software. Software that used to cost tens of thousands of dollars and take months to write now costs thousands of dollars and takes days to write. It’s finally economically feasible to write BTTs to automate tiny, boring business processes.
Inability to see that the collapse in software economics necessitates a different transformation consulting approach. That BTTs are suddenly cheap, fast, and viable to build should be feeding through to the kinds of transformation consulting services businesses buy … but businesses continue to want conventional consulting services that price and deploy oversize, too-expensive transformation solutions based on pre-collapse software economics.

A consulting problem

That last imagination failure is the big one.

When businesses can only imagine transformations that are big and legible, they look for and engage conventional consultants who promise big legible transformations. These conventional consultants scope and deliver expensive and needlessly enterprise-scale solutions to justify their big fees and promises of big transformations.

(It is now an open secret that the big consulting firms are using AI tools internally to cut development time while still charging out at pre-AI prices. They keep the difference and have no incentive to educate clients about these new economics.)

Meanwhile, what the small-medium business actually needs is a different consulting approach entirely: What I call amorphous consulting. In contrast to conventional consulting, amorphous consulting comes into a client engagement with an open mind about the problem and the solution.

Rob’s bundle preparation tool was a solution scoped out of a deep understanding of the user’s real operational problems. The amorphous consultant gets this kind of deep understanding by spending time observing what’s happening in the client’s organisation. This approach to consulting leads to tiny, boring, highly effective solutions. When software is scoped and built by amorphous consultants, it is more likely to be

Minimal and fast to build (because the scope will be limited to only features needed to address the actual problem to be solved),
Functional within actual operational constraints faced by users on the ground,
Actually adopted by users with operational responsibilities, and
Undisruptive to other workflows.

These are all desirable characteristics of transformation software.

How Rob scoped and built his BTT — prompted by a serendipitous observation of his wife doing silly rote work to prepare a bundle manually while on holiday in Champagne — is an example of the kind of amorphous consulting approach that most small-medium businesses actually need.

Unfortunately, most of these businesses don’t know how to procure these kinds of amorphous consulting services where the scopes of work and target deliverables can’t (and shouldn’t be) defined in advance.

Imagination is the hard part

As I wrote in AI’s missing middle, small-medium firms in traditional industries represent a massive market and a massive missed opportunity. (Policymakers and investors might want to take note.) Small-medium firms don’t need fancy conventional consultants and their overscoped and expensively disruptive software. What they do need are Boring Tiny Tools scoped and built by amorphous consultants.

None of this is expensive or technically challenging. All it takes is better imagination.

Even with a silver bullet, you still have to aim

2026-06-30T03:44:53Z

As George Lucas once said, for some reason: history is like poetry. It rhymes.

Our industry is certainly rhyming with a historical era I love to pick on: the boom in tech investment that followed the one-two punch of the Dot Com Bust and the Financial Crisis. Companies and their investors were desperate to find that one silver bullet that would turn everything around. Back then, that silver bullet was Design Thinking. Big tech poached designers from celebrated consultancies and airdropped them into pure chaos without a plan beyond “fix the product for us.”

Ten years after design thinking’s heyday, there’s a new silver bullet that everyone has pinned their hopes on, without a clear idea of what exactly it will do. At this point in the hype cycle, you’ve seen the numbers for yourselves: billions in investments, and loud cricket noises for returns. Anecdotes of individual productivity gains are easy to come by, and yet they do not translate to improvements in the bottom line!

What’s actually making teams move faster is org design. Taking care of organizational dysfunction may be a side effect of “AI transformation” but it sure seems to be offering the majority of the benefits.

❝

Consider this story when the right handpicked people completed in 3 hours, a piece of work estimated at 10-12 weeks. That story was from roughly ten years ago, well before any of the AI tools appeared on the scene. It didn’t rely on magical tooling, it just needed us to undo some organizational dysfunction and to put the right people in the room to collaborate on the problem.

The purpose of a system is what it does

“AI works” vs “AI doesn’t work” is an exhausting debate; no one is ever convinced of anything and the only winners of the arguments are keyboard manufacturers. I’m not going to go over the various hallucination rates and studies on how much longer it takes to check an AI’s output for errors than to just do it yourself. Even with 0% error rates, those figures would be besides the point. The biggest problem — and don’t take my word for it, listen to some Nobel laureate guy — is that the bottleneck lies upstream of your Jira ticket:

❝

Most research on AI productivity is overblown because it overwhelmingly focuses on easy, well-defined tasks where context is clear. These are not representative of the economy, which simply isn't set up with so many of those.

We are not in the future. It is still 2016. It is still 1986, if you are so inclined. We have hardly made any strides in our ability to define tasks and their context. What baby steps we’ve made in the right direction have been swallowed up by a great cloud of noise. For every Toyota Way book written in the 90s, there is a Ford counterexample being eagerly practiced today.

AI has intensified this problem, as it intensifies all things to which it is applied. But the problem did not originate with AI. Before Claude could learn to talk like the median LinkedIn influencer, the median LinkedIn influencer had to arrive at that mindset under their own power:

❝

Criticism of the […] slop genAI produces invariably fails to acknowledge how much slop-ification was already characterising so many discourses prior to retail LLMs: PEEL paragraphs, Powerpoint slides, TED talks, Big Consultancy strategy canvases, PR company produced apologies to customers, LinkedIn posts...

The bottleneck is the context in which the work takes place

For every step a would-be thought leader took towards “synergistic leadership” or “growth-hacking paradigms” they took a step away from reality. And corporate bullshit (not my name for it but a real term of the art, take it up with Cornell!) is not the only layer lubricating a product’s slide into slop. Robert Fabricant has a good layman’s overview of how little a modern product is influenced by a given design decision, and how much is a consequence of business and technology decisions that stack up over time without a single decision-maker deliberately steering things to be so. I recommend giving this one a read, as a means of cognitive estrangement from the ways of working we take for granted, because we are too close to them to realize how ridiculous they really are.

Unfortunately, many tech leaders do not take the time to reflect on whether the system actually makes sense. They shirk the responsibility given to them — to build an effective system — and instead whip the existing, faulty system to run faster and faster, until all the wheels fall off:

❝

“Ship more faster now” is not a product strategy; it’s a psychosis. …

Reframe craft as care for collaborators, users, and the collective aspiration to create something useful, meaningful, and significant. Craft is a social act that binds teams around shared values and takes time.

But how do we make this reframe happen? How do we seize control of our discipline’s story?

That’s where my head’s been at for years — before the Picnic, before my work for NN/g, even before I started publishing with the UX Collective. But only lately have those thoughts come together into something actionable enough to share. I’ll be writing on this topic through July, but the capstone of these ideas will be my one-day workshop with Throughline on July 31st.

Flex those professional development budgets and join me in taking back control of the conversation.

— Pavel at the Product Picnic

untitled

2026-06-21T18:45:46Z

Moving beyond hunch and silencing the sceptics. Over the years, Conway’s Law has resurfaced as a guiding principle to organise teams. Yet, many people still hesitate to accept the idea. Ultimately, it remains a hypothesis. Melvin Conway did not provide any proof when publishing the thesis. However, in the meantime, decades later, a good deal of academic research supports the thesis. Ultimately, Harvard and MIT researchers provided robust empirical validation for the hypothesis across the software industry, further confirmed by the CHAOS Report from the Standish Group, reinforcing The Mirroring Principle as an undeniable organisational reality.

Henderson and Clark observed in 1990 that established firms are unable to adopt architectural innovation because the architectural knowledge is embedded in the communication structures of the organisation, preventing them from adopting new designs.

We show that architectural innovations destroy the usefulness of the architectural knowledge of established firms, and that since architectural knowledge tends to become embedded in the structure and information-processing procedures of established organizations, this destruction is difficult for firms to recognize and hard to correct. Architectural innovation therefore presents established organizations with subtle challenges that may have significant competitive implications.

– Architectural Innovation: The Reconfiguration of Existing Product Technologies and the Failure of Established Firms, Henderson and Clark, 1990

“Architectural innovation” is defined as innovations that change the way product components are linked together, while leaving the core design concepts — and thus the basic knowledge underlying the components — untouched. That does not mean that components remain untouched. Often, architectural innovation is triggered by a change in a component that triggers new interactions and connections with other components.

The study’s outcome is based on the analysis of 20 years of data (1962-1986) from the photolithographic alignment equipment industry, where four organisations subsequently took over market leadership because a competitor uncovered an architectural innovation that the previous leader was unable to detect due to its organisational structure.

There has been growing evidence of numerous technical innovations that involved apparent modest changes to existing technology, but that had dramatic competitive consequences. In the mid-1950s, RCA developed a prototype for a portable radio receiver. It utilised technology that RCA mastered well. Yet, they did not pursue the idea. Until Sony, a small, relatively new company, entered the US market with portable radio receivers. RCA remained a follower. The irony was that Sony produced radios during many years using RCA’s licensed technology. The case of Xerox, in the mid-1970s, although the inventor of the plain-paper copier, was, during eight years of strategic errors, unable to compete with competitors offering smaller, more reliable copiers. Xerox lost half of the market. For a more recent example, we look at the German car industry, which had trouble marketing electric cars until Tesla, and later the Chinese car manufacturers took over the market. Meanwhile, Audi closes its Belgian factory that produces the expensive electric SUV because of a market mismatch.

This inability to adopt architectural innovation happens because an organisation organises itself around the product’s key components that make up the “dominant design”. The “dominant design” is characterised by a general acceptance of a particular product architecture. It incorporates a range of basic design choices that are not revisited — think car platforms, or in software, a CRUD design. It results in communication channels between the design teams that reflect the organisation’s knowledge about the critical interactions between these components. Thus, an organisation’s communication channels embody the organisation’s architectural knowledge that is critical to effective design. From this moment on, organisations incline to focus only on “incremental innovation”, tending to reinforce the competitive positions of established firms as it builds on their core competencies.

As such, the architectural knowledge is a mirror of the communication structures organised around the product being designed. Somehow, this is the knowledge parallel of Conway’s thesis “[Any] organisation which designs systems is constrained to produce designs which are copies of the communication structures of the organisation”. If communication structures dictate design (Conway, 1968), they also dictate what an organisation is capable of knowing (Henderson and Clark, 1990).

Henderson and Clark establish that, given how knowledge is organised in firms, learning about changes in the product architecture is unlikely to happen. It requires explicit management, or even a different organisation. To counter this, the authors suggest cross-functional teams as a possible response to the danger of architectural knowledge becoming embedded within tacit communication lines.

Recently, I was front-seat observing an organisation unable to run only a simple experiment, which would let them eventually discover a potential innovation, merely because of the structures in place.

Because the communication channels are “frozen” around the old product architecture, the knowledge is also frozen; the organisation cannot distinguish or adjust to a possible new architecture reality, or even conceive a new design reality. This is Conway’s Law working back over time (more on this in Dynamics: Conway’s Time Component).

The Aircraft Engine/Automotive Validations (2004, 2007)

In 2004, Sosa et al. published a study that investigated the design process of a commercial aircraft jet engine, the PW4098, at Pratt & Whitney. They studied how design interfaces in the product architecture map onto communication patterns within the development organisation. More specifically, they explored whether organisational problems exist at the boundaries of the components inside the jet engine, i.e. whether a misalignment exists between the design team boundaries and system component interfaces.

According to Henderson and Clarke, architectural knowledge is embedded in the communication structures of organisations; therefore, organisations need to understand how they manage embedded knowledge. This is especially relevant when designing new products. In this context, system design interfaces are the primary sources of team interdependence, or at least, they should be. Meaning, when a specific team designs their own component with an interface to a neighboring component, there should be a corresponding technical interaction between the two respective teams. These are the expected cases. On the other hand, unmatched design interfaces and unmatched team interactions are unexpected. As it seems, these cases were largely ignored in previous research.

However, we believe that considering these cases is important because their existence would indicate that not all known product-related interdependences are addressed by direct technical communication, and that technical communication (where not expected) may uncover undocumented product-related interdependences.

– The Misalignment of Product Architecture and Organizational Structure in Complex Product Development, Sosa et al, 2004

Finally, the study concludes that …

… we provide empirical evidence that product ambiguity exists, and it is more likely to be present across organizational and system boundaries

– The Misalignment of Product Architecture and Organizational Structure in Complex Product Development, Sosa et al, 2004

Meaning, that there is a great likelihood that some of the system component interfaces are not covered by team interactions, which can lead to major setbacks involving significant rework that can be extremely costly. The paper, exactly, refers to two major setbacks during the design of that specific jet engine.

What they are essentially saying here is: If there is a misalignment between the organisation and the product architecture, there will be problems.

In 2007, Gokpinar et al. conducted a study examining the design of cars in the automotive industry.

[…] a key challenge for product development organizations is matching the organization to the product being developed. This involves two fundamental problems: (1) How to assign people to the parts and subsystems that make up the product, and (2) How to ensure that people communicate/collaborate effectively in the performance of design tasks.

– The Impact of Misalignment of Organization Structure and Product Architecture on Quality in Complex Product Development, Gokpinar et al, 2007

The research defined a new metric coordination deficit, which quantifies mismatches between product architecture and organisation structure, and investigated the effect of coordination deficit on product quality.

They concluded that mismatches between product architecture and organisational structure are positively associated with quality problems.

Our results suggest that misalignment of the design organization with the product architecture negatively affects product quality”

– The Impact of Misalignment of Organization Structure and Product Architecture on Quality in Complex Product Development, Gokpinar et al, 2007

Once more, if there is a misalignment between organisation and product architecture, there will be problems.

The Micro Software Validation (2006)

In 2006, Cataldo et al. investigated the consequences of congruence on the speed and efficiency of software development work inside a single software project. They investigated the impact of misalignment.

This is particularly important in product development organizations which organize themselves around their products’ architectures because the main components of their products define the organization’s key sub-tasks.

– Identification of Coordination Requirements: Implications for the Design of Collaboration and Awareness Tools, Cataldo et al, 2006

This matters because, as architectural knowledge is embedded in the communication structures of organisations (Henderson and Clark, 1990), any minor changes to the product architecture can disrupt the organisation’s ability to coordinate effectively.

Ultimately, Cataldo et al. found that tasks were completed more rapidly when the patterns of communication between team members were congruent with the patterns of interdependency between components (more on congruence in Mechanics: The Mathematical & Geometrical Shades).

Congruence between coordination requirements and coordination activities shortened development time.

– Identification of Coordination Requirements: Implications for the Design of Collaboration and Awareness Tools, Cataldo et al, 2006

Fundamentally, the research indicates, if teams are aligned with the components they work on, teams are more efficient.

The Macro Software Validation (2012)

In 2012, we had the first empirical evidence for the software industry provided by the research paper Exploring the Duality between Product and Organizational Architectures: A Test of the “Mirroring” Hypothesis from Baldwin et al.

One of their findings was: software products tend to mirror the structure of the organisation in which they are developed.

We find strong evidence to support the hypothesis that a product’s architecture tends to mirror the structure of the organization in which it is developed.

– Exploring the Duality between Product and Organizational Architectures: A Test of the “Mirroring” Hypothesis, Baldwin, MacCormack and Rusnak, 2012

Source: Sketchplanations https://sketchplanations.com/conways-law

To validate this hypothesis, they compared open-source with closed-source software. Based on the idea that the open-source communities are loosely coupled organisations by design because contributors have diverse goals, belong to different organisations, work in different geographical locations and time zones, and have no formal authority to guide development activities. Whereas closed-source organisations are more tightly-coupled, as they consist of smaller, co-located teams that share common goals, and are dedicated full-time to products and have formal decision-making authority to guide development.

The study’s approach focused on comparing the architectures of open-source and closed-source software products that have similar functions and are of a similar size. They identified five software categories: financial management, word processing, spreadsheets, operating systems and databases. For each category, they selected an open-source and a closed-source software product.

Because commercial firms are somehow reluctant to share their code base, especially for research comparing with “free” equivalents, researchers did not always have a closed-source codebase at their disposal. When commercial options were unavailable, they turned to formerly closed-source products that had been open-sourced. In those instances, they analysed the first open-source version.

So they ended up comparing:

For financial management: GnuCash vs MyBooks
For word processing: Abiword vs StarWriter
For spreadsheets: Gnumeric vs StarCalc
For operating systems: Linux vs Solaris
For databases: MySQL vs Berkeley DB

The paper primarily explores whether organisations with distinctly different forms develop products with distinctly different architectures.

The outcome of the study was that loosely-coupled organisations, such as open-source, tend to develop products with a more modular architecture than tightly-coupled organisations such as commercial firms. This confirms Parnas’ information hiding concept (Parnas 1972), which allows individual, isolated contributors to work independently. It also reaffirms Henderson and Clark’s observation. Loosely coupled, open-source organisations prevent architectural knowledge from becoming a central, siloed trap. They adopt cross-functional, loosely coupled teams to democratise knowledge. Tightly-coupled organisations, on the other hand, stumble when faced with innovations, because knowledge is tightly frozen, siloed, in the structures of the organisation.

The Industry Reality Check (1995-2020)

We know why projects fail, we know how to prevent their failure – so why do they still fail?

– Cobb’s Paradox, Martin Cobb, CIO for the Secretariat of Treasury Board of Canada, CHAOS University, 1995, published in Unfinished Voyages I, Standish Group, 1995

When organisations keep treating software products as mere, rigid projects, the resulting architecture and outcomes for users will be notably poor, leading to surprising failures.

From 1994 until 2020, the Standish Group published the CHAOS Report, a study on the success, failure, and challenge rates of IT projects.

Year	Success	Challenged	Cancelled
1994	16%	53%	31%
2000	28%	49%	23%
2002	34%	51%	15%
2004	29%	53%	18%
2006	35%	46%	19%
2008	32%	44%	24%
2011	39%	39%	22%
2012	37%	46%	17%
2013	41%	40%	19%
2014	36%	47%	17%
2015	36%	45%	19%
2020	31%	50%	19%

– source: CHAOS Reports 1995, 2009, 2015, 2020, Standish Group.

Although this may look baffling at first, this stagnation blindsides no experienced practitioners. The numbers reveal a flatline trend. Despite agile and DevOps, the success rate is capped at approximately 30%.

Till 2015, the Standish Group focused mainly on classic project management requirements to measure success, such as well-defined upfront requirements, proper planning, skilled project managers, as well as user involvement and project sponsor. Success was defined in classic PMI terms, the Iron Triangle, i.e. on time, on budget, with all features as initially specified. Then, in 2015, a first significant shift happened. The success resolution definition changed. It was still on time and on budget, however, with satisfactory results. All requirements no longer have to be implemented, since data shows that some projects meet the Iron Triangle yet do not return value. In the end, satisfaction and value are greater when fewer features are delivered, but meeting obvious user needs. This is aligned with Lean principles, as well as another Standish Group study revealing that 64% of delivered features are rarely used. Later confirmed by the 2019 Pendo Feature Adoption Study, revealing 80% of features see low to zero adoption.

By 2020, the Standish Group had pivoted and reported an astonishing outcome that contradicts their early recommendations. Stop treating software products as a project because, by definition, projects have a start and an end, but products do not. They explicitly stated that project managers harm software products and increase the failure rate. Specifically, the report demonstrates that project managers often generate more paperwork, resulting in slower decision-making that inevitably slows down feedback. While, once more, not surprising for experienced practitioners — it was already established long before, and brought to the public in 2017 with the #NoProjects movement — the Standish Group brought it to the masses.

Ultimately, the Standish Group concluded that project success is defined by Good Sponsor, Good Team, Good Place, and Short Decision Latency. To a large extent, this aligns with the 2001 findings from the book Good to Great. Organisations become great when they have Level 5 Leadership (Good Sponsor) to build enduring greatness through a paradoxical blend of personal humility and professional will. They focus on First Who … Then What (Good Team), first securing the right people before figuring out where to drive it, before vision, before strategy, before tactics, before organisational structure, before technology. Lastly, they Confront the Brutal Facts of Reality (Good Place), by fostering a climate where the truth is heard.

These crucial prerequisites enable Short Decision Latency, allowing information to circulate rapidly. However, low decision latency is entirely dependent on communication paths. Since Conway’s Law dictates that systems reflect the organisation’s communication structures, high-latency organisations stuck in silos are unable to produce high-velocity, decoupled system designs. This architectural congestion hampers communication and impedes overall success.

Ultimately, the CHAOS Report recommends moving away from “projects” and adopting Infinite Flow, which Lean has been promoting since the 1980’s. Somehow, this is the business world finally accepting what Baldwin et al. proved in 2012: tightly coupled organisations produce tightly coupled system architectures that work against flow, while loosely-coupled organisations thrive because they produce loosely coupled system architectures that enable flow, short time to markets, and consequently accelerated feedback to meet user expectations better. The 2020 CHAOS Report Good Place — where truth is heard to reduce Decision Latency — is exactly the type of environment needed to break the “frozen”, siloed, tacit communication lines Henderson and Clark blamed for corporate blindness.

Conclusion

To conclude, the tech industry spent decades trying to fix management processes, only to realise they were fighting a losing battle against the structural gravity of Conway’s Law and its implications on innovation and market leadership. Mediocrity is the default. Success is optional. Just saying …

The Series: Navigating the Shades

Beyond the Shades of Conway’s Law series:

Foundations: The Origin & The Mirroring Principle - How the worlds of organisation and product design observed the same thesis independently.
Validation: The Research & Reality Check - Moving beyond the “hunch”, how researchers proved the Law in different industries, but especially in software.
Mechanics: The Mathematical & Geometrical Shades - The geometry of design: from mathematical isomorphism, homomorphism, congruence to compatibility.
Strategy: Reversing the Law - How the system ultimately forces the organisation to change versus deliberately changing the organisation.
Scale: Conway’s Corollary - The required organisational flexibility.
Dynamics: Conway’s Time Component - The “Engineer Half Life” and why architecture is “sticky” long after teams change.
Conclusion: The Different Lenses - A concluding look at how we perceive organisations and their systems.

Bibliography

How Do Committees Invent?, Melvin Conway, 1968
On the Criteria To Be Used in Decomposing Systems into Modules, Parnas, 1972
Architectural Innovation: The Reconfiguration of Existing Product Technologies and the Failure of Established Firms, Henderson and Clark, 1990
CHAOS Report 1995, Standish Group, 1995
Unfinished Voyages I, Standish Group, 1995
Good to Great, Jim Collins, 2001
The Misalignment of Product Architecture and Organizational Structure in Complex Product Development, Sosa et al, 2004
Identification of Coordination Requirements: Implications for the Design of Collaboration and Awareness Tools, Cataldo 2006
The Impact of Misalignment of Organization Structure and Product Architecture on Quality in Complex Product Development, Gokpinar et al, 2007
CHAOS Report 2009, Standish Group, 2009
Exploring the Duality between Product and Organizational Architecture: A Test of the “Mirroring” Hypothesis, Baldwin, MacCormack, Rusnak, 2012
CHAOS Report 2015, Standish Group, 2015
Standish Group 2015 Chaos Report - Q&A with Jennifer Lynch, InnoQ, 2015
Are 64% of Features Really Rarely or Never Used?, Mike Cohn, 2016
The 2019 Feature Adoption Report, Pendo, 2019
Chaos Report — why this study about IT project management is so unique about the CHAOS Report 2020, Radek, The Story, 2024

untitled

2026-06-14T16:10:31Z

The Plan

In the beginning was the Plan.

And then came the Assumptions.

And the Assumptions were without form.

And darkness was upon the face of the Workers.

And they spoke among themselves, saying, "It is a crock of shit, and it stinketh."

And the workers went unto their Supervisors and said, "It is a pail of dung, and none may abide the odour thereof."

And the Supervisors went unto their Managers, saying, "It is a container of excrement, and it is very strong, such that none may abide by it."

And the Managers went unto their Directors, saying, "It is a vessel of fertiliser, and none may abide its strength."

And the Directors spoke amongst themselves, saying one to another, "It contains that which aids plant growth, and it is very strong."

And the Directors then went onto the Vice Presidents, saying unto them, "It promotes growth and is very powerful."

And the Vice Presidents went unto the President, saying unto him, "This new plan will actively promote the growth and vigour of the company; with powerful effects."

And the President looked upon the Plan, and saw that it was good.

And the Plan became Policy.

This is How Shit Happens.

→ Check out more Funny Shit

untitled

2026-06-14T06:26:18Z

Think about the last time you tracked your progress on a personal goal. Maybe you were trying to exercise more frequently, be more patient with a spouse or partner, or learn a new skill. Chances are, you had a rough sense of how it was going. And chances are that sense was more optimistic than the evidence warranted.

This is not a character flaw. It is one of the most reliable findings in behavioral science: We are systematically poor judges of our own progress, and during periods of active change, we tend to get worse.

One reason is what psychologists Terence Mitchell and Leigh Thompson at the University of Washington termed rosy retrospection. This is our consistent tendency to remember past experiences more favorably than we rated them in the moment. When we look back over a month of trying to exercise more or sleep differently, our memory selectively surfaces the days we succeeded. The days we failed are there, but they are not as memorable. We feel we’ve been doing better than we really have.

A second explanation is subtler. Motivation isn’t a fixed state; it shifts gradually, often well before behavior changes in any visible way. The enthusiasm that felt solid in January has usually started to erode by March. But because the erosion is incremental, we rarely catch it in real time. We trust that how we feel today is roughly how we’ve been feeling all along. It almost never is.

Together, these two patterns create a specific failure mode: We assume the feedback loop is running when it isn’t. We trust that because we know our own goals and intentions, we know how we’re actually doing. But intention and behavior are not the same thing. How we feel about a change is not the same as how we’re enacting it. And until we close that gap with something more rigorous than a gut check, we’re optimizing for a version of our progress that may not exist.

In researching what separates change successes from failures at organizations for my book, my coauthors and I found that leaders designing change were consistently more confident about employee progress on change than the employees’ actual behavior or stated experience supported. They trusted their read on the room. The data, when they gathered it, told a different story. The wider the gap, the more likely the change failure.

The same gap operates in personal change. And the same remedy applies: Use measurement, not instinct.

This doesn’t require an elaborate system. Research by James Pennebaker at the University of Texas at Austin shows that the act of writing specifically about your experience—not vague journaling, but naming exactly what you did, how it felt, and where the friction was—produces measurable improvements in self-understanding and in the likelihood of follow-through. Writing forces the brain to translate impression into information. The gap between what you felt and what actually happened becomes clearer.

Social accountability produces a related effect. A study by psychologist Gail Matthews at Dominican University found that people who wrote down their goals and reported progress to a friend were 42 percent more likely to achieve those goals than people who kept them private. Part of what accountability does is discipline the story you tell yourself. You can’t smooth over the gaps as easily when someone else is watching for them.

Both interventions do the same thing: They replace vague impressions with actual information. When people begin gathering that information honestly, they usually discover not only the gaps in their efforts, but also the progress they had stopped noticing.

Teresa Amabile, a psychologist at Harvard Business School, spent years studying what helps people stay motivated during sustained effort. Her research, drawn from the daily diaries of 238 professionals tracked over several months, produced a counterintuitive finding: The single biggest day-to-day motivator wasn’t recognition from a manager or a meaningful reward. It was the perception of making progress on work that mattered. Small wins, when noticed specifically, produced outsized improvements in engagement and follow-through. The operative word is “noticed.” Progress that goes unobserved doesn’t produce the same effect. It needs to be named to become fuel.

This is what honest self-assessment makes possible. When you journal about your week in specific detail, you often discover you accomplished much less than your vague, optimistic impression suggested—and you'll have a record you can point to as concrete evidence. That specificity matters more than it might seem. Ayelet Fishbach at the University of Chicago Booth School of Business has shown across repeated experiments that concrete progress feedback strengthens goal commitment more than encouragement does. What keeps people going isn’t being told they’re doing well. It’s seeing, with some precision, that they truly are.

Motivation Essential Reads

The story you’re telling yourself about your progress probably has some blind spots. But it may also be overlooking real growth that felt too small or too ordinary to count. Learning to see your progress more clearly, without judgment, isn’t self-punishment. It’s an invaluable form of self-support.

untitled

2026-06-11T19:21:22Z

Chances are you used an AI tool before 9 a.m. today. Chances are most of your users didn't.

It's a simple contrast, but Debbie McMahon, VP of Product at Loveholidays, built an entire MXP London session around it and what it means for every product decision. She called it the AI adoption gap, and she was refreshingly clear-eyed about the ways it had distorted her own judgment.

The AI adoption gap and the 0.04%

An infographic Debbie shared on stage breaks the world's relationship with AI into four tiers:

84% of people have never used AI

16% have tried a free chatbot

0.3% pay for AI

0.04% are actively building with it

That last group is the one that fills rooms like MXP London. And Debbie's point is that this creates a fundamental disconnect. The people designing AI-powered products are a statistical anomaly relative to the people they're designing for.

She made that concrete with a story about her friend Agnes. Agnes is a primary school teacher who spends her days helping children from difficult family circumstances settle into school. She's smart, busy, and has a free ChatGPT account she’s never used. Why? Her school doesn't allow AI tools on work devices. The budget isn't there for anything paid. And nobody on staff is confident enough with the technology to know where to start, even if the first two problems disappeared tomorrow.

Agnes is not an edge case. She's a typical Loveholidays customer, and she's who Debbie asked the MXP London audience to hold in mind.

“

We're living in a bubble. We're the 0.04% and our daily experience of AI has essentially nothing to do with our users’ reality.”

Debbie McMahon

VP of Product, Loveholidays

For a look at how AI engagement is trending across real products, Mixpanel's 2026 AI benchmarks draw on nearly 290 billion AI events across 2.6 billion devices, and the picture is more nuanced than adoption headlines suggest.

Three ways the bubble distorts your decisions

Debbie named three patterns she sees in herself and across the industry.

Over-prioritizing AI features. "Have you put something out there just because it was easy to do?" She raised her own hand. The pressure is real: boards want AI in the roadmap, investors want AI in the roadmap, competitors are shipping. And the tooling makes it fast to build something. But fast to build isn't the same as worth building.

Underestimating friction. Most people are forming their views on AI right now without the benefit of product communities or conference sessions to help them process it. That's not a knock on users. They're figuring out whether to trust the technology at the same moment they're being asked to use it. Dropping a disclaimer like "generated by AI, so may not be accurate" into a set of results doesn't help someone who's already uncertain about the output.

Asking users to change behavior. Behavior change is extraordinarily difficult even when people want to change. For users with an ingrained habit and no clear reason to do anything differently, it's close to impossible. The customers who are confused or skeptical about AI aren't going to reroute their entire search behavior for a feature they weren't asking for. If what they want is a keyword search and a filter, that's what they'll reach for.

Go deeper

See whether your AI features are delivering value, or just getting used:

What the data actually showed

Two case studies from Loveholidays that Debbie shared with the kind of candor that makes MXP sessions worth attending.

The room grouping problem. Hotel listings at Loveholidays had accumulated a long tail of duplicated, poorly named room types, a messy, years-old problem that had resisted every previous attempt to fix it. When a team went after it with AI, they went in with confidence. What they found was that the problem was deterministic: two rooms are either the same or they're not, and AI isn't built for that kind of precision. The model grouped rooms in ways that didn't hold up to scrutiny, renamed options customers recognized, and quietly dropped listings that people had been booking. The output looked coherent, but it wasn't.

In Debbie’s own words: "Don't put AI on top of a broken thing. Fix the actual problem."

The Q&A tool. Another team shipped an LLM-based Q&A feature into the pre-booking journey, built around the hypothesis that customers comparing hotels would want to ask questions. The feature got used, a lot, actually. But when the team looked at who was using it, the data didn't match the hypothesis at all.

The users weren't pre-booking customers weighing their options. They were people who had already made a booking, navigating a long and confusing post-booking experience to find basic information about their upcoming trip. And the questions they were asking weren't the ones the team had imagined. The top two were whether their room came with a kettle and whether it had an iron. Third most common: the wi-fi password.

In other words, customers who needed information Loveholidays already had were threading through a complex AI-powered feature just to ask if they needed to pack a travel iron. The team proposed moving the tool into the post-booking journey where the actual demand was. Debbie shut it down, and the reasoning is worth sitting with.

“

Just because it does answer those users’ questions doesn't mean that this is the right solution. We have that information. It’s a fact. We just need to give it to them in a form that they need it in.”

Debbie McMahon

VP of Product, Loveholidays

The post-booking journey got fixed instead, and the Q&A tool got pulled. The story doesn't end there: the team is relaunching the feature with a genuinely different hypothesis behind it. Since that original experiment, Loveholidays launched its own reviews platform, which changes what a Q&A tool can meaningfully do and who it's actually for. There's a real use case now tied to a specific user group, and that's what "not now" looks like when it's used well.

Go deeper

Further reading on building and measuring product adoption in the AI era:

Building different journeys for different people

Loveholidays went to their customers directly and asked how they'd feel about AI-mediated booking experiences. The words that came back included "anxiety," "I trust filters more," and "I prefer clicking." There was also genuine enthusiasm from a portion of respondents. But the skepticism was real, present across a significant share of the audience, and Debbie didn't try to explain it away.

53% of consumers distrust AI-powered search results, according to a Gartner survey published in September 2025. Debbie cited a similar figure on stage and her read was plain: "I can't ignore that. That’s the reality for my customers, and that's where we're starting from."

Her answer is to segment more precisely. AI comfort and interest now sit alongside demographics, holiday type, and referral route as a dimension Debbie's team uses to personalize the experience. Design one AI-mediated journey for everyone and you either hold back the users who want to explore new things or push unwanted change onto the customers who aren't ready for it.

Debbie's dad started using email about a decade after she did. She uses that as a reference point: adoption follows its own timeline, and you can't compress it by shipping a feature and calling it done.

The playground: A structural answer

Recognizing a problem clearly is useful. Building an organizational system to address it at scale is something else. That's what separates this MXP session from a general mindset piece.

Debbie's team built what she calls "out the bubble," an internal prototyping environment that anyone at Loveholidays with access to Codex or Claude Code can use, with no prior technical setup required. It's pre-integrated with the company's design system, data mesh, live pricing, and real images, which matters more than it might sound. When you put a prototype in front of a customer in a travel context, the first thing they notice is whether the hotel price looks real. This system removes that distraction so the feedback is about the experience itself.

The pipeline works in three stages:

Anyone at the company builds a prototype, sometimes in 10 minutes, sometimes a couple of hours, and pushes it to an internal testing environment for NMD review.

Prototypes that clear that bar are automatically routed to Usertesting.com, with the builder setting up the user panel themselves, no dependency on a separate research team.

The ones that pass user testing move to a small live test, pushed out via socials, CRM, or the homepage to a defined customer segment.

The important shift isn't the speed, though. It's that the people with the best product ideas are no longer gated by their ability to build prototypes. The system makes that capability available to the whole company. Fewer ideas get duplicated. More of them get in front of real customers. And when something earns the full investment of a real build, there's actual evidence behind the decision.

Start with the problem. The tools will follow.

Debbie closed by revisiting the Henry Ford quote, carefully, because she's not entirely sure Ford said it. But the argument holds regardless. The car achieved adoption because Ford understood what his customers actually needed. He built a faster horse, not a different animal entirely.

“

AI doesn't let us skip understanding what our users need. If you want people to adopt things, you have to start with their problem.”

Debbie McMahon

VP of Product, Loveholidays

The tools have changed. The product job hasn't. What AI does change is how big your answers can be once you've understood the question clearly. The potential is genuinely extraordinary, but only after you've done the harder work of understanding what your users need and why they'd change.

Remember Agnes from the opening of this piece? The primary school teacher who supports children through difficult family circumstances, has a ChatGPT account she never uses, and has no budget, no tech support, and no time to figure it out herself. She's never going to ask for an AI-powered solution to her problems. But she'd benefit enormously if someone took the time to understand those problems and built something that addressed them, with AI as part of the answer rather than the starting point.

That's what good product work looks like, and as Debbie put it, that's always been the job.

Build the bridge your users are ready to cross

The product teams pulling ahead right now aren't the ones shipping the most AI features. They're the ones who understand the gap between their own relationship with AI and their users', and who've built the systems to keep closing it.

To understand what your users are doing in your product, Mixpanel AI is built to surface that signal proactively, so the insight finds you rather than the other way around. For technical teams who want to go further, try Mixpanel Headless where AI agents read and drive Mixpanel programmatically.

Social RSS (?)

2026-06-01T15:41:59Z

What’s up with this?

The spirit, of course, is “I want my Google Reader back.”

I should be careful though: I imagine if we really did have it back exactly as it was, it would probably not be as cool as we remember. That was a long-ass time ago and web software just wasn’t as nice as it is now. But: it really did have social features in it! I could leave comments and see comments from my friends. (How “friends” worked back then, I scarcely remember.)

I’m not alone in wanting this. Here’s what I’ve seen so far.

There is Skyreader, and it looks like Tim Disney is still actively working on it:

In my mind (and perhaps not reality?) Skyreader solves a “cold start” problem where signing up for a new social site is a lonely bummer because you don’t have any friends. Well, if all my Bluesky friends were automatically my friends on Skyreader, wouldn’t that be, like, good?

My problem is… I don’t… get it.

Maybe there aren’t really “friends” on Skyreader? If there are, I can’t really find a list of them. If there aren’t, I guess this isn’t what I’m looking for. Or is it just assumed they are the same as Bluesky friends? I dunno, I think I have a lot to learn about this AT Proto stuff.
What happens when I share an article? Does it share on my Bluesky? It doesn’t look like it. Who sees my shared articles then? My friends, if there are such a thing? Other people who explicitly subscribe? How would anyone figure out how to do that?
It looks like I can share and I can highlight stuff specifically in what I share. Who sees the highlight? I imagine they need to use Skyreader to see that highlight? Or does it republish as RSS somehow?
No comments… right? That’s weird to me for a social feed reader. What I think of when I think social feed reader is 90% writing and reading my friends comments.
The UI is confusing to me. I don’t love the expanding list format that jumps around a lot.

The main reason I can’t “really use it,” though, is that it doesn’t sync with Feedbin. I understand that’s a niche concern, but I need one canonical feed source, so everything I’m subscribed to is in one place, and how much I’ve read is also canonical. That might be antithetical to AT Proto? I have no idea.

Indieweb has their own version of all this, but it’s seriously no joke getting involved. From one of the reader sites, Togther:

What You Need

#1 Your own website

On the IndieWeb your website is your identity. It doesn’t need to be advanced, it doesn’t need to be pretty, it just needs to be yours.

#2 IndieAuth

IndieAuth is a technology to allow you to sign into services using only your website.

#3 A MicroSub Server

MicroSub is where the real magic happens. There are 2 parts; the server and the client. Together is the client, but it needs a server to go along with it. The server does a lot of tricky work such as fetching & parsing feeds as well as keeping track of what you have read and how you have everything organized. If you don’t already have a MicroSub server then check out Aperture.

#4 MicroPub (optional)

MicroPub is an optional but very cool piece of the puzzle. If your own website has MicroPub functionality then you can use Together not just to read content, but also to reply and create your own content. You can like, repost and reply to stuff you are following, you can even write full on blog posts from inside Together.

I mean, I’m not above some nerd hackin’, but this is 100% not ever going to be a place with a critical mass of my friends just chillin’ and sharing/commenting on blog posts. Way too hard.

The closest thing I’ve seen is CommonRSS by Brad Coffield which was build in a direct response to my original Bluesky post.

I originally had a gut reaction to the vibe coding nature of it. Like I don’t wanna use this thing you slopped together over a weekend, you gotta get more serious about it first. Which I admit is just a gut reaction and not a fair assessment of the software. Brad does seem quite sincere about making it a thing. But at the same time, it feels like it’s been months without updates, which is a bad sign for super brand new software.

In testing it lately…

It looks great, love the design — but it feels quite slow across the board.
The Feedbin sync (yay!) didn’t work at first, but appears to be working now.
It now claims to have 2-way syncing (yay!) but that doesn’t work for me. The articles I read aren’t read in feedbin. The stars don’t sync.
It’s got the cold-start social problem: there isn’t anyone to follow. The discover page only lists Brad, and clicking the follow button from him doesn’t work.

It feels like it’s close, but it needs to work better. The commenting feature (again, like my main thing for wanting this) requires you to highlight some of the post first, then you can comment, but the comment won’t save.

CommonRSS feels like a solid concept, but it’s both got the cold-start thing in terms of the social network, but also the cold-start thing in terms of a business. My guess is it’s hard for Brad to muster a ton of enthusiasm to work on the project all day every day with low users.

I’m thinking the main problem is that there just isn’t much of a business to be built around RSS. It can build boutique one-person companies with a passion for it, but even then, difficult.

On top of that, maybe there just isn’t much appetite for the social part? There are a number of arguably already-decently-successful boutique RSS apps, so why haven’t they done it? Why hasn’t Feedbin done anything terribly social? Why not Feedly? Remember FlipBoard’s kind weird taking on RSS? Now they have an equally weird spinoff. My guess is it’s just too hard of work for too little payoff.

On wintering.

2026-05-29T15:00:36Z

Abraham Lincoln rode home from Washington in December 1849, with what looked like the end of his career packed into his luggage. He'd served one term in the House, alienated his constituents by opposing the Mexican War, and lost his shot at a federal Land Office appointment.

He went back to Springfield to practice law, a near-broken man. And, for nearly 5 years, he barely participated in national politics.

He rode the Illinois circuit, argued patent disputes, and taught himself geometry from Euclid by candlelight in coach inns. He read newspapers obsessively; he read Shakespeare and the King James Bible until he could quote either from pretty much any starting point.

The folks who saw him in those years said he looked...tired.

When he returned to the spotlight, in October 1854, the Kansas-Nebraska Act had cracked the country open. Lincoln walked onto the stage at Peoria and spoke for 3 hours straight. The man who'd been a country lawyer that morning was a national figure by midnight.

Six years later, he was president.

Lincoln's lost years are the part of the biography American children skip past in school; they get the rail-splitter, the beard, the debates, the war, the emancipation, the address, the assassination.

But the 5 years we skip over are the whole ballgame.

They rebuilt the instrument.

The English writer Katherine May coined the modern usage in her 2020 book Wintering, but the idea is older than the word. Russian peasants called the long quiet stretches between harvests zima and treated them as a season for weaving, sleeping, repairing tools, and telling stories. Japanese Buddhist monasteries built whole liturgies around rohatsu sesshin, the seven-day winter retreat that closes the year. Foragers like the !Kung and the Hadza, spent something like 4 hours a day on subsistence and the rest on…rest.

Productivity is a recent invention; wintering is not.

Cormac McCarthy published Blood Meridian in 1985 to a shrugging response. The New York Times reviewed it in a single column. He'd been writing in El Paso for years, broke and largely forgotten. Friends thought he'd peaked. Then in 1992 All the Pretty Horses came out, won the National Book Award, sold half a million copies, and the back catalog got reissued. McCarthy hadn't been recovering. He'd been finishing something the culture wasn't ready for in 1985 and was ready for by 1992.

He'd been wintering.

Daniel Day-Lewis stopped acting in 1997 and apprenticed as a cobbler in Florence. He came back, played Bill the Butcher in Gangs of New York, and won an Oscar. He stopped again. Came back. Won another Oscar. Stopped again, and by all reports has actually stopped this time, though I wouldn't bet on it. The cobbler years were how he reset the instrument.

In the long winter, organisms route metabolism inward.

Trees pull resources out of leaves, drop the leaves, and push the sugars down into root systems. Bears don't sleep, exactly. Their core temperature drops a few degrees, their metabolism halves, and they cycle slowly through fat reserves while their kidneys learn to recycle urea into protein. They come out in spring with their bones still mineralized and their muscles roughly intact, which is something no human has yet figured out how to do. What the bear performs is one of the most metabolically sophisticated tricks in the animal kingdom.

The Romans understood that a field left fallow for a season produced more in the next cycle than one worked continuously. Norfolk farmers in the 18th century made it a four-course rotation: wheat, turnips, barley, clover, with the clover restoring nitrogen the wheat had pulled out. The land that looks unused is doing the most useful work.

People who winter well are doing something analogous. They route attention inward and downward, into the parts of the system that don't show up on the surface. They read, they revise, they take long walks they can't account for, and they think the same thought 400 times until it cracks.

Most of what gets published, shipped, posted, and announced is washed off the rocks within a quarter. The people doing it are running on a treadmill that resets their position to zero every Monday. They have to keep producing to stay visible, and visibility is how they earn the right to keep producing.

It's a closed loop, and it generates very little compound interest.

The winterer is off the loop. They aren't maintaining a position because they don't have a position to maintain.

In the short term, you pay dearly for it.

People forget you exist. Calls dry up. Old collaborators stop replying. Younger versions of you lap you in the standings.

The benefit is that you can do work that takes longer than a quarter, and longer than a year, and longer than 5 years, because nobody is auditing the line item.

Charles Darwin came back from the Beagle voyage in 1836 with the rough outline of natural selection in his head. He published On the Origin of Species in 1859. The intervening 23 years included long stretches when he wrote almost nothing in his theory notebooks, partly because he was sick, partly because he was writing 8 volumes about barnacles, and partly because he understood the case had to be airtight. When he finally published, the argument was so heavily fortified that the church spent the next 50 years trying to find a hairline crack and failing.

If Darwin had published in 1840, he might be a footnote. His 23 years of comparative silence were the moat.

Robert Caro started his Lyndon Johnson biography in 1976. He's published 4 volumes of an intended 5. He's now 90. He moved to the Texas Hill Country to live among the people Johnson grew up with, because he thought he couldn't write about a man without inhabiting his weather. Each volume took roughly a decade. The publishing world treats him as a slow eccentric. Anyone who's read the books knows he's running a different clock, on a different scale, and that no one currently working at speed is going to produce anything close.

Plenty of people stop and produce nothing. The graveyard of failed comebacks is large, and wintering is dangerous as a strategy because most attempts at it collapse into actual stagnation.

The difference between the two is invisible from the outside, until the end.

The reason the wintering few register as dangerous, when they re-emerge, is that they have something the still-busy don't have: a center of gravity. They've spent enough time alone with a single problem to develop actual opinions about it, opinions that don't move when other people push on them. In a culture optimized for constant repositioning, conviction is a structural advantage. The market doesn't know how to price it.

The winterer has been watching while you weren't looking. They've watched the consensus shift, watched the mistakes pile up. When they come back, they come back with reads you can't get from inside the swirl, because the swirl makes you stupid.

The philosopher Hannah Arendt, writing in The Life of the Mind in the 1970s, described thinking itself as a form of withdrawal. You can't think and act at the same time, she said, because thinking pulls you out of the stream of ongoing events. She was suspicious of people who claimed to do both at once.

The British psychiatrist Anthony Storr, in Solitude (1988), made the case that the most original work of major figures often came out of long isolated stretches. Newton in plague-year Cambridge. Wittgenstein in Norway. Kafka in Zürau. Beckett in his Paris apartment with the curtains drawn. Storr wasn't romanticizing it; the isolated stretches were often miserable, sometimes pathological. But the work that came out of them had a density that wasn't available to people doing it part-time.

Any culture that systematically punishes withdrawal is going to lose its most concentrated thinkers to either burnout or invisibility. The modern knowledge economy, with its ambient pressure to post, ship, and stay in the conversation, is a machine for producing exactly that loss. The people we'll wish we had in 15 years are, right now, being shamed into producing slop they don't believe in, because the alternative is to drop out, and dropping out reads as failure.

The winterers who survive this will be those who can tolerate looking like they failed. This is a real and rare psychological skill, and most people don't have it. It requires you to be okay with the wrong kind of silence around your name for years. It requires you to pass on small wins that would re-establish your position. It requires you to bet that what you're working on is worth more than what you're giving up, when the only person who can evaluate the bet is you, and you might be wrong, and you'll only know in 7 years.

Lincoln didn't know in 1851 that he was wintering.

He thought he was finished.

He told his law partner William Herndon that his political career was over, and he believed it. And then his country produced an emergency that demanded exactly the kind of mind he'd been nurturing, and he was the man of the hour whose hour had finally come.

The people who appear to have stopped, in any given year, are mostly people who have actually stopped. But small fraction of them are doing the other thing.

Our world produces emergencies on a reliable schedule; when the next one comes, watch who walks out of the woods.

10x engineers: the Purple Developer

2026-05-22T06:35:39Z

What makes a software engineer productive? You can list attributes like experience with the language, scientific mindset, intelligence, focus, a personally crafted IDE setup. Yet, in my experience, far and away the biggest factor is: familiarity with the codebase they’re changing.

That knowledge of what the system does and needs to do, it’s completely essential and entirely contextual. The easiest way to get is to create it — that is, to write that software from scratch. And then it feels completely natural. It doesn’t feel like knowledge, it feels like breathing. It feels obvious.

When you have a decent mental model of a system, sharing that with others is hard. You don’t know how much you know. If you’re the purple developer in this picture:

below the line, a software system; above it, a purple person with a mental model of the system. A blue person and a green person with very incomplete models and lots of questions.

then you have a mental model of this system because you built it. The green and blue developers have been assigned to help, but they can’t change the system because they don’t understand it. Meanwhile, you may be changing the system fast enough that it’s impossible for them to get a grasp on it, no matter how smart they are. (I’ve been in their situation.) The solution is to work together on the system, and invest attention in transferring your mental model. Until then, Blue and Green get in your way.

This slide is part of my Collective Problem Solving keynote (also seen as Origins of Opera). People came up to me afterward: “We have a purple developer. Now I don’t feel so bad about being blue or green” or “Oh no, I’m the purple developer!”

Which developer in this picture is ten times more productive than the others? Purple developer!

Is it because they are inherently smarter? Nah. Circumstances lead to this situation.

There is not any “productive” software engineer so much as “productive in this codebase.” Spread the knowledge, spread the productivity.

Native Apps Should Be Avoided Whenever Possible

2026-05-15T05:08:54Z

April 12, 2026

Native Apps Should Be Avoided Whenever Possible

TL;DR: What you should do:

Openly refuse apps, and vocally advocate for the web instead.

Try not to install any apps if you don’t need to.

If a service has a functioning website, use it instead.

Revoke all permissions by default, including background location, microphone, and camera permissions for anything that doesn’t require them to function.

Audit your installed apps. Uninstall all apps you don’t actively need.

Treat every “download our app” prompt with skepticism.

Most native apps collect far more data than their website equivalents ever could. They request permissions to hardware, sensors, and background processes that browsers deliberately restrict. The third-party software embedded in these apps frequently transmits your location, device identifiers, and behavioral data to third parties before you even see a consent prompt. This data is in tandem bought, sold, and aggregated by brokers. It has been used to out individuals, track immigrants, and enable prosecution over reproductive healthcare.

The White House App

On March 27, 2026, the Trump administration released an official White House app for iOS and Android. Within hours, two independent security researchers decompiled it and published their findings [1]. The app is a textbook example of everything wrong with the native app model.

Apple requires apps to submit a privacy manifest disclosing what data they collect. The White House app declared an empty array. Zero data collection. Meanwhile, the actual binary contained ten analytics frameworks, including the full OneSignal SDK with a sub-framework specifically for location tracking [2]. The GPS pipeline polled precise coordinates every 4.5 minutes in the foreground and every 9.5 minutes in the background, syncing everything to OneSignal’s commercial servers. A boolean flag in OneSignal’s server responses could remotely enable or disable GPS tracking without an app update and without Apple review.

An Exodus Privacy audit identified three embedded trackers, one of which was Huawei Mobile Services Core [3]. The app’s privacy policy, last updated January 20, 2025, makes no mention of GPS tracking, OneSignal, or background data collection.

Nearly everything in the app is available on whitehouse.gov. The app’s unique additions are push notifications, a pre-filled text message to the President, and an ICE tip button (also available on ice.gov). What it actually added at scale was a surveillance pipeline: 77% of the app’s network requests go to third parties, not whitehouse.gov.

The Software Embedded in Apps

Most people think of apps as products built by a single company. In practice, the average app is a thin wrapper around dozens of third-party software packages, each with its own data collection pipeline and commercial incentives. When you grant an app permission to access your location, every package embedded in that app inherits that permission. A single package can appear in hundreds of apps, feeding location data on millions of people to a single aggregator.

In January 2025, a hacker breached Gravy Analytics and leaked roughly 30 million location records collected from 3,455 apps — dating, fitness, gaming, and health apps among them [4]. The FTC subsequently banned Gravy Analytics from selling Americans’ location data [5], but by then the data was already circulating on cybercrime forums.

In a separate case, Google paid $391.5 million to settle claims from 40 states for continuing to collect location data even when users explicitly disabled location tracking [6].

Why Everyone Should Care

This data is bought, sold, and aggregated by brokers. It has been used to out individuals, track immigrants, and enable prosecution over reproductive healthcare. In multiple cases, journalists and private groups have purchased app-derived location data to identify specific people based on their movements.

There are virtually no restrictions in the United States on buying, selling, or weaponizing this data. There is no comprehensive federal privacy law. And there isn’t likely to be one soon. The best we can do is minimize the data we share in the first place.

What Apps Can Do That Websites Can’t

The core argument for using the website instead of the app comes down to what each platform is technically capable of doing without your knowledge.

Capability	Native App	Website / PWA
Background location tracking	Yes, can poll GPS continuously	No
Run at device startup	Yes	No
Access biometric hardware	Yes	Limited (WebAuthn, user initiated)
Modify or delete device storage	Yes	No (sandboxed to browser)
Embed invisible third-party software	Yes, all inherit granted permissions	No, scripts visible in page source
Transmit data before consent prompt	Yes (common with third-party software)	Restricted by browser policies
Push notifications while closed	Yes	Yes (PWA, user opt in required)
Access contacts, call logs, SMS	Yes (if permitted)	No
Prevent phone from sleeping	Yes	No
Camera and microphone	Yes (persistent if granted)	Yes (per session, prompted each time)
Offline functionality	Yes	Yes (via service workers)

The browser is the security boundary. Websites operate within it. Native apps bypass it.

The Access Provided by Default is Enough to Do Real Harm

The moment you install an app, before you allow a single permission prompt, it can:

Reach any server on the internet
Read your IP address, device model, OS version, timezone, country, carrier, and network type
Generate and persist a unique identifier tied to your device
Run code at device startup (Android) and wake up in the background
Fingerprint your device by combining the above into a signature that follows you across sessions
Grant all of this same access to every third-party software package embedded in the app
Compare this data to other datasets to infer your identity, demographics, interests, and habits

The runtime permission prompts you actually see (location, camera, contacts) are helpful while annoying, but the majority of the default access permissions do not require your consent.

A website, by contrast, starts with almost none of this: no persistent identifier, no background execution, no third-party software inheritance, no startup hooks.

Some Things Need to Be Apps, But Most Don’t

Some things need to be apps. AR and VR, real-time games, anything talking to NFC or Bluetooth hardware, serious audio and video work, accessibility tools. These are legitimate cases where the browser sandbox is the limitation. In these circumtances, I personally use a full computer as opposed to my phone.

Almost nothing else qualifies. Your banking, your travel, your grocery store, the restaurant down the street — none of it needs an app. And rewards be damned. No rewards are worth the data you are willingly giving them.

Same goes for hardware. If a thermostat or a fitness tracker can’t be set up without a proprietary app, that’s a flaw in the product, not a feature. I immediately avoid such products. You’re buying an ongoing relationship with someone else’s servers and guaranteeing that you’ll forget that corporations are watching everything they can.

Conclusion

I avoid most apps. It turns out this is easier than most people assume, because the app is almost never the only option. It is just the option the company wants you to take and not enough people question.

We are at a very specific time in humanity right now. Where aggregating data is a currency, and it is actively being utilized at a scale never before seen. I recommend you at least take stock of what you’re freely giving away.

References

1. I Decompiled the White House’s New App (Thereallo, March 28, 2026) and Security Analysis of the Official White House iOS App (atomic.computer, March 27, 2026). Two independent researchers decompiled the app on Android and iOS within hours of release.

2. Security Analysis of the Official White House iOS App (atomic.computer). Documents the empty privacy manifest, OneSignal SDK with ten sub-frameworks, and the remote GPS toggle via server response.

3. Exodus Privacy Report: gov.whitehouse.app. Automated audit identifying three embedded trackers including Huawei Mobile Services Core. Additional context in Fedware: 13 Government Apps That Spy Harder Than the Apps They Ban (Sam Bent).

4. A breach of Gravy Analytics’ huge trove of location data threatens the privacy of millions (TechCrunch, January 13, 2025).

5. FTC Finalizes Order Prohibiting Gravy Analytics, Venntel from Selling Sensitive Location Data (FTC, January 14, 2025).

6. Google to pay $391.5 million in location tracking settlement with 40 states (TechCrunch, November 14, 2022).

AI dělá chyby a autisté je dokážou najít. Z jejich jinakosti dělá Češka ve svém startupu přednost

2026-05-07T05:32:58Z

Představte si konkrétní situaci: datoví vědci ve startupu trénují svůj analytický model, ale v určitých případech narážejí na vysokou chybovost v podobě falešně pozitivních výsledků. Místo toho, aby se pálil drahý čas seniorních inženýrů na manuálním čištění datasetů, předá se úkol ven. Výsledek? Najatý tým projde pět iterací dat a pomůže tak o polovinu zlepšit úspěšnost modelu. Právě na tom staví mladý pražský startup Diversight. Jeho tajnou zbraní nejsou pokročilejší algoritmy, nýbrž dedikovaní analytici na autistickém spektru. Ukazuje tak, že vlastnosti, kvůli kterým se jiné firmy těmto lidem někdy vyhýbají, se mohou proměnit na silné stránky.

„Nejsme charita ani neziskovka, jsme klasická technologická firma,“ říká hned na úvod spoluzakladatelka a ředitelka Diversightu Irena Zatloukalová. V českém technologickém prostředí se pohybuje dvacet let, a to zejména v oblasti komunikace, kdy sedm let byla mluvčí tuzemské internetové jedničky Seznam.cz. Mimo jiné působila i jako spolumajitelka PR agentury Fenek a později na volné noze radila s komunikací technologickým firmám jako Heureka, Apify, Liftago či Ackee. Loni v červenci ale všechny klienty předala dál a naplno se vrhla do nového byznysu.

V Diversightu se zaměřuje na takzvaný datový dluh. V éře, kdy se prakticky každá firma snaží do svých procesů implementovat umělou inteligenci, stojí budoucí úspěch primárně na kvalitě vstupních dat. „I ty nejpokročilejší vývojářské týmy totiž musí manuálně odstraňovat duplicity a řešit hraniční případy, v nichž zavedená automatizace zkrátka selhává a modely začínají generovat nesmysly. Diversight funguje jako filtr, který přebírá na přesnost náročnou vrstvu ekosystému a zajišťuje potřebnou lidskou pojistku proti halucinacím,“ vysvětluje pro CzechCrunch Zatloukalová.

Foto: Diversight

Irena Zatloukalová s kolegou Šimonem

Právě v oblasti hledání nepatrných vzorců totiž samotná umělá inteligence běžně naráží na své limity, dodává. Pokud modely generují opakující se chyby, vývojářům nepostačí jen jednoduché přepsání textového promptu. Týmy potřebují člověka, který dokáže precizně pochopit byznysovou logiku a vyčíst přímo ze syrové databáze konkrétní pravidla a odchylky. „V tom máme ohromnou výhodu, protože máme lidi, kterým jednou řeknete pravidla a oni pak po nich neúnavně půjdou, od prvního řádku až po řádek číslo 1 500,“ popisuje specifikum práce Zatloukalová.

Doplňuje přitom, že jejím lidem nedělá problém udržet stejnou hladinu pozornosti na jednom tématu klidně i osm dní v kuse: „Prostě potřebují problém vyřešit, neznámou najít a věc dotáhnout do konce. To pomyšlení na rozlousknutí problému pomáhá jejich fokusu.“

Na samotném začátku příběhu Diversightu přitom nestála Zatloukalová. „Ráda bych to říkala, ale bohužel to nebyl můj nápad,“ usmívá se a jako originálního tvůrce označuje Tomáše Borovičku z technologické společnosti Datamole. Ten na zahraničních trzích narazil na izraelskou firmu Point AI, která budovala data-anotační byznys (zaměřený na označování surových dat) na přesné práci lidí na spektru. Od té doby přemýšlel nad tím, jak vyzkoušet podobný koncept i v Česku.

„Byla to jedna večeře, jedno potkání, tři hodiny povídání. Šli jsme hodně do hloubky a do našich hodnot a na konci mi řekl: Tak jo, pojďme to zkusit,“ vzpomíná Zatloukalová na zlom, který přišel v prosinci 2024. A dodává, že na začátku je propojila Lenka Kučerová ze spolku prg.ai.

Původní plán Diversightu počítal s tím, že se do Prahy přenese jako franšíza přímo izraelský model. Rychlý průzkum trhu ale ukázal, že běžný anotační byznys je v tuzemsku v podstatě jen levná komodita, kde firmy často najímají studenty a na finální kvalitě jim tolik nesejde. Zatímco v Izraeli se data pro zdravotnictví či zbrojní průmysl anotují s kritickou přesností, v lokálních podmínkách by takový model ekonomicky nepřežil. Diversight proto musel okamžitě změnit plány. Zaměřil se na komplexnější analytické úkoly a takzvanou validaci s člověkem v procesu (human-in-the-loop), kde se naplno využije kognitivní preciznost ve složitých datech.

Projekt od začátku strategicky i finančně podpořil zmíněný Datamole, který mu poskytl peníze na bezpečný roční provoz a působí v roli partnera. Mimo jiné nabízí i přímý přístup ke svým seniorním AI architektům. Hlavním byznysovým úkolem pro nadcházející měsíce je tak dostat Diversight do zisku díky příjmům ze zakázek.

Foto: Diversight

Tomáš Borovička a Irena Zatloukalová

Mezi aktuální klienty vedle analytické společnosti BizMachine, pro kterou startup dělá v úvodu zmíněné čištění dat, patří například také InfraHex, jedna z firem holdingu Respilon. Ta vyvinula technologii, která upravuje tepelnou stopu živých i neživých objektů a tím omezuje jejich detekci pomocí termokamer. V současnosti se využívá i na Ukrajině, kde pomáhá vojákům v terénu. I drobné odchylky zde hrají zásadní roli, protože se systém opírá o přesné analýzy obrazů v různých infra spektrech. A tak se na této detailní práci vyhodnocování a výběru dat podílejí lidé na autistickém spektru.

Českým mediálním i firemním prostorem dnes podle Zatloukalové bohužel rezonují převážně extrémní obrazy autismu, kdy na jedné straně stojí příběhy nespolupracujících jedinců s těžkým postižením, na straně druhé zprofanované postavy geniálních vědců, jako je například Sheldon Cooper z Teorie velkého třesku. Diversight ale systematicky hledá velkou množinu lidí s nadprůměrnou analytickou inteligencí ležící mezi těmito dvěma extrémy. Většina z nich má však potíže projít přes výběrová řízení či běžná HR oddělení, protože standardní firemní postupy zkrátka neumí s neurodiverzitou pracovat.

Pokud zahodíte své ego a uznáte, že někdo našel lepší cestu, firmě to hodně pomůže.

„Už jen to, když do pracovního inzerátu vypíšete deset věcí jako ‚nice to have‘. Člověk na spektru si to projde, u třetí položky si řekne, že má jen dva a půl roku zkušeností místo tří, takže to není práce pro něj, a jde dál,“ vysvětluje Zatloukalová jeden z mnoha detailů, proč kandidáti sami rovnou z recruitmentu vypadávají. Značná část lidí na spektru tak i kvůli tomu končí na úřadu práce či na podřadných pozicích a ve firmách běžně narazí kvůli logickému poukazování na neefektivitu a navrhování lepších postupů.

Zatloukalová proto změnila rovnou i náborový proces a přizpůsobila ho lidem na spektru. Přesné zadání úkolů a otázky zasílá s předstihem, kandidáty navíc netestuje pod umělým tlakem, ale zaměřuje se striktně na jejich schopnost strukturovaně přemýšlet. Aktuálně má firma pět lidí – vedle samotné Zatloukalové je to technologický manažer Adam Vesecký a interní tým tří specializovaných analytiků. Záměrem je tým rozšiřovat, nicméně se stropem na hranici deseti zaměstnanců, aby firma nepřišla o klidné prostředí komorního kolektivu.

Zatloukalová navíc varuje před zjednodušováním nebo předsudky, že práce s neurodivergentními lidmi přináší jen těžkosti a komunikační zádrhele. Na první pohled nepříjemné situace, kdy podřízený manažerovi zcela upřímně oznámí, že zadal úkol špatně, totiž vedou k lepším procesům do budoucna, a tím i výsledkům. „Pokud zahodíte své ego a uznáte, že někdo našel lepší cestu, firmě to hodně pomůže,“ vysvětluje Zatloukalová.

Lidé na spektru podle ní přinášejí do byznysu inovace a odlišné úhly pohledu nejen v datové analytice, ale napříč všemi obory od zákaznické podpory po design. „Mají přirozenou schopnost jít striktně po podstatě problému a všímat si i drobných odchylek, které neurotypická většina snadno přehlédne,“ doplňuje. Výměnou za tento výkon často vyžadují jen velmi specifické, avšak finančně nenáročné úpravy pracovního prostředí – typicky přesun stolu dál od rušné kuchyňky, utlumení ostrého kancelářského osvětlení nebo zkrátka jen možnost po náročné schůzce úplně vypnout a odejít domů.

Cítím nespravedlnost. Jsem naštvaná na to, jakým způsobem se společnost staví k lidem, kteří jsou jiní.

Častou chybou některých manažerů je tyto potřeby zpochybňovat jen proto, že zaměstnanec nemá na stole oficiální lékařskou diagnózu. „Moje velká prosba k jakémukoliv manažerovi je: i bez diagnózy člověka poslouchejte. Když říká, že něco potřebuje, respektujte to, nezpochybňujte a hledejte cesty, jak zařídit, aby mohl svoji práci vykonávat co nejlépe,“ apeluje. Jak sama upozorňuje, nevyžaduje to stavbu drahých oddělených kanceláří, často stačí investice do kvalitních sluchátek s aktivním potlačením hluku.

Přístup orientovaný na lidi se specifickými potřebami má podle Zatloukalové přesah do celkového chodu jakékoliv firmy. Veškerá drobná provozní pravidla a limity, které startup explicitně zavádí kvůli neurodivergentním kolegům, totiž v důsledku ulehčují práci všem. „Každý zaměstnanec občas potřebuje ničím nerušený čas na práci, jen se to v běžných korporacích tolik neakcentuje. Funguje zde stejný princip jako v moderním urbanismu. Pokud navrhnete veřejný prostor tak, aby vyhovoval lidem na vozíku, dětem i rodičům s kočárky, vytvoříte ve finále vysoce funkční prostředí, ve kterém se bude dobře žít úplně všem,“ srovnává.

Na druhou stranu Zatloukalová doplňuje, že budování neurodiverzního týmu s sebou pro vedení nese nutnost preciznosti v mezilidské komunikaci. „Tou se živím dvacet let a největší paradox je, že se mi opravdu stává, že si s kolegy nerozumím v zadání,“ přiznává s tím, že sama musela odložit své ego stranou a opět se učit zadávat úkoly maximálně jasně, doslovně a bez zbytečného sarkasmu.

To vše se promítá i do samotného workflow při řešení úkolů pro klienty. „Když řeknu ‚udělej A‘, kolega si začne domýšlet B, C a D. A místo toho, aby se zeptal, radši to rovnou udělá až po to D. Já mu pak musím říkat, že zašel moc daleko. Pro příště ale vím, že potřebuji dát víc kontextu k tomu, proč má v bodě A skončit,“ ilustruje občasné komunikační střety zakladatelka. Doslovnost a analytický rozklad problémů se však startupu okamžitě vrací ve chvíli, kdy tým narazí na nekvalitní a chaotická data v zadání.

„Cítím nespravedlnost. Jsem naštvaná na to, jakým způsobem se společnost staví k lidem, kteří jsou jiní. Ta jinakost nemusí být na první pohled patrná, ale znamená pro ně spoustu překážek, které jim vyrábíme my všichni. Angličtina pro to má slovní spojení ‚disabled by society‘. Jakmile mi ukážete něco takového, jsem totální buldok a jdu po narovnávání té nespravedlnosti,“ vysvětluje Zatloukalová, pro kterou se tak v Diversightu kromě společenského přesahu spojuje vášeň pro data a práce s lidmi. „Baví mě dělat věci, které nikdo předtím moc nedělal nebo neprozkoumal. Diversight má v sobě všechno,“ uzavírá.

Using safe-area-inset to build mobile-safe layouts

2026-05-06T12:14:32Z

Modern phones are not simple rectangles. They have rounded corners, camera cutouts, dynamic islands, and home indicators that double as gesture areas. Browsers know the dimensions of all of these and expose the parts that could obscure content as safe area insets.

The "safe area" is the portion of the screen that is guaranteed to be free from being obscured by system UI. The safe-area-inset is the measurement of how much space the system UI is taking up on each edge of the screen. By using these values in your CSS, you can make sure that important content and controls are not obscured by the system UI.

If you don't want your floating chat button to end up sitting behind the home indicator, where it's unreachable, you need to account for the safe area inset.

Environment variables for safe area insets

With the safe-area-inset environment variables, you can make your layout adapt to the current device's safe area and avoid those bugs. The env() function is how you read those values in CSS:

body {
  padding-top: env(safe-area-inset-top);
  padding-right: env(safe-area-inset-right);
  padding-bottom: env(safe-area-inset-bottom);
  padding-left: env(safe-area-inset-left);
}

Browser support

Safe-area-insets are baseline widely available, which means you can use them in production today and be confident that they will work for almost all your users on mobile devices.

Since it's baseline widely available, you don't really need to think about fallbacks but if you want to be extra safe, you can provide a fallback padding if the browser doesn't support env():

body {
  padding-top: 1rem; 
  padding-top: calc(env(safe-area-inset-top) + 1rem);
}

And for browsers that support env() but not safe-area-inset-* variables, you can provide a fallback value directly in the env() function. If safe-area-inset-top is not supported, it will fall back to 1rem:

body {
  padding-top: env(safe-area-inset-top, 1rem);
}

It's worth noting that this situation is purely theoretical, as all browsers that support env() also support safe-area-inset-* variables. That might not be the case with other environment variables, so it's good to know that this fallback mechanism exists.

Using safe-area-inset values

Before we get into problem areas, let's get a general understanding of how these variables work and how you can use them to affect the layout.

In this demo, change each inset and watch how a realistic mobile UI that takes safe area insets into account shifts to remain usable:

The specific px values here are not particular to any specific device, the point is to show how the device sets these variables and how your layout can respond to them as they change.

On real devices, you can think of these values as constants. They're provided by the browser and while they might change when switching from portrait to landscape or between OS updates that change the device UI as well as simply differ per device, they don't change dynamically as the user scrolls or interacts with the page. The browser provides them as a constant value that you can use to ensure your content is not obscured by the system UI.

Which web pages actually need this?

If you want your pages to look the best they can, all of them.

Browsers by default will prevent your site from being obscured by the notch or home indicator, so your content will be safe without any special handling. That does come with a downside, which is that the browser will give you a smaller viewport to reserve space:

You'll notice the edges here keep the site from being obscured, but they also don't look that great. Ideally we want the content to stretch edge-to-edge, but we want to make sure they're not obscured by system UI. To get that, you need to opt in to the full viewport and handle safe areas yourself.

To do so is a two-step process. First, you need to add viewport-fit=cover to your meta viewport tag:

<meta name="viewport" content="width=device-width, viewport-fit=cover" />

This will make the browser stretch your page edge-to-edge:

As you can see, the content sits behind the notch/dynamic island. viewport-fit=cover tells the browser that you want to be responsible for making sure your content is not obscured by the system UI.

And now we can move those elements away from behind the system UI using env(safe-area-inset-*).

.content {
  padding-right: env(safe-area-inset-right);
  padding-left: env(safe-area-inset-left);
}

If you opt into the full viewport, here are the kinds of things you now have to think about to make sure your content is not obscured by the system UI:

Fixed headers and navigation bars should have enough space above or below for the notch or home indicator
Floating chat or help buttons should remain inside the safe area
Full-screen dialogs and drawers should take up the right height without being obscured by the home indicator
Map or video controls near screen corners

Safe-area-inset doesn't provide margins

Safe-area-insets are defined to be exactly the space that the system UI is taking up. That means they don't provide any margin between the system UI's edge and your content.

If you set padding to just the safe-area-inset value, your content will sit right up against the edge of the safe area, which is right up against the system UI.

To add some breathing room, you can add your own padding on top of the safe area insets by combining things with calc(). This way you can ensure that your content is not only safe from being obscured but also has some space to breathe:

body {
  padding-top: calc(env(safe-area-inset-top) + 1rem);
  padding-right: calc(env(safe-area-inset-right) + 1rem);
  padding-bottom: calc(env(safe-area-inset-bottom) + 1rem);
  padding-left: calc(env(safe-area-inset-left) + 1rem);
}

Safe-area-inset only has non-zero values on mobile devices

env() is supported across browsers and platforms, but the safe-area-inset-* variables really only have non-zero values on mobile devices.

Desktop browsers always return 0 because there is no UI on top of pages in desktop browsers. It's only on mobile devices that these values are non-zero and that you have to account for them.

That's exactly why these bugs are so easy to miss. If you test in the Chrome responsive view, the safe area insets will still be 0 and you won't see any issues during development.

Testing on real devices is often delegated to the end of the project. By then, making layout changes to accommodate safe areas can be expensive, and the bugs can slip through to production. Even worse, they often only affect users on certain devices, so they can go unnoticed for a long time.

Polypane's device emulation supports safe area insets

Polypane is the first and only desktop browser to emulate safe area insets. Every device in Polypane has correct safe area inset values for both portrait and landscape orientations.

Here's Polypane showing the safe area insets in blue, and the small viewport height difference in pink:

We're also the only desktop browser to emulate svh, but that's a topic for another article.

With Polypane's safe-area-inset overlay visualization, you can see exactly where the unsafe areas are on each device.

Solving a specific issue: Floating buttons

Let's look at a specific example of how to use safe-area-inset values to solve a common issue: floating buttons that end up behind the home indicator and become unreachable.

.chat-button {
  position: fixed;
  right: 10px;
  bottom: 10px;
}

Toggle the positioning between 'Fixed position' and 'Using env()' to switch between hard-coded offsets and offsets that use safe-area-inset-bottom and safe-area-inset-right.

`safe-area-max-inset`

Along with safe-area-inset-*, the specification also describes safe-area-max-inset-* variables. Those are not widely supported yet, but they are worth mentioning because they have slightly different behavior:

safe-area-inset-* gives you the current inset value right now. On scroll, the browser chrome can collapse, and the inset value can shrink all the way to 0. Your element moves with that change.
safe-area-max-inset-* gives you the maximum inset the browser can report for that edge. It stays stable even when the browser chrome collapses. Use it when you want a reserved zone that does not jump around.

Drag or scroll the phone to simulate the browser address bar appearing and disappearing.

The green button uses safe-area-inset-bottom and follows the current inset. The blue button uses safe-area-max-inset-bottom and stays put.

OverlayShow safe inset areas

inset vs max-inset

When to use which really depends on your situation. For some UI components it makes sense to move with the live viewport state, like a floating chat button that should always be just above the home indicator.

For other things, like a persistent cookie banner or a full-screen dialog, it can be better to reserve a stable zone that doesn't shift when the browser chrome collapses so that users don't inadvertently tap the wrong button.


.floating-cta {
  bottom: calc(env(safe-area-inset-bottom) + 1rem);
}


.persistent-zone {
  bottom: calc(env(safe-area-max-inset-bottom) + 1rem);
}

Browser support for `safe-area-max-inset-*`

As of now, only Chromium implements safe-area-max-inset-* so there is no support in (mobile) Safari or Firefox. That means you can't depend on it yet and should provide a fallback stack for other browsers:

.bottom-spacer {
  padding-bottom: 1rem;
  padding-bottom: calc(env(safe-area-inset-bottom) + 1rem);
  padding-bottom: calc(env(safe-area-max-inset-bottom, env(safe-area-inset-bottom)) + 1rem);
}

Notice how we're using the env() fallback mechanism to fall back to safe-area-inset-bottom if safe-area-max-inset-bottom is not supported.

Testing safe areas in Polypane

As covered earlier, safe area bugs are easy to miss during development. Chrome's responsive view always reports inset values of 0, and real-device testing tends to get pushed to the end of the project, when fixing layout issues is expensive and bugs have already reached production.

Polypane changes that. You can test safe area insets (and small viewport behavior) directly on your desktop, across multiple devices and orientations simultaneously, as part of your normal development workflow.

Catch real-device layout failures during development, not after shipping.

What to take away

Safe areas are not edge cases for unusual devices. They are the reality of modern mobile devices with camera punch holes, notches, dynamic islands, and home indicator bars. Your users are on those devices, and if you want to give them the best experience, you need to make sure your content is not obscured by the system UI.

Make sure your viewport is set to viewport-fit=cover so you get the full viewport and that you use env(safe-area-inset-*) so all your content is visible and accessible. That gives users the best experience.

Get started with testing safe areas in Polypane today, and catch real-device layout failures during development, not after shipping.

Design Token Naming Conventions: A Practical Guide

2026-05-04T10:30:43Z

Design Token Naming Conventions: A Practical Guide

23rd of April 2026

On This Page:

If you have ever stared at a long token list and wondered whether a name should be button-primary-bg, primary-button-background, or color-button-primary, you are not alone.

Naming design tokens can look and feel simple right up until you have to do it for real. Choose a weak pattern and things get inconsistent fast. Choose a clear pattern and you get a shared language that helps both design and engineering move quicker.

There is no single correct convention. A two-person team shipping one product has very different needs from a large organisation running multiple brands across multiple platforms.

What you can do is pick a convention that is clear, consistent, and scalable. This article walks through the core token tiers, common naming models, and practical rules that help avoid naming drift.

Why naming gets hard faster than expected

Most teams don't struggle creating tokens. They struggle to keep token meaning stable as more people and components work on the Design System.

You can often see multiple names emerge for the same idea:

Code languagejson

["spacing.small", "space-2", "gap-md", "paddingStandard"]

Each name might make sense in isolation. Together, they signal a semantic drift. The naming model is no longer acting as a shared language.

This is why naming quality can affect delivery speed. If token intent is unclear, every usage becomes a local judgement call, and those judgement calls will accumulate into inconsistency.

The examples in this article follow the Design Tokens Community Group specification using $value, $type, and dot-notation aliases like {color.brand.primary}.

What are design tokens, a quick recap

Design tokens are a tooling and platform agnostic way to store design decisions as named values: colour, spacing, typography, radius, shadows, motion, and (much) more.

The key difference is where and how many times you define those values.

Without tokens, you might use #BADA55 in dozens of places across components, stylesheets, and design files. When that green needs to change, you might have to hunt down every instance to make sure you're updating everything that needs to be updated.

With tokens, you create a design token file, a JSON object following the Design Tokens Community Group specification:

Code languagejson

{
  "green": {
    "400": {
      "$value": "#BADA55",
      "$type": "color"
    }
  }
}

You define #BADA55 once as green.400, then you can reference that name everywhere. The value is still defined somewhere, but it is defined in one place with a meaningful name and consumed by both design tools and code by reference via tooling like Tokens Studio or Style Dictionary.

This centralisation helps to give you three major benefits:

Better consistency across UI
Easier theming and re-branding
Safer changes at scale because you update once and propagate everywhere

Tokens are not just a technical detail. They are a shared language between design and development (and the whole team) that ends up in the product(s).

The three token tiers

Most modern token workflows follow a three-tier model. You will see different names for each tier, but the same underlying structure keeps showing up.

Tier 1 - Primitive tokens

These are your raw values.

They describe what something is, not why it exists.

They are also referred to as reference, base, options, or global tokens.

Code languagejson

{
  "blue": {
    "500": {
      "$value": "#0066CC",
      "$type": "color"
    }
  },
  "spacing": {
    "8": {
      "$value": "8px",
      "$type": "dimension"
    }
  }
}

Primitives should not (imho) and are usually not consumed directly by components. They are source values for more specific design decisions and token tiers.

A common failure is skipping straight to component naming because it feels productive in the short term. In practice, that locks in naming decisions before your core scales are stable.

When possible, stabilise primitives first, attach intent later.

Code languagejson

{
  "core": {
    "dimension": {
      "100": { "$value": "8px", "$type": "dimension" },
      "200": { "$value": "16px", "$type": "dimension" },
      "300": { "$value": "24px", "$type": "dimension" }
    }
  }
}

Tier 2 - Semantic tokens

Also known as alias, decision, theme, or system tokens.

Semantic tokens can express intent. They describe why a value is used or what it is for.

Code languagejson

{
  "color": {
    "brand": {
      "primary": {
        "$value": "{blue.500}",
        "$type": "color"
      }
    }
  }
}

This is where tokens become resilient.

If brand blue changes, the semantic name can stay the same while the underlying primitive reference changes.

You can update once and it will change everywhere.

Code languagejson

{
  "layout": {
    "spacing": {
      "formStack": { "$value": "{core.dimension.300}", "$type": "dimension" },
      "contentToButton": {
        "$value": "{core.dimension.200}",
        "$type": "dimension"
      }
    }
  }
}

Tier 3 - Component tokens

Also known as "component specific" or contextual tokens.

This layer maps semantic intent to specific elements, components, and parts thereof.

Component tokens should always reference semantic tokens, never primitives directly. That indirection is what can keep the system flexible.

It is optional, but very useful in larger systems.

Code languagejson

{
  "button": {
    "primary": {
      "background": {
        "$value": "{color.brand.primary}",
        "$type": "color"
      }
    }
  }
}

Component tokens give you precision. You can tune one component without creating side effects and explosions elsewhere.

Common naming conventions

There is no universal winner ... "It depends". Different teams will always optimise for different makeup, needs, and constraints.

These diagrams show the core layers for each convention. Real-world usage may add optional layers like state or namespace for more specificity.

1. Category-Property-Modifier (CPM)

category property role state

This starts broad and gets more specific: category, property, role (and optionally state for interactions). It is easy to scan and keeps naming fairly lightweight.

Code languagejson

{
  "color": {
    "button": {
      "primary": {
        "hover": {
          "$value": "{color.brand.primary-hover}",
          "$type": "color"
        }
      }
    }
  }
}

2. Tier-based naming

primitive semantic component state

Here, the naming pattern changes depending on the tier. Primitives hold values, semantic tokens hold intent, and component tokens hold local UI decisions, which keeps each layer focused. The diagram shows the token tiers; actual nesting depth varies (e.g., primitives are often 2 layers, while component tokens may be 3+).

Code languagejson

{
  "blue": {
    "500": {
      "$value": "#2196F3",
      "$type": "color"
    }
  },
  "color": {
    "brand": {
      "primary": {
        "$value": "{blue.500}",
        "$type": "color"
      }
    }
  },
  "button": {
    "primary": {
      "background": {
        "$value": "{color.brand.primary}",
        "$type": "color"
      }
    }
  }
}

3. Object-Property-Modifier (OPM)

object property modifier state

This one starts with the UI object, then narrows to property and variant. It tends to feel natural in component-led workflows because it mirrors how teams talk about UI day to day.

Code languagejson

{
  "button": {
    "background": {
      "primary": {
        "hover": {
          "$value": "{color.brand.primary-hover}",
          "$type": "color"
        }
      }
    }
  }
}

4. Context-first naming

tier category property modifier

Context (e.g., 'brand' or 'theme') comes first, followed by category, property, and modifier. This ensures you know the layer before the specific decision. That works well when the same semantic token needs to exist across several themes or brands.

Code languagejson

{
  "brand": {
    "color": {
      "primary": {
        "default": {
          "$value": "{primitive.color.blue.500}",
          "$type": "color"
        }
      }
    }
  }
}

5. A Comprehensive Structure

namespace category concept property variant state

Each segment has a specific job, from namespace through to state. This comprehensive structure allows for up to 6 layers, but start with the core 5 and add as needed for complexity. Names are longer, but they are very explicit, which helps when multiple teams and brands share the same system. Nathan Curtis's article Naming Tokens in Design Systems goes much deeper on this taxonomy, it's well worth a reading (and bookmarking).

Code languagejson

{
  "acme": {
    "color": {
      "button": {
        "background": {
          "primary": {
            "hover": {
              "$value": "{acme.color.brand.primary}",
              "$type": "color"
            }
          }
        }
      }
    }
  }
}

Handling variants, states, and modes

Whatever base naming model you choose, you still need clear, predictable state and variant placement patterns.

One practical way to stay coherent is to define where each concept lives in the name:

Variant: what version of the same thing it is (for example primary, secondary, danger)
State: how that version is currently behaving (for example hover, active, disabled)
Mode: which environment it belongs to (for example light, dark, highContrast)

If those three ideas float around in different positions, naming gets noisy fast.

The same decision in different conventions

These examples all express the same meaning: primary button background on hover in dark mode.

Code languagejson

[
  "button.primary.background.hover.dark",
  "button-background-primary-hover-dark",
  "dark.button.primary.background.hover",
  "acme.color.button.background.primary.hover.dark",
  "theme.dark.component.button.primary.background.hover"
]

Pick one ordering model and enforce it consistently. Mixing these patterns inside one system is where discoverability starts to break down.

Variants

Variants describe alternatives at the same hierarchy level, not interaction changes.

Code languagejson

[
  "button.primary.background",
  "button.secondary.background",
  "button.danger.background"
]

Equivalent variant naming in other conventions:

Code languagejson

[
  "button-background-primary",
  "button-background-secondary",
  "button-background-danger",
  "component.button.background.primary",
  "component.button.background.secondary",
  "component.button.background.danger"
]

States

States represent interaction or status changes for the same UI element. Keep state terms predictable (hover, active, focus, disabled) so engineers can find related tokens quickly.

Code languagejson

[
  "button.primary.background.hover",
  "button.primary.background.active",
  "button.primary.background.disabled",
  "button.primary.background.focus"
]

The same state grouping works across naming styles:

Code languagejson

[
  "button-background-primary-hover",
  "button-background-primary-active",
  "button-background-primary-focus",
  "button-background-primary-disabled"
]

Code languagejson

[
  "component.button.background.primary.hover",
  "component.button.background.primary.active",
  "component.button.background.primary.focus",
  "component.button.background.primary.disabled"
]

Scale sizes

Scale tokens create consistent step-based sizing. Whether you use xs-xl labels or numeric steps, use one scale pattern and apply it everywhere to avoid drift.

Code languagejson

{
  "spacing.sm",
  "spacing.md",
  "spacing.lg",
}

For type scales, the same rule applies: keep naming sequential so size relationships are obvious at a glance.

Code languagejson

["font.size.sm", "font.size.md", "font.size.lg"]

Modes and themes

Modes are contextual variants of the same decision (for example light and dark). The token purpose stays the same, only the value changes per mode.

Code languagejson

["color.background.default.light", "color.background.default.dark"]

Other mode-placement patterns you will see in real systems:

Code languagejson

[
  "light.color.background.default",
  "dark.color.background.default",
  "theme.light.color.background.default",
  "theme.dark.color.background.default"
]

Whichever pattern you choose, keep mode placement fixed. A stable suffix or a stable prefix both work; switching between them does not.

Numbered scales

Numbered scales are useful when you need many fine-grained steps, especially for colour ramps and heading systems. The key is to keep the numbering direction and increments consistent.

Code languagejson

[
  "font.size.heading.1",
  "font.size.heading.2",
  "font.size.heading.3",
  "color.gray.50",
  "color.gray.100",
  "color.gray.200"
]

Numbered scales also appear in alternative patterns:

Code languagejson

[
  "size-font-heading-100",
  "size-font-heading-200",
  "size-font-heading-300",
  "scale.type.heading.100",
  "scale.type.heading.200",
  "scale.type.heading.300"
]

Rules that keep token naming healthy

Most strong naming systems follow a predictable anatomy: an optional prefix (namespace or context), a core meaning (category and property), optional modifiers (intent, variant, scale), and an optional suffix (state or mode).

You do not need every part in every token, but when a part is present it should always appear in the same position.

1. Consistency beats perfection

The best naming system is the one your whole team actually follows.

Drifting tends to start not where the convention is broken outright, but where it is ambiguous enough to invite interpretation. A token set that mixes naming styles makes it impossible to predict what a new token should be called.

Avoid: mixing conventions across the same token tier

Code languagejson

[
  "color.button.primary.default",
  "button.background.primary.hover",
  "theme.dark.component.button.primary.background.default"
]

Prefer: one convention applied throughout

Code languagejson

[
  "component.button.primary.background.default",
  "component.button.primary.background.hover",
  "component.button.secondary.background.default"
]

2. Optimise for clarity over brevity

Autocompletion means typing long names is rarely a problem, but reading a token list at a glance still relies on human pattern recognition. Abbreviating too aggressively trades a small typing convenience for an ongoing cognitive overhead every time someone reads or audits the system.

Avoid abbreviations that require decoding

Code languagejson

["btn-bg-pri-hvr", "c-txt-err", "sp-md"]

Prefer names that read on first glance

Code languagejson

["button-background-primary-hover", "color-text-error", "spacing-md"]

3. Keep semantics semantic

A name like color-text-red describes a visual value, which means the name breaks as soon as the colour changes. A name like color-text-error describes intent, which stays stable even when the underlying value does not. At semantic and component levels, name for why, not what.

Avoid: naming for the visual value

Code languagejson

["color.text.red", "color.background.grey", "color.border.green"]

Prefer: naming for the intent

Code languagejson

["color.text.error", "color.background.subtle", "color.border.success"]

4. Order from broad to specific

Names are easier to scan when they move from the widest concept toward specificity. This also means that alphabetically sorted token lists naturally group related tokens together, which makes auditing and searching much faster.

Avoid: specificity before category

Code languagejson

["hover.primary.button.color", "disabled.background.input", "lg.font.heading"]

Prefer: broad category first, narrowing toward specificity

Code languagejson

[
  "color.button.primary.hover",
  "color.input.background.disabled",
  "font.heading.lg"
]

5. Make names self-explanatory

A developer who has never seen your design tokens before should be able to make a reasonable guesstimate about what it does. If a name requires specific knowledge or a colleague to interpret, it is a candidate for renaming.

Avoid: names that only make sense in context

Code languagejson

["token-1", "c-bd-x", "primary-a"]

Prefer: names that explain themselves

Code languagejson

[
  "button.border.radius",
  "color.border.focus",
  "color.button.background.primary"
]

6. Design for growth

Your naming model should be able to absorb more components, more themes, and potentially more brands without a heavy structural rewrite. That does not mean you need to "over engineer" from the start, but it does mean avoiding patterns that only work at small scale, like implicit positional meaning or naming steps after their current count.

Avoid: patterns that hit a ceiling

Code languagejson

["spacing-small", "spacing-smaller", "spacing-new", "spacing-new-2"]

Prefer: a scale that can always grow in either direction

Code languagejson

["spacing.100", "spacing.200", "spacing.300", "spacing.420"]

Choosing your convention

A practical way to decide is to consider your team size, system scope, tooling, and existing patterns together.

Larger teams will need a stronger structure, single product systems can probably stay simpler, and tools like Figma, Tokens Studio, and your build pipeline may naturally favour one style.

It is also usually safer to evolve existing patterns than replace everything at once, and the best outcome comes when designers and developers both buy into the same model.

Start simple, document it clearly, add examples, and revisit as the system grows.

Token names do not need to be academically perfect. They need to be clear enough that your team can apply them without hesitation.

Keep naming alive with lightweight governance

Naming is not a one time decision.

It is an ongoing Design System practice.

If you want conventions to hold up over time, define lightweight governance:

One canonical naming guide with approved examples
A short review checklist for new tokens
Periodic cleanup of duplicate or ambiguous names
Shared ownership across design and engineering

The goal is not "process for process' sake".

A token system in a Design System that people trust is one they will actually use, and consistent naming is how that trust gets built.

Upgrading Forgejo with S3 Object Storage and Actions!

2026-05-03T10:50:05Z

Awhile ago I spun up my own instance of Forgejo. Forgejo, if you are not familiar, is a hard fork of Gitea when they became a for-profit company and is run/managed by Codeberg. If you want more details on why Forgejo came to be, you can read their blog post here. As of now it’s basically a drop-in replacement for Gitea. Though over time this will change as the two projects drift apart. So it may not be one in the future! Anyhow, I’ve been rolling my own instance since September 2023 and, recently I have been moving my entire lab towards a more GitOps approach. To do this and do it securely (in the confines of my internal network), I’ve decided to turn Forgejo into my full CI/CD manager and config storage place. I’ll also be using Bitwarden’s Secret Manager to manage, well, secrets.

Before I begin re-doing my lab for the umpteenth time, I need to change how Forgejo operates currently. That way Forgejo can support the transition from ‘FileOps’ (aka loose files) to GitOps. Now this isn’t the ‘perfect’ setup process as the repo storage still lives in /data, and it’s not currently replicated across my small swarm cluster. This will change when I get to actually rebuilding the cluster from the ground up to utilize NFS mounts for shared storage. Though that requires more hardware, which the wife probably won’t be happy about. 😅

Migrating to S3 Object Storage

Forgejo (in Docker), by default, has you map a /data directory where all the necessary configs/repos/logs/etc. live. As you can see by the example compose file here.

docker-compose.yml


    image: codeberg.org/forgejo/forgejo:7
      - ./forgejo:/data # <---- data volume
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

This is great if you plan on just running a simple setup, or you already have shared storage. For me though, I have no shared storage, and a MinIO server that is currently doing nothing but holding attachments for Outline. I figured it’s time to put it to more use. That, and my day job involves doing AWS things so figured it would be good practice.

Currently, Forgejo supports the following directories (under /appdata) for object storage.

Forgejo Docs: Storage Settings
Subsystem	Directory	app.ini Sections
Attachments	attachments/	[attachment]
LFS	lfs/	[lfs]
Avatars	avatars/	[avatar]
Repository avatars	repo-avatars/	[repo-avatar]
Repository archives	repo-archive/	[repo-archive]
Packages	packages/	[packages]
Actions logs	actions_log/	[storage.actions_log]
Actions Artifacts	actions_artifacts/	[actions.artifacts]

As you can see in the table above there are a few storage directories missing. Mainly repositories. Since repositories do not work well in object storage they are unsupported. Which means they will still be living on one of my hosts temporarily until I finalize my cross cluster storage solution. In order migrate the storage to MinIO, I first need to create a bucket (named forgejo), an Access Identity, and an ACL for it.

access-credentials

Access_key: O3n9bNhrbadkeyAGKOuhIUzMGF
Secret_key: czgQT1fb1l3GD2nRQEPebadsecretbWG0xGIu20hjOeBfoSGo

Access Key User Policy


    "s3:ListBucketMultipartUploads",
    "s3:AbortMultipartUpload",
    "s3:ListMultipartUploadParts",

The above ACL replaces the default one under the “Current User Policy” after enabling the “ON” switch for “Restrict beyond user policy” setting in User > Access Keys > Create access key. It limits the access key to the forgejo bucket. The bucket is also private by default, so I do not have to change the bucket ACL.

MinIO Create Access Key with a custom user policy

Next was to upload the existing files to MinIO so that when Forgejo restarted with the new s3 config it would pick them back up. As you can see in the picture below there are a few absent directories. I decided to condense a few related directories into parent ones for my ease of use. Also, some of them do not exist yet and will be created when needed (e.g. packages/).

Example MinIO bucket directory layout

My current s3 bucket directory layout:

With MinIO ready to go, all I had to do was update the Forgejo container environment variables for the MinIO connection and re-deploy. Forgejo environment variables can be defined using the env -> ini notation. So in your app.ini you have sections that are labeled, such as ‘[security].’ In order to update the INI value for say, ‘INSTALL_LOCK’, you’d need to define it as FORGEJO__security__INSTALL_LOCK.

As you can see in the example snippet file below, I am defining the MinIO connection information, as well as, where to look for specific directories in the bucket.

forgejo-minio-env-example

      - FORGEJO__storage__STORAGE_TYPE="minio"
      - FORGEJO__storage__MINIO_USE_SSL="true"
      - FORGEJO__storage__MINIO_ENDPOINT="s3.${LAB_DOMAIN}"
      - FORGEJO__storage__MINIO_ACCESS_KEY_ID=${FJ_S3_ACCESS_ID}
      - FORGEJO__storage__MINIO_SECRET_ACCESS_KEY=${FJ_S3_ACCESS_KEY}
      - FORGEJO__storage__MINIO_BUCKET="forgejo"
      - FORGEJO__storage__MINIO_LOCATION="us-east-1"
      - FORGEJO__attachment__MINIO_BASE_PATH="attachments/"
      - FORGEJO__lfs__MINIO_BASE_PATH="lfs/"
      - FORGEJO__avatar__MINIO_BASE_PATH="avatars/users/"
      - FORGEJO__repo-avatar__MINIO_BASE_PATH="avatars/repositories/"
      - FORGEJO__repo-archive__MINIO_BASE_PATH="archives/"
      - FORGEJO__packages__MINIO_BASE_PATH="packages/"
      - FORGEJO__storage.actions_log__MINIO_BASE_PATH="actions/logs/"
      - FORGEJO__actions.artifacts__MINIO_BASE_PATH="actions/artifacts/"

Forgejo Compose Example [Expand me]

forgejo.yml


      test: ["CMD", "curl", "-fSs", "localhost:3000/api/healthz"]
    image: codeberg.org/forgejo/forgejo:7.0
      - FORGEJO_APP_NAME="Forgejo"
      - FORGEJO_RUN_MODE="prod"
      - FORGEJO_WORK_PATH="/data/gitea"
      - FORGEJO__repository__ROOT="/data/git/repositories"
      - FORGEJO__repository.local__LOCAL_COPY_PATH="/data/gitea/tmp/local-repo"
      - FORGEJO__repository.pull-request__DEFAULT_MERGE_STYLE="merge"
      - FORGEJO__repository.signing__DEFAULT_TRUST_MODEL="committer"
      - FORGEJO__server__APP_DATA_PATH="/data/gitea"
      - FORGEJO__server__DOMAIN="git.${LAB_DOMAIN}"
      - FORGEJO__server__SSH_DOMAIN="git.${LAB_DOMAIN}"
      - FORGEJO__server__HTTP_PORT="3000"
      - FORGEJO__server__ROOT_URL="https://git.${LAB_DOMAIN}"
      - FORGEJO__server__DISABLE_SSH="false"
      - FORGEJO__server__SSH_PORT="22"
      - FORGEJO__server__SSH_LISTEN_PORT="22"
      - FORGEJO__server__LFS_START_SERVER="true"
      - FORGEJO__server__LFS_JWT_SECRET="${FJ_JWT_SECRET}"
      - FORGEJO__server__OFFLINE_MODE="false"
      - FORGEJO__database__DB_TYPE="postgres"
      - FORGEJO__database__HOST="${FJ_DB_HOST}"
      - FORGEJO__database__NAME="${FJ_DB}"
      - FORGEJO__database__USER="${FJ_DB_USER}"
      - FORGEJO__database__PASSWD="${FJ_DB_PASS}"
      - FORGEJO__database__LOG_SQL="false"
      - FORGEJO__database__SCHEMA=""
      - FORGEJO__database__SSL_MODE="disable"
      - FORGEJO__indexer__ISSUE_INDEXER_PATH="/data/gitea/indexers/issues.bleve"
      - FORGEJO__session__PROVIDER_CONFIG="/data/gitea/sessions"
      - FORGEJO__session__PROVIDER="file"
      - FORGEJO__session__COOKIE_SECURE="true"
      - FORGEJO__log__MODE="console"
      - FORGEJO__log__LEVEL="info"
      - FORGEJO__log__ROOT_PATH="/data/gitea/log"
      - FORGEJO__security__INSTALL_LOCK="true"
      - FORGEJO__security__SECRET_KEY="${FJ_SECRET_KEY}"
      - FORGEJO__security__REVERSE_PROXY_LIMIT="1"
      - FORGEJO__security__REVERSE_PROXY_TRUSTED_PROXIES="${CADDY_INT_IP}"
      - FORGEJO__security__INTERNAL_TOKEN="${FJ_INT_TOKEN}"
      - FORGEJO__security__PASSWORD_HASH_ALGO="${FJ_PASS_ALGO}"
      - FORGEJO__service__DISABLE_REGISTRATION="true"
      - FORGEJO__service__REQUIRE_SIGNIN_VIEW="false"
      - FORGEJO__service__REGISTER_EMAIL_CONFIRM="false"
      - FORGEJO__service__ENABLE_NOTIFY_MAIL="false"
      - FORGEJO__service__ALLOW_ONLY_EXTERNAL_REGISTRATION="false"
      - FORGEJO__service__ENABLE_CAPTCHA="true"
      - FORGEJO__service__CAPTCHA_TYPE="cfturnstile"
      - FORGEJO__service__REQUIRE_CAPTCHA_FOR_LOGIN="true"
      - FORGEJO__service__CF_TURNSTILE_SECRET="${FJ_CFT_SECRET}"
      - FORGEJO__service__CF_TURNSTILE_SITEKEY="${FJ_CFT_SITE_KEY}"
      - FORGEJO__service__DEFAULT_KEEP_EMAIL_PRIVATE="true"
      - FORGEJO__service__DEFAULT_ALLOW_CREATE_ORGANIZATION="true"
      - FORGEJO__service__DEFAULT_ENABLE_TIMETRACKING="true"
      - FORGEJO__service__NO_REPLY_ADDRESS="noreply@${FJ_DOMAIN}"
      - FORGEJO__mailier__ENABLED="false"
      - FORGEJO__openid__ENABLE_OPENID_SIGNIN="false"
      - FORGEJO__openid__ENABLE_OPENID_SIGNUP="false"
      - FROGEJO__cron.update_checker__ENABLED="false"
      - FORGEJO__oauth2__JWT_SECRET="${FJ_OAUTH_SEC}"
      - FORGEJO__actions__ENABLED="true"
      - FORGEJO__actions__DEFAULT_ACTIONS_URL="https://git.${LAB_DOMAIN}"
      - FORGEJO__actions__ARTIFACT_RETENTION_DAYS="90"
      - FORGEJO__storage__STORAGE_TYPE="minio"
      - FORGEJO__storage__MINIO_USE_SSL="true"
      - FORGEJO__storage__MINIO_ENDPOINT="s3.${LAB_DOMAIN}"
      - FORGEJO__storage__MINIO_ACCESS_KEY_ID=${FJ_S3_ACCESS_ID}
      - FORGEJO__storage__MINIO_SECRET_ACCESS_KEY=${FJ_S3_ACCESS_KEY}
      - FORGEJO__storage__MINIO_BUCKET="forgejo"
      - FORGEJO__storage__MINIO_LOCATION="us-east-1"
      - FORGEJO__attachment__MINIO_BASE_PATH="attachments/"
      - FORGEJO__lfs__MINIO_BASE_PATH="lfs/"
      - FORGEJO__avatar__MINIO_BASE_PATH="avatars/users/"
      - FORGEJO__repo-avatar__MINIO_BASE_PATH="avatars/repositories/"
      - FORGEJO__repo-archive__MINIO_BASE_PATH="archives/"
      - FORGEJO__packages__MINIO_BASE_PATH="packages/"
      - FORGEJO__storage.actions_log__MINIO_BASE_PATH="actions/logs/"
      - FORGEJO__actions.artifacts__MINIO_BASE_PATH="actions/artifacts/"
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

With the environment variables updated, and a quick redeploy of Forgejo, I was up and running with object storage via MinIO! A quick trip into the Site Administration > Configuration > Summary in Forgejo shows that LFS was now using MinIO as seen in the below image.

Forgejo site admin config showing LFS using MinIO

Setting up Forgejo Actions

All jokes aside, Forgejo Actions is actually fairly decent. The runner is not production ready (according to Forgejo themselves) but it works well enough that I am going all in on it for my lab. Setting up the action runner was fairly straight forward. It’s got a similar setup process to the self-hosted GitHub runner.

Enable Actions in the app.ini or using environment variables. Restart/Redeploy Forgejo.

See above for an example using environment variables under #Actions Configs.

Create a new runner in Site Administration > Actions > Runners > Create new runner.
Copy the new runner registration token.
Deploy Forgejo Runner and register it with the token from step 3.

Now there are a few ways you could deploy the runner, but I decided to deploy it in Docker. You can bind the runner container to the Docker sock, but I did not. I really do not like doing that so, I ended up using Docker in Docker (DIND). It’s not really recommended to use Docker this way but for my lab it will do fine. The compose file was also pretty easy to set up.

forgejo-actions.yml


    command: ['dockerd', '-H', 'tcp://0.0.0.0:2375', '--tls=false']
    image: code.forgejo.org/forgejo/runner:3.4.1
        condition: service_started
    command: '/bin/sh -c "while : ; do sleep 1 ; done ;"'
    # Change to this command after registration
    #command: '/bin/sh -c "sleep 5; forgejo-runner daemon"'
      - DOCKER_HOST=tcp://dind:2375

As you can see above there are two commands:. '/bin/sh -c "while : ; do sleep 1 ; done ;"' starts up the runner container and does nothing. This allows you to enter the container via docker exec and run the registration command using the token generated in the Site Administration settings. After registering the container, it can be brought down and the current command can be replaced with the commented out one ('/bin/sh -c "sleep 5; forgejo-runner daemon"'). Then after redeploying the container the runner will become available in Forgejo globally. Having a global runner means it can be used by all repositories. Which for me works fine since my Forgejo instance is mainly private except for a few public repositories and a few GitHub mirrors.

Forgejo Manage Runners

With my runner online and working, I am now able to have workflows kick off from steps defined in files located in .forgejo/workflows/. The syntax is fairly similar to GitHub actions so most of the actions you can run there can run in Forgejo. The best part is I can mirror the action repositories and have my runner pull them directly from my Forgejo instance. The first thing I set up was Renovate Bot to keep my Docker images updated. The bot runs on a cron based workflow as seen below and opens a PR for image updates.

.forgejo/workflows/renovate.yml


    - cron: '0 0/6 * * *'  # every 6 hours, every day
  RENOVATE_REPOSITORIES: ${{ github.repository }}
    if: ${{ secrets.RENOVATE_TOKEN != '' }}
      image: renovate/renovate:full
          GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
          RENOVATE_BASE_DIR: ${{ github.workspace }}/.tmp
          RENOVATE_ENDPOINT: ${{ github.server_url }}
          RENOVATE_REPOSITORY_CACHE: 'enabled'
          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
          RENOVATE_GIT_AUTHOR: 'Renovate Bot <renovate@mydomain.dev>'
          GIT_AUTHOR_NAME: 'Renovate'
          GIT_AUTHOR_EMAIL: 'renovate@mydomain.dev'
          GIT_COMMITTER_NAME: 'Renovate Bot'
          GIT_COMMITTER_EMAIL: 'renovate@mydomain.dev'
      - name: Get Ntfy Token from SM
        uses: actions/sm-action@v4
        access_token: ${{ secrets.BITWARDEN_LAB_KEY }}
          some-secret-id-string > NTFY_TOKEN
          some-secret-id-string > LAB_DOMAIN
        uses: actions/ntfy-action@master
          url: 'https://ntfy.${LAB_DOMAIN}'
          headers: '{"authorization": "bearer $NTFY_TOKEN"}'
          details: Renovate Bot failed to run!

The bot is configured via a renovate.json file located in the root of the repository.

renovate.json


  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base"],
  "assignees": ["alexandzors"],
  "dependencyDashboard": true,
  "labels": ["maintenance"],
  "enabledManagers": ["docker-compose"],
    "fileMatch": ["^.*\\.yml$"]
      "managers": ["docker-compose"],
      "depTypeList": ["services"],
      "datasources": ["docker"]

At the moment I have it scanning through all YAML files since I do not name my compose files docker-compose.yml. They are named as the service so: forge.yml, caddy.yml, authelia.yml, etc. They are also located in their own folders. This will be changing as I move more to GitOps but for now it’s fine.

PR Opened by Renovate Bot using the Forgejo Action workflow

Conclusion

This was a fun project to work on in between work and family stuff. Hopefully this inspires one of you to upgrade your own local git setup with an ‘Actions’ runner or object storage! I’ll have a full config dump for MinIO, Forgejo, Forgejo Actions, and an example workflow in the blog-files repo.

On to the next project. 🚀

Rebuilding My Home Lab

2026-05-03T10:46:07Z

_{FTC: Some links in this post are income earning affiliate links.}

It’s been a while since I last posted about my homelab or well posted on the blog. So I figured I’d start off with a banger for 2025. Rebuilding my homelab into a Docker Swarm cluster that supports high availability. Hope you’re ready for a ride, because this was definitely a journey. 😅

_{Side note: I also recently acquired a Bambu Lab A1 3D printer (with the AMS :)) to replace my old Anet A8. I’d say it’s mine, but I’ve only been printing stuff for my wife so far..}

The Plan

The plan for this project was to:

Create a cluster that supported high availability ✅
Setup a distributed storage solution that supported the above ✅
Setup a load balancer ✅
Have the cluster communicate over 2.5gb or faster networking ✅
Have a minimum of 3 nodes ✅
Be small and efficient ✅

Easy enough right?… right?

Since I had two Dell Optiplex 5050 Micros that I was using already, I decided to grab a third micro to complete the 3 node cluster. As for the actual orchestration software I decided to stick with Docker and use Docker Swarm. I’m a Docker guy but never had a chance to use Swarm properly with a shared storage system. Figured it was time to give it a proper go.

Wait… Docker Swarm? Alex you should use Kubernetes!!!!!!

Current v2 Cluster Specs

2x Dell Optiplex 5050 Micros
1x Dell Optiplex 7040 Micros
3x Realtek 8125B 2.5gb A+E NICs
3x Intel 700p 256GB SSDs
3x Crucial BX500 1TB SSDs
Ubiquiti Flex-2.5g Switch
Ubuntu 24.04 LTS
GlusterFS

Cluster v1

_{circa 2023}

Cluster v1 never actually made it to ‘production’. It ended up being a test bed and a learning experience on gotchas when it came to Linux, Intel, drivers, and poor research on my part…

TLDR: I originally bought some Intel i225-v 2.5gb m.2 cards from IOCrest thinking they’d work. Come to find out these particular chips have some driver issues on Linux (specifically Ubuntu).

When originally designing the cluster I had planned on using M.2 2.5g cards since the Micro’s had M.2 M-key slots. That way I could get a less expensive SATA SSD for storage. After doing some research I settled on some Intel 225-V 2.5g cards from IOCrest on AliExpress. The one downside to these cards was the vertical connector pins. The chassis did not have enough space with a SATA SSD installed. So I did what any tinkerer would do and replaced the pins with right angle ones. The first two cards went good, but my old soldering iron doesnt keep temp well. On the third card’s NIC module I messed up and destroyed one of the traces. So I had to fix it with a jumper wire. Live and learn I suppose.

As for the NICs themselves, I started experiencing driver issues after getting two of the nodes up and running in a ‘mock’ swarm. These issues manifested as random disconnects or complete NIC power loss. I tried to fix it by messing with grub configs for pcie_aspm thinking it was related to Active State Power Management (ASPM) at first. I then tried manually setting the autonegotiation via ethtool. Also double checked UEFI settings, etc. Nothing seemed to work. So I ended up scrapping the Intel cards and going with Realtek 8125B cards for cluster v2.

Anyways, I’ll leave you with a few pictures from the v1 prototype, and we can move on to v2.

Cluster v2

_{circa 2024}

My second attempt at the cluster faired a bit better. Well mostly (more on that later). After my first purchase mistake I went back and bought different M.2 adapters. These were A+E keyed and used the WLAN slot rather than the M.2 M-keyed slot which got freed up for more storage. These new adapters also use a Realtek chipset (8125B) rather than an Intel one. So no more odd driver issues.

M.2 A+E keyed adapter installed

With the new adapters using the A+E slot, I decided to use the M slot for a boot drive and the SATA port for cluster storage. For the boot drive I went with some Intel 700p 256GB SSDs I bought off someone from the Ubiquiti Discord server. As for cluster storage I went with 1TB Crucial BX500 SSDs from Amazon. These are not the fastest drives by any means but for my use case they should do just fine.

Storage SSDs

There was one downside to the Intel SSDs though. Their physical size. They are 2230 and the Micro chassis only supports 2260 and 2280. So in order to use them I had to print out some 2230 to 2280 adapter brackets. I also printed out a 2.5in tray for the third Micro since it did not come with one. The print files are linked below.

The OS

For the OS I decided to stick with my tried and true Ubuntu Server (24.04 LTS btw). It’s stable and easy to manage. Though, I may eventually re-image to Rocky Linux for easy live patching when I rebuild again. The process was pretty standard. I booted via my netboot.xyz PXE server, ran through the installer and then used Ansible to configure the OS. I’ll have the playbooks + task files in the blog-files repo after this post is live.

Your browser does not support webm.Ansible playbook running on the cluster nodes

Networking

After the OS was set up on all the nodes, I went ahead configured the network interfaces. During the v2 build Ubiquiti launched their Flex 2.5g switch line. Since this was an SFF cluster the Flex Mini 2.5g was a perfect fit as the cluster’s network backbone. Also with it being powered by POE made for one less cable to deal with. The cluster network is also a fairly simple one. It has no access to the internet and no access to any other networks. Other networks also can’t access it. Don’t need stuff snooping on my unencrypted SQL traffic..

With the network interfaces configured and working, it was time to test the speed using iperf. As you can see by the short video below, I was able to get roughly 2.5gbps of bandwidth between the nodes.

Your browser does not support webm.iperf testing

Distributed File System

This was probably the most difficult part of the cluster build. At the time I had really no experience dealing with distributed file systems. I narrowed down my options to GlusterFS and Ceph. I ended up going with GlusterFS for this iteration. Using the official documentation I was able to get a basic cluster up and running fairly quickly. I also created a dedicated Docker user ‘doc’ and set its home directory to the GlusterFS volume on each of the nodes. I thought this would be a great idea, but it ended up being a pain later on.

GlusterFS setup

Docker Swarm

With the OS, Networking, and DFS setup, it was time to set up Docker Swarm. Setting up Docker and Docker Swarm is fairly straight forward using the official documentation. You basically follow the installation on each of the nodes. Then you initialize the swarm on one and join the other nodes to it. In my case my commands were:

Terminal window


sudo docker swarm init --advertise-addr 192.168.100.2 ## init the swarm on the cluster network interface
sudo docker swarm join --token SWMTKN-1-4c2d6gncjh15gaznem5m8t2twwfn09o0paqpjxwdkqr1n76h8j-chr42vcbaobx7v2nrk86q9e93 192.168.100.2:2377 ## joining other nodes to the swarm

I then promoted the other two nodes to managers. This way I had proper high availability as it requires a minimum of 3 managers to be in the swarm.

Docker Swarm cluster

Keepalived

Next up was setting up Keepalived to handle VRRP for the cluster. This allowed me to have a single IP address that automatically failed over to the next node automatically. The video below shows the failover in action (sped up to keep the video short).

Your browser does not support webm.Keepalived VRRP failover

As you can see the active node switches as each keepalived service is terminated. The lower right shows a constant ping to the VRRP virtual IP (VIP). In this case 10.8.8.11. Having this VIP allows me to bring down hosts for maintenance or updates without having to worry about service connectivity.

HA Caddy

The final step before I could start using the cluster was to set up a load balancer/reverse proxy. There are a few different options but, if you’ve read any of my previous posts you know I love using Caddy. With its easy-to-use configuration and automatic certificate management makes it hard to beat. Also, Caddy is a great choice for this since it automatically clusters itself if all instances use the same storage volume.

First thing I did was create the new overlay network for Caddy. This network would be for any services that I wanted to be able to access from outside the cluster.

Terminal window


sudo docker network create --opt encrypted=true --driver overlay --attachable --internal --subnet=172.0.96.0/20 caddy-internal

Creating the Caddy service file.

caddy.yml


    image: alexandzors/caddy:2.9.1
      - /swarm/volumes/doc/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
      - /swarm/volumes/doc/caddy/.data:/data
      - /swarm/volumes/doc/caddy/configs:/etc/caddy/configs:ro

Then deploying Caddy to the swarm.

Terminal window


sudo docker stack deploy --c caddy.yml caddy

docker service ls command showing Caddy running on all 3 nodes

I eventually need to update the Caddy deployment to use host mode for port assignments. That way I can get the source IP address of incoming requests and not the docker proxy IP. But that will be part of v3.

Cluster v3

_{circa 2025}

So here we are. Cluster v2 has been running great for a while now, but I’ve been having some issues with it. The main, and arguably the largest, issue I have been running into is the slow performance of GlusterFS. I’m not sure if it’s the hardware choices I made, or a misconfiguration? I tried the performance tuning tips from the official documentation but was still running into issues. Like switching to the cluster user ‘doc’ would take anywhere from 30 to 45 seconds to complete. Also, deploying stacks of services could take up to 5 minutes before they were alive and running.

Either way for v3 I’m probably going to move to Ceph as my DFS solution. Since it seems to be the more popular of the two after digging around more. I may need to upgrade the RAM on the nodes to handle it though. I’m also going to try and add a 4th node to the cluster with beefier hardware since I still have one 2.5g port left on the switch. Maybe one with some GPU compute to handle local LLMs?

Current services running on the cluster

However, what won’t be changing is my use of Docker Swarm. 😉

How to smoke

2026-05-02T05:49:10Z

The best part of smoking was always the click of the Zippo.

This week’s question comes to us from Mat Honan:

I used to love to smoke. If it weren’t for the whole lung cancer, emphysema, death thing, would you recommend smoking?

You’re forgetting the smell.

Last week I was coming back from the record store and feeling lazy, so I jumped on the bus. A couple of stops after I got on, a dude got on and sat down next to me. Well-dressed dude, also carrying an Amoeba Records shopping bag. Out and about on a Sunday afternoon, doing his record shopping just like me. Within seconds it became clear this dude had just had a cigarette. And he stunk. The bus was also packed at this point, so I was pretty much stuck in that seat until I got off, which luckily wasn’t for too much longer. Also, it feels kinda shitty when you sit down next to somebody on the bus and they immediately get up. (Unless you’re getting a creeper vibe, of course.) I just had to suck it up for a few more stops. But man, it was rough. I’m not trying to disparage the guy or anything. Especially because I used to be that guy, and you used to be that guy, and I’m guessing a lot of our readers used to be that guy. We just walked around smelling awful.

Which is not to say that’s the worst part of smoking, but you took lung cancer, emphysema, and the dead thing off the table. Which leaves us with the smell.

I started smoking my freshman year in college. All reasons to take up smoking are stupid, but this one might be the stupidest. This being the mid-80s our dorm had a cigarette machine in the lobby. One of those old machines you might still see in old man dive bars, with two rows of big ka-chunky knobs, which felt objectively good to pull. It was like you could feel the entire mechanism of the machine come to life when you pulled that knob. And it took some strength to do it! But you could feel the knob hit a gear, you could hear the gear spin, you could feel a pack of cigarettes get pushed free from deep inside the machine. You could hear it slide down a little ramp, and then you’d see it appear down in the landing zone, where you’d push your hand past the trap door, grab it, and somehow have the pack open, and a cigarette in your mouth before the rest of the pack hit your pocket.

If memory serves, a pack of smokes was between $1.25 and $1.50 around that time. (Fun fact, I attempted to Google this and the slop top on the search results page said 26¢ a pack, which is… not true. But please, continue to rebuild society around such amazing technology.) Being the art school miscreants that we were, and also broke, we discovered that if you pulled a knob halfway out, inserted a quarter, and then pulled the knob the rest of the way out it would release a pack of cigarettes. (Ok, that may not have been the actual process, but it was a long time ago, and it’s very close to the spirit of the actual process, so let’s run with it. So I guess we were getting a pack of cigarettes for what Google’s stupid slop robot said, but it included doing crimes.) The knowledge of how to hack the cigarette machine spread through the dorms like wildfire, and soon we were all smoking. Because we were idiots. Also, it felt like we were getting one over on The Man. But mostly because we were idiots.

Also, being art school kids we were very visually-driven people, and all the photos of cool people that we’d hang up in our dorm rooms showed them holding a cigarette. And we very much wanted to be cool people. (True fact: take a photo of Humphrey Bogart, replace the ever-present cigarette with a vape and Humphrey Bogart will look like an herb.) Again, mostly we were idiots.

The vending machine company tried to patch the hack several times, eventually gave up and just took the machine away. This was our first lesson that sometimes doing crime is in the public interest, but that lesson didn’t occur to us right away. At the time, we were just pissed that we had to pay retail for cigarettes again.

By the time I started smoking society was pretty much done with the pretense that smoking was doing anything but murdering you slowly. I know this because we’d sit around in art classes making collages using old cigarette ads where doctors would tell you smoking was good for your nerves, and we would laugh at people for believing this, as we lit cigarette after cigarette. (Yes, you could smoke in class.) And we thought “Boy, our grandparents sure were chumps for believing cigarettes were healthy.” Then we would have a coughing fit. But there was definitely the sense that doing this thing that we all knew had a very very high probability of killing us wasn’t a big deal, mostly because we were in our 20s when nothing can hurt you, Reagan was president and, just to reiterate—we were idiots.

My first post-college job was at a copy shop, and you got one 15 minute break during your shift. Unless you smoked, then you could get as many breaks as you needed. Several people started smoking while working there.

We all stunk. We’d come in from smoking out back, and immediately walk up to the service counter to help a customer. A customer who either stunk as bad as we did, or had become inured to the stench because it was all around them, emanating from everyone.

We smoked in class. We smoked at the movies. We smoked at the supermarket. We smoked at sporting events. My friend Jeff, who grew up in Boston, tells a good story about going to Celtics games as a kid and having to look past the hovering cloud of smoke between the cheap seats and the court. We smoked in restaurants, where the smoking and non-smoking sections were often divided by nothing more than a paper sign denoting the territorial boundary. We smoked on planes, man.

It wasn’t too long after college that the world began to shift. In 2003 New York City banned smoking in bars. And I was visiting at the time. By 2003, I was no longer “a smoker” but I was very much someone who would look for reasons to bum a smoke from someone if the situation arose, and very likely to put myself in situations where it might. But I remember the rage from several friends and from the owners and bartenders of any bar we’d walk into. The ban was going to kill bars all over the city. It was going to kill nightlife. It was going to completely take down the economy. New York, as we know it, would cease to exist. Which of course, it didn’t. Everyone adjusted. They went outside. They eventually started smoking less because it was cold outside. People enjoyed being able to hang out in rooms that weren’t making them sick, and they enjoyed going home not reeking of cigarette smoke. If I could go back in time I’d reassure all those bartenders that it wasn’t the smoking ban they had to worry about. It was the kids who’d stop going out at all because they needed to sit at home and tend to their AI agents.

I selected your question this week because I’ve actually been thinking of the smoking ban lately. We grew up in a time when smoking, or dealing with other smokers was an inevitability. Even if you didn’t smoke, you’d most likely work next to someone who did, or sit down next to someone who did at a restaurant, or at the movies. And even if they weren’t actively smoking, and covering you in second-hand smoke, you’d go home with the stench of smoke all over you. Airing your clothes out was an inevitability. Having to wash the stench out of your hair was an inevitability. Society smoked, so you did too. Whether you wanted to or not. And then it changed. Most cities in America now have smoking bans and rules about how far away you have to be from a public entrance to smoke, which get enforced to various degrees.

The change came in a couple of very interesting ways. One, cigarettes are now hovering between $12 to $14 a pack. (I had to look this up!) They’re also available in less places. (You used to be able to buy cigarettes at the drug store!) Secondly, people just look at you weird if you start smoking now. Like, what the fuck dude, did you just light a cigarette!? Are you from the past? Gross.

Which of course makes me think of some of the things that we have currently accepted as a society, things which we fully know are not healthy for society, that we are currently tolerating. And also thinking there’s no way it will ever change, because we appear to be in an era of “what if everyone modeled themselves off the stupidest people?”

Right now there is someone firing up ChatGPT because it’s cheap. Right now there is someone writing a prompt in Claude because it brings him closer to his co-workers. Right now there is someone walking a co-worker through his agentic workflow, in the same way we attempted to impress one another by blowing smoke rings. Right now there is someone parking a Cybertruck on your street, believing that leaving his divorce where everyone can see it is somehow impressive. We have always been good at ignoring the warnings that came with the pack.

Our parents packed their homes with asbestos. They heated their homes with coal. They packed their Big Macs in styrofoam. Making mistakes will always be cheaper than fixing them. But nothing is more expensive than ignoring them.

Cultural norms are an ever-changing thing. History is the story of what was once desirable becoming unacceptable. Something that used to be an inevitability is no longer inevitable. Something that used to be tolerated is no longer tolerated. Something that was seen as a cultural norm no longer is. Even when those things were backed by entire industries with very strong lobbies, as the tobacco lobby once was. The same fate will someday befall the NRA. The same fate will someday befall AIPAC. The same fate will someday befall the slop lobby.

There was a time we thought if we prohibited people from smoking in bars it would lead to societal collapse. I think it was a good idea. More importantly, it was an idea that worked. It improved not just our personal health but the health of our communities.

The basic strategy of all addictive technologies is very simple. They make you feel extra capable, they addict you, then they make you feel inadequate without them. They start by making you feel cool, and confident. Relax. Put your feet up. Hang with the fellas. Social anxiety? It’s toasted, dog! Let me write that résumé for you. Anniversary card for your wife? I can write that for you. Light one up. It’s a great way to start a relationship. But that initial boost eventually turns to reliance, and suddenly you can’t get out of bed without a hit. You can’t write your kid a love note without firing up a slop engine. And suddenly an entire industry is telling you that you’re not capable of moving through your day without their help. An entire industry gaslighting you, until it becomes easier to just gaslight yourself into believing that you were never truly capable of things you are very much capable of.

I don’t miss smoking. Maybe I did at one point. But eventually the whiff of cigarette smoke went from smelling nostalgic to just smelling bad. Thankfully, knock on wood, I’ve been able to escape years of smoking without any major lasting effects, but trust that I carry every pack I ever smoked with me every time I walk up a flight of stairs. If I’m doing anything strenuous, it’s always my lungs that give up first.

Thankfully, I still have some lung capacity. I enjoy using it. You should too.

🚬

🙋 Got a question? Ask it! I will somewhat answer it!

📓 Get your sexy copy of my new book How to Die (and other stories)! And if you’re in the Bay Area, come see me and Annalee Newitz talk about it on May 11 at Booksmith!

🚢 My friend Lucy Bellwood made a wonderful comic about loss and building.

🙅 My friend Jason Cosper made a great tool for fucking with Substack’s dollar.

📣 I’ve got a few seats left in next week’s Presenting w/Confidence workshop where you can learn true confidence. The one inside you.

🍉 Please support the children of Palestine.

🏳️‍⚧️ Please support trans kids. And if there is a trans person in your life please tell them you love them.

rarouš.w3b

AI Software Development - What Does The Data Say? - Codemanship's Blog

Share this:

Related

Author: codemanship

Leave a comment Cancel reply

Cloudflare OS is an architecture of distrust

Make the code irrelevant to safety

Never let the agent hold the key

Let it act, but never let it commit

Distrust the outputs, not just the actions

A tell in the contributing guide

Is this wisdom or a cage?

SSD Streaming for AI Models: How to Turn RAM from a Wall into a Dial

When RAM Becomes the Bottleneck, Not the Solution

The Real Problem with Running Large Models Locally

Plans first. Then code.

The Scale Problem Gets Worse with Capability

Mixture of Experts: Why Not All Weights Are Created Equal

What This Means Practically

How SSD Streaming Actually Works

Everyone else built a construction worker.We built the contractor.

The Math Behind It

Prefetching Matters

SSD Speed Is Not Optional

The Trade-off Triangle: RAM, Speed, and Model Quality

What You Gain

Other agents ship a demo. Remy ships an app.

What You Give Up

Who This Is Actually For

Where Cloud Models Change the Calculation

Other agents start typing. Remy starts asking.

Practical Setup Considerations for SSD Streaming

Storage Requirements

Quantization Still Helps

RAM Allocation

Thermal Throttling on Laptops

Frequently Asked Questions

What is SSD streaming for AI models?

Does SSD streaming reduce model quality?

What kind of SSD do I need for AI model streaming?

Why does SSD streaming work better for MoE models?

How does Dwarf Star differ from just using swap space?

Can I use SSD streaming for production applications?

Key Takeaways

New research: The Core Web Vitals thresholds you trust might be wrong for your site

A Boring Tiny Tooling paradigm

Three imagination failures

A consulting problem

Imagination is the hard part

Even with a silver bullet, you still have to aim

The purpose of a system is what it does

The bottleneck is the context in which the work takes place

untitled

The Aircraft Engine/Automotive Validations (2004, 2007)

The Micro Software Validation (2006)

The Macro Software Validation (2012)

The Industry Reality Check (1995-2020)

Conclusion

The Series: Navigating the Shades

Bibliography

untitled

The Plan

untitled

untitled

The AI adoption gap and the 0.04%

Three ways the bubble distorts your decisions

What the data actually showed

Building different journeys for different people

The playground: A structural answer

Start with the problem. The tools will follow.

Build the bridge your users are ready to cross

Social RSS (?)

On wintering.

10x engineers: the Purple Developer

Native Apps Should Be Avoided Whenever Possible

The White House App

The Software Embedded in Apps

Why Everyone Should Care

What Apps Can Do That Websites Can’t

Everyone else built a construction worker.
We built the contractor.

`safe-area-max-inset`

Browser support for `safe-area-max-inset-*`