rarouš.w3b

Native Apps Should Be Avoided Whenever Possible

2026-05-15T05:08:54Z

April 12, 2026

Native Apps Should Be Avoided Whenever Possible

TL;DR: What you should do:

Openly refuse apps, and vocally advocate for the web instead.

Try not to install any apps if you don’t need to.

If a service has a functioning website, use it instead.

Revoke all permissions by default, including background location, microphone, and camera permissions for anything that doesn’t require them to function.

Audit your installed apps. Uninstall all apps you don’t actively need.

Treat every “download our app” prompt with skepticism.

Most native apps collect far more data than their website equivalents ever could. They request permissions to hardware, sensors, and background processes that browsers deliberately restrict. The third-party software embedded in these apps frequently transmits your location, device identifiers, and behavioral data to third parties before you even see a consent prompt. This data is in tandem bought, sold, and aggregated by brokers. It has been used to out individuals, track immigrants, and enable prosecution over reproductive healthcare.

The White House App

On March 27, 2026, the Trump administration released an official White House app for iOS and Android. Within hours, two independent security researchers decompiled it and published their findings [1]. The app is a textbook example of everything wrong with the native app model.

Apple requires apps to submit a privacy manifest disclosing what data they collect. The White House app declared an empty array. Zero data collection. Meanwhile, the actual binary contained ten analytics frameworks, including the full OneSignal SDK with a sub-framework specifically for location tracking [2]. The GPS pipeline polled precise coordinates every 4.5 minutes in the foreground and every 9.5 minutes in the background, syncing everything to OneSignal’s commercial servers. A boolean flag in OneSignal’s server responses could remotely enable or disable GPS tracking without an app update and without Apple review.

An Exodus Privacy audit identified three embedded trackers, one of which was Huawei Mobile Services Core [3]. The app’s privacy policy, last updated January 20, 2025, makes no mention of GPS tracking, OneSignal, or background data collection.

Nearly everything in the app is available on whitehouse.gov. The app’s unique additions are push notifications, a pre-filled text message to the President, and an ICE tip button (also available on ice.gov). What it actually added at scale was a surveillance pipeline: 77% of the app’s network requests go to third parties, not whitehouse.gov.

The Software Embedded in Apps

Most people think of apps as products built by a single company. In practice, the average app is a thin wrapper around dozens of third-party software packages, each with its own data collection pipeline and commercial incentives. When you grant an app permission to access your location, every package embedded in that app inherits that permission. A single package can appear in hundreds of apps, feeding location data on millions of people to a single aggregator.

In January 2025, a hacker breached Gravy Analytics and leaked roughly 30 million location records collected from 3,455 apps — dating, fitness, gaming, and health apps among them [4]. The FTC subsequently banned Gravy Analytics from selling Americans’ location data [5], but by then the data was already circulating on cybercrime forums.

In a separate case, Google paid $391.5 million to settle claims from 40 states for continuing to collect location data even when users explicitly disabled location tracking [6].

Why Everyone Should Care

This data is bought, sold, and aggregated by brokers. It has been used to out individuals, track immigrants, and enable prosecution over reproductive healthcare. In multiple cases, journalists and private groups have purchased app-derived location data to identify specific people based on their movements.

There are virtually no restrictions in the United States on buying, selling, or weaponizing this data. There is no comprehensive federal privacy law. And there isn’t likely to be one soon. The best we can do is minimize the data we share in the first place.

What Apps Can Do That Websites Can’t

The core argument for using the website instead of the app comes down to what each platform is technically capable of doing without your knowledge.

Capability	Native App	Website / PWA
Background location tracking	Yes, can poll GPS continuously	No
Run at device startup	Yes	No
Access biometric hardware	Yes	Limited (WebAuthn, user initiated)
Modify or delete device storage	Yes	No (sandboxed to browser)
Embed invisible third-party software	Yes, all inherit granted permissions	No, scripts visible in page source
Transmit data before consent prompt	Yes (common with third-party software)	Restricted by browser policies
Push notifications while closed	Yes	Yes (PWA, user opt in required)
Access contacts, call logs, SMS	Yes (if permitted)	No
Prevent phone from sleeping	Yes	No
Camera and microphone	Yes (persistent if granted)	Yes (per session, prompted each time)
Offline functionality	Yes	Yes (via service workers)

The browser is the security boundary. Websites operate within it. Native apps bypass it.

The Access Provided by Default is Enough to Do Real Harm

The moment you install an app, before you allow a single permission prompt, it can:

Reach any server on the internet
Read your IP address, device model, OS version, timezone, country, carrier, and network type
Generate and persist a unique identifier tied to your device
Run code at device startup (Android) and wake up in the background
Fingerprint your device by combining the above into a signature that follows you across sessions
Grant all of this same access to every third-party software package embedded in the app
Compare this data to other datasets to infer your identity, demographics, interests, and habits

The runtime permission prompts you actually see (location, camera, contacts) are helpful while annoying, but the majority of the default access permissions do not require your consent.

A website, by contrast, starts with almost none of this: no persistent identifier, no background execution, no third-party software inheritance, no startup hooks.

Some Things Need to Be Apps, But Most Don’t

Some things need to be apps. AR and VR, real-time games, anything talking to NFC or Bluetooth hardware, serious audio and video work, accessibility tools. These are legitimate cases where the browser sandbox is the limitation. In these circumtances, I personally use a full computer as opposed to my phone.

Almost nothing else qualifies. Your banking, your travel, your grocery store, the restaurant down the street — none of it needs an app. And rewards be damned. No rewards are worth the data you are willingly giving them.

Same goes for hardware. If a thermostat or a fitness tracker can’t be set up without a proprietary app, that’s a flaw in the product, not a feature. I immediately avoid such products. You’re buying an ongoing relationship with someone else’s servers and guaranteeing that you’ll forget that corporations are watching everything they can.

Conclusion

I avoid most apps. It turns out this is easier than most people assume, because the app is almost never the only option. It is just the option the company wants you to take and not enough people question.

We are at a very specific time in humanity right now. Where aggregating data is a currency, and it is actively being utilized at a scale never before seen. I recommend you at least take stock of what you’re freely giving away.

References

1. I Decompiled the White House’s New App (Thereallo, March 28, 2026) and Security Analysis of the Official White House iOS App (atomic.computer, March 27, 2026). Two independent researchers decompiled the app on Android and iOS within hours of release.

2. Security Analysis of the Official White House iOS App (atomic.computer). Documents the empty privacy manifest, OneSignal SDK with ten sub-frameworks, and the remote GPS toggle via server response.

3. Exodus Privacy Report: gov.whitehouse.app. Automated audit identifying three embedded trackers including Huawei Mobile Services Core. Additional context in Fedware: 13 Government Apps That Spy Harder Than the Apps They Ban (Sam Bent).

4. A breach of Gravy Analytics’ huge trove of location data threatens the privacy of millions (TechCrunch, January 13, 2025).

5. FTC Finalizes Order Prohibiting Gravy Analytics, Venntel from Selling Sensitive Location Data (FTC, January 14, 2025).

6. Google to pay $391.5 million in location tracking settlement with 40 states (TechCrunch, November 14, 2022).

AI dělá chyby a autisté je dokážou najít. Z jejich jinakosti dělá Češka ve svém startupu přednost

2026-05-07T05:32:58Z

Představte si konkrétní situaci: datoví vědci ve startupu trénují svůj analytický model, ale v určitých případech narážejí na vysokou chybovost v podobě falešně pozitivních výsledků. Místo toho, aby se pálil drahý čas seniorních inženýrů na manuálním čištění datasetů, předá se úkol ven. Výsledek? Najatý tým projde pět iterací dat a pomůže tak o polovinu zlepšit úspěšnost modelu. Právě na tom staví mladý pražský startup Diversight. Jeho tajnou zbraní nejsou pokročilejší algoritmy, nýbrž dedikovaní analytici na autistickém spektru. Ukazuje tak, že vlastnosti, kvůli kterým se jiné firmy těmto lidem někdy vyhýbají, se mohou proměnit na silné stránky.

„Nejsme charita ani neziskovka, jsme klasická technologická firma,“ říká hned na úvod spoluzakladatelka a ředitelka Diversightu Irena Zatloukalová. V českém technologickém prostředí se pohybuje dvacet let, a to zejména v oblasti komunikace, kdy sedm let byla mluvčí tuzemské internetové jedničky Seznam.cz. Mimo jiné působila i jako spolumajitelka PR agentury Fenek a později na volné noze radila s komunikací technologickým firmám jako Heureka, Apify, Liftago či Ackee. Loni v červenci ale všechny klienty předala dál a naplno se vrhla do nového byznysu.

V Diversightu se zaměřuje na takzvaný datový dluh. V éře, kdy se prakticky každá firma snaží do svých procesů implementovat umělou inteligenci, stojí budoucí úspěch primárně na kvalitě vstupních dat. „I ty nejpokročilejší vývojářské týmy totiž musí manuálně odstraňovat duplicity a řešit hraniční případy, v nichž zavedená automatizace zkrátka selhává a modely začínají generovat nesmysly. Diversight funguje jako filtr, který přebírá na přesnost náročnou vrstvu ekosystému a zajišťuje potřebnou lidskou pojistku proti halucinacím,“ vysvětluje pro CzechCrunch Zatloukalová.

Foto: Diversight

Irena Zatloukalová s kolegou Šimonem

Právě v oblasti hledání nepatrných vzorců totiž samotná umělá inteligence běžně naráží na své limity, dodává. Pokud modely generují opakující se chyby, vývojářům nepostačí jen jednoduché přepsání textového promptu. Týmy potřebují člověka, který dokáže precizně pochopit byznysovou logiku a vyčíst přímo ze syrové databáze konkrétní pravidla a odchylky. „V tom máme ohromnou výhodu, protože máme lidi, kterým jednou řeknete pravidla a oni pak po nich neúnavně půjdou, od prvního řádku až po řádek číslo 1 500,“ popisuje specifikum práce Zatloukalová.

Doplňuje přitom, že jejím lidem nedělá problém udržet stejnou hladinu pozornosti na jednom tématu klidně i osm dní v kuse: „Prostě potřebují problém vyřešit, neznámou najít a věc dotáhnout do konce. To pomyšlení na rozlousknutí problému pomáhá jejich fokusu.“

Na samotném začátku příběhu Diversightu přitom nestála Zatloukalová. „Ráda bych to říkala, ale bohužel to nebyl můj nápad,“ usmívá se a jako originálního tvůrce označuje Tomáše Borovičku z technologické společnosti Datamole. Ten na zahraničních trzích narazil na izraelskou firmu Point AI, která budovala data-anotační byznys (zaměřený na označování surových dat) na přesné práci lidí na spektru. Od té doby přemýšlel nad tím, jak vyzkoušet podobný koncept i v Česku.

„Byla to jedna večeře, jedno potkání, tři hodiny povídání. Šli jsme hodně do hloubky a do našich hodnot a na konci mi řekl: Tak jo, pojďme to zkusit,“ vzpomíná Zatloukalová na zlom, který přišel v prosinci 2024. A dodává, že na začátku je propojila Lenka Kučerová ze spolku prg.ai.

Původní plán Diversightu počítal s tím, že se do Prahy přenese jako franšíza přímo izraelský model. Rychlý průzkum trhu ale ukázal, že běžný anotační byznys je v tuzemsku v podstatě jen levná komodita, kde firmy často najímají studenty a na finální kvalitě jim tolik nesejde. Zatímco v Izraeli se data pro zdravotnictví či zbrojní průmysl anotují s kritickou přesností, v lokálních podmínkách by takový model ekonomicky nepřežil. Diversight proto musel okamžitě změnit plány. Zaměřil se na komplexnější analytické úkoly a takzvanou validaci s člověkem v procesu (human-in-the-loop), kde se naplno využije kognitivní preciznost ve složitých datech.

Projekt od začátku strategicky i finančně podpořil zmíněný Datamole, který mu poskytl peníze na bezpečný roční provoz a působí v roli partnera. Mimo jiné nabízí i přímý přístup ke svým seniorním AI architektům. Hlavním byznysovým úkolem pro nadcházející měsíce je tak dostat Diversight do zisku díky příjmům ze zakázek.

Foto: Diversight

Tomáš Borovička a Irena Zatloukalová

Mezi aktuální klienty vedle analytické společnosti BizMachine, pro kterou startup dělá v úvodu zmíněné čištění dat, patří například také InfraHex, jedna z firem holdingu Respilon. Ta vyvinula technologii, která upravuje tepelnou stopu živých i neživých objektů a tím omezuje jejich detekci pomocí termokamer. V současnosti se využívá i na Ukrajině, kde pomáhá vojákům v terénu. I drobné odchylky zde hrají zásadní roli, protože se systém opírá o přesné analýzy obrazů v různých infra spektrech. A tak se na této detailní práci vyhodnocování a výběru dat podílejí lidé na autistickém spektru.

Českým mediálním i firemním prostorem dnes podle Zatloukalové bohužel rezonují převážně extrémní obrazy autismu, kdy na jedné straně stojí příběhy nespolupracujících jedinců s těžkým postižením, na straně druhé zprofanované postavy geniálních vědců, jako je například Sheldon Cooper z Teorie velkého třesku. Diversight ale systematicky hledá velkou množinu lidí s nadprůměrnou analytickou inteligencí ležící mezi těmito dvěma extrémy. Většina z nich má však potíže projít přes výběrová řízení či běžná HR oddělení, protože standardní firemní postupy zkrátka neumí s neurodiverzitou pracovat.

Pokud zahodíte své ego a uznáte, že někdo našel lepší cestu, firmě to hodně pomůže.

„Už jen to, když do pracovního inzerátu vypíšete deset věcí jako ‚nice to have‘. Člověk na spektru si to projde, u třetí položky si řekne, že má jen dva a půl roku zkušeností místo tří, takže to není práce pro něj, a jde dál,“ vysvětluje Zatloukalová jeden z mnoha detailů, proč kandidáti sami rovnou z recruitmentu vypadávají. Značná část lidí na spektru tak i kvůli tomu končí na úřadu práce či na podřadných pozicích a ve firmách běžně narazí kvůli logickému poukazování na neefektivitu a navrhování lepších postupů.

Zatloukalová proto změnila rovnou i náborový proces a přizpůsobila ho lidem na spektru. Přesné zadání úkolů a otázky zasílá s předstihem, kandidáty navíc netestuje pod umělým tlakem, ale zaměřuje se striktně na jejich schopnost strukturovaně přemýšlet. Aktuálně má firma pět lidí – vedle samotné Zatloukalové je to technologický manažer Adam Vesecký a interní tým tří specializovaných analytiků. Záměrem je tým rozšiřovat, nicméně se stropem na hranici deseti zaměstnanců, aby firma nepřišla o klidné prostředí komorního kolektivu.

Zatloukalová navíc varuje před zjednodušováním nebo předsudky, že práce s neurodivergentními lidmi přináší jen těžkosti a komunikační zádrhele. Na první pohled nepříjemné situace, kdy podřízený manažerovi zcela upřímně oznámí, že zadal úkol špatně, totiž vedou k lepším procesům do budoucna, a tím i výsledkům. „Pokud zahodíte své ego a uznáte, že někdo našel lepší cestu, firmě to hodně pomůže,“ vysvětluje Zatloukalová.

Lidé na spektru podle ní přinášejí do byznysu inovace a odlišné úhly pohledu nejen v datové analytice, ale napříč všemi obory od zákaznické podpory po design. „Mají přirozenou schopnost jít striktně po podstatě problému a všímat si i drobných odchylek, které neurotypická většina snadno přehlédne,“ doplňuje. Výměnou za tento výkon často vyžadují jen velmi specifické, avšak finančně nenáročné úpravy pracovního prostředí – typicky přesun stolu dál od rušné kuchyňky, utlumení ostrého kancelářského osvětlení nebo zkrátka jen možnost po náročné schůzce úplně vypnout a odejít domů.

Cítím nespravedlnost. Jsem naštvaná na to, jakým způsobem se společnost staví k lidem, kteří jsou jiní.

Častou chybou některých manažerů je tyto potřeby zpochybňovat jen proto, že zaměstnanec nemá na stole oficiální lékařskou diagnózu. „Moje velká prosba k jakémukoliv manažerovi je: i bez diagnózy člověka poslouchejte. Když říká, že něco potřebuje, respektujte to, nezpochybňujte a hledejte cesty, jak zařídit, aby mohl svoji práci vykonávat co nejlépe,“ apeluje. Jak sama upozorňuje, nevyžaduje to stavbu drahých oddělených kanceláří, často stačí investice do kvalitních sluchátek s aktivním potlačením hluku.

Přístup orientovaný na lidi se specifickými potřebami má podle Zatloukalové přesah do celkového chodu jakékoliv firmy. Veškerá drobná provozní pravidla a limity, které startup explicitně zavádí kvůli neurodivergentním kolegům, totiž v důsledku ulehčují práci všem. „Každý zaměstnanec občas potřebuje ničím nerušený čas na práci, jen se to v běžných korporacích tolik neakcentuje. Funguje zde stejný princip jako v moderním urbanismu. Pokud navrhnete veřejný prostor tak, aby vyhovoval lidem na vozíku, dětem i rodičům s kočárky, vytvoříte ve finále vysoce funkční prostředí, ve kterém se bude dobře žít úplně všem,“ srovnává.

Na druhou stranu Zatloukalová doplňuje, že budování neurodiverzního týmu s sebou pro vedení nese nutnost preciznosti v mezilidské komunikaci. „Tou se živím dvacet let a největší paradox je, že se mi opravdu stává, že si s kolegy nerozumím v zadání,“ přiznává s tím, že sama musela odložit své ego stranou a opět se učit zadávat úkoly maximálně jasně, doslovně a bez zbytečného sarkasmu.

To vše se promítá i do samotného workflow při řešení úkolů pro klienty. „Když řeknu ‚udělej A‘, kolega si začne domýšlet B, C a D. A místo toho, aby se zeptal, radši to rovnou udělá až po to D. Já mu pak musím říkat, že zašel moc daleko. Pro příště ale vím, že potřebuji dát víc kontextu k tomu, proč má v bodě A skončit,“ ilustruje občasné komunikační střety zakladatelka. Doslovnost a analytický rozklad problémů se však startupu okamžitě vrací ve chvíli, kdy tým narazí na nekvalitní a chaotická data v zadání.

„Cítím nespravedlnost. Jsem naštvaná na to, jakým způsobem se společnost staví k lidem, kteří jsou jiní. Ta jinakost nemusí být na první pohled patrná, ale znamená pro ně spoustu překážek, které jim vyrábíme my všichni. Angličtina pro to má slovní spojení ‚disabled by society‘. Jakmile mi ukážete něco takového, jsem totální buldok a jdu po narovnávání té nespravedlnosti,“ vysvětluje Zatloukalová, pro kterou se tak v Diversightu kromě společenského přesahu spojuje vášeň pro data a práce s lidmi. „Baví mě dělat věci, které nikdo předtím moc nedělal nebo neprozkoumal. Diversight má v sobě všechno,“ uzavírá.

Using safe-area-inset to build mobile-safe layouts

2026-05-06T12:14:32Z

Modern phones are not simple rectangles. They have rounded corners, camera cutouts, dynamic islands, and home indicators that double as gesture areas. Browsers know the dimensions of all of these and expose the parts that could obscure content as safe area insets.

The "safe area" is the portion of the screen that is guaranteed to be free from being obscured by system UI. The safe-area-inset is the measurement of how much space the system UI is taking up on each edge of the screen. By using these values in your CSS, you can make sure that important content and controls are not obscured by the system UI.

If you don't want your floating chat button to end up sitting behind the home indicator, where it's unreachable, you need to account for the safe area inset.

Environment variables for safe area insets

With the safe-area-inset environment variables, you can make your layout adapt to the current device's safe area and avoid those bugs. The env() function is how you read those values in CSS:

body {
  padding-top: env(safe-area-inset-top);
  padding-right: env(safe-area-inset-right);
  padding-bottom: env(safe-area-inset-bottom);
  padding-left: env(safe-area-inset-left);
}

Browser support

Safe-area-insets are baseline widely available, which means you can use them in production today and be confident that they will work for almost all your users on mobile devices.

Since it's baseline widely available, you don't really need to think about fallbacks but if you want to be extra safe, you can provide a fallback padding if the browser doesn't support env():

body {
  padding-top: 1rem; 
  padding-top: calc(env(safe-area-inset-top) + 1rem);
}

And for browsers that support env() but not safe-area-inset-* variables, you can provide a fallback value directly in the env() function. If safe-area-inset-top is not supported, it will fall back to 1rem:

body {
  padding-top: env(safe-area-inset-top, 1rem);
}

It's worth noting that this situation is purely theoretical, as all browsers that support env() also support safe-area-inset-* variables. That might not be the case with other environment variables, so it's good to know that this fallback mechanism exists.

Using safe-area-inset values

Before we get into problem areas, let's get a general understanding of how these variables work and how you can use them to affect the layout.

In this demo, change each inset and watch how a realistic mobile UI that takes safe area insets into account shifts to remain usable:

The specific px values here are not particular to any specific device, the point is to show how the device sets these variables and how your layout can respond to them as they change.

On real devices, you can think of these values as constants. They're provided by the browser and while they might change when switching from portrait to landscape or between OS updates that change the device UI as well as simply differ per device, they don't change dynamically as the user scrolls or interacts with the page. The browser provides them as a constant value that you can use to ensure your content is not obscured by the system UI.

Which web pages actually need this?

If you want your pages to look the best they can, all of them.

Browsers by default will prevent your site from being obscured by the notch or home indicator, so your content will be safe without any special handling. That does come with a downside, which is that the browser will give you a smaller viewport to reserve space:

You'll notice the edges here keep the site from being obscured, but they also don't look that great. Ideally we want the content to stretch edge-to-edge, but we want to make sure they're not obscured by system UI. To get that, you need to opt in to the full viewport and handle safe areas yourself.

To do so is a two-step process. First, you need to add viewport-fit=cover to your meta viewport tag:

<meta name="viewport" content="width=device-width, viewport-fit=cover" />

This will make the browser stretch your page edge-to-edge:

As you can see, the content sits behind the notch/dynamic island. viewport-fit=cover tells the browser that you want to be responsible for making sure your content is not obscured by the system UI.

And now we can move those elements away from behind the system UI using env(safe-area-inset-*).

.content {
  padding-right: env(safe-area-inset-right);
  padding-left: env(safe-area-inset-left);
}

If you opt into the full viewport, here are the kinds of things you now have to think about to make sure your content is not obscured by the system UI:

Fixed headers and navigation bars should have enough space above or below for the notch or home indicator
Floating chat or help buttons should remain inside the safe area
Full-screen dialogs and drawers should take up the right height without being obscured by the home indicator
Map or video controls near screen corners

Safe-area-inset doesn't provide margins

Safe-area-insets are defined to be exactly the space that the system UI is taking up. That means they don't provide any margin between the system UI's edge and your content.

If you set padding to just the safe-area-inset value, your content will sit right up against the edge of the safe area, which is right up against the system UI.

To add some breathing room, you can add your own padding on top of the safe area insets by combining things with calc(). This way you can ensure that your content is not only safe from being obscured but also has some space to breathe:

body {
  padding-top: calc(env(safe-area-inset-top) + 1rem);
  padding-right: calc(env(safe-area-inset-right) + 1rem);
  padding-bottom: calc(env(safe-area-inset-bottom) + 1rem);
  padding-left: calc(env(safe-area-inset-left) + 1rem);
}

Safe-area-inset only has non-zero values on mobile devices

env() is supported across browsers and platforms, but the safe-area-inset-* variables really only have non-zero values on mobile devices.

Desktop browsers always return 0 because there is no UI on top of pages in desktop browsers. It's only on mobile devices that these values are non-zero and that you have to account for them.

That's exactly why these bugs are so easy to miss. If you test in the Chrome responsive view, the safe area insets will still be 0 and you won't see any issues during development.

Testing on real devices is often delegated to the end of the project. By then, making layout changes to accommodate safe areas can be expensive, and the bugs can slip through to production. Even worse, they often only affect users on certain devices, so they can go unnoticed for a long time.

Polypane's device emulation supports safe area insets

Polypane is the first and only desktop browser to emulate safe area insets. Every device in Polypane has correct safe area inset values for both portrait and landscape orientations.

Here's Polypane showing the safe area insets in blue, and the small viewport height difference in pink:

We're also the only desktop browser to emulate svh, but that's a topic for another article.

With Polypane's safe-area-inset overlay visualization, you can see exactly where the unsafe areas are on each device.

Solving a specific issue: Floating buttons

Let's look at a specific example of how to use safe-area-inset values to solve a common issue: floating buttons that end up behind the home indicator and become unreachable.

.chat-button {
  position: fixed;
  right: 10px;
  bottom: 10px;
}

Toggle the positioning between 'Fixed position' and 'Using env()' to switch between hard-coded offsets and offsets that use safe-area-inset-bottom and safe-area-inset-right.

`safe-area-max-inset`

Along with safe-area-inset-*, the specification also describes safe-area-max-inset-* variables. Those are not widely supported yet, but they are worth mentioning because they have slightly different behavior:

safe-area-inset-* gives you the current inset value right now. On scroll, the browser chrome can collapse, and the inset value can shrink all the way to 0. Your element moves with that change.
safe-area-max-inset-* gives you the maximum inset the browser can report for that edge. It stays stable even when the browser chrome collapses. Use it when you want a reserved zone that does not jump around.

Drag or scroll the phone to simulate the browser address bar appearing and disappearing.

The green button uses safe-area-inset-bottom and follows the current inset. The blue button uses safe-area-max-inset-bottom and stays put.

OverlayShow safe inset areas

inset vs max-inset

When to use which really depends on your situation. For some UI components it makes sense to move with the live viewport state, like a floating chat button that should always be just above the home indicator.

For other things, like a persistent cookie banner or a full-screen dialog, it can be better to reserve a stable zone that doesn't shift when the browser chrome collapses so that users don't inadvertently tap the wrong button.


.floating-cta {
  bottom: calc(env(safe-area-inset-bottom) + 1rem);
}


.persistent-zone {
  bottom: calc(env(safe-area-max-inset-bottom) + 1rem);
}

Browser support for `safe-area-max-inset-*`

As of now, only Chromium implements safe-area-max-inset-* so there is no support in (mobile) Safari or Firefox. That means you can't depend on it yet and should provide a fallback stack for other browsers:

.bottom-spacer {
  padding-bottom: 1rem;
  padding-bottom: calc(env(safe-area-inset-bottom) + 1rem);
  padding-bottom: calc(env(safe-area-max-inset-bottom, env(safe-area-inset-bottom)) + 1rem);
}

Notice how we're using the env() fallback mechanism to fall back to safe-area-inset-bottom if safe-area-max-inset-bottom is not supported.

Testing safe areas in Polypane

As covered earlier, safe area bugs are easy to miss during development. Chrome's responsive view always reports inset values of 0, and real-device testing tends to get pushed to the end of the project, when fixing layout issues is expensive and bugs have already reached production.

Polypane changes that. You can test safe area insets (and small viewport behavior) directly on your desktop, across multiple devices and orientations simultaneously, as part of your normal development workflow.

Catch real-device layout failures during development, not after shipping.

What to take away

Safe areas are not edge cases for unusual devices. They are the reality of modern mobile devices with camera punch holes, notches, dynamic islands, and home indicator bars. Your users are on those devices, and if you want to give them the best experience, you need to make sure your content is not obscured by the system UI.

Make sure your viewport is set to viewport-fit=cover so you get the full viewport and that you use env(safe-area-inset-*) so all your content is visible and accessible. That gives users the best experience.

Get started with testing safe areas in Polypane today, and catch real-device layout failures during development, not after shipping.

Design Token Naming Conventions: A Practical Guide

2026-05-04T10:30:43Z

Design Token Naming Conventions: A Practical Guide

23rd of April 2026

On This Page:

If you have ever stared at a long token list and wondered whether a name should be button-primary-bg, primary-button-background, or color-button-primary, you are not alone.

Naming design tokens can look and feel simple right up until you have to do it for real. Choose a weak pattern and things get inconsistent fast. Choose a clear pattern and you get a shared language that helps both design and engineering move quicker.

There is no single correct convention. A two-person team shipping one product has very different needs from a large organisation running multiple brands across multiple platforms.

What you can do is pick a convention that is clear, consistent, and scalable. This article walks through the core token tiers, common naming models, and practical rules that help avoid naming drift.

Why naming gets hard faster than expected

Most teams don't struggle creating tokens. They struggle to keep token meaning stable as more people and components work on the Design System.

You can often see multiple names emerge for the same idea:

Code languagejson

["spacing.small", "space-2", "gap-md", "paddingStandard"]

Each name might make sense in isolation. Together, they signal a semantic drift. The naming model is no longer acting as a shared language.

This is why naming quality can affect delivery speed. If token intent is unclear, every usage becomes a local judgement call, and those judgement calls will accumulate into inconsistency.

The examples in this article follow the Design Tokens Community Group specification using $value, $type, and dot-notation aliases like {color.brand.primary}.

What are design tokens, a quick recap

Design tokens are a tooling and platform agnostic way to store design decisions as named values: colour, spacing, typography, radius, shadows, motion, and (much) more.

The key difference is where and how many times you define those values.

Without tokens, you might use #BADA55 in dozens of places across components, stylesheets, and design files. When that green needs to change, you might have to hunt down every instance to make sure you're updating everything that needs to be updated.

With tokens, you create a design token file, a JSON object following the Design Tokens Community Group specification:

Code languagejson

{
  "green": {
    "400": {
      "$value": "#BADA55",
      "$type": "color"
    }
  }
}

You define #BADA55 once as green.400, then you can reference that name everywhere. The value is still defined somewhere, but it is defined in one place with a meaningful name and consumed by both design tools and code by reference via tooling like Tokens Studio or Style Dictionary.

This centralisation helps to give you three major benefits:

Better consistency across UI
Easier theming and re-branding
Safer changes at scale because you update once and propagate everywhere

Tokens are not just a technical detail. They are a shared language between design and development (and the whole team) that ends up in the product(s).

The three token tiers

Most modern token workflows follow a three-tier model. You will see different names for each tier, but the same underlying structure keeps showing up.

Tier 1 - Primitive tokens

These are your raw values.

They describe what something is, not why it exists.

They are also referred to as reference, base, options, or global tokens.

Code languagejson

{
  "blue": {
    "500": {
      "$value": "#0066CC",
      "$type": "color"
    }
  },
  "spacing": {
    "8": {
      "$value": "8px",
      "$type": "dimension"
    }
  }
}

Primitives should not (imho) and are usually not consumed directly by components. They are source values for more specific design decisions and token tiers.

A common failure is skipping straight to component naming because it feels productive in the short term. In practice, that locks in naming decisions before your core scales are stable.

When possible, stabilise primitives first, attach intent later.

Code languagejson

{
  "core": {
    "dimension": {
      "100": { "$value": "8px", "$type": "dimension" },
      "200": { "$value": "16px", "$type": "dimension" },
      "300": { "$value": "24px", "$type": "dimension" }
    }
  }
}

Tier 2 - Semantic tokens

Also known as alias, decision, theme, or system tokens.

Semantic tokens can express intent. They describe why a value is used or what it is for.

Code languagejson

{
  "color": {
    "brand": {
      "primary": {
        "$value": "{blue.500}",
        "$type": "color"
      }
    }
  }
}

This is where tokens become resilient.

If brand blue changes, the semantic name can stay the same while the underlying primitive reference changes.

You can update once and it will change everywhere.

Code languagejson

{
  "layout": {
    "spacing": {
      "formStack": { "$value": "{core.dimension.300}", "$type": "dimension" },
      "contentToButton": {
        "$value": "{core.dimension.200}",
        "$type": "dimension"
      }
    }
  }
}

Tier 3 - Component tokens

Also known as "component specific" or contextual tokens.

This layer maps semantic intent to specific elements, components, and parts thereof.

Component tokens should always reference semantic tokens, never primitives directly. That indirection is what can keep the system flexible.

It is optional, but very useful in larger systems.

Code languagejson

{
  "button": {
    "primary": {
      "background": {
        "$value": "{color.brand.primary}",
        "$type": "color"
      }
    }
  }
}

Component tokens give you precision. You can tune one component without creating side effects and explosions elsewhere.

Common naming conventions

There is no universal winner ... "It depends". Different teams will always optimise for different makeup, needs, and constraints.

These diagrams show the core layers for each convention. Real-world usage may add optional layers like state or namespace for more specificity.

1. Category-Property-Modifier (CPM)

category property role state

This starts broad and gets more specific: category, property, role (and optionally state for interactions). It is easy to scan and keeps naming fairly lightweight.

Code languagejson

{
  "color": {
    "button": {
      "primary": {
        "hover": {
          "$value": "{color.brand.primary-hover}",
          "$type": "color"
        }
      }
    }
  }
}

2. Tier-based naming

primitive semantic component state

Here, the naming pattern changes depending on the tier. Primitives hold values, semantic tokens hold intent, and component tokens hold local UI decisions, which keeps each layer focused. The diagram shows the token tiers; actual nesting depth varies (e.g., primitives are often 2 layers, while component tokens may be 3+).

Code languagejson

{
  "blue": {
    "500": {
      "$value": "#2196F3",
      "$type": "color"
    }
  },
  "color": {
    "brand": {
      "primary": {
        "$value": "{blue.500}",
        "$type": "color"
      }
    }
  },
  "button": {
    "primary": {
      "background": {
        "$value": "{color.brand.primary}",
        "$type": "color"
      }
    }
  }
}

3. Object-Property-Modifier (OPM)

object property modifier state

This one starts with the UI object, then narrows to property and variant. It tends to feel natural in component-led workflows because it mirrors how teams talk about UI day to day.

Code languagejson

{
  "button": {
    "background": {
      "primary": {
        "hover": {
          "$value": "{color.brand.primary-hover}",
          "$type": "color"
        }
      }
    }
  }
}

4. Context-first naming

tier category property modifier

Context (e.g., 'brand' or 'theme') comes first, followed by category, property, and modifier. This ensures you know the layer before the specific decision. That works well when the same semantic token needs to exist across several themes or brands.

Code languagejson

{
  "brand": {
    "color": {
      "primary": {
        "default": {
          "$value": "{primitive.color.blue.500}",
          "$type": "color"
        }
      }
    }
  }
}

5. A Comprehensive Structure

namespace category concept property variant state

Each segment has a specific job, from namespace through to state. This comprehensive structure allows for up to 6 layers, but start with the core 5 and add as needed for complexity. Names are longer, but they are very explicit, which helps when multiple teams and brands share the same system. Nathan Curtis's article Naming Tokens in Design Systems goes much deeper on this taxonomy, it's well worth a reading (and bookmarking).

Code languagejson

{
  "acme": {
    "color": {
      "button": {
        "background": {
          "primary": {
            "hover": {
              "$value": "{acme.color.brand.primary}",
              "$type": "color"
            }
          }
        }
      }
    }
  }
}

Handling variants, states, and modes

Whatever base naming model you choose, you still need clear, predictable state and variant placement patterns.

One practical way to stay coherent is to define where each concept lives in the name:

Variant: what version of the same thing it is (for example primary, secondary, danger)
State: how that version is currently behaving (for example hover, active, disabled)
Mode: which environment it belongs to (for example light, dark, highContrast)

If those three ideas float around in different positions, naming gets noisy fast.

The same decision in different conventions

These examples all express the same meaning: primary button background on hover in dark mode.

Code languagejson

[
  "button.primary.background.hover.dark",
  "button-background-primary-hover-dark",
  "dark.button.primary.background.hover",
  "acme.color.button.background.primary.hover.dark",
  "theme.dark.component.button.primary.background.hover"
]

Pick one ordering model and enforce it consistently. Mixing these patterns inside one system is where discoverability starts to break down.

Variants

Variants describe alternatives at the same hierarchy level, not interaction changes.

Code languagejson

[
  "button.primary.background",
  "button.secondary.background",
  "button.danger.background"
]

Equivalent variant naming in other conventions:

Code languagejson

[
  "button-background-primary",
  "button-background-secondary",
  "button-background-danger",
  "component.button.background.primary",
  "component.button.background.secondary",
  "component.button.background.danger"
]

States

States represent interaction or status changes for the same UI element. Keep state terms predictable (hover, active, focus, disabled) so engineers can find related tokens quickly.

Code languagejson

[
  "button.primary.background.hover",
  "button.primary.background.active",
  "button.primary.background.disabled",
  "button.primary.background.focus"
]

The same state grouping works across naming styles:

Code languagejson

[
  "button-background-primary-hover",
  "button-background-primary-active",
  "button-background-primary-focus",
  "button-background-primary-disabled"
]

Code languagejson

[
  "component.button.background.primary.hover",
  "component.button.background.primary.active",
  "component.button.background.primary.focus",
  "component.button.background.primary.disabled"
]

Scale sizes

Scale tokens create consistent step-based sizing. Whether you use xs-xl labels or numeric steps, use one scale pattern and apply it everywhere to avoid drift.

Code languagejson

{
  "spacing.sm",
  "spacing.md",
  "spacing.lg",
}

For type scales, the same rule applies: keep naming sequential so size relationships are obvious at a glance.

Code languagejson

["font.size.sm", "font.size.md", "font.size.lg"]

Modes and themes

Modes are contextual variants of the same decision (for example light and dark). The token purpose stays the same, only the value changes per mode.

Code languagejson

["color.background.default.light", "color.background.default.dark"]

Other mode-placement patterns you will see in real systems:

Code languagejson

[
  "light.color.background.default",
  "dark.color.background.default",
  "theme.light.color.background.default",
  "theme.dark.color.background.default"
]

Whichever pattern you choose, keep mode placement fixed. A stable suffix or a stable prefix both work; switching between them does not.

Numbered scales

Numbered scales are useful when you need many fine-grained steps, especially for colour ramps and heading systems. The key is to keep the numbering direction and increments consistent.

Code languagejson

[
  "font.size.heading.1",
  "font.size.heading.2",
  "font.size.heading.3",
  "color.gray.50",
  "color.gray.100",
  "color.gray.200"
]

Numbered scales also appear in alternative patterns:

Code languagejson

[
  "size-font-heading-100",
  "size-font-heading-200",
  "size-font-heading-300",
  "scale.type.heading.100",
  "scale.type.heading.200",
  "scale.type.heading.300"
]

Rules that keep token naming healthy

Most strong naming systems follow a predictable anatomy: an optional prefix (namespace or context), a core meaning (category and property), optional modifiers (intent, variant, scale), and an optional suffix (state or mode).

You do not need every part in every token, but when a part is present it should always appear in the same position.

1. Consistency beats perfection

The best naming system is the one your whole team actually follows.

Drifting tends to start not where the convention is broken outright, but where it is ambiguous enough to invite interpretation. A token set that mixes naming styles makes it impossible to predict what a new token should be called.

Avoid: mixing conventions across the same token tier

Code languagejson

[
  "color.button.primary.default",
  "button.background.primary.hover",
  "theme.dark.component.button.primary.background.default"
]

Prefer: one convention applied throughout

Code languagejson

[
  "component.button.primary.background.default",
  "component.button.primary.background.hover",
  "component.button.secondary.background.default"
]

2. Optimise for clarity over brevity

Autocompletion means typing long names is rarely a problem, but reading a token list at a glance still relies on human pattern recognition. Abbreviating too aggressively trades a small typing convenience for an ongoing cognitive overhead every time someone reads or audits the system.

Avoid abbreviations that require decoding

Code languagejson

["btn-bg-pri-hvr", "c-txt-err", "sp-md"]

Prefer names that read on first glance

Code languagejson

["button-background-primary-hover", "color-text-error", "spacing-md"]

3. Keep semantics semantic

A name like color-text-red describes a visual value, which means the name breaks as soon as the colour changes. A name like color-text-error describes intent, which stays stable even when the underlying value does not. At semantic and component levels, name for why, not what.

Avoid: naming for the visual value

Code languagejson

["color.text.red", "color.background.grey", "color.border.green"]

Prefer: naming for the intent

Code languagejson

["color.text.error", "color.background.subtle", "color.border.success"]

4. Order from broad to specific

Names are easier to scan when they move from the widest concept toward specificity. This also means that alphabetically sorted token lists naturally group related tokens together, which makes auditing and searching much faster.

Avoid: specificity before category

Code languagejson

["hover.primary.button.color", "disabled.background.input", "lg.font.heading"]

Prefer: broad category first, narrowing toward specificity

Code languagejson

[
  "color.button.primary.hover",
  "color.input.background.disabled",
  "font.heading.lg"
]

5. Make names self-explanatory

A developer who has never seen your design tokens before should be able to make a reasonable guesstimate about what it does. If a name requires specific knowledge or a colleague to interpret, it is a candidate for renaming.

Avoid: names that only make sense in context

Code languagejson

["token-1", "c-bd-x", "primary-a"]

Prefer: names that explain themselves

Code languagejson

[
  "button.border.radius",
  "color.border.focus",
  "color.button.background.primary"
]

6. Design for growth

Your naming model should be able to absorb more components, more themes, and potentially more brands without a heavy structural rewrite. That does not mean you need to "over engineer" from the start, but it does mean avoiding patterns that only work at small scale, like implicit positional meaning or naming steps after their current count.

Avoid: patterns that hit a ceiling

Code languagejson

["spacing-small", "spacing-smaller", "spacing-new", "spacing-new-2"]

Prefer: a scale that can always grow in either direction

Code languagejson

["spacing.100", "spacing.200", "spacing.300", "spacing.420"]

Choosing your convention

A practical way to decide is to consider your team size, system scope, tooling, and existing patterns together.

Larger teams will need a stronger structure, single product systems can probably stay simpler, and tools like Figma, Tokens Studio, and your build pipeline may naturally favour one style.

It is also usually safer to evolve existing patterns than replace everything at once, and the best outcome comes when designers and developers both buy into the same model.

Start simple, document it clearly, add examples, and revisit as the system grows.

Token names do not need to be academically perfect. They need to be clear enough that your team can apply them without hesitation.

Keep naming alive with lightweight governance

Naming is not a one time decision.

It is an ongoing Design System practice.

If you want conventions to hold up over time, define lightweight governance:

One canonical naming guide with approved examples
A short review checklist for new tokens
Periodic cleanup of duplicate or ambiguous names
Shared ownership across design and engineering

The goal is not "process for process' sake".

A token system in a Design System that people trust is one they will actually use, and consistent naming is how that trust gets built.

Upgrading Forgejo with S3 Object Storage and Actions!

2026-05-03T10:50:05Z

Awhile ago I spun up my own instance of Forgejo. Forgejo, if you are not familiar, is a hard fork of Gitea when they became a for-profit company and is run/managed by Codeberg. If you want more details on why Forgejo came to be, you can read their blog post here. As of now it’s basically a drop-in replacement for Gitea. Though over time this will change as the two projects drift apart. So it may not be one in the future! Anyhow, I’ve been rolling my own instance since September 2023 and, recently I have been moving my entire lab towards a more GitOps approach. To do this and do it securely (in the confines of my internal network), I’ve decided to turn Forgejo into my full CI/CD manager and config storage place. I’ll also be using Bitwarden’s Secret Manager to manage, well, secrets.

Before I begin re-doing my lab for the umpteenth time, I need to change how Forgejo operates currently. That way Forgejo can support the transition from ‘FileOps’ (aka loose files) to GitOps. Now this isn’t the ‘perfect’ setup process as the repo storage still lives in /data, and it’s not currently replicated across my small swarm cluster. This will change when I get to actually rebuilding the cluster from the ground up to utilize NFS mounts for shared storage. Though that requires more hardware, which the wife probably won’t be happy about. 😅

Migrating to S3 Object Storage

Forgejo (in Docker), by default, has you map a /data directory where all the necessary configs/repos/logs/etc. live. As you can see by the example compose file here.

docker-compose.yml


    image: codeberg.org/forgejo/forgejo:7
      - ./forgejo:/data # <---- data volume
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

This is great if you plan on just running a simple setup, or you already have shared storage. For me though, I have no shared storage, and a MinIO server that is currently doing nothing but holding attachments for Outline. I figured it’s time to put it to more use. That, and my day job involves doing AWS things so figured it would be good practice.

Currently, Forgejo supports the following directories (under /appdata) for object storage.

Forgejo Docs: Storage Settings
Subsystem	Directory	app.ini Sections
Attachments	attachments/	[attachment]
LFS	lfs/	[lfs]
Avatars	avatars/	[avatar]
Repository avatars	repo-avatars/	[repo-avatar]
Repository archives	repo-archive/	[repo-archive]
Packages	packages/	[packages]
Actions logs	actions_log/	[storage.actions_log]
Actions Artifacts	actions_artifacts/	[actions.artifacts]

As you can see in the table above there are a few storage directories missing. Mainly repositories. Since repositories do not work well in object storage they are unsupported. Which means they will still be living on one of my hosts temporarily until I finalize my cross cluster storage solution. In order migrate the storage to MinIO, I first need to create a bucket (named forgejo), an Access Identity, and an ACL for it.

access-credentials

Access_key: O3n9bNhrbadkeyAGKOuhIUzMGF
Secret_key: czgQT1fb1l3GD2nRQEPebadsecretbWG0xGIu20hjOeBfoSGo

Access Key User Policy


    "s3:ListBucketMultipartUploads",
    "s3:AbortMultipartUpload",
    "s3:ListMultipartUploadParts",

The above ACL replaces the default one under the “Current User Policy” after enabling the “ON” switch for “Restrict beyond user policy” setting in User > Access Keys > Create access key. It limits the access key to the forgejo bucket. The bucket is also private by default, so I do not have to change the bucket ACL.

MinIO Create Access Key with a custom user policy

Next was to upload the existing files to MinIO so that when Forgejo restarted with the new s3 config it would pick them back up. As you can see in the picture below there are a few absent directories. I decided to condense a few related directories into parent ones for my ease of use. Also, some of them do not exist yet and will be created when needed (e.g. packages/).

Example MinIO bucket directory layout

My current s3 bucket directory layout:

With MinIO ready to go, all I had to do was update the Forgejo container environment variables for the MinIO connection and re-deploy. Forgejo environment variables can be defined using the env -> ini notation. So in your app.ini you have sections that are labeled, such as ‘[security].’ In order to update the INI value for say, ‘INSTALL_LOCK’, you’d need to define it as FORGEJO__security__INSTALL_LOCK.

As you can see in the example snippet file below, I am defining the MinIO connection information, as well as, where to look for specific directories in the bucket.

forgejo-minio-env-example

      - FORGEJO__storage__STORAGE_TYPE="minio"
      - FORGEJO__storage__MINIO_USE_SSL="true"
      - FORGEJO__storage__MINIO_ENDPOINT="s3.${LAB_DOMAIN}"
      - FORGEJO__storage__MINIO_ACCESS_KEY_ID=${FJ_S3_ACCESS_ID}
      - FORGEJO__storage__MINIO_SECRET_ACCESS_KEY=${FJ_S3_ACCESS_KEY}
      - FORGEJO__storage__MINIO_BUCKET="forgejo"
      - FORGEJO__storage__MINIO_LOCATION="us-east-1"
      - FORGEJO__attachment__MINIO_BASE_PATH="attachments/"
      - FORGEJO__lfs__MINIO_BASE_PATH="lfs/"
      - FORGEJO__avatar__MINIO_BASE_PATH="avatars/users/"
      - FORGEJO__repo-avatar__MINIO_BASE_PATH="avatars/repositories/"
      - FORGEJO__repo-archive__MINIO_BASE_PATH="archives/"
      - FORGEJO__packages__MINIO_BASE_PATH="packages/"
      - FORGEJO__storage.actions_log__MINIO_BASE_PATH="actions/logs/"
      - FORGEJO__actions.artifacts__MINIO_BASE_PATH="actions/artifacts/"

Forgejo Compose Example [Expand me]

forgejo.yml


      test: ["CMD", "curl", "-fSs", "localhost:3000/api/healthz"]
    image: codeberg.org/forgejo/forgejo:7.0
      - FORGEJO_APP_NAME="Forgejo"
      - FORGEJO_RUN_MODE="prod"
      - FORGEJO_WORK_PATH="/data/gitea"
      - FORGEJO__repository__ROOT="/data/git/repositories"
      - FORGEJO__repository.local__LOCAL_COPY_PATH="/data/gitea/tmp/local-repo"
      - FORGEJO__repository.pull-request__DEFAULT_MERGE_STYLE="merge"
      - FORGEJO__repository.signing__DEFAULT_TRUST_MODEL="committer"
      - FORGEJO__server__APP_DATA_PATH="/data/gitea"
      - FORGEJO__server__DOMAIN="git.${LAB_DOMAIN}"
      - FORGEJO__server__SSH_DOMAIN="git.${LAB_DOMAIN}"
      - FORGEJO__server__HTTP_PORT="3000"
      - FORGEJO__server__ROOT_URL="https://git.${LAB_DOMAIN}"
      - FORGEJO__server__DISABLE_SSH="false"
      - FORGEJO__server__SSH_PORT="22"
      - FORGEJO__server__SSH_LISTEN_PORT="22"
      - FORGEJO__server__LFS_START_SERVER="true"
      - FORGEJO__server__LFS_JWT_SECRET="${FJ_JWT_SECRET}"
      - FORGEJO__server__OFFLINE_MODE="false"
      - FORGEJO__database__DB_TYPE="postgres"
      - FORGEJO__database__HOST="${FJ_DB_HOST}"
      - FORGEJO__database__NAME="${FJ_DB}"
      - FORGEJO__database__USER="${FJ_DB_USER}"
      - FORGEJO__database__PASSWD="${FJ_DB_PASS}"
      - FORGEJO__database__LOG_SQL="false"
      - FORGEJO__database__SCHEMA=""
      - FORGEJO__database__SSL_MODE="disable"
      - FORGEJO__indexer__ISSUE_INDEXER_PATH="/data/gitea/indexers/issues.bleve"
      - FORGEJO__session__PROVIDER_CONFIG="/data/gitea/sessions"
      - FORGEJO__session__PROVIDER="file"
      - FORGEJO__session__COOKIE_SECURE="true"
      - FORGEJO__log__MODE="console"
      - FORGEJO__log__LEVEL="info"
      - FORGEJO__log__ROOT_PATH="/data/gitea/log"
      - FORGEJO__security__INSTALL_LOCK="true"
      - FORGEJO__security__SECRET_KEY="${FJ_SECRET_KEY}"
      - FORGEJO__security__REVERSE_PROXY_LIMIT="1"
      - FORGEJO__security__REVERSE_PROXY_TRUSTED_PROXIES="${CADDY_INT_IP}"
      - FORGEJO__security__INTERNAL_TOKEN="${FJ_INT_TOKEN}"
      - FORGEJO__security__PASSWORD_HASH_ALGO="${FJ_PASS_ALGO}"
      - FORGEJO__service__DISABLE_REGISTRATION="true"
      - FORGEJO__service__REQUIRE_SIGNIN_VIEW="false"
      - FORGEJO__service__REGISTER_EMAIL_CONFIRM="false"
      - FORGEJO__service__ENABLE_NOTIFY_MAIL="false"
      - FORGEJO__service__ALLOW_ONLY_EXTERNAL_REGISTRATION="false"
      - FORGEJO__service__ENABLE_CAPTCHA="true"
      - FORGEJO__service__CAPTCHA_TYPE="cfturnstile"
      - FORGEJO__service__REQUIRE_CAPTCHA_FOR_LOGIN="true"
      - FORGEJO__service__CF_TURNSTILE_SECRET="${FJ_CFT_SECRET}"
      - FORGEJO__service__CF_TURNSTILE_SITEKEY="${FJ_CFT_SITE_KEY}"
      - FORGEJO__service__DEFAULT_KEEP_EMAIL_PRIVATE="true"
      - FORGEJO__service__DEFAULT_ALLOW_CREATE_ORGANIZATION="true"
      - FORGEJO__service__DEFAULT_ENABLE_TIMETRACKING="true"
      - FORGEJO__service__NO_REPLY_ADDRESS="noreply@${FJ_DOMAIN}"
      - FORGEJO__mailier__ENABLED="false"
      - FORGEJO__openid__ENABLE_OPENID_SIGNIN="false"
      - FORGEJO__openid__ENABLE_OPENID_SIGNUP="false"
      - FROGEJO__cron.update_checker__ENABLED="false"
      - FORGEJO__oauth2__JWT_SECRET="${FJ_OAUTH_SEC}"
      - FORGEJO__actions__ENABLED="true"
      - FORGEJO__actions__DEFAULT_ACTIONS_URL="https://git.${LAB_DOMAIN}"
      - FORGEJO__actions__ARTIFACT_RETENTION_DAYS="90"
      - FORGEJO__storage__STORAGE_TYPE="minio"
      - FORGEJO__storage__MINIO_USE_SSL="true"
      - FORGEJO__storage__MINIO_ENDPOINT="s3.${LAB_DOMAIN}"
      - FORGEJO__storage__MINIO_ACCESS_KEY_ID=${FJ_S3_ACCESS_ID}
      - FORGEJO__storage__MINIO_SECRET_ACCESS_KEY=${FJ_S3_ACCESS_KEY}
      - FORGEJO__storage__MINIO_BUCKET="forgejo"
      - FORGEJO__storage__MINIO_LOCATION="us-east-1"
      - FORGEJO__attachment__MINIO_BASE_PATH="attachments/"
      - FORGEJO__lfs__MINIO_BASE_PATH="lfs/"
      - FORGEJO__avatar__MINIO_BASE_PATH="avatars/users/"
      - FORGEJO__repo-avatar__MINIO_BASE_PATH="avatars/repositories/"
      - FORGEJO__repo-archive__MINIO_BASE_PATH="archives/"
      - FORGEJO__packages__MINIO_BASE_PATH="packages/"
      - FORGEJO__storage.actions_log__MINIO_BASE_PATH="actions/logs/"
      - FORGEJO__actions.artifacts__MINIO_BASE_PATH="actions/artifacts/"
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

With the environment variables updated, and a quick redeploy of Forgejo, I was up and running with object storage via MinIO! A quick trip into the Site Administration > Configuration > Summary in Forgejo shows that LFS was now using MinIO as seen in the below image.

Forgejo site admin config showing LFS using MinIO

Setting up Forgejo Actions

All jokes aside, Forgejo Actions is actually fairly decent. The runner is not production ready (according to Forgejo themselves) but it works well enough that I am going all in on it for my lab. Setting up the action runner was fairly straight forward. It’s got a similar setup process to the self-hosted GitHub runner.

Enable Actions in the app.ini or using environment variables. Restart/Redeploy Forgejo.

See above for an example using environment variables under #Actions Configs.

Create a new runner in Site Administration > Actions > Runners > Create new runner.
Copy the new runner registration token.
Deploy Forgejo Runner and register it with the token from step 3.

Now there are a few ways you could deploy the runner, but I decided to deploy it in Docker. You can bind the runner container to the Docker sock, but I did not. I really do not like doing that so, I ended up using Docker in Docker (DIND). It’s not really recommended to use Docker this way but for my lab it will do fine. The compose file was also pretty easy to set up.

forgejo-actions.yml


    command: ['dockerd', '-H', 'tcp://0.0.0.0:2375', '--tls=false']
    image: code.forgejo.org/forgejo/runner:3.4.1
        condition: service_started
    command: '/bin/sh -c "while : ; do sleep 1 ; done ;"'
    # Change to this command after registration
    #command: '/bin/sh -c "sleep 5; forgejo-runner daemon"'
      - DOCKER_HOST=tcp://dind:2375

As you can see above there are two commands:. '/bin/sh -c "while : ; do sleep 1 ; done ;"' starts up the runner container and does nothing. This allows you to enter the container via docker exec and run the registration command using the token generated in the Site Administration settings. After registering the container, it can be brought down and the current command can be replaced with the commented out one ('/bin/sh -c "sleep 5; forgejo-runner daemon"'). Then after redeploying the container the runner will become available in Forgejo globally. Having a global runner means it can be used by all repositories. Which for me works fine since my Forgejo instance is mainly private except for a few public repositories and a few GitHub mirrors.

Forgejo Manage Runners

With my runner online and working, I am now able to have workflows kick off from steps defined in files located in .forgejo/workflows/. The syntax is fairly similar to GitHub actions so most of the actions you can run there can run in Forgejo. The best part is I can mirror the action repositories and have my runner pull them directly from my Forgejo instance. The first thing I set up was Renovate Bot to keep my Docker images updated. The bot runs on a cron based workflow as seen below and opens a PR for image updates.

.forgejo/workflows/renovate.yml


    - cron: '0 0/6 * * *'  # every 6 hours, every day
  RENOVATE_REPOSITORIES: ${{ github.repository }}
    if: ${{ secrets.RENOVATE_TOKEN != '' }}
      image: renovate/renovate:full
          GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
          RENOVATE_BASE_DIR: ${{ github.workspace }}/.tmp
          RENOVATE_ENDPOINT: ${{ github.server_url }}
          RENOVATE_REPOSITORY_CACHE: 'enabled'
          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
          RENOVATE_GIT_AUTHOR: 'Renovate Bot <renovate@mydomain.dev>'
          GIT_AUTHOR_NAME: 'Renovate'
          GIT_AUTHOR_EMAIL: 'renovate@mydomain.dev'
          GIT_COMMITTER_NAME: 'Renovate Bot'
          GIT_COMMITTER_EMAIL: 'renovate@mydomain.dev'
      - name: Get Ntfy Token from SM
        uses: actions/sm-action@v4
        access_token: ${{ secrets.BITWARDEN_LAB_KEY }}
          some-secret-id-string > NTFY_TOKEN
          some-secret-id-string > LAB_DOMAIN
        uses: actions/ntfy-action@master
          url: 'https://ntfy.${LAB_DOMAIN}'
          headers: '{"authorization": "bearer $NTFY_TOKEN"}'
          details: Renovate Bot failed to run!

The bot is configured via a renovate.json file located in the root of the repository.

renovate.json


  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base"],
  "assignees": ["alexandzors"],
  "dependencyDashboard": true,
  "labels": ["maintenance"],
  "enabledManagers": ["docker-compose"],
    "fileMatch": ["^.*\\.yml$"]
      "managers": ["docker-compose"],
      "depTypeList": ["services"],
      "datasources": ["docker"]

At the moment I have it scanning through all YAML files since I do not name my compose files docker-compose.yml. They are named as the service so: forge.yml, caddy.yml, authelia.yml, etc. They are also located in their own folders. This will be changing as I move more to GitOps but for now it’s fine.

PR Opened by Renovate Bot using the Forgejo Action workflow

Conclusion

This was a fun project to work on in between work and family stuff. Hopefully this inspires one of you to upgrade your own local git setup with an ‘Actions’ runner or object storage! I’ll have a full config dump for MinIO, Forgejo, Forgejo Actions, and an example workflow in the blog-files repo.

On to the next project. 🚀

Rebuilding My Home Lab

2026-05-03T10:46:07Z

_{FTC: Some links in this post are income earning affiliate links.}

It’s been a while since I last posted about my homelab or well posted on the blog. So I figured I’d start off with a banger for 2025. Rebuilding my homelab into a Docker Swarm cluster that supports high availability. Hope you’re ready for a ride, because this was definitely a journey. 😅

_{Side note: I also recently acquired a Bambu Lab A1 3D printer (with the AMS :)) to replace my old Anet A8. I’d say it’s mine, but I’ve only been printing stuff for my wife so far..}

The Plan

The plan for this project was to:

Create a cluster that supported high availability ✅
Setup a distributed storage solution that supported the above ✅
Setup a load balancer ✅
Have the cluster communicate over 2.5gb or faster networking ✅
Have a minimum of 3 nodes ✅
Be small and efficient ✅

Easy enough right?… right?

Since I had two Dell Optiplex 5050 Micros that I was using already, I decided to grab a third micro to complete the 3 node cluster. As for the actual orchestration software I decided to stick with Docker and use Docker Swarm. I’m a Docker guy but never had a chance to use Swarm properly with a shared storage system. Figured it was time to give it a proper go.

Wait… Docker Swarm? Alex you should use Kubernetes!!!!!!

Current v2 Cluster Specs

2x Dell Optiplex 5050 Micros
1x Dell Optiplex 7040 Micros
3x Realtek 8125B 2.5gb A+E NICs
3x Intel 700p 256GB SSDs
3x Crucial BX500 1TB SSDs
Ubiquiti Flex-2.5g Switch
Ubuntu 24.04 LTS
GlusterFS

Cluster v1

_{circa 2023}

Cluster v1 never actually made it to ‘production’. It ended up being a test bed and a learning experience on gotchas when it came to Linux, Intel, drivers, and poor research on my part…

TLDR: I originally bought some Intel i225-v 2.5gb m.2 cards from IOCrest thinking they’d work. Come to find out these particular chips have some driver issues on Linux (specifically Ubuntu).

When originally designing the cluster I had planned on using M.2 2.5g cards since the Micro’s had M.2 M-key slots. That way I could get a less expensive SATA SSD for storage. After doing some research I settled on some Intel 225-V 2.5g cards from IOCrest on AliExpress. The one downside to these cards was the vertical connector pins. The chassis did not have enough space with a SATA SSD installed. So I did what any tinkerer would do and replaced the pins with right angle ones. The first two cards went good, but my old soldering iron doesnt keep temp well. On the third card’s NIC module I messed up and destroyed one of the traces. So I had to fix it with a jumper wire. Live and learn I suppose.

As for the NICs themselves, I started experiencing driver issues after getting two of the nodes up and running in a ‘mock’ swarm. These issues manifested as random disconnects or complete NIC power loss. I tried to fix it by messing with grub configs for pcie_aspm thinking it was related to Active State Power Management (ASPM) at first. I then tried manually setting the autonegotiation via ethtool. Also double checked UEFI settings, etc. Nothing seemed to work. So I ended up scrapping the Intel cards and going with Realtek 8125B cards for cluster v2.

Anyways, I’ll leave you with a few pictures from the v1 prototype, and we can move on to v2.

Cluster v2

_{circa 2024}

My second attempt at the cluster faired a bit better. Well mostly (more on that later). After my first purchase mistake I went back and bought different M.2 adapters. These were A+E keyed and used the WLAN slot rather than the M.2 M-keyed slot which got freed up for more storage. These new adapters also use a Realtek chipset (8125B) rather than an Intel one. So no more odd driver issues.

M.2 A+E keyed adapter installed

With the new adapters using the A+E slot, I decided to use the M slot for a boot drive and the SATA port for cluster storage. For the boot drive I went with some Intel 700p 256GB SSDs I bought off someone from the Ubiquiti Discord server. As for cluster storage I went with 1TB Crucial BX500 SSDs from Amazon. These are not the fastest drives by any means but for my use case they should do just fine.

Storage SSDs

There was one downside to the Intel SSDs though. Their physical size. They are 2230 and the Micro chassis only supports 2260 and 2280. So in order to use them I had to print out some 2230 to 2280 adapter brackets. I also printed out a 2.5in tray for the third Micro since it did not come with one. The print files are linked below.

The OS

For the OS I decided to stick with my tried and true Ubuntu Server (24.04 LTS btw). It’s stable and easy to manage. Though, I may eventually re-image to Rocky Linux for easy live patching when I rebuild again. The process was pretty standard. I booted via my netboot.xyz PXE server, ran through the installer and then used Ansible to configure the OS. I’ll have the playbooks + task files in the blog-files repo after this post is live.

Your browser does not support webm.Ansible playbook running on the cluster nodes

Networking

After the OS was set up on all the nodes, I went ahead configured the network interfaces. During the v2 build Ubiquiti launched their Flex 2.5g switch line. Since this was an SFF cluster the Flex Mini 2.5g was a perfect fit as the cluster’s network backbone. Also with it being powered by POE made for one less cable to deal with. The cluster network is also a fairly simple one. It has no access to the internet and no access to any other networks. Other networks also can’t access it. Don’t need stuff snooping on my unencrypted SQL traffic..

With the network interfaces configured and working, it was time to test the speed using iperf. As you can see by the short video below, I was able to get roughly 2.5gbps of bandwidth between the nodes.

Your browser does not support webm.iperf testing

Distributed File System

This was probably the most difficult part of the cluster build. At the time I had really no experience dealing with distributed file systems. I narrowed down my options to GlusterFS and Ceph. I ended up going with GlusterFS for this iteration. Using the official documentation I was able to get a basic cluster up and running fairly quickly. I also created a dedicated Docker user ‘doc’ and set its home directory to the GlusterFS volume on each of the nodes. I thought this would be a great idea, but it ended up being a pain later on.

GlusterFS setup

Docker Swarm

With the OS, Networking, and DFS setup, it was time to set up Docker Swarm. Setting up Docker and Docker Swarm is fairly straight forward using the official documentation. You basically follow the installation on each of the nodes. Then you initialize the swarm on one and join the other nodes to it. In my case my commands were:

Terminal window


sudo docker swarm init --advertise-addr 192.168.100.2 ## init the swarm on the cluster network interface
sudo docker swarm join --token SWMTKN-1-4c2d6gncjh15gaznem5m8t2twwfn09o0paqpjxwdkqr1n76h8j-chr42vcbaobx7v2nrk86q9e93 192.168.100.2:2377 ## joining other nodes to the swarm

I then promoted the other two nodes to managers. This way I had proper high availability as it requires a minimum of 3 managers to be in the swarm.

Docker Swarm cluster

Keepalived

Next up was setting up Keepalived to handle VRRP for the cluster. This allowed me to have a single IP address that automatically failed over to the next node automatically. The video below shows the failover in action (sped up to keep the video short).

Your browser does not support webm.Keepalived VRRP failover

As you can see the active node switches as each keepalived service is terminated. The lower right shows a constant ping to the VRRP virtual IP (VIP). In this case 10.8.8.11. Having this VIP allows me to bring down hosts for maintenance or updates without having to worry about service connectivity.

HA Caddy

The final step before I could start using the cluster was to set up a load balancer/reverse proxy. There are a few different options but, if you’ve read any of my previous posts you know I love using Caddy. With its easy-to-use configuration and automatic certificate management makes it hard to beat. Also, Caddy is a great choice for this since it automatically clusters itself if all instances use the same storage volume.

First thing I did was create the new overlay network for Caddy. This network would be for any services that I wanted to be able to access from outside the cluster.

Terminal window


sudo docker network create --opt encrypted=true --driver overlay --attachable --internal --subnet=172.0.96.0/20 caddy-internal

Creating the Caddy service file.

caddy.yml


    image: alexandzors/caddy:2.9.1
      - /swarm/volumes/doc/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
      - /swarm/volumes/doc/caddy/.data:/data
      - /swarm/volumes/doc/caddy/configs:/etc/caddy/configs:ro

Then deploying Caddy to the swarm.

Terminal window


sudo docker stack deploy --c caddy.yml caddy

docker service ls command showing Caddy running on all 3 nodes

I eventually need to update the Caddy deployment to use host mode for port assignments. That way I can get the source IP address of incoming requests and not the docker proxy IP. But that will be part of v3.

Cluster v3

_{circa 2025}

So here we are. Cluster v2 has been running great for a while now, but I’ve been having some issues with it. The main, and arguably the largest, issue I have been running into is the slow performance of GlusterFS. I’m not sure if it’s the hardware choices I made, or a misconfiguration? I tried the performance tuning tips from the official documentation but was still running into issues. Like switching to the cluster user ‘doc’ would take anywhere from 30 to 45 seconds to complete. Also, deploying stacks of services could take up to 5 minutes before they were alive and running.

Either way for v3 I’m probably going to move to Ceph as my DFS solution. Since it seems to be the more popular of the two after digging around more. I may need to upgrade the RAM on the nodes to handle it though. I’m also going to try and add a 4th node to the cluster with beefier hardware since I still have one 2.5g port left on the switch. Maybe one with some GPU compute to handle local LLMs?

Current services running on the cluster

However, what won’t be changing is my use of Docker Swarm. 😉

How to smoke

2026-05-02T05:49:10Z

The best part of smoking was always the click of the Zippo.

This week’s question comes to us from Mat Honan:

I used to love to smoke. If it weren’t for the whole lung cancer, emphysema, death thing, would you recommend smoking?

You’re forgetting the smell.

Last week I was coming back from the record store and feeling lazy, so I jumped on the bus. A couple of stops after I got on, a dude got on and sat down next to me. Well-dressed dude, also carrying an Amoeba Records shopping bag. Out and about on a Sunday afternoon, doing his record shopping just like me. Within seconds it became clear this dude had just had a cigarette. And he stunk. The bus was also packed at this point, so I was pretty much stuck in that seat until I got off, which luckily wasn’t for too much longer. Also, it feels kinda shitty when you sit down next to somebody on the bus and they immediately get up. (Unless you’re getting a creeper vibe, of course.) I just had to suck it up for a few more stops. But man, it was rough. I’m not trying to disparage the guy or anything. Especially because I used to be that guy, and you used to be that guy, and I’m guessing a lot of our readers used to be that guy. We just walked around smelling awful.

Which is not to say that’s the worst part of smoking, but you took lung cancer, emphysema, and the dead thing off the table. Which leaves us with the smell.

I started smoking my freshman year in college. All reasons to take up smoking are stupid, but this one might be the stupidest. This being the mid-80s our dorm had a cigarette machine in the lobby. One of those old machines you might still see in old man dive bars, with two rows of big ka-chunky knobs, which felt objectively good to pull. It was like you could feel the entire mechanism of the machine come to life when you pulled that knob. And it took some strength to do it! But you could feel the knob hit a gear, you could hear the gear spin, you could feel a pack of cigarettes get pushed free from deep inside the machine. You could hear it slide down a little ramp, and then you’d see it appear down in the landing zone, where you’d push your hand past the trap door, grab it, and somehow have the pack open, and a cigarette in your mouth before the rest of the pack hit your pocket.

If memory serves, a pack of smokes was between $1.25 and $1.50 around that time. (Fun fact, I attempted to Google this and the slop top on the search results page said 26¢ a pack, which is… not true. But please, continue to rebuild society around such amazing technology.) Being the art school miscreants that we were, and also broke, we discovered that if you pulled a knob halfway out, inserted a quarter, and then pulled the knob the rest of the way out it would release a pack of cigarettes. (Ok, that may not have been the actual process, but it was a long time ago, and it’s very close to the spirit of the actual process, so let’s run with it. So I guess we were getting a pack of cigarettes for what Google’s stupid slop robot said, but it included doing crimes.) The knowledge of how to hack the cigarette machine spread through the dorms like wildfire, and soon we were all smoking. Because we were idiots. Also, it felt like we were getting one over on The Man. But mostly because we were idiots.

Also, being art school kids we were very visually-driven people, and all the photos of cool people that we’d hang up in our dorm rooms showed them holding a cigarette. And we very much wanted to be cool people. (True fact: take a photo of Humphrey Bogart, replace the ever-present cigarette with a vape and Humphrey Bogart will look like an herb.) Again, mostly we were idiots.

The vending machine company tried to patch the hack several times, eventually gave up and just took the machine away. This was our first lesson that sometimes doing crime is in the public interest, but that lesson didn’t occur to us right away. At the time, we were just pissed that we had to pay retail for cigarettes again.

By the time I started smoking society was pretty much done with the pretense that smoking was doing anything but murdering you slowly. I know this because we’d sit around in art classes making collages using old cigarette ads where doctors would tell you smoking was good for your nerves, and we would laugh at people for believing this, as we lit cigarette after cigarette. (Yes, you could smoke in class.) And we thought “Boy, our grandparents sure were chumps for believing cigarettes were healthy.” Then we would have a coughing fit. But there was definitely the sense that doing this thing that we all knew had a very very high probability of killing us wasn’t a big deal, mostly because we were in our 20s when nothing can hurt you, Reagan was president and, just to reiterate—we were idiots.

My first post-college job was at a copy shop, and you got one 15 minute break during your shift. Unless you smoked, then you could get as many breaks as you needed. Several people started smoking while working there.

We all stunk. We’d come in from smoking out back, and immediately walk up to the service counter to help a customer. A customer who either stunk as bad as we did, or had become inured to the stench because it was all around them, emanating from everyone.

We smoked in class. We smoked at the movies. We smoked at the supermarket. We smoked at sporting events. My friend Jeff, who grew up in Boston, tells a good story about going to Celtics games as a kid and having to look past the hovering cloud of smoke between the cheap seats and the court. We smoked in restaurants, where the smoking and non-smoking sections were often divided by nothing more than a paper sign denoting the territorial boundary. We smoked on planes, man.

It wasn’t too long after college that the world began to shift. In 2003 New York City banned smoking in bars. And I was visiting at the time. By 2003, I was no longer “a smoker” but I was very much someone who would look for reasons to bum a smoke from someone if the situation arose, and very likely to put myself in situations where it might. But I remember the rage from several friends and from the owners and bartenders of any bar we’d walk into. The ban was going to kill bars all over the city. It was going to kill nightlife. It was going to completely take down the economy. New York, as we know it, would cease to exist. Which of course, it didn’t. Everyone adjusted. They went outside. They eventually started smoking less because it was cold outside. People enjoyed being able to hang out in rooms that weren’t making them sick, and they enjoyed going home not reeking of cigarette smoke. If I could go back in time I’d reassure all those bartenders that it wasn’t the smoking ban they had to worry about. It was the kids who’d stop going out at all because they needed to sit at home and tend to their AI agents.

I selected your question this week because I’ve actually been thinking of the smoking ban lately. We grew up in a time when smoking, or dealing with other smokers was an inevitability. Even if you didn’t smoke, you’d most likely work next to someone who did, or sit down next to someone who did at a restaurant, or at the movies. And even if they weren’t actively smoking, and covering you in second-hand smoke, you’d go home with the stench of smoke all over you. Airing your clothes out was an inevitability. Having to wash the stench out of your hair was an inevitability. Society smoked, so you did too. Whether you wanted to or not. And then it changed. Most cities in America now have smoking bans and rules about how far away you have to be from a public entrance to smoke, which get enforced to various degrees.

The change came in a couple of very interesting ways. One, cigarettes are now hovering between $12 to $14 a pack. (I had to look this up!) They’re also available in less places. (You used to be able to buy cigarettes at the drug store!) Secondly, people just look at you weird if you start smoking now. Like, what the fuck dude, did you just light a cigarette!? Are you from the past? Gross.

Which of course makes me think of some of the things that we have currently accepted as a society, things which we fully know are not healthy for society, that we are currently tolerating. And also thinking there’s no way it will ever change, because we appear to be in an era of “what if everyone modeled themselves off the stupidest people?”

Right now there is someone firing up ChatGPT because it’s cheap. Right now there is someone writing a prompt in Claude because it brings him closer to his co-workers. Right now there is someone walking a co-worker through his agentic workflow, in the same way we attempted to impress one another by blowing smoke rings. Right now there is someone parking a Cybertruck on your street, believing that leaving his divorce where everyone can see it is somehow impressive. We have always been good at ignoring the warnings that came with the pack.

Our parents packed their homes with asbestos. They heated their homes with coal. They packed their Big Macs in styrofoam. Making mistakes will always be cheaper than fixing them. But nothing is more expensive than ignoring them.

Cultural norms are an ever-changing thing. History is the story of what was once desirable becoming unacceptable. Something that used to be an inevitability is no longer inevitable. Something that used to be tolerated is no longer tolerated. Something that was seen as a cultural norm no longer is. Even when those things were backed by entire industries with very strong lobbies, as the tobacco lobby once was. The same fate will someday befall the NRA. The same fate will someday befall AIPAC. The same fate will someday befall the slop lobby.

There was a time we thought if we prohibited people from smoking in bars it would lead to societal collapse. I think it was a good idea. More importantly, it was an idea that worked. It improved not just our personal health but the health of our communities.

The basic strategy of all addictive technologies is very simple. They make you feel extra capable, they addict you, then they make you feel inadequate without them. They start by making you feel cool, and confident. Relax. Put your feet up. Hang with the fellas. Social anxiety? It’s toasted, dog! Let me write that résumé for you. Anniversary card for your wife? I can write that for you. Light one up. It’s a great way to start a relationship. But that initial boost eventually turns to reliance, and suddenly you can’t get out of bed without a hit. You can’t write your kid a love note without firing up a slop engine. And suddenly an entire industry is telling you that you’re not capable of moving through your day without their help. An entire industry gaslighting you, until it becomes easier to just gaslight yourself into believing that you were never truly capable of things you are very much capable of.

I don’t miss smoking. Maybe I did at one point. But eventually the whiff of cigarette smoke went from smelling nostalgic to just smelling bad. Thankfully, knock on wood, I’ve been able to escape years of smoking without any major lasting effects, but trust that I carry every pack I ever smoked with me every time I walk up a flight of stairs. If I’m doing anything strenuous, it’s always my lungs that give up first.

Thankfully, I still have some lung capacity. I enjoy using it. You should too.

🚬

🙋 Got a question? Ask it! I will somewhat answer it!

📓 Get your sexy copy of my new book How to Die (and other stories)! And if you’re in the Bay Area, come see me and Annalee Newitz talk about it on May 11 at Booksmith!

🚢 My friend Lucy Bellwood made a wonderful comic about loss and building.

🙅 My friend Jason Cosper made a great tool for fucking with Substack’s dollar.

📣 I’ve got a few seats left in next week’s Presenting w/Confidence workshop where you can learn true confidence. The one inside you.

🍉 Please support the children of Palestine.

🏳️‍⚧️ Please support trans kids. And if there is a trans person in your life please tell them you love them.

Crashing hard: why talking about bubbles obscures the real social cost of overinvesting into “Artificial Intelligence”

2026-04-27T08:11:01Z

More and more commentators talk about and warn of an “AI bubble”, and everybody seems to congratulate each other on being such a smart financial analyst. BUT: A bubble pops and you are left with air and maybe a splash of soap somewhere on the floor. A fairly clean affair. This kind of investor speak obscures the severe consequences economic crashes cause, coming from someone’s point of view for whom this is more likely to be a spectacle than a direct threat.

When the “AI” market crashes, there will be NO “reset button”, NO “rollercoaster” continuing on an orderly path after having come down, NO “bubble” that just lets off hot air. These are all metaphors that heavily misrepresent what it means for markets to crash, or, as they say, “correct”. We might be in for a long and painful struggle to at least reduce the grip of “AI” on current core societal functions like government administration, education and research funding. In this article, I want to illustrate the broad range of costs that BOTH the buildup of “AI” overvaluations AND their coming down will have. The current “AI” investments will have long-term costs by creating significant path dependencies: They make harmful things cheaper, speed up the commodification of human labour and shift social norms. Just to be clear: I am referring to the current “AI” boom which is driven mostly by generative AI (“genAI”) applications, not necessarily the things that have been around for decades (e.g. various forms of pattern recognition) and that did not induce companies to spend hundreds of billions on data centres.

To better understand what is going on, let’s first look at the outcomes of previous instances of overinvestment, including the 2000 dot-com “bubble” and the significant piles of money Uber burnt for many years, before turning to contemporary “AI” path dependencies.

Overinvestments shape technological paths

Let’s start with the obvious: The so-called dot-com boom crashed between 2000 and 2002 – this already hints at the fact that “bubble bursting” is a long period during which no one knows when it will end. When it did end, investors lost money, and it is mostly their perspective that was covered in the media (and they whined about a crash being less bad than the pain inflicted by missing out on a boom). Many people lost their jobs, their livelihoods, and needed to find other ways to make ends meet (developers on Reddit gave an account, but they were probably among the more privileged). Unemployment in the US increased from about 4% to almost 6%.
What might be a little less obvious: A few key developments that the dot-com boom had started persisted long after. While the internet was still a mostly academic affair until the 1990s, the dot-com boom kicked off the scale-over-everything, ad-based internet we know today. We saw alignment of advertising business and finance as well as a massive drive to consolidation during the crash. Google, eBay, Amazon, Nvidia, they all became central players in the commercial internet. What now seems inevitable to most people seemed coincidental before the boom – but then today’s driving forces crowded out most other, less commercial forms of existing on the internet.

“AI” investments make harmful things cheaper, speed up the commodification of human labour and shift social norms.

The investment logic has shaped the internet ever since: Uber accumulated almost $34bn USD in losses (excluding losses while it was not yet public between 2009 and 2014) before it started to generate profits in 2023. (And also various other unicorns incur massive losses they may never recoup.) Money also creates habits and legitimises actions: Uber “broke laws, duped police and secretly lobbied governments”, as the Guardian titled in 2022 and its controversies got their own Wikipedia page. But also their very official business model is based on a) normalising precarious working conditions by eroding labour protections as they fought countless lawsuits over giving their workers only freelance and not an employee status, thereby avoiding sick pay, holidays etc., and b) making human work seem more automated and less visible by mediating passengers and drivers through an algorithm in an app, reducing the need for actual human interaction. Both developments shift social norms in ways that are likely to be profitable for businesses even beyond Uber: Uber’s mission can be understood as reducing the value of workers and human interaction, and with that long-term goal it is commercially rational to rack up significant losses in the short to medium term.

The “AI” path dependencies we will not correct

It is plausible to expect something similar to happen with “AI”. The ongoing investment boom is creating significant overcapacity with effects lasting long after many “AI” startups have gone bust. These range from very visible and direct to the more indirect and structural.

Lower costs for compute and energy: In order to sustain the boom, investments follow projections of endless “AI” growth, which translate into hundreds of billions currently being invested into data centre construction, alongside an expansion of energy infrastructure. And once they are built, it does not make sense to stop them, does it? Keeping them running is much cheaper than constructing them. This puts us on a path of energy-intensive technology even once these data centres are no longer needed for “AI” applications. This eradicates any incentive for resource-efficient coding or low-computation technology. At the same time, this infrastructure is not costless to maintain (to my knowledge, chips need to be replaced about every 7 years) – but possibly that cost is still lower than doing anything else, hence continuing on that trajectory will remain cheaper than alternatives for quite some time to come. These artificially low costs of compute and energy will be even further away from the “real” costs when factoring in just the environmental harms they produce. That is very bad news for anybody still hoping for a combined digital and environmental transition.

Sectoral knowledge destruction: Some people may get used to using “AI” for a variety of tasks, even where this is neither actually helpful nor profitable for the “AI” providers. As is widely reported, managers often encourage or force their employees to e.g. code using genAI applications, university students use genAI applications for writing, public bodies are continuing to move onto fancy “AI” clouds and forget how to do on-premise computing, and we are likely to see more diffusion before the boom crashes. Just as Uber’s mission was broader than just individual transport, “AI” has an inbuilt contempt for human interaction (as it is built to automate speech while avoiding any interpersonal friction) and workers (as it seeks to make them even more interchangeable and subordinate to machine processes). Hence, using “AI” often destroys established processes of developing skills and sharing knowledge. Rebuilding them will take much longer and possibly cost more than might have been saved in the meantime.

Again more economic inequality: An economic crisis is not equally dangerous for everyone. Only few companies are benefitting from the “AI” boom and most of the stock market gains in recent years were driven by Big Tech valuations going absolutely through the roof. However, we can expect that the losses will be shared more widely, based on the experience of past financial crises. Big Tech is trying to portray itself as too big to fail, which means that their systemic relevance would prompt governments to inject tax money to reduce any losses they might incur. A financial crisis has ripple effects that go far beyond the market in question – just as the 2008 US housing crisis did not only lead to people losing their homes, but caused a huge recession with a stark increase in unemployment and financial instability.

Why “AI” overinvestments might take a long time to unwind

There is no way of knowing when the “AI” boom is likely to crash – that is the whole point of markets that are supposed to create collective rationality from individual choices. I see a few reasons to suspect an even longer and more painful struggle than the dot-com crash in 2000. First, the boom is orchestrated or arguably planned by a handful of extremely powerful companies. Being few increases the scope to act strategically. Second, these companies are very close not only to the US government, but through their start-up investments also to governments across the world, selling “AI” promises and lies to politicians. The push of small and large “AI” firms into military tech aggravates this dynamic: It is the area in which talking of an “AI race” carries quite intuitive meaning because having more destructive power translates into military power, though not necessarily into better societal outcomes. And third, the intention of reaching systematic relevance is bearing some fruit as more and more institutions are becoming financially invested into “AI success”. Not only VC investors, but large parts of society including life insurance and pension funds will bear the cost of its failure, giving them an incentive to prolong the boom at fairly high costs.

There are a few reasons to suspect an even longer and more painful struggle than the dot-com crash: market concentration, closeness to governments, and financial actors being invested into “AI success”.

What to do and why not to despair

It is important to analyse the abyss, but don’t stare into the abyss, as Jathan Sadowski sometimes says on my currently favourite podcast This Machine Kills. Understanding what is happening is essential to figure out a plan and I am keen to do that with others (i.e. I am aware my suggestions are insufficient). Anything that contributes to not making the boom bigger than it needs to be is helpful (e.g. do not invest into “AI”, tell your friends not to, do not use genAI applications or pay for them). Anything that helps us to talk in less delusional terms about what is going on is helpful (e.g. do not join those talking about “bubbles” suggesting they are merely financial events or that have one moment of coming down after which everything will be okay again). And let’s try to preserve that knowledge that companies are keen to replace with “AI”. We will need it.

Photo by Maxim Hopman on Unsplash

Wrap Text Around Images with CSS shape-outside

2026-04-25T17:37:09Z

Make your text wrap around images perfectly.

By default, text wraps around a boring rectangle. But what if you could make it hug the actual shape of the image?

You can. With just one CSS property:

shape-outside: url(your-img.png);

Add a float and a bit of margin with shape-margin, and your layout feels instantly more refined.

No JavaScript. No layout hacks. Just native CSS support and it’s supported in over 95% of browsers.

Use it with transparent PNGs, SVGs, or even basic shapes like circle() or polygon(). Ideal for editorial layouts, landing pages, or any design that needs more personality.

Checkout the codepen: https://codepen.io/theosoti/pen/ogjjged

shape-outside can produce elegant editorial layouts when image silhouettes stay simple. Test long paragraphs and varied image ratios, because float-based wrapping can become fragile in edge cases.

shape-outside can create editorial layouts with strong flow, but test with varied image sizes and long text. Float-based wrapping can break faster than block layouts.

Use this as a readability tool, not only a visual effect. The best result is when style improves scanning speed without adding cognitive load.

To roll this out safely, start by applying shape-outside: url(your-img.png); in a single UI surface where the benefit is obvious. Then reuse that same pattern in similar contexts so behavior stays consistent and review time stays low.

Before shipping, test shape-outside: url(your-img.png); with both short and long content, then verify behavior in narrow and wide containers.

If you liked this tip, you might enjoy the book, which is packed with similar insights to help you build better websites without relying on JavaScript.

Go check it out https://theosoti.com/you-dont-need-js/ and enjoy 20% OFF for a limited time!

Why The Split? - MeshCore Blog

2026-04-24T04:36:09Z

Since inception, the MeshCore development team have been working hard to build MeshCore.

We’ve released more than 85 versions of the MeshCore Companion, Repeater and Room Server firmwares with support for more than 75 hardware variants. All of this has been hand crafted, by humans.

We have always been wary of AI generated code, but felt everyone is free to do what they want and experiment, etc. But, one of our own, Andy Kirby, decided to branch out and extensively use Claude Code, and has decided to aggressively take over all of the components of the MeshCore ecosystem: standalone devices, mobile app, web flasher and web config tools.

And, he’s kept that small detail a secret - that it’s all majority vibe coded.

We ran a poll recently, and asked in the MeshCore Discord about AI and trust, and these are the results:

The team didn’t feel it was our place to protest, until we recently discovered that Andy applied for the MeshCore Trademark (on the 29th March, according to filings) and didn’t tell any of us. We have tried discussing this, and what his intentions are, but those broke down and we now have no communication with Andy.

It’s been a stressful few months trying to sort this out, and is now a sad day to bring this out to the public. It’s been a slap in the face to the team that have worked so hard on this project, to have an insider team up with a robot and a lawyer.

“Official” MeshCore

The use of the ‘official’ status is what is currently being contested. Andy is adamant that he owns the brand, and is using the word very heavily with his MeshOS line.

Meanwhile, in reality, the only ‘official’ MeshCore is the github repo. It’s the source of truth in terms of what is MeshCore, and Andy has never contributed to that.

Since the internal split, we launched the meshcore.io site, as Andy controls the meshcore.co.uk site and original discord server. We’ve been left with little other recourse. And, since launching the site, Andy copied the look and feel (again, using Claude) even though we asked him not to.

Project Growth

The MeshCore project has been on an incredible journey.

Having only started in January 2025, we have grown extremely fast!

As of this post, the official MeshCore Map shows 38,000+ nodes around the world, and the official MeshCore App has more than 100,000+ active users across Android and iOS.

It’s pretty epic how we’ve all built such an incredible community in such as a short time!

As the project grows, so does our need for a dedicated space that provides you with official information from the core team.

In recent times, we’ve seen an explosion of growth in MeshCore web sites dedicated to specific countries and mesh communities.

To name a few, we’ve seen:

Andy Kirby did do an amazing job helping to promote the MeshCore project on his personal YouTube, but only promotes his own products now.

Where To From Here?

So, the core team are pushing ahead with the meshcore.io website, the ongoing work of firmware feature development, bug fixes, managing PR’s and developer discussions, etc.

We now release change logs, blog posts and technical documentation for all of our new firmware and app releases here.

You’ll also find some familiar faces on our blog posts, such as:

Scott our project founder, lead firmware engineer and developer of the Ripple firmware!
Recrof our official MeshCore Map developer and Firmware Flasher guru. He has shared some insights into the early development of the MeshCore Map.
Liam Cottle the official MeshCore App developer who will be posting useful guides for getting started with the MeshCore App.
FDLamotte who has done epic work on the Python tooling for MeshCore, as well as the STM32 firmware variants.
Oltaco (Che Aporeps) who has done amazing work on the new OTA Fix bootloader that makes firmware updates much more reliable.

The Core Team

The MeshCore team, now consisting of Scott, Liam, Recrof, FDLamotte and now Oltaco remain committed to designing and developing high quality, human-written software.

Our New Home

Please update your bookmarks!

This is where we will be hosting all official releases, technical documentation, and community discussions moving forward.

With the new website, we are also starting fresh with a new Discord server!

This is where you can interact directly with the MeshCore developers, get help with your projects, and contribute to the future of MeshCore.

Thanks for being a part of this journey!

The MeshCore Team

How to Stop A Data Center in Your Backyard ~ L.A. TACO

2026-04-23T13:10:47Z

When the people of Monterey Park found that their local government was going to approve a 250,000-square-foot data center just 500 feet from their homes, they organized.

And within a few months, the developer withdrew their application.

Andrew Yip, an organizer with SGV Progressive Action, tells L.A. TACO that the organization’s success started with their “existing network of volunteers,” noting that “the community was able to jump in at a moment's notice.”

SGV Progressive Action was founded in 2020 to “address the Black Lives Matter uprisings," Yip says. "To support our Black community."

Then it organized local resolutions advocating for a ceasefire in Palestine, and built a lending library in El Monte called Matilija Collective, where they trained volunteers in community defense against ICE, hosted organizers, and stored 20 canopies and a speaker system.

"So that existed," Yip says.

In November, a community member who had come to a council meeting for other business saw the data center on the agenda and called on SGV Progressive Action.

"They asked if we can take a look at this," Yip says. "And see if that's something that communities should be concerned about."

All that was needed was one last council vote. But the developer requested a delay to the next meeting.

"Had they voted that day, it would have been done, right? It would have been done," Yip says. “But we found out about it, and we turned out hundreds of people to the next meeting.”

CA PUBLIC RECORDS ACT

Under California's Sunshine Laws, local governments are required to turn over agendas, meeting minutes, attendance records, and emails. SGV Progressive Action immediately filed public records requests.

"That's really how we found out," Yip says.

The records showed city planners had given their blessings to the data center, saying it would not result in any “significant environmental impacts.” They had used the developer's own impact assessment in place of a more thorough state environmental review.

The records also showed the city had held a series of community meetings to ask residents what should be built at 1977 Saturn Street, but only notified people living within 500 feet. Each meeting drew between 20 and 60 people. Residents who were there told Yip and other organizers that the city clerk brought in people who backed the data center. It won with roughly 20 votes.

“Twenty-something votes determined [that] residents here wanted a data center,” Yip says. ”That just seemed like a weird recommendation coming out of a community town hall.”

Council Member Thomas Wong, who would vote for having the data center, also works at the power company that would sell its electricity.

The developer also bought a larger property at 1980 Saturn Street across the street. The data center trade magazines reported that these were part of a 13-parcel assembly, all meant for data centers. Yip asked the developers what they planned to do with 1980 Saturn; they said they were not authorized to discuss it.

NO DATA CENTER MPK & THE INFORMATION CAMPAIGN

The residents of Monterey Park bought the domain No Data Center MPK.

“And we had a ton of people come out to support, whether it's walking the neighborhoods, distributing fliers, calling folks, creating artwork,” Yip says. “It was a big showing.”

They went door-to-door to tell their neighbors what the city had not told them: The data center would use twice as much electricity as all of Monterey Park. The 14 “backup” generators would burn 200,000 gallons of diesel every year, without a blackout. And more when the grid price was high.

They held a teach-in. 150 people came.

“We have a lot of very smart residents who were able to do a lot of this research and fact-finding. Many of the residents we work with are researchers or hold PhDs, and they work in universities. So they know how to find this information,” Yip says.

One resident 3D-printed a model of the data center to show just how much of the neighborhood’s space it would take up. Another resident mixed noise recordings from data centers and played them over a loudspeaker. You couldn't hear the birds.

Two dozen people in Virginia who lived near a data center were ready to fly out to testify on their own dime.

“Virginia became ground zero for data center proliferation,” says Yip. “The people didn't know enough about data centers at the time.”

The vibrations and noise never stop, the people from Virginia warned. The reported sound levels of 60db and higher are far from the data center. They have taken to sleeping in their basements. Neither a decibel meter nor the law measures vibration.

Yip and the organizers found that almost nobody in Monterey Park that they spoke to had heard of the data center. The city’s notification had only reached 40 people living within 500 feet of its proposed location, in English.

The neighborhood is 65 percent Asian and 27 percent Latino. The community’s outreach was done in at least three languages, five when necessary. It extended to all the surrounding neighborhoods.

“Data centers, their pollution, and their effects don't just stop at the border,” Yip says.

They started a petition in English, Chinese, and Spanish. It grew to 4,500 signatures.

THE DEVELOPERS & THE DISINFORMATION CAMPAIGN

The developers retained a law firm with 1,000 attorneys on four continents. They hired Actum, the lobbying firm that represents Amazon and Clorox. Actum lists Trump’s former chief of staff as a partner and another who exploited a loophole so large that California had to legislate to close it.

"The applicant sent a letter to the city council talking about misinformation being spread, and that was exactly what the council members said," Yip tells us. "They just parroted the same talking points."

The political lobbyists canvassed neighborhoods and local shops.

"One of the public benefits of the project they were touting was a pocket park ... The pocket park is basically just leftover land on their property that they didn't really need for the data center," Yip says.

They promised 200 jobs and, in the same conversation, no traffic.

“Our people would poke holes in it,” Yip says. “Why wouldn't there be cars in that facility if you're going to have a lot of employees?”

No Data Center MPK retained an environmental and land use attorney. They recommended an ordinance banning data centers immediately, then to reinforce it with a ballot measure.

They set up a one-click email so residents could send comments to City Council demanding both.

Hundreds showed up to the next three council meetings.

“We played mahjong on the City Hall lawn while we waited. We had a lion dance right outside to cheer people on as they entered the council chambers,” Yip says.

The chambers filled, overflowing into the aisles and hallways.

The first meeting produced a 45-day ban on data centers, the second brought a ballot measure that would ban them forever.

During the third meeting, opponents of the data center called for a rally at 5:30 p.m. before the 6:30 p.m. meeting. Steven Kung, the Monterey Park resident who purchased the No Data Center MPK domain, addressed the developers, their lawyers, and the lobby firm directly.

“You’re fighting an uphill battle against an entire city that doesn’t want you here and yet you continue to bully your way into this community of color, to pollute the air we breathe, to make electricity more expensive, to devalue our homes, to drain our energy and resources like a parasite,” Kung says. “You think you can take us on? You’ve messed with the wrong city. ”

On March 31, the developer withdrew its application.

“They underestimated the community's passion. And we never underestimated them. And I think that was a good strategy,” says Yip.

The FPPC, California’s political ethics commission, saw the data center would have a “material financial effect” on councilmember Wong. His power to vote on them was stripped.

Yip says the people who joined for the fight over 1977 Saturn Street have overwhelmingly stayed active with SGV Progressive Action.

"They recognize it's not just about data centers. It's about building community and protecting your community," he says.

The volunteers who organized against the data center are now working to strengthen sanctuary ordinances to protect their communities from ICE.

“And now we have a coalition of all these community members coming from La Puente, Avocado Heights, Rowland Heights, and Hacienda Heights coming together to fight this common enemy. And I'm just going to name it the City of Industry,” Yip says.

NO DATA CENTERS SGV COALITION

Samuel Brown Vazquez rode ten miles horseback from Avocado Heights to the Monterey Park council meeting. Vazquez is a community organizer and founding member of the Avocado Heights Vaqueros.

The City of Industry is in the process of approving zoning changes that would clear the way for three data centers and battery energy storage that would affect the residents of Hacienda Heights, La Puente, Walnut, Diamond Bar, West Covina and others.

The Avocado Heights Vaqueros, SGV Progressive Action, and others organized across city and county lines.

“We created an infographic and started mobilizing folks,” Vazquez says. They are No Data Centers SGV Coalition.

Like Monterey Park, they canvassed door-to-door, showed up to every City of Industry meeting, started a petition, and filed public records requests.

The record requests showed the City of Industry had been discussing zoning changes with developers at Puente Hills Mall, Madrid Middle School, and two battery storage facilities near Hacienda Heights. All just feet from homes and schools.

In February, the city unanimously rezoned Puente Hills Mall for battery storage. Months earlier, the city manager, Joshua Nelson, emailed the developers that they were working to allow data centers anywhere in the city.

Again, organizers found that people had no idea what the city was planning to put next to their homes. Battery centers burned for days when they caught fire. One at Moss Landing had spilled 55,000 tons of nickel, cobalt, and lithium.

“Maybe only one or two people had heard," said Sophia Ramirez, an organizer and the daughter of Zacatecan immigrants. “That was pretty shocking.”'

Ramirez, a Cal Poly biology grad, explained in the outreach, in plain language, how PM2.5 and PM10 particles could slip past the body's filters into the lungs, then the blood.

The No Data Center SGV petition grew to 18,000 signatures.

In Monterey Park, the people who organized were also the people who vote. In the City of Industry, there hasn’t been a competitive election in 10 years, and only four since 1957.

State law says when fewer people run than there are open seats for, a city doesn't have to hold an election. The City of Industry council appoints its council members and some of its members are descendents of the original founders.

The City of Industry has 256 residents, the largest financial reserves of any city in the San Gabriel Valley, and a history of building its wealth at the expense of the surrounding working-class Latino and Asian Pacific Islander communities.

"I had normalized what it was like to live next to City of Industry,” Vazquez says. “I thought it was normal to just grow up near all these warehouses."

The people of the surrounding areas, Covina, Diamond Bar, El Monte, La Puente, Pomona, Walnut, and West Covina, would all be affected by the data centers, but have no political power over the City of Industry.

“To think of it in the context of environmental racism, environmental injustices, it’s really crazy that it's like the city of industry has decided that these communities that live around them are not valuable lives,” Ramirez says. “Zonas de sacrificio.”

People from La Puente, Avocado Heights, Rowland Heights, Diamond Bar, Walnut, and Hacienda Heights came to the City of Industry’s March 26 council meeting where they planned to change the city's zoning code to allow data centers.

The city’s email went down before the meeting. People said it often does. The only way to comment is to show up. More than 100 people did, at 9 a.m. on a workday. Before public comment, the council went into closed session.

“They made us all wait outside in the heat,” Ramirez says.

Ramirez said that it was 90 degrees, and there were many elders. After two hours, roughly 40 were let inside. Outside, there was no livestream. They chanted, "Let us in."

When the doors opened, only about 30 people were allowed to speak. Their comment time was cut from three minutes to one minute. Mayor Pro Tem Greubel got up and walked out halfway through.

The City of Industry does not provide interpreters, translated materials, or any way for people who speak other languages to comment. They haven't posted meeting minutes in over a year. More than a quarter of their meetings are called with 24 hours notice.

“Everything about City of Industry is designed to minimize participation of the public,” Vazquez tells us. ”Like, that is not an exaggeration to say that.”

VALLE IMPERIAL RESISTE VS. IMPERIAL COUNTY

Gilberto Manzanarez, an organizer and founder of Valle Imperial Resiste, learned of the data center on Facebook. Someone had leaked the planning commission’s map over Thanksgiving break.

Manzanarez grabbed his camera, drove to the site, and posted a video that spread across the valley.

The Imperial Valley data center would be one of the world’s largest at nearly one million square feet. It would use double the electricity of Imperial Valley and 750,000 gallons of water every day. County planning staff decided they “qualified as a permitted industrial use,” and there would be no need for environmental review.

Manzanarez, also a history teacher, says, “Public health always takes a backseat to economic development in the Imperial Valley.”

The air carries cropland burnings, diesel fumes, and dust from the drying Salton Sea, laced with pesticides and heavy metals. More than one in five children that live around the Salton Sea have asthma.

Despite decades of investment, solar farms, geothermal plants, military bases, and canals, the 80 percent Latino region’s unemployment sits at three times the national average.

“The promise of economic development and jobs, the same thing, these are stories that we already heard when we had the solar farm boom. Those solar panel projects were sold to us,” Manzanarez says. “It's like, this is going to save us. This is going to lift us out of poverty ... Thousands of people across the Imperial Valley were hired, and they were excited to go work. Fast forward 10 years, 2026, guess what? We have the highest unemployment rate in the state of California. Again.”

Bryan Vega, another local organizer, saw the Valle Imperial Resiste post and joined the other activists. They knocked on doors, handed out flyers, and flooded Instagram with informational videos.

“We started to share information about what the data center is and invited folks to submit public comment,” Vega says.

In January, they held a protest on Main Street and Imperial Avenue. They started a petition to enact the Imperial County Data Center Prohibition Act.

On March 26, the Imperial County Board of Supervisors called for an evening meeting, outside of their normal schedule, specifically about the data center.

The main chamber filled 30 minutes before it started. Two overflow rooms opened in a separate building. And when those filled, too, more than 60 people stood in the parking lot.

The developer, Sebastian Rucci, spoke. There were to be no questions.

Vega recalled standing out in the parking lot.

“Someone outside said, ‘Why are we just standing here? Why are we not more upset? They're making decisions about us in there. And we're out here. And we should be in there. We go to the door and we're like, ‘No Data Center. No Data Center.’ And it's like a battle cry from our soul,” he says.

Vega said one of the organizers, from Imperial Valley for Palestine, had a bullhorn in her car, “because, duh, she's an organizer and she's always prepared for these things."

KPBS reported that “county officials paused the meeting and Rucci departed early after protestors drowned him out.”

Videos show Imperial County sheriff’s deputies escorting Rucci and his business partner, Hector Casas, to their car.

Outside, the 60 people who weren’t allowed in chanted “Fuera! Fuera!” at Rucci and Casas as they drove off.

"This meeting was a sham,” Manzanarez says. “They didn't want to educate us. They didn't want to hear people. They just wanted to check a box."

KPBS reported that in 2010, Ohio prosecutors charged Rucci with money laundering, promoting prostitution, and perjury at a Youngstown nightclub. The felonies were thrown out, but he served 30 days for selling alcohol on an expired license. He later opened an addiction treatment center. The state revoked its certification after finding falsified records.

In 2021, the FBI raided the treatment center and seized more than $600,000. No criminal charges were filed. Rucci sued, got the money back, and is still fighting in federal court.

In an interview with KPBS, he said, "I won them all but one."

Rucci and his partner, Hector Casas, targeted Imperial County because the zoning laws allowed them to skip environmental review.

"Our whole goal is speed," Rucci told KPBS. "That is not sneaky. That's just smart."

On April 7, at 8 a.m., 50 men got off a tour bus with identical orange vests that said, "Data centers equal jobs and prosperity."

They were led through the side entrance of the County Administration Building before any members of the community were allowed through the front. A leaked internal county email sent the night before warned of high turnout and increased security. Despite that, the county provided no overflow room.

Over 100 community members were left outside in 96-degree heat for five to six hours. Elders with walkers. No shade, no water, no chairs. One community member got kicked out for calling out the outsiders taking seats from residents.

The board of supervisors voted to approve the lot merger. Only one supervisor voted no.

Senator Padilla's SB 887 would require all new data centers in California to go through environmental review. The bill does not ban data centers, it only requires developers to study their impact and hear from the public before building.

On April 2, the organizers filed the Imperial County Data Center Prohibition Act, the first step toward a ballot measure that would ban data centers across the county.

“Our parents and grandparents sacrificed so much to be able to be in the Imperial Valley ... part of the sacrifice was having to accept a hard hand,“ Vegas says. “Like when I think about what my Mexican farmworker parents had to undergo, it makes it almost intuitive to fight for the environment, intuitive to fight for the things that I know are true to them, but also very simply put, we would not be here without that."

How one programmer's pet project changed how we think about software

2026-04-17T03:53:44Z

This is the story of how one programmer's obsession with simplicity quietly reshaped how the software world thinks about time, immutability, and what it means to write code that lasts. From a sabbatical pet-project to the backbone of one of the world's largest fintechs and a global community that treats their language like a philosophy. This is the story of Clojure. --- This documentary wouldn't exist without the kind support of Nubank: https://building.nubank.com/engineering/ Thanks to Railway, our channel sponsor, for supporting all of our films: Railway is the all-in-one intelligent cloud provider ➡️ https://railway.com/?referralCode=cultrepo --- The Clojure Documentary features: Alessandra Sierra, Alex Miller, Chris Houser, David Nolen, Ed Wible, Eric Normand, Eric Thorsen, Lucas Cavalcanti, Michael Fogus, Nathan Marz, Rich Hickey, Steph Hickey, and Stuart Halloway. Film Credits: Directed by: Cormac Dunne Produced by: Emma Tracey Additional direction: Joey Bania Music supervision and sound design: Tomás Malara --- For a full overview of all the papers, talks, essays, etc. that appear in the film, check this link: https://clojure.org/about/documentary --- Follow us: X: x.com/CultRepo Bluesky: cultrepo.bsky.social Instagram: www.instagram.com/cult.repo LinkedIn: https://www.linkedin.com/company/cult-repo

‘By Design’ Flaw in MCP Could Enable Widespread AI Supply Chain Attacks

2026-04-16T12:15:58Z

Model Context Protocol (MCP) has been a boon to agentic AI users and is widely used and trusted locally by companies adopting agentic AI internally.

Introduced by Anthropic in November 2024, it provides a standard connector between agents and data. Enterprises use it locally to avoid the pain of developing their own connectors, and it is in widespread use as a local STDIO MCP server.

There are multiple providers of MCP servers, almost all inheriting Anthropic’s code. The problem, reports OX Security, is what it terms an architectural flaw in Anthropic’s MCP code embedded within most of these local STDIO MCPs.

In a nutshell, OX Security says this flaw can result in a complete adversarial takeover over the user’s computer system. “And the exploit mechanism is straightforward, MCP’s STDIO interface was designed to launch a local server process. But the command is executed regardless of whether the process starts successfully,” reports OX.

“Pass in a malicious command, receive an error – and the command still runs. No sanitization warnings. No red flags in the developer toolchain. Nothing.”

OX extensively tested whether this ‘flaw’ was exploitable, extensively succeeded, and extensively disclosed its findings to the MCP providers; from Anthropic downward. Initially it had little response. Eventually, the common response was inaction coupled with the suggestion that this behavior was ‘by design’.

Advertisement. Scroll to continue reading.

But OX discovered, and demonstrated, that this ‘by design’ behavior could be easily exploited, leaving potentially millions of downstream users exposed to sensitive data, API key and internal corporate data theft, the exposure of chat histories, and more. If the process that MCP failed included malware, that malware could be silently installed, potentially leading to complete system takeover.

Eventually, the only apparent action from Anthropic was to quietly update its security guidance to recommend MCP adapters be used ‘with caution’ – “leaving the flaw intact and shifting responsibility to developers”.

This is an interesting position to take. It suggests that developers are responsible for the security of what they develop, which is fair. It possibly also suggests that any company so breached is not the responsibility of Anthropic, but the fault of misconfiguring the MCP installation – which certainly does happen. And to be fair, GitHub’s own installation was an exception to the OX testing, proving that security gating on installation is possible.

Learn More at the AI Risk Summit | Ritz-Carlton, Half Moon Bay

But the sheer volume of successful compromises conducted by OX demonstrates that the developers installing MCP servers are failing to install successfully. This should be no surprise when AI is automating so many aspects of security and lowering the bar of security competence among developers.

The OX position is that Anthropic should take responsibility and fix this ‘architectural flaw’ itself. Without doing so, it is leaving industry open to “the mother of all supply chain attacks”, starting from Anthropic, fanning out to many thousands of local MCP users, and from those compromised systems to who knows how many other servers.

During its research, OX adopted a coordinated disclosure process, leading to more than 30 accepted disclosures and more than 10 high and critical vulnerabilities patched. But the underlying design flaw, it says, remains, leaving millions of users and thousands of systems exposed to unauthorized access. “The current implementation of the Model Context Protocol places the entire burden of security on the downstream developer – a structural failure that guarantees vulnerability at scale.”

The OX report on its findings includes details on how Anthropic could solve the problem by deprecating unsanitized STDIO connections, introducing protocol level command sandboxing, including a ‘dangerous mode’ explicit opt-in, and developing marketplace verification standards to include a standardized security manifest.

In the meantime, any company adopting STDIO MCP as part of an agentic AI development should do so ‘with caution’.

font-family Doesn’t Fall Back the Way You Think

2026-04-16T10:53:12Z

10 April, 2026

font-family Doesn’t Fall Back the Way You Think

Written by Harry Roberts on CSS Wizardry.

Table of Contents

Independent writing is brought to you via my wonderful Supporters.

There is a small but surprisingly important nuance in the way font-family works that seems to catch a lot of people out. In my continuing series on web performance for design systems, today we’ll look at font stacks and how, when improperly configured, they can cause unsightly flashes of inappropriate or unexpected fallback text, and in more extreme cases, layout shifts.

Correctly, developers for the most part know that font-family is an inherited property: set a font family on the :root/html/body and, unless told otherwise, descendants will inherit that font:

body {
  font-family: system-ui, sans-serif;
}

So far, so good!

The confusion tends to arrive when we introduce a web or custom font on a child element, e.g.:

h1 {
  font-family: "Open Sans";
}

At a glance, this can feel perfectly sensible. The page should use system-ui, sans-serif; the heading uses "Open Sans"; and while the web font is loading, the browser will presumably just fall back to the parent’s stack—system-ui, sans-serif.

Unfortunately, that isn’t the case.

`font-family` Fallbacks Are Self-Contained

Once you declare a font-family on an element, that declaration stands on its own. The element does not say: I would like "Open Sans", and if that is unavailable right now, please work your way back up the DOM and inherit whatever fallbacks the nearest ancestor might have.

Instead, it says: My font-family is "Open Sans". And that’s all it says.

And if the browser does not yet have "Open Sans" available (yet), it resolves fallback from that declaration, not from the parent’s.

Put another way:

h1 {
  font-family: "Open Sans"; /* « The fallback happens inside here… */
}

/**
 * …not here.
 */
body {
  font-family: system-ui, sans-serif;
}

Why You Get a Flash of Times

If the current element’s font-family declaration contains only one value, and that value is not currently available, the browser falls back to its default for that element, and not to an inheritable font-family from somewhere higher up. For most browsers in their default state, that fallback is likely Times or Times New Roman. That is why you so often see a brief flash of Times New Roman where you were expecting something much more sympathetic or appropriate.

The browser is not forgetting the parent’s font stack; it’s obeying the child’s declaration exactly as written, then exhausting the options available in that declaration, and then falling back to the browser default.

The Fix Is Simple

Whenever you specify a font-family, specify a complete stack. I’m looking at a client’s site right now and I can see this right at the very top of their CSS:

:root {
  --hero-hero: "Clan Pro";
  --heading-x-large: "Clan Pro";
  --heading-large: "Clan Pro";
  --heading-medium: "Clan Pro";
  --heading-medium-subtle: "Clan Pro";
  --heading-small: "Clan Pro";
  --heading-small-subtle: "Clan Pro";
  --heading-x-small: "Clan Pro";
  --paragraph-title-x-large: "Bernina Sans";
  --paragraph-title-large: "Bernina Sans";
  --paragraph-title-medium: "Bernina Sans";
  --paragraph-title-small: "Bernina Sans";
  --paragraph-title-x-small: "Bernina Sans";
  --paragraph-body-x-large: "Bernina Sans";
  --paragraph-body-large: "Bernina Sans";
  --paragraph-body-medium: "Bernina Sans";
  --paragraph-body-small: "Bernina Sans";
  --paragraph-body-x-small: "Bernina Sans";
  --label-3-x-large: "Clan Pro";
  --label-2-x-large: "Clan Pro";
  --label-x-large: "Clan Pro";
  --label-large: "Clan Pro";
  --label-medium: "Clan Pro";
  --label-small: "Clan Pro";
  --label-x-small: "Clan Pro";
}

At the very least, all of these simply need to read:

:root {
  --hero-hero: "Clan Pro", sans-serif;
  --heading-x-large: "Clan Pro", sans-serif;
  --heading-large: "Clan Pro", sans-serif;
  --heading-medium: "Clan Pro", sans-serif;
  --heading-medium-subtle: "Clan Pro", sans-serif;
  --heading-small: "Clan Pro", sans-serif;
  --heading-small-subtle: "Clan Pro", sans-serif;
  --heading-x-small: "Clan Pro", sans-serif;
  --paragraph-title-x-large: "Bernina Sans", sans-serif;
  --paragraph-title-large: "Bernina Sans", sans-serif;
  --paragraph-title-medium: "Bernina Sans", sans-serif;
  --paragraph-title-small: "Bernina Sans", sans-serif;
  --paragraph-title-x-small: "Bernina Sans", sans-serif;
  --paragraph-body-x-large: "Bernina Sans", sans-serif;
  --paragraph-body-large: "Bernina Sans", sans-serif;
  --paragraph-body-medium: "Bernina Sans", sans-serif;
  --paragraph-body-small: "Bernina Sans", sans-serif;
  --paragraph-body-x-small: "Bernina Sans", sans-serif;
  --label-3-x-large: "Clan Pro", sans-serif;
  --label-2-x-large: "Clan Pro", sans-serif;
  --label-x-large: "Clan Pro", sans-serif;
  --label-large: "Clan Pro", sans-serif;
  --label-medium: "Clan Pro", sans-serif;
  --label-small: "Clan Pro", sans-serif;
  --label-x-small: "Clan Pro", sans-serif;
}

Remember, any time you declare a font-family, declare the whole thing. Even if that is just a broad <generic-family> And while this is the bare minimum, at least sans-serif web fonts will actually fall back to sans.

To do a much more thorough job, you can simply hire me to run my Web Performance for Design Systems workshop.

Why This Matters

At its most simple, this is a trivial visual issue: a nascent sans heading briefly rendered in serif just looks wrong.

At the other end of the spectrum, it can have real knock-on effects on Core Web Vitals: if the fallback face is excessively different in width, height, or overall proportions, the eventual swap to the web font can have an impact on your CLS scores.

Closing Thoughts

If a font-family matters enough to override, it matters enough to define properly. This is one of those small details that feels too small to matter right up until you notice it everywhere.

My client has had complaints of noticeable layout shifts while migrating to a new design system, and at the size and scale they’re working at, they were really, really struggling to pin it down. It only took me a few minutes because it’s easy when you know the answer. That’s exactly why you hire consultants.

Under the hood of MDN's new frontend

2026-04-11T15:22:11Z

Last year, we launched a new frontend for MDN. The most noticeable changes were adjustments to our styles; we simplified and unified the MDN design across all of our pages. In truth, the biggest changes were not reader-visible, but rather in the overhauled code that powers our frontend. This post describes what we've done, the technologies we chose, and why we did it at all.

To fully understand the changes we made to MDN's frontend, I should provide some context on how MDN content is assembled into the website you all know and love. MDN's architecture is probably worthy of its own blog post, but to simplify for the sake of this post, pages are published to the site via these major steps:

The documentation is written and maintained in Markdown, across a couple of git repositories, by our fantastic team of technical writers, partners, and invited experts, alongside our enormous community of contributors and translators.
A build tool ingests these Markdown files, converts them into HTML, and saves them as a set of JSON files with supplemental metadata about each page.
Our frontend traverses these JSON files and compiles fully-featured pages, complete with browser compatibility tables, l10n support, navigation menus, and so on - in a step we name (or perhaps misname) server-side rendering (SSR).
At this point, the resulting HTML, CSS, and JavaScript files are uploaded to cloud buckets and delivered to our readers globally.

Rebuilding our frontend had been a long time coming because of how tricky it was to work on MDN's UI. Our previous frontend (called yari) was a React app that, unfortunately, had accumulated quite a lot of technical debt. Maintenance wasn't exactly impossible, but was certainly painful to undertake. Whenever we fixed issues or added new site functionality, we inevitably ended up piling on more technical debt. But how did we get there?

The React app had started life as a "Create React App", but a number of the built-in defaults didn't work for us. Of course, this led to a series of workarounds, and we eventually had to "eject" the configuration. We ended up with an extremely complicated Webpack config as well as some very hacky build scripts.

On the CSS side as well, things were starting to get out of control. We used Sass extensively, then added modern CSS features like CSS variables, which meant we had a bizarre mix of both idioms spread across our files.

The CSS was also incredibly entangled, with poor or nonexistent scoping. When we made a change in one UI component, we'd frequently spot unintended changes in others. These issues, and a lack of build tools to split up the CSS, meant we had to ship a large render-blocking CSS blob to our users, complete with styles for components they might never load.

But by far, the biggest issue was that our React app was merely a wrapper around our static content. To make the React app aware of the HTML content that our build tool generated would have required expensive reparsing of the HTML and an extraordinary amount of logic which we'd have to ship to users in our client-side JavaScript. We didn't want to do this, so the React app boundary essentially ended where our documentation began – we used React's dangerouslySetInnerHTML to insert the content.

Our content is mostly static (prose and code examples), but there were a number of places within this static content where we needed to add interactivity (think things like the "Copy" button on code blocks). For these interactive parts, we ended up using regular DOM APIs, which wasn't very elegant, particularly when the rest of the site was written in React. We couldn't use JSX (React's HTML-like syntax), which limited the maintainability of more complex pieces of interactivity, and we occasionally faced the worst-case scenario of maintaining duplicate implementations - one using React and another using DOM APIs.

As a possible solution to this problem, in 2024, we started experimenting with Lit and web components to see whether they could improve the developer experience when working on this kind of interactivity within content. Our first proper prototype, and eventual production implementation, came out of our work on the MDN Curriculum when we partnered with Scrimba.

Scrimba has a feature called "Scrims" - an interactive learning environment we embed on MDN via an <iframe>. Scrims let learners watch a short coding tutorial and then edit the code themselves, all within the same view — think of them as interactive screencasts.

On our pages, we didn't want to send any user data to Scrimba until a user chose to interact with their content; so we didn't load the <iframe> until after a user clicks to open it. We also wanted to be able to expand the Scrim to fullscreen, without a user leaving MDN, so we used the <dialog> element.

We figured that building a web component would allow us to use a custom element to insert these Scrims directly into our content, thereby skipping a number of rendering steps and avoiding the tricky-to-maintain DOM API implementation.

Our component starts life by extending LitElement:

Within that, we need to define some state. In Lit, we can do this through a static properties attribute:

And we set defaults in our class constructor:

We want to manipulate the URL which has been provided as an attribute to our custom element. Lit provides us with lifecycle methods to do this. We want to compute a value once we know an update to the component will be rendered:

We can then use this state to render our component:

Lit's html template literal is just as convenient as JSX in allowing us to write HTML-ish syntax in JavaScript. The huge advantage over JSX is it doesn't require any compilation to use: it's native JavaScript.

I say "HTML-ish" because you'll notice a few annotations before certain attributes in the template above, namely @close and @click. This is Lit syntax that allows us to bind event listeners to these elements: the close event on the <dialog> and click events on a couple of buttons. We define these in the class too:

When a user clicks to open the Scrim, the #open method fires, which updates the values of _scrimLoaded and _fullscreen. Lit notices the changes to these properties, because we'd defined them in static properties, and automatically re-renders the component, loading the <iframe> and the Scrim inside.

I've simplified the component a little for brevity, you can see the MDNScrimInline source on GitHub. There's a number of additions in there like telemetry, and a dynamically-rendered thumbnail (it was fewer bytes and a simpler implementation than pre-rendering a bunch of images). As you can imagine, this was very straightforward to develop, thanks to the convenience functions we get with Lit; this would've been a massive headache to implement directly with traditional DOM APIs.

In many ways, I found that the implementation in Lit was simpler than in React: you may notice the state we're dealing with isn't particularly complex, and doesn't require a complex component architecture to reflect it. But more importantly, it gives us that custom element we can insert into our curriculum content wherever we need to add a Scrim:

The Scrimba implementation was a good introduction for the team to writing a small component, but what about something more complex? Interactive examples are the components that appear under "Try it" sections at the top of many CSS, JavaScript, and HTML pages.

Improving the infrastructure for these had been on the engineering team backlog for a while; they were difficult for our technical writers and community to maintain and author. The existing implementation was split across four git repositories, and authoring or debugging an example could require synchronizing changes across them all. Worse still, examples had to be written in isolation from the content they'd be included in, so it wasn't possible to show a live preview containing both changes to an example and the content of the MDN page it would be included on.

This complexity came about for good reasons: these interactive examples were too complex to easily engineer and maintain with DOM APIs directly. Instead, we had a separate build system and examples repository which rendered these examples into separate HTML pages which we could load in an <iframe> directly.

We wanted to simplify this architecture, to make writing interactive examples easier for our authors, so again: we reached for Lit to build a web component we could include directly in our content. This was a much more technically-complex implementation than Scrims. Firstly, we needed a number of templates for the different ways interactive examples are displayed:

A code editor and console for JavaScript examples, with the addition of tabs for WASM examples.
A tabbed code editor with rendered output for HTML examples (usually tabs for HTML and CSS).
A table of code editors that can be selected with rendered output for CSS examples (see the CSS background-clip property, for example).

Secondly, we needed a way to render the examples, and the edits users made to them. We had already written that logic to create our interactive Playground, but it was in React: so we needed to port that too.

So we set about doing all that. What made things a lot simpler was that Lit's React integration allowed us to render these web components in our existing React app. So we could port the elements of the Playground we needed piece-by-piece to web components, without having to port the entire thing all at once, and without having to maintain dual implementations.

At a high level, we split our single entangled Playground React component into a series of custom elements:

<play-editor>: A CodeMirror-powered editor.
<play-console>: An element to format and render console messages.
<play-runner>: An element responsible for rendering the current state of each editor.
<play-controller>: An element responsible for passing events and state between each of the above elements.

This made the logic in the Playground simpler and decoupled, which allowed us to reuse these elements in the <interactive-example> element we created. This included logic to search the page for <code> elements the interactive example component should ingest, sending their contents to a <play-controller>, and working out which of the above templates it needed to render: using some combination of the <play-*> elements to do this.

This meant that authors could now add a macro to content - which renders our <interactive-example> custom element behind the scenes - followed by the code blocks the example should use:

You can see the full source for this example on the CSS background-repeat page GitHub. We're not quite sure where we stand on putting custom elements directly in our non-curriculum Markdown content, which could make our architecture even simpler; that's a discussion for another day.

So this is all well and good, web components seem pretty cool and solve some of our problems around adding interactivity within our static content. But wasn't this blog post supposed to be about how we rewrote our entire frontend stack: what happened to that? To answer that, let's get into another problem we had with our old frontend:

I've mentioned our problem with our React app being a "wrapper" and not being able to interact with our content. The fundamental problem is that (at least classically) React apps are Single Page Applications (SPAs), which you figure out how to render on the server, then attempt to figure out how to avoid shipping an absolutely enormous JavaScript bundle to your users.

That last part is necessary. That's because everything you render in an SPA, even if it could be rendered statically on a server or in a compilation step, has to be shipped in your client-side JavaScript bundle and re-rendered in the client – just to verify nothing has changed. The React documentation itself summarises it better than I could:

This pattern means users need to download and parse an additional 75K (gzipped) of libraries, and wait for a second request to fetch the data after the page loads, just to render static content that will not change for the lifetime of the page.
Server Components on react.dev

That's a quote from the documentation for React Server Components (RSC): so the project recognizes that this is a problem, and is working hard on solving it. Unfortunately, using RSC effectively requires using a framework that we aren't already using. Migrating to that would require rewriting a lot of the frontend anyway.

So since a large rewrite was required to address this fundamental issue, we could also re-evaluate what kind of a site MDN is, and how complex it needed to be. Really, MDN isn't a particularly complex site, at least from a "things that require interactivity" standpoint. The vast majority of content on an MDN documentation page is HTML and CSS: we don't need a complex app powering the majority of the site. We essentially have islands of interactivity, which could easily all be implemented as web components.

And if we're implementing all our functionality in isolated web components, it doesn't really matter how they're assembled: we just need to template HTML together, and that can happen multiple times, in multiple places in our overall build system. There's no higher-level "app" that needs to understand the state of the entire page, so there never could be a "wrapper" problem – our markdown to HTML build tool is just as first class a citizen as whatever templating we need to do in our frontend.

This approach solved all three problems at once: there's no SPA shipping redundant JavaScript to re-render static content, there's no "wrapper" that can't reach into our documentation HTML, and each piece of interactivity is a self-contained web component that only loads when it's needed. What remained was deciding how to do the static templating that assembles everything else.

We considered using a dedicated templating language, such as EJS, to do templating in our frontend, but realized there's a lot of benefits to a component-based architecture. While doing static HTML templating on the server avoids shipping logic to users in a client-side JavaScript bundle, this HTML still requires styling. And as you may recall, the CSS in our old frontend was a mess, and we wanted to avoid shipping unnecessary CSS if it wasn't necessary to render the current page.

We first built our own concept of Server Components, using Lit's HTML template literal, which we were already comfortable with. Here's an example of our top navigation bar component:

You'll see the logic is handled in a render method, much like it would be in a Lit component. We don't need any lifecycle methods because this only ever runs once. This component can render other server components like Logo and Menu, as well as web components like <mdn-search-button> and <mdn-search-modal>.

We render this to HTML in NodeJS, using a convenient function Lit provides. This also renders these Lit web components to Declarative Shadow DOM so, in compatible browsers, the Shadow DOM and CSS of our custom elements is rendered before the JavaScript gets loaded.

As I mentioned before, one big problem with an SPA-approach to building websites is everything necessary to render the page needs to be included in the client-side JavaScript bundle. Another similar problem is that it's very easy, and therefore very common, to ship a whole load of code not necessary for rendering the current page, and only necessary for rendering other pages, in one huge client-side bundle. We fell into this trap with our old frontend, both with our JavaScript and our CSS.

Over time we did partition off certain routes into separate chunks, but this was only possible with our JavaScript. Our CSS was too entangled to do this, and our build tool wasn't configured to do it either.

And, it was only possible to do this per route, not for components within the same page. If there was a chance a certain route might load a component, it needed to be included in the bundle if it was to be server-side rendered, even if it wasn't, and that JavaScript was never executed client side.

We wanted to avoid all this in our new frontend: only loading the most minimal CSS and JavaScript bundles required to render the page, and making it interactive; and I wanted to achieve this on an architectural level, so it was nearly impossible to not do. We achieved this in a few ways, but the key to unlocking them all was a flat name-based component structure.

Every component lives in a flat hierarchy under the ./components/ directory, with the following file names reserved for certain pieces of the component:

components/example-component
├── element.css
├── element.js
├── global.css
├── server.css
└── server.js

element.js - A web component, exporting a MDNExampleComponent class and defining a <mdn-example-component> element.
server.js - A server component, extending ServerComponent from components/server/index.js.
server.css - CSS for a server component that will be automatically loaded for this component from the server.
global.css - CSS for the component that gets loaded everywhere all the time.

We enforce parts of this via linting, and some of this by throwing errors if you don't adhere to the naming requirements.

Because we know where each web component lives based on name, we can do some clever things. When our page loads, we run logic like this client-side:

This results in lazy-loading every custom element present in the DOM at load time, entirely async and in parallel. The advantages here are numerous:

Engineers don't have to remember to import web components in server components; they can use them as if they are normal HTML elements.
We can add custom elements to our content markdown (either directly or through a macro), without having to wire up an export elsewhere.
We only ever load the JavaScript for each web component if it's present on the page, automatically. It's not necessary for engineers to think about whether their new component increases the bundle size - if it's not present on the page the user is viewing, that code won't end up being loaded by the users' browser.
Changes to one component should only have minimal impact on the rest of the bundle: a bugfix in one component will require that component's JavaScript to be reloaded, but other components should be cached by the browser and will become interactive almost immediately, as they load in parallel asynchronously.

We also automatically load every web component in our SSR bundle, where Lit renders them into a Declarative Shadow DOM (unless the component opts out because it doesn't make sense to render them on the server). This helps ensure we don't have layout shifts when the JavaScript loads. The result is that the slight delay to interactivity when we load its JavaScript after initial render is imperceptible because the component is already on the page, just not interactive yet.

At least, not entirely interactive: this architecture is flexible enough for us to be very clever with certain components. One of these is <mdn-dropdown>.

This is a component which, as the name might suggest, implements a dropdown. The render method is rather simple:

We render two slots, which can be used like so:

Since we use slots here, <mdn-dropdown>'s shadow DOM is almost irrelevant, and any children of it can be styled entirely as normal: the element only adds interactivity. It does have some styles attached, but those aren't for styling: they're for interactivity too. See, we also have a lifecycle method:

And define our loaded property like so:

If you trace the logic through, you'll see that the dropdown slot isn't hidden by default, and therefore, is visible when we render to DSD on the server. And once the component is loaded=true, we reflect that attribute into the DOM, so it appears like:

The reason we want this is because we also attach the following CSS to the element:

The logic here is a little hard to parse - and I say that as the person who wrote it - but it effectively translates to:

If JavaScript for the mdn-dropdown has loaded, do no styling.
If the JavaScript for the mdn-dropdown hasn't loaded, then:
- If the focus is outside the element, hide the dropdown slot.
- If the focus is within the element, show the dropdown slot. And what does this mean? Well, we have a CSS native dropdown as soon as the page is rendered, which progressively enhances to a JavaScript dropdown, once that code has loaded; other engineers need not know any of this is going on, just that the dropdown component works.

This is hugely important in our top navigation menu, where most of our links are behind dropdowns: those are completely usable as soon as they're rendered to the page. In other components, such as in our theme switcher, while choosing between themes isn't possible until its JavaScript loads, the dropdown being interactive already gives us a few seconds longer load time for that JavaScript before the user clicks on anything requiring it.

What if we don't have DSD

Now, Declarative Shadow DOM isn't widely available yet, so we have to ensure things also work on slightly older browsers. This is where the global.css file comes in: any CSS written in one of these is included on all pages, all the time.

This is obviously necessary for setting things like global CSS variables, global reset styles, and the like. But for components, when DSD isn't available, before its JavaScript has loaded, they'll appear to the browser as an empty inline element with no styling attached. This isn't always optimal, and can cause layout shifts when the JavaScript gets loaded, so we set global styles for certain elements, for example, for our button component:

This is just enough to ensure buttons don't shift the layout before being loaded. There's a small optimisation to be had by only loading this style when an mdn-button is present on the page, rather than on every page: but it's so minimal it's probably not worth the added complexity. It's also important for our aforementioned dropdown component: we still want this to be interactive if DSD hasn't loaded, so we include a similar style in a global.css file.

For server components, the considerations are a little different. The JavaScript itself doesn't need to be lazy loaded or cut down, since it's only being used to SSR our HTML. But what does require careful loading is the CSS used in each server component. We only want to load this if the component gets rendered to the page. But how do we know this?

Our server components extend our ServerComponent class, so we place some tracking logic in its static render method, which runs before and after instantiating each server component:

This gives us a Set, componentsUsed, which only contains the components that rendered anything. We then use this in our OuterLayout component:

The compilationStats object here comes from our build tool, Rspack (more on that later), and this code block gives us a list of <link> tags to include in our <head> containing only the CSS that's necessary to render the page. We also load a CSS file with all the global.css files mentioned before, bundled into one.

Again, this is a simplified version of our ServerComponent and OuterLayout classes, which you can see in their entirety in our repository if you're interested.

You'll note that what I've suggested here results in a number of quite small CSS and JavaScript files being loaded on the page. This goes against the classical wisdom that there's an optimal bundle size where you combine multiple assets into one to reduce the number of round trips a browser needs to make to load them all.

I can't claim to be a performance expert here, but HTTP/2 and HTTP/3 do a lot to change that wisdom. As we can now download assets in parallel, and reuse connections, multiple small assets don't have the overhead they did before, and can be advantageous, particularly given how our web components load.

As I described before, since we load our web components asynchronously and independently after the page has rendered, it's faster to fire these down the wire component by component - so the browser can act on adding interactivity as soon as the code has loaded - rather than all in one blob where the browser has to parse the code for multiple components, perhaps to only add interactivity for one.

Once you throw caching into the mix, things get even faster, and this extends to our CSS too: an update to one component in many cases won't touch the bundled code of the others. So for a user re-visiting MDN, they'll get interactivity for components that have been cached near-instantly, and for any that have changed - or for any server component CSS that has changed - they'll only be waiting for that changed component to download.

Benchmarking these things is always required, and we could always do more, but what we've done so far showed that bundling things together was only as good or slower with a cold cache. We do also have a few levers in our build config to pull to easily bundle smaller components together if future benchmarking shows that that would be better.

We're using quite a number of more modern web technologies here, and we needed an easy way to determine if we could use something, and if we could do without polyfills or progressive enhancement. Luckily enough, we've spent the last few years working on the Baseline project with a cross-vendor range of partners in the WebDX group.

This gave us a fantastically easy way to determine whether to use an API or not. The advice I gave the other engineers was: if it's "Baseline Widely Available", just use it; if it's "Baseline Newly Available", come talk to me first, and we'll figure out a polyfill or if it can be used as a progressive enhancement; and if it's "Baseline Limited Availability" or you need to do something there's no API for yet, think some more about if you really need to do it, then come talk to me.

We ended up using a range of technologies, across all these statuses:

Things like Custom Elements and Shadow DOM have been supported cross-browser for a surprisingly long time these days, and are solidly Baseline Widely Available.

Declarative Shadow DOM, as I mentioned earlier, is supported cross browser but hasn't been in the web platform long enough to be Widely Available, so we use it as a progressive enhancement, with fallbacks in place for older browsers which don't support it yet, like our use of global.css stylesheets. There's also a few things we wanted which are at the very bleeding edge: one of those was extending light-dark to images, where we used PostCSS to define a custom mixin, allowing us to use syntax like this:

Using Baseline allows us to build confidently - knowing the vast majority of users can use a feature once it's reached Widely Available - and also keep our set of polyfills (and their overhead) small as we automatically remove them when features move from Newly Available to Widely Available.

I've saved what I think our best improvement is until last: but I'm a little biased as an engineer working on MDN. Our old frontend development environment pained me every time I had to use it.

The big problems were:

It was slow: the default start command took about two minutes to present you with a functional locally running SPA, not including the time to download npm packages and so on.
It was complex: there were an enormous number of commands in our package.json, some of which gave you a development environment faster by skipping elements of the build, but that required an intricate knowledge of what exactly these complex commands were doing to know if you needed them or not.
It didn't reliably restart: frequently changes, even simple ones like adding a new image, would require restarting the development server - and waiting another two minutes - to see the change.
By default, we only rendered the SPA, with no SSR: that required running a separate command, which only created a production bundle and took even longer to run, which made debugging certain issues with SSR exceptionally difficult to do.

We were very keen to improve this - obviously for our own sake - but also to make the contribution process easier, and we have: the new frontend takes 2 seconds to start, and there's really only one command you need:

An enormous amount of this speed comes from using Rspack as our build tool. Webpack was what the old frontend used, and to its credit is fantastically configurable - something we needed given the approaches we took with our architecture. Rspack has a webpack-compatible API, but it's written in Rust and is incredibly fast.

Though our Rspack config, currently standing at 650 LOC isn't necessarily simple, I would argue it's straightforward. There's very little logic hidden away, magically happening in our build tool. This config does a lot for us, including all the bundling, polyfilling, mixins, and optimizations I described before.

Our architecture is simple enough to rely on a single command that does almost everything. Unlike before, there aren't really separate things to build independently. There's no SPA that we can render without SSR, as server components are fundamental to our architecture. We don't need multiple commands because there's only one way our website is assembled.

This gives us an environment far more similar to our production environment than before, with the main difference being whether we reload the page on every change, and reload and re-render our server components on each request. This means we almost never have to restart our development environment, unless we're making changes to our Rspack config itself or applying various production-level optimizations. We have another command to build a production build with those optimizations and without the dynamic loading, which we can easily run if we need.

Developing in this new environment has been an absolute joy for me, and I'm so glad we managed to make these improvements.

I think you'll agree that this blog post is long enough, and I hear the Slack pings coming in from our content team telling me to finish this post already, but I don't feel like I've even told you half the story of our new frontend architecture!

If you're interested in learning more, or have any questions about anything we've done, please feel free to come chat with us in the #platform channel on our Discord.

If you spot any issues, please raise them in the fred GitHub repository. And if anything there looks like something you'd like to fix, have a go and submit a PR.

Building this new frontend was a complete pleasure: it's been a privilege to be able to use new web technologies to build the website that documents them.

Tech's empiricism problem

2026-03-15T06:36:23Z

The tech industry has extreme difficulty integrating information that doesn't have its source in an overtly rationalist process. In practice, this means that we tend to think that if you can't give a logical chain of deductions that proves that something is the case, your information is worthless. The issue with this is that day-to-day, in the tech world and outside of it, the vast bulk of the information we use to make decisions isn't this kind of information.

Nobody Gets Promoted for Simplicity

2026-03-07T05:55:29Z

“Simplicity is a great virtue, but it requires hard work to achieve and education to appreciate. And to make matters worse, complexity sells better.” — Edsger Dijkstra

I think there’s something quietly screwing up a lot of engineering teams. In interviews, in promotion packets, in design reviews: the engineer who overbuilds gets a compelling narrative, but the one who ships the simplest thing that works gets… nothing.

This isn’t intentional, of course. Nobody sits down and says, “let’s make sure the people who over-engineer things get promoted!” But that’s what can happen (and it has been, over and over again) when companies evaluate work incorrectly.

Picture two engineers on the same team. Engineer A gets assigned a feature. She looks at the problem, considers a few options, and picks the simplest. A straightforward implementation, maybe 50 lines of code. Easy to read, easy to test, easy for the next person to pick up. It works. She ships it in a couple of days and moves on.

Engineer B gets a similar feature. He also looks at the problem, but he sees an opportunity to build something more “robust.” He introduces a new abstraction layer, creates a pub/sub system for communication between components, adds a configuration framework so the feature is “extensible” for future use cases. It takes three weeks. There are multiple PRs. Lots of excited emojis when he shares the document explaining all of this.

Now, promotion time comes around. Engineer B’s work practically writes itself into a promotion packet: “Designed and implemented a scalable event-driven architecture, introduced a reusable abstraction layer adopted by multiple teams, and built a configuration framework enabling future extensibility.” That practically screams Staff+.

But for Engineer A’s work, there’s almost nothing to say. “Implemented feature X.” Three words. Her work was better. But it’s invisible because of how simple she made it look. You can’t write a compelling narrative about the thing you didn’t build. Nobody gets promoted for the complexity they avoided.

Complexity looks smart. Not because it is, but because our systems are set up to reward it. And the incentive problem doesn’t start at promotion time. It starts before you even get the job.

Think about interviews. You’re in a system design round, and you propose a simple solution. A single database, a straightforward API, maybe a caching layer. The interviewer is like: “What about scalability? What if you have ten million users?” So you add services. You add queues. You add sharding. You draw more boxes on the whiteboard. The interviewer finally seems satisfied now.

What you just learned is that complexity impresses people. The simple answer wasn’t wrong. It just wasn’t interesting enough. And you might carry that lesson with you into your career. To be fair, interviewers sometimes have good reasons to push on scale; they want to see how you think under pressure and whether you understand distributed systems. But when the takeaway for the candidate is “simple wasn’t enough,” something’s off.

It also shows up in design reviews. An engineer proposes a clean, simple approach and gets hit with “shouldn’t we future-proof this?” So they go back and add layers they don’t need yet, abstractions for problems that might never materialize, flexibility for requirements nobody has asked for. Not because the problem demanded it, but because the room expected it.

I’ve seen engineers (and have been one myself) create abstractions to avoid duplicating a few lines of code, only to end up with something far harder to understand and maintain than the duplication ever was. Every time, it felt like the right thing to do. The code looked more “professional.” More engineered. But the users didn’t get their feature any faster, and the next engineer to touch it had to spend half a day understanding the abstraction before they could make any changes.

Now, let me be clear: complexity is sometimes the right call. If you’re processing millions of transactions, you might need distributed systems. If you have 10 teams working on the same product, you probably need service boundaries. When the problem is complex, the solution (probably) should be too!

The issue isn’t complexity itself. It’s unearned complexity. There’s a difference between “we’re hitting database limits and need to shard” and “we might hit database limits in three years, so let’s shard now.”

Some engineers understand this. And when you look at their code (and architecture), you think “well, yeah, of course.” There’s no magic, no cleverness, nothing that makes you feel stupid for not understanding it. And that’s exactly the point.

The actual path to seniority isn’t learning more tools and patterns, but learning when not to use them. Anyone can add complexity. It takes experience and confidence to leave it out.

So what do we actually do about this? Because saying “keep it simple” is easy. Changing incentive structures is harder.

If you’re an engineer, learn that simplicity needs to be made visible. The work doesn’t speak for itself; not because it’s not good, but because most systems aren’t designed to hear it.

Start with how you talk about your own work. “Implemented feature X” doesn’t mean much. But “evaluated three approaches including an event-driven architecture and a custom abstraction layer, determined that a straightforward implementation met all current and projected requirements, and shipped in two days with zero incidents over six months”, that’s the same simple work, just described in a way that captures the judgment behind it. The decision not to build something is a decision, an important one! Document it accordingly.

In design reviews, when someone asks “shouldn’t we future-proof this?”, don’t just cave and go add layers. Try: “Here’s what it would take to add that later if we need it, and here’s what it costs us to add it now. I think we wait.” You’re not pushing back, but showing you’ve done your homework. You considered the complexity and chose not to take it on.

And yes, bring this up with your manager. Something like: “I want to make sure the way I document my work reflects the decisions I’m making, not just the code I’m writing. Can we talk about how to frame that for my next review?” Most managers will appreciate this because you’re making their job easier. You’re giving them language they can use to advocate for you.

Now, if you do all of this and your team still only promotes the person who builds the most elaborate system… that’s useful information too. It tells you something about where you work. Some cultures genuinely value simplicity. Others say they do, but reward the opposite. If you’re in the second kind, you can either play the game or find a place where good judgment is actually recognized. But at least you’ll know which one you’re in.

If you’re an engineering leader, this one’s on you more than anyone else. You set the incentives, whether you realize it or not. And the problem is that most promotion criteria are basically designed to reward complexity, even when they don’t intend to. “Impact” gets measured by the size and scope of what someone built, which more often than not matters! But what they avoided should also matter.

So start by changing the questions you ask. In design reviews, instead of “have we thought about scale?”, try “what’s the simplest version we could ship, and what specific signals would tell us we need something more complex?” That one question changes the game: it makes simplicity the default and puts the burden of proof on complexity, not the other way around!

In promotion discussions, push back when someone’s packet is basically a list of impressive-sounding systems. Ask: “Was all of that necessary? Did we actually need a pub/sub system here, or did it just look good on paper?” And when an engineer on your team ships something clean and simple, help them write the narrative. “Evaluated multiple approaches and chose the simplest one that solved the problem” is a compelling promotion case, but only if you actually treat it like one.

One more thing: pay attention to what you celebrate publicly. If every shout-out in your team channel is for the big, complex project, that’s what people will optimize for. Start recognizing the engineer who deleted code. The one who said “we don’t need this yet” and was right.

At the end of the day, if we keep rewarding complexity and ignoring simplicity, we shouldn’t be surprised when that’s exactly what we get. But the fix isn’t complicated. Which, I guess, is kind of the point.

The Frontend Treadmill

2026-03-05T19:30:10Z

A lot of frontend teams are very convinced that rewriting their frontend will lead to the promised land. And I am the bearer of bad tidings.

If you are building a product that you hope has longevity, your frontend framework is the least interesting technical decision for you to make. And all of the time you spend arguing about it is wasted energy.

I will die on this hill.

If your product is still around in 5 years, you’re doing great and you should feel successful. But guess what? Whatever framework you choose will be obsolete in 5 years. That’s just how the frontend community has been operating, and I don’t expect it to change soon. Even the popular frameworks that are still around are completely different. Because change is the name of the game. So they’re gonna rewrite their shit too and just give it a new version number.

Product teams that are smart are getting off the treadmill. Whatever framework you currently have, start investing in getting to know it deeply. Learn the tools until they are not an impediment to your progress. That’s the only option. Replacing it with a shiny new tool is a trap.

I also wanna give a piece of candid advice to engineers who are searching for jobs. If you feel strongly about what framework you want to use, please make that a criteria for your job search. Please stop walking into teams and derailing everything by trying to convince them to switch from framework X to your framework of choice. It’s really annoying and tremendously costly.

I always have to start with the cynical take. It’s just how I am. But I do want to talk about what I think should be happening instead.

Companies that want to reduce the cost of their frontend tech becoming obsoleted so often should be looking to get back to fundamentals. Your teams should be working closer to the web platform with a lot less complex abstractions. We need to relearn what the web is capable of and go back to that.

Let’s be clear, I’m not suggesting this is strictly better and the answer to all of your problems. I’m suggesting this as an intentional business tradeoff that I think provides more value and is less costly in the long run. I believe if you stick closer to core web technologies, you’ll be better able to hire capable engineers in the future without them convincing you they can’t do work without rewriting millions of lines of code.

And if you’re an engineer, you will be able to retain much higher market value over time if you dig into and understand core web technologies. I was here before react, and I’ll be here after it dies. You may trade some job marketability today. But it does a lot more for career longevity than trying to learn every new thing that gets popular. And you see how quickly they discarded us when the market turned anyway. Knowing certain tech won’t save you from those realities.

I couldn’t speak this candidly about this stuff when I held a management role. People can’t help but question my motivations and whatever agenda I may be pushing. Either that or I get into a lot of trouble with my internal team because they think I’m talking about them. But this is just what I’ve seen play out after doing this for 20+ years. And I feel like we need to be able to speak plainly.

This has been brewing in my head for a long time. The frontend ecosystem is kind of broken right now. And it’s frustrating to me for a few different reasons. New developers are having an extremely hard time learning enough skills to be gainfully employed. They are drowning in this complex garbage and feeling really disheartened. As a result, companies are finding it more difficult to do basic hiring. The bar is so high just to get a regular dev job. And everybody loses.

What’s even worse is that I believe a lot of this energy is wasted. People that are learning the current tech ecosystem are absolutely not learning web fundamentals. They are too abstracted away. And when the stack changes again, these folks are going to be at a serious disadvantage when they have to adapt away from what they learned. It’s a deep disservice to people’s professional careers, and it’s going to cause a lot of heartache later.

On a more personal note, this is frustrating to me because I think it’s a big part of why we’re seeing the web stagnate so much. I still run into lots of devs who are creative and enthusiastic about building cool things. They just can’t. They are trying and failing because the tools being recommended to them are just not approachable enough. And at the same time, they’re being convinced that learning fundamentals is a waste of time because it’s so different from what everybody is talking about.

I guess I want to close by stating my biases. I’m a web guy. I’ve been bullish on the web for 20+ years, and I will continue to be. I think it is an extremely capable and unique platform for delivering software. And it has only gotten better over time while retaining an incredible level of backwards compatibility. The underlying tools we have are dope now. But our current framework layer is working against the grain instead of embracing the platform.

This is from a recent thread I wrote on mastodon. Reproduced with only light editing.

A GitHub Issue Title Compromised 4,000 Developer Machines

2026-03-05T19:15:27Z

esc to close

Five steps from a GitHub issue title to 4,000 compromised developer machines. The entry point was natural language.

On February 17, 2026, someone published cline@2.3.0 to npm. The CLI binary was byte-identical to the previous version. The only change was one line in package.json:

"postinstall": "npm install -g openclaw@latest"

For the next eight hours, every developer who installed or updated Cline got OpenClaw - a separate AI agent with full system access - installed globally on their machine without consent. Approximately 4,000 downloads occurred before the package was pulled¹.

The interesting part is not the payload. It is how the attacker got the npm token in the first place: by injecting a prompt into a GitHub issue title, which an AI triage bot read, interpreted as an instruction, and executed.

The full chain

The attack - which Snyk named "Clinejection"² - composes five well-understood vulnerabilities into a single exploit that requires nothing more than opening a GitHub issue.

Step 1: Prompt injection via issue title. Cline had deployed an AI-powered issue triage workflow using Anthropic's claude-code-action. The workflow was configured with allowed_non_write_users: "*", meaning any GitHub user could trigger it by opening an issue. The issue title was interpolated directly into Claude's prompt via ${{ github.event.issue.title }} without sanitisation.

On January 28, an attacker created Issue #8904 with a title crafted to look like a performance report but containing an embedded instruction: install a package from a specific GitHub repository³.

Step 2: The AI bot executes arbitrary code. Claude interpreted the injected instruction as legitimate and ran npm install pointing to the attacker's fork - a typosquatted repository (glthub-actions/cline, note the missing 'i' in 'github'). The fork's package.json contained a preinstall script that fetched and executed a remote shell script.

Step 3: Cache poisoning. The shell script deployed Cacheract, a GitHub Actions cache poisoning tool. It flooded the cache with over 10GB of junk data, triggering GitHub's LRU eviction policy and evicting legitimate cache entries. The poisoned entries were crafted to match the cache key pattern used by Cline's nightly release workflow.

Step 4: Credential theft. When the nightly release workflow ran and restored node_modules from cache, it got the compromised version. The release workflow held the NPM_RELEASE_TOKEN, VSCE_PAT (VS Code Marketplace), and OVSX_PAT (OpenVSX). All three were exfiltrated³.

Step 5: Malicious publish. Using the stolen npm token, the attacker published cline@2.3.0 with the OpenClaw postinstall hook. The compromised version was live for eight hours before StepSecurity's automated monitoring flagged it - approximately 14 minutes after publication¹.

A botched rotation made it worse

Security researcher Adnan Khan had actually discovered the vulnerability chain in late December 2025 and reported it via a GitHub Security Advisory on January 1, 2026. He sent multiple follow-ups over five weeks. None received a response³.

When Khan publicly disclosed on February 9, Cline patched within 30 minutes by removing the AI triage workflows. They began credential rotation the next day.

But the rotation was incomplete. The team deleted the wrong token, leaving the exposed one active⁴. They discovered the error on February 11 and re-rotated. But the attacker had already exfiltrated the credentials, and the npm token remained valid long enough to publish the compromised package six days later.

Khan was not the attacker. A separate, unknown actor found Khan's proof-of-concept on his test repository and weaponised it against Cline directly³.

The new pattern: AI installs AI

The specific vulnerability chain is interesting but not unprecedented. Prompt injection, cache poisoning, and credential theft are all documented attack classes. What makes Clinejection distinct is the outcome: one AI tool silently bootstrapping a second AI agent on developer machines.

This creates a recursion problem in the supply chain. The developer trusts Tool A (Cline). Tool A is compromised to install Tool B (OpenClaw). Tool B has its own capabilities - shell execution, credential access, persistent daemon installation - that are independent of Tool A and invisible to the developer's original trust decision.

OpenClaw as installed could read credentials from ~/.openclaw/, execute shell commands via its Gateway API, and install itself as a persistent system daemon surviving reboots¹. The severity was debated - Endor Labs characterised the payload as closer to a proof-of-concept than a weaponised attack⁵ - but the mechanism is what matters. The next payload will not be a proof-of-concept.

This is the supply chain equivalent of confused deputy: the developer authorises Cline to act on their behalf, and Cline (via compromise) delegates that authority to an entirely separate agent the developer never evaluated, never configured, and never consented to.

Why existing controls did not catch it

npm audit: The postinstall script installs a legitimate, non-malicious package (OpenClaw). There is no malware to detect.

Code review: The CLI binary was byte-identical to the previous version. Only package.json changed, and only by one line. Automated diff checks that focus on binary changes would miss it.

Provenance attestations: Cline was not using OIDC-based npm provenance at the time. The compromised token could publish without provenance metadata, which StepSecurity flagged as anomalous¹.

Permission prompts: The installation happens in a postinstall hook during npm install. No AI coding tool prompts the user before a dependency's lifecycle script runs. The operation is invisible.

The attack exploited the gap between what developers think they are installing (a specific version of Cline) and what actually executes (arbitrary lifecycle scripts from the package and everything it transitively installs).

What Cline changed afterward

Cline's post-mortem⁴ outlines several remediation steps:

Eliminated GitHub Actions cache usage from credential-handling workflows
Adopted OIDC provenance attestations for npm publishing, eliminating long-lived tokens
Added verification requirements for credential rotation
Began working on a formal vulnerability disclosure process with SLAs
Commissioned third-party security audits of CI/CD infrastructure

These are meaningful improvements. The OIDC migration alone would have prevented the attack - a stolen token cannot publish packages when provenance requires a cryptographic attestation from a specific GitHub Actions workflow.

The architectural question

Clinejection is a supply chain attack, but it is also an agent security problem. The entry point was natural language in a GitHub issue title. The first link in the chain was an AI bot that interpreted untrusted text as an instruction and executed it with the privileges of the CI environment.

This is the same structural pattern we have written about in the context of MCP tool poisoning and agent skill registries - untrusted input reaches an agent, the agent acts on it, and nothing evaluates the resulting operations before they execute.

The difference here is that the agent was not a developer's local coding assistant. It was an automated CI workflow that ran on every new issue, with shell access and cached credentials. The blast radius was not one developer's machine - it was the entire project's publication pipeline.

Every team deploying AI agents in CI/CD - for issue triage, code review, automated testing, or any other workflow - has this same exposure. The agent processes untrusted input (issues, PRs, comments) and has access to secrets (tokens, keys, credentials). The question is whether anything evaluates what the agent does with that access.

Per-syscall interception catches this class of attack at the operation layer. When the AI triage bot attempts to run npm install from an unexpected repository, the operation is evaluated against policy before it executes - regardless of what the issue title said. When a lifecycle script attempts to exfiltrate credentials to an external host, the egress is blocked.

The entry point changes. The operations do not. grith was built to catch exactly this class of problem - evaluating every operation at the syscall layer, regardless of which agent triggered it or why.

Container queries and units in action

2026-03-05T06:07:52Z

Published: October 23, 2025

One of the goals when writing CSS is to build component parts that will adapt well to different (and unexpected) contexts. Ideally, a component can be placed inside any "container" element without it feeling broken or out of place. How can you accomplish this in a complex layout like a store where the primary component—the "product"—has to fit into a variety of list layouts, including the sidebar?

Responsive typography with `cqi` units

The first step is to define some basic sizing variables that could be reused across the project—starting with whitespace sizes. But before creating any custom properties, the browser provides some useful named values as CSS units:

1em: the current font size.
1rem: the font size on the :root (html) element.
1lh / 1rlh: the current and root line heights.
1vw: the viewport width.
1vi: the viewport "inline" size (for English, this is the same as vw).
1cqi: the inline size of the nearest "container" (defaulting to the viewport).

You can think of these units as common variables provided by the browser, with a shorthand syntax for multiplication. If you wanted half of a --line-height custom property in CSS, you would need to write the entire calculation calc(0.5 * var(--line-height)), but with the lh unit, you can ask for 0.5lh instead.

Custom properties like --brand-color and --button-background have different meanings and serve a different purpose, even when they result in the same deepPink color (another browser-provided variable). Similarly, 1em might sometimes be equal to 16px, but that's not a stable relationship. Like any other variable, units should be used to express a relationship, rather than an expected value.

Both 1em and 1lh are font-relative units that could be used for spacing, but only one of them has a reliable relationship to the current line height. If this page involved a lot of elements with prose in them—paragraphs and lists, for example—the lh unit would work well for spacing between them:

p, ul, ol {
  margin-block: 1lh;
}

That will maintain a consistent baseline rhythm, without any extra work. But this page has almost no prose. Instead of spacing within a flow of text, this layout requires spacing to push text away from the edge of a card, and spacing between columns and rows in a grid or stacked layout. In this case there are several things to consider:

Multiples (or fractions) of 1lh may still be useful for maintaining vertical rhythm across the page.
The cqi unit would account for the amount of available space in a given context.

By combining these two units in a round() function, the --gap variable is based primarily on the container size, but rounded up to a multiple of quarter-lines:

html {
  --gap: round(up, 2cqi, 0.25lh);
}

If the line-height is 20px, then the --gap will be multiples of 5px—but the exact multiple will depend on available space. If either the line-height or available space change, the --gap variable will adapt to its new context. To make the --gap consistent across the entire design, you could replace the container-relative cqi units with viewport-relative vi units.

The same approach is useful when establishing "fluid" font sizes that respond to available space. This --body-text variable is based on the user-provided font preference, with some range to adapt based on the container. In this case, clamp() ensures the font will be at least as large as the user's preference, can grow some as the container size increases, but will stop growing at some point:

html {
  --body-text: clamp(1rem, 0.875rem + 0.5cqi, 1.25rem);
}

The range is clamped between 1rem and 1.25rem, to stay near the user-selected font size.
The cqi value determines how fast the font will grow or shrink in relation to available space, which is added to a rem value to offset that growth based on the user-selected size.

Keeping the central rem value near 1 and the added cqi value low ensures that users still have significant control when zooming in or out. You can adjust those two values, and then use browser zoom to see how they interact. The closer you get to 100cqi, and the farther you fall below 1rem, the less influence user font preferences will have—and the less font sizes will respond to zooming in or out.

The --item-title variable is slightly larger and more responsive, and the --list-title size responds to the viewport size (using vi) rather than the immediate container size. That way item headings respond to their context, but the main list headings all match in size no matter where they show up.

Defining containers to measure in context

At this point, container query units have been used to create adaptive typography on a web page, but new containers haven't been defined.

By default, 1cqi (1/100 container query inline size) is the same as 1svi (1/100 small viewport inline size) because the "small" viewport acts as the initial container for any web page. In order to take full advantage of the cqi unit, you need to define additional "containers" within the page. The primary layout containers on this page are the product-list and shopping-cart—so they are set to expose their inline-size.

product-list,
shopping-cart {
  container-type: inline-size;
}

Container query units—including cqi—aren't able to measure the element that they are used on. If you set the width of shopping-cart to 25cqi, it would be a paradox to determine the container's width based on its own width! Instead, the result will be based on the next ancestor container in the tree hierarchy that contains shopping-cart.

The product-detail cards are part of a product-list grid that can be changed by the user. The list layout option displays each card at full width. In that case, referring to the parent container size is useful, but both the small-grid and large-grid options cause the card to grow and shrink depending on how many columns fit into the container. Even when the container is quite large, the cards can remain tightly packed into smaller grid cells.

There's currently no way to declare those grid cells as "containers" directly. Instead, an extra element is needed to measure within each cell. That's why each product-detail instance has an <article> element nested directly inside. product-detail > article is the card to be styled, while product-detail itself is used only as a container to measure. That allows the cqi-based text and spacing calculations previously defined to be recalculated for the space available to each card.

Explicit container queries

Container units are powerful, but sometimes it's useful to make more dramatic changes in a component layout when the available size crosses a threshold. These are often called breakpoints—since the fix is applied at the point when a given layout begins to break. You may already be familiar with using @media to add breakpoints based on the viewport size:

main {
  display: grid;
  grid-template:
    'controls' auto
    'cart' 1fr
    'list' auto
    / minmax(min-content, 1fr);

  @media (width > 30em) {
    grid-template:
      'controls controls' auto
      'list cart' 1fr
      / 2fr 1fr
    ;
  }
}

Here, shopping-cart appears above the main product-list on small screens—but at a certain point, there's more horizontal space, and a sidebar makes more sense. Since that shift depends on the overall viewport, a media query is used to handle the change. However, the product-list and product-detail components might appear in different contexts, somewhat independent of the viewport size. When the viewport grows wider than the sidebar breakpoint, the shopping-cart component suddenly becomes smaller, providing less space for products inside. This is where container queries become necessary. The syntax is nearly identical to a media query, but uses @container rather than @media at the start of the rule:

article {
  display: grid;
  grid-template:
    'image' auto 'title' auto
    'summary' auto
    'button' auto / auto
  ;

  @container (inline-size > 40ch) {
    --image-ratio: 1;
    grid-template:
      'image title' auto
      'image summary' 1fr
      'image button' auto
      / minmax(50px, min(20%, 500px)) 1fr
    ;
  }

  @container (inline-size > 50ch) {
    grid-template:
      'image title title' auto
      'image summary button' 1fr
      / minmax(50px, min(20%, 500px)) 1fr fit-content(20%);
  }
}

The initial goal of this post is that components should be able to respond to any context. To make that work, each component defines its own internal behavior, without explicit knowledge of the surrounding components. Container queries help us accomplish that. The product-list and product-detail components don't need to be aware of why they have more or less space in a given context, they only need to know how much space is currently available.

Transition grid templates and visibility

With a layout that's changing often based on container queries, how do we smooth out all of the transitions? When using grids for layout, it's possible to animate the size of a column or row, as well as the gap between columns and rows. In this case, the cart area and gap are expanded from 0 width when the cart is opened. In order to animate grid templates like this, two things are required:

The initial and end states must have the same number of tracks (columns or rows).
The animated tracks must use comparable units.

While it would be possible to change from a one-column grid to a two-column grid when the sidebar is hidden, the empty sidebar column is instead resized to 0, and the column-gap is also set to 0. When the cart is open in the sidebar, the 0-width column transitions to calc(15em + 1cqi). Since the calculation results in a normal length value, the transition can be animated from length 0. The gap also animates from one length to another—from 0 to var(--gap)—which was defined earlier:

main {
  transition: grid-template 250ms, gap 250ms;
}

Non-Baseline animation enhancements

The shopping-cart and product-detail components are also animated when hidden or shown, and this is done using two recent features that are not yet Baseline, but work as progressive enhancements for browsers that do have support:

Applying interpolate-size: allow-keywords allows transitioning element dimensions from 0 to auto. This is used for transitioning the products to and from a block-size of 0 when they are added or removed from the cart. Since the interpolate-size property inherits, that only needs to be defined once on the <html> element, and it is available to every other element on the page. This is only supported in Chrome-based browsers (Chrome, Edge, and others), but can be used as a progressive enhancement. The fallback works as expected, just without the animated transition.
"Discrete" properties like display can now be transitioned as well, even though there are no intermediate values between grid and none. Instead the transition is applied at the start or end of the duration provided. To achieve that, allow-discrete is added to the transition-behavior property. While transition-behavior is otherwise Baseline, animating the display property is not yet supported in Firefox. But again, this works well as a progressive enhancement.

There are many situations where container queries and units can be used to replace media queries and viewport units, and that's great when it helps you express the intent of a design more clearly. But one is not meant to replace the other, and there's not a better query or unit that will work in every situation. CSS works best when the relationships established in code match the goals and purpose of the design. When you want text and spacing that is relative to immediate context, container queries provide that functionality, but when you want consistent sizing relative to the overall viewport, media queries and viewport units are still an excellent option. Most websites will likely involve a mix of both.

rarouš.w3b

Native Apps Should Be Avoided Whenever Possible

The White House App

The Software Embedded in Apps

Why Everyone Should Care

What Apps Can Do That Websites Can’t

The Access Provided by Default is Enough to Do Real Harm

Some Things Need to Be Apps, But Most Don’t

Conclusion

References

AI dělá chyby a autisté je dokážou najít. Z jejich jinakosti dělá Češka ve svém startupu přednost

Using safe-area-inset to build mobile-safe layouts

Environment variables for safe area insets

Browser support

Using safe-area-inset values

Which web pages actually need this?

Safe-area-inset doesn't provide margins

Safe-area-inset only has non-zero values on mobile devices

Polypane's device emulation supports safe area insets

Solving a specific issue: Floating buttons

safe-area-max-inset

Browser support for safe-area-max-inset-*

Testing safe areas in Polypane

What to take away

Design Token Naming Conventions: A Practical Guide

Design Token Naming Conventions: A Practical Guide

Why naming gets hard faster than expected

What are design tokens, a quick recap

The three token tiers

Tier 1 - Primitive tokens

Tier 2 - Semantic tokens

Tier 3 - Component tokens

Common naming conventions

1. Category-Property-Modifier (CPM)

2. Tier-based naming

3. Object-Property-Modifier (OPM)

4. Context-first naming

5. A Comprehensive Structure

Handling variants, states, and modes

The same decision in different conventions

Variants

States

Scale sizes

Modes and themes

Numbered scales

Rules that keep token naming healthy

1. Consistency beats perfection

2. Optimise for clarity over brevity

3. Keep semantics semantic

4. Order from broad to specific

5. Make names self-explanatory

6. Design for growth

Choosing your convention

Keep naming alive with lightweight governance

Upgrading Forgejo with S3 Object Storage and Actions!

Migrating to S3 Object Storage

Setting up Forgejo Actions

Conclusion

Rebuilding My Home Lab

The Plan

Current v2 Cluster Specs

Cluster v1

Cluster v2

The OS

Networking

Distributed File System

Docker Swarm

Keepalived

HA Caddy

Cluster v3

How to smoke

Crashing hard: why talking about bubbles obscures the real social cost of overinvesting into “Artificial Intelligence”

Overinvestments shape technological paths

“AI” investments make harmful things cheaper, speed up the commodification of human labour and shift social norms.

The “AI” path dependencies we will not correct

Why “AI” overinvestments might take a long time to unwind

There are a few reasons to suspect an even longer and more painful struggle than the dot-com crash: market concentration, closeness to governments, and financial actors being invested into “AI success”.

What to do and why not to despair

Wrap Text Around Images with CSS shape-outside

Make your text wrap around images perfectly.

`safe-area-max-inset`

Browser support for `safe-area-max-inset-*`

`font-family` Fallbacks Are Self-Contained

Responsive typography with `cqi` units