Last week, we reported on Anthropic seemingly being on a speed run to break devs’ goodwill by silently “nerfing” Claude Code, banning corporate accounts without warning, and a weird growth experiment involving revoking Claude Code and then restoring it. This week, a dev on the $20/month Pro plan had Claude Code removed just days into their subscription:

This week, Anthropic announced a big data center expansion, and relaxing previous usage limitations, while Elon Musk’s SpaceX / xAI ( a single company after a merger) is renting its complete Colossus 1 data center to Anthropic. From the announcement:

“Colossus 1 features over 220,000 NVIDIA GPUs, including dense deployments of H100, H200, and next-generation GB200 accelerators. The cluster delivers extreme parallel performance for large language models, multimodal systems, scientific simulations, and generative AI at frontier scale.

Anthropic plans to use this additional compute to directly improve capacity for Claude Pro and Claude Max subscribers.”

In parallel with this release, Anthropic announced:

• Doubling Claude Code’s current 5-hour limits for Pro, Max, Team, and seat-based Enterprise plans
• Removing peak hours limit reduction on Claude Code for Pro and Max plans
• Substantially raising API rate limits for Opus models

Is it possible that capacity issues are what led Anthropic to make Claude worse? It’s confirmed the company has struggled with capacity for months. Conveniently, Claude Code being “nerfed” led to lower compute load, while removing Claude Code access from cheap plans could look like rate limiting. Even the banning of corporate accounts could be seen as scaling back at a time when the business has struggled to serve existing growth. Yesterday, (6 May), at the Code with Claude event hosted by Anthropic, CEO, Dario Amodei, said:

“We originally planned for 10x growth, and we’ve seen something more like 80x growth in revenue and usage over the last period of time.”

SpaceX / xAI renting a good chunk of its capacity to Anthropic is ironic, considering that xAI (Musk’s AI startup) builds Grok, a frontier model and direct rival of Claude, and also in January, Anthropic banned xAI developers from Claude. As covered at the time:

“It’s common for an AI lab to not allow another AI lab to use its model, like at OpenAI, Anthropic, and Google. On the other side, there’s also the pertinent question of why a leading AI lab would even want to use a rival for its own day-to-day work?

Turns out, xAI (Elon Musk’s AI lab) was relying on Cursor to write code, which we know because they got cut off.”

Anthropic likely banned xAI to stop Claude from being potentially distilled while it tried to improve Grok’s coding capability. Meanwhile, Musk called Anthropic “misanthropic and evil” earlier this year, and said the new tenant “hates Western civilization”. But both parties seem happy to put that behind them and strike a deal, so perhaps there’s something else at play.

Could SpaceX / xAI be checking out of the frontier-AI model wars? Leasing a good chunk of its data center capacity might suggest that. SpaceX / xAI has two data centers: Colossus 1 and Colossus 2. Colossus 1 represents somewhere around 45% of current SpaceX / xAI capacity, and 20-25% of planned total capacity.

Giving up as much capacity as this might indicate a lack of demand, or capacity sitting idle. It also means Grok is losing out in market share to Claude, ChatGPT, and other leading models. In February’s AI tooling survey we found scarce mention of Grok, which lagged in usage behind open models like DeepSeek and Qwen.

To be fair, unlike Anthropic and OpenAI, Grok never had a B2C nor B2B business that took off. The biggest consumer use case for Grok seems to be its integration into the social media platform, X; at least, I don’t know of any tech company using the model for serious work.

“The enemy of my enemy is my friend”, says the maxim, and if there’s one company Musk hates, it’s OpenAI. He is currently suing OpenAI, claiming it betrayed its founding nonprofit mission to develop safe AGI for humanity’s benefit by shifting to a profit-driven model backed by Microsoft. Musk also claims that despite investing about $40M, he has no ownership of the company.

He wants $150B in damages, the removal of Sam Altman and Greg Brockman, and for OpenAI to return to a full nonprofit, as per when he invested in the company. We covered more about OpenAI’s own ethical challenges between nonprofit and for-profit right after the firing of Sam Altman in 2023, in the deepdive What is OpenAI, really?

Similarly, Anthropic may well have an issue with OpenAI, if CEO Dario Amodei’s failure to join hands with Sam Altman while sharing a stage with the Prime Minister of India earlier this year is anything to go by.

Capacity issues hurting Anthropic would benefit OpenAI, and so by offering significant capacity to Anthropic, Musk is making it harder for OpenAI to win the market. That would be ironic, given he’s a former investor.

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse. This week’s issue covers:

1. Forward deployed engineering heats up again. Massive demand for the role at Google, OpenAI, and Anthropic. The latest version of the FDE role looks like the consultant / solution architect role done by many early-junior engineers.
2. Why are layoffs spiking? Tech job cuts are higher than since early 2023 for various reasons: smaller teams prompt reorgs and reduce the need for middle management. Meanwhile, poorly performing companies make layoffs without the influence of AI.
3. New trend: self-reporting 100% AI generated code at Microsoft. With mid-year performance reviews looming, some managers advise their reports to claim they use AI for everything.
4. Industry Pulse. Tokenmaxxing at Amazon, too, SaaS companies grow faster than before – perhaps partly due to AI, Bun rewritten in Rust with AI works well, Anthropic overtakes OpenAI in enterprise spend, and more.
5. Vibe coding & agentic engineering get uncomfortably close. A relatable observation by software engineer, Simon Willison, about reviewing AI agents’ code less than would be ideal.

Pay transparency has always been an issue in tech, especially in Europe. For a while, I assumed that the most that a senior+ software engineer could make in London or Amsterdam would be in the realm of £100K / €100K. Once you reach that level, you've made it. You’re now at the very top of the market! Or are you?

So when I was making £93K in London, working as a principal engineer at Skyscanner in 2016, I was not expecting that I could be compensated meaningfully better. Pay surveys kept confirming that I'm well above the median, and into the 90th percentile of pay grades.

Imagine my surprise when I got an offer from Uber, in Amsterdam, that effectively doubled by compensation, into the realm of around €220-250K ($260-295K). By year four, I made €283K ($332K):

It felt like I discovered a "secret, upper-tier" of the market that no one else knew about. When I became a manager at Uber, and started hiring for my team, several strong software engineers were hesitant to move forward with the process, because they assumed that they were at the very top of the market – but they still made ~half of what we would have offered! I had no way of telling them "your data is wrong, this place pays a lot more!" and so several of them just never bothered to interview, assuming the most raise they would get would be 5-10%. When they could have potentially doubled their compensation…

I saw first-hand that not having good compensation information works against us, developers, and decided to try and change this. I collected data points from closer to 200 engineers working in the Netherlands, and explained that there's a third, "hidden" tier of compensation in The Trimodal Nature of Software Engineering Salaries in the Netherlands and Europe.

After the success of the article, I decided to "open source" compensation data points I collected, and thus TechPays was born:

I built this site together with Zsombor Erdődy-Nagy. We paid attention to support compensation anonymization, capture freelancer compensation, and break down how compensation packages were put together. We've received so many heart-warming stories on how you've been able to negotiate better compensation packages, thanks to having access to this information.

Knowing that we're making a difference kept us going for a few years, as a side project. However, over time, both Zsombor and I got busier with other projects. For me, it was The Pragmatic Engineer taking up more of my time. We wanted to find a way to keep TechPays running, and get the care it deserves.

Levels.fyi will be taking over operating TechPays – and taking learnings about European compensation packages, and integrating into their global pay transparency platform. I've known Levels.fyi founders Zuhayeer and Zaheer for years, and we share our drive to make compensation as transparent as possible, across the tech industry.

With TechPays, there are no changes: you get to browse the data, as before. And expect even more, high-quality data points on Levels.fyi, for Europe, and globally.

To get more details on compensation, check out Levels.fyi. And read the Trimodal nature of tech compensation in the US, UK and India, based on Levels.fyi data points:

GitHub’s reliability has been beyond unacceptable recently: last month, third party measurements pinned it at one nine (right at 90%). This month, reliability has been down to zero nines – 86% – as per a third-party tracker, and last week, things got even worse: a frankly embarrassing data integrity incident, more outages, and a partial explanation from GitHub, eventually.

Data integrity incident

Last Thursday (23 April), this happened: PRs merged via the merge queue using the squash merge method produced incorrect merge commits, when the merge group contained more than one PR. Commits were reverted from subsequent merges: basically, commits were “lost” in the code that was merged!

Thanks to a bug GitHub introduced, the service broke its integrity promise that pull requests would be merged as expected when using squash merge, which is a technique typically used to merge multiple small commits into a single, meaningful commit. This is a big deal: as data integrity promises are some of the most important ones, for services like GitHub.

A total of 2,092 pull requests were impacted, and companies hit by the outage included Modal and Zipline. Effectively, GitHub pushed a bunch of work on affected customers who had to manually untangle and recover lost commits, which GitHub could offer zero assistance with.

Customers had to manually go through their git history and restore missing code. After following manual recovery steps (reverting the squash commit and re-applying commits one by one), all commits should have been recovered.

GitHub later emailed the list of affected commits to customers, but it’s odd that GitHub executives seemed to downplay the nature of this outage. After all, an outage that messes with data integrity is a much bigger deal than something like a fall in availability where no data is corrupted.

Can Duruk, software engineer at Modal, was unhappy about GitHub’s muted response to the outage:

“The COO going out of their way to find a huge denominator to make the impact appear small feels very dishonest; versus a sincere apology about how this invalidates their entire promise to their customers. We had to dig into their status page about this to even realize they just casually f***ed up our repo.”

Outages don’t stop

On Monday (27 April), pull requests and issues disappeared from GitHub’s web UI:

This had to do with an Elasticsearch outage on GitHub’s backend: the cluster became overloaded and went down. So, while pull requests, issues, and projects didn’t vanish altogether, they also didn’t show up during the 6-hour-long outage.

There were other outages this week:

• Some pull requests not showing up (Tuesday, 28 April)
• Problems with some GitHub Actions (the same day)
• Incomplete pull requests in repositories (Wednesday, 29 April)

Also on Tuesday (28 April), security firm Wiz disclosed a critical security issue, where a bad actor could get access to all repositories on GitHub and GitHub Enterprise server by using only a git push command. GitHub fixed the issue on GitHub.com within six hours, but GitHub Enterprise servers that were not updated remain vulnerable.

Famous open source contributor quits GitHub in frustration

On Tuesday, Mitchell Hashimoto, founder of HashiCorp, creator of Ghostty, announced GitHub was unfit for professional work and that he was moving off to Ghostty, the open source terminal that’s his main focus. Mitchell’s reasoning was dead simple: being on GitHub makes him unproductive (emphasis mine:)

“The past month I’ve kept a journal where I put an “X” next to every date where a GitHub outage has negatively impacted my ability to work. Almost every day has an X. On the day I am writing this post, I’ve been unable to do any PR review for ~2 hours because there is a GitHub Actions outage. This is no longer a place for serious work if it just blocks you out for hours per day, every day.

It’s not a fun place for me to be anymore. I want to be there, but it doesn’t want me to be there. I want to get work done and it doesn’t want me to get work done. I want to ship software and it doesn’t want me to ship software.

I want it to be better, but I also want to code. And I can’t code with GitHub anymore. I’m sorry. After 18 years, I’ve got to go. I’d love to come back one day, but this will have to be predicated on real results and improvements, not words and promises.”

Mitchell’s experience suggests that GitHub’s official status page is inaccurate from the point of view of a heavy user like himself. The third-party “missing GitHub status page” is likely to be a better estimation: where GitHub’s reliability is at zero nines: at 85.51% uptime. That means that a part of GitHub was down for 2-3 hours, per day, on average, for the last 90 days (!!)

Mitchell’s complaint sounds straightforward:

1. As a professional software engineer, it’s important to have tools that help you get work done
2. For months, GitHub has got in the way of his work on open source projects via a flood of outages
3. It makes no sense to use a product unfit for professional work.
4. As GitHub shows no signs of improvement, it’s worthwhile to move to a different solution which just works

CTO blames AI agent-fuelled load spike

GitHub CTO, Vlad Fedorov, shared an update on why reliability has been terrible for months at GitHub. He identified the load from agents being much bigger than expected as the culprit. Charts illustrating this were shared by GitHub:

This chart looks eye-catching – but there’s just one tiny issue: no Y axis! So, while it tells the story of the load going up slowly and then very fast, we’re not told by how much. However, I managed to get data from GitHub, and below is the chart showing the actual load increase over two years:

A load increase of ~3.5x, spread across two years, doesn’t seem so brutal at first glance. It is nothing like a load increase of 10x in a month, and a good chunk of it occurred in recent months. So, why can’t GitHub handle it? In a blog post, Fedorov said:

“A pull request can touch Git storage, mergeability checks, branch protection, GitHub Actions, search, notifications, permissions, webhooks, APIs, background jobs, caches, and databases. At large scale, small inefficiencies compound: queues deepen, cache misses become database load, indexes fall behind, retries amplify traffic, and one slow dependency can affect several product experiences.”

Here’s how the per-second load numbers from January 2023 and today compare:

GitHub took 15 years to achieve the 2023 numbers, and maybe it expected to continue growing in a comparable way in the future. If so, some engineering decisions about long-term infrastructure improvements would have been made obsolete by the arrival of AI agents.

To add to GitHub’s challenges, the company is in the midst of a migration from its own data centers → Azure. In October last year, GitHub started to move over to Azure – a project expected to take 12 months – because it already had constraints on its own data center capacity.

Such large-scale infrastructure migrations are hard enough when the load on a service is relatively stable; just making sure nothing breaks takes a lot of effort. But moving at a time when load is spiking means that bugs can cause more visible outages. Of course, GitHub can secure a lot more compute capacity on Azure, now they know what to expect.

But other major companies prepared for a 10x increase in infra load, so why not Microsoft / GitHub? A year ago, I did research on how Big Tech was preparing to respond to the impact of AI on their business. Google was improving its internal systems to accommodate for a 10x increase in load. As we covered in The Pragmatic Engineer, in July last year:

“Google is preparing for 10x more code to be shipped. A former Google Site Reliability Engineer (SRE) told me:

“What I’m hearing from SRE friends is that they are preparing for 10x the lines of code making their way into production.”

If any company has data on the likely impact of AI tools, it’s Google. 10x as much code generated will likely also mean 10x more: code review, deployments, feature flags, source control footprint and, perhaps, even bugs and outages, if not handled with care.”

Predicted enormous load increases were not secret knowledge within the industry, yet it seems GitHub was blissfully ignorant of their potential size. According to Vlad, GitHub did eventually plan for a need to increase capacity by 10x, but this was in October 2025, months later. In February 2026, the company is now adjusting that expectation to 30x. He wrote:

“We started executing our plan to increase GitHub’s capacity by 10X in October 2025 with a goal of substantially improving reliability and failover. By February 2026, it was clear that we needed to design for a future that requires 30X today’s scale.”

There’s also the question of whether GitHub miscalculated how much time it had to prepare for explosive load growth, and whether it was caught off guard when that growth materialized months sooner than expected at the start of this year.

Given GitHub only started to prepare for a major load increase in October, its current problems are unsurprising. At the scale of GitHub, it’s common enough for each team owning a service to plan a year ahead on how much load their service will have, and hardware resources like storage, VMs, and networking are allocated accordingly. Load planning can account for up to half of the preparations, and when reality doesn’t conform to plans, some systems can struggle to scale up.

So, on one hand, dealing with a 3.5x increase in load over 2 years should not be such a big deal for most services; especially not ones which can be horizontally scaled (when there’s not much state, and scaling is achieved simply by adding new nodes.) But GitHub probably stores a lot more state with pull requests, workflows, projects, etc. This probably makes scaling more tricky when it comes to databases and systems running workflows.

GitHub also has 18 years of tech debt on its hands, and thousands of staff to align as “organizational overhead.” As its service load grows faster than before, responding is harder due to all that accumulated “debt”:

• Tech debt: many systems at the company are 10+ years old and are likely patched up, making them more difficult and risky to change
• Organizational debt: around 4,000 people work at GitHub, of whom 1,000 are engineers. Teams have dependencies with each other, and even seemingly simple work can require dozens of engineers to work together
• Customer expectations: GitHub cannot break customer workflows, even if doing so would mean changes to systems happen faster

GitHub finds itself in the ‘innovator’s dilemma’: the company became successful because it built developer workflows that made sense, pre-AI, and it used to be able to accurately forecast service load changes. But now that engineering teams’ workflows include AI agents, GitHub’s own workflows are not necessarily the best fit, and the company failed to forecast service-level changes.

Other vendors floored by AI load? Not really

One thing that doesn’t add up about the situation is that other vendors who are presumably experiencing similar load spikes don’t appear to be suffering with reliability issues as much. Vercel, Linear, Resend, Railway, Sentry, and other infra providers see record-level growth thanks to AI, but keep up with the load.

Yes, it’s true that AI vendors like Anthropic, OpenAI, and Cursor have some reliability issues, but it’s not at the scale of GitHub’s. GitHub’s direct competitors, GitLab and Bitbucket, presumably see load going up similarly, but they’re not going down as much.

An obvious question is how much of GitHub’s pain is self-inflicted? With Microsoft as owner, it has more resources at its disposal than any competitor or startup, and yet failed to predict load increases and is too big to respond with the nimbleness of a startup.

It’s undeniable that solving for a major load increase is a hard challenge; it’s when the difference between average and standout engineering teams is apparent. GitHub hasn’t been responding like a world-class engineering org.

GitHub alternatives?

Every regular user of GitHub feels the pain of ongoing outages. As a dev, you can either hope Microsoft will eventually improve reliability, or seek alternatives. As covered above, Mitchell has chosen to quit and is currently deciding where to take Ghostty.

The obvious alternatives are GitHub’s biggest competitors, GitLab, and Bitbucket. Each offers Git hosting, and neither comes with the uptime woes that GitHub is suffering from.

Self-hosted solutions are also an option, like self-hosting your git repo, or going with a self-hosted forge like Forgejo, which is an open source, local-first GitHub alternative.

I also suspect that, soon enough, we’ll see startups offering GitHub-like code hosting capabilities, while offering more robust uptime and being architected to handle the 30x-or-more scale which GitHub hopes one day to support.

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse. This week’s issue covers:

1. Did Anthropic turn hostile on devs because capacity was running low?
2. Amazon finally allows Claude Code and Codex usage
3. Meta forcefully assigns engineers to data labelling ahead of job cuts
4. New trend: small “AI-forward” teams
5. Industry Pulse: why Meta tracks employees’ computer activity, OpenAI starts to move off Datadog, Apple lets slip it uses Claude Code, GitHub → Xbox transfers at Microsoft, VS Code inserted “coathored by Copilot” even when Copilot did nothing, analysis of the Coinbase layoffs

Last week, we covered the slightly perverse trend of “tokenmaxxing” across the industry, where devs run agents with the sole aim of boosting their personal “token stats” in an effort to rank higher on internal token leaderboards, and not be seen as a Luddite who doesn’t use AI tools enough compared to peers.

This week, I spoke with a software engineer at a large company and another at a seed-stage place. Both shared almost identical stories: at their latest all-hands, company leadership expressed concerns about the fast-rising costs of tokens. At both places, token spend has increased by ~10x in the last six months – with no signs of slowing down.

I wanted to find out about this trend, so I talked to devs at 15 businesses. Below is what I learned about what’s happening in workplaces of all sizes. Names are anonymized.

Large companies

Setting the default model to a cheaper one: 10,000+ person SaaS company, offices on all continents

Inside a large SaaS company, most devs use an internal background coding tool for coding. This model defaults to Claude Sonnet, which is the cheaper Claude version. Model selection is not persisted, so devs who prefer working with Opus, for instance, must reselect it on every subsequent startup.

This tool supports all major frontier models such as Sonnet, Opus, GPT, and Gemini. Devs at the company whom I talked to are very heavy users of the tool and have not encountered usage limitations.

Fintech company, US, Series D, ~8,000 people. Staff engineer:

“The cost in token spend is off the charts – and leadership has shared this trend with us. They have not said anything beyond showing growth in spend, and mentioning that this won’t be sustainable. So, nothing specific yet, but my sense is that something will have to change. Limits or prioritizing cheaper models, cutting back on hiring? Who knows.”

Infra company, US, publicly traded, ~5,000 people. Engineering Director:

“We’re monitoring but not restricting. We are spot checking the heaviest users, but we are seeing the business cases working out.

We are offering some guidance on model selection - e.g., turn off the new high-effort setting in Claude. Some users are trying open source models – but open source model usage is a bottom-up initiative, not a top-down one.”

Information technology, US, 10,000+ people. Director of Engineering:

“We have already had to raise our API budget limits multiple times in April. We recently switched to a much higher-effort level for Claude, which significantly increased the cost per PR.

One reason for the cost spike is using state-of-the-art models for demanding tasks. We are using that high-effort setting even for fairly trivial tasks that could have been handled by much cheaper models, or even by lower-effort Claude loops. Despite a few of us pointing this out, leadership has basically said budget is not the concern right now.

I sense that the budget increase has not been forecasted, and we’re in for a reckoning. I suspect the attitude changes once finance and other cost-conscious parts of the org realize we are spending hundreds of dollars per day, per highly-engaged developer. For now, fear of missing out and not wanting to fall behind seems to be outweighing cost discipline.”

Games studio, US+Europe, ~5,000 people. Senior developer:

“What budget increase? It’s very hard to get a budget for AI here! Claude Code is still not rolled out because $200/month/dev is seen as too high a cost. I talk with people at startups where $1,000/month in spending is totally normal, and it’s night and day here.”

Fintech company, US+Europe, late stage, ~5,000 people. Staff engineer:

“Some developers are now spending $500 a day (!!) on Claude Code. Practically speaking, this means that employee costs have doubled. Productivity has increased, in my view, but now the bottleneck is code reviews. AI can spit out code quite quickly, but we still have human reviews in place. Leadership encourages using AI for code review, but my team will not blindly trust AI.

The push from AI is coming from the top. This year’s performance review had a section on AI, rating devs by how well they used AI, so this is another reason everyone just uses it as much as they can.”

Mid-sized companies

SaaS industry, US, ~2,000 people. Dev Productivity Lead:

“Model routing helped keep our costs growing less dramatically. For example, changing the default model reduced cost by 30%. This is our strategy with AI spend, summarized:Short term: spend, spend, spend! Experiment and use whatever models make sense.Measure the impact. Measure key outcomes and report on spend, monthly.When spend vs results diverge: adjust. When our spend increases dramatically, but outcomes don’t follow: see what we can do to adjust the delta. More spend should mean better outcomes. If not, we are doing something wrong.”

Finance industry, US, ~2,000 people. VP of AI:

“We have Cursor and Claude Desktop, both of which have around 800-1,200 total users. Token usage is growing somewhat unexpectedly. Estimates are being adjusted on the fly; the initial plan to have strict limits (say, $100 per user) is breaking when reality hits, and people exhaust them in 3-5 working days.

Using expensive models is a problem. In regards to Cursor, many devs are defaulting to the most expensive models without realizing that going with Opus gives single percentage gains in intelligence compared to Sonnet, for example, while exhausting their budgets almost immediately.

We are working on blocking/managing out the most expensive models [with Cursor], as going into thousands of dollars per user, per month is not sustainable on our scale. Cursor is a good partner and we’re working with them to switch to a “pooled spend” model where heavy users can tap into a pool of extra spend.

Claude is a similar story. We were at $100 of Claude Desktop limit for everyone, but as we are moving forward, I can see that we would need to go much higher, especially for business-critical use cases.”

Infra company, US, late-stage, ~700 people. Founder:

“We haven’t had much of an issue. Most folks police themselves for runaway costs; for example, we had someone hit like $10K in a week because they messed up caching, but it was caught and they corrected their harness.

For the most part, we don’t see our high-end folks spending more than ~$1K/week. Now, to be clear, this is not a small amount! BUT it’s already a small subset of the population.

We’re just factoring it into engineering costs at this point: if it’s, say, $2K/month per employee, that’s $24K per year.

Who cares, then, when engineers already cost $200-400K/year in cash comp? Okay, so what if it’s $5K/month. That’s $60K/year.

Our bet is that token costs will stabilize and we’ll eventually end up with local-ish models.

Now, it could be five years before they stabilize, but overall, spend today isn’t that insane to me.

There’s a lot of people who are just dumb about it, but most legit execs push back on this. Take the Ralph loops or other insanity where someone spends $1K/day, $5K/week or stuff like this. That’s all just people being fools thinking they’re doing “R&D,” or somehow that they’re smarter than everyone else, but they’re just producing junk that never ships or is not useful.

We saw a bit of “stupid overspend” in the first couple months, but that’s all gone now. Costs could go up even more if we would “crack the whip” in wanting to see even more output, but we’re not doing that.”

Healthcare industry, US, ~500 people. Senior engineering manager:

“We are not holding back on spend, and have a monthly spend leaderboard. And we WANT devs to spend more on tokens! For example, one of my engineers spent $1,400 on a long Claude Code session in a single day.

We are seeing massive leverage, and we do more with the same number of people. This is why we are okay with our spending spiking. Our traffic is growing more than 10x, year-on-year, and we have managed to keep things running with the same team, and these AI tools.

Engineering is now blocked on Product and Design – which never happened before! This is how fast execution has become. We now have Staff+ engineers writing Product PRDs so we can move faster.

I’ve been in tech for close to 15 years and I never saw dramatic change like this. I just came back after a 3-month break, and every single thing is different in my day! I feel these AI agents are the biggest change in the industry since high-level languages became widespread.”

E-commerce company, US & Europe, ~2,000 devs. Head of Engineering:

“The increase in spend is INSANE. It’s about usage going up, with no signs of stopping. Usage is off the charts.

We currently do not have limits in place, and are not pausing now. Our CEO is AI-pilled and won’t let us slow down.

We do buy tokens at a discount. They start from 5% and go up with usage with the vendors we use (the usual suspects.)

We don’t let devs use anything lower than Opus 4.7 for coding. Cheaper models might work better, but a slight error pushed to prod would result in hours of toil.”

Small companies

Series A, US, ~50 people. Principal Engineer:

“About 15 devs are heavy users of AI and costs are rising very fast. Almost everyone uses Claude and Claude Code. We are considering four potential options:Increase AI budget, and start measuring more. Continue doing what we are, but allow devs to use more tokens instead of hiring limits. The precise ROI is hard to quantify, but we’ll start to measure and track both AI adoption and impact.Optimize token consumption. Use cheaper models for simpler tasks, review token usage, and see where we can cut usage. Downside: this approach could become one with diminishing returns, fast.Integrate more AI providers in the company. Find wrappers to abstract LLMs. The problem is: how do you replace Claude Code, for instance?Pivot to local models: such as Kimi, Qwen, and so on. The problem is it’s a big investment in high-end hardware or cloud GPUs. Upside: it offers better long-term cost control, once done.

We are likely to go with option #1: increase spend BUT maintain momentum and put the right measurements in place. We can do #2, #3 and #4 later. But if we kill AI usage momentum inside the company, the outcome will probably be worse.”

AI infra, US, seed stage, ~15 people. Founder:

“We saw a 15x increase in 6 months:Six months ago our spend per developer was ~$200/monthToday, it’s around $3,000/developer/month, for our seven devs
We’re not slowing usage, especially as we are building an AI infra product. The increase was much faster than expected, though.”

Small, bootstrapped company, Europe. Founding engineer:

“Our current strategy in dealing with the increase in costs is to switch to a cheaper model; unfortunately, from Opus to Sonnet in our case. That said, Sonnet is quite decent.”

How businesses manage token spend

Regardless of company size, there seems to be two strategies for how companies deal with increased spending. A summary:

Strategy #1: “let it rip and start measuring.” Around half of respondents say AI spend is rising dramatically, and they have decided to do nothing about it. They want devs to use AI as much as it makes sense to, and to help the work as much as possible.

However, because the cost is rising dramatically, these companies are now starting to measure usage and attempting to measure the impact of their AI tools.

There’s a few companies where the impact seems to be very positive, already. Smaller startups whose business is exploding in numbers of customers, load, and revenue, see that they don’t need to hire more staff because existing engineers can keep supporting the growth with AI tools.

Strategy #2: curb spending. Commonly mentioned cost-saving approaches:

• Use cheaper models for simpler tasks
• Set default models to less capable ones
• Set a spending cap and make it hard for engineers to exceed it, or require consent for doing so

Most companies using strategy #1 have briefly considered going with this approach, but threw it away, because they see this approach as optimizing on the wrong thing: cutting costs before the productivity impact of using state-of-the-art tools is even known!

Discounts exist when the spend is in the millions of dollars. I asked several people if they are getting discounts from vendors when buying tokens at scale. There were no exact numbers, but this is what I gathered in aggregate about possible custom agreements:

• Cursor: open to discounts above a few million dollars in spend. Companies have negotiated discounts with Cursor after crossing $1M of spending. Some companies negotiated tiered discounts from this level, starting at 5% and going higher as their spend goes up.
• Anthropic: no discounts. I talked with companies spending $5M+ per year on Claude which have received no discounts. If Anthropic offers discounts, it will likely be at a much higher tier.
• All discounts are custom, so try to negotiate – it’s free! Pricing discounts are on a per-customer basis, and highly custom. The easiest way to see if a discount is available is to ask the vendors!

—-

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse. This week’s issue covers:

1. Load from AI breaks GitHub – but why not other vendors? GitHub’s reliability is less than one nine, and getting worse. Prolific open source contributor, Mitchell Hashimoto, is quitting GitHub because he thinks it’s not suited for professional work. GitHub’s leadership blames the 3.5x increase in service load as the cause of degradation – or it might be self-inflicted.
2. Anthropic’s speedrun to destroy trust. Anthropic could do no wrong until recently, but in the past month, that’s all changed. Silently nerfing Claude Code, banning companies from Claude, and baffling price rises all add to a sense that Anthropic is in its “extraction” era of generating more revenue for the same or worse service.
3. Industry pulse. Dramatic price increases at GitHub Copilot, explosive growth at Codex, Google scrambling to build a good coding model, Cursor might be bought by SpaceX, AI agent deletes car business, and more.
4. Mitchell Hashimoto & the “building block economy.” Ghostty’s creator finds that open source “building blocks” are the best way to win massive adoption by software components – but it’s got harder to build a business on top of open building blocks.

Inside Meta, an engineer created a “token leaderboard” that ranks employees by token usage. Last week, The Information reported:

“Employees at Meta Platforms who want to show off their AI superuser chops are competing on an internal leaderboard for status as a “Session Immortal”— or, even better, “Token Legend.”

The rankings, set up by a Meta employee on its intranet using company data, measure how many tokens — the units of data processed by AI models — employees are burning through. Dubbed “Claudeonomics” after the flagship product of AI startup Anthropic, the leaderboard aggregates AI usage from more than 85,000 Meta employees, listing the top 250 power users.

The practice is emblematic of Silicon Valley’s newest form of conspicuous consumption, known as “tokenmaxxing,” which has turned token usage into a benchmark for productivity and a competitive measure of who is most AI native. Workers are maximizing their prompts, coding sessions and the number of agents working in parallel to climb internal rankings at Meta and other companies and demonstrate their value as AI automates functions such as coding.”

I spoke with a few engineers at Meta about what’s happening, and this is what they said:

• Massive waste. Plenty of devs are running an OpenClaw-like internal agent that burns massive amounts of tokens for little to no outcome.
• Outages caused by AI overuse. A dev mentioned that some SEVs were caused by what looked like careless AI code generation; almost like a dev behind the SEV was more concerned with churning out massive amounts of code with AI than with product quality.
• Gamified leaderboard. Those at the top of the leaderboard produce throwaway, wasteful work. This is painfully clear to anyone who checks Trajectories (AI prompts), which can be viewed.

As per The Information, Meta employees used a total of 60.2 trillion AI tokens (!!) in 30 days. If this was charged at Anthropic’s API prices, it would cost $900M. Of course, Meta is likely purchasing tokens at a discount, but that could still come in at $100M+ – in large part from senseless “tokenmaxxing”.

After backlash on social media, Meta abolished the internal leaderboard last week. One day after The Information revealed details about the incredible tokenmaxxing numbers, I confirmed that Meta has taken down its leaderboard; perhaps they realized that the incentive created enormous and unnecessary waste. If so, it’s a bit surprising that it took media coverage for the social media giant to reach that conclusion.

One engineer at Meta told me they think Meta had a different goal with the token leaderboard. A long-tenured engineer suspects increasing AI usage actually was the real goal. They said:

“Putting a leaderboard in place was always going to incentivize much more AI usage. And more AI usage means producing a lot more real-world traces. These traces can then be used to train Meta’s next-generation coding model better.

I believe this was the goal, even if no one said it out loud.

It’s an expensive way to generate data for training, but if any company has the means to do so, it’s Meta.”

Microsoft: full-force tokenmaxxing

Similarly, Microsoft has had an internal token leaderboard like Meta’s since January, and it started pretty well, as I reported back at the time: there’s an internal token dashboard that displays the individuals who use the most tokens in order to promote the use of tokens and experimentation with LLMs. At the Windows maker, this leaderboard is interesting:

• Very senior engineers – distinguished-level folks – are in the top 5 across the whole company, despite the fact that this group generally wrote little code in the past.
• VP-level folks make the top 10 and top 20, despite often being in meetings for most of the day and rarely writing code.

However, what starts as a metric for performance reviews or promotions can quickly become a target for devs. I talked with a software engineer at the Windows maker who admitted they’re full-on “tokenmaxxing” – not to get on the leaderboard, but rather because they don’t want to be seen as using too few tokens:

“We have internal dashboards and metrics tracking AI usage, token usage, percentage of code written by AI vs hand-written code.

I am conscious of not wanting to be seen as “uses too little AI,” and I’m not ashamed to say I need to do tokenmaxxing to do this. Things I do to inflate my token usage metrics:Ask AI questions about the code already in the documentation. The AI pulls up the documentation, processes it, and gives me results 10x slower, but while burning lots of tokens. I could use “readthedocs” [an internal product], but then my token numbers would be lowerAsk the AI to prototype a feature that I have no intention of working on. Prompt it a few more times, then throw the whole thing awayDefault to always using the agent, even when I know I could do the work by hand much faster. Then watch it fail”

This engineer is relatively new at the company, so is concerned about job security, and is playing this game to avoid being tagged as insufficiently “AI-native” by burning far more tokens than necessary.

Salesforce: burning tokens to hit “minimum” & “ideal” targets

Elsewhere, Salesforce has created “tokenmaxxing” incentives, as well. Talking with an engineer there, I learned that the company built two tools that effectively incentivize excessive spending on tokens:

1. “Minimum” incentives with a tracking tool. There’s a Mac widget that shows your own spend, updated every 15 minutes. It also displays minimum expected spend. Last week, the target was $100 on Claude Code, and $70 on Cursor.
2. Showing everyone’s spend. A web-based tool to see the token spend of any colleague. It’s used to check where team mates’ usage is at.
3. “Maximum” spend limits that can be exceeded. Up to a week ago, there was also a maximum monthly limit of $250 for Claude Code and $170 for Cursor. However, this can be exceeded with the simple press of a button if the limit is reached. I’ve learned that last week, some engineering organisations at Salesforce had their “maximum” limit removed in order to “remove any friction from the development process.”

The message Salesforce sends to staff is clear: “use a minimum of $170/month tokens or be flagged.” Who wants to get flagged for using too few tokens? The outcome is somewhat wasteful token spend:

• Burning tokens for nothing. Devs ask Claude or Cursor: “build me X,” where X is a project or product with nothing to do with their work, and not something they’d ever ship. It’s just a way to burn tokens
• Calibrating token spend to be above average. Plenty of devs browse peers’ token spend to figure out the slightly-above average point, then use the tokens needed to hit that mark

Shopify: an example on how to avoid tokenmaxxing

The first-ever token leaderboard that I’m aware of was built by Shopify in 2025. And it worked well! Last June, the Head of Engineering at Shopify, Farhan Thawar, told me on The Pragmatic Engineer Podcast:

“We have a leaderboard where we actively celebrate the people who use the most tokens because we want to make sure they are [celebrated] if they’re doing great work with AI.

[And for the top people on the leaderboard,] I want to see why they spent say $1,000 a month in credits for Cursor. Maybe that’s because they’re building something great and they have an agent workforce underneath them!”

I asked Farhan for details on how it’s gone since. Here’s what he told me:

“We have since renamed the token leaderboard to usage dashboard: for obvious reasons, as we don’t want to encourage “competing” to make it to the top of this board. We have token spend on our internal wiki profile as well as on the usage dashboard.

We also have circuit breakers to catch “runaway agents.” So if personal spend spikes within a day, we can cut off access immediately, and you can renew if the usage spike was deliberate, or if it was a runaway agent. The circuit breaker worked well for us: we’ve not only caught runaway agents, but found bugs in our infra this way!”

Shopify’s approach seems to have worked for a few reasons:

• The usage dashboard served as a “push” for devs to use AI tools, early-on. Last year, devs were mostly experimenting with AI tools because they were not as performant as today. The usage dashboard encouraged developers to try new tools, and highlighted power users.
• Circuit breakers helped. Cutting off spend when usage spikes helped catch “runaway agents.”
• High usage is looked at. Farhan checks-in with top-spending individuals to understand the use cases. Any tokenmaxxing would likely have been spotted at this stage, which would have been a bit embarrassing for the user!

One more interesting learning Farhan shared with me: it’s more interesting to not look at “who spent the most in overall token cost?” but instead, “whose tokens cost the most?” Devs who generate tokens that come out as expensive have turned out to do in-depth work that was interesting to learn about!

Tokenmaxxing: great for AI vendors, bad for everyone else

I see very few rational reasons why incentivizing tokenmaxxing makes sense for any company. It results in increasing AI spend – by a lot! – in return for little to no value. Heck, in some cases it actually incentivises slower work – as shown by devs using the AI to answer questions when documentation is readily available – and encouraging ‘busywork’ where devs prompt projects that they don’t even want to ship. Tokenmaxxing seems to push devs to focus on stuff that makes no difference to a business.

It feels to me that a good part of the industry is using token count numbers similarly to how the lines-of-code-produced metric was used years ago. There was a time when the number of lines written daily or monthly was an important metric in programmer productivity, until it became clear that it’s a terrible thing to focus on. A lines-of-code metric can easily be gamed by writing boilerplate or throwaway code. Also, the best developers are not necessarily those who write the most code; they’re the ones who solve hard problems for the business quickly and reliably with – or without – code!

Similarly, the number of tokens a dev generates can easily be gamed, and if this metric is measured then devs will indeed game it. But doing so generates a massive accompanying AI bill!

—-

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse. This week’s issue covers:

1. New trend: token spend breaks budgets – what next? In the past 2-3 months, spending on AI agents has exploded at many tech companies, and the ramifications of this are starting to dawn on engineering leaders. We’ve sourced details from 15 companies, including the different ways they are coping with this realization.
2. New trend: more AI vendors can’t keep up with demand. Related to massively increased spending, GitHub Copilot and Anthropic are starting to limit less-profitable individual users, so they can serve business users whose spend has easily 10x’d in the last few months. The exception is OpenAI and Codex.
3. Morale at Meta hits all-time low? Business is booming but devs at Meta are furious and worried due to looming layoffs, and an invasive tracking program rolled out to all US employees.

We’re used to highly reliable systems which target four-nines of availability (99.99%, meaning about 52 minutes of downtime per year), and for it to be embarrassing to barely hit three nines (around 9 hours of downtime per year.) And yet, in the past month, GitHub’s reliability is down to one nine!

Here’s data from the third-party, “missing GitHub status page”, which was built after GitHub stopped updating its own status page due to terrible availability. Recently, things have looked poor:

This means that for every 30 days, GitHub had issues on 3 days, or issues/degradations for 2.5 hours daily (around 10% of the time.)

GitHub seems unable to keep up with the massive increase in infra load from agents. One software engineer built a clever website called “Claude’s Code” that tracks Claude Code bot contributions across GitHub. Growth in the past three months has been enormous:

Stream of GitHub outages from infra overload

GitHub’s CTO, Vladimir Fedorov, addressed availability issues in a blog post and covered three major incidents:

• 2 February: security policies unintentionally blocked access to virtual machine metadata
• 9 February: a database cluster got overloaded
• 5 March: writes failed on a Redis cluster

Software engineer Lori Hochstein did a helpful analysis of these outages and the CTO’s response, and has interesting observations:

• Saturation: the database cluster incident (9 Feb) was a case of the database getting saturated, due to higher-than-expected usage. Databases are harder to scale up than stateless services. GitHub also underestimated how much additional traffic there would be.
• Failover + telemetry gap: the 2 Feb incident was a combination of an infra issue in one region failing over to a healthy region, and making things worse with a telemetry gap (incorrect security policies were applied in the new regions which blocked access to VM metadata)
• Failover + configuration issue: the 5 March incident was uncannily similar: after a failover, a configuration issue blocked writes on a Redis cluster

It is certainly nice to get details from GitHub on these outages. It feels to me that infra strains are causing more infra issues → they trigger constraints faster → failovers are not as smooth as they should be. Could it be because GitHub keeps changing their existing systems?

Startup shows GitHub how it’s done

While GitHub struggles to keep up with the increase in load from AI agents generating more code and pull requests, a new startup called Pierre Computer claims to have built an “AI-native” solution for AI agents pushing code, which scales far beyond what GitHub can do. Pierre was founded by Jacob Thornton: formerly an engineer at Coinbase, Medium, and Twitter, and also the creator of the once-very popular Bootstrap CSS library.

Here’s what Pierre supports, which GitHub does not:

“In October [2025], Github shared they were averaging ~230 new repos per minute.

Last week we [at Pierre Computer] hit a sustained peak of > 15,000 repos per minute for 3 hours.

And in the last 30 days customers have created > 9M repos”

These are incredible numbers – if also self-reported – and something that GitHub clearly cannot get close to, at least not today! There are few details about customers, while the product – called Code.storage – seems to be in closed beta.

Still, this is the type of “git for AI agents” that GitHub has failed to build, and the type of infrastructure it needs badly.

Has GitHub lost focus and purpose?

GitHub’s reliability issues are acute enough that, if it keeps up, teams will start giving alternatives like small startups such as Pierre a try, or perhaps even consider self-hosting Git. But how did the largest Git host in the world neglect its customers, and fail to prepare its infra for an increase in code commits and pull requests?

Mitchell Hashimoto, founder of Ghostty, and a heavy user of GitHub himself, had advice on what he would do if he was in charge of GitHub, after growing frustrated with the state of its core offering. He writes (emphasis mine)

“Here’s what I’d do if I was in charge of GitHub, in order:

1. Establish a North Star plan around being critical infrastructure for agentic code lifecycles and determine a set of ways to measure that.

2. Fire everyone who works on or advocates for Copilot and shut it down. It’s not about the people, I’m sure there’s many talented people; you’re just working at the wrong company.

3. Buy Pierre and launch agentic repo hosting as the first agentic product. Repos would be separate from the legacy web product to start, since they’re likely burdened with legacy cross product interactions.

4. Re-evaluate all product lines and initiatives against the new North Star. I suspect 50% get cut (to make room for different ones).

The big idea is all agentic interactions should critically rely on GitHub APIs. Code review should be agentic but the labs should be building that into GH (not bolted in through GHA like today, real first class platform primitives). GH should absolutely launch an agent chat primitive, agent mailboxes are obviously good. GH should be a platform and not an agent itself.

This is going to be very obviously lacking since I only have external ideas to work off of and have no idea how GitHub internals are working, what their KPIs are or what North Star they define, etc.

But, with imperfect information, this is what I’d do.”

My sense is that GitHub has three concurrent problems:

• GitHub and Copilot are entangled with Microsoft’s internal politics. GitHub’s Copilot in 2021 was the first massively successful “AI product.” Microsoft took the “Copilot” brand and used it across all of their product lines, creating low-quality AI integrations. Simultaneously, internal Microsoft orgs like Azure and Microsoft AI were trying to get their hands on GitHub, which is one of the most positive developer brands at Microsoft.
• GitHub has no leader, seemingly by design. GitHub’s last CEO was Thomas Dohmke, who stepped down voluntarily, and Microsoft never backfilled the CEO role; instead carrying out a reorg to make GitHub part of Microsoft’s AI group and stripping its independence. It seems the “Microsoft AI” side won that battle.
• GitHub has no focus, and is stuck chasing Copilot as a revenue source. GitHub has no CEO and is caught up in internal politics, so, what can GitHub teams do? The safest bet is to increase revenue and the best way to do that is by investing more into GitHub Copilot, and ignoring long-term issues like reliability.

I agree with Mitchell: GitHub has no “North Star” and we see a large org being dysfunctional. That lack of vision – and CEO – is hitting hard:

• GitHub Copilot went from the most-used AI agent in 2021, to be overtaken by Claude Code, and is soon to be overtaken by Cursor.
• As a platform, GitHub has no vision for how to evolve to support AI agents. Sure, GitHub has an MCP server, but it has no “AI-native git platform” that can handle the massive load AI agents generate.
• GitHub keeps shipping small features and improvements without direction. For example, in October 2025, they started to work on stacked diffs. However, when it ships, the stacked diffs workflow might be mostly obsolete – at least with AI agents!

It’s easy to win a market when you do one thing better than anyone else in the world. Right now, GitHub is doing too many things and doing a subpar job with Copilot, its platform, and AI infra.

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse.

Catch up with recent The Pragmatic Engineer issues:

• Scaling Uber with Thuan Pham (Uber’s first CTO — podcast). We went into topics like scaling Uber from constant outages to global infrastructure, the shift to microservices and platform teams, and how AI is reshaping engineering.
• Building WhatsApp with Jean Lee (podcast): Jean Lee, engineer #19 at WhatsApp, on scaling the app with a tiny team, the Facebook acquisition, and what it reveals about the future of engineering.
• What will the Staff Engineer role look like in 2027 and beyond? What happens to the Staff engineer role when agents write more code? Actually, they could be more in demand than ever!

An interesting trend highlighted by The Wall Street Journal: companies want to hire for FDE roles, but devs are just not that interested:

“Job postings on Indeed grew more than 10-fold in 2025 compared with 2024. The number of public company transcripts mentioning the role jumped to 50 from eight over the same period, according to data from AlphaSense.

The only problem? Few engineers want the job, which has historically been seen as demanding, undesirable, and less prestigious than product-focused engineering roles.

“Everyone wants them and there’s only maybe 10% of the market that wants that role,” said Patrick Kellenberger, president and chief operating officer at Betts Recruiting.“

Last summer, we covered the rise of the FDE role, and looked into what it’s like. Back then, this is how I visualized what was then a very hot role:

At the companies where I interviewed FDE folks – OpenAI and Ramp – the role seemed to live up to this visualization. However, I’ve since talked with two engineers who took FDE roles and were disappointed. This is how they saw it, in practice:

The role seems akin to a “sales engineer” where FDEs help close the deals, or a solutions engineer (or even consultant), where FDEs deploy to a customer to build them a solution. They don’t contribute back into the platform, and don’t do much that’s considered “software engineering” beyond integrating software which the product team built.

Some engineers figure out the nature of the role during the interview process and pass on it. Meanwhile, some others take the job and later quit. Here’s what a dev told me who accept an FDE role at a company, but didn’t find what they expected:

“This FDE job was a typical IT services mindset. The company wanted to use me more on the engagement lead side, and nothing on software development. It’s not what I signed up for, and I didn’t like the vibe and culture. I quit 4 weeks later.”

In today’s job market, if there’s high demand for a role which pays decently but attracts little interest from engineers, there’s always a reason!

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse.

Catch up with recent The Pragmatic Engineer issues:

• Building WhatsApp with Jean Lee (podcast): Jean Lee, engineer #19 at WhatsApp, on scaling the app with a tiny team, the Facebook acquisition, and what it reveals about the future of engineering.
• The Pulse: What will the Staff Engineer role look like in 2027 and beyond? What happens to the Staff engineer role when agents write more code? Actually, they could be more in demand than ever!
• From IDEs to AI Agents with Steve Yegge (podcast): Steve Yegge on how AI is reshaping software engineering, the rise of “vibe coding,” and why developers must adapt to a rapidly changing craft.

If you’ve been forwarded this email, you can subscribe here to get issues like this in your inbox.

Today’s issue of The Pulse focuses on a single event because it’s a significant one with major potential ripple effects. On Tuesday, Cloudflare shocked the dev world by announcing that they have rewritten Next.js in just one week, with a single developer who used only $1,100 in tokens:

There are several layers to dig into here:

1. The Next.js ecosystem: a recap. Close to half of React devs use Next.js, and the best place to deploy Next.js is on Vercel – partly thanks to its proprietary build output.
2. What Cloudflare did with Next.js. Replacing the build engine in Next.js with the more standard Vite one, allowing Next.js apps to be easily deployed on Cloudflare.
3. AI brings the impossible within reach. What would take years in engineering terms was executed in one week with some tokens.
4. “AI slop” still an issue. Contrary to Cloudflare’s claims, vinext is not production-ready, and will need plenty of cleanup and auditing to make it on par with Next.js.

1. The Next.js ecosystem: a recap

First, some background. Next.js is the most popular fullstack React framework and around half of all React devs use it, as per recent research such as the 2025 Stack Overflow developer survey. Next.js is an open source project, built and mostly maintained by Vercel, which is the preferred deployment target for Next.js applications for many reasons. One of them is that Next.js is ideal to deploy to Vercel because Next.js applications are built with Vercel’s Turbopack build tool. The output of a build is a proprietary format. As Netlify engineer Eduardo Bouças writes:

“The output of a Next.js build has a proprietary and undocumented format that is used in Vercel deployments to provision the infrastructure needed to power the application.

This means that any hosting providers other than Vercel must build on top of undocumented APIs that can introduce unannounced breaking changes in minor or patch releases. (And they have)”.

Next.js is an interestingly built project, where everything is open source, and the best place to deploy a Next.js application is on Vercel, as it’s optimized to run undocumented build artifacts the most efficiently. This is a smart strategy from Vercel which competitors will dislike, as any hosting provider would prefer Next.js to produce a standard build format. To do this, the build engine, Turbopack, would need to be replaced with something more standard.

Let’s talk about build tools for web development. According to the State of JS 2025 survey, the most popular in the web ecosystem are:

1. Vite: the most popular choice for new projects due to its speed and developer experience. Uses projects like esbuild and Rollup under the hood
2. Webpack: a legacy tool that’s not very performant, but still widely deployed in older projects
3. Turbopack: Created by Vercel and optimized for larger Next.js applications. Built in Rust and intended to be more performant
4. Bun: a relatively new, all-in-one runtime and bundler. Anthropic acquired the team in December, and some Bun folks are now focused on improving Claude Code’s performance.

So, most of the web ecosystem uses Vite as a build tool; Next.js uses Turbopack, and the majority of React applications with a full-stack React framework use Next.js. Basically, most devs using Next.js are likely to use Vite as their build tool.

2. What Cloudflare did with Next.js

Here’s a naive idea: what if Next.js used Vite to generate build outputs? In that case, build outputs would be standardized and would run equally well on any cloud provider, as there would be nothing proprietary or undocumented to Vercel.

And this is what Cloudflare did: replace Turbopack with Vite and call the new package ‘vinext’:

Buried midway in the announcement is how this project is experimental and not at all guaranteed to work okay: it’s a ‘use-at-own-risk’ project. Still, the mere fact of this development feels like an earthquake in the tech world because of how it was pulled off.

3. AI brings the impossible within reach

In a blog post announcing the project, Cloudflare claims only one engineer “rebuilt” the whole thing in a way that’s trivial to deploy to Cloudflare’s own infrastructure, and only cost $1,100 in tokens. From Cloudflare’s statement:

“Last week, one engineer and an AI model rebuilt the most popular front-end framework from scratch. The result, vinext (pronounced “vee-next”), is a drop-in replacement for Next.js, built on Vite, that deploys to Cloudflare Workers with a single command. In early benchmarks, it builds production apps up to 4x faster and produces client bundles up to 57% smaller. And we already have customers running it in production.

The whole thing cost about $1,100 in tokens”.

What Cloudflare did:

• Took the Next.js public API
• Reimplemented behaviour using Vite
• Created build output whose behaviour matches the “original” Next.js implementation

After 10 years, the core of Next has around 194,000 lines of code (LOC)**. Meanwhile, vinext is about 67,000 lines of code which suggests a much leaner implementation: for example, vinext does not need to support legacy Next APIs, and vinext currently supports 94% of the Next.js API (and it’s safe to assume they left complex edge cases in the remaining 6%).

** the Next.js repository is closer to 2M lines of code: 1M is bundled dependencies (eg React bundles, CSS build etc), tests are 308,000 LOC, Turbopack 311,000 LOC.

Pre-AI, this reimplementation would have taken years of engineering time to complete. Doing what Cloudflare did was always possible in theory, but never seemed practical. I mean, why have a team of engineers spend potentially years on generating a standardized build output for Next.js apps? Even if they did, the dev community would have doubts about whether Cloudflare would maintain the project.

This is the thing with forking or rewriting open source projects: a major value proposition for commercial open source is to know that they will be maintained. Vercel has proved it’s a reliable custodian of Next.js for the past 10 years. Without AI, it could be assumed that any new reimplementation would eventually run out of steam.

Separately but relatedly, Cloudflare has now proved that the cost of rewriting existing software has become ~100x cheaper, thanks to AI, and this economy is likely to be the case for maintenance, too. Considering how trivial it was to rebuild one of the more complex open source projects, this augers well for it being trivial and much cheaper to maintain in the future. Potentially, Cloudflare no longer needs to budget an engineering team only for maintenance, if a single engineer could maintain the project, part-time!

Cloudflare had a project measured in engineering years, and completed it in one engineering week! It just took a single engineer using OpenCode (open source coding agent), Opus 4.5, and a bunch of tokens, then: ‘boom’, vinext was born.

4. “AI slop” still an issue

There are questions about the quality of vinext, though. Vercel, naturally, is unhappy and hit out at the obvious weakness that vinext is unfit for production usage because it’s insecure. Vercel CEO, Guillermo Rauch, did not miss a beat by tying Cloudflare’s effort to the “vibe coding” stereotype of sloppy work executed with a lack of understanding:

Guillermo has a point: anyone who stopped reading Cloudflare’s launch announcement after the first few sentences would assume it’s production-ready, with the first paragraph of this announcement closing with:

“And we already have customers running it in production.”

However, Cloudflare doesn’t share the rather crucial detail that “running in production” means that vinext has been deployed onto a beta site, until more than 1,000 words (around 2–3 pages) into the announcement:

“We want to be clear: vinext is experimental. It’s not even one week old, and it has not yet been battle-tested with any meaningful traffic at scale. (...)

We’ve been working with National Design Studio, a team that’s aiming to modernize every government interface, on one of their beta sites, CIO.gov.

Oh. So, “customers running it in production” at Cloudflare apparently means “customer running a beta site in production without meaningful traffic.” This is a first from the infrastructure giant, which usually prides itself on accurate statements!

This detail was also absent when Cloudflare’s CEO and CTO were boosting vinext like it was a mature, battle-tested product. In that context, Vercel’s raising of the issue of security vulnerabilities is more than fair game, in my view.

Still, all that doesn’t alter the core learning from this project: that AI has the power to drastically reduce engineering time by up to ~100x and deliver usable-enough output, for relatively negligible financial cost. Just keep in mind that security and reliability issues will probably take plenty of extra time and effort to address.

5. New attack vector on commercial open source?

If arch-rivalries exist in tech, then Cloudflare and Vercel are a prime example. Both are gunning to become the most popular platform for developers to deploy their code, and the CEOs are regularly seen in public taking shots at the other side. One such spat happened in March, as covered at the time:

“Things kicked off on social media, with developers confused about the severity of the incident, and about why Next.js seemed silent, and also why Cloudflare sites were breaking due to its fix for the CVE causing its own issues. It was at that point that Cloudflare’s CEO, Matthew Prince, entered the chat to accuse Vercel of not caring about security:

Given the security incident was ongoing, this felt a bit “below the belt” by the Cloudflare chief. Criticizing rivals is fair game, but why not wait until the incident is over? The punch landed, and Vercel’s CEO Guillermo Rauch is not someone to take it lying down, so he hit back.

Cloudflare’s CEO then responded with a cartoon implying that although Vercel is much larger than its competitor Netlify, Cloudflare is 100x bigger than both, and could stomp them into the ground at will.”

Serving the public interest wasn’t why Cloudflare rewrote Next.js: they did it because they want Next.js sites to be deployed onto Cloudflare, but doing so made little sense until now because Next.js produced bespoke build output optimized for Vercel’s infrastructure. With this change, Cloudflare claims it provides superior performance when hosting Next.js apps, according to their own measurements.

I’d just add that performance is important for developers, but other things matter, too. Cost, reliability, developer experience, and how much devs like a company, are all factors in choosing between vendors. Also, performance measurements from a vendor about its own service must be taken with a large pinch of salt.

Zooming out from this episode, it seems that AI is bringing the value of existing commercial open source moats into question. Vercel carved out a clever open source strategy that helped turn its open source investment into business revenue:

1. Build and maintain Next.js, delivering the best developer experience (DX).
2. Optimize Vercel to serve the specific (and undocumented) build output of Next.js.
3. Most developers onboarding to Next.js will decide to deploy on Vercel to get the most benefit, in terms of DX and performance.
4. … repeat for years while the business becomes worth billions! (Vercel was valued at $9B last October).

Underpinning this success are some assumptions:

1. Next.js will remain the #1 choice for developers to build React applications, thanks to ongoing investment.
2. It is expensive to rewrite Next.js to be deployable and performant on another cloud vendor.
3. Even if someone did #2, developers would be skeptical and not switch over.

Vercel can invest in #1 to keep Next as best-in-class, while knowing that the risk of #2 occurring is minor. However, Cloudflare has now “cloned” Next, and can easily keep up with all changes in the future, and port them back to vinext.

But AI makes it trivial to “piggyback” off any commercial open source project, which is a massive problem for commercial open source startups. It puts all the effort and investment into building and maintaining Next.js, while Cloudflare enjoys the benefit of this hard work (the Next.js public API) which is easily deployable to Cloudflare, and it can now undercut Vercel on price. For all future Next.js changes, Cloudflare will just sync it to vinext, using AI!

WordPress had a similar problem, with WP Engine “piggybacking” off its work and undercutting their pricing in 2024. As I analyzed at the time:

“Free-riding on permissive open source is too tempting to pass on for other vendors. WP Engine uses a common loophole of contributing almost nothing in R&D to WordPress, while selling it as a managed service. This means that they could either easily undercut the pricing of larger players like Automattic which do spend on WordPress’s R&D. Alternatively, a company like WP Engine could charge as much, or more, as Automattic, but be able to spend a lot more on marketing, while being similarly profitable. “Saving” on R&D gives the “free-riders” plenty of options to grow their businesses: options not necessarily open to Automattic while they invest as much into R&D as they do.

Commercial open source vendors pressure to end “freeriding”. Automattic is likely facing lower revenue growth, with customers choosing vendors like WP Engine which offer a similar service — getting these customers either via a cheaper price or thanks to more marketing spend. This legal fight could be an effort to force WP Engine to stop eating Automattic’s lunch, or perhaps get WP Engine to sell to Automattic, which would cement its leading status in managed Wordpress, while also boosting revenue by $400M a year – according to its own figures”.

Vercel managed to avoid the “free-riding” problem with Next.js, but that’s no longer possible now that AI makes it trivial to rewrite.

6. Defense or offense?

How should commercial open source companies respond to the threat that a competitor can easily rewrite the software behind the managed solutions which they sell as services?

One obvious response is to make tests private, so that replication is harder for AI. One thing that made it so easy for Cloudflare to rewrite Next was the project’s comprehensive test suite. From their announcement (emphasis mine):

“We also want to acknowledge the Next.js team. They’ve spent years building a framework that raised the bar for what React development could look like. The fact that their API surface is so well-documented and their test suite so comprehensive is a big part of what made this project possible.”

Database solution SQLite is famous for its incredible test suite. What some people don’t know is that while core SQLite tests are open source, its most comprehensive test suite – TH3 – is closed source. SQLite monetizes its advanced infrastructure as a service for purchase. This is a fair tradeoff: for most contributors, the basic open source tests work well enough. For enterprise users or customers who really care about correctness, it makes sense to purchase advanced testing services from the service’s creator.

Open source canvas project, tldraw, announced it will relocate its test suite to a closed source repository; a move which makes plenty of sense. Here’s commentary from Simon Willison:

“It’s become very apparent over the past few months that a comprehensive test suite is enough to build a completely fresh implementation of any open source library from scratch, potentially in a different language.”

In the event, tldraw’s announcement turned out to be a joke, but who’s laughing now? An open source project with excellent tests is an easy target for an AI agent to execute a full rewrite of it.

Could new licenses be created for the AI era? Existing open source licenses were created on the assumption that humans read open source code, and humans modify it. Agents break that assumption.

Could we see new license types emerge to ban AI agents from modifying projects’ source code? It seems pretty far-fetched and hard to implement, but not beyond the realms of possibility.

AI agents are still very new, and going mainstream in tech. Once they break into other industries, I wouldn’t be surprised if legal frameworks are reworded to also apply to AI agents. If and when this happens, it would open the path for open source licenses to distinguish between agents and humans.

What is a moat, if code can be trivially ported? A team operating a popular open source project can no longer assume it’s expensive to fork or to be completely rewritten, meaning it makes sense to focus on other moats, such as:

• Outstanding (paid) support. AI could make this much easier at a higher quality, if done right.
• Smaller open core, larger closed source part. “Open core” as a business model has been dominant for commercial open source: keep the core of the software open source, while advanced enterprise features are source available or closed source. I would expect more companies to move their additional services to closed source, not source available.
• In-person connection and community. Projects with a real-world community will form a sense of connection that goes beyond code. For example, it’s hard to imagine vinext meetups popping up – whereas there are many Next.js communities.
• Infrastructure and hardware remains a massive moat. In a world where software is trivial to copy, infrastructure remains a moat. Commercial open source might make most sense for players that own and operate superior infrastructure layers than their rivals: and being able to offer lower cost, higher reliability, lower latency, higher performance, or a combination of these.

7. AI-world reality

One of the single best AI use cases is full-on rewrites of well-tested products. I estimate that AI sped up the creation of vinext by at least 100x, which is massive. But we don’t really see efficiency boosts of anything like that with AI tools, in general. As Laura Tacho shared at The Pragmatic Summit in San Francisco, the average self-reported efficiency ‘AI gain’ seems to be circa 10%.

I suspect this vast chasm in efficiency boosts is because AI is many times more efficient at “no-brainer tasks” where correctness can be verified with tests, versus those which are more open ended or involve more creativity.

In general, tests are incredibly important for efficient AI usage. On The Pragmatic Engineer Podcast, Peter Steinberger stressed how important “closing the loop” in his developer flow is by instructing the AI to test itself, and ensuring the AI has tests to run that verify correctness.

Automated tests were always considered a best practice for creating maintainable code. Now, having a codebase with extensive tests is the baseline to make AI agents work productively for refactors, rewrites – or even adding new features and verifying that things did not break!

Vendors will start to deploy “migration AI agents” to move customers over to their own stacks. This got lost in Cloudflare’s announcement, but it’s important:

vinext includes an Agent Skill that handles migration for you. It works with Claude Code, OpenCode, Cursor, Codex, and dozens of other AI coding tools. Install it, open your Next.js project, and tell the AI to migrate:

> npx skills add cloudflare/vinext

Then open your Next.js project in any supported tool and say:

> migrate this project to vinext

The skill handles compatibility checking, dependency installation, config generation, and dev server startup. It knows what vinext supports and will flag anything that needs manual attention.

This is very clever from Cloudflare, and a true “AI-native” move. They have not only used AI to migrate Next.js, but also built an “AI plugin” (a skill) to help customers migrate their existing codebases over to vinext – and deploy on Cloudflare!

This move will surely be copied by other vendors, since migrations which are tedious for humans are much less effort with agents.

AI is making the tech industry more ruthless when it comes to business practices. Laura Tacho said something interesting at The Pragmatic Summit:

“AI is an accelerator, it’s a multiplier, and it is moving organizations in different directions.”

AI seems to be accelerating the ruthlessness of competition for customers and the speed at which this happens. In one week, Cloudflare rebuilt Next.js, and it’s attacking Vercel full-on: claiming their “vibe coded” alternative is more performant and production-ready, and burying at the foot of the launch announcement the crucial information that vinext is very much experimental.

I sense vendors are realizing that there’s a limited amount of time in which to use AI to their advantage, and some will decide to use it like Cloudflare has.

On the other hand, AI could be great news for non-commercial open source. AI presents as a threat to commercial open source because it removes existing moats which make code hard to fully rewrite. However, beyond that, AI could help non-commercial open source to thrive:

• With AI, it’s easy to fork an open source project and keep the fork in-sync with the original.
• It’s trivial to instruct AI to rewrite an open source project to another language or framework.
• …and it’s equally trivial for AI to add features to a fork.

For these reasons, I believe there could be a lot more forks and rewrites to come, and more open source projects and code, in general.

Takeaways

Personally, I could not have imagined things changing this quickly in software. Rewriting Next.js in a single week, even to a version that is not quite there – but mostly works? This was out of the question as recently as a few months ago.

Things changed around last December, when Opus 4.5 and GPT-5.2 came out and proved capable of writing most of the code. What used to be expensive is now cheap – like rewriting complete projects – and we still need to learn what the “new” expensive parts of software engineering are.

All this is new territory for everyone. To succeed in the tech industry, you need to be able to capitalize upon change, as Cloudflare has clearly done in this case by making the most of an opportunity created by new technology. It’s unclear how popular vinext will become, and how much of a moat Vercel has around the broader Next.js ecosystem, but I suspect that it’d take more than a Next rewrite to make Cloudflare into a viable Next.js platform-as-a-service provider.

1. SaaS is a pure software product. People who pay SaaS vendors do so because it’s cheaper to buy this software than build it.
2. LLMs dramatically reduce the time and cost of building custom software.
3. Therefore, most SaaS vendors will go out of business because most companies/teams will prompt an LLM to write the software they need, such as for ticketing, meetings, customer relationship management, etc.

The reason for my scepticism has been that SaaS such as HR software Workday is more than just software. Workday, for example, keeps up with compliance requirements (e.g., for holiday pay in different countries), guarantees correctness (e.g., payslips that comply with local regulations), and over time the software keeps up to date with changes in the external and internal environments.

However, this week I had first-hand experience of how ridiculously easy it is now to replace SaaS with LLMs. On my website – pragmaticengineer.com – I have a testimonials section, which displays real LinkedIn and X posts about this publication. It cost $120/year for a small service called Shoutout.io, and looked like this:

And this is the backend: nothing fancy, just a way to add, edit, reorganize, and delete testimonials.

I was a customer for four years and logged in perhaps once a year. My latest login was to get an annual invoice for my expenses. Unfortunately, the billing section was broken, so I emailed support and they sent me a broken link instead of the invoice. This was frustrating: why pay for a SaaS with broken billing? I couldn’t even tell what they would charge me next year.

So I asked myself if I could rebuild my own use case with an LLM, and do it rapidly. My use case was much simpler than the SaaS itself:

• Display existing testimonials in a similar way
• Make it easy to add new ones, e.g., store testimonials in some JSON format
• Make it look good

To my surprise, this whole effort from start to finish took exactly 20 minutes with Codex. The steps I took were straightforward enough:

• Asked Codex to make a plan on how to remove this third-party dependency and host all testimonials in my codebase (a GitHub repo, deployed onto Netlify)
• Tweaked the plan: I pushed for a modular approach where testimonials are in a separate JSON file, and they get generated into HTML with a compile-time build step
• Added this build step both locally and as a build trigger on Netlify
• Tested the solution
• Tweaked the UX and generated a schema
• Deployed it

The end result is visually the same as before, except I no longer have a third-party dependency rendering all of this!

What does this mean for SaaS products and software engineers?

What it means for software engineers:

• Devs are (probably) a lot more comfortable using the command line for future updates than regular users. To add a future testimonial, I’ll need to turn to my AI agent to insert it in my codebase, and I’ll then need to verify that it looks good. This is not a big deal for me, but it might be a dealbreaker for someone not comfortable with verifying the code output of an LLM.
• It’s a lot faster for a dev to “port” a SaaS than for anyone else. I first told Codex to copy the UI and it got things wrong because it tried to use a flexbox model. I had to tell it that this UI layout was not what I wanted, and then make the decision on which framework to use for the UI layout. A non-developer could probably figure all this out, but it would take longer.
• Honestly, it’s fun and interesting to rewrite a third-party feature. I recommend it. Part of why I took on this project is because I expected it to be an interesting challenge. I thought the effort would be more than what it was, and I’ve learned more about how well these tools work. I also used Codex in order to experience it more.

What this could mean for SaaS software:

• Rebuilding a SaaS still feels much harder than rebuilding your specific use case. I did not “rebuild” Shoutout in any way. Shoutout has 10x or more features, like adding quotes from 10 different platforms, authentication, billing (which didn’t work for me), and more.
• A SaaS that doesn’t give ongoing value is at risk of being replaced by customers. Shoutout doesn’t provide ongoing value after it displays my testimonials, and this static nature means it’s easy to replace. In contrast, it would be harder to rebuild if I paid for the platform to stay compliant, provide analytics or alerting, and do other real-time things that helped my business.
• Buying and selling SaaS businesses could become less profitable. The original version of Shoutout that I signed up for in 2021 was built in 2020 by an independent developer. In 2022, this developer sold this micro-SaaS to a product studio. Then, in 2025, Shoutout was sold again to new developers. From my point of view, nothing changed except that the billing system broke. I assume the buyers of this SaaS figured that revenue could keep rising with zero investment. But perhaps at some point that ceases to be true when people get fed up with a broken product and quit – especially when doing so is cheaper.

“Broken windows” not being fixed is less acceptable than it used to be. My journey away from Shoutout began with its billing system being broken. For example, below is what I saw when I went to my billing section to see the invoices:

As well as this, the customer support sent me a broken link in response to my email. That was enough for me to decide to replace this dependency, and I was surprised by how easy this was with an LLM and knowing what I wanted it to build. By the time customer support sent me a working link two hours later, I had finished migrating off the SaaS.

It feels like something valuable is being taken away, and suddenly. It took a lot of effort to get good at coding and to learn how to write code that works, to read and understand complex code, and to debug and fix when code doesn’t work as it should. I still remember how daunting my first “real” programming class was at university (learning C), how lost I felt on my first job with a complex codebase, and how it took years of practice, learning from other devs, books, and blogs, to get better at the craft. Once you’re pretty good, you have something that’s valuable and easy to validate by writing code that works!

Some of my best memories of building software are about coding. Being “locked in” and balancing several ideas while typing them out, of being in the zone, then compiling the code, running it and seeing that “YES”, it worked as expected!

It’s been a love-hate relationship, to be fair, based on the amount of focus needed to write complex code. Then there’s all the conflicts that time estimates caused: time passes differently when you’re locked in and working on a hard problem.

Now, all that looks like it will be history.

I wonder if I’ll still get the same sense of satisfaction from the fact that writing complicated code is hard? Yes, AI is convenient, but there’s also a loss.

Or perhaps with AI agents, being “in the zone” will shift to thinking about higher-level problems, while instructing more complex code to be written?

This was a section from my analysis piece When AI writes almost all code, what happens to software engineering?. Read the full one here.

A mere two weeks after Cloudflare suffered a major outage and took down half the internet, the same thing has happened again. Last Friday, 5th December, thousands of sites went down or partially down once more, in a global Cloudflare outage lasting 25 minutes.

As per last time, Cloudflare was speedy to share a full postmortem on the same day. It estimated that 28% of Cloudflare’s HTTP traffic was impacted. The cause of this latest outage was Cloudflare making a seemingly innocent – but global – configuration change that went on to take out a good portion of Cloudflare, globally, until being reverted. Here’s what happened:

• Cloudflare was rolling out a fix for a nasty React security vulnerability
• The fix caused an error in an internal testing tool
• The Cloudflare team disabled the testing tool with a global killswitch
• As this global configuration change was made, the killswitch unexpectedly caused a bug that resulted in HTTP 500 errors across Cloudflare’s network

In this latest outage, Cloudflare was burnt by yet another global configuration change. The previous outage in November happened thanks to a global database permissions change. In the postmortem of that incident, the Cloudflare team closed with this action item:

“Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input”

This change would make it so that Cloudflare’s configuration files do not propagate immediately to the full network, as they still do now. But making all global configuration files have staged rollouts is a large implementation that could take months. Evidently, there wasn’t time to make it yet, and it has come back to bite Cloudflare.

Unfortunately for Cloudflare, customers are likely to find unacceptable a second outage with similar causes to a previous one, only weeks ago. If Cloudflare proves unreliable, customers should plan to onboard to backup CDNs at the very least, and a backup CDN vendor will do its best to convince new customers to use it as the primary CDN.

Cloudflare’s value-add rests on rock-solid reliability without customers needing to budget for a backup CDN. Yes, publishing postmortems on the same day as an outage occurs helps restore trust, but that will crumble anyway with repeated large outages.

To be fair, the company is doubling down on implementing staged configuration rollouts. In its postmortem, Cloudflare is its own biggest critic. CTO Dane Knecht reflected:

“[Global configuration changes rolling out globally] remains our first priority across the organization. In particular, the projects outlined below should help contain the impact of these kinds of changes:Enhanced Rollouts & Versioning: Similar to how we slowly deploy software with strict health validation, data used for rapid threat response and general configuration needs to have the same safety and blast mitigation features. This includes health validation and quick rollback capabilities among other things.Streamlined break glass capabilities: Ensure that critical operations can still be achieved in the face of additional types of failures. This applies to internal services as well as all standard methods of interaction with the Cloudflare control plane used by all Cloudflare customers.“Fail-Open” Error Handling: As part of the resilience effort, we are replacing the incorrectly applied hard-fail logic across all critical Cloudflare data-plane components. If a configuration file is corrupt or out-of-range (e.g., exceeding feature caps), the system will log the error and default to a known-good state or pass traffic without scoring, rather than dropping requests. Some services will likely give the customer the option to fail open or closed in certain scenarios. This will include drift-prevention capabilities to ensure this is enforced continuously.
These kinds of incidents, and how closely they are clustered together, are not acceptable for a network like ours”.

Global configuration errors often trigger large outages

There’s a pattern of implicit or explicit global configuration errors causing large outages, and some of the biggest ones in recent years were caused by a single change being rolled out to a whole network of machines:

• DNS and DNS-related systems like BGP: DNS changes are global by default, so it’s no wonder that DNS changes can cause global outages. Meta’s 7-hour outage in 2021 was related to DNS changes (more specifically, Border Gateway Protocol changes.) Meanwhile, the AWS outage in October started with the internal DNS system.
• OS updates happening at the same time, globally: Datadog’s 2023 outage cost the company $5M and was caused by Datadog’s Ubuntu machines executing an OS update within the same time window, globally. It caused issues with networking, and it didn’t help that Datadog ran its infra on 3 different cloud providers across 3 networks. The same kind of Ubuntu update also caused a global outage for Heroku in 2024.

Globally replicating configs: in 2024, a configuration policy change was rolled out globally and crashed every Spanner database node straight away. As Google concluded in its postmortem: “Given the global nature of quota management, this metadata was replicated globally within seconds”.

Implementing gradual rollouts for all configuration files is a lot of work. It’s also invisible labor because when done well, then its benefits will be undetectable, except in the absence of incidents, thanks to better infrastructure!

The largest systems in the world will likely have to implement safer ways to roll out configs – but not everybody needs to. Staged configuration rollout doesn’t make much sense for smaller companies and products because this infra work slows down product development.

It doesn’t just slow down building, but every deployment, too, and this friction is designed to make everything slower. As such, they don’t make much sense unless the stability of mature systems is more important than fast iteration.

Software engineering is a field where tradeoffs are a fact of life, and universal solutions don’t exist. The development which worked for a system with 1/100th of the load and users a year ago, may not make sense today.

This was one out of the four topics covered in this week’s The Pulse. The full edition additionally covers:

1. Industry Pulse. Poor capacity planning at AWS, Meta moves to a “closed AI” approach, a looming RAM shortage, early-stage startups hiring slower than before, how long it takes to earn $600K at Amazon and Meta, Apple loses execs to Meta, and more
2. How the engineering team at Oxide uses LLMs. They find LLMs great for reading documents and lightweight research, mixed for coding and code review, and a poor choice for writing documents – or any kind of writing, really!
3. Linux officially supports Rust in the kernel. Rust is now a first-class language inside the Linux kernel, eight months after a Linux Foundation Fellow predicted more support for Rust. A summary of the pros and cons of Rust support for Linux

Read the full The Pulse issue.

A year ago, Amazon became the first tech giant to bring staff back into the office for the full five days per week. Back then, I analyzed the reasons for the change, and whether other workplaces would follow suit by dropping the widespread hybrid policy of 2-3 days/week in the office.

Now, Meta employees in the Instagram division have become the latest subjects of a full return to the office, following an announcement by the social media platform this week.

Instagram’s 5-day return to office

Instagram employees received the unexpected email on Monday, reports fellow Substacker, Alex Heath, who acquired a copy of the message. It was sent internally by Instagram CEO Adam Mosseri, who wrote:

“1. Back to the office: I believe that we are more creative and collaborative when we are together in-person. (...)

2. Fewer meetings: We all spend too much time in meetings that are not effective, and it’s slowing us down. Every six months, we’ll cancel all recurring meetings and only re-add the ones that are absolutely necessary (...)

3. More demos, less decks: Most product overviews should be prototypes instead of decks.

4. Faster decision-making: We’re going to have a more formalized unblocking process with DRIs, and I’ll be at the priorities progress unblocking meeting every week.”

This decision by Meta affects around a quarter of company staff, and it’s hard to imagine other divisions not following Instagram’s lead; after all, everything in Mosseri’s memo likely applies across the business.

Five years ago, CEO Mark Zuckerberg predicted 50% of Meta staff would work remotely by now, which didn’t happen. Indeed, with Instagram’s new 5-day RTO, I’d be surprised if 5% of Meta folks work remotely in two years’ time.

The reason for Insta’s RTO seems rooted in the leadership’s belief that in-office is more productive, as indicated by the top bullet point of Mosseri’s message. That message in full:

“I believe that we are more creative and collaborative when we are together in-person. I felt this pre-COVID and I feel it any time I go to our New York office where the in-person culture is strong.

Starting February 2, I’m asking everyone in my rollup based in a US office with assigned desks to come back full time (five days a week). The specifics:

• You’ll still have the flexibility to work from home when you need to, since I recognize there will be times you won’t be able to come into the office. I trust you all to use your best judgment in figuring out how to adapt to this schedule.
• In the NY office, we won’t expect you to come back full time until we’ve alleviated the space constraints. We’ll share more once we have a better sense of timeline.
• In MPK [Menlo Park, the HQ], we’ll move from MPK21 to MPK22 on January 26 so everyone has an assigned desk. We’re also offering the option to transfer from the MPK to SF office for those people whose commute would be the same or better with that change. We’ll reach out directly to those people with more info.
• XFN [cross-functional] partners will continue to follow their own org norms.
• There is no change for employees who are currently remote”.

From what I’ve seen of Mosseri from afar, he seems like a pretty straight shooter. It’s clear that he feels in-office creates more energy, and in Mosseri’s defense, I hear similar from many startup founders and leaders who say remote work causes a bunch of headaches: it’s harder to spot motivational problems and performance issues, information travels more slowly, and rallying teams is harder.

There’s no doubt that running a full-remote company is a lot of effort. There’s often-overlooked labor involved in hiring, onboarding, performance management, team celebrations, and even company-wide meetings – none of it is easy.

Linear is a full-remote company with nearly 50 people working there, which recently published details about how it operates. They’re introducing the concept of “coworking hubs”, flying in teams for in-person events, and holding regular off-sites, while being careful to hire people who fit the culture.

My feeling is that remote work policies at tech companies are going to become questions of their leaders’ preferences. Many devs prefer remote work: there’s fewer interruptions, more deep focus, and less commuting. Most of us would probably be just as productive – and probably more so – than when being interrupted in-office.

Leaders who prefer full-remote can cite flexibility and easier hiring from a larger pool of candidates as clear benefits. Meanwhile, those most comfortable with in-person will always have enough reasons to justify a 5-day RTO, along the lines of Mosseri’s reasoning. Advocates of hybrid setups cite balancing of focus time and efficiency.

In today’s job market, any company that pays closer to the top of the market can probably get away with five-days-a-week RTO. Meta is in this space, and although I’m sure plenty of devs will dislike the change, the alternative is to go out on the job market, accept a pay cut to join a new company, and start rebuilding your internal network.

Since we’re in the midst of a weird job market, it makes switching jobs more difficult than before, when the job market was very hot. In this respect, Instagram has external conditions on its side. For devs at Meta, one upside is that Big Tech experience opens more doors, even in this tough job market.

One caveat is that a 5-day RTO is unlikely in places where it’s hard to hire the right people. So, AI engineers and those working on AI products should be pretty safe, for instance, because those roles are incredibly in-demand, as indicated by the trend of higher base salaries for AI engineers. Based on that, few companies should want to push those workers to quit to join competitors.

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Many subscribers expense The Pragmatic Engineer Newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

One amusing detail of the November 2025 Cloudflare outage is that the realtime outage and monitoring service, Downdetector, went down, revealing a key dependency on Cloudflare. At first, this looks odd; after all, Downdetector is about monitoring uptime, so why would it take on a key dependency like Cloudflare if it means this can happen?

Downdetector was built multi-region and multi-cloud, which I confirmed by talking with Senior Director of Engineering, Dhruv Arora, at Ookla, the company behind Downdetector. Multi-cloud resilience makes little sense for most products, but Downdetector was built to detect cloud provider outages, as well. And for this, they needed to be multi-cloud!

Still, Downdetector uses Cloudflare for DNS, Content Delivery (CDN), and Bot Protection. So, why would it take on this one key dependency, as opposed to hosting everything on its own servers?

A CDN has advantages that are hard to ignore, such as:

• Drastically lower bandwidth costs – assets cached on the CDN are much faster
• Faster load times because assets on a CDN are served from Edge nodes nearer users
• Protection from sudden traffic spikes, as would be common for Downdetector, especially during outages! Without a CDN, those spikes could overload their services
• DDoS protection from bad actors taking the site offline with a distributed denial of service attack
• Reduced infrastructure requirements, as Downdetector can run on fewer servers

Downdetector’s usage patterns reflect that it’s a service very heavily used by consumers whom the business doesn’t really monetize (Downdetector is free to use.) So, Downdetector could get rid of Cloudflare, but costs would surge, the site would become slower to load, and revenue wouldn’t change.

In the end, Downdetector’s dependence on Cloudflare could be a pragmatic choice based on the business model, and how removing its upstream dependency upon Cloudflare could get very expensive!

Dhruv confirmed this and sharing more about the design choices at Downdetector:

“Building redundancy at the DNS & CDN layers would require enormous overhead. This is especially true as Cloudflare’s Bot Protection is world-class, and building similar functionality would be a lot of effort. There are hyperscalers [cloud providers] that have this kind of redundancy built in. We will look into what we can do, but with a team size in the double digits, building up a core piece of infra like this is a pretty tall order: not just for us, but for any mid-sized team.

We’ve learned that there are more things that we can improve, for the future. For example, during the outage, the Cloudflare control pane was down, but their API wasn’t. So, us having more Infrastructure as Code could have helped bring back Downdetector sooner.

On our end, we also noticed that the outage wasn’t global, so we were able to shift traffic around and reduce the impact.

One more interesting detail: Cloudflare’s Bot Protection went haywire during the outage, and started to block legitimate traffic. So, our team had to turn that off temporarily”.

Thanks very much to Dhruv and the Downdetector team for sharing details.

An unexpected highlight of publishing the book was ending up in Mongolia in June of this year, at a small-but-mighty startup called Nasha Tech. This was because the startup translated my book into Mongolian. Here's the completed book:

Here’s what happened:

A little over a year ago, a small startup from Mongolia reached out, asking if they could translate the book. I was skeptical it would happen because the unit economics appeared pretty unfavorable. Mongolia’s population is 3.5 million; much smaller than other countries where professional publishers had offered to do a translation (Taiwan: 23M, South Korea: 51M, Germany: 84M, Japan: 122M, China: 1.43B people).

But I agreed to the initiative, and expected to hear nothing back. To my surprise, nine months later the translation was ready, and the startup printed 500 copies on the first run. They invited me to a book signing in the capital city of Ulaanbaatar, and soon I was on my way to meet the team, and to understand why a small tech company translated my book!

Japanese startup vibes in Mongolia

The startup behind the translation is called Nasha Tech; a mix of a startup and a digital agency. Founded in 2018, its main business has been agency work, mainly for companies in Japan. They are a group of 30 people, mostly software engineers.

Their offices resembled a mansion more than a typical workplace, and everyone takes their shoes off when arriving at work and switches to “office slippers”. I encountered the same vibe later at Cursor’s headquarters in San Francisco, in the US.

Nasha Tech found a niche of working for Japanese companies thanks to one of its cofounders studying in Japan, and building up connections while there. Interestingly, another cofounder later moved to Silicon Valley, and advises the company from afar.

The business builds the “Uber Eats of Mongolia”. Outside of working as an agency, Nasha Tech builds its own products. The most notable is called TokTok, the “UberEats of Mongolia”, which is the leading food delivery app in the capital city. The only difference between TokTok and other food delivery apps is scale: the local market is smaller than in some other cities. At a few thousand orders per day, it might not be worthwhile for an international player like Uber or Deliveroo to enter the market.

The tech stack Nasha Tech typically uses:

• Frontend: React / Next, Vue / Nuxt, TypeScript, Electron, Tailwind, Element UI
• Backend and API: NodeJS (Express, Hono, Deno, NestJS), Python (FastAPI, Flask), Ruby on Rails, PHP (Laravel), GraphQL, Socket, Recoil
• Mobile: Flutter, React Native, Fastlane
• Infra: AWS, GCP, Docker, Kubernetes, Terraform
• AI & ML: GCP Vertex, AWS Bedrock, Elasticsearch, LangChain, Langfuse

AI tools are very much widespread, and today the team uses Cursor, GitHub Copilot, Claude Code, OpenAI Codex, and Junie by Jetbrains.

I detected very few differences between Nasha Tech and other “typical” startups I’ve visited, in terms of the vibe and tech stack. Devs working on TokTok were very passionate about how to improve the app and reduce the tech debt accumulated by prioritizing the launch. A difference for me was the language and target market: the main language in the office is, obviously, Mongolian, and the products they build like TokTok also target the Mongolian market, or the Japanese one when working with clients.

One thing I learned was that awareness about the latest tools has no borders: back in June, a dev at Nasha Tech was already telling me that Claude Code was their daily driver, even though the tool had been released for barely a month at that point!

Why translate the book into Mongolian?

Nasha Tech was the only non-book publisher to express interest in translating the book. But why did they do it?

I was told the idea came from software engineer Suuribaatar Sainjargal, who bought and enjoyed the English-language version. He suggested translating the book so that everyone at the company could read it, not only those fluent in English.

Nasha Tech actually had some in-house experience of translation. A year earlier, in 2024, the company translated Matt Mochary’s The Great CEO Within as a way to uplevel their leadership team, and to help the broader Mongolian tech ecosystem.

Also, the company’s General Manager, Batutsengel Davaa, happened to have been involved in translating more than 10 books in a previous role. He took the lead in organizing this work, and here’s how the timelines played out:

• Professional translator: 3 months
• Technical editor revising the draft translation: 1 month
• Technical editing #2 by a Support Engineer in Japan: 2 months
• Technical revision: 15 engineers at Nasha Tech revised the book, with a “divide and conquer” approach: 2 months
• Final edit and print: 1 month

This was a real team effort. Somehow, this startup managed to produce a high-quality translation in around the same time as it took professional book publishers in my part of the world to do the same!

A secondary goal that Nasha Tech had was to advance the tech ecosystem in Mongolia. There’s understandably high demand for books in the mother tongue; I observed a number of book stands selling these books, and book fairs are also popular. The translation of my book has been selling well, where you can buy the book for 70,000 MNTs (~$19).

Book signing and the Mongolian startup scene

The book launch event was at Mongolia’s startup hub, called IT Park, which offers space for startups to operate in. I met a few working in the AI and fintech spaces – and even one startup producing comics.

I had the impression that the government and private sector are investing heavily in startups, and want to help more companies to become breakout success stories:

• IT Park report: the country’s tech sector is growing ~20%, year-on-year. The combined valuation of all startups in Mongolia is at $130M, today. It’s worth remembering that location is important for startups: being in hubs like the US, UK, and India confers advantages that can be reflected in valuations.
• Mongolian Startup Ecosystem Report 2023: the average pre-seed valuation of a startup in Mongolia is $170K, seed valuation at $330K, and Series A valuation at $870K. The numbers reflect market size; for savvy investors, this could also be an opportunity to invest early. I met a Staff Software Engineer at the book signing event who is working in Silicon Valley at Google, and invests and advises in startups in Mongolia.
• Mongolian startup ecosystem Map: better-known startups in the country.

Two promising startups from Mongolia: Chimege (an AI+voice startup) AND Global (fintech). Thanks very much to the Nasha Tech team for translating the book – keep up the great work!

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Before we start: I’m excited to share something new: The Pragmatic Summit.

Four years ago, The Pragmatic Engineer started as a small newsletter: me writing about topics relevant for engineers and engineering leaders at Big Tech and startups. Fast forward to today, and the newsletter crossed one million readers, and the publication expanded with a podcast as well.

One thing that was always missing: meeting in person. Engineers, leaders, founders—people who want to meet others in this community, and learn from each other. Until now that is:

In partnership with Statsig, I’m hosting the first-ever Pragmatic Summit. Seats are limited, and tickets are priced at $499, covering the venue, meals, and production—we’re not aiming to make any profit from this event.

Apply to attend the Summit

I hope to see many of you there!

Cloudflare takes down half the internet – but shares a great postmortem

On Tuesday came another reminder about how much of the internet depends on Cloudflare’s content delivery network (CDN), when thousands of sites went fully or partially offline in an outage that lasted 6 hours. Some of the higher-profile victims included:

• ChatGPT and Claude
• Canva, Dropbox, Spotify,
• Uber, Coinbase, Zoom
• X and Reddit

Separately, you may or may not recall that during a different recent outage caused by AWS, Elon Musk noted on his website, X, that AWS is a hard dependency for Signal, meaning an AWS outage could take down the secure messaging service at any moment. In response, a dev pointed out that it is the same for X with Cloudflare – and so it proved earlier this week, when X was broken by the Cloudflare outage.

That AWS outage was in the company’s us-east-1 region and took down a good part of the internet last month. AWS released incident details three days later – unusually speedy for the e-commerce giant – although that postmortem was high-level and we never learned exactly what caused AWS’s DNS Enactor service to slow down, triggering an unexpected race condition that kicked off the outage.

What happened this time with Cloudflare?

Within hours of mitigating the outage, Cloudflare’s CEO Matthew Prince shared an unusually detailed report of what exactly went wrong. The root cause was to do with propagating a configuration file to Cloudflare’s Bot Management module. The file crashed Bot Management, which took Cloudflare’s proxy functionality offline.

Here’s a brief overview of how Cloudflare’s proxy layer works at a high level. It’s the layer that protects the “origin” resources of customers – minimizing network traffic to them by blocking malicious requests and caching static resources in Cloudflare’s CDN:

Here’s how the incident unfolded:

A database permissions change in ClickHouse kicked things off. Before the permissions changed, all queries to fetch feature metadata (to be used by the Bot Management module) would have only been run on distributed tables in Clickhouse, in a database called “default” which contains 60 features.

Until now, these queries were running using a shared system account. Cloudflare’s engineering team wanted to improve system security and reliability, and move from this shared system account to individual user accounts. User accounts already had access to another database called “r0”, so the team made the database permission change for access to r0 to be implicit instead of explicit.

As a side effect of this, the same query collecting the features to be passed to Bot Management started to fetch from the r0 database, and return many more features than expected:

The Bot Management module does not allow loading of more than 200 features. This limit was well above the production usage of 60, and was put in place for performance reasons: the Bot Management module pre-allocates memory for up to 200 features, and it will not operate with more than this number.

A system panic hit machines served with the incorrect feature file. Cloudflare was nice enough to share the exact code that caused this panic, which was this unwrap() function:

What likely happened:

• The append_with_names() function likely checked for a limit of 200 features
• If it saw more than 200 features, it likely returned an error
• … and when writing the code, it was not expected that append_with_names() would return an error…
• … and so .unwrap() panicked and crashed the system!

Edge nodes started to crash, one by one, seemingly randomly. The feature file was being generated every 5 minutes, and gradually rolled out to Edge nodes. So, initially, it was only a few nodes that crashed, and then over time, more became non-responsive. At one point, both good and bad configuration files were being distributed, making failed nodes that received the good configuration file start working – for a while!

Why so long to find the root cause?

It took Cloudflare engineers unusually long – 2.5 hours! – to figure all this out, and that an incorrect configuration file propagating to Edge servers was to blame for their proxy going down. Turns out, an unrelated failure made the Cloudflare team suspect that they were under a coordinated botnet attack, as when a few of the Edge nodes started to go offline, the company’s status page did, too:

The team tried to gather details about the attack, but there was no attack, meaning they wasted time looking in the wrong place. In reality, the status page going down was a coincidence and unrelated to the outage. But it’s easy to see why their first reaction was to figure out if there was a distributed denial of service (DDoS) attack.

As mentioned, it eventually took 2.5 hours to pinpoint the incorrect configuration files as the source of the outage, and another hour to stop the propagation of new files, and create a new and correct file, which was deployed 3.5 hours after the start of the incident. Cleanup took another 2.5 hours, and at 17:06 UTC, the outage was resolved, ~6 hours after it started.

Cloudflare shared a detailed review of the incident and learnings, which can be read here.

How did the postmortem come so fast?

One thing that keeps being surprising about Cloudflare is how they have a very detailed postmortem up in less than 24 hours after the incident is resolved. Cofounfer and CEO Matthew Prince explained how this was possible:

• Matthew was part of the outage call.
• After the outage was resolved, he wrote a first version of the incident review, at home. Matthew was in Lisbon, in Cloudflare’s European HQ, so this was early evening
• The team circulated a Google Doc with this initial writeup, and questions that needed to be reviewed
• In a few hours, all questions were answered
• Matthew: “None of us were happy [about the incident] — we were embarrassed by what had happened — but we declared it [the postmortem] true and accurate.
• Sent the draft over to the SF team, who did one more sweep, the posted it

Talk about moving with the speed of a startup, despite being a publicly traded company!

Learnings

There is much to learn from this incident, such as:

Be explicit about logging errors when you raise them! Cloudflare could probably have identified the root cause of this error much faster if the line of code that returned an error, also logged the error, and if Cloudflare had alerts set up when certain errors spiked on its nodes. It could have surely shaved an hour or two off the time it took to mitigate.

Of course, logging errors before throwing them is extra work, but when done with monitoring or log analysis, it can help find the source of errors much faster.

Global database changes are always risky. You never know what part of the system you might hit. The incident started with a seemingly innocuous database permissions change that impacted a wide range of queries. Unfortunately, there is no good way to test the impact of such changes (if you know one, please leave a comment below!)

Cloudflare was making the right kind of change by removing global systems accounts; it’s a good direction to go in for security and reliability. It was extremely hard to predict the change would end up taking down a part of their system – and the web.

Two things going wrong at the same time can really throw an engineering team. If Cloudflare’s status page did not go offline, the engineering team would have surely pinpointed the problem much faster than they did. But in the heat of the moment, it’s easy to assume that two small outages are connected, until there’s evidence that they’re not. Cloudflare is a service that’s continuously under attack, so the engineering team can’t be blamed for assuming it might be more of the same.

CDNs are the backbone of the internet, and this outage doesn’t change that. The outage hit lots of large businesses, resulting in lost revenue for many. But could affected companies have prepared better for Cloudflare going down?

The problem is that this is hard: using a CDN means taking on a hard dependency in order to reduce traffic on your own servers (the origin servers), while serving internet users faster and more cheaply:

When using a CDN, you propagate addresses that point to that CDN server’s IP or domain. When the CDN goes down, you could start to redirect traffic to your own origin servers (and deal with the traffic spike), or utilize a backup CDN, if you prepared for this eventuality.

Both these are expensive to pull off:

• Redirecting to the origin servers likely means needing to suddenly scale up backend infrastructure
• Having a backup CDN means there must be a contract and payment for a CDN partner which will most likely sit idle. As and when it is needed, you must switch over and warm up their cache: it’s a lot of effort and money to do this!

A case study in the trickiness of dealing with a CDN going offline is the story of Downdetector, including inside details on why Downdetector went down during Cloudflare’s latest outage, and what they learned from it.

This was one out of the five topics covered in this week’s The Pulse. The full edition additionally covers:

1. Downdetector & the real cost of no upstream dependencies. During the Cloudflare outage, Downdetector was also unavailable. I got details from the team about why they have a hard dependency on Cloudflare, and why that won’t change anytime soon.
2. Antigravity: Google’s new AI IDE – that its devs cannot use. Google wants to become a serious player in AI coding tools, but Antigravity contains remnants of Windsurf. Interestingly, devs at Google aren’t allowed to use Antigravity for work
3. Industry pulse. Gemini 3 launch, Anthropic valued at $350B, Jeff Bezos funds an AI company, and unusually slow headcount growth at startups persists.
4. Five AI fakers caught in 1 month by crypto startup. Candidates who fake their backgrounds and change their looks in remote interviews continue to plague companies hiring full-remote – especially crypto startups.

Read the full The Pulse