Last week, we covered the slightly perverse trend of “tokenmaxxing” across the industry, where devs run agents with the sole aim of boosting their personal “token stats” in an effort to rank higher on internal token leaderboards, and not be seen as a Luddite who doesn’t use AI tools enough compared to peers.

This week, I spoke with a software engineer at a large company and another at a seed-stage place. Both shared almost identical stories: at their latest all-hands, company leadership expressed concerns about the fast-rising costs of tokens. At both places, token spend has increased by ~10x in the last six months – with no signs of slowing down.

I wanted to find out about this trend, so I talked to devs at 15 businesses. Below is what I learned about what’s happening in workplaces of all sizes. Names are anonymized.

Large companies

Setting the default model to a cheaper one: 10,000+ person SaaS company, offices on all continents

Inside a large SaaS company, most devs use an internal background coding tool for coding. This model defaults to Claude Sonnet, which is the cheaper Claude version. Model selection is not persisted, so devs who prefer working with Opus, for instance, must reselect it on every subsequent startup.

This tool supports all major frontier models such as Sonnet, Opus, GPT, and Gemini. Devs at the company whom I talked to are very heavy users of the tool and have not encountered usage limitations.

Fintech company, US, Series D, ~8,000 people. Staff engineer:

“The cost in token spend is off the charts – and leadership has shared this trend with us. They have not said anything beyond showing growth in spend, and mentioning that this won’t be sustainable. So, nothing specific yet, but my sense is that something will have to change. Limits or prioritizing cheaper models, cutting back on hiring? Who knows.”

Infra company, US, publicly traded, ~5,000 people. Engineering Director:

“We’re monitoring but not restricting. We are spot checking the heaviest users, but we are seeing the business cases working out.

We are offering some guidance on model selection - e.g., turn off the new high-effort setting in Claude. Some users are trying open source models – but open source model usage is a bottom-up initiative, not a top-down one.”

Information technology, US, 10,000+ people. Director of Engineering:

“We have already had to raise our API budget limits multiple times in April. We recently switched to a much higher-effort level for Claude, which significantly increased the cost per PR.

One reason for the cost spike is using state-of-the-art models for demanding tasks. We are using that high-effort setting even for fairly trivial tasks that could have been handled by much cheaper models, or even by lower-effort Claude loops. Despite a few of us pointing this out, leadership has basically said budget is not the concern right now.

I sense that the budget increase has not been forecasted, and we’re in for a reckoning. I suspect the attitude changes once finance and other cost-conscious parts of the org realize we are spending hundreds of dollars per day, per highly-engaged developer. For now, fear of missing out and not wanting to fall behind seems to be outweighing cost discipline.”

Games studio, US+Europe, ~5,000 people. Senior developer:

“What budget increase? It’s very hard to get a budget for AI here! Claude Code is still not rolled out because $200/month/dev is seen as too high a cost. I talk with people at startups where $1,000/month in spending is totally normal, and it’s night and day here.”

Fintech company, US+Europe, late stage, ~5,000 people. Staff engineer:

“Some developers are now spending $500 a day (!!) on Claude Code. Practically speaking, this means that employee costs have doubled. Productivity has increased, in my view, but now the bottleneck is code reviews. AI can spit out code quite quickly, but we still have human reviews in place. Leadership encourages using AI for code review, but my team will not blindly trust AI.

The push from AI is coming from the top. This year’s performance review had a section on AI, rating devs by how well they used AI, so this is another reason everyone just uses it as much as they can.”

Mid-sized companies

SaaS industry, US, ~2,000 people. Dev Productivity Lead:

“Model routing helped keep our costs growing less dramatically. For example, changing the default model reduced cost by 30%. This is our strategy with AI spend, summarized:Short term: spend, spend, spend! Experiment and use whatever models make sense.Measure the impact. Measure key outcomes and report on spend, monthly.When spend vs results diverge: adjust. When our spend increases dramatically, but outcomes don’t follow: see what we can do to adjust the delta. More spend should mean better outcomes. If not, we are doing something wrong.”

Finance industry, US, ~2,000 people. VP of AI:

“We have Cursor and Claude Desktop, both of which have around 800-1,200 total users. Token usage is growing somewhat unexpectedly. Estimates are being adjusted on the fly; the initial plan to have strict limits (say, $100 per user) is breaking when reality hits, and people exhaust them in 3-5 working days.

Using expensive models is a problem. In regards to Cursor, many devs are defaulting to the most expensive models without realizing that going with Opus gives single percentage gains in intelligence compared to Sonnet, for example, while exhausting their budgets almost immediately.

We are working on blocking/managing out the most expensive models [with Cursor], as going into thousands of dollars per user, per month is not sustainable on our scale. Cursor is a good partner and we’re working with them to switch to a “pooled spend” model where heavy users can tap into a pool of extra spend.

Claude is a similar story. We were at $100 of Claude Desktop limit for everyone, but as we are moving forward, I can see that we would need to go much higher, especially for business-critical use cases.”

Infra company, US, late-stage, ~700 people. Founder:

“We haven’t had much of an issue. Most folks police themselves for runaway costs; for example, we had someone hit like $10K in a week because they messed up caching, but it was caught and they corrected their harness.

For the most part, we don’t see our high-end folks spending more than ~$1K/week. Now, to be clear, this is not a small amount! BUT it’s already a small subset of the population.

We’re just factoring it into engineering costs at this point: if it’s, say, $2K/month per employee, that’s $24K per year.

Who cares, then, when engineers already cost $200-400K/year in cash comp? Okay, so what if it’s $5K/month. That’s $60K/year.

Our bet is that token costs will stabilize and we’ll eventually end up with local-ish models.

Now, it could be five years before they stabilize, but overall, spend today isn’t that insane to me.

There’s a lot of people who are just dumb about it, but most legit execs push back on this. Take the Ralph loops or other insanity where someone spends $1K/day, $5K/week or stuff like this. That’s all just people being fools thinking they’re doing “R&D,” or somehow that they’re smarter than everyone else, but they’re just producing junk that never ships or is not useful.

We saw a bit of “stupid overspend” in the first couple months, but that’s all gone now. Costs could go up even more if we would “crack the whip” in wanting to see even more output, but we’re not doing that.”

Healthcare industry, US, ~500 people. Senior engineering manager:

“We are not holding back on spend, and have a monthly spend leaderboard. And we WANT devs to spend more on tokens! For example, one of my engineers spent $1,400 on a long Claude Code session in a single day.

We are seeing massive leverage, and we do more with the same number of people. This is why we are okay with our spending spiking. Our traffic is growing more than 10x, year-on-year, and we have managed to keep things running with the same team, and these AI tools.

Engineering is now blocked on Product and Design – which never happened before! This is how fast execution has become. We now have Staff+ engineers writing Product PRDs so we can move faster.

I’ve been in tech for close to 15 years and I never saw dramatic change like this. I just came back after a 3-month break, and every single thing is different in my day! I feel these AI agents are the biggest change in the industry since high-level languages became widespread.”

E-commerce company, US & Europe, ~2,000 devs. Head of Engineering:

“The increase in spend is INSANE. It’s about usage going up, with no signs of stopping. Usage is off the charts.

We currently do not have limits in place, and are not pausing now. Our CEO is AI-pilled and won’t let us slow down.

We do buy tokens at a discount. They start from 5% and go up with usage with the vendors we use (the usual suspects.)

We don’t let devs use anything lower than Opus 4.7 for coding. Cheaper models might work better, but a slight error pushed to prod would result in hours of toil.”

Small companies

Series A, US, ~50 people. Principal Engineer:

“About 15 devs are heavy users of AI and costs are rising very fast. Almost everyone uses Claude and Claude Code. We are considering four potential options:Increase AI budget, and start measuring more. Continue doing what we are, but allow devs to use more tokens instead of hiring limits. The precise ROI is hard to quantify, but we’ll start to measure and track both AI adoption and impact.Optimize token consumption. Use cheaper models for simpler tasks, review token usage, and see where we can cut usage. Downside: this approach could become one with diminishing returns, fast.Integrate more AI providers in the company. Find wrappers to abstract LLMs. The problem is: how do you replace Claude Code, for instance?Pivot to local models: such as Kimi, Qwen, and so on. The problem is it’s a big investment in high-end hardware or cloud GPUs. Upside: it offers better long-term cost control, once done.

We are likely to go with option #1: increase spend BUT maintain momentum and put the right measurements in place. We can do #2, #3 and #4 later. But if we kill AI usage momentum inside the company, the outcome will probably be worse.”

AI infra, US, seed stage, ~15 people. Founder:

“We saw a 15x increase in 6 months:Six months ago our spend per developer was ~$200/monthToday, it’s around $3,000/developer/month, for our seven devs
We’re not slowing usage, especially as we are building an AI infra product. The increase was much faster than expected, though.”

Small, bootstrapped company, Europe. Founding engineer:

“Our current strategy in dealing with the increase in costs is to switch to a cheaper model; unfortunately, from Opus to Sonnet in our case. That said, Sonnet is quite decent.”

How businesses manage token spend

Regardless of company size, there seems to be two strategies for how companies deal with increased spending. A summary:

Strategy #1: “let it rip and start measuring.” Around half of respondents say AI spend is rising dramatically, and they have decided to do nothing about it. They want devs to use AI as much as it makes sense to, and to help the work as much as possible.

However, because the cost is rising dramatically, these companies are now starting to measure usage and attempting to measure the impact of their AI tools.

There’s a few companies where the impact seems to be very positive, already. Smaller startups whose business is exploding in numbers of customers, load, and revenue, see that they don’t need to hire more staff because existing engineers can keep supporting the growth with AI tools.

Strategy #2: curb spending. Commonly mentioned cost-saving approaches:

• Use cheaper models for simpler tasks
• Set default models to less capable ones
• Set a spending cap and make it hard for engineers to exceed it, or require consent for doing so

Most companies using strategy #1 have briefly considered going with this approach, but threw it away, because they see this approach as optimizing on the wrong thing: cutting costs before the productivity impact of using state-of-the-art tools is even known!

Discounts exist when the spend is in the millions of dollars. I asked several people if they are getting discounts from vendors when buying tokens at scale. There were no exact numbers, but this is what I gathered in aggregate about possible custom agreements:

• Cursor: open to discounts above a few million dollars in spend. Companies have negotiated discounts with Cursor after crossing $1M of spending. Some companies negotiated tiered discounts from this level, starting at 5% and going higher as their spend goes up.
• Anthropic: no discounts. I talked with companies spending $5M+ per year on Claude which have received no discounts. If Anthropic offers discounts, it will likely be at a much higher tier.
• All discounts are custom, so try to negotiate – it’s free! Pricing discounts are on a per-customer basis, and highly custom. The easiest way to see if a discount is available is to ask the vendors!

—-

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse. This week’s issue covers:

1. Load from AI breaks GitHub – but why not other vendors? GitHub’s reliability is less than one nine, and getting worse. Prolific open source contributor, Mitchell Hashimoto, is quitting GitHub because he thinks it’s not suited for professional work. GitHub’s leadership blames the 3.5x increase in service load as the cause of degradation – or it might be self-inflicted.
2. Anthropic’s speedrun to destroy trust. Anthropic could do no wrong until recently, but in the past month, that’s all changed. Silently nerfing Claude Code, banning companies from Claude, and baffling price rises all add to a sense that Anthropic is in its “extraction” era of generating more revenue for the same or worse service.
3. Industry pulse. Dramatic price increases at GitHub Copilot, explosive growth at Codex, Google scrambling to build a good coding model, Cursor might be bought by SpaceX, AI agent deletes car business, and more.
4. Mitchell Hashimoto & the “building block economy.” Ghostty’s creator finds that open source “building blocks” are the best way to win massive adoption by software components – but it’s got harder to build a business on top of open building blocks.

Inside Meta, an engineer created a “token leaderboard” that ranks employees by token usage. Last week, The Information reported:

“Employees at Meta Platforms who want to show off their AI superuser chops are competing on an internal leaderboard for status as a “Session Immortal”— or, even better, “Token Legend.”

The rankings, set up by a Meta employee on its intranet using company data, measure how many tokens — the units of data processed by AI models — employees are burning through. Dubbed “Claudeonomics” after the flagship product of AI startup Anthropic, the leaderboard aggregates AI usage from more than 85,000 Meta employees, listing the top 250 power users.

The practice is emblematic of Silicon Valley’s newest form of conspicuous consumption, known as “tokenmaxxing,” which has turned token usage into a benchmark for productivity and a competitive measure of who is most AI native. Workers are maximizing their prompts, coding sessions and the number of agents working in parallel to climb internal rankings at Meta and other companies and demonstrate their value as AI automates functions such as coding.”

I spoke with a few engineers at Meta about what’s happening, and this is what they said:

• Massive waste. Plenty of devs are running an OpenClaw-like internal agent that burns massive amounts of tokens for little to no outcome.
• Outages caused by AI overuse. A dev mentioned that some SEVs were caused by what looked like careless AI code generation; almost like a dev behind the SEV was more concerned with churning out massive amounts of code with AI than with product quality.
• Gamified leaderboard. Those at the top of the leaderboard produce throwaway, wasteful work. This is painfully clear to anyone who checks Trajectories (AI prompts), which can be viewed.

As per The Information, Meta employees used a total of 60.2 trillion AI tokens (!!) in 30 days. If this was charged at Anthropic’s API prices, it would cost $900M. Of course, Meta is likely purchasing tokens at a discount, but that could still come in at $100M+ – in large part from senseless “tokenmaxxing”.

After backlash on social media, Meta abolished the internal leaderboard last week. One day after The Information revealed details about the incredible tokenmaxxing numbers, I confirmed that Meta has taken down its leaderboard; perhaps they realized that the incentive created enormous and unnecessary waste. If so, it’s a bit surprising that it took media coverage for the social media giant to reach that conclusion.

One engineer at Meta told me they think Meta had a different goal with the token leaderboard. A long-tenured engineer suspects increasing AI usage actually was the real goal. They said:

“Putting a leaderboard in place was always going to incentivize much more AI usage. And more AI usage means producing a lot more real-world traces. These traces can then be used to train Meta’s next-generation coding model better.

I believe this was the goal, even if no one said it out loud.

It’s an expensive way to generate data for training, but if any company has the means to do so, it’s Meta.”

Microsoft: full-force tokenmaxxing

Similarly, Microsoft has had an internal token leaderboard like Meta’s since January, and it started pretty well, as I reported back at the time: there’s an internal token dashboard that displays the individuals who use the most tokens in order to promote the use of tokens and experimentation with LLMs. At the Windows maker, this leaderboard is interesting:

• Very senior engineers – distinguished-level folks – are in the top 5 across the whole company, despite the fact that this group generally wrote little code in the past.
• VP-level folks make the top 10 and top 20, despite often being in meetings for most of the day and rarely writing code.

However, what starts as a metric for performance reviews or promotions can quickly become a target for devs. I talked with a software engineer at the Windows maker who admitted they’re full-on “tokenmaxxing” – not to get on the leaderboard, but rather because they don’t want to be seen as using too few tokens:

“We have internal dashboards and metrics tracking AI usage, token usage, percentage of code written by AI vs hand-written code.

I am conscious of not wanting to be seen as “uses too little AI,” and I’m not ashamed to say I need to do tokenmaxxing to do this. Things I do to inflate my token usage metrics:Ask AI questions about the code already in the documentation. The AI pulls up the documentation, processes it, and gives me results 10x slower, but while burning lots of tokens. I could use “readthedocs” [an internal product], but then my token numbers would be lowerAsk the AI to prototype a feature that I have no intention of working on. Prompt it a few more times, then throw the whole thing awayDefault to always using the agent, even when I know I could do the work by hand much faster. Then watch it fail”

This engineer is relatively new at the company, so is concerned about job security, and is playing this game to avoid being tagged as insufficiently “AI-native” by burning far more tokens than necessary.

Salesforce: burning tokens to hit “minimum” & “ideal” targets

Elsewhere, Salesforce has created “tokenmaxxing” incentives, as well. Talking with an engineer there, I learned that the company built two tools that effectively incentivize excessive spending on tokens:

1. “Minimum” incentives with a tracking tool. There’s a Mac widget that shows your own spend, updated every 15 minutes. It also displays minimum expected spend. Last week, the target was $100 on Claude Code, and $70 on Cursor.
2. Showing everyone’s spend. A web-based tool to see the token spend of any colleague. It’s used to check where team mates’ usage is at.
3. “Maximum” spend limits that can be exceeded. Up to a week ago, there was also a maximum monthly limit of $250 for Claude Code and $170 for Cursor. However, this can be exceeded with the simple press of a button if the limit is reached. I’ve learned that last week, some engineering organisations at Salesforce had their “maximum” limit removed in order to “remove any friction from the development process.”

The message Salesforce sends to staff is clear: “use a minimum of $170/month tokens or be flagged.” Who wants to get flagged for using too few tokens? The outcome is somewhat wasteful token spend:

• Burning tokens for nothing. Devs ask Claude or Cursor: “build me X,” where X is a project or product with nothing to do with their work, and not something they’d ever ship. It’s just a way to burn tokens
• Calibrating token spend to be above average. Plenty of devs browse peers’ token spend to figure out the slightly-above average point, then use the tokens needed to hit that mark

Shopify: an example on how to avoid tokenmaxxing

The first-ever token leaderboard that I’m aware of was built by Shopify in 2025. And it worked well! Last June, the Head of Engineering at Shopify, Farhan Thawar, told me on The Pragmatic Engineer Podcast:

“We have a leaderboard where we actively celebrate the people who use the most tokens because we want to make sure they are [celebrated] if they’re doing great work with AI.

[And for the top people on the leaderboard,] I want to see why they spent say $1,000 a month in credits for Cursor. Maybe that’s because they’re building something great and they have an agent workforce underneath them!”

I asked Farhan for details on how it’s gone since. Here’s what he told me:

“We have since renamed the token leaderboard to usage dashboard: for obvious reasons, as we don’t want to encourage “competing” to make it to the top of this board. We have token spend on our internal wiki profile as well as on the usage dashboard.

We also have circuit breakers to catch “runaway agents.” So if personal spend spikes within a day, we can cut off access immediately, and you can renew if the usage spike was deliberate, or if it was a runaway agent. The circuit breaker worked well for us: we’ve not only caught runaway agents, but found bugs in our infra this way!”

Shopify’s approach seems to have worked for a few reasons:

• The usage dashboard served as a “push” for devs to use AI tools, early-on. Last year, devs were mostly experimenting with AI tools because they were not as performant as today. The usage dashboard encouraged developers to try new tools, and highlighted power users.
• Circuit breakers helped. Cutting off spend when usage spikes helped catch “runaway agents.”
• High usage is looked at. Farhan checks-in with top-spending individuals to understand the use cases. Any tokenmaxxing would likely have been spotted at this stage, which would have been a bit embarrassing for the user!

One more interesting learning Farhan shared with me: it’s more interesting to not look at “who spent the most in overall token cost?” but instead, “whose tokens cost the most?” Devs who generate tokens that come out as expensive have turned out to do in-depth work that was interesting to learn about!

Tokenmaxxing: great for AI vendors, bad for everyone else

I see very few rational reasons why incentivizing tokenmaxxing makes sense for any company. It results in increasing AI spend – by a lot! – in return for little to no value. Heck, in some cases it actually incentivises slower work – as shown by devs using the AI to answer questions when documentation is readily available – and encouraging ‘busywork’ where devs prompt projects that they don’t even want to ship. Tokenmaxxing seems to push devs to focus on stuff that makes no difference to a business.

It feels to me that a good part of the industry is using token count numbers similarly to how the lines-of-code-produced metric was used years ago. There was a time when the number of lines written daily or monthly was an important metric in programmer productivity, until it became clear that it’s a terrible thing to focus on. A lines-of-code metric can easily be gamed by writing boilerplate or throwaway code. Also, the best developers are not necessarily those who write the most code; they’re the ones who solve hard problems for the business quickly and reliably with – or without – code!

Similarly, the number of tokens a dev generates can easily be gamed, and if this metric is measured then devs will indeed game it. But doing so generates a massive accompanying AI bill!

—-

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse. This week’s issue covers:

1. New trend: token spend breaks budgets – what next? In the past 2-3 months, spending on AI agents has exploded at many tech companies, and the ramifications of this are starting to dawn on engineering leaders. We’ve sourced details from 15 companies, including the different ways they are coping with this realization.
2. New trend: more AI vendors can’t keep up with demand. Related to massively increased spending, GitHub Copilot and Anthropic are starting to limit less-profitable individual users, so they can serve business users whose spend has easily 10x’d in the last few months. The exception is OpenAI and Codex.
3. Morale at Meta hits all-time low? Business is booming but devs at Meta are furious and worried due to looming layoffs, and an invasive tracking program rolled out to all US employees.

We’re used to highly reliable systems which target four-nines of availability (99.99%, meaning about 52 minutes of downtime per year), and for it to be embarrassing to barely hit three nines (around 9 hours of downtime per year.) And yet, in the past month, GitHub’s reliability is down to one nine!

Here’s data from the third-party, “missing GitHub status page”, which was built after GitHub stopped updating its own status page due to terrible availability. Recently, things have looked poor:

This means that for every 30 days, GitHub had issues on 3 days, or issues/degradations for 2.5 hours daily (around 10% of the time.)

GitHub seems unable to keep up with the massive increase in infra load from agents. One software engineer built a clever website called “Claude’s Code” that tracks Claude Code bot contributions across GitHub. Growth in the past three months has been enormous:

Stream of GitHub outages from infra overload

GitHub’s CTO, Vladimir Fedorov, addressed availability issues in a blog post and covered three major incidents:

• 2 February: security policies unintentionally blocked access to virtual machine metadata
• 9 February: a database cluster got overloaded
• 5 March: writes failed on a Redis cluster

Software engineer Lori Hochstein did a helpful analysis of these outages and the CTO’s response, and has interesting observations:

• Saturation: the database cluster incident (9 Feb) was a case of the database getting saturated, due to higher-than-expected usage. Databases are harder to scale up than stateless services. GitHub also underestimated how much additional traffic there would be.
• Failover + telemetry gap: the 2 Feb incident was a combination of an infra issue in one region failing over to a healthy region, and making things worse with a telemetry gap (incorrect security policies were applied in the new regions which blocked access to VM metadata)
• Failover + configuration issue: the 5 March incident was uncannily similar: after a failover, a configuration issue blocked writes on a Redis cluster

It is certainly nice to get details from GitHub on these outages. It feels to me that infra strains are causing more infra issues → they trigger constraints faster → failovers are not as smooth as they should be. Could it be because GitHub keeps changing their existing systems?

Startup shows GitHub how it’s done

While GitHub struggles to keep up with the increase in load from AI agents generating more code and pull requests, a new startup called Pierre Computer claims to have built an “AI-native” solution for AI agents pushing code, which scales far beyond what GitHub can do. Pierre was founded by Jacob Thornton: formerly an engineer at Coinbase, Medium, and Twitter, and also the creator of the once-very popular Bootstrap CSS library.

Here’s what Pierre supports, which GitHub does not:

“In October [2025], Github shared they were averaging ~230 new repos per minute.

Last week we [at Pierre Computer] hit a sustained peak of > 15,000 repos per minute for 3 hours.

And in the last 30 days customers have created > 9M repos”

These are incredible numbers – if also self-reported – and something that GitHub clearly cannot get close to, at least not today! There are few details about customers, while the product – called Code.storage – seems to be in closed beta.

Still, this is the type of “git for AI agents” that GitHub has failed to build, and the type of infrastructure it needs badly.

Has GitHub lost focus and purpose?

GitHub’s reliability issues are acute enough that, if it keeps up, teams will start giving alternatives like small startups such as Pierre a try, or perhaps even consider self-hosting Git. But how did the largest Git host in the world neglect its customers, and fail to prepare its infra for an increase in code commits and pull requests?

Mitchell Hashimoto, founder of Ghostty, and a heavy user of GitHub himself, had advice on what he would do if he was in charge of GitHub, after growing frustrated with the state of its core offering. He writes (emphasis mine)

“Here’s what I’d do if I was in charge of GitHub, in order:

1. Establish a North Star plan around being critical infrastructure for agentic code lifecycles and determine a set of ways to measure that.

2. Fire everyone who works on or advocates for Copilot and shut it down. It’s not about the people, I’m sure there’s many talented people; you’re just working at the wrong company.

3. Buy Pierre and launch agentic repo hosting as the first agentic product. Repos would be separate from the legacy web product to start, since they’re likely burdened with legacy cross product interactions.

4. Re-evaluate all product lines and initiatives against the new North Star. I suspect 50% get cut (to make room for different ones).

The big idea is all agentic interactions should critically rely on GitHub APIs. Code review should be agentic but the labs should be building that into GH (not bolted in through GHA like today, real first class platform primitives). GH should absolutely launch an agent chat primitive, agent mailboxes are obviously good. GH should be a platform and not an agent itself.

This is going to be very obviously lacking since I only have external ideas to work off of and have no idea how GitHub internals are working, what their KPIs are or what North Star they define, etc.

But, with imperfect information, this is what I’d do.”

My sense is that GitHub has three concurrent problems:

• GitHub and Copilot are entangled with Microsoft’s internal politics. GitHub’s Copilot in 2021 was the first massively successful “AI product.” Microsoft took the “Copilot” brand and used it across all of their product lines, creating low-quality AI integrations. Simultaneously, internal Microsoft orgs like Azure and Microsoft AI were trying to get their hands on GitHub, which is one of the most positive developer brands at Microsoft.
• GitHub has no leader, seemingly by design. GitHub’s last CEO was Thomas Dohmke, who stepped down voluntarily, and Microsoft never backfilled the CEO role; instead carrying out a reorg to make GitHub part of Microsoft’s AI group and stripping its independence. It seems the “Microsoft AI” side won that battle.
• GitHub has no focus, and is stuck chasing Copilot as a revenue source. GitHub has no CEO and is caught up in internal politics, so, what can GitHub teams do? The safest bet is to increase revenue and the best way to do that is by investing more into GitHub Copilot, and ignoring long-term issues like reliability.

I agree with Mitchell: GitHub has no “North Star” and we see a large org being dysfunctional. That lack of vision – and CEO – is hitting hard:

• GitHub Copilot went from the most-used AI agent in 2021, to be overtaken by Claude Code, and is soon to be overtaken by Cursor.
• As a platform, GitHub has no vision for how to evolve to support AI agents. Sure, GitHub has an MCP server, but it has no “AI-native git platform” that can handle the massive load AI agents generate.
• GitHub keeps shipping small features and improvements without direction. For example, in October 2025, they started to work on stacked diffs. However, when it ships, the stacked diffs workflow might be mostly obsolete – at least with AI agents!

It’s easy to win a market when you do one thing better than anyone else in the world. Right now, GitHub is doing too many things and doing a subpar job with Copilot, its platform, and AI infra.

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse.

Catch up with recent The Pragmatic Engineer issues:

• Scaling Uber with Thuan Pham (Uber’s first CTO — podcast). We went into topics like scaling Uber from constant outages to global infrastructure, the shift to microservices and platform teams, and how AI is reshaping engineering.
• Building WhatsApp with Jean Lee (podcast): Jean Lee, engineer #19 at WhatsApp, on scaling the app with a tiny team, the Facebook acquisition, and what it reveals about the future of engineering.
• What will the Staff Engineer role look like in 2027 and beyond? What happens to the Staff engineer role when agents write more code? Actually, they could be more in demand than ever!

An interesting trend highlighted by The Wall Street Journal: companies want to hire for FDE roles, but devs are just not that interested:

“Job postings on Indeed grew more than 10-fold in 2025 compared with 2024. The number of public company transcripts mentioning the role jumped to 50 from eight over the same period, according to data from AlphaSense.

The only problem? Few engineers want the job, which has historically been seen as demanding, undesirable, and less prestigious than product-focused engineering roles.

“Everyone wants them and there’s only maybe 10% of the market that wants that role,” said Patrick Kellenberger, president and chief operating officer at Betts Recruiting.“

Last summer, we covered the rise of the FDE role, and looked into what it’s like. Back then, this is how I visualized what was then a very hot role:

At the companies where I interviewed FDE folks – OpenAI and Ramp – the role seemed to live up to this visualization. However, I’ve since talked with two engineers who took FDE roles and were disappointed. This is how they saw it, in practice:

The role seems akin to a “sales engineer” where FDEs help close the deals, or a solutions engineer (or even consultant), where FDEs deploy to a customer to build them a solution. They don’t contribute back into the platform, and don’t do much that’s considered “software engineering” beyond integrating software which the product team built.

Some engineers figure out the nature of the role during the interview process and pass on it. Meanwhile, some others take the job and later quit. Here’s what a dev told me who accept an FDE role at a company, but didn’t find what they expected:

“This FDE job was a typical IT services mindset. The company wanted to use me more on the engagement lead side, and nothing on software development. It’s not what I signed up for, and I didn’t like the vibe and culture. I quit 4 weeks later.”

In today’s job market, if there’s high demand for a role which pays decently but attracts little interest from engineers, there’s always a reason!

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse.

Catch up with recent The Pragmatic Engineer issues:

• Building WhatsApp with Jean Lee (podcast): Jean Lee, engineer #19 at WhatsApp, on scaling the app with a tiny team, the Facebook acquisition, and what it reveals about the future of engineering.
• The Pulse: What will the Staff Engineer role look like in 2027 and beyond? What happens to the Staff engineer role when agents write more code? Actually, they could be more in demand than ever!
• From IDEs to AI Agents with Steve Yegge (podcast): Steve Yegge on how AI is reshaping software engineering, the rise of “vibe coding,” and why developers must adapt to a rapidly changing craft.

If you’ve been forwarded this email, you can subscribe here to get issues like this in your inbox.

Today’s issue of The Pulse focuses on a single event because it’s a significant one with major potential ripple effects. On Tuesday, Cloudflare shocked the dev world by announcing that they have rewritten Next.js in just one week, with a single developer who used only $1,100 in tokens:

There are several layers to dig into here:

1. The Next.js ecosystem: a recap. Close to half of React devs use Next.js, and the best place to deploy Next.js is on Vercel – partly thanks to its proprietary build output.
2. What Cloudflare did with Next.js. Replacing the build engine in Next.js with the more standard Vite one, allowing Next.js apps to be easily deployed on Cloudflare.
3. AI brings the impossible within reach. What would take years in engineering terms was executed in one week with some tokens.
4. “AI slop” still an issue. Contrary to Cloudflare’s claims, vinext is not production-ready, and will need plenty of cleanup and auditing to make it on par with Next.js.

1. The Next.js ecosystem: a recap

First, some background. Next.js is the most popular fullstack React framework and around half of all React devs use it, as per recent research such as the 2025 Stack Overflow developer survey. Next.js is an open source project, built and mostly maintained by Vercel, which is the preferred deployment target for Next.js applications for many reasons. One of them is that Next.js is ideal to deploy to Vercel because Next.js applications are built with Vercel’s Turbopack build tool. The output of a build is a proprietary format. As Netlify engineer Eduardo Bouças writes:

“The output of a Next.js build has a proprietary and undocumented format that is used in Vercel deployments to provision the infrastructure needed to power the application.

This means that any hosting providers other than Vercel must build on top of undocumented APIs that can introduce unannounced breaking changes in minor or patch releases. (And they have)”.

Next.js is an interestingly built project, where everything is open source, and the best place to deploy a Next.js application is on Vercel, as it’s optimized to run undocumented build artifacts the most efficiently. This is a smart strategy from Vercel which competitors will dislike, as any hosting provider would prefer Next.js to produce a standard build format. To do this, the build engine, Turbopack, would need to be replaced with something more standard.

Let’s talk about build tools for web development. According to the State of JS 2025 survey, the most popular in the web ecosystem are:

1. Vite: the most popular choice for new projects due to its speed and developer experience. Uses projects like esbuild and Rollup under the hood
2. Webpack: a legacy tool that’s not very performant, but still widely deployed in older projects
3. Turbopack: Created by Vercel and optimized for larger Next.js applications. Built in Rust and intended to be more performant
4. Bun: a relatively new, all-in-one runtime and bundler. Anthropic acquired the team in December, and some Bun folks are now focused on improving Claude Code’s performance.

So, most of the web ecosystem uses Vite as a build tool; Next.js uses Turbopack, and the majority of React applications with a full-stack React framework use Next.js. Basically, most devs using Next.js are likely to use Vite as their build tool.

2. What Cloudflare did with Next.js

Here’s a naive idea: what if Next.js used Vite to generate build outputs? In that case, build outputs would be standardized and would run equally well on any cloud provider, as there would be nothing proprietary or undocumented to Vercel.

And this is what Cloudflare did: replace Turbopack with Vite and call the new package ‘vinext’:

Buried midway in the announcement is how this project is experimental and not at all guaranteed to work okay: it’s a ‘use-at-own-risk’ project. Still, the mere fact of this development feels like an earthquake in the tech world because of how it was pulled off.

3. AI brings the impossible within reach

In a blog post announcing the project, Cloudflare claims only one engineer “rebuilt” the whole thing in a way that’s trivial to deploy to Cloudflare’s own infrastructure, and only cost $1,100 in tokens. From Cloudflare’s statement:

“Last week, one engineer and an AI model rebuilt the most popular front-end framework from scratch. The result, vinext (pronounced “vee-next”), is a drop-in replacement for Next.js, built on Vite, that deploys to Cloudflare Workers with a single command. In early benchmarks, it builds production apps up to 4x faster and produces client bundles up to 57% smaller. And we already have customers running it in production.

The whole thing cost about $1,100 in tokens”.

What Cloudflare did:

• Took the Next.js public API
• Reimplemented behaviour using Vite
• Created build output whose behaviour matches the “original” Next.js implementation

After 10 years, the core of Next has around 194,000 lines of code (LOC)**. Meanwhile, vinext is about 67,000 lines of code which suggests a much leaner implementation: for example, vinext does not need to support legacy Next APIs, and vinext currently supports 94% of the Next.js API (and it’s safe to assume they left complex edge cases in the remaining 6%).

** the Next.js repository is closer to 2M lines of code: 1M is bundled dependencies (eg React bundles, CSS build etc), tests are 308,000 LOC, Turbopack 311,000 LOC.

Pre-AI, this reimplementation would have taken years of engineering time to complete. Doing what Cloudflare did was always possible in theory, but never seemed practical. I mean, why have a team of engineers spend potentially years on generating a standardized build output for Next.js apps? Even if they did, the dev community would have doubts about whether Cloudflare would maintain the project.

This is the thing with forking or rewriting open source projects: a major value proposition for commercial open source is to know that they will be maintained. Vercel has proved it’s a reliable custodian of Next.js for the past 10 years. Without AI, it could be assumed that any new reimplementation would eventually run out of steam.

Separately but relatedly, Cloudflare has now proved that the cost of rewriting existing software has become ~100x cheaper, thanks to AI, and this economy is likely to be the case for maintenance, too. Considering how trivial it was to rebuild one of the more complex open source projects, this augers well for it being trivial and much cheaper to maintain in the future. Potentially, Cloudflare no longer needs to budget an engineering team only for maintenance, if a single engineer could maintain the project, part-time!

Cloudflare had a project measured in engineering years, and completed it in one engineering week! It just took a single engineer using OpenCode (open source coding agent), Opus 4.5, and a bunch of tokens, then: ‘boom’, vinext was born.

4. “AI slop” still an issue

There are questions about the quality of vinext, though. Vercel, naturally, is unhappy and hit out at the obvious weakness that vinext is unfit for production usage because it’s insecure. Vercel CEO, Guillermo Rauch, did not miss a beat by tying Cloudflare’s effort to the “vibe coding” stereotype of sloppy work executed with a lack of understanding:

Guillermo has a point: anyone who stopped reading Cloudflare’s launch announcement after the first few sentences would assume it’s production-ready, with the first paragraph of this announcement closing with:

“And we already have customers running it in production.”

However, Cloudflare doesn’t share the rather crucial detail that “running in production” means that vinext has been deployed onto a beta site, until more than 1,000 words (around 2–3 pages) into the announcement:

“We want to be clear: vinext is experimental. It’s not even one week old, and it has not yet been battle-tested with any meaningful traffic at scale. (...)

We’ve been working with National Design Studio, a team that’s aiming to modernize every government interface, on one of their beta sites, CIO.gov.

Oh. So, “customers running it in production” at Cloudflare apparently means “customer running a beta site in production without meaningful traffic.” This is a first from the infrastructure giant, which usually prides itself on accurate statements!

This detail was also absent when Cloudflare’s CEO and CTO were boosting vinext like it was a mature, battle-tested product. In that context, Vercel’s raising of the issue of security vulnerabilities is more than fair game, in my view.

Still, all that doesn’t alter the core learning from this project: that AI has the power to drastically reduce engineering time by up to ~100x and deliver usable-enough output, for relatively negligible financial cost. Just keep in mind that security and reliability issues will probably take plenty of extra time and effort to address.

5. New attack vector on commercial open source?

If arch-rivalries exist in tech, then Cloudflare and Vercel are a prime example. Both are gunning to become the most popular platform for developers to deploy their code, and the CEOs are regularly seen in public taking shots at the other side. One such spat happened in March, as covered at the time:

“Things kicked off on social media, with developers confused about the severity of the incident, and about why Next.js seemed silent, and also why Cloudflare sites were breaking due to its fix for the CVE causing its own issues. It was at that point that Cloudflare’s CEO, Matthew Prince, entered the chat to accuse Vercel of not caring about security:

Given the security incident was ongoing, this felt a bit “below the belt” by the Cloudflare chief. Criticizing rivals is fair game, but why not wait until the incident is over? The punch landed, and Vercel’s CEO Guillermo Rauch is not someone to take it lying down, so he hit back.

Cloudflare’s CEO then responded with a cartoon implying that although Vercel is much larger than its competitor Netlify, Cloudflare is 100x bigger than both, and could stomp them into the ground at will.”

Serving the public interest wasn’t why Cloudflare rewrote Next.js: they did it because they want Next.js sites to be deployed onto Cloudflare, but doing so made little sense until now because Next.js produced bespoke build output optimized for Vercel’s infrastructure. With this change, Cloudflare claims it provides superior performance when hosting Next.js apps, according to their own measurements.

I’d just add that performance is important for developers, but other things matter, too. Cost, reliability, developer experience, and how much devs like a company, are all factors in choosing between vendors. Also, performance measurements from a vendor about its own service must be taken with a large pinch of salt.

Zooming out from this episode, it seems that AI is bringing the value of existing commercial open source moats into question. Vercel carved out a clever open source strategy that helped turn its open source investment into business revenue:

1. Build and maintain Next.js, delivering the best developer experience (DX).
2. Optimize Vercel to serve the specific (and undocumented) build output of Next.js.
3. Most developers onboarding to Next.js will decide to deploy on Vercel to get the most benefit, in terms of DX and performance.
4. … repeat for years while the business becomes worth billions! (Vercel was valued at $9B last October).

Underpinning this success are some assumptions:

1. Next.js will remain the #1 choice for developers to build React applications, thanks to ongoing investment.
2. It is expensive to rewrite Next.js to be deployable and performant on another cloud vendor.
3. Even if someone did #2, developers would be skeptical and not switch over.

Vercel can invest in #1 to keep Next as best-in-class, while knowing that the risk of #2 occurring is minor. However, Cloudflare has now “cloned” Next, and can easily keep up with all changes in the future, and port them back to vinext.

But AI makes it trivial to “piggyback” off any commercial open source project, which is a massive problem for commercial open source startups. It puts all the effort and investment into building and maintaining Next.js, while Cloudflare enjoys the benefit of this hard work (the Next.js public API) which is easily deployable to Cloudflare, and it can now undercut Vercel on price. For all future Next.js changes, Cloudflare will just sync it to vinext, using AI!

WordPress had a similar problem, with WP Engine “piggybacking” off its work and undercutting their pricing in 2024. As I analyzed at the time:

“Free-riding on permissive open source is too tempting to pass on for other vendors. WP Engine uses a common loophole of contributing almost nothing in R&D to WordPress, while selling it as a managed service. This means that they could either easily undercut the pricing of larger players like Automattic which do spend on WordPress’s R&D. Alternatively, a company like WP Engine could charge as much, or more, as Automattic, but be able to spend a lot more on marketing, while being similarly profitable. “Saving” on R&D gives the “free-riders” plenty of options to grow their businesses: options not necessarily open to Automattic while they invest as much into R&D as they do.

Commercial open source vendors pressure to end “freeriding”. Automattic is likely facing lower revenue growth, with customers choosing vendors like WP Engine which offer a similar service — getting these customers either via a cheaper price or thanks to more marketing spend. This legal fight could be an effort to force WP Engine to stop eating Automattic’s lunch, or perhaps get WP Engine to sell to Automattic, which would cement its leading status in managed Wordpress, while also boosting revenue by $400M a year – according to its own figures”.

Vercel managed to avoid the “free-riding” problem with Next.js, but that’s no longer possible now that AI makes it trivial to rewrite.

6. Defense or offense?

How should commercial open source companies respond to the threat that a competitor can easily rewrite the software behind the managed solutions which they sell as services?

One obvious response is to make tests private, so that replication is harder for AI. One thing that made it so easy for Cloudflare to rewrite Next was the project’s comprehensive test suite. From their announcement (emphasis mine):

“We also want to acknowledge the Next.js team. They’ve spent years building a framework that raised the bar for what React development could look like. The fact that their API surface is so well-documented and their test suite so comprehensive is a big part of what made this project possible.”

Database solution SQLite is famous for its incredible test suite. What some people don’t know is that while core SQLite tests are open source, its most comprehensive test suite – TH3 – is closed source. SQLite monetizes its advanced infrastructure as a service for purchase. This is a fair tradeoff: for most contributors, the basic open source tests work well enough. For enterprise users or customers who really care about correctness, it makes sense to purchase advanced testing services from the service’s creator.

Open source canvas project, tldraw, announced it will relocate its test suite to a closed source repository; a move which makes plenty of sense. Here’s commentary from Simon Willison:

“It’s become very apparent over the past few months that a comprehensive test suite is enough to build a completely fresh implementation of any open source library from scratch, potentially in a different language.”

In the event, tldraw’s announcement turned out to be a joke, but who’s laughing now? An open source project with excellent tests is an easy target for an AI agent to execute a full rewrite of it.

Could new licenses be created for the AI era? Existing open source licenses were created on the assumption that humans read open source code, and humans modify it. Agents break that assumption.

Could we see new license types emerge to ban AI agents from modifying projects’ source code? It seems pretty far-fetched and hard to implement, but not beyond the realms of possibility.

AI agents are still very new, and going mainstream in tech. Once they break into other industries, I wouldn’t be surprised if legal frameworks are reworded to also apply to AI agents. If and when this happens, it would open the path for open source licenses to distinguish between agents and humans.

What is a moat, if code can be trivially ported? A team operating a popular open source project can no longer assume it’s expensive to fork or to be completely rewritten, meaning it makes sense to focus on other moats, such as:

• Outstanding (paid) support. AI could make this much easier at a higher quality, if done right.
• Smaller open core, larger closed source part. “Open core” as a business model has been dominant for commercial open source: keep the core of the software open source, while advanced enterprise features are source available or closed source. I would expect more companies to move their additional services to closed source, not source available.
• In-person connection and community. Projects with a real-world community will form a sense of connection that goes beyond code. For example, it’s hard to imagine vinext meetups popping up – whereas there are many Next.js communities.
• Infrastructure and hardware remains a massive moat. In a world where software is trivial to copy, infrastructure remains a moat. Commercial open source might make most sense for players that own and operate superior infrastructure layers than their rivals: and being able to offer lower cost, higher reliability, lower latency, higher performance, or a combination of these.

7. AI-world reality

One of the single best AI use cases is full-on rewrites of well-tested products. I estimate that AI sped up the creation of vinext by at least 100x, which is massive. But we don’t really see efficiency boosts of anything like that with AI tools, in general. As Laura Tacho shared at The Pragmatic Summit in San Francisco, the average self-reported efficiency ‘AI gain’ seems to be circa 10%.

I suspect this vast chasm in efficiency boosts is because AI is many times more efficient at “no-brainer tasks” where correctness can be verified with tests, versus those which are more open ended or involve more creativity.

In general, tests are incredibly important for efficient AI usage. On The Pragmatic Engineer Podcast, Peter Steinberger stressed how important “closing the loop” in his developer flow is by instructing the AI to test itself, and ensuring the AI has tests to run that verify correctness.

Automated tests were always considered a best practice for creating maintainable code. Now, having a codebase with extensive tests is the baseline to make AI agents work productively for refactors, rewrites – or even adding new features and verifying that things did not break!

Vendors will start to deploy “migration AI agents” to move customers over to their own stacks. This got lost in Cloudflare’s announcement, but it’s important:

vinext includes an Agent Skill that handles migration for you. It works with Claude Code, OpenCode, Cursor, Codex, and dozens of other AI coding tools. Install it, open your Next.js project, and tell the AI to migrate:

> npx skills add cloudflare/vinext

Then open your Next.js project in any supported tool and say:

> migrate this project to vinext

The skill handles compatibility checking, dependency installation, config generation, and dev server startup. It knows what vinext supports and will flag anything that needs manual attention.

This is very clever from Cloudflare, and a true “AI-native” move. They have not only used AI to migrate Next.js, but also built an “AI plugin” (a skill) to help customers migrate their existing codebases over to vinext – and deploy on Cloudflare!

This move will surely be copied by other vendors, since migrations which are tedious for humans are much less effort with agents.

AI is making the tech industry more ruthless when it comes to business practices. Laura Tacho said something interesting at The Pragmatic Summit:

“AI is an accelerator, it’s a multiplier, and it is moving organizations in different directions.”

AI seems to be accelerating the ruthlessness of competition for customers and the speed at which this happens. In one week, Cloudflare rebuilt Next.js, and it’s attacking Vercel full-on: claiming their “vibe coded” alternative is more performant and production-ready, and burying at the foot of the launch announcement the crucial information that vinext is very much experimental.

I sense vendors are realizing that there’s a limited amount of time in which to use AI to their advantage, and some will decide to use it like Cloudflare has.

On the other hand, AI could be great news for non-commercial open source. AI presents as a threat to commercial open source because it removes existing moats which make code hard to fully rewrite. However, beyond that, AI could help non-commercial open source to thrive:

• With AI, it’s easy to fork an open source project and keep the fork in-sync with the original.
• It’s trivial to instruct AI to rewrite an open source project to another language or framework.
• …and it’s equally trivial for AI to add features to a fork.

For these reasons, I believe there could be a lot more forks and rewrites to come, and more open source projects and code, in general.

Takeaways

Personally, I could not have imagined things changing this quickly in software. Rewriting Next.js in a single week, even to a version that is not quite there – but mostly works? This was out of the question as recently as a few months ago.

Things changed around last December, when Opus 4.5 and GPT-5.2 came out and proved capable of writing most of the code. What used to be expensive is now cheap – like rewriting complete projects – and we still need to learn what the “new” expensive parts of software engineering are.

All this is new territory for everyone. To succeed in the tech industry, you need to be able to capitalize upon change, as Cloudflare has clearly done in this case by making the most of an opportunity created by new technology. It’s unclear how popular vinext will become, and how much of a moat Vercel has around the broader Next.js ecosystem, but I suspect that it’d take more than a Next rewrite to make Cloudflare into a viable Next.js platform-as-a-service provider.

1. SaaS is a pure software product. People who pay SaaS vendors do so because it’s cheaper to buy this software than build it.
2. LLMs dramatically reduce the time and cost of building custom software.
3. Therefore, most SaaS vendors will go out of business because most companies/teams will prompt an LLM to write the software they need, such as for ticketing, meetings, customer relationship management, etc.

The reason for my scepticism has been that SaaS such as HR software Workday is more than just software. Workday, for example, keeps up with compliance requirements (e.g., for holiday pay in different countries), guarantees correctness (e.g., payslips that comply with local regulations), and over time the software keeps up to date with changes in the external and internal environments.

However, this week I had first-hand experience of how ridiculously easy it is now to replace SaaS with LLMs. On my website – pragmaticengineer.com – I have a testimonials section, which displays real LinkedIn and X posts about this publication. It cost $120/year for a small service called Shoutout.io, and looked like this:

And this is the backend: nothing fancy, just a way to add, edit, reorganize, and delete testimonials.

I was a customer for four years and logged in perhaps once a year. My latest login was to get an annual invoice for my expenses. Unfortunately, the billing section was broken, so I emailed support and they sent me a broken link instead of the invoice. This was frustrating: why pay for a SaaS with broken billing? I couldn’t even tell what they would charge me next year.

So I asked myself if I could rebuild my own use case with an LLM, and do it rapidly. My use case was much simpler than the SaaS itself:

• Display existing testimonials in a similar way
• Make it easy to add new ones, e.g., store testimonials in some JSON format
• Make it look good

To my surprise, this whole effort from start to finish took exactly 20 minutes with Codex. The steps I took were straightforward enough:

• Asked Codex to make a plan on how to remove this third-party dependency and host all testimonials in my codebase (a GitHub repo, deployed onto Netlify)
• Tweaked the plan: I pushed for a modular approach where testimonials are in a separate JSON file, and they get generated into HTML with a compile-time build step
• Added this build step both locally and as a build trigger on Netlify
• Tested the solution
• Tweaked the UX and generated a schema
• Deployed it

The end result is visually the same as before, except I no longer have a third-party dependency rendering all of this!

What does this mean for SaaS products and software engineers?

What it means for software engineers:

• Devs are (probably) a lot more comfortable using the command line for future updates than regular users. To add a future testimonial, I’ll need to turn to my AI agent to insert it in my codebase, and I’ll then need to verify that it looks good. This is not a big deal for me, but it might be a dealbreaker for someone not comfortable with verifying the code output of an LLM.
• It’s a lot faster for a dev to “port” a SaaS than for anyone else. I first told Codex to copy the UI and it got things wrong because it tried to use a flexbox model. I had to tell it that this UI layout was not what I wanted, and then make the decision on which framework to use for the UI layout. A non-developer could probably figure all this out, but it would take longer.
• Honestly, it’s fun and interesting to rewrite a third-party feature. I recommend it. Part of why I took on this project is because I expected it to be an interesting challenge. I thought the effort would be more than what it was, and I’ve learned more about how well these tools work. I also used Codex in order to experience it more.

What this could mean for SaaS software:

• Rebuilding a SaaS still feels much harder than rebuilding your specific use case. I did not “rebuild” Shoutout in any way. Shoutout has 10x or more features, like adding quotes from 10 different platforms, authentication, billing (which didn’t work for me), and more.
• A SaaS that doesn’t give ongoing value is at risk of being replaced by customers. Shoutout doesn’t provide ongoing value after it displays my testimonials, and this static nature means it’s easy to replace. In contrast, it would be harder to rebuild if I paid for the platform to stay compliant, provide analytics or alerting, and do other real-time things that helped my business.
• Buying and selling SaaS businesses could become less profitable. The original version of Shoutout that I signed up for in 2021 was built in 2020 by an independent developer. In 2022, this developer sold this micro-SaaS to a product studio. Then, in 2025, Shoutout was sold again to new developers. From my point of view, nothing changed except that the billing system broke. I assume the buyers of this SaaS figured that revenue could keep rising with zero investment. But perhaps at some point that ceases to be true when people get fed up with a broken product and quit – especially when doing so is cheaper.

“Broken windows” not being fixed is less acceptable than it used to be. My journey away from Shoutout began with its billing system being broken. For example, below is what I saw when I went to my billing section to see the invoices:

As well as this, the customer support sent me a broken link in response to my email. That was enough for me to decide to replace this dependency, and I was surprised by how easy this was with an LLM and knowing what I wanted it to build. By the time customer support sent me a working link two hours later, I had finished migrating off the SaaS.

It feels like something valuable is being taken away, and suddenly. It took a lot of effort to get good at coding and to learn how to write code that works, to read and understand complex code, and to debug and fix when code doesn’t work as it should. I still remember how daunting my first “real” programming class was at university (learning C), how lost I felt on my first job with a complex codebase, and how it took years of practice, learning from other devs, books, and blogs, to get better at the craft. Once you’re pretty good, you have something that’s valuable and easy to validate by writing code that works!

Some of my best memories of building software are about coding. Being “locked in” and balancing several ideas while typing them out, of being in the zone, then compiling the code, running it and seeing that “YES”, it worked as expected!

It’s been a love-hate relationship, to be fair, based on the amount of focus needed to write complex code. Then there’s all the conflicts that time estimates caused: time passes differently when you’re locked in and working on a hard problem.

Now, all that looks like it will be history.

I wonder if I’ll still get the same sense of satisfaction from the fact that writing complicated code is hard? Yes, AI is convenient, but there’s also a loss.

Or perhaps with AI agents, being “in the zone” will shift to thinking about higher-level problems, while instructing more complex code to be written?

This was a section from my analysis piece When AI writes almost all code, what happens to software engineering?. Read the full one here.

A mere two weeks after Cloudflare suffered a major outage and took down half the internet, the same thing has happened again. Last Friday, 5th December, thousands of sites went down or partially down once more, in a global Cloudflare outage lasting 25 minutes.

As per last time, Cloudflare was speedy to share a full postmortem on the same day. It estimated that 28% of Cloudflare’s HTTP traffic was impacted. The cause of this latest outage was Cloudflare making a seemingly innocent – but global – configuration change that went on to take out a good portion of Cloudflare, globally, until being reverted. Here’s what happened:

• Cloudflare was rolling out a fix for a nasty React security vulnerability
• The fix caused an error in an internal testing tool
• The Cloudflare team disabled the testing tool with a global killswitch
• As this global configuration change was made, the killswitch unexpectedly caused a bug that resulted in HTTP 500 errors across Cloudflare’s network

In this latest outage, Cloudflare was burnt by yet another global configuration change. The previous outage in November happened thanks to a global database permissions change. In the postmortem of that incident, the Cloudflare team closed with this action item:

“Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input”

This change would make it so that Cloudflare’s configuration files do not propagate immediately to the full network, as they still do now. But making all global configuration files have staged rollouts is a large implementation that could take months. Evidently, there wasn’t time to make it yet, and it has come back to bite Cloudflare.

Unfortunately for Cloudflare, customers are likely to find unacceptable a second outage with similar causes to a previous one, only weeks ago. If Cloudflare proves unreliable, customers should plan to onboard to backup CDNs at the very least, and a backup CDN vendor will do its best to convince new customers to use it as the primary CDN.

Cloudflare’s value-add rests on rock-solid reliability without customers needing to budget for a backup CDN. Yes, publishing postmortems on the same day as an outage occurs helps restore trust, but that will crumble anyway with repeated large outages.

To be fair, the company is doubling down on implementing staged configuration rollouts. In its postmortem, Cloudflare is its own biggest critic. CTO Dane Knecht reflected:

“[Global configuration changes rolling out globally] remains our first priority across the organization. In particular, the projects outlined below should help contain the impact of these kinds of changes:Enhanced Rollouts & Versioning: Similar to how we slowly deploy software with strict health validation, data used for rapid threat response and general configuration needs to have the same safety and blast mitigation features. This includes health validation and quick rollback capabilities among other things.Streamlined break glass capabilities: Ensure that critical operations can still be achieved in the face of additional types of failures. This applies to internal services as well as all standard methods of interaction with the Cloudflare control plane used by all Cloudflare customers.“Fail-Open” Error Handling: As part of the resilience effort, we are replacing the incorrectly applied hard-fail logic across all critical Cloudflare data-plane components. If a configuration file is corrupt or out-of-range (e.g., exceeding feature caps), the system will log the error and default to a known-good state or pass traffic without scoring, rather than dropping requests. Some services will likely give the customer the option to fail open or closed in certain scenarios. This will include drift-prevention capabilities to ensure this is enforced continuously.
These kinds of incidents, and how closely they are clustered together, are not acceptable for a network like ours”.

Global configuration errors often trigger large outages

There’s a pattern of implicit or explicit global configuration errors causing large outages, and some of the biggest ones in recent years were caused by a single change being rolled out to a whole network of machines:

• DNS and DNS-related systems like BGP: DNS changes are global by default, so it’s no wonder that DNS changes can cause global outages. Meta’s 7-hour outage in 2021 was related to DNS changes (more specifically, Border Gateway Protocol changes.) Meanwhile, the AWS outage in October started with the internal DNS system.
• OS updates happening at the same time, globally: Datadog’s 2023 outage cost the company $5M and was caused by Datadog’s Ubuntu machines executing an OS update within the same time window, globally. It caused issues with networking, and it didn’t help that Datadog ran its infra on 3 different cloud providers across 3 networks. The same kind of Ubuntu update also caused a global outage for Heroku in 2024.

Globally replicating configs: in 2024, a configuration policy change was rolled out globally and crashed every Spanner database node straight away. As Google concluded in its postmortem: “Given the global nature of quota management, this metadata was replicated globally within seconds”.

Implementing gradual rollouts for all configuration files is a lot of work. It’s also invisible labor because when done well, then its benefits will be undetectable, except in the absence of incidents, thanks to better infrastructure!

The largest systems in the world will likely have to implement safer ways to roll out configs – but not everybody needs to. Staged configuration rollout doesn’t make much sense for smaller companies and products because this infra work slows down product development.

It doesn’t just slow down building, but every deployment, too, and this friction is designed to make everything slower. As such, they don’t make much sense unless the stability of mature systems is more important than fast iteration.

Software engineering is a field where tradeoffs are a fact of life, and universal solutions don’t exist. The development which worked for a system with 1/100th of the load and users a year ago, may not make sense today.

This was one out of the four topics covered in this week’s The Pulse. The full edition additionally covers:

1. Industry Pulse. Poor capacity planning at AWS, Meta moves to a “closed AI” approach, a looming RAM shortage, early-stage startups hiring slower than before, how long it takes to earn $600K at Amazon and Meta, Apple loses execs to Meta, and more
2. How the engineering team at Oxide uses LLMs. They find LLMs great for reading documents and lightweight research, mixed for coding and code review, and a poor choice for writing documents – or any kind of writing, really!
3. Linux officially supports Rust in the kernel. Rust is now a first-class language inside the Linux kernel, eight months after a Linux Foundation Fellow predicted more support for Rust. A summary of the pros and cons of Rust support for Linux

Read the full The Pulse issue.

A year ago, Amazon became the first tech giant to bring staff back into the office for the full five days per week. Back then, I analyzed the reasons for the change, and whether other workplaces would follow suit by dropping the widespread hybrid policy of 2-3 days/week in the office.

Now, Meta employees in the Instagram division have become the latest subjects of a full return to the office, following an announcement by the social media platform this week.

Instagram’s 5-day return to office

Instagram employees received the unexpected email on Monday, reports fellow Substacker, Alex Heath, who acquired a copy of the message. It was sent internally by Instagram CEO Adam Mosseri, who wrote:

“1. Back to the office: I believe that we are more creative and collaborative when we are together in-person. (...)

2. Fewer meetings: We all spend too much time in meetings that are not effective, and it’s slowing us down. Every six months, we’ll cancel all recurring meetings and only re-add the ones that are absolutely necessary (...)

3. More demos, less decks: Most product overviews should be prototypes instead of decks.

4. Faster decision-making: We’re going to have a more formalized unblocking process with DRIs, and I’ll be at the priorities progress unblocking meeting every week.”

This decision by Meta affects around a quarter of company staff, and it’s hard to imagine other divisions not following Instagram’s lead; after all, everything in Mosseri’s memo likely applies across the business.

Five years ago, CEO Mark Zuckerberg predicted 50% of Meta staff would work remotely by now, which didn’t happen. Indeed, with Instagram’s new 5-day RTO, I’d be surprised if 5% of Meta folks work remotely in two years’ time.

The reason for Insta’s RTO seems rooted in the leadership’s belief that in-office is more productive, as indicated by the top bullet point of Mosseri’s message. That message in full:

“I believe that we are more creative and collaborative when we are together in-person. I felt this pre-COVID and I feel it any time I go to our New York office where the in-person culture is strong.

Starting February 2, I’m asking everyone in my rollup based in a US office with assigned desks to come back full time (five days a week). The specifics:

• You’ll still have the flexibility to work from home when you need to, since I recognize there will be times you won’t be able to come into the office. I trust you all to use your best judgment in figuring out how to adapt to this schedule.
• In the NY office, we won’t expect you to come back full time until we’ve alleviated the space constraints. We’ll share more once we have a better sense of timeline.
• In MPK [Menlo Park, the HQ], we’ll move from MPK21 to MPK22 on January 26 so everyone has an assigned desk. We’re also offering the option to transfer from the MPK to SF office for those people whose commute would be the same or better with that change. We’ll reach out directly to those people with more info.
• XFN [cross-functional] partners will continue to follow their own org norms.
• There is no change for employees who are currently remote”.

From what I’ve seen of Mosseri from afar, he seems like a pretty straight shooter. It’s clear that he feels in-office creates more energy, and in Mosseri’s defense, I hear similar from many startup founders and leaders who say remote work causes a bunch of headaches: it’s harder to spot motivational problems and performance issues, information travels more slowly, and rallying teams is harder.

There’s no doubt that running a full-remote company is a lot of effort. There’s often-overlooked labor involved in hiring, onboarding, performance management, team celebrations, and even company-wide meetings – none of it is easy.

Linear is a full-remote company with nearly 50 people working there, which recently published details about how it operates. They’re introducing the concept of “coworking hubs”, flying in teams for in-person events, and holding regular off-sites, while being careful to hire people who fit the culture.

My feeling is that remote work policies at tech companies are going to become questions of their leaders’ preferences. Many devs prefer remote work: there’s fewer interruptions, more deep focus, and less commuting. Most of us would probably be just as productive – and probably more so – than when being interrupted in-office.

Leaders who prefer full-remote can cite flexibility and easier hiring from a larger pool of candidates as clear benefits. Meanwhile, those most comfortable with in-person will always have enough reasons to justify a 5-day RTO, along the lines of Mosseri’s reasoning. Advocates of hybrid setups cite balancing of focus time and efficiency.

In today’s job market, any company that pays closer to the top of the market can probably get away with five-days-a-week RTO. Meta is in this space, and although I’m sure plenty of devs will dislike the change, the alternative is to go out on the job market, accept a pay cut to join a new company, and start rebuilding your internal network.

Since we’re in the midst of a weird job market, it makes switching jobs more difficult than before, when the job market was very hot. In this respect, Instagram has external conditions on its side. For devs at Meta, one upside is that Big Tech experience opens more doors, even in this tough job market.

One caveat is that a 5-day RTO is unlikely in places where it’s hard to hire the right people. So, AI engineers and those working on AI products should be pretty safe, for instance, because those roles are incredibly in-demand, as indicated by the trend of higher base salaries for AI engineers. Based on that, few companies should want to push those workers to quit to join competitors.

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Many subscribers expense The Pragmatic Engineer Newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

One amusing detail of the November 2025 Cloudflare outage is that the realtime outage and monitoring service, Downdetector, went down, revealing a key dependency on Cloudflare. At first, this looks odd; after all, Downdetector is about monitoring uptime, so why would it take on a key dependency like Cloudflare if it means this can happen?

Downdetector was built multi-region and multi-cloud, which I confirmed by talking with Senior Director of Engineering, Dhruv Arora, at Ookla, the company behind Downdetector. Multi-cloud resilience makes little sense for most products, but Downdetector was built to detect cloud provider outages, as well. And for this, they needed to be multi-cloud!

Still, Downdetector uses Cloudflare for DNS, Content Delivery (CDN), and Bot Protection. So, why would it take on this one key dependency, as opposed to hosting everything on its own servers?

A CDN has advantages that are hard to ignore, such as:

• Drastically lower bandwidth costs – assets cached on the CDN are much faster
• Faster load times because assets on a CDN are served from Edge nodes nearer users
• Protection from sudden traffic spikes, as would be common for Downdetector, especially during outages! Without a CDN, those spikes could overload their services
• DDoS protection from bad actors taking the site offline with a distributed denial of service attack
• Reduced infrastructure requirements, as Downdetector can run on fewer servers

Downdetector’s usage patterns reflect that it’s a service very heavily used by consumers whom the business doesn’t really monetize (Downdetector is free to use.) So, Downdetector could get rid of Cloudflare, but costs would surge, the site would become slower to load, and revenue wouldn’t change.

In the end, Downdetector’s dependence on Cloudflare could be a pragmatic choice based on the business model, and how removing its upstream dependency upon Cloudflare could get very expensive!

Dhruv confirmed this and sharing more about the design choices at Downdetector:

“Building redundancy at the DNS & CDN layers would require enormous overhead. This is especially true as Cloudflare’s Bot Protection is world-class, and building similar functionality would be a lot of effort. There are hyperscalers [cloud providers] that have this kind of redundancy built in. We will look into what we can do, but with a team size in the double digits, building up a core piece of infra like this is a pretty tall order: not just for us, but for any mid-sized team.

We’ve learned that there are more things that we can improve, for the future. For example, during the outage, the Cloudflare control pane was down, but their API wasn’t. So, us having more Infrastructure as Code could have helped bring back Downdetector sooner.

On our end, we also noticed that the outage wasn’t global, so we were able to shift traffic around and reduce the impact.

One more interesting detail: Cloudflare’s Bot Protection went haywire during the outage, and started to block legitimate traffic. So, our team had to turn that off temporarily”.

Thanks very much to Dhruv and the Downdetector team for sharing details.

An unexpected highlight of publishing the book was ending up in Mongolia in June of this year, at a small-but-mighty startup called Nasha Tech. This was because the startup translated my book into Mongolian. Here's the completed book:

Here’s what happened:

A little over a year ago, a small startup from Mongolia reached out, asking if they could translate the book. I was skeptical it would happen because the unit economics appeared pretty unfavorable. Mongolia’s population is 3.5 million; much smaller than other countries where professional publishers had offered to do a translation (Taiwan: 23M, South Korea: 51M, Germany: 84M, Japan: 122M, China: 1.43B people).

But I agreed to the initiative, and expected to hear nothing back. To my surprise, nine months later the translation was ready, and the startup printed 500 copies on the first run. They invited me to a book signing in the capital city of Ulaanbaatar, and soon I was on my way to meet the team, and to understand why a small tech company translated my book!

Japanese startup vibes in Mongolia

The startup behind the translation is called Nasha Tech; a mix of a startup and a digital agency. Founded in 2018, its main business has been agency work, mainly for companies in Japan. They are a group of 30 people, mostly software engineers.

Their offices resembled a mansion more than a typical workplace, and everyone takes their shoes off when arriving at work and switches to “office slippers”. I encountered the same vibe later at Cursor’s headquarters in San Francisco, in the US.

Nasha Tech found a niche of working for Japanese companies thanks to one of its cofounders studying in Japan, and building up connections while there. Interestingly, another cofounder later moved to Silicon Valley, and advises the company from afar.

The business builds the “Uber Eats of Mongolia”. Outside of working as an agency, Nasha Tech builds its own products. The most notable is called TokTok, the “UberEats of Mongolia”, which is the leading food delivery app in the capital city. The only difference between TokTok and other food delivery apps is scale: the local market is smaller than in some other cities. At a few thousand orders per day, it might not be worthwhile for an international player like Uber or Deliveroo to enter the market.

The tech stack Nasha Tech typically uses:

• Frontend: React / Next, Vue / Nuxt, TypeScript, Electron, Tailwind, Element UI
• Backend and API: NodeJS (Express, Hono, Deno, NestJS), Python (FastAPI, Flask), Ruby on Rails, PHP (Laravel), GraphQL, Socket, Recoil
• Mobile: Flutter, React Native, Fastlane
• Infra: AWS, GCP, Docker, Kubernetes, Terraform
• AI & ML: GCP Vertex, AWS Bedrock, Elasticsearch, LangChain, Langfuse

AI tools are very much widespread, and today the team uses Cursor, GitHub Copilot, Claude Code, OpenAI Codex, and Junie by Jetbrains.

I detected very few differences between Nasha Tech and other “typical” startups I’ve visited, in terms of the vibe and tech stack. Devs working on TokTok were very passionate about how to improve the app and reduce the tech debt accumulated by prioritizing the launch. A difference for me was the language and target market: the main language in the office is, obviously, Mongolian, and the products they build like TokTok also target the Mongolian market, or the Japanese one when working with clients.

One thing I learned was that awareness about the latest tools has no borders: back in June, a dev at Nasha Tech was already telling me that Claude Code was their daily driver, even though the tool had been released for barely a month at that point!

Why translate the book into Mongolian?

Nasha Tech was the only non-book publisher to express interest in translating the book. But why did they do it?

I was told the idea came from software engineer Suuribaatar Sainjargal, who bought and enjoyed the English-language version. He suggested translating the book so that everyone at the company could read it, not only those fluent in English.

Nasha Tech actually had some in-house experience of translation. A year earlier, in 2024, the company translated Matt Mochary’s The Great CEO Within as a way to uplevel their leadership team, and to help the broader Mongolian tech ecosystem.

Also, the company’s General Manager, Batutsengel Davaa, happened to have been involved in translating more than 10 books in a previous role. He took the lead in organizing this work, and here’s how the timelines played out:

• Professional translator: 3 months
• Technical editor revising the draft translation: 1 month
• Technical editing #2 by a Support Engineer in Japan: 2 months
• Technical revision: 15 engineers at Nasha Tech revised the book, with a “divide and conquer” approach: 2 months
• Final edit and print: 1 month

This was a real team effort. Somehow, this startup managed to produce a high-quality translation in around the same time as it took professional book publishers in my part of the world to do the same!

A secondary goal that Nasha Tech had was to advance the tech ecosystem in Mongolia. There’s understandably high demand for books in the mother tongue; I observed a number of book stands selling these books, and book fairs are also popular. The translation of my book has been selling well, where you can buy the book for 70,000 MNTs (~$19).

Book signing and the Mongolian startup scene

The book launch event was at Mongolia’s startup hub, called IT Park, which offers space for startups to operate in. I met a few working in the AI and fintech spaces – and even one startup producing comics.

I had the impression that the government and private sector are investing heavily in startups, and want to help more companies to become breakout success stories:

• IT Park report: the country’s tech sector is growing ~20%, year-on-year. The combined valuation of all startups in Mongolia is at $130M, today. It’s worth remembering that location is important for startups: being in hubs like the US, UK, and India confers advantages that can be reflected in valuations.
• Mongolian Startup Ecosystem Report 2023: the average pre-seed valuation of a startup in Mongolia is $170K, seed valuation at $330K, and Series A valuation at $870K. The numbers reflect market size; for savvy investors, this could also be an opportunity to invest early. I met a Staff Software Engineer at the book signing event who is working in Silicon Valley at Google, and invests and advises in startups in Mongolia.
• Mongolian startup ecosystem Map: better-known startups in the country.

Two promising startups from Mongolia: Chimege (an AI+voice startup) AND Global (fintech). Thanks very much to the Nasha Tech team for translating the book – keep up the great work!

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Before we start: I’m excited to share something new: The Pragmatic Summit.

Four years ago, The Pragmatic Engineer started as a small newsletter: me writing about topics relevant for engineers and engineering leaders at Big Tech and startups. Fast forward to today, and the newsletter crossed one million readers, and the publication expanded with a podcast as well.

One thing that was always missing: meeting in person. Engineers, leaders, founders—people who want to meet others in this community, and learn from each other. Until now that is:

In partnership with Statsig, I’m hosting the first-ever Pragmatic Summit. Seats are limited, and tickets are priced at $499, covering the venue, meals, and production—we’re not aiming to make any profit from this event.

Apply to attend the Summit

I hope to see many of you there!

Cloudflare takes down half the internet – but shares a great postmortem

On Tuesday came another reminder about how much of the internet depends on Cloudflare’s content delivery network (CDN), when thousands of sites went fully or partially offline in an outage that lasted 6 hours. Some of the higher-profile victims included:

• ChatGPT and Claude
• Canva, Dropbox, Spotify,
• Uber, Coinbase, Zoom
• X and Reddit

Separately, you may or may not recall that during a different recent outage caused by AWS, Elon Musk noted on his website, X, that AWS is a hard dependency for Signal, meaning an AWS outage could take down the secure messaging service at any moment. In response, a dev pointed out that it is the same for X with Cloudflare – and so it proved earlier this week, when X was broken by the Cloudflare outage.

That AWS outage was in the company’s us-east-1 region and took down a good part of the internet last month. AWS released incident details three days later – unusually speedy for the e-commerce giant – although that postmortem was high-level and we never learned exactly what caused AWS’s DNS Enactor service to slow down, triggering an unexpected race condition that kicked off the outage.

What happened this time with Cloudflare?

Within hours of mitigating the outage, Cloudflare’s CEO Matthew Prince shared an unusually detailed report of what exactly went wrong. The root cause was to do with propagating a configuration file to Cloudflare’s Bot Management module. The file crashed Bot Management, which took Cloudflare’s proxy functionality offline.

Here’s a brief overview of how Cloudflare’s proxy layer works at a high level. It’s the layer that protects the “origin” resources of customers – minimizing network traffic to them by blocking malicious requests and caching static resources in Cloudflare’s CDN:

Here’s how the incident unfolded:

A database permissions change in ClickHouse kicked things off. Before the permissions changed, all queries to fetch feature metadata (to be used by the Bot Management module) would have only been run on distributed tables in Clickhouse, in a database called “default” which contains 60 features.

Until now, these queries were running using a shared system account. Cloudflare’s engineering team wanted to improve system security and reliability, and move from this shared system account to individual user accounts. User accounts already had access to another database called “r0”, so the team made the database permission change for access to r0 to be implicit instead of explicit.

As a side effect of this, the same query collecting the features to be passed to Bot Management started to fetch from the r0 database, and return many more features than expected:

The Bot Management module does not allow loading of more than 200 features. This limit was well above the production usage of 60, and was put in place for performance reasons: the Bot Management module pre-allocates memory for up to 200 features, and it will not operate with more than this number.

A system panic hit machines served with the incorrect feature file. Cloudflare was nice enough to share the exact code that caused this panic, which was this unwrap() function:

What likely happened:

• The append_with_names() function likely checked for a limit of 200 features
• If it saw more than 200 features, it likely returned an error
• … and when writing the code, it was not expected that append_with_names() would return an error…
• … and so .unwrap() panicked and crashed the system!

Edge nodes started to crash, one by one, seemingly randomly. The feature file was being generated every 5 minutes, and gradually rolled out to Edge nodes. So, initially, it was only a few nodes that crashed, and then over time, more became non-responsive. At one point, both good and bad configuration files were being distributed, making failed nodes that received the good configuration file start working – for a while!

Why so long to find the root cause?

It took Cloudflare engineers unusually long – 2.5 hours! – to figure all this out, and that an incorrect configuration file propagating to Edge servers was to blame for their proxy going down. Turns out, an unrelated failure made the Cloudflare team suspect that they were under a coordinated botnet attack, as when a few of the Edge nodes started to go offline, the company’s status page did, too:

The team tried to gather details about the attack, but there was no attack, meaning they wasted time looking in the wrong place. In reality, the status page going down was a coincidence and unrelated to the outage. But it’s easy to see why their first reaction was to figure out if there was a distributed denial of service (DDoS) attack.

As mentioned, it eventually took 2.5 hours to pinpoint the incorrect configuration files as the source of the outage, and another hour to stop the propagation of new files, and create a new and correct file, which was deployed 3.5 hours after the start of the incident. Cleanup took another 2.5 hours, and at 17:06 UTC, the outage was resolved, ~6 hours after it started.

Cloudflare shared a detailed review of the incident and learnings, which can be read here.

How did the postmortem come so fast?

One thing that keeps being surprising about Cloudflare is how they have a very detailed postmortem up in less than 24 hours after the incident is resolved. Cofounfer and CEO Matthew Prince explained how this was possible:

• Matthew was part of the outage call.
• After the outage was resolved, he wrote a first version of the incident review, at home. Matthew was in Lisbon, in Cloudflare’s European HQ, so this was early evening
• The team circulated a Google Doc with this initial writeup, and questions that needed to be reviewed
• In a few hours, all questions were answered
• Matthew: “None of us were happy [about the incident] — we were embarrassed by what had happened — but we declared it [the postmortem] true and accurate.
• Sent the draft over to the SF team, who did one more sweep, the posted it

Talk about moving with the speed of a startup, despite being a publicly traded company!

Learnings

There is much to learn from this incident, such as:

Be explicit about logging errors when you raise them! Cloudflare could probably have identified the root cause of this error much faster if the line of code that returned an error, also logged the error, and if Cloudflare had alerts set up when certain errors spiked on its nodes. It could have surely shaved an hour or two off the time it took to mitigate.

Of course, logging errors before throwing them is extra work, but when done with monitoring or log analysis, it can help find the source of errors much faster.

Global database changes are always risky. You never know what part of the system you might hit. The incident started with a seemingly innocuous database permissions change that impacted a wide range of queries. Unfortunately, there is no good way to test the impact of such changes (if you know one, please leave a comment below!)

Cloudflare was making the right kind of change by removing global systems accounts; it’s a good direction to go in for security and reliability. It was extremely hard to predict the change would end up taking down a part of their system – and the web.

Two things going wrong at the same time can really throw an engineering team. If Cloudflare’s status page did not go offline, the engineering team would have surely pinpointed the problem much faster than they did. But in the heat of the moment, it’s easy to assume that two small outages are connected, until there’s evidence that they’re not. Cloudflare is a service that’s continuously under attack, so the engineering team can’t be blamed for assuming it might be more of the same.

CDNs are the backbone of the internet, and this outage doesn’t change that. The outage hit lots of large businesses, resulting in lost revenue for many. But could affected companies have prepared better for Cloudflare going down?

The problem is that this is hard: using a CDN means taking on a hard dependency in order to reduce traffic on your own servers (the origin servers), while serving internet users faster and more cheaply:

When using a CDN, you propagate addresses that point to that CDN server’s IP or domain. When the CDN goes down, you could start to redirect traffic to your own origin servers (and deal with the traffic spike), or utilize a backup CDN, if you prepared for this eventuality.

Both these are expensive to pull off:

• Redirecting to the origin servers likely means needing to suddenly scale up backend infrastructure
• Having a backup CDN means there must be a contract and payment for a CDN partner which will most likely sit idle. As and when it is needed, you must switch over and warm up their cache: it’s a lot of effort and money to do this!

A case study in the trickiness of dealing with a CDN going offline is the story of Downdetector, including inside details on why Downdetector went down during Cloudflare’s latest outage, and what they learned from it.

This was one out of the five topics covered in this week’s The Pulse. The full edition additionally covers:

1. Downdetector & the real cost of no upstream dependencies. During the Cloudflare outage, Downdetector was also unavailable. I got details from the team about why they have a hard dependency on Cloudflare, and why that won’t change anytime soon.
2. Antigravity: Google’s new AI IDE – that its devs cannot use. Google wants to become a serious player in AI coding tools, but Antigravity contains remnants of Windsurf. Interestingly, devs at Google aren’t allowed to use Antigravity for work
3. Industry pulse. Gemini 3 launch, Anthropic valued at $350B, Jeff Bezos funds an AI company, and unusually slow headcount growth at startups persists.
4. Five AI fakers caught in 1 month by crypto startup. Candidates who fake their backgrounds and change their looks in remote interviews continue to plague companies hiring full-remote – especially crypto startups.

Read the full The Pulse

In the end, this process took several times longer; 4 years, in fact! Happily, it was worth it: readers’ feedback about The Software Engineer’s Guidebook has been overwhelmingly positive, and on launch, the book became a #1 bestseller among all titles in two Amazon markets (the Netherlands and Poland), as well as a top 100-selling book in most Amazon markets. In 24 months it sold around 40,000 copies, and was translated into German, Korean, Mongolian and Traditional Chines – with the Japanese and simplified Chinese versions releasing later this month.

A lot of people ask why I chose to self publish, and it would be nice to say this was always the goal, but it wasn’t! Originally, I wanted to work with a top tech publisher, who would get the book to market fast, and give it a higher profile. This didn’t happen, but during the process I learned a lot about how publishing works, how to pitch a book, and how to choose which publishing route might be the right one. 

This article shares my learnings from writing and publishing a book which has done pretty well with readers, and it includes the experience working with an established publishing house:

1. Tech book publishing landscape
2. Financials of publishing
3. Publishing process and the publisher’s role
4. My book pitch
5. Working with a publisher
6. Breaking up with a publisher

1. Tech book publishing landscape

Today, there are reputable book publishers whose titles are good and authoritative, and there are other publishers whom this doesn’t apply to. Each publisher also has a subject area: some are mainstream and publish titles about every software engineering area from languages to engineering management. Meanwhile, others stick to a topic of expertise they focus on.

Here’s my mental model of the book publishing industry in 2025:

Highly reputable mainstream publishers

In tech book publishing, three publishing houses really stand out, in my opinion, and form a ‘big three’ among all players in this sector: 

• O’Reilly: if I had to pick a #1 tech book publisher, it would be O’Reilly. They publish some of the most referenced books – like Designing Data Intensive Applications by Martin Kleppmann, Tidy First by Kent Beck, The Staff Engineer’s Path by Tanya Reilly, and more. The book covers are distinctive, using images of animals.
• Manning: a broad range of titles on both specific and general tech topics, which employ historical figures on the covers.
• The Pragmatic Bookshelf: also referred to as the “Prags.” Founded by Andy Hunt and Dave Thomas, the authors of what might be the best-selling tech book ever; The Pragmatic Programmer. Since its founding, The Prags has refused digital rights management (DRM) on their ebooks.

High reputable “mainstream” publishers that are tough to pitch to

The publishers in this section have strong reputations, like those above. However, they are harder to pitch to, usually because they publish fewer tech books. I couldn’t find an author pitch template, or clear pitching instructions, and contributes to a sense of “don’t find us, we’ll find you” among the following publishing houses: 

• Addison-Wesley: one of the best-known brands in tech. It has been an imprint (a trade name within a publication) of Pearson since 1988, and is the publisher of many “classic” book titles like Clean Code by Robert C. Martin, The Pragmatic Programmer by Andy Hunt and Dave Thomas, and some recent ones like Modern Software Engineering by Dave Farley. I couldn’t find any way to pitch to this publisher, and new books they publish seem to be by established authors.
• Pearson: This business owns the Addison-Wesley imprint. Recently, it started to publish tech books as “Pearson” instead, author Martin Fowler shared.
• Wiley: formerly a well-known tech book publisher behind the “X for Dummies” series. It publishes lots of computer science textbooks, but I can’t find recently-published, well-known tech books for software engineers.
• Springer: another massive publisher for whom tech books are a small part of the business. I couldn’t find how to pitch tech books to them.
• Morgan Kaufmann: a well-known tech books publisher founded in 1984, and acquired in 2001 by Elsevier. As I understand, these days it prints far fewer technology book, and focuses on academic topics. No clear way to pitch to them.

Highly reputable “niche” publishers

The following publishers are standout in quality, covering fewer topics than those above.

• No Starch Press: “The finest in geek entertainment” is the tagline, featuring fun visuals, and high-quality content on specific technologies like machine learning, Python, JavaScript, etc.
• IT Revolution: titles for technology leaders: DevOps, technology delivery, workplace culture, and similar. Publisher of The Phoenix Project, Team Topologies, and Accelerate.
• Artima: focuses on Scala.
• CRC Press: publishes on technology, engineering, math, and medicine.
• Stripe Press: “works about technological, economic, and scientific advancement.”
• MIT Press: “a distinctive collection of influential books curated for scholars and libraries worldwide.”

Other mainstream book publishers

Apress is a reputable publisher with a lower profile, which publishes on a wide range of topics, from specific technologies and frameworks, to more generic topics on computing. Because they publish many books on many topics, they are usually open to pitches.

Packt. A tech book publisher with a focus on quantity over quality, it feels to me. There is limited support and feedback for authors, and titles could often use more editing. But also, Packt is likely to say “yes” to a serious proposal.

2. Financials of publishing

Financial matters really come into play when your proposal is accepted by a publisher and you receive a contract offer.

Advance: $2,000 – $5,000. An advance payment to the writer is a tried and tested way to make them deliver a completed manuscript. It’s often paid in chunks: 50% when a milestone is hit, and 50% when a full draft appears.

The “big three” publishers typically offer $5,000, usually as a flat, non-negotiable rate; at least, it’s what I was offered. Smaller publishers offer closer to $2,000 for more niche books. The advance is non-refundable; even if your book sells zero copies, you keep it. The publisher is making an investment in you, and taking a risk.

As an aside: if you are thinking of writing a book: for guest authors in The Pragmatic Engineer Newsletter guest authors I offer a $4,000 per article payment – and you can later publish your guest article in a book. Several authors working on their book have written a guest articles such as Lou Franco on Paying down tech debt or Apurva Chitnis on Thriving as a founding engineer. Writing a guest post can help refine ideas, broaden your reach, and prove helpful when publishing the article.

Paperback royalty: 7-15% 

Royalties are earned on book sales, and taken from the net price of the book. Net price is what a publisher gets after the retailer (e.g. Amazon, or a bookshop) takes their cut. Let’s see how it works for a $40 book:

It matters financially where your title is purchased; be it an online shop, physical book store, or purchased directly from the publisher. Many tech books are sold on Amazon and online stores. Amazon’s 40% cut seems high, but it’s actually the lowest among book retailers. Up to 60% is a common cut for a physical bookshop.

Most publishers offer 10-12.5% royalties, is my understanding, and Packt around 15-20%. Keep in mind that brand reputation plays a role; for example, Packt’s reputation is less elevated than Manning, which can make a difference to sales.

Ebook royalties: 10-25%

For ebooks, several publishers pay 25% royalties, but not all. But even with a higher royalty rate, an author might end up making less per sale. For example, on the Kindle platform, the cut for Amazon is high at 65%. Let’s look at a $30 ebook with a 20% royalty rate:

Ebooks are almost always priced lower than physical books, and when sold on Kindle, generate much less revenue for the author, while earning more per copy than the paperback version. I was offered 10% royalties on ebook sales, which is at the low end.

“Earning out” 

When an author needs to pay back an advance before being paid anything, this is called “earning out”. If you get a $5,000 advance for a title costing $40 per hard copy and $25 for the ebook version, and most sales happen on Amazon, it means:

• ~2,080 paperback sales on Amazon
• Or ~2,850 Kindle book sales
• Or ~1,250 paperback sales on the publisher website

The author needs to sell at least 1,000 copies across various platforms to “earn out.” The good news is that a publisher sends quarterly or annual royalty payments if a book keeps generating revenue, which would effectively be passive income.

The Prags’ unique approach

One publisher that calculates rates differently is The Pragmatic Bookshelf. Instead of offering a low-digit number on revenue, they offer a 50% split on profit.

50% on profit sounds much higher than 10% on revenue, right? However, the devil is in the details, because paying on profit means that the upfront publisher costs – editors, cover design, printing, distribution, marketing – all are deducted before any profit split.

Authors who have used this approach tell me the numbers end up pretty similar to the revenue model.

Real-world case studies with actual earnings

Designing Data Intensive Applications author, Martin Kleppmann, shared the cumulative royalties he made in 6 years. The breakdown is interesting; ebook and Safari Online sales generated more revenue for the writer than the print version.

Cloud Native Infrastructure earnings: author Justin Garrison published with O’Reilly, and was offered 10% for print and 25% for ebooks (split into half, thanks to working with a coauthor). His book sold 1,337 copies in 4 months; and made about $22,000 for the two authors (and around $11,000 for Justin.) Justin concluded:

“Going into this project I had a rough estimate in my head to make about $2000–3000 so this is much better than I expected. Set your expectations accordingly.”

Don’t forget that publishers are also in this to make a positive return. This means that it is unlikely for a highly reputable publisher to invest into a book that they do not believe would sell at least a few thousand copies. I don’t have the data here: but if I was a publisher, I would reject any book that didn’t look like it could hit 1,000 copies sold in the first year of publishing.

3. The publishing process, and publisher roles

Why does a publisher take so much of the revenue? Part of this is because they do a lot of the work around publishing, and need to hire (and pay!) people for those roles. Here is my understanding of how the publishing process works, based on four months of pitching to publishers; two months of working with one of them; and researching how the rest of the process works:

Here are people I worked with, and my experience with them:

The acquisitions editor. If you write a technical blog, you might get a reachout from someone called an acquisitions editor, who will ask if you would consider publishing a book. Also, when you submit a pitch to a publisher, chances are that you will first communicate with an acquisitions editor.

A publisher’s goal is to publish books that will be profitable for them. They find authors who could write these books two ways:

• Inbound pitches coming from authors – reviewed by editors or acquisitions editors
• External reachouts done by acquisitions editors

These people need to have a good understanding of what kinds of books sell well at the publisher (and why); what their current catalogue is; what the gaps are; and what competitor publishers are commissioning.

When I pitched my book to 3 respected publishers, in two cases I talked with (and worked with) the acquisitions editor to improve my pitch. The acquisitions editors were my “champions” at the publisher. Their goal was to get a pitch that the company would say yes to.

The development editor works on the structure of the book. They ask the author to come up with a detailed table of contents – in my case, they asked me to estimate even the length of the chapters. They also help develop – and maintain – the narrative of the book.

Had I not worked with a publisher, I would have had no appreciation of this “high-level editing” – which, turns out, is key for writing a well-structured tech book!

The project manager checks in with timelines, organizes reviews—like editorial reviews—and helps keep you accountable. One of the best things about working with a publisher is that you are on a tight deadline—without which it would take you several times longer to publish the book!

The publisher owns a lot of rights for your book! One thing that I realized only after signing with a publisher is that while publishers help a lot with writing the book – and taking a higher cut is sensible because of this – they also hold on to a lot of rights that impact your book! These are all things that you give up on, versus when self-publishing. These are:

• Global publishing rights. Although you are the author of the book – and usually hold the copyright to it – the publisher own wordlwide publishing rights. This means that they are the only ones who can publish the book, or longer excerpts of it. In practice, this means you need to get permission if you’d like to publish some parts of your book on e.g. your blog, or social media. They’ll usually grant this as it’s good marketing – but it’s still that you need to ask, as the author.
• Foreign rights. The publisher will own the publishing right, and will usually be the one who owns selling foreign rights. In theory, this could sound like you are losing out on things. In pratice, publishers are much better positioned to sell and administer these rights. Most publishers offer a 50% cut on these rights – it’s what my publisher offered. Also, the majority of tech books are not translated to other languages – a book that “only” sells 2,000 copies in English is unlikely to sell a significant number in a non-English market!
• The cover. The publisher decides what cover they will design, though they tend to check the author for feedback.
• The title. One of the surprises for me was how the publisher ultimately decides on the title and subtitle.

In short: this book is owned by the publisher. You are the author, but they are the only ones who can distribute it. In practice, many authors would prefer to have it this way – because all the work related to distributing the book is taken on by the publisher. However, it’s good to know that you need to give up all the above when working with a publisher.

4. My book pitch

My secret hope, back in 2019, was to get a contract with one of the “Big 3” tech book publishers: O’Reilly, Manning or The Prags. I pitched my book to all three: got a “no” from two, but a “yes” (and a contract) from one. Here’s how I went about my pitch.

Write a “one-pager” about your book

What will this book be about? Who is it for? What will readers take away when reading it? Answer these in a short pitch, before even seeking out publishers. Here’s what I put together as my “one-pager:”

Do some market research

What are similar books in the market that would be competing with this book, directly or indirectly? How is this book different from them?

What is the demographic of people who would be interested in buying this book? Can you estimate how large this crowd is? Realistically, what percentage of this group could be interested in buying the book – assuming they know about it? Don’t forget that publishers will invest into books that can generate decent sales: it’s good to do a little research to help confirm your title could be one of these!

Shortlist publishers you would be interested working with

While there are quite a few publishers out there: what are your top preferences? And what are ones you’re willing to consider, even if your “top” choices turn you away?

Self-publishing is always an option (I’ll cover more on how I went about this in later parts). However, going with a good publisher can significantly speed up your book production, while also improving the quality.

Write a draft table of contents and a draft chapter

Some publishers will want to look at what a draft chapter will look like – but not all of them. Still, I found it helpful to do writing before submitting to a publisher. If for no other reason, this was to confirm that I’d enjoy longform writing!

I spent about a week putting together a table of content, and around four months writing drafts of chapters. These chapters turned out to be helpful later on.

Submit a tailored pitch your the publisher(s)

Once you identified your top publisher choices, submit a pitch. Most book publishers have a pitch document they want you to follow. Here are common ones:

O’Reilly’s pitch template:

• Description
• About the topic
• Audience
• Keywords
• Competing titles
• Related O’Reilly titles
• Book outline
• Writing schedule

Manning’s pitch template:

• About the author
• About the book topic
• The book plan
• Q&A
• Reader overview
• Book competition
• Book length and illustrations
• Writing schedule
• Table of contents

The Pragmatic Bookshelf template:

• Overview
• Outline
• Bio
• Competing books
• PragProg books
• Market size
• Promotional ideas
• Writing samples

Most of these templates ask for similar content, so if you completed one pitch: the others are much easier. Here are some tips I’d have for building a pitch.

Put yourself in the shoes of the publisher. This book is a huge deal to you: but it’s just one of the dozens that the publisher will publish just this year. You want to write an amazing book: but the publisher wants to publish one that will sell.

And these are major differences! The publisher will care very much about competition for the book, and how their existing titles relate to them. Like a VC firm, a publisher will not want to fund two investments competing on the exact same market: so if the publisher recently published a book that is a deepdive on Go; they will almost certainly pass on the next one, no matter how good your pitch is.

Pitching to several publishers parallel is totally fine and you should do it! This is one thing I wish I’d done differently. In my mind, I was 100% certain that my first publisher-of-choice would jump on the opportunity to publish this book. I thus felt that it would be “unfair” if I pitched to other publishers, without hearing back.

In hindsight, as a first-time author, this strategy was a waste of time on my end. Most publishers are unlikely to take a risk on a first-time author with no books published in the past – like I was in 2019. And so the likely outcome is rejection in most cases.

In my case, I spent about two and a half months waiting on the response from this first publisher. My acquisitions editor was championing the book – making the case for the publisher to offer a contract – but in the end, the publisher chose another book with a similar topic that was in their pipeline. This made perfect business sense for them – but for me, I was spent waiting for months, instead of pitching the book to other publishers!

My book pitch ended up being a helpful resource on my self-publishing journey. Even though I did not release with a publisher: pitching to publishers helped the book become an eventual success. It was for these reasons:

• Defining the structure. I had my table of contents well thought-out by the time I submitted the pitch. This structure changed later, but it was a solid start.
• Positioning the book. I had a good idea of the “competitive” landscape, and what books my title would “go up against.” It also helped me focus on how my book is different to what is already out there.
• Forcing me to think about marketing. The Pragmatic Bookshelf asked for a section on promotional ideas. This forced me to think about where (and how) I would promote the book – even before getting into the thick of writing. When going with a publisher, it’s safe to assume that the publisher’s brand will do some marketing. However, authors will still do the lion’s share of marketing – and it’s good to think about this ahead of time.

5. Working with a publisher

I got lucky with one of the three publishers, in the end. This publisher was looking for a book just like mine, right at that time! What happened was one of their best sellers had to be pulled from publication, for reasons outside the control of the publisher. Apparently, when my pitch arrived, they had just started a search for a book that could plug the hole – and they saw my book being a perfect fit for a “software career advice” book.

At the time, this felt like great luck. In hindsight, my relationship with the publisher might have soured exactly because they were looking for me to write a specific kind of book that would be similar enough to this old book – but I had no intention of doing so. More on how things went sour in the section after this one.

From signing the contract, I worked with a publisher for about a month – so I’m not exactly the most experienced in this front. However, a couple of things stood out as strong positives – and things that I “lost” when deciding to self publish, in the end.

Strong pressure to write – thanks to the contract. My contract had pretty strict deadlines included. We signed it on 11 January 2020, and these deadlines were part of the contract:

“The Author shall prepare and deliver to the Publisher a machine-readable electronic copy of the manuscript for the Work, including all its illustrations, code listings, and exercises, as mutually agreed upon by the Publisher and the Author as follows:

- Not later than March 15, 2020, a partial manuscript for the Work totaling not less than one third of the planned finished Work.

- Not later than June 1, 2020, a partial manuscript for the Work totaling not less than two thirds of the planned finished Work.

- Not later than August 15, 2020, a draft of the complete manuscript for the Work suitable for review.

- Not later than September 1, 2020, the final, revised and complete manuscript for the Work acceptable to the Publisher for publication.”

Talk about pressure! Also, my first payout was tied to reaching the first milestone – which was delivering at least a third of the finished work. My publisher also set up regular check-ins to help me stay accountable. And this kind of pressure was good – because without it, I would have pushed back writing, or got stuck on relatively trivial parts! 

6. Breaking up with the publisher

While I greatly appreciated that a publisher took a chance on me, lots of things felt wrong from the start. A month into working together, I felt that things were getting worse, and not better.

The small things that I dismissed, in the beginning:

• A (very) opinionated structure. This publisher had strongly opinionated templates I was told to use for all chapters. They included each chapter to start by stating what the reader will learn; and then summarize this at the end of the chapter. It wasn’t how I imagined my book to be – but it didn’t seem I had a choice. I figured, I’ll give it a go. The publisher knows better after all, as they’ve done this hundreds of times. Right?
• Needing to ask for permission to share drafts on social media. I originally planned to share screenshots of some of the parts I am writing to get feedback as I go – and to also increase visibility of the book. I thought that this is a no-brainer. Not only does this kind of “early sharing” makes the book better: but it will also make more people excited about the book, leading to more eventual customers. To my surprise, my contact at the publisher said I will need to ask for permission whether I can do this. Permission? For something that will market the book? Yes: because the publisher owns all publishing rights, including for the draft!
• I won’t decide on what the title will be. I had strong opinions about what I’d like the book’s title to be. My publishing contact also had ideas on what they thought would be good to add to it – like introducing the “mentoring” term either to the title or the subtitle: which was an idea I disliked. As I talked with them, it became clear that the publisher will set the final title: not me. Hmm – odd, no? It’s another reminder that, although it’s my book: it’s really the publisher’s book, and they have the final say on all important decisions.
• Nudges to “dumb down” the book. My editor was giving more suggestions on how to edit the content to make it more “beginner-friendly” and suggested I introduce e.g. “Alice and Bob” examples to make it easier to digest the contents. One of the recently best-selling books of the publisher heavily used Alice and Bob, and it seems the publisher thought it helped their sales.

The first major editorial review was where I decided we should part ways with the publisher. About a month-and-a-half in, the publisher pulled together several experienced editors, and offered suggestions on how I could improve the book. The suggestions were these:

• Focus on reader engagement. Tell stories and develop them with emotion, mystery, aha moments, and unexpected conclusions. Tell the stories from the "we" or "they" perspective -- make stories team-oriented.
• Exercises. Develop exercises for use within the chapters (not just end) or a story about what happened when one person did the exercise.
• Mini-projects. Guide readers to discover and come to conclusions on their own (see Donald Saari story in What the Best College Teachers Do). Mini project topics: testing, architectures.
• Word of the day feature. Example: Dependency injection (what is it)? Scatter these across the book.
• Quotes. Include quotes from luminaries such as [Well-known-person 1] and [Well-known-person 2] that relate to advice given. Ask other [Publisher] authors to relate experience about how they followed similar advice and were successful.
• Tech map. Create a diagram of the current technology landscape. Example big-picture topics: architecture demystified, distributed systems demystified.

While I appreciated the suggestions: I hated all of them. I saw what implementing them would do: they would turn this book – which I already had reservations with the “forced” style on me – to something I would not want to read. Much less write!

I envisioned writing a more matter-of-the-fact book that doesn’t have exercises, “mini projects” or “word of the day” gimmicks.

I sat down to reflect why I chose to work with a publisher, to start with. As an author, I’m giving up a lot of things: editorial control, the bulk of revenue, all publishing rights… and for what? For the publisher to make the process easier, and for the end result book to be better than if I was working alone.

But I felt that this book would be far worse if I continued with my publisher: and the only way to get it back to what I envisioned was if I spent a lot of time and energy pushing back on them.

It would cost me less energy to self-publish. So I decided to terminate my agreement because it didn’t feel my publisher was helping write the book that I wanted to write.

My publisher was understanding and professional in terminating the contract. I explained to them that all the feedback suggested they wanted to see a very different book to what I wanted to write. And that, frankly, I am not the author to write that kind of book.

Truth be told, I was embarrassed that I had wasted their resources – working with their development editor and the editing team – for these two months. At the same time, I was vocal in voicing to my editor that I was hesitant about this mandated style. I also made the decision that there is no point in continuing at the first formal feedback session. I’m not sure I could have come to this conclusion any further, as I was still learning how this book publisher worked, up until that point.

To show how professional this team was, this is the termination letter they sent as a signed PDF:

“This letter is in reference to our Publishing Agreement with you for [what would become The Software Engineer’s Guidebook] dated January 11, 2020. By mutual agreement, we are terminating the publishing contract.

Since no advance was paid to you under the terms of this contract, all rights in the content you originally submitted will hereby return to you and we will consider this matter concluded.

The decision to cancel a project is never an easy one to make. We thank you for all the efforts on this project that you made and wish you the best in your future endeavors.”

At this point, I learned enough about publishers and myself to decide: I’m doing it by myself. Having my book accepted by a major publisher gave external validation that there’s a strong business case for The Software Engineer’s Guidebook. And working with an opinionated publisher – and continuously pushing back on styling suggestions made me realize that I already have my own opinonated style that I like using.

I did lose a very important thing by deciding to self-publish: the accountability of meeting a publishing deadline. Working with the publisher, this book would have been out fall 2020 or spring 2021. Self-publishing, I launched it November 2023.

One of the reasons for publishing my book two years later than it would have taken with a publisher was because I now knew I could no longer rely on a well-known publisher to lend my book their brand. For my book to have an even slim chance of being successful: I would have to compensate for the lack of being associated with a publisher, and fill the gap in marketing and awareness, leading up to the book launch.

Not having a publisher was a reason I started writing The Pragmatic Engineer Newsletter in August 2021 (a year-and-a-half after breaking up with this publisher) – and the sudden success of this newsletter gave me less time to wrap up the book. At the same time, by the time the book was ready, there were plenty of people who looked forward to reading it: and many of them were already readers of the newsletter!

I’ll cover more about how I went about the actual self-publishing process in a follow-up article, how the book ended up selling, and other learnings. Subscribe to The Pragmatic Engineer to get notified when it is out.

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Online retail giant Amazon unexpectedly announced 14,000 job cuts earlier last week. The massive round of layoffs at the company follows other mass redundancies in recent years:

• January 2023: 18,000 people laid off.
• March 2023: another 9,000 people let go
• November 2023: hundreds of people let go inside the Alexa team, as Amazon was looking to shift Alexa more toward GenAI
• April 2024: hundreds of people let go from AWS

Software engineers, unfortunately, seem hit hard by the latest layoffs: of 2,300 employees laid off in Washington State, 25% are software engineers, GeekWire reports. We can only speculate about the ratio across the rest of the company, but if cuts at HQ are heavy on engineering, then things don’t look promising for other locations, sadly.

The memo from Beth Galetti, Senior Vice President of People Experience and Technology, to workers didn’t explain much:

“Some may ask why we’re reducing roles when the company is performing well. Across our businesses, we’re delivering great customer experiences every day, innovating at a rapid rate, and producing strong business results. What we need to remember is that the world is changing quickly. This generation of AI is the most transformative technology we’ve seen since the Internet, and it’s enabling companies to innovate much faster than ever before (in existing market segments and altogether new ones). We’re convinced that we need to be organized more leanly, with fewer layers and more ownership, to move as quickly as possible for our customers and business.”

The statement is utterly confusing, as encapsulated by its message that “business is great, but we need to do layoffs”. Job cuts usually mean a business is in trouble, which obviously isn’t the case for Amazon. So, why are these layoffs really happening?

Layoffs to boost efficiency?

The company’s memo states:

“We’re convinced that we need to be organized more leanly, with fewer layers and more ownership, to move as quickly as possible for our customers and business.”

If this line of reasoning sounds familiar, it’s because most of the layoffs in 2023 were justified the same way. The tech industry overhired during the pandemic in 2020-2021, making orgs more bloated and decision-making slower. In February 2023, I reported on the trend of fewer middle managers, with Meta the first Big Tech giant to reduce its management layers. In 2023, most of Big Tech followed this approach with layoffs or reorgs. Managers acquired more reports, and tech companies cut down the number of layers between the CEO and individual contributors.

Given Amazon did other massive layoffs in 2023, it’s unlikely they missed the industrywide trend for fewer managers. While the current layoffs seem to be targeting managers quite a bit – from the Washington State layoffs, 20% of those let go are managers – there are still more ICs laid off than managers, overall. So, this official explanation doesn’t pass my personal “smell test”.

Layoffs to buy more GPUs?

The day after its jobs announcement, Amazon had more big news, this time about AI: it unveiled Project Rainer, the largest AI computing platform AWS has ever built. It already has 500,000 Trainium2 chips (built by Amazon), This capacity is already 70% larger than any AI computing platform in AWS’s history, and Anthropic is using all of it (!!) to train its next models. Below is an image of one of the several Project Rainer data centers packed with Amazon GPUs:

Building data centers is incredibly capital-intensive: Amazon has spent $11B on Project Rainer alone. Even though very profitable, Amazon might want to invest more cash than it currently has into building data centers. So, one reason for job cuts could be to reallocate financial resources from paying salaries and compensation towards building more data centers.

Before doing the math, a couple of concepts are important to understand:

• Free cash flow (FCF): profit after payment for things like capital expenditure (CapEx), such as financing data centers. If a company wants to operate with as little debt as possible, FCF is usually very important. If Amazon wants to avoid loans and not touch its reserves, then data center investment would come from FCF, reducing FCF further.
• Cash reserves: a company’s liquid reserve investments, usually an accumulation of investments in financial instruments like bonds and securities, or cash deposits.

Let’s run Amazon’s numbers:

• Cash reserves: $93B. This is how much Amazon has in reserve.
• FCF: $32B. This is the rough free cash flow Amazon has currently, as per its latest quarterly report. This is after deducting current data center investments.
• Savings from layoffs: $2-4B. This is my estimate of the rough total compensation of 14,000 employees.

So, the savings from these layoffs wouldn’t even pay for half of Project Rainer ($11B in total), and Amazon could easily build 3x Project Rainers in the next year, without needing to dip into its savings! Of course, Amazon has its famous frugality principle, but this massive layoff of 14,000 people won’t make a big difference to how much it can invest in data centers; It can already spend much more, if it wants!

Leanness and AI fail job cuts “smell test”

It’s not only me who doesn’t buy the explanation that these layoffs are to streamline the company, or to redirect resources to AI. Arne Knudson worked at Amazon for nearly two decades, most recently as a software development manager (SDM), before leaving the company earlier this year. He shared his analysis, with some insider detail:

“In my 18 years at Amazon, I went through a few layoffs and hiring freezes.

This is the first time I’ve seen multiple years of significant layoffs essentially back-to-back. Even in the depths after the .com bubble, it wasn’t this bad. They’ve been laying people off now for almost 3 straight years.

The explanation that this is downsizing after hiring too many at the height of the pandemic doesn’t pass the smell test, at least to me. That was 3 years ago; they’re not that dumb to keep those people around for 3 extra years. Those folks were laid off back in ‘22.

I’m also not convinced that this is optimization due to AI. My degree’s in AI, and I worked on AI stuff at Amazon; I don’t think there’s enough automation yet, and it’s not accurate enough yet, to replace 30,000 people. The cost of inaccuracies seems too high. But I could be wrong; maybe they’ve gotten their false negative & false positive rates low enough to avoid too many region-wide AWS outages. (Or not.)

One of the articles I read said this was going to be in HR, and I can tell you as a former manager, my experience working with HR had been steadily worsening over the past 5-7 years. They outsourced so much of the work, overworked the people they had, and had such high turnover that I never knew who I was supposed to work with. When I needed to put someone on a performance plan or help a new hire receive some kind of accommodation, it seemed like it was a different person each time. If they really are laying off tens of thousands more HR folks, this is only going to get worse.

And, I suspect, it means they don’t plan on hiring MORE people in any of the business units for a year or more. So, by the smell-o-meter, this seems more significant than streamlining the workforce, improved AI, and “nah, we don’t need as many HR folks.”

US economy to blame for Amazon layoffs?

It’s safe to assume AWS as a business unit is doing just fine, as suggested by Project Rainer’s existence and the agenda for building data centers. But how is the e-commerce side of the business performing, and what’s its outlook?

If one business should have its finger on the pulse of the US economy, it’s Amazon with its size and self-professed, relentless customer focus, providing a window into people’s spending habits across the country. Flashing lights on the dashboard of the national economy may signal tough times ahead in e-commerce, which could be a reason to start cutting costs early.

There are concerning signs from other sectors about the US economy. Below is the CEO of the restaurant chain, Chipotle.

“Earlier this year, as consumer sentiment declined sharply, we saw a broad-based pullback in frequency across all income cohorts.

Since then, the gap has widened, with low to middle-income guests further reducing frequency. We believe that this guest with household income below $100,000, represents about 40% of our total sales. And, based on our data, they are dining out less often due to concerns about the economy, and inflation.

A particularly challenged cohort is the 25- to 35-year-old age group. We believe that this trend is not unique to Chipotle and is occurring across all restaurants as well as many discretionary categories. This group is facing several headwinds, including unemployment, increased student loan repayment and slower real wage growth. We tend to skew younger and slightly over-indexed to this group relative to the broader restaurant industry”.

Chipotle is saying that everyone is eating out less, particularly 25-35 year olds, because of inflation. If people spend less on Chipotle because of rising prices, then they may also spend less in other areas of their lives for the same reason, including on Amazon.

In the e-commerce supply chain, there’s evidence of this trend, which would mean delivery services like UPS have fewer parcels to deliver. Speaking of UPS, two days ago, it announced massive layoffs:

“United Parcel Service (UPS) has slashed 48,000 jobs this year — one of the largest single-year reductions by a US company since the pandemic — as the package giant scrambled to contain costs and revive its lagging stock price.

The Atlanta-based delivery behemoth disclosed the reductions Tuesday while reporting third-quarter earnings that beat Wall Street expectations.

UPS said 34,000 of the cuts hit drivers and warehouse operations, while 14,000 targeted management (...)

UPS shares jumped nearly 9% in Tuesday afternoon trading, even as the company reported weaker revenue and profits”.

UPS’s revenue is down on last year, which suggests that there are, indeed, fewer deliveries (or lower value ones). As with the latest job cuts at Amazon, these drastic layoffs could be explained by a lot of things, most easily by UPS expecting reduced trade in the future.

If US consumer spending is trending down, then the e-commerce sector will be among the first to feel this. It could explain why Amazon is making these layoffs now. It can also explain why Google, Meta, and Microsoft might not be seeing their businesses impacted: they’re not involved in retail like Amazon is, and the AI sector is very much booming.

Among all of Big Tech, Amazon is best positioned to detect changes in US consumer spending. Google’s and Meta’s revenue is more dependent on advertising, and Microsoft’s more on enterprise spend. Like Amazon, Apple is well placed to feel market changes with its range of smartphones and watches, and other consumer tech.

I believe Amazon is highly commercially rational, so it’s worth understanding the actual reason for its second major mass layoffs in just two years, following deep cuts in 2023. I’d put my money on this reason being the economy, and how Amazon probably expects customers to cut back their spending everywhere, including on Amazon.

This was one out of the four topics covered in last week’s The Pulse. Read the full article.

This week’s The Pulse additionally covers:

1. Cursor and GitHub double down on agents. Each company is focusing on agents: Cursor with its multi-agent mode, and GitHub with its “Agent HQ.” Cursor is increasingly a direct rival to GitHub.
2. Industry pulse. Meta rolls out AI-assisted interviews, Cursor and Windsurf believed to be using Chinese open source AI models, South Korean government pays price of ignoring backup “101,” startups growing much faster in the US than in Europe, companies using AI tools buy more JIRA seats, a neat uptime service called Updog, and more.
3. OpenAI inflating the bubble? OpenAI signs another massive deal with AWS based on predicted growth, and seeks taxpayer protection to borrow more.
4. Large tech companies struggle to build their AI integrations. Apple admits failure to modernize Siri by paying Google $1B per year for its LLM. Perplexity to pay Snap $400M to be its AI search interface.
5. How much do Directors of Engineering earn at startups? Data from Carta says it’s more than any other Director: $215-230K at companies valued at $25-250M.

Read the full issue here

Puneet Patwari recently accepted an offer to join Atlassian as a Principal Software Engineer. In three months, he did more than 60 interviews at 11 companies, he told me – while dropping out of 3 more interview processes after accepting the Atlassian offer, including that of Meta. Following that endeavour, he has compared the interview processes of the largest companies:

A few more observations that Puneet shared with me:

Amazon: the Amazon Hiring Manager round was one of the most unique I ever experienced. We got so engrossed in the discussion that it took 160 minutes instead of the scheduled 60 minutes! We had to take a break in between the interview process.

Atlassian: The leadership craft (LC) & values were two interview rounds which were very crucial in determining that I’ll be levelled at the Principal level. Of course, the Systems Design interview was also key here. Atlassian puts a lot of emphasis on LC for Principal engineers.

Salesforce: the system design round was based on the actual job requirement. It was a migration problem where the interviewer wanted to check if I can own a project end-to-end with customers at the centre of it.

Confluent: when I say it was the most mentally demanding interview, what I mean is how every skill was tested with two interviews! So 2x data structures and algorithms (DSA), 2x System Design 2x behavioural interview rounds.

I cannot stress enough how important behavioural interviews are at the Staff+ levels. Doing well on these interviews were decisive in getting Staff and Principal-level offers. Of course, you needed to do well on coding and systems design: but my sense was that the behavioural parts were make or break for levelling and getting an offer.

A few things stand out to me from Puneet’s account of his interviews at leading tech companies:

• Algorithmical coding interviews are everywhere! For senior+ positions, you need to get really good at these, including challenging topics like dynamic programming. We cover how to perform well in these in the article, How experienced engineers get unstuck in coding interviews
• Interviews are tough, and time consuming. Even after Puneet had offers, no company shortened their process. Puneet had to decline 3 more interviews – including one at Meta – because by the time the interviews would have come around, he already had an offer he had accepted at Atlassian.
• In a tough job market, “top” candidates are still in demand. We’ve covered how challenging the current tech labor market is for jobseekers, but Puneet interviewed at 11 companies and got 6 offers. His applications had to have a lot going for them in order to pass the resume screenings: 10+ years of experience, and working as a Senior Software Engineer at Microsoft. He also showed up really well prepared.
• Bad luck can strike at any time. Puneet’s interview experience at Uber seems to have been a bit unlucky: the interviewer presented as rigid and not open to dialogue. Perhaps they were having a tough day, or wanted to get the interview over with. Or it could be what Steve Yegge describes as the interviewer anti-loop

Congrats to Puneet for accepting the Atlassian position, and thanks for sharing all these learnings!

This was one out of five topics covered in The Pulse #149. The full edition additionally covers:

• New trend: programming by kicking off parallel AI agents. More devs are experimenting with kicking off coding agents in parallel
• ACP protocol. A new protocol built by the Zed team, which tries to make it easier to build AI tooling for IDEs than the MCP protocol allows
• AI security tooling works surprisingly well? AI-powered security tools seem good at identifying security flaws in mature open source projects
• Is AI the only engine of US economic growth? Forty percent of US GDP this year is based on AI-related spend, while 60% of venture capital goes into AI. Hopefully, it won’t end up as a bubble which bursts like in 2001

Read the full issue here, and check out today’s The Pulse here.