We’re used to highly reliable systems which target four-nines of availability (99.99%, meaning about 52 minutes of downtime per year), and for it to be embarrassing to barely hit three nines (around 9 hours of downtime per year.) And yet, in the past month, GitHub’s reliability is down to one nine!

Here’s data from the third-party, “missing GitHub status page”, which was built after GitHub stopped updating its own status page due to terrible availability. Recently, things have looked poor:

This means that for every 30 days, GitHub had issues on 3 days, or issues/degradations for 2.5 hours daily (around 10% of the time.)

GitHub seems unable to keep up with the massive increase in infra load from agents. One software engineer built a clever website called “Claude’s Code” that tracks Claude Code bot contributions across GitHub. Growth in the past three months has been enormous:

Stream of GitHub outages from infra overload

GitHub’s CTO, Vladimir Fedorov, addressed availability issues in a blog post and covered three major incidents:

• 2 February: security policies unintentionally blocked access to virtual machine metadata
• 9 February: a database cluster got overloaded
• 5 March: writes failed on a Redis cluster

Software engineer Lori Hochstein did a helpful analysis of these outages and the CTO’s response, and has interesting observations:

• Saturation: the database cluster incident (9 Feb) was a case of the database getting saturated, due to higher-than-expected usage. Databases are harder to scale up than stateless services. GitHub also underestimated how much additional traffic there would be.
• Failover + telemetry gap: the 2 Feb incident was a combination of an infra issue in one region failing over to a healthy region, and making things worse with a telemetry gap (incorrect security policies were applied in the new regions which blocked access to VM metadata)
• Failover + configuration issue: the 5 March incident was uncannily similar: after a failover, a configuration issue blocked writes on a Redis cluster

It is certainly nice to get details from GitHub on these outages. It feels to me that infra strains are causing more infra issues → they trigger constraints faster → failovers are not as smooth as they should be. Could it be because GitHub keeps changing their existing systems?

Startup shows GitHub how it’s done

While GitHub struggles to keep up with the increase in load from AI agents generating more code and pull requests, a new startup called Pierre Computer claims to have built an “AI-native” solution for AI agents pushing code, which scales far beyond what GitHub can do. Pierre was founded by Jacob Thornton: formerly an engineer at Coinbase, Medium, and Twitter, and also the creator of the once-very popular Bootstrap CSS library.

Here’s what Pierre supports, which GitHub does not:

“In October [2025], Github shared they were averaging ~230 new repos per minute.

Last week we [at Pierre Computer] hit a sustained peak of > 15,000 repos per minute for 3 hours.

And in the last 30 days customers have created > 9M repos”

These are incredible numbers – if also self-reported – and something that GitHub clearly cannot get close to, at least not today! There are few details about customers, while the product – called Code.storage – seems to be in closed beta.

Still, this is the type of “git for AI agents” that GitHub has failed to build, and the type of infrastructure it needs badly.

Has GitHub lost focus and purpose?

GitHub’s reliability issues are acute enough that, if it keeps up, teams will start giving alternatives like small startups such as Pierre a try, or perhaps even consider self-hosting Git. But how did the largest Git host in the world neglect its customers, and fail to prepare its infra for an increase in code commits and pull requests?

Mitchell Hashimoto, founder of Ghostty, and a heavy user of GitHub himself, had advice on what he would do if he was in charge of GitHub, after growing frustrated with the state of its core offering. He writes (emphasis mine)

“Here’s what I’d do if I was in charge of GitHub, in order:

1. Establish a North Star plan around being critical infrastructure for agentic code lifecycles and determine a set of ways to measure that.

2. Fire everyone who works on or advocates for Copilot and shut it down. It’s not about the people, I’m sure there’s many talented people; you’re just working at the wrong company.

3. Buy Pierre and launch agentic repo hosting as the first agentic product. Repos would be separate from the legacy web product to start, since they’re likely burdened with legacy cross product interactions.

4. Re-evaluate all product lines and initiatives against the new North Star. I suspect 50% get cut (to make room for different ones).

The big idea is all agentic interactions should critically rely on GitHub APIs. Code review should be agentic but the labs should be building that into GH (not bolted in through GHA like today, real first class platform primitives). GH should absolutely launch an agent chat primitive, agent mailboxes are obviously good. GH should be a platform and not an agent itself.

This is going to be very obviously lacking since I only have external ideas to work off of and have no idea how GitHub internals are working, what their KPIs are or what North Star they define, etc.

But, with imperfect information, this is what I’d do.”

My sense is that GitHub has three concurrent problems:

• GitHub and Copilot are entangled with Microsoft’s internal politics. GitHub’s Copilot in 2021 was the first massively successful “AI product.” Microsoft took the “Copilot” brand and used it across all of their product lines, creating low-quality AI integrations. Simultaneously, internal Microsoft orgs like Azure and Microsoft AI were trying to get their hands on GitHub, which is one of the most positive developer brands at Microsoft.
• GitHub has no leader, seemingly by design. GitHub’s last CEO was Thomas Dohmke, who stepped down voluntarily, and Microsoft never backfilled the CEO role; instead carrying out a reorg to make GitHub part of Microsoft’s AI group and stripping its independence. It seems the “Microsoft AI” side won that battle.
• GitHub has no focus, and is stuck chasing Copilot as a revenue source. GitHub has no CEO and is caught up in internal politics, so, what can GitHub teams do? The safest bet is to increase revenue and the best way to do that is by investing more into GitHub Copilot, and ignoring long-term issues like reliability.

I agree with Mitchell: GitHub has no “North Star” and we see a large org being dysfunctional. That lack of vision – and CEO – is hitting hard:

• GitHub Copilot went from the most-used AI agent in 2021, to be overtaken by Claude Code, and is soon to be overtaken by Cursor.
• As a platform, GitHub has no vision for how to evolve to support AI agents. Sure, GitHub has an MCP server, but it has no “AI-native git platform” that can handle the massive load AI agents generate.
• GitHub keeps shipping small features and improvements without direction. For example, in October 2025, they started to work on stacked diffs. However, when it ships, the stacked diffs workflow might be mostly obsolete – at least with AI agents!

It’s easy to win a market when you do one thing better than anyone else in the world. Right now, GitHub is doing too many things and doing a subpar job with Copilot, its platform, and AI infra.

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse.

Catch up with recent The Pragmatic Engineer issues:

• Scaling Uber with Thuan Pham (Uber’s first CTO — podcast). We went into topics like scaling Uber from constant outages to global infrastructure, the shift to microservices and platform teams, and how AI is reshaping engineering.
• Building WhatsApp with Jean Lee (podcast): Jean Lee, engineer #19 at WhatsApp, on scaling the app with a tiny team, the Facebook acquisition, and what it reveals about the future of engineering.
• What will the Staff Engineer role look like in 2027 and beyond? What happens to the Staff engineer role when agents write more code? Actually, they could be more in demand than ever!

An interesting trend highlighted by The Wall Street Journal: companies want to hire for FDE roles, but devs are just not that interested:

“Job postings on Indeed grew more than 10-fold in 2025 compared with 2024. The number of public company transcripts mentioning the role jumped to 50 from eight over the same period, according to data from AlphaSense.

The only problem? Few engineers want the job, which has historically been seen as demanding, undesirable, and less prestigious than product-focused engineering roles.

“Everyone wants them and there’s only maybe 10% of the market that wants that role,” said Patrick Kellenberger, president and chief operating officer at Betts Recruiting.“

Last summer, we covered the rise of the FDE role, and looked into what it’s like. Back then, this is how I visualized what was then a very hot role:

At the companies where I interviewed FDE folks – OpenAI and Ramp – the role seemed to live up to this visualization. However, I’ve since talked with two engineers who took FDE roles and were disappointed. This is how they saw it, in practice:

The role seems akin to a “sales engineer” where FDEs help close the deals, or a solutions engineer (or even consultant), where FDEs deploy to a customer to build them a solution. They don’t contribute back into the platform, and don’t do much that’s considered “software engineering” beyond integrating software which the product team built.

Some engineers figure out the nature of the role during the interview process and pass on it. Meanwhile, some others take the job and later quit. Here’s what a dev told me who accept an FDE role at a company, but didn’t find what they expected:

“This FDE job was a typical IT services mindset. The company wanted to use me more on the engagement lead side, and nothing on software development. It’s not what I signed up for, and I didn’t like the vibe and culture. I quit 4 weeks later.”

In today’s job market, if there’s high demand for a role which pays decently but attracts little interest from engineers, there’s always a reason!

Read the full issue of last week’s The Pulse, or check out this week’s The Pulse.

Catch up with recent The Pragmatic Engineer issues:

• Building WhatsApp with Jean Lee (podcast): Jean Lee, engineer #19 at WhatsApp, on scaling the app with a tiny team, the Facebook acquisition, and what it reveals about the future of engineering.
• The Pulse: What will the Staff Engineer role look like in 2027 and beyond? What happens to the Staff engineer role when agents write more code? Actually, they could be more in demand than ever!
• From IDEs to AI Agents with Steve Yegge (podcast): Steve Yegge on how AI is reshaping software engineering, the rise of “vibe coding,” and why developers must adapt to a rapidly changing craft.

If you’ve been forwarded this email, you can subscribe here to get issues like this in your inbox.

Today’s issue of The Pulse focuses on a single event because it’s a significant one with major potential ripple effects. On Tuesday, Cloudflare shocked the dev world by announcing that they have rewritten Next.js in just one week, with a single developer who used only $1,100 in tokens:

There are several layers to dig into here:

1. The Next.js ecosystem: a recap. Close to half of React devs use Next.js, and the best place to deploy Next.js is on Vercel – partly thanks to its proprietary build output.
2. What Cloudflare did with Next.js. Replacing the build engine in Next.js with the more standard Vite one, allowing Next.js apps to be easily deployed on Cloudflare.
3. AI brings the impossible within reach. What would take years in engineering terms was executed in one week with some tokens.
4. “AI slop” still an issue. Contrary to Cloudflare’s claims, vinext is not production-ready, and will need plenty of cleanup and auditing to make it on par with Next.js.

1. The Next.js ecosystem: a recap

First, some background. Next.js is the most popular fullstack React framework and around half of all React devs use it, as per recent research such as the 2025 Stack Overflow developer survey. Next.js is an open source project, built and mostly maintained by Vercel, which is the preferred deployment target for Next.js applications for many reasons. One of them is that Next.js is ideal to deploy to Vercel because Next.js applications are built with Vercel’s Turbopack build tool. The output of a build is a proprietary format. As Netlify engineer Eduardo Bouças writes:

“The output of a Next.js build has a proprietary and undocumented format that is used in Vercel deployments to provision the infrastructure needed to power the application.

This means that any hosting providers other than Vercel must build on top of undocumented APIs that can introduce unannounced breaking changes in minor or patch releases. (And they have)”.

Next.js is an interestingly built project, where everything is open source, and the best place to deploy a Next.js application is on Vercel, as it’s optimized to run undocumented build artifacts the most efficiently. This is a smart strategy from Vercel which competitors will dislike, as any hosting provider would prefer Next.js to produce a standard build format. To do this, the build engine, Turbopack, would need to be replaced with something more standard.

Let’s talk about build tools for web development. According to the State of JS 2025 survey, the most popular in the web ecosystem are:

1. Vite: the most popular choice for new projects due to its speed and developer experience. Uses projects like esbuild and Rollup under the hood
2. Webpack: a legacy tool that’s not very performant, but still widely deployed in older projects
3. Turbopack: Created by Vercel and optimized for larger Next.js applications. Built in Rust and intended to be more performant
4. Bun: a relatively new, all-in-one runtime and bundler. Anthropic acquired the team in December, and some Bun folks are now focused on improving Claude Code’s performance.

So, most of the web ecosystem uses Vite as a build tool; Next.js uses Turbopack, and the majority of React applications with a full-stack React framework use Next.js. Basically, most devs using Next.js are likely to use Vite as their build tool.

2. What Cloudflare did with Next.js

Here’s a naive idea: what if Next.js used Vite to generate build outputs? In that case, build outputs would be standardized and would run equally well on any cloud provider, as there would be nothing proprietary or undocumented to Vercel.

And this is what Cloudflare did: replace Turbopack with Vite and call the new package ‘vinext’:

Buried midway in the announcement is how this project is experimental and not at all guaranteed to work okay: it’s a ‘use-at-own-risk’ project. Still, the mere fact of this development feels like an earthquake in the tech world because of how it was pulled off.

3. AI brings the impossible within reach

In a blog post announcing the project, Cloudflare claims only one engineer “rebuilt” the whole thing in a way that’s trivial to deploy to Cloudflare’s own infrastructure, and only cost $1,100 in tokens. From Cloudflare’s statement:

“Last week, one engineer and an AI model rebuilt the most popular front-end framework from scratch. The result, vinext (pronounced “vee-next”), is a drop-in replacement for Next.js, built on Vite, that deploys to Cloudflare Workers with a single command. In early benchmarks, it builds production apps up to 4x faster and produces client bundles up to 57% smaller. And we already have customers running it in production.

The whole thing cost about $1,100 in tokens”.

What Cloudflare did:

• Took the Next.js public API
• Reimplemented behaviour using Vite
• Created build output whose behaviour matches the “original” Next.js implementation

After 10 years, the core of Next has around 194,000 lines of code (LOC)**. Meanwhile, vinext is about 67,000 lines of code which suggests a much leaner implementation: for example, vinext does not need to support legacy Next APIs, and vinext currently supports 94% of the Next.js API (and it’s safe to assume they left complex edge cases in the remaining 6%).

** the Next.js repository is closer to 2M lines of code: 1M is bundled dependencies (eg React bundles, CSS build etc), tests are 308,000 LOC, Turbopack 311,000 LOC.

Pre-AI, this reimplementation would have taken years of engineering time to complete. Doing what Cloudflare did was always possible in theory, but never seemed practical. I mean, why have a team of engineers spend potentially years on generating a standardized build output for Next.js apps? Even if they did, the dev community would have doubts about whether Cloudflare would maintain the project.

This is the thing with forking or rewriting open source projects: a major value proposition for commercial open source is to know that they will be maintained. Vercel has proved it’s a reliable custodian of Next.js for the past 10 years. Without AI, it could be assumed that any new reimplementation would eventually run out of steam.

Separately but relatedly, Cloudflare has now proved that the cost of rewriting existing software has become ~100x cheaper, thanks to AI, and this economy is likely to be the case for maintenance, too. Considering how trivial it was to rebuild one of the more complex open source projects, this augers well for it being trivial and much cheaper to maintain in the future. Potentially, Cloudflare no longer needs to budget an engineering team only for maintenance, if a single engineer could maintain the project, part-time!

Cloudflare had a project measured in engineering years, and completed it in one engineering week! It just took a single engineer using OpenCode (open source coding agent), Opus 4.5, and a bunch of tokens, then: ‘boom’, vinext was born.

4. “AI slop” still an issue

There are questions about the quality of vinext, though. Vercel, naturally, is unhappy and hit out at the obvious weakness that vinext is unfit for production usage because it’s insecure. Vercel CEO, Guillermo Rauch, did not miss a beat by tying Cloudflare’s effort to the “vibe coding” stereotype of sloppy work executed with a lack of understanding:

Guillermo has a point: anyone who stopped reading Cloudflare’s launch announcement after the first few sentences would assume it’s production-ready, with the first paragraph of this announcement closing with:

“And we already have customers running it in production.”

However, Cloudflare doesn’t share the rather crucial detail that “running in production” means that vinext has been deployed onto a beta site, until more than 1,000 words (around 2–3 pages) into the announcement:

“We want to be clear: vinext is experimental. It’s not even one week old, and it has not yet been battle-tested with any meaningful traffic at scale. (...)

We’ve been working with National Design Studio, a team that’s aiming to modernize every government interface, on one of their beta sites, CIO.gov.

Oh. So, “customers running it in production” at Cloudflare apparently means “customer running a beta site in production without meaningful traffic.” This is a first from the infrastructure giant, which usually prides itself on accurate statements!

This detail was also absent when Cloudflare’s CEO and CTO were boosting vinext like it was a mature, battle-tested product. In that context, Vercel’s raising of the issue of security vulnerabilities is more than fair game, in my view.

Still, all that doesn’t alter the core learning from this project: that AI has the power to drastically reduce engineering time by up to ~100x and deliver usable-enough output, for relatively negligible financial cost. Just keep in mind that security and reliability issues will probably take plenty of extra time and effort to address.

5. New attack vector on commercial open source?

If arch-rivalries exist in tech, then Cloudflare and Vercel are a prime example. Both are gunning to become the most popular platform for developers to deploy their code, and the CEOs are regularly seen in public taking shots at the other side. One such spat happened in March, as covered at the time:

“Things kicked off on social media, with developers confused about the severity of the incident, and about why Next.js seemed silent, and also why Cloudflare sites were breaking due to its fix for the CVE causing its own issues. It was at that point that Cloudflare’s CEO, Matthew Prince, entered the chat to accuse Vercel of not caring about security:

Given the security incident was ongoing, this felt a bit “below the belt” by the Cloudflare chief. Criticizing rivals is fair game, but why not wait until the incident is over? The punch landed, and Vercel’s CEO Guillermo Rauch is not someone to take it lying down, so he hit back.

Cloudflare’s CEO then responded with a cartoon implying that although Vercel is much larger than its competitor Netlify, Cloudflare is 100x bigger than both, and could stomp them into the ground at will.”

Serving the public interest wasn’t why Cloudflare rewrote Next.js: they did it because they want Next.js sites to be deployed onto Cloudflare, but doing so made little sense until now because Next.js produced bespoke build output optimized for Vercel’s infrastructure. With this change, Cloudflare claims it provides superior performance when hosting Next.js apps, according to their own measurements.

I’d just add that performance is important for developers, but other things matter, too. Cost, reliability, developer experience, and how much devs like a company, are all factors in choosing between vendors. Also, performance measurements from a vendor about its own service must be taken with a large pinch of salt.

Zooming out from this episode, it seems that AI is bringing the value of existing commercial open source moats into question. Vercel carved out a clever open source strategy that helped turn its open source investment into business revenue:

1. Build and maintain Next.js, delivering the best developer experience (DX).
2. Optimize Vercel to serve the specific (and undocumented) build output of Next.js.
3. Most developers onboarding to Next.js will decide to deploy on Vercel to get the most benefit, in terms of DX and performance.
4. … repeat for years while the business becomes worth billions! (Vercel was valued at $9B last October).

Underpinning this success are some assumptions:

1. Next.js will remain the #1 choice for developers to build React applications, thanks to ongoing investment.
2. It is expensive to rewrite Next.js to be deployable and performant on another cloud vendor.
3. Even if someone did #2, developers would be skeptical and not switch over.

Vercel can invest in #1 to keep Next as best-in-class, while knowing that the risk of #2 occurring is minor. However, Cloudflare has now “cloned” Next, and can easily keep up with all changes in the future, and port them back to vinext.

But AI makes it trivial to “piggyback” off any commercial open source project, which is a massive problem for commercial open source startups. It puts all the effort and investment into building and maintaining Next.js, while Cloudflare enjoys the benefit of this hard work (the Next.js public API) which is easily deployable to Cloudflare, and it can now undercut Vercel on price. For all future Next.js changes, Cloudflare will just sync it to vinext, using AI!

WordPress had a similar problem, with WP Engine “piggybacking” off its work and undercutting their pricing in 2024. As I analyzed at the time:

“Free-riding on permissive open source is too tempting to pass on for other vendors. WP Engine uses a common loophole of contributing almost nothing in R&D to WordPress, while selling it as a managed service. This means that they could either easily undercut the pricing of larger players like Automattic which do spend on WordPress’s R&D. Alternatively, a company like WP Engine could charge as much, or more, as Automattic, but be able to spend a lot more on marketing, while being similarly profitable. “Saving” on R&D gives the “free-riders” plenty of options to grow their businesses: options not necessarily open to Automattic while they invest as much into R&D as they do.

Commercial open source vendors pressure to end “freeriding”. Automattic is likely facing lower revenue growth, with customers choosing vendors like WP Engine which offer a similar service — getting these customers either via a cheaper price or thanks to more marketing spend. This legal fight could be an effort to force WP Engine to stop eating Automattic’s lunch, or perhaps get WP Engine to sell to Automattic, which would cement its leading status in managed Wordpress, while also boosting revenue by $400M a year – according to its own figures”.

Vercel managed to avoid the “free-riding” problem with Next.js, but that’s no longer possible now that AI makes it trivial to rewrite.

6. Defense or offense?

How should commercial open source companies respond to the threat that a competitor can easily rewrite the software behind the managed solutions which they sell as services?

One obvious response is to make tests private, so that replication is harder for AI. One thing that made it so easy for Cloudflare to rewrite Next was the project’s comprehensive test suite. From their announcement (emphasis mine):

“We also want to acknowledge the Next.js team. They’ve spent years building a framework that raised the bar for what React development could look like. The fact that their API surface is so well-documented and their test suite so comprehensive is a big part of what made this project possible.”

Database solution SQLite is famous for its incredible test suite. What some people don’t know is that while core SQLite tests are open source, its most comprehensive test suite – TH3 – is closed source. SQLite monetizes its advanced infrastructure as a service for purchase. This is a fair tradeoff: for most contributors, the basic open source tests work well enough. For enterprise users or customers who really care about correctness, it makes sense to purchase advanced testing services from the service’s creator.

Open source canvas project, tldraw, announced it will relocate its test suite to a closed source repository; a move which makes plenty of sense. Here’s commentary from Simon Willison:

“It’s become very apparent over the past few months that a comprehensive test suite is enough to build a completely fresh implementation of any open source library from scratch, potentially in a different language.”

In the event, tldraw’s announcement turned out to be a joke, but who’s laughing now? An open source project with excellent tests is an easy target for an AI agent to execute a full rewrite of it.

Could new licenses be created for the AI era? Existing open source licenses were created on the assumption that humans read open source code, and humans modify it. Agents break that assumption.

Could we see new license types emerge to ban AI agents from modifying projects’ source code? It seems pretty far-fetched and hard to implement, but not beyond the realms of possibility.

AI agents are still very new, and going mainstream in tech. Once they break into other industries, I wouldn’t be surprised if legal frameworks are reworded to also apply to AI agents. If and when this happens, it would open the path for open source licenses to distinguish between agents and humans.

What is a moat, if code can be trivially ported? A team operating a popular open source project can no longer assume it’s expensive to fork or to be completely rewritten, meaning it makes sense to focus on other moats, such as:

• Outstanding (paid) support. AI could make this much easier at a higher quality, if done right.
• Smaller open core, larger closed source part. “Open core” as a business model has been dominant for commercial open source: keep the core of the software open source, while advanced enterprise features are source available or closed source. I would expect more companies to move their additional services to closed source, not source available.
• In-person connection and community. Projects with a real-world community will form a sense of connection that goes beyond code. For example, it’s hard to imagine vinext meetups popping up – whereas there are many Next.js communities.
• Infrastructure and hardware remains a massive moat. In a world where software is trivial to copy, infrastructure remains a moat. Commercial open source might make most sense for players that own and operate superior infrastructure layers than their rivals: and being able to offer lower cost, higher reliability, lower latency, higher performance, or a combination of these.

7. AI-world reality

One of the single best AI use cases is full-on rewrites of well-tested products. I estimate that AI sped up the creation of vinext by at least 100x, which is massive. But we don’t really see efficiency boosts of anything like that with AI tools, in general. As Laura Tacho shared at The Pragmatic Summit in San Francisco, the average self-reported efficiency ‘AI gain’ seems to be circa 10%.

I suspect this vast chasm in efficiency boosts is because AI is many times more efficient at “no-brainer tasks” where correctness can be verified with tests, versus those which are more open ended or involve more creativity.

In general, tests are incredibly important for efficient AI usage. On The Pragmatic Engineer Podcast, Peter Steinberger stressed how important “closing the loop” in his developer flow is by instructing the AI to test itself, and ensuring the AI has tests to run that verify correctness.

Automated tests were always considered a best practice for creating maintainable code. Now, having a codebase with extensive tests is the baseline to make AI agents work productively for refactors, rewrites – or even adding new features and verifying that things did not break!

Vendors will start to deploy “migration AI agents” to move customers over to their own stacks. This got lost in Cloudflare’s announcement, but it’s important:

vinext includes an Agent Skill that handles migration for you. It works with Claude Code, OpenCode, Cursor, Codex, and dozens of other AI coding tools. Install it, open your Next.js project, and tell the AI to migrate:

> npx skills add cloudflare/vinext

Then open your Next.js project in any supported tool and say:

> migrate this project to vinext

The skill handles compatibility checking, dependency installation, config generation, and dev server startup. It knows what vinext supports and will flag anything that needs manual attention.

This is very clever from Cloudflare, and a true “AI-native” move. They have not only used AI to migrate Next.js, but also built an “AI plugin” (a skill) to help customers migrate their existing codebases over to vinext – and deploy on Cloudflare!

This move will surely be copied by other vendors, since migrations which are tedious for humans are much less effort with agents.

AI is making the tech industry more ruthless when it comes to business practices. Laura Tacho said something interesting at The Pragmatic Summit:

“AI is an accelerator, it’s a multiplier, and it is moving organizations in different directions.”

AI seems to be accelerating the ruthlessness of competition for customers and the speed at which this happens. In one week, Cloudflare rebuilt Next.js, and it’s attacking Vercel full-on: claiming their “vibe coded” alternative is more performant and production-ready, and burying at the foot of the launch announcement the crucial information that vinext is very much experimental.

I sense vendors are realizing that there’s a limited amount of time in which to use AI to their advantage, and some will decide to use it like Cloudflare has.

On the other hand, AI could be great news for non-commercial open source. AI presents as a threat to commercial open source because it removes existing moats which make code hard to fully rewrite. However, beyond that, AI could help non-commercial open source to thrive:

• With AI, it’s easy to fork an open source project and keep the fork in-sync with the original.
• It’s trivial to instruct AI to rewrite an open source project to another language or framework.
• …and it’s equally trivial for AI to add features to a fork.

For these reasons, I believe there could be a lot more forks and rewrites to come, and more open source projects and code, in general.

Takeaways

Personally, I could not have imagined things changing this quickly in software. Rewriting Next.js in a single week, even to a version that is not quite there – but mostly works? This was out of the question as recently as a few months ago.

Things changed around last December, when Opus 4.5 and GPT-5.2 came out and proved capable of writing most of the code. What used to be expensive is now cheap – like rewriting complete projects – and we still need to learn what the “new” expensive parts of software engineering are.

All this is new territory for everyone. To succeed in the tech industry, you need to be able to capitalize upon change, as Cloudflare has clearly done in this case by making the most of an opportunity created by new technology. It’s unclear how popular vinext will become, and how much of a moat Vercel has around the broader Next.js ecosystem, but I suspect that it’d take more than a Next rewrite to make Cloudflare into a viable Next.js platform-as-a-service provider.

1. SaaS is a pure software product. People who pay SaaS vendors do so because it’s cheaper to buy this software than build it.
2. LLMs dramatically reduce the time and cost of building custom software.
3. Therefore, most SaaS vendors will go out of business because most companies/teams will prompt an LLM to write the software they need, such as for ticketing, meetings, customer relationship management, etc.

The reason for my scepticism has been that SaaS such as HR software Workday is more than just software. Workday, for example, keeps up with compliance requirements (e.g., for holiday pay in different countries), guarantees correctness (e.g., payslips that comply with local regulations), and over time the software keeps up to date with changes in the external and internal environments.

However, this week I had first-hand experience of how ridiculously easy it is now to replace SaaS with LLMs. On my website – pragmaticengineer.com – I have a testimonials section, which displays real LinkedIn and X posts about this publication. It cost $120/year for a small service called Shoutout.io, and looked like this:

And this is the backend: nothing fancy, just a way to add, edit, reorganize, and delete testimonials.

I was a customer for four years and logged in perhaps once a year. My latest login was to get an annual invoice for my expenses. Unfortunately, the billing section was broken, so I emailed support and they sent me a broken link instead of the invoice. This was frustrating: why pay for a SaaS with broken billing? I couldn’t even tell what they would charge me next year.

So I asked myself if I could rebuild my own use case with an LLM, and do it rapidly. My use case was much simpler than the SaaS itself:

• Display existing testimonials in a similar way
• Make it easy to add new ones, e.g., store testimonials in some JSON format
• Make it look good

To my surprise, this whole effort from start to finish took exactly 20 minutes with Codex. The steps I took were straightforward enough:

• Asked Codex to make a plan on how to remove this third-party dependency and host all testimonials in my codebase (a GitHub repo, deployed onto Netlify)
• Tweaked the plan: I pushed for a modular approach where testimonials are in a separate JSON file, and they get generated into HTML with a compile-time build step
• Added this build step both locally and as a build trigger on Netlify
• Tested the solution
• Tweaked the UX and generated a schema
• Deployed it

The end result is visually the same as before, except I no longer have a third-party dependency rendering all of this!

What does this mean for SaaS products and software engineers?

What it means for software engineers:

• Devs are (probably) a lot more comfortable using the command line for future updates than regular users. To add a future testimonial, I’ll need to turn to my AI agent to insert it in my codebase, and I’ll then need to verify that it looks good. This is not a big deal for me, but it might be a dealbreaker for someone not comfortable with verifying the code output of an LLM.
• It’s a lot faster for a dev to “port” a SaaS than for anyone else. I first told Codex to copy the UI and it got things wrong because it tried to use a flexbox model. I had to tell it that this UI layout was not what I wanted, and then make the decision on which framework to use for the UI layout. A non-developer could probably figure all this out, but it would take longer.
• Honestly, it’s fun and interesting to rewrite a third-party feature. I recommend it. Part of why I took on this project is because I expected it to be an interesting challenge. I thought the effort would be more than what it was, and I’ve learned more about how well these tools work. I also used Codex in order to experience it more.

What this could mean for SaaS software:

• Rebuilding a SaaS still feels much harder than rebuilding your specific use case. I did not “rebuild” Shoutout in any way. Shoutout has 10x or more features, like adding quotes from 10 different platforms, authentication, billing (which didn’t work for me), and more.
• A SaaS that doesn’t give ongoing value is at risk of being replaced by customers. Shoutout doesn’t provide ongoing value after it displays my testimonials, and this static nature means it’s easy to replace. In contrast, it would be harder to rebuild if I paid for the platform to stay compliant, provide analytics or alerting, and do other real-time things that helped my business.
• Buying and selling SaaS businesses could become less profitable. The original version of Shoutout that I signed up for in 2021 was built in 2020 by an independent developer. In 2022, this developer sold this micro-SaaS to a product studio. Then, in 2025, Shoutout was sold again to new developers. From my point of view, nothing changed except that the billing system broke. I assume the buyers of this SaaS figured that revenue could keep rising with zero investment. But perhaps at some point that ceases to be true when people get fed up with a broken product and quit – especially when doing so is cheaper.

“Broken windows” not being fixed is less acceptable than it used to be. My journey away from Shoutout began with its billing system being broken. For example, below is what I saw when I went to my billing section to see the invoices:

As well as this, the customer support sent me a broken link in response to my email. That was enough for me to decide to replace this dependency, and I was surprised by how easy this was with an LLM and knowing what I wanted it to build. By the time customer support sent me a working link two hours later, I had finished migrating off the SaaS.

It feels like something valuable is being taken away, and suddenly. It took a lot of effort to get good at coding and to learn how to write code that works, to read and understand complex code, and to debug and fix when code doesn’t work as it should. I still remember how daunting my first “real” programming class was at university (learning C), how lost I felt on my first job with a complex codebase, and how it took years of practice, learning from other devs, books, and blogs, to get better at the craft. Once you’re pretty good, you have something that’s valuable and easy to validate by writing code that works!

Some of my best memories of building software are about coding. Being “locked in” and balancing several ideas while typing them out, of being in the zone, then compiling the code, running it and seeing that “YES”, it worked as expected!

It’s been a love-hate relationship, to be fair, based on the amount of focus needed to write complex code. Then there’s all the conflicts that time estimates caused: time passes differently when you’re locked in and working on a hard problem.

Now, all that looks like it will be history.

I wonder if I’ll still get the same sense of satisfaction from the fact that writing complicated code is hard? Yes, AI is convenient, but there’s also a loss.

Or perhaps with AI agents, being “in the zone” will shift to thinking about higher-level problems, while instructing more complex code to be written?

This was a section from my analysis piece When AI writes almost all code, what happens to software engineering?. Read the full one here.

A mere two weeks after Cloudflare suffered a major outage and took down half the internet, the same thing has happened again. Last Friday, 5th December, thousands of sites went down or partially down once more, in a global Cloudflare outage lasting 25 minutes.

As per last time, Cloudflare was speedy to share a full postmortem on the same day. It estimated that 28% of Cloudflare’s HTTP traffic was impacted. The cause of this latest outage was Cloudflare making a seemingly innocent – but global – configuration change that went on to take out a good portion of Cloudflare, globally, until being reverted. Here’s what happened:

• Cloudflare was rolling out a fix for a nasty React security vulnerability
• The fix caused an error in an internal testing tool
• The Cloudflare team disabled the testing tool with a global killswitch
• As this global configuration change was made, the killswitch unexpectedly caused a bug that resulted in HTTP 500 errors across Cloudflare’s network

In this latest outage, Cloudflare was burnt by yet another global configuration change. The previous outage in November happened thanks to a global database permissions change. In the postmortem of that incident, the Cloudflare team closed with this action item:

“Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input”

This change would make it so that Cloudflare’s configuration files do not propagate immediately to the full network, as they still do now. But making all global configuration files have staged rollouts is a large implementation that could take months. Evidently, there wasn’t time to make it yet, and it has come back to bite Cloudflare.

Unfortunately for Cloudflare, customers are likely to find unacceptable a second outage with similar causes to a previous one, only weeks ago. If Cloudflare proves unreliable, customers should plan to onboard to backup CDNs at the very least, and a backup CDN vendor will do its best to convince new customers to use it as the primary CDN.

Cloudflare’s value-add rests on rock-solid reliability without customers needing to budget for a backup CDN. Yes, publishing postmortems on the same day as an outage occurs helps restore trust, but that will crumble anyway with repeated large outages.

To be fair, the company is doubling down on implementing staged configuration rollouts. In its postmortem, Cloudflare is its own biggest critic. CTO Dane Knecht reflected:

“[Global configuration changes rolling out globally] remains our first priority across the organization. In particular, the projects outlined below should help contain the impact of these kinds of changes:Enhanced Rollouts & Versioning: Similar to how we slowly deploy software with strict health validation, data used for rapid threat response and general configuration needs to have the same safety and blast mitigation features. This includes health validation and quick rollback capabilities among other things.Streamlined break glass capabilities: Ensure that critical operations can still be achieved in the face of additional types of failures. This applies to internal services as well as all standard methods of interaction with the Cloudflare control plane used by all Cloudflare customers.“Fail-Open” Error Handling: As part of the resilience effort, we are replacing the incorrectly applied hard-fail logic across all critical Cloudflare data-plane components. If a configuration file is corrupt or out-of-range (e.g., exceeding feature caps), the system will log the error and default to a known-good state or pass traffic without scoring, rather than dropping requests. Some services will likely give the customer the option to fail open or closed in certain scenarios. This will include drift-prevention capabilities to ensure this is enforced continuously.
These kinds of incidents, and how closely they are clustered together, are not acceptable for a network like ours”.

Global configuration errors often trigger large outages

There’s a pattern of implicit or explicit global configuration errors causing large outages, and some of the biggest ones in recent years were caused by a single change being rolled out to a whole network of machines:

• DNS and DNS-related systems like BGP: DNS changes are global by default, so it’s no wonder that DNS changes can cause global outages. Meta’s 7-hour outage in 2021 was related to DNS changes (more specifically, Border Gateway Protocol changes.) Meanwhile, the AWS outage in October started with the internal DNS system.
• OS updates happening at the same time, globally: Datadog’s 2023 outage cost the company $5M and was caused by Datadog’s Ubuntu machines executing an OS update within the same time window, globally. It caused issues with networking, and it didn’t help that Datadog ran its infra on 3 different cloud providers across 3 networks. The same kind of Ubuntu update also caused a global outage for Heroku in 2024.

Globally replicating configs: in 2024, a configuration policy change was rolled out globally and crashed every Spanner database node straight away. As Google concluded in its postmortem: “Given the global nature of quota management, this metadata was replicated globally within seconds”.

Implementing gradual rollouts for all configuration files is a lot of work. It’s also invisible labor because when done well, then its benefits will be undetectable, except in the absence of incidents, thanks to better infrastructure!

The largest systems in the world will likely have to implement safer ways to roll out configs – but not everybody needs to. Staged configuration rollout doesn’t make much sense for smaller companies and products because this infra work slows down product development.

It doesn’t just slow down building, but every deployment, too, and this friction is designed to make everything slower. As such, they don’t make much sense unless the stability of mature systems is more important than fast iteration.

Software engineering is a field where tradeoffs are a fact of life, and universal solutions don’t exist. The development which worked for a system with 1/100th of the load and users a year ago, may not make sense today.

This was one out of the four topics covered in this week’s The Pulse. The full edition additionally covers:

1. Industry Pulse. Poor capacity planning at AWS, Meta moves to a “closed AI” approach, a looming RAM shortage, early-stage startups hiring slower than before, how long it takes to earn $600K at Amazon and Meta, Apple loses execs to Meta, and more
2. How the engineering team at Oxide uses LLMs. They find LLMs great for reading documents and lightweight research, mixed for coding and code review, and a poor choice for writing documents – or any kind of writing, really!
3. Linux officially supports Rust in the kernel. Rust is now a first-class language inside the Linux kernel, eight months after a Linux Foundation Fellow predicted more support for Rust. A summary of the pros and cons of Rust support for Linux

Read the full The Pulse issue.

A year ago, Amazon became the first tech giant to bring staff back into the office for the full five days per week. Back then, I analyzed the reasons for the change, and whether other workplaces would follow suit by dropping the widespread hybrid policy of 2-3 days/week in the office.

Now, Meta employees in the Instagram division have become the latest subjects of a full return to the office, following an announcement by the social media platform this week.

Instagram’s 5-day return to office

Instagram employees received the unexpected email on Monday, reports fellow Substacker, Alex Heath, who acquired a copy of the message. It was sent internally by Instagram CEO Adam Mosseri, who wrote:

“1. Back to the office: I believe that we are more creative and collaborative when we are together in-person. (...)

2. Fewer meetings: We all spend too much time in meetings that are not effective, and it’s slowing us down. Every six months, we’ll cancel all recurring meetings and only re-add the ones that are absolutely necessary (...)

3. More demos, less decks: Most product overviews should be prototypes instead of decks.

4. Faster decision-making: We’re going to have a more formalized unblocking process with DRIs, and I’ll be at the priorities progress unblocking meeting every week.”

This decision by Meta affects around a quarter of company staff, and it’s hard to imagine other divisions not following Instagram’s lead; after all, everything in Mosseri’s memo likely applies across the business.

Five years ago, CEO Mark Zuckerberg predicted 50% of Meta staff would work remotely by now, which didn’t happen. Indeed, with Instagram’s new 5-day RTO, I’d be surprised if 5% of Meta folks work remotely in two years’ time.

The reason for Insta’s RTO seems rooted in the leadership’s belief that in-office is more productive, as indicated by the top bullet point of Mosseri’s message. That message in full:

“I believe that we are more creative and collaborative when we are together in-person. I felt this pre-COVID and I feel it any time I go to our New York office where the in-person culture is strong.

Starting February 2, I’m asking everyone in my rollup based in a US office with assigned desks to come back full time (five days a week). The specifics:

• You’ll still have the flexibility to work from home when you need to, since I recognize there will be times you won’t be able to come into the office. I trust you all to use your best judgment in figuring out how to adapt to this schedule.
• In the NY office, we won’t expect you to come back full time until we’ve alleviated the space constraints. We’ll share more once we have a better sense of timeline.
• In MPK [Menlo Park, the HQ], we’ll move from MPK21 to MPK22 on January 26 so everyone has an assigned desk. We’re also offering the option to transfer from the MPK to SF office for those people whose commute would be the same or better with that change. We’ll reach out directly to those people with more info.
• XFN [cross-functional] partners will continue to follow their own org norms.
• There is no change for employees who are currently remote”.

From what I’ve seen of Mosseri from afar, he seems like a pretty straight shooter. It’s clear that he feels in-office creates more energy, and in Mosseri’s defense, I hear similar from many startup founders and leaders who say remote work causes a bunch of headaches: it’s harder to spot motivational problems and performance issues, information travels more slowly, and rallying teams is harder.

There’s no doubt that running a full-remote company is a lot of effort. There’s often-overlooked labor involved in hiring, onboarding, performance management, team celebrations, and even company-wide meetings – none of it is easy.

Linear is a full-remote company with nearly 50 people working there, which recently published details about how it operates. They’re introducing the concept of “coworking hubs”, flying in teams for in-person events, and holding regular off-sites, while being careful to hire people who fit the culture.

My feeling is that remote work policies at tech companies are going to become questions of their leaders’ preferences. Many devs prefer remote work: there’s fewer interruptions, more deep focus, and less commuting. Most of us would probably be just as productive – and probably more so – than when being interrupted in-office.

Leaders who prefer full-remote can cite flexibility and easier hiring from a larger pool of candidates as clear benefits. Meanwhile, those most comfortable with in-person will always have enough reasons to justify a 5-day RTO, along the lines of Mosseri’s reasoning. Advocates of hybrid setups cite balancing of focus time and efficiency.

In today’s job market, any company that pays closer to the top of the market can probably get away with five-days-a-week RTO. Meta is in this space, and although I’m sure plenty of devs will dislike the change, the alternative is to go out on the job market, accept a pay cut to join a new company, and start rebuilding your internal network.

Since we’re in the midst of a weird job market, it makes switching jobs more difficult than before, when the job market was very hot. In this respect, Instagram has external conditions on its side. For devs at Meta, one upside is that Big Tech experience opens more doors, even in this tough job market.

One caveat is that a 5-day RTO is unlikely in places where it’s hard to hire the right people. So, AI engineers and those working on AI products should be pretty safe, for instance, because those roles are incredibly in-demand, as indicated by the trend of higher base salaries for AI engineers. Based on that, few companies should want to push those workers to quit to join competitors.

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Many subscribers expense The Pragmatic Engineer Newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

One amusing detail of the November 2025 Cloudflare outage is that the realtime outage and monitoring service, Downdetector, went down, revealing a key dependency on Cloudflare. At first, this looks odd; after all, Downdetector is about monitoring uptime, so why would it take on a key dependency like Cloudflare if it means this can happen?

Downdetector was built multi-region and multi-cloud, which I confirmed by talking with Senior Director of Engineering, Dhruv Arora, at Ookla, the company behind Downdetector. Multi-cloud resilience makes little sense for most products, but Downdetector was built to detect cloud provider outages, as well. And for this, they needed to be multi-cloud!

Still, Downdetector uses Cloudflare for DNS, Content Delivery (CDN), and Bot Protection. So, why would it take on this one key dependency, as opposed to hosting everything on its own servers?

A CDN has advantages that are hard to ignore, such as:

• Drastically lower bandwidth costs – assets cached on the CDN are much faster
• Faster load times because assets on a CDN are served from Edge nodes nearer users
• Protection from sudden traffic spikes, as would be common for Downdetector, especially during outages! Without a CDN, those spikes could overload their services
• DDoS protection from bad actors taking the site offline with a distributed denial of service attack
• Reduced infrastructure requirements, as Downdetector can run on fewer servers

Downdetector’s usage patterns reflect that it’s a service very heavily used by consumers whom the business doesn’t really monetize (Downdetector is free to use.) So, Downdetector could get rid of Cloudflare, but costs would surge, the site would become slower to load, and revenue wouldn’t change.

In the end, Downdetector’s dependence on Cloudflare could be a pragmatic choice based on the business model, and how removing its upstream dependency upon Cloudflare could get very expensive!

Dhruv confirmed this and sharing more about the design choices at Downdetector:

“Building redundancy at the DNS & CDN layers would require enormous overhead. This is especially true as Cloudflare’s Bot Protection is world-class, and building similar functionality would be a lot of effort. There are hyperscalers [cloud providers] that have this kind of redundancy built in. We will look into what we can do, but with a team size in the double digits, building up a core piece of infra like this is a pretty tall order: not just for us, but for any mid-sized team.

We’ve learned that there are more things that we can improve, for the future. For example, during the outage, the Cloudflare control pane was down, but their API wasn’t. So, us having more Infrastructure as Code could have helped bring back Downdetector sooner.

On our end, we also noticed that the outage wasn’t global, so we were able to shift traffic around and reduce the impact.

One more interesting detail: Cloudflare’s Bot Protection went haywire during the outage, and started to block legitimate traffic. So, our team had to turn that off temporarily”.

Thanks very much to Dhruv and the Downdetector team for sharing details.

An unexpected highlight of publishing the book was ending up in Mongolia in June of this year, at a small-but-mighty startup called Nasha Tech. This was because the startup translated my book into Mongolian. Here's the completed book:

Here’s what happened:

A little over a year ago, a small startup from Mongolia reached out, asking if they could translate the book. I was skeptical it would happen because the unit economics appeared pretty unfavorable. Mongolia’s population is 3.5 million; much smaller than other countries where professional publishers had offered to do a translation (Taiwan: 23M, South Korea: 51M, Germany: 84M, Japan: 122M, China: 1.43B people).

But I agreed to the initiative, and expected to hear nothing back. To my surprise, nine months later the translation was ready, and the startup printed 500 copies on the first run. They invited me to a book signing in the capital city of Ulaanbaatar, and soon I was on my way to meet the team, and to understand why a small tech company translated my book!

Japanese startup vibes in Mongolia

The startup behind the translation is called Nasha Tech; a mix of a startup and a digital agency. Founded in 2018, its main business has been agency work, mainly for companies in Japan. They are a group of 30 people, mostly software engineers.

Their offices resembled a mansion more than a typical workplace, and everyone takes their shoes off when arriving at work and switches to “office slippers”. I encountered the same vibe later at Cursor’s headquarters in San Francisco, in the US.

Nasha Tech found a niche of working for Japanese companies thanks to one of its cofounders studying in Japan, and building up connections while there. Interestingly, another cofounder later moved to Silicon Valley, and advises the company from afar.

The business builds the “Uber Eats of Mongolia”. Outside of working as an agency, Nasha Tech builds its own products. The most notable is called TokTok, the “UberEats of Mongolia”, which is the leading food delivery app in the capital city. The only difference between TokTok and other food delivery apps is scale: the local market is smaller than in some other cities. At a few thousand orders per day, it might not be worthwhile for an international player like Uber or Deliveroo to enter the market.

The tech stack Nasha Tech typically uses:

• Frontend: React / Next, Vue / Nuxt, TypeScript, Electron, Tailwind, Element UI
• Backend and API: NodeJS (Express, Hono, Deno, NestJS), Python (FastAPI, Flask), Ruby on Rails, PHP (Laravel), GraphQL, Socket, Recoil
• Mobile: Flutter, React Native, Fastlane
• Infra: AWS, GCP, Docker, Kubernetes, Terraform
• AI & ML: GCP Vertex, AWS Bedrock, Elasticsearch, LangChain, Langfuse

AI tools are very much widespread, and today the team uses Cursor, GitHub Copilot, Claude Code, OpenAI Codex, and Junie by Jetbrains.

I detected very few differences between Nasha Tech and other “typical” startups I’ve visited, in terms of the vibe and tech stack. Devs working on TokTok were very passionate about how to improve the app and reduce the tech debt accumulated by prioritizing the launch. A difference for me was the language and target market: the main language in the office is, obviously, Mongolian, and the products they build like TokTok also target the Mongolian market, or the Japanese one when working with clients.

One thing I learned was that awareness about the latest tools has no borders: back in June, a dev at Nasha Tech was already telling me that Claude Code was their daily driver, even though the tool had been released for barely a month at that point!

Why translate the book into Mongolian?

Nasha Tech was the only non-book publisher to express interest in translating the book. But why did they do it?

I was told the idea came from software engineer Suuribaatar Sainjargal, who bought and enjoyed the English-language version. He suggested translating the book so that everyone at the company could read it, not only those fluent in English.

Nasha Tech actually had some in-house experience of translation. A year earlier, in 2024, the company translated Matt Mochary’s The Great CEO Within as a way to uplevel their leadership team, and to help the broader Mongolian tech ecosystem.

Also, the company’s General Manager, Batutsengel Davaa, happened to have been involved in translating more than 10 books in a previous role. He took the lead in organizing this work, and here’s how the timelines played out:

• Professional translator: 3 months
• Technical editor revising the draft translation: 1 month
• Technical editing #2 by a Support Engineer in Japan: 2 months
• Technical revision: 15 engineers at Nasha Tech revised the book, with a “divide and conquer” approach: 2 months
• Final edit and print: 1 month

This was a real team effort. Somehow, this startup managed to produce a high-quality translation in around the same time as it took professional book publishers in my part of the world to do the same!

A secondary goal that Nasha Tech had was to advance the tech ecosystem in Mongolia. There’s understandably high demand for books in the mother tongue; I observed a number of book stands selling these books, and book fairs are also popular. The translation of my book has been selling well, where you can buy the book for 70,000 MNTs (~$19).

Book signing and the Mongolian startup scene

The book launch event was at Mongolia’s startup hub, called IT Park, which offers space for startups to operate in. I met a few working in the AI and fintech spaces – and even one startup producing comics.

I had the impression that the government and private sector are investing heavily in startups, and want to help more companies to become breakout success stories:

• IT Park report: the country’s tech sector is growing ~20%, year-on-year. The combined valuation of all startups in Mongolia is at $130M, today. It’s worth remembering that location is important for startups: being in hubs like the US, UK, and India confers advantages that can be reflected in valuations.
• Mongolian Startup Ecosystem Report 2023: the average pre-seed valuation of a startup in Mongolia is $170K, seed valuation at $330K, and Series A valuation at $870K. The numbers reflect market size; for savvy investors, this could also be an opportunity to invest early. I met a Staff Software Engineer at the book signing event who is working in Silicon Valley at Google, and invests and advises in startups in Mongolia.
• Mongolian startup ecosystem Map: better-known startups in the country.

Two promising startups from Mongolia: Chimege (an AI+voice startup) AND Global (fintech). Thanks very much to the Nasha Tech team for translating the book – keep up the great work!

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Before we start: I’m excited to share something new: The Pragmatic Summit.

Four years ago, The Pragmatic Engineer started as a small newsletter: me writing about topics relevant for engineers and engineering leaders at Big Tech and startups. Fast forward to today, and the newsletter crossed one million readers, and the publication expanded with a podcast as well.

One thing that was always missing: meeting in person. Engineers, leaders, founders—people who want to meet others in this community, and learn from each other. Until now that is:

In partnership with Statsig, I’m hosting the first-ever Pragmatic Summit. Seats are limited, and tickets are priced at $499, covering the venue, meals, and production—we’re not aiming to make any profit from this event.

Apply to attend the Summit

I hope to see many of you there!

Cloudflare takes down half the internet – but shares a great postmortem

On Tuesday came another reminder about how much of the internet depends on Cloudflare’s content delivery network (CDN), when thousands of sites went fully or partially offline in an outage that lasted 6 hours. Some of the higher-profile victims included:

• ChatGPT and Claude
• Canva, Dropbox, Spotify,
• Uber, Coinbase, Zoom
• X and Reddit

Separately, you may or may not recall that during a different recent outage caused by AWS, Elon Musk noted on his website, X, that AWS is a hard dependency for Signal, meaning an AWS outage could take down the secure messaging service at any moment. In response, a dev pointed out that it is the same for X with Cloudflare – and so it proved earlier this week, when X was broken by the Cloudflare outage.

That AWS outage was in the company’s us-east-1 region and took down a good part of the internet last month. AWS released incident details three days later – unusually speedy for the e-commerce giant – although that postmortem was high-level and we never learned exactly what caused AWS’s DNS Enactor service to slow down, triggering an unexpected race condition that kicked off the outage.

What happened this time with Cloudflare?

Within hours of mitigating the outage, Cloudflare’s CEO Matthew Prince shared an unusually detailed report of what exactly went wrong. The root cause was to do with propagating a configuration file to Cloudflare’s Bot Management module. The file crashed Bot Management, which took Cloudflare’s proxy functionality offline.

Here’s a brief overview of how Cloudflare’s proxy layer works at a high level. It’s the layer that protects the “origin” resources of customers – minimizing network traffic to them by blocking malicious requests and caching static resources in Cloudflare’s CDN:

Here’s how the incident unfolded:

A database permissions change in ClickHouse kicked things off. Before the permissions changed, all queries to fetch feature metadata (to be used by the Bot Management module) would have only been run on distributed tables in Clickhouse, in a database called “default” which contains 60 features.

Until now, these queries were running using a shared system account. Cloudflare’s engineering team wanted to improve system security and reliability, and move from this shared system account to individual user accounts. User accounts already had access to another database called “r0”, so the team made the database permission change for access to r0 to be implicit instead of explicit.

As a side effect of this, the same query collecting the features to be passed to Bot Management started to fetch from the r0 database, and return many more features than expected:

The Bot Management module does not allow loading of more than 200 features. This limit was well above the production usage of 60, and was put in place for performance reasons: the Bot Management module pre-allocates memory for up to 200 features, and it will not operate with more than this number.

A system panic hit machines served with the incorrect feature file. Cloudflare was nice enough to share the exact code that caused this panic, which was this unwrap() function:

What likely happened:

• The append_with_names() function likely checked for a limit of 200 features
• If it saw more than 200 features, it likely returned an error
• … and when writing the code, it was not expected that append_with_names() would return an error…
• … and so .unwrap() panicked and crashed the system!

Edge nodes started to crash, one by one, seemingly randomly. The feature file was being generated every 5 minutes, and gradually rolled out to Edge nodes. So, initially, it was only a few nodes that crashed, and then over time, more became non-responsive. At one point, both good and bad configuration files were being distributed, making failed nodes that received the good configuration file start working – for a while!

Why so long to find the root cause?

It took Cloudflare engineers unusually long – 2.5 hours! – to figure all this out, and that an incorrect configuration file propagating to Edge servers was to blame for their proxy going down. Turns out, an unrelated failure made the Cloudflare team suspect that they were under a coordinated botnet attack, as when a few of the Edge nodes started to go offline, the company’s status page did, too:

The team tried to gather details about the attack, but there was no attack, meaning they wasted time looking in the wrong place. In reality, the status page going down was a coincidence and unrelated to the outage. But it’s easy to see why their first reaction was to figure out if there was a distributed denial of service (DDoS) attack.

As mentioned, it eventually took 2.5 hours to pinpoint the incorrect configuration files as the source of the outage, and another hour to stop the propagation of new files, and create a new and correct file, which was deployed 3.5 hours after the start of the incident. Cleanup took another 2.5 hours, and at 17:06 UTC, the outage was resolved, ~6 hours after it started.

Cloudflare shared a detailed review of the incident and learnings, which can be read here.

How did the postmortem come so fast?

One thing that keeps being surprising about Cloudflare is how they have a very detailed postmortem up in less than 24 hours after the incident is resolved. Cofounfer and CEO Matthew Prince explained how this was possible:

• Matthew was part of the outage call.
• After the outage was resolved, he wrote a first version of the incident review, at home. Matthew was in Lisbon, in Cloudflare’s European HQ, so this was early evening
• The team circulated a Google Doc with this initial writeup, and questions that needed to be reviewed
• In a few hours, all questions were answered
• Matthew: “None of us were happy [about the incident] — we were embarrassed by what had happened — but we declared it [the postmortem] true and accurate.
• Sent the draft over to the SF team, who did one more sweep, the posted it

Talk about moving with the speed of a startup, despite being a publicly traded company!

Learnings

There is much to learn from this incident, such as:

Be explicit about logging errors when you raise them! Cloudflare could probably have identified the root cause of this error much faster if the line of code that returned an error, also logged the error, and if Cloudflare had alerts set up when certain errors spiked on its nodes. It could have surely shaved an hour or two off the time it took to mitigate.

Of course, logging errors before throwing them is extra work, but when done with monitoring or log analysis, it can help find the source of errors much faster.

Global database changes are always risky. You never know what part of the system you might hit. The incident started with a seemingly innocuous database permissions change that impacted a wide range of queries. Unfortunately, there is no good way to test the impact of such changes (if you know one, please leave a comment below!)

Cloudflare was making the right kind of change by removing global systems accounts; it’s a good direction to go in for security and reliability. It was extremely hard to predict the change would end up taking down a part of their system – and the web.

Two things going wrong at the same time can really throw an engineering team. If Cloudflare’s status page did not go offline, the engineering team would have surely pinpointed the problem much faster than they did. But in the heat of the moment, it’s easy to assume that two small outages are connected, until there’s evidence that they’re not. Cloudflare is a service that’s continuously under attack, so the engineering team can’t be blamed for assuming it might be more of the same.

CDNs are the backbone of the internet, and this outage doesn’t change that. The outage hit lots of large businesses, resulting in lost revenue for many. But could affected companies have prepared better for Cloudflare going down?

The problem is that this is hard: using a CDN means taking on a hard dependency in order to reduce traffic on your own servers (the origin servers), while serving internet users faster and more cheaply:

When using a CDN, you propagate addresses that point to that CDN server’s IP or domain. When the CDN goes down, you could start to redirect traffic to your own origin servers (and deal with the traffic spike), or utilize a backup CDN, if you prepared for this eventuality.

Both these are expensive to pull off:

• Redirecting to the origin servers likely means needing to suddenly scale up backend infrastructure
• Having a backup CDN means there must be a contract and payment for a CDN partner which will most likely sit idle. As and when it is needed, you must switch over and warm up their cache: it’s a lot of effort and money to do this!

A case study in the trickiness of dealing with a CDN going offline is the story of Downdetector, including inside details on why Downdetector went down during Cloudflare’s latest outage, and what they learned from it.

This was one out of the five topics covered in this week’s The Pulse. The full edition additionally covers:

1. Downdetector & the real cost of no upstream dependencies. During the Cloudflare outage, Downdetector was also unavailable. I got details from the team about why they have a hard dependency on Cloudflare, and why that won’t change anytime soon.
2. Antigravity: Google’s new AI IDE – that its devs cannot use. Google wants to become a serious player in AI coding tools, but Antigravity contains remnants of Windsurf. Interestingly, devs at Google aren’t allowed to use Antigravity for work
3. Industry pulse. Gemini 3 launch, Anthropic valued at $350B, Jeff Bezos funds an AI company, and unusually slow headcount growth at startups persists.
4. Five AI fakers caught in 1 month by crypto startup. Candidates who fake their backgrounds and change their looks in remote interviews continue to plague companies hiring full-remote – especially crypto startups.

Read the full The Pulse

In the end, this process took several times longer; 4 years, in fact! Happily, it was worth it: readers’ feedback about The Software Engineer’s Guidebook has been overwhelmingly positive, and on launch, the book became a #1 bestseller among all titles in two Amazon markets (the Netherlands and Poland), as well as a top 100-selling book in most Amazon markets. In 24 months it sold around 40,000 copies, and was translated into German, Korean, Mongolian and Traditional Chines – with the Japanese and simplified Chinese versions releasing later this month.

A lot of people ask why I chose to self publish, and it would be nice to say this was always the goal, but it wasn’t! Originally, I wanted to work with a top tech publisher, who would get the book to market fast, and give it a higher profile. This didn’t happen, but during the process I learned a lot about how publishing works, how to pitch a book, and how to choose which publishing route might be the right one. 

This article shares my learnings from writing and publishing a book which has done pretty well with readers, and it includes the experience working with an established publishing house:

1. Tech book publishing landscape
2. Financials of publishing
3. Publishing process and the publisher’s role
4. My book pitch
5. Working with a publisher
6. Breaking up with a publisher

1. Tech book publishing landscape

Today, there are reputable book publishers whose titles are good and authoritative, and there are other publishers whom this doesn’t apply to. Each publisher also has a subject area: some are mainstream and publish titles about every software engineering area from languages to engineering management. Meanwhile, others stick to a topic of expertise they focus on.

Here’s my mental model of the book publishing industry in 2025:

Highly reputable mainstream publishers

In tech book publishing, three publishing houses really stand out, in my opinion, and form a ‘big three’ among all players in this sector: 

• O’Reilly: if I had to pick a #1 tech book publisher, it would be O’Reilly. They publish some of the most referenced books – like Designing Data Intensive Applications by Martin Kleppmann, Tidy First by Kent Beck, The Staff Engineer’s Path by Tanya Reilly, and more. The book covers are distinctive, using images of animals.
• Manning: a broad range of titles on both specific and general tech topics, which employ historical figures on the covers.
• The Pragmatic Bookshelf: also referred to as the “Prags.” Founded by Andy Hunt and Dave Thomas, the authors of what might be the best-selling tech book ever; The Pragmatic Programmer. Since its founding, The Prags has refused digital rights management (DRM) on their ebooks.

High reputable “mainstream” publishers that are tough to pitch to

The publishers in this section have strong reputations, like those above. However, they are harder to pitch to, usually because they publish fewer tech books. I couldn’t find an author pitch template, or clear pitching instructions, and contributes to a sense of “don’t find us, we’ll find you” among the following publishing houses: 

• Addison-Wesley: one of the best-known brands in tech. It has been an imprint (a trade name within a publication) of Pearson since 1988, and is the publisher of many “classic” book titles like Clean Code by Robert C. Martin, The Pragmatic Programmer by Andy Hunt and Dave Thomas, and some recent ones like Modern Software Engineering by Dave Farley. I couldn’t find any way to pitch to this publisher, and new books they publish seem to be by established authors.
• Pearson: This business owns the Addison-Wesley imprint. Recently, it started to publish tech books as “Pearson” instead, author Martin Fowler shared.
• Wiley: formerly a well-known tech book publisher behind the “X for Dummies” series. It publishes lots of computer science textbooks, but I can’t find recently-published, well-known tech books for software engineers.
• Springer: another massive publisher for whom tech books are a small part of the business. I couldn’t find how to pitch tech books to them.
• Morgan Kaufmann: a well-known tech books publisher founded in 1984, and acquired in 2001 by Elsevier. As I understand, these days it prints far fewer technology book, and focuses on academic topics. No clear way to pitch to them.

Highly reputable “niche” publishers

The following publishers are standout in quality, covering fewer topics than those above.

• No Starch Press: “The finest in geek entertainment” is the tagline, featuring fun visuals, and high-quality content on specific technologies like machine learning, Python, JavaScript, etc.
• IT Revolution: titles for technology leaders: DevOps, technology delivery, workplace culture, and similar. Publisher of The Phoenix Project, Team Topologies, and Accelerate.
• Artima: focuses on Scala.
• CRC Press: publishes on technology, engineering, math, and medicine.
• Stripe Press: “works about technological, economic, and scientific advancement.”
• MIT Press: “a distinctive collection of influential books curated for scholars and libraries worldwide.”

Other mainstream book publishers

Apress is a reputable publisher with a lower profile, which publishes on a wide range of topics, from specific technologies and frameworks, to more generic topics on computing. Because they publish many books on many topics, they are usually open to pitches.

Packt. A tech book publisher with a focus on quantity over quality, it feels to me. There is limited support and feedback for authors, and titles could often use more editing. But also, Packt is likely to say “yes” to a serious proposal.

2. Financials of publishing

Financial matters really come into play when your proposal is accepted by a publisher and you receive a contract offer.

Advance: $2,000 – $5,000. An advance payment to the writer is a tried and tested way to make them deliver a completed manuscript. It’s often paid in chunks: 50% when a milestone is hit, and 50% when a full draft appears.

The “big three” publishers typically offer $5,000, usually as a flat, non-negotiable rate; at least, it’s what I was offered. Smaller publishers offer closer to $2,000 for more niche books. The advance is non-refundable; even if your book sells zero copies, you keep it. The publisher is making an investment in you, and taking a risk.

As an aside: if you are thinking of writing a book: for guest authors in The Pragmatic Engineer Newsletter guest authors I offer a $4,000 per article payment – and you can later publish your guest article in a book. Several authors working on their book have written a guest articles such as Lou Franco on Paying down tech debt or Apurva Chitnis on Thriving as a founding engineer. Writing a guest post can help refine ideas, broaden your reach, and prove helpful when publishing the article.

Paperback royalty: 7-15% 

Royalties are earned on book sales, and taken from the net price of the book. Net price is what a publisher gets after the retailer (e.g. Amazon, or a bookshop) takes their cut. Let’s see how it works for a $40 book:

It matters financially where your title is purchased; be it an online shop, physical book store, or purchased directly from the publisher. Many tech books are sold on Amazon and online stores. Amazon’s 40% cut seems high, but it’s actually the lowest among book retailers. Up to 60% is a common cut for a physical bookshop.

Most publishers offer 10-12.5% royalties, is my understanding, and Packt around 15-20%. Keep in mind that brand reputation plays a role; for example, Packt’s reputation is less elevated than Manning, which can make a difference to sales.

Ebook royalties: 10-25%

For ebooks, several publishers pay 25% royalties, but not all. But even with a higher royalty rate, an author might end up making less per sale. For example, on the Kindle platform, the cut for Amazon is high at 65%. Let’s look at a $30 ebook with a 20% royalty rate:

Ebooks are almost always priced lower than physical books, and when sold on Kindle, generate much less revenue for the author, while earning more per copy than the paperback version. I was offered 10% royalties on ebook sales, which is at the low end.

“Earning out” 

When an author needs to pay back an advance before being paid anything, this is called “earning out”. If you get a $5,000 advance for a title costing $40 per hard copy and $25 for the ebook version, and most sales happen on Amazon, it means:

• ~2,080 paperback sales on Amazon
• Or ~2,850 Kindle book sales
• Or ~1,250 paperback sales on the publisher website

The author needs to sell at least 1,000 copies across various platforms to “earn out.” The good news is that a publisher sends quarterly or annual royalty payments if a book keeps generating revenue, which would effectively be passive income.

The Prags’ unique approach

One publisher that calculates rates differently is The Pragmatic Bookshelf. Instead of offering a low-digit number on revenue, they offer a 50% split on profit.

50% on profit sounds much higher than 10% on revenue, right? However, the devil is in the details, because paying on profit means that the upfront publisher costs – editors, cover design, printing, distribution, marketing – all are deducted before any profit split.

Authors who have used this approach tell me the numbers end up pretty similar to the revenue model.

Real-world case studies with actual earnings

Designing Data Intensive Applications author, Martin Kleppmann, shared the cumulative royalties he made in 6 years. The breakdown is interesting; ebook and Safari Online sales generated more revenue for the writer than the print version.

Cloud Native Infrastructure earnings: author Justin Garrison published with O’Reilly, and was offered 10% for print and 25% for ebooks (split into half, thanks to working with a coauthor). His book sold 1,337 copies in 4 months; and made about $22,000 for the two authors (and around $11,000 for Justin.) Justin concluded:

“Going into this project I had a rough estimate in my head to make about $2000–3000 so this is much better than I expected. Set your expectations accordingly.”

Don’t forget that publishers are also in this to make a positive return. This means that it is unlikely for a highly reputable publisher to invest into a book that they do not believe would sell at least a few thousand copies. I don’t have the data here: but if I was a publisher, I would reject any book that didn’t look like it could hit 1,000 copies sold in the first year of publishing.

3. The publishing process, and publisher roles

Why does a publisher take so much of the revenue? Part of this is because they do a lot of the work around publishing, and need to hire (and pay!) people for those roles. Here is my understanding of how the publishing process works, based on four months of pitching to publishers; two months of working with one of them; and researching how the rest of the process works:

Here are people I worked with, and my experience with them:

The acquisitions editor. If you write a technical blog, you might get a reachout from someone called an acquisitions editor, who will ask if you would consider publishing a book. Also, when you submit a pitch to a publisher, chances are that you will first communicate with an acquisitions editor.

A publisher’s goal is to publish books that will be profitable for them. They find authors who could write these books two ways:

• Inbound pitches coming from authors – reviewed by editors or acquisitions editors
• External reachouts done by acquisitions editors

These people need to have a good understanding of what kinds of books sell well at the publisher (and why); what their current catalogue is; what the gaps are; and what competitor publishers are commissioning.

When I pitched my book to 3 respected publishers, in two cases I talked with (and worked with) the acquisitions editor to improve my pitch. The acquisitions editors were my “champions” at the publisher. Their goal was to get a pitch that the company would say yes to.

The development editor works on the structure of the book. They ask the author to come up with a detailed table of contents – in my case, they asked me to estimate even the length of the chapters. They also help develop – and maintain – the narrative of the book.

Had I not worked with a publisher, I would have had no appreciation of this “high-level editing” – which, turns out, is key for writing a well-structured tech book!

The project manager checks in with timelines, organizes reviews—like editorial reviews—and helps keep you accountable. One of the best things about working with a publisher is that you are on a tight deadline—without which it would take you several times longer to publish the book!

The publisher owns a lot of rights for your book! One thing that I realized only after signing with a publisher is that while publishers help a lot with writing the book – and taking a higher cut is sensible because of this – they also hold on to a lot of rights that impact your book! These are all things that you give up on, versus when self-publishing. These are:

• Global publishing rights. Although you are the author of the book – and usually hold the copyright to it – the publisher own wordlwide publishing rights. This means that they are the only ones who can publish the book, or longer excerpts of it. In practice, this means you need to get permission if you’d like to publish some parts of your book on e.g. your blog, or social media. They’ll usually grant this as it’s good marketing – but it’s still that you need to ask, as the author.
• Foreign rights. The publisher will own the publishing right, and will usually be the one who owns selling foreign rights. In theory, this could sound like you are losing out on things. In pratice, publishers are much better positioned to sell and administer these rights. Most publishers offer a 50% cut on these rights – it’s what my publisher offered. Also, the majority of tech books are not translated to other languages – a book that “only” sells 2,000 copies in English is unlikely to sell a significant number in a non-English market!
• The cover. The publisher decides what cover they will design, though they tend to check the author for feedback.
• The title. One of the surprises for me was how the publisher ultimately decides on the title and subtitle.

In short: this book is owned by the publisher. You are the author, but they are the only ones who can distribute it. In practice, many authors would prefer to have it this way – because all the work related to distributing the book is taken on by the publisher. However, it’s good to know that you need to give up all the above when working with a publisher.

4. My book pitch

My secret hope, back in 2019, was to get a contract with one of the “Big 3” tech book publishers: O’Reilly, Manning or The Prags. I pitched my book to all three: got a “no” from two, but a “yes” (and a contract) from one. Here’s how I went about my pitch.

Write a “one-pager” about your book

What will this book be about? Who is it for? What will readers take away when reading it? Answer these in a short pitch, before even seeking out publishers. Here’s what I put together as my “one-pager:”

Do some market research

What are similar books in the market that would be competing with this book, directly or indirectly? How is this book different from them?

What is the demographic of people who would be interested in buying this book? Can you estimate how large this crowd is? Realistically, what percentage of this group could be interested in buying the book – assuming they know about it? Don’t forget that publishers will invest into books that can generate decent sales: it’s good to do a little research to help confirm your title could be one of these!

Shortlist publishers you would be interested working with

While there are quite a few publishers out there: what are your top preferences? And what are ones you’re willing to consider, even if your “top” choices turn you away?

Self-publishing is always an option (I’ll cover more on how I went about this in later parts). However, going with a good publisher can significantly speed up your book production, while also improving the quality.

Write a draft table of contents and a draft chapter

Some publishers will want to look at what a draft chapter will look like – but not all of them. Still, I found it helpful to do writing before submitting to a publisher. If for no other reason, this was to confirm that I’d enjoy longform writing!

I spent about a week putting together a table of content, and around four months writing drafts of chapters. These chapters turned out to be helpful later on.

Submit a tailored pitch your the publisher(s)

Once you identified your top publisher choices, submit a pitch. Most book publishers have a pitch document they want you to follow. Here are common ones:

O’Reilly’s pitch template:

• Description
• About the topic
• Audience
• Keywords
• Competing titles
• Related O’Reilly titles
• Book outline
• Writing schedule

Manning’s pitch template:

• About the author
• About the book topic
• The book plan
• Q&A
• Reader overview
• Book competition
• Book length and illustrations
• Writing schedule
• Table of contents

The Pragmatic Bookshelf template:

• Overview
• Outline
• Bio
• Competing books
• PragProg books
• Market size
• Promotional ideas
• Writing samples

Most of these templates ask for similar content, so if you completed one pitch: the others are much easier. Here are some tips I’d have for building a pitch.

Put yourself in the shoes of the publisher. This book is a huge deal to you: but it’s just one of the dozens that the publisher will publish just this year. You want to write an amazing book: but the publisher wants to publish one that will sell.

And these are major differences! The publisher will care very much about competition for the book, and how their existing titles relate to them. Like a VC firm, a publisher will not want to fund two investments competing on the exact same market: so if the publisher recently published a book that is a deepdive on Go; they will almost certainly pass on the next one, no matter how good your pitch is.

Pitching to several publishers parallel is totally fine and you should do it! This is one thing I wish I’d done differently. In my mind, I was 100% certain that my first publisher-of-choice would jump on the opportunity to publish this book. I thus felt that it would be “unfair” if I pitched to other publishers, without hearing back.

In hindsight, as a first-time author, this strategy was a waste of time on my end. Most publishers are unlikely to take a risk on a first-time author with no books published in the past – like I was in 2019. And so the likely outcome is rejection in most cases.

In my case, I spent about two and a half months waiting on the response from this first publisher. My acquisitions editor was championing the book – making the case for the publisher to offer a contract – but in the end, the publisher chose another book with a similar topic that was in their pipeline. This made perfect business sense for them – but for me, I was spent waiting for months, instead of pitching the book to other publishers!

My book pitch ended up being a helpful resource on my self-publishing journey. Even though I did not release with a publisher: pitching to publishers helped the book become an eventual success. It was for these reasons:

• Defining the structure. I had my table of contents well thought-out by the time I submitted the pitch. This structure changed later, but it was a solid start.
• Positioning the book. I had a good idea of the “competitive” landscape, and what books my title would “go up against.” It also helped me focus on how my book is different to what is already out there.
• Forcing me to think about marketing. The Pragmatic Bookshelf asked for a section on promotional ideas. This forced me to think about where (and how) I would promote the book – even before getting into the thick of writing. When going with a publisher, it’s safe to assume that the publisher’s brand will do some marketing. However, authors will still do the lion’s share of marketing – and it’s good to think about this ahead of time.

5. Working with a publisher

I got lucky with one of the three publishers, in the end. This publisher was looking for a book just like mine, right at that time! What happened was one of their best sellers had to be pulled from publication, for reasons outside the control of the publisher. Apparently, when my pitch arrived, they had just started a search for a book that could plug the hole – and they saw my book being a perfect fit for a “software career advice” book.

At the time, this felt like great luck. In hindsight, my relationship with the publisher might have soured exactly because they were looking for me to write a specific kind of book that would be similar enough to this old book – but I had no intention of doing so. More on how things went sour in the section after this one.

From signing the contract, I worked with a publisher for about a month – so I’m not exactly the most experienced in this front. However, a couple of things stood out as strong positives – and things that I “lost” when deciding to self publish, in the end.

Strong pressure to write – thanks to the contract. My contract had pretty strict deadlines included. We signed it on 11 January 2020, and these deadlines were part of the contract:

“The Author shall prepare and deliver to the Publisher a machine-readable electronic copy of the manuscript for the Work, including all its illustrations, code listings, and exercises, as mutually agreed upon by the Publisher and the Author as follows:

- Not later than March 15, 2020, a partial manuscript for the Work totaling not less than one third of the planned finished Work.

- Not later than June 1, 2020, a partial manuscript for the Work totaling not less than two thirds of the planned finished Work.

- Not later than August 15, 2020, a draft of the complete manuscript for the Work suitable for review.

- Not later than September 1, 2020, the final, revised and complete manuscript for the Work acceptable to the Publisher for publication.”

Talk about pressure! Also, my first payout was tied to reaching the first milestone – which was delivering at least a third of the finished work. My publisher also set up regular check-ins to help me stay accountable. And this kind of pressure was good – because without it, I would have pushed back writing, or got stuck on relatively trivial parts! 

6. Breaking up with the publisher

While I greatly appreciated that a publisher took a chance on me, lots of things felt wrong from the start. A month into working together, I felt that things were getting worse, and not better.

The small things that I dismissed, in the beginning:

• A (very) opinionated structure. This publisher had strongly opinionated templates I was told to use for all chapters. They included each chapter to start by stating what the reader will learn; and then summarize this at the end of the chapter. It wasn’t how I imagined my book to be – but it didn’t seem I had a choice. I figured, I’ll give it a go. The publisher knows better after all, as they’ve done this hundreds of times. Right?
• Needing to ask for permission to share drafts on social media. I originally planned to share screenshots of some of the parts I am writing to get feedback as I go – and to also increase visibility of the book. I thought that this is a no-brainer. Not only does this kind of “early sharing” makes the book better: but it will also make more people excited about the book, leading to more eventual customers. To my surprise, my contact at the publisher said I will need to ask for permission whether I can do this. Permission? For something that will market the book? Yes: because the publisher owns all publishing rights, including for the draft!
• I won’t decide on what the title will be. I had strong opinions about what I’d like the book’s title to be. My publishing contact also had ideas on what they thought would be good to add to it – like introducing the “mentoring” term either to the title or the subtitle: which was an idea I disliked. As I talked with them, it became clear that the publisher will set the final title: not me. Hmm – odd, no? It’s another reminder that, although it’s my book: it’s really the publisher’s book, and they have the final say on all important decisions.
• Nudges to “dumb down” the book. My editor was giving more suggestions on how to edit the content to make it more “beginner-friendly” and suggested I introduce e.g. “Alice and Bob” examples to make it easier to digest the contents. One of the recently best-selling books of the publisher heavily used Alice and Bob, and it seems the publisher thought it helped their sales.

The first major editorial review was where I decided we should part ways with the publisher. About a month-and-a-half in, the publisher pulled together several experienced editors, and offered suggestions on how I could improve the book. The suggestions were these:

• Focus on reader engagement. Tell stories and develop them with emotion, mystery, aha moments, and unexpected conclusions. Tell the stories from the "we" or "they" perspective -- make stories team-oriented.
• Exercises. Develop exercises for use within the chapters (not just end) or a story about what happened when one person did the exercise.
• Mini-projects. Guide readers to discover and come to conclusions on their own (see Donald Saari story in What the Best College Teachers Do). Mini project topics: testing, architectures.
• Word of the day feature. Example: Dependency injection (what is it)? Scatter these across the book.
• Quotes. Include quotes from luminaries such as [Well-known-person 1] and [Well-known-person 2] that relate to advice given. Ask other [Publisher] authors to relate experience about how they followed similar advice and were successful.
• Tech map. Create a diagram of the current technology landscape. Example big-picture topics: architecture demystified, distributed systems demystified.

While I appreciated the suggestions: I hated all of them. I saw what implementing them would do: they would turn this book – which I already had reservations with the “forced” style on me – to something I would not want to read. Much less write!

I envisioned writing a more matter-of-the-fact book that doesn’t have exercises, “mini projects” or “word of the day” gimmicks.

I sat down to reflect why I chose to work with a publisher, to start with. As an author, I’m giving up a lot of things: editorial control, the bulk of revenue, all publishing rights… and for what? For the publisher to make the process easier, and for the end result book to be better than if I was working alone.

But I felt that this book would be far worse if I continued with my publisher: and the only way to get it back to what I envisioned was if I spent a lot of time and energy pushing back on them.

It would cost me less energy to self-publish. So I decided to terminate my agreement because it didn’t feel my publisher was helping write the book that I wanted to write.

My publisher was understanding and professional in terminating the contract. I explained to them that all the feedback suggested they wanted to see a very different book to what I wanted to write. And that, frankly, I am not the author to write that kind of book.

Truth be told, I was embarrassed that I had wasted their resources – working with their development editor and the editing team – for these two months. At the same time, I was vocal in voicing to my editor that I was hesitant about this mandated style. I also made the decision that there is no point in continuing at the first formal feedback session. I’m not sure I could have come to this conclusion any further, as I was still learning how this book publisher worked, up until that point.

To show how professional this team was, this is the termination letter they sent as a signed PDF:

“This letter is in reference to our Publishing Agreement with you for [what would become The Software Engineer’s Guidebook] dated January 11, 2020. By mutual agreement, we are terminating the publishing contract.

Since no advance was paid to you under the terms of this contract, all rights in the content you originally submitted will hereby return to you and we will consider this matter concluded.

The decision to cancel a project is never an easy one to make. We thank you for all the efforts on this project that you made and wish you the best in your future endeavors.”

At this point, I learned enough about publishers and myself to decide: I’m doing it by myself. Having my book accepted by a major publisher gave external validation that there’s a strong business case for The Software Engineer’s Guidebook. And working with an opinionated publisher – and continuously pushing back on styling suggestions made me realize that I already have my own opinonated style that I like using.

I did lose a very important thing by deciding to self-publish: the accountability of meeting a publishing deadline. Working with the publisher, this book would have been out fall 2020 or spring 2021. Self-publishing, I launched it November 2023.

One of the reasons for publishing my book two years later than it would have taken with a publisher was because I now knew I could no longer rely on a well-known publisher to lend my book their brand. For my book to have an even slim chance of being successful: I would have to compensate for the lack of being associated with a publisher, and fill the gap in marketing and awareness, leading up to the book launch.

Not having a publisher was a reason I started writing The Pragmatic Engineer Newsletter in August 2021 (a year-and-a-half after breaking up with this publisher) – and the sudden success of this newsletter gave me less time to wrap up the book. At the same time, by the time the book was ready, there were plenty of people who looked forward to reading it: and many of them were already readers of the newsletter!

I’ll cover more about how I went about the actual self-publishing process in a follow-up article, how the book ended up selling, and other learnings. Subscribe to The Pragmatic Engineer to get notified when it is out.

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Online retail giant Amazon unexpectedly announced 14,000 job cuts earlier last week. The massive round of layoffs at the company follows other mass redundancies in recent years:

• January 2023: 18,000 people laid off.
• March 2023: another 9,000 people let go
• November 2023: hundreds of people let go inside the Alexa team, as Amazon was looking to shift Alexa more toward GenAI
• April 2024: hundreds of people let go from AWS

Software engineers, unfortunately, seem hit hard by the latest layoffs: of 2,300 employees laid off in Washington State, 25% are software engineers, GeekWire reports. We can only speculate about the ratio across the rest of the company, but if cuts at HQ are heavy on engineering, then things don’t look promising for other locations, sadly.

The memo from Beth Galetti, Senior Vice President of People Experience and Technology, to workers didn’t explain much:

“Some may ask why we’re reducing roles when the company is performing well. Across our businesses, we’re delivering great customer experiences every day, innovating at a rapid rate, and producing strong business results. What we need to remember is that the world is changing quickly. This generation of AI is the most transformative technology we’ve seen since the Internet, and it’s enabling companies to innovate much faster than ever before (in existing market segments and altogether new ones). We’re convinced that we need to be organized more leanly, with fewer layers and more ownership, to move as quickly as possible for our customers and business.”

The statement is utterly confusing, as encapsulated by its message that “business is great, but we need to do layoffs”. Job cuts usually mean a business is in trouble, which obviously isn’t the case for Amazon. So, why are these layoffs really happening?

Layoffs to boost efficiency?

The company’s memo states:

“We’re convinced that we need to be organized more leanly, with fewer layers and more ownership, to move as quickly as possible for our customers and business.”

If this line of reasoning sounds familiar, it’s because most of the layoffs in 2023 were justified the same way. The tech industry overhired during the pandemic in 2020-2021, making orgs more bloated and decision-making slower. In February 2023, I reported on the trend of fewer middle managers, with Meta the first Big Tech giant to reduce its management layers. In 2023, most of Big Tech followed this approach with layoffs or reorgs. Managers acquired more reports, and tech companies cut down the number of layers between the CEO and individual contributors.

Given Amazon did other massive layoffs in 2023, it’s unlikely they missed the industrywide trend for fewer managers. While the current layoffs seem to be targeting managers quite a bit – from the Washington State layoffs, 20% of those let go are managers – there are still more ICs laid off than managers, overall. So, this official explanation doesn’t pass my personal “smell test”.

Layoffs to buy more GPUs?

The day after its jobs announcement, Amazon had more big news, this time about AI: it unveiled Project Rainer, the largest AI computing platform AWS has ever built. It already has 500,000 Trainium2 chips (built by Amazon), This capacity is already 70% larger than any AI computing platform in AWS’s history, and Anthropic is using all of it (!!) to train its next models. Below is an image of one of the several Project Rainer data centers packed with Amazon GPUs:

Building data centers is incredibly capital-intensive: Amazon has spent $11B on Project Rainer alone. Even though very profitable, Amazon might want to invest more cash than it currently has into building data centers. So, one reason for job cuts could be to reallocate financial resources from paying salaries and compensation towards building more data centers.

Before doing the math, a couple of concepts are important to understand:

• Free cash flow (FCF): profit after payment for things like capital expenditure (CapEx), such as financing data centers. If a company wants to operate with as little debt as possible, FCF is usually very important. If Amazon wants to avoid loans and not touch its reserves, then data center investment would come from FCF, reducing FCF further.
• Cash reserves: a company’s liquid reserve investments, usually an accumulation of investments in financial instruments like bonds and securities, or cash deposits.

Let’s run Amazon’s numbers:

• Cash reserves: $93B. This is how much Amazon has in reserve.
• FCF: $32B. This is the rough free cash flow Amazon has currently, as per its latest quarterly report. This is after deducting current data center investments.
• Savings from layoffs: $2-4B. This is my estimate of the rough total compensation of 14,000 employees.

So, the savings from these layoffs wouldn’t even pay for half of Project Rainer ($11B in total), and Amazon could easily build 3x Project Rainers in the next year, without needing to dip into its savings! Of course, Amazon has its famous frugality principle, but this massive layoff of 14,000 people won’t make a big difference to how much it can invest in data centers; It can already spend much more, if it wants!

Leanness and AI fail job cuts “smell test”

It’s not only me who doesn’t buy the explanation that these layoffs are to streamline the company, or to redirect resources to AI. Arne Knudson worked at Amazon for nearly two decades, most recently as a software development manager (SDM), before leaving the company earlier this year. He shared his analysis, with some insider detail:

“In my 18 years at Amazon, I went through a few layoffs and hiring freezes.

This is the first time I’ve seen multiple years of significant layoffs essentially back-to-back. Even in the depths after the .com bubble, it wasn’t this bad. They’ve been laying people off now for almost 3 straight years.

The explanation that this is downsizing after hiring too many at the height of the pandemic doesn’t pass the smell test, at least to me. That was 3 years ago; they’re not that dumb to keep those people around for 3 extra years. Those folks were laid off back in ‘22.

I’m also not convinced that this is optimization due to AI. My degree’s in AI, and I worked on AI stuff at Amazon; I don’t think there’s enough automation yet, and it’s not accurate enough yet, to replace 30,000 people. The cost of inaccuracies seems too high. But I could be wrong; maybe they’ve gotten their false negative & false positive rates low enough to avoid too many region-wide AWS outages. (Or not.)

One of the articles I read said this was going to be in HR, and I can tell you as a former manager, my experience working with HR had been steadily worsening over the past 5-7 years. They outsourced so much of the work, overworked the people they had, and had such high turnover that I never knew who I was supposed to work with. When I needed to put someone on a performance plan or help a new hire receive some kind of accommodation, it seemed like it was a different person each time. If they really are laying off tens of thousands more HR folks, this is only going to get worse.

And, I suspect, it means they don’t plan on hiring MORE people in any of the business units for a year or more. So, by the smell-o-meter, this seems more significant than streamlining the workforce, improved AI, and “nah, we don’t need as many HR folks.”

US economy to blame for Amazon layoffs?

It’s safe to assume AWS as a business unit is doing just fine, as suggested by Project Rainer’s existence and the agenda for building data centers. But how is the e-commerce side of the business performing, and what’s its outlook?

If one business should have its finger on the pulse of the US economy, it’s Amazon with its size and self-professed, relentless customer focus, providing a window into people’s spending habits across the country. Flashing lights on the dashboard of the national economy may signal tough times ahead in e-commerce, which could be a reason to start cutting costs early.

There are concerning signs from other sectors about the US economy. Below is the CEO of the restaurant chain, Chipotle.

“Earlier this year, as consumer sentiment declined sharply, we saw a broad-based pullback in frequency across all income cohorts.

Since then, the gap has widened, with low to middle-income guests further reducing frequency. We believe that this guest with household income below $100,000, represents about 40% of our total sales. And, based on our data, they are dining out less often due to concerns about the economy, and inflation.

A particularly challenged cohort is the 25- to 35-year-old age group. We believe that this trend is not unique to Chipotle and is occurring across all restaurants as well as many discretionary categories. This group is facing several headwinds, including unemployment, increased student loan repayment and slower real wage growth. We tend to skew younger and slightly over-indexed to this group relative to the broader restaurant industry”.

Chipotle is saying that everyone is eating out less, particularly 25-35 year olds, because of inflation. If people spend less on Chipotle because of rising prices, then they may also spend less in other areas of their lives for the same reason, including on Amazon.

In the e-commerce supply chain, there’s evidence of this trend, which would mean delivery services like UPS have fewer parcels to deliver. Speaking of UPS, two days ago, it announced massive layoffs:

“United Parcel Service (UPS) has slashed 48,000 jobs this year — one of the largest single-year reductions by a US company since the pandemic — as the package giant scrambled to contain costs and revive its lagging stock price.

The Atlanta-based delivery behemoth disclosed the reductions Tuesday while reporting third-quarter earnings that beat Wall Street expectations.

UPS said 34,000 of the cuts hit drivers and warehouse operations, while 14,000 targeted management (...)

UPS shares jumped nearly 9% in Tuesday afternoon trading, even as the company reported weaker revenue and profits”.

UPS’s revenue is down on last year, which suggests that there are, indeed, fewer deliveries (or lower value ones). As with the latest job cuts at Amazon, these drastic layoffs could be explained by a lot of things, most easily by UPS expecting reduced trade in the future.

If US consumer spending is trending down, then the e-commerce sector will be among the first to feel this. It could explain why Amazon is making these layoffs now. It can also explain why Google, Meta, and Microsoft might not be seeing their businesses impacted: they’re not involved in retail like Amazon is, and the AI sector is very much booming.

Among all of Big Tech, Amazon is best positioned to detect changes in US consumer spending. Google’s and Meta’s revenue is more dependent on advertising, and Microsoft’s more on enterprise spend. Like Amazon, Apple is well placed to feel market changes with its range of smartphones and watches, and other consumer tech.

I believe Amazon is highly commercially rational, so it’s worth understanding the actual reason for its second major mass layoffs in just two years, following deep cuts in 2023. I’d put my money on this reason being the economy, and how Amazon probably expects customers to cut back their spending everywhere, including on Amazon.

This was one out of the four topics covered in last week’s The Pulse. Read the full article.

This week’s The Pulse additionally covers:

1. Cursor and GitHub double down on agents. Each company is focusing on agents: Cursor with its multi-agent mode, and GitHub with its “Agent HQ.” Cursor is increasingly a direct rival to GitHub.
2. Industry pulse. Meta rolls out AI-assisted interviews, Cursor and Windsurf believed to be using Chinese open source AI models, South Korean government pays price of ignoring backup “101,” startups growing much faster in the US than in Europe, companies using AI tools buy more JIRA seats, a neat uptime service called Updog, and more.
3. OpenAI inflating the bubble? OpenAI signs another massive deal with AWS based on predicted growth, and seeks taxpayer protection to borrow more.
4. Large tech companies struggle to build their AI integrations. Apple admits failure to modernize Siri by paying Google $1B per year for its LLM. Perplexity to pay Snap $400M to be its AI search interface.
5. How much do Directors of Engineering earn at startups? Data from Carta says it’s more than any other Director: $215-230K at companies valued at $25-250M.

Read the full issue here

Puneet Patwari recently accepted an offer to join Atlassian as a Principal Software Engineer. In three months, he did more than 60 interviews at 11 companies, he told me – while dropping out of 3 more interview processes after accepting the Atlassian offer, including that of Meta. Following that endeavour, he has compared the interview processes of the largest companies:

A few more observations that Puneet shared with me:

Amazon: the Amazon Hiring Manager round was one of the most unique I ever experienced. We got so engrossed in the discussion that it took 160 minutes instead of the scheduled 60 minutes! We had to take a break in between the interview process.

Atlassian: The leadership craft (LC) & values were two interview rounds which were very crucial in determining that I’ll be levelled at the Principal level. Of course, the Systems Design interview was also key here. Atlassian puts a lot of emphasis on LC for Principal engineers.

Salesforce: the system design round was based on the actual job requirement. It was a migration problem where the interviewer wanted to check if I can own a project end-to-end with customers at the centre of it.

Confluent: when I say it was the most mentally demanding interview, what I mean is how every skill was tested with two interviews! So 2x data structures and algorithms (DSA), 2x System Design 2x behavioural interview rounds.

I cannot stress enough how important behavioural interviews are at the Staff+ levels. Doing well on these interviews were decisive in getting Staff and Principal-level offers. Of course, you needed to do well on coding and systems design: but my sense was that the behavioural parts were make or break for levelling and getting an offer.

A few things stand out to me from Puneet’s account of his interviews at leading tech companies:

• Algorithmical coding interviews are everywhere! For senior+ positions, you need to get really good at these, including challenging topics like dynamic programming. We cover how to perform well in these in the article, How experienced engineers get unstuck in coding interviews
• Interviews are tough, and time consuming. Even after Puneet had offers, no company shortened their process. Puneet had to decline 3 more interviews – including one at Meta – because by the time the interviews would have come around, he already had an offer he had accepted at Atlassian.
• In a tough job market, “top” candidates are still in demand. We’ve covered how challenging the current tech labor market is for jobseekers, but Puneet interviewed at 11 companies and got 6 offers. His applications had to have a lot going for them in order to pass the resume screenings: 10+ years of experience, and working as a Senior Software Engineer at Microsoft. He also showed up really well prepared.
• Bad luck can strike at any time. Puneet’s interview experience at Uber seems to have been a bit unlucky: the interviewer presented as rigid and not open to dialogue. Perhaps they were having a tough day, or wanted to get the interview over with. Or it could be what Steve Yegge describes as the interviewer anti-loop

Congrats to Puneet for accepting the Atlassian position, and thanks for sharing all these learnings!

This was one out of five topics covered in The Pulse #149. The full edition additionally covers:

• New trend: programming by kicking off parallel AI agents. More devs are experimenting with kicking off coding agents in parallel
• ACP protocol. A new protocol built by the Zed team, which tries to make it easier to build AI tooling for IDEs than the MCP protocol allows
• AI security tooling works surprisingly well? AI-powered security tools seem good at identifying security flaws in mature open source projects
• Is AI the only engine of US economic growth? Forty percent of US GDP this year is based on AI-related spend, while 60% of venture capital goes into AI. Hopefully, it won’t end up as a bubble which bursts like in 2001

Read the full issue here, and check out today’s The Pulse here.

With agentic command line interfaces like Claude Code, OpenAI Codex, Cursor, and many others going mainstream, I’m seeing a trend of more software engineers experimenting with kicking off work with several agents simultaneously on separate tasks:

I talked with Anthropic engineer Sid Bidasaria about how Claude Code is built, and at the end of our conversation, he mentioned that he’d had a few agents running throughout and that it made him more productive with work. Similarly, software engineer Simon Willison, whom I consider an AI engineering expert, has posted about “embracing the parallel coding agent lifestyle.” He writes:

“For a while now, I’ve been hearing from engineers who run multiple coding agents at once—firing up several Claude Code or OpenAI Codex instances at the same time, sometimes in the same repo, sometimes against multiple checkouts or git worktrees.

I was pretty skeptical about this at first. AI-generated code needs to be reviewed, which means the natural bottleneck on all of this is how fast I can review the results. It’s tough keeping up with just a single LLM given how fast they can churn things out, where’s the benefit from running more than one at a time if it just leaves me further behind?

Despite my misgivings, over the past few weeks I’ve noticed myself quietly starting to embrace the parallel coding agent lifestyle.

I can only focus on reviewing and landing one significant change at a time, but I’m finding an increasing number of tasks that can still be fired off in parallel without adding too much cognitive overhead to my primary work.”

Simon shares advice about what works for him, with research, maintenance tasks, and directed work all mentioned as use cases.

It’s interesting to consider whether parallel work with agents has the potential to overturn decades of software engineering practices. Let’s assume software engineers who kick off multiple agents at once do become more productive than “single-threaded” peers who work on one problem at a time. If so, then this practice has a chance to spread, should enough software engineers seek to be more productive – or want to avoid being left behind by some colleagues doing more than before.

But engineering in the pre-AI era was all about being in the flow for many productive engineers. A flow state goes something like this:

• Understand the moving parts
• Build a solution, validate it, iterate on it
• When satisfied with how it works, submit a pull request for code review — or, if no review is needed, just merge and ship it

Interrupting this process disrupts the flow state, and it takes time to get back into it: it’s why software engineers tend to prioritize focus time, to make progress with coding work.

Of course, this isn’t universal among all highly productive engineers; when I was an engineering manager, the most productive engineers on my team did a lot of context switching and were adept at juggling several things at once. Here’s an average-looking day for a senior engineer acting as a tech lead:

• Code reviews. Arrive at office, go through open code reviews from the previous night
• Coding. Get some of their own coding work done
• Standup. The usual
• More coding. Get the work done. At least, that’s the idea. In reality:
• Interruptions: code reviews, requests for help, taps on shoulder. The most productive engineer on a team regularly gets messages requesting code reviews to unblock teammates, or to help someone else who’s stuck, or the manager (me – sorry!) tapping them on the shoulder for help with something.

I wonder if senior+ engineers will be “naturals” at working with parallel AI agents, based on their existing habits and what they do currently:

• Keep parallel workflows in their heads; e.g., what team members are doing at any one time.
• Code reviews across several workstreams: they’re the go-to code reviewer, and usually review all code changes across 2-5 workstreams. They may not do the work, but know when it’s correct.
• Can handle interruptions: they’ve learned how to make progress when their focus is continually being broken.
• Good at directing colleagues: because they’re regularly interrupted, they’ve also learned how to delegate and explain urgent work to team members.
• Writing skill: these engineers write a lot of code reviews, draw up documents like RFCs that outline work, create tickets to break down projects, and critique colleagues’ efforts; all this involves communicating effectively in writing.

With AI agents, the qualities that make a good tech lead are within reach for engineers who want to be more productive. So far, the only people I’ve heard are using parallel agents successfully are senior+ engineers.

Then again, this workflow hasn’t stuck with everyone: I asked Flask creator Armin Ronacher about his experience with parallel agents. He told me:

“I sometimes kick off parallel agents, but not as much as I used to do.

The thing is: it’s only so much my mind can review!”

But we’re in new territory now that any dev can kick off parallel coding with coding agents. Will it make engineers more productive, or will it just make people feel like they’re more productive? Perhaps engineers who do one thing at a time and keep focus will be shown to produce more reliable software, over time. Or maybe it’ll turn out that working with parallel agents leads to more issues slipping through and more iterations, which destroys any gains.

We will find out. Personally, I can only see more devs experimenting with parallel agents.

My sense is that software engineering basics matter more when working with AI agents. I’ve started to use AI agents for my own side projects, with success so far. I do a few things:

• Testing: all side projects have unit tests because I learned to not trust my own work without validation
• Small, descriptive tasks: I give tasks small enough in scope, which I explain, and give examples of
• Refactoring: every third or fourth task is for the agent to refactor some code they wrote (e.g., extract into a method, move to a new class)
• Review: I track what the agent does
• Do small things personally: I keep my IDE open and do anything that’s a few lines to change by hand, so I stay aware of the codebase

I keep hearing the same from other engineers: “mandating” engineering practices like having the agent pass all tests before continuing, leads to better results. This is unsurprising and it’s why these practices are getting popular. AI agents are non-deterministic and to some extent unreliable; these practices make them a lot more reliable and usable.

This was one out of the five topics covered in The Pulse #149. The full edition additionally covers: The full issue additionally covers:

• ACP protocol. A new protocol built by the Zed team, which tries to make it easier to build AI tooling for IDEs than the MCP protocol allows
• AI security tooling works surprisingly well? AI-powered security tools seem good at identifying security flaws in mature open source projects
• Is AI the only engine of US economic growth? Forty percent of US GDP this year is based on AI-related spend, while 60% of venture capital goes into AI. Hopefully, it won’t end up as a bubble which bursts like in 2001
• Comparing interviews at 8 large tech companies. Puneet Patwari applied to 8 major tech companies, and received 6 offers. He compares his interview experiences at Meta, Amazon, Uber, and 5 other workplaces

Read the full issue here, and check out today’s The Pulse here.

Monday was an interesting day: Signal stopped working, Slack and Zoom had issues, and most Amazon services were also down, together with thousands of websites and apps, across the globe. The cause was a 14-hour-long AWS outage in the us-east-1 region.

Today, we look into what caused this outage.

To its credit, AWS posted continuous updates throughout the outage. Three days after the incident, they released a detailed postmortem – much faster than the 4 months it took in 2023 after a similarly large event.

The latest outage was caused by DynamoDB’s DNS failure. DynamoDB is a serverless NoSQL database built for durability and high availability, which promises 99.99% uptime as its service level agreement (SLA), when set to multi-availability zone (AZ) replication. Basically, when operated in a single region, DynamoDB promises – and delivers! – very high uptime with low latency. Even better, while the default consistency model for DynamoDB is eventual consistency (reads might not yet reflect the actual status), reads can also be set to use strong consistency (guaranteed to return the actual status).

All these traits make DynamoDB an attractive choice for data storage for pretty much any application, and many of AWS’s own services also depend heavily on DynamoDB. Plus, DynamoDB has a track record of delivering on its SLA promises, so the question is often not why to use DynamoDB, but rather, why not to use this highly reliable data storage. Potential reasons for not using it include complex querying, complex data models, or storing large amounts of data when storage costs are not worth it compared to other bulk storage solutions.

In this outage, DynamoDB went down, and the dynamodb.us-east-1.amazonaws.com address returned an empty DNS record. To every service – external to AWS or AWS internal – it seemed like DynamoDB in this AWS region disappeared off the face of earth! To understand what happened, we need to look into DynamoDB DNS management.

How DynamoDB DNS management happens

Here’s an overview:

How it works:

• DNS planner: this service monitors load balancer (LB) health. As you can imagine, DynamoDB runs at a massive scale, and LBs can easily get overloaded or under-utilized. When overloading happens, new LBs need to be added, and when there’s underutilization, they need to be removed. The DNS planner created DNS plans. Each DNS plan is a set of LB sets, and assigning them weights on how much traffic to give the LB.
• DNS enactor: the service responsible for updating the routes in Amazon’s DNS service called Route 53. For resiliency, there is one DNS enactor running in each availability zone (AZ). In us-east-1, there are 3 AZs, and 3x DNS Enactor instances.
• Race conditions are expected: with 3x parallel DNS Enactors working simultaneously, race conditions are expected. The system deals with this by assuming eventual consistency: even if a DNS Enactor updates Route 53 with an “old” plan, DNS plans are consistent with one another. Plus, updating happens quickly, and DNS Enactors only use the latest plans from the DNS Planner.

DynamoDB down for 3 hours

Several independent events combined to knock DynamoDB’s DNS offline:

1. High delays on a DNS Enactor #1. Updating DNS took unusually long for one DNS Enactor for some reason. Usually, these updates are rapid, but weren’t on 20 October.
2. DNS Planner turns up the pace in churning out DNS plans. Just as DNS updates turned slow, the DNS planner started to produce new plans at a much higher pace than before.
3. DNS Enactor #2 rapidly processes DNS plans. While DNS Enactor #1 was applying DNS plans at snail’s pace, DNS Enactor #2 was storming through them. As soon as it finished writing these plans to Route 53, it went back to DNS Planner and deleted the old plans.

These three things pushed the system into an inconsistent state and emptied out DynamoDB DNS:

1. DNS Enactor #1 unknowingly uses an old DNS plan. When DNS Enactor #2 finished applying the newest DNS plan, it went back to DNS Planner and deleted all older plans. Doing so should have meant that other DNS Enactors did not use old plans; but remember, DNS Enactor #1 was slow and still processing through an old plan! As a result, the check by DNS Enactor #1 was stale.
2. DNS Enactor #2 detects the old plan being used and clears DNS records. DNS Enactors have another cleanup check: if they detect an old plan being used, they delete the plan itself. Deleting a plan means removing all IP addresses for the regional endpoints in Route 53. So DNS Enactor #2 turned the dynamodb.us-east-1.amazonaws.com DNS empty!

DynamoDB going down also took down all AWS services dependent on the us-east-1 DynamoDB services. From the AWS postmortem:

“All systems needing to connect to the DynamoDB service in the N. Virginia (us-east-1) Region via the public endpoint immediately began experiencing DNS failures and failed to connect to DynamoDB. This included customer traffic as well as traffic from internal AWS services that rely on DynamoDB. Customers with DynamoDB global tables were able to successfully connect to and issue requests against their replica tables in other Regions, but experienced prolonged replication lag to and from the replica tables in the N. Virginia (us-east-1) Region.”

The DynamoDB outage lasted around 3 hours; I can only imagine AWS engineers scratching their heads and wondering how the DNS records were emptied. Eventually, engineers manually intervened, and brought back DynamoDB. It’s worth remembering that bringing up DynamoDB might have included avoiding the thundering herd issue that is typical of restarting large services.

To be honest, I’m sensing key details were omitted from the postmortem. Things unmentioned which are key to understanding what really happened:

• Why did DNS Enactor #1 slow down in updating DNS, compared to DNS Enactor #2?
• Why did DNS Enactor #2 delete all DNS records as part of cleanup? This really made no sense, and it feels like there’s an underlying reason.
• The race condition of one DNS Enactor being well ahead of others seems to be easy enough to forecast. Was this the first time it happened? If not, what happened after previous, similar incidents?
• Most pressingly, how will the team fix this vulnerability which could happen anytime in the future?

Amazon EC2 down for 12 more hours

With DynamoDB restored, the pain was still not over for AWS. In fact, Amazon EC2’s problems just got worse. To understand what happened, we need to understand how EC2 works:

• DropletWorkflow Manager (DWFM) is the subsystem that manages physical servers for EC2. Think of it as the “Kubernetes for EC2.” EC2 instances are called “droplets.”
• Lease: DropletWorkflow Manager tracks the lease for each droplet (server) so it knows if and when a server is occupied, or can be allocated to an EC2 customer. The DWFM does a status check of the server every few minutes to determine its state.

State check results are stored in DynamoDB, so the DynamoDB outage caused problems:

1. Leases started to time out. With state check results not returning due to the DynamoDB outage, the DropletWorkflow Manager started to mark droplets as not available.

2. Insufficient capacity errors on EC2: with most leases timed out, DWFM started to return “insufficient capacity error” messages to EC2 customers. It thought servers were not available, after all.

3. DynamoDB’s return didn’t help: when DynamoDB came back online, it should have been possible to update the status of droplets. But that didn’t happen. From the postmortem:

“Due to the large number of droplets, efforts to establish new droplet leases took long enough that the work could not be completed before they timed out. Additional work was queued to reattempt establishing the droplet lease. At this point, DWFM had entered a state of congestive collapse and was unable to make forward progress in recovering droplet leases.”

It took engineers 3 more hours to come up with mitigations to get EC2 instance allocation working again.

Network propagation errors took another 5 hours to fix. Even when EC2 was looking healthy on the inside, instances could not communicate with the outside world, and congestion built up inside a system called Network Manager. Also from the postmortem:

“Network Manager started to experience increased latencies in network propagation times as it worked to process the backlog of network state changes. While new EC2 instances could be launched successfully, they would not have the necessary network connectivity due to the delays in network state propagation. Engineers worked to reduce the load on Network Manager to address network configuration propagation times and took action to accelerate recovery. By 10:36 AM PT [11 hours after the start of the outage], network configuration propagation times had returned to normal levels, and new EC2 instance launches were once again operating normally.”

Final cleanup took another 3 hours. After all 3 systems fixed – DynamoDB, EC2’s DropletWorkflow Manager and Network Manager – there was a bit of cleanup left to do:

“The final step towards EC2 recovery was to fully remove the request throttles that had been put in place to reduce the load on the various EC2 subsystems. As API calls and new EC2 instance launch requests stabilized, at 11:23 AM PDT [12 hours after the outage started] our engineers began relaxing request throttles as they worked towards full recovery. At 1:50 PM [14 hours after the outage started], all EC2 APIs and new EC2 instance launches were operating normally.”

Phew – that was a lot of work! Props to the AWS team for working through it all in what must have been a stressful night’s work. You can read the full postmortem here, which details the impact on other services like the Network Load Balancer (NLB), Lambda functions, Amazon Elastic Container Service (ECS), Elastic Kubernetes Service (EKS), Fargate, Amazon Connect, AWS Security Token Service, and AWS Management Console.

This was one out of four topics from today’s The Pulse, analyzing this large AWS outage. The full issue additionally covers:

1. Worldwide impact. From Ring cameras, Robinhood, Snapchat, and Duolingo, all the way to Substack – sites and services went down in their thousands.
2. Unexpected AWS dependencies. Status pages using Atlassian’s Statuspage product could not be updated, Eight Sleep mattresses were effectively bricked for users, Postman was unusable, UK taxpayers couldn’t access the HMRC portal, and a Premier League game was interrupted.
3. Why such dependency on us-east-1? It feels like half of the internet is on us-east-1 for its low pricing and high capacity. Meanwhile, some AWS services are themselves dependent on this region.

Read the full article here.