1. SaaS is a pure software product. People who pay SaaS vendors do so because it’s cheaper to buy this software than build it.
2. LLMs dramatically reduce the time and cost of building custom software.
3. Therefore, most SaaS vendors will go out of business because most companies/teams will prompt an LLM to write the software they need, such as for ticketing, meetings, customer relationship management, etc.

The reason for my scepticism has been that SaaS such as HR software Workday is more than just software. Workday, for example, keeps up with compliance requirements (e.g., for holiday pay in different countries), guarantees correctness (e.g., payslips that comply with local regulations), and over time the software keeps up to date with changes in the external and internal environments.

However, this week I had first-hand experience of how ridiculously easy it is now to replace SaaS with LLMs. On my website – pragmaticengineer.com – I have a testimonials section, which displays real LinkedIn and X posts about this publication. It cost $120/year for a small service called Shoutout.io, and looked like this:

And this is the backend: nothing fancy, just a way to add, edit, reorganize, and delete testimonials.

I was a customer for four years and logged in perhaps once a year. My latest login was to get an annual invoice for my expenses. Unfortunately, the billing section was broken, so I emailed support and they sent me a broken link instead of the invoice. This was frustrating: why pay for a SaaS with broken billing? I couldn’t even tell what they would charge me next year.

So I asked myself if I could rebuild my own use case with an LLM, and do it rapidly. My use case was much simpler than the SaaS itself:

• Display existing testimonials in a similar way
• Make it easy to add new ones, e.g., store testimonials in some JSON format
• Make it look good

To my surprise, this whole effort from start to finish took exactly 20 minutes with Codex. The steps I took were straightforward enough:

• Asked Codex to make a plan on how to remove this third-party dependency and host all testimonials in my codebase (a GitHub repo, deployed onto Netlify)
• Tweaked the plan: I pushed for a modular approach where testimonials are in a separate JSON file, and they get generated into HTML with a compile-time build step
• Added this build step both locally and as a build trigger on Netlify
• Tested the solution
• Tweaked the UX and generated a schema
• Deployed it

The end result is visually the same as before, except I no longer have a third-party dependency rendering all of this!

What does this mean for SaaS products and software engineers?

What it means for software engineers:

• Devs are (probably) a lot more comfortable using the command line for future updates than regular users. To add a future testimonial, I’ll need to turn to my AI agent to insert it in my codebase, and I’ll then need to verify that it looks good. This is not a big deal for me, but it might be a dealbreaker for someone not comfortable with verifying the code output of an LLM.
• It’s a lot faster for a dev to “port” a SaaS than for anyone else. I first told Codex to copy the UI and it got things wrong because it tried to use a flexbox model. I had to tell it that this UI layout was not what I wanted, and then make the decision on which framework to use for the UI layout. A non-developer could probably figure all this out, but it would take longer.
• Honestly, it’s fun and interesting to rewrite a third-party feature. I recommend it. Part of why I took on this project is because I expected it to be an interesting challenge. I thought the effort would be more than what it was, and I’ve learned more about how well these tools work. I also used Codex in order to experience it more.

What this could mean for SaaS software:

• Rebuilding a SaaS still feels much harder than rebuilding your specific use case. I did not “rebuild” Shoutout in any way. Shoutout has 10x or more features, like adding quotes from 10 different platforms, authentication, billing (which didn’t work for me), and more.
• A SaaS that doesn’t give ongoing value is at risk of being replaced by customers. Shoutout doesn’t provide ongoing value after it displays my testimonials, and this static nature means it’s easy to replace. In contrast, it would be harder to rebuild if I paid for the platform to stay compliant, provide analytics or alerting, and do other real-time things that helped my business.
• Buying and selling SaaS businesses could become less profitable. The original version of Shoutout that I signed up for in 2021 was built in 2020 by an independent developer. In 2022, this developer sold this micro-SaaS to a product studio. Then, in 2025, Shoutout was sold again to new developers. From my point of view, nothing changed except that the billing system broke. I assume the buyers of this SaaS figured that revenue could keep rising with zero investment. But perhaps at some point that ceases to be true when people get fed up with a broken product and quit – especially when doing so is cheaper.

“Broken windows” not being fixed is less acceptable than it used to be. My journey away from Shoutout began with its billing system being broken. For example, below is what I saw when I went to my billing section to see the invoices:

As well as this, the customer support sent me a broken link in response to my email. That was enough for me to decide to replace this dependency, and I was surprised by how easy this was with an LLM and knowing what I wanted it to build. By the time customer support sent me a working link two hours later, I had finished migrating off the SaaS.

It feels like something valuable is being taken away, and suddenly. It took a lot of effort to get good at coding and to learn how to write code that works, to read and understand complex code, and to debug and fix when code doesn’t work as it should. I still remember how daunting my first “real” programming class was at university (learning C), how lost I felt on my first job with a complex codebase, and how it took years of practice, learning from other devs, books, and blogs, to get better at the craft. Once you’re pretty good, you have something that’s valuable and easy to validate by writing code that works!

Some of my best memories of building software are about coding. Being “locked in” and balancing several ideas while typing them out, of being in the zone, then compiling the code, running it and seeing that “YES”, it worked as expected!

It’s been a love-hate relationship, to be fair, based on the amount of focus needed to write complex code. Then there’s all the conflicts that time estimates caused: time passes differently when you’re locked in and working on a hard problem.

Now, all that looks like it will be history.

I wonder if I’ll still get the same sense of satisfaction from the fact that writing complicated code is hard? Yes, AI is convenient, but there’s also a loss.

Or perhaps with AI agents, being “in the zone” will shift to thinking about higher-level problems, while instructing more complex code to be written?

This was a section from my analysis piece When AI writes almost all code, what happens to software engineering?. Read the full one here.

A mere two weeks after Cloudflare suffered a major outage and took down half the internet, the same thing has happened again. Last Friday, 5th December, thousands of sites went down or partially down once more, in a global Cloudflare outage lasting 25 minutes.

As per last time, Cloudflare was speedy to share a full postmortem on the same day. It estimated that 28% of Cloudflare’s HTTP traffic was impacted. The cause of this latest outage was Cloudflare making a seemingly innocent – but global – configuration change that went on to take out a good portion of Cloudflare, globally, until being reverted. Here’s what happened:

• Cloudflare was rolling out a fix for a nasty React security vulnerability
• The fix caused an error in an internal testing tool
• The Cloudflare team disabled the testing tool with a global killswitch
• As this global configuration change was made, the killswitch unexpectedly caused a bug that resulted in HTTP 500 errors across Cloudflare’s network

In this latest outage, Cloudflare was burnt by yet another global configuration change. The previous outage in November happened thanks to a global database permissions change. In the postmortem of that incident, the Cloudflare team closed with this action item:

“Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input”

This change would make it so that Cloudflare’s configuration files do not propagate immediately to the full network, as they still do now. But making all global configuration files have staged rollouts is a large implementation that could take months. Evidently, there wasn’t time to make it yet, and it has come back to bite Cloudflare.

Unfortunately for Cloudflare, customers are likely to find unacceptable a second outage with similar causes to a previous one, only weeks ago. If Cloudflare proves unreliable, customers should plan to onboard to backup CDNs at the very least, and a backup CDN vendor will do its best to convince new customers to use it as the primary CDN.

Cloudflare’s value-add rests on rock-solid reliability without customers needing to budget for a backup CDN. Yes, publishing postmortems on the same day as an outage occurs helps restore trust, but that will crumble anyway with repeated large outages.

To be fair, the company is doubling down on implementing staged configuration rollouts. In its postmortem, Cloudflare is its own biggest critic. CTO Dane Knecht reflected:

“[Global configuration changes rolling out globally] remains our first priority across the organization. In particular, the projects outlined below should help contain the impact of these kinds of changes:Enhanced Rollouts & Versioning: Similar to how we slowly deploy software with strict health validation, data used for rapid threat response and general configuration needs to have the same safety and blast mitigation features. This includes health validation and quick rollback capabilities among other things.Streamlined break glass capabilities: Ensure that critical operations can still be achieved in the face of additional types of failures. This applies to internal services as well as all standard methods of interaction with the Cloudflare control plane used by all Cloudflare customers.“Fail-Open” Error Handling: As part of the resilience effort, we are replacing the incorrectly applied hard-fail logic across all critical Cloudflare data-plane components. If a configuration file is corrupt or out-of-range (e.g., exceeding feature caps), the system will log the error and default to a known-good state or pass traffic without scoring, rather than dropping requests. Some services will likely give the customer the option to fail open or closed in certain scenarios. This will include drift-prevention capabilities to ensure this is enforced continuously.
These kinds of incidents, and how closely they are clustered together, are not acceptable for a network like ours”.

Global configuration errors often trigger large outages

There’s a pattern of implicit or explicit global configuration errors causing large outages, and some of the biggest ones in recent years were caused by a single change being rolled out to a whole network of machines:

• DNS and DNS-related systems like BGP: DNS changes are global by default, so it’s no wonder that DNS changes can cause global outages. Meta’s 7-hour outage in 2021 was related to DNS changes (more specifically, Border Gateway Protocol changes.) Meanwhile, the AWS outage in October started with the internal DNS system.
• OS updates happening at the same time, globally: Datadog’s 2023 outage cost the company $5M and was caused by Datadog’s Ubuntu machines executing an OS update within the same time window, globally. It caused issues with networking, and it didn’t help that Datadog ran its infra on 3 different cloud providers across 3 networks. The same kind of Ubuntu update also caused a global outage for Heroku in 2024.

Globally replicating configs: in 2024, a configuration policy change was rolled out globally and crashed every Spanner database node straight away. As Google concluded in its postmortem: “Given the global nature of quota management, this metadata was replicated globally within seconds”.

Implementing gradual rollouts for all configuration files is a lot of work. It’s also invisible labor because when done well, then its benefits will be undetectable, except in the absence of incidents, thanks to better infrastructure!

The largest systems in the world will likely have to implement safer ways to roll out configs – but not everybody needs to. Staged configuration rollout doesn’t make much sense for smaller companies and products because this infra work slows down product development.

It doesn’t just slow down building, but every deployment, too, and this friction is designed to make everything slower. As such, they don’t make much sense unless the stability of mature systems is more important than fast iteration.

Software engineering is a field where tradeoffs are a fact of life, and universal solutions don’t exist. The development which worked for a system with 1/100th of the load and users a year ago, may not make sense today.

This was one out of the four topics covered in this week’s The Pulse. The full edition additionally covers:

1. Industry Pulse. Poor capacity planning at AWS, Meta moves to a “closed AI” approach, a looming RAM shortage, early-stage startups hiring slower than before, how long it takes to earn $600K at Amazon and Meta, Apple loses execs to Meta, and more
2. How the engineering team at Oxide uses LLMs. They find LLMs great for reading documents and lightweight research, mixed for coding and code review, and a poor choice for writing documents – or any kind of writing, really!
3. Linux officially supports Rust in the kernel. Rust is now a first-class language inside the Linux kernel, eight months after a Linux Foundation Fellow predicted more support for Rust. A summary of the pros and cons of Rust support for Linux

Read the full The Pulse issue.

A year ago, Amazon became the first tech giant to bring staff back into the office for the full five days per week. Back then, I analyzed the reasons for the change, and whether other workplaces would follow suit by dropping the widespread hybrid policy of 2-3 days/week in the office.

Now, Meta employees in the Instagram division have become the latest subjects of a full return to the office, following an announcement by the social media platform this week.

Instagram’s 5-day return to office

Instagram employees received the unexpected email on Monday, reports fellow Substacker, Alex Heath, who acquired a copy of the message. It was sent internally by Instagram CEO Adam Mosseri, who wrote:

“1. Back to the office: I believe that we are more creative and collaborative when we are together in-person. (...)

2. Fewer meetings: We all spend too much time in meetings that are not effective, and it’s slowing us down. Every six months, we’ll cancel all recurring meetings and only re-add the ones that are absolutely necessary (...)

3. More demos, less decks: Most product overviews should be prototypes instead of decks.

4. Faster decision-making: We’re going to have a more formalized unblocking process with DRIs, and I’ll be at the priorities progress unblocking meeting every week.”

This decision by Meta affects around a quarter of company staff, and it’s hard to imagine other divisions not following Instagram’s lead; after all, everything in Mosseri’s memo likely applies across the business.

Five years ago, CEO Mark Zuckerberg predicted 50% of Meta staff would work remotely by now, which didn’t happen. Indeed, with Instagram’s new 5-day RTO, I’d be surprised if 5% of Meta folks work remotely in two years’ time.

The reason for Insta’s RTO seems rooted in the leadership’s belief that in-office is more productive, as indicated by the top bullet point of Mosseri’s message. That message in full:

“I believe that we are more creative and collaborative when we are together in-person. I felt this pre-COVID and I feel it any time I go to our New York office where the in-person culture is strong.

Starting February 2, I’m asking everyone in my rollup based in a US office with assigned desks to come back full time (five days a week). The specifics:

• You’ll still have the flexibility to work from home when you need to, since I recognize there will be times you won’t be able to come into the office. I trust you all to use your best judgment in figuring out how to adapt to this schedule.
• In the NY office, we won’t expect you to come back full time until we’ve alleviated the space constraints. We’ll share more once we have a better sense of timeline.
• In MPK [Menlo Park, the HQ], we’ll move from MPK21 to MPK22 on January 26 so everyone has an assigned desk. We’re also offering the option to transfer from the MPK to SF office for those people whose commute would be the same or better with that change. We’ll reach out directly to those people with more info.
• XFN [cross-functional] partners will continue to follow their own org norms.
• There is no change for employees who are currently remote”.

From what I’ve seen of Mosseri from afar, he seems like a pretty straight shooter. It’s clear that he feels in-office creates more energy, and in Mosseri’s defense, I hear similar from many startup founders and leaders who say remote work causes a bunch of headaches: it’s harder to spot motivational problems and performance issues, information travels more slowly, and rallying teams is harder.

There’s no doubt that running a full-remote company is a lot of effort. There’s often-overlooked labor involved in hiring, onboarding, performance management, team celebrations, and even company-wide meetings – none of it is easy.

Linear is a full-remote company with nearly 50 people working there, which recently published details about how it operates. They’re introducing the concept of “coworking hubs”, flying in teams for in-person events, and holding regular off-sites, while being careful to hire people who fit the culture.

My feeling is that remote work policies at tech companies are going to become questions of their leaders’ preferences. Many devs prefer remote work: there’s fewer interruptions, more deep focus, and less commuting. Most of us would probably be just as productive – and probably more so – than when being interrupted in-office.

Leaders who prefer full-remote can cite flexibility and easier hiring from a larger pool of candidates as clear benefits. Meanwhile, those most comfortable with in-person will always have enough reasons to justify a 5-day RTO, along the lines of Mosseri’s reasoning. Advocates of hybrid setups cite balancing of focus time and efficiency.

In today’s job market, any company that pays closer to the top of the market can probably get away with five-days-a-week RTO. Meta is in this space, and although I’m sure plenty of devs will dislike the change, the alternative is to go out on the job market, accept a pay cut to join a new company, and start rebuilding your internal network.

Since we’re in the midst of a weird job market, it makes switching jobs more difficult than before, when the job market was very hot. In this respect, Instagram has external conditions on its side. For devs at Meta, one upside is that Big Tech experience opens more doors, even in this tough job market.

One caveat is that a 5-day RTO is unlikely in places where it’s hard to hire the right people. So, AI engineers and those working on AI products should be pretty safe, for instance, because those roles are incredibly in-demand, as indicated by the trend of higher base salaries for AI engineers. Based on that, few companies should want to push those workers to quit to join competitors.

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Many subscribers expense The Pragmatic Engineer Newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

One amusing detail of the November 2025 Cloudflare outage is that the realtime outage and monitoring service, Downdetector, went down, revealing a key dependency on Cloudflare. At first, this looks odd; after all, Downdetector is about monitoring uptime, so why would it take on a key dependency like Cloudflare if it means this can happen?

Downdetector was built multi-region and multi-cloud, which I confirmed by talking with Senior Director of Engineering, Dhruv Arora, at Ookla, the company behind Downdetector. Multi-cloud resilience makes little sense for most products, but Downdetector was built to detect cloud provider outages, as well. And for this, they needed to be multi-cloud!

Still, Downdetector uses Cloudflare for DNS, Content Delivery (CDN), and Bot Protection. So, why would it take on this one key dependency, as opposed to hosting everything on its own servers?

A CDN has advantages that are hard to ignore, such as:

• Drastically lower bandwidth costs – assets cached on the CDN are much faster
• Faster load times because assets on a CDN are served from Edge nodes nearer users
• Protection from sudden traffic spikes, as would be common for Downdetector, especially during outages! Without a CDN, those spikes could overload their services
• DDoS protection from bad actors taking the site offline with a distributed denial of service attack
• Reduced infrastructure requirements, as Downdetector can run on fewer servers

Downdetector’s usage patterns reflect that it’s a service very heavily used by consumers whom the business doesn’t really monetize (Downdetector is free to use.) So, Downdetector could get rid of Cloudflare, but costs would surge, the site would become slower to load, and revenue wouldn’t change.

In the end, Downdetector’s dependence on Cloudflare could be a pragmatic choice based on the business model, and how removing its upstream dependency upon Cloudflare could get very expensive!

Dhruv confirmed this and sharing more about the design choices at Downdetector:

“Building redundancy at the DNS & CDN layers would require enormous overhead. This is especially true as Cloudflare’s Bot Protection is world-class, and building similar functionality would be a lot of effort. There are hyperscalers [cloud providers] that have this kind of redundancy built in. We will look into what we can do, but with a team size in the double digits, building up a core piece of infra like this is a pretty tall order: not just for us, but for any mid-sized team.

We’ve learned that there are more things that we can improve, for the future. For example, during the outage, the Cloudflare control pane was down, but their API wasn’t. So, us having more Infrastructure as Code could have helped bring back Downdetector sooner.

On our end, we also noticed that the outage wasn’t global, so we were able to shift traffic around and reduce the impact.

One more interesting detail: Cloudflare’s Bot Protection went haywire during the outage, and started to block legitimate traffic. So, our team had to turn that off temporarily”.

Thanks very much to Dhruv and the Downdetector team for sharing details.

An unexpected highlight of publishing the book was ending up in Mongolia in June of this year, at a small-but-mighty startup called Nasha Tech. This was because the startup translated my book into Mongolian. Here's the completed book:

Here’s what happened:

A little over a year ago, a small startup from Mongolia reached out, asking if they could translate the book. I was skeptical it would happen because the unit economics appeared pretty unfavorable. Mongolia’s population is 3.5 million; much smaller than other countries where professional publishers had offered to do a translation (Taiwan: 23M, South Korea: 51M, Germany: 84M, Japan: 122M, China: 1.43B people).

But I agreed to the initiative, and expected to hear nothing back. To my surprise, nine months later the translation was ready, and the startup printed 500 copies on the first run. They invited me to a book signing in the capital city of Ulaanbaatar, and soon I was on my way to meet the team, and to understand why a small tech company translated my book!

Japanese startup vibes in Mongolia

The startup behind the translation is called Nasha Tech; a mix of a startup and a digital agency. Founded in 2018, its main business has been agency work, mainly for companies in Japan. They are a group of 30 people, mostly software engineers.

Their offices resembled a mansion more than a typical workplace, and everyone takes their shoes off when arriving at work and switches to “office slippers”. I encountered the same vibe later at Cursor’s headquarters in San Francisco, in the US.

Nasha Tech found a niche of working for Japanese companies thanks to one of its cofounders studying in Japan, and building up connections while there. Interestingly, another cofounder later moved to Silicon Valley, and advises the company from afar.

The business builds the “Uber Eats of Mongolia”. Outside of working as an agency, Nasha Tech builds its own products. The most notable is called TokTok, the “UberEats of Mongolia”, which is the leading food delivery app in the capital city. The only difference between TokTok and other food delivery apps is scale: the local market is smaller than in some other cities. At a few thousand orders per day, it might not be worthwhile for an international player like Uber or Deliveroo to enter the market.

The tech stack Nasha Tech typically uses:

• Frontend: React / Next, Vue / Nuxt, TypeScript, Electron, Tailwind, Element UI
• Backend and API: NodeJS (Express, Hono, Deno, NestJS), Python (FastAPI, Flask), Ruby on Rails, PHP (Laravel), GraphQL, Socket, Recoil
• Mobile: Flutter, React Native, Fastlane
• Infra: AWS, GCP, Docker, Kubernetes, Terraform
• AI & ML: GCP Vertex, AWS Bedrock, Elasticsearch, LangChain, Langfuse

AI tools are very much widespread, and today the team uses Cursor, GitHub Copilot, Claude Code, OpenAI Codex, and Junie by Jetbrains.

I detected very few differences between Nasha Tech and other “typical” startups I’ve visited, in terms of the vibe and tech stack. Devs working on TokTok were very passionate about how to improve the app and reduce the tech debt accumulated by prioritizing the launch. A difference for me was the language and target market: the main language in the office is, obviously, Mongolian, and the products they build like TokTok also target the Mongolian market, or the Japanese one when working with clients.

One thing I learned was that awareness about the latest tools has no borders: back in June, a dev at Nasha Tech was already telling me that Claude Code was their daily driver, even though the tool had been released for barely a month at that point!

Why translate the book into Mongolian?

Nasha Tech was the only non-book publisher to express interest in translating the book. But why did they do it?

I was told the idea came from software engineer Suuribaatar Sainjargal, who bought and enjoyed the English-language version. He suggested translating the book so that everyone at the company could read it, not only those fluent in English.

Nasha Tech actually had some in-house experience of translation. A year earlier, in 2024, the company translated Matt Mochary’s The Great CEO Within as a way to uplevel their leadership team, and to help the broader Mongolian tech ecosystem.

Also, the company’s General Manager, Batutsengel Davaa, happened to have been involved in translating more than 10 books in a previous role. He took the lead in organizing this work, and here’s how the timelines played out:

• Professional translator: 3 months
• Technical editor revising the draft translation: 1 month
• Technical editing #2 by a Support Engineer in Japan: 2 months
• Technical revision: 15 engineers at Nasha Tech revised the book, with a “divide and conquer” approach: 2 months
• Final edit and print: 1 month

This was a real team effort. Somehow, this startup managed to produce a high-quality translation in around the same time as it took professional book publishers in my part of the world to do the same!

A secondary goal that Nasha Tech had was to advance the tech ecosystem in Mongolia. There’s understandably high demand for books in the mother tongue; I observed a number of book stands selling these books, and book fairs are also popular. The translation of my book has been selling well, where you can buy the book for 70,000 MNTs (~$19).

Book signing and the Mongolian startup scene

The book launch event was at Mongolia’s startup hub, called IT Park, which offers space for startups to operate in. I met a few working in the AI and fintech spaces – and even one startup producing comics.

I had the impression that the government and private sector are investing heavily in startups, and want to help more companies to become breakout success stories:

• IT Park report: the country’s tech sector is growing ~20%, year-on-year. The combined valuation of all startups in Mongolia is at $130M, today. It’s worth remembering that location is important for startups: being in hubs like the US, UK, and India confers advantages that can be reflected in valuations.
• Mongolian Startup Ecosystem Report 2023: the average pre-seed valuation of a startup in Mongolia is $170K, seed valuation at $330K, and Series A valuation at $870K. The numbers reflect market size; for savvy investors, this could also be an opportunity to invest early. I met a Staff Software Engineer at the book signing event who is working in Silicon Valley at Google, and invests and advises in startups in Mongolia.
• Mongolian startup ecosystem Map: better-known startups in the country.

Two promising startups from Mongolia: Chimege (an AI+voice startup) AND Global (fintech). Thanks very much to the Nasha Tech team for translating the book – keep up the great work!

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Before we start: I’m excited to share something new: The Pragmatic Summit.

Four years ago, The Pragmatic Engineer started as a small newsletter: me writing about topics relevant for engineers and engineering leaders at Big Tech and startups. Fast forward to today, and the newsletter crossed one million readers, and the publication expanded with a podcast as well.

One thing that was always missing: meeting in person. Engineers, leaders, founders—people who want to meet others in this community, and learn from each other. Until now that is:

In partnership with Statsig, I’m hosting the first-ever Pragmatic Summit. Seats are limited, and tickets are priced at $499, covering the venue, meals, and production—we’re not aiming to make any profit from this event.

Apply to attend the Summit

I hope to see many of you there!

Cloudflare takes down half the internet – but shares a great postmortem

On Tuesday came another reminder about how much of the internet depends on Cloudflare’s content delivery network (CDN), when thousands of sites went fully or partially offline in an outage that lasted 6 hours. Some of the higher-profile victims included:

• ChatGPT and Claude
• Canva, Dropbox, Spotify,
• Uber, Coinbase, Zoom
• X and Reddit

Separately, you may or may not recall that during a different recent outage caused by AWS, Elon Musk noted on his website, X, that AWS is a hard dependency for Signal, meaning an AWS outage could take down the secure messaging service at any moment. In response, a dev pointed out that it is the same for X with Cloudflare – and so it proved earlier this week, when X was broken by the Cloudflare outage.

That AWS outage was in the company’s us-east-1 region and took down a good part of the internet last month. AWS released incident details three days later – unusually speedy for the e-commerce giant – although that postmortem was high-level and we never learned exactly what caused AWS’s DNS Enactor service to slow down, triggering an unexpected race condition that kicked off the outage.

What happened this time with Cloudflare?

Within hours of mitigating the outage, Cloudflare’s CEO Matthew Prince shared an unusually detailed report of what exactly went wrong. The root cause was to do with propagating a configuration file to Cloudflare’s Bot Management module. The file crashed Bot Management, which took Cloudflare’s proxy functionality offline.

Here’s a brief overview of how Cloudflare’s proxy layer works at a high level. It’s the layer that protects the “origin” resources of customers – minimizing network traffic to them by blocking malicious requests and caching static resources in Cloudflare’s CDN:

Here’s how the incident unfolded:

A database permissions change in ClickHouse kicked things off. Before the permissions changed, all queries to fetch feature metadata (to be used by the Bot Management module) would have only been run on distributed tables in Clickhouse, in a database called “default” which contains 60 features.

Until now, these queries were running using a shared system account. Cloudflare’s engineering team wanted to improve system security and reliability, and move from this shared system account to individual user accounts. User accounts already had access to another database called “r0”, so the team made the database permission change for access to r0 to be implicit instead of explicit.

As a side effect of this, the same query collecting the features to be passed to Bot Management started to fetch from the r0 database, and return many more features than expected:

The Bot Management module does not allow loading of more than 200 features. This limit was well above the production usage of 60, and was put in place for performance reasons: the Bot Management module pre-allocates memory for up to 200 features, and it will not operate with more than this number.

A system panic hit machines served with the incorrect feature file. Cloudflare was nice enough to share the exact code that caused this panic, which was this unwrap() function:

What likely happened:

• The append_with_names() function likely checked for a limit of 200 features
• If it saw more than 200 features, it likely returned an error
• … and when writing the code, it was not expected that append_with_names() would return an error…
• … and so .unwrap() panicked and crashed the system!

Edge nodes started to crash, one by one, seemingly randomly. The feature file was being generated every 5 minutes, and gradually rolled out to Edge nodes. So, initially, it was only a few nodes that crashed, and then over time, more became non-responsive. At one point, both good and bad configuration files were being distributed, making failed nodes that received the good configuration file start working – for a while!

Why so long to find the root cause?

It took Cloudflare engineers unusually long – 2.5 hours! – to figure all this out, and that an incorrect configuration file propagating to Edge servers was to blame for their proxy going down. Turns out, an unrelated failure made the Cloudflare team suspect that they were under a coordinated botnet attack, as when a few of the Edge nodes started to go offline, the company’s status page did, too:

The team tried to gather details about the attack, but there was no attack, meaning they wasted time looking in the wrong place. In reality, the status page going down was a coincidence and unrelated to the outage. But it’s easy to see why their first reaction was to figure out if there was a distributed denial of service (DDoS) attack.

As mentioned, it eventually took 2.5 hours to pinpoint the incorrect configuration files as the source of the outage, and another hour to stop the propagation of new files, and create a new and correct file, which was deployed 3.5 hours after the start of the incident. Cleanup took another 2.5 hours, and at 17:06 UTC, the outage was resolved, ~6 hours after it started.

Cloudflare shared a detailed review of the incident and learnings, which can be read here.

How did the postmortem come so fast?

One thing that keeps being surprising about Cloudflare is how they have a very detailed postmortem up in less than 24 hours after the incident is resolved. Cofounfer and CEO Matthew Prince explained how this was possible:

• Matthew was part of the outage call.
• After the outage was resolved, he wrote a first version of the incident review, at home. Matthew was in Lisbon, in Cloudflare’s European HQ, so this was early evening
• The team circulated a Google Doc with this initial writeup, and questions that needed to be reviewed
• In a few hours, all questions were answered
• Matthew: “None of us were happy [about the incident] — we were embarrassed by what had happened — but we declared it [the postmortem] true and accurate.
• Sent the draft over to the SF team, who did one more sweep, the posted it

Talk about moving with the speed of a startup, despite being a publicly traded company!

Learnings

There is much to learn from this incident, such as:

Be explicit about logging errors when you raise them! Cloudflare could probably have identified the root cause of this error much faster if the line of code that returned an error, also logged the error, and if Cloudflare had alerts set up when certain errors spiked on its nodes. It could have surely shaved an hour or two off the time it took to mitigate.

Of course, logging errors before throwing them is extra work, but when done with monitoring or log analysis, it can help find the source of errors much faster.

Global database changes are always risky. You never know what part of the system you might hit. The incident started with a seemingly innocuous database permissions change that impacted a wide range of queries. Unfortunately, there is no good way to test the impact of such changes (if you know one, please leave a comment below!)

Cloudflare was making the right kind of change by removing global systems accounts; it’s a good direction to go in for security and reliability. It was extremely hard to predict the change would end up taking down a part of their system – and the web.

Two things going wrong at the same time can really throw an engineering team. If Cloudflare’s status page did not go offline, the engineering team would have surely pinpointed the problem much faster than they did. But in the heat of the moment, it’s easy to assume that two small outages are connected, until there’s evidence that they’re not. Cloudflare is a service that’s continuously under attack, so the engineering team can’t be blamed for assuming it might be more of the same.

CDNs are the backbone of the internet, and this outage doesn’t change that. The outage hit lots of large businesses, resulting in lost revenue for many. But could affected companies have prepared better for Cloudflare going down?

The problem is that this is hard: using a CDN means taking on a hard dependency in order to reduce traffic on your own servers (the origin servers), while serving internet users faster and more cheaply:

When using a CDN, you propagate addresses that point to that CDN server’s IP or domain. When the CDN goes down, you could start to redirect traffic to your own origin servers (and deal with the traffic spike), or utilize a backup CDN, if you prepared for this eventuality.

Both these are expensive to pull off:

• Redirecting to the origin servers likely means needing to suddenly scale up backend infrastructure
• Having a backup CDN means there must be a contract and payment for a CDN partner which will most likely sit idle. As and when it is needed, you must switch over and warm up their cache: it’s a lot of effort and money to do this!

A case study in the trickiness of dealing with a CDN going offline is the story of Downdetector, including inside details on why Downdetector went down during Cloudflare’s latest outage, and what they learned from it.

This was one out of the five topics covered in this week’s The Pulse. The full edition additionally covers:

1. Downdetector & the real cost of no upstream dependencies. During the Cloudflare outage, Downdetector was also unavailable. I got details from the team about why they have a hard dependency on Cloudflare, and why that won’t change anytime soon.
2. Antigravity: Google’s new AI IDE – that its devs cannot use. Google wants to become a serious player in AI coding tools, but Antigravity contains remnants of Windsurf. Interestingly, devs at Google aren’t allowed to use Antigravity for work
3. Industry pulse. Gemini 3 launch, Anthropic valued at $350B, Jeff Bezos funds an AI company, and unusually slow headcount growth at startups persists.
4. Five AI fakers caught in 1 month by crypto startup. Candidates who fake their backgrounds and change their looks in remote interviews continue to plague companies hiring full-remote – especially crypto startups.

Read the full The Pulse

In the end, this process took several times longer; 4 years, in fact! Happily, it was worth it: readers’ feedback about The Software Engineer’s Guidebook has been overwhelmingly positive, and on launch, the book became a #1 bestseller among all titles in two Amazon markets (the Netherlands and Poland), as well as a top 100-selling book in most Amazon markets. In 24 months it sold around 40,000 copies, and was translated into German, Korean, Mongolian and Traditional Chines – with the Japanese and simplified Chinese versions releasing later this month.

A lot of people ask why I chose to self publish, and it would be nice to say this was always the goal, but it wasn’t! Originally, I wanted to work with a top tech publisher, who would get the book to market fast, and give it a higher profile. This didn’t happen, but during the process I learned a lot about how publishing works, how to pitch a book, and how to choose which publishing route might be the right one. 

This article shares my learnings from writing and publishing a book which has done pretty well with readers, and it includes the experience working with an established publishing house:

1. Tech book publishing landscape
2. Financials of publishing
3. Publishing process and the publisher’s role
4. My book pitch
5. Working with a publisher
6. Breaking up with a publisher

1. Tech book publishing landscape

Today, there are reputable book publishers whose titles are good and authoritative, and there are other publishers whom this doesn’t apply to. Each publisher also has a subject area: some are mainstream and publish titles about every software engineering area from languages to engineering management. Meanwhile, others stick to a topic of expertise they focus on.

Here’s my mental model of the book publishing industry in 2025:

Highly reputable mainstream publishers

In tech book publishing, three publishing houses really stand out, in my opinion, and form a ‘big three’ among all players in this sector: 

• O’Reilly: if I had to pick a #1 tech book publisher, it would be O’Reilly. They publish some of the most referenced books – like Designing Data Intensive Applications by Martin Kleppmann, Tidy First by Kent Beck, The Staff Engineer’s Path by Tanya Reilly, and more. The book covers are distinctive, using images of animals.
• Manning: a broad range of titles on both specific and general tech topics, which employ historical figures on the covers.
• The Pragmatic Bookshelf: also referred to as the “Prags.” Founded by Andy Hunt and Dave Thomas, the authors of what might be the best-selling tech book ever; The Pragmatic Programmer. Since its founding, The Prags has refused digital rights management (DRM) on their ebooks.

High reputable “mainstream” publishers that are tough to pitch to

The publishers in this section have strong reputations, like those above. However, they are harder to pitch to, usually because they publish fewer tech books. I couldn’t find an author pitch template, or clear pitching instructions, and contributes to a sense of “don’t find us, we’ll find you” among the following publishing houses: 

• Addison-Wesley: one of the best-known brands in tech. It has been an imprint (a trade name within a publication) of Pearson since 1988, and is the publisher of many “classic” book titles like Clean Code by Robert C. Martin, The Pragmatic Programmer by Andy Hunt and Dave Thomas, and some recent ones like Modern Software Engineering by Dave Farley. I couldn’t find any way to pitch to this publisher, and new books they publish seem to be by established authors.
• Pearson: This business owns the Addison-Wesley imprint. Recently, it started to publish tech books as “Pearson” instead, author Martin Fowler shared.
• Wiley: formerly a well-known tech book publisher behind the “X for Dummies” series. It publishes lots of computer science textbooks, but I can’t find recently-published, well-known tech books for software engineers.
• Springer: another massive publisher for whom tech books are a small part of the business. I couldn’t find how to pitch tech books to them.
• Morgan Kaufmann: a well-known tech books publisher founded in 1984, and acquired in 2001 by Elsevier. As I understand, these days it prints far fewer technology book, and focuses on academic topics. No clear way to pitch to them.

Highly reputable “niche” publishers

The following publishers are standout in quality, covering fewer topics than those above.

• No Starch Press: “The finest in geek entertainment” is the tagline, featuring fun visuals, and high-quality content on specific technologies like machine learning, Python, JavaScript, etc.
• IT Revolution: titles for technology leaders: DevOps, technology delivery, workplace culture, and similar. Publisher of The Phoenix Project, Team Topologies, and Accelerate.
• Artima: focuses on Scala.
• CRC Press: publishes on technology, engineering, math, and medicine.
• Stripe Press: “works about technological, economic, and scientific advancement.”
• MIT Press: “a distinctive collection of influential books curated for scholars and libraries worldwide.”

Other mainstream book publishers

Apress is a reputable publisher with a lower profile, which publishes on a wide range of topics, from specific technologies and frameworks, to more generic topics on computing. Because they publish many books on many topics, they are usually open to pitches.

Packt. A tech book publisher with a focus on quantity over quality, it feels to me. There is limited support and feedback for authors, and titles could often use more editing. But also, Packt is likely to say “yes” to a serious proposal.

2. Financials of publishing

Financial matters really come into play when your proposal is accepted by a publisher and you receive a contract offer.

Advance: $2,000 – $5,000. An advance payment to the writer is a tried and tested way to make them deliver a completed manuscript. It’s often paid in chunks: 50% when a milestone is hit, and 50% when a full draft appears.

The “big three” publishers typically offer $5,000, usually as a flat, non-negotiable rate; at least, it’s what I was offered. Smaller publishers offer closer to $2,000 for more niche books. The advance is non-refundable; even if your book sells zero copies, you keep it. The publisher is making an investment in you, and taking a risk.

As an aside: if you are thinking of writing a book: for guest authors in The Pragmatic Engineer Newsletter guest authors I offer a $4,000 per article payment – and you can later publish your guest article in a book. Several authors working on their book have written a guest articles such as Lou Franco on Paying down tech debt or Apurva Chitnis on Thriving as a founding engineer. Writing a guest post can help refine ideas, broaden your reach, and prove helpful when publishing the article.

Paperback royalty: 7-15% 

Royalties are earned on book sales, and taken from the net price of the book. Net price is what a publisher gets after the retailer (e.g. Amazon, or a bookshop) takes their cut. Let’s see how it works for a $40 book:

It matters financially where your title is purchased; be it an online shop, physical book store, or purchased directly from the publisher. Many tech books are sold on Amazon and online stores. Amazon’s 40% cut seems high, but it’s actually the lowest among book retailers. Up to 60% is a common cut for a physical bookshop.

Most publishers offer 10-12.5% royalties, is my understanding, and Packt around 15-20%. Keep in mind that brand reputation plays a role; for example, Packt’s reputation is less elevated than Manning, which can make a difference to sales.

Ebook royalties: 10-25%

For ebooks, several publishers pay 25% royalties, but not all. But even with a higher royalty rate, an author might end up making less per sale. For example, on the Kindle platform, the cut for Amazon is high at 65%. Let’s look at a $30 ebook with a 20% royalty rate:

Ebooks are almost always priced lower than physical books, and when sold on Kindle, generate much less revenue for the author, while earning more per copy than the paperback version. I was offered 10% royalties on ebook sales, which is at the low end.

“Earning out” 

When an author needs to pay back an advance before being paid anything, this is called “earning out”. If you get a $5,000 advance for a title costing $40 per hard copy and $25 for the ebook version, and most sales happen on Amazon, it means:

• ~2,080 paperback sales on Amazon
• Or ~2,850 Kindle book sales
• Or ~1,250 paperback sales on the publisher website

The author needs to sell at least 1,000 copies across various platforms to “earn out.” The good news is that a publisher sends quarterly or annual royalty payments if a book keeps generating revenue, which would effectively be passive income.

The Prags’ unique approach

One publisher that calculates rates differently is The Pragmatic Bookshelf. Instead of offering a low-digit number on revenue, they offer a 50% split on profit.

50% on profit sounds much higher than 10% on revenue, right? However, the devil is in the details, because paying on profit means that the upfront publisher costs – editors, cover design, printing, distribution, marketing – all are deducted before any profit split.

Authors who have used this approach tell me the numbers end up pretty similar to the revenue model.

Real-world case studies with actual earnings

Designing Data Intensive Applications author, Martin Kleppmann, shared the cumulative royalties he made in 6 years. The breakdown is interesting; ebook and Safari Online sales generated more revenue for the writer than the print version.

Cloud Native Infrastructure earnings: author Justin Garrison published with O’Reilly, and was offered 10% for print and 25% for ebooks (split into half, thanks to working with a coauthor). His book sold 1,337 copies in 4 months; and made about $22,000 for the two authors (and around $11,000 for Justin.) Justin concluded:

“Going into this project I had a rough estimate in my head to make about $2000–3000 so this is much better than I expected. Set your expectations accordingly.”

Don’t forget that publishers are also in this to make a positive return. This means that it is unlikely for a highly reputable publisher to invest into a book that they do not believe would sell at least a few thousand copies. I don’t have the data here: but if I was a publisher, I would reject any book that didn’t look like it could hit 1,000 copies sold in the first year of publishing.

3. The publishing process, and publisher roles

Why does a publisher take so much of the revenue? Part of this is because they do a lot of the work around publishing, and need to hire (and pay!) people for those roles. Here is my understanding of how the publishing process works, based on four months of pitching to publishers; two months of working with one of them; and researching how the rest of the process works:

Here are people I worked with, and my experience with them:

The acquisitions editor. If you write a technical blog, you might get a reachout from someone called an acquisitions editor, who will ask if you would consider publishing a book. Also, when you submit a pitch to a publisher, chances are that you will first communicate with an acquisitions editor.

A publisher’s goal is to publish books that will be profitable for them. They find authors who could write these books two ways:

• Inbound pitches coming from authors – reviewed by editors or acquisitions editors
• External reachouts done by acquisitions editors

These people need to have a good understanding of what kinds of books sell well at the publisher (and why); what their current catalogue is; what the gaps are; and what competitor publishers are commissioning.

When I pitched my book to 3 respected publishers, in two cases I talked with (and worked with) the acquisitions editor to improve my pitch. The acquisitions editors were my “champions” at the publisher. Their goal was to get a pitch that the company would say yes to.

The development editor works on the structure of the book. They ask the author to come up with a detailed table of contents – in my case, they asked me to estimate even the length of the chapters. They also help develop – and maintain – the narrative of the book.

Had I not worked with a publisher, I would have had no appreciation of this “high-level editing” – which, turns out, is key for writing a well-structured tech book!

The project manager checks in with timelines, organizes reviews—like editorial reviews—and helps keep you accountable. One of the best things about working with a publisher is that you are on a tight deadline—without which it would take you several times longer to publish the book!

The publisher owns a lot of rights for your book! One thing that I realized only after signing with a publisher is that while publishers help a lot with writing the book – and taking a higher cut is sensible because of this – they also hold on to a lot of rights that impact your book! These are all things that you give up on, versus when self-publishing. These are:

• Global publishing rights. Although you are the author of the book – and usually hold the copyright to it – the publisher own wordlwide publishing rights. This means that they are the only ones who can publish the book, or longer excerpts of it. In practice, this means you need to get permission if you’d like to publish some parts of your book on e.g. your blog, or social media. They’ll usually grant this as it’s good marketing – but it’s still that you need to ask, as the author.
• Foreign rights. The publisher will own the publishing right, and will usually be the one who owns selling foreign rights. In theory, this could sound like you are losing out on things. In pratice, publishers are much better positioned to sell and administer these rights. Most publishers offer a 50% cut on these rights – it’s what my publisher offered. Also, the majority of tech books are not translated to other languages – a book that “only” sells 2,000 copies in English is unlikely to sell a significant number in a non-English market!
• The cover. The publisher decides what cover they will design, though they tend to check the author for feedback.
• The title. One of the surprises for me was how the publisher ultimately decides on the title and subtitle.

In short: this book is owned by the publisher. You are the author, but they are the only ones who can distribute it. In practice, many authors would prefer to have it this way – because all the work related to distributing the book is taken on by the publisher. However, it’s good to know that you need to give up all the above when working with a publisher.

4. My book pitch

My secret hope, back in 2019, was to get a contract with one of the “Big 3” tech book publishers: O’Reilly, Manning or The Prags. I pitched my book to all three: got a “no” from two, but a “yes” (and a contract) from one. Here’s how I went about my pitch.

Write a “one-pager” about your book

What will this book be about? Who is it for? What will readers take away when reading it? Answer these in a short pitch, before even seeking out publishers. Here’s what I put together as my “one-pager:”

Do some market research

What are similar books in the market that would be competing with this book, directly or indirectly? How is this book different from them?

What is the demographic of people who would be interested in buying this book? Can you estimate how large this crowd is? Realistically, what percentage of this group could be interested in buying the book – assuming they know about it? Don’t forget that publishers will invest into books that can generate decent sales: it’s good to do a little research to help confirm your title could be one of these!

Shortlist publishers you would be interested working with

While there are quite a few publishers out there: what are your top preferences? And what are ones you’re willing to consider, even if your “top” choices turn you away?

Self-publishing is always an option (I’ll cover more on how I went about this in later parts). However, going with a good publisher can significantly speed up your book production, while also improving the quality.

Write a draft table of contents and a draft chapter

Some publishers will want to look at what a draft chapter will look like – but not all of them. Still, I found it helpful to do writing before submitting to a publisher. If for no other reason, this was to confirm that I’d enjoy longform writing!

I spent about a week putting together a table of content, and around four months writing drafts of chapters. These chapters turned out to be helpful later on.

Submit a tailored pitch your the publisher(s)

Once you identified your top publisher choices, submit a pitch. Most book publishers have a pitch document they want you to follow. Here are common ones:

O’Reilly’s pitch template:

• Description
• About the topic
• Audience
• Keywords
• Competing titles
• Related O’Reilly titles
• Book outline
• Writing schedule

Manning’s pitch template:

• About the author
• About the book topic
• The book plan
• Q&A
• Reader overview
• Book competition
• Book length and illustrations
• Writing schedule
• Table of contents

The Pragmatic Bookshelf template:

• Overview
• Outline
• Bio
• Competing books
• PragProg books
• Market size
• Promotional ideas
• Writing samples

Most of these templates ask for similar content, so if you completed one pitch: the others are much easier. Here are some tips I’d have for building a pitch.

Put yourself in the shoes of the publisher. This book is a huge deal to you: but it’s just one of the dozens that the publisher will publish just this year. You want to write an amazing book: but the publisher wants to publish one that will sell.

And these are major differences! The publisher will care very much about competition for the book, and how their existing titles relate to them. Like a VC firm, a publisher will not want to fund two investments competing on the exact same market: so if the publisher recently published a book that is a deepdive on Go; they will almost certainly pass on the next one, no matter how good your pitch is.

Pitching to several publishers parallel is totally fine and you should do it! This is one thing I wish I’d done differently. In my mind, I was 100% certain that my first publisher-of-choice would jump on the opportunity to publish this book. I thus felt that it would be “unfair” if I pitched to other publishers, without hearing back.

In hindsight, as a first-time author, this strategy was a waste of time on my end. Most publishers are unlikely to take a risk on a first-time author with no books published in the past – like I was in 2019. And so the likely outcome is rejection in most cases.

In my case, I spent about two and a half months waiting on the response from this first publisher. My acquisitions editor was championing the book – making the case for the publisher to offer a contract – but in the end, the publisher chose another book with a similar topic that was in their pipeline. This made perfect business sense for them – but for me, I was spent waiting for months, instead of pitching the book to other publishers!

My book pitch ended up being a helpful resource on my self-publishing journey. Even though I did not release with a publisher: pitching to publishers helped the book become an eventual success. It was for these reasons:

• Defining the structure. I had my table of contents well thought-out by the time I submitted the pitch. This structure changed later, but it was a solid start.
• Positioning the book. I had a good idea of the “competitive” landscape, and what books my title would “go up against.” It also helped me focus on how my book is different to what is already out there.
• Forcing me to think about marketing. The Pragmatic Bookshelf asked for a section on promotional ideas. This forced me to think about where (and how) I would promote the book – even before getting into the thick of writing. When going with a publisher, it’s safe to assume that the publisher’s brand will do some marketing. However, authors will still do the lion’s share of marketing – and it’s good to think about this ahead of time.

5. Working with a publisher

I got lucky with one of the three publishers, in the end. This publisher was looking for a book just like mine, right at that time! What happened was one of their best sellers had to be pulled from publication, for reasons outside the control of the publisher. Apparently, when my pitch arrived, they had just started a search for a book that could plug the hole – and they saw my book being a perfect fit for a “software career advice” book.

At the time, this felt like great luck. In hindsight, my relationship with the publisher might have soured exactly because they were looking for me to write a specific kind of book that would be similar enough to this old book – but I had no intention of doing so. More on how things went sour in the section after this one.

From signing the contract, I worked with a publisher for about a month – so I’m not exactly the most experienced in this front. However, a couple of things stood out as strong positives – and things that I “lost” when deciding to self publish, in the end.

Strong pressure to write – thanks to the contract. My contract had pretty strict deadlines included. We signed it on 11 January 2020, and these deadlines were part of the contract:

“The Author shall prepare and deliver to the Publisher a machine-readable electronic copy of the manuscript for the Work, including all its illustrations, code listings, and exercises, as mutually agreed upon by the Publisher and the Author as follows:

- Not later than March 15, 2020, a partial manuscript for the Work totaling not less than one third of the planned finished Work.

- Not later than June 1, 2020, a partial manuscript for the Work totaling not less than two thirds of the planned finished Work.

- Not later than August 15, 2020, a draft of the complete manuscript for the Work suitable for review.

- Not later than September 1, 2020, the final, revised and complete manuscript for the Work acceptable to the Publisher for publication.”

Talk about pressure! Also, my first payout was tied to reaching the first milestone – which was delivering at least a third of the finished work. My publisher also set up regular check-ins to help me stay accountable. And this kind of pressure was good – because without it, I would have pushed back writing, or got stuck on relatively trivial parts! 

6. Breaking up with the publisher

While I greatly appreciated that a publisher took a chance on me, lots of things felt wrong from the start. A month into working together, I felt that things were getting worse, and not better.

The small things that I dismissed, in the beginning:

• A (very) opinionated structure. This publisher had strongly opinionated templates I was told to use for all chapters. They included each chapter to start by stating what the reader will learn; and then summarize this at the end of the chapter. It wasn’t how I imagined my book to be – but it didn’t seem I had a choice. I figured, I’ll give it a go. The publisher knows better after all, as they’ve done this hundreds of times. Right?
• Needing to ask for permission to share drafts on social media. I originally planned to share screenshots of some of the parts I am writing to get feedback as I go – and to also increase visibility of the book. I thought that this is a no-brainer. Not only does this kind of “early sharing” makes the book better: but it will also make more people excited about the book, leading to more eventual customers. To my surprise, my contact at the publisher said I will need to ask for permission whether I can do this. Permission? For something that will market the book? Yes: because the publisher owns all publishing rights, including for the draft!
• I won’t decide on what the title will be. I had strong opinions about what I’d like the book’s title to be. My publishing contact also had ideas on what they thought would be good to add to it – like introducing the “mentoring” term either to the title or the subtitle: which was an idea I disliked. As I talked with them, it became clear that the publisher will set the final title: not me. Hmm – odd, no? It’s another reminder that, although it’s my book: it’s really the publisher’s book, and they have the final say on all important decisions.
• Nudges to “dumb down” the book. My editor was giving more suggestions on how to edit the content to make it more “beginner-friendly” and suggested I introduce e.g. “Alice and Bob” examples to make it easier to digest the contents. One of the recently best-selling books of the publisher heavily used Alice and Bob, and it seems the publisher thought it helped their sales.

The first major editorial review was where I decided we should part ways with the publisher. About a month-and-a-half in, the publisher pulled together several experienced editors, and offered suggestions on how I could improve the book. The suggestions were these:

• Focus on reader engagement. Tell stories and develop them with emotion, mystery, aha moments, and unexpected conclusions. Tell the stories from the "we" or "they" perspective -- make stories team-oriented.
• Exercises. Develop exercises for use within the chapters (not just end) or a story about what happened when one person did the exercise.
• Mini-projects. Guide readers to discover and come to conclusions on their own (see Donald Saari story in What the Best College Teachers Do). Mini project topics: testing, architectures.
• Word of the day feature. Example: Dependency injection (what is it)? Scatter these across the book.
• Quotes. Include quotes from luminaries such as [Well-known-person 1] and [Well-known-person 2] that relate to advice given. Ask other [Publisher] authors to relate experience about how they followed similar advice and were successful.
• Tech map. Create a diagram of the current technology landscape. Example big-picture topics: architecture demystified, distributed systems demystified.

While I appreciated the suggestions: I hated all of them. I saw what implementing them would do: they would turn this book – which I already had reservations with the “forced” style on me – to something I would not want to read. Much less write!

I envisioned writing a more matter-of-the-fact book that doesn’t have exercises, “mini projects” or “word of the day” gimmicks.

I sat down to reflect why I chose to work with a publisher, to start with. As an author, I’m giving up a lot of things: editorial control, the bulk of revenue, all publishing rights… and for what? For the publisher to make the process easier, and for the end result book to be better than if I was working alone.

But I felt that this book would be far worse if I continued with my publisher: and the only way to get it back to what I envisioned was if I spent a lot of time and energy pushing back on them.

It would cost me less energy to self-publish. So I decided to terminate my agreement because it didn’t feel my publisher was helping write the book that I wanted to write.

My publisher was understanding and professional in terminating the contract. I explained to them that all the feedback suggested they wanted to see a very different book to what I wanted to write. And that, frankly, I am not the author to write that kind of book.

Truth be told, I was embarrassed that I had wasted their resources – working with their development editor and the editing team – for these two months. At the same time, I was vocal in voicing to my editor that I was hesitant about this mandated style. I also made the decision that there is no point in continuing at the first formal feedback session. I’m not sure I could have come to this conclusion any further, as I was still learning how this book publisher worked, up until that point.

To show how professional this team was, this is the termination letter they sent as a signed PDF:

“This letter is in reference to our Publishing Agreement with you for [what would become The Software Engineer’s Guidebook] dated January 11, 2020. By mutual agreement, we are terminating the publishing contract.

Since no advance was paid to you under the terms of this contract, all rights in the content you originally submitted will hereby return to you and we will consider this matter concluded.

The decision to cancel a project is never an easy one to make. We thank you for all the efforts on this project that you made and wish you the best in your future endeavors.”

At this point, I learned enough about publishers and myself to decide: I’m doing it by myself. Having my book accepted by a major publisher gave external validation that there’s a strong business case for The Software Engineer’s Guidebook. And working with an opinionated publisher – and continuously pushing back on styling suggestions made me realize that I already have my own opinonated style that I like using.

I did lose a very important thing by deciding to self-publish: the accountability of meeting a publishing deadline. Working with the publisher, this book would have been out fall 2020 or spring 2021. Self-publishing, I launched it November 2023.

One of the reasons for publishing my book two years later than it would have taken with a publisher was because I now knew I could no longer rely on a well-known publisher to lend my book their brand. For my book to have an even slim chance of being successful: I would have to compensate for the lack of being associated with a publisher, and fill the gap in marketing and awareness, leading up to the book launch.

Not having a publisher was a reason I started writing The Pragmatic Engineer Newsletter in August 2021 (a year-and-a-half after breaking up with this publisher) – and the sudden success of this newsletter gave me less time to wrap up the book. At the same time, by the time the book was ready, there were plenty of people who looked forward to reading it: and many of them were already readers of the newsletter!

I’ll cover more about how I went about the actual self-publishing process in a follow-up article, how the book ended up selling, and other learnings. Subscribe to The Pragmatic Engineer to get notified when it is out.

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Online retail giant Amazon unexpectedly announced 14,000 job cuts earlier last week. The massive round of layoffs at the company follows other mass redundancies in recent years:

• January 2023: 18,000 people laid off.
• March 2023: another 9,000 people let go
• November 2023: hundreds of people let go inside the Alexa team, as Amazon was looking to shift Alexa more toward GenAI
• April 2024: hundreds of people let go from AWS

Software engineers, unfortunately, seem hit hard by the latest layoffs: of 2,300 employees laid off in Washington State, 25% are software engineers, GeekWire reports. We can only speculate about the ratio across the rest of the company, but if cuts at HQ are heavy on engineering, then things don’t look promising for other locations, sadly.

The memo from Beth Galetti, Senior Vice President of People Experience and Technology, to workers didn’t explain much:

“Some may ask why we’re reducing roles when the company is performing well. Across our businesses, we’re delivering great customer experiences every day, innovating at a rapid rate, and producing strong business results. What we need to remember is that the world is changing quickly. This generation of AI is the most transformative technology we’ve seen since the Internet, and it’s enabling companies to innovate much faster than ever before (in existing market segments and altogether new ones). We’re convinced that we need to be organized more leanly, with fewer layers and more ownership, to move as quickly as possible for our customers and business.”

The statement is utterly confusing, as encapsulated by its message that “business is great, but we need to do layoffs”. Job cuts usually mean a business is in trouble, which obviously isn’t the case for Amazon. So, why are these layoffs really happening?

Layoffs to boost efficiency?

The company’s memo states:

“We’re convinced that we need to be organized more leanly, with fewer layers and more ownership, to move as quickly as possible for our customers and business.”

If this line of reasoning sounds familiar, it’s because most of the layoffs in 2023 were justified the same way. The tech industry overhired during the pandemic in 2020-2021, making orgs more bloated and decision-making slower. In February 2023, I reported on the trend of fewer middle managers, with Meta the first Big Tech giant to reduce its management layers. In 2023, most of Big Tech followed this approach with layoffs or reorgs. Managers acquired more reports, and tech companies cut down the number of layers between the CEO and individual contributors.

Given Amazon did other massive layoffs in 2023, it’s unlikely they missed the industrywide trend for fewer managers. While the current layoffs seem to be targeting managers quite a bit – from the Washington State layoffs, 20% of those let go are managers – there are still more ICs laid off than managers, overall. So, this official explanation doesn’t pass my personal “smell test”.

Layoffs to buy more GPUs?

The day after its jobs announcement, Amazon had more big news, this time about AI: it unveiled Project Rainer, the largest AI computing platform AWS has ever built. It already has 500,000 Trainium2 chips (built by Amazon), This capacity is already 70% larger than any AI computing platform in AWS’s history, and Anthropic is using all of it (!!) to train its next models. Below is an image of one of the several Project Rainer data centers packed with Amazon GPUs:

Building data centers is incredibly capital-intensive: Amazon has spent $11B on Project Rainer alone. Even though very profitable, Amazon might want to invest more cash than it currently has into building data centers. So, one reason for job cuts could be to reallocate financial resources from paying salaries and compensation towards building more data centers.

Before doing the math, a couple of concepts are important to understand:

• Free cash flow (FCF): profit after payment for things like capital expenditure (CapEx), such as financing data centers. If a company wants to operate with as little debt as possible, FCF is usually very important. If Amazon wants to avoid loans and not touch its reserves, then data center investment would come from FCF, reducing FCF further.
• Cash reserves: a company’s liquid reserve investments, usually an accumulation of investments in financial instruments like bonds and securities, or cash deposits.

Let’s run Amazon’s numbers:

• Cash reserves: $93B. This is how much Amazon has in reserve.
• FCF: $32B. This is the rough free cash flow Amazon has currently, as per its latest quarterly report. This is after deducting current data center investments.
• Savings from layoffs: $2-4B. This is my estimate of the rough total compensation of 14,000 employees.

So, the savings from these layoffs wouldn’t even pay for half of Project Rainer ($11B in total), and Amazon could easily build 3x Project Rainers in the next year, without needing to dip into its savings! Of course, Amazon has its famous frugality principle, but this massive layoff of 14,000 people won’t make a big difference to how much it can invest in data centers; It can already spend much more, if it wants!

Leanness and AI fail job cuts “smell test”

It’s not only me who doesn’t buy the explanation that these layoffs are to streamline the company, or to redirect resources to AI. Arne Knudson worked at Amazon for nearly two decades, most recently as a software development manager (SDM), before leaving the company earlier this year. He shared his analysis, with some insider detail:

“In my 18 years at Amazon, I went through a few layoffs and hiring freezes.

This is the first time I’ve seen multiple years of significant layoffs essentially back-to-back. Even in the depths after the .com bubble, it wasn’t this bad. They’ve been laying people off now for almost 3 straight years.

The explanation that this is downsizing after hiring too many at the height of the pandemic doesn’t pass the smell test, at least to me. That was 3 years ago; they’re not that dumb to keep those people around for 3 extra years. Those folks were laid off back in ‘22.

I’m also not convinced that this is optimization due to AI. My degree’s in AI, and I worked on AI stuff at Amazon; I don’t think there’s enough automation yet, and it’s not accurate enough yet, to replace 30,000 people. The cost of inaccuracies seems too high. But I could be wrong; maybe they’ve gotten their false negative & false positive rates low enough to avoid too many region-wide AWS outages. (Or not.)

One of the articles I read said this was going to be in HR, and I can tell you as a former manager, my experience working with HR had been steadily worsening over the past 5-7 years. They outsourced so much of the work, overworked the people they had, and had such high turnover that I never knew who I was supposed to work with. When I needed to put someone on a performance plan or help a new hire receive some kind of accommodation, it seemed like it was a different person each time. If they really are laying off tens of thousands more HR folks, this is only going to get worse.

And, I suspect, it means they don’t plan on hiring MORE people in any of the business units for a year or more. So, by the smell-o-meter, this seems more significant than streamlining the workforce, improved AI, and “nah, we don’t need as many HR folks.”

US economy to blame for Amazon layoffs?

It’s safe to assume AWS as a business unit is doing just fine, as suggested by Project Rainer’s existence and the agenda for building data centers. But how is the e-commerce side of the business performing, and what’s its outlook?

If one business should have its finger on the pulse of the US economy, it’s Amazon with its size and self-professed, relentless customer focus, providing a window into people’s spending habits across the country. Flashing lights on the dashboard of the national economy may signal tough times ahead in e-commerce, which could be a reason to start cutting costs early.

There are concerning signs from other sectors about the US economy. Below is the CEO of the restaurant chain, Chipotle.

“Earlier this year, as consumer sentiment declined sharply, we saw a broad-based pullback in frequency across all income cohorts.

Since then, the gap has widened, with low to middle-income guests further reducing frequency. We believe that this guest with household income below $100,000, represents about 40% of our total sales. And, based on our data, they are dining out less often due to concerns about the economy, and inflation.

A particularly challenged cohort is the 25- to 35-year-old age group. We believe that this trend is not unique to Chipotle and is occurring across all restaurants as well as many discretionary categories. This group is facing several headwinds, including unemployment, increased student loan repayment and slower real wage growth. We tend to skew younger and slightly over-indexed to this group relative to the broader restaurant industry”.

Chipotle is saying that everyone is eating out less, particularly 25-35 year olds, because of inflation. If people spend less on Chipotle because of rising prices, then they may also spend less in other areas of their lives for the same reason, including on Amazon.

In the e-commerce supply chain, there’s evidence of this trend, which would mean delivery services like UPS have fewer parcels to deliver. Speaking of UPS, two days ago, it announced massive layoffs:

“United Parcel Service (UPS) has slashed 48,000 jobs this year — one of the largest single-year reductions by a US company since the pandemic — as the package giant scrambled to contain costs and revive its lagging stock price.

The Atlanta-based delivery behemoth disclosed the reductions Tuesday while reporting third-quarter earnings that beat Wall Street expectations.

UPS said 34,000 of the cuts hit drivers and warehouse operations, while 14,000 targeted management (...)

UPS shares jumped nearly 9% in Tuesday afternoon trading, even as the company reported weaker revenue and profits”.

UPS’s revenue is down on last year, which suggests that there are, indeed, fewer deliveries (or lower value ones). As with the latest job cuts at Amazon, these drastic layoffs could be explained by a lot of things, most easily by UPS expecting reduced trade in the future.

If US consumer spending is trending down, then the e-commerce sector will be among the first to feel this. It could explain why Amazon is making these layoffs now. It can also explain why Google, Meta, and Microsoft might not be seeing their businesses impacted: they’re not involved in retail like Amazon is, and the AI sector is very much booming.

Among all of Big Tech, Amazon is best positioned to detect changes in US consumer spending. Google’s and Meta’s revenue is more dependent on advertising, and Microsoft’s more on enterprise spend. Like Amazon, Apple is well placed to feel market changes with its range of smartphones and watches, and other consumer tech.

I believe Amazon is highly commercially rational, so it’s worth understanding the actual reason for its second major mass layoffs in just two years, following deep cuts in 2023. I’d put my money on this reason being the economy, and how Amazon probably expects customers to cut back their spending everywhere, including on Amazon.

This was one out of the four topics covered in last week’s The Pulse. Read the full article.

This week’s The Pulse additionally covers:

1. Cursor and GitHub double down on agents. Each company is focusing on agents: Cursor with its multi-agent mode, and GitHub with its “Agent HQ.” Cursor is increasingly a direct rival to GitHub.
2. Industry pulse. Meta rolls out AI-assisted interviews, Cursor and Windsurf believed to be using Chinese open source AI models, South Korean government pays price of ignoring backup “101,” startups growing much faster in the US than in Europe, companies using AI tools buy more JIRA seats, a neat uptime service called Updog, and more.
3. OpenAI inflating the bubble? OpenAI signs another massive deal with AWS based on predicted growth, and seeks taxpayer protection to borrow more.
4. Large tech companies struggle to build their AI integrations. Apple admits failure to modernize Siri by paying Google $1B per year for its LLM. Perplexity to pay Snap $400M to be its AI search interface.
5. How much do Directors of Engineering earn at startups? Data from Carta says it’s more than any other Director: $215-230K at companies valued at $25-250M.

Read the full issue here

Puneet Patwari recently accepted an offer to join Atlassian as a Principal Software Engineer. In three months, he did more than 60 interviews at 11 companies, he told me – while dropping out of 3 more interview processes after accepting the Atlassian offer, including that of Meta. Following that endeavour, he has compared the interview processes of the largest companies:

A few more observations that Puneet shared with me:

Amazon: the Amazon Hiring Manager round was one of the most unique I ever experienced. We got so engrossed in the discussion that it took 160 minutes instead of the scheduled 60 minutes! We had to take a break in between the interview process.

Atlassian: The leadership craft (LC) & values were two interview rounds which were very crucial in determining that I’ll be levelled at the Principal level. Of course, the Systems Design interview was also key here. Atlassian puts a lot of emphasis on LC for Principal engineers.

Salesforce: the system design round was based on the actual job requirement. It was a migration problem where the interviewer wanted to check if I can own a project end-to-end with customers at the centre of it.

Confluent: when I say it was the most mentally demanding interview, what I mean is how every skill was tested with two interviews! So 2x data structures and algorithms (DSA), 2x System Design 2x behavioural interview rounds.

I cannot stress enough how important behavioural interviews are at the Staff+ levels. Doing well on these interviews were decisive in getting Staff and Principal-level offers. Of course, you needed to do well on coding and systems design: but my sense was that the behavioural parts were make or break for levelling and getting an offer.

A few things stand out to me from Puneet’s account of his interviews at leading tech companies:

• Algorithmical coding interviews are everywhere! For senior+ positions, you need to get really good at these, including challenging topics like dynamic programming. We cover how to perform well in these in the article, How experienced engineers get unstuck in coding interviews
• Interviews are tough, and time consuming. Even after Puneet had offers, no company shortened their process. Puneet had to decline 3 more interviews – including one at Meta – because by the time the interviews would have come around, he already had an offer he had accepted at Atlassian.
• In a tough job market, “top” candidates are still in demand. We’ve covered how challenging the current tech labor market is for jobseekers, but Puneet interviewed at 11 companies and got 6 offers. His applications had to have a lot going for them in order to pass the resume screenings: 10+ years of experience, and working as a Senior Software Engineer at Microsoft. He also showed up really well prepared.
• Bad luck can strike at any time. Puneet’s interview experience at Uber seems to have been a bit unlucky: the interviewer presented as rigid and not open to dialogue. Perhaps they were having a tough day, or wanted to get the interview over with. Or it could be what Steve Yegge describes as the interviewer anti-loop

Congrats to Puneet for accepting the Atlassian position, and thanks for sharing all these learnings!

This was one out of five topics covered in The Pulse #149. The full edition additionally covers:

• New trend: programming by kicking off parallel AI agents. More devs are experimenting with kicking off coding agents in parallel
• ACP protocol. A new protocol built by the Zed team, which tries to make it easier to build AI tooling for IDEs than the MCP protocol allows
• AI security tooling works surprisingly well? AI-powered security tools seem good at identifying security flaws in mature open source projects
• Is AI the only engine of US economic growth? Forty percent of US GDP this year is based on AI-related spend, while 60% of venture capital goes into AI. Hopefully, it won’t end up as a bubble which bursts like in 2001

Read the full issue here, and check out today’s The Pulse here.

With agentic command line interfaces like Claude Code, OpenAI Codex, Cursor, and many others going mainstream, I’m seeing a trend of more software engineers experimenting with kicking off work with several agents simultaneously on separate tasks:

I talked with Anthropic engineer Sid Bidasaria about how Claude Code is built, and at the end of our conversation, he mentioned that he’d had a few agents running throughout and that it made him more productive with work. Similarly, software engineer Simon Willison, whom I consider an AI engineering expert, has posted about “embracing the parallel coding agent lifestyle.” He writes:

“For a while now, I’ve been hearing from engineers who run multiple coding agents at once—firing up several Claude Code or OpenAI Codex instances at the same time, sometimes in the same repo, sometimes against multiple checkouts or git worktrees.

I was pretty skeptical about this at first. AI-generated code needs to be reviewed, which means the natural bottleneck on all of this is how fast I can review the results. It’s tough keeping up with just a single LLM given how fast they can churn things out, where’s the benefit from running more than one at a time if it just leaves me further behind?

Despite my misgivings, over the past few weeks I’ve noticed myself quietly starting to embrace the parallel coding agent lifestyle.

I can only focus on reviewing and landing one significant change at a time, but I’m finding an increasing number of tasks that can still be fired off in parallel without adding too much cognitive overhead to my primary work.”

Simon shares advice about what works for him, with research, maintenance tasks, and directed work all mentioned as use cases.

It’s interesting to consider whether parallel work with agents has the potential to overturn decades of software engineering practices. Let’s assume software engineers who kick off multiple agents at once do become more productive than “single-threaded” peers who work on one problem at a time. If so, then this practice has a chance to spread, should enough software engineers seek to be more productive – or want to avoid being left behind by some colleagues doing more than before.

But engineering in the pre-AI era was all about being in the flow for many productive engineers. A flow state goes something like this:

• Understand the moving parts
• Build a solution, validate it, iterate on it
• When satisfied with how it works, submit a pull request for code review — or, if no review is needed, just merge and ship it

Interrupting this process disrupts the flow state, and it takes time to get back into it: it’s why software engineers tend to prioritize focus time, to make progress with coding work.

Of course, this isn’t universal among all highly productive engineers; when I was an engineering manager, the most productive engineers on my team did a lot of context switching and were adept at juggling several things at once. Here’s an average-looking day for a senior engineer acting as a tech lead:

• Code reviews. Arrive at office, go through open code reviews from the previous night
• Coding. Get some of their own coding work done
• Standup. The usual
• More coding. Get the work done. At least, that’s the idea. In reality:
• Interruptions: code reviews, requests for help, taps on shoulder. The most productive engineer on a team regularly gets messages requesting code reviews to unblock teammates, or to help someone else who’s stuck, or the manager (me – sorry!) tapping them on the shoulder for help with something.

I wonder if senior+ engineers will be “naturals” at working with parallel AI agents, based on their existing habits and what they do currently:

• Keep parallel workflows in their heads; e.g., what team members are doing at any one time.
• Code reviews across several workstreams: they’re the go-to code reviewer, and usually review all code changes across 2-5 workstreams. They may not do the work, but know when it’s correct.
• Can handle interruptions: they’ve learned how to make progress when their focus is continually being broken.
• Good at directing colleagues: because they’re regularly interrupted, they’ve also learned how to delegate and explain urgent work to team members.
• Writing skill: these engineers write a lot of code reviews, draw up documents like RFCs that outline work, create tickets to break down projects, and critique colleagues’ efforts; all this involves communicating effectively in writing.

With AI agents, the qualities that make a good tech lead are within reach for engineers who want to be more productive. So far, the only people I’ve heard are using parallel agents successfully are senior+ engineers.

Then again, this workflow hasn’t stuck with everyone: I asked Flask creator Armin Ronacher about his experience with parallel agents. He told me:

“I sometimes kick off parallel agents, but not as much as I used to do.

The thing is: it’s only so much my mind can review!”

But we’re in new territory now that any dev can kick off parallel coding with coding agents. Will it make engineers more productive, or will it just make people feel like they’re more productive? Perhaps engineers who do one thing at a time and keep focus will be shown to produce more reliable software, over time. Or maybe it’ll turn out that working with parallel agents leads to more issues slipping through and more iterations, which destroys any gains.

We will find out. Personally, I can only see more devs experimenting with parallel agents.

My sense is that software engineering basics matter more when working with AI agents. I’ve started to use AI agents for my own side projects, with success so far. I do a few things:

• Testing: all side projects have unit tests because I learned to not trust my own work without validation
• Small, descriptive tasks: I give tasks small enough in scope, which I explain, and give examples of
• Refactoring: every third or fourth task is for the agent to refactor some code they wrote (e.g., extract into a method, move to a new class)
• Review: I track what the agent does
• Do small things personally: I keep my IDE open and do anything that’s a few lines to change by hand, so I stay aware of the codebase

I keep hearing the same from other engineers: “mandating” engineering practices like having the agent pass all tests before continuing, leads to better results. This is unsurprising and it’s why these practices are getting popular. AI agents are non-deterministic and to some extent unreliable; these practices make them a lot more reliable and usable.

This was one out of the five topics covered in The Pulse #149. The full edition additionally covers: The full issue additionally covers:

• ACP protocol. A new protocol built by the Zed team, which tries to make it easier to build AI tooling for IDEs than the MCP protocol allows
• AI security tooling works surprisingly well? AI-powered security tools seem good at identifying security flaws in mature open source projects
• Is AI the only engine of US economic growth? Forty percent of US GDP this year is based on AI-related spend, while 60% of venture capital goes into AI. Hopefully, it won’t end up as a bubble which bursts like in 2001
• Comparing interviews at 8 large tech companies. Puneet Patwari applied to 8 major tech companies, and received 6 offers. He compares his interview experiences at Meta, Amazon, Uber, and 5 other workplaces

Read the full issue here, and check out today’s The Pulse here.

Monday was an interesting day: Signal stopped working, Slack and Zoom had issues, and most Amazon services were also down, together with thousands of websites and apps, across the globe. The cause was a 14-hour-long AWS outage in the us-east-1 region.

Today, we look into what caused this outage.

To its credit, AWS posted continuous updates throughout the outage. Three days after the incident, they released a detailed postmortem – much faster than the 4 months it took in 2023 after a similarly large event.

The latest outage was caused by DynamoDB’s DNS failure. DynamoDB is a serverless NoSQL database built for durability and high availability, which promises 99.99% uptime as its service level agreement (SLA), when set to multi-availability zone (AZ) replication. Basically, when operated in a single region, DynamoDB promises – and delivers! – very high uptime with low latency. Even better, while the default consistency model for DynamoDB is eventual consistency (reads might not yet reflect the actual status), reads can also be set to use strong consistency (guaranteed to return the actual status).

All these traits make DynamoDB an attractive choice for data storage for pretty much any application, and many of AWS’s own services also depend heavily on DynamoDB. Plus, DynamoDB has a track record of delivering on its SLA promises, so the question is often not why to use DynamoDB, but rather, why not to use this highly reliable data storage. Potential reasons for not using it include complex querying, complex data models, or storing large amounts of data when storage costs are not worth it compared to other bulk storage solutions.

In this outage, DynamoDB went down, and the dynamodb.us-east-1.amazonaws.com address returned an empty DNS record. To every service – external to AWS or AWS internal – it seemed like DynamoDB in this AWS region disappeared off the face of earth! To understand what happened, we need to look into DynamoDB DNS management.

How DynamoDB DNS management happens

Here’s an overview:

How it works:

• DNS planner: this service monitors load balancer (LB) health. As you can imagine, DynamoDB runs at a massive scale, and LBs can easily get overloaded or under-utilized. When overloading happens, new LBs need to be added, and when there’s underutilization, they need to be removed. The DNS planner created DNS plans. Each DNS plan is a set of LB sets, and assigning them weights on how much traffic to give the LB.
• DNS enactor: the service responsible for updating the routes in Amazon’s DNS service called Route 53. For resiliency, there is one DNS enactor running in each availability zone (AZ). In us-east-1, there are 3 AZs, and 3x DNS Enactor instances.
• Race conditions are expected: with 3x parallel DNS Enactors working simultaneously, race conditions are expected. The system deals with this by assuming eventual consistency: even if a DNS Enactor updates Route 53 with an “old” plan, DNS plans are consistent with one another. Plus, updating happens quickly, and DNS Enactors only use the latest plans from the DNS Planner.

DynamoDB down for 3 hours

Several independent events combined to knock DynamoDB’s DNS offline:

1. High delays on a DNS Enactor #1. Updating DNS took unusually long for one DNS Enactor for some reason. Usually, these updates are rapid, but weren’t on 20 October.
2. DNS Planner turns up the pace in churning out DNS plans. Just as DNS updates turned slow, the DNS planner started to produce new plans at a much higher pace than before.
3. DNS Enactor #2 rapidly processes DNS plans. While DNS Enactor #1 was applying DNS plans at snail’s pace, DNS Enactor #2 was storming through them. As soon as it finished writing these plans to Route 53, it went back to DNS Planner and deleted the old plans.

These three things pushed the system into an inconsistent state and emptied out DynamoDB DNS:

1. DNS Enactor #1 unknowingly uses an old DNS plan. When DNS Enactor #2 finished applying the newest DNS plan, it went back to DNS Planner and deleted all older plans. Doing so should have meant that other DNS Enactors did not use old plans; but remember, DNS Enactor #1 was slow and still processing through an old plan! As a result, the check by DNS Enactor #1 was stale.
2. DNS Enactor #2 detects the old plan being used and clears DNS records. DNS Enactors have another cleanup check: if they detect an old plan being used, they delete the plan itself. Deleting a plan means removing all IP addresses for the regional endpoints in Route 53. So DNS Enactor #2 turned the dynamodb.us-east-1.amazonaws.com DNS empty!

DynamoDB going down also took down all AWS services dependent on the us-east-1 DynamoDB services. From the AWS postmortem:

“All systems needing to connect to the DynamoDB service in the N. Virginia (us-east-1) Region via the public endpoint immediately began experiencing DNS failures and failed to connect to DynamoDB. This included customer traffic as well as traffic from internal AWS services that rely on DynamoDB. Customers with DynamoDB global tables were able to successfully connect to and issue requests against their replica tables in other Regions, but experienced prolonged replication lag to and from the replica tables in the N. Virginia (us-east-1) Region.”

The DynamoDB outage lasted around 3 hours; I can only imagine AWS engineers scratching their heads and wondering how the DNS records were emptied. Eventually, engineers manually intervened, and brought back DynamoDB. It’s worth remembering that bringing up DynamoDB might have included avoiding the thundering herd issue that is typical of restarting large services.

To be honest, I’m sensing key details were omitted from the postmortem. Things unmentioned which are key to understanding what really happened:

• Why did DNS Enactor #1 slow down in updating DNS, compared to DNS Enactor #2?
• Why did DNS Enactor #2 delete all DNS records as part of cleanup? This really made no sense, and it feels like there’s an underlying reason.
• The race condition of one DNS Enactor being well ahead of others seems to be easy enough to forecast. Was this the first time it happened? If not, what happened after previous, similar incidents?
• Most pressingly, how will the team fix this vulnerability which could happen anytime in the future?

Amazon EC2 down for 12 more hours

With DynamoDB restored, the pain was still not over for AWS. In fact, Amazon EC2’s problems just got worse. To understand what happened, we need to understand how EC2 works:

• DropletWorkflow Manager (DWFM) is the subsystem that manages physical servers for EC2. Think of it as the “Kubernetes for EC2.” EC2 instances are called “droplets.”
• Lease: DropletWorkflow Manager tracks the lease for each droplet (server) so it knows if and when a server is occupied, or can be allocated to an EC2 customer. The DWFM does a status check of the server every few minutes to determine its state.

State check results are stored in DynamoDB, so the DynamoDB outage caused problems:

1. Leases started to time out. With state check results not returning due to the DynamoDB outage, the DropletWorkflow Manager started to mark droplets as not available.

2. Insufficient capacity errors on EC2: with most leases timed out, DWFM started to return “insufficient capacity error” messages to EC2 customers. It thought servers were not available, after all.

3. DynamoDB’s return didn’t help: when DynamoDB came back online, it should have been possible to update the status of droplets. But that didn’t happen. From the postmortem:

“Due to the large number of droplets, efforts to establish new droplet leases took long enough that the work could not be completed before they timed out. Additional work was queued to reattempt establishing the droplet lease. At this point, DWFM had entered a state of congestive collapse and was unable to make forward progress in recovering droplet leases.”

It took engineers 3 more hours to come up with mitigations to get EC2 instance allocation working again.

Network propagation errors took another 5 hours to fix. Even when EC2 was looking healthy on the inside, instances could not communicate with the outside world, and congestion built up inside a system called Network Manager. Also from the postmortem:

“Network Manager started to experience increased latencies in network propagation times as it worked to process the backlog of network state changes. While new EC2 instances could be launched successfully, they would not have the necessary network connectivity due to the delays in network state propagation. Engineers worked to reduce the load on Network Manager to address network configuration propagation times and took action to accelerate recovery. By 10:36 AM PT [11 hours after the start of the outage], network configuration propagation times had returned to normal levels, and new EC2 instance launches were once again operating normally.”

Final cleanup took another 3 hours. After all 3 systems fixed – DynamoDB, EC2’s DropletWorkflow Manager and Network Manager – there was a bit of cleanup left to do:

“The final step towards EC2 recovery was to fully remove the request throttles that had been put in place to reduce the load on the various EC2 subsystems. As API calls and new EC2 instance launch requests stabilized, at 11:23 AM PDT [12 hours after the outage started] our engineers began relaxing request throttles as they worked towards full recovery. At 1:50 PM [14 hours after the outage started], all EC2 APIs and new EC2 instance launches were operating normally.”

Phew – that was a lot of work! Props to the AWS team for working through it all in what must have been a stressful night’s work. You can read the full postmortem here, which details the impact on other services like the Network Load Balancer (NLB), Lambda functions, Amazon Elastic Container Service (ECS), Elastic Kubernetes Service (EKS), Fargate, Amazon Connect, AWS Security Token Service, and AWS Management Console.

This was one out of four topics from today’s The Pulse, analyzing this large AWS outage. The full issue additionally covers:

1. Worldwide impact. From Ring cameras, Robinhood, Snapchat, and Duolingo, all the way to Substack – sites and services went down in their thousands.
2. Unexpected AWS dependencies. Status pages using Atlassian’s Statuspage product could not be updated, Eight Sleep mattresses were effectively bricked for users, Postman was unusable, UK taxpayers couldn’t access the HMRC portal, and a Premier League game was interrupted.
3. Why such dependency on us-east-1? It feels like half of the internet is on us-east-1 for its low pricing and high capacity. Meanwhile, some AWS services are themselves dependent on this region.

Read the full article here.

Open source maintenance fee: an interesting experiment

Wix Toolset is a set of open source tools for creating Windows installers. As with most open source projects, the long-term financial sustainability of the project was uncertain: who would pay for core maintainers to spend time on this project, rather than on their work or other side projects? A project like Wix needs to be continuously updated to support new versions of Windows, and security and other issues must be responded to by maintainers, who also review pull requests.

The team decided to adopt a relatively new concept called the Open Source Maintenance Fee. How it works:

• The code is freely available under an open source license
• Support: paid. Opening issues or commenting on them, or participating in releases or pull requests requires paying the fee
• Binary releases: paid. To download any binary release of the project, one needs to pay the fee. But the binary can be compiled from the code - one just has to build it themselves.

Basically, if an individual or business uses Wix Toolset as part of its revenue-generating activity, and someone from that company wants to ask questions or submit changes, or download a pre-built binary, then it’s necessary to become a sponsor and pay:

• $10/month when working at a company up to 20 people
• $40/month for companies of 20-100 employees
• $60/month for companies with 100+ employees

So far, the fee seems to be working: The project currently has 64 sponsors, including Microsoft (which needs to pay the $60/month fee). This could be generating anything from $640/month up to a few thousand dollars per month, which can be used to cover costs, such as compensating core contributors. As maintainer Rob Menshing elaborated:

“The maintenance fee is intended to go to those doing maintenance. In my project, there are two of us working to keep the project running. We have contributors who do stuff they find interesting, but none of them do ‘boring but important stuff’.

So far, it is working.”

What I like about this structure is that it removes the burden on maintainers who invest their time, and turns it into an incentive for commercial users to pay for their time.

The idea of the fee came after observing the XZ Utils supply chain attack incident. As Rob shared on the backstory of the Open Source Maintenance Fee:

“I touched on it in the video but I felt compelled to do something about the sustainability problems facing Open Source Maintainers. My personal frustration dealing with entitled consumers for the last few years—and watching other maintainers go through the same thing—hit a breaking point with the XZ Utils incident.

Two things happened in that incident. First, we saw a maintainer—who was vulnerable due to the lack of project sustainability—manipulated by attackers to sneak a backdoor into Linux… and then nothing changed. Second, in my viral tweet-thread everyone agreed that something should be done. I’ve never seen that many people on the internet agree about anything.

The cognitive dissonance that something should be done but nothing was done, fundamentally bothered me. I couldn’t stop thinking about the problem. It wasn’t until early July that the Maintenance Fee idea really started to come together.”

And you know what - I agree! As an OSS maintainer, it can feel that users are entitled: demanding a lot, but offering nothing in return. More than a decade ago, I also burnt out maintaining a Windows Phone library called AdRotator that helped users generate revenue for their apps. I released my library thinking it will help others make more money, and everyone will sort out their own issues. I was not expected to see demands coming from users that I implement new features, or fix issued they are seeing. Back then, I was relieved to hand over maintenance to a fellow contributor who had far more energy to support users with questions and feature requests.

I naively assumed open source was about offering the source as open, and everyone taking it, and sorting out their issues by themselves. I was not prepared that by releasing something as open source, I’d get a bunch of maintenance burden to have to deal with — something I never signed up for or asked for.

I wonder if GitHub’s popularity means more projects will implement a similar fee. GitHub has become the de facto place for open source projects to live: its simple, intuitive UI makes it very easy to open an issue and raise a pull request.

However, open source maintainers of more popular projects face a steady stream of often low-quality issues raised, and pull requests which aren’t ready to be merged. Meanwhile, users expect responses and can get frustrated when issues are not quickly resolved.

For open source projects with significant commercial usage, an open source maintenance fee could be a viable way to reduce this kind of noise, and generate some revenue for the time maintainers spend on them. For example, the Wix Toolset project currently has 775 open issues (!) and another 6,500 closed ones. Just keeping on top of the issues raised looks like a lot of work!

For projects that do not use GitHub, the barrier to entry for creating issues and pull requests is a lot higher. For example, Linux still runs on email mailing lists, and patches need to be sent via email. To contribute to Linux, a lot of upfront work is necessary first. We cover more about contributing to Linux in the episode of the Pragmatic Engineer Podcast, How Linux is built with Greg Kroah-Hartman.

A fee like this could help keep open source projects as free and open source software (FOSS). Someone might argue that the existence of a fee goes against the concept of “free”. However, the “free” in FOSS stands for the freedom to:

1. Run the program as you wish for any purpose (you can do this!)
2. Study how the program works and modify it as you wish (you can do this; just get the code and change it, or fork it!)
3. Redistribute copies to help others (you can do this)
4. Distribute copies of modified versions (again, you can do this)

FOSS shouldn’t mean that someone works for free in replying to issues and responding to change requests. In the case of Wix Toolset, maintainer Rob Menshing worked hard with lawyers to ensure that this Open Source Maintainer Fee is fully compliant with FOSS principles and expectations. As Rob put it:

“OSS does not mean that everything is available for no cost.”

The more open source projects on GitHub are sustainable, the better it will be for the tech ecosystem. An open source project that is sustainable long-term – with maintainers having a reasonable workload, keeping up with comments, and releasing new versions when needed – is a lot better than one that puts up no such barriers and leads maintainers to throw in the towel and quit.

Don’t forget, any company or individual who does not want to pay this fee but does want to make modifications to the code can go ahead and do it! They just need to create their fork and then can go ahead with the modification. The cost of keeping this fork up-to-date with the project then becomes theirs to pay and it could be more expensive than the $10-60/month fee asked for by the project.

When introducing the fee, the Wix Toolset project had a long discussion; read the thread for more details of the thinking behind this move.

Enterprise-only features: how uv’s creator plans to make money

In the Python world, the uv package manager is surging in popularity: it is ultra-fast and 10-100x speedier than the formerly popular package manager, pip. Part of this speed is thanks to uv being written from scratch, in Rust, for performance. It does lots of clever things like parallel downloading and installing of packages, and maintaining an optimized module cache. uv is so useful that an AI startup told me that moving over to this package manager improved productivity more than any AI tool they trialed. From Software engineering with LLMs in 2025, reality check: 

“An interesting detail emerged when I asked how they would compare the impact of AI tools to other innovations in the field. This engineer said that for their domain, the impact of the uv project manager and ruff linter has been greater than AI tools, since uv made their development experience visibly faster!

uv is built by a startup called Astral that has raised $4M in seed funding in 2023. With VC funding, Astral has to generate revenue, but how to do this with a free open source package manager? 

We now have an answer:

Astral created a private package registry called pyx, a paid-enterprise focused package registry. It’s like uv, but with additional features for security and GPU support. Astral has signed up companies like Ramp and Intercom as customers already.

This approach of charging for enterprise features is clever and great news for the open source community: if Astral can get traction for pyx, they could have a business offering standout tools for free to individual developers, and also more advanced paid versions with enterprise features for companies. We have covered the pressure on commercial open source companies to make money, so fingers crossed that Astral succeeds!

This was an excerpt from The Pulse #143. The full issue additionally covers:

1. Industry pulse. Microsoft’s US compensation bands revealed, a reader gets in touch about Forward Deployed Engineers, why OpenAI has released an open model, Canva trials AI coding tools for interviews, and more.
2. Tailwind CSS team burnt by LLM’s inconsistent code generation. A project estimated to take 6 weeks without AI, took 12 weeks with Claude Code.
3. Meta and OpenAI in talent & compensation war. 15 years after aggressively poaching engineers from Google, Meta is doing the same to OpenAI, which is responding with six and seven-figure retainer bonuses.
4. What people think improves AI applications vs what actually does. Instead of keeping up with the latest AI headlines, why not talk to users? Author Chip Huyen has practical tips for AI engineers.

Read the full issue here

Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of four topics from this week’s The Pulse issue. To get articles like this in your inbox, every week, subscribe here.

“996” stands for “from 9am to 9pm, 6 days a week”, and used to be a common work pattern at Chinese tech companies until it was officially banned starting from 2021. Such extreme working hours have long been rejected in the US. It is even outlawed in Europe because excessive hours tend to lead to burnout and other health issues, longer term.

Despite that, more AI startups are adopting something similar to the 996 work pattern, including Cognition, which expects staff to put in 80+ hours per week. Indeed, the CEO, Scott Wu, was unapologetic about the company’s hardcore culture in a post he shared:

“People have asked about our culture and recent employee communications. Cognition has an extreme performance culture, and we’re upfront about this in hiring so there are no surprises later.

We routinely are at the office through the weekend and do some of our best work late into the night. Many of us literally live where we work.

We know that people who joined Windsurf didn’t expect to join Cognition and while we’re proud of how we work, we understand it’s not for everyone”.

There are several other cases of AI startups mandating grueling hours for workers:

• Lovable: job descriptions by the company detail “Long hours, high pace. Candidates must thrive under high urgency, with AGI timelines approaching.”
• Replit: CEO, Amjad Masad, posted photos of the whole team grinding in the office at midnight, preparing to ship something big.
• xAI: Zeeshan Pate, an engineer at Elon Musk’s AI startup, shared that the team was “grinding in the office day and night”.
• CodeRabbit: just today, the team was still shipping at 2:30am in the San Francisco headquarters
• Icon (ad maker): the founder and CEO shared that the business “only hires [the] top 0.01% engineers with no life”, and expects them to work all week: “why do 6 [days of work] when you can do 7”.
• Google’s AI unit: Google cofounder Sergey Brin told staff in the tech giant’s AI unit: “60 hours a week is the sweet spot of productivity”.

These extreme hours are justified by a sprint to achieve Artificial General Intelligence (AGI) in just a matter of months. That’s because plenty of AI professionals believe that when this point is reached, it will be “game over” for most companies in the segment, with a new, solidified, status quo in place, and AGI being able to improve itself with more resources. From then on, companies with AGI will dominate the industry. This is the incentive for using every means possible (including employees’ labor) to get to AGI – and quickly! Personally, I don’t buy this simplistic prediction about AGI and what might happen when or if it’s reached, but it is the driving force behind the thinking of many founders with commercial pressures.

It’s nearly three years since ChatGPT was released, and there are still no signs of AGI, even though LLMs continuously improve. But what about the exhausting work patterns that are meant to be in place for a few months; could they stay in place for years, and become standard?

Asking or demanding staff to put in very long hours is a recipe for making the pace of work slow down, and for individuals to burn out. We live in an economic system in which companies try to “extract” as much as possible from employees across all industries, so why don’t employers in other sectors also make staff work 80-hour weeks?

In many countries, regulations mandating sensible working conditions are one reason, and unions advocate for this. Another is that the downside of long working hours soon becomes visible:

• Staff take more time off for sickness
• Productivity drops
• More employees quit for places with shorter working weeks

Plus, mandating long working hours automatically excludes many strong potential candidates who…

• have family duties outside of working hours
• live further from the office, and for whom the commute would be too long
• prioritize personal, non-work, activities outside of core hours

Of course, if you hire young professionals, burnout won’t occur so fast – and some people can work for years like this. Plus, if they don’t have families or a busy social life, they may be comfortable with working what looks like “crazy” hours.

Another powerful incentive for an AI startup to create a long-hours culture: the promise of generational wealth. Consider these two questions:

1. Would you work 6 days a week at a startup, and pull 80+ hours per week indefinitely, for less than you’d earn at a company with a 40-hour work week?
2. Would you work 6 days a week and put in 80+ hour weeks for 1-3 years, then walk away with $10M in compensation?

The answer to #1 is likely “no”, but the answer to #2 is “obviously yes” for most people! And the incredible growth in the AI industry means that #2 seems achievable to many. Take Windsurf: just 10 months after launching the Windsurf IDE, 40 employees from the team were acquired by Google. The founders likely made hundreds of millions, and some engineers may have made $10M+ in compensation! That’s not bad for a few years’ work!

Now, put yourself in the shoes of founders who are set to make not $10M, but a multiple of that. For them, it was absolutely worth putting in the long hours and pushing their team to do the same.

Based on that, we can expect founders to keep pushing staff to spend every waking moment at work, or thinking about work.

While there's the promise of making it big with AI startups, these working hours will likely stay. Within AI startups, founders face pressure to ship fast or be out-executed by rivals. Speed is essential to win in AI, time-to-market is essential, and the most obvious way to attempt to get things done faster is to push people to work more.

For the importance of time to market: just look at startups like Magic.dev. A year ago, it raised $515M in funding and claimed it could support 100M token context windows, at a time when most models could not even support 100K. That meant Magic’s model would be a 1,000x improvement on mainstream LLMs! However, a year later and Google’s Gemini already supports 2M tokens, and Claude added support for 1M tokens. So Magic’s lead is cut to 50-100x – which is still considerable, but much reduced.

At the same time, there have not been updates from Magic, and this 100M context window model is not publicly available for use. If the company does not ship a public-facing product soon, they could see mainstream LLMs catch up in context window length, and thereby lose most of their potential customer base in the dev market.

Still, right now there’s a massive business opportunity to make lots of money with AI products, and for early employees to create generational wealth via generous equity – if their startup executes well, and is later acquired for a huge sum.

But long hours alone don’t guarantee success, and there are signs of this. Cognition is proud of its “extreme performance culture”, but its office culture hasn’t quite led to business success: Devin was one of the least-referenced AI tools in The Pragmatic Engineer 2025 survey.

Another sign that grueling hours don’t automatically generate success is the ad maker, Icon. Despite demanding 7-day working weeks from staff, the company seems to be pivoting to become just another advertising agency, offering to create unlimited versions of ads for $1,000/month. Ad agencies are pretty good lifestyle businesses, but rarely the kind of high-growth ventures valued in the billions!

I expect that things will eventually return to normal in a few years' time, when every tech company will also be an “AI company” and it will be business as usual. But for now, the fear of missing out (FOMO) is so strong across the industry that very long workweeks look set to spread.

Everyone’s situation is different, so figure out if you want or need to work extreme hours. But there will be more of a push from AI startups to weed out applicants who are resistant to very long work weeks. Accepting such a position can be an amazing career boost; yes, long hours have many downsides, but you “gel” better with colleagues, and relationships forged under pressure can last decades. Of course, all the stress can also lead to burnout and health issues.

In the end, just know that this trend is real and likely to stick around. If your workplace employs a more “normal” work pattern, it’s worth knowing that this is not necessarily something you can take for granted across tech!

Long hours are not a new thing in tech, and could be just be as much about excitement about LLMs. A now retired software engineer told me how the above summary brought back memories to her of doing pretty much the same, back 50 years ago:

“Thank you for that brief trip down memory lane. Of course this, current generation is working exhausting hours, this technology is new and exciting.

My generation did the same thing with computers back in the 1970s....

Storing cases of beer in the cooling system beneath the raised tiles.

Eating burgers at 7:00 am having pulled an all nighter after system crashes.

I suppose it is all new to you youngers!”

This was one out of four topics from this week's The Pulse issue. Read the full article here.

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

An interesting study has been published by the nonprofit org, Model Evaluation and Threat Research (METR). They recruited 16 experienced developers who worked on large open source repositories, to fix 136 real issues, for pay of $150/hour. Some devs were assigned AI tools to use, and others were not. The study recorded devs’ screens, and then examined and analyzed 146 hours of footage. The takeaway:

“Surprisingly, we find that when developers use AI tools, they take 19% longer than without. AI makes them slower. (...) This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.”

This result is very surprising! But what is going on? Looking closely at the research paper:

The research is about Cursor’s impact on developer productivity. The AI tool of choice for pretty much all participants was Cursor, using Sonnet 3.5 or 3.7. A total of 44% of developers had never used Cursor before, and most others had used it for up to 50 hours.

Those using AI spent less time on coding to complete the work – but took more time, overall. They also spent less time on researching and testing. But they took longer on promoting, waiting on the AI, reviewing its output, and on “IDE overhead”, than those not using AI. In the end, additional time spent with the AI wiped out the time it saved on coding, research, and testing, the study found.

It’s worth pointing out that this finding applies to all AI tools, and not only to Cursor, which just happens to be the tool chosen for this study.

Developers are over optimistic in their estimates about AI’s productivity impact – initially, at least. From the survey:

“Both experts and developers drastically overestimate the usefulness of AI on developer productivity, even after they have spent many hours using the tools. This underscores the importance of conducting field experiments with robust outcome measures, compared to relying solely on expert forecasts or developer surveys.”

The one dev who had used Cursor for 50+ hours saw a lot of speedup! In the study, there was a single developer who had used Cursor for a total of more than 50 hours, previously. This dev saw a very impressive 38% increase in speed. Then again, a sample size of one is not very representative of a group of 16:

Software engineer Simon Willison – whom I consider an unbiased expert on AI dev tools – interprets the survey like this:

“My intuition here is that this study mainly demonstrated that the learning curve of AI-assisted development is high enough that asking developers to bake it into their existing workflows reduces their performance while they climb that learning curve.”

Indeed, he made a similar point on an episode of the Pragmatic Engineer podcast: “you have to put in so much effort to learn, to explore and experiment, and learn how to use it. And there's no guidance.”

In research on AI tools by this publication, based on input from circa 200 software engineers, we found supporting evidence of that: those who hadn’t used AI tools for longer than 6 months were more likely to have a negative perception of them. Very common feedback from engineers who didn’t use AI tooling was that they’d tried it, but it didn’t meet expectations, so they stopped.

The engineer who saw a 38% “speed-up” versus non-AI devs has an interesting take. That lone engineer with 50+ hours of Cursor experience is PhD student, Quentin Anthony. Here’s what he says about the study, and how AI tools impact developer efficiency:

“1. AI speedup is very weakly correlated to anyone's ability as a dev. All the devs in this study are very good. I think it has more to do with falling into failure modes, both in the LLM's ability and the human's workflow. I work with a ton of amazing pretraining devs, and I think people face many of the same problems.

We like to say that LLMs are tools, but treat them more like a magic bullet.

Literally any dev can attest to the satisfaction of finally debugging a thorny issue. LLMs are a big dopamine shortcut button that may one-shot your problem. Do you keep pressing the button that has a 1% chance of fixing everything? It's a lot more enjoyable than the grueling alternative, at least to me.

2. LLMs today have super spiky capability distributions. I think this has more to do with:what coding tasks we have lots of clean data forwhat benchmarks/evals LLM labs are using to measure success.

As an example, LLMs are all horrible at low-level systems code (GPU kernels, parallelism/communication, etc). This is because their code data is relatively rare, and evaluating model capabilities is hard (I discuss this in more detail here).

Since these tasks are a large part of what I do as a pretraining dev, I know what parts of my work are amenable to LLMs (writing tests, understanding unfamiliar code, etc) and which are not (writing kernels, understanding communication synchronization semantics, etc). I only use LLMs when I know they can reliably handle the task.

When determining whether some new task is amenable to an LLM, I try to aggressively time-box my time working with the LLM so that I don't go down a rabbit hole. Again, tearing yourself away from an LLM when "it's just so close!" is hard!

3. It's super easy to get distracted in the downtime while LLMs are generating. The social media attention economy is brutal, and I think people spend 30 mins scrolling while "waiting" for their 30-second generation.

All I can say on this one is that we should know our own pitfalls and try to fill LLM-generation time productively:If the task requires high-focus, spend this time either working on a subtask, or thinking about follow-up questions. Even if the model one-shots my question, what else don't I understand?If the task requires low-focus, do another small task in the meantime (respond to email/slack, read or edit another paragraph, etc).

As always, small digital hygiene steps help with this (website blockers, phone on dnd, etc). Sorry to be a grampy, but it works for me :)”

Quentin concludes:

“LLMs are a tool, and we need to start learning its pitfalls and have some self-awareness. A big reason people enjoy Andrej Karpathy's talks is because he's a highly introspective LLM user, which he arrived at a bit early due to his involvement in pretraining some of them.

If we expect to use this new tool well, we need to understand its (and our own!) shortcomings and adapt to them.”

I wonder if context switching could become the Achilles Heel of AI coding tools. As a dev, the most productive work I do is when I’m in “the zone”, just locked into a problem with no distractions, and when my sole focus is work! I know how expensive it is to get back into the zone after you fall out of it.

But I cannot stay in the zone when using a time-saving AI coding tool; I need to do something else while code is being generated, so context switches are forced, and each one slows me down. It’s a distraction.

What if the constraint of being “in the zone” when writing code is a feature, not a bug? And what if experienced devs not using AI tools outperform most others with AI because they consciously stay in “the zone” and focus more? Could those without AI tools have been “in the zone” and working at a higher performance level than devs forced into repeated context switches by their AI tools?

There’s food for thought here about how time saved on coding doesn’t automatically translate into higher productivity when building software.

This was one out of four topics from last week's The Pulse. The full issue also covers:

1. Industry pulse. Why 1.1.1.1 went down for an hour, Microsoft cut jobs to buy more GPUs, Meta’s incredible AI data center spend, and the “industry-wide problem” of fake job candidates from North Korea.
2. Windsurf sale: a complicated story of OpenAI, Microsoft, Google, and Cognition. OpenAI wanted to buy Windsurf but couldn’t because of Microsoft. Google then hired the founders and core team of Windsurf, and Cognition (the maker of Devin) bought the rest of the company. It’s a weird story that could not happen outside of California – thanks to California having a ban on noncompetes.
3. Beginning of the end for VC-subsidized tokens? Cursor angered devs by silently imposing limits on its “unlimited” tier. Us devs face the reality that LLM usage is getting more expensive – and VC funding will probably stop subsidizing the real cost of tokens.

This week's The Pulse issue – sent out to full subscribers – covers:

1. Mystery solved about the cause of June 10th outages. Heroku went down for a day due to an update to the systemd process on Ubuntu Linux. Turns out that dozens of other companies including OpenAI, Zapier, and GitLab, were also hit by the same issue, with outages of up to 6 hours.
2. Replit AI secretly deletes prod – oops! Cautionary tale of why vibe-coding apps are not yet production-ready, which makes it hard to foresee production-hardened apps being shipped with no software engineering expertise involved.
3. Industry pulse. Fresh details about the Windsurf sale, Zed editor allows all AI functionality to be turned off, government agencies using Microsoft Sharepoint hacked, GitHub releases vibe coding tool, and more.
4. Reflections on a year at OpenAI. Software engineer Calvin French-Owen summarized his impressions of OpenAI, sharing how the company runs on Slack and Azure, capacity planning challenges for OpenAI Codex launch, learnings from working on a large Python codebase, and more.

Read The Pulse issues here.