Both approaches are valid, and which approach a team takes is balanced against the incentives the team operates in. For a Helpdesk team, which is often under-staffed, throughput tends to be highly prioritized. For a Dev team with on-call responsibilities, growth tends to be the norm to make sure problems can be handled by anyone and to stop burning out your senior talent. Security teams tend to be a blend of both; with ticket-based security reviews urging throughput mindsets, where incident-response prioritizes growth. For Platform teams that tend to be involved in a lot of incidents due to running the systems that problems happen on, even if the problem wasn't actually the platform system, throughput mindsets are hard to avoid.
Now add coding LLMs to the mix.
The on-label reason to use agents such as Claude is throughput: spend less time working on tasks to improve your throughput. For Engineering Directors this is a great thing, because you can get throughput advances in your reporting teams without sacrificing engineering maturity, letting you lean on the product roadmap a little harder.
I'd argue that the throughput nature of LLM agents actually sacrifices some growth at population-scale when used in currently typical tech-companies. By moving from a purely generative mindset when building code to a revisionary mindset, going from writing code to reviewing and editing generated code, engineers spend less time learning problem-spaces in detail. Less time learning means taking more calendar time building up true domain knowledge. Coding agents are a somewhat different thing than the decades of "just add another abstraction layer and forget about the lower level details" we've been doing since 1970. It is true that most engineers in SaaS companies aren't spending lots of time hand-tuning assembly for execution on their ISA, and doing so is actually very bad practice unless you're in extremely specific problem domains. The difference between yet another abstraction and coding agent is the difference between abstraction and synthesis.
Abstraction is a distillation of a problem-space into an API, which can represented by grpc protobufs or function-calls, or whatever. You have inputs, expected actions, and expected results; the abstraction handles the details so you don't have to and often someone else is responsible for updates over time.
Synthesis is building novel functionality through combining multiple things to create a new thing. Humans traditionally have been the synthesis engines of code-production, but coding agents are beginning to automate portions of this.
Writing code is synthesis. Until the advent of coding agents, your IDE would give you support through syntax highlighting and API short-cuts, reducing the cognitive load of bare syntax problems to let you focus on higher level problems like how a function should handle bad inputs. The IDE gives you support for abstractions, but leaves the synthesis to you.
Once coding agents get added in, you shift from generating code, synthesis, to reviewing automatically generated code whenever the agent creates something for you. This is when cognitive biases come into play. Remember the mindset thing I opened this post with? An engineer under throughput stress is going to be less diligent about checking generated code in detail, and will shift more of their domain learning to troubleshooting test-failures and incident-response. Less domain knowledge will be developed during the initial writing. An engineer with no throughput stress, perhaps they're doing some for-fun coding, is more likely to take a fine toothed comb to generated code to learn what the generated code is trying to do, why it works the way it does, and what best-practices it seems to be following; in this throughput-free case the coding agent is a driver of growth.
Coding agents end up magnifying the biases present in the team's environment, enhancing positive feedback loops. The advertised reason to adopt coding agents is to increase developer throughput! The throughput bias is baked into the technology and its marketing, which means organizations requiring coding agent use need to take steps to provide some negative feedback loop enhancement to avoid large-scale degradation in coding quality and incident severity. Use of these agents make an organization more sensitive to growth/throughput bias shifts prompted by org-chart and quarterly goal shifts.
Combine the bias-enhancing quality of coding agents with the industry-wide retraction in US-based jobs for economic reasons, and you should be seeing a general retraction in overall growth mindsets among US-based engineers.
]]>There actually has not been enough reflection in the tech community about DOGE
all I see are "they're not engineers" "that's not what engineering is" I want a thick good really real piece to sink my teeth in about all the parts of this that ARE "like engineering"
I would write it except I want to stay safe
Engineering can be so much better than this and more than this. It's absurd how much we're resigned to this entire area of human work being held hostage by this kind of culture.
I do not believe that the majority of the software people I have worked with all these years sit around thinking, "I want cancer research to be destroyed." We have mountains to climb no doubt, we are trapped in some bad cultures no doubt, but I do not believe this of you for one moment.
Our tech community needs SO much more repair, processing, and collective reflection. The people who worked to support government and science were so betrayed by other groups. The people in flashy big tech cos feeling like their values were betrayed. Just so much repair needed here
I absolutely agree that our tech community needs quite a lot of collective reflection, processing, and work on repair. For a variety of reasons I've done a fair amount of casual research into recovering from trauma of various types. Some of this comes from having lived through the dot-com, great recession, and profitability-crunch contractions in the tech market, and some from good old fashioned life experience outside of the workplace. Trauma is trauma, and we have some pretty good ideas what chronic (ongoing, persistent) trauma does vs acute (single traumatic event) trauma.
Acute trauma: sudden-death layoff.
Chronic trauma: never being allowed to stay on one project long enough to get something good for your performance review, making you constantly fear the next layoff will have your name in the list.
Each of these affects the body and mind differently, and when entire populations are subjected to chronic traumas you get population-level reactions. When those chronic traumas are structural, as is the case with management culture in all of big US-tech, remediation becomes next to impossible. When individuals in this chronically traumatized population can't fix it, you get three big trauma-responses:
Big-tech management likes workers in the trauma harder category, is somewhat tolerant of cynics, and is designed from the org-chart out to prevent the heroes from getting anywhere useful and to redirect their energies in positive directions like burnishing the company's reputation among diversity hires. Do this for a few decades and you have an entire population of highly educated workers who have been trained their entire careers to look at the next sprint/month/quarter's deliverable for your team and kinda ignore what the rest of the company is doing. How your quarter's project to reduce stream latency by 15% at the p95 level relates to the efficiency of data-analysis in ICE isn't always obvious, don't look up or you might find out.
So take these workers who live in this pit of oppression every day for their day-jobs, and put them into an open-source community for their fun-time activity. What happens then?
Fixing this sort of thing requires so much work.
You can't do this overnight, it will take a revolution of some kind. Some revolutions are slow, like the heroes getting somewhere with governmental support allowing unionization to creep in higher and higher numbers until union contracts dominate worker terms and conditions rather than Radford salary reports. Some revolutions come quick like whole industries getting nationalized after a socialist junta and remodeled away from oligarchic control. Some come generationally, like China overtaking the US for big-tech exports forcing the oligarchs to look elsewhere to stay fat.
Our tech community needs SO much more repair, processing, and collective reflection.
]]>@xgranade The common complaint is that you need to be a professional software developer to use Linux. But to use Windows 11 (and have it be usable) you need to be a professional sysadmin who uses terms like "group policy".
https://aus.social/@natarasee/115501020317612539
Because this is spot on. I'd argue Linux is usable by non-devs these days, but you still need a tolerance for fiddling and non-standard UI. The Windows side is extremely true. Windows in a corporate context is way more tolerable than Windows in a home context because the corporate context has a group of grumpy Windows sysadmins setting new Group Policy every time a security or feature release comes out to turn down the suck. Those grumpy sysadmins are as grumpy at Microsoft pulling this consent-violating shit as you are, and Windows lets you centrally shut it off (in a corporate context.)
Windows has been losing desktop market-share to Apple for years, and the old "Wintel" cash-cow they used to enjoy is not milking as much as it used to. When software makers see flagging revenue and soft user demand, it's time to do demand forcing! And demand forcing leads to shittier experiences as the use more software! message gets ever more aggressive.
The long time followers of this blog have seen enough of this industry to know the cycle when they see it.
Do this for enough years and you get the sclerotic Windows 11, full of demand-forcing promptware that pisses your customers off.
]]>Here’s the lightning sketch of Paul’s Treatise Against Efficiency that I’ve never written:
1. Efficiency is asymptotically inefficient: as costs approach zero, the cost of further reducing them approaches infinity.
2. Efficiency prioritizes the measurable over the difficult-to-measure.
3. Efficiency prioritizes what those in power see (or imagine) over on-the-ground reality.
4. Following from 2 and 3, efficiency reduces the amount and quality of information flowing into a human system.
5. Efficiency foments institutional inflexibility.
6. By removing slack, efficiency causes small failures to cascade more readily and increases the risk of catastrophic failure.
7. Following rom 4, 5, and 6, efficiency trades small costs for massive risks: from failures, from missed opportunities, and from inability to adjust.
8. Efficiency, when pushed, strangles the emergent phenomena that in the long term create all new things of value.
9. Thus, although it can be a by-product of evolution, efficiency as a goal in itself strangles evolution.
10. Efficiency as a goal strangles joy.
It turns out I do have time to write an article about that, below the fold.
]]> 1. Efficiency is asymptotically inefficient: as costs approach zero, the cost of further reducing them approaches infinity.This is a long-standing axiom of "nines" thinking. Where each "nine" of uptime costs rather more than the previous nine. I won't spend time on this, because this is widely understood. Fighting for better uptime, no matter how good it already was, is a big part of why Site Reliability Engineering came into being: we needed a way to quantify this, and frameworks for deciding when moar uptime! was not actually called for given business realities.
In areas where efficiency is not a proxy for uptime, where we have an entire engineering practice created to manage realities surrounding the topic, you get what we had before SRE was everywhere: managers attempting to score points by driving up efficiencies of operation and damn the costs! Improved efficiency will pay for itself! It wasn't a good time in the tech industry.
2. Efficiency prioritizes the measurable over the difficult-to-measure.
This is an axiom of management overall: you can't manage what you can't measure. Efficiency is a management problem, therefore efficiency is based on what you can measure. QED. Managing through vibes is not based in science.
3. Efficiency prioritizes what those in power see (or imagine) over on-the-ground reality.
This is another emergent axiom out of US tech industry management practices. In any organization with more than three levels on the org chart, the people making the big decisions on costs are at a great remove from the people doing the work. In theory, if your quantization is good enough (point 2) this shouldn't matter. Theory is not reality, though; which is why the big tech orgs often have Staff+ engineers (Staff, Principal, Distinguished Engineers, Architects) partnering with higher level managers to improve the odds of ground-truth being an input to the big decision. Improving the odds is not the same as 'shall.' The management context always wins in a conflict with ground-reality.
4. Following from 2 and 3, efficiency reduces the amount and quality of information flowing into a human system.
By restricting information solely to things that can be quantized (point 2) and with a tall enough org chart that the people who make the big decisions are heavily divorced from the people doing the work (point 3) you are making decisions on heavily summarized data based on a restricted subset of all the data, which makes deriving a high quality solution rather harder. This is a fundamental feature of modern management and data practices.
5. Efficiency foments institutional inflexibility.
Efficiency is a form of optimization.
Optimization requires building your organization and management structure to be fit for the given tasks, with no surplus.
Every time the task changes, you have to reoptimize. Reoptimization has a real cost, so don't change the task for little things.
This leads to inflexibility, and missed innovation opportunities; it simply costs too much to experiment with a new thing.
6. By removing slack, efficiency causes small failures to cascade more readily and increases the risk of catastrophic failure.
Efficiency of management becomes brittle when one failure causes other managers to get overloaded with work even with routine 'failures' like parental leaves. Slack allows you to better buffer these routine failures, and improves the odds of the whole thing avoiding a catastrophic failure when some unplanned failure happens.
Technical efficiency in distributed systems becomes brittle when the ability to scale up to meet demand is compromised, or a changed usage-pattern causes cascading failures due to lack of optimization in the new pattern. Slack, inefficiency, allows better buffering of changed usage patterns, allowing time to reoptimize for the new shape.
7. Following rom 4, 5, and 6, efficiency trades small costs for massive risks: from failures, from missed opportunities, and from inability to adjust.
As efficiency increases, slack decreases.
As slack decreases, tolerance of failure decreases. Even some 'planned' failures can prompt greater incidence-rates of major failure. Unplanned failures are far more likely to result in major failure.
As slack decreases, the cost of innovation increases, leading to missed opportunities. Certain innovations become classed as failures as a result.
8. Efficiency, when pushed, strangles the emergent phenomena that in the long term create all new things of value.
Efficiency by necessity creates brittleness and reduced failure tolerance.
Lack of failure tolerance means lack of tolerance for experimentation, which is the foundation of innovation. Such systems are more likely to wall innovation off into a 'safe' part of the organization, and be chronically underfunded.
9. Thus, although it can be a by-product of evolution, efficiency as a goal in itself strangles evolution.
Efficiency is optimization. While evolution can produce some amazingly optimized builds, such builds do not tolerate changed circumstances. Take the oxygen out of the system, and most of life as we know it has a much harder time working. Efficiency increases change intolerance, which slows evolution.
10. Efficiency as a goal strangles joy.
The joy of making is introducing change. A new, more efficient process. A better way of handling changed usage patterns. A fundamental rethink of how features are planned. A better platform system that's less annoying to keep running.
Efficiency increases change intolerance. Change is joy. Efficiency decreases joy.
]]>For 2025:
Yes, we're getting way more spam than legitimate questions right now, and have all year.
The Metasmoke people are doing a lot to make sure you rarely see it, but it's still a burden on the moderation staff due to the need to delete the spam-users. That deletion helps the StackExchange anti-spam systems identify bad IPs and others which increases the burden on spam-campaigners like these.
]]>At the base, I ended up writing a book for Platform teams looking to deliver internally deployed observability systems. That's not quite what I had in mind when I started writing in 2020, but that's where it lives five years later. My actual goal was to write a book that was usable by people in the SaaS industry, but also in businesses where the main user of internally developed software was internal users. The non-SaaS population often gets ignored in book targeting, and I wanted something that would let these people feel seen in a way that reading yet another Observability for Cloud Systems book would. In 2025, this book is a Platform book.
In 2019 and early 2020 when I was working with Manning on the title and terms, the word "observability" came up. It seems hard to remember in 2025, but "observability' was still a vague term that didn't yet have industry consensus behind it. OpenTelemetry was a thing at the time, but the "metrics" leg of OTel was still in beta, and "logs" was merely roadmapped. In 2025 there are debates around whether the fourth pillar is profiles, performance traces, or errors, which could be stack-dumps or a category of logs. If we had decided to use "observability" instead of "telemetry" the book may have sold better, but the term "telemetry" works better for me because observability is a practice built on top of telemetry signals. I wasn't writing a book about practice, I was writing about herding signals.
Herding signals, not interpreting them.
In 2025, most of the herding is supposed to be done through OpenTelemetry these days. Or if it isn't OTel, the signals are being herded through other systems like Apache Spark. This is industry consensus; instrument your code, add attributes in the emitters and collectors, change your vendors as you need to, build dashboards in your vendor's platform. A rewrite of Software Telemetry would reference OTel far more often than I did, but I would still make sure to mention non-OTel styles due to OTel not actually being supported (or in some cases, a good fit) in certain environments like network telemetry.
Whatever the API format of the signals getting herded, platform engineers need to know the fundamentals of how telemetry systems operate and that's what I wrote about. But also, I wrote about storing those signals, which is something that OpenTelemetry deliberately leaves out as a detail for the implementer. As I extensively wrote about, storing signals and creating a reporting interface is a hard enough part of telemetry that you can build a business around it. In fact, the Observability Tools market in 2025 is valued at around $2.75 Billion, and they all would love for you to use OTel to send them data to store and present.
In the language of my book, OpenTelemetry is an early shipping stage technology. Early because it has no role in storage. OTel arguably has a role in the emitting stage through explicit markup in code itself. OpenTelemetry's impact to the presentation stage is mostly in tagging and attribute schemas and how they get represented in storage. Observability needs to consider every stage, but also the SRE Guide problems of figuring out what to instrument, to which markup standards, following which procedures to ensure reliability. Observability sits on top of telemetry.
One of the consistent comments I got during the pre-publication reviews was: "I want to know what to track."
My answer was simple: that's not the book I'm writing.
This book is for you, the growth engineer tasked with taking a Kafka topic (or group of topics) of logging data, sent there by OTel, and transform it in the big Databricks instance with all the other business data.
This book is for you, the network engineer tasked with extracting network metrics out of a proprietary system, so you can chart network things in the main engineering dashboarding platform.
This book is for you, the security engineer tasked with extracting security event data out of a cloud provider to put into the SIEM system.
This book is for you, the project manager who has just been given a digital transformation project to revitalize how all the internally developed apps will produce telemetry, and how engineers will observe the system.
]]>RSS never tracked you.
https://mastodon.social/@Daojoan/114587431688413845M
Email never throttled you.
Blogs never begged for dopamine.
The old web wasn’t perfect.
But it was yours.
I was there for the rise and fall of blogging, so the rest of this post is me over thinking this particular post.
]]>RSS never tracked you
RSS never tracked you the way HTTP or HTML never tracked you. What tracks you with HTTP/HTML are rendering engines for HTML and Javascript. I can say with absolute certainty that tracking-tech during blogging's height was used to track audience, click-through rates, and other site-engagement metrics. RSS was the loss-leader for clicks on sites. This particular post uses one of those tricks; an opening paragraph promising more, which if you're reading this through RSS you will have to click through to read. Tracking RSS feeds was done through tracking pixels back in the day because you couldn't trust Javascript to run in the RSS readers but HTML rendering generally worked.
Email never throttled you.
Email absolutely did, absolutely. The problem with spinning up a self-hosted newsletter service in 2025 is getting your IP reputation good enough that the big mail-box vendors (Google, Microsoft, Yahoo/AOL) let you deliver. This issue was in its infancy in 2005, but was an emerging problem as IP reputation became clear as a low-cost first-pass anti-spam technique. Mailing list operators ran into this all the time back in 2005 as various subscribers moved behind security appliances doing IP reputation.
Blogs never begged for dopamine.
On a factual basis, this is false. Sole operators like myself lived for comments, that dopamine hit from people liking what I wrote. If I couldn't get that, I'd enjoy reshare statistics (see the first point about RSS tracking). Many commercial blogging platforms even created RSS feeds for comments on specific articles to make it easier to keep up on discussions. If that isn't dopamine, I don't know what is.
The old web wasn’t perfect.
But it was yours.
True, to a point. I tagged this article blogger because that's what I hosted this blog through for most of its first decade. You know, a centralized blogging platform akin to LiveJournal or Dreamwidth back then, or Medium and Substack today. I didn't own the platform. After I moved to my own domain and Movable Type, this was true. And yeah, it wasn't perfect but it was most definitely mine.
There is another layer to this post beyond the simply factual, and that's a critique of platforms. Blogging in its heyday was perhaps the last major gasp of the Old Internet, where a bunch of hobbyists create something beautiful and widely adopted, which then gets enclosed by commercial interests. Blogging was absolutely not centralized. That lack of centralization means there was no unitary profit motive looking to drive engagement to increase sales, those were limited to individual sites.
Blogging's decline is traceable to two big trends:
The fact that Google Reader's death functionally killed RSS-based distribution is proof that blogging was already beginning to centralize. Google couldn't control production and distribution, only engagement; and the real money was in controlling all three. Google wanted what Twitter had, which was the entire production, distribution, and engagement framework on the same site with the same owners and tracking infrastructure. Once they had that, start engineering dopamine feedback loops to improve stickiness and engagement; and we have all the algorithmic and dark pattern pathologies we know and loathe today.
Modern internet users have been trained for a decade in a half to expect social media to involve a single, or small number of sites to do everything, and individual posts are short enough to deal with while waiting for a bus or the kids to walk from school to car. The old blog+rss model simply can't compete with this, and better fits as attention-competition to a given user's news media consumption. Medium and Substack both offering paid subscription options for individual blogs is proof that news media is the competition to blogging, not social media.
The nostalgic exhortation to set up a blog and distribute through RSS is in the same vein as calls to return to IRC and dump Slack/Teams. Some people/groups can do that, but the platform features are different enough that the old tools don't feel feature-complete and that lack nudges folk back into the commercial walled gardens.
]]>The SaaS revolution has utterly transformed the office automation space. The job I had in 2005, in the early years of this blog, only exists in small pockets anymore. So many office systems have been SaaSified that the old problems I used to blog about around backups and storage tech are much less pressing in the modern era. Where we have stuff like that are places that have decades of old file data, starting in the mid to late 1980s, that is still being hauled around. Even when I was still doing this in the late 2000s the needle was shifting to large arrays of cheap disks replacing tape arrays.
Where you still see tape being used here are offices with policies for "off-site" or "offline" storage of key office data. A lot of that stuff is also done on disk these days, but some offices still kept their tape libraries. The InfoSec space is keen to point out you can't crypto-locker an offline tape, so offline tape is a useful tool in recovering from a ransomware incident. I suspect a lot of what DoGE found was in this category of offices retaining tape infrastructure. Is disk cheaper here? Marginally, the true savings will be much less than the $1M headline rate.
But there is another area where tape continues to be the economical option, and it's another area DoGE is going to run into: large scientific datasets.
To explain why, I want to use a contrasting example: A vacation picture you took on an iPhone in 2011, put into Dropbox, shared twice, and haven't looked at in 14 years. That file has followed you to new laptops and phones, unseen, unloved, but available. A lot goes into making sure it's available.
All the big object-stores like S3, and file-sync-and-share services (like Dropbox, Box, MS live, Google Drive, Proton Drive, etc) use a common architecture because this architecture has been proven to be reliable at avoiding visible data-loss:
The end result of the above is that the 1MB vacation picture is written to disk 6 to 14 different times. The nice thing about the above is you can lose an entire rack-row of a datacenter and not lose data; you might lose 2 of your 5 copies of a given block, but you have 3 left to rebuild, and your other region still has full copies.
But I mentioned this 1MB file has been kept online for 14 years. Assuming an average disk life-span of 5 years, each block has been migrated to new hardware 3 times in those years. Meaning each 4KB block of that file has been resident on between 24 and 42 hardrives; or more, if your provider replicates to more than 2 discrete geographic region. Those drives have been spinning and using power (and therefore requiring cooling) the entire time.
These systems need to go to all of this effort because they need to be sure that all files are available all the time, when you need it, where you need it, as fast as possible. If a person in that vacation photo retires, and you suddenly need that picture for the Retirement Montage at their going away party, you don't want to wait hours for it to come off tape. You want it now.
Contrast this to a scientific dataset. Once the data has stopped being used for Science! it can safely be archived until someone else needs to use it. This is the use-case behind AWS S3 Glacier: you pay a lot less for storing data, so long as you're willing to accept delays measurable in hours before you can access it. This is also the use-case where tape shines.
A lab gets done chewing on a dataset sized at 100TB, which is pretty chonky for 2011. They send it to cold storage. Their IT section dutifully copies the 100TB dataset onto LTO-5 drives at 1.5TB per tape, for a stack of 67 tapes, and removes the dataset from their disk-based storage arrays.
Time passes, as with the Dropbox-style data. LTO drives can read between 1 and 2 generations prior. Assuming the lab IT section keeps up on tape technology, it would be the advent of LTO-7 in 2015 that would prompt a great restore and rearchive effort of all LTO-5 and previous media. LTO-7 can do 6TB per tape, for a much smaller stack of 17 tapes.
LTO-8 changed this, with only a one version lookback. So when LTO-8 comes out in 2017 with a 9TB capacity, a read restore/rearchive effort runs again, changing our stack of tapes from 17 to 12. LTO-9 comes out in 2021 with 18TB per tape, and that stack reduces to 6 tapes to hold 100TB.
All in all, our cold dataset had to relocate to new media three times, same as the disk-based stuff. However, keeping stacks of tape in a climate controlled room is vastly cheaper than a room of powered, spinning disk. The actual reality is somewhat different, as the few data archive people I know mention they do great restore/archive runs about every 8 to 10 years, largely driven by changes in drive connectivity (SCSI, SATA, FibreChannel, Infiniband, SAS, etc), OS and software support, and corporate purchasing cycles. Keeping old drives around for as long as possible is fiscally smart, so the true recopy events for our example data is likely "1".
So another lab wants to use that dataset and puts in a request. A day later, the data is on a disk-array for usage. Done. Carrying costs for that data in the intervening 14 years are significantly lower than the always available model of S3 and Dropbox.
Tape: still quite useful in the right contexts.
]]>Meanwhile, everyone in the tech-disaster industry peeps over the shoulders of environmental disaster recoveries like hurricanes and earthquakes. You can learn a lot by watching the pros. I've talked about some of what we learned, mostly it has been procedural in nature:
Since then, the United States elected a guy who wants to be dictator, and a Congress who seems willing to let it happen. For those of us in the disliked minority of the moment, we're facing concerted efforts to roll back our ability to exist in public. That's risk. Below the fold I talk about using what I learned from IT risk management and how I apply those techniques to assess my own risks. It turns out building risks for "dictatorship in America" can't rely on prior art as much as risks for "datacenter going offline," which absolutely has prior art to include; and even incident rates to factor in.
]]> The risk analysis process in overview is:That's a lot of steps, but I'll walk you through some examples to help you figure out what I'm talking about. As computer professionals in the disaster recovery space, it's extremely easy to jump to the worse case scenario ("organized murders of my minority in my area," also known as an organized genocide) and base all of your planning around that. Doing so will turn you into a giant ball of anxiety, so going through the above process will give your limbic system a break.
You probably already have specific fears, like the "organized murders of my minority in my area" one I used above. Write that down, and write everything else that may push you out of "ignore/fight" and into "do something." This will be emotional work, just get things on the page. We'll qualify the risks later on, but we need a starting set to even begin the analysis.
For the analysis I ran regarding when "flee" becomes a good idea, I came up with the following response menu:
Most of this work is identifying which responses have lead-times, and determining how much time you need. This is a dependency analysis, so it's safe to get into the weeds a bit regarding procedures and lead-times. For international relocation, there ended up being a huge menu of ways to do that, multiple for each country and this one response ended up taking the majority of the time for me to qualify.
You're looking for two lead-times:
I also learned a few things. For the Domestic move option, there is a difference between ASAP and Planned move cases, and for a same-state vs different-state relocation. An ASAP move would involve moving into short term housing, such as a hotel, and then deciding if house-hunting or apartment-hunting needs to happen next. For a planned move, you can do the apartment/house hunt before executing the move. If you're employed, your employer may place restrictions on interstate moves! Find that out, because it'll affect your ability to relocate to another state.
Interstate moves as a remote employee
Due to how tech employers in the US do geobanding, each state and city is sorted into typically three geobands. The geobands are in broad agreement due to the common use of Radford services to assess market salary bands, and run from Tier 1 (SF/NYC) to Tier 3 (cheapest places). Employers typically want you to move within a band. Or better, down a band which will let them pay you less. You might even get relocation cost-support by moving down a band.
Many companies will require approval to move to a new state. This has to do with how employment taxes are computed in America. Your employer needs to have an entity in your state in order to pay you. Smaller employers may be willing to set up an entity just for you. Larger ones are unlikely to do so.
International moves have a menu of possibilities, especially if you're a late career technical professional who has been reaping great big RSUs for a while and has significant taxable savings:
Your financial situation will tell you how many of these are viable. This may require you to change employer in order to gain the possibility of an internal transfer to another country. If so, add this to your "prep" time. If the investment or retirement visas are viable, now is the time to look at countries and where you can go and how many of them will allow you to be you.
Of the above, the Employer Sponsored is probably the fastest to execute once things get going. Your employer probably has professionals to work on obtaining the visa. Even so, it could take up to 6 months for you and your stuff to get to your target company. The Investment Visa option requires vetting by your target country's consulate, which can take up to a year. Or more, in the case of Portugal who is seeing a huge influx of applicants. Retirement visas are easier to get, but they require enough passive income so fewer people under IRA mandatory distribution age will qualify.
The right-wing rise is everywhere that speaks English
Everywhere English-speaking is having a bad time of it right now. Everywhere. The UK fell to right wing extremism before the US did. New Zealand has a right wing government. India has had a right-wing government for years. Australia has major anti-immigrant sentiment. Canada was following the US until the US pissed them off enough they decided to fight, so there may be hope there; but they're also increasingly likely to cut off immigration from the US as a result of the hostilities.
If you're like me, and a member of the trans community, the US of March 2025 is still one of the best places around to be. That spot is actively degrading, but be aware that where you jump to may not be better. Your affordable hope may simply be "won't get as bad," and will be actively worse until the US catches up.
The most extreme response, Overnight Evacuation, is a special one because it tells you what the execute time needs to be: 12 hours. The big work for Overnight Evacuation is planning and prep, such as maintaining go-bags and having a plan for where you can run. Go-bag maintenance takes time, especially if you're middle-aged and have a lot of prescribed medications. Depending on your circumstances, you may have to run evacuation drills to refine your procedures. For the Overnight Evacuation option you'll likely need to decide when you want to start actively prepping the 12-hour evacuation capability, because maintaining that readiness for two plus years will be a lot of work.
After assessment, my response table revised somewhat drastically.
| Response | Preparation time | Execution time |
|---|---|---|
| Endure it | N/A | N/A |
| Fight it | N/A | N/A |
| ASAP domestic move | 2 weeks | 2 weeks |
| Planned domestic move | 1 month | 1 to 3 months |
| Work-visa move | 1 year (job hunt) | 2-6 months |
| Investor/retirement visa move | 3 months | 6-18 months |
| Evacuate overnight | 1 month + regular refreshes | 12 hours |
The international moves have incredibly long lead-times! In the absence of refugee lanes for Americans fleeing dictatorship, an international relocation simply can't be the response to an emergent problem. Knowing this tells us that any risk with "international move" as a response needs to be analyzed for what events heralding the need for an international relocation will happen 6 to 18 months beforehand.
Now that we've gone through all the work of qualifying our responses and understanding lead times, now we look at our list of risks and apply responses to them. This is a judgment call based on vibes. If you want to get scientific about each risk and diagram out dependency chains, do whatever makes your brain happy.
One thing to note: risks rated Evacuate Overnight are likely part of a rising pattern of risks. Step 5 is where we analyze these dependencies, and Step 6 is where we start grouping risks. Don't get too hung up on dependencies here, you want to get your brainstorm list completed. Feel free to evict risks if you find any that you don't feel merit response any more, or add new risks should thinking about others remind you of risks you missed.
For the risks with responses having a long lead time, we need to analyze those risks to identify what sort of events are upstream of the risk, things that need to happen before our risk happens. Such analysis will let us know which prelude events we should track before entering planning. For example, let's look at the extreme option I opened with, "organized murders in my area," which is an Evacuate Overnight risk. One dependent chain could be:
This list demonstrates an escalation from a call for action to action in my area. This also elides probabilities, which is analysis you will have to do yourself. America is a huge place. The areas most at risk for a rapid escalation from 1 to 6 are deeply red areas in Republican controlled states, where State authorities are less likely to investigate murders of disliked minorities. The areas least at risk are blue cities in Democrat held states, where State authorities are quite likely to investigate coordinated murders; making it harder for murder militias to safely operate.
For blue state residents, violence will rise in neighboring red states before it starts leaking across borders. This affects timing. After judging the dependent chain, we can assign risks to higher levels of the chain.
We have a new risk: regular media reports calls for murders, without condemning the murders. This new risk is a herald of the much more severe murders in my area risk.
Repeat this exercise for each of your long-lead risks. Now is the time to get into the weeds.
One risk on my brainstorm list was "ACA is repealed," which was removed from my brainstorm list due to finding discussions about how the right widely views ACA as settled (their electoral drubbing in 2018 was earned by being too aggressive about repealing ACA) and attacks are focusing on Medicaid block-grants. Such grants don't affect me, so "ACA is repealed" was drop from the risks list. Taking its place was, "Coverage of trans-related care banned for ACA plans," the court-cases for which are already underway.
Your dependency chain analysis likely revealed a few risks that share a dependent chain, or are actual precursors to risks with higher severity levels of response. In my analysis I was able to condense the huge brainstorm list down to four risk-types, and eliminated several more risks that weren't eliminated in Step 5. My risks were pretty broad, but cover similar themes.
Murder, kidnapping for ransom/torture, state disappearance squads, and my state losing safe-haven status. This was a far shorter list than I started with. This list is also extremely dark. The good thing is we're no where near anything on this list as of March 2025. Now the work turns to figuring out the joined dependencies and coming up with prelude events.
This is repeating step 5, but using the merged risks. I found diagramming flow-charts to be quite helpful. While doing your work, consider what events would constitute the trigger for the following stages of response:
Now that you've done all of this work, it's time to build the summary. This aggregated list is the one to check back on once a week or so to see if any lines have been crossed. My list is highly personal, but has 10 different events on it. We are no where near any of those 10 events I identified as significant heralds of greater danger.
Then refresh your analysis once a month to square your analysis with what has happened in the previous four months. My March analysis involved a major rewrite because my February one failed to identify the mass secret-order detention of citizens risk, in spite of it being the number one thing dictators want above all else.
Whenever you feel an anxiety spike watching the news, and the urge to flee begin to take root, review your list of events. It calms me down, and once in a while I make notes for the next monthly refresh. This methodical approach absolutely did get me out of a doom-spiral, all thanks to a career in planning for disasters.
]]>Why did you start blogging in the first place?
I covered a lot of that in 20 years of this nonsense from about a year ago. The quick version is I was charged with creating a "Static pages from your NetWare home directory" project and needed something to test with, so here we are. That version was done with Blogger before the Google acquisition, when they still supported publish-by-ftp (which I also had to set up as part of the same project).
What platform are you using to manage your blog, and why do you use it?
When blogger got rid of the publish-by-ftp method, I had to move. I came to my own domain and went looking for blogging software. On advice from an author I like, I kept in mind the slashdot effect so wanted to be sure if I had an order of magnitude more traffic for an article it wouldn't melt the server it was one. So I wanted something relatively light weight, which at the time was Movable Type. Wordpress required database hits for every webpage, which didn't seem to scale.
I stuck with it because Movable Type continues to do the job quite well, and be ergonomic for me. I turned off comments a while ago, as that was an anti-spam nightmare I needed recency to solve. Movable Type now requires a thousand dollars a year for a subscription, which pencils out to about $125 per blog post at my current posting rate. Not worth it.
Have you blogged on other platforms before?
Like just about everyone my age, I was on Livejournal. I don't remember if this blog or LJ came first, and I'm not going to go check. I had another blog on Blogger for a while, about local politics. It has been lost to time, though is still visible on archive.org if you know where to look for it.
How do you write your posts?
Most are spur of the moment. I have a topic, and time, and remember I can be long-form about it. Once in a while I'll get into something on social media and realize I need actual wordcount to do it justice, so I do it here instead. The advent of twitter absolutely slowed down my posting rate here!
Once I have the words in, I schedule a post for a few days hence.
When do you feel most inspired to write?
As with all writers, it comes when it comes. Sometimes I set out goals and I stick to them. But blogging hasn't been a focus of mine for a long time, so it's entirely whim. I do know I need an hour or so of mostly uninterrupted time to get my thoughts in order, which is hard to come by without arranging for it.
Do you normally publish immediately after writing, or do you let it simmer a bit?
As mentioned above, I use scheduled-post. Typically for 9am, unless I've got something spicy and don't care. This is rare, I've also learned that posting spicy takes absolutely needs a cooling off period. I've pulled posts after writing them because I realize they didn't actually need to get posted, I merely needed to write them.
What's your favorite post on your blog?
That's changed a lot over the years as I've changed.
I'm not stopping blogging any time soon. At some point the dependency chain for Movable Type will rot and I'll have to port to something else, probably a static site generator. I believe I'm spoiled for choice in that domain.
]]>A few organizations go so far as to have a fully separate process for the 'High' and 'Critical' urgencies of events, maybe calling them Disaster Recovery events instead of Incidents. DR events need to be rare, which means that process isn't as well exercised as Incident response. However, a separate process makes it abundantly clear that certain urgencies and scopes require different process overall. More on this in a later blog-post.
This is the later blog post.
The SaaS industry as a whole has been referring to the California Fire Command (now known as the Incident Command System) model for inspiration on handling technical incidents. The basic structure is familiar to any SaaS engineer:
There may be additional roles available depending on organizational specifics:
Again, familiar. But truly large incidents put stress on this model. In a given year the vast majority of incidents experienced by an engineering organization will be the grass fire variety that can be handled by a team of four people in under 30 minutes. What happens when a major event happens?
The example I'm using here is a private information disclosure by a hostile party using a compromised credential. Someone not employed by the company dumped a database they shouldn't have had access to, and that database involved data that requires disclosure in the case of compromise. Given this, we already know some of the workstreams that incident response will be doing once this activity is discovered:
An antiseptic list, but a scary one. The moment the company officially notices a breach of private information, legislation world-wide starts timers on when privacy regulators or the public need to be informed. For a profit driven company, this is admitting fault in public which is something none of them do lightly due to the lawsuits that will result. For publicly traded companies, stockholder notification will also need to be generated. Incidents like this look very little like an availability SLA breach SEV of the kind that happens 2-3 times a month in different systems.
Based on the rubric I showed back in November, an incident of this type is of Critical urgency due to regulated timelines, and will require either Cross-Org or C-level response depending on the size of the company. What's more, the need to figure out where the attacker went blocks later stages of response, so this response process will actually be a 24 hour operation and likely run several days. No one person can safely stay awake for 4+ days straight.
The Incident Command Process defines three types of command structure:
Incidents of the scale of our private information breach lean into the Area Command style for a few reasons. First and foremost, there are discrete workstreams that need to be executed by different groups; such as the security review to isolate scope, building regulated notifications, and cycling credentials. All those workstreams need people to run them, and those workstream leads need to report to incident command. That looks a lot like Area Command to me.
If your daily incident experience are 4-7 person team responses, how ready are you to be involved in an Area Command style response? Not at all.
If you've been there for years and have seen a few multi-org responses in your time, how ready are you to handle Area Command style response? Better, you might be a good workstream lead.
One thing the Incident Command Process makes clear is that Area Commanders do not have an operational role, meaning they're not involved in the technical remediation. Their job is coordination, logistics, and high level decision making across response areas. For our pretend SaaS company, a good Area Commander will be someone:
Is your company equipped to handle this scale of response?
In many cases, probably not. Companies handle incidents of this type a few different ways. As I mentioned in the earlier post, some categorize problems like this as a disaster instead of an incident and invoke a different process. This has the advantage of making it clear the response for these is different, at the cost of having far fewer people familiar with the response methods. You make up for the lack of in situ training, learn by doing, by regularly re-certifying key leaders on the process.
Other companies extend the existing incident response process on the fly rather than risk having a separate process that will get stale. This works so long as you have some people around who kind of know what they're doing and can herd others into the right shape. Though, after the second disaster of this scale, people will start talking about how to formalize procedures.
Whichever way your company goes, start thinking about this. Unless you're working for the hyperscalers, incidents of this response scope are going to be rare. This means you need to schedule quarterly time to train, practice, and certify your Area Commanders and workstream leads. This will speed up response time overall, because less time will be spent arguing over command and feedback structures.
]]>How Elasticsearch is columnar (or not)
First of all, Elasticsearch isn't exactly columnar but it can fake it to a point. You use Elasticsearch when you need full indexing and tokenization of every field in order to accelerate query-time performance. Born as it was in the early part of the 2010s, Elasticsearch balances ingestion-side complexity in order to optimize read-side performance. There is a reason that if you have a search field in your app, there is a good chance that Elasticsearch or OpenSearch is involved in the business logic. While Elasticsearch is "schema-less," schema still matters, and there are clear limits to how many fields you can add to an Elasticsearch index.
Each Elasticsearch index or datastream has defined fields. Fields can be defined at index/datastream creation, or configured to auto-create on first use. Both are quite handy in telemetry contexts. Each document in an index or datastream has a reference for every defined field, even if the contents of that field are null. If you have 30K fields, and one document has only 19 fields defined, the rest will still exist on the document but be nulled; which in turn makes that 19 defined-field document rather larger than the same document in an index/datastream with only 300 defined fields.
Larger average document size slows down search for everything in general, due to the number and size of field-indexes the system has to keep track of. This also balloons index/datastream size, which has operational impacts when it comes to routine operations like patching and maintenance. As I mentioned in Software Telemetry, Elasticsearch's cardinality problem manifests in number of fields, not in unique values in each field.
If you are willing to get complicated in your ingestion pipeline through careful crafting of telemetry shape, and ingestion into multiple index/datastreams to bucket types of telemetry into shards of similarly shaped telemetry, you can mitigate some of the above problems. Create an alias to use as your search endpoint, and populate the alias with the index/datastreams of your various shards. Elasticsearch is smart enough to know where to search, which lets you bucket your field-count cardinality problems in ways that will perform faster and save space. However, this is clearly adding complexity that you have to manage yourself.
How Apache Spark is columnar
Spark is pretty clearly columnar, which is why it's the de facto platform of choice for Business Intelligence operations. You know, telemetry for business ops not engineering ops. A table defined in Spark (and most of its backing databases like Parquet or Hive) can have arbitrary columns defined in it. Data for each column is stored in separate files, which means queries like the following looking to build a histogram of log-entries per hour "COUNT timestamp GROUP BY hour(timestamp)" are extremely efficient as the system only needs to look at a single file out of thousands.
Columnar databases have to do quite a bit of read-time and ingestion-time optimization to truly perform fast, which demonstrates some of the tradeoffs of the style. Where Elasticsearch was trading ingestion-time complexity to speed up read-time performance, columnar databases are tilting the needle more towards increasing read-time complexity in order to optimize overall resource usage. In short columnar databases have better scaling profiles than something like Elasticsearch, but they don't query as fast as a result of the changed priorities. This is a far easier trade-off to make in 2024 than it was in 2014!
Columnar databases also don't tokenize the way Elasticsearch does. Have a free-text field that you want to do sub-string searches on? Elasticsearch is built from the bolts out to make that search as fast as possible. Columnar databases, on the other hand, do all of the string walking and searching at query-time instead of pulling the values out of some b-trees.
Where Elasticsearch suffers performance issues when field-count rises, Spark only encounters this problems if the query is designed to encounter it through use of "select *" or similar constructs. The files hit by the query will only be the ones for columns referenced in the query! Have a table with 30K columns in it? So long as you query right, it should perform quite well; the 19 defined fields in a row problem shouldn't be a problem so long as you're only referencing one of those 19 fields/columns.
Why columnar is neat
A good centralized logging system can stand in for both metrics and traces, and in large part can do so because the backing databases for centralized logging are often columnar or columnar-like. There is nothing stopping you from creating metric_name and metric_value fields in your logging system, and building a bunch of metrics-type queries using those rows.
As for emulating tracing, this isn't done through OpenTelemetry, this is done old-school through hacking. Chapter 5 in Software Telemetry covered how the Presentation Stage uses correlation identifiers:
"A correlation identifier is a string or number that uniquely identifies that specific execution or workflow."
Correlation identifiers allow you to build the charts that tracing systems like Jaeger, Tempo, and Honeycomb are known for. There is nothing stopping you creating an array-of-strings type field named "span_id" where you dump the span-stack for each log-line. Want to see all the logs for a given Span? Here you are. Given a sophisticated enough visualization engine, you can even emulate the waterfall diagrams in dedicated tracing platforms.
The reason we haven't used columnar databases for metrics systems has to do with cost. If you're willing to accept cardinality limits, you can store a far greater number of metrics for the same amount of money as doing it in a columnar database. However, the biggest companies already are using columnar datastores for engineering metrics, and nearly all companies are using columnar for business metrics.
But if you're willing to spend the extra resources to use a columnar-like datasource for metrics, you can start answering questions like "how many 5xx response-codes did accounts with the Iridium subscription encounter on October 19th." Traditional metrics system would consider subscription-type to be too highly cardinal, where columnar databases shrug and move on.
What this means for the future of telemetry and observability
Telemetry over the last 60 years of computing has gone from digging through the SYSLOG printout from one of your two servers, to digging through /var/log/syslog, to the creation of dedicated metrics systems, to the creation of tracing techniques. Every decade's evolution of telemetry has been constrained by the compute and storage performance envelope available to the average system operator.
We're part way through the 2020s, and it's already clear to me that columnar databases are probably where telemetry systems are going to end up by the end of the decade. Business intelligence is already using them, so most of our companies have them in our infrastructure already. Barriers to adoption are going to be finding ways to handle the different retention and granularity requirements of what we now call the three pillars of observability:
Having columnar databases at the core allows a convergence of the pillars of observability. How far we get in convergence over the next five years remains to be seen, and I look forward to finding out.
]]>A good incident response program needs to be approachable by anyone in the company, meaning anyone looking to open one should have reasonable success in picking incident attributes right. The incident automation industry, tools such as PagerDuty's Jeli and the Rootly platform, has settled on a pick-one list for severity, with sometimes support for additional fields. Unless a company is looking to home build their own incident automation for creating slack channels, managing the post-incident review process, and tracking remediation action items, these de facto conventions constrain the options available to an incident response program.
As Honeycomb pointed out, there are two axis that need to be captured by "severity," and they are: urgency, and level of response. I propose the following pair of attributes:
Urgency
Level of response
Is Private? Yes/No - If yes, only the people involved in the response are notified of the incident and updates. Useful for Security and Compliance type incidents, where discoverability is actually bad. Some incidents qualify as Material Non-Public Information, which matters to companies with stocks being traded.
The combinatorics indicate that 5*5=25 pairs, 50 if you include Is Private, which makes for an unwieldy pick-one list. However, like stellar types there is a kind of main sequence of pairs that are more common, with problematic outliers that make simple solutions a troublesome fit. Let's look at a few pairs that are on the main sequence of event types:
Six of 25 combinations. But some of the others are still viable, even if they don't look plausible on the surface. Let's look at a few:
How is an organization supposed to build a pick-one list from this mess that is usable? This is hard work!
Some organizations solve this by bucketing incidents using another field, and allowing the pick-one list to mean different things based on what that other field says. A Security SEV1 gets a different scale of response than a Revenue SEV1, which in turn gets a different type of response than an Availability SEV1. Systems like this have problems with incidents that cross buckets, such as a Security issue that also affects Availability. It's for this reason that Honeycomb has an 'ambiguous' bucket.
A few organizations go so far as to have a fully separate process for the 'High' and 'Critical' urgencies of events, maybe calling them Disaster Recovery events instead of Incidents. DR events need to be rare, which means that process isn't as well exercised as Incident response. However, a separate process makes it abundantly clear that certain urgencies and scopes require different process overall. More on this in a later blog-post.
Other orgs handle the outlier problem differently, taking them out of incidents and into another process all together. Longer flow problems, low urgency above, get called something like a Code Yellow after a Google effort, or Code Red for the Critical + C-Team level to handle long flow big problems.
Honeycomb took the bucketing idea one step further and dropped urgency and level of response entirely, focusing instead on incident type. A process like this still needs ways to manage urgency and response-scope differences, but this is being handled at a layer below incident automation. In my opinion, a setup like this works best when Engineering is around Dunbar's Number or less in size, allowing informal relationships to carry a lot of weight. Companies with deeper management chains, and thus more engineers, will need more formalism to determine cross-org interaction and prioritization.
Another approach is to go super broad with your pick-one list, and make it apply to everyone. While this approach disambiguates pretty well between the SEV 1 highest urgency problems and SEV 2 urgent but not pants on fire urgent, they're less good at disambiguating SEV 3 and SEV 4 incidents. Those incidents tend to only have local scope, so local definitions will prevail, meaning only locals will know how to correctly categorize issues.
There are several simple answers for this problem, but each simplification has it's own problem. Your job is to pick the problems your org will put up with.
One of the products offered by my employer is written in Elixir, which is built on top of Erlang. Elixir had an 8 or so month period of fame, which is when the decision to write that product was made. We picked Elixir because the Erlang engine gives you a lot of concurrency and async processing for relatively easy. And it worked! That product was a beast in relatively little CPU. We had a few cases of 10x usage from customers, and it just scaled up no muss no fuss.
Where the problems with the product came wasn't in the writing, but in the maintaining and productionizing. Some of the issues we've had over the years, many of which got better as Elixir as an ecosystem matured:
My opinion is that the dismissiveness from this particular Linux Kernel Maintainer had to do with this list. The Linux kernel and module ecosystem is massive, with highly complex build processes spanning many organizations, and regression testing frameworks to match. Ecosystem maturity matters way more for CI, regression, and repeatable build problems than language maturity.
Rust has something Elixir never had: durable mindshare. Yeah, the kernel rebuild process has taken many years, and has many years to go. Durable mindshare means that engineers are sticking with it, instead of chasing the next hot new memory safe language.
]]>Any company big enough to get past the running out of money is the biggest disaster phase has probably spent some time thinking about what to do if things go wrong. But how do you, the engineer in the room, get the deciders to think about disasters in productive ways?
The really big disasters are obvious:
All obvious stuff, and building to deal with them will let you tick the box for compliance DR. Cool.
But there are other disasters, the sneaky ones that make you think and take a hard look at process and procedures in a way that the "oops we lost everything of [x] type" disasters generally don't.
The above are all kinda "security" disasters, and that's my point. SysAdmins sometimes think of these, but even we are guilty of not having the right mental models to rattle these off the top of our head when asked. Asking about disasters like this list should start conversations that generally don't happen. Or you get the bad case: people shrug and say "that's Security's problem, not ours," which is a sign you have a toxic reliability culture.
Security-type disasters have a phase that merely technical disasters lack: how do we restore trust in production systems? In technical disasters, you can start recovery as soon as you've detected the disaster. For security disasters recovery has to wait until the attacker has been evicted, which can take a while. This security delay means key recovery concepts like Recovery Time and Recovery Point Objectives (RTO/RPO) will be subtly different.
If you're trying to knock loose some ossified DR thinking, these security type disasters can crack open new opportunities to make your job safer.
]]>