Where was I?

Right – the problem is most of the promised benefits of DeFi still amount to illusory hand-waving [1]. The first and most obvious problem with “DeFi” as a concept is that everyone will end up operating through a centralized exchange (a la Coinbase) for convenience anyway, defeating the purpose of “decentralization.” And if you don’t, we’re going to lose all the mechanisms that protect us from ourselves and that we take for granted in our current financial system:

• Forgot your password? Customer service can help you regain access.
• Added an extra 0 to your order? Transferred your money to the wrong recipient? No problem, transactions can be reversed due to mistakes.
• Your five year old got on and accidentally traded your net worth for doge? No problem, call in and fix it.

The main problem/feature of the blockchain that breaks finance is that it attempts to completely dissociate technical rules from legal ones. It is well established that you cannot legally keep money that was sent to you by accident. Most blockchains though, are irreversible by design, even in cases of obvious fraud. This poses real problems for law enforcement and umm, anyone who cares about fairness. Dealing with exceptions, reversing mistaken & fraudulent transactions, and the whole web of “human judgment and intermediaries” is treated like unnecessary baggage….and the collateral damage is treated with a shrug. Until those are solved, we’re trading one set of problems for another. But crypto enthusiasts refuse to even acknowledge the tradeoffs – at least until they (or their exchange) get hacked, and they learn the actual meaning of “irreversible.”

And its frustrating because there could be real and tangible benefits to cryptocurrency and blockchain technology. Perhaps the greatest embarrassment of 21st century human technology is that we haven’t yet solved cross-border payments yet. We might as well try a different approach with crypto.

It’s just that very few people are actually working on solving real problems like this with blockchain. Overt manipulation, tax avoidance, speculative bubbles, organized crime, and crazy volatility seems to be par for the course. You might think “it seems kind of silly to try and build a financial system on top of such a foundation” and I wouldn’t blame you. Luckily, almost nobody is even trying: crypto is mostly used for crypto. Speculative investments, “yield farming,” exit scams, and money laundering seem to be the main use cases. Almost every Ethereum project is either an outright scam or Yet Another Way to Manage your Cryptocurrency Wallets. It would be as if the bittorrent inventors could only think of “transferring bittorrent clients to each other over bittorrent” as their main use case.

It’s a sad joke.

[1] The one possible exception: if it drags the world financial system into the modern age without changing anything else, it will be an unalloyed good. There’s no real reason money transfers can only occur on weekdays, or for stock trades to take 2 days to settle.

How did we get here?

ARM started out as the upstart competitor optimized for low power and great performance per watt. So we stuck em on smartphones and tablets. Now perf per watt matters everywhere – datacenters and mobile devices are where the volume is. In addition, the performance gap is closing. x86 is still ahead in many single-threaded workloads you just want completed as fast as possible. But once that Rubicon is crossed, there will be very few cases where x86 makes sense as a CPU architecture that anyone intentionally uses.

Most major hardware OEMs are either shipping or looking into ARM. The battery life implications are too hard to ignore. The Apple M1 means Apple is unlikely to release another x86 device ever again. Microsoft has been partnering with Qualcomm to design ARM CPUs for years now, and Windows 10 on ARM has x86-64 emulation in preview as of December 2020. In other words, we are about 1 or 2 hardware cycles away from the mass availability of ARM devices and platforms that support ~99% of use cases consumers need [1].

The major cloud providers will eventually be forced to migrate to ARM for the cost savings. At AWS there’s already work underway to switch over internal workloads to ARM. Azure may be secretly testing ARM for internal workloads too. The cost savings is just too great not to do it. Google Cloud will come around to it once they start caring about margins.

For customer facing workloads, Amazon graviton2 VMs will soon be a reasonable default for customers, given it can be 40% faster and 20% cheaper than x86. For many cloud services that aren’t VMs, the ARM switch will be completely transparent, since HTTP and other layer 7 protocols don’t care what your underlying CPU architecture is. Or layers 1 – 6.

On a semi-related note, new Intel CEO Pat Gelsinger is taking over while under assault from all sides. Reports are that Intel has been in trouble for years now, with fat margins that are increasingly unsustainable and under attack from AMD, Amazon, and now Apple. Then you have dedicated fabs (but mostly TSMC) that have been pulling away and are now probably a full generation ahead of Intel. So in summary, Intel makes CPUs on a legacy architecture that consumes vastly more power than the competition, and is slowly losing its performance edge without anything on the horizon that can reverse the trend, with manufacturing capability is as much as 5 years behind the state of the art, and can’t even produce 10nm chips in mass quantities with decent yield even in early 2021. Yikes.

[1] I’m not sure how x86 on ARM virtualization will work, which may be important for some workstation scenarios so developers can build and test all their software. But the single most important event of the last 2 years has been Apple getting ARM laptops to developers, which lit a fire under folks like Docker to get cracking on ARM support.

We are talking about a roughly 97% decrease in price per GB over 20 years. In fact it’s kind of hard to see on a linear scale, so let’s switch to logarithmic for this time period:

Yup. Definitely logarithmic. Let’s zoom in to the past 10 years or so (linear scale):

For fun we can also superimpose a chart from BackBlaze, a backup service that shared their average price per hard drive through 2Q 2017:

The 2011 Thailand monsoons left a noticeable blip on prices in early 2012, but otherwise the trend is pretty clear: there’s been a deceleration of progress, with gains getting a lot slower starting around 2016. To date there’s been a lot of resistance getting below $0.02 per GB. So the era of just assuming storage will get significantly cheaper every year is over, at least for now. In fact, prices don’t seem to have really dropped at all since mid 2019 when some drives started selling for $0.019 / GB.

However, there’s a nuance we’re missing if we look at only price per GB without accounting for density. Let’s look at a snapshot of hard drive prices at various capacities from December 2020 on Newegg:

The prices for 3.5″ desktop drives range from $40 for a 500GB drive to ~$645 for an enterprise 18TB. We can conclude that at $40 the cost of the drive is almost entirely dominated by fixed costs, since the price doesn’t actually change at all between 250GB and 1TB, and barely changes at 2TB (the cheapest 2TB drive is roughly $50).

Because of this fixed cost issue, it’s very hard to get good “bang for buck” at lower drive capacities. Here’s the cost per GB as a function of drive capacity:

And here’s the same chart on a log scale:

For many years the sweet spot was a 4TB drive, but now it looks like the absolute best deal might be the 8TB drives. However, I’m surprised to see some 14TB and 16TB drives get within striking distance ($0.021 per GB vs 0.019 per GB). If the larger capacity drives are roughly as reliable, then going for 16TB drives is a no-brainer, since they’ll consume less space and power than two 8TB drives.

Because of this, and despite $/GB not really dropping much, big datacenters and cloud providers are likely to continue seeing marginal decreases in total cost of ownership and operational costs per GB-month, but absent new breakthrough technology, not that much progress.

What are the implications of this?

A number of startup darlings have succeeded by betting on trends in underlying technologies to make their business viable (e.g., YouTube was founded 15 years ago and would’ve remained hopelessly unprofitable if not for the massive drop in bandwidth prices). And many a startup capitalized on the drop in storage prices to build attractive data warehousing and database solutions, replacing technology that assumed a GB cost $15. But if you’re betting on the price of say, AWS S3 pricing to go down significantly in the near future to make your startup profitable, well, bad news:

Amazon literally hasn’t reduced prices for S3 since 2016, almost 5 years now.

But it’s easy to fall into this trap by espousing something that sounds like an opinionated stance without actually taking one, like “The codebase should look like it was written by one person!”

No. Stop. Don’t tell me what ideals you want. Tell me what you are prepared to sacrifice for it. Is there a comprehensive style guide? Has everyone been trained on it? Have you adopted a process for how to amend it? Do you make ship-blocking change requests for style (citing the style guide)?

Saying you want the codebase to be uniform is like saying you want to be a published author, or a PhD. It’s not the declaration of desire that matters. Are you committing the time and effort it takes to bend reality in that direction?

It’s more important to live up to cultural values than to have ones that are the nicest-sounding.

The reasoning is sound: Engineers generally aren’t drawn to listings with “must be able to grind through mountains of red tape, bureaucracy, and slow process. We deploy once a quarter.” And every tech company thinks it is fast-paced[1] the way every driver thinks s/he is above average.

But what, exactly, does fast paced mean? No, I don’t think you can use meter sticks like “we release every (day|week|month)” – or at least, you leave a lot of room for gaming that kind of metric (“I FTP’d a new config into the production server today”).

I think there’s a reasonable test that is a good proxy for how fast-paced any engineering environment is, phrased as a question: “Say the app has a one character typo in some production code. How long, in the best case, does it take to go from someone in the company knowing about this to it being fixed?”

Yes, this is stuff they teach in Six Sigma called “lead time” and its a metric so simple that it is probably simplistic. But you learn a lot about the maturity of an organization with this one question, because you can ask this of a specific engineer on a specific team and they’re going to start going down the mental checklist of how to actually deploy changes and then you can have a deep conversation about QA, automation, build times, and deployment process.

You might be surprised about the answers you get!

[1] Conflating ‘long hours’ with ‘fast paced’ is

I am talking, of course, about automobiles.

It’s been more than 100 years since mass production at scale, so we can safely say automobiles are a mature industry. Most of the incumbents are quite old, and new startups that disrupt them are rare – Tesla being a notable exception[1]. You typically do not see rapid advances that make older models obsolete within a few short years. Things today look roughly how they looked a decade ago. Etc.

In contrast, dial-up became broadband became WiFi became 4G LTE in short order. LiveJournal gave way to Friendster gave way to MySpace which gave way to Facebook. The conventional wisdom is that disruption is always just around the corner. That the right college dropout could change the game again at any moment. But perhaps our faith in the perpetual motion machine of the technology industry is…unfounded? What if we were just in the crazy early wild west days, and are starting to settle into a long haul “maturity” phase? What if it’s already happened?

There is some evidence that this has at least started. Here is Crunchbase data on startup formation over the last 10 years:

There is clearly a worrying trend towards fewer startups being founded after peaking in 2014.

US IPOs are noisier, but data going back to 2005 suggests a slightly declining trend as well:

More worrying is how much the “fat head” has accumulated. The top 5 largest technology companies today (Apple, Amazon, Microsoft, Google, and Facebook) have a collective market cap of $6.8 trillion (reminder: that’s $6,800 billion, or as many as 6,800,000 millionaires). They’re worth more than the combined market cap of every health care company in the S&P 500.

There are other signs of maturity in the technology industry. Regulation and enforcement that was unthinkable just a few years ago are now the norm (see: Europe). Countries all over the world are exerting more control over technology and information companies than ever before (see: Google in Europe, Facebook in China, TikTok in the US, Apple in…everywhere).

What does this mean for policymakers? For entrepreneurs?

For policymakers, there’s some low hanging fruit around the elimination of startup killers like employment non-competes, and maybe immigration reform that stops discouraging talent from contributing to the industry here. But the secular decline in new company formation follows a yearslong trend that isn’t just limited to technology startups, suggesting the problem is at least partially “dynamism in the broader economy,” and not just some attribute of the technology industry specifically [2].

For entrepreneurs, it can be deceptively expensive to start a new business, depending on your industry. Yes, Amazon Web Services is cheap. But the real costs are regulatory (and the people to deal with the regulations, and the health plans for those people, and the people to deal with the health plans, ad nauseum), especially if you want to reach the broadest audience possible. Learn to love acronyms like CASL, CCPA, and GDPR. And know that regulation will generally have the unintended effect of entrenching the existing big players, so today’s incumbents are more likely to stay dominant than in the past. That in turn suggests that a successful new entrant’s best hope is to be acquired, rather than to “disrupt” the likes of, say, FAANG.

In other words, we are very, very, far from “On the internet, nobody knows you’re a dog.” It isn’t necessarily a bad thing if software and the information industry is growing up. But it’s almost certainly a mistake to assume the churn from the early years will continue at the same pace.

[1] Tesla is the leader in the still niche battery-electric vehicle category, but still has only ~1% US automotive market share, which is a tremendouss achievement! But also goes to show how low the bar for disruption is.

[2] We could fix healthcare, which would help tremendously with new business formation across the board, but that’s not low hanging fruit.

There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:

It’s harder to read code than to write it.

This is why code reuse is so hard. This is why everybody on your team has a different function they like to use for splitting strings into arrays of strings. They write their own function because it’s easier and more fun than figuring out how the old function works.

It’s been more than 20 years since, so this is no longer considered a “subtle reason,” having firmly graduated into the sphere of common knowledge. But Joel never really explored in depth why reading code is more difficult than writing it. Many explanations have been offered over the years:

• Because when you are writing code, you often don’t immediately clutter it with all the exception handling and edge cases, but this is often present when reading code.
• Because programming languages attempt to balance readability with brevity.
• In most languages, small and seemingly insignificant notational changes can change the meaning of code.
• Because programming languages can’t be read aloud.
• Because when you are writing code you know the problem context, the mental model, and you know what the code is supposed to do.
• Because reading code requires working memory to remember and hold references to other places and things (functions, variable names, imported classes from other files, etc), only some subset of which will turn out to be relevant. In contrast, when writing code, you only need to remember precisely what you need.

Contrast this with, say, English prose: The meaning of the text does not generally depend on making references to other concepts defined only elsewhere in the text. It can be read aloud, in a linear order. And the grammar and syntax in English is more loose, because ambiguous constructions are allowed (“John greeted my dog and then he left.”).

So it’s hard to argue that working memory requirements, references, and syntax makes code hard to read. They certainly do. But everything mentioned so far hits at peripheral and contributory issues without truly nailing the central reason: human language is designed exclusively to encode ideas. Its only purpose is to record and transmit meaning. But programming languages are instead designed to encode computer commands unambiguously; documenting the meaning of the problem to be solved is left to optional activities like choosing good variable and method names, adding code comments, and generally things that have nothing to do with the correctness or successful execution of the program.

So reading code is hard because we must simultaneously decode the literal meaning of the instructions alongside the documentation about the problem it is designed to solve. And as most professional software developers come to learn, it takes great skill to write non-trivial programs correctly, but even more skill to do it in a way that simultaneously encodes the intent and meaning of the code too.

We should…

Hire and pay for world class people
Hire economically

Be together
Work anywhere

Emphasize quality
Emphasize speed

Emphasize market share growth
Emphasize profitability

Be highly leveraged
Be highly self sufficient

Prioritize technology updates
Prioritize technology stability

Focus on experience
Focus on performance

Build for generalists
Build for developers

Be disciplined and methodical
Have urgency and bias for action

Stretch ourselves
Be predictable

Be compassionate
Be right

Build simplicity
Build robustness

Think long term
Think “winner takes all”

Value correctness
Value shipping early

Be collaborative
Be decisive

Deploy and iterate
Stabilize the product

Value working software first
Value complete and comprehensive documentation

Deliver on schedule
Always deliver quality

Be transparent
Exercise discretion

Value professional growth over team stability
Value stability over learning and growth opportunities

Try to fix problems, even if they’re not ours
Stay in our lane

Be radically candid, even if it hurts
All get along

You can distill this down into a few easy calls that apply in almost all situations:

• No, don’t code your own database from scratch, use an existing open source or commercial one instead.
• No, don’t self-host a database on your own hardware, if you can use an equivalent managed cloud service instead.
• No, definitely don’t build and self-host your own database on your own hardware.

There are a few nuances and exceptions to this rule for small startups:

At the very earliest “founders and 1-2 employees” phase, it can make sense to avoid expensive managed services, if you can save precious cash for something else. This is because there can often be a marginal 9th (or 10th…or 11th) hour in a working day that doesn’t actually displace anything else — in other words, build some sweat equity in lieu of spending cash, which might work out if it lets you delay or skip fundraising, and keep more of the pie for yourself. If you’re a small founding team, that might be a reasonable trade to bet on (the benefits get too diffuse once headcount exceeds a dozen or two). This happens to be important for an ancillary reason, namely, it can be a sign that early folks are resourceful enough to not need money to solve every problem. I cringe at co-founder pairs spending hundreds a month on “time tracking” and “project management” software when Google Sheets and Trello is a thing.

For an early stage company with more than a handful of employees, sometimes DIY or self-hosting can still make sense even if the math doesn’t appear to pan out. This may be because 1) at this moment there isn’t anything else that would be more valuable for that engineer to build, and/or 2) it doesn’t represent incremental cash outflow – you’re already paying an engineer, so that salary commitment is a sunk cost; they should do whatever work results in the most cash savings or increased revenue — even if individual projects don’t always cover the cost of the employee’s salary. In other words, it might be smarter to avoid the option that definitely increases cash burn (buying a new service).

Additional build vs buy factors to consider include a few things that are easy to get wrong, and sometimes effectively unknowable:

• “The innocent hack that ends up being a mission-critical beast”: If something custom but innocuous gets turned into this irreplaceable mission-critical thing, then it becomes a chain around your neck forever and gets harder to rip out as you grow more dependent on it. And you’ll have this unique snowflake with its own documentation, training and support cost that you can’t hire externally for. It becomes debt in the classic sense, in that it commits future resources to its care and feeding.
• Feature set and customization: It can be tempting to either “build your own” (or fork something, which is almost the same) rather than using just things off the shelf because “nothing else suits our use case.” This almost always causes operational issues later [1], and is generally symptomatic of some other issue — either the team is insufficiently imaginative/adaptable, has adopted unusually broken workflows, or is just plain unskilled enough to actually read existing code and documentation. Some teams do truly need to build something custom for certain business processes, but this should be very rare early on.

So, a couple takeaways.

Frugality as a value only moves in one direction.

Have an exit strategy if you DIY, since you usually don’t want to be trapped maintaining and extending a homebrew solution forever that doesn’t create a competitive strength for you.

If you keep saying “nothing out there suits my use case” in every domain you encounter, the odds are you’re doing it wrong.

[1] I always like to say that forking is easy, merging is hard. This happens to be true in a number of domains (like processes), not just code libraries.

What intrigues me is how often new software engineers are changing jobs to get significant raises 1, 2, and 3 years outside of school. I see a lot of resumes and it’s actually somewhat unusual if I don’t see a new job every ~2 years on average. That seems to be a lot of wasted domain expertise and skills, as its an immediate net loss when a productive engineer leaves an employer to go be a training burden on a different one.

It seems like it’d be better, overall, if engineers switched jobs less often, because then they wouldn’t be constantly throwing out experience and domain knowledge from their old employers all the time, and the economy would be bigger from all that extra accumulated productivity. So why is this not the case?

One reason I think the observed behavior makes sense: Relevant real world experience dramatically changes the productivity of a software engineer, and the early gains can be the steepest. And I’ve mentioned before that I suspect many engineers at many companies aren’t being re-leveled fast enough to keep pace with the increase in their productivity early in their careers. So maybe the engineer has become much more productive and does the rational thing by jumping to an employer that is willing to share more of that value with higher compensation. I would expect this to be true in thick markets like SF, NYC, and Boston (but haven’t yet found data to back it up).

Also — and I recognize this is true because the dot-com bust happened before I started working — the entire market has been expanding for basically my entire career (OK, markets were in free fall in 2008, but software merely hit the pause button compared to the rest of the economy). So maybe there’s some kind of wage elasticity / stickiness problem…I think in labor economics it’s understood that when demand goes down, it’s very hard to straight up cut employee salaries, so correspondingly, salaries don’t magically move upward when the market moves up. Actually, now that we mention it, many firms are still coming out of Great Recession mode, where it seems like everything is on the table except raising salaries. Combine that with the fact that demand has been ahead of supply for a straight decade, and that seems like an environment with lots of natural incentives for engineers to hop around every year or two to get raises (really, just salaries at the new market rate). And so then you get tables like this:

And then there’s the typical whining about millennial job hopping, how they’re the least engaged generation in the workplace, etc. Employers have become less attached to their workforce in the past generation and vice versa, so there’s probably a self-reinforcing loop there, but also…that isn’t specific to the software industry so it’s not super interesting to talk deeply about. But the pain of interviewing and starting a new job just usually doesn’t make sense unless 1) you’re making more comp / advancing your career, or 2) you’re optimizing for something else, like lifestyle or geography or deciding you need your career to be rewarding and fulfilling because if I’m checking my email and answering pagerduty on nights and weekends I want to not hate every moment of it. Maybe there’s some deeper cultural thing here that’s really hard to untangle and there’s nothing anyone can do about it ¯\_(ツ)_/¯

So I guess there are a few not-exactly-exclusive reasons the current state of the world makes total sense:

• It’s a “multi-armed bandit” problem, where engineers job hop even if they are satisfied, exploring if they might be even more satisfied somewhere else.
• Demand is outstripping supply slightly (and in some markets, a lot), but employers are in denial because they kind of had it easy from roughly 2008 to 2011, and aren’t responding to market conditions fast enough. There’s some evidence of this if you look at quit rates over a recent 10 year period:

• Engineers get more productive with experience and for some self-sabotaging reason, firms are making it easy for their own engineers to get a raise by switching employers.

The other fun thing to note is that developer salaries are definitely headed up, but outside of a few metro areas the rise in national salaries has been relatively modest and not that far ahead of inflation:

But also, the industry has roughly doubled in these years, so account for the likely fact that the seniority of the average developer has dropped during this observation period.

References

• https://fred.stlouisfed.org/series/LEU0254477200A
• https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm