The following article originally appeared on Jennifer Pahlka’s Eating Policy website and is being republished here with the author’s permission.

I read Norman Maclean’s Young Men and Fire when I was a teenager, I think, so it’s been many years, but I still remember its turning point vividly. It’s set in 1949 in Montana, at the Gates of the Mountains Wilderness, about an hour north of Helena. A fire is burning, and the Forest Service sends out their smokejumpers to fight it. But the fire changes direction without warning, and a group of smokejumpers working in the Mann Gulch find themselves trapped, facing certain death. Instead of running, the foreman, Wag Dodge, pulls out matches and does the unthinkable: He lights a fire.

Today we know what he was doing. The escape fire consumed the fuel around him, allowing the main fire to pass over him and a few of his colleagues. But in 1949, the families of the 13 other smokejumpers who died accused Wag of causing their deaths. To them, what he had done made no sense.

I love that Marina Nitze, Matthew Weaver, and Mikey Dickerson chose this story as a framing device for their new book, Crisis Engineering: Time-Tested Tools for Turning Chaos Into Clarity, out now. Not just because it brought back the memory of a book that I once loved, but because Maclean’s obsessive investigation of what had happened back then (he wrote the book years after the incident) seemed to me almost as heroic as the bravery of the smokejumpers. And indeed, his insistence on making sense of what happened has probably saved lives. Escape fires are now formally recognized and taught as a last resort tactic when training new firefighters.

The Dodge escape fire wouldn’t seem to have much to do with Three Mile Island or healthcare.gov or the pandemic unemployment insurance backlogs, but the authors use it to make a point about how action and understanding interact in a crisis. One key is exactly what Maclean himself did so well: sensemaking. In a crisis like Mann Gulch, sensemaking disintegrates: a broken radio, wind so strong communication is impossible, fire whose behavior violates well-tested assumptions, and a team scattered. You don’t achieve sensemaking by staring at a map; you achieve it by acting and observing results. Wag Dodge didn’t understand fire behavior well enough to explain the escape fire in advance. But his actions created the understanding itself—retrospectively, as all real sensemaking is.

The book’s key claim is that crises are opportunities, and the authors leverage Daniel Kahneman’s Thinking, Fast and Slow to explain why crises are the only real windows for organizational change—and why everything else, the incentives, the logical arguments, the reorganizations, mostly doesn’t work. Most organizations, most of the time, run on autopilot. People habituate to their environment, rationalize away small surprises, and build stable stories about how things work. A crisis breaks this. When surprise accumulates faster than the brain’s “surprise-removing machinery” can rationalize it away, the whole apparatus jams, and organizations become, briefly, reprogrammable.

An institution resolves a crisis in one of three ways, according to the authors. It makes durable deliberate change, it dies, or, most commonly, it rationalizes the failure into an accepted new normal. “Most large organizations contain programs and departments that passively accept abject failure: infinitely long backlogs, hospitals that kill patients, devastating school closures that do little to affect a pandemic. These are fossils of past crises where the organization failed to adapt.”

Too many of our public institutions have failed to adapt, and the idea that they might be reprogrammable at all is a bit radical. We live in an era when too many people have given up on them, willing to burn them to the ground rather than renovate them. If crises represent the chance for true transformation, then we’d better get a lot better at using them for that. This is explicitly why Crisis Engineering exists, and it’s a detailed, practical book—the theory and framing devices are well used, but there’s a ton of pragmatic substance here you’ll be grateful for when the moment comes.

I remember when I was working in the White House and frustrated by the slow pace of progress. My UK mentor Mike Bracken told me: “Hold on, you just need a crisis. You Americans only ever change in crisis.” Boom. About two months later, healthcare.gov had its inauspicious start. And he was right. Change followed. Not all the change we needed, but a start. Marina, Weaver, and Mikey are three of the people who drove that change. I got to work with them again the first summer of the pandemic on California’s unemployment insurance claims backlog. I’m not a crisis engineer, but their strategies and tactics have deeply influenced how I think about the work I do and how I think we’re going to get from the institutions we have today to the ones we need.

We may be living in an era when too many people have given up on institutions, but we are also likely entering an era of crisis, and even polycrisis. This makes for uncomfortable math, but also drives home the need for a new generation of crisis engineers.

When I first read about Mann Gulch, so many years ago, I remember being in awe of the ingenuity and courage it took to start Wag Dodge’s escape fire. Today I think a lot about that pattern: the controlled burns that reduce the risk of megafires, the little earthquakes that take the pressure off faults under great tension, the managed crises that, if we’re skilled enough to use them, keep our institutions from the kind of collapse that comes when nothing has been allowed to give for too long. Dodge didn’t burn things down. He burned a path through. We’re going to have to get good at that.

As I get older, I increasingly think about whether I’m spending my time the right way to advance my career and my life. This is also a question that your company asks about you every performance cycle: Is this engineering manager spending their time effectively to advance the company or their organization?

Confusingly, in my experience, answering these nominally similar questions has surprisingly little in common. This piece spends some time exploring both questions in the particularly odd moment we live in today, where managers are being told they’ve spent the last decade doing the wrong things and need to engage with a new model of engineering management in order to be valued by the latest iteration of the industry.

Good Leadership Is a Fad

When I started my software career at Yahoo in the late 2000s, I had two 1:1s with my manager over the course of two years. The first one came a few months after I started, and he mostly asked me about a colleague’s work quality. The second came when I gave notice that I was leaving to join Digg. A modern evaluation of this manager would be scathing, but his management style closely resembled that of the team leader in The Soul of a New Machine: identifying an important opportunity for the team and navigating the broader organization that might impede progress towards that goal. He was, in the context we were working in, an effective manager.

Compare that leadership style to the expectations of the 2010s, where attracting, retaining, and motivating engineers was emphasized as the most important leadership criteria in many organizations. This made sense in the era of hypergrowth, where budgets were uncapped and many companies viewed hiring strong engineers as their constraint on growth. This was an era where managers were explicitly told to stop writing software as the first step of their transition into management, and it was good advice! Looking back we can argue it was bad guidance by today’s standards, but it aligned the managers with the leadership expectations of the moment.

Then think about our current era, that started in late 2022, where higher interest rates killed zero interest-rate policy (ZIRP) and productized large language models are positioned as killing deep engineering organizations. We’ve flattened engineering organizations where many roles that previously focused on coordination are now expected to be hands-on keyboard, working deep in the details. Once again, the best managers of the prior era—who did exactly what the industry asked them to do—are now reframed as bureaucrats rather than integral leaders.

In each of these transitions, the business environment shifted, leading to a new formulation of ideal leadership. That makes a lot of sense: Of course we want leaders to fit the necessary patterns of today. Where things get weird is that in each case a morality tale was subsequently superimposed on top of the transition:

• In the 2010s, the morality tale was that it was all about empowering engineers as a fundamental good. Sure, I can get excited for that, but I don’t really believe that narrative: It happened because hiring was competitive.
• In the 2020s, the morality tale is that bureaucratic middle management has made organizations stale and inefficient. The lack of experts has crippled organizational efficiency. Once again, I can get behind that—there’s truth here—but the much larger drivers aren’t about morality; they’re about ZIRP ending and optimism about productivity gains from AI tooling.

The conclusion here is clear: The industry will want different things from you as it evolves, and it will tell you that each of those shifts is because of some complex moral change, but it’s pretty much always about business realities changing. If you take any current morality tale as true, then you’re setting yourself up to be severely out of position when the industry shifts again in a few years, because “good leadership” is just a fad.

Earlier this summer, I also gave a presentation at the Y Combinator CTO summit on this specific topic of the evolution of engineering management. You can watch a recorded practice run of that talk on YouTube as well, and see the slides.

Self-Development Across Leadership Fads

If you accept the argument that the specifically desired leadership skills of today are the result of fads that frequently shift, then it leads to an important followup question: What are the right skills to develop to be effective today and to be impactful across fads?

Having been and worked with engineering managers for some time, I think there are eight foundational engineering management skills, which I want to personally group into two clusters: core skills that are essential to operate in all roles (including entry-level management roles), and growth skills whose presence—or absence—determines how far you can go in your career.

The core skills are:

Execution: Lead the team to deliver expected tangible and intangible work. Fundamentally, management is about getting things done, and you’ll neither get an opportunity to begin managing nor stay long as a manager if your teams don’t execute. Examples: Shipping projects, managing on-call rotation, sprint planning, managing incidents.

Team: Shape the team and the environment such that they succeed. This is not working for the team, nor is it working for your leadership. It is finding the balance between the two that works for both. Examples: Hiring, coaching, performance management, advocating with your management.

Ownership: Navigate reality to make consistent progress, even when reality is difficult. Find a way to get things done rather than finding a way that not getting it done is someone else’s fault. Examples: Doing hard things, showing up when it’s uncomfortable, being accountable despite systemic issues.

Alignment: Build shared understanding across leadership, stakeholders, your team, and the problem space. Find a realistic plan that meets the moment, without surprising or being surprised by those around you. Examples: Documenting and sharing top problems, and updates during crises.

The growth skills are:

Taste: Exercise discerning judgment about what “good” looks like—technically, in business terms, and in process/strategy. Taste is a broad church, and my experience is that broad taste is a somewhat universal criteria for truly senior roles. In some ways, taste is a prerequisite to Amazon’s “Are Right, A Lot.” Examples: Refining proposed product concept, avoiding high-risk rewrites, finding usability issues in team’s work.

Clarity: Your team, stakeholders, and leadership know what you’re doing and why, and agree that it makes sense. In particular, they understand how you are overcoming your biggest problems. So clarity is not “struggling with scalability issues” but instead “sharding the user logins database in a new cluster to reduce load.” Examples: Identifying levers to progress, creating plan to exit a crisis, showing progress on implementing that plan.

Navigating ambiguity: Work from complex problem to opinionated, viable approach. If you’re given an extremely messy, open-ended problem, can you still find a way to make progress? (I’ve written previously about this topic.) Examples: Launching a new business line, improving developer experience, going from 1 to N cloud regions.

Working across timescales: Ensure your areas of responsibility make progress across both the short and long term. There are many ways to appear successful by cutting corners today that end in disaster tomorrow. Success requires understanding, and being accountable for, how different timescales interact. Examples: Having an explicit destination, ensuring short-term work steers towards it, being long-term rigid and short-term flexible.

Having spent a fair amount of time pressure-testing these, I’m pretty sure most effective managers, and manager archetypes, can be fit into these boxes.

Self-assessing these skills

There’s no perfect way to measure anything complex, but here are some thinking questions for you to spend time with as you assess where you stand on each of these skills:

Execution

• When did your team last have friction delivering work? Is that a recurring issue?
• What’s something hard you shipped that went really, really well?
• When were you last pulled onto solving a time-sensitive, executive-visible project?

Team

• Who was the last strong performer you hired?
• Have you retained your strongest performers?
• What strong performers want to join your team?
• Which peers consider your team highly effective?
• When did an executive describe your team as exceptional?

Ownership

• When did you or your team overcome the odds to deliver something important? (Would your stakeholders agree?)
• What’s the last difficult problem you solved that stayed solved (rather than reoccurring)?
• When did you last solve the problem first before addressing cross-team gaps?

Alignment

• When was the last time you were surprised by a stakeholder? What could you do to prevent that reoccurring?
• How does a new stakeholder understand your prioritization trade-offs (including rationale)?
• When did you last disappoint a stakeholder without damaging your relationship?
• What stakeholders would join your company because they trust you?

Taste

• What’s a recent decision that is meaningfully better because you were present?
• If your product counterpart left, what decisions would you struggle to make?
• Where’s a subtle clarification that significantly changed a design or launch?
• How have you inflected the team’s outcomes by seeing around corners?

Clarity

• What’s a difficult trade-off you recently helped your team make?
• How could you enable them to make that same trade-off without your direct participation?
• What’s a recent decision you made that was undone? How?

Navigating ambiguity

• What problem have you worked on that was stuck before you assisted and unstuck afterwards?
• How did you unstick it?
• Do senior leaders bring ambiguous problems to you? Why?

Working across timescales

• What’s a recent trade-off you made between short- and long-term priorities?
• How do you inform these trade-offs across timescales?
• What long-term goals are you protecting at significant short-term cost?

Most of these questions stand on their own, but it’s worth briefly explaining the “Have you ever been pulled into a SpecificSortOfProject by an executive?” questions. My experience is that in most companies, executives will try to poach you onto their most important problems that correspond to your strengths. So if they’re never attempting to pull you in then either you’re not considered as particularly strong on those dimensions or you’re already very saturated with other work such that it doesn’t seem possible to pull you in.

Are “core skills” the same over time?

While those groupings of “core” and “growth” skills are obvious groupings to me, what I came to appreciate while writing this is that some skills swap between core to growth as the fads evolve. Where execution is a foundational skill today, it was less of a core skill in the hypergrowth era, and even less in the investor era.

This is the fundamentally tricky part of succeeding as an engineering manager across fads: You need a sufficiently broad base across each of these skills to be successful, otherwise you’re very likely to be viewed as a weak manager when the eras unpredictably end.

Stay Energized to Stay Engaged

The “Manage Your Priorities and Energy” chapter in The Engineering Executive’s Primer captures an important reality that took me too long to understand: The perfect allocation of work is not the mathematically ideal allocation that maximizes impact. Instead, it’s the balance between that mathematical ideal and doing things that energize you enough to stay motivated over the long haul. If you’re someone who loves writing software, that might involve writing a bit more than helpful to your team. If you’re someone who loves streamlining an organization, it might be improving a friction-filled process that is a personal affront, even if it’s not causing that much overall inefficiency.

Forty-Year Career

Similarly to the question of prioritizing activities to stay energized, there’s also understanding where you are in your career, an idea I explored in “A Forty-Year Career.”

For each role, you have the chance to prioritize across different dimensions like pace, people, prestige, profit, or learning. There’s no “right decision,” and there are always trade-offs. The decisions you make early in your career will compound over the following forty years. You also have to operate within the constraints of your life today and your possible lives tomorrow. Early in my career, I had few responsibilities to others and had the opportunity to work extremely hard at places like Uber. Today, with more family responsibilities, I am unwilling to make the trade-offs to consistently work that way, which has real implications on how I think about which roles to prioritize over time.

Recognizing these trade-offs, and making them deliberately, is one of the highest value things you can do to shape your career. Most importantly, it’s extremely hard to have a career at all if you don’t think about these dimensions and have a healthy amount of self-awareness to understand the trade-offs that will allow you to stay engaged over half a lifetime.

We anticipate that 2026 will be a year of increased accountability. Expect enterprises to shift focus from experimentation to measurable business outcomes and sustainable AI costs. There are promising productivity and efficiency gains to be had in software engineering and development, operations, security, and product design, but significant challenges also persist.  

Bigger picture, the industry is still grappling with what AI is and where we’re headed. Is AI a worker that will take all our jobs? Is AGI imminent? Is the bubble about to burst? Economic uncertainty, layoffs, and shifting AI hiring expectations have undeniably created stark career anxiety throughout the industry. But as Tim O’Reilly pointedly argues, “AI is not taking jobs: The decisions of people deploying it are.” No one has quite figured out how to make money yet, but the organizations that succeed will do so by creating solutions that “genuinely improve. . .customers’ lives.” That won’t happen by shoehorning AI into existing workflows but by first determining where AI can actually improve upon them, then taking an “AI first” approach to developing products around these insights.

As Tim O’Reilly and Mike Loukides recently explained, “At O’Reilly, we don’t believe in predicting the future. But we do believe you can see signs of the future in the present.” We’re watching a number of “possible futures taking shape.” AI will undoubtedly be integrated more deeply into industries, products, and the wider workforce in 2026 as use cases continue to be discovered and shared. Topics we’re keeping tabs on include context engineering for building more reliable, performant AI systems; LLM posttraining techniques, in particular fine-tuning as a means to build more specialized, domain-specific models; the growth of agents, as well as the protocols, like MCP, to support them; and computer vision and multimodal AI more generally to enable the development of physical/embodied AI and the creation of world models. 

Here are some of the other trends that are pointing the way forward.

Software Development

In 2025, AI was embedded in software developers’ everyday work, transforming their roles—in some cases dramatically. A multitude of AI tools are now available to create code, and workflows are undergoing a transformation shaped by new concepts including vibe coding, agentic development, context engineering, eval- and spec-driven development, and more.

In 2026, we’ll see an increased focus on agents and the protocols, like MCP, that support them; new coding workflows; and the impact of AI on assisting with legacy code. But even as software development practices evolve, fundamental skills such as code review, design patterns, debugging, testing, and documentation are as vital as ever.

And despite major disruption from GenAI, programming languages aren’t going anywhere. Type-safe languages like TypeScript, Java, and C# provide compile-time validation that catches AI errors before production, helping mitigate the risks of AI-generated code. Memory safety mandates will drive interest in Rust and Zig for systems programming: Major players such as Google, Microsoft, Amazon, and Meta have adopted Rust for critical systems, and Zig is behind Anthropic’s most recent acquisition, Bun. And Python is central to creating powerful AI and machine learning frameworks, driving complex intelligent automation that extends far beyond simple scripting. It’s also ideal for edge computing and robotics, two areas where AI is likely to make inroads in the coming year.

Takeaways

Which AI tools programmers use matter less than how they use them. With a wide choice of tools now available in the IDE and on the command line, and new options being introduced all the time, it’s useful to focus on the skills needed to produce good code rather than focusing on the tool itself. After all, whatever tool they use, developers are ultimately responsible for the code it produces.

Effectively communicating with AI models is the key to doing good work. The more background AI tools are given about a project, the better the code they generate will be. Developers have to understand both how to manage what the AI knows about their project (context engineering) and how to communicate it (prompt engineering) to get useful outputs.

AI isn’t just a pair programmer; it’s an entire team of developers. Software engineers have moved beyond single coding assistants. They’re building and deploying custom agents, often within complex setups involving multi-agent scenarios, teams of coding agents, and agent swarms. But as the engineering workflow shifts from conducting AI to orchestrating AI, the fundamentals of building and maintaining good software—code review, design patterns, debugging, testing, and documentation—stay the same and will be what elevates purposeful AI-assisted code above the crowd.

Software Architecture

AI has progressed from being something architects might have to consider to something that is now essential to their work. They can use LLMs to accelerate or optimize architecture tasks; they can add AI to existing software systems or use it to modernize those systems; and they can design AI-native architectures, an approach that requires new considerations and patterns for system design. And even if they aren’t working with AI (yet), architects still need to understand how AI relates to other parts of their system and be able to communicate their decisions to stakeholders at all levels.

Takeaways

AI-enhanced and AI-native architectures bring new considerations and patterns for system design. Event-driven models can enable AI agents to act on incoming triggers rather than fixed prompts. In 2026, evolving architectures will become more important as architects look for ways to modernize existing systems for AI. And the rise of agentic AI means architects need to stay up-to-date on emerging protocols like MCP.

Many of the concerns from 2025 will carry over into the new year. Considerations such as incorporating LLMs and RAG into existing architectures, emerging architecture patterns and antipatterns specifically for AI systems, and the focus on API and data integrations elevated by MCP are critical.

The fundamentals still matter. Tools and frameworks are making it possible to automate more tasks. However, to successfully leverage these capabilities to design sustainable architecture, enterprise architects must have a full command of the principles behind them: when to add an agent or a microservice, how to consider cost, how to define boundaries, and how to act on the knowledge they already have.

Infrastructure and Operations

The InfraOps space is undergoing its most significant transformation since cloud computing, as AI evolves from a workload to be managed to an active participant in managing infrastructure itself. With infrastructure sprawling across multicloud environments, edge deployments, and specialized AI accelerators, manual management is becoming nearly impossible. In 2026, the industry will keep moving toward self-healing systems and predictive observability—infrastructure that continuously optimizes itself, shifting the human role from manual maintenance to system oversight, architecture, and long-term strategy.

Platform engineering makes this transformation operational, abstracting infrastructure complexity behind self-service interfaces, which lets developers deploy AI workloads, implement observability, and maintain security without deep infrastructure expertise. The best platforms will evolve into orchestration layers for autonomous systems. While fully autonomous systems remain on the horizon, the trajectory is clear.

Takeaways

AI is becoming a primary driver of infrastructure architecture. AI-native workloads demand GPU orchestration at scale, specialized networking protocols optimized for model training and inference, and frameworks like Ray on Kubernetes that can distribute compute intelligently. Organizations are redesigning infrastructure stacks to accommodate these demands and are increasingly considering hybrid environments and alternatives to hyperscalers to power their AI workloads—“neocloud” platforms like CoreWeave, Lambda, and Vultr.

AI is augmenting the work of operations teams with real-time intelligence. Organizations are turning to AIOps platforms to predict failures before they cascade, identify anomalies humans would miss, and surface optimization opportunities in telemetry data. These systems aim to amplify human judgment, giving operators superhuman pattern recognition across complex environments.

AI is evolving into an autonomous operator that makes its own infrastructure decisions. Companies will implement emerging “agentic SRE” practices: systems that reason about infrastructure problems, form hypotheses about root causes, and take independent corrective action, replicating the cognitive workload that SREs perform, not just following predetermined scripts.

Data

The big story of the back half of 2025 was agents. While the groundwork has been laid, in 2026 we expect focus on the development of agentic systems to persist—and this will necessitate new tools and techniques, particularly on the data side. AI and data platforms continue to converge, with vendors like Snowflake, Databricks, and Salesforce releasing products to help customers build and deploy agents. 

Beyond agents, AI is making its influence felt across the entire data stack, as data professionals target their workflows to support enterprise AI. Significant trends include real-time analytics, enhanced data privacy and security, and the increasing use of low-code/no-code tools to democratize data access. Sustainability also remains a concern, and data professionals need to consider ESG compliance, carbon-aware tooling, and resource-optimized architectures when designing for AI workloads.

Takeaways

Data infrastructure continues to consolidate. The consolidation trend has not only affected the modern data stack but also more traditional areas like the database space. In response, organizations are being more intentional about what kind of databases they deploy. At the same time, modern data stacks have fragmented across cloud platforms and open ecosystems, so engineers must increasingly design for interoperability. 

A multiple database approach is more important than ever. Vector databases like Pinecone, Milvus, Qdrant, and Weaviate help power agentic AI—while they’re a new technology, companies are beginning to adopt vector databases more widely. DuckDB’s popularity is growing for running analytical queries. And even though it’s been around for a while, ClickHouse, an open source distributed OLAP database used for real-time analytics, has finally broken through with data professionals.

The infrastructure to support autonomous agents is coming together. GitOps, observability, identity management, and zero-trust orchestration will all play key roles. And we’re following a number of new initiatives that facilitate agentic development, including AgentDB, a database designed specifically to work effectively with AI agents; Databricks’ recently announced Lakebase, a Postgres database/OLTP engine integrated within the data lakehouse; and Tiger Data’s Agentic Postgres, a database “designed from the ground up” to support agents.

Security

AI is a threat multiplier—59% of tech professionals cited AI-driven cyberthreats as their biggest concern in a recent survey. In response, the cybersecurity analyst role is shifting from low-level human-in-the-loop tasks to complex threat hunting, AI governance, advanced data analysis and coding, and human-AI teaming oversight. But addressing AI-generated threats will also require a fundamental transformation in defensive strategy and skill acquisition—and the sooner it happens, the better.

Takeaways

Security professionals now have to defend a broader attack surface. The proliferation of AI agents expands the attack surface. Security tools must evolve to protect it. Implementing zero trust for machine identities is a smart opening move to mitigate sprawl and nonhuman traffic. Security professionals must also harden their AI systems against common threats such as prompt injection and model manipulation.

Organizations are struggling with governance and compliance. Striking a balance between data utility and vulnerability requires adherence to data governance best practices (e.g., least privilege). Government agencies, industry and professional groups, and technology companies are developing a range of AI governance frameworks to help guide organizations, but it’s up to companies to translate these technical governance frameworks into board-level risk decisions and actionable policy controls.

The security operations center (SOC) is evolving. The velocity and scale of AI-driven attacks can overwhelm traditional SIEM/SOAR solutions. Expect increased adoption of agentic SOC—a system of specialized, coordinated AI agents for triage and response. This shifts the focus of the SOC analyst from reactive alert triage to proactive threat hunting, complex analysis, and AI system oversight.

Product Management and Design

Business focus in 2025 shifted from scattered AI experiments to the challenge of building defensible, AI-native businesses. Next year we’re likely to see product teams moving from proof of concept to proof of value. 

One thing to look for: Design and product responsibilities may consolidate under a “product builder”—a full stack generalist in product, design, and engineering who can rapidly build, validate, and launch new products. Companies are currently hiring for this role, although few people actually possess the full skill set at the moment. But regardless of whether product builders become ascendant, product folks in 2026 and beyond will need the ability to combine product validation, good-enough engineering, and rapid design, all enabled by AI as a core accelerator. We’re already seeing the “product manager” role becoming more technical as AI spreads throughout the product development process. Nearly all PMs use AI, but they’ll increasingly employ purpose-built AI workflows for research, user-testing, data analysis, and prototyping.

Takeaways

Companies need to bridge the AI product strategy gap. Most companies have moved past simple AI experiments but are now facing a strategic crisis. Their existing product playbooks (how to size markets, roadmapping, UX) weren’t designed for AI-native products. Organizations must develop clear frameworks for building a portfolio of differentiated AI products, managing new risks, and creating sustainable value. 

AI product evaluation is now mission-critical. As AI becomes a core product component and strategy matures, rigorous evaluation is the key to turning products that are good on paper into those that are great in production. Teams should start by defining what “good” means for their specific context, then build reliable evals for models, agents, and conversational UIs to ensure they’re hitting that target.

Design’s new frontier is conversations and interactions. Generative AI has pushed user experience beyond static screens into probabilistic new multimodal territory. This means a harder shift toward designing nonlinear, conversational systems, including AI agents. In 2026, we’re likely to see increased demand for AI conversational designers and AI interaction designers to devise conversation flows for chatbots and even design a model’s behavior and personality.

What It All Means

While big questions about AI remain unanswered, the best way to plan for uncertainty is to consider the real value you can create for your users and for your teams themselves right now. The tools will improve, as they always do, and the strategies to use them will grow more complex. Being deeply versed in the core knowledge of your area of expertise gives you the foundation you’ll need to take advantage of these quickly evolving technologies—and ensure that whatever you create will be built on bedrock, not shaky ground.

We call this configuration drift, and for decades, our industry’s solution has been to throw more scripts at the problem. We write massive, brittle ETL (Extract-Transform-Load) pipelines that attempt to scrape the world and shove it into a relational database. It never works. The “world”—especially the modern cloud native world—moves too fast.

We realized we couldn’t solve this problem by writing better scripts. We had to change the fundamental architecture of how we sync data. We stopped trying to boil the ocean and fix the entire enterprise at once. Instead, we focused on one notoriously difficult environment: Kubernetes. If we could build an autonomous agent capable of reasoning about the complex, ephemeral state of a Kubernetes cluster, we could prove a pattern that works everywhere else. This article explores how we used the newly open-sourced Codex CLI and theModel Context Protocol (MCP) to build that agent. In the process, we moved from passive code generation to active infrastructure operation, transforming the “stale CMDB” problem from a data entry task into a logic puzzle.

The Shift: From Code Generation to Infrastructure Operation with Codex CLI and MCP

The reason most CMDB initiatives fail is ambition. They try to track every switch port, virtual machine, and SaaS license simultaneously. The result is a data swamp—too much noise, not enough signal. We took a different approach. We drew a small circle around a specific domain: Kubernetes workloads. Kubernetes is the perfect testing ground for AI agents because it’s high-velocity and declarative. Things change constantly. Pods die; deployments roll over; services change selectors. A static script struggles to distinguish between a CrashLoopBackOff (a temporary error state) and a purposeful scale-down. We hypothesized that a large language model (LLM), acting as an operator, could understand this nuance. It wouldn’t just copy data; it would interpret it.

The Codex CLI turned this hypothesis into a tangible architecture by enabling a shift from “code generation” to “infrastructure operation.” Instead of treating the LLM as a junior programmer that writes scripts for humans to review and run, Codex empowers the model to execute code itself. We provide it with tools—executable functions that act as its hands and eyes—via the Model Context Protocol. MCP defines a clear interface between the AI model and the outside world, allowing us to expose high-level capabilities like cmdb_stage_transaction without teaching the model the complex internal API of our CMDB. The model learns to use the tool, not the underlying API.

The architecture of agency

Our system, which we call k8s-agent, consists of three distinct layers. This isn’t a single script running top to bottom; it’s a cognitive architecture.

The cognitive layer (Codex + contextual instructions): This is the Codex CLI running a specific system prompt. We don’t fine-tune the model weights. Infrastructure moves too fast for fine-tuning: A model trained on Kubernetes v1.25 would be hallucinating by v1.30. Instead, we use context engineering—the art of designing the environment in which the AI operates. This involves tool design (creating atomic, deterministic functions), prompt architecture (structuring the system prompt), and information architecture (deciding what information to hide or expose). We feed the model a persistent context file (AGENTS.md) that defines its persona: “You are a meticulous infrastructure auditor. Your goal is to ensure the CMDB accurately reflects the state of the Kubernetes cluster. You must prioritize safety: Do not delete records unless you have positive confirmation; they are orphans.”

The tool layer: Using MCP, we expose deterministic Python functions to the agent.

• Sensors: k8s_list_workloads, cmdb_query_service, k8s_get_deployment_spec
• Actuators: cmdb_stage_create, cmdb_stage_update, cmdb_stage_delete

Note that we track workloads (Deployments, StatefulSets), not Pods. Pods are ephemeral; tracking them in a CMDB is an antipattern that creates noise. The agent understands this distinction—a semantic rule that is hard to enforce in a rigid script.

The state layer (the safety net): LLMs are probabilistic; infrastructure must be deterministic. We bridge this gap with a staging pattern. The agent never writes directly to the production database. It writes to a staged diff. This allows a human (or a policy engine) to review the proposed changes before they are committed.

The OODA Loop in Action

How does this differ from a standard sync script? A script follows a linear path: Connect → Fetch → Write. If any step fails or returns unexpected data, the script crashes or corrupts data. Our agent follows the Observe-Orient-Decide-Act (OODA) loop, popularized by military strategists. Unlike a linear script that executes blindly, the OODA loop forces the agent to pause and synthesize information before taking action. This cycle allows it to handle incomplete data, verify assumptions, and adapt to changing conditions—traits essential for operating in a distributed system.

Let’s walk through a real scenario we encountered during our pilot, the Ghost Deployment, to explore the benefits of using an OODA loop. A developer had deleted a deployment named payment-processor-v1 from the cluster but forgot to remove the record from the CMDB. A standard script might pull the list of deployments, see payment-processor-v1 is missing, and immediately issue a DELETE to the database. The risk is obvious: What if the API server was just timing out? What if the script had a bug in its pagination logic? The script blindly destroys data based on the absence of evidence. 

The agent approach is fundamentally different. First, it observes: Calling k8s_list_workloads and cmdb_query_service, noticing the discrepancy. Second, it orients: Checking its context instructions to “verify orphans before deletion” and deciding to call k8s_get_event_history. Third, it decides: Seeing a “delete” event in the logs, it reasons that the resource is missing and that there’s been a deletion event. Finally, it acts: Calling cmdb_stage_delete with a comment confirming the deletion. The agent didn’t just sync data; it investigated. It handled the ambiguity that usually breaks automation.

Solving the “Semantic Gap”

This specific Kubernetes use case highlights a broader problem in IT operations: the “semantic gap.” The data in our infrastructure (JSON, YAML, logs) is full of implicit meaning. A label “env: production” changes the criticality of a resource. A status CrashLoopBackOff means “broken,” but Completed means “finished successfully.” Traditional scripts require us to hardcode every permutation of this logic, resulting in thousands of lines of unmaintainable if/else statements. With the Codex CLI, we replace those thousands of lines of code with a few sentences of English in the system prompt: “Ignore jobs that have completed successfully. Sync failing Jobs so we can track instability.” The LLM bridges the semantic gap. It understands what “instability” implies in the context of a job status. We’re describing our intent, and the agent is handling the implementation.

Scaling Beyond Kubernetes

We started with Kubernetes because it’s the “hard mode” of configuration management. In a production environment with thousands of workloads, things change constantly. A standard script sees a snapshot and often gets it wrong. An agent, however, can work through the complexity. It might run its OODA loop multiple times to solve a single issue—by checking logs, verifying dependencies, and confirming rules before it ever makes a change. This ability to connect reasoning steps allows it to handle the scale and uncertainty that breaks traditional automation.

But the pattern we established, agentic OODA Loops via MCP, is universal. Once we proved the model worked for Pods and Services, we realized we could extend it. For legacy infrastructure, we can give the agent tools to SSH into Linux VMs. For SaaS management, we can give it access to Salesforce or GitHub APIs. For cloud governance, we can ask it to audit AWS Security Groups. The beauty of this architecture is that the “brain” (the Codex CLI) stays the same. To support a new environment, we don’t need to rewrite the engine; we just hand it a new set of tools. However, shifting to an agentic model forces us to confront new trade-offs. The most immediate is cost versus context. We learned the hard way that you shouldn’t give the AI the raw YAML of a Kubernetes deployment—it consumes too many tokens and distracts the model with irrelevant details. Instead, you create a tool that returns a digest—a simplified JSON object with only the fields that matter. This is context optimization, and it is the key to running agents cost-effectively.

Conclusion: The Human in the Cockpit

There’s a fear that AI will replace the DevOps engineer. Our experience with the Codex CLI suggests the opposite. This technology does not remove the human; it elevates them. It promotes the engineer from a “script writer” to a “mission commander.” The stale CMDB was never really a data problem; it was a labor problem. It was simply too much work for humans to manually track and too complex for simple scripts to automate. By introducing an agent that can reason, we finally have a mechanism capable of keeping up with the cloud. 

We started with a small Kubernetes cluster. But the destination is an infrastructure that is self-documenting, self-healing, and fundamentally intelligible. The era of the brittle sync script is over. The era of infrastructure as intent has begun!

Few professional software developers will find it surprising that software development teams are respondents said that productivity is the biggest challenge their organization faced, and another 19% said that time to market and deployment speed are the biggest challenges. Those two answers are almost the same: decreasing time to market requires increasing productivity, and improving deployment speed is itself an increase in productivity. Together, those two answers represented 48% of the respondents, just short of half.

HR issues were the second-most-important challenge, but they’re nowhere near as pressing. 12% of the respondents reported that job satisfaction is the greatest challenge; 11% said that there aren’t good job candidates to hire; and 10% said that employee retention is the biggest issue. Those three challenges total 33%, just one-third of the respondents.

It’s heartening to realize that hiring and retention are still challenges in this time of massive layoffs, but it’s also important to realize that these issues are less important than productivity.

But the big issue, the issue we wanted to explore, isn’t the challenges themselves; it’s what organizations are doing to meet them. A surprisingly large percentage of respondents (28%) aren’t making any changes to become more productive. But 20% are changing their onboarding and upskilling processes, 15% are hiring new developers, and 13% are using self-service engineering platforms.

We found that the biggest struggle for developers working with new tools is training (34%), and another 12% said the biggest struggle is “ease of use.” Together, that’s almost half of all respondents (46%). That was a surprise, since many of these tools are supposed to be low- or no-code. We’re thinking specifically about tools like GitHub Copilot, Amazon CodeWhisperer, and other code generators, but almost all productivity tools claim to make life simpler. At least at first, that’s clearly not true. There’s a learning curve, and it appears to be steeper than we’d have guessed. It’s also worth noting that 13% of the respondents said that the tools “didn’t effectively solve the problems that developers face.”

Over half of the respondents (51%) said that their organizations are using self-service deployment pipelines to increase productivity. Another 13% said that while they’re using self-service pipelines, they haven’t seen an increase in productivity. So almost two-thirds of the respondents are using self-service pipelines for deployment, and for most of them, the pipelines are working—reducing the overhead required to put new projects into production.

Finally, we wanted to know specifically about the effect of GitHub Copilot, ChatGPT, and other AI-based programming tools. Two-thirds of the respondents (67%) reported that these tools aren’t in use at their organizations. We suspect this estimate is lowballing Copilot’s actual usage. Back in the early 2000s, a widely quoted survey reported that CIOs almost unanimously said that their IT organizations weren’t making use of open source. How little they knew! Actual usage of Copilot, ChatGPT, and similar tools is likely to be much higher than 33%. We’re sure that even if they aren’t using Copilot or ChatGPT on the job, many programmers are experimenting with these tools or using them on personal projects.

What about the 33% who reported that Copilot and ChatGPT are in use at their organizations? First, realize that these are early adopters: Copilot was only released a year and a half ago, and ChatGPT has been out for less than a year. It’s certainly significant that they (and similar tools) have grabbed a third of the market in that short a period. It’s also significant that making a commitment to a new way of programming—and these tools are nothing if not a new kind of programming—is a much bigger change than, say, signing up for a ChatGPT account.

11% of the respondents said their organizations use Copilot and ChatGPT, and that the tools are primarily useful to junior developers; 13% said they’re primarily useful to senior developers. Another 9% said that the tools haven’t yielded an increase in productivity. The difference between junior and senior developers is closer than we expected. Common wisdom is that Copilot is more of an advantage to senior programmers, who are better able to describe the problem they need to solve in an intricate set of prompts and to notice bugs in the generated code quickly. Our survey hints that the difference between senior and junior developers is relatively small—although they’re almost certainly using Copilot in different ways. Junior developers are using it to learn and to spend less time solving problems by looking up solutions on Stack Overflow or searching online documentation. Senior developers are using it to help design and structure systems, and even to create production code.

Is developer productivity an issue? Of course; it always is. Part of the solution is improved tooling: self-service deployment, code-generation tools, and other new technologies and ideas. Productivity tools—and specifically the successors to tools like Copilot—are remaking software development in radical ways. Software developers are getting value from these tools, but don’t let the buzz fool you: that value doesn’t come for free. Nobody’s going to sit down with ChatGPT, type “Generate an enterprise application for selling shoes,” and come away with something worthwhile. Each has its own learning curve, and it’s easy to underestimate how steep that curve can be. Developer productivity tools will be a big part of the future; but to take full advantage of those tools, organizations will need to plan for skills development.

Paradigm Waves

Paradigm shifts in computing are as regular as waves on a beach, it’s hard to see where they came from and even harder to see where they are going. We have seen shifts from mainframe computers to personal computers, and from servers to the cloud. Each shift presented new challenges and opportunities, shaping the way we interact with technology. The most recent large-scale shift was from servers to the cloud, driven by an acknowledgment that using commodity servers run by experts is a better choice for most businesses. Serverless APIs are the culmination of the cloud commoditizing the old hardware-based paradigm. The same process of commoditization that gave rise to the cloud will also bring about the next paradigm, creating a new wave of abstractions and a rising tide for tomorrow’s applications.

“Make yourself a monopoly by growing the markets around you … Smart companies try to commoditize their products’ complements.”
— Joel Spolsky

This iconic Joel Spolsky quote is a testament to his deep understanding of the technology industry and its market dynamics. Spolsky, a renowned software engineer and entrepreneur, co-founded Fog Creek Software, Stack Overflow, and Trello. With years of experience in the field, he has developed keen insights into business strategies and the importance of commoditization in the tech sector. His quote emphasizes the need for companies to create monopolies by commoditizing complementary products, which has proven to be a successful approach for many businesses. This means making the hardware supply chain into a commodity if you make PCs, making PCs into commodities if you sell operating systems, and making servers a commodity by promoting serverless function execution if you sell cloud. What goes around comes around as the cloud becomes the next commodity, and the independent crew of cloudless innovators, the next monopoly breakers.

From the cloud to the network

The new paradigm shift is from the cloud to the protocol network. Protocol networks are groups of loosely affiliated enterprises that provide globally available services like ledger, compute, and storage. Just as serverless is the culmination of the cloud, this move to protocol networks will culminate in cloudless APIs, leading to applications driven by protocols with incentives and capabilities that go beyond what the cloud’s location-based paradigm can offer. They run on any cloud or other data-center and reward service providers through fees they collect from users.

The new paradigm shift is from the cloud to the protocol network. Protocol networks are groups of loosely affiliated enterprises that provide globally available services like ledger, compute, and storage. Just as serverless is the culmination of the cloud, this move to protocol networks will culminate in cloudless APIs, leading to applications driven by protocols with incentives and capabilities that go beyond what the cloud’s location-based paradigm can offer. They run on any cloud or other data-center and reward service providers through fees they collect from users.

We call this new paradigm of network protocol based infrastructure cloudless. Its benefits include cost and security improvements as well as lower cognitive overhead and operational burden for developers, users, operators, and enterprises stemming from its location-independence and cryptographic verifiability. This is a technical consequence of content addressing, the hash based identifier system widely used in storage networks and leveraged by peer-to-peer networks for global addressability. We’ll discuss the technical underpinnings of cloudless later in this article.

As the cloud becomes commoditized and more developers, businesses, and users become aware of Cloudless computing’s advantages, such as increased data privacy, greater resilience, and lower costs, there will likely be a stronger inclination to embrace these new platforms. The move to more abstract APIs has an element of structural inevitability. As protocol networks emerge and gain traction, we can anticipate a phase change in the technological landscape, akin to the formation of a solid crystal structure from a less stable liquid state. The availability of these new networks serves as a catalyst for change, driving a more rapid transition from cloud-based systems to cloudless computing.

Most consumer-facing apps have not been written in a location-independent way thus far, primarily because the required infrastructure was not yet available to realize the benefits. However, with the advent of Cloudless protocols, we are witnessing a new wave of applications that harness the potential of these technologies. Early adopters have focused on smart contracts and decentralized apps (dApps), but the next wave is much more extensive, encompassing social applications, provable AI execution and training data provenance, big data processing like transcode or map-reduce, and asset delivery for gaming, metaverse, and media. These use cases exemplify the transformative nature of cloudless computing, showcasing its potential to revolutionize various industries and redefine the way we interact with technology.

Cost Savings and New Business Models

The democratizing effects of the cloudless paradigm are poised to revolutionize the future of applications. For instance, even beginners can create applications that go viral without incurring centralized costs. Imagine crafting a social media app (or remixing an existing one), sharing it with friends, and witnessing it go viral—all without having to pay a bill. Cloudless computing reduces cognitive overhead for developers, users, operators, and enterprises, leading to more cost-effective solutions.

Without the need for apps to pay a centralized hosting bill, we can expect to see a broader range of voices and perspectives represented. The user-driven nature of cryptographic identity makes it easier to write new apps using existing data sets, as users can easily grant innovative new chat apps or photo galleries access to their data. This will lead to the development of novel applications we can’t even imagine yet. Data DAOs can sponsor the storage of content for the apps they represent. This is a great option for archives and other long-term storage institutions.

With cloudless computing, the inherent cacheability allows for time savings and increased focus on problem-solving. This cacheability goes beyond peer-to-peer and into the foundations of computing. Automated data analysis workloads, in the research world and commercially, are heavy users of shared data and reusable computing, and have the most to gain from automatic data provenance tracking and verification. Data scientists coding in notebooks like Databricks frequently rerun the same transformations on source data. Cacheability is part of what helped leading database vendor Snowflake dominate the market, and now, with cloudless, microservices can be upgraded to use verifiable data and deterministic computing, leveraging cache liveness provided by the protocol network.

Decentralized hosting costs and management enable developers to overcome resource limitations and handle computational loads more effectively. Protocol nodes can be placed in retail locations or edge sensors, allowing for compute to be performed over data at rest at the edge. This eliminates the need for unnecessary data transfers and enables faster, more efficient querying of data. As cryptographic verifiability makes trust more fungible, enterprises will be able to run their businesses on a competitive market of computing infrastructure and specialized algorithm vendors, resulting in lower costs and more robust applications.

Cryptographic identity, verifiable data, and deterministic computing make cloudless apps possible. With location-independent links to data and compute, anyone can access data and execute functions anywhere. By making intelligent use of today’s existing cloud providers and network infrastructure, Cloudless protocols add a layer of resilience and trust, enabling new applications and business models, and paving the way for a more interconnected and efficient computing landscape.

Commoditizing the Cloud: How is Cloudless Possible?

Cloudless computing is built upon the principles of decentralization, collaboration, and shared innovation, and its success is dependent on embracing open source and open standards. This approach ensures that the underlying technologies can be continuously improved, adapted, and maintained by a diverse community of stakeholders, eliminating the risk of vendor lock-in, promoting interoperability, and enabling a more resilient and flexible infrastructure. Cloudless computing offers several advantages over serverless cloud computing, such as cost savings, increased choice for developers, and the potential for new business models centered on app and data ownership. These benefits are made possible by the core foundations of cloudless computing: cryptographic identity, verifiable data, and deterministic compute. In the following sections, we will delve into the features that make cloudless apps possible.

Cryptographic Identity

Cryptographic identity is fundamental to cloudless computing. It addresses the identity problem and its challenges by leveraging the increasing familiarity with private keys, signing transactions, and verifying hashes. Recent advancements in user experience, such as TouchID/FaceID and secure enclave, have made cryptographic key pairs more accessible to average users, setting the stage for the next generation of applications that take advantage of cryptographic guarantees. The operating system and browser vendors offer password management products that many people are familiar with. Cloudless capability delegation feels a lot like a password manager, only instead of passwords it uses secure signatures, reducing the risk of leaks and hacking.

In this self-sovereign model, users control their own crypto keyrings, granting them greater visibility and authority over their data and online interactions. This eliminates the need for reliance on centralized service providers and prevents lock-in. Access to accounts is maintained by delegating capabilities to other cryptographic actors, such as other devices or account recovery services.

In other words, “No lockouts/no lock-in.”

Verifiable Data

Verifiable data enables the storage and retrieval of data that is independently verifiable and authenticated using cryptographic techniques. The peer-to-peer web protocol, IPFS (InterPlanetary File System), for example, uses hash-based Content Identifiers (CIDs) to ensure data integrity and authenticity. These CIDs allow data to be fetched from any location, using location-independent URIs, and provide a layer of safety that systems relying on location-based addresses (like URLs) cannot offer. Because hash-based identifiers are deterministically and uniquely derived from the content they reference, they are unforgeable and tamper-proof, providing the robust foundation for cloudless applications like smart contracts, distributed identity, storage, and compute.

Applications that use verifiable data can benefit from improved security, lower computing costs, and better performance. Global addressability means CIDs enable data to migrate to the most appropriate provider without any loss of trust, and the immutable nature of these addresses allows for efficient caching and acceleration.

Deterministic Compute

Deterministic computing allows for consistent and predictable computations, regardless of data location or infrastructure. It requires a container runtime or execution environment, a way to address data consistently (such as with CIDs), and a secure and verifiable method to invoke the computation.

The benefits of deterministic computing include faster second runs, cost-effective and performant location selection, workload sharing and reuse, edge computing for reduced network costs and improved performance, and the ability to coalesce workloads for cost savings and accelerated output.

By moving from location-dependent APIs to location-agnostic APIs, cloudless computing can optimize data routing and enable greater flexibility and cost savings. This is exemplified by compute-over-data projects like Bacalhau, which leverage the guarantees of cryptographic identity, verifiable data, and deterministic computing to create a competitive marketplace for computing infrastructure and algorithm vendors.

Early adopters of verifiable data include industries like smart contracts, NFTs, and DAOs, as well as organizations focused on maintaining journalistic integrity in reporting. One notable example is the Starling Lab, a nonprofit academic research center that uses cryptography and decentralized protocols to maintain trust in sensitive digital records giving journalistic data the standard of evidence that can be used in war crimes trials. The lab employs the Starling Framework of “Capture, Store, Verify” for digital media, leveraging IPFS to provide a powerful solution for trust and integrity. Their work demonstrates how verifiable data is essential for preserving and maintaining trust in critical historical data, which can be applied to various other use cases.

The growing demand for verifiable data is shaping the future of cloudless computing and distributed identity systems. Factors such as the rise of cryptocurrencies, blockchain technology, regulations like GDPR, and advancements in AI and machine learning have contributed to the increasing need for verifiable data. As tools mature and the learning curve becomes less steep, organizations working on mission-critical data applications will increasingly adopt these technologies. Existing tools, such as programming notebooks and static site hosting, will evolve to use cloudless technology, further driving the adoption and impact of verifiable data in various industries and applications.

Now that we’ve reviewed the core enabling technologies, you can see how cloudless makes it possible to reduce costs and gain capabilities. Combining cryptographic identity, verifiable data, and deterministic compute allow for a more cost-efficient and flexible computing landscape, where users and applications can interact in ways not possible with traditional cloud-based systems. By leveraging cryptographic guarantees, cloudless computing unlocks a world of possibilities that extend beyond mere optimizations and cost savings, setting the stage for a future filled with new voices, applications, and opportunities.

Data Privacy and Ownership

As our digital lives become increasingly interconnected, the need for secure and user-friendly distributed identity systems grows more pressing. These systems are vital for protecting individual privacy and granting users control over their data. However, realizing the full potential of distributed identity systems requires overcoming numerous challenges, chief among them being the user experience. This section delves into the importance of UX in distributed identity systems, examining the latest innovations and trends that have improved security and usability, while also discussing the remaining challenges and how they can be addressed.

User experience (UX) is crucial for distributed identity systems, as it ensures ease of use, accessibility, and adoption for users of varying technical expertise. One of the most significant challenges is keypair management. Non-extractable keypairs, recently made available to the mainstream via WebAuthn and biometric authentication systems like TouchID and FaceID, have significantly improved the security and user experience in distributed identity systems.

WebAuthn is a modern web authentication standard that relies on authenticators, such as hardware security keys or platform-based authenticators like fingerprint scanners, to create and manage public-private key pairs securely. The private key remains securely stored on the authenticator and is never exposed, reducing the risk of key theft or unauthorized access.

The increasing familiarity with cryptography, fueled by the widespread adoption of cryptocurrency wallets like MetaMask, has also contributed to a better user experience in distributed identity systems.

Companies like Apple have played a significant role in improving the UX of distributed identity systems. Innovations like TouchID and FaceID, especially when used with open standards like WebAuthn, have made it easier for users to interact with such systems securely. WebAuthn supports non-extractable keypairs, providing enhanced security by ensuring that private keys are securely stored within authenticators and never exposed or extractable.

In addition to these security features, Apple’s iPhone setup process, which uses local radio and camera/screen inputs for secure pairing, is a great example of seamless user experience. This approach allows for easy capability delegation between device keys, ensuring that users can quickly and securely transfer data and settings between devices. It is worth noting that UCAN, a distributed authorization protocol, also leverages non-extractable keypairs and employs a similar delegation approach for enhanced security and user experience. Both Apple and UCAN demonstrate how integrating these concepts into distributed identity systems can result in a more intuitive and secure user experience.

In contrast, the open-source community often faces challenges in improving UX for distributed identity solutions. Solutions that cater to technical users may not be accessible or user-friendly for non-technical users. For instance, mnemonic passphrase private key sharing in cryptocurrency wallets may be suitable for tech-savvy users but not for the general population. To achieve a better user experience, developers need to invest time and effort in creating robust, user-friendly solutions.

As users become more sophisticated and technology becomes more user-friendly, the challenges of catering to users with less computing experience are gradually being addressed. A range of solutions for multi-signature recovery of crypto assets is available, spanning from powerful tools for geeks to easy-to-use options for non-technical users. The market will reward those with the most trustworthy UX, driving continuous improvement.

Emerging trends, technologies, and practices, such as the increasing demand for verifiable data, will contribute to improved data privacy and ownership through better UX in distributed identity systems. As enterprises recognize the cost-saving and performance-enhancing benefits of data verification, investment in UX for cryptographically aware toolchains will grow, resulting in more accessible and user-friendly cloudless solutions.

Not bullish on bearer tokens

In this section, we’ll dive into some technical details around the limitation of bearer tokens, the modern equivalent of cookies, as well as explore alternative cloudless solutions that promise enhanced security and efficiency. Although this discussion is a bit more technical in nature, we encourage readers of all backgrounds to stay engaged, as there is valuable information applicable to everyone. Following this section, we will broaden our focus to address further implications and opportunities in the realm of cloudless computing, data privacy, and distributed identity systems.

Bearer tokens, also known as access tokens or API keys, are commonly used in modern authentication and authorization systems to grant access to protected resources. They are typically issued by an authorization server and are passed along with each request to a resource server, which uses the token to determine whether the client has permission to access the requested resource. While bearer tokens have become a popular choice for authentication and authorization, they also come with several significant limitations.

One major issue with bearer tokens is that they encourage an architecture that routinely proxies data through multiple services. In many cases, a user’s device must send a request to a central service, which then forwards the request to another service with the bearer token attached. This process may be repeated multiple times before the data is ultimately returned to the user’s device. This proxying of data through multiple services is done to keep the bearer token secret and prevent it from being intercepted by a malicious actor, but it exacts a heavy cost in terms of performance, reliability, and resource use.

This proxying process is highly inefficient, as it adds multiple extra steps to the data retrieval process and can slow down the overall performance of the application. Additionally, it increases the risk of security breaches, as each service that handles the bearer token is a potential point of failure. Because bearer tokens are simply strings of characters that are passed along with each request, they can be easily intercepted and used by unauthorized parties if they are not properly protected. The more services that handle the bearer token, the greater the risk that it will be intercepted by a malicious actor.

Instead of dwelling on the risks of bearer tokens, let’s explore an alternative solution that leverages client-side cryptographic keys to create capabilities, delegations, invocations, and receipts that are safe to store-and-forward without the danger of replay attacks. This approach utilizes cryptographic proofs rather than bearer tokens. By signing each invocation as it is created, the client can safely send it to anyone on the network, who can route it to the service which will run it. This allows workloads to be coalesced and moved to the most cost-effective infrastructure, as described earlier as among the benefits of deterministic computing.

Centralized authentication systems, which often rely on bearer tokens, have their own set of issues. They are controlled by a single entity or organization, which can wield significant power over users and their data. These systems are also vulnerable to data breaches and hacking, resulting in sensitive information falling into the wrong hands. Furthermore, they favor data silos, making it difficult for users to share data across different platforms and services.

UCAN, or User-Controlled Authorization Networks, offers a decentralized access control protocol that enables secure and verifiable data routing by allowing users to delegate access to their capabilities using public key cryptography. Users can grant permission to access their data to other actors through the use of public keys, without the need for a central authority to manage authentication. With UCANs, users control the keys and delegations, and services can cryptographically verify proofs about the authorization data. UCANs rely on cryptographic signatures, reducing the risk of token leakage, stealing, and expiration.

This decentralized and location-independent approach to coding allows services to be composed without the need for a location-based proxy secret model and the risk of bearer token leakage. The “compute over data” model enables computations to be performed on the data, rather than the data being transported to the computation. This makes data routing possible in a secure and efficient manner, with computations performed and results signed by service providers without relying on intermediaries to handle and transmit the data.

By using verifiable data and UCAN, Cloudless computing demonstrates the benefits of a more secure, efficient, and user-controlled approach to authentication and authorization, moving away from the limitations and risks associated with traditional bearer tokens and centralized systems.

The democratization of app development

In the world of cloudless computing, a community of hobbyist developers can collaborate on a project, adding features and making modifications to the code as they see fit. Each member can spin up a copy of the app to experiment with, test, and improve. As the app evolves and attracts attention from others, it can grow and fork as new communities adopt the app. The cloudless nature of the project means there are no hosting bills, and the developers can avoid the crippling costs that often accompany the sudden popularity of a traditional application. This democratization of app development enables hobbyist developers to create and adapt applications without the limitations imposed by traditional platforms.

The relationship between hobbyist developers and platforms like GitHub fosters a thriving developer ecosystem. For example, the open-source project “TodoMVC” demonstrates the power of collaboration and forking on platforms like GitHub. Developers can easily compare different implementations of the same app using various frameworks and libraries, leading to numerous forks and adaptations as developers experiment and personalize the application. This collaborative environment is integral to the growth and success of open-source projects.

Decentralization empowers even hobbyist developers to address the same markets as mainstream applications, enabling them to create popular open-source projects without the constraints of traditional platforms. This leads to a more innovative and diverse app ecosystem, benefiting both developers and users alike.

Cloudless computing enables a wide range of innovative applications, such as:

1. Decentralized finance (DeFi) platforms and secure document databases for smart contracts.
2. Secure voting systems for election processes and corporate governance.
3. Supply chain management systems for traceability and transparency of goods and products.
4. Healthcare record management systems that are secure, immutable, and accessible.
5. Legal document management systems for secure and tamper-proof tracking of legal agreements and contracts.
6. Asset tracking systems for real-time tracking and management of physical assets.
7. Identity management systems for secure and decentralized authentication and authorization. In the new world, these can be as easy-to-use and informal as is Openlinks, which makes verifiable link-in-profile pages, or as serious as digital drivers licenses.
8. Environmental monitoring systems for real-time tracking of environmental conditions and data.
9. Compliance management systems for secure and transparent tracking of regulatory compliance.
10. Real estate management systems for secure and transparent tracking of property transactions and ownership.

These new types of apps present unique opportunities for hobbyist developers to create innovative solutions in various sectors, further driving the democratization of app development.

Automation tools, such as GitHub Actions, have emerged from the democratization of app development, supporting hobbyist developers and fostering a more inclusive developer ecosystem. By streamlining the software development process, these tools optimize developer productivity, ensure consistency, and elevate the overall project standard. Continuous integration and deployment enabled by automation tools allow developers to automatically test and build their code upon each commit, ensuring code quality and alignment with project standards. This approach reduces friction between team members, promotes a positive environment, and encourages open-source contributors to feel valued and respected. The result is a thriving, innovative, and successful developer community that benefits from collaboration and shared expertise.

The resilience and accessibility of cloudless computing pave the way for a Cambrian explosion of app developer voices. This democratization of app development breaks down barriers and empowers a diverse range of developers, including hobbyists, to create innovative applications without the constraints of traditional platforms. As we have seen with the unstoppable nature of crypto smart contracts, decentralization can lead to a flourishing ecosystem that transcends geographical, economic, and technical limitations.

The cloudless computing paradigm not only reduces costs and fosters collaboration but also enables developers to create secure, scalable, and efficient solutions across various industries. By embracing the potential of cloudless computing and learning from the success of peer-to-peer technologies like IPFS and Ethereum, we can expect a new wave of groundbreaking applications that enrich the lives of users worldwide.

Ultimately, this democratization of app development will lead to a more inclusive, innovative, and robust ecosystem, where diverse developer voices contribute to a brighter and more connected future.

Real-world examples

The advent of Cloudless computing has brought forth numerous groundbreaking applications and protocols that are already transforming the technological landscape. These early Cloudless applications not only showcase the innovative potential of this technology but also highlight the far-reaching impact it can have across various industries.

Smart contracts on platforms like Ethereum are one of the first and most well-known use cases of Cloudless computing. These self-executing contracts allow for secure and automated transactions on the blockchain, eliminating the need for intermediaries and reducing costs.

In the networking sphere, Socket Supply provides a runtime for decentralized applications, enabling developers to build and deploy their apps in a Cloudless environment. This approach promotes efficiency, security, and user control over data and logic.

For storage, Filecoin has emerged as a popular Cloudless solution that allows users to rent out their unused storage space and earn tokens in return. Filecoin leverages a decentralized network of storage providers, ensuring data redundancy and security.

Tableland, an API for decentralized databases, enables developers to build and deploy applications with user-owned data, ensuring privacy and data sovereignty.

Fission and BlueSky are also leading the charge in the Cloudless movement, focusing on giving users control over their data and the logic of the applications they interact with. These technologies empower users by decentralizing ownership and control of data and software, ensuring a more equitable and transparent digital landscape.

Long-standing protocols, such as DNS and HTTP, have paved the way for large-scale cooperation by insulating apps from implementation-specific details. Similarly, Ethereum and other blockchain technologies harness the power of peer-to-peer networks to create immutable logs, while the SWIFT message format enables secure store-and-forward messaging.

As Cloudless computing continues to evolve and mature, we can expect to see even more transformative applications and use cases across various industries. This paradigm shift will empower individuals, foster innovation, and ultimately reshape the digital world as we know it.

Cloudless computing has the potential to democratize the app development process, enabling hobbyist developers to create and share apps without the need for expensive hosting services. The transformative nature of Cloudless computing has already led to the emergence of innovative solutions in various industries, from healthcare to finance. With the development of decentralized hosting and management solutions, the cost and management of computational loads are reduced, allowing developers to handle resource limitations more effectively. The deployment of protocol nodes at the edge enables compute to be performed over data at rest, eliminating the need for unnecessary data transfers and improving the efficiency of querying data. As cryptographic verifiability makes trust more fungible, enterprises will be able to run their businesses on a competitive market of computing infrastructure and specialized algorithm vendors, resulting in lower costs and more robust applications. With Cloudless computing, we can expect a Cambrian explosion of app developer voices and unstoppable smart contract-powered experiences that will transform the way we interact with technology.

In Conclusion

As we look towards the future, the transformative power of cloudless computing is becoming increasingly evident. This revolutionary approach to application development and deployment offers numerous benefits, including reduced environmental impact, democratization of app development, enhanced data privacy, and new opportunities for developers and creators alike.

The environmental impact of cloudless computing cannot be overstated. By distributing computational resources across numerous devices and minimizing reliance on centralized data centers, energy consumption and carbon emissions can be significantly reduced. This decentralized approach to computing infrastructure not only promotes sustainability but also encourages innovative solutions for further reducing our digital footprint.

Developers are incentivized by the cloudless computing paradigm as it grants them greater freedom, flexibility, and access to markets previously dominated by mainstream applications. The ease of entry for hobbyist developers, facilitated by platforms like GitHub, fosters a vibrant and inclusive ecosystem that encourages creativity and collaboration.

The democratization of app development is further bolstered by the cloudless paradigm, breaking down barriers for independent developers and leveling the playing field. With the support of collaboration tools and automation like GitHub Actions, a more diverse range of developers can contribute to and benefit from this rapidly growing field.

Data privacy is another critical aspect of cloudless computing. By eliminating reliance on centralized cloud services, users can maintain greater control over their data and ensure that their information remains secure and private. This heightened level of privacy is particularly important in an era where data breaches and privacy concerns are increasingly common.

The rise of Web3 technologies and their impact on creators and rent-taker issues is also noteworthy. Decentralized platforms enable creators to retain control over their content, reduce fees paid to intermediaries, and foster more direct relationships with their audiences. As the Web3 ecosystem continues to evolve, cloudless computing will play a vital role in empowering creators and minimizing rent-seeking behaviors.

In addition to these broader benefits, cloudless computing brings forth a myriad of specific technologies and innovations. The use of UCAN invocations, IPFS, Merkle DAGs, immutable CIDs, and CAR transactions, are just a few examples of the tools that are shaping the future of cloudless computing. These advancements in data structures will eventually resemble GraphQL, SQL, and NoSQL database APIs, highlighting the potential for creating developer-friendly solutions.

Compute-over-Data (CoD) has become a practical way to run computations across large data archives, with compute-over-data projects like Bacalhau using immutable references to code and data to enable low cost big-data processing. Developers are increasingly leveraging tools like w3up and w3ui to delegate data uploads, reducing runtime requirements and avoiding unnecessary data transfers.

Optimized data routing and features like IPLD for HTML enable apps to run in the browser while still making UCAN calls that can be executed, cached, and stored anywhere on the network. The Saturn content delivery network will allow anyone to be compensated for accelerating these workloads.

Lastly, cloudless computing enables new capabilities and opportunities in the computing world. Innovative applications, such as secure voting systems, supply chain management systems, healthcare record management systems, and asset tracking systems, are just a few examples of the potential that cloudless computing offers. As more developers adopt this paradigm, we can expect to see even more groundbreaking innovations and advancements in the technology landscape.

The future of the cloudless paradigm is one of increased efficiency, cost savings, and empowerment for enterprises, developers, and individuals alike. As a market for computing and algorithms develops, data storage and serverless execution will transition from the centralized cathedral of big cloud providers to the decentralized bazaar of networked protocol participants.

A special thanks to the peer-to-peer and distributed data community for their invaluable contributions to the field of cloudless computing. Their dedication and innovation have significantly impacted this transformative technology, fostering a more decentralized, collaborative, and secure digital future. We appreciate their efforts and look forward to the continued growth of cloudless computing, thanks to their inspiring work and visionary leadership. Heartfelt thanks to the editors and individuals who provided feedback on the early drafts of this article. Your insights, suggestions, and attention to detail have been instrumental in shaping the final version.

Most of that money is spent on digital ads, like the ones that follow you across websites to offer you deals on items you’ve just bought. Those are typically based on your online activity. Ad networks trail behind you as you browse the web, trying to get an idea of who you are and what you’re likely to buy, so they can pair you with hopeful merchants.

While merchants are clearly happy with targeted ads—at least, I’d hope so, given how much they’re spending—consumers have, understandably, expressed concerns over personal privacy. Apple took note, and limited iOS apps’ ability to track users across sites. Google has announced changes that would further limit advertisers’ reach. Who knows? Maybe the next step will be that the ad industry gets stronger regulations.

There’s also the question of whether targeted advertising even works.  While the ad networks aren’t required to disclose their stats, there are even people inside those companies who think that their product is “almost all crap.”

Maybe it’s time for a different approach? Recently, Disney’s video streaming service, Disney+, threw its hat into the advertising ring by announcing a new ad-supported plan. (Credit where it’s due: I originally found this in Les Echos, which may be paywalled. Here’s the official, English-language press release from Disney.)

It may be easy to disregard this Disney+ move, since so much of the online world is ad-supported these days. But I think this merits more attention than it may seem on the surface.

To be clear: I have no inside information here. But it at least looks like Disney+ can run its ad platform in a fairly low-tech fashion while also preserving privacy. That’s a pretty big deal for Disney, for consumers, and for the wider space of online advertising.

Everything old is new again

To understand why, let’s first consider the idea of “content marketing.” This is a new term for the age-old practice of selling ad space next to curated content that aligns with a particular theme. For example, let’s say you’ve created a magazine about cars. Motoring enthusiasts will read your magazine, which means advertisers (merchants) who want to reach them will place ads in your pages. The content is what draws readers and advertisers to the same spot.

What’s nice about content marketing is that the ad’s placement is based on the content, not the specific person reading it.

This addresses the privacy concern at the core of targeted advertising, because content marketing doesn’t require that you build a detailed profile of a person based on their every browsing habit. You’re not pairing an ad to a person; you’re pairing an ad to a piece of content. So you shift your analytical focus from the reader to what they’re reading.

The mouse has a large library

Now, consider Disney: its catalog spans decades’ worth of cartoons, tween sitcoms, and movies. Its recent acquisition of the Star Wars franchise gives it access to an even wider fanbase. And don’t forget that Disney owns ESPN, which adds sports content to the portfolio. It now makes that content available through its video-on-demand (VOD) platform of Disney+.

Disney already has to keep track of that catalog of content as part of its day-to-day business, which means we can reasonably assume that every show, movie, and sporting event on Disney+ has been assigned some number of descriptive tags or labels.

From the perspective of content marketing, all of this adds up to Disney+ being able to place ads on that content without having to do much extra work. The parent company, Disney, already owns the content and it’s already been tagged. The depth and breadth of the video catalog will certainly attract a large number and wide variety of viewers. That shifts the heavy lifting to the ad-matching system, which connects advertisers with the content.

Tracking your ad budget

You’ve likely heard the John Wanamaker adage: “Half the money I spend on advertising is wasted; the trouble is, I don’t know which half.” It’s a well-founded complaint about billboard or magazine advertising, since an advertiser can’t really tell how many people saw a given ad.

(Some early advertising pioneers, David Ogilvy among them, learned to supply coupons with print ads so stores could track which one had resonated the most. While this added a new level of analytical rigor to the field, it still wasn’t a perfect solution to Wanamaker’s plight.)

Delivering content-based ads through a well-curated streaming platform addresses that somewhat. Disney+ can provide an advertiser a detailed analysis of their ad spend without revealing any individual’s identity: “N number of people watched Variant V, your ad for Product P, during Show S, with the following breakdowns for time of day…”

And that leads me to my next point:

Minimal ML/AI

When you review the setup—a curated and labeled catalog, with broad-brush marketing characteristics—Disney+ has the ability to run this ad service using minimal ML/AI.

(Once again: I’m speculating from the outside here. I don’t know for sure how much ML/AI Disney+ is using or plans to use. I’m working through one hypothetical-yet-seemingly-plausible scenario.)

Disney+ can use those content labels—”pro football,” “tween comedy,” “gen-X cartoon”—to pair a piece of content with an advertisement. They may not get a perfect hit rate on these ads; but given that they’re building on top of work they’ve already done (the catalog and the streaming platform) then the ad system can run at a relatively low cost. And providing stats to advertisers is a matter of counting. Since those calculations are so trivial, I expect the toughest part of that BI will be scaling it to Disney’s audience size.

Can Disney+ still use ML/AI in places? They most certainly can, but they don’t have to. Disney+ has the option to run this using a smaller team of data scientists and a far smaller data analysis infrastructure. Whether you call this “smaller budget” or “higher margins,” the net effect is the same: the company ends the day with money in its pocket.

Disney+ can task that ML team with building models that better tag content, or that improve matches between content and advertisers. They don’t have to spend money analyzing the specific actions of a specific individual in the hopes of placing ads.

Future-proofing the ad system

Assuming that the Disney+ ad system will indeed run on a content marketing concept, that means the company has one more card to play: They have just sidestepped potential future privacy laws that limit the use of personal information.

Yes, Disney+ can get a person’s contact information when they subscribe to the service. Yes, the company can track customer behavior on- and off-platform, through a mix of first- and third-party data. But, contrary to targeted advertising, they don’t need all of that to run ads. All the company needs is to pair content with an advertisement. Given that this is the modern-day equivalent of a billboard or newspaper article, I imagine it would be difficult for Disney+ to run afoul of any present-day or upcoming privacy regulation with such an ad setup.

There’s still some room for trouble…

Going back to our car magazine example, Disney’s library is the equivalent of hundreds or even thousands of magazines. And if a single magazine is a hint as to a single interest, what can a larger number of magazines tell us?

By tracking what content a person watches, how they watch it (phone, tablet, TV), and what time of day, Disney+ could infer quite a bit about that person and household: the number and age of adults; marital or relationship status; age and number of children; whether this is a multi-generational household; and even some clues as to viewers’ gender. (I emphasize the term “infer” here, since it would hardly be perfect.)

In turn, Disney could use this for ad targeting, or to provide even more-detailed breakdowns to advertisers, or even find ways to share the data with other companies. This could get creepy quickly, so let’s hope they don’t take this route. And based on what we’ve covered thus far, Disney+ has every opportunity to run an ad network that preserves a reasonable amount of privacy.

Could the tail someday wag the dog?

Another possible wrinkle would be in how advertising weighs on future content.

Disney already has a good eye for what people will want to watch. And right now, those viewers are Disney’s customers. But when Disney+ becomes an ad marketplace, they’ll officially be a middleman, which means they’ll have to keep both sides of the ad equation happy. At what point does Disney use the Disney+ advertising as a compass, feeding back into decisions around what content to create?

And would Disney ever stretch beyond its own character lines, to build TV and movies around someone else’s toys?  It’s not too far-fetched of an idea. In The Great Beanie Baby Bubble, author Zac Bisonette points out that:

[A TV show deal] was the kind of product-based programming that was responsible for billions per year in sales and could turn toys that no one wanted into hits through sheer exposure. Lines such as He-Man, My Little Pony, and the ThunderCats had all become hundred-million-dollar brands with the help of the product-based TV shows that accompanied their launches.

Creating content in one side of the businesses while running ads in the other, it’s not unlike running an investment bank and retail bank under one roof: sure, it can lead to all kinds of interesting business opportunities.  It can also lead to trouble.

When it comes to content marketing, you need to strike a balance: you want to create evergreen content, so you can continue to run ads. And when that content is going into the Disney catalog—some of which currently spans multiple generations—it has to be absolutely timeless. Giving in to the whims of a single advertiser, or a single fad, can lead to short-term gains but also short-lived content.

Beyond the Magic Kingdom

Despite those challenges, content marketing has huge potential for generating revenue, preserving privacy, and avoiding future regulation that could hinder targeted advertising. By building this system on BI and content tagging, Disney could do so at a smaller price tag than an AI-based, targeted-ad marketplace.

And this isn’t just a Disney opportunity. I’ve focused on them in this piece but other VOD providers have already seen the benefit in monetizing their catalog. According to Jason Kilar, former CEO of WarnerMedia, “Close to 50% of every new [HBO Max] subscriber is choosing the ad tier. Hulu, the last stat they shared publicly, is they are north of 60%.” Amazon will rename its ad-supported IMDb TV service to Freevee. (I first saw this in Der Spiegel; I’ve since found a US  press release.)  And Netflix, long a holdout in the ad-supported space, hinted at plans for a similar offering.

To be clear, content marketing at this scale is not exactly a get-rich-quick scheme. It works best for groups that already have a large amount of content—video, image, text, audio—that they can monetize. This certainly holds true for the platforms I’ve just mentioned. Maybe it’s also true for your company?

It may require getting creative as you comb through your attic. And maybe there’s an option for a new kind of ad marketplace, one that groups people with a small amount of content into a larger content ecosystem. Sort of like what EthicalAds does for developer documentation. If low-cost, non-invasive content marketing is an option, it can’t hurt to try.

Many thanks to Chris Butler for reviewing an early draft of this article. I always appreciate his insights. The section on the tail wagging the dog was based on his idea and I give him full credit for pointing this out to me.

Greed will bring you profits in the short term. Probably in the long term, as well.  As a middleman, though, your greed is an existential threat.  When you abuse your position and mistreat the parties you connect–when your cost outweighs your value–they’ll find a way to replace you. Maybe not today, maybe not tomorrow, but it will happen.

Luckily, you can make money as a middleman and still keep everyone happy.  Here’s how to create that win-win-win triangle:

Keep refining your platform

Running a marketplace is a game of continuous improvement. You need to keep asking yourself: how can I make this better for the people who interact through the marketplace?

To start, you can look for ways to make your platform more attractive to existing customers. I emphasize both customers, not just one side of the marketplace. Mistreating one side to favor the other may work for a time, but it will eventually fall through. Frustration has a way of helping people overcome switching costs.

Some stock exchanges designate market makers (“specialists,” if you’re old-school), firms that are always ready to both buy and sell shares of a given stock. If I want to offload a thousand shares and there’s no one who wants to buy them from me, the market maker steps in to play the role of the buyer. By guaranteeing that there will always be someone on the other side of the bid or ask, exchanges keep everyone happy.

If you constantly review how the two parties interact, you can look for opportunities to mitigate their risk, create new services, or otherwise reduce friction. Most platforms connect strangers, right?  So if you look at your business through the lens of safety, you’ll find a lot of work to do. Note how eBay’s review system provides extra assurance for buyers and sellers to trade with people they’ve never met.  Similarly, in the early days of online commerce, credit card issuers limited shoppers’ fraud risk to just $50 per purchase.  This improved consumers’ trust in online shopping, which helped make e-commerce the everyday norm that it is today.

Safety improvements also extend to communications. Do the parties really need to swap e-mail addresses or phone numbers?  If they’re just confirming a rideshare pickup or flirting through a dating app, probably not.  As a middleman, you are perfectly positioned to serve as the conduit;  one that provides an appropriate level of masking or pseudonymity.  And the money you invest in deploying a custom messaging system or temporary phone numbers (Twilio, anyone?) will pay off in terms of improved adoption and retention.

Design new products and services

If you understand how your parties interact and what they want to achieve, you’re in a position to spot new product opportunities that will make your customers happy.

From a conversation with Cyril Nigg, Director of Analytics at Reverb, the music-gear marketplace was “founded by music makers, for music makers.”  Musicians like to try new gear, but they want to offload it if it doesn’t pan out. Reverb has therefore built tools around pricing assistance to help musicians with their product listings: You want to sell this distortion pedal within 7 days? List it as $X. This extra assurance that they’ll be able to resell a piece of equipment, in short order, reduces apprehensions about buying. (Going back to the point about keeping both sides of the marketplace happy: Cyril also pointed out that a Reverb customer may act as both buyer and seller across different transactions.  That means the company can’t skimp on one side of the experience.)

People on a dating site want to communicate, so an easy win there is to keep an eye on new communications tools. Maybe your platform started out with an asynchronous, text-based tool that resembled e-mail.  Can you add an option for real-time chat?  What would it take to move up to voice? And ultimately, video? Each step in the progression requires advances in technology, so you may have to wait before you can actually deploy something. But if you can envision the system you want, you can keep an eye on the tech and be poised to pounce when it is generally available.

Unlike dating sites, financial exchanges are marketplaces for opposing views. One person thinks that some event will happen, they seek a counterpart who thinks that it will not, and fate determines the winner.  This can be as vanilla as people buying or selling shares of stock, where the counterparties believe the share price will rise or fall, respectively.  You also see situations that call for more exotic tools.  In the lead-up to what would become the 2008 financial crisis, investors wanted to stake claims around mortgage-backed securities but there wasn’t a way to express the belief that those prices would fall. In response to this desire, a group of banks dusted off the credit default swap (CDS) concept and devised a standard, easily-tradable contract.  Now there was a way for people to take either side of the trade, and for the banks to collect fees in the middle.  A win-win-win situation.

(Well, the actual trade was a win-win-win. The long-term outcome was more of a lose-lose-win. Mortgage defaults rose, sending prices for the associated mortgage-backed securities into decline, leading to big payouts for the “I told you this was going to happen” side of each CDS contract. The banks that served double-duty as both market participant and middleman took on sizable losses as a result. Let this be a lesson to you: part of why a middleman makes money is precisely because they have no stake in the long-term outcome of putting the parties together. Stay in the middle if you want to play it safe.)

Granted, you don’t have to roll out every possible product or feature on your first day. You have to let the marketplace grow and mature somewhat, to see what will actually be useful. Still, you want to plan ahead. As you watch the marketplace, you will spot opportunities well in advance, so you can position yourself to implement them before the need is urgent.

Focus on your business

Besides making things easier for customers, being a better middleman means improving how your business runs.

To start, identify and eliminate inefficiencies in your operations. I don’t mean that you should cut corners, as that will come back to bite you later.  I mean that you can check for genuine money leaks. The easy candidates will be right there on your balance sheet: have you actually used Service ABC in the last year?  If not, maybe it’s time to cut it. Is there an equivalent to Service XYZ at a lower price? Once you’ve confirmed that the cheaper service is indeed a suitable replacement, it’s time to make the switch.

A more subtle candidate is your codebase. Custom code is a weird form of debt. It requires steady, ongoing maintenance just like payments in a loan. It may also require disruptive changes if you encounter a bug. (Imagine that your mortgage lender occasionally demanded a surprise lump sum in mid-month.) Can you replace that home-grown system with an off-the-shelf tool or a third-party service, for a cheaper and more predictable payment schedule?

You also want to check on the size of your total addressable market (TAM).  What happens when you’ve reached everyone who will ever join? It’s emotionally reassuring to tell yourself that the entire planet will use your service, sure. But do you really want to base revenue projections on customers you can’t realistically acquire or retain? At some point, your customer numbers will plateau (and, after that, sink). You need to have a difficult conversation with yourself, your leadership team, and your investors around how you’ll handle that. And you need to have that conversation well in advance. Once you hit that limit on your TAM, you’ll need to be ready to deliver improvements that reduce churn. Perhaps you can offer new services, which may extend your addressable market into new territory, but even that has its limits.

What are you doing for risk management? A risk represents a possible future entry on your balance sheet, one of indeterminate size. Maybe it’s a code bug that spirals out of control under an edge case. Or a lingering complaint that blossoms into a full-scale PR issue. To be blunt: good risk management will save you money. Possibly lots of money. While it’s tempting to let some potential problems linger, understand that it’s easier and cheaper to address them early and on your own schedule. That’s much nicer than being under pressure to fix a surprise in real-time.

Sharp-eyed readers will catch that subtle tradeoff between “addressing inefficiencies” and “proactively mitigating risks.” Risk management often requires that you leave extra slack in the system, such as higher staff headcount, or extra machines that mostly sit idle. This slack serves as a cushion in the event of a surge in customer activity but it also costs money.  There’s no easy answer here. It’s a blend of art and science to spot the difference between slack and waste.

Most of all, as a marketplace, you want to mature with your customers and the field overall. The term “innovate” gets some much-deserved flack, but it’s not complete hogwash. Be prepared to invest in research so you can see what changes are on the horizon, and then adapt accordingly. Also, keep an eye on the new features your customers are asking for, or the complaints they raise about your service. You’ll  otherwise fall into the very trap described in The Innovator’s Dilemma. Don’t become the slow-moving, inattentive behemoth that some nimble upstart will work to unseat.

Use technology as a force multiplier

Bad middlemen squeeze the parties they connect; good middlemen squeeze technology.

Done well, technology is a source of asymmetric advantage. Putting code in the right places allows you to accomplish more work, more consistently, with fewer people, and in less time. All of the efficiencies you get through code will leave more money to split between yourself and your customers.  That is a solid retention strategy.

To start, you can apply software to real and artificial scarcity that exists in other middlemen. A greenfield operation can start with lower headcount, less (or zero!) office space, and so on.

Tech staffing, for example, is a matching problem at its core. A smart staffing firm would start with self-service search tools so a company could easily find people to match their open roles. No need to interact with a human recruiter. It could also standardize contract language to reduce legal overhead (no one wants a thousand slightly-different contracts laying around, anyway) and use electronic signatures to make it easier to store paperwork for future reference.

You don’t even have to do anything fancy. Sometimes, the very act of putting something online is a huge step up from the incumbent solution. Craigslist, simply by running classified ads on a website, gave people a much-improved experience over the print-newspaper version. People had more space to write (goodbye, obscure acronyms), had search functionality (why skim all the listings to find what you’re after?), and could pull their ad when it had been resolved (no more getting phone calls for an extra week just because the print ad is still visible).

Technology also makes it easier to manage resources. Love or loathe them, rideshare companies like Lyft and Uber can scale to a greater number of drivers and riders than the old-school taxi companies that rely on radio dispatch and flag-pulls. And they can do it with less friction. Why call a company and tell them your pickup location, when an app can use your phone’s GPS? And why should that dispatcher have to radio around in search of a driver? To arrange a ride, you need to match three elements–pickup location, dropoff location, and number of passengers–to an available driver. This is a trivial effort for a computer. Throw in mobile apps for drivers and passengers, and you have a system that can scale very well.

(Some may argue that the rideshare companies get extra scale because their drivers are classified as independent contractors, and because they don’t require expensive taxi medallions. I don’t disagree. I just want to point out that the companies’ technology is also a strong enabler.)

Being at the center of the marketplace means you get to see the entire system at once. You can analyze the data around customer activity, and pass on insights to market participants to make their lives easier. Airbnb, for example, has deep insight into how different properties perform. Their research team determined that listings with high-quality photos tend to earn more revenue. They publicized this information to help hosts and, to sweeten the deal, the company then built a service to connect hosts with professional photographers.

What about ML/AI? While I hardly believe that it’s ready to eat every job, I do see opportunities for AI to make a smaller team of people more effective. ML models are well-suited for decisions that are too fuzzy or cumbersome to be expressed as hard rules in software, but not so nuanced that they require human judgment. Putting AI in the seat for those decisions frees up your team for things that genuinely merit a human’s eyes and expertise.

I’ve argued before that a lot of machine learning is high-powered matching. What is “classification,” if not rating one item’s similarity to an archetype?  A marketplace that deals in the long tail of goods can use ML to help with that matching.

Take Reverb, where most pieces of gear are unique but still similar to other items. They’re neither completely fungible, nor completely non-fungible.  They’re sort of semi-fungible. To simplify search, then, Director of Analytics Cyril Nigg says that the company groups related items into ML-based canonical products (where some specific Product X is really part of a wider Canonical Product Y). “[We use] ML to match listings to a product–say, matching on title, price point, or some other attribute. This tells us, with a high degree of confidence, that a seller’s used Fender guitar is actually an American Standard Stratocaster. Now that we know the make and model, a buyer can easily compare all the different listings within that product to help them find the best option. This ML system learns over time, so that a seller can upload a listing and the system can file it under the proper canonical product.”

Machine-based matching works for food as well as guitars. Resham Sarkar heads up data science at Slice, which gives local pizzerias the tools, technology and guidance they need to thrive. In a 2021 interview, she told me how her team applies ML to answer the age-old question: will Person X enjoy Pizza Y at Restaurant Z? Slice’s recommendations give eaters the confidence to try a new flavor in a new location, which helps them (maybe they’ll develop a new favorite) and also helps pizzerias (they get new customers). This is especially useful when a pizza lover lands in a new city and doesn’t know where to get their fix.

Any discussion of technology wouldn’t be complete without a nod to emerging tech. Yes, keeping up with the Shiny New Thing of the Moment means having to wade through plenty of hype. But if you look closely, you may also find some real game-changers for your business. This was certainly true of the 1990s internet boom. We’ve seen it in the past decade of what we now call AI, across all of its rebrandings. And yes, I expect that blockchain technologies will prove more useful than the curmudgeons want to let on.  (Even NFTs. Or, especially NFTs.)

Skip past the success stories and vendor pitches, though. Do your own homework on what the new technology really is and what it can do. Then, engage an expert to help you fill in the gaps and sort out what is possible with your business. The way a new technology addresses your challenges may not align with whatever is being hyped in the news, but who cares? All that matters is that it drives improvements for your use cases.

Watch your tech

Technology is a double-edged sword. It’s like using leverage in the stock market: employing software or AI exposes you to higher highs when things go right, but also lower lows when things unravel.

One benefit to employing people to perform a task is that they can notice when something is wrong and then stop working. A piece of code, by comparison, has no idea that it is operating out of its depth. The same tools that let you do so much more, with far fewer people, also expose you to a sizable risk: one bug or environmental disconnect can trigger a series of errors, at machine speeds, cascading into a massive failure.

All it takes is for a few smaller problems to collide. Consider the case of Knight Capital. This experienced, heavyweight market-maker once managed $21BN in daily transaction volume on the NYSE. One day in 2012, an inconsistent software deployment met a branch of old code, which in turn collided with a new order type on the exchange. This led to a meltdown in which Knight Capital lost $440M in under an hour.

The lesson here is that some of the money you save from reduced headcount should be reinvested in the company in the form of people and tools to keep an eye on the larger system. You’ll want to separate responsibilities in order to provide checks and balances, such as assigning someone who is not a developer to manage and review code deployments. Install monitors that provide fine-grained information about the state of your systems. Borrowing a line from a colleague: you can almost never have too many dimensions of data when troubleshooting.

You’ll also need people to step in when someone gets caught in your web of automation. Have you ever called a company’s customer service line, only to wind up in a phone-tree dead-end? That can be very frustrating. You don’t want that for your customers, so you need to build escape hatches that route them to a person. That holds for your AI-driven chatbot as much as your self-help customer service workflows. And especially for any place where people can report a bug or an emergency situation.

Most of all, this level of automation requires a high-caliber team. Don’t skimp on hiring. Pay a premium for very experienced people to build and manage your technology. If you can, hire someone who has built trading systems on Wall St. That culture is wired to identify and handle risk in complex, automated systems where there is a lot of real money at stake.  And they have seen technology fail in ways that you cannot imagine.

Markets, everywhere

I’ve often said that problems in technology are rarely tech-related; they’re people-related. The same holds for building a marketplace, where the big problem is really human greed.

Don’t fall for the greed trap. You can certainly run the business in a way that brings you revenue, keeps customers happy, and attracts new prospects. Identify inefficiencies in your business operations, and keep thinking of ways to make the platform better for your customers. That’s it.  A proper application of software and AI, risk management, and research into emerging technologies should help you with both. And the money you save, you can split with your user base.

If you’re willing to blur the lines a little, you will probably find markets in not-so-obvious places. An airline sits between passengers and destinations. Grocery stores sit between shoppers and suppliers. Employers sit between employees and clients. And so on. Once you find the right angle, you can borrow ideas from the established, well-run middlemen to improve your business.

(Many thanks to Chris Butler for his thoughtful and insightful feedback on early drafts of this article.)

Two-sided markets are a fascinating study. They are also quite common in the business world, and therefore, so are middlemen. Record labels, rideshare companies, even dating apps all fall under this umbrella.  The role has plenty of perks, as well as some sizable pitfalls.  “Middleman” often carries a negative connotation because, in all fairness, some of them provide little value compared to what they ask in return.

Still, there’s room for everyone involved—Party A, Party B, and the middleman—to engage in a happy and healthy relationship.  In this first article, I’ll explain more about the middleman’s role and the challenges they face.  In the next article, I’ll explore what it takes to make a better middleman and how technology can play a role.

Paving the Path

When I say that middlemen make interactions easier, I mean that they address a variety of barriers:

• Discovery: “Where do I find the other side of my need or transaction?” Dating apps like OKCupid, classified ads services such as Craigslist, and directory sites like Angi (formerly Angie’s List) are all a twist on a search engine. Party A posts a description of themself or their service, Party B scrolls and sifts the list while evaluating potential matches for fit.
  
• Matching: “Should we interact? Are our needs compatible?” Many middlemen that help with discovery also handle the matching for you, as with ride-share apps.  Instead of you having to scroll through lists of drivers, Uber and Lyft use your phone’s GPS to pair you with someone nearby.  (Compared to the Discovery case, Matching works best when one or both counterparties are easily interchangeable.)
  
• Standardization: “The middleman sets the rules of engagement, so we all know what to expect.”  A common example would be when a middleman like eBay sets the accepted methods of payment.  By narrowing the scope of what’s possible—by limiting options—the middleman standardizes how the parties interact.
  
• Safety: “I don’t have to know you in order to exchange money with you.” Stock market exchanges and credit card companies build trust with Party A and Party B, individually, so the two parties (indirectly) trust each other through the transitive property.
  
• Simplicity: “You two already know each other; I’ll insert myself into the middle, to make the relationship smoother.” Stripe and Squarespace make it easier for companies to sell goods and services by handling payments.  And then there’s Squire, which co-founder Songe Laron describes as the “operating system for the barber shop, [handling] everything from the booking, to the payment, to the point of sales system, to payroll,” and a host of other frictions between barber and customer.  In all cases, each party gets to focus on what it does best (selling goods or cutting hair) while the middleman handles the drudgework.

Nice Work, If You can Get It

As far as their business model, middlemen usually take a cut of transactions as value moves from Party A to Party B. And this arrangement has its benefits.

For one, you’re first in line to get paid: Party A pays you, you take a cut, then you pass the rest on to Party B.  Record labels and book publishers are a common example.  They pair a creator with an audience.  All of the business deals for that creator’s work run through the middleman, who collects the revenue from sales and takes their share along the way.

(The music biz is littered with stories of artists getting a raw deal—making a small percentage of revenue from their albums, while the label takes the lion’s share—but that’s another story.)

Then there’s the opportunity for recurring revenue, if Party A and Party B have an ongoing relationship.  Companies often turn to tech staffing agencies to find staff-augmentation contractors.  Those agencies typically take a cut for the entire duration of the project or engagement, which can run anywhere from a few weeks to more than a decade.  The staffing agency makes one hell of a return on their efforts when placing such a long-term contractor. Nice work, if you can get it.

Staffing agencies may have to refund a customer’s money if a contractor performs poorly.  Some middlemen, however, make money no matter how the deal ultimately turns out.  Did I foolishly believe my friend’s hot stock tip, in his drunken reverie, and pour my savings into a bad investment? Well, NYSE isn’t going to refund my money, which means they aren’t about to lose their cut.

A middleman also gets a bird’s-eye view of the relationships it enables.  It sees who interacts with whom, and how that all happens.  Middlemen that run online platforms have the opportunity to double-dip on their revenue model: first by taking their cut from an interaction, then by collecting and analyzing data around each interaction.  Everything from an end-user’s contact or demographic details, to exploring patterns of how they communicate with other users, can be packaged up and resold.  (This is, admittedly, a little shady. We’ll get to middlemen’s abuse of privilege shortly.)

Saddling Some Burdens, Too

Before you rush out to build your own middleman company, recognize that it isn’t all easy revenue.  You first need to breathe the platform into existence, so the parties can interact.  Depending on the field, this can involve a significant outlay of capital, time, and effort.  Then you need to market the platform so that everyone knows where to go to find the Party B to their Party A.

Once it’s up and running, maintenance costs can be low if you keep things simple.  (Consider the rideshare companies that own the technology platform, but not the vehicles in which passengers ride.) But until you reach that cruising altitude, you’re crossing your fingers that things pan out in your favor.  That can mean a lot of sleepless nights and stressful investor calls.

The middleman’s other big challenge is that they need to keep all of those N sides of the N-sided market happy.  The market only exists because all of the parties want to come together, and your service persists only because they want to come together through you.  If one side gets mad and leaves, the other side(s) will soon follow.  Keeping the peace can be a touchy balancing act.

Consider Airbnb.  Early in the pandemic they earned praise from guests by allowing them to cancel certain bookings without penalty.  It then passed those “savings” on to hosts, who weren’t too happy about the lost revenue.  (Airbnb later created a fund to support hosts, but some say it still fell short.)  The action sent a clear—though, likely, unintentional and incorrect—message that Airbnb valued guests more than hosts.  A modern-day version of robbing Peter to pay Paul.

Keeping all sides happy is a tough line for a middleman to walk.  Mohambir Sawhney, from Northwestern University’s McCormick Foundation, summed this up well: “In any two-sided market, you always have to figure out who you’re going to subsidize more, and who you’re going to actually screw more.” It’s easy for outsiders to say that Airbnb should have just eaten the losses—refunded guests’ money while letting hosts keep their take—but that sounds much easier said than done.  In the end, the company still has to subsidize itself, right?

The subsidize versus screw decision calculus gets even more complicated when one side only wants you but doesn’t need you.  In the Airbnb case, the company effectively serves as a marketing arm and payments processor for property owners.  Any sufficiently motivated owner is just one step away from handling that on their own, so even a small negative nudge can send them packing.  (In economics terms, we say that those owners’ switching costs are low.)

The same holds for the tech sector, where independent contractors can bypass staffing firms to hang their own shingle.  Even rideshare drivers have a choice.  While it would be tougher for them to get their own taxi medallion, they can switch from Uber to Lyft.  Or, as many do, they can sign up with both services so that switching costs are effectively zero: “delete Uber app, keep the Lyft app running, done.”

Making Enemies

Even with those challenges, delivering on the middleman’s raison d’être—”keep all parties happy”—should be a straightforward affair.  (I don’t say “easy,” just “straightforward.” There’s a difference.) Parties A and B clearly want to be together, you’re helping them be together, so the experience should be a win all around.

Why, then, do middlemen have such a terrible reputation?  It mostly boils down to greed.

Once a middleman becomes a sufficiently large and/or established player, they become the de facto place for the parties to meet.  This is a near-monopoly status. The middleman no longer needs to care about keeping one or even both parties happy, they figure, because those groups either interact through the middleman or they don’t interact at all. (This also holds true for the near-cartel status of a group of equally unpleasant middlemen.)

Maybe the middleman suddenly raises fees, or sets onerous terms of service, or simply mistreats one side of the pairing.  This raises the dollar, effort, and emotional cost to the parties since they don’t have many options to leave.

Consider food-delivery apps, which consumers love but can take as much as a 30% cut of an order’s revenue.  That’s a large bite, but easier to swallow when a restaurant has a modest take-away business alongside a much larger dine-in experience. It’s quite another story when take-away is suddenly your entire business and you’re still paying rent on the empty dining room space. Most restaurants found themselves in just this position early in the COVID-19 pandemic. Some hung signs in their windows, asking customers to call them directly instead of using the delivery apps.

Involving a middleman in a relationship can also lead to weird principal-agent problems.  Tech staffing agencies (even those that paint themselves as “consultancies”) have earned a special place here.  Big companies hand such “preferred vendors” a strong moat by requiring contractors to pass through them in lieu of establishing a direct relationship. Since the middlemen can play this Work Through Us, or Don’t Work at All card, it’s no surprise that they’ve been known to take as much as 50% of the money as it passes from client to contractor.  The client companies don’t always know this, so they are happy that the staffing agency has helped them find software developers and DBAs. The contractors, many of whom are aware of the large cuts, aren’t so keen on the arrangement.

This is on top of limiting a tech contractor’s ability to work through a competing agency.  I’ve seen everything from thinly-veiled threats (“if the client sees your resume from more than one agency, they’ll just throw it out”) to written agreements (“this contract says you won’t go through another agency to work with this client”).   What if you’ve found a different agency that will take a smaller cut, so you get more money?  Or what if Agency 1 has done a poor job of representing you, while you know that Agency 2 will get it right?  In both cases, the answer is: tough luck.

A middleman can also resort to more subtle ways to mistreat the parties.  Uber has reportedly used a variety of techniques from behavioral science—such as the gamification of male managers pretending to be women—to encourage drivers to work more.  They’ve also been accused of showing drivers and passengers different routes, charging the passenger for the longer way and paying the driver for the shorter way.

It’s Not All Easy Money

To be fair, middlemen do earn some of their cut. They provide value in that they reduce friction for both the buy and sell sides of an interaction.

This goes above and beyond building the technology for a platform.  Part of how the Deliveroos and Doordashes of the world connect diners to restaurants is by coordinating fleets of delivery drivers.  It would be expensive for a restaurant to do this on their own: hiring multiple drivers, managing the schedule, accounting for demand … and hoping business stays hot so that the drivers aren’t paid to sit idle. Similarly, tech staffing firms don’t just introduce you to contract talent. They also handle time-tracking, invoicing, and legal agreements. The client company cuts one large check to the staffing firm, which cuts lots of smaller checks to the individual contractors.

Don’t forget that handling contracts and processing payments come with extra regulatory requirements. Rules often vary by locale, and the middleman has to spend money to keep track of those rules.  So it’s not all profit.

(They can also build tools to avoid rules, such as Uber’s infamous “greyball” system … but that’s another story.)

That said, a middleman’s benefit varies by the industry vertical and even by the client.  Some argue that their revenue cut far exceeds the value they provide. In the case of tech staffing firms, I’ve heard plenty of complaints that recruiters take far too much money for  just “having a phone number” (having a client relationship) and cutting a check, when it’s the contractor who does the actual work of building software or managing systems for the client.

A Win-Win-Win Triangle

Running a middleman has its challenges and risks.  It can also be tempting to misuse the role’s power.  Still, I say that there’s a way to build an N-sided marketplace where everyone can be happy.  I’ll explore that in the next article in this series.

(Many thanks to Chris Butler for his thoughtful and insightful feedback on early drafts of this article.  I’d also like to thank Mike Loukides for shepherding this piece into its final form.)

1. The vast majority of business ideas fail to generate a positive impact.
2. Most companies are unaware of this.
3. It is unlikely that companies will increase the success rate for their business ideas.

These are lessons that could profoundly change how businesses operate. In what follows, we flesh out the three assertions above with the bulk of the content explaining why it may be difficult to improve the poor success rate for business ideas. Despite the challenges, we conclude with some recommendations for better managing your business.

(1) The vast majority of business ideas fail to generate positive results

To properly measure the outcomes of business ideas, companies are embracing experimentation (a.k.a. randomized controlled trials or A/B testing). The process is simple in concept. Before rolling out a business idea, you test; you try the idea out on a subset group of customers¹ while another group—a control group—is not exposed to the new idea. When properly sampled, the two groups will exhibit the same attributes (demographics, geographics, etc.) and behaviors (purchase rates, life-time-value, etc.). Therefore, when the intervention is introduced—ie. the exposure to the new business idea—any changes in behavior can be causally attributed to the new business idea. This is the gold standard in scientific measurement used in clinical trials for medical research, biological studies, pharmaceutical trials, and now to test business ideas.

For the very first time in many business domains, experimentation reveals the causal impact of our business ideas. The results are humbling. They indicate that the vast majority of our business ideas fail to generate positive results. It’s not uncommon for 70-90% of ideas to either have no impact at all or actually move the metrics in the opposite direction of what was intended. Here are some statistics from a few notable companies that have disclosed their success rates publicly:

• Microsoft declared that roughly one-third of their ideas yield negative results, one-third yield no results, and one-third yield positive results (Kohavi and Thomke, 2017).
• Streaming service Netflix believes that 90% of its ideas are wrong (Moran, 2007).
• Google reported that as much as 96.1% of their ideas fail to generate positive results (Thomke, 2020).
• Travel site Booking.com shared that 9 out of 10 of their ideas fail to improve metrics (Thomke, 2020).

To be sure, the statistics cited above reflect a tiny subset of the ideas implemented by companies. Further, they probably reflect a particular class of ideas: those that are conducive to experimentation² such as changes to user interfaces, new ad creatives, subtle messaging variants, and so on. Moreover, the companies represented are all relatively young and either in the tech sector or leverage technology as a medium for their business. This is far from a random sample of all companies and business ideas. So, while it’s possible that the high failure rates are specific to the types of companies and ideas that are convenient to test experimentally, it seems more plausible that the high failure rates are reflective of business ideas in general and that the disparity in perception of their success can be attributed to the method of measurement. We shouldn’t be surprised; high failure rates are common in many domains. Venture capitalists invest in many companies because most fail; similarly, most stock portfolio managers fail to outperform the S&P 500; in biology, most mutations are unsuccessful; and so on. The more surprising aspect of the low success rates for business ideas is most of us don’t seem to know about it.

(2) Most companies are unaware of the low success rates for their business ideas

Those statistics should be sobering to any organization. Collectively, business ideas represent the roadmap companies rely upon to hit their goals and objectives. However, the dismal failure rates appear to be known only to the few companies that regularly conduct experiments to scientifically measure the impact of their ideas. Most companies do not appear to employ such a practice and seem to have the impression that all or most of their ideas are or will be successful. Planners, strategists, and functional leaders rarely convey any doubts about their ideas. To the contrary, they set expectations on the predicted impact of their ideas and plan for them as if they are certain. They attach revenue goals and even their own bonuses to those predictions. But, how much do they really know about the outcomes of those ideas? If they don’t have an experimentation practice, they likely know very little about the impact their roadmap is actually having.

Without experimentation, companies either don’t measure the outcomes of their ideas at all or use flimsy methods to assess their impacts. In some situations, ideas are acted upon so fluidly that they are not recognized as something that merits measurement.  For example, in some companies an idea such as “we’ll lower prices to increase demand by 10%” might be made on a whim by a marketing exec and there will be no follow up at all to see if it had the expected impact on demand. In other situations, a post-implementation assessment of a business idea is done, but in terms of execution, not impact (“Was it implemented on time?” “Did it meet requirements?” etc., not “What was the causal impact on business metrics?”). In other cases still, post hoc analysis is performed in an attempt to quantify the impact of the idea. But, this is often done using subjective or less-than-rigorous methods to justify the idea as a success. That is, the team responsible for doing the analysis often is motivated either implicitly or explicitly to find evidence of success. Bonuses are often tied to the outcomes of business ideas. Or, perhaps the VP whose idea it was is the one commissioning the analysis. In either case, there is a strong motivation to find success. For example, a company may seek qualitative customer feedback on the new loyalty program in order to craft a narrative for how it is received. Yet, the customers willing to give feedback are often biased towards the positive. Even if more objective feedback were to be acquired it would still not be a measure of impact; customers often behave differently from the sentiments they express. In still other cases, empirical analysis is performed on transaction data in an attempt to quantify the impact. But, without experimentation, at best, such analysis can only capture correlation—not causation. Business metrics are influenced simultaneously by many factors, including random fluctuations. Without properly controlling for these factors, it can be tempting to attribute any uptick in metrics as a result of the new business idea. The combination of malleable measurements and strong incentives to show success likely explain why so many business initiatives are perceived to be successful.

By contrast, the results of experimentation are numeric and austere. They do not care about the hard work that went into executing on a business initiative. They are unswayed by well-crafted narratives, emotional reviews by customers, or an executive’s influence. In short, they are brutally honest and often hard-to-accept.³ Without experimentation, companies don’t learn the sobering truth about their high failure rate. While ignorance is bliss, it is not an effective way to run your business.

(3) It is unlikely that companies will increase the success rate for their business ideas.

At this point, you may be thinking, “we need to get better at separating the wheat from the chaff, so that we only allocate resources to the good ideas.” Sadly, without experimentation, we see little reason for optimism as there are forces that will actively work against your efforts.

One force that is actively working against us is the way we reason about our companies.

We like to reason about our businesses as if they are simple, predictable systems. We build models of their component parts and manage them as if they are levers we can pull in order to predictably manage the business to a desired state. For example, a marketer seeking to increase demand builds a model that allows her to associate each possible price with a predicted level of demand. The scope of the model is intentionally narrow so that she can isolate the impact price has on demand. Other factors like consumer perception, the competitive assortment, operational capacity, the macroeconomic landscape, and so on are out of her control and assumed to remain constant. Equipped with such an intuitive model, she can identify the price that optimizes demand. She’s in control and hitting her goal is merely a matter of execution.

However, experimentation reveals that our predictions for the impact of new business ideas can be radically off—not just a little off in terms of magnitude, but often in the completely wrong direction. We lower prices and see demand go down. We launch a new loyalty program and it hurts retention. Such unintuitive results are far more common than you might think.

The problem is that many businesses behave as complex systems which cannot be understood by studying its components in isolation. Customers, competitors, partners, market force—each can adjust in response to the intervention in ways that are not observable from simple models of the components. Just as you can’t learn about an ant colony by studying the behaviors of an individual ant (Mauboussin, 2009), the insights derived from modeling individual components of a business in isolation often have little relevance to the way the business behaves as a whole.

It’s important to note that our use of the term complex does not just mean ‘not simple.’ Complexity is an entire area of research within Systems Theory. Complexity arises in systems with many interacting agents that react and adapt to one another and their environment. Examples of complex systems include weather systems, rain forest ecology, economies, the nervous system, cities, and yes, many businesses.

Reasoning about complex systems requires a different approach. Rather than focusing on component parts, attention needs to be directed at system-wide behaviors. These behaviors are often termed “emergent,” to indicate that they are very hard to anticipate. This frame orients us around learning, not executing. It encourages more trial and error with less attachment to the outcomes of a narrow set of ideas. As complexity researcher Scott E. Page says, “An actor in a complex system controls almost nothing but influences almost everything” (Page, 2009).

Another force that is actively working against our efforts to discern good ideas from bad is our cognitive biases. You might be thinking: “Thank goodness my company has processes that filter away bad ideas, so that we only invest in great ideas!” Unfortunately, all companies probably try hard to select only the best ideas, and yet we assert that they are not particularly successful at separating good from bad ideas. We suggest that this is because these processes are deeply human in nature, leaving them vulnerable to cognitive biases.

Cognitive biases are systematic errors in human thinking and decision making (Tversky & Kahneman, 1974). They result from the core thinking and decision making processes that we developed over our evolutionary history. Unfortunately, evolution adapted us to an environment with many differences from the modern world. This can lead to a habit of poor decision making. To illustrate: we know that a healthy bundle of kale is better for our bodies than a big juicy burger. Yet, we have an innate preference for the burger. Many of us will decide to eat the burger tonight. And tomorrow night. And again next week. We know we shouldn’t. But yet our society continues consuming too much meat, fat, and sugar. Obesity is now a major public health problem. Why are we doing this to ourselves? Why are we imbued with such a strong urge—a literal gut instinct—to repeatedly make decisions that have negative consequences for us? It’s because meat, fat, and sugar were scarce and precious resources for most of our evolutionary history. Consuming these resources at every opportunity was an adaptive behavior, and so humans evolved a strong desire to do so. Unfortunately, we remain imbued with this desire despite the modern world’s abundance of burger joints.

Cognitive biases are predictable and pervasive. We fall prey to them despite believing that we are rational and objective thinkers. Business leaders (ourselves included) are not immune. These biases compromise our ability to filter out bad business ideas. They can also make us feel extremely confident as we make a bad business decision. See the following sidebar for examples of cognitive biases manifesting in business environments and producing bad decisions.

A final force that is actively working against efforts to discern good ideas from bad is your business maturing. A thought experiment: Suppose a local high school coach told NBA superstar Stephen Curry how to adjust his jump shot. Would implementing these changes improve or hurt his performance? It is hard to imagine it would help. Now, suppose the coach gave this advice to a local 6th grader. It seems likely that it would help the kid’s game.

Now, imagine a consultant telling Google how to improve their search algorithm versus advising a startup on setting up a database. It’s easier to imagine the consultant helping the startup. Why? Well, Google search is a cutting edge system that has received extensive attention from numerous world class experts—kind of like Steph Curry. It’s going to be hard to offer a new great idea. In contrast, the startup will benefit from getting pointed in a variety of good directions—kind of like a 6th grader.

To use a more analytic framework, imagine a hill which represents a company’s objective function⁵ like profit, revenue, or retention. The company’s goal is to climb to the peak, where it’s objective is maximized. However, the company can’t see very far in this landscape. It doesn’t know where the peak is. It can only assess (if it’s careful and uses experimentation) whether it’s going up or downhill by taking small steps in different directions—perhaps by tweaking it’s pricing strategy and measuring if revenue goes up.

When a company (or basketball player) is young, its position on this objective function (profit, etc.) landscape is low. It can step in many directions and go uphill. Through this process, a company can grow (walk up Mount Revenue). However, as it climbs the mountain, a smaller proportion of the possible directions to step will lead uphill. At the summit a step in any direction will take you downhill.

This is admittedly a simple model  of a business (and we already discussed the follies of using simple models). However, all companies will eventually face the truism that as they improve, there are fewer ways to continue to improve (the low apples have been plucked), as well as the extrinsic constraints of market saturation, commoditization, etc. that make it harder to improve your business as it matures.⁶

So, what to do

We’ve argued that most business ideas fail to deliver on their promised goals. We’ve also explained that there are systematic reasons that make it unlikely that companies will get better, just by trying harder. So where does this leave you? Are you destined to implement mostly bad ideas? Here are a few recommendations that might help:

1. Run experiments and exercise your optionality. Recognize that your business may be a complex system, making it very difficult to predict how it will respond to your business ideas. Instead of rolling out your new business ideas to all customers, try them on a sample of customers as an experiment. This will show you the impact your idea has on the company. You can then make an informed decision about whether or not to roll out your idea. If your idea has a positive impact, great. Roll it out to all customers. But in the more likely event that your idea does not have the positive impact you were hoping for you can end the experiment and kill the idea. It may seem wasteful to use company resources to implement a business idea only to later kill it, but this is better than unknowingly providing on-going support to an idea that is doing nothing or actually hurting your metrics—which is what happens most of the time.
2. Recognize your cognitive biases, collect a priori predictions, and celebrate learnings. Your company’s ability to filter out bad business ideas will be limited by your team member’s cognitive biases. You can start building a culture that appreciates this fact by sending a survey to all of a project’s stakeholders before your next big release. Ask everyone to predict how the metrics will move. Make an anonymized version of these predictions and accuracy available for employees. We expect your team members will become less confident in their predictions over time. This process may also reveal that big wins tend to emerge from a string of experiments, rather than a single stroke of inspiration. So celebrate all of the necessary stepping stones on the way to a big win.
3. Recognize that it’s going to get harder to find successful ideas, so try more things, and get more skeptical. As your company matures, it may get harder to find ways to improve it. We see three ways to address this challenge. First, try more ideas. It will be hard to increase the success rate of your ideas, so try more ideas. Consider building a leverageable and reusable experimentation platform to increase your bandwidth. Follow the lead of the venture world: fund a lot of ideas to get a few big wins.⁷ Second, as your company matures, you might want to adjust the amount of evidence that is required before you roll out a change—a more mature company should require a higher degree of statistical certainty before inferring that a new feature has improved metrics. In experimental lingo, you might want to adjust the “p-value thresholds” that you use to assess an experiment. Or to use our metaphor, a 6th grader should probably just listen whenever a coach tells them to adjust their jump shot, but Steph Curry should require a lot of evidence before he adjusts his.

This may be a hard message to accept. It’s easier to assume that all of our ideas are having the positive impact that we intended. It’s more inspiring to believe that successful ideas and companies are the result of brilliance rather than trial and error. But, consider the deference we give to mother nature. She is able to produce such exquisite creatures—the giraffe, the mighty oak tree, even us humans—each so perfectly adapted to their environment that we see them as the rightful owners of their respective niches. Yet, mother nature achieves this not through grandiose ideas, but through trial and error… with a success rate far more dismal than that of our business ideas. It’s an effective strategy if we can convince our egos to embrace it.

References

Arkes, H. R., & Blumer, C. (1985), The psychology of sunk costs. Organizational Behavior and Human Decision Processes, 35, 124-140.

Gneezy, U., & Rustichini, A. (2000). A Fine is a Price. The Journal of Legal Studies, 29(1), 1-17. doi:10.1086/468061

Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526. https://doi.org/10.1037/a0016755

Kohavi, R. & Thomke, S. “The Surprising Power of Online Experiments,” Harvard Business Review 95, no. 5 (September-October 2017)

Mauboussin, M. J. (2009). Think Twice: Harnessing the Power of Counterintuition. Harvard Business Review Press.

Milgram, S. (1963). “Behavioral Study of obedience”. The Journal of Abnormal and Social Psychology. 67 (4): 371–378.

Moran, M. Do It Wrong Quickly: How the Web Changes the Old Marketing Rules . s.l. : IBM Press, 2007. 0132255960.

Nickerson, R. S. (1998), “Confirmation bias: A ubiquitous phenomenon in many guises”, Review of General Psychology, 2 (2): 175–220.

Page, S. E. (2009). Understanding Complexity – The Great Courses – Lecture Transcript and Course Guidebook (1st ed.). The Teaching Company.

Thomke, S. H. (2020). Experimentation Works: The Surprising Power of Business Experiments. Harvard Business Review Press.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124-1131.

Whyte, W. H., (1952). “Groupthink”. Fortune, 114-117, 142, 146.

Footnotes

1. Do not confuse the term ‘test’ to mean a process by which a nascent idea is vetted to get feedback. In an experiment, the test group receives a full-featured implementation of an idea. The goal of the experiment is to measure impact—not get feedback.
2. In some cases there may be insufficient sample size, ethical concerns, lack of a suitable control group, and many other conditions that can inhibit experimentation.
3. Even trained statisticians can fall victim to pressures to cajole the data. “P-hacking”, “significance chasing” and other terms refer to the temptation to use flawed methods in statistical analysis.
4. We believe that these types of factors are only obvious in hindsight because the signals are often unobserved until we know to look for them (Kahneman & Klein, 2009).
5. One reason among many why this mental picture is oversimplified is that it implicitly takes business conditions and the world at large to be static—the company “state vector” that maximizes the objective function today is the same as what maximizes the objective function tomorrow. In other words, it ignores that, in reality, the hill is changing shape under our feet as we try to climb it. Still, it’s a useful toy model.
6. Finding a new market (jumping to a new “hill” in the “Mount Revenue” metaphor), as recommended in the next section, is one way to continue improving business metrics even as your company matures.
7. VCs are able to learn about the outcomes of the startups even without experimentation. This is because the outcomes are far more readily apparent than that of business ideas. It’s difficult to cajole results to show a successful outcome when the company is out of business.

But what are “trends”? All too often, trends degenerate into horse races over languages and platforms. Look at all the angst heating up social media when TIOBE or RedMonk releases their reports on language rankings. Those reports are valuable, but their value isn’t in knowing what languages are popular in any given month. And that’s what I’d like to get to here: the real trends that aren’t reflected (or at best, are indirectly reflected) by the horse races. Sometimes they’re only apparent if you look carefully at the data; sometimes it’s just a matter of keeping your ear to the ground.

In either case, there’s a difference between “trends” and “trendy.” Trendy, fashionable things are often a flash in the pan, forgotten or regretted a year or two later (like Pet Rocks or Chia Pets). Real trends unfold on much longer time scales and may take several steps backward during the process: civil rights, for example. Something is happening and, over the long arc of history, it’s not going to stop. In our industry, cloud computing might be a good example.

Methodology

This study is based on title usage on O’Reilly online learning. The data includes all usage of our platform, not just content that O’Reilly has published, and certainly not just books. We’ve explored usage across all publishing partners and learning modes, from live training courses and online events to interactive functionality provided by Katacoda and Jupyter notebooks. We’ve included search data in the graphs, although we have avoided using search data in our analysis. Search data is distorted by how quickly customers find what they want: if they don’t succeed, they may try a similar search with many of the same terms. (But don’t even think of searching for R or C!) Usage data shows what content our members actually use, though we admit it has its own problems: usage is biased by the content that’s available, and there’s no data for topics that are so new that content hasn’t been developed.

We haven’t combined data from multiple terms. Because we’re doing simple pattern matching against titles, usage for “AWS security” is a subset of the usage for “security.” We made a (very) few exceptions, usually when there are two different ways to search for the same concept. For example, we combined “SRE” with “site reliability engineering,” and “object oriented” with “object-oriented.”

The results are, of course, biased by the makeup of the user population of O’Reilly online learning itself. Our members are a mix of individuals (professionals, students, hobbyists) and corporate users (employees of a company with a corporate account). We suspect that the latter group is somewhat more conservative than the former. In practice, this means that we may have less meaningful data on the latest JavaScript frameworks or the newest programming languages. New frameworks appear every day (literally), and our corporate clients won’t suddenly tell their staff to reimplement the ecommerce site just because last year’s hot framework is no longer fashionable.

Usage and query data for each group are normalized to the highest value in each group. Practically, this means that you can compare topics within a group, but you can’t compare the groups with each other. Year-over-year (YOY) growth compares January through September 2020 with the same months of 2019. Small fluctuations (under 5% or so) are likely to be noise rather than a sign of a real trend.

Enough preliminaries. Let’s look at the data, starting at the highest level: O’Reilly online learning itself.

O’Reilly Online Learning

Usage of O’Reilly online learning grew steadily in 2020, with 24% growth since 2019. That may not be surprising, given the COVID-19 pandemic and the resulting changes in the technology industry. Companies that once resisted working from home were suddenly shutting down their offices and asking their staff to work remotely. Many have said that remote work will remain an option indefinitely. COVID had a significant effect on training: in-person training (whether on- or off-site) was no longer an option, so organizations of all sizes increased their participation in live online training, which grew by 96%. More traditional modes also saw increases: usage of books increased by 11%, while videos were up 24%. We also added two new learning modes, Katacoda scenarios and Jupyter notebooks, during the year; we don’t yet have enough data to see how they’re trending.

It’s important to place our growth data in this context. We frequently say that 10% growth in a topic is “healthy,” and we’ll stand by that, but remember that O’Reilly online learning itself showed 24% growth. So while a technology whose usage is growing 10% annually is healthy, it’s not keeping up with the platform.

As travel ground to a halt, so did traditional in-person conferences. We closed our conference business in March, replacing it with live virtual Superstreams. While we can’t compare in-person conference data with virtual event data, we can make a few observations. The most successful superstream series focused on software architecture and infrastructure and operations. Why? The in-person O’Reilly Software Architecture Conference was small but growing. But when the pandemic hit, companies found out that they really were online businesses—and if they weren’t, they had to become online to survive. Even small restaurants and farm markets were adding online ordering features to their websites. Suddenly, the ability to design, build, and operate applications at scale wasn’t optional; it was necessary for survival.

Programming Languages

Although we’re not fans of the language horse race, programming languages are as good a place as any to start. Figure 1 shows usage, year-over-year growth in usage, and the number of search queries for several popular languages. The top languages for O’Reilly online learning are Python (up 27%), Java (down 3%), C++ (up 10%), C (up 12%), and JavaScript (up 40%). Looking at 2020 usage rather than year-over-year changes, it’s surprising to see JavaScript so far behind Python and Java. (JavaScript usage is 20% of Python’s, and 33% of Java’s.)

Past the top five languages, we see healthy growth in Go (16%) and Rust (94%). Although we believe that Rust’s popularity will continue to grow, don’t get too excited; it’s easy to grow 94% when you’re starting from a small base. Go has clearly established itself, particularly as a language for concurrent programming, and Rust is likely to establish itself for “system programming”: building new operating systems and tooling for cloud operations. Julia, a language designed for mathematical computation, is an interesting wild card. It’s slightly down over the past year, but we’re optimistic about its long term chances.

We shouldn’t separate usage of titles specifically aimed at learning a programming language from titles applying the language or using frameworks based on it. After all, many Java developers use Spring, and searching for “Java” misses content only has the word “Spring” in the title. The same is true for JavaScript, with the React, Angular, and Node.js frameworks. With Python, the most heavily used libraries are PyTorch and scikit-learn. Figure 2 shows what happens when you add the use of content about Python, Java, and JavaScript to the most important frameworks for those languages.

It probably isn’t a surprise that the results are similar, but there are some key differences. Adding usage and search query data for Spring (up 7%) reverses Java’s apparent decline (net-zero growth). Zero growth isn’t inappropriate for an established enterprise language, particularly one owned by a company that has mired the language in controversy. Looking further at JavaScript, if you add in usage for the most popular frameworks (React, Angular, and Node.js), JavaScript usage on O’Reilly online learning rises to 50% of Python’s, only slightly behind Java and its frameworks. However, Python, when added to the heavily used frameworks PyTorch and scikit-learn, remains the clear leader.

It’s important to understand what we’ve done though. We’re trying to build a more comprehensive picture of language use that includes the use of various frameworks. We’re not pretending the frameworks themselves are comparable—Spring is primarily for backend and middleware development (though it includes a web framework); React and Angular are for frontend development; and scikit-learn and PyTorch are machine learning libraries. And although it’s widely used, we didn’t assign TensorFlow to any language; it has bindings for Python, Java, C++, and JavaScript, and it’s not clear which language predominates. (Google Trends suggests C++.) We also ignored thousands (literally) of minor platforms, frameworks, and libraries for all these languages; once you get past the top few, you’re into the noise.

We aren’t advocating for Python, Java, or any other language. None of these top languages are going away, though their stock may rise or fall as fashions change and the software industry evolves. We’re just saying that when you make comparisons, you have to be careful about exactly what you’re comparing. The horse race? That’s just what it is. Fun to watch, and have a mint julep when it’s over, but don’t bet your savings (or your job) on it.

If the horse race isn’t significant, just what are the important trends for programming languages? We see several factors changing pro‐ gramming in significant ways:

• Multiparadigm languages
  Since last year, O’Reilly online learning has seen a 14% increase in the use of content on functional programming. However, Haskell and Erlang, the classic functional languages, aren’t where the action is; neither shows significant usage, and both are headed down (roughly 20% decline year over year). Object oriented programming is up even more than functional programming: 29% growth since last year. This suggests that the real story is the integration of functional features into procedural and object-oriented languages. Starting with Python 3.0 in 2008 and continuing with Java 8 in 2014, programming languages have added higher-order functions (lambdas) and other “functional” features. Several popular languages (including JavaScript and Go) have had functional features from the beginning. This trend started over 20 years ago (with the Standard Template Library for C++), and we expect it to continue.
  
• Concurrent programming
  Platform data for concurrency shows an 8% year-over-year increase. This isn’t a large number, but don’t miss the story because the numbers are small. Java was the first widely used language to support concurrency as part of the language. In the mid-’90s, thread support was a luxury; Moore’s law had plenty of room to grow. That’s no longer the case, and support for concurrency, like support for functional programming, has become table stakes. Go, Rust, and most other modern languages have built-in support for concurrency. Concurrency has always been one of Python’s weaknesses.
  
• Dynamic versus static typing
  This is another important paradigmatic axis. The distinction between languages with dynamic typing (like Ruby and JavaScript) and statically typed languages (like Java and Go) is arguably more important than the distinction between functional and object-oriented languages. Not long ago, the idea of adding static typing to dynamic languages would have started a brawl. No longer. Combining paradigms to form a hybrid is taking a hold here too. Python 3.5 added type hinting, and more recent versions have added additional static typing features. TypeScript, which adds static typing to JavaScript, is coming into its own (12% year-over-year increase).
  
• Low-code and no-code computing
  It’s hard for a learning platform to gather data about a trend that minimizes the need to learn, but low-code is real and is bound to have an effect. Spreadsheets were the forerunner of low-code computing. When VisiCalc was first released in 1979, it enabled millions to do significant and important computation without learning a programming language. Democratization is an important trend in many areas of technology; it would be surprising if programming were any different.

What’s important isn’t the horse race so much as the features that languages are acquiring, and why. Given that we’ve run to the end of Moore’s law, concurrency will be central to the future of programming. We can’t just get faster processors. We’ll be working with microservices and serverless/functions-as-a-service in the cloud for a long time–and these are inherently concurrent systems. Functional programming doesn’t solve the problem of concurrency—but the discipline of immutability certainly helps avoid pitfalls. (And who doesn’t love first-class functions?) As software projects inevitably become larger and more complex, it makes eminent sense for languages to extend themselves by mixing in functional features. We need programmers who are thinking about how to use functional and object-oriented features together; what practices and patterns make sense when building enterprise-scale concurrent software?

Low-code and no-code programming will inevitably change the nature of programming and programming languages:

• There will be new languages, new libraries, and new tools to support no- or low-code programmers. They’ll be very simple. (Horrors, will they look like BASIC? Please no.) Whatever form they take, it will take programmers to build and maintain them.
  
• We’ll certainly see sophisticated computer-aided coding as an aid to experienced programmers. Whether that means “pair programming with a machine” or algorithms that can write simple programs on their own remains to be seen. These tools won’t eliminate programmers; they’ll make programmers more productive.

There will be a predictable backlash against letting the great unwashed into the programmers’ domain. Ignore it. Low-code is part of a democratization movement that puts the power of computing into more peoples’ hands, and that’s almost always a good thing. Programmers who realize what this movement means won’t be put out of jobs by nonprogrammers. They’ll be the ones becoming more productive and writing the tools that others will use.

Whether you’re a technology leader or a new programmer, pay attention to these slow, long-term trends. They’re the ones that will change the face of our industry.

Operations or DevOps or SRE

The science (or art) of IT operations has changed radically in the last decade. There’s been a lot of discussion about operations culture (the movement frequently known as DevOps), continuous integration and deployment (CI/CD), and site reliability engineering (SRE). Cloud computing has replaced data centers, colocation facilities, and in-house machine rooms. Containers allow much closer integration between developers and operations and do a lot to standardize deployment.

Operations isn’t going away; there’s no such thing as NoOps. Technologies like Function as a Service (a.k.a. FaaS, a.k.a. serverless, a.k.a. AWS Lambda) only change the nature of the beast. The number of people needed to manage an infrastructure of a given size has shrunk, but the infrastructures we’re building have expanded, sometimes by orders of magnitude. It’s easy to round up tens of thousands of nodes to train or deploy a complex AI application. Even if those machines are all in Amazon’s giant data centers and managed in bulk using highly automated tools, operations staff still need to keep systems running smoothly, monitoring, troubleshooting, and ensuring that you’re not paying for resources you don’t need. Serverless and other cloud technologies allow the same operations team to manage much larger infrastructures; they don’t make operations go away.

The terminology used to describe this job fluctuates, but we don’t see any real changes. The term “DevOps” has fallen on hard times. Usage of DevOps-titled content in O’Reilly online learning has dropped by 17% in the past year, while SRE (including “site reliability engineering”) has climbed by 37%, and the term “operations” is up 25%. While SRE and DevOps are distinct concepts, for many customers SRE is DevOps at Google scale–and who doesn’t want that kind of growth? Both SRE and DevOps emphasize similar practices: version control (62% growth for GitHub, and 48% for Git), testing (high usage, though no year-over-year growth), continuous deployment (down 20%), monitoring (up 9%), and observability (up 128%). Terraform, HashiCorp’s open source tool for automating the configuration of cloud infrastructure, also shows strong (53%) growth.

It’s more interesting to look at the story the data tells about the tools. Docker is close to flat (5% decline year over year), but usage of content about containers skyrocketed by 99%. So yes, containerization is clearly a big deal. Docker itself may have stalled—we’ll know more next year—but Kubernetes’s dominance as the tool for container orchestration keeps containers central. Docker was the enabling technology, but Kubernetes made it possible to deploy containers at scale.

Kubernetes itself is the other superstar, with 47% growth, along with the highest usage (and the most search queries) in this group. Kubernetes isn’t just an orchestration tool; it’s the cloud’s operating system (or, as Kelsey Hightower has said, “Kubernetes will be the Linux of distributed systems”). But the data doesn’t show the number of conversations we’ve had with people who think that Kubernetes is just “too complex.” We see three possible solutions:

• A “simplified” version of Kubernetes that isn’t as flexible, but trades off a lot of the complexity. K3s is a possible step in this direction. The question is, What’s the trade-off? Here’s my version of the Pareto principle, also known as the 80/20 rule. Given any system (like Kubernetes), it’s usually possible to build something simpler by keeping the most widely used 80% of the features and cutting the other 20%. And some applications will fit within the 80% of the features that were kept. But most applications (maybe 80% of them?) will require at least one of the features that were sacrificed to make the system simpler.
  
• An entirely new approach, some tool that isn’t yet on the horizon. We have no idea what that tool is. In Yeats’s words, “What rough beast…slouches towards Bethlehem to be born”?
  
• An integrated solution from a cloud vendor (for example, Microsoft’s open source Dapr distributed runtime). I don’t mean cloud vendors that provide Kubernetes as a service; we already have those. What if the cloud vendors integrate Kubernetes’s functionality into their stack in such a way that that functionality disappears into some kind of management console? Then the question becomes, What features do you lose, and do you need them? And what kind of vendor lock-in games do you want to play?

The rich ecosystem of tools surrounding Kubernetes (Istio, Helm, and others) shows how valuable it is. But where do we go from here? Even if Kubernetes is the right tool to manage the complexity of modern applications that run in the cloud, the desire for simpler solutions will eventually lead to higher-level abstractions. Will they be adequate?

Observability saw the greatest growth in the past year (128%), while monitoring is only up 9%. While observability is a richer, more powerful capability than monitoring—observability is the ability to find the information you need to analyze or debug software, while monitoring requires predicting in advance what data will be useful—we suspect that this shift is largely cosmetic. “Observability” risks becoming the new name for monitoring. And that’s unfortunate. If you think observability is merely a more fashionable term for monitoring, you’re missing its value. Complex systems running in the cloud will need true observability to be manageable.

Infrastructure is code, and we’ve seen plenty of tools for automating configuration. But Chef and Puppet, two leaders in this movement, are both significantly down (49% and 40% respectively), as is Salt. Ansible is the only tool from this group that’s up (34%). Two trends are responsible for this. Ansible appears to have supplanted Chef and Puppet, possibly because Ansible is multilingual, while Chef and Puppet are tied to Ruby. Second, Docker and Kubernetes have changed the configuration game. Our data shows that Chef and Puppet peaked in 2017, when Kubernetes started an almost exponential growth spurt, as Figure 4 shows. (Each curve is normalized separately to 1; we wanted to emphasize the inflection points rather than compare usage.) Containerized deployment appears to minimize the problem of reproducible configuration, since a container is a complete software package. You have a container; you can deploy it many times, getting the same result each time. In reality, it’s never that simple, but it certainly looks that simple–and that apparent simplicity reduces the need for tools like Chef and Puppet.

The biggest challenge facing operations teams in the coming year, and the biggest challenge facing data engineers, will be learning how to deploy AI systems effectively. In the past decade, a lot of ideas and technologies have come out of the DevOps movement: the source repository as the single source of truth, rapid automated deployment, constant testing, and more. They’ve been very effective, but AI breaks the assumptions that lie behind them, and deployment is frequently the greatest barrier to AI success.

AI breaks these assumptions because data is more important than code. We don’t yet have adequate tools for versioning data (though DVC is a start). Models are neither code nor data, and we don’t have adequate tools for versioning models either (though tools like MLflow are a start). Frequent deployment assumes that the software can be built relatively quickly, but training a model can take days. It’s been suggested that model training doesn’t need to be part of the build process, but that’s really the most important part of the application. Testing is critical to continuous deployment, but the behavior of AI systems is probabilistic, not deterministic, so it’s harder to say that this test or that test failed. It’s particularly difficult if testing includes issues like fairness and bias.

Although there is a nascent MLOps movement, our data doesn’t show that people are using (or searching for) content in these areas in significant numbers. Usage is easily explainable; in many of these areas, content doesn’t exist yet. But users will search for content whether or not it exists, so the small number of searches shows that most of our users aren’t yet aware of the problem. Operations staff too frequently assume that an AI system is just another application—but they’re wrong. And AI developers too frequently assume that an operations team will be able to deploy their software, and they’ll be able to move on to the next project—but they’re also wrong. This situation is a train wreck in slow motion, and the big question is whether we can stop the trains before they crash. These problems will be solved eventually, with a new generation of tools—indeed, those tools are already being built—but we’re not there yet.

AI, Machine Learning, and Data

Healthy growth in artificial intelligence has continued: machine learning is up 14%, while AI is up 64%; data science is up 16%, and statistics is up 47%. While AI and machine learning are distinct concepts, there’s enough confusion about definitions that they’re frequently used interchangeably. We informally define machine learning as “the part of AI that works”; AI itself is more research oriented and aspirational. If you accept that definition, it’s not surprising that content about machine learning has seen the heaviest usage: it’s about taking research out of the lab and putting it into practice. It’s also not surprising that we see solid growth for AI, because that’s where bleeding-edge engineers are looking for new ideas to turn into machine learning.

Have the skepticism, fear, and criticism surrounding AI taken a toll, or are “reports of AI’s death greatly exaggerated”? We don’t see that in our data, though there are certainly some metrics to say that artificial intelligence has stalled. Many projects never make it to production, and while the last year has seen amazing progress in natural language processing (up 21%), such as OpenAI’s GPT-3, we’re seeing fewer spectacular results like winning Go games. It’s possible that AI (along with machine learning, data, big data, and all their fellow travelers) is descending into the trough of the hype cycle. We don’t think so, but we’re prepared to be wrong. As Ben Lorica has said (in conversation), many years of work will be needed to bring current research into commercial products.

It’s certainly true that there’s been a (deserved) backlash over heavy handed use of AI. A backlash is only to be expected when deep learning applications are used to justify arresting the wrong people, and when some police departments are comfortable using software with a 98% false positive rate. A backlash is only to be expected when software systems designed to maximize “engagement” end up spreading misinformation and conspiracy theories. A backlash is only to be expected when software developers don’t take into account issues of power and abuse. And a backlash is only to be expected when too many executives see AI as a “magic sauce” that will turn their organization around without pain or, frankly, a whole lot of work.

But we don’t think those issues, as important as they are, say a lot about the future of AI. The future of AI is less about breathtaking breakthroughs and creepy face or voice recognition than it is about small, mundane applications. Think quality control in a factory; think intelligent search on O’Reilly online learning; think optimizing data compression; think tracking progress on a construction site. I’ve seen too many articles saying that AI hasn’t helped in the struggle against COVID, as if someone was going to click a button on their MacBook and a superdrug was going to pop out of a USB-C port. (And AI has played a huge role in COVID vaccine development.) AI is playing an important supporting role—and that’s exactly the role we should expect. It’s enabling researchers to navigate tens of thousands of research papers and reports, design drugs and engineer genes that might work, and analyze millions of health records. Without automating these tasks, getting to the end of the pandemic will be impossible.

So here’s the future we see for AI and machine learning:

• Natural language has been (and will continue to be) a big deal. GPT-3 has changed the world. We’ll see AI being used to create “fake news,” and we’ll find that AI gives us the best tools for detecting what’s fake and what isn’t.
  
• Many companies are placing significant bets on using AI to automate customer service. We’ve made great strides in our ability to synthesize speech, generate realistic answers, and search for solutions.
  
• We’ll see lots of tiny, embedded AI systems in everything from medical sensors to appliances to factory floors. Anyone interested in the future of technology should watch Pete Warden’s work on TinyML very carefully.
  
• We still haven’t faced squarely the issue of user interfaces for collaboration between humans and AI. We don’t want AI oracles that just replace human errors with machine-generated errors at scale; we want the ability to collaborate with AI to produce results better than either humans or machines could alone. Researchers are starting to catch on.

TensorFlow is the leader among machine learning platforms; it gets the most searches, while usage has stabilized at 6% growth. Content about scikit-learn, Python’s machine learning library, is used almost as heavily, with 11% year-over-year growth. PyTorch is in third place (yes, this is a horse race), but usage of PyTorch content has gone up 159% year over year. That increase is no doubt influenced by the popularity of Jeremy Howard’s Practical Deep Learning for Coders course and the PyTorch-based fastai library (no data for 2019). It also appears that PyTorch is more popular among researchers, while TensorFlow remains dominant in production. But as Jeremy’s students move into industry, and as researchers migrate toward production positions, we expect to see the balance between PyTorch and TensorFlow shift.

Kafka is a crucial tool for building data pipelines; it’s stable, with 6% growth and usage similar to Spark. Pulsar, Kafka’s “next generation” competition, isn’t yet on the map.

Tools for automating AI and machine learning development (IBM’s AutoAI, Google’s Cloud AutoML, Microsoft’s AutoML, and Amazon’s SageMaker) have gotten a lot of press attention in the past year, but we don’t see any signs that they’re making a significant dent in the market. That content usage is nonexistent isn’t a surprise; O’Reilly members can’t use content that doesn’t exist. But our members aren’t searching for these topics either. It may be that AutoAI is relatively new or that users don’t think they need to search for supplementary training material.

What about data science? The report What Is Data Science is a decade old, but surprisingly for a 10-year-old paper, views are up 142% over 2019. The tooling has changed though. Hadoop was at the center of the data science world a decade ago. It’s still around, but now it’s a legacy system, with a 23% decline since 2019. Spark is now the dominant data platform, and it’s certainly the tool engineers want to learn about: usage of Spark content is about three times that of Hadoop. But even Spark is down 11% since last year. Ray, a newcomer that promises to make it easier to build distributed applications, doesn’t yet show usage to match Spark (or even Hadoop), but it does show 189% growth. And there are other tools on the horizon: Dask has seen nearly 400% growth.

It’s been exciting to watch the discussion of data ethics and activism in the past year. Broader societal movements (such as #BlackLivesMatter), along with increased industry awareness of diversity and inclusion, have made it more difficult to ignore issues like fairness, power, and transparency. What’s sad is that our data shows little evidence that this is more than a discussion. Usage of general content (not specific to AI and ML) about diversity and inclusion is up significantly (87%), but the absolute numbers are still small. Topics like ethics, fairness, transparency, and explainability don’t make a dent in our data. That may be because few books have been published and few training courses have been offered—but that’s a problem in itself.

Web Development

Since the invention of HTML in the early 1990s, the first web servers, and the first browsers, the web has exploded (or degenerated) into a proliferation of platforms. Those platforms make web development infinitely more flexible: They make it possible to support a host of devices and screen sizes. They make it possible to build sophisticated applications that run in the browser. And with every new year, “desktop” applications look more old-fashioned.

So what does the world of web frameworks look like? React leads in usage of content and also shows significant growth (34% year over year). Despite rumors that Angular is fading, it’s the #2 platform, with 10% growth. And usage of content about the server-side platform Node.js is just behind Angular, with 15% growth. None of this is surprising.

It’s more surprising that Ruby on Rails shows extremely strong growth (77% year over year) after several years of moderate, stable performance. Likewise, Django (which appeared at roughly the same time as Rails) shows both heavy usage and 63% growth. You might wonder whether this growth holds for all older platforms; it doesn’t. Usage of content about PHP is relatively low and declining (8% drop), even though it’s still used by almost 80% of all websites. (It will be interesting to see how PHP 8 changes the picture.) And while jQuery shows healthy 18% growth, usage of jQuery content was lower than any other platform we looked at. (Keep in mind, though, that there are literally thousands of web platforms. A complete study would be either heroic or foolish. Or both.)

Vue and Flask make surprisingly weak showings: for both platforms, content usage is about one-eighth of React’s. Usage of Vue-related content declined 13% in the past year, while Flask grew 10%. Neither is challenging the dominant players. It’s tempting to think of Flask and Vue as “new” platforms, but they were released in 2010 and 2014, respectively; they’ve had time to establish themselves. Two of the most promising new platforms, Svelte and Next.js, don’t yet produce enough data to chart—possibly because there isn’t yet much content to use. Likewise, WebAssembly (Wasm) doesn’t show up. (It’s also too new, with little content or training material available.) But WebAssembly represents a major rethinking of web programming and bears watching closely. Could WebAssembly turn JavaScript’s dominance of web development on its head? We suspect that nothing will happen quickly. Enterprise customers will be reluctant to bear the cost of moving from an older framework like PHP to a more fashionable JavaScript framework. It costs little to stick with an old stalwart.

The foundational technologies HTML, CSS, and JavaScript are all showing healthy growth in usage (22%, 46%, and 40%, respectively), though they’re behind the leading frameworks. We’ve already noted that JavaScript is one of the top programming languages—and the modern web platforms are nothing if not the apotheosis of JavaScript. We find that chilling. The original vision for the World Wide Web was radically empowering and democratizing. You didn’t need to be a techno-geek; you didn’t even need to program—you could just click “view source” in the browser and copy bits you liked from other sites. Twenty-five years later, that’s no longer true: you can still “view source,” but all you’ll see is a lot of incomprehensible JavaScript. Ironically, just as other technologies are democratizing, web development is increasingly the domain of programmers. Will that trend be reversed by a new generation of platforms, or by a reformulation of the web itself? We shall see.

Clouds of All Kinds

It’s no surprise that the cloud is growing rapidly. Usage of content about the cloud is up 41% since last year. Usage of cloud titles that don’t mention a specific vendor (e.g., Amazon Web Services, Microsoft Azure, or Google Cloud) grew at an even faster rate (46%). Our customers don’t see the cloud through the lens of any single platform. We’re only at the beginning of cloud adoption; while most companies are using cloud services in some form, and many have moved significant business-critical applications and datasets to the cloud, we have a long way to go. If there’s one technology trend you need to be on top of, this is it.

The horse race between the leading cloud vendors, AWS, Azure, and Google Cloud, doesn’t present any surprises. Amazon is winning, even ahead of the generic “cloud”—but Microsoft and Google are catching up, and Amazon’s growth has stalled (only 5%). Use of content about Azure shows 136% growth—more than any of the competitors—while Google Cloud’s 84% growth is hardly shabby. When you dominate a market the way AWS dominates the cloud, there’s nowhere to go but down. But with the growth that Azure and Google Cloud are showing, Amazon’s dominance could be short-lived.

What’s behind this story? Microsoft has done an excellent job of reinventing itself as a cloud company. In the past decade, it’s rethought every aspect of its business: Microsoft has become a leader in open source; it owns GitHub; it owns LinkedIn. It’s hard to think of any corporate transformation so radical. This clearly isn’t the Microsoft that declared Linux a “cancer,” and that Microsoft could never have succeeded with Azure.

Google faces a different set of problems. Twelve years ago, the company arguably delivered serverless with App Engine. It open sourced Kubernetes and bet very heavily on its leadership in AI, with the leading AI platform TensorFlow highly optimized to run on Google hardware. So why is it in third place? Google’s problem hasn’t been its ability to deliver leading-edge technology but rather its ability to reach customers—a problem that Thomas Kurian, Google Cloud’s CEO, is attempting to address. Ironically, part of Google’s customer problem is its focus on engineering to the detriment of the customers themselves. Any number of people have told us that they stay away from Google because they’re too likely to say, “Oh, that service you rely on? We’re shutting it down; we have a better solution.” Amazon and Microsoft don’t do that; they understand that a cloud provider has to support legacy software, and that all software is legacy the moment it’s released.

While our data shows very strong growth (41%) in usage for content about the cloud, it doesn’t show significant usage for terms like “multicloud” and “hybrid cloud” or for specific hybrid cloud products like Google’s Anthos or Microsoft’s Azure Arc. These are new products, for which little content exists, so low usage isn’t surprising. But the usage of specific cloud technologies isn’t that important in this context; what’s more important is that usage of all the cloud platforms is growing, particularly content that isn’t tied to any vendor. We also see that our corporate clients are using content that spans all the cloud vendors; it’s difficult to find anyone who’s looking at a single vendor.

Not long ago, we were skeptical about hybrid and multicloud. It’s easy to assume that these concepts are pipe dreams springing from the minds of vendors who are in second, third, fourth, or fifth place: if you can’t win customers from Amazon, at least you can get a slice of their business. That story isn’t compelling—but it’s also the wrong story to tell. Cloud computing is hybrid by nature. Think about how companies “get into the cloud.” It’s often a chaotic grassroots process rather than a carefully planned strategy. An engineer can’t get the resources for some project, so they create an AWS account, billed to the company credit card. Then someone in another group runs into the same problem, but goes with Azure. Next there’s an acquisition, and the new company has built its infrastructure on Google Cloud. And there’s petabytes of data on-premises, and that data is subject to regulatory requirements that make it difficult to move. The result? Companies have hybrid clouds long before anyone at the C-level perceives the need for a coherent cloud strategy. By the time the C suite is building a master plan, there are already mission-critical apps in marketing, sales, and product development. And the one way to fail is to dictate that “we’ve decided to unify on cloud X.”

All the cloud vendors, including Amazon (which until recently didn’t even allow its partners to use the word multicloud), are being drawn to a strategy based not on locking customers into a specific cloud but on facilitating management of a hybrid cloud, and all offer tools to support hybrid cloud development. They know that support for hybrid clouds is key to cloud adoption–and, if there is any lock in, it will be around management. As IBM’s Rob Thomas has frequently said, “Cloud is a capability, not a location.”

As expected, we see a lot of interest in microservices, with a 10% year-over-year increase—not large, but still healthy. Serverless (a.k.a. functions as a service) also shows a 10% increase, but with lower usage. That’s important: while it “feels like” serverless adoption has stalled, our data suggests that it’s growing in parallel with microservices.

Security and Privacy

Security has always been a problematic discipline: defenders have to get thousands of things right, while an attacker only has to discover one mistake. And that mistake might have been made by a careless user rather than someone on the IT staff. On top of that, companies have often underinvested in security: when the best sign of success is that “nothing bad happened,” it’s very difficult to say whether money was well spent. Was the team successful or just lucky?

Yet the last decade has been full of high-profile break-ins that have cost billions of dollars (including increasingly hefty penalties) and led to the resignations and firings of C-suite executives. Have companies learned their lessons?

The data doesn’t tell a clear story. While we’ve avoided discussing absolute usage, usage of content about security is very high—higher than for any other topic except for the major programming languages like Java and Python. Perhaps a better comparison would be to compare security with a general topic like programming or cloud. If we take that approach, programming usage is heavier than security, and security is only slightly behind cloud. So the usage of content about security is high, indeed, with year-over-year growth of 35%.

But what content are people using? Certification resources, certainly: CISSP content and training is 66% of general security content, with a slight (2%) decrease since 2019. Usage of content about the CompTIA Security+ certification is about 33% of general security, with a strong 58% increase.

There’s a fair amount of interest in hacking, which shows 16% growth. Interestingly, ethical hacking (a subset of hacking) shows about half as much usage as hacking, with 33% growth. So we’re evenly split between good and bad actors, but the good guys are increasing more rapidly. Penetration testing, which should be considered a kind of ethical hacking, shows a 14% decrease; this shift may only reflect which term is more popular.

Beyond those categories, we get into the long tail: there’s only minimal usage of content about specific topics like phishing and ransomware, though ransomware shows a huge year-over-year increase (155%); that increase no doubt reflects the frequency and severity of ransomware attacks in the past year. There’s also a 130% increase in content about “zero trust,” a technology used to build defensible networks—though again, usage is small.

It’s disappointing that we see so little interest in content about privacy, including content about specific regulatory requirements such as GDPR. We don’t see heavy usage; we don’t see growth; we don’t even see significant numbers of search queries. This doesn’t bode well.

Not the End of the Story

We’ve taken a tour through a significant portion of the technology landscape. We’ve reported on the horse races along with the deeper stories underlying those races. Trends aren’t just the latest fashions; they’re also long-term processes. Containerization goes back to Unix version 7 in 1979; and didn’t Sun Microsystems invent the cloud in the 1990s with its workstations and Sun Ray terminals? We may talk about “internet time,” but the most important trends span decades, not months or years—and often involve reinventing technology that was useful but forgotten, or technology that surfaced before its time.

With that in mind, let’s take several steps back and think about the big picture. How are we going to harness the computing power needed for AI applications? We’ve talked about concurrency for decades, but it was only an exotic capability important for huge number-crunching tasks. That’s no longer true; we’ve run out of Moore’s law, and concurrency is table stakes. We’ve talked about system administration for decades, and during that time, the ratio of IT staff to computers managed has gone from many-to-one (one mainframe, many operators) to one-to-thousands (monitoring infrastructure in the cloud). As part of that evolution, automation has also gone from an option to a necessity.

We’ve all heard that “everyone should learn to program.” This may be correct…or maybe not. It doesn’t mean that everyone should be a professional programmer but that everyone should be able to use computers effectively, and that requires programming. Will that be true in the future? No-code and low-code products are reaching the market, allowing users to build everything from business applications to AI prototypes. Again, this trend goes way back: in the late 1950s, the first modern programming languages made programming much easier. And yes, even back then there were those who said “real men use machine language.” (And that sexism was no doubt intentional, since the first generation of programmers included many women.) Will our future bring further democratization? Or a return to a cult of “wizards”? Low-code AI and complex JavaScript web platforms offer conflicting visions of what the future may bring.

Finally, the most important trend may not yet appear in our data at all. Technology has largely gotten a free ride as far as regulation and legislation are concerned. Yes, there are heavily regulated sectors like healthcare and finance, but social media, much of machine learning, and even much of online commerce have only been lightly regulated. That free ride is coming to an end. Between GDPR, the California Consumer Privacy Act (which will probably be copied by many states), California Propositions 22 and 24, many city ordinances regarding the use of face recognition, and rethinking the meaning of Section 230 of the Communications Decency Act, laws and regulations will play a big role in shaping technology in the coming years. Some of that regulation was inevitable, but a lot of it is a direct response to an industry that moved too fast and broke too many things. In this light, the lack of interest in privacy and related topics is unhealthy. Twenty years ago, we built a future that we don’t really want to live in. The question facing us now is simple:

What future will we build?

You can also see other highlights from the event.

This keynote was sponsored by Oracle Cloud Infrastructure.

You can also see other highlights from the event.

You can also see other highlights from the event.