Sometimes good engineering looks like over-engineering.

I was recently putting together a talk that walks through some architecture and engineering decisions I’ve made over the last eight-plus years when something stood out to me.

Without context, many of those decisions probably look like over-engineering.

🤔 An Example: Operational Toggles

In several mission-critical payment applications, we follow a simple philosophy:

If a dependency isn’t required to process customer traffic, we should be able to turn it off.

For example, metrics.

Metrics are important, but they aren’t more important than processing transactions. If something goes wrong with our metrics implementation, we want to be able to turn off metrics while continuing to process traffic.

The same applies to out-of-band management messages and data updates. We typically use message brokers for this, and if there is an issue with that integration or a bug in the message handlers, we want the ability to stop consuming those messages without taking down the application.

Why?

Because we want fine-grained operational control over our platform. A non-critical capability shouldn’t be able to take down the critical path.

🤯 Isn’t That Over-Engineering?

Maybe.

These operational toggles add complexity to the system: flags that need to be managed, tests that need to be written and executed, and operational procedures for failures that might never happen.

For many applications, it’s probably not worth it.

If a metrics library causes a memory leak in an application that does batch processing, restarting to release the memory is perfectly acceptable.

If a message handler crashes an application that is automatically restarted, it might be ok depending on the application. Not great, but if requests are retried and nothing truly fails, no harm, no foul.

Not every application needs the complexity of these operational toggles.

But when an application crash could cause impact, when the system is mission-critical, and every failure is potentially impactful, the equation changes.

🧠 Context Determines Everything

For the batch and non-critical application examples, these operational toggles could easily be considered over-engineering.

But for a mission-critical, always-on, large-scale platform, when a situation occurs and these operational toggles are used, they become incredibly valuable.

The above examples are not hypothetical.

A change in metrics causes a spike in memory usage. An unsafe type assertion in a data sync handler from a message broker causes an application to crash.

These are real scenarios, scenarios that I’ve seen in mission-critical systems. Because these operational toggles existed, these scenarios were not disastrous or even inconvenient.

These operational toggles let us turn off the cause of an issue. More importantly, they gave us time. Time to understand what happened, build and properly test a fix, and validate that the fix actually solved the root problem.

All without rushing a change to production while the platform was unstable.

🧐 Final Thoughts

If you look at the way we use operational toggles, it could easily feel like over-engineering. But building critical systems means planning for failure scenarios even if they are unlikely.

Sometimes that additional complexity might look unnecessary. But when something goes wrong, you’re thankful it’s there.

Originally posted on #Bengineering.

For the canonical version and future updates, read it here: Sometimes good engineering looks like over-engineering

Sometimes good engineering looks like over-engineering was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

AI has made code cheap to create, but more code means more overhead.

Before coding agents, creating more code cost time and effort because engineers had to write it.

That naturally encouraged us to reuse code as much as possible: libraries, frameworks, keeping implementations concise, and thinking twice before rebuilding functionality that already existed.

Coding agents have dramatically reduced that natural cost.

Today, generating another implementation costs almost nothing. But you still have to maintain it.

🤖 Just Let the Agent Build It

There’s a growing mindset that because agents can generate code quickly, there’s less reason to worry about reuse.

Why spend time finding an existing library when an agent can recreate the functionality in seconds? And initially, that might feel faster, better.

But let’s say you have six applications that all need to validate email addresses against your internal requirements.

Instead of creating a common package, you let your coding agent implement email validation directly in each application. As long as the prompt is the same, it’s fine, right? Not necessarily. You can easily end up with six implementations, all slightly different.

Then you find a bug.

🪳 Creation is Cheap, Maintenance Isn’t

Fixing the application with the bug is easy.

Tell the agent about the bug; it fixes the implementation, maybe it adds some tests, done.

What about those five other implementations? Do they also have this bug?

Maybe, maybe not. Because each application has a different implementation, they may have the same bug or different bugs.

You’ve essentially found yourself in a situation where, depending on the application, an email might work, or it might not. Maybe it works in three of the six, or two. What happens if that email works for a registration page, but not the login page?

Not a great user experience.

🧱 Reuse Still Matters

Compare that to using a shared package. You write it once, everyone uses the same package.

You find a bug, fix it once, add tests, and release a new version.

Every instance using that new version has that bug fixed.

Now your suite of applications is consistent, and consistency matters most. Rejecting a valid email isn’t great, but if you’re consistent about it, that’s a much better experience than it sometimes working and sometimes not.

That consistency is much easier to manage if the implementation is in one reusable library.

⚖️ Don’t Go DRY Crazy

Don’t use the above example to go to the other extreme, where every similar three lines of code turns into a shared library. Breaking everything into its own little shared library creates overhead costs, too.

It’s important to strike a balance between reusable and duplicative.

If you’re writing something with complex or nuanced behavior that’s needed across multiple applications, or more importantly, needs to behave consistently across them, a shared library is probably a good idea.

If it’s simple, situationally unique, and unlikely to change, don’t bother.

🧐 Final Thoughts

The cost of creating code has changed significantly. But the cost of managing it hasn’t.

Testing it, patching it, securing it, and keeping implementations consistent all become more expensive as the amount of code you own grows.

AI has made it incredibly cheap to create more code. Don’t confuse cheap to generate with cheap to own.

Originally posted on #Bengineering.

For the canonical version and future updates, read it here: AI makes code cheap to create, not cheap to own

AI makes code cheap to create, not cheap to own was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’m a believer that, as an engineer, you should be able to run your software locally. But I hear it often: “We can’t run locally because of some reason.”

Sometimes it’s valid. There are architectures and platforms out there that prevent running locally. But more often than not, when it comes to backend distributed systems, it’s a design or implementation decision that nobody has challenged.

Why This Matters

Everyone these days is focused on speeding up software delivery. Today, that generally means coding agents, AI tooling, and code generation.

But writing code faster doesn’t matter much if validating a one-line change takes an hour.

The speed of software delivery is often dictated by validation, not code generation.

The Problem with Shared Dev Environments

Many teams still rely on shared development environments as their primary way of validating changes.

That process usually looks something like this:

• Make a change
• Commit the change
• Push the branch
• Wait for a build
• Wait for a deployment
• Run tests

If everything goes well, you can validate your change. If a mistake was made, or something doesn’t work right, you have to fix it and start over.

The result is a slow feedback loop, which can sometimes influence behavior.

The more friction there is to validate a change, the larger that change becomes. If testing takes a long time, you will naturally focus on making sure everything is perfect before spending time validating.

Local Validation Influences Good Behavior

Compare the above process to one that uses a locally running service.

• Make a change
• Build
• Start the service
• Run tests

The process is dramatically shorter and, more importantly, faster. This encourages engineers to make smaller changes, test more frequently, iterate their implementations, and find mistakes earlier.

This all results in better software.

The Excuses

“My service depends on too many other services.”

Run the services you own locally. Mock the services you don’t.

“I need a database or message broker.”

Run those locally too. There’s very likely a Docker container for each of them.

“We use cloud services.”

This one can be harder, but emulators exist for some services. Other services might have a Dockerized open-source alternative. For those that don’t, you could mock them.

There will always be exceptions: mainframes, specialized hardware, or unique managed services.

But I find many teams jump straight to “we can’t run locally” before they’ve seriously explored how they could.

Running Local Doesn’t Mean Everything

You don’t need to recreate your entire production environment on a laptop, though if you can, go for it.

The goal is to validate your change without depending on a shared or non-local environment that takes forever to test against.

Local development isn’t about recreating production. It’s about creating enough of the environment to validate your change. If that means turning off some functionality or creating mock services, it might be worth it.

Why This Matters Even More Now

Validating changes quickly is becoming more important as teams adopt coding agents. An agent can generate code in seconds.

But if every iteration with your coding agent requires committing, pushing, building, deploying, and waiting, the feedback loop becomes the bottleneck.

Final Thoughts

Agents make mistakes. Humans make mistakes.

The faster you can validate a change, the faster you can correct those mistakes. That’s why local execution isn’t just a convenience.

It’s one of the most effective ways to shorten the feedback loop.

“We can’t run locally” is usually a design smell was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the simplest reliability rules I’ve learned is this: Every dependency is another way for your service to fail.

Why Dependencies Matter

Every service has dependencies.

• Databases
• Caches
• Configuration services
• Secrets managers
• Logging pipelines
• Tracing backends

All of these dependencies can fail, and when they do, the typical service will fail with them.

The more dependencies a service has, the more failures it inherits. Every dependency adds features, but it also adds failure modes.

Reliability is often about deciding which dependencies are actually worth the tradeoff.

Why Edge Systems Tend to Be Dependency-Light

I recently wrote about how systems closest to the customer carry the greatest responsibility for availability.

This is one reason edge systems tend to be dependency-light.

Load balancers, API gateways, and routers are responsible for availability.

Every dependency added to these systems creates another opportunity to take down the entire platform. So they tend to avoid dependencies whenever possible.

Eliminating Dependencies Isn’t Always Necessary

Sometimes removing a dependency entirely isn’t practical.

A better question is:

• What happens if the dependency disappears?
• Can the service continue operating?
• Can it use cached data?
• Can it fall back to a last-known-good configuration?
• Can it degrade gracefully?

If the answer is yes, you’ve significantly improved reliability even though the dependency still exists.

A Real-World Example

Take Envoy Proxy. Envoy can receive configuration from an xDS service. But it doesn’t call xDS for every request.

Instead:

• Configuration is fetched periodically
• Stored in memory
• Used locally during request processing

If the xDS service becomes unavailable, Envoy continues routing traffic using its last known configuration. The dependency still exists, but request processing no longer depends on its availability.

That’s a very different reliability model.

Don’t Make Observability a Hard Dependency

One of the most common mistakes I see is making observability a required dependency.

If your logging or tracing backend becomes unavailable, should customer traffic stop flowing? No.

In most cases, observability should be a best effort.

Use asynchronous logging, buffering, and truncation policies so customer traffic continues to flow even when your observability platform is experiencing issues.

Operational visibility is important, but customer availability is more important.

Final Thoughts

You’ll never eliminate every dependency. But you can eliminate unnecessary ones.

And for the remaining dependencies, you can design your service to survive failures.

The most reliable services aren’t dependency-free. They’re designed to survive dependency failures.

To make a service more stable, eliminate dependencies was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

You’ve all heard the advice: “Avoid caching because caching is difficult to get right.” I agree with part of that statement. Caching can absolutely be difficult. But I think the reality is more nuanced.

The difficulty level of caching depends heavily on the type of data you are caching.

Not All Data Is Equal

When engineers talk about caching complexity, they’re usually thinking about data that changes frequently and requires strong consistency.

That’s the hardest kind of data to cache. Because you need to answer some difficult questions.

• How do you invalidate the cache?
• How do you handle updates?
• What happens when the cache population fails?
• How stale is too stale?

These problems are real, but they don’t apply to all data.

Frequently Updated Data

Some data changes constantly and has strict accuracy requirements. Account balances, inventory counts, active orders, and similar records. This is where caching is difficult.

Every stale read has a consequence, and in some cases, it’s not worth the complexity.

Infrequently Updated Data

Some data changes occasionally, once an hour, once a day, or even longer. Configuration data, product catalogs, country codes, and similar records.

In these cases, a small amount of staleness might be fine. If data changes hourly and your cache refreshes every few minutes, that’s often a reasonable tradeoff. The less frequently data changes, the less sensitive you become to cache staleness.

Caching is much easier with infrequently updated data.

Immutable Data

Immutable data is the easiest to cache. Because it never changes. Ledger entries, event records, receipts, and other immutable records.

Once immutable data is loaded into a cache, cache invalidation largely disappears as a problem. Cache complexity for immutable data is more about balancing performance and hit-miss ratios.

Most of the time, immutable data is an ideal candidate for caching.

The Real Question

Instead of asking, “Should I cache?” Ask yourself, “How often does this data change? How accurate does it need to be?”

The answers will give you an idea of how complex caching will be.

Final Thoughts

Caching isn’t inherently hard.

Maintaining correctness for frequently changing data is hard.

The more stable the data, the less complex the caching needs to be.

Like most things in system design, you can’t just take all-or-none answers like “always cache” or “never cache.” It’s important to understand the trade-offs, constraints, and consistency requirements of the data you’re working with.

Caching isn’t hard. Some data is hard to cache was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

The closer a component is to the customer, the greater its responsibility for keeping the entire platform available, even when everything behind it is having a bad day.

Not All Services Carry the Same Reliability Burden

Let’s consider a typical platform.

Customer -> Load Balancer -> API Gateway -> Orchestrator -> Microservices -> Database

Every layer has a different job. But every layer also has a different level of responsibility for resiliency.

As you move toward the customer, that responsibility increases.

Deep Services Focus on Business Logic

At the deepest layers of the platform, services are usually focused on business capabilities.

They process orders, transfer money, manage inventory, and store data.

These services often have databases, business rules, stateful operations, and multiple dependencies.

Resiliency matters, but it’s often focused on correctness.

If a database call fails:

• Should the transaction roll back?
• Should the service fail over?
• Should a compensating transaction occur?

These services are primarily concerned with business outcomes.

The Middle Layers Absorb Failures

Move up a layer, and you often find orchestrators and workflow services. These components coordinate work across multiple services.

If one service fails, the orchestrator may retry, execute fallback logic, trigger compensating actions, or roll back a workflow. Their job is not just executing business logic, it’s ensuring execution succeeds despite failures.

The Edge Exists to Protect Everything Behind It

At the edge, things change.

Load balancers and API gateways are often stateless, dependency-light, highly available, and extremely fast.

Why?

Because their primary responsibility is availability. Everything behind them is allowed to fail, and they absorb as much of that failure as possible.

They:

• Route around failures
• Shed load
• Fail over traffic
• Enforce timeouts
• Apply retries
• Protect backend systems

The edge isn’t just resilient for itself. It’s resilient on behalf of everything behind it.

Final Thoughts

The deepest services in a platform should be focused on business logic. The edge should be focused on availability.

The more failures your edge can absorb, the less every downstream service needs to care. That’s why the closer you get to the customer, the more stable the platform must become.

The closer to the edge, the more stable a platform must be was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.