The closer a component is to the customer, the greater its responsibility for keeping the entire platform available, even when everything behind it is having a bad day.

Not All Services Carry the Same Reliability Burden

Let’s consider a typical platform.

Customer -> Load Balancer -> API Gateway -> Orchestrator -> Microservices -> Database

Every layer has a different job. But every layer also has a different level of responsibility for resiliency.

As you move toward the customer, that responsibility increases.

Deep Services Focus on Business Logic

At the deepest layers of the platform, services are usually focused on business capabilities.

They process orders, transfer money, manage inventory, and store data.

These services often have databases, business rules, stateful operations, and multiple dependencies.

Resiliency matters, but it’s often focused on correctness.

If a database call fails:

• Should the transaction roll back?
• Should the service fail over?
• Should a compensating transaction occur?

These services are primarily concerned with business outcomes.

The Middle Layers Absorb Failures

Move up a layer, and you often find orchestrators and workflow services. These components coordinate work across multiple services.

If one service fails, the orchestrator may retry, execute fallback logic, trigger compensating actions, or roll back a workflow. Their job is not just executing business logic, it’s ensuring execution succeeds despite failures.

The Edge Exists to Protect Everything Behind It

At the edge, things change.

Load balancers and API gateways are often stateless, dependency-light, highly available, and extremely fast.

Why?

Because their primary responsibility is availability. Everything behind them is allowed to fail, and they absorb as much of that failure as possible.

They:

• Route around failures
• Shed load
• Fail over traffic
• Enforce timeouts
• Apply retries
• Protect backend systems

The edge isn’t just resilient for itself. It’s resilient on behalf of everything behind it.

Final Thoughts

The deepest services in a platform should be focused on business logic. The edge should be focused on availability.

The more failures your edge can absorb, the less every downstream service needs to care. That’s why the closer you get to the customer, the more stable the platform must become.

Most resiliency discussions focus on retries, timeouts, and circuit breakers. But some of the most important resiliency patterns happen after the failure.

That’s where compensating transactions come in.

Resiliency Is About Recovery

A common mistake is thinking resiliency means preventing failures.

Failures are inevitable.

Networks fail. Services crash. Messages get lost. Requests time out.

True resiliency is accepting that failures will occur and having a plan to recover when they do.

Sometimes recovery means correcting actions that may already have occurred.

A Real-World Example

Have you ever swiped your card at a store, received a charge notification, only to have the terminal report an error and later discover that the charge was nowhere to be found?

What happened?

The point-of-sale terminal sent an authorization request.

But before it received a response, something failed: a network issue, a timeout, or a problem somewhere in the payment flow.

Now the terminal doesn’t know whether the transaction succeeded or failed.

The authorization request may have reached the card issuer and been processed, or it may not have.

Rather than risk leaving the customer incorrectly charged, the terminal sends a second transaction: a reversal.

The Compensating Transaction

That reversal is a real-world example of a compensating transaction.

Its purpose is simple:

Undo the effects of a previous action if that action completed successfully. If the original authorization never happened, nothing changes.

If it did happen, the reversal corrects it. Instead of determining exactly what happened, the system performs a corrective action.

Beyond Payments

Compensating transactions show up in many real-world distributed systems:

• Releasing inventory after a failed order
• Refunding a payment after a fulfillment failure
• Canceling a reservation when verification times out

The pattern is always the same: something failed, and the system takes a corrective action.

Final Thoughts

When engineers think about resiliency, they often focus on preventing failures. But distributed systems fail in unexpected ways.

Sometimes the most resilient thing a system can do isn’t retry. It’s correct the mistake and move forward.

Sometimes, the most resilient thing a system can do isn’t retry was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

This debate comes up constantly. Should resiliency live in the platform components, or should the application own it?

Like most things in distributed systems, the answer is: It depends.

Infrastructure Understands Traffic

Service meshes, API gateways, and load balancers are great at handling generic resiliency concerns.

Things like:

• Connection timeouts
• Automatic retries
• Circuit breakers
• Request-level failover

The advantage is obvious. You remove complexity from the application and apply resiliency consistently across services.

For many scenarios, this is the right answer.

Infrastructure Doesn’t Understand Intent

The challenge is that platform services only understand traffic. They don’t understand why the request is made.

They don’t know whether a request is reading customer profile data, reserving inventory, transferring money, or uploading cat videos.

To the service mesh, they’re all just requests. Some requests might even have custom timeouts and retries configured. Most of the time, that’s fine.

Sometimes, it’s not.

Context-Aware Resiliency

Some resiliency decisions require application context.

Consider a financial transaction. Should a timeout trigger a retry? Maybe.

Does the request have an idempotency key? If yes, blindly retrying might be safe. If not, things become more complicated.

Did the request reach the downstream system? Was it partially processed?

Do you need a compensating transaction before retrying?

At this point, the retry is no longer a networking decision. It’s a business decision. And business decisions belong in the application.

The Real Answer

The best architectures usually use both approaches.

Use platform resiliency whenever the decision can be made without application context. But when correctness depends on understanding the request itself, move that logic into the application.

There should be a strong preference toward offloading complexity when possible. Just don’t offload decisions that require business context.

Final Thoughts

I often see teams over-index on platform-level resiliency. And while reducing application complexity is valuable, it isn’t free.

Infrastructure understands traffic and request characteristics. Applications understand business intent.

When resiliency decisions depend on business intent, they belong in the application.

Should retries and timeouts live in your application or your service mesh? was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dual writes are one approach that deserves more attention.

Most database migrations fall into one of a few buckets:

• Export and import
• Replication between two databases
• Services specifically built to synchronize data

All of those approaches can work well. But sometimes you need both databases active while gradually migrating traffic from one to the other.

That’s where dual writes become interesting.

What Are Dual Writes?

The idea is exactly what it sounds like. Every write is sent to both databases.

Insert a row? Write it twice.

Update a record? Update it twice.

Delete something? Yup, delete it twice.

The goal is to keep the old and new databases synchronized while both are active.

How It Works

The most common approach is implementing dual writes directly in the application.

Instead of maintaining one database connection pool, the application maintains two and executes write operations against both systems.

In some cases, infrastructure can help as well.

For example, Envoy supports request mirroring patterns that can be useful when migrating certain technologies, such as Redis.

The implementation will vary, but the concept remains the same: each write is performed twice.

The Hard Part: Failure Handling

The hard part is not writing twice, but rather handling partial success.

What happens when Database A succeeds, but Database B fails?

Now the two databases disagree.

Do you retry? Can the operation be safely retried?

Can you roll back the successful write?

Do you have locking or reconciliation mechanisms?

This is where dual writes become significantly more complex than they initially sound.

Writing twice is easy. Maintaining correctness is the hard part.

Where Dual Writes Work Best

Dual writes tend to work best when:

• Updates occur frequently
• Eventual consistency is acceptable
• Reconciliation processes exist
• Future updates naturally correct drift

They become much harder in systems that require strict consistency guarantees for every operation.

Why Teams Use Them

Despite the complexity, dual writes enable a powerful migration approach.

You can:

• Introduce a new database platform or table
• Keep it synchronized with the old platform
• Gradually migrate read traffic
• Eventually retire the old database

All without requiring a large downtime window.

Final Thoughts

Like most migration strategies, dual writes are a tradeoff.

They add complexity.

But they also enable something extremely valuable: migrating between database platforms while the system remains live.

When downtime is not an option, dual writes can be one of the most powerful migration techniques available.

Need to migrate from one database to another without downtime? was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

I recently talked about using glue services (Anti-Corruption Layers) to isolate modern platforms from legacy integrations.

Today I want to talk about another type of glue service: data synchronization services.

🗃️ The Real Modernization Problem

One of the hardest parts of replacing a legacy platform is usually not the application itself.

It’s the data.

In a perfect world, you could: take downtime, export the database, import it into the new platform, and switch traffic.

Simple.

But large downtime windows are increasingly rare. And big-bang migrations are risky enough that many organizations actively avoid them.

That means old and new systems often need to run side-by-side for a while.

Both platforms stay active.

Both serve customers.

Both need access to the same data. This is where modernization becomes difficult.

🔄 Keeping Two Systems in Sync

When you are changing both the application and the underlying database, keeping data synchronized becomes difficult very quickly.

Especially when the two systems use different schemas, database technologies, data models, and update patterns.

You can’t simply export/import anymore. You need continuous synchronization.

👨🏻‍🤝‍👨🏾 Enter the Synchronization Service

One useful pattern is to build a dedicated synchronization service whose sole responsibility is to keep data aligned across systems.

How synchronization works depends entirely on the platform.

I’ve used several approaches over the years.

⚙️ Database Triggers

One approach is to use database triggers or change-capture mechanisms in the legacy system.

When data changes, it’s detected, the synchronization service processes it, and the new platform is updated.

📩 Event-Based Synchronization

Another approach is to have the legacy platform emit events via a message broker.

The synchronization service consumes those events and updates the new database/platform accordingly. This tends to work especially well when modernizing toward event-driven systems.

⏳ Temporary Infrastructure

The important thing to remember is that these synchronization services are usually temporary. Their job is not to become a permanent platform.

Their purpose is to enable safe migration while reducing downtime and risk.

Once the migration is complete, the glue service disappears.

⚠️ One Important Warning

Bi-directional synchronization becomes extremely difficult very quickly.

With bi-directional synchronization, you have to solve:

• Conflict resolution
• Consistency problems
• Update ordering
• Partial failures

Whenever possible, avoid bi-directional synchronization.

In many cases, it is simpler to migrate the application layer first, then the database layer.

🧐 Final Thought

Modernization is rarely just replacing code. A major challenge is safely bridging old and new systems during transition periods.

Synchronization glue services are often what make zero-downtime migrations possible.

Glue Services: Part Two — Data Synchronization was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

When modernizing legacy systems, don’t be afraid to build glue services.

One of the biggest mistakes I see during modernization efforts is letting legacy integrations dictate the design of the new platform.

That usually leads to putting fresh paint on the same old house. Rebuilding the same architecture with a newer tech stack.

😴 The Dream vs. Reality

The dream project is building a brand-new platform with no existing users, integrations, or constraints.

You can design everything “the right way” from day one. But most real-world projects are not like that.

Most projects are modernization efforts. And most modernization efforts are weighed down by existing integrations, legacy protocols, and operational dependencies.

Changing customer behavior and expectations is often harder than rebuilding the platform itself.

🥲 The Common Mistake

Many teams respond to this by centering their new platform on the old integration model.

If customers use XML over Message Brokers, the new platform may speak JSON, but it still inherits the event-driven constraints — even when they no longer make sense.

If the legacy system exchanges files, the new platform is usually heavily batch-based.

The problem is:

Your modernization effort becomes constrained by the past.

🤯 A Better Approach: Glue Services

The formal term for this pattern is an Anti-Corruption Layer.

Personally, I think “glue service” explains it better. It makes the concept easier to understand.

Build the internal platform the way you actually want it designed.

Then build lightweight edge services that translate between legacy integrations and your modern platform.

• XML over Message Brokers? Use gRPC internally.
• Files? Break them into APIs.
• Long-lived ISO8583 TCP connections? Terminate them at the edge and use gRPC + microservices behind them.

The glue service has one responsibility: to translate between old and new worlds.

🤔 Why This Matters

These glue services give you two major advantages.

First, your internal architecture stays modern and optimized for current engineering practices. Second, your customers and integrations do not need to migrate immediately.

That dramatically reduces modernization risk. Temporary glue services can be huge modernization enablers.

But sometimes these glue services live forever, which is ok.

The important thing is that legacy integrations no longer constrain your modern platform.

🔁 It Works at Both Ends

This pattern applies both to inbound (clients calling your platform) and outbound (your platform calling legacy systems) integrations.

In many large systems, you’ll find glue services on both sides of the platform.

At the edge entering the system, and again leaving it.

🧐 Final Thoughts

Modernization is rarely just rewriting code. A major challenge is safely bridging old and new systems during transition periods.

I’ve used this pattern many times.

• Breaking files into APIs
• Translating ISO8583 and ISO20022 messages into Protobuf
• Terminating long-lived TCP sessions at the edge of a gRPC-based microservices platform

Glue services may not be glamorous, but they are one of the safest ways to modernize systems without dragging the past into the future.

When modernizing legacy systems, don’t be afraid to build glue services was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Coding agents can’t see your architecture diagrams — fix that.

I’ve been talking a lot about architecture documentation and how it helps both humans and agents.

From my experience, most teams spend 90% of their effort on diagrams and 10% on text.

Why? Because they are the fastest way to communicate a system:

Diagrams show how components interact, how data flows, and what depends on what. They help humans understand complex systems quickly.

That’s why most architecture documentation leans heavily on diagrams.

🤔 The Problem

Most diagrams are images. Images are great for humans, but not for agents.

Some agents can interpret images, but not reliably or consistently. And even when an agent can interpret an image, it can’t reliably reason about it or keep it up to date.

🧠 Make Diagrams Understandable

You don’t need to move away from diagrams to embrace agents. Just make them understandable to agents.

If diagrams are the most valuable part of your architecture documentation, make them readable as code.

📝 Use Code-Based Diagrams

Tools like Mermaid turn text into diagrams.

That means agents can read them, reason about them, and even keep them up to date.

A Mermaid diagram isn’t just documentation — it’s structured context.

💡 Why This Matters

When your diagrams are code:

• They live with your system
• They evolve with changes (as long as you update them)
• They can be versioned
• They can be generated or modified by agents

And most importantly, they become a living, usable context.

🔄 Keep Them Up to Date

Keeping architecture documentation up to date is always a pain. Images make that even harder.

Code-based diagrams are much easier to keep up to date. Especially when you use agents and the architecture sits next to your code.

When you make a change, direct the agent to update the architecture documentation as well. Even if you don’t fully trust agents with code changes, architecture documentation is low risk.

Instead of letting it drift over time, let agents keep it up to date.

Coding agents can’t see your architecture diagrams — fix that was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most teams put low-level architecture in the wrong place, if they document it at all.

I recently wrote about keeping architecture documentation in Git and making it available to coding agents.

A common pattern I see is teams maintaining a central architecture repository that spans multiple services and platforms. It’s an approach I use as well, but there’s a nuance.

🏯 Architecture Has Layers

Good architecture documentation spans multiple levels.

At a high level, you have:

Business Architecture 👨‍💼

Capabilities, domains, and high-level concepts.

Solution Architecture 👩‍🎨

Mapping capabilities to systems.

Defining which systems own what.

What gets replaced.

What gets invested in.

System Architecture 👷

Services, databases, jobs, infrastructure, and how they interact.

These three layers belong in a central architecture repository.

But there is one more layer that many teams forget.

🔍 The Missing Layer

Component Architecture 🧱

This is where architecture stops being abstract and starts becoming actionable.

Things like:

• Why middleware X was chosen over Y
• How a request flows through a handler
• What interfaces exist and why
• What metrics and tracing are required
• How a feature is expected to behave

These decisions are too detailed for a central architecture repo-and too important to leave undocumented.

📦 Put It in the Codebase

Component-level architecture belongs with the code.

Not in a separate repo, not in a wiki, and certainly not only in tribal knowledge.

Why does this matter? Because this is where architecture stops being abstract and becomes a specification.

A specification for engineers, reviewers, and now, coding agents.

🤖 Why This Matters for Agents

Agents don’t understand your system by osmosis the way human engineers do.

They only know what you give them. If your architecture decisions live far away from your code, your agent:

• Misses constraints: Uses technologies that aren’t allowed in your environment
• Guesses at design: Assumes REST when you expect event-driven messaging
• Makes inconsistent choices: Pulls in the wrong libraries or dependencies

But when that context lives in the repo, architecture:

• Evolves with the code
• Is versioned
• Is available at the moment it’s needed

🧠 Final Thought

I’ve always believed the best engineers don’t just understand one codebase; they learn the end-to-end system.

The same applies to agents.

If you want better output from an agent, give it the same level of detail you’d give a senior engineer.

Originally published at https://bencane.com on June 4, 2026.

Most teams put low-level architecture in the wrong place was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Your coding agent is missing one thing: architectural context.

I’ve been a big believer in Architecture Decision Records ( ADRs) long before coding agents came along.

Documenting decisions gives engineers context:

Why is the system designed a certain way? What constraints existed at the time? What tradeoffs were made?

That context matters. It also matters for agents.

🤖 Agents Need Context Too

Unlike human engineers, agents don’t get context from hallway conversations, shadowing others, or tribal knowledge.

They only know what you capture. The best way to capture architectural context? Write it down as a decision record-and make it accessible to agents.

The only question is, what’s the best way to make decision records accessible?

🏗️ Option 1: MCP Server

If your ADRs live in a wiki or documentation system, you can expose them through an MCP server.

This works well when documentation is spread across teams or multiple systems that need to be aggregated.

You want a unified interface for agents. MCP is a good approach, but it comes with some infrastructure overhead.

🧱 Option 2: Keep ADRs in Git

I’ve long preferred storing ADRs in Git.

It provides versioning, review workflows, automated validation, and is where engineering work happens. Storing ADRs in Git, ideally alongside your code, is the fastest way to give agents usable context.

The challenge is that architecture often spans multiple services and repositories. So many centralize their architecture into a single repository, which is not where your code lives.

🌉 Bridging the Gap

Most modern coding agents let you include additional directories or sources at runtime, either through slash commands or CLI options.

That means you can: open your codebase, include your architecture repository, and run the agent with context.

Just adding another directory gives your agent an understanding of system constraints, architecture decisions, technology choices, and surrounding systems. These are not things an agent can reliably infer from code alone.

💡 Final Thought: Why Context Matters

With architectural context, agents produce code that aligns with your system.

When engineers understand the system end to end, they make better decisions. The same applies to agents.

If you want better results, give better context.

Originally published at https://bencane.com on May 28, 2026.

Your coding agent is missing one thing: architectural context was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the easiest ways to break a gRPC service in production is health-checking the wrong listener.

A common issue I see teams run into when adopting gRPC is leaving readiness checks pointed at their HTTP listener while production traffic actually flows through gRPC.

Everything looks fine until it suddenly doesn’t.

🤔 The Problem

Many gRPC services run two listeners: one for HTTP and one for gRPC.

The HTTP listener often exists for metrics, liveness checks, and management APIs. Teams moving to gRPC often reuse the HTTP health checks they set up for their REST-based services.

It’s generally a good idea to reuse what you already have, but in this case, it can be misleading.

⚠️ Health-Check What Serves Traffic

If customers connect through gRPC, your first readiness check should too.

Your HTTP listener can be perfectly healthy while the gRPC listener is misconfigured, hung, or otherwise failing.

Meanwhile, Kubernetes, load balancers, and dashboards might all show green. ✅

This happens more often than people think.

🩺 Better Ways to Monitor gRPC

There are better ways to monitor your gRPC service.

gRPC Health Probe ✅

Use a real gRPC health check request against the listener.

This validates the actual serving path and confirms the service can respond over gRPC.

A strong default option.

Build a Status gRPC Service 📋

Expose an internal status method in your gRPC API.

This gives you flexibility to check deeper dependencies, such as database readiness, downstream systems, internal state, and maintenance toggles.

It’s more work, but more control.

Use a Single Shared Listener ☝️

Because gRPC runs on top of HTTP/2, many languages and frameworks can serve HTTP and gRPC traffic on the same listener.

That means an HTTP health endpoint may be acceptable because it checks the same network path. It still does not fully validate gRPC behavior, but it is better than checking an entirely separate listener.

🧠 Final Thoughts

gRPC is awesome.

But making a service production-ready means revisiting configurations inherited from REST services.

• Health checks
• Load balancing behavior
• Connection management
• Contracts
• Operational tooling

None of these changes are difficult. They’re just easy to miss.

Originally published at https://bencane.com on May 21, 2026.