Go has historically had a relatively straightforward but hands-on OpenTelemetry instrumentation model: add the OpenTelemetry SDK, initialize it, instrument the libraries you use, and create custom spans where application-specific context matters.

That approach still gives you the most control. But it is no longer the only practical option.

The manual instrumentation approach requires so much work from engineers that we decided to create a general AI Skill for instrumenting applications with OpenTelemetry SDK, regardless of the runtime/SDK.

Two newer approaches can instrument Go applications with little or no manual source-code instrumentation:

• OpenTelemetry eBPF Instrumentation (OBI) instruments applications at runtime.
• OpenTelemetry Go Compile-Time Instrumentation (otelc) injects instrumentation during the build.

Both are part of the OpenTelemetry ecosystem. Both can produce OpenTelemetry telemetry without requiring developers to manually wrap every HTTP handler, database call, gRPC client, or messaging operation.

Both can be described as “zero-code instrumentation”, but they solve the problem at completely different points in the software lifecycle.

OBI asks:

How can we observe this application without changing or rebuilding it?

Otelc asks:

How can we build this application with instrumentation already inside it without manually modifying its source code?

This article looks at both approaches from the perspective of a developer, SRE, DevOps engineer, or engineering manager who needs to make a practical decision about instrumenting a Go application.

The traditional way to instrument Go with OpenTelemetry

The conventional way to instrument a Go application is to explicitly add OpenTelemetry support to the application using the SDK for Go.

At a high level, that usually means:

1. Adding the OpenTelemetry Go API and SDK.
2. Configuring a TracerProvider.
3. Configuring exporters.
4. Instrumenting libraries such as HTTP servers and clients, gRPC, databases, and messaging clients.
5. Adding custom spans around important application operations.

A simplified example might look like this:

tracer := otel.Tracer("checkout")
ctx, span := tracer.Start(ctx, "reserve_inventory")
defer span.End()
if err := inventory.Reserve(ctx, order); err != nil {
  span.RecordError(err)
  return err
}

For a more complete example, see our Gin service instrumentation example: https://github.com/sematext/sematext-otel-onboarding/tree/main/go

This model has important advantages:

• You explicitly control where spans start and end.
• You can attach application-specific attributes.
• You can model important business operations.
• You can decide what should and should not become telemetry.

The downside is obvious: instrumentation becomes part of the application.

For a sufficiently large system, that can mean touching many services, maintaining instrumentation dependencies, reviewing instrumentation changes, and deciding how deeply each library and operation should be instrumented. This is the type of stuff we faced at Sematext when we said “OK, let’s go and instrument all our Go services now”.

A useful way to think about the landscape is:

OBI and Otelc occupy different places on that spectrum.

Note that manual instrumentation is far from obsolete, you and your team can still choose this approach. It’s just that the zero-code approaches provide another layer of options. As a matter of fact, as you read this article you will learn that manual instrumentation is still critical in certain situations and complements the auto-instrumentation approaches.

Why zero-code instrumentation is harder in Go

Automatic instrumentation is relatively familiar in languages with highly dynamic runtimes.

Java agents can modify or intercept bytecode. Python can wrap functions dynamically. Other runtimes provide mechanisms that make it possible to insert instrumentation after an application has been built.

But Go is different. A Go application is typically compiled into a native binary. Once the binary exists, there is no general-purpose equivalent of loading a Java agent and rewriting the application’s bytecode.

That creates a fundamental choice for automatic instrumentation:

Do you instrument the process from outside, or do you modify the application during compilation?

OBI takes the first approach and Otelc takes the second.

The architectures look like this.

OBI: runtime instrumentation

Otelc: compile-time instrumentation

This difference affects deployment, security, portability, ownership, and operational complexity.

Let’s look at each approach in detail.

OBI: Instrumenting a running Go process

OpenTelemetry eBPF Instrumentation (OBI) is the OpenTelemetry project’s eBPF-based automatic instrumentation technology.

It was originally based on technology from Grafana Beyla and is now developed as an OpenTelemetry project. OBI’s first OpenTelemetry release was announced in late 2025, and the project has continued to evolve since then.

We are big fans of eBPF at Sematext and have been using it for 10+ years now. When we first started instrumenting our Go applications we chose the OBI approach and included it in our OTel examples for instrumenting Go applications.

The central idea is simple:

Run instrumentation outside the application process and observe the application while it runs.

Unlike a traditional OpenTelemetry SDK integration, OBI does not require adding OpenTelemetry code to the application binary.

See also: OpenTelemetry eBPF Instrumentation documentation

How OBI works

OBI uses eBPF to observe application and system behavior. Depending on what is being instrumented, it can capture activity at protocol boundaries and use language-specific techniques, including user-space probes, or uprobes, for supported Go instrumentation.

A simplified flow looks like this:

         Client
           │
           │ HTTP request
           ▼
┌──────────────────────┐
│      Go service      │
│                      │
│   net/http handler   │◄───── OBI observes
│          │           │       supported operations
│          ▼           │
│     application      │
│       logic          │
│          │           │
│          ▼           │
│    database/sql      │◄───── OBI observes
│                      │
└──────────────────────┘
           │
           │ SQL
           ▼
       PostgreSQL
           │
           ▼
   ┌───────────────┐
   │      OBI      │
   └───────┬───────┘
           │
           ▼
    Trace + metrics

OBI can be deployed as:

• a standalone process,
• a Docker container,
• a Kubernetes sidecar,
• or a Kubernetes DaemonSet.

The exact deployment model depends on how broadly you want to instrument the environment.

What can OBI instrument in Go?

OBI supports a combination of protocol-level and Go library-level instrumentation.

The current OpenTelemetry documentation lists Go support for technologies including:

• net/http
• HTTP/2
• gorilla/mux
• Gin
• gRPC
• database/sql
• MySQL drivers
• PostgreSQL drivers
• Redis
• Kafka
• Sarama
• pgx

Compatibility depends on the specific library and version. For example, Go library-level instrumentation is documented for Go 1.17+, while some context propagation capabilities require Go 1.18+.

This is an important practical point. OBI is not simply “watching packets.”

For supported Go libraries, it can use Go-specific instrumentation to capture richer application behavior.

At the same time, it remains fundamentally an out-of-process instrumentation system.

What using OBI looks like

Imagine you have an existing Go service, say payment-api, and that it is already running in production.

As such, you may not want to:

• modify its source code,
• change its dependencies,
• rebuild it,
• or restart it purely to add instrumentation.

OBI can be deployed separately and configured to discover or target the application. This is particularly useful for existing workloads.

OBI’s runtime model is one of its strongest characteristics: it can observe applications without making OpenTelemetry instrumentation part of the application’s build artifact. The OpenTelemetry project describes OBI as out-of-process instrumentation that can provide telemetry without application code changes or application restarts in supported scenarios.

Advantages of OBI

1. No source-code changes

The obvious benefit is that developers do not need to edit the application.

There is no need to:

git clone --> add instrumentation --> test instrumentation --> commit --> build --> deploy

Instead, instrumentation can be introduced independently of the application code.

This is particularly attractive when:

• the application is maintained by another team,
• source code is unavailable,
• the application is legacy,
• you want to instrument an existing fleet,
• or the platform team owns observability deployment.

2. No application rebuild

OBI can instrument supported workloads without requiring you to produce a new binary. This is one of the clearest differences between OBI and Otelc.

Otelc, on the other hand, requires the ability to run the application through an instrumented build process.

3. Instrumentation can be centrally operated

OBI is well suited to an infrastructure-oriented operating model. For example, the platform team could choose to deploy OBI as a Kubernetes DaemonSet and have it handle the instrumentation of Go, Java, and Python applications. In other words, the application teams do not necessarily have to independently add and maintain instrumentation.

That can be valuable in organizations with many teams and inconsistent OpenTelemetry adoption.

4. It is not Go-specific

OBI can observe applications written in multiple languages. That means an SRE or platform engineering team can use a common instrumentation mechanism across a heterogeneous environment. For organizations operating Go, Java, Python, Node.js, NGINX, and other workloads, that can simplify initial telemetry coverage. This is one of the OBI aspects that we have benefited from at Sematext, as some of our legacy code still uses Java/Kotlin and Node.js.

The limitations and trade-offs of OBI

1. OBI depends on the runtime environment

OBI is fundamentally a Linux and eBPF-based technology.

That means your deployment environment must support the capabilities OBI needs.

The OpenTelemetry documentation describes OBI as a Linux process that can inspect other running processes and requires elevated privileges or the appropriate Linux capabilities, depending on deployment and configuration.

In containers and Kubernetes, this can become an architectural decision rather than a simple configuration change.

For example, some OBI deployments require or may use:

• privileged containers,
• CAP_SYS_ADMIN,
• CAP_PERFMON,
• host or shared process namespaces,
• access to /proc.

The exact requirements depend on the instrumentation and deployment model. Recent kernel security changes can also affect Go instrumentation because OBI uses uprobes for Go-specific instrumentation.

This does not mean OBI is inherently unsuitable for production.It means that the security model must be evaluated by the platform team.

Of course, if your applications don’t run on Linux then OBI is not an option for you at all until other platforms, like Windows, get the needed eBPF support. See https://github.com/microsoft/ebpf-for-windows for what Microsoft is doing about that for Windows.

2. Coverage depends on what OBI understands

OBI can automatically observe supported protocols and libraries, but it does not automatically understand arbitrary application code.

Consider:

func CalculateEnterpriseDiscount(
    customer Customer,
    contract Contract,
) (Discount, error) {
    // 500 lines of business logic
}

There is no general way for an external runtime observer to know that this function represents an important business operation. OBI can show what happens around it:

HTTP request
    │
    ├── database query
    ├── Redis lookup
    ├── gRPC call
    └── Kafka publish

But it cannot automatically know that CalculateEnterpriseDiscount is an important domain-level operation.

3. Automatic service names and routes may need tuning

Because OBI observes applications externally, automatically derived service names, routes, and URLs may not always match how your organization wants to identify services.

The OpenTelemetry documentation specifically calls out route configuration and decoration as something that should be reviewed when generating traces with OBI.

That means you should validate the resulting telemetry rather than assuming that automatic discovery will always produce exactly the naming and cardinality you want.

Otelc: Instrumenting Go during compilation

The second approach moves instrumentation from runtime to build time.

Otelc, the OpenTelemetry Go Compile-Time Instrumentation tool, modifies the Go build process so that supported instrumentation is injected while the application is being compiled.

The resulting application binary contains the instrumentation.

The normal Go build looks like this:

Source code ──► go build ──► Go binary

While with Otelc it looks like this:

Source code ──► otelc + go build ──► Instrumented Go binary

So if you choose Otelc you will not need to change the application source code, but you will need to alter the build process.

How Otelc works

According to the OpenTelemetry Go compile-time instrumentation documentation, Otelc:

1. Intercepts compilation using the Go toolchain’s -toolexec mechanism.
2. Matches packages and functions against instrumentation rules.
3. Injects lightweight hook points.
4. Links those hooks to OpenTelemetry instrumentation code.

The resulting binary contains the instrumentation, so there is no separate runtime instrumentation agent that needs to attach to the process. Operationally, this is simpler because there is no additional moving piece running in production.

Visually things look like this:

Go application (source) ──► Otelc (match rules, inject hooks, link OTel code) ──► Instrumented Go binary ──► OTLP

The Otelc project uses techniques including trampoline code injection and function hook mechanisms to connect instrumented functions with OpenTelemetry logic.

What does using Otelc look like?

A simple workflow can look like:

otelc go build -o myapp .

Alternatively, Otelc can be integrated with the standard Go toolchain using -toolexec.

A documented pattern is:

otelc setup
export GOFLAGS="${GOFLAGS} '-toolexec=otelc toolexec'"
go build -o myapp .

This can be useful when a build command is controlled by an existing Makefile, CI system, or another build tool.

Otelc can also be installed as a Go tool dependency in supported Go versions, allowing builds such as:

go tool otelc go build -o myapp .

Otelc also supports generating or maintaining instrumentation configuration based on the application’s dependency graph.

What can Otelc instrument?

The currently documented set of supported instrumentation includes:

• net/http
• gRPC
• database/sql
• Gin
• Redis
• MongoDB
• Kubernetes client-go
• OpenAI Go SDK
• Anthropic Go SDK
• Kafka
• AWS SDK for Go v2
• selected logging libraries for trace/span correlation

The supported set will continue to evolve, so it is worth checking the project’s current supported-library documentation before choosing it for a specific application.

Advantages of Otelc

1. No manual instrumentation changes

Developers do not have to manually add instrumentation to every supported library boundary.

The source can remain:

http.HandleFunc("/checkout", checkoutHandler)

rather than becoming:

handler := otelhttp.NewHandler(
    http.HandlerFunc(checkoutHandler),
    "checkout",
)
http.Handle("/checkout", handler)

The instrumentation is introduced during compilation instead.

2. No privileged runtime instrumentation process

Once the application has been built, there is no eBPF process that needs to attach to it.

This can make Otelc attractive in environments where:

• privileged containers are prohibited,
• eBPF is unavailable,
• security policy restricts process instrumentation,
• or platform teams do not want observability software attaching to production workloads.

The OpenTelemetry documentation explicitly identifies this as a use case for compile-time instrumentation.

3. Instrumentation can reach supported dependencies

Because Otelc participates in the compilation process, it can instrument supported third-party dependencies that are part of the application’s build. That is useful when you use a library that you do not own but still want to instrument.

For example:

Your application
       │
       ├── Gin
       ├── gRPC
       ├── database/sql
       ├── Redis
       └── AWS SDK

Otelc can apply instrumentation rules to supported parts of that dependency graph without requiring you to fork or edit those dependencies.

4. The build artifact contains the instrumentation

This changes who owns the operational problem.

With OBI: application deployment + runtime instrumentation deployment

With Otelc: only the instrumented application artifact

The instrumentation becomes part of the software artifact produced by the build. That can fit naturally into organizations where application teams already own their build and deployment pipelines.

The limitations and trade-offs of Otelc

Nothing in this world seems to come without downsides… let’s look at Otelc’s cons.

1. You must control the build

Otelc requires access to the build process because that is where instrumentation is introduced. This is the most important limitation. If you have a precompiled production binary but cannot rebuild it, Otelc is simply not an option.

2. The build pipeline becomes part of the instrumentation architecture

Adding Otelc is not the same as adding another environment variable.

You now need to think about:

• local developer builds,
• CI builds,
• release builds,
• test builds,
• reproducibility,
• dependency management,
• cross-compilation,
• monorepos,
• and build caching.

The good news is that Otelc is designed to work with the normal Go build workflow and documents approaches for integrating through go tool, direct build wrapping, and -toolexec.

But this should still be treated as a build-system change and tested accordingly.

3. Coverage is limited to available instrumentation

Like OBI, Otelc does not automatically understand every Go package. It needs instrumentation rules for the libraries and functions you want to observe.

If you use some framework that is not yet supported by the instrumentation, Otelc will not magically infer its semantics.

You may need to:

• add manual instrumentation,
• create instrumentation for that library,
• or accept that the library is not automatically traced.

Luckily, the Otelc project includes an instrumentation model and documentation for adding support for additional libraries, so this scenario can be handled.

4. Automatic instrumentation is still not the same as manual application instrumentation

Otelc can add spans around supported framework and library operations. It does not automatically know which parts of your business logic are important.

For example:

POST /checkout
       │
       ▼
ValidateOrder
       │
       ├── ReserveInventory
       │
       ├── CalculateDiscount
       │
       ├── ProcessPayment
       │
       └── CreateShipment

Automatic instrumentation may produce excellent visibility into:

HTTP server span
       │
       ├── SQL query
       ├── Redis operation
       ├── HTTP call to payment provider
       └── Kafka publish

But it may not tell you how much time was spent specifically in CalculateDiscount unless you explicitly instrument that operation.

OBI vs. Otelc: side-by-side comparison

The following table summarizes the practical differences.

OBI vs. Otelc: how to choose which one to use

A simple decision matrix

A quick way to determine if you should be considering OBI or Otelc is by considering the following scenarios and asking a few questions.

If your starting point is:

“I have a binary already running and I don’t want to rebuild it.”

Start with OBI.

Existing binary?
      │
      ├── Yes ──► OBI is the practical zero-code option
      │
      └── No

“I control the build but don’t want to modify the application source.”

Look at Otelc.

Control the Go build?
      │
      ├── Yes ──► Otelc is a strong candidate
      │
      └── No ──► Consider OBI

“I cannot run privileged instrumentation in production.”

Look at Otelc.

eBPF / runtime privileges allowed?
      │
      ├── No ──► Otelc
      │
      └── Yes ──► OBI or Otelc

“I need to instrument services in several languages.”

OBI may provide a better common platform-level approach.

Go + Java + Python + Node.js
             │
             ▼
            OBI

“I need to instrument important internal business operations.”

Neither automatic approach completely solves the problem. You will probably still want manual instrumentation. Ooops! ;)

A practical decision tree

Here is another approach, a decision tree, that will help you quickly see which approach is more suitable.

If you are deciding how to instrument a Go application, start with this:

                      ┌──────────────────────┐
                      │ Need OpenTelemetry?  │
                      └──────────┬───────────┘
                                 │
                                 ▼
                  ┌────────────────────────────────┐
                  │ Can you modify the application │
                  │ source code?                   │
                  └──────────┬─────────────────────┘
                             │
                ┌────────────┴────────────┐
                │                         │
               Yes                        No
                │                         │
                ▼                         ▼
        Manual instrumentation     Can you rebuild
        is available                the application?
                                      │
                           ┌──────────┴──────────┐
                           │                     │
                          Yes                    No
                           │                     │
                           ▼                     ▼
                     Consider Otelc        Consider OBI
                           │                     │
                           ▼                     ▼
                    Do you need             Does your runtime
                    domain-level spans?     support OBI/eBPF?
                           │                     │
                           ▼                     ▼
                  Add manual spans        Deploy and validate
                  where they matter       supported coverage

In practice, the decision often reduces to four questions.

Question 1: Do I control the build?

If yes, Otelc becomes an option.

If no, it does not.

Question 2: Can I run eBPF instrumentation in production?

If yes, OBI becomes an option.

If no, Otelc may be easier operationally.

Question 3: Do I need to instrument existing binaries?

If yes, OBI is the more natural fit.

Question 4: How much application-specific context do I need?

If the answer is “a lot,” neither zero-code approach is likely to be sufficient by itself. Plan for some manual instrumentation.

Why automatic instrumentation does not eliminate the need for manual instrumentation

This is perhaps the most important point in this entire article. It is tempting to think of automatic instrumentation as a replacement for manual instrumentation. In our experience, it usually is not.

Automatic instrumentation and manual instrumentation solve different problems.

Automatic instrumentation is excellent at finding and instrumenting common technical boundaries:

• HTTP requests,
• gRPC calls,
• database queries,
• Redis operations,
• messaging operations,
• cloud SDK calls,
• and other supported libraries.

Manual instrumentation is where you describe what your application actually does.

Consider a checkout service.

Automatic instrumentation may give you:

POST /checkout                           820 ms
│
├── SELECT customer                      12 ms
├── SELECT inventory                     18 ms
├── Redis GET                             3 ms
├── POST payment-provider               410 ms
└── Kafka publish                         8 ms

This is indeed already extremely useful and you should aim for this as your first step.

But your engineering team may care about something different:

Checkout
│
├── ValidateOrder
├── ReserveInventory
├── CalculateDiscount
├── ProcessPayment
└── CreateShipment

Those are domain operations. Neither OBI or Otelc can reliably infer that these operations are important simply by observing technical behavior.

The most effective approach is often a hybrid.

                HTTP request
                       │
                       ▼
        ┌──────────────────────────┐
        │ Automatic instrumentation│
        └────────────┬─────────────┘
                     │
              Application code
                     │
                     ▼
        ┌─────────────────────────┐
        │   Manual business spans │
        │                         │
        │ ReserveInventory        │
        │ CalculateDiscount       │
        │ ProcessPayment          │
        └────────────┬────────────┘
                     │
                     ▼
        ┌──────────────────────────┐
        │ Automatic instrumentation│
        │ SQL / Redis / gRPC / etc │
        └──────────────────────────┘

For OBI specifically, OpenTelemetry also provides the Go Instrumentation Auto SDK, which is intended to help integrate manually created spans with eBPF-generated spans and shared trace context.

That makes the hybrid model especially relevant:

This avoids two extremes:

Extreme 1: Instrument nothing automatically

Every team has to manually instrument every HTTP framework, database driver, messaging library, and client.

Extreme 2: Instrument everything automatically and assume the result is sufficient

You get infrastructure-level telemetry but may lack the domain context required to answer questions such as:

• Why is checkout slow?
• Which business operation failed?
• How much time is spent calculating pricing?
• Which customer workflow is affected?
• Did the payment provider fail, or did our own validation logic reject the request?

The best observability, as I hope I’ve illustrated so far in this article, usually combines the two approaches.

Conclusion

OBI and Otelc represent two fundamentally different approaches to zero-code OpenTelemetry instrumentation for Go.

OBI instruments from the outside.

It is attractive when you want to observe applications that already exist, especially when rebuilding or modifying them is difficult. It can be deployed independently of the application and can support a centralized, platform-owned instrumentation model.

Otelc instruments from the inside—during the build.

It is attractive when you control the Go build process and want supported OpenTelemetry instrumentation to become part of the resulting binary without manually modifying application source code. It also avoids the need for a privileged runtime eBPF instrumentation process.

Neither approach eliminates the value of manual instrumentation.

A practical architecture for many teams will look like this:

┌─────────────────────────────────────────────┐
│           Automatic instrumentation         │
│                                             │
│   OBI or Otelc                              │
│                                             │
│   HTTP • gRPC • SQL • Redis • Kafka • etc.  │
└───────────────────────┬─────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────┐
│           Manual instrumentation            │
│                                             │
│   Business operations                       │
│   Domain-specific spans                     │
│   Important attributes                      │
│   High-value application context            │
└───────────────────────┬─────────────────────┘
                        │
                        ▼
              OpenTelemetry backend

The practical goal should not necessarily be to choose one instrumentation method and use it everywhere. Instead, choose the method that best fits the part of the system you are trying to observe.

• Need visibility into existing workloads without rebuilding them? Start with OBI.
• Control the Go build and want instrumentation baked into the binary? Look at Otelc.
• Need detailed visibility into business operations? Add manual instrumentation.
• Need all three? A hybrid approach may be the most useful architecture.

For implementation details and current compatibility information, start with the authoritative project documentation:

• OpenTelemetry eBPF Instrumentation (OBI)
• OBI Go instrumentation compatibility and supported libraries
• OpenTelemetry Go compile-time instrumentation (Otelc)
• Otelc getting started guide and supported libraries
• OpenTelemetry Go Instrumentation Auto SDK

Start Free Trial

The post Zero-Code OpenTelemetry for Go: Runtime Instrumentation with OBI vs. Compile-Time Instrumentation with Otelc appeared first on Sematext.

Key Functionality to Compare

At the basic level, an SLO defines the reliability target for a service—say, 99.9% successful requests over 30 days. SLO monitoring turns that target into something actionable by tracking the underlying indicator, calculating error-budget consumption, and warning me when the budget is being burned too quickly.

The core functionality I look for is:

• Flexible SLIs based on availability, latency, errors, or custom metrics.
• Rolling and calendar-based time windows.
• Error budgets, including remaining budget and burn rate.
• Burn-rate alerts, ideally with multi-window alerting.
• Good integration with the telemetry I already collect—metrics, traces, logs, or synthetic checks.
• Enough flexibility to model real services, without forcing every SLO into a simplistic uptime template.

The rest of this comparison focuses on the practical trade-offs: how each tool models SLOs, what data it can use, how usable the UI is, and how much complexity I have to accept to get reliable alerting. Note that service-level objectives are one of those things that sound straightforward until you try to operationalize them, so in addition to reviewing SLO monitoring tools it’s important to spend some time thinking about the SLOs themselves.

Quick comparison

I reviewed 9 tools. Most offer SLO as part of their wider observability platform, while one is a pure synthetic monitoring and another a purely SLO-focused tool. Here is a very quick high level comparison. There are several additional tables below that I suggest you look through.

Best SLO Monitoring Tools: A Practical Comparison

SLO monitoring has become one of the more useful ways to answer a deceptively simple question: is my service actually reliable enough?

I don’t mean “are all the dashboards green?” or “did CPU stay below 80%?” I mean whether users are getting the level of availability and performance that the service is supposed to provide.

A Service Level Indicator (SLI) is the measurement. A Service Level Objective (SLO) is the target for that measurement over a period of time. If my API successfully serves 99.9% of requests over 30 days, the SLO might be 99.9% availability. The remaining 0.1% is the error budget—the amount of unreliability I can afford before missing the objective.

SLO monitoring matters because raw monitoring data does not tell me how much a problem matters. A service can generate thousands of errors and still be within its reliability objective. Conversely, a relatively short outage can consume a huge chunk of the error budget and require immediate attention.

The core capabilities I look for are:

• Flexible ways to define SLIs for availability, latency, errors, throughput, or custom business signals.
• Rolling and calendar-based evaluation windows.
• Accurate error-budget calculations.
• Burn-rate monitoring and alerting.
• Support for grouping, filtering, and managing large numbers of SLOs.
• Good integration with the telemetry I already collect.
• Automation through APIs, Terraform, or configuration-as-code.
• A practical path from an SLO violation to the logs, metrics, traces, or synthetic checks that explain it.

The tools below take noticeably different approaches. Some treat SLOs as one capability inside a larger observability platform. Others, especially Nobl9, treat SLO management as the product itself.

Tool Comparison

1. Sematext

Sematext takes a relatively straightforward approach to SLO monitoring. Its current SLO functionality is built around synthetic monitoring, allowing me to create objectives from existing HTTP and browser monitors rather than requiring new instrumentation or a separate telemetry pipeline. That makes it particularly suitable for services where the thing I actually want to measure is externally observable behavior: can the API be reached, does the website respond quickly enough, and can a browser complete a critical user journey?

The implementation tracks compliance over configurable windows and provides error-budget visibility and early warnings. Sematext also connects the reliability view with the rest of its monitoring stack, including logs, infrastructure monitoring, and tracing. The main trade-off is that this is not trying to be a universal, vendor-neutral SLO layer that can model every imaginable SLI from every backend. It is simpler and more opinionated.

Key features

• SLOs built from HTTP and browser synthetic monitors.
• Availability and performance-oriented objectives.
• Configurable compliance targets and time windows.
• Remaining error-budget tracking.
• Historical and live compliance views.
• Early warning and alerting before an objective is violated.
• Synthetic checks from multiple locations.
• Monitoring of APIs, websites, user journeys, and third-party dependencies.
• Correlation with other Sematext telemetry.

Pros

• Very little setup if synthetic monitors already exist.
• Good fit for externally visible services and user-facing reliability.
• Easy to understand without becoming an SRE research project.
• Useful for monitoring third-party APIs against expected reliability.
• Integrates with logs, metrics, tracing, and other Sematext capabilities.

Cons

• Less flexible than tools that can define SLOs directly from arbitrary metrics or multiple external data sources.
• Not as strong for organizations that want every SLO managed through Git and a formal SLO-as-code workflow.
• Synthetic-monitor-based SLOs are not a replacement for every internal service-level indicator.

Pricing

Sematext currently offers a 14-day free trial. Synthetic Monitoring starts at $2 per monitor per month on monthly pricing, or $1.80 per monitor per month with annual pricing. SLO functionality is associated with the synthetic monitoring workflow, so the practical cost depends primarily on the monitors required to measure the service.

My opinion

What I like is the lack of ceremony. If I already have HTTP or browser monitors, turning those measurements into an SLO is a pretty natural next step. I also like the fact that an external SLO can represent what users actually experience rather than what an internal metric claims is happening.

What I dislike is the relative lack of flexibility compared with something like Nobl9 or a metric-centric implementation. If I want to define complex internal SLIs across arbitrary telemetry sources, this would not be my first choice.

2. Datadog

Datadog has one of the more mature general-purpose SLO implementations. It supports metric-based, monitor-based, and time-slice SLOs, which gives me several different ways to model reliability. Metric-based SLOs work well when I can clearly define good and bad events. Monitor-based SLOs build on existing monitors, synthetic checks, or service checks. Time-slice SLOs are useful when reliability is defined as a metric satisfying a condition during discrete periods of time.

Datadog also provides error-budget and burn-rate alerting, SLO tagging, search, historical views, APIs, and Terraform support. If my infrastructure, APM, logs, and synthetics are already in Datadog, this is an obvious place to keep SLOs because the data is already there. The downside is the same one that applies to Datadog generally: the platform is broad, powerful, and easy to expand inside, but the pricing model can become difficult to reason about as more products and telemetry are added.

Key features

• Metric-based SLOs.
• Monitor-based SLOs.
• Time-slice SLOs.
• Availability, latency, and custom metric use cases.
• Error-budget tracking.
• Burn-rate indicators and alerts.
• Rolling windows.
• Grouped SLOs and tags.
• SLO search and management views.
• API and Terraform support.
• Integration with APM, logs, RUM, synthetics, and infrastructure metrics.

Pros

• Very flexible SLI modeling.
• Excellent fit if Datadog is already the telemetry platform.
• Good support for both event-based and time-based reliability measurements.
• Mature burn-rate alerting.
• Strong automation support.
• Easy to move from an SLO problem into the rest of the observability stack.

Cons

• Pricing can be complicated because SLOs are only one part of a larger set of billable products.
• The number of possible ways to model an SLO can be confusing for teams new to SRE practices.
• Monitor-based SLOs can introduce dependencies on the underlying monitor configuration.
• The best experience assumes the relevant telemetry already lives in Datadog.

Pricing

Datadog’s pricing is modular and depends on the products used to collect and evaluate the underlying telemetry. Host-based products, including infrastructure monitoring and some APM offerings, use different billing models, while other capabilities are billed according to consumption. Datadog provides public list pricing, but in practice I would model the cost based on the complete telemetry architecture rather than looking for a standalone “SLO price.”

My opinion

What I like most is flexibility. Datadog can handle the classic “good events divided by total events” model, monitor-based availability, and time-slice definitions without forcing me into a single interpretation of an SLI.

What I dislike is that I would hesitate to introduce Datadog purely for SLO monitoring. It makes the most sense when the organization is already committed to the platform. Otherwise, I am paying for and operating inside a much larger system than the specific SLO problem requires.

3. New Relic

New Relic’s Service Level Management is tightly integrated into the rest of the New Relic platform. I can create service levels ranging from relatively simple one-click configurations to more advanced and customizable definitions, then view them alongside applications, workloads, and other observability data. The product also provides alerts and analysis views for tracking reliability over time and investigating breaches.

The advantage is that SLOs are not isolated objects sitting in a separate reliability tool. They are part of the APM and observability workflow. That can be useful when the next step after discovering that an error budget is being consumed is immediately opening the affected service, transaction, trace, or other telemetry.

New Relic’s pricing is also different from the traditional per-host approach. The current public model combines data ingest with user or compute-based access options. That can work well, but I would spend time modeling access requirements before committing.

Key features

• Guided and advanced service-level creation.
• SLI and SLO definitions integrated with New Relic services.
• Reliability views across Navigator and Workloads.
• Alerting for degradation and breaches.
• Period-over-period analysis.
• Investigation workflows around SLO breaches.
• Integration with APM, infrastructure, logs, synthetics, and other New Relic capabilities.
• Support for both simple and more customizable SLO definitions.

Pros

• Strong integration with the rest of the observability platform.
• Relatively approachable SLO creation.
• Good fit for teams already using New Relic APM.
• Useful free tier for evaluation and smaller deployments.
• No need to count hosts as a primary pricing dimension.

Cons

• Pricing can become more complicated once data volume and platform-user requirements grow.
• The experience is best when New Relic is already the primary observability platform.
• Some organizations may find the user-access model less attractive than purely telemetry-based pricing.
• Less attractive as a standalone, vendor-neutral SLO layer.

Pricing

New Relic’s free tier includes 100 GB of data ingest per month, unlimited basic users, and one free full-platform user. Beyond that, public pricing lists original data ingest at $0.40/GB beyond the included allowance. User pricing varies by edition, while New Relic also offers a compute-based model for eligible customers.

My opinion

I like New Relic’s integrated approach. If I am already using New Relic for APM, I don’t want to export the same telemetry somewhere else just to calculate an error budget. Keeping the SLO next to the application data is operationally sensible.

What I dislike is the pricing complexity around data, user types, editions, and newer compute models. It is more transparent than some historical observability pricing models, but I would still build a realistic cost model before assuming the free tier or entry pricing reflects production costs.

4. Grafana Cloud

Grafana Cloud is particularly interesting to me because it fits naturally into Prometheus and OpenTelemetry-oriented environments. Grafana SLO provides a dedicated workflow for creating and managing SLOs, generating dashboards and alerts, tracking error budgets, and automating configuration through APIs and Terraform.

The biggest advantage is architectural familiarity. If my engineering organization already thinks in PromQL, metrics, recording rules, infrastructure-as-code, and Git-based workflows, Grafana SLO feels like an extension of the existing stack rather than a new conceptual layer. The product can generate supporting dashboards, recording rules, and alerting components instead of requiring me to hand-build everything.

The limitation is that Grafana SLO is a Grafana Cloud capability rather than a general feature of self-hosted open-source Grafana. That distinction matters if my primary reason for choosing Grafana is self-hosting and avoiding a managed SaaS dependency.

Key features

• Guided SLO creation.
• Metric-based SLIs.
• Error-budget tracking.
• SLO dashboards.
• Error-budget alerts.
• Generated recording rules.
• Generated alerting rules.
• API support.
• Terraform support.
• SLO-as-code workflows.
• Integration with the broader Grafana observability stack.

Pros

• Excellent fit for Prometheus-centric teams.
• Strong infrastructure-as-code story.
• Familiar workflow for teams already using Grafana.
• Good separation between raw telemetry and reliability objectives.
• Free tier and relatively accessible entry pricing.
• Less architectural lock-in than some all-in-one observability platforms.

Cons

• The managed SLO capability is specifically part of Grafana Cloud.
• Metric modeling still requires Prometheus/Grafana expertise.
• Usage-based pricing can become harder to predict as metric cardinality grows.
• The SLO implementation is less useful if my relevant data is not accessible through the supported Grafana Cloud metric workflow.

Pricing

Grafana Cloud has a free tier and a Pro plan starting at $19 per month plus usage. The pricing page currently includes 10,000 active metric series in the platform fee, with additional metrics starting at $6.50 per 1,000 series before volume discounts. Enterprise starts with a $25,000 annual spend commitment.

My opinion

I like Grafana Cloud when the organization already has Prometheus expertise. The SLO-as-code and Terraform story is especially important for teams that do not want critical reliability definitions to exist only as manually configured UI objects.

What I dislike is that I still need to understand the underlying metric model. That is not necessarily a flaw—SLOs should be based on carefully chosen indicators—but it means Grafana is not always the easiest tool for a team that wants a highly opinionated, guided reliability workflow.

5. Dynatrace

Dynatrace has a fairly powerful SLO implementation, especially for organizations already using its broader observability platform. I can create SLOs from predefined templates or define custom SLIs using DQL, which means the underlying indicator does not have to be limited to standard availability or latency metrics. Dynatrace can use data available through Grail, including metrics and other data types that can be queried into a time series. The current SLO experience includes error-budget tracking, visualization, and management through a dedicated application, while APIs and SDKs provide automation options. Dynatrace is particularly interesting for complex environments because its topology and entity model can provide useful context around the services being measured. The downside is that the platform is large and opinionated, so using it just for SLO monitoring would usually be excessive.

Key features

• Template-based SLO creation.
• Custom SLI definitions using DQL.
• Service availability and performance objectives.
• Infrastructure and Kubernetes-oriented templates.
• Error-budget tracking.
• SLO visualization and dashboard integration.
• Entity-aware observability context.
• API and SDK support.
• Access to multiple Grail data types for SLI definitions.

Pros

• Very flexible SLI definitions.
• Strong fit for large and complex environments.
• Good topology and entity context.
• Templates make common SLOs easier to create.
• Custom DQL opens up nontraditional SLI use cases.
• Good API and SDK support.

Cons

• A large platform if SLO monitoring is the only requirement.
• DQL adds another query language to learn.
• Pricing is not simple to evaluate from the SLO feature alone.
• The platform can feel more opinionated than Prometheus-centric alternatives.

Pricing

Dynatrace pricing is based on the broader platform and the products or capabilities being used rather than a simple standalone SLO price. I would treat SLO cost as part of the overall observability architecture and request a realistic quote based on data, monitoring scope, and the Dynatrace platform capabilities required.

My opinion

What I like is the flexibility. Being able to build an SLI from a custom DQL query means I am not limited to a small set of predefined reliability models. I also like the surrounding context: if an SLO is degrading, topology and entity information can help connect that reliability problem to the actual system.

What I dislike is the weight of the platform. If my team just wants straightforward SLOs on Prometheus metrics, Dynatrace would probably feel like bringing an entire observability platform to solve a narrower problem.

6. Elastic Observability

Elastic has one of the more flexible SLO implementations among full-stack observability platforms. I can create SLIs from APM availability or latency, synthetic availability, custom metrics, histogram metrics, timeslice metrics, or custom KQL queries against data in Elasticsearch. That is a much broader set of options than tools that restrict SLOs to predefined monitor types.

Elastic supports both rolling and calendar-aligned windows, occurrences- and timeslice-based budgeting, error budgets, and burn-rate alerting. The SLO overview also makes it possible to see historical SLI performance and budget consumption without manually assembling dashboards.

I think Elastic is particularly interesting for teams that already use Elasticsearch for logs or observability data and want to define SLOs from that data without exporting it elsewhere.

Key features

• APM availability and latency SLIs.
• Synthetic availability SLIs.
• Custom KQL-based SLIs.
• Custom metric and histogram metric SLIs.
• Timeslice and occurrences budgeting.
• Rolling and calendar-aligned windows.
• Error-budget tracking.
• Burn-rate alerts.
• Historical SLI and error-budget views.
• Dashboard integration.
• OpenTelemetry and Prometheus support across the broader observability platform.

Pros

• One of the most flexible sets of SLI types in this comparison.
• Can build SLOs from logs, metrics, APM, or synthetic data.
• Supports both rolling and calendar windows.
• Good support for error budgets and burn rates.
• Strong fit for teams already using Elastic.
• Useful combination of internal observability and external digital-experience monitoring.

Cons

• Requires learning Elastic’s data model and query language if I want to use the most flexible SLI types.
• Not a lightweight standalone SLO tool.
• SLO availability depends on the appropriate Elastic deployment and licensing.
• Self-managed deployments can require more operational work than SaaS-only alternatives.

Pricing

Elastic’s current serverless Observability Complete tier includes SLO functionality. Serverless pricing is usage-based: Elastic currently lists metrics ingest from $0.023/GB, other observability data from $0.09/GB, plus separate retention and egress charges. Synthetic monitoring is available as an add-on. I would calculate the real cost based on total telemetry volume rather than thinking of SLOs as a separately priced feature.

My opinion

What I like most is flexibility. Elastic lets me define an SLO from the data I actually have instead of forcing me to translate everything into one specific metric format. I also like the combination of APM, logs, synthetics, and SLOs in the same platform.

What I dislike is the complexity. Elastic can do a lot, but that also means I need to understand how my data is structured before I can take full advantage of the SLO functionality.

7. Honeycomb

Honeycomb takes a more engineering-centric approach to observability than traditional metrics-first platforms, and its SLO implementation fits that model. SLOs are built from events, which makes them a natural fit for tracing and high-cardinality telemetry. I can define what successful behavior looks like and then track the remaining error budget over the selected time period.

The alerting model is particularly interesting. Honeycomb supports both Exhaustion Time alerts, which estimate when the error budget will run out, and Budget Rate alerts, which trigger when the budget is being consumed faster than expected. The UI also provides a budget-burndown graph that helps tune alerts before blindly picking a burn-rate threshold.

For teams already using OpenTelemetry and distributed tracing heavily, Honeycomb is one of the more natural SLO implementations I would evaluate.

Key features

• Event-based SLOs.
• Error-budget tracking.
• Budget burndown visualization.
• Historical burn-rate analysis.
• Exhaustion Time burn alerts.
• Budget Rate burn alerts.
• Slack and PagerDuty notification support.
• Distributed tracing and OpenTelemetry integration.
• High-cardinality telemetry support.
• Service-level SLO workflows.

Pros

• Very natural fit for tracing and event-based telemetry.
• Excellent burn-alert model.
• Budget-burndown visualization helps tune alerts.
• Strong OpenTelemetry support.
• Particularly well suited to modern distributed systems.
• More engineering-focused than dashboard-heavy.

Cons

• Less attractive if the organization primarily thinks in Prometheus metrics.
• SLOs are not available on the free plan.
• The Pro plan includes only two SLOs.
• Teams unfamiliar with event-based observability may need to adjust their mental model.

Pricing

Honeycomb has a free plan with up to 20 million events per month and 100 million metric data points per month. The Pro plan starts at $150 per month and includes 2 SLOs. Enterprise pricing is custom and starts with significantly larger trigger and SLO allowances.

My opinion

I like Honeycomb because the SLO implementation feels connected to how engineers actually investigate modern distributed systems. If the SLO burns, I can work from the relevant events and traces rather than switching into an entirely separate monitoring model.

What I dislike is the relatively limited number of SLOs in the Pro plan. Two SLOs is enough for evaluation or a small service footprint, but it is restrictive if I want to make SLOs a standard part of every production service.

8. Checkly

Checkly represents a different approach from the large observability platforms. It is primarily a synthetic monitoring platform designed around developers and monitoring-as-code. I can define API checks, browser checks, Playwright test suites, uptime monitors, and multistep checks, then manage them through the Checkly CLI, Terraform, or Pulumi.

For SLO-style monitoring, this is useful when the thing I care about is externally observable behavior. Instead of defining availability from an internal metric, I can measure whether an API responds correctly or whether a user can actually complete a critical workflow.

I would not put Checkly in the same category as Nobl9 or Elastic when it comes to arbitrary SLI definitions. Its strength is narrower but useful: taking synthetic tests and treating them as production reliability signals that can be managed like code.

Key features

• HTTP, TCP, DNS, ICMP, and heartbeat monitoring.
• API and multistep checks.
• Browser checks using Playwright.
• Playwright Check Suites.
• Global and private monitoring locations.
• Automatic retries.
• Monitoring-as-code workflows.
• Checkly CLI.
• Terraform provider.
• Pulumi provider.
• Prometheus metrics export.
• Status pages and alerting integrations.

Pros

• Excellent developer experience.
• Strong monitoring-as-code support.
• Native Playwright integration.
• Useful for testing critical user journeys.
• Terraform, Pulumi, and CLI support.
• Good fit for CI/CD and production monitoring workflows.

Cons

• Less flexible for arbitrary internal metric-based SLIs.
• Synthetic checks can become expensive at high frequency and across many locations.
• Not a full observability platform.
• Best suited to externally observable services and workflows.

Pricing

Checkly has a free Hobby plan with 10 uptime monitors, 10,000 API check runs, and 1,000 browser check runs per month. The Starter plan starts at $24 per month, while Team starts at $64 per month. Enterprise pricing is custom. Pricing scales through monitor counts and synthetic check runs.

My opinion

What I like is the developer workflow. I would rather keep monitoring definitions in Git and deploy them with the rest of the application infrastructure than manually create hundreds of checks in a UI.

What I dislike is that it is easy to confuse “synthetic monitoring” with complete SLO management. Checkly is excellent when my SLI is based on externally observable behavior, but I would use another tool if I needed to define reliability objectives from arbitrary application metrics.

9. Nobl9

Nobl9 is the most specialized SLO product in this comparison. Rather than assuming that SLOs should live inside one observability backend, it acts as a dedicated reliability layer that can connect to multiple existing telemetry systems. Its platform includes error-budget alerting, composite SLOs, backtesting, service-health views, reporting, and a strong SLO-as-code workflow.

That vendor-neutral approach is its biggest differentiator. A large organization may have Datadog in one team, Prometheus in another, New Relic somewhere else, and cloud-native telemetry in yet another environment. Nobl9 is designed to put SLO definitions above those individual systems rather than requiring telemetry consolidation first.

It also has one of the strongest configuration-as-code stories through OpenSLO, YAML-based definitions, Git workflows, validation tooling, and automation. The trade-off is additional platform complexity and a separate product to operate.

Key features

• Vendor-neutral SLO management.
• Multiple telemetry integrations.
• Error-budget tracking and alerting.
• Composite SLOs.
• SLO backtesting.
• Service Health Dashboard.
• SLO annotations.
• Reporting.
• OpenSLO support.
• YAML and Git-based workflows.
• OpenSLO validation through the Oslo CLI.
• SLO-as-code automation.

Pros

• Purpose-built for SLO management.
• Works across heterogeneous observability environments.
• Excellent SLO-as-code support.
• Strong OpenSLO ecosystem involvement.
• Useful for large organizations standardizing reliability practices.
• Advanced capabilities such as composite SLOs and backtesting.

Cons

• Another platform to buy, integrate, and maintain.
• Probably excessive for a small team with a handful of services.
• Less compelling if all telemetry already lives comfortably in one observability platform.
• Public pricing is not as simple as self-service competitors.

Pricing

Nobl9 provides pricing through its sales process rather than publishing a simple per-monitor or per-host price. The pricing offering is aimed at selecting an option based on organizational requirements, so I would expect to request a quote for a real deployment.

My opinion

Nobl9 is the one I would look at if SLOs themselves are becoming a platform concern. If multiple teams use different monitoring systems and I need a consistent reliability model across all of them, a dedicated abstraction layer makes sense.

What I dislike is the obvious trade-off: if I only have 10 services and everything already lives in Grafana, Datadog, or New Relic, adding another product may solve a problem I don’t actually have.

Feature comparison

Here is a product-feature matrix for all SLO monitoring tools we are comparing here.

* Checkly is the outlier in this table. I would describe it as a synthetic monitoring and monitoring-as-code tool rather than a full native SLO management platform. It can measure availability and latency and can be used to implement reliability targets from checks, but it does not provide the same general-purpose SLI, error-budget, and multi-window burn-rate model as Datadog, Grafana Cloud, Dynatrace, Elastic, Honeycomb, or Nobl9.

This distinction is useful because it prevents the comparison from making Checkly look weaker at something it is not primarily designed to do. Its real strength is defining production checks as code and using them to monitor APIs and critical user journeys.

Which SLO monitoring tool would I choose?

My choice would depend less on the number of features in the product and more on where my telemetry already lives and what I am actually trying to measure.

If I had to narrow the list down based on the problem rather than the vendor:

The important distinction for me is that Checkly and Sematext are strongest when the SLI represents externally observable behavior, while Datadog, New Relic, Grafana Cloud, Dynatrace, Elastic, and Honeycomb can build SLOs from deeper application telemetry. Nobl9 is different again: its main value is separating the SLO/reliability layer from the underlying observability system.

So I would not pick a winner based on a feature checklist. I would first decide where the SLI should come from, and then choose the tool that makes that workflow the least painful.

Final thoughts

The most important thing I have learned about SLO monitoring is that the tool is rarely the hard part. The difficult part is choosing an SLI that actually represents user experience and setting an objective that is neither meaningless nor impossible.

A monitoring tool can calculate an error budget perfectly and still give me the wrong answer if I am measuring the wrong thing.

So before comparing vendors too deeply, I would start with a few concrete questions:

1. What user behavior am I trying to protect?
2. What counts as a good event and a bad event?
3. Should every request count equally, or is time-based availability more appropriate?
4. How much unreliability can the business actually tolerate?
5. What should happen when the error budget is being consumed too quickly?
6. Where will engineers go next to investigate the problem?

Once those answers are clear, the choice usually becomes much easier. For most teams, I would strongly prefer using the SLO capability already available in the observability platform they trust—unless they have a real need for vendor-neutral, organization-wide SLO management.

Start Free Trial

The post Best 9 SLO Monitoring Tools in 2026: Review and Comparison Tables appeared first on Sematext.

That shift is why network monitoring in 2026 looks very different from a decade ago. Traditional SNMP-based tools still have a place, but cloud-native teams need service-aware visibility, automatic topology discovery, and tight correlation with logs, metrics, and traces. The right tool can cut hours off incident response, surface cost leaks, and replace the architecture diagrams that are always getting outdated that nobody wants to maintain.

This guide compares the 12 best network monitoring tools available in 2026. For each one, you will find the features that matter, honest pros and cons, real pricing, and the kind of team it fits best.

What is Network Monitoring?

Network monitoring is the continuous practice of collecting data about network devices, traffic, and connections to detect performance issues, outages, and security events before they affect users. The data can come from many sources: SNMP polling for device health, NetFlow or sFlow for traffic analysis, packet captures for deep inspection, eBPF for kernel-level connection visibility, and synthetic probes for path validation.

The category has expanded beyond LAN and WAN. In 2026, network monitoring also covers:

• Pod-to-pod and node-to-node traffic inside Kubernetes clusters
• East-west traffic between microservices
• Cross-region and cross-cloud connectivity costs
• Connections to managed databases, message queues, and SaaS APIs
• Suspicious outbound traffic that might signal a compromise

Good network monitoring helps teams answer questions like: which services are talking to which databases right now, where is traffic getting slow, which connections are eating our cloud egress budget, and what process on which host opened that unexpected outbound connection.

What to Look for in a Network Monitoring Tool

Before comparing tools, here are the criteria that matter most in 2026.

Topology Discovery and Visualization

A good tool draws the map for you. Static Visio diagrams go stale the moment someone deploys a new service. Look for automatic discovery and a real-time topology view that updates as your infrastructure changes.

Kubernetes and Cloud-Native Support

Containerized workloads are short-lived and dynamic. Tools designed for static IP-based device lists struggle here. Native Kubernetes support, pod-aware identity, and service detection are now table stakes for any team running clusters.

Protocol Visibility

Knowing that two hosts exchanged 500 MB tells you less than knowing that an application server made 12,000 PostgreSQL queries to a specific database. Tools that detect protocols (HTTP, PostgreSQL, MongoDB, Kafka, Redis, and others) give you context, not just byte counts.

Auto-Detection of Services

Manually labeling every host and process does not scale. The best tools recognize databases, caches, message queues, web servers, and runtimes automatically based on process names, container images, network signatures, and orchestrator metadata.

Correlation with Logs, Metrics, and Traces

A network alert in isolation is not very useful. Tools that let you pivot from a network connection to the related logs, application metrics, or distributed traces dramatically reduce time to root cause.

Alerting and Thresholds

Alerts that fire on every minor blip get ignored. Look for tools that support tunable thresholds, baseline-aware anomaly detection, and clear escalation paths.

Pricing Transparency

Network monitoring is one of the categories where cost surprises are most common, especially with consumption-based pricing on flow data or device counts. Predictable, published pricing matters more than free tiers that turn into bill shock at scale.

The 12 Best Network Monitoring Tools in 2026

1. Sematext Network Map

Best for: Cloud-native teams that want eBPF-powered service and infrastructure visibility inside a full observability platform

Sematext Network Map gives you a real-time, visual representation of your entire infrastructure topology. Using eBPF-powered network insights collected by the Sematext Agent, you can see how services, pods, containers, and processes communicate across Kubernetes clusters and standalone hosts without configuring anything beyond installing the agent.

Instead of polling devices and inferring relationships, Network Map shows the actual connections happening right now. When something breaks, you can immediately see which services are affected and trace the problem to its source. Two complementary views, Services View and Infrastructure View (see details in the docs), let you flip between application-level dependency analysis and Kubernetes infrastructure drill-down.

Key Features:

• eBPF-based connection capture: Kernel-level visibility into every connection, no port mirroring or packet capture required. Next to zero overhead.
• Two views, one map: Services View for application dependencies, Infrastructure View for cluster, node, pod, container, and process drill-down
• Automatic service detection: Over 100 service types recognized out of the box, including PostgreSQL, MySQL, MongoDB, Redis, Kafka, RabbitMQ, Elasticsearch, Nginx, HAProxy, Envoy, and more
• Protocol-aware connections: Detects HTTP, PostgreSQL, MongoDB, Kafka, Redis, and dozens of other protocols with traffic volume per connection
• Key metrics: network latency, round trip time (RTT), packet loss, retransmissions, CPU, memory, disk I/O, HTTP latency, HTTP response codes, etc.
• Custom thresholds: Configurable warning and critical levels for CPU, memory, network I/O, and disk I/O so the map highlights what matters in your environment
• Color-coded health: Green, yellow, and red service cards and connection lines make problems visible at a glance
• Kubernetes-native: Cluster to node to pod to container to process navigation with namespace, deployment, and workload context
• Standalone host support: Works equally well for VMs and bare-metal servers, not just containers
• Filtering and search: Quickly focus on a namespace, service type, host, or specific service when the topology gets dense
• Unified observability: Pivot from network connections to logs, metrics, traces, and synthetic checks in the same UI

Pros:

• Truly cloud-native and Kubernetes-first, designed for dynamic infrastructure rather than retrofitted onto SNMP foundations
• eBPF means no application code changes, no sidecars, and no instrumentation work, and next to zero overhead
• Service detection works automatically for the technologies most teams actually run
• Integrated with the rest of the Sematext platform, so traces, logs, and metrics live next to your topology
• Significantly cheaper than the major enterprise observability platforms
• Useful for discovering unknown dependencies, suspicious connections, data exfiltration patterns, and cross-region cost leakage
• Replaces stale architecture diagrams with a live picture of what is actually running

Cons:

• Newer product compared to legacy NMS incumbents, less brand recognition among traditional network teams
• Focuses on host, container, and service visibility rather than deep packet inspection or SNMP-style switch and router polling

Pricing: Starts at $1.68 per host per month. Network Map scales with your Sematext Infrastructure Monitoring plan and retention. 14-day free trial, no credit card required.

Best For: DevOps and SRE teams running Kubernetes or hybrid cloud infrastructure who want service-aware network visibility tightly integrated with logs, metrics, and traces. Teams that find traditional NMS tools too device-centric and modern APM platforms too expensive.

Get started with Sematext Network Map or check out Network Map docs.

2. Datadog Network Monitoring

Best for: Enterprises already on Datadog who want network visibility inside the same platform

Datadog offers two related products: Network Performance Monitoring (NPM) for host-to-host and pod-to-pod traffic analysis, and Cloud Network Monitoring for cloud provider flow logs and load balancer telemetry. Both plug into the Datadog platform alongside APM, logs, and infrastructure monitoring.

Key Features:

• Host and Kubernetes pod-level traffic analysis via the Datadog Agent
• VPC flow log analysis for AWS, Azure, and GCP
• DNS monitoring with query-level visibility
• Cloud load balancer and gateway telemetry
• Integration with Datadog APM, logs, and infrastructure metrics

Pros:

• Strong if you already use Datadog for everything else
• Detailed flow analysis with rich filtering
• AI-assisted anomaly detection through Watchdog

Cons:

• Each Datadog product is billed separately, and bills add up quickly at scale
• “Bill shock” is a recurring complaint from customers
• Network products require agent and integrations setup separately from core APM

Pricing: Network Performance Monitoring starts at $5 per host per month. Cloud Network Monitoring is priced per analyzed flow. Costs scale with hosts, flows, and retention.

Best For: Mid-size to large enterprises already standardized on Datadog who can absorb the platform-wide cost.

3. Dynatrace

Best for: Large enterprises wanting AI-driven full-stack observability with network context

Dynatrace covers network visibility as part of its broader Davis AI observability platform. The OneAgent captures process-level network connections alongside code-level traces and infrastructure metrics.

Key Features:

• OneAgent captures host, process, and connection-level data automatically
• Smartscape topology view combining services, processes, and infrastructure
• Davis AI for root cause analysis across signals
• Multi-cloud and Kubernetes coverage
• Strong support for traditional enterprise workloads like SAP and mainframe

Pros:

• Truly automatic instrumentation with minimal manual configuration
• AI correlation across network, traces, logs, and metrics
• Strong enterprise governance and compliance features

Cons:

• Premium pricing, often the highest in the market
• OneAgent has a significant resource footprint compared to lightweight collectors
• Steep learning curve across the many Dynatrace apps and modules

Pricing: Custom pricing through sales. Typically positioned at the high end of the enterprise observability market.

Best For: Large enterprises with complex hybrid environments and the budget for a premium AI-powered observability platform.

4. Kentik

Best for: Network teams running large hybrid, multi-cloud, and internet-facing infrastructure

Kentik focuses on what it calls network intelligence: deep traffic analysis, BGP and internet path visibility, and AI-assisted investigation for hybrid and multi-cloud networks. It is particularly strong for service providers, SaaS companies, and large enterprises with significant cloud and internet exposure.

Key Features:

• NetFlow, sFlow, IPFIX, VPC flow logs, and synthetic test ingestion
• BGP and internet performance visibility
• Kentik AI Advisor and Cause Analysis for guided troubleshooting
• DDoS detection and mitigation analytics
• Hybrid and multi-cloud topology

Pros:

• Industry-leading depth for traffic analytics and internet path visibility
• Strong for cloud egress cost analysis
• AI-assisted investigation reduces time on complex incidents

Cons:

• Pricing aimed at mid-market and enterprise, not small teams
• Steeper learning curve than general-purpose monitoring tools
• Focused on NetOps workflows more than application-level observability

Pricing: Custom pricing through sales. Aimed at enterprise budgets.

Best For: Mid-market and enterprise NetOps and SRE teams managing complex hybrid and internet-connected networks.

5. Cisco ThousandEyes

Best for: Teams that need path visibility across the public internet and SaaS providers

ThousandEyes monitors network paths from your users and applications to wherever they need to go: SaaS apps, cloud regions, third-party APIs, and your own services. It is the go-to tool for understanding ISP, CDN, and public internet performance.

Key Features:

• Synthetic agents on user devices, in cloud regions, and on enterprise networks
• Hop-by-hop path visualization across the internet
• BGP route monitoring and outage detection
• SaaS application performance tests (Microsoft 365, Salesforce, Zoom, and similar)
• Internet outage detection and notifications

Pros:

• Best-in-class for public internet and SaaS path visibility
• Wide global agent network
• Strong correlation between user experience and underlying network paths

Cons:

• Does not cover host or pod-level traffic inside your infrastructure
• Pricing scales quickly with agent counts and test frequency
• Less useful as a standalone tool, usually has to be paired with another monitoring platform

Pricing: Subscription-based, priced per agent and per test. Custom quotes through sales.

Best For: Enterprises with significant SaaS, hybrid work, and internet-facing service dependencies that need to prove whether the problem is them, their ISP, or a SaaS provider.

6. SolarWinds Network Performance Monitor

Best for: Traditional enterprise IT teams managing on-premises networks

SolarWinds Network Performance Monitor is one of the longest-running enterprise NPM products. It excels at SNMP-based monitoring of routers, switches, firewalls, wireless access points, and other traditional network gear.

Key Features:

• SNMP v1, v2c, and v3 polling with deep multi-vendor support
• Automated Layer 2 and Layer 3 topology mapping
• NetPath for hop-by-hop path analysis
• Wireless network monitoring and heat maps
• Integration with the broader SolarWinds Observability portfolio

Pros:

• Deep coverage for traditional network gear from Cisco, Juniper, Aruba, Fortinet, and many others
• Mature alerting and reporting features
• Familiar to most network engineers

Cons:

• Built for static network device inventories rather than dynamic cloud-native workloads
• Self-hosted deployment requires Windows Server infrastructure
• Modernization toward SaaS has been gradual

Pricing: Starts around $1,995 for SolarWinds NPM perpetual license. SaaS pricing through SolarWinds Observability is consumption-based.

Best For: Enterprise NetOps teams managing campus, branch, or data center networks where SNMP and traditional NPM workflows dominate.

7. Auvik

Best for: MSPs and IT teams managing many distributed sites

Auvik is a cloud-based network monitoring platform aimed primarily at managed service providers and multi-site IT teams. Its automated topology mapping and per-site collector model make it easy to onboard new networks quickly.

Key Features:

• Cloud-managed with lightweight on-premises collectors per site
• Automatic Layer 1, 2, and 3 topology discovery and visualization
• TrafficInsights for NetFlow, sFlow, J-Flow, and IPFIX analysis
• Configuration backup and change tracking
• Multi-tenant architecture for MSPs

Pros:

• Fast time to value, sites usually online within an hour
• Excellent automated topology maps that update in real time
• Multi-tenant model purpose-built for MSPs
• Configuration management and syslog included

Cons:

• Per-billable-device pricing can get expensive at scale
• Less depth than Kentik or SolarWinds NPM for deep traffic analytics
• Not designed for application-level or container-level visibility

Pricing: Per-billable-device pricing through sales. Free trial available.

Best For: MSPs and IT teams managing dozens or hundreds of distributed sites where rapid onboarding matters.

8. Paessler PRTG Network Monitor

Best for: Small and mid-size organizations that prefer sensor-based, all-in-one monitoring

PRTG is a long-established Windows-based monitoring tool that uses a sensor model: each metric, port, or check is a sensor, and you pay for the total number of sensors. It covers networks, servers, applications, and IoT devices through SNMP, WMI, NetFlow, and HTTP probes.

Key Features:

• 250+ pre-built sensor types for network devices, servers, and applications
• SNMP, WMI, NetFlow, sFlow, J-Flow, and IPFIX support
• Maps, dashboards, and reporting included
• On-premises and PRTG Hosted options
• Free tier up to 100 sensors

Pros:

• Easy to install and configure on Windows
• Broad sensor catalog covers network and infrastructure in one tool
• Free tier is genuinely useful for small environments
• Strong fit for SMB and mid-market IT generalists

Cons:

• Sensor-based licensing gets complex and expensive as environments grow
• Windows-centric architecture (probes can be on Linux but the core is Windows-only)
• Less suitable for dynamic Kubernetes workloads

Pricing: Perpetual licenses from around $2,149 for 500 sensors. PRTG Hosted available as SaaS. Free up to 100 sensors.

Best For: Small and mid-size organizations running mixed Windows and Linux infrastructure that want a single tool for network and server monitoring.

9. ManageEngine OpManager

Best for: IT teams that want broad multi-vendor monitoring at a moderate price

OpManager is part of the ManageEngine portfolio. It provides SNMP, WMI, CLI, and Telnet-based monitoring with extensive multi-vendor support and offers both subscription and perpetual licensing.

Key Features:

• 200+ pre-built device templates across Cisco, Juniper, HP, Dell, Fortinet, and others
• Network Configuration Manager add-on for backup and compliance
• Workflow automation for routine remediation
• NetFlow Analyzer add-on for traffic analysis
• On-premises deployment with broad OS support

Pros:

• Significantly cheaper than SolarWinds NPM with comparable feature depth
• Perpetual licensing option for teams that prefer capital over operating expenses
• Strong multi-vendor coverage out of the box

Cons:

• Interface feels dated compared to modern SaaS tools
• Add-on modules increase the total cost
• Self-hosted only, no fully managed SaaS option

Pricing: Starts around $245 per year for 25 devices. Perpetual licenses available. NetFlow Analyzer and Network Configuration Manager priced separately.

Best For: Mid-market IT teams wanting broad on-premises network and server monitoring at a moderate price.

10. LogicMonitor

Best for: Hybrid infrastructure teams wanting SaaS-based monitoring across networks, servers, and cloud

LogicMonitor is a SaaS infrastructure monitoring platform that covers network devices, servers, cloud resources, and containers from a single console. The Edwin AI engine adds anomaly detection and event correlation across signals.

Key Features:

• 3,000+ integrations and modules across networks, servers, and cloud
• Cloud-managed with on-premises collectors
• Edwin AI for anomaly detection and alert correlation
• Per-device licensing rather than per-sensor or per-interface
• Topology mapping and synthetic checks

Pros:

• One of the broadest integration libraries in the market
• Predictable per-device pricing
• Strong fit for hybrid environments mixing data center and cloud

Cons:

• Enterprise pricing, not aimed at small teams
• Less focused on application-level or service-level visibility than Datadog or Dynatrace
• Some users report complex initial setup for advanced features

Pricing: Custom pricing through sales. Aimed at mid-market and enterprise.

Best For: Mid-market and enterprise IT teams managing hybrid infrastructure that want a single SaaS tool for networks, servers, and cloud.

11. Zabbix

Best for: Teams wanting a free, open-source enterprise-grade monitoring platform

Zabbix is one of the most widely deployed open-source monitoring tools. It supports SNMP, agent-based, agentless, and API-based monitoring for networks, servers, applications, and cloud services. The project has been actively developed for over two decades.

Key Features:

• SNMP, IPMI, JMX, ODBC, agent, and agentless monitoring
• Network discovery and topology maps
• Flexible templating and macros
• Notifications across many channels
• Distributed proxy architecture for large environments

Pros:

• Completely free and open-source under AGPL
• Mature, production-proven at very large scale
• Active community and commercial support available
• Flexible enough to cover networks, servers, and applications

Cons:

• Requires operational expertise to deploy, scale, and maintain
• UI feels less polished than commercial SaaS tools
• Initial configuration can be time-consuming

Pricing: Free. Commercial support and training available through Zabbix LLC.

Best For: Teams with the operational capacity to run their own monitoring infrastructure and that want full control without licensing costs.

12. Cilium Hubble

Best for: Kubernetes teams wanting open-source eBPF-native network observability

Cilium Hubble is the observability layer for Cilium, the CNCF-graduated eBPF-based networking and security project. It provides flow-level visibility, service maps, and policy enforcement insights for Kubernetes clusters.

Key Features:

• eBPF-based flow capture without sidecars
• L3, L4, and L7 visibility including HTTP, gRPC, Kafka, and DNS
• Service map visualization
• Network policy verification and dropped-flow analysis
• Integration with Prometheus and Grafana

Pros:

• Completely free and open-source under Apache 2.0
• Native eBPF approach with low overhead
• Strong for Kubernetes security and network policy use cases
• Tight integration with Cilium CNI

Cons:

• Requires Cilium as the CNI plugin, which is a significant architectural choice
• Operational expertise needed to deploy and scale Hubble
• Less suited for standalone hosts, VMs, or non-Kubernetes workloads

Pricing: Free. Commercial support through Isovalent (now part of Cisco).

Best For: Kubernetes-native teams that have adopted or are willing to adopt Cilium as their CNI and want deep eBPF-based flow observability without commercial licensing.

Network Monitoring Tools Comparison Table

How to Choose the Right Network Monitoring Tool

Start With Your Stack

A team running mostly Cisco switches and on-premises servers has a very different set of needs from a team running Kubernetes on EKS. SNMP-based tools like SolarWinds, ManageEngine, and PRTG remain the right answer for traditional network gear. For Kubernetes and dynamic cloud workloads, eBPF-based tools like Sematext Network Map and Cilium Hubble give you visibility that legacy NMS products cannot match.

Decide Whether You Want a Standalone or Integrated Tool

Network monitoring used to be a separate tool maintained by a separate team. That model still works for some organizations, but most modern teams benefit from network visibility that lives next to logs, metrics, and traces. Sematext, Datadog, Dynatrace, and LogicMonitor all let you correlate network signals with application data. Standalone specialists like Kentik and ThousandEyes go deeper in their niche, often paired with a broader observability platform.

Model Your Cost at Scale

Pricing models vary widely. Per-sensor models like PRTG can balloon as you add checks. Per-device models like Auvik and LogicMonitor are predictable but can become expensive as device counts grow. Per-host models like Sematext and Datadog scale with infrastructure size. Always run a quick projection for your expected scale rather than going by starter-tier pricing.

Test Before You Buy

Most commercial tools offer free trials. Open-source tools cost only your time. Pick two or three candidates that fit your environment and run them on a real workload for a week or two. You will learn more from a short pilot than from any vendor demo.

Do Not Forget About People

A tool nobody uses is worse than no tool at all. Whatever you choose, make sure your on-call engineers find the UI usable, the alerts trustworthy, and the data easy to share across teams.

Conclusion

Network monitoring in 2026 is no longer a single category. Some teams need deep SNMP visibility for traditional gear. Others need eBPF-based service maps for Kubernetes. Many need both, plus correlation with logs, metrics, and traces.

For cloud-native and hybrid teams, Sematext Network Map stands out as the most pragmatic choice in 2026. It gives you eBPF-powered service and infrastructure visibility, automatic detection of over 100 service types, Kubernetes-native drill-down, and integration with logs, metrics, traces, and synthetic monitoring in one platform. All of that comes at a fraction of the cost of the major enterprise observability incumbents.

If your environment is dominated by traditional network gear, SolarWinds, ManageEngine, and Auvik remain solid choices. If you need internet and SaaS path visibility, ThousandEyes is hard to beat. If you want fully open-source, Zabbix covers the broad case and Cilium Hubble covers the Kubernetes-native case.

Whatever you choose, prioritize tools that fit how your infrastructure actually works today, not how it worked five years ago.

Ready to see your infrastructure as it really is? Try Sematext Network Map free for 14 days, no credit card required.

FAQ

What is the difference between network monitoring and observability?

Network monitoring focuses on the health and behavior of network infrastructure: devices, interfaces, traffic, paths, and connections. Observability is a broader practice that combines logs, metrics, traces, and topology to understand the health of the entire system, including applications. Modern tools like Sematext, Datadog, and Dynatrace blur the line by treating network visibility as one signal among many.

Do I need separate network monitoring if I already have APM?

APM tells you what your application code is doing. Network monitoring tells you what is happening at the network connection and infrastructure level. The two are complementary. A slow database query shows up in APM. A flaky network path, a saturated link, or a rogue process making outbound calls shows up in network monitoring, but not in APM. Most teams benefit from both, ideally on the same platform.

What is eBPF and why does it matter for network monitoring?

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that lets tools safely run sandboxed programs inside the kernel to observe events such as network connections, system calls, and packet processing. For network monitoring, eBPF allows tools to capture connection-level data with very low overhead and without modifying applications. Tools like Sematext Network Map and Cilium Hubble use eBPF to build accurate, real-time topology views without sidecars, port mirroring, or packet captures.

Can network monitoring tools work with Kubernetes?

Yes, but with very different levels of effectiveness. Tools designed for static device inventories struggle with the dynamic nature of pods and containers. eBPF-based and Kubernetes-aware tools like Sematext Network Map, Cilium Hubble, Datadog NPM, and Dynatrace are built for this environment. Traditional SNMP-focused tools can still monitor the underlying nodes, but they will not give you pod-level or service-level visibility.

What are the best free or open-source network monitoring tools in 2026?

Zabbix remains the most widely deployed open-source choice for general infrastructure and network monitoring. Nagios and Icinga are still in active use for traditional environments. LibreNMS and OpenNMS are strong for SNMP-heavy networks. For Kubernetes-native, eBPF-based observability, Cilium Hubble is the leading open-source option. All of these are free to use but require operational expertise to deploy and scale.

How much do network monitoring tools cost?

Pricing varies enormously. Open-source tools are free but carry operational cost. Per-device tools like Auvik and OpManager typically range from a few dollars to tens of dollars per device per month. Per-host tools like Sematext Network Map start around $1.68 per host per month. Per-sensor tools like PRTG depend on how many checks you configure. Enterprise platforms like Datadog, Dynatrace, Kentik, and LogicMonitor are usually priced through sales and can run from tens of thousands to hundreds of thousands of dollars per year depending on scale.

Start Free Trial

The post Top 12 Network Monitoring Tools in 2026: Complete Comparison & Reviews appeared first on Sematext.

But there’s a part of the OTel pitch that often gets glossed over: somebody still has to instrument the application. And that part isn’t quick or easy. Even now.

The instrumentation tax

Modern applications aren’t a single binary anymore. At Sematext we run 30+ microservices to power the Sematext Cloud platform: alerts, metrics receivers and consumers, log receivers and consumers, user experience services, tracing pipelines, network map, various APIs and more, across Java (Spring Boot), Go, and a few other stacks.

That’s a lot of surface area, several languages, multiple frameworks, different build systems for different stacks. None of this is unusual for a mature system. This means that instrumentation effort grows with that diversity, so any mechanism that helps minimize instrumentation mistakes will be welcomed by engineers tasked with instrumentation.

If you want end-to-end tracing through that stack, the kind that actually tells you where a slow request spent its time, you can’t just instrument one service. You have to instrument the whole chain: frontend → API gateway → backend service A → backend service B → database. Skip a hop and the trace breaks. The dependency graph the trace gives you stops being useful exactly at the boundary you didn’t instrument.

So in practice, “let’s adopt OpenTelemetry” turns into a checklist of dozens of services that each need their own instrumentation work. The good news is that it doesn’t have to happen all at once and AI can help.

How much does OpenTelemetry instrumentation cost?

By cost, we mean the cost to an engineer, the team, and the organization. We can look at it as a non-monetary cost, but if we trace (pun intended!) this cost all the way down then yes, there is also a financial cost associated with this effort.

Three things make this hard, even with OTel:

Prioritization. Instrumenting a service competes with shipping features and fixing bugs. It’s preventive work; its value shows up the next time something breaks at 3am, not this sprint. That’s a hard sell to a product manager.

Unknown territory. When the chain spans services you didn’t write, in languages you don’t use day to day, you’re spending most of your time on context switch overhead. You’re not adding instrumentation; you’re re-learning a framework you saw once two years ago.

Time needed even for auto-instrumentation. “Auto-instrumentation” means no code changes. It doesn’t mean no work. For one service the loop typically goes:

1. Read the right OTel SDK docs for your language
2. Pick the right auto-instrumentation package (there are usually three options, only one of which is current)
3. Install it in the build (pom.xml, package.json, requirements.txt, …)
4. Configure the OTLP endpoint, the auth header, the service name
5. Restart, hit the service, watch what happens
6. Debug the first attempt: wrong port (4317 vs 4318 vs 4338), wrong protocol (http/protobuf vs grpc), wrong auth header (Bearer vs vendor-specific), region mismatch on the endpoint
7. Verify the data lands in the right place in your observability tool
8. Multiply by the number of services in your chain

Forty minutes to two hours per service if you’re moving carefully, and that’s just for traces and metrics. The OTel auto-instrumentation packages don’t ship logs in most SDKs. For logs you need to switch to manual instrumentation, which is another SDK init block per service.

And then there’s custom OpenTelemetry instrumentation

The above buys you “spans for every incoming HTTP request” and a generic metrics set. The moment you want anything specific (e.g., a custom span attribute for the user’s account tier, a business metric counting checkouts, a log enriched with the trace ID so you can correlate logs and traces for faster Root Cause Analysis), you’re back in manual-instrumentation land, writing SDK code in every service you care about. The auto path ends; the per-language SDK learning curve begins.

For one service that’s an afternoon. For thirty services that’s a quarter. What can we do about this?

Can we use AI to instrument applications with OpenTelemetry?

The instrumentation work (pick the SDK, set the env vars, debug the endpoint, verify it landed) is exactly the kind of structured, repetitive, well-documented task an AI agent does well. The blockers aren’t intellectual; they’re “look up the right thing, paste it in the right place, watch for the obvious gotcha.”

That’s not “let the AI do your engineering.” It’s “let the AI do the parts that already had a right answer, written down somewhere, and just needed someone to fetch it.”

We tried this for instrumenting applications against Sematext Cloud. The result is a small,but highly valuable open-source artifact: a Claude Code Agent Skill that walks an engineer through OTel instrumentation conversationally. It’s plain markdown, lives in our public Github repository, and works with any AI agent that can read a URL.

What the OTel instrumentation AI skill does

The Sematext OTel skill at sematext-otel-onboarding/blob/main/skills/SKILL.md is the AI-readable version of “how to wire your application to Sematext.” When loaded into Claude Code (or any agent that can fetch a markdown URL), it triages the user through six short questions:

1. Sematext region (US or EU)
2. Which App types you’re wiring (Tracing, Logs, Monitoring, any combination)
3. Flow: managed OTLP endpoint or Sematext Agent
4. Protocol: HTTP (default) or gRPC
5. Language and deployment environment
6. Auto or manual instrumentation

Then it produces the exact env-var block, parameterized to your answers, including:

• The correct OTLP endpoint URL for your region and protocol
• The Sematext-specific X-API-TOKEN header (different from the standard Authorization: Bearer … most OTel docs show, easy to miss)
• One header per signal type, so you only configure what you’re using
• A pointer to a runnable reference example in the same repo, in your language

Similarly, you can use the skill not only to add instrumentation to uninstrumented applications, but also to fix broken instrumentation that’s not really working. Auto-instrumentation doesn’t ship logs? The skill flags that and asks if you want to switch to manual. Region-token mismatch? The skill warns explicitly. Custom header convention? Documented. The skill is opinionated and aware of the setup required where the official OpenTelemetry docs may be silent or difficult to understand and follow.

What an instrumentation session looks like

In practice, an engineer with Claude Code in their editor opens their service’s directory and pastes:

Use https://github.com/sematext/sematext-otel-onboarding/blob/main/skills/SKILL.md to instrument this app for Sematext.
Region: US. App type: Tracing. Token: <pasted-from-Sematext-UI>.

Claude loads the skill, reads the project’s files to figure out the language and framework, asks the two remaining triage questions, then proposes the diff to the project (adds the OTel SDK to the build file, adds the env vars to docker-compose or systemd or .env or wherever they belong, and shows you what to expect in the Sematext UI within 60 seconds). You review the diff. You apply. You restart. You see traces.

Compared to the manual path (read docs, pick SDK, install, configure, debug, verify), we’d expect the instrumentation effort time to first data to drop from the typical 40 to 120 minutes per service to a handful of minutes. For an organization adopting OTel across dozens of services, that compounds quickly. A quarter of part-time effort becomes a couple of focused days. Thousands of dollars in engineering time drops to a much more sane number. The effort has a positive ROI.

What the skill doesn’t do

This is where AI posts usually start hand-waving. Here’s the honest list:

• It doesn’t write custom span attributes or business metrics for you. It writes the boilerplate that gets you to the point where you can write those. The judgment about what to measure is still yours. The benefit is that the skill gives you all the scaffolding, a working instrumentation, so the effort of collecting custom/business metrics becomes significantly lower.
• It doesn’t psychic-debug your network. If your service can’t reach the OTLP endpoint on first run (corporate proxy, missing TLS cert chain, wrong port), the skill points at the common causes, but you still have to look at the service’s own logs to confirm what happened.
• It doesn’t change OTel’s reality. Auto-instrumentation still doesn’t ship logs in most SDKs. AI doesn’t fix the SDK. But it does tell you up front, so you don’t spend an hour wondering why your Logs App is empty.

Try it, the skill is vendor-agnostic

The skill is open-source and lives in our OTel onboarding repo. The same repo has runnable reference apps for Node.js, Java, Python, .NET, and PHP across baremetal, Docker, and Kubernetes deployments, so if you want to see what the skill is going to walk you through, the reference is right there.

If you have Claude Code, point it at the URL above. If you use a different AI agent, the skill is just markdown. Load it however your agent loads documentation. There’s no install step; there’s no vendor lock-in. The skill shared is not Sematext-specific. Sematext’s contribution is the knowledge, encoded in a format an AI can act on.

Where this is going

The OTel skill is one example of a broader pattern we think makes sense for observability tools: knowledge as something an AI can use, not just something a human can read.

A few directions we’re exploring:

• Per-language sub-skills for deeper, opinionated guidance when a language has subtle gotchas (Node async hooks, Java agent attach, Python startup ordering)
• In-product wiring so the App creation page in Sematext Cloud gives you a one-click “use AI to set this up” alongside the existing manual instructions, with your region and token pre-filled into the prompt
• Skills for the rest of the observability journey (picking sensible default alerts, creating dashboards, interpreting RCA results), each as a small, auditable markdown file you can use, fork, or ignore

The bet is that the value of an observability platform isn’t just the data it collects; it’s how quickly an engineer can go from “we should monitor this” to “we’re monitoring it and we know what to do when it breaks.” AI doesn’t replace the judgment in that loop as of yet. But it can absolutely replace the busywork around it.

If you give the skill a try and find a gap, open a PR. The fastest way for this pattern to get good is for more people to use it on more apps.

Start Free Trial

The post Using AI to Instrument Applications with OpenTelemetry appeared first on Sematext.

It starts in late November, 2025. This marks the beginning of increased AI usage (for coding) in Sematext. That’s when the LLMs got better. It roughly matches the change in velocity as visualized in JIRA.

Individual AI Adoption

The blurred part are PR author names, which we can use for filtering. If we look at trends of individuals we can spot early adopters like this one:

Or another individual who started making more use of AI later:

Source: Github WebHook Events

This data comes into Sematext via Github Webhook Events. It takes about 5-10 minutes to set up. It can be set up at the Github organization level or for individual repositories.

A Word of Caution

1. There are many software development styles. There are people who commit frequently and incrementally and there are those who keep things to themselves until everything is nearly done. This is a fun chart to look at and is helpful when you want to get the feel for the “pulse” of a team or even an individual. But be careful not to judge people on this sort of data alone. Use this with a grain of salt and in combination with other inputs, observations, etc.
2. Creating more code or PRs doesn’t always equal better code or higher effectiveness. A person may be tapping in the dark trying to debug or implement something with the help of AI and, in the process, creating a lot of (temporary?) code and PRs.
3. As velocity increases, so will regressions, unless you take countermeasures. See Faster Coding with AI and Increased Regressions.

Start Free Trial

The post Pull Request Velocity as a Proxy for AI Usage for Software Development appeared first on Sematext.

Running OTel across a handful of services and running it across a few hundred are genuinely different problems. The instrumentation part stays roughly the same. Everything around it — how you collect the data, how you route it, how you make sure a traffic spike in one region does not take down your entire observability pipeline — that is where teams either build something resilient or spend the next six months fire-fighting because of inadequate planning or suboptimal architecture.

I wrote this article to share the patterns that actually hold up at scale: collector tiers, load balancing strategies, sampling at volume, and multi-cluster setups. Everything comes with real config examples because “it depends” is only useful advice if you can see what it depends on.

See How OpenTelemetry changes the way teams do observability for why OpenTelemetry matters and how it shifts focus from traditional metrics and logs to full, end-to-end observability.

Why a Single Collector Falls Apart (and When)

Most OTel tutorials show you a single collector instance receiving spans from all your services and forwarding everything to a backend. That setup works until about the point where it stops working, which tends to happen quietly and at the worst possible time. You are not going to notice a single collector struggling until it is already dropping data, buffering is maxed out, and your traces have gaps you cannot explain.

The core issue is that a single collector is both a single point of failure and a resource bottleneck. At low traffic it sits there looking fine. Add a few dozen services, let traffic spike during a product launch or a retry storm, and you will watch it start falling behind. The exporter queue fills up. Backpressure kicks in. Services start dropping spans rather than blocking on the export. By the time anyone notices, you have lost the exact telemetry you needed to understand what just happened.

The solution is to stop thinking about the collector as a single process and start thinking about it as a tier. Two tiers cover most production scenarios. Three tiers cover the rest. The architecture you need depends on your traffic, whether you need tail-based sampling, and how many backends you are exporting to.

Let me make this more specific: if you have fewer than 20 services and under 500 requests per second total, a single well-configured collector will likely hold up (yes, of course it depends on the underlying hardware/resources). At 20 to 80 services or 500 to 5,000 RPS, the two-tier model becomes worthwhile. Above 80 services or 5,000 RPS, you need the full tiered setup with trace-aware load balancing and tail-based sampling at the gateway. 

For more information on common production pitfalls and strategies to prevent them, see OpenTelemetry Production Monitoring: What Breaks and How to Prevent It.

Collector Tiers: The Architecture That Actually Scales

The tiered collector model separates two concerns that should never have been combined in the first place: getting data off your services quickly, and doing something intelligent with that data before it hits your backend.

Before getting into the architecture, it helps to know that the OTel Collector can run in three modes — and in a scaled setup, you will use all three:

• Agent — a collector running on the same host as your services, collecting telemetry locally and forwarding it upstream. It stays thin: no heavy processing, just receive-and-forward.
• Gateway — a collector running as a standalone service, receiving data from agents (or directly from SDKs) and doing the heavier work: sampling, routing, fan-out to backends, attribute redaction.

Combined — the full pattern, where agent collectors feed into gateway collectors. Agents handle what only makes sense per-host (host metrics, file logs, resource detection). Gateways handle what only makes sense centrally (tail-based sampling, cross-service routing, policy management). The OTel Collector deployment docs call this the combined deployment pattern.

The tiered setup this article describes is the combined pattern. Here is what it looks like:

Tier 1: Collectors running as agents

The agent tier runs as a sidecar. Its job is exactly one thing: receive telemetry from the services and forward it as fast as possible. No tail-based sampling, no complex routing logic, no fan-out to multiple backends. The only processing you want at this tier is cheap and stateless: adding resource attributes like cluster name, node name, and environment; batching spans to reduce connection overhead; and basic filtering to drop genuinely worthless spans like health check endpoints generating thousands of spans per minute and telling you nothing.

# Tier 1: runs as DaemonSet, minimal processing
receivers:
  otlp:
    protocols:
      grpc: {endpoint: "0.0.0.0:4317"}
      http: {endpoint: "0.0.0.0:4318"}

processors:
  batch:                    # batch before forwarding
    send_batch_size: 1024
    timeout: 5s
  resourcedetection:        # stamp node/pod metadata
    detectors: [k8snode, env]
  filter/drop_healthchecks:
    spans:
      exclude:
        match_type: regexp
        attributes:
          - {key: http.route, value: ".*/health.*"}

exporters:
  otlp:
    # forward to gateway tier, not directly to backend
    endpoint: "otel-gateway:4317"
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 500

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [batch, resourcedetection, filter/drop_healthchecks]
      exporters:  [otlp]

exporters:
  loadbalancing:
    routing_key: "traceID"   # hash by trace ID, not round-robin
    resolver:
      k8s:                    # auto-discover gateway pods via DNS
        service: "otel-gateway"
        ports: [4317]
    protocol:
      otlp:
        timeout: 1s
        sending_queue:
          enabled: true
          queue_size: 1000

processors:
  tail_sampling:
    decision_wait: 10s      # wait for all spans before deciding
    num_traces: 100000      # traces held in memory simultaneously
    expected_new_traces_per_sec: 1000
    policies:
      # always keep error traces
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}

      # always keep slow traces (adjust threshold to your SLO)
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}

      # keep 100% of checkout and payment — business critical
      - name: keep-critical-services
        type: string_attribute
        string_attribute:
          key: service.name
          values: [checkout-api, payment-service]

      # probabilistic baseline for everything else
      - name: baseline-sample
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

# nginx — add inside your proxy_pass block
proxy_set_header traceparent $http_traceparent;
proxy_set_header tracestate  $http_tracestate;

---

# Envoy — request_headers_to_add in HttpConnectionManager
route_config:
  request_headers_to_add:
    - header: { key: traceparent }
    - header: { key: tracestate }

processors:
  transform/redact_pii:
    trace_statements:
      - context: span
        statements:
          # hash user IDs rather than drop
          - set(attributes["user.id"], SHA256(attributes["user.id"]))
          # drop email entirely
          - delete_key(attributes, "user.email")
          # truncate IP to /24 for geo without individual tracking
          - replace_pattern(attributes["net.peer.ip"], "\\d+$", "0")

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 15s
          static_configs:
            - targets: ["localhost:8888"]

# Expose collector's own telemetry via its service config
service:
  telemetry:
    metrics:
      level: detailed   # basic | normal | detailed
      address: 0.0.0.0:8888
    logs:
      level: warn       # keep collector logs quiet in production

Start Free Trial

The post Running OpenTelemetry at Scale: Architecture Patterns for 100s of Services appeared first on Sematext.

This is what life looks like when observability is an afterthought: logs in one place, metrics in another, and a custom monitoring agent that only works for two services because the third one was written in a language nobody on the team uses anymore. It works, technically. Until it does not.

OpenTelemetry came out of a genuine frustration with this fragmented mess. It is an open-source observability framework that gives you a vendor-neutral, standardized way to instrument your applications and then connect that instrumentation to service health, error budgets, and eventually SLOs that your entire organization actually understands. This article walks through what that shift looks like in practice, and why it matters for more than just the people who are on call.

The Old World: Logs, APM Agents, and the Dashboard Graveyard

Let’s be direct about how most teams actually do observability before they invest in it properly. You have application logs going into a log management platform, with varying levels of structure depending on who wrote which service. You have an APM tool that auto-instruments some of your services but not all of them, and the traces it produces are siloed within its own ecosystem. And you have a monitoring dashboard that someone built eighteen months ago and that might or might not reflect how the service actually behaves today.

The deeper problem is not the tools themselves; it is that each one has its own instrumentation model. Your APM agent captures HTTP spans one way. Your custom metrics library reports latency percentiles slightly differently. Your logs do not correlate to your traces automatically. So when something breaks, you are stitching together three different narratives instead of reading one coherent story about what happened.

What OpenTelemetry Actually Is (Without the Fluff)

OpenTelemetry standardizes how you generate, collect, and export telemetry data. It covers three signal types (with more to come), which are the foundation of everything else in this article:

What makes OTel different from what came before is not magic; it is the fact that all three signals share the same context propagation model. A trace ID that starts in your frontend propagates through every instrumented microservice call, and if your logs are also emitting that trace ID, you can jump from a log line to its trace in seconds. Not minutes. Seconds. If you are the person doing production troubleshooting you know how valuable this difference is!

SLOs: What They Are and Why OTel Makes Them Achievable

Service Level Objectives have been a thing since Google wrote about them in the Site Reliability Engineering book, and they have been misunderstood and poorly implemented since roughly the same time. The core idea is simple: you agree on a target for how reliable a service needs to be, you measure it consistently, and you manage your engineering work in relation to how much reliability budget you have consumed or have left.

The reason SLOs often fail is not the concept; it is that teams try to define them before they have reliable telemetry. You cannot set a meaningful availability target for a service if your metrics come from three different monitoring agents that measure availability in subtly different ways. You end up with SLOs that nobody trusts, which means nobody uses them to make decisions.

When your SLIs are computed from OTel metrics, specifically from the semantic conventions that define how HTTP span duration and status should be recorded, you get consistency across services by default. The latency histogram for your Go service and the one for your .NET service use the same bucket boundaries. The error classification follows the same logic. Suddenly your SLOs are comparing apples to apples, and that changes what you can do with them.

The Correlation Story: How one Trace ID Connects Everything

One of the things that sounds academic until you experience it is trace context propagation. When a request comes into your frontend and you are using OTel instrumentation, a trace ID gets generated and passed along to every downstream service call via HTTP headers, gRPC metadata, message queue attributes, or whatever transport you are using. Every span in that trace carries the same trace ID, and your logs carry it too if you have set up log correlation.

What this means in practice: when your error rate alert fires because the checkout service just breached its error budget, you do not start by guessing. You go to the traces for that time window, filter for error spans, and you are already looking at the full call path: frontend, checkout API, inventory service, payment gateway, with timing for each hop. If the inventory service was slow, you will see a long span there. If the payment gateway returned a 503, you will see that in the span status. No grep-ing through logs trying to find a request ID that someone may or may not have remembered to log. For a step-by-step breakdown of what these patterns look like in real incidents, troubleshooting microservices with distributed tracing is a good companion read.

Auto-Instrumentation: Getting Value Without Rewriting Everything

One of the biggest objections to investing in observability is the instrumentation cost. If you have thirty microservices and each one needs to be manually instrumented before you see any benefit, that is a project with a very long feedback loop. This is actually what we saw with our initial distributed tracing implementation at Sematext back in 2015 – adoption of a challenge due to how much work engineers would have to invest in instrumenting their applications. OTel’s auto-instrumentation libraries change that equation significantly.

For Java, the OTel Java agent attaches to your JVM at startup and automatically instruments common frameworks such as Spring Boot, gRPC, JDBC, and Kafka without any code changes. For Python, opentelemetry-instrument does the same for Flask, Django, FastAPI, and SQLAlchemy. The .NET ecosystem has similar coverage through the automatic instrumentation package. You get spans for every incoming HTTP request, every outgoing call, and every database query without touching the application code. If you want to skip the boilerplate and start from something that already works, these language-specific OTel examples cover the setup end to end.

What to Actually Watch Out For

None of this comes without tradeoffs, and articles that only cover the benefits are setting you up for some unpleasant surprises. A few things will bite you if you do not plan for them.

A deep dive into OpenTelemetry instrumentation best practices covers all of these in detail, but here is the short version.

Cardinality explodes if you are not careful

OTel metrics support rich attribute sets, which is great for debugging but problematic for storage if you start adding high-cardinality attributes like user IDs or request IDs to your metrics. The OTel metrics spec includes cardinality limits, and you should understand them before you start attaching attributes to everything.

Sampling is necessary at scale and confusing to get right

Sending 100 percent of traces when you are handling thousands of requests per second is expensive. Head-based sampling, where you decide at the start of a trace whether to keep it, is simple but means you might drop the interesting traces. Tail-based sampling, where you decide after seeing the whole trace, keeps the errors but requires the OTel Collector to buffer spans, which adds complexity. There is no right answer, only tradeoffs that depend on your volume and budget.

Auto-instrumentation vs manual instrumentation: the honest tradeoff

Auto instrumentation gets you running in an afternoon with zero code changes and gives consistent coverage across your entire fleet from day one. The trade off is that it understands frameworks, not business intent. It can tell you a database query took 800 ms but not that it was pricing a cart for a high value customer.

Manual instrumentation fills the gap that actually matters for SLOs. Checkout completion time, order processing latency by fulfillment partner, or time to first search result. It takes more effort, but it is what turns a latency alert into a business conversation.

In practice, auto instrumentation provides the foundational 80 percent. Requests, error rates, and durations (aka RED) from day one. You then layer manual instrumentation on top for the business critical signals your SLOs should be measuring.

The Collector configuration gets complex fast

Once you start running multiple pipelines, applying transforms, doing tail-based sampling, and exporting to multiple backends, your collector config becomes something that needs to be tested and versioned like application code. Treat it that way from the start.

Starting Without Starting Over

The most common mistake teams make when adopting OTel is treating it as a big-bang migration. You do not need to instrument every service before any of it becomes useful. Pick one service, ideally something that sits in the middle of your call graph so you can see upstream and downstream spans, and get it fully instrumented with OTel, exporting to a collector and from there to whatever backend you already have. Define one or two SLIs for it. Watch them for a week and see if they match your intuition about how the service is performing.

That first service will teach you things that no amount of reading can. You will find out how your framework handles context propagation. You will discover that your log format does not include trace IDs and need to fix that. You will learn what your normal latency histogram looks like and be surprised by the long tail. Do that before you roll out to thirty services, and the rollout will go much faster. 

To get started see the Sematext step-by-step setup guide for OpenTelemetry tracing. Once you have that in place, the article on building a troubleshooting workflow with Sematext tracing shows how to use those first traces to investigate issues and iterate on your instrumentation.

Start Free Trial

The post From Debugging to SLOs: How OpenTelemetry Changes the Way Teams Do Observability appeared first on Sematext.

However, production has a way of humbling even the most carefully prepared setups. When real traffic hits, and it always spikes sooner or later, you start seeing dropped spans. Collector memory climbs until the process gets killed, and if you are running a single-instance collector, you can forget about collecting any telemetry until you bring it back up. Costs climb faster than anyone budgeted for. A few traces look incomplete. The bossman asks why latency increased by 12% after “just adding observability.”

None of this means OpenTelemetry is broken. It means production behaves differently than demos. This guide walks through what actually breaks when OpenTelemetry meets real-world scale, and what you can do about it before it becomes a 2 AM incident. Catching these issues early is the difference between a boring Tuesday and a war room.

For a practical setup of OpenTelemetry in microservices, see our step-by-step guide on distributed tracing with auto-instrumentation.

The First Production Surprise: Cardinality Explosions

High cardinality is one of the fastest ways to destabilize an otherwise healthy observability setup, and it almost always starts innocently. Someone with the best intension adds a genuinely helpful attribute:

• user_id
• session_id
• request_uuid
• a fully expanded URL path

In development, nothing bad happens. In production, that single decision can create millions of unique time series. For example, if a request counter is labeled with user_idand you have two million users, you have just created two million distinct metric series for one metric. Multiply that across services and dimensions, and storage, memory, and the performance of your observability tool degrades quickly.

You will notice it in a few ways: dashboards become noisy or slow, request latency increases, storage costs spike, and collector memory usage grows for no obvious reason.

The fix is not complicated, but it requires discipline. Metrics should use low-cardinality dimensions only, things like environment (prod, staging), service name, endpoint patterns rather than full URLs, and HTTP status classes (2xx, 4xx, 5xx). Anything that is essentially unique per request does not belong on a metric.

With auto-instrumentation, you do not always control attribute creation directly, but you can still suppress high-cardinality attributes via agent configuration, or drop and transform attributes in the collector using processors like filter, attributes, or transform. With manual instrumentation, you have full control and full responsibility. If you truly need high-cardinality identifiers, consider hashing or aggregating them before attaching them.

The key habit is to monitor cardinality continuously, not just after a cost spike. Keep an eye on the collector metrics that look like processor_accepted_metric_points broken down by metric name. These reveal which metrics are growing out of control before they degrade performance or inflate your bill.

For more guidance on instrumentation hygiene and preventing cardinality issues from the start, see our OpenTelemetry instrumentation best practices.

Scaling Pressure in OpenTelemetry Production Pipelines

OpenTelemetry components, SDKs, agents, and collectors, are not magic. They are software services that can be overloaded, and in high-throughput systems they often are.

In busy environments, traces can be generated at hundreds of thousands per second. Metrics multiply across services, containers, and pods. If batching, memory limits, and exporter throughput are not tuned, the pipeline itself becomes the bottleneck. The symptoms are predictable: processor_refused_spans starts increasing, collector memory climbs steadily, export failures appear, and telemetry arrives late or gets dropped entirely.

To understand where these bottlenecks occur, consider the overall OpenTelemetry production pipeline:

If you are using manual SDK instrumentation, you can tune batching and flush intervals directly. Larger batches reduce per-span overhead but increase memory pressure in the application itself, raising the risk of an OOM kill for containerized workloads. Smaller batches reduce memory but increase network calls. There is a balance, and you find it through load testing rather than guesswork.

With auto-instrumentation agents, you do not have direct SDK access, but most agents expose equivalent environment variables for batch size and schedule delay. These matter in production just as much as they do with manual instrumentation. A simple example showing where these settings live can save a lot of trial and error:

OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512

OTEL_BSP_SCHEDULE_DELAY=5000

For detailed information, see Environment Variable Specification. 

Regardless of instrumentation type, the collector itself must be treated like any other production service. Monitor its CPU and memory, scale it horizontally when needed, use load balancing with trace ID based routing so spans for the same trace land on the same collector instance, and watch queue lengths in the batch processor. If your collector is not monitored, you do not have observability, you have a single point of failure. 

For detailed guidance, see OpenTelemetry Collector architecture and best practices.

Sampling Strategies for OpenTelemetry in Production

At some point, you realize capturing 100% of traces is not sustainable. Sampling becomes necessary. However, sampling is not just a cost decision, it also changes what you can see, so it deserves more thought than simply dialing a number down.

Agent-Level Sampling

Agent-level sampling makes the decision immediately when a request starts, before a single span hits the collector. The benefit is immediate volume reduction: CPU, memory, and network overhead all drop. The trade-off is permanent blindness for discarded traces. If an error happens in a trace that was not sampled, it simply does not exist in your backend. There is no way to recover it after the fact.

Agent-level sampling works well as a baseline control mechanism. Many production systems start at 5 to 10% and adjust based on throughput and debugging needs. It is particularly useful when throughput is extremely high, infrastructure or observability vendor cost is the primary concern, or you need to protect the collector from being overwhelmed. Just keep in mind that it does not guarantee you will retain slow or rare traces that would have been most useful during an incident.

Tail Sampling

Tail sampling moves the decision to the collector, after the entire trace has been observed. This enables smarter decisions: keep slow traces, keep error traces, retain 100% of traffic from business-critical services, and sample normal traffic probabilistically.

This is more powerful, but it comes with real operational weight. The collector has to buffer complete traces in memory while waiting for all spans to arrive, which means memory usage is meaningfully higher than with head-based sampling. It also adds latency to trace delivery, since the collector has to wait for the full trace before deciding whether to keep it. If your typical transaction takes 90 seconds to complete, your collector is buffering 90 seconds of trace data before it can act, which is a lot of memory at scale, and your traces will arrive in your backend 90 or more seconds after the fact. For short-lived transactions this is barely noticeable. For long-running workflows, plan accordingly.

In distributed systems, spans for the same trace can arrive at multiple collector instances. If each collector makes independent sampling decisions, traces become fragmented, leaving gaps that make debugging much harder. Using tail sampling with load-balanced routing, where all spans for a trace are routed to the same collector instance using trace ID hashing, keeps traces intact and reliable. To be precise – this sticky routing is required for well-functioning tail-sampling.

The most effective production strategy usually combines both approaches: use agent-level sampling to cut down overall span volume and prevent the collector from being overwhelmed, then use tail sampling at the collector to make sure high-value traces, slow requests, errors, and critical transactions, are preserved. Sampling is not random volume reduction. It is selecting the traces that help you debug real incidents.

For the official OpenTelemetry guidance, refer to the OpenTelemetry sampling specification.

How to Set Tail Sampling Policies in Practice

Before writing any tail sampling policy, start by asking yourself a few practical questions: what types of incidents happen most often? Are latency regressions more frequent than hard failures? Which services are business-critical or compliance-sensitive? The answers should guide your sampling decisions, not the other way around.

For example, if most of your incidents are latency-related, prioritize keeping slow traces. A common starting point is to retain 100% of traces slower than twice your SLO, while sampling just 5 to 10% of normal traffic. For compliance-sensitive endpoints, always keep those traces intact. For business-critical services, bias your sampling to capture a higher proportion of requests, perhaps 50% from your payment service but only 5% from static content services.

It is also worth maintaining a small baseline sample across all services, around 5 to 10% of overall traffic, even for well-behaved paths. This gives you trend data and lets you detect unknown failure modes you did not anticipate when writing the policies. Without that baseline, you lose visibility into normal system behavior and can miss gradual degradations that do not trigger your explicit rules.

Agent and Collector Stability: The Hidden Risk

Agents and collectors are not passive observers. They are active components in your application infrastructure, and they can fail like any other component.

The collector is the more straightforward case. OpenTelemetry SDKs instrument your application code directly, and the collector runs as a separate process (or set of processes) that receives, processes, and exports telemetry. When a collector crashes, all buffered data is lost, including any traces that were being held in memory for tail sampling decisions. Memory spikes can trigger OOM kills, and if you are running a single collector instance, the entire observability pipeline goes dark until it recovers.

The common causes are predictable: exporters fall behind because the backend is slow or throttling ingest, queues grow, memory fills, and eventually the collector crashes. The practical safeguard against this is the memory limiter processor, which watches the collector’s overall memory consumption and temporarily refuses incoming data when it crosses your configured threshold, giving the collector room to catch up.

processors:

  memory_limiter:

    check_interval: 1s

    limit_mib: 2000

    spike_limit_mib: 400

service:

  pipelines:

    traces:

      receivers: [otlp]

      processors: [memory_limiter, batch]

      exporters: [otlphttp]

This is one of those configurations that feels optional until the day it is not.

Auto-instrumentation adds another layer of complexity. Java agents rewrite bytecode at runtime, async context propagation in .NET or Node.js can behave unexpectedly under load, and in high-throughput systems you may spend measurable CPU time just recording spans. This is why load testing your instrumentation matters as much as load testing your application. Before rolling out to production, measure baseline latency without instrumentation, then measure P50, P95, and P99 latency with it enabled. A 5 to 10% latency increase is often acceptable. Triple-digit millisecond overhead per request is not.

For detailed instructions by language, see the OpenTelemetry auto-instrumentation documentation.

Exporter Bottlenecks: When the Backend Cannot Keep Up

Even if your SDKs and collectors are perfectly tuned, the backend you are exporting to may not be. When the backend is slow, throttling requests, or simply unable to absorb your telemetry volume, batches start piling up in the exporter queues inside the collector. Left unchecked, this cascades into collector instability.

The signals to watch for are otelcol_exporter_send_failed_spans (a counter visible in the collector’s own self-monitoring metrics), growing exporter queue lengths, increased export latency, and rising memory pressure in the collector process.

For self-hosted backends like Elasticsearch, OpenSearch, or Prometheus, ingestion capacity must match telemetry throughput and cardinality. For external vendors, you need to understand their API rate limits, network latency characteristics, and burst handling policies before you are under pressure. An asynchronous exporter with buffering, retry logic, and exponential backoff is essential. Without it, a temporary backend slowdown cascades through the entire pipeline. Your observability stack is only as reliable as its slowest component.

Why This Matters in Real Systems

Many OpenTelemetry tutorials and examples show instrumentation working out of the box, which it does, in a demo environment with predictable traffic and no cost constraints. Real production systems are a different beast entirely: high throughput, distributed microservices, partial network failures, uneven traffic spikes, and budgets that someone is accountable for.

OpenTelemetry is genuinely powerful, but it requires operational discipline. When you adopt it, you are not just instrumenting a few services. You are operating an observability pipeline that itself needs capacity planning, monitoring, load testing, a clear sampling strategy, and ongoing cardinality governance. Treat it as first-class infrastructure and it becomes a strong foundation for understanding your systems. Treat it as a set-and-forget library and it becomes your next incident.

Start Free Trial

The post OpenTelemetry Production Monitoring: What Breaks, and How to Prevent It appeared first on Sematext.

This guide covers practical, trace-based troubleshooting patterns for production microservices. You’ll learn how to use OpenTelemetry distributed traces to diagnose the most common, and most frustrating, problems that surface in distributed architectures.

What you’ll learn:

• How to identify latency bottlenecks using trace waterfall analysis
• Detecting N+1 query patterns and database performance issues in traces
• Diagnosing retry storms, timeout cascades, and circuit breaker failures
• Using error propagation traces to find root causes across service boundaries
• Spotting connection pool exhaustion, cache misses, and queue backlogs
• Correlating traces with logs and metrics for full-context debugging

For step-by-step instrumentation setup, see our companion guide: How to Implement Distributed Tracing in Microservices with OpenTelemetry Auto-Instrumentation. For production-hardening your instrumentation, see OpenTelemetry Instrumentation Best Practices for Microservices Observability.

Why Traces Are the Best Tool for Microservices Troubleshooting

Logs, metrics, and traces each serve a different purpose. But when a production incident hits a distributed system, traces are uniquely positioned to answer the hardest questions, especially those that span service boundaries.

The key insight is that traces give you causation, not just correlation. When service A calls service B, which calls service C, and C fails, a trace shows you the entire chain, the timing of each call, and exactly where things went wrong.

Anatomy of a Troubleshooting Trace

Before diving into specific patterns, let’s establish what you’re looking at in a trace waterfall. Understanding the structure makes pattern recognition faster during incidents.

A distributed trace consists of spans organized in a parent-child hierarchy, as defined by the OpenTelemetry Trace specification. Each span represents a single operation: an HTTP request, a database query, a cache lookup, a message publish. The root span represents the entry point, and child spans represent downstream operations.

[Root Span: GET /api/orders/12345] ─────────── 1,247ms

├── [auth-service: POST /validate] ── 23ms
├── [order-service: GET /orders/12345] ────── 1,180ms
│     ├── [PostgreSQL: SELECT * FROM orders] ── 12ms
│     ├── [inventory-service: GET /stock] ─── 890ms  ← BOTTLENECK
│     │     ├── [Redis: GET inventory:12345] ── 2ms (miss)
│     │     └── [PostgreSQL: SELECT ...] ── 875ms  ← ROOT CAUSE
│     └── [pricing-service: GET /calculate] ── 45ms
└── [notification-service: POST /email] ── 18ms

In this trace, the total request took 1,247ms. The trace waterfall immediately shows that inventory-service consumed 890ms, and within it, a database query took 875ms following a cache miss. Without the trace, you’d see a slow /api/orders endpoint in your metrics and have to investigate each service individually.

Key span attributes to examine during troubleshooting (see the full OpenTelemetry Semantic Conventions for reference):

Diagnosing Latency Bottlenecks with Trace Waterfall Analysis

Latency issues are the most common reason teams reach for traces. The waterfall view transforms a vague “the API is slow” complaint into a precise diagnosis.

Pattern: The Slow Database Query

Symptoms in metrics: Elevated p95/p99 latency on a specific endpoint. Database CPU or connection usage may appear normal.

What the trace reveals:

[order-service: GET /orders] ────────── 2,340ms

├── [PostgreSQL: SELECT o.*, oi.* FROM orders o
│    JOIN order_items oi ON o.id = oi.order_id
│    WHERE o.customer_id = $1
│    ORDER BY o.created_at DESC] ──── 2,280ms  ← Problem
└── [Redis: SET order-cache:customer:789] ── 3ms

The trace shows a single database query consuming 97% of the request time. The db.statement attribute reveals the actual SQL, which is a full table scan joining orders with order items, likely missing an index on customer_id.

What to look for in spans:

• db.statement: Check for missing WHERE clauses, full table scans, large JOINs, or unoptimized queries. Use EXPLAIN to confirm.
• Span duration vs. typical duration: Compare against baseline traces for the same operation
• Sequential vs. parallel queries: Are queries running sequentially when they could be parallelized?

Pattern: Sequential Service Calls (Missed Parallelization)

Symptoms in metrics: High latency that seems disproportionate to what any single service reports.

What the trace reveals:

[api-gateway: GET /dashboard] ──────────── 1,850ms

├── [user-service: GET /profile] ── 320ms
├── [order-service: GET /recent] ─── 480ms    (starts after user-svc)
├── [notification-svc: GET /unread] ── 410ms  (starts after order-svc)
└── [recommendation-svc: GET /for-you] ── 590ms (starts after notif.)

The waterfall reveals that four independent service calls are executing sequentially. Total time is the sum of all calls (1,800ms) instead of the max (590ms), a 3x penalty. The trace makes this immediately visible because spans don’t overlap.

The fix: Refactor to concurrent calls. With parallelization, the trace collapses to ~620ms as all four spans overlap.

Pattern: Fan-out Amplification

Symptoms in metrics: Latency increases with load, but individual service latencies look normal.

The trace reveals a product catalog page making 50 individual HTTP calls to the inventory service, one per product. Each call is fast (45–60ms), but the accumulated overhead of 50 sequential HTTP roundtrips adds up to over 3 seconds.

The fix: Replace individual calls with a batch API (GET /stock?skus=A001,A002,…,A050) or use a GraphQL-style query that returns all needed data in a single request.

Detecting N+1 Query Patterns in Traces

N+1 queries are one of the most common performance killers in microservices, and traces make them trivially easy to spot. The pattern appears as one initial query followed by N repetitive queries, and in the trace waterfall, it’s unmistakable.

Pattern: Classic ORM N+1

What the trace reveals:

[order-service: GET /orders] ─────────── 1,890ms

├── [PostgreSQL: SELECT * FROM orders WHERE status = 'active'
│    LIMIT 50] ── 15ms                              (1 query)
├── [PostgreSQL: SELECT * FROM customers WHERE id = 101] ── 8ms
├── [PostgreSQL: SELECT * FROM customers WHERE id = 102] ── 9ms
├── [PostgreSQL: SELECT * FROM customers WHERE id = 103] ── 7ms
│   ... (47 more identical-pattern queries)
└── [PostgreSQL: SELECT * FROM customers WHERE id = 150] ── 11ms

The trace shows 1 query to fetch orders + 50 individual queries to fetch each order’s customer. ORM lazy loading is the usual culprit. Each query is fast individually, but 51 database roundtrips add up to nearly 2 seconds.

How to spot N+1 patterns in your tracing tool:

• High span count on a single trace: A trace with 50+ database spans for a simple endpoint is almost always an N+1
• Repetitive db.statement patterns: Same query template with different parameter values
• Low individual span duration but high total trace duration: Each query is fast, but there are too many

The fix: Replace lazy loading with eager loading (JOIN or IN clause):

— Instead of 51 queries, use 1:

SELECT o.*, c.* FROM orders o

JOIN customers c ON o.customer_id = c.id

WHERE o.status = 'active' LIMIT 50

Pattern: Service-Level N+1 (Microservice Fan-out)

The N+1 pattern isn’t limited to databases. It manifests across service boundaries too:

[checkout-service: POST /checkout] ───────── 4,100ms
├── [cart-service: GET /cart/items] ── 35ms
│    Response: [{productId: "P1"}, ..., {productId: "P20"}]
├── [product-service: GET /products/P1] ── 120ms
├── [product-service: GET /products/P2] ── 135ms
│   ... (18 more calls)
└── [product-service: GET /products/P20] ── 128ms

The checkout service fetches cart items, then calls the product service individually for each item. The fix: implement a batch endpoint (POST /products/batch accepting a list of IDs) or use request collapsing.

Diagnosing Timeout Cascades and Retry Storms

Timeout cascades are among the most dangerous failure modes in microservices. Patterns like the circuit breaker exist specifically to contain them. A single slow dependency can cause cascading failures across your entire system, and traces are the fastest way to understand the chain reaction.

Pattern: Timeout Cascade

Symptoms in metrics: Multiple services show elevated error rates simultaneously. Latency spikes propagate across services.

What the trace reveals:

[api-gateway: POST /orders] ──────────── 30,012ms (TIMEOUT)
└── [order-service: POST /create] ─────── 30,005ms (TIMEOUT)
├── [inventory-svc: POST /reserve] ──── 30,001ms (TIMEOUT)
│     └── [PostgreSQL: UPDATE inventory ...] ── 30,000ms
│           otel.status_code: ERROR
│           exception.message: "Lock wait timeout exceeded"
└── [payment-service: POST /charge] (NOT REACHED)

The trace reveals the cascade: a database lock timeout in inventory causes inventory to time out, which causes order-service to time out, which causes the gateway to time out. Without the trace, you’d see three services all timing out and might investigate the wrong one first.

Key diagnostic signals in timeout traces:

• Span duration equals the configured timeout value exactly (e.g., 30,000ms), which confirms a timeout rather than slow processing
• otel.status_code: ERROR with timeout-related exception messages
• Child spans that were never started (like payment-service above), which confirms the timeout interrupted the flow
• Multiple parent spans with identical durations, meaning each parent waited for the full timeout of its child

Pattern: Retry Storm

Symptoms in metrics: Sudden traffic spike to a downstream service. Error rates increase rather than decrease.

What the trace reveals:

[order-service: POST /create] ─────────── 12,450ms
├── [inventory-svc: POST /reserve] ── 5,001ms TIMEOUT
├── [inventory-svc: POST /reserve] ── 5,002ms TIMEOUT (retry 1)
├── [inventory-svc: POST /reserve] ── 2,410ms TIMEOUT (retry 2)
│     exception.message: "Connection pool exhausted"
└── Result: ERROR "Failed after 3 retries"

The trace shows the order service retrying the inventory call three times. With 100 concurrent requests all doing the same, the inventory service receives 300 requests instead of 100, a 3x amplification. The connection pool exhaustion on retry 2 confirms the retry storm is making things worse.

Multi-layer retry amplification: When multiple layers retry, the multiplication compounds:

Gateway (3 retries) → Order Service (3 retries) → Inventory

= 3 × 3 = 9 requests to inventory per user request

Troubleshooting Error Propagation Across Service Boundaries

When an error surfaces at the API boundary, the root cause often lies several services deep. Traces let you follow the error propagation chain backwards from symptom to cause.

Pattern: Hidden Error Origin

Symptoms: Users see “Internal Server Error” on the checkout page. Logs show 500 errors cascading through services.

What the trace reveals in a single view:

[api-gateway: POST /checkout] ─ 500 Internal Server Error
└── [checkout-service: POST /process] ─ 500
├── [cart-service: GET /cart] ─ 200 OK (45ms)
└── [payment-service: POST /charge] ─ 500
└── [fraud-service: POST /evaluate] ─ 500
└── [ML model: POST /predict] ─ 503

exception.message: "Model server OOM:
cannot allocate 2GB for inference batch"

The trace cuts through four levels of error wrapping and reveals the actual root cause: the ML model server ran out of memory. Without the trace, the on-call engineer would start by investigating the checkout service, then the payment service, before eventually reaching the fraud detection service, potentially losing 30+ minutes following the chain manually.

Pattern: Silent Error Swallowing

Sometimes errors don’t propagate. Instead, they get silently caught, and the system returns degraded results instead of errors:

[product-service: GET /product/123] ─ 200 OK (890ms)
├── [PostgreSQL: SELECT ...] ── 12ms ─ 200 OK
├── [review-service: GET /reviews] ── 5,001ms ─ TIMEOUT
│     otel.status_code: ERROR
├── [recommendation-svc: GET /similar] ── 5,002ms ─ TIMEOUT
│     otel.status_code: ERROR
└── [Redis: SET product-cache:123] ── 3ms

The product page returns 200 OK, but the trace reveals two child services timed out. Metrics show 200 OK and ~900ms latency. Only the trace reveals the degraded user experience.

To catch this pattern: Filter traces by spans with otel.status_code: ERROR even when the root span shows success.

Spotting Connection Pool Exhaustion

Connection pool exhaustion is subtle. It doesn’t always produce errors, but it silently adds latency to every request as threads wait for available connections.

Pattern: Pool Wait Time

What the trace reveals:

[order-service: GET /orders] ───────── 2,340ms
├── [PostgreSQL: SELECT ...] ── 15ms
├── [gap: 1,800ms]  ← No spans, just waiting
└── [PostgreSQL: SELECT ...] ── 12ms

The telltale sign is gaps between spans, periods where the service is doing nothing visible. The 1,800ms gap between the first and second database query indicates the thread was waiting for a connection from the pool.

Diagnostic approach: Look for consistent gaps in trace waterfalls that don’t correspond to any span. When you see this pattern across multiple traces for the same service, check connection pool metrics (active connections, wait queue depth, pool size). The trace points you to the exact service experiencing pool pressure, and metrics confirm the diagnosis.

Diagnosing Cache Effectiveness Issues

Caches are supposed to reduce latency, but misconfigured caches can make things worse. Traces reveal cache behavior that’s invisible in aggregate metrics.

Pattern: Cache Miss Cascade

[product-service: GET /product/456] ─────── 1,250ms
├── [Redis: GET product:456] ── 1ms (MISS)
├── [PostgreSQL: SELECT * FROM products ...] ── 85ms
├── [Redis: GET product:456:reviews] ── 1ms (MISS)
├── [review-service: GET /reviews] ── 890ms
│     ├── [PostgreSQL: SELECT ...reviews...] ── 45ms
│     └── [PostgreSQL: SELECT ...users...] ── 830ms  ← Slow join
├── [Redis: SET product:456] ── 2ms
└── [Redis: SET product:456:reviews] ── 1ms

The trace shows: both cache lookups missed, forcing expensive database queries and service calls. The review service’s slow user join (830ms) is the real latency contributor, normally hidden behind a cache hit.

To monitor cache effectiveness with traces: Add custom span attributes for cache hit/miss status. Then in your tracing tool, filter and group by this attribute to see miss rates per operation, not just aggregate miss rates.

# Python example: Adding cache status to spans

from opentelemetry import trace

tracer = trace.get_tracer("cache-instrumentation")

def get_from_cache(key):

with tracer.start_as_current_span("cache.lookup") as span:

span.set_attribute("cache.key", key)

result = redis_client.get(key)

span.set_attribute("cache.hit", result is not None)

return result

Pattern: Cache Stampede

When a popular cache key expires, many concurrent requests simultaneously miss the cache and hit the database, a problem known as cache stampede. Looking at multiple traces for the same endpoint around the same timestamp reveals the stampede: each trace shows a cache miss, and database query durations increase progressively as the database becomes overloaded. All traces set the same cache key, resulting in redundant writes.

Troubleshooting Message Queue Issues

Asynchronous messaging adds complexity to troubleshooting because the producer and consumer execute at different times. OpenTelemetry’s context propagation via W3C Trace Context headers connects these spans into a single trace.

Pattern: Consumer Lag

[order-service: POST /orders] ─ (publishes to Kafka)

├── [Kafka: produce to orders-topic] ── 5ms
│     messaging.kafka.partition: 3
│     messaging.kafka.offset: 1847293
│
│  ~~~ 45,000ms gap (consumer lag) ~~~
│
└── [fulfillment-svc: consume from orders-topic] ── 120ms
└── [PostgreSQL: INSERT INTO fulfillment_queue] ── 8ms 

The trace links the producer span (order-service) to the consumer span (fulfillment-service) through propagated context. The 45-second gap between produce and consume timestamps reveals consumer lag. The consumer itself processes quickly (120ms), so the problem is in Kafka consumer group throughput, not processing logic.

Pattern: Poison Message / Dead Letter

[order-service: produce to orders-topic] ── 3ms

→ [fulfillment-svc: consume attempt 1] ── 15ms ── ERROR
│    exception.message: "Invalid product SKU format: null"
→ [fulfillment-svc: consume attempt 2] ── 12ms ── ERROR
→ [fulfillment-svc: consume attempt 3] ── 14ms ── ERROR
→ [dead-letter-queue: produce to orders-dlq] ── 4ms 

The trace shows a message being consumed, failing, retried twice, and finally sent to the dead letter queue. The exception message reveals the root cause: a null product SKU, likely a producer-side validation issue.

Using Trace-Based Alerting for Proactive Troubleshooting

Reactive troubleshooting (waiting for users to complain) isn’t good enough. Modern tracing tools support alerting on trace-derived signals that catch issues before they impact users.

Alert on RED Metrics Derived from Traces

Trace-Specific Alerts

Beyond RED metrics, some conditions are only visible through trace analysis:

• Span count anomaly: Alert when average spans-per-trace exceeds a threshold, catching N+1 regressions after deployments
• New error types: Alert when exception.type values appear that haven’t been seen in the last 7 days
• Missing service in trace: Alert when an expected service stops appearing in traces for a critical flow

Building a Troubleshooting Workflow with Sematext Tracing

Sematext Tracing provides the trace analysis capabilities needed to apply all the patterns described above. Here’s how to build an effective troubleshooting workflow.

Step 1: Start with the Service Overview

The Tracing Overview dashboard provides RED metrics (Rate, Error, Duration) across all instrumented services. This is your starting point: identify which service has elevated error rates or latency, and in which time window the problem started.

Step 2: Drill into the Trace Explorer

Use the Trace Explorer to filter traces by the affected service, time window, and error status. Sort by duration to find the slowest traces, or filter by otel.status_code: ERROR to find failures.

Key filters for troubleshooting:

• By service name: Isolate traces involving a specific service
• By minimum duration: Find traces exceeding your latency SLO
• By status: Filter for error traces only
• By operation: Focus on a specific endpoint or database operation
• By custom attributes: Filter by customer ID, order ID, or other business context

Step 3: Analyze the Trace Waterfall

Open the Trace Details view for a representative trace. The waterfall visualization shows the complete request flow with timing for each span. Look for the patterns described in this guide: long spans, gaps between spans, high span counts, and error spans.

Step 4: Set Up Alerts

Configure alerts on the RED metrics derived from your traces. Start with error rate and p95 latency alerts for your most critical services and endpoints, then expand to more specific alerts as you learn your system’s failure patterns.

Troubleshooting Checklist for Production Incidents

When an incident hits, use this trace-based workflow to minimize time-to-resolution:

1. Identify the scope: Check the service overview: is the issue isolated to one service or affecting multiple? Are error rates or latency elevated?
2. Find representative traces: Use the trace explorer to filter for affected traces. Sort by duration for latency issues, filter by error status for failures.
3. Read the waterfall: Open 3–5 representative traces. Look for: the longest span (bottleneck), error spans (root cause), gaps between spans (pool exhaustion), high span counts (N+1 patterns), and missing expected spans (service unreachable).
4. Check span attributes: Examine db.statement for bad queries, http.status_code for upstream failures, exception.message for error details, and custom attributes for business context.
5. Correlate with other signals: Jump to logs for detailed error messages and stack traces. Check infrastructure metrics for resource exhaustion. Look at deployment events for recent changes.
6. Verify the fix: After applying a fix, compare new traces against the problematic ones. Confirm the bottleneck span duration decreased, error spans disappeared, or the N+1 pattern resolved.

Summary

Distributed tracing transforms microservices troubleshooting from guesswork into systematic diagnosis. The patterns covered in this guide, including latency bottlenecks, N+1 queries, timeout cascades, retry storms, error propagation, connection pool exhaustion, cache failures, and message queue issues, account for the vast majority of production incidents in distributed systems.

The key is developing pattern recognition: learn what healthy traces look like for your critical flows, and the unhealthy patterns will stand out immediately during incidents. OpenTelemetry auto-instrumentation provides the data foundation, and a capable tracing backend like Sematext Tracing gives you the analysis tools to turn that data into fast resolution.

Next steps:

• Not yet instrumented? Start with How to Implement Distributed Tracing in Microservices with OpenTelemetry Auto-Instrumentation
• Need to optimize your instrumentation? Read OpenTelemetry Instrumentation Best Practices for Microservices Observability
• Want to extract higher-level insights? See From Raw Traces to Operational Intelligence (coming soon – contact us)
• Ready to try? Start your free Sematext trial, no credit card required

Start Free Trial

The post Troubleshooting Microservices with OpenTelemetry Distributed Tracing appeared first on Sematext.

In other words, how do you use it to make your life as an SRE/DevOps/biz person easier?

How do you have to set things up to truly solve production issues faster?

And does doing that require you to spend more money on observability or can you be smart about how you set things up so that OTel doesn’t break the bank?

While we were putting the finishing touches on Sematext’s OTel support, I asked one of my friends about their experience with and use of OTel in the context of questions like the ones above. The friend, the company, and the monitoring vendor they used will go unnamed, but here are the experiences and the practices my friend shared.

We’re a mid-sized org with about 30 frontend and backend developers. We know our way around observability, but have not adopted OpenTelemetry until late 2025. When we first rolled out OpenTelemetry in production, it felt like we had finally “done observability right.”

Every service was instrumented. OK, almost every service. ;)
Every request had a trace.
Every component had a metric.
Logs were nicely structured and correlated.

It was not quick and easy to set it all up, but we split the work among several team members and we did it.

However, within about two weeks we started observing – pun intended – problems:

• our storage bill doubled
• dashboards became slow
• our team stopped opening traces
• cardinality exploded
• and we started sampling randomly just to survive

It became apparent pretty quickly that just adopting OpenTelemetry is not automatically going to give us good monitoring. OpenTelemetry doesn’t give you a signal strategy. Out of the box, with naive usage, it just gives you a firehose and enables you to drown in your own telemetry more quickly.

We kept this new firehose on, but we had to quickly start making decisions around things like:

• what belongs in metrics
• what belongs in traces
• what belongs in logs
• and, perhaps most importantly, what should never be emitted at all!

How I Think About the Three Telemetry Signals Now

Early on, we treated metrics, logs, and traces as three different ways to describe the same thing. They are not. That was a mistake. They are different tools with different costs and different failure modes.

Now I think about them like this:

• Metrics answer: “Is the system healthy?” (both from tech/engineering perspective and business – we use metrics to understand the business side of things, too)
• Traces answer: “Where did the time go?”
• Logs answer: “What exactly happened?”

This separation of concerns feels simple and straightforward. As long as the observability tool you’re using has good UX for cross-connecting and correlating these signals, this separation should serve you well.

The Architecture We Ended Up With

This is the shape that finally worked for us:

The key idea is simple:
Applications emit everything. The collector acts as a filter, among other things, and decides what survives.

If you try to enforce strategy in application code, you’ll fail. Teams move too fast, especially now with AI. You need one place where you can say:

• keep error traces
• drop noisy attributes
• batch aggressively
• deduplicate
• enforce memory limits
• …

That place is the collector.

Metrics: What We Actually Trust During Incidents

The first real incident after we adopted OpenTelemetry was a checkout latency spike. Nobody opened a trace first. We all looked at metrics because our alert notifications pointed us there.

Metrics are what we trust when:

• we get an alert notification
• the CTO asks “are we down?”
• a deploy goes wrong

So we designed metrics to answer only three questions:

• How many requests?
• How many errors?
• How slow are they?

Sounds familiar? 👌Yes, RED!

Here’s a snippet from the relevant Python application.

Example (Python)

from opentelemetry import metrics

meter = metrics.get_meter("checkout")

request_counter = meter.create_counter(
    "http.server.requests",
    description="Total HTTP requests"
)

latency_histogram = meter.create_histogram(
    "http.server.duration",
    unit="ms"
)

def handle_request():
    request_counter.add(1, {"route": "/checkout", "status": "200"})
    latency_histogram.record(245, {"route": "/checkout"})

Hard Rule We Learned

It’s actually very simple: If a label (aka tag) can be different for every request, it does not belong in metrics.

These caused real problems for us:

• user_id
• email
• request_id
• order_id

You see where this is going? Yeah, cardinality. Cardinality tends to kill storage, makes certain UI elements unusable (think dropdowns with 1000+ values – fun!), etc.

See The First Production Surprise: Cardinality Explosions for more details on cardinality problems in OpenTelemetry.

Traces: How We Debugged Slow Requests

When it comes to traces you might think that they are like logs and you want to have them all so you can really dig in when you need to troubleshoot. However, for us at least, traces became useful only after we stopped trying to store all of them.

At first, we sampled at 100%. Meaning we didn’t sample at all.
Then we realized how much that was going to cost us.
Then we went for the other extreme and sampled at 1%.
But then we missed the interesting traces.

What finally worked was tail-based sampling:
We decide after the trace finishes whether it’s worth keeping.

Earlier, I mentioned a collector acting as a filter that decides what survives. This is a perfect example of that. Here’s the collector config for sampling.

Tail Sampling Config

processors:
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      - name: slow
        type: latency
        latency:
          threshold_ms: 500

So now what we have does this:

• slow requests survive
• failed requests survive
• boring 200ms health checks die

This changed traces from “expensive noise” into “high-signal debugging data.”

We also learned to be careful with attributes.
Anything that explodes into millions of values makes sampling useless.

For more details on sampling strategies, see our article The First Production Surprise: Cardinality Explosions.

Logs: The Last Mile of Debugging

We still rely on logs like we relied on them before, except with tracing in place oftentimes logs are what we read after traces tell us “this DB call is slow” and we need to know why beyond what we can see through traces themselves.

So the big change – the key – for us was correlating logs with traces.

Here’s how we do it with Python. You’d do something like this in any language. Note how we get the trace_id and span_id from the context and include it in the log event.

Python Logging with Trace Context

from opentelemetry.trace import get_current_span
import logging

logger = logging.getLogger(__name__)

span = get_current_span()
ctx = span.get_span_context()

logger.error(
    "payment failed",
    extra={
        "trace_id": format(ctx.trace_id, "x"),
        "span_id": format(ctx.span_id, "x"),
        "order_id": 1234
    }
)

Once we did this, debugging became a flow instead of a search:

Alert → metric → trace → log.
That’s the loop we optimized for.

The Collector Is Where Strategy Lives

Here’s a simplified version of the collector config we ended up with:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    limit_mib: 400
  batch:
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500

exporters:
  otlp:
    endpoint: backend:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

This let us:

• tune sampling without redeploying apps
• cap memory
• drop junk centrally

What Scaling Telemetry Really Means

When people say “scaling OpenTelemetry,” they usually mean handling “more traffic/observability data.”

Based on our experience, though, what we actually hit first was:

• cardinality
• storage
• query performance
• human attention

And thus, what scaling really meant for us in this context was:

• having fewer but better metrics
• having fewer but selectively chosen traces
• well structured logs that we can not just search but really slice and dice

What I’d Do Again (and What I Wouldn’t)

The Gist

OpenTelemetry is neither an observability strategy or solution. It’s a transport, a spec, an implementation in the form of SDKs. It’s just a tool. And one capable of drowning you in your own telemetry.

The strategy is being smart about how you set it up and how you use it. I strongly suggest counting on needing to spend some time on this. It pays off in the long run. Questions to answer:

• what questions you want answered
• what data you’re willing to pay for
• what engineers will actually use

Metrics tell me when things break.
Traces tell me where they break.
Logs tell me why they break.

Everything else …….send to /dev/null?

Start Free Trial

The post OpenTelemetry in Production: Design for Order, High Signal, Low Noise, and Survival appeared first on Sematext.