AI observability is often described as “monitoring LLMs”, but that definition is way too narrow. There is so much more that AI applications do that needs monitoring.

A production AI application can involve an LLM, multiple model providers, prompts, embeddings, vector databases, RAG pipelines, agents, tools, external APIs, memory, guardrails, GPUs, and traditional application infrastructure. An AI application can also be perfectly healthy from an infrastructure perspective while producing inaccurate, unsafe, expensive, or completely useless results.

All these are aspects of AI that could and should be observed.

In traditional IT observability we asked questions such as: Is the application available? Is it fast? Are there errors? Are resources healthy? For AI observability we have to go further: What did the AI do? Why did it do it? Was the result good? Was it safe? How much did it cost? And did it actually accomplish what the user wanted?

In this article, I explore the different dimensions of AI observability and the tools available for monitoring them.

1. AI Infrastructure

At the foundation is the infrastructure running your AI workloads.

For self-hosted models, this includes the usual infrastructure metrics (CPU, memory, disk, network, availability…), but also GPU utilization, GPU memory, accelerator utilization, model-loading time, batch sizes, queue depth, and concurrency. GPU utilization is particularly important because an expensive GPU that spends most of its time idle is an expensive resource that’s not doing much work.

Model-serving performance also introduces metrics that aren’t particularly common in traditional application monitoring. Time to first token (TTFT), time between tokens, tokens per second, queueing time, and total generation time can have a direct impact on user experience.

Tools such as Datadog, New Relic, and Dynatrace can monitor the infrastructure and application components surrounding AI workloads. For organizations running their own GPU infrastructure, specialized GPU monitoring can be added to provide deeper visibility into accelerator utilization and performance.

2. LLM Requests and Responses

Every call to an LLM is an important observable event.

You want to know which model was called, which provider served the request, how long it took, whether it succeeded, how many input and output tokens were consumed, what parameters were used, and why generation stopped. This is the AI equivalent of monitoring or tracing HTTP requests or database queries, but with considerably more semantic information.

This data is particularly useful when something changes. A new model version may increase latency. A prompt change may double token consumption. A provider may start returning more errors. A model that looked inexpensive during development may become surprisingly expensive in production.

Langfuse provides open-source LLM tracing with visibility into requests, responses, latency, tokens, and costs. W&B Weave provides similar tracing capabilities and connects LLM calls to evaluations and experiments, making it easier to understand not just how an individual request performed but whether changes to the application improved it.

3. End-to-End AI Traces

A user request rarely results in a single model call.

Consider a customer asking an AI assistant a question. The application might retrieve some documents, send them to an LLM, have the LLM decide to call a tool, invoke an external API, send the result back to the model, make another model call, and finally generate the answer.

A single LLM trace isn’t enough to understand this. You need an end-to-end trace showing the entire execution path and the relationships between its individual operations. That allows you to answer questions such as Why did this request take 15 seconds?, Which model call was responsible?, or Why did the agent call this API?

While there are commercial tools built around tracing and evaluation of AI applications, OpenTelemetry is also becoming increasingly relevant here. Its emerging GenAI conventions provide a standardized way to represent AI operations alongside traditional distributed traces.

4. Prompts and Context

With conventional software, developers generally think of code as the program.

With LLM applications, the prompt is part of the program.

Changing a system prompt, adding an instruction, modifying a few-shot example, changing the conversation history, or altering the model parameters can dramatically change application behavior. Consequently, observing which prompt was used and even which version of that prompt was used, is essential for debugging.

Context is equally important. To be meaningful, a production trace may need to show the system instructions, conversation history, retrieved documents, user context, tool descriptions, and other information supplied to the model.

This information can contain sensitive data, so prompt and context observability needs appropriate access controls, redaction, and retention policies.

Langfuse combines observability with prompt management and versioning, making it possible to associate production traces with the prompt version that generated them. Arize Phoenix also provides prompt management and experimentation capabilities. Sematext AI Agent Watch looks for Personally Identifiable Information (PII) in prompts.

5. AI Output Quality

This is where AI observability starts to diverge significantly from traditional observability.

A conventional application can return HTTP 200 and still be wrong, but in most applications a successful response provides a reasonably strong indication that the operation worked. With AI, a perfectly successful API request can produce a completely useless answer.

AI observability therefore needs to measure things such as correctness, relevance, completeness, helpfulness, instruction-following, and format compliance.

Some of these properties can be evaluated deterministically. Others require reference answers, human evaluation, specialized models, or LLM-as-a-judge techniques. The important point is that an AI system needs to be observed at the level of behavior and outcomes, not just infrastructure.

We need to evaluate the quality of AI outputs for the same reason that traditional information retrieval systems. For comparison, there we had very established metrics like precision, recall, mean reciprocal rank, and normalized discounted cumulative gain, etc.

Arize Phoenix provides evaluators that can be applied to production traces, including LLM-as-a-judge evaluations. W&B Weave similarly supports custom scorers and LLM judges, allowing teams to continuously evaluate production behavior.

6. Hallucinations and Groundedness

Few things are more important to monitor in generative AI than hallucinations.

As we’ve all seen, an LLM can produce an answer that sounds authoritative and convincing while being completely fabricated. Monitoring hallucinations means determining whether the answer is supported by the information available to the model and whether the claims it makes are consistent with that information.

This is particularly important for RAG applications. When an answer is wrong, you want to know whether the problem was retrieval, generation, or both. Did the system retrieve the wrong documents? Did it retrieve the right documents but fail to use them? Or did the model simply invent information despite having the correct context?

Ragas is an open-source framework focused heavily on evaluating RAG applications, including metrics around faithfulness and answer relevance. Arize Phoenix provides RAG and grounding-related evaluations that can be connected directly to application traces.

7. RAG and Retrieval

Similar to the above, for every retrieval operation, you may want to know the query that was generated, which documents or chunks were returned, their relevance scores, how many were retrieved, which ones actually made it into the prompt, and how long retrieval took.

This lets you distinguish between “the LLM couldn’t answer the question” and “the information required to answer the question was never retrieved.”

You can also monitor the components underneath retrieval: embedding generation, embedding-model versions, chunking strategies, vector search latency, filtering, reranking, and retrieval relevance. Changes to any of these can affect the final answer even when the LLM itself hasn’t changed.

Arize Phoenix provides tracing and evaluation for RAG pipelines, while W&B Weave can trace retrieval operations and evaluate the quality of retrieved information alongside the generated answer.

8. AI Agents

AI agents make observability considerably more complicated.

A conventional LLM application might send a prompt and receive an answer. An agent can decide what to do next, and that can include a whole bunch of things. It can call a tool, inspect the result, make another decision, call another tool, retry an operation, delegate to another agent, and eventually take an action.

For agents, observability therefore needs to capture the trajectory of an execution: what decisions were made, what tools were selected, what arguments were supplied, what results came back, how many iterations occurred, and how the agent ultimately reached its outcome.

This is one of the areas where AI observability is evolving particularly quickly and Sematext AI Agent Watch excels at capturing this.

9. Tool Calls and Actions

Tools deserve special attention because they turn AI from something that generates text into something that can do things, and that opens a whole new can of security worms.

An agent might have access to a database, browser, CRM, ticketing system, email, internal API, payment system, or cloud infrastructure. AI observability solutions have to be able to show which tools were invoked, when they were invoked, what arguments were passed, what they returned, and whether the action succeeded.

Tool monitoring is also where AI observability starts overlapping heavily with security.

An agent might make a technically valid API call but use the wrong API, access data it shouldn’t access, repeatedly retry an expensive operation, or perform an action that violates organizational policy. Monitoring tool usage creates both an operational record and a behavioral baseline.

Sematext AI Agent Watch, Langfuse, LangSmith, and Arize Phoenix all expose tool calls as part of AI execution traces. OpenTelemetry is also working toward standardized semantic conventions for tool execution, making tool calls increasingly interoperable with traditional distributed tracing.

10. AI Memory and State

Many agentic applications maintain state between interactions.

That state might contain conversation history, user information, summaries, retrieved facts, intermediate results, or persistent agent memory. Observability should show what was written to memory, what was subsequently retrieved, and how that information affected the agent’s behavior.

Memory creates some unusual failure modes. An agent can remember something that is wrong, retrieve stale information, accumulate excessive context, expose information from one user to another, or become increasingly expensive as its context grows.

Langfuse and LangSmith can incorporate application state and intermediate operations into their traces. OpenTelemetry’s emerging GenAI conventions also recognize memory operations as an observable part of AI application execution.

11. Token Usage and Cost

Every model call potentially has a price attached to it. This aspect of AI monitoring is probably the most familiar to everyone because we all experienced the need for this information the moment we started using AI tools and paying for them.

Observability should therefore track input tokens, output tokens, model-specific costs, cost per request, cost per user, cost per application, and cost over time. For agentic applications, it can be particularly useful to identify which steps in an agent trajectory are consuming the most tokens and money.

I should note that cost monitoring isn’t merely an accounting exercise. It is not of interest only to the CFO and the company management. Unexpected token consumption can reveal poorly designed prompts, unnecessarily large context windows, inefficient RAG pipelines, or agents caught in loops. As such, having insight into token usage and cost also matters for engineering teams. Especially if they want to preempt having their managers or CFOs on their backs.

Langfuse has token and cost tracking built directly into its LLM observability capabilities. W&B Weave exposes token usage and cost alongside traces and evaluations, making it possible to correlate cost with latency and quality. LLM token usage and cost monitoring is also built into Sematext AI Agent Watch, plus it allows pricing comparison across all known models, not just those used in your organization or by your AI applications.

12. Model and Prompt Drift

How many new models and model versions have we seen in the last 12 months? It would take me a while to list them all. Well, no, I could just ask AI, couldn’t I? But my point is that AI applications don’t stay static, far from it.

Models are upgraded. Providers change model versions. Prompts evolve. Retrieval data changes. Fine-tuned models are retrained. User populations change. All of these can and do alter application behavior.

Observability therefore needs to identify behavioral changes over time and help correlate those changes with what changed in the system. A sudden decline in answer quality may have nothing to do with application code, it could be the result of a model update or a change to the retrieval dataset.

W&B Weave provides versioning and experiment tracking for models, prompts, datasets, and other components of AI applications. MLflow provides open-source tracking of models, experiments, versions, and deployments, making it particularly useful for organizations operating both traditional ML and generative AI systems.

13. AI Security

AI introduces an entirely new class of security problems. We have already seen some, like zero-day vulnerabilities being discovered (which is all fine until the wrong people do that first).

These include prompt injection, jailbreaks, sensitive-data exposure, malicious inputs, unsafe outputs, data exfiltration, excessive permissions, tool abuse, and attacks against AI supply chains.

Security observability needs to monitor not only what users send to the AI system but also what the AI system attempts to do. For an agent, that can mean detecting unusual tool sequences, attempts to access sensitive resources, policy violations, or behavior that deviates from established patterns.

This AI security monitoring is one of the key foci of Sematext AI Agent Watch. Lakera focuses on runtime protection for generative AI applications, including threats such as prompt injection and sensitive-data exposure. NVIDIA NeMo Guardrails is an open-source framework for implementing programmable controls around AI applications. Protect AI focuses more broadly on security across the machine-learning and AI software supply chain.

14. AI Inventory and Governance

There is one final problem that becomes increasingly important as organizations deploy, knowingly or not, AI at scale:

Do you actually know what AI you have running?

An organization may have dozens or hundreds of models, LLM applications, AI agents, MCP servers, vector databases, and autonomous workflows. Some may have been formally approved. Others may have been deployed by individual engineering teams or employees without centralized visibility.

AI observability therefore increasingly overlaps with AI governance. Organizations need to know which models and agents exist, who owns them, which models they use, what data they can access, what tools they can invoke, and what policies govern their behavior.

This is especially important for autonomous agents. Traditional monitoring asks whether a service is healthy. AI governance asks a different question:

Is this AI allowed to be doing what it is doing?

15. Responsible AI

This may feel a little abstract to some readers, and it’s rather different from the more “techy” aspects of AI monitoring, but it is really important at the level of our global society. AI systems can fail in ways that aren’t captured by conventional notions of correctness.

Depending on the application, teams may need to monitor toxicity, harmful content, bias, inappropriate refusals, unfair treatment of different user groups, or systematic differences in model performance.

These properties are usually best monitored through continuous evaluation of production traffic, combined with segmentation and human review. The goal isn’t simply to find one problematic response. It is to discover patterns.

Arize Phoenix supports automated evaluations, human annotations, and user feedback, allowing safety and quality signals to be attached to individual traces. W&B Weave supports production monitors that can use evaluators and LLM judges to score live application behavior for characteristics such as correctness, helpfulness, and other application-specific criteria.

16. User Feedback and Business Outcomes

Ultimately, AI systems are deployed to accomplish something useful. Without that, what’s the point?

That might mean answering a customer’s question, resolving a support case, helping a developer write code, recommending a product, detecting fraud, or completing an agentic workflow. AI observability should therefore connect AI telemetry to real-world outcomes.

This is easier said that done, but at the end of the day we want to be able to answer questions such as: Did the user accept the answer? Did they ask the same question again? Did they correct the AI? Did they escalate to a human? Did the recommendation result in a purchase? Did the coding agent actually produce working code?

W&B Weave supports collecting human feedback and associating it with application traces. Arize Phoenix likewise supports human annotations and user feedback, allowing teams to connect subjective assessments with the exact AI execution that produced them.

Bringing AI Observability Together

These 16 dimensions can seem like a lot… and, in my opinion, they are indeed a lot. That’s because modern AI applications are a lot more complicated than simply calling an LLM API. The good news is that all of these are more or less optional. Sort of. ;)

I grouped all the angles of AI monitoring into four layers that I think are a useful way to think about the landscape:

First is infrastructure observability: GPUs, CPU, memory, network, availability, latency, throughput, and model-serving performance.

Second is AI execution and security observability: model calls, prompts, tokens, context, retrieval, embeddings, tool calls, agents, memory, and end-to-end traces.

Third is AI quality and safety observability: hallucinations, groundedness, correctness, relevance, bias, toxicity, security violations, and other behavioral characteristics.

Finally, there is outcome observability: cost, user satisfaction, task completion, conversion, business impact, and governance.

The first layer looks a lot like traditional observability. The higher you go, the more AI-specific the problems become. And that’s the fundamental difference between monitoring an AI application and monitoring a conventional application.

Traditional observability can tell you that the system is running, how fast (think performance metrics) and what it did (think log messages).

AI observability needs to tell you what the AI did, why it did it, whether it was correct and safe, how much it cost, and whether it accomplished what you wanted.

That is a much bigger problem and a much more interesting one. This is why we are seeing the mushrooming of AI monitoring solutions covering various angles of AI that I described here.

Start Free Trial

The post AI Observability: What Should You Monitor in AI Applications? appeared first on Sematext.

What are AI Agents?

Let’s define this first. For the purposes of this post we can think of them as applications that are “clients” of LLM models through providers of AI services, such as OpenAI, Anthropic, and so on. These AI agents are popping up like mushrooms after the rain and are becoming real production software. They are no longer just chatbots that answer questions, but systems that can call APIs, execute shell commands, read and modify files, query databases, browse the web, send messages, and delegate work to other agents.

That radically changes the monitoring problem and actually adds a whole new bucket of things one ought to monitor now.

When you run a normal application in production, you mostly want to know whether it is available, fast, and error-free. With AI agents running everywhere now, those questions still matter, but they’re no longer enough. An agent can be technically healthy while doing something completely wrong.

It can successfully call every API and still misunderstand the task. It can spend hundreds of dollars on model calls because it got stuck in a loop. It can select the wrong tool, expose information it shouldn’t, or take an action that nobody explicitly intended it to take.

And there is another problem that I think will become increasingly important: sometimes the organization may not even know that an AI agent is running on its infrastructure.

That is why I think “AI agent monitoring” is really becoming two related but distinct categories:

1. Monitoring the AI application you intentionally built, including traces, model calls, latency, cost, and quality.
2. Discovering and monitoring autonomous agents running on your infrastructure, including what they are doing and whether they create security or governance risks.

What is AI agent monitoring?

AI agent monitoring sits somewhere between several things we already know from software engineering, but it isn’t exactly the same as any one of them:

• Application performance monitoring (APM). Agents are still software, so you need to know whether they’re slow, failing, retrying excessively, or consuming unreasonable amounts of CPU, memory, or other resources. Traditional application monitoring covers part of this problem, but it usually can’t explain why an agent decided to take a particular action.
• Distributed tracing. A useful agent trace is more than a request flowing through microservices. An agent might call an LLM, decide to invoke a tool, inspect the result, call another model, delegate work to another agent, and repeat the process several times. You want to be able to reconstruct that chain and understand how one decision led to the next.
• LLM observability. This adds visibility into things traditional tracing doesn’t normally care about: prompts, model responses, token consumption, model selection, context size, and inference costs. This is essential when the model itself is part of the application’s decision-making process.
• Evaluation and testing. With traditional software, I can often write a deterministic test: given input X, expect output Y. Agents are different. They may take different paths and still produce acceptable results, or they may successfully complete every API call while producing a completely wrong answer. Monitoring should therefore include some way of measuring quality, not just whether the software crashed.
• Security monitoring. This is the area I think will become much more important as agents get more autonomy. Once an agent can execute commands, read files, access APIs, use credentials, browse the web, or communicate with other systems, you absolutely need visibility into what it is actually doing—not just what it was supposed to do, but what it did, which commands it called, while files it read, created, deleted or modified, which information is sent out and where, and so on.
• Infrastructure monitoring. Finally, there is a very practical problem that most AI observability tools don’t solve: how do I know which agents are running in the first place? An organization may have agents running on developer machines, servers, containers, CI/CD infrastructure, or Kubernetes clusters without a central inventory of them.

That combination is what makes AI agent monitoring different from simply adding another dashboard to an existing observability stack.

So for those two distinct categories I described earlier, there are also two fundamentally different questions we want monitoring to answer.

The first is:

Is my agent working correctly?

For that, we care about the agent’s execution path. What did it decide? Which model did it call? Which tools did it use? Where did it fail? How long did it take? How much did it cost? The first four bullets from the above list are all about this.

The second question is more uncomfortable:

What AI agents are actually running in my environment, and what are they doing?

That means being able to determine whether an agent is accessing files, executing shell commands, connecting to external systems, using credentials, or behaving in ways that violate security policies. That’s what the last two bullet points are about.

Why AI agent monitoring is important

The obvious reason for monitoring agents (in that first category mentioned above) is debugging. Agents are complicated systems. When something goes wrong, you need to understand what happened.

But I think the bigger reason will increasingly be security. That’s that second category.

We are giving software systems capabilities that we would traditionally give only to trusted applications or humans. Agents can access production systems, use credentials, execute commands, modify files, communicate with external services, and sometimes make decisions with relatively little human intervention.

That creates a muuuuuuuch larger attack surface.

Recent security incidents involving major AI organizations and AI infrastructure, including incidents affecting organizations such as OpenAI and the broader open-source AI ecosystem around Hugging Face, are reminders that AI infrastructure is not somehow separate from normal security problems. It contains credentials, proprietary models, sensitive data, source code, APIs, and increasingly powerful workloads. Keep in mind that whatever you read about incidents like the one from OpenAI/Hugging Face is likely not the full scope of the incident. Not everything is shared. Especially when it may hurt those organizations. Too much interest, too much money is at stake.

At the same time, research into autonomous agents is showing behavior that should make engineers pay attention. In controlled environments, agents have found unexpected ways around constraints, coordinated with other agents, left persistent information for other agents, exploited loopholes in their environments, and, in some evaluations, attempted to hide or preserve behavior when their goals conflicted with being shut down or modified.

Again, I think the problem is much bigger than what is publicly being shared. This is not science fiction any more, this is reality today. We must not ignore what these early incidents  are showing us.

We are building systems with:

• more autonomy,
• access to more tools,
• more persistent memory,
• more access to infrastructure,
• more ability to communicate,
• and fewer humans directly involved in every decision.

It is entirely reasonable to expect that these systems will eventually do things their designers did not anticipate. And let’s actually be real: they are really already doing that today.

Eventually, some of those things will be malicious, not necessarily because the underlying model suddenly develops malicious intent, but because an agent can be manipulated, compromised, given a harmful objective, or simply find an unexpected path toward accomplishing the objective it was given. It doesn’t have quite the full reasoning we have. It doesn’t really have empathy. It doesn’t really discern the way (good) humans do.

Security engineers have spent decades learning that you don’t wait until something becomes a widespread incident before you start monitoring it. In other words, even if you are not running any AI agents in your infrastructure today, I’m of the opinion that you should be prepared and put in place measures and tools to catch agents when they do show up. This is very much like e.g., virus or malware monitoring on your own computer. You don’t install an antivirus after it’s already on your computer, after it has already stolen your credentials, read and sent out sensitive data, spread to the rest of the network, encrypted your files and started blackmailing you, etc. No, smart people have antivirus running well ahead of such incidents and catch issues before they happen.

That’s how I think companies should approach AI agents.

What should an AI agent monitoring tool actually do?

If I were evaluating these products, I wouldn’t start with a long feature checklist. I’d start with a few practical questions. So back to my 2-category starting point:

For an agent application that my team built, I want to know whether I can follow an execution from beginning to end. I want to see the model calls, tools, sub-agents, latency, failures, and cost. I also want some way of determining whether the agent’s output was actually good.

For infrastructure-level monitoring, my questions are different.

Can the system tell me which agents exist without requiring every developer to manually register them?

Can it show me what those agents are doing?

Can I see shell commands, file activity, network connections, tool execution, and access to sensitive information?

Can I identify unusual or risky behavior?

Can I create policies and alerts?

The first category is mostly about observability and engineering quality.

The second is increasingly about security and governance.

I don’t think those two categories will necessarily converge into one product. In fact, I suspect most companies will use different tools for each.

1. Sematext AI Agent Watch

Description

Sematext AI Agent Watch addresses a different problem from most of the products in this article.

LangSmith, Langfuse, Phoenix, and similar tools, which I review below, generally start from the assumption that you know about the agent. You built it, you instrumented it, and want to understand its execution.

But not all AI agents running on your infrastructure will be your agents – the agents that you have built and instrumented. There will be third party agents that somebody has installed on your servers for who knows what reason. Thus, AI Agent Watch starts with a more fundamental question:

What AI agents are actually running on my infrastructure?

It is designed to first discover agents running on monitored hosts and then observe what they are doing. That can include agent sessions, shell commands, tool execution, file activity, network connections, and potential exposure of sensitive information.

I think this is a super important distinction. This is that second category from the beginning of the article.

Additionally, an AI agent, including your own, can be perfectly instrumented at the application level and still be a security problem. Conversely, an agent might be running somewhere in the environment without anyone having instrumented it at all. But you should still want to know about it and its activity, right?

This type of monitoring is about discovering autonomous software and understanding its real-world behavior.

Key features

• Automatic AI agent discovery
• Host-level monitoring
• Agent session tracking
• Shell command visibility
• Tool execution monitoring
• File read and write monitoring
• Network connection monitoring
• Sensitive-data detection
• Trusted and untrusted agent tracking
• Risk scoring
• Alert rules and reporting

Pros

• Solves a problem most LLM observability tools don’t address
• Can discover agents without application-level instrumentation
• Strong infrastructure and security perspective
• Useful for detecting risky behavior
• Focuses on real actions rather than only prompts and responses

Cons

• Not a replacement for detailed LLM tracing
• Not primarily an evaluation platform
• Most useful where you control or monitor the underlying infrastructure

Pricing

Sematext AI Agent Watch pricing is tied to the monitored infrastructure and Sematext plan. It starts at $1.68/host/month.

My Opinion

What I like: I think this addresses a problem that will become increasingly important: you can’t govern or secure agents if you don’t know they exist.

Traditional AI observability starts after someone deliberately instruments an agent. Infrastructure-level discovery starts earlier.

It asks:

What autonomous software is running, where, and what is it actually doing?

I think that will eventually become a standard security question.

What I dislike: I wouldn’t use it alone for agent quality, prompt debugging, or detailed model-level evaluation. For those problems, I would pair it with Langfuse, LangSmith, or Phoenix.

2. LangSmith

Description

LangSmith is one of the most complete AI application observability platforms. It covers tracing, evaluation, prompt management, datasets, experiments, production monitoring, and feedback collection.

Its biggest strength is that it makes complicated agent executions relatively easy to inspect. If an agent calls multiple models, invokes tools, uses retrievers, and delegates work to sub-agents, LangSmith gives you a structured view of that execution instead of forcing you to reconstruct it from logs.

It is particularly compelling for teams using LangChain or LangGraph, although it is no longer limited to those frameworks.

I see LangSmith as a platform rather than just a tracing tool. That’s useful if you want one place for much of your AI engineering workflow. The trade-off is that you’re adopting a fairly large proprietary platform, and I would think about that before making it the center of everything.

Key features

• Agent and LLM tracing
• Tool and sub-agent visibility
• Production monitoring
• Offline and online evaluation
• Datasets and experiments
• Prompt management
• Token and cost tracking
• User feedback
• Dashboards and alerting

Pros

• One of the best tracing experiences in the category
• Excellent LangChain and LangGraph integration
• Strong combination of observability and evaluation
• Mature production workflow
• Good cost visibility

Cons

• Proprietary platform
• Strongest integration is naturally within the LangChain ecosystem
• Self-hosting is not the default experience
• Pricing is complicated: costs can grow with both usage and team size

Pricing

LangSmith offers a free developer tier and paid plans that combine seats and usage. Pricing changes fairly regularly, so I would check the current pricing page before making a decision, particularly for production-scale workloads.

My Opinion

What I like: The tracing UI is genuinely useful. When debugging a complicated agent, being able to visually follow the execution tree saves a lot of time.

What I dislike: I don’t love the idea of tightly coupling my entire AI development workflow to one proprietary platform. If interoperability and deployment flexibility are important to me, I’d look closely at Langfuse or Phoenix as well.

3. Langfuse

Description

Langfuse is probably my default recommendation for a general-purpose engineering team that wants AI observability without committing too heavily to a single proprietary ecosystem. Keep in mind that when I refer to “AI observability” here I mean that first category described further above.

It provides the things I expect from an AI observability platform: traces, nested agent activity, sessions, token and cost tracking, prompts, datasets, experiments, and evaluations. The important difference is that Langfuse is open source and can be self-hosted.

That makes it particularly interesting for companies that have data residency requirements or simply don’t want every prompt and model response to live in someone else’s SaaS platform.

I also like its OpenTelemetry direction. I think standardization matters in this space, especially because AI frameworks are changing so quickly.

The downside is that flexibility comes with more choices. Langfuse doesn’t force one particular way of building your AI engineering workflow, which I see as an advantage, but some teams may prefer something more opinionated.

Key features

• LLM and agent tracing
• Agent graphs
• Sessions
• Token and cost tracking
• Prompt management
• Datasets and experiments
• Online and offline evaluations
• OpenTelemetry support
• Python and JavaScript SDKs
• Self-hosting

Pros

• Open-source core
• Strong self-hosting story
• Framework-agnostic
• OpenTelemetry support
• Good combination of tracing and evaluation
• Reasonable entry pricing

Cons

• Self-hosting means operational responsibility
• Requires more architectural decisions than highly opinionated platforms
• Evaluation still requires engineering work and good datasets
• Pricing units require some understanding at scale

Pricing

Langfuse has a free cloud tier, paid cloud plans, and a self-hosted open-source option. This is one of the things that’s good about it: small teams can get started cheaply, while larger organizations have the option to keep more control over their deployment.

My Opinion

What I like: This is probably where I would start for many teams. I like the combination of open source, OpenTelemetry, self-hosting, tracing, and evaluation.

What I dislike: It’s less of a “just follow this workflow and everything works” product. If my team wanted maximum convenience and was already committed to LangGraph, LangSmith might be easier.

4. Arize Phoenix and Arize AX

Description

Keep in mind that Dynatrace acquired Arize in August 2026. This means things may change in the future, but for now, this is how I see things:

Phoenix takes a more engineering-oriented approach to AI observability.

The open-source Phoenix project provides tracing and evaluation capabilities, while Arize’s commercial platform adds managed production monitoring and additional enterprise features.

What I find interesting about Phoenix is its emphasis on OpenTelemetry and open infrastructure. That makes it attractive if I don’t want my instrumentation strategy tied to a specific AI framework.

Phoenix is particularly useful for inspecting complex LLM and retrieval workflows and for building evaluation pipelines around them. The managed Arize platform makes more sense when you want those capabilities without operating everything yourself.

The main trade-off is that Phoenix feels more like an engineering platform than a polished, highly opinionated SaaS workflow.

That’s a good thing if your team wants flexibility. It’s less attractive if you just want to install an SDK and immediately have everything configured for you.

Key features

• OpenTelemetry-based tracing
• LLM and agent observability
• Retrieval and RAG inspection
• Evaluations
• Custom metrics
• Production monitoring through Arize
• Open-source deployment

Pros

• Strong OpenTelemetry support
• Open-source option
• Good fit for engineering-heavy teams
• Strong evaluation capabilities
• No immediate vendor lock-in

Cons

• More engineering effort than some SaaS alternatives
• Phoenix and Arize AX can be confusing as separate products
• Less turnkey for teams without observability experience
• Some uncertainty around the future of both Phoenix and Arize XA after acquisition by Dynatrace, known for being expensive and targeting large enterprises with deep pockets.

Pricing

Phoenix is available as open source. Arize also offers managed plans with usage-based pricing and enterprise options.

My Opinion

What I like: I like the architectural direction. OpenTelemetry and open-source instrumentation give me more confidence that I won’t have to completely redo my observability stack when the next AI framework becomes popular.

What I dislike: It’s not the product I would choose for a team that wants the simplest possible onboarding experience.

5. Braintrust

Description

Braintrust is the tool in this list I would look at first if my biggest concern was not debugging, but preventing AI quality regressions.

Tracing is available, but the center of gravity is evaluation. Braintrust is built around datasets, experiments, scorers, production evaluations, and measuring whether AI systems are actually improving.

That philosophy makes sense to me. My background is in text search where similar frameworks exist for pretty much the same reasons – evaluating the quality of results, tuning, feedback, etc.

A beautiful trace can tell me exactly what happened. It still doesn’t tell me whether the agent did a good job.

Braintrust makes evaluation part of the normal engineering workflow. I can define test cases, compare versions, create scoring functions, and evaluate production behavior.

The trade-off is that this approach requires investment. You need meaningful datasets and useful evaluators. The tool can’t invent a good definition of quality for your application.

Key features

• Agent tracing
• Datasets and experiments
• Custom scorers
• Automated evaluation
• LLM-as-a-judge
• Production quality monitoring
• Token and cost tracking

Pros

• Excellent evaluation workflow
• Strong support for regression testing
• Good production quality monitoring
• Useful experiment comparison

Cons

• More expensive than lightweight tracing tools
• Requires serious investment in evaluation design
• Proprietary platform
• Can be more than a small team needs

Pricing

Braintrust offers a free entry tier and paid plans for larger-scale evaluation and production use. Pricing is worth checking carefully because the platform makes the most sense when you are running a meaningful evaluation workflow rather than just collecting traces.

My Opinion

What I like: I agree with the underlying philosophy. AI quality needs to be tested continuously, not treated as something developers inspect manually when a customer complains.

What I dislike: If I only need tracing and debugging, I wouldn’t start here.

6. AgentOps

Description

AgentOps is more explicitly focused on the practical problem of understanding what an agent did. This is indeed useful. It sounds simple, but it’s not.

When an agent behaves unexpectedly, reproducing the problem can be difficult because LLM-based systems aren’t fully deterministic. AgentOps focuses heavily on tracing agent activity, tool calls, model usage, and the ability to inspect or replay sessions.

I particularly like the idea of replaying an agent’s execution. Traditional debugging tools have taught developers to expect breakpoints, stack traces, and reproducible failures. Agent systems don’t always provide that.

AgentOps tries to close that gap.

Compared with evaluation-first tools like Braintrust, I see AgentOps as more focused on debugging and operational visibility.

Key features

• Agent tracing
• Tool-call visibility
• Multi-agent monitoring
• Session tracking
• Replay and debugging
• Token and cost monitoring
• Logs and errors
• Framework integrations

Pros

• Strong focus on practical agent debugging
• Useful replay capabilities
• Good multi-agent visibility
• Broad ecosystem support
• Straightforward cost tracking

Cons

• Evaluation capabilities are not its strongest differentiator
• Proprietary platform
• Usage pricing needs to be monitored
• Doesn’t solve infrastructure-level agent discovery

Pricing

AgentOps provides a free tier and paid plans based on usage and organizational requirements.

My Opinion

What I like: I like the focus on debugging. When an agent does something weird, my first instinct is to ask what sequence of events led to that behavior.

What I dislike: For serious quality engineering, I would probably combine this type of tooling with a stronger evaluation platform.

7. W&B Weave

Description

Weave makes the most sense in organizations that are already using Weights & Biases for ML experimentation. It extends the W&B ecosystem into LLM applications and agents by providing tracing, evaluations, monitoring, experiments, feedback, and production analysis.

The advantage of this approach is continuity. A team that already uses W&B for models and experiments doesn’t have to introduce a completely separate platform for AI application monitoring.

The idea of monitoring production traffic using automated scoring and evaluators also makes sense. That is closer to how I think mature AI applications should operate: continuously checking whether behavior is degrading rather than waiting for users to notice.

The disadvantage is that if you are not already using W&B, the ecosystem advantage is much smaller.

Key features

• End-to-end tracing
• Token and cost monitoring
• Evaluations
• Experiment tracking
• Production monitoring
• Human feedback
• Automated scoring
• Safety and quality monitoring

Pros

• Strong evaluation and experimentation capabilities
• Good fit for existing W&B users
• Connects development and production workflows
• Supports multimodal AI workloads

Cons

• Most compelling inside the W&B ecosystem
• Broader platform than some teams need
• Not focused on infrastructure-level agent discovery

Pricing

Weave pricing depends on the broader W&B platform and organizational usage. I would check the current pricing directly before evaluating it.

My Opinion

What I like: If my organization already used W&B, I would absolutely evaluate Weave before adding another specialized observability vendor.

What I dislike: For a conventional software engineering team with no existing ML platform investment, it may feel like more ecosystem than necessary.

Comparison table

How I would choose between these tools

If I were building an agent application today, I would start by deciding which problem I actually need to solve, which of those two categories of AI monitoring I am primarily after.

If the problem is debugging agent behavior, I would look first at LangSmith, Langfuse, Phoenix, or AgentOps. (Category 1 below)

If the problem is measuring quality and preventing regressions, Braintrust, LangSmith, Langfuse, Phoenix, and Weave are stronger candidates. (Category 1 below)

If OpenTelemetry, open source, and deployment control matter most, I would look closely at Langfuse and Phoenix. (Category 1 below)

If my organization already runs heavily on LangChain/LangGraph, LangSmith would be the obvious first evaluation.

And if the question is:

What AI agents are running on my servers, containers, and infrastructure—and what are they actually doing?

Then I need something different from application-level tracing. That’s where infrastructure-level monitoring and agent discovery tools become relevant. (Category 2 below)

Category 1: AI application observability

For tracing, debugging, costs, and evaluations:

• Langfuse is my default general-purpose recommendation.
• LangSmith is the obvious choice for many LangChain/LangGraph teams.
• Phoenix is attractive if OpenTelemetry and open source matter.
• Braintrust is strongest when evaluations are central to your engineering process.
• AgentOps is particularly good for agent execution debugging.
• W&B Weave makes the most sense for organizations already invested in the W&B ecosystem.

Category 2: AI agent infrastructure and security monitoring

This is a different problem. Here I want to know:

Which agents are running on my servers, what are they doing, and are they creating risk?

For that problem, traditional LLM observability products generally aren’t enough because they typically depend on application instrumentation and focus on prompts, model calls, and traces.

That is where a tool such as Sematext AI Agent Watch fits.

My preferred architecture for a serious production environment would probably be two layers:

       AI Agent Application
               │
┌──────────────┴──────────────┐
│                             │
▼                             ▼
AI Observability              Infrastructure
Langfuse / LangSmith          AI Agent Watch
Phoenix / Braintrust          │
│                             │
▼                             ▼
Traces, Evals, Cost           Discovery, Files,
Quality, Latency              Commands, Network,
                              PII, Risk, Alerts

Feature comparison

Here is another way to compare tools reviewed in this article. You can clearly see the two categories of solutions addressing different angles of AI monitoring.

My final opinion

I don’t think there is a single best AI agent monitoring tool because the category is splitting into two layers.

The first layer is AI application observability.

This is where tools such as Langfuse, LangSmith, Phoenix, Braintrust, AgentOps, and Weave operate. They help me understand execution, latency, costs, prompts, tools, and quality.

The second layer is AI agent discovery, governance, and infrastructure monitoring.

This layer answers a different question.

Not:

“Why did my application agent make this decision?”

But:

“What autonomous agents are running in my environment, what capabilities do they have, and what are they actually doing?”

I think the first layer is already becoming a standard part of AI application engineering.

The second layer is earlier in its evolution. It’s only emerging. But I suspect it won’t stay that way for long. As agents become more autonomous and get access to more systems, companies are going to need an inventory of those agents. They will need to know which agents can execute commands, access files, use credentials, and communicate externally.

And eventually, monitoring alone won’t be enough. Organizations will want AI policies, AI agent governance, and enforcement around what agents are allowed to do.

So if I were designing a production AI observability today, I would probably use two layers:

An AI observability platform to understand whether the agent works correctly.

An infrastructure-level agent monitoring platform to understand whether the agent is doing something it shouldn’t.

Those are different problems and both (are going to) matter.

Start Free Trial

The post Best AI Agent Monitoring Tools in 2026: Reviews & Comparisons appeared first on Sematext.

After more than a decade of building Sematext, one thing stands out to me:

Many of the problems we are solving today are remarkably similar to the problems we were trying to solve 10 or 15 years ago.

We still want to know:

• What is happening in our systems?
• Why is this application slow?
• Where did this request spend its time?
• Which services depend on which other services?
• What changed?
• Where is the bottleneck?
• What is consuming the CPU?
• What is actually happening on the network?
• And, increasingly: what are our AI agents doing?

The questions have stayed with us. Ask any developers or sysadmins (that was the term back then) from the end of the last millenium and you’ll see they were asking the same questions 30+ years ago.

What has changed, sometimes dramatically, is the technology available to answer them. And new use cases have emerged (see towards the end of this post).

Looking back at Sematext’s own history is an interesting way to see this technology cycle play out.

Platform gravity

One of the things that has changed significantly over the last decade is the expectation that monitoring and observability capabilities should work together.

Today, every major monitoring vendor is a platform.

Metrics are not enough. Logs are not enough. Traces are not enough. Customers expect to be able to move between signals, correlate them, share context, and investigate problems without constantly switching tools and losing information along the way. That is now the default expectation.

But it wasn’t always like this. This was not the case as recently as 10-15 years ago.

Sematext was built around this idea early. Back in 2013 and 2014, we were already combining monitoring and log management in what would eventually evolve into Sematext Cloud. At the time, these were treated as separate product categories: New Relic was the leader in monitoring and Splunk in the log monitoring category.  For us, however, they were already parts of the same operational problem. For several years I was scratching my head and wondering why these big vendors attached only these individual observability dimensions and didn’t see what looked obvious to me, that these signals belong together:

• If a metric suddenly spikes, you want to see the logs.
• If an error appears in the logs, you want to understand what was happening with the infrastructure and application.

The value was not simply in collecting multiple kinds of data. The value was in connecting them. That basic idea has only become stronger over time.

As far as I am aware, Sematext was the first solution that combined metrics, logs, and later traces into a single observability platform, and later synthetic monitoring, user experience monitoring, infrastructure visibility, maps, profiling-related capabilities, and increasingly AI-related observability in one environment.

Things that have changed are architectures, data volumes, and terminology, but the underlying idea has remained remarkably stable:

Operational data becomes much more valuable when it can be understood in context.

Observability verticals

The rise of platforms does not mean the end of focused, single-purpose products. In fact, depending on the goal, there may be more opportunities than ever for specialists.

Platforms are powerful, but they have a natural disadvantage: they are large. Adding a completely new capability to an existing platform involves product strategy, engineering, storage architecture, data models, pricing, user experience, support, documentation, integrations, and the inevitable question of how the new capability fits with everything that already exists.

A specialist can simply focus on solving one problem exceptionally well. That creates an interesting dynamic. A startup can identify an emerging problem, move quickly, build deep expertise, and create technology that would take a larger platform years to reproduce.

Eventually, all platforms face build-versus-buy decisions and oftentimes buying is the obvious answer.

A very recent example is Dash0’s acquisition of Polar Signals. Polar Signals focused deeply on continuous profiling, including CPU, memory, and GPU/CUDA profiling. Dash0, meanwhile, is building a broader observability platform. The acquisition brings Polar Signals’ profiling technology and engineering expertise into that larger platform.

That is a model I expect we will continue to see:

1. A focused company identifies a new problem.
2. It goes deep.
3. It builds expertise and technology.
4. A larger platform recognizes that reproducing all of that internally would be expensive and slow.
5. The specialist becomes part of the platform.

We are seeing similar dynamics around AI. There is a great deal of innovation happening in narrowly focused companies building capabilities around AI agents, AI infrastructure, LLM observability, governance, security, coding agents, and autonomous operations.

Some of the specialist companies will become platforms themselves. Others may become highly valuable pieces of much larger platforms. Some will not make it.

The existence of platform gravity does not eliminate the opportunity for specialists. In many cases, it creates it.

Some recent examples of the AI acquisitions in the observability space:

• Dynatrace bought Arize AI (August 2026)
• Elastic bought Deductive AI (August 2026)
• Datadog bought Adaptive ML (June 2026)

Distributed tracing before “the three pillars of observability”

Distributed tracing is a good example of how terminology can arrive (and also evaporate!) after the technology.

Sematext added distributed transaction tracing in 2015. It was very different from what most people think of when they hear “distributed tracing” today. Our initial implementation focused on JVM applications and used Java bytecode instrumentation. The goal, however, was already familiar: follow transactions through an application and understand where time was being spent.

Today, we have OpenTelemetry and with that we have  standardized APIs and SDKs.

We have auto-instrumentation across many languages, sometimes even multiple choices, like in case of Go (see Zero-Code OpenTelemetry for Go: Runtime Instrumentation with OBI vs. Compile-Time Instrumentation with Otelc).

We have collectors, semantic conventions, OTLP, and increasingly broad ecosystem support.

But the fundamental problem has not changed.

We still want the answer to the same question:

Where did the request go, and where did it spend its time?

In 2015, the now-familiar “three pillars of observability” framing, referring to metrics, logs, and traces, had not yet become a part of the industry vocabulary. Today, the trifecta of these observability signals is so widely accepted that it is easy to forget how much of the ecosystem around these signals was still forming only a decade ago. And the name, “three pillars of observability” came and went. Nobody uses it any more. The “pillars” keep getting added as observability platforms widen their scope in an endless race to increase customer lifetime value, among other things.

OpenTelemetry itself provides an interesting perspective on how long technology maturation can take.

OpenTelemetry now feels as though it is reaching a much more mature stage in 2026. Its core signals are widely adopted, the tooling is substantially better, and the ecosystem around it is far more developed than it was just a couple of years ago.

But this did not happen overnight. The industry has spent well over a decade converging toward more standardized approaches to collecting and exchanging telemetry.

From AppMap to Service Map

In August 2015, Sematext built what we called AppMap. I dug out the original AppMap announcements from archive.org. The UI looks super retro 🙂

But behind that old interface was a problem that is still very much with us today:

How do you understand the relationships and dependencies inside a distributed system?

The technology we had available at the time was different from what we have at our disposal today, of course. There was no OpenTelemetry. There was no broadly adopted, vendor-neutral tracing standard. So we used the signals we could obtain: network packet analysis.

Fast-forward 11 years.

Today, Sematext has a slick Service Dependency Map, built on top of OpenTelemetry and modern distributed tracing. A bunch of things are different:

• The implementation is fundamentally different
• The instrumentation model is different
• The data model is different
• The backend is different
• Most obviously, the UI 😂

But at the core, both products are trying to answer the same set of questions. We still want and need to know the same:

2015                                     2026
AppMap                                   Service Map
-----------------------------------------------------------------
What talks to what?                      What talks to what?
What depends on what?                    What depends on what?
Where are dependencies?                  Where are dependencies?
Where might the bottleneck be?           Where is the bottleneck?

How we go about getting answers to these questions has changed, but the core problem remains.

Network Map: major technology change

Here is another example. We also built an early Network Map. We called it NetMap.

It was built so long ago that the original announcement is no longer available on our blog. Somewhere along the way, probably during one of those SEO and content-cleanup projects that every company eventually goes through, the original post disappeared.

The Wayback Machine still has an archived version of the old Network Monitoring product page from 2019, although the original NetMap itself was built several years before that.

Now, roughly a decade later, Sematext has a new Network Map.

The technology behind it is entirely different. The old and new versions are not incremental versions of the same implementation. The agent, the data collection technology, the backend, the frontend, they are all very different and, because of that, the capabilities and usability are substantially better.

The new Network Map is built using eBPF.

And this is where the technology cycle becomes particularly interesting (more towards the end).

eBPF gives us a fundamentally different way to observe what is happening inside modern systems. It allows us to create visibility that would have been much more difficult, or in some cases simply impractical, to build with the technologies available to us a decade ago.

And what are we doing with this, what problems are we solving, what questions are we answering? Well, we are once again answering questions such as:

• Which processes are communicating?
• Which services depend on each other?
• What is the network topology?
• Where are unexpected connections?
• What are the communication paths through the infrastructure?

In other words:

The implementation is radically different. The core operational question is not.

Profiling in 2016 vs profiling in 2026: OpenTelemetry

Another example is application profiling.

In March 2016, Sematext announced on-demand Java profiling. It was very different from what we think of as modern continuous profiling. It was limited to JVM applications. It was on-demand. You would run it for a few minutes to capture and investigate what was happening inside an application.

The goal? Understand where the application was spending its time and resources.

Fast-forward more than 10 years to 2026. In March 2026, OpenTelemetry Profiles officially entered public Alpha.

The goal is not just to define another data format. It is to make continuous, low-overhead production profiling a first-class, standardized observability signal that can be correlated with traces, metrics, and logs, and that can be used to profile all kinds for runtimes.

2016
│
├── On-demand profiling
├── JVM-focused
└── Run it when you need to investigate
 │
 ▼
2026
│
├── Continuous profiling
├── Multiple runtimes and environments
├── Low-overhead production collection
├── eBPF-based approaches
└── Standardization through OpenTelemetry

The need for finding bottlenecks inside applications running in production, but without having to attach computationally expensive and impractical profiles, has been around for decades. We finally have the means to do that. The open-source standards and technology simply needed time to catch up.

Incremental additions to observability

Looking back over all of this, I think there is a broader lesson.

Observability is often presented as a sequence of new categories.

• First metrics
• Then logs
• Then traces
• Then profiling
• Then eBPF (that provides a substantially better tech for these 4 telemetry signals)
• Then OpenTelemetry
• Then AI observability

And, yes, new technologies genuinely do create new capabilities. But they also repeatedly improve our ability to solve old problems.

Network visibility did not begin with eBPF. Distributed tracing did not begin with OpenTelemetry. Profiling did not begin with continuous profiling platforms. Service dependency visualization did not begin with today’s Service Maps.

The problems often existed first! The technology caught up later. And when new technology arrives, it often does two things at once:

1. It makes existing use cases dramatically better.
2. It makes entirely new use cases possible.

eBPF is a good example. It can be used to solve a whole set of diverse problems:

• It can help build a new generation of Network Maps
• It can also be used for profiling
• It can contribute to tracing and application visibility
• It can be used for security, network routing, filtering, etc.

And the same fundamental ability to observe what software is actually doing can be extended into completely new areas……. including AI.

The next frontier: observing AI agents

The AI era is upon us and AI agents introduce a new category of operational questions. It is no longer enough to ask the health of applications, or how much CPU they use, or what are the main bottlenecks, and so on. We now increasingly need to ask questions such as:

• Which AI agents are running in our environment?
• What are they doing?
• Which systems are they accessing?
• What tools are they using?
• What actions are they taking?
• Are they behaving according to policy?
• Are they creating security or operational risks?
• What sensitive data are they sending out to third party services?
• ….

I think we can all relate to this. That is why we recently introduced Sematext’s AI Agent Watch capabilities. This sort of functionality is desperately needed right around now, whether or not every one of us recognizes this yet or not.

And so the underlying technologies and techniques behind modern observability continue to find new uses. The same industry that spent years learning how to observe servers, processes, services, requests, and code is now applying similar thinking to AI agents.

And this is only the beginning.

Start Free Trial

The post What 13 Years of Building Observability Has Taught Us: The Problems Stay, the Technology Changes appeared first on Sematext.

Go has historically had a relatively straightforward but hands-on OpenTelemetry instrumentation model: add the OpenTelemetry SDK, initialize it, instrument the libraries you use, and create custom spans where application-specific context matters.

That approach still gives you the most control. But it is no longer the only practical option.

The manual instrumentation approach requires so much work from engineers that we decided to create a general AI Skill for instrumenting applications with OpenTelemetry SDK, regardless of the runtime/SDK.

Two newer approaches can instrument Go applications with little or no manual source-code instrumentation:

• OpenTelemetry eBPF Instrumentation (OBI) instruments applications at runtime.
• OpenTelemetry Go Compile-Time Instrumentation (otelc) injects instrumentation during the build.

Both are part of the OpenTelemetry ecosystem. Both can produce OpenTelemetry telemetry without requiring developers to manually wrap every HTTP handler, database call, gRPC client, or messaging operation.

Both can be described as “zero-code instrumentation”, but they solve the problem at completely different points in the software lifecycle.

OBI asks:

How can we observe this application without changing or rebuilding it?

Otelc asks:

How can we build this application with instrumentation already inside it without manually modifying its source code?

This article looks at both approaches from the perspective of a developer, SRE, DevOps engineer, or engineering manager who needs to make a practical decision about instrumenting a Go application.

The traditional way to instrument Go with OpenTelemetry

The conventional way to instrument a Go application is to explicitly add OpenTelemetry support to the application using the SDK for Go.

At a high level, that usually means:

1. Adding the OpenTelemetry Go API and SDK.
2. Configuring a TracerProvider.
3. Configuring exporters.
4. Instrumenting libraries such as HTTP servers and clients, gRPC, databases, and messaging clients.
5. Adding custom spans around important application operations.

A simplified example might look like this:

tracer := otel.Tracer("checkout")
ctx, span := tracer.Start(ctx, "reserve_inventory")
defer span.End()
if err := inventory.Reserve(ctx, order); err != nil {
  span.RecordError(err)
  return err
}

For a more complete example, see our Gin service instrumentation example: https://github.com/sematext/sematext-otel-onboarding/tree/main/go

This model has important advantages:

• You explicitly control where spans start and end.
• You can attach application-specific attributes.
• You can model important business operations.
• You can decide what should and should not become telemetry.

The downside is obvious: instrumentation becomes part of the application.

For a sufficiently large system, that can mean touching many services, maintaining instrumentation dependencies, reviewing instrumentation changes, and deciding how deeply each library and operation should be instrumented. This is the type of stuff we faced at Sematext when we said “OK, let’s go and instrument all our Go services now”.

A useful way to think about the landscape is:

OBI and Otelc occupy different places on that spectrum.

Note that manual instrumentation is far from obsolete, you and your team can still choose this approach. It’s just that the zero-code approaches provide another layer of options. As a matter of fact, as you read this article you will learn that manual instrumentation is still critical in certain situations and complements the auto-instrumentation approaches.

Why zero-code instrumentation is harder in Go

Automatic instrumentation is relatively familiar in languages with highly dynamic runtimes.

Java agents can modify or intercept bytecode. Python can wrap functions dynamically. Other runtimes provide mechanisms that make it possible to insert instrumentation after an application has been built.

But Go is different. A Go application is typically compiled into a native binary. Once the binary exists, there is no general-purpose equivalent of loading a Java agent and rewriting the application’s bytecode.

That creates a fundamental choice for automatic instrumentation:

Do you instrument the process from outside, or do you modify the application during compilation?

OBI takes the first approach and Otelc takes the second.

The architectures look like this.

OBI: runtime instrumentation

Otelc: compile-time instrumentation

This difference affects deployment, security, portability, ownership, and operational complexity.

Let’s look at each approach in detail.

OBI: Instrumenting a running Go process

OpenTelemetry eBPF Instrumentation (OBI) is the OpenTelemetry project’s eBPF-based automatic instrumentation technology.

It was originally based on technology from Grafana Beyla and is now developed as an OpenTelemetry project. OBI’s first OpenTelemetry release was announced in late 2025, and the project has continued to evolve since then.

We are big fans of eBPF at Sematext and have been using it for 10+ years now. When we first started instrumenting our Go applications we chose the OBI approach and included it in our OTel examples for instrumenting Go applications.

The central idea is simple:

Run instrumentation outside the application process and observe the application while it runs.

Unlike a traditional OpenTelemetry SDK integration, OBI does not require adding OpenTelemetry code to the application binary.

See also: OpenTelemetry eBPF Instrumentation documentation

How OBI works

OBI uses eBPF to observe application and system behavior. Depending on what is being instrumented, it can capture activity at protocol boundaries and use language-specific techniques, including user-space probes, or uprobes, for supported Go instrumentation.

A simplified flow looks like this:

         Client
           │
           │ HTTP request
           ▼
┌──────────────────────┐
│      Go service      │
│                      │
│   net/http handler   │◄───── OBI observes
│          │           │       supported operations
│          ▼           │
│     application      │
│       logic          │
│          │           │
│          ▼           │
│    database/sql      │◄───── OBI observes
│                      │
└──────────────────────┘
           │
           │ SQL
           ▼
       PostgreSQL
           │
           ▼
   ┌───────────────┐
   │      OBI      │
   └───────┬───────┘
           │
           ▼
    Trace + metrics

OBI can be deployed as:

• a standalone process,
• a Docker container,
• a Kubernetes sidecar,
• or a Kubernetes DaemonSet.

The exact deployment model depends on how broadly you want to instrument the environment.

What can OBI instrument in Go?

OBI supports a combination of protocol-level and Go library-level instrumentation.

The current OpenTelemetry documentation lists Go support for technologies including:

• net/http
• HTTP/2
• gorilla/mux
• Gin
• gRPC
• database/sql
• MySQL drivers
• PostgreSQL drivers
• Redis
• Kafka
• Sarama
• pgx

Compatibility depends on the specific library and version. For example, Go library-level instrumentation is documented for Go 1.17+, while some context propagation capabilities require Go 1.18+.

This is an important practical point. OBI is not simply “watching packets.”

For supported Go libraries, it can use Go-specific instrumentation to capture richer application behavior.

At the same time, it remains fundamentally an out-of-process instrumentation system.

What using OBI looks like

Imagine you have an existing Go service, say payment-api, and that it is already running in production.

As such, you may not want to:

• modify its source code,
• change its dependencies,
• rebuild it,
• or restart it purely to add instrumentation.

OBI can be deployed separately and configured to discover or target the application. This is particularly useful for existing workloads.

OBI’s runtime model is one of its strongest characteristics: it can observe applications without making OpenTelemetry instrumentation part of the application’s build artifact. The OpenTelemetry project describes OBI as out-of-process instrumentation that can provide telemetry without application code changes or application restarts in supported scenarios.

Advantages of OBI

1. No source-code changes

The obvious benefit is that developers do not need to edit the application.

There is no need to:

git clone --> add instrumentation --> test instrumentation --> commit --> build --> deploy

Instead, instrumentation can be introduced independently of the application code.

This is particularly attractive when:

• the application is maintained by another team,
• source code is unavailable,
• the application is legacy,
• you want to instrument an existing fleet,
• or the platform team owns observability deployment.

2. No application rebuild

OBI can instrument supported workloads without requiring you to produce a new binary. This is one of the clearest differences between OBI and Otelc.

Otelc, on the other hand, requires the ability to run the application through an instrumented build process.

3. Instrumentation can be centrally operated

OBI is well suited to an infrastructure-oriented operating model. For example, the platform team could choose to deploy OBI as a Kubernetes DaemonSet and have it handle the instrumentation of Go, Java, and Python applications. In other words, the application teams do not necessarily have to independently add and maintain instrumentation.

That can be valuable in organizations with many teams and inconsistent OpenTelemetry adoption.

4. It is not Go-specific

OBI can observe applications written in multiple languages. That means an SRE or platform engineering team can use a common instrumentation mechanism across a heterogeneous environment. For organizations operating Go, Java, Python, Node.js, NGINX, and other workloads, that can simplify initial telemetry coverage. This is one of the OBI aspects that we have benefited from at Sematext, as some of our legacy code still uses Java/Kotlin and Node.js.

The limitations and trade-offs of OBI

1. OBI depends on the runtime environment

OBI is fundamentally a Linux and eBPF-based technology.

That means your deployment environment must support the capabilities OBI needs.

The OpenTelemetry documentation describes OBI as a Linux process that can inspect other running processes and requires elevated privileges or the appropriate Linux capabilities, depending on deployment and configuration.

In containers and Kubernetes, this can become an architectural decision rather than a simple configuration change.

For example, some OBI deployments require or may use:

• privileged containers,
• CAP_SYS_ADMIN,
• CAP_PERFMON,
• host or shared process namespaces,
• access to /proc.

The exact requirements depend on the instrumentation and deployment model. Recent kernel security changes can also affect Go instrumentation because OBI uses uprobes for Go-specific instrumentation.

This does not mean OBI is inherently unsuitable for production.It means that the security model must be evaluated by the platform team.

Of course, if your applications don’t run on Linux then OBI is not an option for you at all until other platforms, like Windows, get the needed eBPF support. See https://github.com/microsoft/ebpf-for-windows for what Microsoft is doing about that for Windows.

2. Coverage depends on what OBI understands

OBI can automatically observe supported protocols and libraries, but it does not automatically understand arbitrary application code.

Consider:

func CalculateEnterpriseDiscount(
    customer Customer,
    contract Contract,
) (Discount, error) {
    // 500 lines of business logic
}

There is no general way for an external runtime observer to know that this function represents an important business operation. OBI can show what happens around it:

HTTP request
    │
    ├── database query
    ├── Redis lookup
    ├── gRPC call
    └── Kafka publish

But it cannot automatically know that CalculateEnterpriseDiscount is an important domain-level operation.

3. Automatic service names and routes may need tuning

Because OBI observes applications externally, automatically derived service names, routes, and URLs may not always match how your organization wants to identify services.

The OpenTelemetry documentation specifically calls out route configuration and decoration as something that should be reviewed when generating traces with OBI.

That means you should validate the resulting telemetry rather than assuming that automatic discovery will always produce exactly the naming and cardinality you want.

Otelc: Instrumenting Go during compilation

The second approach moves instrumentation from runtime to build time.

Otelc, the OpenTelemetry Go Compile-Time Instrumentation tool, modifies the Go build process so that supported instrumentation is injected while the application is being compiled.

The resulting application binary contains the instrumentation.

The normal Go build looks like this:

Source code ──► go build ──► Go binary

While with Otelc it looks like this:

Source code ──► otelc + go build ──► Instrumented Go binary

So if you choose Otelc you will not need to change the application source code, but you will need to alter the build process.

How Otelc works

According to the OpenTelemetry Go compile-time instrumentation documentation, Otelc:

1. Intercepts compilation using the Go toolchain’s -toolexec mechanism.
2. Matches packages and functions against instrumentation rules.
3. Injects lightweight hook points.
4. Links those hooks to OpenTelemetry instrumentation code.

The resulting binary contains the instrumentation, so there is no separate runtime instrumentation agent that needs to attach to the process. Operationally, this is simpler because there is no additional moving piece running in production.

Visually things look like this:

Go application (source) ──► Otelc (match rules, inject hooks, link OTel code) ──► Instrumented Go binary ──► OTLP

The Otelc project uses techniques including trampoline code injection and function hook mechanisms to connect instrumented functions with OpenTelemetry logic.

What does using Otelc look like?

A simple workflow can look like:

otelc go build -o myapp .

Alternatively, Otelc can be integrated with the standard Go toolchain using -toolexec.

A documented pattern is:

otelc setup
export GOFLAGS="${GOFLAGS} '-toolexec=otelc toolexec'"
go build -o myapp .

This can be useful when a build command is controlled by an existing Makefile, CI system, or another build tool.

Otelc can also be installed as a Go tool dependency in supported Go versions, allowing builds such as:

go tool otelc go build -o myapp .

Otelc also supports generating or maintaining instrumentation configuration based on the application’s dependency graph.

What can Otelc instrument?

The currently documented set of supported instrumentation includes:

• net/http
• gRPC
• database/sql
• Gin
• Redis
• MongoDB
• Kubernetes client-go
• OpenAI Go SDK
• Anthropic Go SDK
• Kafka
• AWS SDK for Go v2
• selected logging libraries for trace/span correlation

The supported set will continue to evolve, so it is worth checking the project’s current supported-library documentation before choosing it for a specific application.

Advantages of Otelc

1. No manual instrumentation changes

Developers do not have to manually add instrumentation to every supported library boundary.

The source can remain:

http.HandleFunc("/checkout", checkoutHandler)

rather than becoming:

handler := otelhttp.NewHandler(
    http.HandlerFunc(checkoutHandler),
    "checkout",
)
http.Handle("/checkout", handler)

The instrumentation is introduced during compilation instead.

2. No privileged runtime instrumentation process

Once the application has been built, there is no eBPF process that needs to attach to it.

This can make Otelc attractive in environments where:

• privileged containers are prohibited,
• eBPF is unavailable,
• security policy restricts process instrumentation,
• or platform teams do not want observability software attaching to production workloads.

The OpenTelemetry documentation explicitly identifies this as a use case for compile-time instrumentation.

3. Instrumentation can reach supported dependencies

Because Otelc participates in the compilation process, it can instrument supported third-party dependencies that are part of the application’s build. That is useful when you use a library that you do not own but still want to instrument.

For example:

Your application
       │
       ├── Gin
       ├── gRPC
       ├── database/sql
       ├── Redis
       └── AWS SDK

Otelc can apply instrumentation rules to supported parts of that dependency graph without requiring you to fork or edit those dependencies.

4. The build artifact contains the instrumentation

This changes who owns the operational problem.

With OBI: application deployment + runtime instrumentation deployment

With Otelc: only the instrumented application artifact

The instrumentation becomes part of the software artifact produced by the build. That can fit naturally into organizations where application teams already own their build and deployment pipelines.

The limitations and trade-offs of Otelc

Nothing in this world seems to come without downsides… let’s look at Otelc’s cons.

1. You must control the build

Otelc requires access to the build process because that is where instrumentation is introduced. This is the most important limitation. If you have a precompiled production binary but cannot rebuild it, Otelc is simply not an option.

2. The build pipeline becomes part of the instrumentation architecture

Adding Otelc is not the same as adding another environment variable.

You now need to think about:

• local developer builds,
• CI builds,
• release builds,
• test builds,
• reproducibility,
• dependency management,
• cross-compilation,
• monorepos,
• and build caching.

The good news is that Otelc is designed to work with the normal Go build workflow and documents approaches for integrating through go tool, direct build wrapping, and -toolexec.

But this should still be treated as a build-system change and tested accordingly.

3. Coverage is limited to available instrumentation

Like OBI, Otelc does not automatically understand every Go package. It needs instrumentation rules for the libraries and functions you want to observe.

If you use some framework that is not yet supported by the instrumentation, Otelc will not magically infer its semantics.

You may need to:

• add manual instrumentation,
• create instrumentation for that library,
• or accept that the library is not automatically traced.

Luckily, the Otelc project includes an instrumentation model and documentation for adding support for additional libraries, so this scenario can be handled.

4. Automatic instrumentation is still not the same as manual application instrumentation

Otelc can add spans around supported framework and library operations. It does not automatically know which parts of your business logic are important.

For example:

POST /checkout
       │
       ▼
ValidateOrder
       │
       ├── ReserveInventory
       │
       ├── CalculateDiscount
       │
       ├── ProcessPayment
       │
       └── CreateShipment

Automatic instrumentation may produce excellent visibility into:

HTTP server span
       │
       ├── SQL query
       ├── Redis operation
       ├── HTTP call to payment provider
       └── Kafka publish

But it may not tell you how much time was spent specifically in CalculateDiscount unless you explicitly instrument that operation.

OBI vs. Otelc: side-by-side comparison

The following table summarizes the practical differences.

OBI vs. Otelc: how to choose which one to use

A simple decision matrix

A quick way to determine if you should be considering OBI or Otelc is by considering the following scenarios and asking a few questions.

If your starting point is:

“I have a binary already running and I don’t want to rebuild it.”

Start with OBI.

Existing binary?
      │
      ├── Yes ──► OBI is the practical zero-code option
      │
      └── No

“I control the build but don’t want to modify the application source.”

Look at Otelc.

Control the Go build?
      │
      ├── Yes ──► Otelc is a strong candidate
      │
      └── No ──► Consider OBI

“I cannot run privileged instrumentation in production.”

Look at Otelc.

eBPF / runtime privileges allowed?
      │
      ├── No ──► Otelc
      │
      └── Yes ──► OBI or Otelc

“I need to instrument services in several languages.”

OBI may provide a better common platform-level approach.

Go + Java + Python + Node.js
             │
             ▼
            OBI

“I need to instrument important internal business operations.”

Neither automatic approach completely solves the problem. You will probably still want manual instrumentation. Ooops! ;)

A practical decision tree

Here is another approach, a decision tree, that will help you quickly see which approach is more suitable.

If you are deciding how to instrument a Go application, start with this:

                      ┌──────────────────────┐
                      │ Need OpenTelemetry?  │
                      └──────────┬───────────┘
                                 │
                                 ▼
                  ┌────────────────────────────────┐
                  │ Can you modify the application │
                  │ source code?                   │
                  └──────────┬─────────────────────┘
                             │
                ┌────────────┴────────────┐
                │                         │
               Yes                        No
                │                         │
                ▼                         ▼
        Manual instrumentation     Can you rebuild
        is available                the application?
                                      │
                           ┌──────────┴──────────┐
                           │                     │
                          Yes                    No
                           │                     │
                           ▼                     ▼
                     Consider Otelc        Consider OBI
                           │                     │
                           ▼                     ▼
                    Do you need             Does your runtime
                    domain-level spans?     support OBI/eBPF?
                           │                     │
                           ▼                     ▼
                  Add manual spans        Deploy and validate
                  where they matter       supported coverage

In practice, the decision often reduces to four questions.

Question 1: Do I control the build?

If yes, Otelc becomes an option.

If no, it does not.

Question 2: Can I run eBPF instrumentation in production?

If yes, OBI becomes an option.

If no, Otelc may be easier operationally.

Question 3: Do I need to instrument existing binaries?

If yes, OBI is the more natural fit.

Question 4: How much application-specific context do I need?

If the answer is “a lot,” neither zero-code approach is likely to be sufficient by itself. Plan for some manual instrumentation.

Why automatic instrumentation does not eliminate the need for manual instrumentation

This is perhaps the most important point in this entire article. It is tempting to think of automatic instrumentation as a replacement for manual instrumentation. In our experience, it usually is not.

Automatic instrumentation and manual instrumentation solve different problems.

Automatic instrumentation is excellent at finding and instrumenting common technical boundaries:

• HTTP requests,
• gRPC calls,
• database queries,
• Redis operations,
• messaging operations,
• cloud SDK calls,
• and other supported libraries.

Manual instrumentation is where you describe what your application actually does.

Consider a checkout service.

Automatic instrumentation may give you:

POST /checkout                           820 ms
│
├── SELECT customer                      12 ms
├── SELECT inventory                     18 ms
├── Redis GET                             3 ms
├── POST payment-provider               410 ms
└── Kafka publish                         8 ms

This is indeed already extremely useful and you should aim for this as your first step.

But your engineering team may care about something different:

Checkout
│
├── ValidateOrder
├── ReserveInventory
├── CalculateDiscount
├── ProcessPayment
└── CreateShipment

Those are domain operations. Neither OBI or Otelc can reliably infer that these operations are important simply by observing technical behavior.

The most effective approach is often a hybrid.

                HTTP request
                       │
                       ▼
        ┌──────────────────────────┐
        │ Automatic instrumentation│
        └────────────┬─────────────┘
                     │
              Application code
                     │
                     ▼
        ┌─────────────────────────┐
        │   Manual business spans │
        │                         │
        │ ReserveInventory        │
        │ CalculateDiscount       │
        │ ProcessPayment          │
        └────────────┬────────────┘
                     │
                     ▼
        ┌──────────────────────────┐
        │ Automatic instrumentation│
        │ SQL / Redis / gRPC / etc │
        └──────────────────────────┘

For OBI specifically, OpenTelemetry also provides the Go Instrumentation Auto SDK, which is intended to help integrate manually created spans with eBPF-generated spans and shared trace context.

That makes the hybrid model especially relevant:

This avoids two extremes:

Extreme 1: Instrument nothing automatically

Every team has to manually instrument every HTTP framework, database driver, messaging library, and client.

Extreme 2: Instrument everything automatically and assume the result is sufficient

You get infrastructure-level telemetry but may lack the domain context required to answer questions such as:

• Why is checkout slow?
• Which business operation failed?
• How much time is spent calculating pricing?
• Which customer workflow is affected?
• Did the payment provider fail, or did our own validation logic reject the request?

The best observability, as I hope I’ve illustrated so far in this article, usually combines the two approaches.

Conclusion

OBI and Otelc represent two fundamentally different approaches to zero-code OpenTelemetry instrumentation for Go.

OBI instruments from the outside.

It is attractive when you want to observe applications that already exist, especially when rebuilding or modifying them is difficult. It can be deployed independently of the application and can support a centralized, platform-owned instrumentation model.

Otelc instruments from the inside—during the build.

It is attractive when you control the Go build process and want supported OpenTelemetry instrumentation to become part of the resulting binary without manually modifying application source code. It also avoids the need for a privileged runtime eBPF instrumentation process.

Neither approach eliminates the value of manual instrumentation.

A practical architecture for many teams will look like this:

┌─────────────────────────────────────────────┐
│           Automatic instrumentation         │
│                                             │
│   OBI or Otelc                              │
│                                             │
│   HTTP • gRPC • SQL • Redis • Kafka • etc.  │
└───────────────────────┬─────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────┐
│           Manual instrumentation            │
│                                             │
│   Business operations                       │
│   Domain-specific spans                     │
│   Important attributes                      │
│   High-value application context            │
└───────────────────────┬─────────────────────┘
                        │
                        ▼
              OpenTelemetry backend

The practical goal should not necessarily be to choose one instrumentation method and use it everywhere. Instead, choose the method that best fits the part of the system you are trying to observe.

• Need visibility into existing workloads without rebuilding them? Start with OBI.
• Control the Go build and want instrumentation baked into the binary? Look at Otelc.
• Need detailed visibility into business operations? Add manual instrumentation.
• Need all three? A hybrid approach may be the most useful architecture.

For implementation details and current compatibility information, start with the authoritative project documentation:

• OpenTelemetry eBPF Instrumentation (OBI)
• OBI Go instrumentation compatibility and supported libraries
• OpenTelemetry Go compile-time instrumentation (Otelc)
• Otelc getting started guide and supported libraries
• OpenTelemetry Go Instrumentation Auto SDK

Start Free Trial

The post Zero-Code OpenTelemetry for Go: Runtime Instrumentation with OBI vs. Compile-Time Instrumentation with Otelc appeared first on Sematext.

Key Functionality to Compare

At the basic level, an SLO defines the reliability target for a service—say, 99.9% successful requests over 30 days. SLO monitoring turns that target into something actionable by tracking the underlying indicator, calculating error-budget consumption, and warning me when the budget is being burned too quickly.

The core functionality I look for is:

• Flexible SLIs based on availability, latency, errors, or custom metrics.
• Rolling and calendar-based time windows.
• Error budgets, including remaining budget and burn rate.
• Burn-rate alerts, ideally with multi-window alerting.
• Good integration with the telemetry I already collect—metrics, traces, logs, or synthetic checks.
• Enough flexibility to model real services, without forcing every SLO into a simplistic uptime template.

The rest of this comparison focuses on the practical trade-offs: how each tool models SLOs, what data it can use, how usable the UI is, and how much complexity I have to accept to get reliable alerting. Note that service-level objectives are one of those things that sound straightforward until you try to operationalize them, so in addition to reviewing SLO monitoring tools it’s important to spend some time thinking about the SLOs themselves.

Quick comparison

I reviewed 9 tools. Most offer SLO as part of their wider observability platform, while one is a pure synthetic monitoring and another a purely SLO-focused tool. Here is a very quick high level comparison. There are several additional tables below that I suggest you look through.

SLO Monitoring Tools: A Practical Comparison

SLO monitoring has become one of the more useful ways to answer a deceptively simple question: is my service actually reliable enough?

I don’t mean “are all the dashboards green?” or “did CPU stay below 80%?” I mean whether users are getting the level of availability and performance that the service is supposed to provide.

A Service Level Indicator (SLI) is the measurement. A Service Level Objective (SLO) is the target for that measurement over a period of time. If my API successfully serves 99.9% of requests over 30 days, the SLO might be 99.9% availability. The remaining 0.1% is the error budget—the amount of unreliability I can afford before missing the objective.

SLO monitoring matters because raw monitoring data does not tell me how much a problem matters. A service can generate thousands of errors and still be within its reliability objective. Conversely, a relatively short outage can consume a huge chunk of the error budget and require immediate attention.

The core capabilities I look for are:

• Flexible ways to define SLIs for availability, latency, errors, throughput, or custom business signals.
• Rolling and calendar-based evaluation windows.
• Accurate error-budget calculations.
• Burn-rate monitoring and alerting.
• Support for grouping, filtering, and managing large numbers of SLOs.
• Good integration with the telemetry I already collect.
• Automation through APIs, Terraform, or configuration-as-code.
• A practical path from an SLO violation to the logs, metrics, traces, or synthetic checks that explain it.

The tools below take noticeably different approaches. Some treat SLOs as one capability inside a larger observability platform. Others, especially Nobl9, treat SLO management as the product itself.

Tool Comparison

1. Sematext

Sematext takes a relatively straightforward approach to SLO monitoring. Its current SLO functionality is built around synthetic monitoring, allowing me to create objectives from existing HTTP and browser monitors rather than requiring new instrumentation or a separate telemetry pipeline. That makes it particularly suitable for services where the thing I actually want to measure is externally observable behavior: can the API be reached, does the website respond quickly enough, and can a browser complete a critical user journey?

The implementation tracks compliance over configurable windows and provides error-budget visibility and early warnings. Sematext also connects the reliability view with the rest of its monitoring stack, including logs, infrastructure monitoring, and tracing. The main trade-off is that this is not trying to be a universal, vendor-neutral SLO layer that can model every imaginable SLI from every backend. It is simpler and more opinionated.

Key features

• SLOs built from HTTP and browser synthetic monitors.
• Availability and performance-oriented objectives.
• Configurable compliance targets and time windows.
• Remaining error-budget tracking.
• Historical and live compliance views.
• Early warning and alerting before an objective is violated.
• Synthetic checks from multiple locations.
• Monitoring of APIs, websites, user journeys, and third-party dependencies.
• Correlation with other Sematext telemetry.

Pros

• Very little setup if synthetic monitors already exist.
• Good fit for externally visible services and user-facing reliability.
• Easy to understand without becoming an SRE research project.
• Useful for monitoring third-party APIs against expected reliability.
• Integrates with logs, metrics, tracing, and other Sematext capabilities.

Cons

• Less flexible than tools that can define SLOs directly from arbitrary metrics or multiple external data sources.
• Not as strong for organizations that want every SLO managed through Git and a formal SLO-as-code workflow.
• Synthetic-monitor-based SLOs are not a replacement for every internal service-level indicator.

Pricing

Sematext currently offers a 14-day free trial. Synthetic Monitoring starts at $2 per monitor per month on monthly pricing, or $1.80 per monitor per month with annual pricing. SLO functionality is associated with the synthetic monitoring workflow, so the practical cost depends primarily on the monitors required to measure the service.

My opinion

What I like is the lack of ceremony. If I already have HTTP or browser monitors, turning those measurements into an SLO is a pretty natural next step. I also like the fact that an external SLO can represent what users actually experience rather than what an internal metric claims is happening.

What I dislike is the relative lack of flexibility compared with something like Nobl9 or a metric-centric implementation. If I want to define complex internal SLIs across arbitrary telemetry sources, this would not be my first choice.

2. Datadog

Datadog has one of the more mature general-purpose SLO implementations. It supports metric-based, monitor-based, and time-slice SLOs, which gives me several different ways to model reliability. Metric-based SLOs work well when I can clearly define good and bad events. Monitor-based SLOs build on existing monitors, synthetic checks, or service checks. Time-slice SLOs are useful when reliability is defined as a metric satisfying a condition during discrete periods of time.

Datadog also provides error-budget and burn-rate alerting, SLO tagging, search, historical views, APIs, and Terraform support. If my infrastructure, APM, logs, and synthetics are already in Datadog, this is an obvious place to keep SLOs because the data is already there. The downside is the same one that applies to Datadog generally: the platform is broad, powerful, and easy to expand inside, but the pricing model can become difficult to reason about as more products and telemetry are added.

Key features

• Metric-based SLOs.
• Monitor-based SLOs.
• Time-slice SLOs.
• Availability, latency, and custom metric use cases.
• Error-budget tracking.
• Burn-rate indicators and alerts.
• Rolling windows.
• Grouped SLOs and tags.
• SLO search and management views.
• API and Terraform support.
• Integration with APM, logs, RUM, synthetics, and infrastructure metrics.

Pros

• Very flexible SLI modeling.
• Excellent fit if Datadog is already the telemetry platform.
• Good support for both event-based and time-based reliability measurements.
• Mature burn-rate alerting.
• Strong automation support.
• Easy to move from an SLO problem into the rest of the observability stack.

Cons

• Pricing can be complicated because SLOs are only one part of a larger set of billable products.
• The number of possible ways to model an SLO can be confusing for teams new to SRE practices.
• Monitor-based SLOs can introduce dependencies on the underlying monitor configuration.
• The best experience assumes the relevant telemetry already lives in Datadog.

Pricing

Datadog’s pricing is modular and depends on the products used to collect and evaluate the underlying telemetry. Host-based products, including infrastructure monitoring and some APM offerings, use different billing models, while other capabilities are billed according to consumption. Datadog provides public list pricing, but in practice I would model the cost based on the complete telemetry architecture rather than looking for a standalone “SLO price.”

My opinion

What I like most is flexibility. Datadog can handle the classic “good events divided by total events” model, monitor-based availability, and time-slice definitions without forcing me into a single interpretation of an SLI.

What I dislike is that I would hesitate to introduce Datadog purely for SLO monitoring. It makes the most sense when the organization is already committed to the platform. Otherwise, I am paying for and operating inside a much larger system than the specific SLO problem requires.

3. New Relic

New Relic’s Service Level Management is tightly integrated into the rest of the New Relic platform. I can create service levels ranging from relatively simple one-click configurations to more advanced and customizable definitions, then view them alongside applications, workloads, and other observability data. The product also provides alerts and analysis views for tracking reliability over time and investigating breaches.

The advantage is that SLOs are not isolated objects sitting in a separate reliability tool. They are part of the APM and observability workflow. That can be useful when the next step after discovering that an error budget is being consumed is immediately opening the affected service, transaction, trace, or other telemetry.

New Relic’s pricing is also different from the traditional per-host approach. The current public model combines data ingest with user or compute-based access options. That can work well, but I would spend time modeling access requirements before committing.

Key features

• Guided and advanced service-level creation.
• SLI and SLO definitions integrated with New Relic services.
• Reliability views across Navigator and Workloads.
• Alerting for degradation and breaches.
• Period-over-period analysis.
• Investigation workflows around SLO breaches.
• Integration with APM, infrastructure, logs, synthetics, and other New Relic capabilities.
• Support for both simple and more customizable SLO definitions.

Pros

• Strong integration with the rest of the observability platform.
• Relatively approachable SLO creation.
• Good fit for teams already using New Relic APM.
• Useful free tier for evaluation and smaller deployments.
• No need to count hosts as a primary pricing dimension.

Cons

• Pricing can become more complicated once data volume and platform-user requirements grow.
• The experience is best when New Relic is already the primary observability platform.
• Some organizations may find the user-access model less attractive than purely telemetry-based pricing.
• Less attractive as a standalone, vendor-neutral SLO layer.

Pricing

New Relic’s free tier includes 100 GB of data ingest per month, unlimited basic users, and one free full-platform user. Beyond that, public pricing lists original data ingest at $0.40/GB beyond the included allowance. User pricing varies by edition, while New Relic also offers a compute-based model for eligible customers.

My opinion

I like New Relic’s integrated approach. If I am already using New Relic for APM, I don’t want to export the same telemetry somewhere else just to calculate an error budget. Keeping the SLO next to the application data is operationally sensible.

What I dislike is the pricing complexity around data, user types, editions, and newer compute models. It is more transparent than some historical observability pricing models, but I would still build a realistic cost model before assuming the free tier or entry pricing reflects production costs.

4. Grafana Cloud

Grafana Cloud is particularly interesting to me because it fits naturally into Prometheus and OpenTelemetry-oriented environments. Grafana SLO provides a dedicated workflow for creating and managing SLOs, generating dashboards and alerts, tracking error budgets, and automating configuration through APIs and Terraform.

The biggest advantage is architectural familiarity. If my engineering organization already thinks in PromQL, metrics, recording rules, infrastructure-as-code, and Git-based workflows, Grafana SLO feels like an extension of the existing stack rather than a new conceptual layer. The product can generate supporting dashboards, recording rules, and alerting components instead of requiring me to hand-build everything.

The limitation is that Grafana SLO is a Grafana Cloud capability rather than a general feature of self-hosted open-source Grafana. That distinction matters if my primary reason for choosing Grafana is self-hosting and avoiding a managed SaaS dependency.

Key features

• Guided SLO creation.
• Metric-based SLIs.
• Error-budget tracking.
• SLO dashboards.
• Error-budget alerts.
• Generated recording rules.
• Generated alerting rules.
• API support.
• Terraform support.
• SLO-as-code workflows.
• Integration with the broader Grafana observability stack.

Pros

• Excellent fit for Prometheus-centric teams.
• Strong infrastructure-as-code story.
• Familiar workflow for teams already using Grafana.
• Good separation between raw telemetry and reliability objectives.
• Free tier and relatively accessible entry pricing.
• Less architectural lock-in than some all-in-one observability platforms.

Cons

• The managed SLO capability is specifically part of Grafana Cloud.
• Metric modeling still requires Prometheus/Grafana expertise.
• Usage-based pricing can become harder to predict as metric cardinality grows.
• The SLO implementation is less useful if my relevant data is not accessible through the supported Grafana Cloud metric workflow.

Pricing

Grafana Cloud has a free tier and a Pro plan starting at $19 per month plus usage. The pricing page currently includes 10,000 active metric series in the platform fee, with additional metrics starting at $6.50 per 1,000 series before volume discounts. Enterprise starts with a $25,000 annual spend commitment.

My opinion

I like Grafana Cloud when the organization already has Prometheus expertise. The SLO-as-code and Terraform story is especially important for teams that do not want critical reliability definitions to exist only as manually configured UI objects.

What I dislike is that I still need to understand the underlying metric model. That is not necessarily a flaw—SLOs should be based on carefully chosen indicators—but it means Grafana is not always the easiest tool for a team that wants a highly opinionated, guided reliability workflow.

5. Dynatrace

Dynatrace has a fairly powerful SLO implementation, especially for organizations already using its broader observability platform. I can create SLOs from predefined templates or define custom SLIs using DQL, which means the underlying indicator does not have to be limited to standard availability or latency metrics. Dynatrace can use data available through Grail, including metrics and other data types that can be queried into a time series. The current SLO experience includes error-budget tracking, visualization, and management through a dedicated application, while APIs and SDKs provide automation options. Dynatrace is particularly interesting for complex environments because its topology and entity model can provide useful context around the services being measured. The downside is that the platform is large and opinionated, so using it just for SLO monitoring would usually be excessive.

Key features

• Template-based SLO creation.
• Custom SLI definitions using DQL.
• Service availability and performance objectives.
• Infrastructure and Kubernetes-oriented templates.
• Error-budget tracking.
• SLO visualization and dashboard integration.
• Entity-aware observability context.
• API and SDK support.
• Access to multiple Grail data types for SLI definitions.

Pros

• Very flexible SLI definitions.
• Strong fit for large and complex environments.
• Good topology and entity context.
• Templates make common SLOs easier to create.
• Custom DQL opens up nontraditional SLI use cases.
• Good API and SDK support.

Cons

• A large platform if SLO monitoring is the only requirement.
• DQL adds another query language to learn.
• Pricing is not simple to evaluate from the SLO feature alone.
• The platform can feel more opinionated than Prometheus-centric alternatives.

Pricing

Dynatrace pricing is based on the broader platform and the products or capabilities being used rather than a simple standalone SLO price. I would treat SLO cost as part of the overall observability architecture and request a realistic quote based on data, monitoring scope, and the Dynatrace platform capabilities required.

My opinion

What I like is the flexibility. Being able to build an SLI from a custom DQL query means I am not limited to a small set of predefined reliability models. I also like the surrounding context: if an SLO is degrading, topology and entity information can help connect that reliability problem to the actual system.

What I dislike is the weight of the platform. If my team just wants straightforward SLOs on Prometheus metrics, Dynatrace would probably feel like bringing an entire observability platform to solve a narrower problem.

6. Elastic Observability

Elastic has one of the more flexible SLO implementations among full-stack observability platforms. I can create SLIs from APM availability or latency, synthetic availability, custom metrics, histogram metrics, timeslice metrics, or custom KQL queries against data in Elasticsearch. That is a much broader set of options than tools that restrict SLOs to predefined monitor types.

Elastic supports both rolling and calendar-aligned windows, occurrences- and timeslice-based budgeting, error budgets, and burn-rate alerting. The SLO overview also makes it possible to see historical SLI performance and budget consumption without manually assembling dashboards.

I think Elastic is particularly interesting for teams that already use Elasticsearch for logs or observability data and want to define SLOs from that data without exporting it elsewhere.

Key features

• APM availability and latency SLIs.
• Synthetic availability SLIs.
• Custom KQL-based SLIs.
• Custom metric and histogram metric SLIs.
• Timeslice and occurrences budgeting.
• Rolling and calendar-aligned windows.
• Error-budget tracking.
• Burn-rate alerts.
• Historical SLI and error-budget views.
• Dashboard integration.
• OpenTelemetry and Prometheus support across the broader observability platform.

Pros

• One of the most flexible sets of SLI types in this comparison.
• Can build SLOs from logs, metrics, APM, or synthetic data.
• Supports both rolling and calendar windows.
• Good support for error budgets and burn rates.
• Strong fit for teams already using Elastic.
• Useful combination of internal observability and external digital-experience monitoring.

Cons

• Requires learning Elastic’s data model and query language if I want to use the most flexible SLI types.
• Not a lightweight standalone SLO tool.
• SLO availability depends on the appropriate Elastic deployment and licensing.
• Self-managed deployments can require more operational work than SaaS-only alternatives.

Pricing

Elastic’s current serverless Observability Complete tier includes SLO functionality. Serverless pricing is usage-based: Elastic currently lists metrics ingest from $0.023/GB, other observability data from $0.09/GB, plus separate retention and egress charges. Synthetic monitoring is available as an add-on. I would calculate the real cost based on total telemetry volume rather than thinking of SLOs as a separately priced feature.

My opinion

What I like most is flexibility. Elastic lets me define an SLO from the data I actually have instead of forcing me to translate everything into one specific metric format. I also like the combination of APM, logs, synthetics, and SLOs in the same platform.

What I dislike is the complexity. Elastic can do a lot, but that also means I need to understand how my data is structured before I can take full advantage of the SLO functionality.

7. Honeycomb

Honeycomb takes a more engineering-centric approach to observability than traditional metrics-first platforms, and its SLO implementation fits that model. SLOs are built from events, which makes them a natural fit for tracing and high-cardinality telemetry. I can define what successful behavior looks like and then track the remaining error budget over the selected time period.

The alerting model is particularly interesting. Honeycomb supports both Exhaustion Time alerts, which estimate when the error budget will run out, and Budget Rate alerts, which trigger when the budget is being consumed faster than expected. The UI also provides a budget-burndown graph that helps tune alerts before blindly picking a burn-rate threshold.

For teams already using OpenTelemetry and distributed tracing heavily, Honeycomb is one of the more natural SLO implementations I would evaluate.

Key features

• Event-based SLOs.
• Error-budget tracking.
• Budget burndown visualization.
• Historical burn-rate analysis.
• Exhaustion Time burn alerts.
• Budget Rate burn alerts.
• Slack and PagerDuty notification support.
• Distributed tracing and OpenTelemetry integration.
• High-cardinality telemetry support.
• Service-level SLO workflows.

Pros

• Very natural fit for tracing and event-based telemetry.
• Excellent burn-alert model.
• Budget-burndown visualization helps tune alerts.
• Strong OpenTelemetry support.
• Particularly well suited to modern distributed systems.
• More engineering-focused than dashboard-heavy.

Cons

• Less attractive if the organization primarily thinks in Prometheus metrics.
• SLOs are not available on the free plan.
• The Pro plan includes only two SLOs.
• Teams unfamiliar with event-based observability may need to adjust their mental model.

Pricing

Honeycomb has a free plan with up to 20 million events per month and 100 million metric data points per month. The Pro plan starts at $150 per month and includes 2 SLOs. Enterprise pricing is custom and starts with significantly larger trigger and SLO allowances.

My opinion

I like Honeycomb because the SLO implementation feels connected to how engineers actually investigate modern distributed systems. If the SLO burns, I can work from the relevant events and traces rather than switching into an entirely separate monitoring model.

What I dislike is the relatively limited number of SLOs in the Pro plan. Two SLOs is enough for evaluation or a small service footprint, but it is restrictive if I want to make SLOs a standard part of every production service.

8. Checkly

Checkly represents a different approach from the large observability platforms. It is primarily a synthetic monitoring platform designed around developers and monitoring-as-code. I can define API checks, browser checks, Playwright test suites, uptime monitors, and multistep checks, then manage them through the Checkly CLI, Terraform, or Pulumi.

For SLO-style monitoring, this is useful when the thing I care about is externally observable behavior. Instead of defining availability from an internal metric, I can measure whether an API responds correctly or whether a user can actually complete a critical workflow.

I would not put Checkly in the same category as Nobl9 or Elastic when it comes to arbitrary SLI definitions. Its strength is narrower but useful: taking synthetic tests and treating them as production reliability signals that can be managed like code.

Key features

• HTTP, TCP, DNS, ICMP, and heartbeat monitoring.
• API and multistep checks.
• Browser checks using Playwright.
• Playwright Check Suites.
• Global and private monitoring locations.
• Automatic retries.
• Monitoring-as-code workflows.
• Checkly CLI.
• Terraform provider.
• Pulumi provider.
• Prometheus metrics export.
• Status pages and alerting integrations.

Pros

• Excellent developer experience.
• Strong monitoring-as-code support.
• Native Playwright integration.
• Useful for testing critical user journeys.
• Terraform, Pulumi, and CLI support.
• Good fit for CI/CD and production monitoring workflows.

Cons

• Less flexible for arbitrary internal metric-based SLIs.
• Synthetic checks can become expensive at high frequency and across many locations.
• Not a full observability platform.
• Best suited to externally observable services and workflows.

Pricing

Checkly has a free Hobby plan with 10 uptime monitors, 10,000 API check runs, and 1,000 browser check runs per month. The Starter plan starts at $24 per month, while Team starts at $64 per month. Enterprise pricing is custom. Pricing scales through monitor counts and synthetic check runs.

My opinion

What I like is the developer workflow. I would rather keep monitoring definitions in Git and deploy them with the rest of the application infrastructure than manually create hundreds of checks in a UI.

What I dislike is that it is easy to confuse “synthetic monitoring” with complete SLO management. Checkly is excellent when my SLI is based on externally observable behavior, but I would use another tool if I needed to define reliability objectives from arbitrary application metrics.

9. Nobl9

Nobl9 is the most specialized SLO product in this comparison. Rather than assuming that SLOs should live inside one observability backend, it acts as a dedicated reliability layer that can connect to multiple existing telemetry systems. Its platform includes error-budget alerting, composite SLOs, backtesting, service-health views, reporting, and a strong SLO-as-code workflow.

That vendor-neutral approach is its biggest differentiator. A large organization may have Datadog in one team, Prometheus in another, New Relic somewhere else, and cloud-native telemetry in yet another environment. Nobl9 is designed to put SLO definitions above those individual systems rather than requiring telemetry consolidation first.

It also has one of the strongest configuration-as-code stories through OpenSLO, YAML-based definitions, Git workflows, validation tooling, and automation. The trade-off is additional platform complexity and a separate product to operate.

Key features

• Vendor-neutral SLO management.
• Multiple telemetry integrations.
• Error-budget tracking and alerting.
• Composite SLOs.
• SLO backtesting.
• Service Health Dashboard.
• SLO annotations.
• Reporting.
• OpenSLO support.
• YAML and Git-based workflows.
• OpenSLO validation through the Oslo CLI.
• SLO-as-code automation.

Pros

• Purpose-built for SLO management.
• Works across heterogeneous observability environments.
• Excellent SLO-as-code support.
• Strong OpenSLO ecosystem involvement.
• Useful for large organizations standardizing reliability practices.
• Advanced capabilities such as composite SLOs and backtesting.

Cons

• Another platform to buy, integrate, and maintain.
• Probably excessive for a small team with a handful of services.
• Less compelling if all telemetry already lives comfortably in one observability platform.
• Public pricing is not as simple as self-service competitors.

Pricing

Nobl9 provides pricing through its sales process rather than publishing a simple per-monitor or per-host price. The pricing offering is aimed at selecting an option based on organizational requirements, so I would expect to request a quote for a real deployment.

My opinion

Nobl9 is the one I would look at if SLOs themselves are becoming a platform concern. If multiple teams use different monitoring systems and I need a consistent reliability model across all of them, a dedicated abstraction layer makes sense.

What I dislike is the obvious trade-off: if I only have 10 services and everything already lives in Grafana, Datadog, or New Relic, adding another product may solve a problem I don’t actually have.

Feature comparison

Here is a product-feature matrix for all SLO monitoring tools we are comparing here.

* Checkly is the outlier in this table. I would describe it as a synthetic monitoring and monitoring-as-code tool rather than a full native SLO management platform. It can measure availability and latency and can be used to implement reliability targets from checks, but it does not provide the same general-purpose SLI, error-budget, and multi-window burn-rate model as Datadog, Grafana Cloud, Dynatrace, Elastic, Honeycomb, or Nobl9.

This distinction is useful because it prevents the comparison from making Checkly look weaker at something it is not primarily designed to do. Its real strength is defining production checks as code and using them to monitor APIs and critical user journeys.

Which SLO monitoring tool would I choose?

My choice would depend less on the number of features in the product and more on where my telemetry already lives and what I am actually trying to measure.

If I had to narrow the list down based on the problem rather than the vendor:

The important distinction for me is that Checkly and Sematext are strongest when the SLI represents externally observable behavior, while Datadog, New Relic, Grafana Cloud, Dynatrace, Elastic, and Honeycomb can build SLOs from deeper application telemetry. Nobl9 is different again: its main value is separating the SLO/reliability layer from the underlying observability system.

So I would not pick a winner based on a feature checklist. I would first decide where the SLI should come from, and then choose the tool that makes that workflow the least painful.

Final thoughts

The most important thing I have learned about SLO monitoring is that the tool is rarely the hard part. The difficult part is choosing an SLI that actually represents user experience and setting an objective that is neither meaningless nor impossible.

A monitoring tool can calculate an error budget perfectly and still give me the wrong answer if I am measuring the wrong thing.

So before comparing vendors too deeply, I would start with a few concrete questions:

1. What user behavior am I trying to protect?
2. What counts as a good event and a bad event?
3. Should every request count equally, or is time-based availability more appropriate?
4. How much unreliability can the business actually tolerate?
5. What should happen when the error budget is being consumed too quickly?
6. Where will engineers go next to investigate the problem?

Once those answers are clear, the choice usually becomes much easier. For most teams, I would strongly prefer using the SLO capability already available in the observability platform they trust—unless they have a real need for vendor-neutral, organization-wide SLO management.

Start Free Trial

The post Best 9 SLO Monitoring Tools in 2026: Review and Comparison Tables appeared first on Sematext.

That shift is why network monitoring in 2026 looks very different from a decade ago. Traditional SNMP-based tools still have a place, but cloud-native teams need service-aware visibility, automatic topology discovery, and tight correlation with logs, metrics, and traces. The right tool can cut hours off incident response, surface cost leaks, and replace the architecture diagrams that are always getting outdated that nobody wants to maintain.

This guide compares the 12 best network monitoring tools available in 2026. For each one, you will find the features that matter, honest pros and cons, real pricing, and the kind of team it fits best.

What is Network Monitoring?

Network monitoring is the continuous practice of collecting data about network devices, traffic, and connections to detect performance issues, outages, and security events before they affect users. The data can come from many sources: SNMP polling for device health, NetFlow or sFlow for traffic analysis, packet captures for deep inspection, eBPF for kernel-level connection visibility, and synthetic probes for path validation.

The category has expanded beyond LAN and WAN. In 2026, network monitoring also covers:

• Pod-to-pod and node-to-node traffic inside Kubernetes clusters
• East-west traffic between microservices
• Cross-region and cross-cloud connectivity costs
• Connections to managed databases, message queues, and SaaS APIs
• Suspicious outbound traffic that might signal a compromise

Good network monitoring helps teams answer questions like: which services are talking to which databases right now, where is traffic getting slow, which connections are eating our cloud egress budget, and what process on which host opened that unexpected outbound connection.

What to Look for in a Network Monitoring Tool

Before comparing tools, here are the criteria that matter most in 2026.

Topology Discovery and Visualization

A good tool draws the map for you. Static Visio diagrams go stale the moment someone deploys a new service. Look for automatic discovery and a real-time topology view that updates as your infrastructure changes.

Kubernetes and Cloud-Native Support

Containerized workloads are short-lived and dynamic. Tools designed for static IP-based device lists struggle here. Native Kubernetes support, pod-aware identity, and service detection are now table stakes for any team running clusters.

Protocol Visibility

Knowing that two hosts exchanged 500 MB tells you less than knowing that an application server made 12,000 PostgreSQL queries to a specific database. Tools that detect protocols (HTTP, PostgreSQL, MongoDB, Kafka, Redis, and others) give you context, not just byte counts.

Auto-Detection of Services

Manually labeling every host and process does not scale. The best tools recognize databases, caches, message queues, web servers, and runtimes automatically based on process names, container images, network signatures, and orchestrator metadata.

Correlation with Logs, Metrics, and Traces

A network alert in isolation is not very useful. Tools that let you pivot from a network connection to the related logs, application metrics, or distributed traces dramatically reduce time to root cause.

Alerting and Thresholds

Alerts that fire on every minor blip get ignored. Look for tools that support tunable thresholds, baseline-aware anomaly detection, and clear escalation paths.

Pricing Transparency

Network monitoring is one of the categories where cost surprises are most common, especially with consumption-based pricing on flow data or device counts. Predictable, published pricing matters more than free tiers that turn into bill shock at scale.

The 12 Best Network Monitoring Tools in 2026

1. Sematext Network Map

Best for: Cloud-native teams that want eBPF-powered service and infrastructure visibility inside a full observability platform

Sematext Network Map gives you a real-time, visual representation of your entire infrastructure topology. Using eBPF-powered network insights collected by the Sematext Agent, you can see how services, pods, containers, and processes communicate across Kubernetes clusters and standalone hosts without configuring anything beyond installing the agent.

Instead of polling devices and inferring relationships, Network Map shows the actual connections happening right now. When something breaks, you can immediately see which services are affected and trace the problem to its source. Two complementary views, Services View and Infrastructure View (see details in the docs), let you flip between application-level dependency analysis and Kubernetes infrastructure drill-down.

Key Features:

• eBPF-based connection capture: Kernel-level visibility into every connection, no port mirroring or packet capture required. Next to zero overhead.
• Two views, one map: Services View for application dependencies, Infrastructure View for cluster, node, pod, container, and process drill-down
• Automatic service detection: Over 100 service types recognized out of the box, including PostgreSQL, MySQL, MongoDB, Redis, Kafka, RabbitMQ, Elasticsearch, Nginx, HAProxy, Envoy, and more
• Protocol-aware connections: Detects HTTP, PostgreSQL, MongoDB, Kafka, Redis, and dozens of other protocols with traffic volume per connection
• Key metrics: network latency, round trip time (RTT), packet loss, retransmissions, CPU, memory, disk I/O, HTTP latency, HTTP response codes, etc.
• Custom thresholds: Configurable warning and critical levels for CPU, memory, network I/O, and disk I/O so the map highlights what matters in your environment
• Color-coded health: Green, yellow, and red service cards and connection lines make problems visible at a glance
• Kubernetes-native: Cluster to node to pod to container to process navigation with namespace, deployment, and workload context
• Standalone host support: Works equally well for VMs and bare-metal servers, not just containers
• Filtering and search: Quickly focus on a namespace, service type, host, or specific service when the topology gets dense
• Unified observability: Pivot from network connections to logs, metrics, traces, and synthetic checks in the same UI

Pros:

• Truly cloud-native and Kubernetes-first, designed for dynamic infrastructure rather than retrofitted onto SNMP foundations
• eBPF means no application code changes, no sidecars, and no instrumentation work, and next to zero overhead
• Service detection works automatically for the technologies most teams actually run
• Integrated with the rest of the Sematext platform, so traces, logs, and metrics live next to your topology
• Significantly cheaper than the major enterprise observability platforms
• Useful for discovering unknown dependencies, suspicious connections, data exfiltration patterns, and cross-region cost leakage
• Replaces stale architecture diagrams with a live picture of what is actually running

Cons:

• Newer product compared to legacy NMS incumbents, less brand recognition among traditional network teams
• Focuses on host, container, and service visibility rather than deep packet inspection or SNMP-style switch and router polling

Pricing: Starts at $1.68 per host per month. Network Map scales with your Sematext Infrastructure Monitoring plan and retention. 14-day free trial, no credit card required.

Best For: DevOps and SRE teams running Kubernetes or hybrid cloud infrastructure who want service-aware network visibility tightly integrated with logs, metrics, and traces. Teams that find traditional NMS tools too device-centric and modern APM platforms too expensive.

Get started with Sematext Network Map or check out Network Map docs.

2. Datadog Network Monitoring

Best for: Enterprises already on Datadog who want network visibility inside the same platform

Datadog offers two related products: Network Performance Monitoring (NPM) for host-to-host and pod-to-pod traffic analysis, and Cloud Network Monitoring for cloud provider flow logs and load balancer telemetry. Both plug into the Datadog platform alongside APM, logs, and infrastructure monitoring.

Key Features:

• Host and Kubernetes pod-level traffic analysis via the Datadog Agent
• VPC flow log analysis for AWS, Azure, and GCP
• DNS monitoring with query-level visibility
• Cloud load balancer and gateway telemetry
• Integration with Datadog APM, logs, and infrastructure metrics

Pros:

• Strong if you already use Datadog for everything else
• Detailed flow analysis with rich filtering
• AI-assisted anomaly detection through Watchdog

Cons:

• Each Datadog product is billed separately, and bills add up quickly at scale
• “Bill shock” is a recurring complaint from customers
• Network products require agent and integrations setup separately from core APM

Pricing: Network Performance Monitoring starts at $5 per host per month. Cloud Network Monitoring is priced per analyzed flow. Costs scale with hosts, flows, and retention.

Best For: Mid-size to large enterprises already standardized on Datadog who can absorb the platform-wide cost.

3. Dynatrace

Best for: Large enterprises wanting AI-driven full-stack observability with network context

Dynatrace covers network visibility as part of its broader Davis AI observability platform. The OneAgent captures process-level network connections alongside code-level traces and infrastructure metrics.

Key Features:

• OneAgent captures host, process, and connection-level data automatically
• Smartscape topology view combining services, processes, and infrastructure
• Davis AI for root cause analysis across signals
• Multi-cloud and Kubernetes coverage
• Strong support for traditional enterprise workloads like SAP and mainframe

Pros:

• Truly automatic instrumentation with minimal manual configuration
• AI correlation across network, traces, logs, and metrics
• Strong enterprise governance and compliance features

Cons:

• Premium pricing, often the highest in the market
• OneAgent has a significant resource footprint compared to lightweight collectors
• Steep learning curve across the many Dynatrace apps and modules

Pricing: Custom pricing through sales. Typically positioned at the high end of the enterprise observability market.

Best For: Large enterprises with complex hybrid environments and the budget for a premium AI-powered observability platform.

4. Kentik

Best for: Network teams running large hybrid, multi-cloud, and internet-facing infrastructure

Kentik focuses on what it calls network intelligence: deep traffic analysis, BGP and internet path visibility, and AI-assisted investigation for hybrid and multi-cloud networks. It is particularly strong for service providers, SaaS companies, and large enterprises with significant cloud and internet exposure.

Key Features:

• NetFlow, sFlow, IPFIX, VPC flow logs, and synthetic test ingestion
• BGP and internet performance visibility
• Kentik AI Advisor and Cause Analysis for guided troubleshooting
• DDoS detection and mitigation analytics
• Hybrid and multi-cloud topology

Pros:

• Industry-leading depth for traffic analytics and internet path visibility
• Strong for cloud egress cost analysis
• AI-assisted investigation reduces time on complex incidents

Cons:

• Pricing aimed at mid-market and enterprise, not small teams
• Steeper learning curve than general-purpose monitoring tools
• Focused on NetOps workflows more than application-level observability

Pricing: Custom pricing through sales. Aimed at enterprise budgets.

Best For: Mid-market and enterprise NetOps and SRE teams managing complex hybrid and internet-connected networks.

5. Cisco ThousandEyes

Best for: Teams that need path visibility across the public internet and SaaS providers

ThousandEyes monitors network paths from your users and applications to wherever they need to go: SaaS apps, cloud regions, third-party APIs, and your own services. It is the go-to tool for understanding ISP, CDN, and public internet performance.

Key Features:

• Synthetic agents on user devices, in cloud regions, and on enterprise networks
• Hop-by-hop path visualization across the internet
• BGP route monitoring and outage detection
• SaaS application performance tests (Microsoft 365, Salesforce, Zoom, and similar)
• Internet outage detection and notifications

Pros:

• Best-in-class for public internet and SaaS path visibility
• Wide global agent network
• Strong correlation between user experience and underlying network paths

Cons:

• Does not cover host or pod-level traffic inside your infrastructure
• Pricing scales quickly with agent counts and test frequency
• Less useful as a standalone tool, usually has to be paired with another monitoring platform

Pricing: Subscription-based, priced per agent and per test. Custom quotes through sales.

Best For: Enterprises with significant SaaS, hybrid work, and internet-facing service dependencies that need to prove whether the problem is them, their ISP, or a SaaS provider.

6. SolarWinds Network Performance Monitor

Best for: Traditional enterprise IT teams managing on-premises networks

SolarWinds Network Performance Monitor is one of the longest-running enterprise NPM products. It excels at SNMP-based monitoring of routers, switches, firewalls, wireless access points, and other traditional network gear.

Key Features:

• SNMP v1, v2c, and v3 polling with deep multi-vendor support
• Automated Layer 2 and Layer 3 topology mapping
• NetPath for hop-by-hop path analysis
• Wireless network monitoring and heat maps
• Integration with the broader SolarWinds Observability portfolio

Pros:

• Deep coverage for traditional network gear from Cisco, Juniper, Aruba, Fortinet, and many others
• Mature alerting and reporting features
• Familiar to most network engineers

Cons:

• Built for static network device inventories rather than dynamic cloud-native workloads
• Self-hosted deployment requires Windows Server infrastructure
• Modernization toward SaaS has been gradual

Pricing: Starts around $1,995 for SolarWinds NPM perpetual license. SaaS pricing through SolarWinds Observability is consumption-based.

Best For: Enterprise NetOps teams managing campus, branch, or data center networks where SNMP and traditional NPM workflows dominate.

7. Auvik

Best for: MSPs and IT teams managing many distributed sites

Auvik is a cloud-based network monitoring platform aimed primarily at managed service providers and multi-site IT teams. Its automated topology mapping and per-site collector model make it easy to onboard new networks quickly.

Key Features:

• Cloud-managed with lightweight on-premises collectors per site
• Automatic Layer 1, 2, and 3 topology discovery and visualization
• TrafficInsights for NetFlow, sFlow, J-Flow, and IPFIX analysis
• Configuration backup and change tracking
• Multi-tenant architecture for MSPs

Pros:

• Fast time to value, sites usually online within an hour
• Excellent automated topology maps that update in real time
• Multi-tenant model purpose-built for MSPs
• Configuration management and syslog included

Cons:

• Per-billable-device pricing can get expensive at scale
• Less depth than Kentik or SolarWinds NPM for deep traffic analytics
• Not designed for application-level or container-level visibility

Pricing: Per-billable-device pricing through sales. Free trial available.

Best For: MSPs and IT teams managing dozens or hundreds of distributed sites where rapid onboarding matters.

8. Paessler PRTG Network Monitor

Best for: Small and mid-size organizations that prefer sensor-based, all-in-one monitoring

PRTG is a long-established Windows-based monitoring tool that uses a sensor model: each metric, port, or check is a sensor, and you pay for the total number of sensors. It covers networks, servers, applications, and IoT devices through SNMP, WMI, NetFlow, and HTTP probes.

Key Features:

• 250+ pre-built sensor types for network devices, servers, and applications
• SNMP, WMI, NetFlow, sFlow, J-Flow, and IPFIX support
• Maps, dashboards, and reporting included
• On-premises and PRTG Hosted options
• Free tier up to 100 sensors

Pros:

• Easy to install and configure on Windows
• Broad sensor catalog covers network and infrastructure in one tool
• Free tier is genuinely useful for small environments
• Strong fit for SMB and mid-market IT generalists

Cons:

• Sensor-based licensing gets complex and expensive as environments grow
• Windows-centric architecture (probes can be on Linux but the core is Windows-only)
• Less suitable for dynamic Kubernetes workloads

Pricing: Perpetual licenses from around $2,149 for 500 sensors. PRTG Hosted available as SaaS. Free up to 100 sensors.

Best For: Small and mid-size organizations running mixed Windows and Linux infrastructure that want a single tool for network and server monitoring.

9. ManageEngine OpManager

Best for: IT teams that want broad multi-vendor monitoring at a moderate price

OpManager is part of the ManageEngine portfolio. It provides SNMP, WMI, CLI, and Telnet-based monitoring with extensive multi-vendor support and offers both subscription and perpetual licensing.

Key Features:

• 200+ pre-built device templates across Cisco, Juniper, HP, Dell, Fortinet, and others
• Network Configuration Manager add-on for backup and compliance
• Workflow automation for routine remediation
• NetFlow Analyzer add-on for traffic analysis
• On-premises deployment with broad OS support

Pros:

• Significantly cheaper than SolarWinds NPM with comparable feature depth
• Perpetual licensing option for teams that prefer capital over operating expenses
• Strong multi-vendor coverage out of the box

Cons:

• Interface feels dated compared to modern SaaS tools
• Add-on modules increase the total cost
• Self-hosted only, no fully managed SaaS option

Pricing: Starts around $245 per year for 25 devices. Perpetual licenses available. NetFlow Analyzer and Network Configuration Manager priced separately.

Best For: Mid-market IT teams wanting broad on-premises network and server monitoring at a moderate price.

10. LogicMonitor

Best for: Hybrid infrastructure teams wanting SaaS-based monitoring across networks, servers, and cloud

LogicMonitor is a SaaS infrastructure monitoring platform that covers network devices, servers, cloud resources, and containers from a single console. The Edwin AI engine adds anomaly detection and event correlation across signals.

Key Features:

• 3,000+ integrations and modules across networks, servers, and cloud
• Cloud-managed with on-premises collectors
• Edwin AI for anomaly detection and alert correlation
• Per-device licensing rather than per-sensor or per-interface
• Topology mapping and synthetic checks

Pros:

• One of the broadest integration libraries in the market
• Predictable per-device pricing
• Strong fit for hybrid environments mixing data center and cloud

Cons:

• Enterprise pricing, not aimed at small teams
• Less focused on application-level or service-level visibility than Datadog or Dynatrace
• Some users report complex initial setup for advanced features

Pricing: Custom pricing through sales. Aimed at mid-market and enterprise.

Best For: Mid-market and enterprise IT teams managing hybrid infrastructure that want a single SaaS tool for networks, servers, and cloud.

11. Zabbix

Best for: Teams wanting a free, open-source enterprise-grade monitoring platform

Zabbix is one of the most widely deployed open-source monitoring tools. It supports SNMP, agent-based, agentless, and API-based monitoring for networks, servers, applications, and cloud services. The project has been actively developed for over two decades.

Key Features:

• SNMP, IPMI, JMX, ODBC, agent, and agentless monitoring
• Network discovery and topology maps
• Flexible templating and macros
• Notifications across many channels
• Distributed proxy architecture for large environments

Pros:

• Completely free and open-source under AGPL
• Mature, production-proven at very large scale
• Active community and commercial support available
• Flexible enough to cover networks, servers, and applications

Cons:

• Requires operational expertise to deploy, scale, and maintain
• UI feels less polished than commercial SaaS tools
• Initial configuration can be time-consuming

Pricing: Free. Commercial support and training available through Zabbix LLC.

Best For: Teams with the operational capacity to run their own monitoring infrastructure and that want full control without licensing costs.

12. Cilium Hubble

Best for: Kubernetes teams wanting open-source eBPF-native network observability

Cilium Hubble is the observability layer for Cilium, the CNCF-graduated eBPF-based networking and security project. It provides flow-level visibility, service maps, and policy enforcement insights for Kubernetes clusters.

Key Features:

• eBPF-based flow capture without sidecars
• L3, L4, and L7 visibility including HTTP, gRPC, Kafka, and DNS
• Service map visualization
• Network policy verification and dropped-flow analysis
• Integration with Prometheus and Grafana

Pros:

• Completely free and open-source under Apache 2.0
• Native eBPF approach with low overhead
• Strong for Kubernetes security and network policy use cases
• Tight integration with Cilium CNI

Cons:

• Requires Cilium as the CNI plugin, which is a significant architectural choice
• Operational expertise needed to deploy and scale Hubble
• Less suited for standalone hosts, VMs, or non-Kubernetes workloads

Pricing: Free. Commercial support through Isovalent (now part of Cisco).

Best For: Kubernetes-native teams that have adopted or are willing to adopt Cilium as their CNI and want deep eBPF-based flow observability without commercial licensing.

Network Monitoring Tools Comparison Table

How to Choose the Right Network Monitoring Tool

Start With Your Stack

A team running mostly Cisco switches and on-premises servers has a very different set of needs from a team running Kubernetes on EKS. SNMP-based tools like SolarWinds, ManageEngine, and PRTG remain the right answer for traditional network gear. For Kubernetes and dynamic cloud workloads, eBPF-based tools like Sematext Network Map and Cilium Hubble give you visibility that legacy NMS products cannot match.

Decide Whether You Want a Standalone or Integrated Tool

Network monitoring used to be a separate tool maintained by a separate team. That model still works for some organizations, but most modern teams benefit from network visibility that lives next to logs, metrics, and traces. Sematext, Datadog, Dynatrace, and LogicMonitor all let you correlate network signals with application data. Standalone specialists like Kentik and ThousandEyes go deeper in their niche, often paired with a broader observability platform.

Model Your Cost at Scale

Pricing models vary widely. Per-sensor models like PRTG can balloon as you add checks. Per-device models like Auvik and LogicMonitor are predictable but can become expensive as device counts grow. Per-host models like Sematext and Datadog scale with infrastructure size. Always run a quick projection for your expected scale rather than going by starter-tier pricing.

Test Before You Buy

Most commercial tools offer free trials. Open-source tools cost only your time. Pick two or three candidates that fit your environment and run them on a real workload for a week or two. You will learn more from a short pilot than from any vendor demo.

Do Not Forget About People

A tool nobody uses is worse than no tool at all. Whatever you choose, make sure your on-call engineers find the UI usable, the alerts trustworthy, and the data easy to share across teams.

Conclusion

Network monitoring in 2026 is no longer a single category. Some teams need deep SNMP visibility for traditional gear. Others need eBPF-based service maps for Kubernetes. Many need both, plus correlation with logs, metrics, and traces.

For cloud-native and hybrid teams, Sematext Network Map stands out as the most pragmatic choice in 2026. It gives you eBPF-powered service and infrastructure visibility, automatic detection of over 100 service types, Kubernetes-native drill-down, and integration with logs, metrics, traces, and synthetic monitoring in one platform. All of that comes at a fraction of the cost of the major enterprise observability incumbents.

If your environment is dominated by traditional network gear, SolarWinds, ManageEngine, and Auvik remain solid choices. If you need internet and SaaS path visibility, ThousandEyes is hard to beat. If you want fully open-source, Zabbix covers the broad case and Cilium Hubble covers the Kubernetes-native case.

Whatever you choose, prioritize tools that fit how your infrastructure actually works today, not how it worked five years ago.

Ready to see your infrastructure as it really is? Try Sematext Network Map free for 14 days, no credit card required.

FAQ

What is the difference between network monitoring and observability?

Network monitoring focuses on the health and behavior of network infrastructure: devices, interfaces, traffic, paths, and connections. Observability is a broader practice that combines logs, metrics, traces, and topology to understand the health of the entire system, including applications. Modern tools like Sematext, Datadog, and Dynatrace blur the line by treating network visibility as one signal among many.

Do I need separate network monitoring if I already have APM?

APM tells you what your application code is doing. Network monitoring tells you what is happening at the network connection and infrastructure level. The two are complementary. A slow database query shows up in APM. A flaky network path, a saturated link, or a rogue process making outbound calls shows up in network monitoring, but not in APM. Most teams benefit from both, ideally on the same platform.

What is eBPF and why does it matter for network monitoring?

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that lets tools safely run sandboxed programs inside the kernel to observe events such as network connections, system calls, and packet processing. For network monitoring, eBPF allows tools to capture connection-level data with very low overhead and without modifying applications. Tools like Sematext Network Map and Cilium Hubble use eBPF to build accurate, real-time topology views without sidecars, port mirroring, or packet captures.

Can network monitoring tools work with Kubernetes?

Yes, but with very different levels of effectiveness. Tools designed for static device inventories struggle with the dynamic nature of pods and containers. eBPF-based and Kubernetes-aware tools like Sematext Network Map, Cilium Hubble, Datadog NPM, and Dynatrace are built for this environment. Traditional SNMP-focused tools can still monitor the underlying nodes, but they will not give you pod-level or service-level visibility.

What are the best free or open-source network monitoring tools in 2026?

Zabbix remains the most widely deployed open-source choice for general infrastructure and network monitoring. Nagios and Icinga are still in active use for traditional environments. LibreNMS and OpenNMS are strong for SNMP-heavy networks. For Kubernetes-native, eBPF-based observability, Cilium Hubble is the leading open-source option. All of these are free to use but require operational expertise to deploy and scale.

How much do network monitoring tools cost?

Pricing varies enormously. Open-source tools are free but carry operational cost. Per-device tools like Auvik and OpManager typically range from a few dollars to tens of dollars per device per month. Per-host tools like Sematext Network Map start around $1.68 per host per month. Per-sensor tools like PRTG depend on how many checks you configure. Enterprise platforms like Datadog, Dynatrace, Kentik, and LogicMonitor are usually priced through sales and can run from tens of thousands to hundreds of thousands of dollars per year depending on scale.

Start Free Trial

The post Top 12 Network Monitoring Tools in 2026: Complete Comparison & Reviews appeared first on Sematext.

But there’s a part of the OTel pitch that often gets glossed over: somebody still has to instrument the application. And that part isn’t quick or easy. Even now.

The instrumentation tax

Modern applications aren’t a single binary anymore. At Sematext we run 30+ microservices to power the Sematext Cloud platform: alerts, metrics receivers and consumers, log receivers and consumers, user experience services, tracing pipelines, network map, various APIs and more, across Java (Spring Boot), Go, and a few other stacks.

That’s a lot of surface area, several languages, multiple frameworks, different build systems for different stacks. None of this is unusual for a mature system. This means that instrumentation effort grows with that diversity, so any mechanism that helps minimize instrumentation mistakes will be welcomed by engineers tasked with instrumentation.

If you want end-to-end tracing through that stack, the kind that actually tells you where a slow request spent its time, you can’t just instrument one service. You have to instrument the whole chain: frontend → API gateway → backend service A → backend service B → database. Skip a hop and the trace breaks. The dependency graph the trace gives you stops being useful exactly at the boundary you didn’t instrument.

So in practice, “let’s adopt OpenTelemetry” turns into a checklist of dozens of services that each need their own instrumentation work. The good news is that it doesn’t have to happen all at once and AI can help.

How much does OpenTelemetry instrumentation cost?

By cost, we mean the cost to an engineer, the team, and the organization. We can look at it as a non-monetary cost, but if we trace (pun intended!) this cost all the way down then yes, there is also a financial cost associated with this effort.

Three things make this hard, even with OTel:

Prioritization. Instrumenting a service competes with shipping features and fixing bugs. It’s preventive work; its value shows up the next time something breaks at 3am, not this sprint. That’s a hard sell to a product manager.

Unknown territory. When the chain spans services you didn’t write, in languages you don’t use day to day, you’re spending most of your time on context switch overhead. You’re not adding instrumentation; you’re re-learning a framework you saw once two years ago.

Time needed even for auto-instrumentation. “Auto-instrumentation” means no code changes. It doesn’t mean no work. For one service the loop typically goes:

1. Read the right OTel SDK docs for your language
2. Pick the right auto-instrumentation package (there are usually three options, only one of which is current)
3. Install it in the build (pom.xml, package.json, requirements.txt, …)
4. Configure the OTLP endpoint, the auth header, the service name
5. Restart, hit the service, watch what happens
6. Debug the first attempt: wrong port (4317 vs 4318 vs 4338), wrong protocol (http/protobuf vs grpc), wrong auth header (Bearer vs vendor-specific), region mismatch on the endpoint
7. Verify the data lands in the right place in your observability tool
8. Multiply by the number of services in your chain

Forty minutes to two hours per service if you’re moving carefully, and that’s just for traces and metrics. The OTel auto-instrumentation packages don’t ship logs in most SDKs. For logs you need to switch to manual instrumentation, which is another SDK init block per service.

And then there’s custom OpenTelemetry instrumentation

The above buys you “spans for every incoming HTTP request” and a generic metrics set. The moment you want anything specific (e.g., a custom span attribute for the user’s account tier, a business metric counting checkouts, a log enriched with the trace ID so you can correlate logs and traces for faster Root Cause Analysis), you’re back in manual-instrumentation land, writing SDK code in every service you care about. The auto path ends; the per-language SDK learning curve begins.

For one service that’s an afternoon. For thirty services that’s a quarter. What can we do about this?

Can we use AI to instrument applications with OpenTelemetry?

The instrumentation work (pick the SDK, set the env vars, debug the endpoint, verify it landed) is exactly the kind of structured, repetitive, well-documented task an AI agent does well. The blockers aren’t intellectual; they’re “look up the right thing, paste it in the right place, watch for the obvious gotcha.”

That’s not “let the AI do your engineering.” It’s “let the AI do the parts that already had a right answer, written down somewhere, and just needed someone to fetch it.”

We tried this for instrumenting applications against Sematext Cloud. The result is a small,but highly valuable open-source artifact: a Claude Code Agent Skill that walks an engineer through OTel instrumentation conversationally. It’s plain markdown, lives in our public Github repository, and works with any AI agent that can read a URL.

What the OTel instrumentation AI skill does

The Sematext OTel skill at sematext-otel-onboarding/blob/main/skills/SKILL.md is the AI-readable version of “how to wire your application to Sematext.” When loaded into Claude Code (or any agent that can fetch a markdown URL), it triages the user through six short questions:

1. Sematext region (US or EU)
2. Which App types you’re wiring (Tracing, Logs, Monitoring, any combination)
3. Flow: managed OTLP endpoint or Sematext Agent
4. Protocol: HTTP (default) or gRPC
5. Language and deployment environment
6. Auto or manual instrumentation

Then it produces the exact env-var block, parameterized to your answers, including:

• The correct OTLP endpoint URL for your region and protocol
• The Sematext-specific X-API-TOKEN header (different from the standard Authorization: Bearer … most OTel docs show, easy to miss)
• One header per signal type, so you only configure what you’re using
• A pointer to a runnable reference example in the same repo, in your language

Similarly, you can use the skill not only to add instrumentation to uninstrumented applications, but also to fix broken instrumentation that’s not really working. Auto-instrumentation doesn’t ship logs? The skill flags that and asks if you want to switch to manual. Region-token mismatch? The skill warns explicitly. Custom header convention? Documented. The skill is opinionated and aware of the setup required where the official OpenTelemetry docs may be silent or difficult to understand and follow.

What an instrumentation session looks like

In practice, an engineer with Claude Code in their editor opens their service’s directory and pastes:

Use https://github.com/sematext/sematext-otel-onboarding/blob/main/skills/SKILL.md to instrument this app for Sematext.
Region: US. App type: Tracing. Token: <pasted-from-Sematext-UI>.

Claude loads the skill, reads the project’s files to figure out the language and framework, asks the two remaining triage questions, then proposes the diff to the project (adds the OTel SDK to the build file, adds the env vars to docker-compose or systemd or .env or wherever they belong, and shows you what to expect in the Sematext UI within 60 seconds). You review the diff. You apply. You restart. You see traces.

Compared to the manual path (read docs, pick SDK, install, configure, debug, verify), we’d expect the instrumentation effort time to first data to drop from the typical 40 to 120 minutes per service to a handful of minutes. For an organization adopting OTel across dozens of services, that compounds quickly. A quarter of part-time effort becomes a couple of focused days. Thousands of dollars in engineering time drops to a much more sane number. The effort has a positive ROI.

What the skill doesn’t do

This is where AI posts usually start hand-waving. Here’s the honest list:

• It doesn’t write custom span attributes or business metrics for you. It writes the boilerplate that gets you to the point where you can write those. The judgment about what to measure is still yours. The benefit is that the skill gives you all the scaffolding, a working instrumentation, so the effort of collecting custom/business metrics becomes significantly lower.
• It doesn’t psychic-debug your network. If your service can’t reach the OTLP endpoint on first run (corporate proxy, missing TLS cert chain, wrong port), the skill points at the common causes, but you still have to look at the service’s own logs to confirm what happened.
• It doesn’t change OTel’s reality. Auto-instrumentation still doesn’t ship logs in most SDKs. AI doesn’t fix the SDK. But it does tell you up front, so you don’t spend an hour wondering why your Logs App is empty.

Try it, the skill is vendor-agnostic

The skill is open-source and lives in our OTel onboarding repo. The same repo has runnable reference apps for Node.js, Java, Python, .NET, and PHP across baremetal, Docker, and Kubernetes deployments, so if you want to see what the skill is going to walk you through, the reference is right there.

If you have Claude Code, point it at the URL above. If you use a different AI agent, the skill is just markdown. Load it however your agent loads documentation. There’s no install step; there’s no vendor lock-in. The skill shared is not Sematext-specific. Sematext’s contribution is the knowledge, encoded in a format an AI can act on.

Where this is going

The OTel skill is one example of a broader pattern we think makes sense for observability tools: knowledge as something an AI can use, not just something a human can read.

A few directions we’re exploring:

• Per-language sub-skills for deeper, opinionated guidance when a language has subtle gotchas (Node async hooks, Java agent attach, Python startup ordering)
• In-product wiring so the App creation page in Sematext Cloud gives you a one-click “use AI to set this up” alongside the existing manual instructions, with your region and token pre-filled into the prompt
• Skills for the rest of the observability journey (picking sensible default alerts, creating dashboards, interpreting RCA results), each as a small, auditable markdown file you can use, fork, or ignore

The bet is that the value of an observability platform isn’t just the data it collects; it’s how quickly an engineer can go from “we should monitor this” to “we’re monitoring it and we know what to do when it breaks.” AI doesn’t replace the judgment in that loop as of yet. But it can absolutely replace the busywork around it.

If you give the skill a try and find a gap, open a PR. The fastest way for this pattern to get good is for more people to use it on more apps.

Start Free Trial

The post Using AI to Instrument Applications with OpenTelemetry appeared first on Sematext.

It starts in late November, 2025. This marks the beginning of increased AI usage (for coding) in Sematext. That’s when the LLMs got better. It roughly matches the change in velocity as visualized in JIRA.

Individual AI Adoption

The blurred part are PR author names, which we can use for filtering. If we look at trends of individuals we can spot early adopters like this one:

Or another individual who started making more use of AI later:

Source: Github WebHook Events

This data comes into Sematext via Github Webhook Events. It takes about 5-10 minutes to set up. It can be set up at the Github organization level or for individual repositories.

A Word of Caution

1. There are many software development styles. There are people who commit frequently and incrementally and there are those who keep things to themselves until everything is nearly done. This is a fun chart to look at and is helpful when you want to get the feel for the “pulse” of a team or even an individual. But be careful not to judge people on this sort of data alone. Use this with a grain of salt and in combination with other inputs, observations, etc.
2. Creating more code or PRs doesn’t always equal better code or higher effectiveness. A person may be tapping in the dark trying to debug or implement something with the help of AI and, in the process, creating a lot of (temporary?) code and PRs.
3. As velocity increases, so will regressions, unless you take countermeasures. See Faster Coding with AI and Increased Regressions.

Start Free Trial

The post Pull Request Velocity as a Proxy for AI Usage for Software Development appeared first on Sematext.

Running OTel across a handful of services and running it across a few hundred are genuinely different problems. The instrumentation part stays roughly the same. Everything around it — how you collect the data, how you route it, how you make sure a traffic spike in one region does not take down your entire observability pipeline — that is where teams either build something resilient or spend the next six months fire-fighting because of inadequate planning or suboptimal architecture.

I wrote this article to share the patterns that actually hold up at scale: collector tiers, load balancing strategies, sampling at volume, and multi-cluster setups. Everything comes with real config examples because “it depends” is only useful advice if you can see what it depends on.

See How OpenTelemetry changes the way teams do observability for why OpenTelemetry matters and how it shifts focus from traditional metrics and logs to full, end-to-end observability.

Why a Single Collector Falls Apart (and When)

Most OTel tutorials show you a single collector instance receiving spans from all your services and forwarding everything to a backend. That setup works until about the point where it stops working, which tends to happen quietly and at the worst possible time. You are not going to notice a single collector struggling until it is already dropping data, buffering is maxed out, and your traces have gaps you cannot explain.

The core issue is that a single collector is both a single point of failure and a resource bottleneck. At low traffic it sits there looking fine. Add a few dozen services, let traffic spike during a product launch or a retry storm, and you will watch it start falling behind. The exporter queue fills up. Backpressure kicks in. Services start dropping spans rather than blocking on the export. By the time anyone notices, you have lost the exact telemetry you needed to understand what just happened.

The solution is to stop thinking about the collector as a single process and start thinking about it as a tier. Two tiers cover most production scenarios. Three tiers cover the rest. The architecture you need depends on your traffic, whether you need tail-based sampling, and how many backends you are exporting to.

Let me make this more specific: if you have fewer than 20 services and under 500 requests per second total, a single well-configured collector will likely hold up (yes, of course it depends on the underlying hardware/resources). At 20 to 80 services or 500 to 5,000 RPS, the two-tier model becomes worthwhile. Above 80 services or 5,000 RPS, you need the full tiered setup with trace-aware load balancing and tail-based sampling at the gateway. 

For more information on common production pitfalls and strategies to prevent them, see OpenTelemetry Production Monitoring: What Breaks and How to Prevent It.

Collector Tiers: The Architecture That Actually Scales

The tiered collector model separates two concerns that should never have been combined in the first place: getting data off your services quickly, and doing something intelligent with that data before it hits your backend.

Before getting into the architecture, it helps to know that the OTel Collector can run in three modes — and in a scaled setup, you will use all three:

• Agent — a collector running on the same host as your services, collecting telemetry locally and forwarding it upstream. It stays thin: no heavy processing, just receive-and-forward.
• Gateway — a collector running as a standalone service, receiving data from agents (or directly from SDKs) and doing the heavier work: sampling, routing, fan-out to backends, attribute redaction.

Combined — the full pattern, where agent collectors feed into gateway collectors. Agents handle what only makes sense per-host (host metrics, file logs, resource detection). Gateways handle what only makes sense centrally (tail-based sampling, cross-service routing, policy management). The OTel Collector deployment docs call this the combined deployment pattern.

The tiered setup this article describes is the combined pattern. Here is what it looks like:

Tier 1: Collectors running as agents

The agent tier runs as a sidecar. Its job is exactly one thing: receive telemetry from the services and forward it as fast as possible. No tail-based sampling, no complex routing logic, no fan-out to multiple backends. The only processing you want at this tier is cheap and stateless: adding resource attributes like cluster name, node name, and environment; batching spans to reduce connection overhead; and basic filtering to drop genuinely worthless spans like health check endpoints generating thousands of spans per minute and telling you nothing.

# Tier 1: runs as DaemonSet, minimal processing
receivers:
  otlp:
    protocols:
      grpc: {endpoint: "0.0.0.0:4317"}
      http: {endpoint: "0.0.0.0:4318"}

processors:
  batch:                    # batch before forwarding
    send_batch_size: 1024
    timeout: 5s
  resourcedetection:        # stamp node/pod metadata
    detectors: [k8snode, env]
  filter/drop_healthchecks:
    spans:
      exclude:
        match_type: regexp
        attributes:
          - {key: http.route, value: ".*/health.*"}

exporters:
  otlp:
    # forward to gateway tier, not directly to backend
    endpoint: "otel-gateway:4317"
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 500

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [batch, resourcedetection, filter/drop_healthchecks]
      exporters:  [otlp]

exporters:
  loadbalancing:
    routing_key: "traceID"   # hash by trace ID, not round-robin
    resolver:
      k8s:                    # auto-discover gateway pods via DNS
        service: "otel-gateway"
        ports: [4317]
    protocol:
      otlp:
        timeout: 1s
        sending_queue:
          enabled: true
          queue_size: 1000

processors:
  tail_sampling:
    decision_wait: 10s      # wait for all spans before deciding
    num_traces: 100000      # traces held in memory simultaneously
    expected_new_traces_per_sec: 1000
    policies:
      # always keep error traces
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}

      # always keep slow traces (adjust threshold to your SLO)
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}

      # keep 100% of checkout and payment — business critical
      - name: keep-critical-services
        type: string_attribute
        string_attribute:
          key: service.name
          values: [checkout-api, payment-service]

      # probabilistic baseline for everything else
      - name: baseline-sample
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

# nginx — add inside your proxy_pass block
proxy_set_header traceparent $http_traceparent;
proxy_set_header tracestate  $http_tracestate;

---

# Envoy — request_headers_to_add in HttpConnectionManager
route_config:
  request_headers_to_add:
    - header: { key: traceparent }
    - header: { key: tracestate }

processors:
  transform/redact_pii:
    trace_statements:
      - context: span
        statements:
          # hash user IDs rather than drop
          - set(attributes["user.id"], SHA256(attributes["user.id"]))
          # drop email entirely
          - delete_key(attributes, "user.email")
          # truncate IP to /24 for geo without individual tracking
          - replace_pattern(attributes["net.peer.ip"], "\\d+$", "0")

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 15s
          static_configs:
            - targets: ["localhost:8888"]

# Expose collector's own telemetry via its service config
service:
  telemetry:
    metrics:
      level: detailed   # basic | normal | detailed
      address: 0.0.0.0:8888
    logs:
      level: warn       # keep collector logs quiet in production

Start Free Trial

The post Running OpenTelemetry at Scale: Architecture Patterns for 100s of Services appeared first on Sematext.

This is what life looks like when observability is an afterthought: logs in one place, metrics in another, and a custom monitoring agent that only works for two services because the third one was written in a language nobody on the team uses anymore. It works, technically. Until it does not.

OpenTelemetry came out of a genuine frustration with this fragmented mess. It is an open-source observability framework that gives you a vendor-neutral, standardized way to instrument your applications and then connect that instrumentation to service health, error budgets, and eventually SLOs that your entire organization actually understands. This article walks through what that shift looks like in practice, and why it matters for more than just the people who are on call.

The Old World: Logs, APM Agents, and the Dashboard Graveyard

Let’s be direct about how most teams actually do observability before they invest in it properly. You have application logs going into a log management platform, with varying levels of structure depending on who wrote which service. You have an APM tool that auto-instruments some of your services but not all of them, and the traces it produces are siloed within its own ecosystem. And you have a monitoring dashboard that someone built eighteen months ago and that might or might not reflect how the service actually behaves today.

The deeper problem is not the tools themselves; it is that each one has its own instrumentation model. Your APM agent captures HTTP spans one way. Your custom metrics library reports latency percentiles slightly differently. Your logs do not correlate to your traces automatically. So when something breaks, you are stitching together three different narratives instead of reading one coherent story about what happened.

What OpenTelemetry Actually Is (Without the Fluff)

OpenTelemetry standardizes how you generate, collect, and export telemetry data. It covers three signal types (with more to come), which are the foundation of everything else in this article:

What makes OTel different from what came before is not magic; it is the fact that all three signals share the same context propagation model. A trace ID that starts in your frontend propagates through every instrumented microservice call, and if your logs are also emitting that trace ID, you can jump from a log line to its trace in seconds. Not minutes. Seconds. If you are the person doing production troubleshooting you know how valuable this difference is!

SLOs: What They Are and Why OTel Makes Them Achievable

Service Level Objectives have been a thing since Google wrote about them in the Site Reliability Engineering book, and they have been misunderstood and poorly implemented since roughly the same time. The core idea is simple: you agree on a target for how reliable a service needs to be, you measure it consistently, and you manage your engineering work in relation to how much reliability budget you have consumed or have left.

The reason SLOs often fail is not the concept; it is that teams try to define them before they have reliable telemetry. You cannot set a meaningful availability target for a service if your metrics come from three different monitoring agents that measure availability in subtly different ways. You end up with SLOs that nobody trusts, which means nobody uses them to make decisions.

When your SLIs are computed from OTel metrics, specifically from the semantic conventions that define how HTTP span duration and status should be recorded, you get consistency across services by default. The latency histogram for your Go service and the one for your .NET service use the same bucket boundaries. The error classification follows the same logic. Suddenly your SLOs are comparing apples to apples, and that changes what you can do with them.

The Correlation Story: How one Trace ID Connects Everything

One of the things that sounds academic until you experience it is trace context propagation. When a request comes into your frontend and you are using OTel instrumentation, a trace ID gets generated and passed along to every downstream service call via HTTP headers, gRPC metadata, message queue attributes, or whatever transport you are using. Every span in that trace carries the same trace ID, and your logs carry it too if you have set up log correlation.

What this means in practice: when your error rate alert fires because the checkout service just breached its error budget, you do not start by guessing. You go to the traces for that time window, filter for error spans, and you are already looking at the full call path: frontend, checkout API, inventory service, payment gateway, with timing for each hop. If the inventory service was slow, you will see a long span there. If the payment gateway returned a 503, you will see that in the span status. No grep-ing through logs trying to find a request ID that someone may or may not have remembered to log. For a step-by-step breakdown of what these patterns look like in real incidents, troubleshooting microservices with distributed tracing is a good companion read.

Auto-Instrumentation: Getting Value Without Rewriting Everything

One of the biggest objections to investing in observability is the instrumentation cost. If you have thirty microservices and each one needs to be manually instrumented before you see any benefit, that is a project with a very long feedback loop. This is actually what we saw with our initial distributed tracing implementation at Sematext back in 2015 – adoption of a challenge due to how much work engineers would have to invest in instrumenting their applications. OTel’s auto-instrumentation libraries change that equation significantly.

For Java, the OTel Java agent attaches to your JVM at startup and automatically instruments common frameworks such as Spring Boot, gRPC, JDBC, and Kafka without any code changes. For Python, opentelemetry-instrument does the same for Flask, Django, FastAPI, and SQLAlchemy. The .NET ecosystem has similar coverage through the automatic instrumentation package. You get spans for every incoming HTTP request, every outgoing call, and every database query without touching the application code. If you want to skip the boilerplate and start from something that already works, these language-specific OTel examples cover the setup end to end.

What to Actually Watch Out For

None of this comes without tradeoffs, and articles that only cover the benefits are setting you up for some unpleasant surprises. A few things will bite you if you do not plan for them.

A deep dive into OpenTelemetry instrumentation best practices covers all of these in detail, but here is the short version.

Cardinality explodes if you are not careful

OTel metrics support rich attribute sets, which is great for debugging but problematic for storage if you start adding high-cardinality attributes like user IDs or request IDs to your metrics. The OTel metrics spec includes cardinality limits, and you should understand them before you start attaching attributes to everything.

Sampling is necessary at scale and confusing to get right

Sending 100 percent of traces when you are handling thousands of requests per second is expensive. Head-based sampling, where you decide at the start of a trace whether to keep it, is simple but means you might drop the interesting traces. Tail-based sampling, where you decide after seeing the whole trace, keeps the errors but requires the OTel Collector to buffer spans, which adds complexity. There is no right answer, only tradeoffs that depend on your volume and budget.

Auto-instrumentation vs manual instrumentation: the honest tradeoff

Auto instrumentation gets you running in an afternoon with zero code changes and gives consistent coverage across your entire fleet from day one. The trade off is that it understands frameworks, not business intent. It can tell you a database query took 800 ms but not that it was pricing a cart for a high value customer.

Manual instrumentation fills the gap that actually matters for SLOs. Checkout completion time, order processing latency by fulfillment partner, or time to first search result. It takes more effort, but it is what turns a latency alert into a business conversation.

In practice, auto instrumentation provides the foundational 80 percent. Requests, error rates, and durations (aka RED) from day one. You then layer manual instrumentation on top for the business critical signals your SLOs should be measuring.

The Collector configuration gets complex fast

Once you start running multiple pipelines, applying transforms, doing tail-based sampling, and exporting to multiple backends, your collector config becomes something that needs to be tested and versioned like application code. Treat it that way from the start.

Starting Without Starting Over

The most common mistake teams make when adopting OTel is treating it as a big-bang migration. You do not need to instrument every service before any of it becomes useful. Pick one service, ideally something that sits in the middle of your call graph so you can see upstream and downstream spans, and get it fully instrumented with OTel, exporting to a collector and from there to whatever backend you already have. Define one or two SLIs for it. Watch them for a week and see if they match your intuition about how the service is performing.

That first service will teach you things that no amount of reading can. You will find out how your framework handles context propagation. You will discover that your log format does not include trace IDs and need to fix that. You will learn what your normal latency histogram looks like and be surprised by the long tail. Do that before you roll out to thirty services, and the rollout will go much faster. 

To get started see the Sematext step-by-step setup guide for OpenTelemetry tracing. Once you have that in place, the article on building a troubleshooting workflow with Sematext tracing shows how to use those first traces to investigate issues and iterate on your instrumentation.

Start Free Trial

The post From Debugging to SLOs: How OpenTelemetry Changes the Way Teams Do Observability appeared first on Sematext.