Manual Tracing, Scores, and Evaluation with Langfuse (Self-Hosted)

In this lesson, you will learn how to take full control of LLM observability using the Langfuse manual tracing API. While Lesson 1 demonstrated the benefits of decorator-based tracing, real-world LLM systems often require deeper visibility. This includes custom spans, step-level metadata, evaluation scores, and multi-stage inspection for RAG pipelines and agent workflows.

In this lesson, you will build a fully instrumented pipeline where every step, every decision, and every model output is recorded with precision inside your self-hosted Langfuse dashboard.

This lesson is the 2nd in a 3-part series on LLM observability with Langfuse:

1. LLM Observability with Self-Hosted Langfuse and vLLM
2. Manual Tracing, Scores, and Evaluation with Langfuse (Self-Hosted) (this tutorial)
3. Lesson 3

To learn how to build manual traces, attach custom spans, and evaluate LLM outputs with scoring metadata, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Why Manual Tracing Matters for LLM Observability

In Lesson 1, we built the foundations of LLM observability with a fully self-hosted Langfuse stack, a local vLLM server, and a complete decorator-based tracing pipeline. With just a few @observe decorators, we captured prompts, outputs, latency, token usage, and nested spans, all visualized instantly in the Langfuse dashboard. That approach was simple, powerful, and ideal for most LLM applications.

However, real production systems require more control than a decorator can provide.

Decorator-based tracing works well when function boundaries align with observability boundaries. Once a pipeline becomes dynamic, for example by involving multiple retrieval steps, conditional branches, tool calls, retries, validations, re-ranking, scoring, or multi-agent planning, you must explicitly decide what gets traced, how traces are grouped, and what metadata is recorded at each stage. In these scenarios, manual tracing becomes essential.

Manual tracing allows you to open and close spans at will, attach arbitrary metadata, log intermediate states, record evaluation scores, and capture execution steps that do not live inside a function, including loops, conditionals, streaming tokens, or retry logic. In short, decorator tracing provides automation, while manual tracing provides precision.

This lesson shows you how to construct traces explicitly, starting from creating the root trace and continuing through building child spans and attaching fine-grained metadata and custom evaluation signals. You will also integrate evaluation_metrics.py, which introduces lightweight scoring for model generations. This makes it possible to track correctness, response length, latency thresholds, or any domain-specific metric directly inside Langfuse as structured metadata.

By the end of this section, you will understand not only why manual tracing matters, but also when it becomes indispensable. Common use cases include debugging RAG pipelines, analyzing retrieval failures, tracking hallucination hotspots, validating agent actions, and building complex multi-step LLM systems where you need complete visibility into what happened and why.

If you are ready to take full control of your observability pipeline, including manual spans and rich evaluation metadata, the following sections will guide you through the process step by step.

Decorator vs Manual Tracing: When to Use Which

In Lesson 1, the @observe decorator gave us an elegant and almost magical tracing experience. You wrapped a function, ran your pipeline, and Langfuse automatically produced a structured trace with child spans, latency, token usage, and full metadata. This approach works well when your application is composed of clean, well-defined functions, such as a simple “generate answer” pipeline or a single LLM call with minimal branching.

However, decorators have an important limitation. They observe function boundaries, not logic boundaries.

If your real pipeline involves conditional flows, loops, retries, branching, retrieval, ranking, tool invocation, or agent-style decision-making, tracing only the outer function hides much of the interesting behavior. The decorator cannot see inside reasoning steps, iterative refinements, or internal calls unless those steps are wrapped in separate functions. As systems become more dynamic and non-linear, decorator-based tracing begins to fall short.

This is where manual tracing becomes essential.

Manual spans allow you to mark exactly where a step begins and ends, even when that step is not a function. You can record intermediate artifacts such as retrieved documents, scoring signals, latency thresholds, or model reasoning stages. You can attach custom metadata to any span and build a detailed step-by-step view of how your LLM pipeline behaves, rather than only seeing which functions were invoked.

In practice, the most effective approach is hybrid. Use decorators for high-level structure, and use manual spans when precision is required.

This lesson focuses on building that precision.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Manual Tracing with the Langfuse Low-Level API

In Lesson 1, you used @observe decorators to add observability with almost no effort: just annotate your functions and Langfuse automatically created traces, spans, usage metadata, and latency metrics.

In this lesson, we take the opposite approach: full manual control.

Manual tracing exposes the entire underlying API used by Langfuse itself. You decide:

• when traces are created
• how spans relate to each other
• what metadata you attach
• how token usage is recorded
• how latencies are measured
• how deeply nested your pipeline becomes

This approach is critical for advanced LLM workflows where decorators are either too restrictive or too magical.

You will see exactly how Langfuse stores a trace internally, and why this skill becomes essential when building complex RAG, evaluation, or multi-agent systems.

Why Manual Tracing Matters (Even If You Use Decorators)

The decorator API is elegant but sometimes too simple.

Manual tracing is required when you need:

• Full control over trace structure: Define parent → child → subchild relationships explicitly.
• Dynamic spans: When you do not know upfront how many steps your pipeline will generate.
• Conditional traces: e.g., only log LLM calls above 2 seconds latency.
• Custom metadata injection: Dynamic context, retrieval sources, ranking scores, chain-of-thought summaries, etc.
• Advanced RAG + agent observability: Where each tool call needs explicit naming and structure.

In short:

The decorator API is the convenience layer.

Manual tracing is the power-user layer.

Full Manual Tracing Implementation with Langfuse

Below is your complete script, src/tracing_manual.py, unmodified and shown entirely so readers can reference it line-by-line.

"""
Manual Tracing with Low-Level Langfuse API

Shows explicit trace creation and management using Langfuse SDK directly.
This gives you full control but requires more code compared to decorators.
"""

from langfuse import Langfuse
from llm_utils import get_llm_client
from config import get_llm_config
import time

# Initialize Langfuse client
langfuse = Langfuse()

# Initialize vLLM client
client, model = get_llm_client(load_model_from_config=True)

# Get configuration
llm_config = get_llm_config()
temperature = llm_config.get("temperature", 0.7)
max_tokens = llm_config.get("max_tokens", 300)


def generate_with_manual_tracing(question: str) -> str:
    """
    Generate answer WITH manual trace creation.
   
    This gives you full control over every trace property:
    - Custom trace names and IDs
    - Granular span creation
    - Manual token counting
    - Custom metadata
    """
   
    print("Calling LLM with manual tracing...")
   
    # 1. Create trace manually
    trace = langfuse.trace(
        name="manual_llm_call",
        metadata={"method": "manual", "question": question}
    )
   
    # 2. Create span for LLM generation
    start_time = time.time()
   
    generation = trace.generation(
        name="llm_generation",
        model=model,
        input=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": question}
        ],
        metadata={
            "temperature": temperature,
            "max_tokens": max_tokens
        }
    )
   
    # 3. Make the actual LLM call
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": question}
        ],
        temperature=temperature,
        max_tokens=max_tokens
    )
   
    latency_ms = (time.time() - start_time) * 1000
    answer = response.choices[0].message.content
   
    # 4. Update generation with results
    generation.update(
        output=answer,
        usage={
            "input": response.usage.prompt_tokens,
            "output": response.usage.completion_tokens,
            "total": response.usage.total_tokens
        },
        metadata={
            "latency_ms": round(latency_ms, 2)
        }
    )
   
    print(f"   Tokens used: {response.usage.total_tokens}")
    print(f"   Latency: {latency_ms:.2f}ms")
    print(f"   ✅ Manually logged to Langfuse")
    print(f"   🔍 Trace ID: {trace.id}\n")
   
    return answer


if __name__ == "__main__":
    print("\n" + "="*70)
    print("Manual Tracing Demo")
    print("="*70 + "\n")
   
    question = "What is deep learning?"
    print(f"Question: {question}\n")
    print("-" * 70 + "\n")
   
    # Generate with manual tracing
    answer = generate_with_manual_tracing(question)
    print(f"Answer: {answer}\n")
   
    print("=" * 70)
    print("\n📊 Manual Tracing vs Decorators:")
    print("   Manual (this file):")
    print("   • Full control over trace structure")
    print("   • More verbose code")
    print("   • Good for complex custom logging")
    print()
    print("   Decorators (recommended):")
    print("   • Clean @observe annotation")
    print("   • Less boilerplate")
    print("   • Automatic nesting")
    print("   • See: src/tracing_decorator.py")
    print("\n🔍 Check your dashboard: https://cloud.langfuse.com")
    print("=" * 70 + "\n")
   
    # Flush traces
    langfuse.flush()

Code Walkthrough: Langfuse Manual Tracing Pipeline

Let us break this down into meaningful building blocks.

Initializing Langfuse + vLLM

langfuse = Langfuse()
client, model = get_llm_client(load_model_from_config=True)
llm_config = get_llm_config()

Here, we:

• connect to the self-hosted Langfuse Server
• initialize a vLLM OpenAI-compatible client
• load generation parameters such as temperature and max_tokens

Nothing happens yet. This is just configuration.

The real magic begins once we create a trace.

Important: Manual tracing gives you full control over the trace lifecycle.

Creating Manual Traces in Langfuse

trace = langfuse.trace(
    name="manual_llm_call",
    metadata={"method": "manual", "question": question}
)

A trace is the root object that represents the entire request.

You define:

• trace name
• metadata
• context

This is equivalent to @observe(name="llm_pipeline"), but explicit.

Creating a Generation Span

generation = trace.generation(
    name="llm_generation",
    model=model,
    input=[ ... ],
    metadata={ ... }
)

This is the part decorators automatically create.

A generation span:

• represents a single LLM model call
• stores the prompt
• stores parameters (temperature, max_tokens)
• links itself as a child of the main trace

This is a foundational building block for RAG and agent pipelines.

Making the Actual LLM Call

response = client.chat.completions.create(...)

Here, the raw LLM execution happens.

No tracing occurs automatically; the span must be updated manually afterward.

Recording Results (Tokens, Latency, Outputs)

generation.update(
    output=answer,
    usage={...},
    metadata={ "latency_ms": round(latency_ms, 2) }
)

In manual mode, you choose what to log.

This is how you capture:

• latency
• token usage
• answer text
• any additional metadata
• final span status

This is where evaluators, reward functions, safety signals, etc., get attached.

Flushing Traces

Short scripts exit before Langfuse can finish sending data.

langfuse.flush()

This guarantees the trace appears in the Langfuse dashboard immediately.

Running the Langfuse Manual Tracing Script

Right after the “run this script” block:

$ python src/tracing_manual.py

You should see the output, as shown in Figure 3:

Viewing Manual Traces in the Langfuse Dashboard

After running the manual tracing script, open the printed trace URL in your browser.

You should see a page similar to the screenshot below, showing the full structure of your manually created trace.

This view includes:

• Root trace: manual_llm_call
• Child span: llm_generation
• Token usage summary: 32 → 300 (332 total)
• Metadata:
  • method: "manual"
  • question: "What is deep learning?"
• Input and output placeholders:
  • (These appear as null until the generation span updates, since the child span holds the actual LLM data.)

This is the clearest demonstration of what manual tracing gives you: explicit control over the structure, metadata, and nesting of your trace.

Manual vs Decorator Tracing in Langfuse

In this section, you learned how to build an entire trace manually:

• creating a root trace
• adding a generation span
• logging prompts
• recording latency
• logging token usage
• updating metadata
• flushing results

Manual tracing is verbose, but incredibly powerful for custom workflows, evaluation, and multi-step LLM applications.

LLM Evaluation Metrics and Quality Scoring with Langfuse

Observability is more than latency and tokens. In real LLM systems, you also need to evaluate:

• “Was the answer good?”
• “Was it long enough?”
• “Was it too slow?”
• “Did model quality silently degrade?”

This section introduces evaluation metrics, custom scoring, and decorator-based tracing for quality analysis. You will learn how to attach accuracy/quality metadata to traces, visualize scores inside Langfuse, and detect degraded model outputs in real time.

We will do this using the file evaluation_metrics.py, which combines:

• the @observe decorator
• custom scoring logic
• latency checks
• trace scoring
• evaluation pipeline wrapper

By the end, you will have a complete scoring pipeline with metrics displayed inside the Langfuse dashboard.

Adding LLM Evaluation Metrics Beyond Manual Tracing

This file builds on everything from Sections 2 and 3:

This script adds 4 major improvements:

• Automated tracing using @observe
• Custom quality metric (using answer_length as a proxy)
• latency threshold warnings
• score logging inside Langfuse (visible as a numerical “quality” score)

This turns your traces from “LLM diagnostics” into LLM evaluation and monitoring.

Code Walkthrough: evaluation_metrics.py

Below is the full annotated walkthrough.

Initialize Langfuse + LLM Client

langfuse = Langfuse()
client, model = get_llm_client(load_model_from_config=True)

We initialize 2 systems:

• Langfuse (manual scoring only): decorators handle tracing, but Langfuse() is needed for scoring.
• vLLM client: same OpenAI-compatible API as Lesson 1.

The Main Function: generate_and_score()

@observe(name="generate_and_score")
def generate_and_score(question: str) -> tuple[str, float]:

The @observe decorator automatically creates a trace and an associated observation.

The rest of the function focuses on:

• LLM call
• latency measurement
• quality scoring
• updating the observation
• recording a score

Load Configurations

llm_config = get_llm_config()
eval_config = get_evaluation_config()
temperature = llm_config.get("temperature", 0.7)
max_tokens = llm_config.get("max_tokens", 300)
   
min_length = eval_config.get("min_length", 20)
good_length_threshold = eval_config.get("good_length_threshold", 100)
max_latency_ms = eval_config.get("max_latency_ms", 5000)

From config.yaml, we load:

LLM Parameters

• temperature
• max_tokens

Evaluation Parameters

• min_length
• good_length_threshold
• max_latency_ms

This means your scoring logic is configurable without touching Python code.

Log Input

langfuse_context.update_current_observation(
    input={"question": question, "model": model}
)

langfuse_context.update_current_observation(...) is used to attach new information to the current observation in a Langfuse trace.

Think of a trace as one full request, and an observation as one step inside that request (e.g., LLM call, embedding call, retrieval step).

Perform the LLM Call + Measure Latency

start_time = time.time()
# Make LLM call
response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": question}
    ],
    temperature=temperature,
    max_tokens=max_tokens
)
# Calculate latency
latency_ms = (time.time() - start_time) * 1000

This gives us:

• Real wall-clock latency
• First-token + completion latency combined
• Values used for threshold checking

Compute Answer Length + Quality Score

answer_length = len(answer)
# Calculate quality score
if answer_length < min_length:
    quality_score = 0.3
elif answer_length >= good_length_threshold:
    quality_score = 1.0
else:
    quality_score = 0.3 + (
        0.7 * (answer_length - min_length) /
        (good_length_threshold - min_length)
    )

This snippet measures the length of the generated answer and uses it to compute a simple quality score: if the answer is too short (below min_length), it assigns a low score of 0.3; if it exceeds the good_length_threshold, it gives a perfect score of 1.0. Otherwise, it linearly scales the score between 0.3 and 1.0 based on how close the answer_length is to the ideal range. This provides a lightweight heuristic for judging response completeness without requiring complex evaluation logic.

Update the Observation (Output + Usage + Metadata)

# Update observation with results and custom metrics
langfuse_context.update_current_observation(
    output={"answer": answer, "quality_score": quality_score},
    usage={
        "input": response.usage.prompt_tokens,
        "output": response.usage.completion_tokens,
        "total": response.usage.total_tokens
    },
    metadata={
        "latency_ms": round(latency_ms, 2),
        "answer_length": answer_length
    }
)

This block updates the current Langfuse observation with everything needed to record the model’s performance: it logs the generated answer and its quality score, tracks token usage from the model response (input, output, and total), and attaches custom metadata such as request latency and the length of the returned answer. Together, these fields give you a complete view of each evaluation run, including what the model produced, how much it cost, and how efficiently it responded, making it easier to analyze and compare results across experiments.

What this adds to Langfuse:

• Answer text
• Quality score
• Token usage
• Latency
• Derived metrics (answer_length)

This gives you the same view you would see in enterprise-grade observability tools.

Attach a Score to the Trace

# Score the trace
langfuse_context.score_current_observation(
    name="quality",
    value=quality_score,
    comment=f"Based on answer length ({answer_length} chars)"
)

This line evaluates the current Langfuse observation by attaching a custom score named “quality” to the trace. It records the numerical quality_score, your own metric for evaluating the model’s answer, and adds a short comment explaining the basis of that score, in this case referencing the answer_length. Scoring observations like this makes it easy to compare model responses, analyze performance over time, and visualize quality trends directly in the Langfuse dashboard.

In short, this creates a visible, numeric score inside the Langfuse dashboard.

This is extremely powerful for:

• model comparisons
• regression testing
• degradation alerts
• ranking model performance

Running the Evaluation Pipeline

@observe(name="evaluation_pipeline")
def run_evaluation(question: str):
    """Wrapper to create a trace context for the evaluation."""
    from datetime import datetime
   
    # Add timestamp to make each run unique
    langfuse_context.update_current_trace(
        metadata={"run_time": datetime.now().isoformat()}
    )
   
    answer, score = generate_and_score(question)
   
    print(f"\n✅ Answer: {answer}\n")
    print(f"📊 Quality Score: {score:.2f}\n")
   
    trace_id = langfuse_context.get_current_trace_id()
    if trace_id:
        print(f"🔍 View trace with scores: https://cloud.langfuse.com/trace/{trace_id}")
        print(f"📋 Trace ID: {trace_id}")
    print("="*50 + "\n")
   
    return answer, score

This function defines an evaluation pipeline using the @observe decorator, which tells Langfuse to treat every call as a traced, observable run. When the function starts, it imports datetime and immediately updates the active Langfuse trace with a timestamp so each evaluation run is uniquely identifiable. This metadata is helpful when you are comparing multiple experiments, debugging behavior, or tracking quality trends over time.

The core of the function calls generate_and_score(question), which returns an AI-generated answer along with a numerical quality score. Both values are printed in a human-friendly format, and the function then retrieves the current trace_id from Langfuse. If a trace exists, it prints a direct link to view the full run, including metrics and scores, in the Langfuse dashboard.

Finally, the function returns the answer and score so they can be used downstream, while also visually marking the end of the run in the terminal output.

It adds:

• timestamp metadata
• parent-level trace context
• output printing
• a link to view the trace

Running the LLM Evaluation Metrics Pipeline

A typical terminal run will show:

==================================================
Evaluation with Custom Scoring
==================================================

Question: What are neural networks?

📊 Quality Score: 0.82 (answer length: 112 chars)
📊 Latency: 212.45ms
📊 Tokens: 14 → 72

🔍 View trace with scores: http://localhost:3000/trace/01HY3SJQH9...
==================================================
⏳ Flushing traces to Langfuse...
✅ Traces sent!

This output must appear in the lesson. It helps the reader validate correctness.

Conceptual Mockup: Evaluation Trace in Langfuse

Before looking at the real dashboard output, here is a clean conceptual view of what an evaluation trace looks like inside Langfuse.

Real Trace from Our Self-Hosted Langfuse Dashboard

Now, let us look at the actual trace generated by our evaluation script.

This is exactly what you should see when running:

$ python src/evaluation_metrics.py

Your Langfuse dashboard will show:

• evaluation_pipeline: as the parent trace
• generate_and_score: as the nested span
• full inputs (question, system message, model config)
• full outputs (LLM answer + quality score)
• token usage (input, output, total)
• latency measured manually
• metadata from config.yaml
• score badge showing the computed quality metric

While Figure 6 shows the actual Langfuse trace captured during execution, the diagram below abstracts the same process into a clear evaluation pipeline. It highlights how the LLM response is generated, how evaluation metrics are computed, and how both the raw outputs and derived quality scores are attached to a single trace before being logged to Langfuse.

Why LLM Evaluation Metrics Matter

By adding evaluation metrics:

• You detect model degradation
• You compare models or prompts
• You measure latency regressions
• You track token cost spikes
• You get quality insights per request

This pushes your system beyond “debuggable” into evaluated, which is critical for anything involving RAG, agents, or multi-step pipelines.

In this section, you learned how to:

• Instrument LLM calls with decorators
• Compute custom evaluation metrics
• Attach quality scores to traces
• Visualize scores, latency, and tokens inside Langfuse
• Wrap everything inside an evaluation_pipeline

With this, tracing evolves from simple diagnostics into actual LLM evaluation.

vLLM Diagnostics and Health Checks for LLM Observability

Before we evaluate model outputs or analyze Langfuse traces, we need to make sure the underlying engine vLLM is alive, reachable, and responding correctly. If vLLM is down, every script in this lesson fails. If the model is still loading, requests time out. If ports are wrong, you will get cryptic errors that look like Langfuse problems but are actually vLLM issues.

To prevent all of that, we use health_check.py, a dedicated diagnostic tool that validates your entire local LLM runtime before you run any tracing or scoring scripts.

This script confirms 3 things:

• Is the vLLM server running and responding?
• Are models actually loaded?
• Can the model generate text?

If all 3 pass, your observability stack is ready.

What the vLLM Health Check Script Validates

health_check.py performs 3 layers of validation:

Layer 1: Infrastructure health

• Calls /health endpoint
• Checks whether the vLLM server is reachable
• Confirms that the port and base URL match your config

Layer 2: Model readiness

• Calls /v1/models
• Ensures at least one model is loaded
• Detects if vLLM is still downloading or initializing the model

Layer 3: LLM generation test

• Sends a simple prompt: “Say ‘OK’ if you’re working.”
• Ensures the model produces an actual response

This prevents 95% of “It’s not working” confusion.

Code Walkthrough: health_check.py

We now walk through the entire script, grouped logically rather than line by line, following typical PyImageSearch style.

Configuration and Imports

import sys
import httpx
from llm_utils import get_llm_client
from config import get_llm_config

The script uses:

• httpx: for fast HTTP checks
• get_llm_client(): to issue a test generation
• get_llm_config(): to load the base URL from your YAML config

No hard-coded URLs, which keeps the system in sync with config.yaml.

Checking vLLM Health

def check_vllm_health(base_url: str = None, timeout: int = 5) -> bool:
    """
    Check if vLLM server is healthy.
   
    Args:
        base_url: vLLM server base URL (defaults to config.yaml)
        timeout: Request timeout in seconds
       
    Returns:
        True if server is healthy, False otherwise
    """
    # Load base_url from config if not provided
    if base_url is None:
        llm_config = get_llm_config()
        base_url = llm_config.get("base_url", "http://localhost:8000/v1")
        base_url = base_url.rstrip("/v1")
   
    health_url = f"{base_url}/health"
    models_url = f"{base_url}/v1/models"
   
    print(f"🔍 Checking vLLM server at {base_url}...")

If base_url is not provided, the vLLM URL is loaded from config.yaml.

Next:

Health endpoint check

try:
        # Check health endpoint
        with httpx.Client(timeout=timeout) as client:
            response = client.get(health_url)
            if response.status_code == 200:
                print(f"  ✅ Health check passed")
            else:
                print(f"  ❌ Health check failed (status: {response.status_code})")
                return False

A healthy vLLM server returns:

{"status": "ok"}

If this fails, vLLM is down, so no tracing or scoring will work.

Models endpoint check

       # Check models endpoint
        with httpx.Client(timeout=timeout) as client:
            response = client.get(models_url)
            if response.status_code == 200:
                models = response.json().get("data", [])
                if models:
                    print(f"  ✅ Models available: {[m['id'] for m in models]}")
                else:
                    print(f"  ⚠️  No models loaded yet (still initializing?)")
                    return False
            else:
                print(f"  ❌ Models endpoint failed (status: {response.status_code})")
                return False
       
        return True

A healthy response contains:

{
  "data": [
     {"id": "meta-llama/Llama-2-7b-chat-hf"}
   ]
}

If this list is empty, the model is still loading.

Error handling

The script gracefully handles:

• connection failure
• timeouts
• unexpected JSON
• wrong ports
• wrong base_url

And prints clear, actionable fixes.

Testing LLM Generation

def test_llm_generation() -> bool:
    """Test simple LLM generation."""
    print("\n🔍 Testing LLM generation...")
   
    try:
        client = get_llm_client(timeout=30)
        response = client.chat.completions.create(
            model="meta-llama/Llama-2-7b-chat-hf",
            messages=[{"role": "user", "content": "Say 'OK' if you're working."}],
            max_tokens=10
        )
       
        answer = response.choices[0].message.content
        print(f"  ✅ Generation successful: {answer[:50]}...")
        return True
       
    except Exception as e:
        print(f"  ❌ Generation failed: {e}")
        return False

This test:

• Instantiates the OpenAI client
• Sends a tiny one-line prompt
• Validates the model answers with at least something

If the model cannot generate, your entire tracing pipeline will also fail.

The Entry Point

if __name__ == "__main__":
     main()

This is the command you will run before every other script.

It prints:

• vLLM health
• Model availability
• Generation test

And guides you through failures with friendly hints:

“Start vLLM: docker-compose up -d”

“Wait 2-3 minutes for model download”

“Check docker logs”

This makes beginner troubleshooting seamless.

Why vLLM Health Checks Matter for LLM Observability

If vLLM is unhealthy, every tracing script fails.

This script prevents:

• Running manual tracing while vLLM is down
• Chasing decorator errors that are actually connection errors
• Confusing Langfuse ingestion errors with model-loading delays
• token errors caused by uninitialized models
• Timeouts that look like Langfuse bugs

It gives readers a clean, deterministic start before diving into observability.

In this section, you learned:

• How to verify vLLM health
• Why the /health and /v1/models endpoints matter
• How to test real generation
• How to diagnose common startup issues
• How to ensure the entire tracing pipeline will work

With your environment confirmed healthy, you are ready to score model outputs and analyze evaluation traces in Langfuse.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: June 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, you moved beyond simply capturing traces and learned how to measure, score, and diagnose the quality of your LLM pipeline. Lesson 1 gave you observability; Lesson 2 gave you interpretation.

You began by understanding why manual tracing still matters even when decorators exist. Manual spans give you full control over trace structure, metadata, and custom logging, making them essential for debugging agent loops, multi-step pipelines, and retrieval-heavy systems. You then revisited the decorator pattern and learned when to use each approach so your real-world projects can choose the right instrumentation strategy.

Next, you implemented true evaluation-driven observability using the Langfuse scoring interface. You wrapped LLM calls with @observe, computed a custom “quality score,” tracked latency and token usage, and attached structured metrics directly to your traces. This transformed your dashboard from a simple trace viewer into a performance analytics console.

Finally, you validated your infrastructure using a robust health-check system. Before any tracing or scoring happens, health_check.py ensures vLLM is running, the model is loaded, and real generation works end-to-end. This eliminates guesswork and gives you a reliable foundation for more advanced workflows.

By the end of this lesson, your observability pipeline now supports:

• manual low-level traces
• decorator-based nested traces
• latency instrumentation
• token usage insights
• custom evaluation scores
• metadata-rich pipeline summaries
• infrastructure-level diagnostics

Together, these upgrades elevate your system from “traced” to measured, from “visible” to actionable.

Citation Information

Singh, V. “Manual Tracing, Scores, and Evaluation with Langfuse (Self-Hosted),” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/24p06

@incollection{Singh_2026_manual-tracing-scores-evaluation-langfuse-self-hosted,
  author = {Vikram Singh},
  title = {{Manual Tracing, Scores, and Evaluation with Langfuse (Self-Hosted)}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/24p06},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Manual Tracing, Scores, and Evaluation with Langfuse (Self-Hosted) appeared first on PyImageSearch.

LLM Observability with Self-Hosted Langfuse and vLLM

In this lesson, you will finally demystify what Large Language Model (LLM) observability actually is. It is not just logs or print statements. It is a full, end-to-end view of how your AI system behaves in real-world conditions.

You will learn why modern LLM apps need more than “it works on my machine,” and how traces, token usage, latency, and model interactions become powerful tools for debugging and optimization.

Next, you will roll up your sleeves and self-host Langfuse locally, connect it to a vLLM server, and run your first fully instrumented LLM pipeline from prompt to response.

By the end, you will be exploring live traces in the Langfuse UI, inspecting individual requests, understanding where time is spent, and building a solid foundation for debugging, improving, and scaling every LLM workflow you create.

This lesson is the 1st in a 3-part series on LLM observability with Langfuse:

1. LLM Observability with Self-Hosted Langfuse and vLLM (this tutorial)
2. Lesson 2
3. Lesson 3

To learn how to self-host Langfuse, connect it to vLLM, and build end-to-end LLM observability from the ground up, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Introduction to LLM Observability with Langfuse

Modern LLM applications behave very differently from traditional software. They are probabilistic, non-deterministic, sensitive to prompt phrasing, and often expensive to run. Debugging them requires far more than print statements or simple application logs — you need visibility into how your entire LLM pipeline behaves at runtime.

This section introduces the foundations of LLM observability, explains why classical ML monitoring tools fall short, and sets the stage for building a complete self-hosted Langfuse + vLLM observability stack.

What Problem Does LLM Observability Solve?

LLMs fail in ways ordinary software doesn’t:

• They hallucinate confidently.
• They produce different answers for the same input.
• They slow down under load due to tokenizer/model server issues.
• They cost real money per token.
• They silently degrade when context windows overflow.
• They chain multiple steps, making errors hard to pinpoint.

Without observability, you are essentially debugging blind.

LLM observability gives you visibility into:

• What prompt was sent?
• What did the LLM actually output?
• How long did it take?
• How many tokens did it use?
• Where did a pipeline fail?
• Was this output good or bad?
• What downstream component was impacted?

In short: Observability turns your LLM pipeline from a black box into a glass box.

Logs vs Metrics vs Traces (Why Logs Alone Fail)

Modern systems use 3 observability pillars:

Logs

Unstructured text messages. Good for errors; terrible for understanding multi-step LLM pipelines.

Metrics

Numerical time-series (e.g., latency, tokens/sec). Good for dashboards and alerts.

Traces

End-to-end structured records of what happened across a pipeline.

Traces are THE critical component for LLM apps because a single request may produce:

• multiple sub-steps
• multiple model calls
• embeddings
• retrieval calls
• tool invocations
• agent planning
• scoring
• post-processing

Logs tell you what happened. Metrics tell you how often. Traces tell you why.

Why LLM Apps Require Traces, Not Just Logs

LLM-specific debugging demands visibility into things you cannot get from logging alone:

• Prompt tracking: See every prompt, system message, and user message.
• Chain-of-thought structure: (Even if hidden, you can capture high-level execution steps.)
• Latency breakdown: Where time is spent: tokenization? forward pass? retrieval?
• Token usage visibility: Cost control + throughput estimation.
• Hallucination hotspots: Which prompts or contexts fail most?
• Pipeline correctness: Observations from retrieval → reasoning → generation.

What Is Langfuse? (And Why It Is the Right Tool)

Langfuse is an open-source observability platform designed specifically for LLM apps. It captures:

• Traces
• Spans
• Prompt metadata
• Inputs and outputs
• Token usage
• Latencies
• Scores (quality, correctness, safety)

…and displays them in a clean, production-grade UI.

You can think of it as:

“Prometheus + Grafana + MLflow, but specifically for LLM pipelines.”

Why Not MLflow or Weights & Biases?

How Langfuse Fits into an LLM Observability Stack

Before building anything, consider the mental model:

• Your Python LLM app: Sends prompts and metadata
• Langfuse SDK: Records traces locally inside your code
• vLLM Server (port 8000): Handles the actual model inference
• Langfuse Server (port 3000): Receives trace data from the SDK
• Langfuse Worker: Aggregates, transforms, and prepares data for the dashboard
• PostgreSQL database: Stores all traces, spans, scores, and token counts
• Langfuse UI dashboard: Displays everything in real time

This flow is the backbone of LLM observability.

Why Self-Hosted Langfuse Instead of Cloud?

When we first integrated Langfuse Cloud during development, we immediately ran into:

• trace delivery delays
• out-of-order spans
• slow UI updates
• unreliable real-time feedback

This is a problem when you are developing an agent or RAG system and need to see:

• the exact prompt
• the exact context
• the exact output
• the exact cost
• immediately after running your script.

So we switched to:

Self-Hosted Langfuse + Local vLLM

Benefits:

• Real-time, near-instant traces
• Fully local development
• No Internet dependency
• Faster iteration loops
• Full control of database and dashboard
• Ideal for agent debugging and RAG evaluation

📌 OPTIONAL CALLOUT

One short bullet note: We still show the Cloud API flow briefly, but everything you build in this module uses the self-hosted setup for real-time performance.

What You Will Build in This Lesson

By the end of Lesson 1, you will have a complete local observability foundation:

Infrastructure

• Self-hosted Langfuse Server
• Langfuse Worker (required for dashboards)
• PostgreSQL database
• vLLM model server (OpenAI-compatible API)

Tracing Skills

• How to instrument an LLM call
• How to build hierarchical traces (pipeline → model call)
• How to log prompts, outputs, latencies, and token usage
• How to visualize traces instantly in the Langfuse User Interface (UI)

What You Will Actually Run

• Decorator-based tracing (tracing_decorator.py)
• Baseline app with no tracing (basic_llm_app.py)
• vLLM-connected LLM client (llm_utils.py)
• Config loaders (config.py)

Langfuse Architecture for LLM Observability

Before we start installing anything, let us zoom out and understand the architecture of the observability stack you are about to build. Langfuse is not just a dashboard. It is a coordinated system of services that receives traces, stores them, aggregates them, and displays them in real time. Your LLM app, Langfuse, vLLM, and PostgreSQL all work together to form a complete observability pipeline.

Think of this section as building your mental model. Once you understand these flows, all the Docker configuration, YAML files, keys, and scripts will make perfect sense.

The High-Level Architecture

At the core, your pipeline is simple:

• Your Python LLM app: executes inference and logs traces
• Langfuse Python SDK: captures all observability data
• vLLM Server: handles the actual LLM generation
• Langfuse Server: receives trace, span, and token data
• PostgreSQL: stores all traces, metadata, and scores
• Langfuse Worker: aggregates data for dashboards
• Langfuse UI: visualizes everything instantly

This architecture ensures that every LLM call becomes a structured trace that you can drill into, including latencies, inputs, outputs, steps, errors, and token details.

How a Single LLM Request Turns Into a Trace

Every time your code calls client.chat.completions.create(...), Langfuse performs 3 major steps behind the scenes:

• Observe the call: capture input, parameters, metadata.
• Record the output: LLM response, tokens, shapes, errors.
• Create and update a trace hierarchy: pipeline spans, child spans, nested steps.

For example:

llm_pipeline (trace)
    ├── retrieve_context (span)
    ├── rerank_candidates (span)
    └── generate_answer (span)

Even in Lesson 1 (where we only use decorators), you will already produce parent → child traces automatically.

Without this structure, debugging multi-step LLM pipelines becomes guesswork.

The Four Core Components You Will Deploy

You will deploy 4 services using Docker Compose:

1. vLLM Server (Port 8000)

Your local LLM inference engine.

It exposes an OpenAI-compatible Application Programming Interface (API) endpoint:

http://localhost:8000/v1

Your Python scripts send prompts here.

2. Langfuse Server (Port 3000)

The brains of the observability system.

It receives traces from the Python SDK, stores them, and exposes the dashboard.

3. Langfuse Worker

Most tutorials miss this, but you cannot get dashboards without the Langfuse Worker.

It processes:

• aggregations
• analytics
• score updates
• background tasks

Without the Langfuse Worker:

you will see traces, but your dashboard will be empty.

4. PostgreSQL (Port 5433 → 5432)

Stores everything:

• traces
• spans
• metadata
• scores
• projects
• settings

It provides the persistence layer that the Langfuse Server depends on.

How These Components Communicate (Data Flow)

Let us make the full pipeline explicit:

• Your script sends an inference request to vLLM.
• The Langfuse SDK in your script sends trace info to Langfuse Server.
• Langfuse Server writes raw trace data into PostgreSQL.
• Langfuse Worker processes raw data to generate:
  • analytics
  • histograms
  • span trees
  • scores
• The Langfuse Web UI reads processed data and displays:
  • full trace trees
  • input/output pairs
  • token usage
  • latency heatmaps
  • error stacks

This is the “observability heartbeat” that runs for every request.

Why Understanding LLM Observability Architecture Matters

Before diving into code, it is important to visualize this system because:

• It prevents confusion when running Docker for the first time.
• You will instantly understand errors like “Worker not running” or “Database unavailable”.
• You will know exactly where to look when traces do not appear.
• You will develop intuition about how requests become saved spans.

Once this architectural layer clicks, every file in docker-compose.yml, every script in src/, and every dashboard panel will feel obvious.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Setting Up a Self-Hosted Langfuse and vLLM Stack

Before we can trace a single LLM call, we need to set up a clean project skeleton and a fully functioning self-hosted observability stack. In this section, you will configure the environment, install dependencies, review the project layout, understand each configuration file, and bring up the Langfuse + vLLM infrastructure using Docker Compose.

Everything that comes later (tracing, scoring, evaluation, debugging) depends on getting this foundation right.

Project Structure Overview

Here is the complete repository structure we will use throughout this lesson:

├── configs
│   └── config.yaml
├── docker-compose.yml
├── README.md
├── requirements.txt
└── src
    ├── basic_llm_app.py
    ├── config.py
    ├── evaluation_metrics.py
    ├── health_check.py
    ├── llm_utils.py
    ├── run_all_examples.py
    ├── tracing_decorator.py
    └── tracing_manual.py

At a high level:

• configs/: stores global configuration used by every example.
• src/: contains the LLM application scripts, utilities, and tracing examples.
• docker-compose.yml: defines the entire Langfuse + vLLM infrastructure.
• requirements.txt: defines Python dependencies.
• .env.example: defines required environment variables.

We will walk through each piece, focusing not on the logic inside every file, but on how the system is designed and how everything connects.

Installing Dependencies

Install the required Python packages:

pip install -r requirements.txt

The key dependencies in this project are intentionally minimal. We use the following packages:

• langfuse>=2.0.0: provides the observability SDK and the @observe decorator
• openai>=1.0.0: is required because vLLM exposes an OpenAI-compatible API endpoint
• python-dotenv: loads .env environment variables
• pyyaml: reads configuration values from config.yaml
• httpx and requests: handle health checks and HTTP communication
• numpy: supports scoring and numeric utilities

Together, these packages form the lightweight foundation for our self-hosted observability stack.

Configuring Environment Variables

Copy .env.example into .env:

cp .env.example .env

Then update the following values after starting Langfuse:

LANGFUSE_PUBLIC_KEY="pk-lf-xxxx"
LANGFUSE_SECRET_KEY="sk-lf-xxxx"
LANGFUSE_HOST=http://localhost:3000

OPENAI_BASE_URL=http://localhost:8000/v1
OPENAI_API_KEY=dummy

A few key points:

• Langfuse keys come from your local dashboard once you create a project.
• vLLM does not require authentication, but the OpenAI client still requires an API key value, so "dummy" works.
• If you use Hugging Face models that are not cached, you may need a token.

This .env file becomes the backbone for all examples.

Centralized Configuration (configs/config.yaml)

Instead of scattering options across scripts, everything is configured through one YAML file:

llm:
  base_url: "http://localhost:8000/v1"
  model: "meta-llama/Llama-2-7b-chat-hf"
  temperature: 0.7
  max_tokens: 300

langfuse:
  host: "http://localhost:3000"
  project_name: "llm-observability-selfhosted"

evaluation:
  enable_scoring: true
  max_latency_ms: 5000
  min_length: 20
  good_length_threshold: 100

This allows you to:

• Switch models without changing code
• Tune evaluation logic centrally
• Redirect LLM traffic to remote endpoints if needed
• Adjust Langfuse Server location

Every script loads from this file automatically.

Utility Modules (src/config.py and src/llm_utils.py)

These 2 utilities prevent duplication across all examples.

config.py: Central Configuration Loader

This module provides:

• load_config(): returns parsed YAML config
• get_llm_config(): returns model, temp, max_tokens
• get_langfuse_config(): returns host, project_name
• get_evaluation_config(): returns scoring thresholds

This keeps every script flexible and model-agnostic.

llm_utils.py: Consistent vLLM Client Factory

vLLM supports the OpenAI Python client natively:

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

This module wraps it into a reusable function:

• Validates environment variables
• Loads model name from config.yaml
• Handles default base_url
• Sets request timeouts
• Returns the (client, model) tuple when requested

Every tracing example uses this function.

The Self-Hosted Stack (docker-compose.yml)

This is the heart of the system.

docker-compose.yml defines:

Langfuse Server

• Runs the frontend + API
• Exposes port 3000
• Performs authentication, API key creation, and trace storage

Langfuse Worker

• Mandatory for dashboards
• Processes traces
• Updates analytics, charts, latency heatmaps

PostgreSQL

• Persistence layer for traces, spans, scores
• Exposed on port 5433 (to avoid conflicts)

vLLM Model Server (GPU or CPU)

• Exposes OpenAI-compatible API at http://localhost:8000/v1
• Runs Llama 2 by default
• Enables fast, local inference for testing

You can start everything with:

docker-compose --profile gpu up -d

Or if you don’t have a GPU:

docker-compose --profile cpu up -d

Verify services:

docker-compose ps

Visit the Langfuse dashboard: http://localhost:3000

Note: If you are running the server on a remote machine, do not forget to use SSH port forwarding. Otherwise, you will not be able to access the Langfuse UI dashboard from your local machine.

Bringing Up the Entire Observability Stack

Once configuration is in place:

docker-compose --profile gpu up -d

Then:

• Go to http://localhost:3000
• Create a project
• Copy your public and secret keys
• Paste them into .env
• Restart your Python script

You now have:

• A live model server (vLLM)
• A local observability platform (Langfuse)
• A database storing every trace
• Real-time dashboards
• A clean Python project ready for tracing

The foundation is complete. Next, we will write and trace our first LLM call.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Baseline LLM Application (Before Observability)

Before we wire in Langfuse, we need a clean baseline: a tiny LLM app that talks to vLLM, prints an answer, and knows nothing about traces, latency, or tokens.

This section walks through src/basic_llm_app.py end-to-end so we have a clear “before” picture of life without observability.

The Full Baseline Script

Here is the full file we will dissect:

"""
Basic LLM Application (No Tracing Baseline)

Simple pipeline using local vLLM server.
This version has NO tracing - compare with tracing_decorator.py
"""

from llm_utils import get_llm_client
from config import get_llm_config

# Initialize vLLM client with model from config
client, model = get_llm_client(load_model_from_config=True)

The docstring sets the tone very clearly:

this is a “no tracing” baseline that we will later compare against a traced version.

We import:

• get_llm_client: a reusable helper that knows how to connect to vLLM using OPENAI_BASE_URL and OPENAI_API_KEY.
• get_llm_config: a small wrapper around config.yaml so we don’t hardcode model parameters in the code.

client, model = get_llm_client(load_model_from_config=True) gives us:

• an OpenAI-compatible client already pointed at http://localhost:8000/v1
• the model name loaded from configs/config.yaml.

At this point, the app can already talk to vLLM, but we still have zero observability.

Generating an Answer (With No Tracing at All)

def generate_answer(question: str) -> str:
    """Generate answer using vLLM - NO tracing."""
    # Load config
    llm_config = get_llm_config()
    temperature = llm_config.get("temperature", 0.7)
    max_tokens = llm_config.get("max_tokens", 300)
   
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": question}
            ],
            temperature=temperature,
            max_tokens=max_tokens
        )
        return response.choices[0].message.content
    except Exception as e:
        print(f"Error: {e}")
        print("Tip: Make sure vLLM is running (docker-compose up -d)")
        raise

Loading config per call

Inside generate_answer, we first pull generation settings from YAML:

• llm_config = get_llm_config() loads the llm: section from configs/config.yaml.
• temperature and max_tokens are read with sensible defaults (0.7 and 300) in case the config is missing keys.

This keeps your generation parameters config-driven, not hardcoded, which is great for experiments, but still does not give you any tracing.

Making the chat completion request

The try block does a standard OpenAI-style chat completion call:

• model=model uses the vLLM-hosted Llama model from your config.
• messages=[...] constructs a simple conversation with:
  • a system message: "You are a helpful assistant."
  • a user message: the question string passed into the function.
• temperature and max_tokens control creativity and output length.

vLLM behaves like the OpenAI API here, so response.choices[0].message.content gives us the generated answer, which is then returned.

Error handling (still without observability)

If anything goes wrong (vLLM not running, bad network, misconfiguration), the except block:

• Prints the raw error message.
• Prints a helpful hint: Make sure vLLM is running (docker-compose up -d).
• Re-raises the exception so the script fails loudly.

This is basic error handling, but notice what is still missing:

• No trace of which prompt failed.
• No structured record of latency or context.
• No way to inspect this error later in a dashboard.

Even errors are invisible beyond your terminal scrollback.

Running the “Invisible” Pipeline

def run_simple_pipeline(question: str):
    """Simple pipeline without tracing - baseline example."""
    print(f"\n{'='*50}")
    print(f"Question: {question}")
    print(f"{'='*50}\n")
   
    print("Generating answer (no tracing)...")
    answer = generate_answer(question)
   
    print(f"✅ Answer:\n{answer}\n")
    print(f"{'='*50}\n")

run_simple_pipeline is deliberately small and linear:

• It prints a visual separator and echoes the question.
• It calls generate_answer(question), the black-box LLM call.
• It prints the answer and another separator.

This gives you a nice terminal UX, but again, it is only surface-level:

• You see the question and final answer.
• You do not see any internal steps.
• You do not know how long it took.
• You do not know how many tokens it used or how much it cost.
• You cannot compare this run with previous ones.

For anything beyond a toy demo, this is not enough.

The __main__ Block

if __name__ == "__main__":
    question = "What is machine learning?"
    run_simple_pipeline(question)

The entry point is intentionally as minimal as possible:

• It defines a simple default question: "What is machine learning?"
• It calls run_simple_pipeline(question)

This makes basic_llm_app.py runnable as a one-shot script:

python src/basic_llm_app.py

It is perfect for quick manual testing and serves as a control group when we later add Langfuse tracing and see how much more we can observe.

Why This Baseline Is Not Enough

With this script, your entire view of the system is:

• one printed question
• one printed answer
• and maybe an error line if something crashes

You cannot answer:

• “Why was this slow?”
• “What exact prompt + params did we send?”
• “How many tokens did we consume?”
• “Where did the pipeline fail?”
• “Why is today’s behavior different from yesterday’s?”

For serious LLM work involving RAG systems, agents, evaluation runs, and A/B testing, this is debugging in the dark.

That is exactly what Langfuse is going to fix.

Adding LLM Observability with the Langfuse @observe Decorator

At this point, you have seen how an uninstrumented LLM pipeline behaves: it works, but it hides everything that matters. Now it is time to unlock real observability using the Langfuse @observe decorator, the cleanest and most powerful way to add tracing in Langfuse 2.x.

In this section, we will transform the baseline pipeline into a fully observable workflow, capturing:

• prompts
• outputs
• latency
• token usage
• metadata
• hierarchy of steps (pipeline → model call)
• trace IDs you can click and inspect instantly in Langfuse

This is where everything finally becomes visible.

Imports, Initialization, and Configuration Logging

import os
from langfuse.decorators import observe, langfuse_context
from llm_utils import get_llm_client
from config import get_llm_config

We import:

• observe → adds tracing automatically
• langfuse_context → lets us update spans programmatically
• our reusable LLM client and config loaders

Before anything happens, the script prints the Langfuse configuration:

print("\n" + "="*70)
print("🔧 LANGFUSE CONFIGURATION")
print("="*70)
print(f"📍 LANGFUSE_HOST: {os.getenv('LANGFUSE_HOST', 'NOT SET')}")
print(f"🔑 LANGFUSE_PUBLIC_KEY: {os.getenv('LANGFUSE_PUBLIC_KEY', 'NOT SET')[:20]}...")
print(f"🔐 LANGFUSE_SECRET_KEY: {os.getenv('LANGFUSE_SECRET_KEY', 'NOT SET')[:20]}...")
print("="*70 + "\n")

This is extremely practical.

It confirms:

• Langfuse host
• truncated keys
• environment setup correctness

If anything is misconfigured, this block saves you debugging time before you even send a single request.

Finally, we initialize the LLM client:

client, model = get_llm_client(load_model_from_config=True)

The model name and base URL automatically load from the YAML config.

Tracing a Single LLM Call with @observe

Here is the traced model-call function:

@observe(name="generate_answer")
def generate_answer(question: str) -> str:

This single decorator:

• creates a new observation
• wraps the function execution
• automatically timestamps execution
• links child spans to parent spans

Step 1: Recording Inputs

Inside the function, the first thing we do is explicitly log the input:

langfuse_context.update_current_observation(
    input={"question": question, "model": model}
)

This ensures Langfuse displays:

• full question
• selected model
• temperature and max_tokens (we will update outputs later)

Step 2: Tracking Latency Manually

Although Langfuse timestamps spans automatically, we want explicit latency measurement:

import time
start_time = time.time()

Then we perform the vLLM call:

response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": question}
    ],
    temperature=temperature,
    max_tokens=max_tokens
)

Step 3: Computing Latency + Extracting Answer

latency_ms = (time.time() - start_time) * 1000
answer = response.choices[0].message.content

Adding Outputs, Token Usage, and Metadata

This is the heart of observability:

langfuse_context.update_current_observation(
    output={"answer": answer},
    usage={
        "input": response.usage.prompt_tokens,
        "output": response.usage.completion_tokens,
        "total": response.usage.total_tokens
    },
    metadata={"latency_ms": round(latency_ms, 2)}
)

With a single update call, you give Langfuse:

Outputs

• final LLM response

Usage

• prompt tokens
• completion tokens
• total tokens

Essential for:

• cost analysis
• throughput understanding
• debugging prompt inflation

Metadata

• latency_ms (explicit + human-readable)

This is exactly what the baseline pipeline could not show.

Print statements reinforce visibility:

print(f"📊 Latency: {latency_ms:.2f}ms")
print(f"📊 Tokens: {response.usage.prompt_tokens} → {response.usage.completion_tokens} (total: {response.usage.total_tokens})")

Building Nested Traces with run_pipeline()

The pipeline function also uses @observe:

@observe(name="llm_pipeline")
def run_pipeline(question: str):

This creates a parent span.

Any traced function called inside run_pipeline() automatically becomes a child span.

Updating the Trace Metadata

langfuse_context.update_current_trace(
    name="decorator_pipeline",
    metadata={"method": "decorator"}
)

This changes the trace title in the Langfuse UI and adds custom metadata so you always know which instrumentation method you used.

Calling the Nested Span

answer = generate_answer(question)

This produces:

llm_pipeline (parent)
└── generate_answer (child)

The tree structure appears instantly in Langfuse.

Linking Back to the UI

trace_id = langfuse_context.get_current_trace_id()
print(f"🔍 View trace: {langfuse_host}/trace/{trace_id}")

This clickable URL directly opens the exact trace and is extremely useful while iterating locally.

Flushing Traces Before Exit

Short-lived scripts often exit before Langfuse sends data.

This line ensures nothing is lost:

langfuse_context.flush()
print("✅ Traces sent!\n")

Without flushing, traces may appear incomplete or missing entirely.

Why the Decorator Approach Is the Best Default

This is why nearly every modern Langfuse tutorial and production workflow recommends decorators as the first instrumentation layer.

What You Just Built

Your LLM pipeline now has:

• Clickable traces
• Per-step metadata
• Latency and token breakdown
• Nested trace hierarchy
• Real-time Langfuse UI updates
• Automatic error propagation

This completes the transformation from:

a blind LLM script → a fully observable workflow.

Running and Verifying a Self-Hosted Langfuse Observability Stack

By now, we have all the moving parts ready:

• the Langfuse Server + Langfuse Worker + PostgreSQL
• the vLLM model server
• our traced LLM pipeline using the @observe decorator

In this section, we will bring everything online, verify the system health, and run the traced pipeline end-to-end. By the end, you will see your first real traces appear instantly inside the Langfuse dashboard.

Start the Self-Hosted Stack

All core services, including Langfuse Server, Langfuse Worker, PostgreSQL, and vLLM, run through your project’s docker-compose.yml.

To start everything with GPU acceleration:

docker compose --profile gpu up -d

Or, if you don’t have a GPU:

docker compose --profile cpu up -d

This launches:

You can check everything is healthy using:

docker compose ps

Expected output (sample):

NAME                 STATUS              PORTS
langfuse-server      healthy             0.0.0.0:3000->3000/tcp
langfuse-worker      running            
langfuse-postgres    healthy             0.0.0.0:5433->5432/tcp
vllm-server          healthy             host:8000->8000/tcp

Tip:

If langfuse-worker is not running, your dashboard will be empty.

If vllm-server is not healthy, your LLM calls will fail.

Verify Each Component Individually

Langfuse Server (UI)

Open:

http://localhost:3000

You should see:

• The Langfuse login screen
• The dashboard panel
• Empty traces (for now)

vLLM Health

Visit:

http://localhost:8000/health

Expected JSON:

{"status": "ok"}

If this endpoint fails, no LLM calls will work.

PostgreSQL Health (optional)

Inside Docker:

docker compose logs langfuse-postgres

Look for:

database system is ready to accept connections

Run Your First Traced Pipeline

Now run the decorator-instrumented script:

python src/tracing_decorator.py

You should see terminal output like:

==================================================
Question: Explain neural networks briefly
==================================================

Generating answer with tracing...
📊 Latency: 312.45ms
📊 Tokens: 12 → 88 (total: 100)
🔍 View trace: http://localhost:3000/trace/01HXF...

⏳ Flushing traces to Langfuse...
✅ Traces sent!

This confirms:

• the decorator worked
• Langfuse received the trace
• the worker processed it

View the Trace in Langfuse

Open the printed URL, for example:

http://localhost:3000/trace/01HXFG23P9...

You will see:

The parent trace

decorator_pipeline

A nested span

generate_answer

Full metadata

• prompt
• output
• latency
• token usage
• model
• system and user messages

This is the moment where the entire pipeline becomes visible.

Your Observability Stack Is Live

By the end of this section, you now have:

• Langfuse Server + Langfuse Worker + PostgreSQL running locally
• vLLM inference server healthy at port 8000
• traced LLM requests flowing into the dashboard
• real-time visibility into latency, prompts, outputs, and token usage

This forms the foundation for everything in Lesson 2:

• scores
• evaluations
• diagnostics
• advanced tracing patterns

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: June 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, you built the core foundation for modern LLM observability. You began by understanding why LLM applications need far more than traditional logs or metrics. They require visibility into prompts, responses, latency, token usage, and multi-step pipelines. This led naturally to Langfuse, a tool purpose-built for tracing and monitoring LLM workloads.

You then deployed a fully self-hosted observability stack using Docker Compose: Langfuse Server, Langfuse Worker, PostgreSQL, and a local vLLM model server. With the project structure, configuration files, and environment variables in place, your development environment became capable of real-time local trace analysis.

Next, you examined your baseline LLM script, a simple “send a question, print an answer” pipeline that works but offers zero visibility. No prompts, no timing, no token counts, and no traceability. This served as the perfect starting point to highlight why observability is essential.

With the Langfuse @observe decorator, you then transformed that invisible pipeline into a fully instrumented one. Every request now captures structured traces: inputs, outputs, latency, token usage, and parent-child spans. Running the script produced your first real trace inside the Langfuse dashboard, revealing exactly what the model did and how the pipeline behaved.

By the end of the lesson, your LLM application evolved from a black box into a transparent, debuggable system running locally with self-hosted components.

In the next lesson, you will go deeper by adding manual tracing, scoring, evaluation logic, latency checks, and health diagnostics, building on the foundation you created today.

Citation Information

Singh, V. “LLM Observability with Self-Hosted Langfuse and vLLM,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/tadoh

@incollection{Singh_2026_llm-observability-self-hosted-langfuse-vllm,
  author = {Vikram Singh},
  title = {{LLM Observability with Self-Hosted Langfuse and vLLM}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/tadoh},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post LLM Observability with Self-Hosted Langfuse and vLLM appeared first on PyImageSearch.

Building and Training a Kimi-K2 Model Using DeepSeek-V3 Components

The landscape of large language models (LLMs) is undergoing a fundamental transformation toward agentic intelligence, where models can autonomously perceive, plan, reason, and act within complex and dynamic environments. This paradigm shift moves beyond traditional static imitation learning toward models that actively learn through interaction, acquire skills beyond their training distribution, and adapt their behavior based on experience. Agentic intelligence represents a critical capability for the next generation of foundation models, with transformative implications for tool use, software development, and real-world autonomy.

Kimi-K2 stands at the forefront of this revolution. As a 1.04 trillion-parameter Mixture-of-Experts (MoE) language model with 32 billion activated parameters, Kimi-K2 was purposefully designed to address the core challenges of agentic capability development. The model achieves remarkable performance across diverse benchmarks:

• 66.1 on Tau2-bench
• 76.5 on ACEBench (en)
• 65.8 on SWE-bench Verified
• 53.7 on LiveCodeBench v6
• 75.1 on GPQA-Diamond

On the LMSYS (Large Model Systems Organization) Arena leaderboard, Kimi-K2 ranks as the top open-source model and 5th overall, competing closely with Claude 4 Opus and Claude 4 Sonnet.

In this lesson, we dive deep into the technical innovations behind Kimi-K2, focusing on its architectural differences from DeepSeek-V3, the revolutionary MuonClip optimizer, and training data improvements. We also provide a complete implementation guide using DeepSeek-V3 components as building blocks.

Kimi-K2 vs DeepSeek-V3: Key Architecture Differences in LLM Design

While Kimi-K2 builds on DeepSeek-V3’s architecture, several strategic modifications were made to optimize agentic capabilities and inference efficiency. Understanding these architectural differences is crucial for implementing the model effectively (Table 1).

Mixture of Experts Scaling in Kimi-K2: Model Size, Sparsity, and Efficiency

The most significant architectural departure lies in Kimi-K2’s aggressive sparsity scaling. Through carefully controlled small-scale experiments, the Kimi team developed a sparsity scaling law that demonstrated a clear relationship: with the number of activated parameters held constant (i.e., constant FLOPs), increasing the total number of experts consistently lowers both training and validation loss. This finding led to a dramatic increase in model sparsity.

Kimi-K2 employs 384 experts compared to DeepSeek-V3’s 256 experts, representing a 50% increase. Despite this, the model maintains 8 active experts per token, resulting in a sparsity ratio of 48 (384/8) versus DeepSeek-V3’s 32 (256/8). This increased sparsity comes with a trade-off: while total parameters grow to 1.04 trillion (54% more than DeepSeek-V3’s 671B), the number of activated parameters actually decreases to 32.6B (13% less than DeepSeek-V3’s 37B). This design choice optimizes the compute-performance frontier, achieving superior model quality while maintaining efficient inference.

Attention Head Optimization in Kimi-K2 for Efficient Long-Context LLMs

A critical optimization for agentic applications involves the number of attention heads. DeepSeek-V3 sets the number of attention heads to roughly twice the number of model layers (128 heads for 61 layers) to better utilize memory bandwidth. However, as context length increases, this design incurs significant inference overhead.

For agentic applications requiring efficient long-context processing, this becomes prohibitive. With a 128k sequence length, increasing attention heads from 64 to 128 (while keeping 384 total experts) leads to an 83% increase in inference FLOPs. Through controlled experiments, the Kimi team found that doubling the number of attention heads yields only modest improvements in validation loss (0.5% to 1.2%) under iso-token training conditions.

Given that sparsity 48 already provides strong performance, the marginal gains from doubling attention heads do not justify the inference cost. Kimi-K2 therefore uses 64 attention heads (half of DeepSeek-V3’s 128), dramatically reducing inference costs for long-context agentic workloads while maintaining competitive performance.

MuonClip Optimizer: Stabilizing Large-Scale LLM Training in Kimi-K2

The MuonClip optimizer represents one of the most significant innovations in Kimi-K2’s development, addressing the fundamental challenge of training stability at trillion-parameter scale while maintaining token efficiency. Understanding MuonClip requires examining both the underlying Muon optimizer and the novel QK-Clip mechanism that makes it stable for large-scale training.

Token Efficiency in LLM Training: Why It Matters for Kimi-K2

Given the increasingly limited availability of high-quality human data, token efficiency has emerged as a critical factor in LLM scaling. Token efficiency refers to how much performance improvement is achieved per token consumed during training. The Muon optimizer, introduced by Jordan et al. (2024), substantially outperforms AdamW under the same compute budget, model size, and training data volume.

Previous work in Moonlight demonstrated that Muon’s token efficiency gains make it an ideal choice for maximizing the intelligence extracted from limited high-quality tokens. However, scaling Muon to trillion-parameter models revealed a critical challenge: training instability due to exploding attention logits.

Attention Logit Explosion in LLMs: Training Instability and Challenges

During medium-scale training runs using vanilla Muon, attention logits rapidly exceeded magnitudes of 1000, leading to numerical instabilities and occasional training divergence (Figure 1). This phenomenon occurred more frequently with Muon than with AdamW, suggesting that Muon’s aggressive optimization dynamics amplify instabilities in the attention mechanism.

Existing mitigation strategies proved insufficient:

• Logit soft-capping (used in Gemma) directly clips attention logits, but the dot products between queries and keys can still grow excessively before capping is applied
• Query-Key Normalization (QK-Norm) (Dehghani et al., 2023) is incompatible with Multi-head Latent Attention (MLA) because full key matrices are not explicitly materialized during inference

QK-Clip: Preventing Attention Logit Explosion in Kimi-K2 Training

To address this fundamental challenge, the Kimi team proposed QK-Clip, a novel weight-clipping mechanism that explicitly constrains attention logits by rescaling the query and key projection weights post-update. The elegance of QK-Clip lies in its simplicity: it does not alter forward and backward computation in the current step but instead uses maximum logits as a guiding signal to control weight growth (Figure 2).

For each attention head , the attention mechanism computes:

The attention output is:

QK-Clip defines the max logit per head as:

where is the current batch and index different tokens.

When exceeds a threshold (set to 100 for Kimi-K2), QK-Clip rescales the weights. Critically, the rescaling is applied per-head rather than globally, minimizing intervention on heads that remain stable:

.

This per-head, component-aware clipping represents a substantial refinement over naive global clipping strategies.

Figure 3 describes the complete algorithm for MuonClip Optimizer.

Training Data Optimization for Kimi-K2: Improving Token Utility in LLMs

Beyond architectural and optimizer innovations, Kimi-K2’s superior performance stems significantly from strategic improvements in training data. With high-quality human-generated data becoming increasingly scarce, the focus shifts to increasing token utility, defined as the effective learning signal each token contributes to model updates.

Token Utility in LLM Training: Maximizing Learning per Token

Token efficiency in pre-training encompasses 2 related but distinct concepts:

• Optimizer efficiency: How effectively the optimizer extracts signal from each gradient update (addressed by MuonClip)
• Token utility: The inherent information density and learning signal in each token

Increasing token utility directly improves token efficiency. A naive approach involves repeated exposure to the same tokens across multiple epochs, but this leads to overfitting and reduced generalization. The key innovation in Kimi-K2 lies in a sophisticated synthetic data generation strategy that amplifies high-quality tokens without inducing overfitting.

Knowledge Data Rephrasing for LLMs: Improving Training Data Quality

Pre-training on knowledge-intensive text presents a fundamental trade-off: a single epoch is insufficient for comprehensive knowledge absorption, while multi-epoch repetition yields diminishing returns. To resolve this tension, Kimi-K2 employs a synthetic rephrasing framework with the following 3 key components.

Style- and Perspective-Diverse Prompting

To enhance linguistic diversity while maintaining factual integrity, carefully engineered prompts guide a large language model to generate faithful rephrasings in varied styles and perspectives. This approach ensures that while surface-level linguistic features change, the underlying factual content remains consistent. The diversity of expressions forces the model to learn robust representations of the same knowledge across multiple linguistic realizations.

Chunk-wise Autoregressive Generation

Long documents pose a challenge for standard LLM-based rewriting due to implicit output length limitations. Kimi-K2 addresses this through a chunk-based autoregressive strategy: documents are segmented, each segment is rephrased individually with preserved context, and segments are stitched back together to form complete passages. This methodology prevents information loss and maintains global coherence across extended texts (Figure 4).

Fidelity Verification

To ensure consistency between original and rewritten content, fidelity checks compare the semantic alignment of each rephrased passage with its source. This quality control step prevents the introduction of hallucinations or factual errors during the rephrasing process.

Mathematics Data Rephrasing

To enhance mathematical reasoning capabilities, high-quality mathematical documents are rewritten into a “learning-note” style following SwallowMath methodology (Figure 5). This transformation converts dense mathematical exposition into more pedagogical formats that better support learning. Additionally, data diversity is increased through the translation of high-quality mathematical materials from other languages into English, effectively multiplying the available high-quality mathematical training data.

Overall Pre-training Corpus

The complete Kimi-K2 pre-training corpus comprises 15.5 trillion tokens of curated, high-quality data spanning 4 primary domains:

• Web Text: General knowledge and natural language understanding
• Code: Programming and structured reasoning
• Mathematics: Quantitative reasoning and formal problem-solving
• Knowledge: Domain-specific expertise and factual information

Kimi-K2 Implementation: Training an Open-Source LLM with DeepSeek-V3

In this section, we walk through the key implementation details for training Kimi-K2, focusing specifically on the components that differ from the standard DeepSeek-V3 implementation. We’ll examine the enhanced Multi-head Latent Attention with max logit tracking, the MuonClip optimizer implementation, and the custom training setup.

Multi-Head Latent Attention (MLA) with Max Logit Tracking in Kimi-K2

The Multi-head Latent Attention (MLA) mechanism in Kimi-K2 extends DeepSeek-V3’s implementation with critical modifications to support QK-Clip. The key enhancement is per-head max-logit tracking during the forward pass, which provides the signal needed for weight clipping by the optimizer.

class MultiheadLatentAttention(nn.Module):
    
    def __init__(self, config: DeepSeekConfig):
        super().__init__()
        self.config = config
        self.n_embd = config.n_embd
        self.n_head = config.n_head
        self.head_dim = config.n_embd // config.n_head

        # Compression dimensions
        self.kv_lora_rank = config.kv_lora_rank
        self.q_lora_rank = config.q_lora_rank
        self.rope_dim = config.rope_dim

        # KV compression
        self.kv_proj = nn.Linear(self.n_embd, self.kv_lora_rank, bias=False)
        self.kv_norm = RMSNorm(self.kv_lora_rank)

        # KV decompression
        self.k_decompress = nn.Linear(self.kv_lora_rank, self.n_head * self.head_dim, bias=False)
        self.v_decompress = nn.Linear(self.kv_lora_rank, self.n_head * self.head_dim, bias=False)

        # Query compression
        self.q_proj = nn.Linear(self.n_embd, self.q_lora_rank, bias=False)
        self.q_decompress = nn.Linear(self.q_lora_rank, self.n_head * self.head_dim, bias=False)

        # RoPE projections
        self.k_rope_proj = nn.Linear(self.n_embd, self.n_head * self.rope_dim, bias=False)
        self.q_rope_proj = nn.Linear(self.q_lora_rank, self.n_head * self.rope_dim, bias=False)

        # Output projection
        self.o_proj = nn.Linear(self.n_head * self.head_dim, self.n_embd, bias=config.bias)

        # Dropout
        self.attn_dropout = nn.Dropout(config.dropout)
        self.resid_dropout = nn.Dropout(config.dropout)

        # RoPE
        self.rope = RotaryEmbedding(self.rope_dim, config.block_size)

        # Causal mask
        self.register_buffer(
            "causal_mask",
            torch.tril(torch.ones(config.block_size, config.block_size)).view(
                1, 1, config.block_size, config.block_size
            )
        )

        self.max_logits = 0.0  # Track maximum attention logits

On Lines 1-47, we define the MLA architecture following DeepSeek-V3’s design with compression and decompression of queries and key-values through low-rank projections. The key innovation appears on Line 49, where we initialize self.max_logits = 0.0, a critical state variable that tracks the maximum attention logits across heads. This tracking mechanism is essential for QK-Clip to function properly.

    def forward(self, x: torch.Tensor, attention_mask: Optional[torch.Tensor] = None):
        B, T, C = x.size()

        # Compression phase
        kv_compressed = self.kv_norm(self.kv_proj(x))
        q_compressed = self.q_proj(x)

        # Decompression phase
        k_content = self.k_decompress(kv_compressed)
        v = self.v_decompress(kv_compressed)
        q_content = self.q_decompress(q_compressed)

        # RoPE components
        k_rope = self.k_rope_proj(x)
        q_rope = self.q_rope_proj(q_compressed)

        # Reshape for multi-head attention
        k_content = k_content.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        v = v.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        q_content = q_content.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        k_rope = k_rope.view(B, T, self.n_head, self.rope_dim).transpose(1, 2)
        q_rope = q_rope.view(B, T, self.n_head, self.rope_dim).transpose(1, 2)

        # Apply RoPE
        cos, sin = self.rope(x, T)
        q_rope = apply_rope(q_rope, cos, sin)
        k_rope = apply_rope(k_rope, cos, sin)

        # Concatenate content and rope parts
        q = torch.cat([q_content, q_rope], dim=-1)
        k = torch.cat([k_content, k_rope], dim=-1)
     

On Lines 52-82, we implement the standard forward pass through the compression-decompression pipeline. The input undergoes compression via kv_proj and q_proj, followed by decompression through dedicated linear layers. We then reshape tensors for multi-head processing and apply Rotary Position Embeddings (RoPE) separately to content and positional components. This separation allows per-head QK-Clip to target only the appropriate components without affecting shared rotary embeddings.

        # Concatenate content and rope parts
        q = torch.cat([q_content, q_rope], dim=-1)
        k = torch.cat([k_content, k_rope], dim=-1)

        # Attention computation
        scale = 1.0 / math.sqrt(q.size(-1))
        scores = torch.matmul(q, k.transpose(-2, -1)) * scale

        with torch.no_grad():
            # self.max_logits = torch.max(scores, dim=1).item()
            self.max_logits = list(torch.max(scores.transpose(1, 0).contiguous().view(scores.shape[1], -1), dim=-1)[0])

        # Apply causal mask
        scores = scores.masked_fill(self.causal_mask[:, :, :T, :T] == 0, float('-inf'))

        # Apply padding mask if provided
        if attention_mask is not None:
            padding_mask_additive = (1 - attention_mask).unsqueeze(1).unsqueeze(2) * float('-inf')
            scores = scores + padding_mask_additive

        # Softmax and dropout
        attn_weights = F.softmax(scores, dim=-1)
        attn_weights = self.attn_dropout(attn_weights)

        # Apply attention to values
        out = torch.matmul(attn_weights, v)

        # Reshape and project
        out = out.transpose(1, 2).contiguous().view(B, T, self.n_head * self.head_dim)
        out = self.resid_dropout(self.o_proj(out))

        return out

On Lines 89-94, we compute attention scores and implement the crucial max logit tracking. The score computation follows standard scaled dot-product attention. However, Lines 92-94 represent a key departure from vanilla DeepSeek-V3: we track the maximum attention logit per head using torch.no_grad() to avoid affecting gradients. The scores tensor has shape [batch, num_heads, seq_len, seq_len], and we transpose and reshape to extract per-head maximum values. This per-head granularity enables targeted intervention only on heads exhibiting logit explosion, minimizing disruption to stable heads.

On Lines 97-113, we complete the attention mechanism with causal masking, optional padding masks, softmax normalization, and dropout. The final output projection maintains the standard MLA architecture. The elegance of this implementation lies in its non-invasiveness: max logit tracking adds minimal computational overhead (a single max operation under torch.no_grad) while providing the critical signal for optimizer-level weight clipping.

Implementing the MuonClip Optimizer for Stable LLM Training

The MuonClip optimizer represents the core innovation enabling stable trillion-parameter training. Our implementation integrates Newton-Schulz orthogonalization, RMS matching, weight decay, and per-head QK-Clip into a unified optimizer.

def apply_qk_clip_per_head(
    query_weights: torch.Tensor,
    key_weights: torch.Tensor,
    max_logits_per_head: Union[List[float], torch.Tensor],
    tau: float = 100.0
) -> None:
        if isinstance(max_logits_per_head, list):
        max_logits_per_head = torch.tensor(
            max_logits_per_head,
            device=query_weights.device,
            dtype=query_weights.dtype
        )
    apply_qk_clip_vectorized(query_weights, key_weights, max_logits_per_head, tau)

On Lines 1-13, we define the entry point for the QK-Clip application. The function accepts query and key projection weights along with per-head max logits and a threshold (defaulting to 100). We handle both list and tensor inputs for flexibility, converting lists to tensors on the appropriate device with matching dtype. The critical design choice here is in-place modification: we directly modify weight tensors to avoid memory allocation overhead during optimization.

def apply_qk_clip_per_head(
    query_weights: torch.Tensor,
    key_weights: torch.Tensor,
    max_logits_per_head: Union[List[float], torch.Tensor],
    tau: float = 100.0
) -> None:
        if isinstance(max_logits_per_head, list):
        max_logits_per_head = torch.tensor(
            max_logits_per_head,
            device=query_weights.device,
            dtype=query_weights.dtype
        )
    apply_qk_clip_vectorized(query_weights, key_weights, max_logits_per_head, tau)

@torch.no_grad()
def apply_qk_clip_vectorized(
    query_weights: torch.Tensor,
    key_weights: torch.Tensor,
    max_logits_per_head: torch.Tensor,
    tau: float = 100.0
) -> None:
    
    q_out, q_in = query_weights.shape[0], query_weights.shape[1]
    k_out, k_in = key_weights.shape[0], key_weights.shape[1]
    num_heads = len(max_logits_per_head)
    d_k = q_out // num_heads

    # Ensure tensor type
    if not isinstance(max_logits_per_head, torch.Tensor):
        max_logits_per_head = torch.tensor(
            max_logits_per_head,
            device=query_weights.device,
            dtype=query_weights.dtype
        )

    # Compute scaling factors: gamma = tau / max_logit where max_logit > tau
    needs_clip = max_logits_per_head > tau

On Lines 15-48, we extract dimensions and ensure tensor type compatibility. We first extract dimensions and compute the per-head scaling factor only for heads where .

@torch.no_grad()
def apply_qk_clip_vectorized(
    query_weights: torch.Tensor,
    key_weights: torch.Tensor,
    max_logits_per_head: torch.Tensor,
    tau: float = 100.0
) -> None:
    
    q_out, q_in = query_weights.shape[0], query_weights.shape[1]
    k_out, k_in = key_weights.shape[0], key_weights.shape[1]
    num_heads = len(max_logits_per_head)
    d_k = q_out // num_heads

    # Ensure tensor type
    if not isinstance(max_logits_per_head, torch.Tensor):
        max_logits_per_head = torch.tensor(
            max_logits_per_head,
            device=query_weights.device,
            dtype=query_weights.dtype
        )

    # Compute scaling factors: gamma = tau / max_logit where max_logit > tau
    needs_clip = max_logits_per_head > tau

    # If no clipping needed, return early
    if not needs_clip.any():
        return

    gamma = torch.where(
        needs_clip,
        tau / max_logits_per_head.clamp(min=1e-8),
        torch.ones_like(max_logits_per_head)
    )
    sqrt_gamma = torch.sqrt(gamma)

    # Reshape weights to [d_model, num_heads, d_k] for per-head scaling
    # Views share underlying storage, so in-place ops modify original tensor
    q_reshaped = query_weights.view(q_out // num_heads, num_heads, q_in)
    k_reshaped = key_weights.view(k_out // num_heads, num_heads, k_in)

    # Apply per-head scaling IN-PLACE: broadcast sqrt_gamma [num_heads] over [d_model, num_heads, d_k]
    q_reshaped.mul_(sqrt_gamma.view(1, num_heads, 1))
    k_reshaped.mul_(sqrt_gamma.view(1, num_heads, 1))

    q_reshaped = q_reshaped.view(q_out, q_in)
    k_reshaped = k_reshaped.view(k_out, k_in)

On Lines 80-97, we perform the actual weight clipping through careful tensor reshaping and in-place multiplication. The weights are reshaped from [d_model, d_model] to [d_model/num_heads, num_heads, d_k] to expose the head dimension. We then apply scaling using in-place multiplication (mul_) with broadcasting. The square root scaling ensures that when query and key both receive , their dot product receives the full scaling. This elegant mathematical property allows us to clip attention logits by rescaling the weights that produce them, rather than clipping logits directly after they’re computed.

Lines 77 and 78 implement early exit if no head requires clipping, which becomes a common case later in training when attention logits stabilize. This optimization avoids unnecessary computation when the model is well-behaved.

class MuonClip(torch.optim.Optimizer):
    def __init__(
        self,
        params,
        lr: float = 1e-3,
        momentum: float = 0.95,
        weight_decay: float = 0.01,
        tau: float = 100.0,
        ns_steps: int = 5,
        eps: float = 1e-7
    ):
        if lr < 0.0:
            raise ValueError(f"Invalid learning rate: {lr}")
        if not 0.0 <= momentum <= 1.0:
            raise ValueError(f"Invalid momentum value: {momentum}")
        if weight_decay < 0.0:
            raise ValueError(f"Invalid weight_decay value: {weight_decay}")
        if tau <= 0.0:
            raise ValueError(f"Invalid tau value: {tau}")

        defaults = dict(
            lr=lr,
            momentum=momentum,
            weight_decay=weight_decay,
            tau=tau,
            ns_steps=ns_steps,
            eps=eps
        )
        super().__init__(params, defaults)

        # For QK-Clip functionality
        self.model = None
        self.attention_layers = []

    def set_model(self, model: nn.Module):
        self.model = model
        if hasattr(model, 'get_attention_layers'):
            self.attention_layers = model.get_attention_layers()

On Lines 1-33, we define the MuonClip optimizer class, inheriting from PyTorch’s base Optimizer. The constructor accepts standard hyperparameters (learning rate, momentum, weight decay) plus QK-Clip-specific parameters ( and Newton-Schulz steps). We validate all parameters and initialize state tracking. Critically, Lines 35-38 implement model registration through set_model(), which extracts attention layers for later QK-Clip application. This design separates optimizer logic from model architecture, allowing the optimizer to operate on any model exposing a get_attention_layers() method.

    @torch.no_grad()
    def step(self, closure: Optional[Callable] = None) -> Optional[float]:
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            lr = group['lr']
            momentum = group['momentum']
            weight_decay = group['weight_decay']
            ns_steps = group['ns_steps']
            eps = group['eps']

            for p in group['params']:
                if p.grad is None:
                    continue

                grad = p.grad
                state = self.state[p]

                # Initialize momentum buffer
                if len(state) == 0:
                    state['momentum_buffer'] = torch.zeros_like(p)

                buf = state['momentum_buffer']

                # Apply momentum: Mt = μMt−1 + Gt
                buf.mul_(momentum).add_(grad)

                if p.ndim >= 2:  # 2D+ parameters - use Muon
                    # Apply Newton-Schulz orthogonalization
                    if p.ndim > 2:
                        original_shape = buf.shape
                        buf_2d = buf.view(buf.shape[0], -1)
                        orthogonal_update = newton_schulz(buf_2d, ns_steps, eps)
                        orthogonal_update = orthogonal_update.view(original_shape)
                    else:
                        orthogonal_update = newton_schulz(buf, ns_steps, eps)

                    # RMS matching factor: √(max(n,m) × 0.2)
                    n, m = p.shape[0], p.shape[1] if p.ndim > 1 else 1
                    rms_factor = math.sqrt(max(n, m) * 0.2)
                    orthogonal_update = orthogonal_update * rms_factor

                    # Update: Wt = Wt−1 − η(Ot + λWt−1)
                    p.add_(orthogonal_update + weight_decay * p, alpha=-lr)
                else:
                    # 1D parameters - standard momentum
                    p.add_(buf + weight_decay * p, alpha=-lr)

        # Apply QK-Clip
        self._apply_qk_clip()

        return loss

On Lines 41-94, we implement the core optimization step integrating Muon updates with QK-Clip. The step begins with standard closure handling and parameter group iteration. Lines 41-68 implement momentum accumulation () using in-place operations for memory efficiency. The critical branching occurs at Line 70: parameters with 2+ dimensions receive Muon treatment.

On Lines 72-83, we apply the Muon update for matrix parameters. Newton-Schulz orthogonalization produces an orthogonal approximation of the momentum buffer, which we then scale by to match AdamW’s RMS characteristics. This scaling ensures Muon’s updates have similar magnitudes to AdamW, enabling easier hyperparameter transfer. Finally, Line 86 applies the update with weight decay: . Line 89 applies standard momentum updates to 1D parameters such as biases and normalization layers.

    def _apply_qk_clip(self):
        """Apply QK-Clip to attention layers to prevent logit explosion."""
        if not self.attention_layers:
            return

        tau = self.param_groups[0]['tau']

        for attention_layer in self.attention_layers:
            if not hasattr(attention_layer, 'max_logits'):
                continue

            max_logits = attention_layer.max_logits
            if not max_logits:
                continue


            # Handle both scalar and per-head max logits
            if isinstance(max_logits, (int, float)):
                max_logits = [max_logits]


            apply_qk_clip_per_head(
                    attention_layer.k_decompress.weight.data,
                    attention_layer.q_decompress.weight.data,
                    max_logits,
                    tau
            )

On Lines 96-122, we apply QK-Clip after all weight updates. The _apply_qk_clip() method iterates through all registered attention layers, extracts their max_logits attribute (populated during forward pass), and applies per-head clipping to the query and key decompression weights. This post-update clipping ensures weights don’t grow unboundedly across training steps while preserving gradient information within each step.

Complete Kimi-K2 Training Pipeline: Setup, Config, and Optimization

Finally, we bring everything together in a complete training configuration:

config = DeepSeekConfig()
config.multi_token_predict = 0
config.n_experts = 8
config.n_head = 4

training_args = TrainingArguments(
    output_dir="./kimik2_checkpoints",
    num_train_epochs=2,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=4,
    learning_rate=5e-4,
    warmup_steps=10,
    weight_decay=0.01,
    logging_dir="./kimik2_checkpoints/logs",
    logging_steps=50,
    save_steps=50,
    save_total_limit=3,
    eval_steps=50,
    eval_strategy="steps",
    save_strategy="steps",
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    gradient_accumulation_steps=4,
    fp16=True,
    dataloader_num_workers=2,
    remove_unused_columns=False,
    report_to="none",
    push_to_hub=False,
    save_safetensors=False,
)

On Lines 1-4, we configure the model architecture. Kimi-K2 does not use Multi-Token Prediction, so we disable multi-token prediction (multi_token_predict=0) to simplify training and focus on core capabilities. We use 8 experts for this educational implementation rather than the hundreds used in production-scale Kimi-K2 and DeepSeek-V3 models. We also use 4 attention heads for this small-scale educational implementation, compared to the production-scale configurations used in DeepSeek-V3 and Kimi-K2.

On Lines 6-30, we define training arguments following best practices for small-scale experiments. We use gradient accumulation (4 steps) to simulate larger batch sizes with limited GPU memory, enable mixed-precision training (fp16=True) for speed and memory efficiency, and configure regular evaluation and checkpointing every 50 steps. The learning rate of 5e-4 is conservative for stable training, with a brief 10-step warmup.

model = DeepSeek(config)

data_collator = DeepSeekDataCollator(tokenizer)

optimizer = MuonClip(model.parameters(), lr=5e-3)
optimizer.set_model(model)

# Create trainer
trainer = DeepSeekTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=data_collator,
    optimizers=(optimizer, None)
)

print("✓ Trainer created. Starting training...")
print("=" * 80)

# Train!
trainer.train()

print("=" * 80)
print("✓ Training complete!")

# Save final model
trainer.save_model("./kimik2_final")
tokenizer.save_pretrained("./kimik2_final")
print("✓ Model saved to ./kimik2_final")

On Lines 31-36, we initialize the model and create a MuonClip optimizer. Critically, Line 36 registers the model with the optimizer using set_model(), enabling QK-Clip to access attention layers. This registration must occur before training begins.

On Lines 39-60, we instantiate the custom trainer with all components and launch training. The optimizers=(optimizer, None) argument provides our custom optimizer to Hugging Face Trainer, overriding its default optimizer creation. After training completes, we save both the model weights and tokenizer for later inference.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: June 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

We began by detailing how to train Kimi-K2 from scratch using DeepSeek-V3 components, emphasizing the architectural differences that set Kimi-K2 apart. We explored the model’s scale and sparsity, showing that reducing the number of attention heads allowed us to balance efficiency and performance. A key part of this journey was the introduction of the MuonClip optimizer, which stabilizes training while pushing the limits of large-scale language modeling.

We then turned to the challenges of token efficiency and the attention logit explosion problem. To address these, we introduced the QK-Clip innovation, which helped us control runaway logits and improve overall stability. Alongside this, we refined our training data pipeline, focusing on token utility and knowledge data rephrasing to ensure that every token contributed meaningfully to the model’s learning process. These improvements allowed us to maximize the value of the data while keeping training efficient.

Finally, we described the implementation details, including enhanced multi-head latent attention with max logit tracking and the practical integration of the MuonClip optimizer. We concluded with a complete training setup, showing how all these innovations came together to make Kimi-K2 a robust, efficient, and scalable model. By combining architectural refinements, optimizer breakthroughs, and data improvements, this lesson demonstrated how these techniques push the boundaries of what’s possible in modern language model training.

Citation Information

Mangla, P. “Building and Training a Kimi-K2 Model Using DeepSeek-V3 Components,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/d3tge

@incollection{Mangla_2026_building-training-kimi-k2-model-using-deepseek-v3,
  author = {Puneet Mangla},
  title = {{Building and Training a Kimi-K2 Model Using DeepSeek-V3 Components}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/d3tge},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Building and Training a Kimi-K2 Model Using DeepSeek-V3 Components appeared first on PyImageSearch.

Semantic Caching for LLMs: TTLs, Confidence, and Cache Safety

In this lesson, you will learn how to harden a semantic cache for LLMs, one of the most important LLMOps patterns for reducing redundant inference costs, and move from a working semantic caching prototype to a system that can survive real-world usage with TTL validation, confidence scoring, deduplication, and cache poisoning prevention.

This lesson is the last in a 2-part series on Semantic Caching for LLMs:

1. Semantic Caching for LLMs: FastAPI, Redis, and Embeddings
2. Semantic Caching for LLMs: TTLs, Confidence, and Cache Safety (this tutorial)

To learn how to harden a semantic cache for LLMs and make it safe, reliable, and production-ready, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Why Semantic Caching for LLMs Requires Production Hardening

In Lesson 1, we built a semantic cache that works end-to-end. It correctly avoids redundant LLM calls, reuses responses for identical queries, and even handles paraphrased inputs via semantic similarity. For many tutorials, that would be the end of the story.

In real systems, however, working is only the starting point.

A semantic cache that works under ideal conditions can still fail in subtle and dangerous ways when exposed to real users, long-running processes, and evolving information. These failures do not usually appear as crashes or explicit errors. Instead, they show up as silent correctness issues, degraded user trust, and unpredictable behavior over time.

What Lesson 1 Solved — and What It Didn’t

Lesson 1 focused on the correctness of flow:

• Requests move through exact match → semantic match → LLM fallback (generation)
• Cached responses are reused when appropriate
• The system is observable and debuggable
• Nothing is hidden behind abstractions

What it intentionally did not address was long-term safety.

We did not ask:

• How old is this cached response, and should we still trust it?
• What happens if the LLM returns an error or partial output?
• What if the cache slowly fills with duplicates?
• What if similarity is high but the answer is no longer valid?

Those questions only matter once the system runs for days or weeks, not minutes.

Real-World Failure Modes in Semantic Caching

Semantic caching introduces failure modes that rarely exist in traditional exact-match caches.

For example:

• A cached answer with very high similarity may still be stale
• An error response may be accidentally cached and reused
• Slight variations of the same query may create duplicate entries
• Old but similar answers may appear correct while being subtly wrong

None of these issues breaks the system outright. Instead, they quietly degrade correctness and user trust over time.

These are the hardest bugs to detect because the system continues to respond quickly and confidently.

Why “It Works” Does Not Mean “It’s Safe”

A semantic cache sits directly in the decision path of an LLM system. When it makes a mistake, that mistake is amplified through reuse.

If an unsafe response enters the cache:

• It can be served repeatedly
• It can outlive the conditions that made it valid
• It can be returned with high confidence

This is why semantic caching requires more discipline, not less, than direct LLM calls.

In this lesson, we will take the working system from Lesson 1 and begin hardening it. We will introduce explicit safeguards for staleness, confidence, duplication, and safety — without changing the core architecture.

The goal is not to make the system perfect, but to make its failures controlled, visible, and predictable.

That is the difference between a demo and a system you can trust.

Cache TTL in Semantic Caching: Preventing Stale LLM Responses

Once a semantic cache is deployed and begins reusing LLM responses, a new question immediately arises:

How long should a cached response be trusted?

Unlike traditional caches that store deterministic outputs, semantic caches store model-generated answers. These answers are only valid within a certain window of time and context. Without explicit controls, a semantic cache can continue serving responses that are technically valid but practically wrong.

This section explains why cached LLM responses become stale, how TTLs help, and what it means for a cache entry to be unsafe.

Why Cached LLM Responses Become Stale

LLM responses are not timeless.

They are influenced by:

• evolving APIs and libraries
• changing business logic or documentation
• updated prompts or system behavior
• newly introduced edge cases

A cached answer that was correct an hour ago may no longer reflect the current state of the world.

Semantic caching amplifies this risk because:

• responses are reused aggressively
• high similarity can mask outdated content
• cached answers are returned with confidence

Without staleness controls, the cache slowly becomes a museum of old truths.

TTL as a Safety Mechanism

A time-to-live (TTL) specifies how long a cache entry remains valid.

Once the TTL expires:

• the entry is treated as unsafe
• it should no longer be reused
• a fresh LLM response must be generated

TTL does not guarantee correctness, but it limits the blast radius of staleness.

In semantic caching, TTL is not an optimization. It is a correctness safeguard.

Application-Level TTL vs Redis: EXPIRE

There are 2 common ways to implement TTLs when using Redis:

Redis EXPIRE

• Redis automatically deletes keys after a fixed duration
• Expired entries are removed entirely
• The application has no visibility into expired data

Application-Level TTL (Used Here)

• Entries remain stored in Redis
• Expiration is checked at read time by the application
• The application decides whether an entry is safe to reuse

In this system, TTL is enforced at the application layer rather than using Redis TTL via the native EXPIRE command, a deliberate choice that prioritizes observability over automation.

This choice allows us to:

• inspect expired entries during debugging
• apply custom expiration logic
• combine TTL with other safety signals (such as confidence)

We trade automatic deletion for control and observability.

When a Cache Entry Becomes Unsafe

In this system, a cache entry is considered unsafe when any of the following are true:

• its TTL has expired
• its content is malformed or erroneous
• its confidence score falls below an acceptable threshold

TTL is the first and most basic of these checks.

If an entry fails the TTL check, semantic similarity is irrelevant.

Reusing it would prioritize speed over correctness.

Designing TTLs for LLM Workloads

There is no universal “correct” TTL for LLM responses.

Instead, TTLs should be chosen based on:

• how fast the underlying information changes
• how costly incorrect answers are
• how frequently similar queries appear

Short TTLs:

• reduce staleness risk
• increase LLM calls

Long TTLs:

• improve cache hit rate
• increase risk of outdated responses

In Lesson 1, we used a conservative default TTL to keep behavior predictable. In this lesson, we will focus on how TTLs are enforced rather than on tuning them for a specific domain.

TTL design is a policy decision. TTL enforcement is a correctness requirement.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

MLOps Project Structure for Semantic Caching with FastAPI and Redis

Before diving into individual components, let’s take a moment to understand how the project is organized.

A clear directory structure is especially important in LLM-backed systems, where responsibilities span API orchestration, caching, embeddings, model calls, and observability. In this project, each concern is isolated into its own module so the request flow remains easy to trace and reason about.

After downloading the source code from the “Downloads” section, your directory structure should look like this:

.
├── app
│   ├── api
│   │   ├── __init__.py
│   │   └── ask.py
│   ├── cache
│   │   ├── __init__.py
│   │   ├── poisoning.py
│   │   ├── schemas.py
│   │   ├── semantic_cache.py
│   │   └── ttl.py
│   ├── config
│   │   ├── __init__.py
│   │   └── settings.py
│   ├── embeddings
│   │   ├── __init__.py
│   │   └── embedder.py
│   ├── llm
│   │   ├── __init__.py
│   │   └── ollama_client.py
│   ├── main.py
│   └── observability
│       └── metrics.py
├── complete-codebase.txt
├── docker-compose.yml
├── Dockerfile
├── README.md
└── requirements.txt

Let’s break this down at a high level.

The app/ Package

The app/ directory contains all runtime application code. Nothing outside this folder is imported at runtime.

This keeps the service self-contained and makes it easy to reason about deployment and dependencies.

app/main.py: Application Entry Point

This file defines the FastAPI application and registers all routers.

It contains no business logic — only service wiring. Every request to the system enters through this file.

app/api/: API Layer

The api/ package defines HTTP-facing endpoints.

• ask.py: Implements the /ask endpoint and acts as the orchestration layer for the entire semantic caching pipeline.

The API layer is responsible for:

• validating input
• enforcing cache ordering
• coordinating cache, embeddings, and LLM calls
• returning structured debug information

It does not implement caching or similarity logic directly.

app/cache/: Caching Logic

This package contains all cache-related functionality.

• semantic_cache.py: Core semantic cache implementation (exact match, semantic match, Redis storage, similarity search).
• schemas.py: Defines the cache entry schema used for Redis storage.
• ttl.py: Application-level TTL configuration and expiration checks.
• poisoning.py: Safety checks to prevent invalid or error responses from being reused.

By isolating caching logic here, the API layer stays clean and reusable.

app/embeddings/: Embedding Generation

• embedder.py: Handles embedding generation via Ollama’s embedding endpoint.

This module has a single responsibility: converting text into semantic vectors.

It does not cache, rank, or validate embeddings.

app/llm/: LLM Client

• ollama_client.py: Wraps calls to the Ollama text-generation endpoint.

Isolating LLM interaction allows the rest of the system to remain model-agnostic.

app/observability/: Metrics

• metrics.py: Implements simple in-memory counters for cache hits, misses, and LLM calls.

These metrics are intentionally lightweight and meant for learning and debugging, not production monitoring.

Configuration and Infrastructure

Outside the app/ directory:

• config/settings.py: Centralizes environment-based configuration (Redis host, TTLs, model names).
• Dockerfile and docker-compose.yml: Define a reproducible runtime environment for the API and Redis.
• requirements.txt: Lists all Python dependencies required to run the service.

How to Implement Cache TTL Validation in Python and Redis

In the previous section, we discussed why cached LLM responses become stale and why TTLs are necessary. In this section, we move from concept to code and look at how TTL validation is enforced in practice.

The key idea is simple but important:

Cache entries are not deleted automatically. They are validated at read time.

This design choice keeps cache behavior explicit, observable, and safe.

The Default TTL Configuration

TTL configuration is centralized in a single helper function:

File: app/cache/ttl.py

def default_ttl():
    return settings.CACHE_TTL_SECONDS

Rather than hardcoding a value, the TTL is loaded from configuration. This allows different environments to use different TTLs without changing the code.

At this stage, the specific TTL value is not important. What matters is that:

• every cache entry receives a TTL at creation time
• TTL is treated as metadata, not as a Redis feature

Checking Whether an Entry Has Expired

TTL enforcement happens through a dedicated validation function:

def is_expired(entry):
    try:
        created_at = int(entry["created_at"])
        ttl = int(entry["ttl"])
        now = int(time.time())
        return now > (created_at + ttl)
    except (KeyError, ValueError, TypeError):
        return True

This function answers 1 question:

Is this cache entry still safe to reuse?

If the current time exceeds created_at + ttl, the entry is considered expired and must not be reused.

Fail-Safe Expiration Behavior

Notice the exception handling at the end of is_expired().

If the entry:

• is missing required fields
• contains malformed values
• cannot be parsed safely

…it is treated as expired by default.

This is a deliberate fail-safe design.

When dealing with cached LLM responses, silently trusting malformed data is more dangerous than recomputing a response. If the system is unsure, it expires the entry and falls back to the LLM.

Correctness always wins over reuse.

Best-Effort Cleanup During Cache Reads

TTL validation does more than reject expired entries — it also performs opportunistic cleanup during cache searches.

Inside the semantic cache search logic:

• expired entries are detected
• expired keys are removed from Redis
• the cache continues scanning remaining entries

This cleanup happens:

• without background workers
• without scheduled jobs
• without blocking the request

This is not a full garbage collector. It is a best-effort hygiene mechanism that keeps the cache from accumulating junk over time.

Why We Validate on Read, Not Delete on Write

At this point, a natural question arises:

Why not just use Redis EXPIRE and let Redis delete entries automatically?

There are 3 reasons this system validates TTLs on read instead:

• Visibility: Expired entries remain inspectable during debugging.
• Control: The application decides what “expired” means, not Redis.
• Composability: TTL checks can be combined with confidence scoring, poisoning detection, and other safety signals.

By validating at read time, TTL becomes part of the decision-making pipeline rather than an invisible background mechanism.

Confidence Scoring in Semantic Caching: Beyond Similarity for LLMs

Up to this point, semantic caching decisions have relied heavily on semantic similarity. If a cached response is similar enough to a new query, it feels reasonable to reuse it.

In practice, this assumption breaks down.

High similarity answers an important question — “Is this response about the same thing?” — but it does not answer an equally important one:

“Is this response still safe to reuse right now?”

Confidence scoring exists to bridge that gap.

Why High Similarity Can Still Be Wrong

Semantic similarity measures closeness in meaning, not correctness over time.

Consider a cached response that:

• has very high embedding similarity to the current query
• was generated hours or days ago
• refers to information that has since changed

From a vector perspective, the response still appears “correct.”

From a system perspective, it may no longer be trustworthy.

This problem is subtle because:

• similarity scores remain high
• responses look fluent and confident
• failures are silent rather than catastrophic

Without an additional signal, the cache has no way to distinguish relevant but stale from relevant and safe.

Combining Semantic Similarity with Freshness

Confidence scoring introduces a second dimension: freshness.

Rather than deciding reuse based on similarity alone, the cache evaluates a combined signal that reflects:

• how semantically close the response is
• how recently the response was generated

At a high level, confidence answers the question:

“How comfortable are we reusing this response right now?”

Fresh responses with high similarity score high confidence.

Old responses, even with high similarity, gradually lose confidence as they age.

This ensures that time acts as a natural decay mechanism.

Understanding the Confidence Score (High-Level)

In this system, confidence is a weighted combination of:

• semantic similarity
• freshness relative to TTL

You do not need to think about exact formulas at this stage. What matters is the behavior:

• Confidence starts high when an entry is created
• Confidence decreases as the entry ages
• Confidence is capped by semantic similarity
• Expired entries always fail confidence checks

Confidence is not a probability. It is a reuse heuristic designed to favor correctness over speed.

How Confidence Affects Cache Reuse Decisions

Confidence scoring acts as a gatekeeper in the cache pipeline.

Even if:

• the entry is not expired
• the semantic similarity is above threshold

…the cache will reject reuse if confidence falls below an acceptable level.

When this happens:

• the cache treats the entry as unsafe
• the request falls back to the LLM
• a fresh response is generated and stored

This behavior ensures that the cache degrades gracefully.

As uncertainty increases, the system automatically shifts work back to the LLM rather than returning questionable results.

Why Confidence Belongs in the Cache (Not the LLM)

It’s tempting to push this logic downstream and let the LLM “fix” stale responses.

That approach fails for two reasons:

• the LLM has no context about cache age
• the LLM cannot distinguish reused content from fresh inference

Confidence must be enforced before reuse, not after generation.

By embedding confidence checks directly into the cache, we ensure that reuse decisions are explicit, explainable, and controllable.

Implementing Confidence Scoring for LLM Cache Optimization (Code Walkthrough)

In the previous section, we introduced confidence scoring as a conceptual safeguard: a way to prevent semantically similar but stale responses from being reused.

In this section, we make that idea concrete by implementing it.

We will walk through where confidence is computed, where it is enforced, and what happens when a cached entry is rejected.

Where Confidence Is Computed

Confidence is computed inside the semantic cache, alongside similarity scoring.

def compute_confidence(similarity: float, created_at: int, ttl: int) -> float:
    age = time.time() - created_at

    if ttl <= 0:
        freshness = 1.0
    else:
        freshness = max(0.0, 1.0 - (age / ttl))

    confidence = (0.7 * similarity) + (0.3 * freshness)
    return round(confidence, 3)

This function combines 2 signals:

• Semantic similarity: how close the meanings are
• Freshness: how recent the response is relative to its TTL

The exact weights are not important here. What matters is the behavior:

• Fresh, similar responses score high confidence
• Old responses lose confidence over time
• Expired entries collapse to low confidence

Confidence is therefore bounded, decaying, and explicitly defined.

Why Confidence Is Computed in the Cache

Notice that confidence is computed inside the cache layer, not in the API.

This ensures:

• all reuse decisions are centralized
• confidence logic is applied consistently
• the API remains an orchestration layer, not a policy engine

The API does not need to understand how confidence is computed — only whether it is acceptable.

Where Confidence Is Enforced

Confidence enforcement happens in the request pipeline in ask.py.

elif cached.get("confidence", 0.0) < 0.7:
    miss_reason = "low_confidence"

This check occurs after:

• exact or semantic matching
• TTL validation
• poisoning checks

And before a cached response is returned.

If confidence is below the threshold:

• the cache entry is rejected
• the request is treated as a cache miss
• the pipeline falls back to the LLM

This ensures that reuse happens only when confidence meets an acceptable threshold.

Why Rejection Is Safer Than Reuse

When confidence is low, the system has 2 choices:

• reuse a response it does not fully trust
• generate a fresh response

This implementation always chooses the second option.

The cost of an extra LLM call is predictable.

The cost of serving an incorrect response is not.

By rejecting low-confidence entries, the cache degrades gracefully instead of failing silently.

What Happens After Rejection

Once a cached entry is rejected:

• the request proceeds to the LLM
• a new response is generated
• the new response is stored with a fresh timestamp and TTL

Over time, this naturally refreshes the cache without requiring explicit invalidation logic.

Making Rejections Observable

Confidence-based rejections are not hidden.

They are surfaced via:

• miss_reason = "low_confidence"
• debug metadata returned to the client
• cache miss metrics

This makes it possible to understand why the cache did not reuse a response — a critical property when tuning thresholds later.

Query Normalization and Deduplication for Efficient Semantic Caching

At this point, our semantic cache is safe against stale and low-confidence responses. However, there is another failure mode that appears once the system runs for longer periods of time:

The cache slowly fills with duplicate entries representing the same query.

This problem does not break correctness, but it can silently degrade cache quality and efficiency.

Why Duplicate Cache Entries Are a Problem

In natural language systems, users rarely type queries the same way twice.

Consider the following inputs:

• What is semantic caching?
• What is semantic caching
• What   is   semantic   caching?

From a human perspective, these queries are identical.

From a naïve cache’s perspective, they are completely different strings.

If we store each variation separately:

• cache size grows unnecessarily
• similarity scans become slower
• cache hit rate decreases
• identical LLM work is repeated

This is not a semantic problem — it is a normalization problem.

Normalizing Queries Before Caching

To prevent this, the cache normalizes queries before storing them.

def _hash_query(query: str) -> str:
    normalized = " ".join(query.lower().split())
    return hashlib.sha256(normalized.encode()).hexdigest()

This function performs 3 important steps:

• Lowercasing: Ensures case-insensitive matching
• Whitespace normalization: Collapses extra spaces and removes leading/trailing whitespace
• Hashing: Produces a fixed-length identifier for fast comparison

The result is a stable representation of the query’s structure, not its formatting.

Deduplication at Store Time

Deduplication happens when a new cache entry is about to be written.

query_hash = self._hash_query(query)

for key in self.r.smembers(f"{self.namespace}:keys"):
    data = self.r.hgetall(key)
    if data and data.get("query_hash") == query_hash:
        return

Before storing a new entry, the cache checks whether an entry with the same normalized hash already exists in the cache.

If it does:

• the new entry is not stored
• the cache avoids creating a duplicate
• storage space and future scans are preserved

This approach ensures that identical queries map to a single cache entry, regardless of how they were formatted.

Why Deduplication Happens in the Cache Layer

Deduplication is enforced inside the cache rather than in the API layer.

This design ensures:

• all cache writes are normalized consistently
• deduplication logic lives next to storage logic
• API code remains simple and declarative

The API does not need to care how deduplication works — only that the cache remains clean.

Why Hash-Based Deduplication Works Well Here

Using a hash instead of raw strings provides several advantages:

• fixed-length comparisons
• efficient storage
• no dependency on query length
• practical collision resistance

For this system, SHA-256 is more than sufficient. The goal is stability and simplicity, not cryptographic security.

What Deduplication Does Not Solve

It’s important to understand the limits of this approach.

Hash-based deduplication:

• prevents exact duplicates after normalization
• does not merge semantically similar queries
• does not replace semantic caching

In other words:

• deduplication keeps the cache clean
• semantic similarity keeps the cache useful

They solve different problems and complement each other.

Preventing Cache Poisoning in Semantic Caching for LLM Systems

So far, we’ve protected the semantic cache against staleness, low confidence, and duplicate entries. There is one more failure mode that can silently undermine the entire system if left unchecked:

Cache poisoning — storing responses that should never be reused.

Cache poisoning does not usually crash the system. Instead, it causes the cache to confidently serve bad answers repeatedly, amplifying a single failure into many incorrect responses.

What Cache Poisoning Looks Like in LLM Systems

In the context of LLM-backed systems, cache poisoning typically happens when:

• the LLM returns an error message
• the response is empty or incomplete
• the output is malformed due to a timeout or partial generation

If these responses are cached, every future “hit” returns the same failure instantly — fast, but incorrect.

This is especially dangerous because:

• the cache appears to be working
• responses are returned quickly
• the system looks healthy from the outside

Poisoning Prevention Strategy

Rather than trying to detect every possible bad response, this system uses a simple, conservative heuristic:

If a response looks unsafe, do not cache it.

This keeps the logic easy to reason about and avoids false positives.

Detecting Poisoned Entries

Poisoning detection is implemented in a dedicated helper function.

def is_poisoned(entry):
    resp = entry.get("response", "")
    if not resp or resp.startswith("[LLM Error]"):
        return True
    return False

This function flags an entry as poisoned if:

• the response is empty, or
• the response is an explicit LLM error

These conditions are intentionally strict. When in doubt, the entry is treated as unsafe.

Where Poisoning Is Enforced

Poisoning checks are applied before any cached response is reused in ask.py.

elif is_poisoned(cached):
    miss_reason = "poisoned"

If a cached entry is poisoned:

• it is rejected immediately
• the request is treated as a cache miss
• the pipeline falls back to the LLM

This ensures that invalid responses are never reused, even if they have high similarity or appear fresh.

Why Poisoned Entries Are Rejected, Not Repaired

The cache does not attempt to “fix” poisoned entries.

Trying to repair cached LLM output introduces:

• ambiguity
• hidden transformations
• unpredictable behavior

Instead, the system takes the safest possible action:

• reject the entry
• generate a fresh response
• overwrite with a clean result

This keeps the cache behavior explicit and predictable.

Making Poisoning Visible

Just like low-confidence rejections, poisoning is not silent.

The reason is surfaced via:

• miss_reason = "poisoned"
• debug metadata returned to the client
• cache miss metrics

This makes it possible to distinguish between:

• semantic misses
• safety rejections
• forced fallbacks

Visibility is a critical part of safety.

What This Approach Does Not Cover

This poisoning strategy is intentionally simple.

It does not attempt to:

• analyze response quality
• validate structured output
• detect hallucinations
• score semantic correctness

Those checks are domain-specific and belong outside the cache.

The cache’s responsibility is narrow:

Do not reuse responses that are obviously unsafe.

End-to-End Semantic Cache Hardening: TTL, Confidence, and Safety Demos

In Lesson 1, we verified that semantic caching works.

In this lesson, we harden that system by watching each safety mechanism activate in practice.

The goal of these demos is not performance testing.

The goal is behavioral verification.

Each demo isolates one hardening feature and makes its effect visible through the response payload.

Demo Case 1: TTL Expiration Forces a Cache Miss

Start by sending a query and populating the cache:

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "Explain semantic caching for LLMs"}'

This first request falls back to the LLM and stores a new cache entry.

After waiting longer than the configured TTL, send the same request again:

sleep 61
curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "Explain semantic caching for LLMs"}'

Expected Behavior

• Exact-match lookup finds an entry
• TTL validation fails
• Entry is rejected
• LLM is called again

Example response

{
  "from_cache": false,
  "debug": {
    "hit": false,
    "miss_reason": "no_match"
  }
}

This confirms that stale responses are not reused.

Demo Case 2: Semantic Reuse When Confidence Remains High

Now consider a cached response that is still within TTL and retains sufficient confidence.

Send a semantically similar query:

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "How does semantic caching reduce LLM calls?"}'

Expected Behavior

• Semantic similarity match found
• Confidence computed
• Confidence above threshold
• Cached response reused

Example response

{
  "from_cache": true,
  "debug": {
    "hit": true,
    "cache_path": "semantic_match",
    "confidence": 0.81
  }
}

This demonstrates that semantic reuse is allowed when both relevance and freshness remain acceptable.

Demo Case 3: Failed LLM Responses Are Never Cached

A safe semantic cache must ensure that failed LLM responses are never reused. This demo demonstrates write-time cache poisoning prevention.

This system enforces that rule at write time.

if not response.startswith("[LLM Error]"):
    cache.store(...)

Only valid responses are ever written to Redis.

How We Demonstrate This

We do not shut down Ollama or the embedding service.

Network failures abort the request before caching logic runs and are not suitable demos.

Instead, we simulate an LLM failure.

Step 1: Temporarily Simulate an LLM Error

In generate_llm_response():

if "simulate_error" in prompt.lower():
    return "[LLM Error] Simulated failure"

Step 2: Send a Query

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "Simulate error in semantic caching"}'

Expected Behavior

• from_cache = false
• Cache miss
• Error response returned

Step 3: Send the Same Query Again

Expected Result

• Cache miss again
• LLM called again
• No cached response reused

Why the Miss Reason Is no_match

• Failed responses are never stored
• No cache entry exists to reject or evaluate
• Cache poisoning checks apply only to existing entries

This is intentional and correct.

Demo Case 4: Deduplication Under Query Variations

Send a query with unusual spacing:

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "   What   is   semantic   caching?   "}'

Then send the normalized version:

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "What is semantic caching?"}'

Expected Behavior

• Both queries map to the same normalized hash
• Only one cache entry exists
• Exact-match reuse occurs

Example response

{
  "from_cache": true,
  "debug": {
    "hit": true,
    "cache_path": "exact_match"
  }
}

This confirms deduplication is working correctly.

Demo Case 5: Observing Metrics After Hardening

After running several demos, inspect the metrics endpoint:

curl http://localhost:8000/internal/metrics

Example response

{
  "hits": 3,
  "misses": 4,
  "llm_calls": 4,
  "_note": "In-memory metrics. Reset on restart. Not production-ready."
}

Metrics help you verify that:

• safety rejections increase misses
• LLM calls rise when reuse is unsafe
• the system degrades gracefully

What These Demos Prove

Across these scenarios, we verified that:

• Stale entries are rejected
• Low-confidence reuse is prevented
• Poisoned responses are never cached
• Duplicate entries are avoided
• Cache behavior is observable and explainable

The cache no longer optimizes for speed alone.

It optimizes for safe reuse.

Semantic Caching Limitations: Trade-Offs in LLM Optimization Systems

By this point, we’ve built a semantic cache that is not only functional, but also hardened against common failure modes: staleness, low confidence, poisoning, duplication, and silent reuse.

However, no system design is complete without clearly stating what it does not attempt to solve.

This section makes those boundaries explicit.

Why This Cache Still Uses O(N) Scans

All semantic lookups in this implementation perform a linear scan over cached entries.

That means:

• every semantic search compares the query embedding against all stored embeddings
• time complexity grows linearly with cache size

This is not an oversight.

It is a deliberate design choice made for:

• teaching clarity
• transparency
• small-to-medium cache sizes

By avoiding ANN indexes or vector databases, every decision remains visible and debuggable. You can trace exactly why a match was selected or rejected.

For educational systems and low-volume services, this trade-off is acceptable — and often desirable.

What We Intentionally Did Not Implement

To keep the system focused and understandable, several production features were intentionally left out:

• Approximate nearest neighbor (ANN) indexing
• Redis Vector Search or RediSearch
• Background garbage collection workers
• Distributed locks for thundering herd prevention
• Request coalescing or single-flight patterns
• Multi-process or persistent metrics
• Cache warming strategies

Each of these adds complexity that would obscure the core ideas being taught.

This cache is designed to explain semantic caching, not to compete with specialized retrieval infrastructure.

When This Design Is “Good Enough”

This architecture works well when:

• cache size is modest (hundreds to low thousands of entries)
• traffic is low to moderate
• correctness and explainability matter more than raw throughput
• you are experimenting with semantic reuse behavior
• you want to understand cache dynamics before scaling

Typical examples include:

• internal tools
• developer-facing APIs
• research prototypes
• educational systems
• early-stage LLM applications

In these contexts, the simplicity of the design is a strength, not a weakness.

When You Need a Vector Database or ANN Index

As usage grows, linear scans eventually become the bottleneck.

You should consider a dedicated vector search solution when:

• cache size grows into tens or hundreds of thousands of entries
• latency requirements become strict
• multiple workers or services share the same cache
• semantic search dominates request time

At that point, technologies such as the following:

• FAISS (Facebook AI Similarity Search)
• Milvus
• Pinecone
• Redis Vector Search

become appropriate.

Importantly, the hardening concepts from this lesson still apply. TTLs, confidence scoring, poisoning prevention, and observability remain relevant even when the storage backend changes.

The Core Trade-Off, Revisited

This lesson deliberately favors:

• clarity over cleverness
• explicit decisions over hidden automation
• safety over aggressive reuse

That makes it an ideal foundation, not a final destination.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: June 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, we took a working semantic cache and made it safe, bounded, and explainable.

Rather than focusing on improving cache hit rates at all costs, we introduced guardrails to ensure cached LLM responses are reused only when they are trustworthy.

We added application-level TTL validation to prevent stale responses from persisting indefinitely, combined semantic similarity with freshness through confidence scoring, and enforced explicit rejection paths for low-confidence and expired entries.

We also addressed subtle but dangerous failure modes that appear in real systems over time. Query normalization and deduplication prevent silent cache bloat, and poisoning checks ensure that error responses are never reused.

Observability signals make every cache decision inspectable rather than implicit. Together, these changes transform the cache from a performance optimization into a reliability component.

Finally, we made the system’s limitations explicit. This design favors clarity, correctness, and debuggability over raw scalability. It deliberately avoids ANN indexes, vector databases, and distributed coordination, making it suitable for small-to-medium systems and educational use cases.

As workloads grow, the same hardening principles apply even when the underlying storage or retrieval strategy changes.

With this lesson, semantic caching is no longer just fast. It is defensive, explainable, and production-aware.

Citation Information

Singh, V. “Semantic Caching for LLMs: TTLs, Confidence, and Cache Safety,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/ahr3p

@incollection{Singh_2026_semantic-caching-llms-ttls-confidence-cache-safety,
  author = {Vikram Singh},
  title = {{Semantic Caching for LLMs: TTLs, Confidence, and Cache Safety}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/ahr3p},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Semantic Caching for LLMs: TTLs, Confidence, and Cache Safety appeared first on PyImageSearch.

Semantic Caching for LLMs: FastAPI, Redis, and Embeddings

In this lesson, you will learn how to build a semantic cache for LLM applications using FastAPI, Redis, and embedding-based similarity search, and how requests flow from exact matches to semantic matches before falling back to the LLM.

This lesson is the 1st in a 2-part series on Semantic Caching for LLMs:

1. Semantic Caching for LLMs: FastAPI, Redis, and Embeddings (this tutorial)
2. Lesson 2

To learn how to build a semantic cache for LLM applications using embeddings and Redis, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Introduction: Why Semantic Caching Matters for LLM Systems

Cost, Latency, and Redundant LLM Calls

Large language models are powerful, but they are not cheap. Every request to an LLM involves tokenization, inference, decoding, and network overhead. Even when models are hosted locally, response times are measured in hundreds of milliseconds or seconds rather than microseconds.

In real applications, this cost compounds quickly. Users often ask similar questions repeatedly, either across sessions or within the same workflow. Each request is treated as a fresh LLM invocation, even when the underlying intent has already been handled before.

This leads to 3 systemic problems:

• High latency: Users wait for responses that could have been reused instantly
• Increased cost: Identical reasoning is paid for multiple times
• Wasted capacity: LLM throughput is consumed by redundant requests

These issues become especially visible under load, where repeated paraphrased queries can overwhelm an otherwise well-sized system.

Why Exact-Match Caching Breaks Down for Natural Language

Traditional caching assumes that identical inputs produce identical outputs. This works well for APIs, database queries, and deterministic functions. It fails for natural language.

From a string-matching perspective, the following queries are completely unrelated:

• “What is semantic caching?”
• “Can you explain how semantic caching works?”
• “How does caching based on embeddings work for LLMs?”

A traditional cache keyed on raw strings will miss all three. As a result, the system calls the LLM three times, even though a human would expect the same answer.

This brittleness causes exact-match caches to have extremely low hit rates in LLM-backed systems. Worse, it gives a false sense of optimization. The cache exists, but it almost never helps in practice.

Where Semantic Caching Fits in Real Systems

Semantic caching addresses this mismatch by caching meaning instead of exact text.

Rather than asking “have I seen this string before?”, a semantic cache asks “have I answered something semantically similar before?”. It does this by converting queries into embeddings and comparing them using a similarity metric such as cosine similarity.

In a real system, semantic caching sits between the application layer and the LLM:

• The application sends a query
• The cache evaluates whether a prior response is reusable
• Only true cache misses reach the LLM

When designed correctly, this layer is invisible to the user. Responses feel faster, costs drop, and the system scales more gracefully without changing the frontend or prompt logic.

This lesson focuses on building that layer explicitly and transparently, using FastAPI, Redis, and embeddings, without hiding the mechanics behind heavy abstractions.

Exact-match caching treats paraphrased queries as unique requests, resulting in repeated LLM calls. Semantic caching groups similar queries by meaning, allowing responses to be reused and reducing both latency and cost.

How Semantic Caching Works for LLMs: Embeddings and Similarity Search Explained

Section 1 explained why semantic caching exists.

This section explains how it works, conceptually, before we touch any FastAPI, Redis, or code.

The goal here is to give the reader a mental execution model they can keep in their head while reading the implementation.

From Text to Meaning: Embeddings as the Cache Key

Semantic caching replaces raw text comparison with vector similarity.

Instead of caching responses under the literal query string, the system converts each query into an embedding: a high-dimensional numeric vector that captures semantic meaning. Queries that are worded differently but mean the same thing produce embeddings that are close together in vector space.

This is what allows the cache to recognize paraphrases as equivalent:

• “How do I reset my password?”
• “I forgot my password, what should I do?”
• “Guide me through password recovery”

Exact strings differ. Embeddings do not.

At a high level, semantic caching works by:

• Generating an embedding for the incoming query
• Comparing it against embeddings stored in the cache
• Reusing a cached response if similarity is high enough

The similarity metric used in this lesson is cosine similarity, which measures the angle between two vectors rather than their raw magnitude.

Why a Layered Cache Beats Semantic-Only Caching

While semantic matching is powerful, it is also computationally expensive.

Embedding generation requires a model call. Similarity search requires vector math. Doing this for every request, even when the exact same query has already been seen, would be wasteful.

That is why this lesson uses a layered caching strategy.

Layer 1: Exact Match (Fast Path)

The query is normalized and hashed.

If the same query has already been answered, the response is returned immediately.

• No embedding generation
• No similarity computation
• Minimal latency

This handles repeated identical queries efficiently.

Layer 2: Semantic Match (Flexible Path)

If no exact match exists, the query is embedded and compared against cached embeddings.

This layer catches:

• paraphrases
• minor wording differences
• reordered phrases

Semantic matches trade compute cost for much higher cache hit rates.

Layer 3: LLM Fallback (Slow Path)

If neither exact nor semantic matches succeed, the request is forwarded to the LLM.

The response is then stored in the cache so future requests can reuse it.

This layered approach ensures:

• the cheapest checks happen first
• expensive operations are only used when necessary

Confidence, Freshness, and Cache Safety

Semantic similarity alone is not enough to decide whether a cached response should be reused.

This lesson introduces the idea of confidence scoring, which combines:

• Similarity: how close the embeddings are
• Freshness: how old the cached entry is

A highly similar but stale response should not necessarily be trusted. Likewise, a fresh response with low similarity should be rejected.

In addition, cached entries are validated to prevent:

• expired responses
• poisoned entries (errors, empty outputs)

These checks ensure the cache improves correctness and performance rather than degrading them.

Incoming queries first attempt an exact-match lookup, then fall back to semantic similarity search using embeddings, and finally call the LLM only on cache miss. This ordering minimizes latency and unnecessary model calls.

Note: In this lesson, we implement this flow using Redis as a simple embedding store with linear similarity scans, rather than a dedicated vector database.

Semantic Caching Architecture and Request Flow

In Section 2, you learned how semantic caching works conceptually.

In this section, we map that mental model to a real request flow in an LLM-backed service.

The goal is to answer one question clearly:

What happens, step by step, when a user sends a request to this system?

We will stay implementation-aware, but not code-specific yet. That comes next.

High-Level System Components

At a high level, the system consists of 5 logical components:

• API layer: Receives user requests and orchestrates the caching pipeline.
• Exact-match cache: Performs fast hash-based lookups for identical queries.
• Embedding model: Converts text queries into semantic vectors when needed.
• Semantic cache: Stores embeddings and responses and performs similarity matching.
• LLM: Acts as the final fallback when no cache entry is suitable.

Each component has a narrowly defined responsibility. This separation is intentional and keeps the system easy to reason about and extend.

In this implementation:

• The API layer is built using FastAPI and acts as the orchestration point.
• Redis is used as the backing store for both exact-match and semantic cache layers.
• Ollama provides both embedding generation and LLM inference locally.

These choices keep the system lightweight, self-contained, and easy to reason about while still reflecting real production patterns.

End-to-End Request Flow

When a user sends a query, the system processes it in the following order.

Step 1: Request enters the API

The API receives a text query along with optional flags, such as whether to use the bypass_cache. Input validation happens immediately to prevent meaningless or malformed queries from entering the pipeline.

This ensures the cache is not polluted with empty or invalid entries.

Step 2: Exact-match cache lookup

The query is normalized and hashed.

The system checks whether an identical query has already been answered.

• If an exact match exists and is valid, the response is returned immediately.
• No embeddings are generated.
• The LLM is not touched.

This is the fastest possible path through the system.

Step 3: Embedding generation

If the exact-match lookup fails, the query is passed to the embedding model.

The model converts the text into a numeric vector that captures semantic meaning. This vector becomes the key for semantic comparison.

This step is intentionally skipped when an exact match succeeds.

Step 4: Semantic cache lookup

The embedding is compared against cached embeddings using a similarity metric.

A cached response is reused only if:

• similarity exceeds a defined threshold
• the entry has not expired
• the entry is not poisoned
• the computed confidence is high enough

If a suitable match is found, the response is returned to the user without calling the LLM.

Step 5: LLM fallback and cache population

If both cache layers miss, the request is forwarded to the LLM.

Once a response is generated:

• it is returned to the user
• it is stored in the cache with metadata, timestamps, and TTL (Time To Live)

This ensures future requests can reuse the result.

Why This Architecture Works Well

This architecture is intentionally conservative and explicit.

• Cheap operations happen first.
• Expensive operations are deferred.
• Every step is observable and debuggable.
• No component hides complexity behind opaque abstractions.

Most importantly, the system degrades gracefully. Even when the cache provides no benefit, the request still succeeds via the LLM.

User queries enter the API, attempt an exact-match lookup, fall back to semantic similarity search using embeddings, and call the LLM only when both cache layers miss. Successful LLM responses are stored for future reuse.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Configuring Your Environment for Semantic Caching: FastAPI, Redis, and Ollama Setup

To follow this guide, you need a small set of Python libraries and system services that support API orchestration, vector similarity, and LLM interaction. The goal is to keep the environment lightweight, reproducible, and easy to reason about.

At a minimum, you will need:

• Python 3.10 or newer
• Redis (used as the cache backing store)
• An LLM + embedding provider (Ollama in this tutorial)

All required Python dependencies are pip-installable.

Installing Python Dependencies

Create and activate a virtual environment (recommended), then install the required packages:

$ pip install fastapi uvicorn redis httpx python-dotenv numpy

These libraries provide the following functionality:

• fastapi: API layer and request orchestration
• uvicorn: ASGI server for running the service
• redis: client Communication with the cache store
• httpx: HTTP client for embedding and LLM calls
• numpy: Vector math for cosine similarity
• python-dotenv: Environment-based configuration

Verifying Redis

This lesson assumes Redis is running locally on the default port.

You can verify Redis is available with:

$ redis-cli ping
PONG

If Redis is not installed, you can start it quickly using Docker (but you also can spin it up using the docker-compose.yml we provide in the code zip):

$ docker run -p 6379:6379 redis:7

Setting Up Ollama

This system uses Ollama for both embedding generation and LLM inference. Make sure Ollama is installed and running, and that the required models are available.

For example:

$ ollama pull nomic-embed-text
$ ollama pull llama3.2

Once running, Ollama exposes local HTTP endpoints that the application will call directly for embeddings and text generation.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Project Structure

Before diving into individual components, let’s take a moment to understand how the project is organized.

A clear directory structure is especially important in LLM-backed systems, where responsibilities span API orchestration, caching, embeddings, model calls, and observability. In this project, each concern is isolated into its own module so the request flow remains easy to trace and reason about.

After downloading the source code from the “Downloads” section, your directory structure should look like this:

.
├── app
│   ├── api
│   │   ├── __init__.py
│   │   └── ask.py
│   ├── cache
│   │   ├── __init__.py
│   │   ├── poisoning.py
│   │   ├── schemas.py
│   │   ├── semantic_cache.py
│   │   └── ttl.py
│   ├── config
│   │   ├── __init__.py
│   │   └── settings.py
│   ├── embeddings
│   │   ├── __init__.py
│   │   └── embedder.py
│   ├── llm
│   │   ├── __init__.py
│   │   └── ollama_client.py
│   ├── main.py
│   └── observability
│       └── metrics.py
├── complete-codebase.txt
├── docker-compose.yml
├── Dockerfile
├── README.md
└── requirements.txt

Let’s break this down at a high level.

The app/ Package

The app/ directory contains all runtime application code. Nothing outside this folder is imported at execution time.

This keeps the service self-contained and makes it easy to reason about deployment and dependencies.

app/main.py: Application Entry Point

This file defines the FastAPI application and registers all routers.

It contains no business logic — only service wiring. Every request into the system enters through this file.

app/api/: API Layer

The api/ package defines HTTP-facing endpoints.

• ask.py: Implements the /ask endpoint and acts as the orchestration layer for the entire semantic caching pipeline.

The API layer is responsible for:

• input validation
• enforcing cache ordering
• coordinating cache, embeddings, and LLM calls
• returning structured debug information

It does not implement caching or similarity logic directly.

app/cache/: Caching Logic

This package contains all cache-related functionality.

• semantic_cache.py: Core semantic cache implementation (exact match, semantic match, Redis storage, similarity search).
• schemas.py: Defines the cache entry schema used for Redis storage.
• ttl.py: Application-level TTL configuration and expiration checks.
• poisoning.py: Safety checks to prevent invalid or error responses from being reused.

By isolating caching logic here, the API layer stays clean and reusable.

app/embeddings/: Embedding Generation

• embedder.py: Handles embedding generation via Ollama’s embedding endpoint.

This module has a single responsibility: convert text into semantic vectors.

It does not cache, rank, or validate embeddings.

app/llm/: LLM Client

• ollama_client.py: Wraps calls to the Ollama text-generation endpoint.

Keeping LLM interaction isolated allows the rest of the system to remain model-agnostic.

app/observability/: Metrics

• metrics.py: Implements simple in-memory counters for cache hits, misses, and LLM calls.

These metrics are intentionally lightweight and meant for learning and debugging, not production monitoring.

Configuration and Infrastructure

Outside the app/ directory:

• config/settings.py: Centralizes environment-based configuration (Redis host, TTLs, model names).
• Dockerfile and docker-compose.yml: Define a reproducible runtime environment for the API and Redis.
• requirements.txt: Lists all Python dependencies required to run the service.

FastAPI Entry Point for Semantic Caching: Wiring the API Service

Before we look at caching logic, embeddings, or Redis, it’s important to understand how the service itself is wired together. Every request to the semantic cache enters the system through a single FastAPI application, defined in app/main.py.

This file acts as the entry point of the service. Its responsibility is not to implement business logic, but to connect the application components and expose HTTP routes.

Application Entry Point (app/main.py)

from fastapi import FastAPI
from api.ask import router as ask_router

app = FastAPI(title="Semantic Cache Basics")
app.include_router(ask_router)

Let’s break this down.

The FastAPI() call creates the application object. This object represents the entire web service and is what the ASGI (Asynchronous Server Gateway Interface) server (uvicorn) runs when the container starts.

The application itself contains no knowledge of caching, embeddings, or LLMs. It simply defines a runtime container that will host those capabilities.

Router Registration

Instead of defining endpoints directly in main.py, the application imports a router from api/ask.py and registers it using include_router().

This pattern serves several purposes:

• Separation of concerns: Routing and request handling live outside the application entry point.
• Scalability: As the system grows, additional routers (for health checks, metrics, or admin endpoints) can be added without modifying core application wiring.
• Readability: main.py remains easy to understand at a glance, even as the codebase expands.

At runtime, FastAPI merges the routes defined in ask_router into the main application. When a request arrives at the /ask endpoint, FastAPI resolves it through the registered router and forwards it to the appropriate handler function.

Why This Matters

Keeping the entry point minimal is intentional. It ensures that:

• The application startup process is predictable
• Routing logic is easy to trace
• Core functionality can evolve independently of service wiring

With the application structure in place, we can now focus on what actually happens when a request reaches the system.

In the next section, we will walk through the /ask endpoint and see how it orchestrates exact-match caching, semantic search, and LLM fallback step by step.

FastAPI Ask Endpoint: End-to-End Semantic Caching Request Flow

This section makes the architecture concrete. We now walk through the /ask endpoint, which orchestrates the entire semantic caching pipeline from request arrival to response delivery.

The goal here is not to memorize code, but to understand why each step exists, where it lives, and how it protects performance, cost, and correctness.

The Role of the Ask Endpoint

The Ask endpoint is the control plane of the system.

It does not:

• Compute similarity
• Store embeddings
• Talk directly to Redis internals

Instead, it:

• Validates input
• Decides which cache layers to consult
• Enforces ordering between cheap and expensive operations
• Collects observability signals
• Guarantees a response even on cache failure

This separation is intentional. Cache logic remains reusable and testable, while orchestration logic stays explicit at the API boundary.

Defining the API Contract

We begin by defining the request and response models.

class AskRequest(BaseModel):
    query: str
    bypass_cache: bool = False

The request consists of a user query and an optional bypass_cache flag. This flag allows us to force a cache miss during debugging or testing, ensuring that the LLM and embedding pipeline still function correctly.

Before the request ever reaches the cache, the query field is validated.

@field_validator('query')
@classmethod
def validate_query(cls, v: str) -> str:
    if not v or not v.strip():
        raise ValueError("Query cannot be empty or whitespace-only")
    return v.strip()

This validation step protects the system at the boundary. Rejecting empty or whitespace-only queries prevents:

• wasted embedding computation
• cache pollution with meaningless entries
• unnecessary LLM calls

This is a recurring pattern in production systems: fail fast, before expensive operations are triggered.

class AskResponse(BaseModel):
    response: str
    from_cache: bool
    similarity: float
    debug: dict

The response model intentionally exposes diagnostic information through fields such as from_cache, similarity, and debug. During development, this makes cache behavior transparent rather than opaque.

Initializing the Cache

Before handling requests, we create a SemanticCache instance:

cache = SemanticCache()

The endpoint itself remains stateless. All persistence and reuse live inside the cache layer.

Step 1: Entering the Endpoint

The endpoint is registered using FastAPI’s routing mechanism:

@router.post("/ask", response_model=AskResponse)
def ask_endpoint(request: AskRequest):

FastAPI automatically validates incoming requests and outgoing responses using the schemas defined earlier. If invalid data enters or exits the system, FastAPI raises an error instead of silently failing.

Inside the handler, we extract the query and initialize tracking state:

query = request.query
miss_reason = None

The miss_reason variable exists purely for observability. Rather than treating cache misses as a black box, we explicitly track why a miss occurred.

Step 2: Exact-Match Cache Lookup (Fast Path)

The first decision point is the exact-match cache lookup:

if not request.bypass_cache:
    cached = cache.search(None, exact_query=query)

This is the cheapest path through the system.

If the same query has already been answered, the response can be returned immediately:

• no embeddings are generated
• no similarity computation occurs
• the LLM is not touched

If a cached entry is found, it is validated:

if is_expired(cached):
    miss_reason = "expired"
elif is_poisoned(cached):
    miss_reason = "poisoned"
elif cached.get("confidence", 0.0) < 0.7:
    miss_reason = "low_confidence"

Only entries that are fresh, valid, and confident are allowed to short-circuit the pipeline.

When all checks pass, the endpoint returns immediately:

metrics.cache_hit()
return AskResponse(...)

This path typically completes in milliseconds and handles repeated identical queries efficiently.

Step 3: Embedding Generation (Escalation Point)

If the exact-match lookup fails or is bypassed, the endpoint escalates:

embedding = embed_text(query)

Embedding generation is expensive, even when running locally. For this reason, it is intentionally delayed until all cheaper options have been exhausted.

This single design choice has a significant impact on system efficiency.

Step 4: Semantic Cache Lookup

With the embedding available, the endpoint attempts a semantic search:

cached = cache.search(embedding)

This path catches paraphrased and reworded queries. As before, cached entries are validated to ensure they are safe to reuse.

If a suitable match is found, the response is returned without calling the LLM.

Step 5: Explicit Cache Bypass

The bypass_cache flag is handled explicitly:

if request.bypass_cache:
    miss_reason = "bypass"

This allows controlled testing and debugging without modifying code or disabling cache logic globally.

Step 6: LLM Fallback and Cache Population

If both cache layers miss, the request is forwarded to the LLM:

metrics.cache_miss()
response = generate_llm_response(query)
metrics.llm_call()

This is the slowest path through the system, but it guarantees correctness.

Successful responses are stored in the cache:

if not response.startswith("[LLM Error]"):
    cache.store(query, embedding, response, metadata=metadata)

Responses beginning with [LLM Error] are intentionally not cached, preventing cache poisoning and ensuring failures do not propagate to future requests.

Control Flow Summary

The endpoint follows a simple, explicit sequence:

Every expensive operation is deferred until absolutely necessary.

Embeddings: Turning Text into Semantic Vectors

Up to this point, we have treated embeddings as a black box: something expensive that we try to avoid unless absolutely necessary.

In this section, we will open that box just enough to understand what embeddings are, when they are generated, and why they enable semantic caching without diving into vector math or model internals.

Why Embeddings Exist in This System

Exact-match caching works only when queries are identical at the string level. As soon as wording changes, exact matching breaks down.

Embeddings solve this problem by converting text into a numeric representation that captures meaning rather than surface form.

Queries that mean the same thing tend to produce vectors that are close together in vector space, even if their wording differs significantly.

This is the foundation that makes semantic caching possible.

Embedding Generation Happens on Demand

In our implementation, embeddings are generated only after the exact-match cache fails.

This decision is intentional.

Embedding generation involves:

• a model invocation
• network overhead
• serialization and deserialization
• non-trivial latency

Because of this cost, embeddings are treated as an escalation step, not a default operation.

This is why the /ask endpoint first attempts an exact-match lookup before calling embed_text().

The embed_text Function

def embed_text(text: str):

This function has one responsibility: Convert input text into a semantic vector representation.

It does not perform caching, similarity search, or validation. Those concerns live elsewhere.

Calling the Embedding Model

url = f"http://{settings.OLLAMA_HOST}:{settings.OLLAMA_PORT}/api/embeddings"

Here, we construct the Ollama embedding endpoint using configuration values (e.g., settings.OLLAMA_HOST, settings.OLLAMA_PORT, etc.).

This allows the embedding service to run locally, inside Docker, or on a remote host without changing code.

resp = httpx.post(
    url,
    json={"model": settings.EMBEDDING_MODEL, "prompt": text},
    timeout=10.0
)

This request sends 2 key pieces of information to the embedding service:

• the embedding model name (e.g., nomic-embed-text)
• the input text to embed

The timeout ensures the request does not hang indefinitely. Embedding generation is expensive, but it should still fail fast if something goes wrong.

Handling the Response

resp.raise_for_status()
return resp.json().get("embedding", [])

If the request succeeds, the embedding model returns a numeric vector — typically a list of floating-point values.

This vector represents the semantic meaning of the input text and becomes the key used for similarity comparison in the cache.

At this stage, we treat the vector as an opaque object. We do not inspect its dimensionality or normalize it here.

Error Handling Strategy

except Exception as e:
    raise RuntimeError(f"Failed to generate embedding: {e}")

If embedding generation fails for any reason (network issues, model errors, timeouts), the function raises an exception.

This is intentional.

If embeddings cannot be generated, the system cannot safely perform semantic matching. Silently continuing would lead to unpredictable behavior, so we fail loudly instead.

Why the Embedder Is Intentionally Simple

Notice what this function does not do:

• it does not store embeddings
• it does not perform similarity search
• it does not retry failed requests
• it does not fall back to alternative models

Those decisions are deliberate.

For Lesson 1, the embedder exists purely to convert text into vectors. Keeping it small and focused makes the system easier to understand and test.

How the Embedder Is Used in the Pipeline

At runtime, the embedder is called only when necessary:

• Exact-match cache fails
• The query is passed to embed_text()
• The returned vector is sent to the semantic cache
• Similarity is computed against stored embeddings

This ensures embeddings are generated only when cheaper paths have already failed.

Key Takeaways

• Embeddings are generated via a simple HTTP call to a local model
• The embedder has a single responsibility
• Errors are surfaced immediately
• Embeddings act as semantic keys for cache lookup

With embedding generation understood, we are now ready to look at the semantic cache itself, how embeddings and responses are stored, scanned, and matched.

In the next section, we will walk through the semantic cache implementation, starting with a deliberately naive but correct linear scan approach.

The Semantic Cache: Cosine Similarity, Redis Storage, and Reusing Meaning

At this point, we understand how queries enter the system and how text is converted into embeddings. What remains is the component that ties everything together: the semantic cache itself.

The semantic cache is responsible for 2 things:

• Storing past queries, embeddings, and responses
• Retrieving the best reusable response for a new query

In Lesson 1, we intentionally implement the cache in the simplest correct way possible: a linear scan over cached entries. This keeps the implementation easy to reason about and makes the request flow fully transparent.

The Semantic Cache Module

The cache logic lives in semantic_cache.py:

class SemanticCache:

This class encapsulates all Redis interaction and similarity logic. The API layer never talks to Redis directly.

Initializing the Cache

def __init__(self):
    self.r = redis.Redis(
        host=settings.REDIS_HOST,
        port=settings.REDIS_PORT,
        decode_responses=True
    )
    self.similarity_threshold = 0.85
    self.namespace = "semantic_cache:v1"

Here we establish a Redis connection and configure 2 important parameters:

• Similarity threshold: Only responses with sufficiently high semantic similarity are eligible for reuse.
• Namespace prefix: All Redis keys are namespaced to avoid collisions and allow future versioning.

For Lesson 1, the exact threshold value is not important. What matters is that a threshold exists and is applied consistently.

Storing Cache Entries

The first core operation is storing new entries.

def store(self, query, embedding, response, metadata=None):

This method is called only after a successful LLM response.

Creating a Cache Entry

entry = CacheEntry(
    id=entry_uuid,
    query=query,
    query_hash=query_hash,
    embedding=json.dumps(embedding),
    response=response,
    created_at=int(time.time()),
    ttl=default_ttl(),
    metadata=metadata or {}
)

Each cache entry stores:

• the original query
• a normalized query hash (used for exact matching)
• the embedding (serialized for Redis storage)
• the LLM response
• timestamps and TTL
• optional metadata for observability

This structure allows the cache to support both exact-match and semantic lookups.

Writing to Redis

self.r.hset(redis_key, mapping=entry.dict())
self.r.sadd(f"{self.namespace}:keys", redis_key)

Each cache entry is stored as a Redis hash, and all entry keys are tracked in a Redis set.

This allows the cache to iterate over all entries during search operations.

For Lesson 1, this approach is intentionally simple and explicit.

Searching the Cache

The second core operation is lookup.

def search(self, embedding, exact_query=None):

This method supports 2 search modes, which map directly to the layered cache strategy used in the API.

Exact-Match Lookup (Fast Path)

if exact_query:
    query_hash = self._hash_query(exact_query)

When an exact query is provided, the cache first attempts a hash-based lookup.

Each cached entry is scanned until a matching hash is found. If found, the entry is returned immediately with a similarity score of 1.0.

No embeddings are involved in this path.

Semantic Lookup (Flexible Path)

If no exact match is found and an embedding is provided, the cache performs a semantic search:

sim = self.cosine_similarity(query_embedding, cached_embedding)

Each cached embedding is compared against the query embedding using cosine similarity.

Only entries that exceed the configured similarity threshold are considered candidates.

Selecting the Best Match

During the scan, the cache tracks the highest similarity score and returns the best matching entry.

This ensures that even when multiple entries are similar, the most relevant response is reused.

Why This Implementation Is O(N)

Every search scans all cached entries.

This is not an accident.

For Lesson 1, a linear scan has 3 advantages:

• the behavior is easy to understand
• the logic is fully visible
• debugging is straightforward

More advanced indexing strategies belong in later lessons.

Why Expired Entries Are Cleaned During Search

While scanning entries, expired items are removed opportunistically.

This prevents stale data from accumulating indefinitely without introducing background workers or schedulers.

Key Takeaways

• The semantic cache owns all Redis interactions
• Exact-match lookup is attempted before semantic matching
• Semantic similarity is computed using embeddings
• A linear scan trades performance for clarity
• The cache returns the best reusable response, not just the first match

At this stage, the system is fully functional: queries can be answered, cached, and reused.

Cache Entries: What Exactly Gets Stored?

So far, we’ve treated the cache as a logical concept: something that stores queries, embeddings, and responses.

In this section, we’ll make that concrete by looking at the structure of a cache entry. Understanding this structure is important because it explains why the cache can support both exact-match and semantic lookup — without duplicating data or logic.

The Cache Entry Schema

Cache entries are defined using a Pydantic model:

class CacheEntry(BaseModel):
    id: str
    query: str
    query_hash: str
    embedding: str
    response: str
    created_at: int
    ttl: int
    metadata: Optional[Dict] = Field(default_factory=dict)

Each field exists for a specific reason. Let’s walk through them one by one.

Identity and Query Fields

id: str
query: str
query_hash: str

• id: uniquely identifies the cache entry and is used to construct the Redis key.
• query: stores the original user input. This is useful for debugging and inspection.
• query_hash: stores a normalized hash of the query and enables exact-match lookup.

At this stage, it’s enough to know that the hash ensures identical queries can be matched quickly. We’ll revisit how and why this normalization matters in a later lesson.

Embedding Storage

embedding: str

Embeddings are stored as a JSON-serialized string, not as a raw Python list.

This choice is deliberate:

• Redis stores strings efficiently
• Serialization keeps the schema simple
• Deserialization happens only when similarity needs to be computed

For Lesson 1, the important takeaway is that embeddings are stored once, alongside the response they produced.

Response and Timing Information

response: str
created_at: int
ttl: int

• response: is the text returned by the LLM.
• created_at: records when the entry was generated.
• ttl: defines how long the entry is considered valid.

The cache does not rely on Redis expiration here. Instead, validity is checked at read time. This gives the application full control over when an entry should be reused or rejected.

We intentionally avoid deeper TTL semantics in this lesson.

Metadata and Safety

metadata: Optional[Dict] = Field(default_factory=dict)

Metadata allows the cache to store contextual information such as:

• pipeline name
• model identifier
• request origin

The use of default_factory=dict avoids shared mutable state across cache entries — a subtle but important correctness detail.

At this stage, metadata is informational rather than functional.

Why This Schema Works Well

This schema supports the layered caching strategy naturally:

• Exact match uses query_hash
• Semantic match uses embedding
• Freshness checks use created_at and ttl
• Safety checks use response and metadata

All required information is co-located in a single cache entry, making lookup and validation straightforward.

End-to-End Demo: Verifying Core Cache Behavior

In this section, we will verify that the semantic cache behaves as expected under a small set of controlled scenarios.

These examples are meant to be run locally by the reader. The responses shown below are representative and may vary slightly depending on the model and configuration.

Demo Case 1: Cold Request (LLM Fallback)

We begin with a query that has not been seen before.

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "What is semantic caching?"}'

Expected behavior

• Exact-match cache miss
• Semantic cache miss
• LLM call
• Cache population

Response

The key signal here is "from_cache": false, confirming the request fell back to the LLM.

Demo Case 2: Exact-Match Cache Hit

Now we send the same query again.

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "What is semantic caching?"}'

Expected behavior

• Exact-match cache hit
• No embedding generation
• No LLM call

Example response

Here, the cache reused the response immediately using an exact-match lookup.

Optional Demo: Whitespace Normalization

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "   What   is   semantic   caching?   "}'

This will hit the exact-match cache due to query normalization.

Demo Case 3: Semantic Cache Hit (Paraphrased Query)

Next, we send a paraphrased version of the original query.

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "Can you explain how semantic caching works?"}'

Expected behavior

• Exact-match cache miss
• Embedding generation
• Semantic cache hit
• No LLM call

Example response

Even though the query text is different, the cache successfully reused the response based on semantic similarity.

Demo Case 4: Forcing a Cache Miss with bypass_cache

The bypass_cache flag allows us to force the system to skip both cache layers.

curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "What is semantic caching?", "bypass_cache": true}'

Expected behavior

• Exact-match cache skipped
• Semantic cache skipped
• LLM called unconditionally

Example response

This is useful for debugging and validating that the LLM pipeline still works independently of the cache.

Observing Cache Metrics (Optional)

You can inspect basic cache statistics using the /internal/metrics endpoint:

curl http://localhost:8000/internal/metrics

Example response

These metrics make cache behavior observable without requiring external tooling.

If you can reproduce these behaviors locally, you’ve successfully implemented a working semantic cache.

In the next lesson, we will take this system and begin hardening it for real-world use.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: June 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, we built a complete semantic caching system for LLM applications from the ground up. We started by wiring a FastAPI service and defining a clean request–response contract, then implemented a layered caching strategy that prioritizes cheap exact-match lookups before escalating to semantic similarity and, finally, LLM inference.

We walked through how text queries are converted into embeddings on demand, how cached responses and embeddings are stored in Redis, and how the cache decides whether a prior response can be safely reused. By keeping the implementation intentionally simple and explicit, every step in the request flow remains observable and easy to reason about.

Finally, we verified the system end-to-end by running controlled demos: a cold request falling back to the LLM, an exact-match cache hit, a semantic cache hit for a paraphrased query, and an explicit cache bypass. At this point, you have a working semantic cache that behaves correctly, makes its decisions visible, and serves as a solid foundation for further hardening and optimization.

Citation Information

Singh, V. “Semantic Caching for LLMs: FastAPI, Redis, and Embeddings,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/yso6f

@incollection{Singh_2026_semantic-caching-for-llms-fastapi-redis-and-embeddings,
  author = {Vikram Singh},
  title = {{Semantic Caching for LLMs: FastAPI, Redis, and Embeddings}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/yso6f},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Semantic Caching for LLMs: FastAPI, Redis, and Embeddings appeared first on PyImageSearch.