CI/CD/CS — The Silver Bullet for Quality in AI
// Software Architecture & AI Engineering · New Concept · February 2026
LangChain published a reflection that resonated in the community: quality in AI systems cannot reach the same level we have achieved in traditional software development. As a senior developer with years in the trenches of quality gates, my first reaction was… to agree. But only halfway.
The premise is correct. The problem is that the implicit conclusion — that we must simply accept this limitation — is not. There is something fundamentally new to propose. Something that, in twenty years of software engineering, we never needed. And it has a name: Continuous Supervision.
// 01
The Original Sin: LLMs Are Not Deterministic
When we implement a quality gate in a traditional CI/CD pipeline, we are betting on a sacred principle of engineering: given the same input, we will get the same output. That promise allowed us to build unit tests, integration tests, smoke tests, automated regressions. The entire cathedral of modern quality rests on that deterministic foundation.
Large Language Models do not live in that cathedral. They were born in a different building — one constructed on transformers, attention mechanisms and probabilistic token prediction in a vector space of thousands of dimensions. Their fundamental nature is fuzzy analysis: approximating, completing, inferring patterns in a world where “meaning” is not a fixed point but a cloud of probabilities.
“An LLM does not calculate. An LLM talks about calculations. The difference is not philosophical — it is architectural, and it has devastating practical consequences when we try to apply the quality standards of classical software to it.”
This is not a defect that can be patched with a better prompt. It is the direct consequence of these models being trained primarily on text — human discourse, narrative, argumentation — not on formal axioms or symbolic rewriting systems.
// 02
Why AI Cannot Do Math — But Can Talk About It
This point deserves attention because it is counterintuitive and, at the same time, perfectly illuminates the quality problem. An LLM can explain Gödel’s proof, summarize linear algebra or generate Python code that solves equations. But if you ask it to calculate, in the strict sense of the term, the result can be spectacularly wrong — and it will present it with the same confidence as when it is right.
⚠ Example of a “Silent Failure” in a multi-agent system
Scenario: A multi-agent RAG system is asked to synthesize financial disclosures for a specific ticker.
The Error: The agent follows the reasoning chain perfectly and produces a professional summary. However, it uses a non-existent API parameter to fetch the data or misinterprets a subtle semantic nuance in the disclosure, leading to a factually wrong but logically "perfect" conclusion.
The Verdict: This isn't a bug you can patch with a unit test. It’s an inherent byproduct of probabilistic architecture.
These are not bugs in a “bad” model. They are emergent behaviors of architectures trained to predict the most likely next token, not to execute formal operations on a precise internal state.
The problem is amplified in complex RAG systems, agent chains or flows with multiple calls where a small intermediate reasoning error propagates and amplifies. Quality is not just non-deterministic — it can be chaotic in the mathematical sense: small variations in input produce large variations in output.
// 03
Current Solutions: Necessary, Insufficient
The industry has not stood idle. In the last two years a rich ecosystem of techniques has emerged to improve quality in AI systems. I know them well — I have implemented them, evaluated them and, in several cases, suffered their limitations in production.
| Current Components | Strength | Limitation |
|---|---|---|
| Self-RAG | Model evaluates in real-time if it needs additional retrieval and critiques its own response | Reduces hallucinations but increases latency and cost |
| CRAG | Adds document relevance evaluation; falls back to web search if quality is low | Improves edge cases but not the core of the problem |
| Rerank + CoT | Reordering by semantic relevance combined with Chain-of-Thought forces explicit reasoning | Excellent for logical coherence, does not guarantee factual correctness |
| Inject Memory | Persistence of relevant context between sessions | Improves longitudinal consistency but does not protect against real-time errors |
| RAGAS | Measures faithfulness, answer relevancy, context precision and recall | Powerful for offline evaluation — blind to drift in production |
| Arize Phoenix | LLM observability: spans, latency, tokens and evaluations | Excellent monitoring layer — reactive, not predictive or corrective |
Why are they insufficient? Not because they are bad techniques and tools—they are excellent. But we are currently mixing architectural patterns (like Self-RAG and CoT) with observability platforms (like Arize Phoenix). The problem is that neither category proposes a lifecycle paradigm. They are point solutions in the problem space. None defines how they should relate to each other, how the system should learn from its own past errors, or how to integrate all of this into the DNA of the development process.
It is like having antibiotics, vaccines and operating rooms, but without preventive medicine, without clinical records, without epidemiology. The tools exist. The healthcare system, not yet.
// 04
The Proposal: Extending CI/CD to CI/CD/CS
Here I propose something that, as far as I can see, has not been explicitly articulated as a paradigm: the extension of the modern development pipeline with a third continuous dimension.
CI → CD → CS
Continuous Integration · Continuous Delivery · Continuous Supervision
CI and CD are well known. We have lived in them for two decades. CS is different in nature — not as a cosmetic extension, but as a structural response to the non-deterministic nature of AI systems.
Continuous Supervision (CS) is the continuous process of lexical, syntactic and semantic analysis of an AI system’s responses, operating both in real-time during inference and in post-mortem feedback cycles, with the goal of detecting, classifying and learning from the system’s errors throughout its entire production lifecycle.
The key is in the adverb: continuous. This is not about evaluating the system before deployment — we already do that with RAGAS and similar tools. It is about the system developing, over time, memory of its own failures. Post-mortem analysis must be a first-class citizen in the architecture, not an afterthought from the QA team.
// 05
The Three Layers of Continuous Supervision
Lexical Analysis Analysis of the vocabulary and tokens in the response. Detects terminological hallucinations, use of non-existent entities, inconsistencies in proper names, dates or figures. Operates at the token stream level — can be integrated in real-time as a post-processing guardrail.
Syntactic Analysis Analysis of the logical and discursive structure of the response. Do the conclusions follow from the premises? Does the reasoning have formal coherence? Is the chain of arguments valid? This is where much of CoT’s value lives as a supervision tool, not just a generation technique.
Semantic Analysis Analysis of meaning and relevance in context. Is the response faithful to the retrieved documents? Is there drift from the user’s intent? Is the system answering what was asked or what it expected to be asked? This level requires vector comparison and evaluators with access to full context.
// 06
The Continuous Learning Cycle: Where CS Becomes Systemic
Pre-inference Guard → Validation of the incoming query: detection of ambiguous intent, prompt injection, or queries outside the system’s domain. First checkpoint of CS.
Real-time Supervision → Self-RAG, CRAG, Reranking and CoT act as real-time supervisors. In the CS model, these are not isolated techniques — they are sensors at the inference layer that feed the supervision system.
Post-inference Evaluation → RAGAS, Arize Phoenix and internal/external evaluators process the complete response. They generate a multidimensional score that is recorded in the supervision history.
Post-mortem Analysis → The transformation of error logs into Guardrail Layers. This is where the system analyzes error patterns over time to identify temporal drift.
Supervised Feedback Loop → Findings from the post-mortem are not just archived; they feed the next cycle by updating RAG index weights or refining the system prompt to prevent the same error from recurring. The system builds a “semantic memory” of its own failures to prevent future drift.
This cycle transforms supervision from a one-time event into an evolutionary process. A system under CS is not only evaluated — it improves structurally over time as a direct result of its own supervision. That is what distinguishes CS from existing evaluation tools.
// 07
What This Changes for Engineering Teams
Adopting CI/CD/CS implies organizational changes, not just technical ones. It requires AI teams to have explicit supervision roles, that production logs are treated as first-class training data, and that the concept of “technical debt” extends to the domain of supervision debt: the accumulated unanalyzed errors that the system repeats because it never learned from them.
LangChain is right that we cannot simply transfer TDD rigor or quality gates to the LLM world. But the answer is not resignation — it is invention. CI/CD was invented because manual integration methods did not scale. CS must be invented because static evaluation methods do not capture the dynamic and evolving nature of AI systems in production.
“We cannot apply the same quality standards of classical software to AI. But we can — and must — build new standards worthy of the complexity of the problem.”
Non-determinism is not an excuse. It is the specification of the problem. And engineering, at its best, has always transformed problem specifications into the foundations of the solution.
CI/CD/CS. The contract is broken. It’s time to build a new one.
! |
|---|
| CI/CD/CS |
Published February 2026 · AI Systems Quality · Tags: Continuous Supervision, RAG, LLM Engineering, CI/CD, AI Architecture
]]>
