Code & Cluster

Everyone Is Arguing About MCP vs Skills vs Tools vs CLI. They're Missing the Point.

2026-06-15T18:06:38.728-07:00

The agentic AI ecosystem has developed a terminology problem.

Depending on which startup, or conference talk or group you're listening to, the future of AI agents is supposedly built around:

Tools
Skills
MCP
CLI environments

After building and studying agent systems for the last couple of years, I feel these debates are missing the real question. It isn't which abstraction to use.

It's: where should orchestration live? Once you understand that, tools, skills, MCP, and CLI all start to make sense.

To illustrate this, let's imagine we're building a fictional AI Operations Agent.

The agent can:

investigate incidents
analyze logs and metrics
review deployments
create Jira tickets
optimize cloud costs
assist with root cause analysis

As we'll see, the architecture evolves dramatically as the system grows.

The Fundamental Problem

At its core, an LLM cannot actually perform operations work.

It cannot:

query Datadog
inspect GitHub commits
retrieve deployment history
restart services
create Jira tickets

All it can do is reason over information provided in its context.

Everything useful in agentic AI comes from connecting the model to external capabilities.

The question becomes:

How should those capabilities be exposed?

The Tool-Based Agent

Most agent systems start with tools.

A tool is simply a primitive capability exposed to the model.

For our operations agent, the tools might look like:


def query_logs(service_name: str):
    ...

def query_metrics(service_name: str):
    ...

def get_recent_deployments(service_name: str):
    ...

def create_jira_ticket(summary: str):
    ...

The architecture is straightforward.


User
 ↓
LLM decides next action
 ↓
Agent Runtime
 ↓
Tool Execution
 ↓
Results returned to LLM

Suppose a user asks:

Why is checkout-service returning elevated 500 errors?

The interaction loop might look like this:

The LLM determines that logs are required.
The runtime executes query_logs().
Results are returned.
The LLM determines deployment history is also needed.
The runtime executes get_recent_deployments().
Results are returned.
The LLM generates a root cause hypothesis.

This works surprisingly well. Initially.

The Hidden Scaling Problem

The challenge appears when the system grows.

At first the agent has only a handful of tools.


query_logs()
query_metrics()
query_deployments()
create_ticket()

Six months later:


query_logs()
query_metrics()
query_traces()
query_alerts()
query_runbooks()
restart_service()
rollback_deployment()
create_ticket()
query_cost_data()
analyze_utilization()
deploy_hotfix()
...

Now the LLM is no longer merely selecting tools.

It is orchestrating an operational workflow.

It must decide:

which tools to invoke
in what order
when enough information has been gathered
how to recover from failures
what actions should be taken

In other words:

Tool-based systems push orchestration into the model.

This turns out to be both powerful and dangerous.

The problem has three dimensions. First, reliability: too much execution logic is living inside a probabilistic system that may sequence it differently on every run. Second, cost: every invocation pays LLM inference tokens to rediscover a workflow that never changes. Third, latency: each tool-selection step is a round-trip to the model — a waterfall of LLM calls where a single deterministic function would do.

The sequence:

query logs → query metrics → retrieve deployment history → retrieve recent alerts

is completely deterministic. You already know exactly what to fetch. Forcing the LLM to reason its way to that sequence on every invocation is not intelligence — it's waste. Unreliable, slow, and expensive waste.

We are slowly realizing that not all orchestration belongs inside the model. This realization leads directly to skills.

The Two Orchestration Layers

Every agent system, no matter how it's described, is really making decisions at two different layers:

Semantic orchestration — deciding what needs to happen. Which capability is relevant. What the user actually meant. This is inherently ambiguous, and it's the one job LLMs are genuinely good at.

Execution orchestration — deciding how it happens. Ordering, parallelism, retries, error handling, state. This is inherently deterministic, and it's the one job LLMs are bad at — slow, expensive, and inconsistent across runs.

Most of the confusion in this space comes from putting both layers in the same place. Tools, skills, MCP, and CLI environments are best understood as different ways of drawing the line between these two layers — not as competing abstractions. Keep this distinction in mind; every section below maps back to it.

Skills: Coarse-Grained Semantic Capabilities

Skills are often described vaguely, so let's define them carefully. A skill is not simply a larger tool. Nor is it necessarily another autonomous agent.

More precisely: a skill is a subgraph with a semantic boundary. It encapsulates three things the LLM should not have to think about: execution order, parallelism, and retry policy. The planner sees a named capability. The skill owns how that capability runs.

For our operations platform, skills might include: IncidentAnalysisSkill, DeploymentReviewSkill, CapacityPlanningSkill, CostOptimizationSkill.

Tools describe primitive operations. Skills describe business-level capabilities and the execution policy that implements them.

The Planner LLM Still Matters

Without an LLM, mapping human requests to skills becomes extremely difficult.

Consider:

Why did checkout-service start failing after lunch?

Is this:

an incident investigation?
a deployment review?
a performance analysis?

The ambiguity lives in natural language.

The LLM remains extremely useful for:

intent understanding
semantic routing
contextual interpretation

The architecture increasingly becomes:


User Request
 ↓
Planner LLM
 ↓
Skill Selection
 ↓
Skill Execution
 ↓
Results
 ↓
Planner LLM
 ↓
Response

Skills Move Execution Orchestration Out Of The Model

Skills are the clearest example of this split in action: semantic orchestration stays with the planner, execution orchestration moves into the skill. This is the architectural shift many modern systems are making.

For example:


class IncidentAnalysisSkill:

  async def execute(self, service_name):

    # Fetch concurrently — no reason to serialize
    logs, metrics, deployments, alerts = (
      await asyncio.gather(
        query_logs(service_name),
        query_metrics(service_name),
        get_recent_deployments(service_name),
        get_recent_alerts(service_name)
      )
    )

    return {
      "logs": logs,
      "metrics": metrics,
      "deployments": deployments,
      "alerts": alerts
    }

The planner LLM still decides: I need IncidentAnalysisSkill. But once selected, deterministic execution logic takes over — including decisions the LLM would never make well, like whether to fetch in parallel or how to handle a partial failure.

Note: this example is deliberately simplified. In production, skills typically need streaming partial results, per-tool timeouts, and circuit breakers. The point is that all of that complexity lives in the skill — not in the model.

Where MCP Fits

MCP doesn't touch this split at all — it just makes the capabilities on the execution side portable across runtimes. This is where many online discussions become confused.

MCP is not competing with skills.

MCP solves a completely different problem.

The ecosystem currently suffers from massive fragmentation.

Every framework invents:

different tool schemas
different invocation mechanisms
different discovery protocols

MCP attempts to standardize that layer.

The easiest way to think about MCP is:

MCP is trying to become the interoperability layer for AI capabilities.

Closer to OpenAPI for agent ecosystems — a shared interface contract that defines how capabilities are described, discovered, and invoked, without caring what carries the bytes underneath.

What MCP Exposes

Most coverage of MCP focuses only on tools. But the protocol defines three primitives, and the distinction matters:

Tools — callable functions the model can invoke. This is what most people mean when they talk about MCP.

Resources — structured data the model can read: log files, database records, API responses. Resources are pull-based; the model requests them rather than the server pushing them.

Prompts — reusable prompt templates with parameters, surfaced by the server. This lets server authors encode domain knowledge — e.g., a Datadog MCP server can expose a prompt template for structured incident analysis, not just raw metric endpoints.

For our operations agent, a well-designed Datadog MCP server doesn't just expose query_metrics(). It might also expose a resource for the current service topology and a prompt template that structures how the model should reason about an anomaly. That's a qualitatively richer integration than a tool-only mental model suggests.

from mcp.server import Server
server = Server("datadog-server")

@server.tool()
async def query_service_metrics(service):

...

@server.resource("topology://services")
async def get_service_topology():
...

@server.prompt()
async def analyze_anomaly(service, window):
return f"Given metrics for {service} over {window}"

One practical implication: an MCP server built today for Claude can work tomorrow with any other MCP-compatible runtime. That portability is the actual value proposition — not the protocol mechanics themselves.

Why Coding Agents Love CLI Environments

Now let's discuss the most interesting trend in the ecosystem. CLI environments push the line further than either tools or skills: here, the model itself participates in execution orchestration by writing commands, which is exactly why sandboxing becomes non-negotiable.

Coding and operations agents are increasingly bypassing structured tools altogether.

Instead of exposing:


query_git_diff()
run_tests()
deploy_service()

they expose:


git
docker
kubectl
terraform
pytest

The shell becomes the capability layer.

The agent generates commands directly.


kubectl logs checkout-service
git diff
terraform plan

This is a radically different philosophy.

Instead of carefully defining capabilities, you expose a real execution environment.

Why CLI Agents Feel More Powerful

Unix environments already solved composability decades ago. Commands naturally chain together. The shell already provides: composition, state, interoperability, flexibility.

Instead of creating hundreds of tool schemas, the environment itself becomes the API. This is why modern coding agents often feel dramatically more capable than traditional tool-calling agents.

The Tradeoff

Of course, flexibility comes with risk. A structured tool:

restart_service(service_name)

is easier to constrain than:

rm -rf /

This is not a theoretical concern. A CLI agent operating against a production Kubernetes cluster with broad permissions is one hallucinated flag away from an outage. The blast radius of a mistake is bounded only by what the execution environment allows — not by what the tool schema permits. With structured tools, you constrain at the interface. With CLI, you constrain at the environment.

This is why sandboxed execution environments have become infrastructure in their own right. Tools like E2B, Daytona, and Modal provide ephemeral, isolated runtimes specifically built for agent workloads: short-lived containers with scoped filesystem access, network egress controls, and resource limits. The architecture shifts from "give the agent a shell" to "give the agent a shell inside a box it can't break out of." This also changes the observability story — a sandboxed run produces a complete audit trail of every command executed, which structured tool-calling often doesn't.

The deeper implication for our orchestration framework: CLI agents are the one place where the semantic/execution boundary genuinely blurs. The model is no longer just selecting a capability — it is participating in execution orchestration by writing the commands themselves. That is both the source of their power and the reason the safety layer beneath them has to be more sophisticated than a permission flag.Where The Industry Appears To Be Heading

If you step back from the terminology wars, a pattern begins to emerge.

Modern agent systems increasingly look like this:


                 LLM
                  ↓
        Semantic Planning
                  ↓
               Skills
                  ↓
      Deterministic Workflows
            ↓          ↓
         Tools       MCP
            ↓          ↓
      Local Systems  External Systems
                  ↓
            CLI Runtime

Notice what happened.

The LLM is not "the system."

It has become one component inside a larger architecture.

Reasoning remains probabilistic.

Execution becomes deterministic.

Interoperability becomes protocol-driven.

Capabilities become reusable infrastructure.

Final Thoughts

Tools, Skills, MCP, and CLI environments are different answers to where the line falls between semantic and execution orchestration — not competing abstractions.

They exist at different layers of the stack.

Tools expose primitive capabilities.

Skills expose semantic capabilities.

MCP standardizes interoperability.

CLI environments expose raw execution contexts.

The real architectural question is not:

Which abstraction wins?

It is:

Where should orchestration live?

The most successful systems increasingly place semantic orchestration inside LLMs and execution orchestration inside deterministic software.

The future of agentic AI looks less like autonomous chatbots wandering through APIs and more like carefully engineered systems where probabilistic reasoning and deterministic execution work together.

Why Deterministic AI Agents Are The Wrong Goal ?

2026-04-09T17:22:00.000-07:00

1. The Probabilistic Machine

In the rush to build "enterprise-grade" AI agents, many teams are chasing a seductive idea: what if we could make AI fully deterministic — predictable, repeatable, always correct?

It sounds reasonable. That's how traditional software works. But it starts with a fundamental misunderstanding of what these systems actually are.

Large Language Models are not knowledge databases or rules engines. At their core, they are statistical machines — predicting the probability of the next token in a sequence, over and over, until a response takes shape. When you craft a precise prompt or inject carefully curated context, you are not overriding that mechanism. You are nudging it. Good context narrows the probability distribution and raises the likelihood of a useful answer. But it doesn't change the fundamental nature of the system. The model remains probabilistic. You can engineer the inputs to the ceiling — you cannot engineer your way out of uncertainty.

2. The Enterprise Trap

This creates a real tension — because the properties that make LLMs powerful are almost perfectly opposed to what enterprises want from software.

Enterprises don't want suggestions. They want answers. They don't want "usually correct" — they want auditable, repeatable, defensible outputs. When something goes wrong, someone needs to explain exactly why the system did what it did.

A probabilistic system doesn't give you that cleanly. And so the instinct is to reach for determinism — to constrain the model until it behaves like a well-behaved service. Temperature to zero. Rigid output schemas. Exhaustive prompt engineering. Rules stacked on rules.

This isn't irrational. Repeatability, auditability, compliance — these are legitimate needs. But chasing determinism in the model itself is solving the wrong problem. You end up with a system that is neither reliably deterministic nor making full use of what the model can actually do. You've neutered the intelligence without gaining the guarantees you wanted.

The goal shouldn't be a deterministic model. It should be a reliable system. Those are not the same thing — and confusing them is where most enterprise AI projects go wrong.

3. What Actually Works: Hybrid Architecture

The real breakthrough of modern AI systems wasn't just model quality. It was a design philosophy.

Tools like ChatGPT and Claude succeeded not because they eliminated uncertainty, but because they made uncertainty part of the interaction. They don't say "here is the correct answer." They say "here's a strong answer — want me to refine it?" That subtle shift changes everything. The human stays in the loop. The model doesn't pretend to be infallible. And because of that, users actually trust the output enough to act on it.

This points toward the pattern that wins in production: a hybrid architecture where determinism and intelligence each live where they belong.

The structure looks like this. A deterministic shell handles everything that must be correct and repeatable — workflows, APIs, validation rules, policy enforcement. A probabilistic core handles everything that requires reasoning — summarization, analysis, decision support, generation. And control points sit between them — confidence thresholds, structured outputs, human approvals — managing the boundary between the two.

The LLM proposes. The system validates. The human, when needed, decides.

Determinism doesn't disappear. It moves to where it actually belongs.

4. Vestra: A Concrete Example

To make this real, consider Vestra — an AI-powered investment analysis agent I've been building.

A naive approach would let the LLM do everything: fetch market data, apply financial rules, generate investment decisions. That system would fail badly. It would hallucinate stock picks, misinterpret regulations, produce outputs that couldn't survive a compliance review.

So Vestra is deliberately split into three layers.

The deterministic shell ingests user portfolio data, computes financial metrics, and enforces business rules. This is traditional code. It behaves identically every time.

The probabilistic core — the LLM — receives only clean, verified data from that shell. It doesn't touch raw market feeds. It reasons over what it's given: evaluating the portfolio against the user's goals, time horizon, and macro context, and generating high-level strategic insights. Rebalance allocations. Increase international exposure. Add diversified index funds.

The control points manage what happens next. The LLM is constrained to return structured JSON, so downstream code can validate and process the output safely. And Vestra never acts autonomously — it presents recommendations to the user with clear disclosure that they are AI-generated. The user accepts, rejects, or refines them, and is encouraged to seek professional advice for significant decisions.

The LLM doesn't fetch data. It doesn't execute trades. It doesn't make final calls.

Determinism lives in the code. Intelligence lives in the model. Control lives with the user.

5. The Broader Principle

There's a persistent belief that keeping humans in the loop is a temporary crutch — something to tolerate until the models get good enough to go fully autonomous. That belief keeps getting disproven in production.

Human involvement isn't a limitation to engineer around. It's a design pattern that makes systems more accurate and more trusted. Users who can steer, refine, and push back on AI outputs consistently get better results than users handed a black-box decision. The interaction is where the value compounds.

That said, not every system should work this way. Where the stakes are high but the timeline allows deliberation — investment analysis, legal drafting, code review, customer support — interactivity is a feature, not a compromise. The human brings judgment the model lacks; the model brings breadth the human can't match.

But in real-time fraud detection, high-throughput automation, or millisecond trading systems, that loop collapses. There's no time for human approval. These systems need hard rules and hard thresholds, with AI informing the design rather than driving the execution.

The mistake isn't choosing the wrong architecture. It's assuming one architecture fits everything.

6. The Real Opportunity

A year or two ago, the bold prediction was that autonomous agents would replace entire workflows by now. That hasn't happened — not because the models aren't capable enough, but because the systems around them weren't designed for it. Fully autonomous agents keep failing in the same ways: confidently wrong outputs, no graceful degradation, users who don't trust them enough to act on what they produce.

What's actually working is quieter and less glamorous. Systems that surface their own uncertainty. Systems where the human is a genuine collaborator, not an afterthought. Systems where the experience is designed so carefully that raw model capability almost becomes secondary.

The opportunity isn't in building perfect AI agents. It's in building systems that help humans navigate imperfection — that make strong suggestions, explain their reasoning, accept feedback, and improve through iteration.

Progress doesn't come from eliminating uncertainty. It comes from designing systems that help people work with it.

Chasing deterministic AI agents may feel like building the future. But the real future belongs to systems that are interactive, collaborative, and intelligently imperfect.

And that's not a limitation. That's the breakthrough.

AI Context Explained: The Real Engineering Behind Modern AI Systems

2026-03-07T08:48:00.000-08:00

Most discussions about large language models focus on prompts — how to phrase instructions to get better responses. But in real AI systems, prompts are only a small part of the story.

What actually determines the quality of an AI system is context: the information available to the model when it generates a response. This includes prompts, conversation history, retrieved documents, tool outputs, and sometimes structured application state. Designing how this information is assembled and provided to the model is what many engineers now call context engineering.

Providing the right context to the LLM is the only reliable way to get accurate, production-grade answers. In this blog, I explore what context actually is, the hidden dangers of massive context windows, and how it should be used in Agentic AI.

Context

Context refers to all the information that is not in the user's immediate question, but is required to help the LLM generate a relevant, highly specific answer. It is the data that gives the LLM situational awareness.

Consider this basic prompt:

Prompt: What is a good stock mutual fund to invest in? Response (Abbreviated): 1. T. Rowe Price Global Technology Fund (PRGTX) 2. Wasatch Ultra Growth Fund (WAMCX).

For many investors, both of these are far too aggressive, high-risk, and expensive. Let's change the prompt slightly to inject some context:

Prompt: What is a good stock mutual fund to invest in? I am 56 years old, nearing retirement. I prefer low-risk, low-cost, highly diversified funds. Response (Abbreviated): 1. Vanguard Target Retirement 2035 Fund (VTTHX) 2. Fidelity ZERO Total Market Index Fund (FZROX) 3. Vanguard Total Bond Market Index Fund (VBTLX).

This response is drastically different and entirely appropriate for a conservative investor. The phrase "56 years old, nearing retirement. I prefer low-risk, low-cost, highly diversified funds" is the context. Without it, asking the LLM the same question multiple times will yield scattered, generic, or even dangerous financial advice.

How is Context Passed to the Model?

Whether you use native provider APIs (OpenAI, Google) or orchestration frameworks like LangChain, context is not a separate magical parameter. It is embedded directly into the input messages.

A raw API call looks like this:

# Python
client.responses.create(
    model="gpt-5.2",
    messages=[ 
        {"role": "system", "content": "You are a financial advisor."}, # system prompt
        {"role": "user", "content": "What is a good fund to invest in?"} # User prompt or query
        {"role": "user", "content": "I am 56 and prefer low risk."}, # Context    ],
    temperature=0.0
)

Everything the LLM knows is stuffed into that messages array. In an agentic system, it is all about getting the right information into that array at the right time.

Context generally falls into three categories:

Static: Data that rarely changes (e.g., "User is a male, NY Yankees fan, foodie").
Dynamic: Data that evolves as the agent runs and interacts with tools (e.g., the results of a real-time stock price lookup).
Long-Lived: Data that spans across multiple sessions or days (e.g., "User already rejected the Vanguard recommendation yesterday").

In practice, building AI systems is often less about “prompt engineering” and more about deciding what information should be included in the model’s context at the moment of inference.

The data comes from a variety of sources. Cache would have the most recent data. The databases have the system of record. The log have events as they occurs. Context engineering involves getting the right data from the right place at the right time.

Note that context can go stale. If you feed in stale context, you get less the accurate answers. Keeping it current is part of the engineering.

The Illusion of the Infinite Context Window

The context window refers to how much of your context the LLM retains for a conversation. The more it can remember, the better. Right ?

LLM providers aggressively advertise their context window sizes. Bigger appears better, but that is a dangerous trap for developers.

The context window simply represents the hard cap on how much text an LLM can "see" at once. Look at the landscape in early 2026:

Meta Llama 4 Scout: ~10 Million tokens
Gemini 3 Pro: ~1.0M - 2.0M tokens
OpenAI GPT-5.2: ~400,000 tokens
Claude 4.5 Sonnet: ~1.0M tokens
DeepSeek-R1: ~164,000 tokens

However, the context window is a model attribute, not an agent capability. Research in 2025 and 2026 has consistently proven that models severely degrade well before hitting their upper limits. This phenomenon is known as Context Rot.

Just because a model can accept 1 million tokens (about 8 full novels) doesn't mean it pays equal attention to all of them. Studies show that when a context window passes 50% capacity, models begin to heavily favor tokens at the very beginning or the very end of the prompt, completely ignoring critical constraints buried in the middle.

The industry is now focusing on the Maximum Effective Context Window (MECW). A model might advertise 1 million tokens, but its MECW—the point where accuracy actually drops off a cliff—might be only 130k tokens.

The Agent Loop

Because of Context Rot, you cannot just dump an entire database into the LLM and expect it to figure things out. This is why we build Agents.

An LLM is a stateless text predictor. An Agent is a software loop that uses the LLM as a reasoning engine to manage its own context. Agents operate in a continuous cycle: Observe → Think → Act.

Imagine building an AI-based investment analysis product. The agent doesn't just ask the LLM one massive question. It loops:

Observe: The user asks, "Should I adjust my portfolio for the upcoming rate cuts?"
Think (LLM): The model realizes it lacks context. It outputs a tool-call: get_user_portfolio() and get_risk_tolerance().
Act (Code): Agent code queries a PostgreSQL database to fetch the financial profile.
Update Context: It appends appends only the relevant portfolio metrics into the messages array.
Loop: The agent sends this newly enriched, highly specific context back to the LLM to generate the final advice.

In this loop, the context is actively mutating. The agent is continuously pruning the messages array, summarizing old turns, and injecting fresh tool outputs to keep the token count well within the Maximum Effective Context Window.

Prompt Engineering vs. Context Engineering

If prompt engineering is about how you ask the question, context engineering is about giving the model a little more data before it attempts to answer.

To use an operating system analogy: The LLM is the CPU, the Prompt is the executable command, and the Context Window is the RAM and context is the data in RAM.

Prompt Engineering is writing a better command. It is user-facing, static, and brittle.
Context Engineering is managing the data in RAM. It is developer-facing, dynamic, and systemic.

As we move toward enterprise-grade AI, prompts are no longer enough. Context engineering involves building the infrastructure that feeds the model. It encompasses Retrieval-Augmented Generation (RAG) to find specific documents, Episodic Memory Graphs to track user decisions over time, and Context Pruning to prevent token overflow.

The Frontier: Context Graphs

While context engineering today is mostly about managing lists of messages, the future of enterprise AI lies in Context Graphs. Current LLM context is linear—a flat, chronological scroll of "User said X, Agent did Y." This works for chat, but it fails for complex enterprise workflows. Real-world business data isn't a timeline; it's a web of relationships.

Enter the Context Graph. Instead of dumping raw logs into the window, advanced agents now build and maintain a dynamic graph structure. Nodes represent entities (User, File, Decision, Error). Edges represent causality or relationships (e.g., User Upload caused Error 500, which triggered Retry Logic).

This structure transforms the context from a "temporary scratchpad" into an organizational brain. If a human auditor later asks, "Why did the agent reject this loan application?", a linear log forces the LLM to re-read thousands of lines of text to guess the reason. A Context Graph simply traverses the edge: Loan Application -> {rejected\_because}} -> Risk Score > 80.

For enterprise applications, this is the missing link. It allows agents to reason across disconnected data points (e.g., linking a Slack message from Tuesday to a Code Commit on Friday) without needing a massive, expensive context window to hold all the noise in between.

Conclusion

A perfectly engineered prompt might get you a clever answer once. But a well-engineered context pipeline ensures your Agent gets the accurate answer securely, cost-effectively, and consistently, every single time.

Popular LLM models are advanced and sophisticated. But everyone has access to them. You have no advantage because you an LLM. Your advantage and intellectual property is in how you manage and feed context to the LLM. Though not new, how you collect, store and retrieve the data that forms the context is the real engineering.

What is (Agentic) AI Memory ?

2026-02-09T10:41:00.000-08:00

I have seen a lot of posts on X and LinkedIn on the importance of Agentic AI memory. What exactly is it ? Is it just another name for RAG ? Why is it different from any other application memory ? In this blog, I try to answer these questions.

What do people mean when they say "AI Memory" ?

Most production LLM interactions rely on external memory systems. Everything called “memory” today is mostly external.

At their core LLMs are stateless functions. You make a request with a prompt and some context data and it provides you with a response

In real systems, AI memory usually means:

Storing past interactions, user preferences, decisions, goals, or facts.
Retrieving relevant parts later
Feeding a compressed version back into the prompt

So yes — at its core:

Memory = save → retrieve → summarize → inject into context

Nothing magical. But is that all ? seems just like a regular cache ? Read on.

Is this just RAG (Retrieval Augmented Generation) ?

They are related but not the same.

RAG (Retrieval Augmented Generation)

Purpose:

Bring external knowledge into the LLM
Docs, PDFs, financial data, code, policies

Typical traits:

retrieval is stateless per query
Large text chunks
Query-driven retrieval
“What additional data can we provide to LLM to help answer this question?”

Agent / User Memory

Purpose:

Maintain continuity
Personalization
Learning user intent and preferences over time

Typical traits:

Long-lived
Highly structured
Small, distilled facts
“What can I provide to LLM so it remembers this user?”

Think of it this way:

They often use can use the same retrieval tools, but they serve different roles.

Where is the memory ?

Option 1: Agent process memory

Any suitable data structure like a HashMap.
Suitable for cases where the Agent loop is short and no persistence is needed.

Option 2: Redis /Cache

Suitable for session info, recent conversation history, tool results cache, temporary state.

Option 3: PostgreSQL/RDBMS

Suitable when you need durability, auditability, explainability.

Option 4: Vector databases

Suitable for semantic search.

Option 5: AI memory tools

Such as LangGraph memory, LlamaIndex memory, Memgpt. They try to make it easier for agents to store and retrieve.

Here is example of data that might be stored in memory:

{

"user_id": "123",

"fact": "User prefers concise python code",

"source": "conversation_turn_5",

"timestamp": "2026-02-09"

}

The mental model for AI memory

Short term memory

This is about recent interactions. It is data relevant to the current topic being discussed. For example, the user prefers conservative answers.

Long term memory

This is stored externally, perhaps even to persistent storage. It is retrieved and inserted into context selectively. For example, the user is a vegetarian or the user's risk tolerance is low.

Memory and the LLM

The LLM takes as input only messages. Agent has to read the data from memory and insert it into the text message. This is what they refer to as context.

You do not want add large amount of arbitrary data as context because:

text is converted to token and token cost spirals
LLM attention degrades with noise
Latency increases
Reasoning quality declines

Real Agentic Memory

At the start of the blog, I asked "is this just a regular cache ?".

To be useful in the agentic way, what is stored in the memory needs to evolve. Older or maybe irrelevant data in the memory needed to be "forgotten" or evicted based on intelligence (not standard algorithms like FIFO, LIFO etc). Updates and evictions need to happen based on recent interactions. If the historical information is too long and should not be evicted, it might need to be compressed.

Agentic systems require more dynamic memory evolution than typical CRUD applications. In the case of long running agents, the quality of data in the memory has to get better with interactions over time.

How exactly that can be implemented is beyond the scope of this blog and could be a topic for a future one.

Considerations

Memory != Raw History

Bad Use : Here are the last 47 conversations ......

Better Use : We were talking about my retirement goals with this income and number of years to retire.

Summarize and abstract to extract intelligence - as opposed to dumping large quantity of data.

In conclusion

AI memory is structured state, sometimes summarized that is retrieved when needed and included as LLM input as "context".

Conceptually, it is similar to RAG but they apply to different use cases.

Better and smaller contexts beat large contexts and large memory.

Agentic AI Memory adds value only when

The system changes behavior ( for the better ) because of it
It produces better response, explanations, reasonings
It saves time

These ideas are not purely theoretical. While building Vestra — an AI agent focused on personal financial planning and modeling — I’ve had to think deeply about what should be remembered, what should be abstracted, and what should be discarded. In financial reasoning especially, raw history is far less useful than structured, evolving state.

But yes, Agentic memory will be different than what we know as memory in regular apps — in the ways it is updated, evicted, and retrieved.

Unique ID generation

2026-01-15T12:27:00.000-08:00

Unique ID generation is a deceptively simple task in application development. While it can seem trivial in a monolithic system, getting it right in a distributed system presents a few challenges. In this blog, I explore the commonly used ID generation techniques, where they fail and how to improve on them.

Why are unique Ids needed in applications ?

They are needed to prevent duplicates. Say the system creates users or products or any other entity. You could use name at the key for uniqueness constraint. But there could be two users or products with the same name. For users, email is an option but that might not be available for other kinds of entities.

Sometimes you are consuming messages from other systems over platforms like Kafka. Processing the same message multiple times can lead to errors. Duplicate delivery can happen for a variety of reasons that can be out of control of the consumer. Senders therefore include a uniqueId with the message so that consumer can ignore ids already processed (Idempotency) .

They can be useful for ordering. Ordering is useful to determine which event happened before others. It can be useful if you want to query for the most recent events or ignore older events.

What should the size and form of an id be ?

Should it be a number or should it be alphanumeric? Should it be 32 bit, 64 bit or larger.

With 32 bit the maximum numeric id you can generate is ~ 4 billion. Sufficient for most systems but not enough if your product is internet scale. With 64 bit, you can generate id into the hundreds of trillions.

But size is not the only issue. It is not the case that ids will generated in sequence in one place one after the other. In any system of even moderate scale, the unique ids will need to be generated from multiple nodes.

On the topic of form, numerics ids are generally preferred as the take up less storage and can be easily ordered and indexed.

In the rest of the blog, I go over some unique id generation techniques I have come across.

ID GenerationTechniques

1. Auto increment feature of the database

If your services uses a database, this becomes an obvious choice as every database supports auto increment.

With postgresql, you would set up the table as

CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT not null);

With mysql,

CREATE TABLE users (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(255) not null);

Every insert into the table generates an id in the id column.

INSERT INTO users(name) VALUES ('JOHN') ;
INSERT INTO users(name) VALUES ('MARY') ;

id | name |

-----+---------

1 | JOHN |

2 | MARY |

While this is simple to setup and use, it is appropriate for simple crud applications with low usage. The disadvantages are

It requires an insert to the db to get an id
If the insert fails for other reasons, you can have gaps
For distributed applications, it is a single point of failure

The single point of failure issue can be solved setting up multiple databases in a multi master arrangement as show below. In fact, here we are handing out ids in batches of hundred to reduce the load on the database.

2. UUID

These are 128 bit alpha numeric strings that are quite easy to generate and can be suitable for ids because they are unique. An example UUID is e3d6d0e9-b99f-4704-b672-b7f3cdaf5618

The advantages of UUID are:

easy to generate. Just add UUID generator to the service.
low probability of collision
Each replica will generate unique id

Disadvantages:

128 bits for each id will eventually take up space
No ordering. Ids are random

UUID v1 has a time component and could be sortable but is not unique enough. UUID v4 is completely random. UUID v7 has a 48 bit timestamp followed by 78 bits of random data. So if you are using UUID, you want to use v7 or higher.

3. Flickr ticket server

The Flickr ticket server is an improvement of the auto increment feature. In the example we have in 1, the id is tied to the users table. You get a new id only when you insert into the user table. if you needed unique ids for another entity, you would need to add autoincrement column to that table. But what if you needed unique ids in increasing order across tables or tables as would be the case in a distributed system ?

We could create a generic table

CREATE TABLE idgenerator (id SERIAL PRIMARY KEY);

This can work but it would keep accumulating rows of ids which would also be store elsewhere.

What they did at flicker was this

create table ticketserver (id int auto_increment primary key, stub char(1));

When they need an id , the do

replace into tickerserver(stub) values ('a') ;
select LAST_INSERT_ID();

This table always have only 1 row, because everytime you need an id, you are replacing the previous row. The above code is Mysql specific.

For SQL that will work on postgresql you can do it little differently

create table ticketserver( stub char(1) primary key, id bigint notnull default 0)

INSERT into ticketserver(stub,id)
VALUES('a', COALESE((SELECT ID from ticketserver where stub = 'a'),0)
ON CONFLICT(stub) DO UPDATE
SET id = ticketserver.id + 1
returning id

4. Snow flake twitter approach

This Twitter approach that does not rely on the database or any third party service. It can be generated in code just following the specification.

It is a 64 bit id.

The left most bit is a sign bit for future use.
The next 41 bits are used for the timestamp. Current time in ms since the epoch Nov 4 2010 1:42:54 UTC. But you can use any epoch.
The next 5 bits are for datacenter id - that gives you 2 power 5 or 32 data centers.
The next 5 bits are for machine id. 32 machines per datacenter.
The last 12 bits are a sequence id - giving you 2 power 12 = 4096 ids per ms ( per datacenter per machine)

The value of 2 power 41 - 1 is 2,199,023,255,551. With 41 bits for timestamp , you get a lot ids. This will last almost 70 years from epoch.

You can change things to reduce the size from 64 bits if needed. You may not need a datacenter id or you can decide to use fewer bits for the timestamp.

The advantages of this approach are

Decentralized generation. No dependency on DB. Lower latency
Time based ordering
Higher throughput
Datacenter id and machine id can help in debugging

The disadvantages are

clocks need to be synchronized
datacenters/machines need unique id
epoch needs to be choose wisely

Other considerations

Clock drift: In distributed systems where timestamp is part of the generated ID, you need to be aware of clock drift and take steps to mitigate it.

Sequential vs Monotonic: Most the time IDs need to be only monotonic, that is always increasing. They not need to be sequential.

Batching: If strictly increasing is not a requirement and you are using the database, you can reduce the load on the database by having the database handout ids to services in batches ( eg. 100 at a time)

Summary

ID generation evolves with your systems scale. When you are starting out, it is normal to keep things simple and go with auto increment. But sooner or later you will need to scale (good problem to have). But the Flicker and Twitter methods are solid. I personally like the Twitter approach as it has no dependency on the database. It offers an excellent balance of decentralization, ordering and efficiency but requires clock synchronization. Whatever approach you choose, you need to ensure that aligns with your systems consistency requirements, scaling needs and tolerance for operational complexity.

Reference

1. Flickr Ticket Server

2. Twitter Snowflake

Review : Facebook Memcache Paper

2025-11-30T17:34:00.000-08:00

Introduction

Facebook (Meta) application handles billions of requests per second. The responses are built using thousands of items that needs to retrieved at low latencies to ensure good user experience. Low latency is achieved by retrieving the items from a cache. Once their business grew, a single node cache like Memcached would obviously not be able to handle the load. This paper is about how they solved the problem by building a distributed cache on top of Memcached.

In 2025, some might say the paper is a little dated ( 2013). But I think it is still interesting for several reasons. It is one of the early papers from a time when cloud and distributed systems exploded. I see it in the same category as the Dynamo paper from Amazon. While better technology is available today, this paper teaches important concepts. More importantly the paper shows how to take technology available and make more out it. In this case, they took a single node cache and built a distributed cache on top of it.

This is my summary of the paper Scaling Memcache at Facebook.

Requirements at Facebook

- Allow near real time communication
- aggregate content on the fly
- access and update popular shared content
- scale to millions of user requests per second

The starting point was single node Memcached servers.
The target was a general purpose memory distributed key value store - called Memcache, that would be used by a variety of application use cases.

In the paper and in this blog, Memcached refers to the popular open source in memory key value store and Memcache is the distributed cached that Facebook built.

Observations:

Read volumes are several orders of magnitude higher than write volumes.
Data fetched from multiple sources - HDFS, MySql etc
Memcached supports simple primitives - set, get, delete

Details

The diagram below shows the Memcache architecture.

Memcache is a demand filled look aside cache. There can be thousands of memcached servers within a memcache cluster.

When an application needs to read data, it tries to get it from memcache. If not found in Memcache, it gets the data from the original source and populates Memcache.

When the application needs to write data, it writes to original source and the key in Memcache is invalidated.

Wide fan out: When the front end servers scale, the backend cache needs to scale too.

Items are distributed across Memcached servers using consistent hashing.

To handle a request, front end server might need to get data from many Memcached servers.
Front end servers use a Memcache client to talk to memcached servers. Client is either a library or a proxy called mcrouter. Memcached servers do not communicate with each other.

Invalidation of the Memcached key is done by code running on the storage. It is not done by the client.

Communication

UDP is used for get requests. (Surprised ? clearly an optimization). Direct from client in webserver to Memcached. Dropped requests are treated as a cache miss.
TCP via mcrouter used for set and delete requests. Using mcrouter helps manage connections to storage.
Client implements flow control to limit load on backend components

Leases

Leases were implemented to address stale sets and thundering herd. Stale sets are caused by updating the cache with invalid values. Requiring a lease lets the system check that update is valid.
Thundering herd happens when there is heavy read write activity on the same key at the same time. By handing out leases only every so often say 10 sec, they slow things down.

Memcache pools

This is a general purpose caching layer used by different applications, different workloads with different requirements. To avoid interference, the clusters servers are partitioned into pools. For example, a pool for keys that are accessed frequently and cannot tolerate cache miss. Another pool for infrequently accessed keys. Each pool can scaled separately depending on requirement.

Failures

For small failures, the requests are directed to a set dedicated backup servers called gutters. When a large number of servers in the cluster down, the entire cluster is considered offline and traffic is routed to another cluster.

Topology

A frontend cluster is a group of webservers with Memcache.
A region is frontend cluster plus storage.
This keeps the failure domains small.
Storage is the final source of truth. Use mcsql and mcrouter to invalidate cache.

When going across data centers and across geo regions, the storage master in one region replicates to replicas in other regions. On an update, when the Memcached needs to be invalidated by the storage, it is not a problem if the update is in the master region. But if the update is in the replica region, then the read after write might read a stale data as the replica might not have caught up. In replica regions, markers are used to ensure that only data in sync with the master is read.

Summary

In summary, this paper show how Facebook took a single node Memcached and scaled it to its growing needs. No new fundamental theories are applied or discussed. But this demonstrated innovation and trade offs in engineering to scale and grow in product in production that needs to scale to meet user demand.

Key point is that separation cache and storage allowed each to be scaled separately. Kept focus on monitoring, debugging and operational efficiency. Gradual rollout and rollback of features kept a running system running.

Some might say that Memcache is a hack. But some loss in architectural purity is worth it -- if your users and business stay happy.

Memcache and Facebook were developed together with application and system programmers working together to evolve and scale the system. This does not happen when teams work in isolation.

Data Storage For Analytics And AI

2025-10-05T16:09:00.000-07:00

For a small or medium sized company storing all the data in relational database like Postgresql or MySQL is sufficient. Perhaps if analytics is needed they might also use a columnar store, more like a data warehouse.

What if you have large amounts of unstructured data as well ? May be logs from your e-commerce site, emails, support logs etc. Those need to be queried, aggregated, summarized and reported as well.

If your business grows to handle large volumes of unstructured data—maybe logs from your e-commerce site, emails, support tickets, images, or customer audio—storing everything in a single RDBMS becomes impossible. These new data types require specialized architectures designed for scale, flexibility, and advanced analytics (like Machine Learning and Generative AI).

Here is a guide to the key data storage paradigms you will encounter:

This is a brief introduction to the options.

1. Relational Database Management Systems (RDBMS)

This needs no introduction.

Primary Use Case: Online Transaction Processing (OLTP). Applications requiring fast, frequent reads and writes, and ACID compliance.

Data Structure: Data is modeled at normalized rows and columns. Explicit relationships are enforced using foreign keys. In most cases storage is implemented as a B+ tree.

Schema Approach: The schema must be defined and enforced before data can be written.

Examples: PostgreSQL (Open Source), MySQL (Open Source/Commercial), Oracle, Microsoft SQL Server.

2. Data Warehouse (DW)

Primary Use Case: Online Analytical Processing (OLAP). Business Intelligence (BI), historical reporting, and generating complex aggregated reports across years of data.

Data Structure: Columnar data store. Data often denormalized into Star or Snowflake schemas to optimize large, analytical JOIN queries.

Schema Approach: Schema-on-Write: Data is cleaned, transformed, and structured via ETL/ELT pipelines before loading.

Examples: Snowflake, Google BigQuery, Amazon Redshift, Apache Pinot, Apache Druid, ClickHouse

3. Data Lake

Primary Use Case: Storing all data (raw and processed) at massive scale for Data Science, Machine Learning (ML), and exploratory analytics.

Data Structure: Stores data in its native, raw format—structured, semi-structured (JSON, XML), and unstructured (logs, images, audio).

Schema Approach: Schema-on-Read: Structure is applied dynamically by the query engine when the data is read. This offers maximum flexibility.

Examples: Amazon S3 (storage), Azure Data Lake Storage (ADLS), Apache Hadoop, Delta Lake, Apache Hudi

4. Data Lakehouse

Primary Use Case: Unifying the scale and flexibility of a Data Lake with the reliability and performance of a Data Warehouse.

Data Structure: Hybrid: Stores all raw data in the lake but adds a metadata and transaction layer (e.g., Delta Lake) to enforce quality and provide table-like features.

Schema Approach: Hybrid: Allows Schema-on-Read for raw ingestion while enforcing Schema Enforcement and ACID transactions for curated tables.

Example: Databricks, Apache Iceberg

5. NOSQL Database

Primary Use Case: High-volume, dynamic, operational use cases where schemas change frequently and extreme horizontal scaling is needed (e.g., user profiles, content management).

Data Structure: Varies (Document, Key-Value, Graph, Wide-Column). Data is often stored as flexible records or objects without strict relationships.

Schema Approach:Schema-less or Dynamic Schema: Structure can evolve on a per-document basis without downtime.

Example: MongoDB (Document), Redis (Key-Value/Cache), Apache Cassandra (Wide-Column), Neo4j (Graph).

6. Vector Database

Given the rise of LLMs and Generative AI, this is one more specialized option critical for working with unstructured data:

This is designed to store and index vector embeddings—numerical representations of unstructured data (text, images, audio) created by AI models. They allow for similarity search (finding "like" data) rather than exact keyword matches.

Primary Use Case: Retrieval-Augmented Generation (RAG), semantic search, recommendation engines, and high-dimensional ML applications.

Example: Pinecone (Commercial), Weaviate (Open Source/Commercial), Qdrant.

Summary

All of these options, from the structured RDBMS to the fluid Vector DB, combine to form a modern enterprise data architecture.

In essence, the modern enterprise no longer relies on a single data storage solution. The journey usually starts with the RDBMS for transactional integrity, moves to the Data Warehouse for structured BI, and expands into the Data Lake to capture all raw, unstructured data necessary for Machine Learning and discovery.

The Data Lakehouse is the cutting-edge step, unifying these functions by bringing governance and performance directly to the lake. Vector Databases bridge the gap between unstructured data and the world of Generative AI.

Understanding the specialized role of each platform is the first and most critical step in designing a future-proof data strategy that extracts maximum value from every piece of information your business creates.

Note that there is some overlap between the categories. For example Postgresql supports JSONB and vector storage, making it useful for some NoSql and AI use cases. Some products that started of as data lakes added features to become lakehouses.

What Does Adding AI To Your Product Even Mean?

2025-09-13T19:09:00.000-07:00

Introduction

I have been asked this question multiple times: My management sent out a directive to all teams to add AI to the product. But I have no idea what that means ?

In this blog I discuss what adding AI actually entails, moving beyond the hype to practical applications and what are some things you might try.

At its core, adding AI to a product means using an AI model, either the more popular large language model (LLM) or a traditional ML model to either

predict answers
generate new data - text, image , audio etc

The effect of that is it enable the product to

do a better job of responding to queries
automate repetitive tasks
personalize responses
extract insights
Reduce manual labor

It's about making your product smarter, more efficient, and more valuable by giving it capabilities it didn't have before.

Any domain where there is a huge domain of published knowledge (programming, healthcare) or vast quantities of data (e-commerce, financial services, health, manufacturing etc), too large for the human brain to comprehend, AI has a place and will outperform what we currently do.

So how do you go about adding AI ?

Thanks to social media, AI has developed the aura of being super-complicated. But if reality, if you use off the shelf models, it is not that hard. Training models is hard. But 97% of us, will never have to do it. Below is a simple 5 step approach to adding AI to your system.

1. Requirements

It is really important that you nail down the requirement before proceeding any further. What task is being automated ? What questions are you attempting to answer ?

The AI solution will need to evaluated against this requirement. Not once or twice but on a continuous basis.

2. Model

Pick a model.

The recent explosion of interest in AI is largely due to Large Language Models (LLMs) like ChatGPT. At its core, the LLM is a text prediction engine. Give it some text and it will give you text that likely to follow.

But beyond text generation, LLMs have been been trained with a lot of published digital data and they retain associations between text. On top of it, they are trained with real world examples of questions and answers. For example, the reason they do such a good job at generating "programming code" is because they are trained with real source code from github repositories.

What model to use ?

The choices are:

Commercial LLMs like ChatGpt, Claude, Gemini etc
Open source LLMs like Llama, Mistral, DeepSeek etc
Traditional ML models

Choosing the right model can make a difference to the results. There might be a model specially tuned for your problem domain.

Cost, latency and accuracy are some parameters that are used to evaluate models.

3. Agent

Develop one or more agents.

Agent is the modern evolution of a service. Agent is the glue that ties the AI model to the rest of your system.

The Agent is the orchestration layer that:

Accepts requests either from a UI or another service
Makes requests to the model on behalf of your system
Makes multiple API calls to systems to fetch data
May search the internet
May save state to a database at various times
In the end, returns a response or start some process to finish a task

It is unlikely that you will develop a model. But it is very likely that you will develop one or more agents.

4. Data pipeline

Bring your data.

A generic AI model can only do so much. Even without additional training, just adding your data to the prompts can yield better results.

The data pipeline is what makes the data in your databases, logs, ticket systems, github, Jira etc available to the models and agents.

get the data from source
clean it
format it
transform it
use it in either prompts or to further train the model

5. Monitoring

Monitor, tune, refine.

Lastly you need to continuously monitor results to ensure quality. LLMs are known to hallucinate and even drift. When the results are not good, your will try tweaking the prompt data, model parameters among other things.

Now let us seem how these concepts translate into some very simple real-world applications across different industries.

Examples

1. Healthcare: Enhancing Diagnostics and Patient Experience

Adding AI can mean:

Personalized Treatment Pathways: An AI Agent can analyze vast amounts of research papers, clinical trial data, and individual patient responses to suggest the most effective treatment plan tailored to a specific patient's profile.
- Example: For a person with high cholesterol, an AI agent can come up with a personalized diet and exercise plan.

2. Finance: Personalized Investing

Adding AI could mean:

Personalized Financial Advice: Here, an AI Agent can serve as a "advisor" to offer highly tailored investment portfolios and financial planning advice.
- Example: A banking app's AI agent uses an LLM to understand your financial goals and then uses its "tools" to connect to your accounts, pull real-time market data, and recommend trades on your behalf. It can then use its LLM to explain in simple terms why it made a specific trade or rebalanced your portfolio.

3. E-commerce: Customer Experience

Adding AI could mean:

Personalized shopping: AI models can find the right product at the right price with the right characteristics for user requirement
- Example: Instead of me shopping and comparing for hours, AI does it for me and makes a recommendation on the final product to purchase.

In Conclusion

Adding AI to your product to make it better means using the proven power of AI models

To better answer customer request with insights
To automate repetitive time consuming task
To make predictions that were hard earlier
To gain insights into vast bodies of knowledge

The tools are there. But to get results you need discipline, patience and process.

Start small. Focus on one specific business problem you want to solve, and build from there.

CRDT Tutorial: Conflict Free Replication Data Types

2025-09-06T06:50:00.000-07:00

Have you ever wondered how Google docs, Figma, Notion provide real time collaborative editing?

The challenge is : What happens when 2 users edit the same part of the document at the same time.

User A at position 5: types X
User B at position 5: types Y

This is a concurrency problem. A traditional implementation would need to lock the document to handle this. But that would destroy real-time responsiveness. There is a need to automatically resolve conflicts so that every one ends up with same document state.

In Google docs, CRDTs are used to handle concurrent text edits, ensuring that if users insert text at the same position, the system is able to resolve the order without conflicts.

What is a CRDT?

CRDT stands for conflict free replication data type.

A CRDT is a specially designed data structure for distributed systems that:

Can be replicated across multiple nodes or regions.
Allows each replica to be updated independently and concurrently (without locks or central coordination).
Guarantees that all replicas will converge to the same state eventually, without conflicts, even if updates are applied in different orders.

Why do we need CRDTs?

In collaborative editing (like Google Docs, Notion, Figma):

Many users may edit the same document concurrently.
Network latency or partitions mean updates may arrive in different orders at different servers.
We can’t just “last-write-wins” — that would lose user edits.
We want low-latency local edits (user sees their change immediately), with eventual consistency across the system.
Typical in distributed systems

CRDTs give us a way to allow users to edit locally first and let the system reconcile changes without central locks.

Types of CRDTs

There are two broad families:

State-based (Convergent CRDTs, CvRDTs)
- Each replica occasionally sends its full state to others.
- Merging = applying a mathematical "join" function (e.g., union, max).
Operation-based (Commutative CRDTs, CmRDTs)

Each replica sends only the operations performed (e.g., "insert X at position 2").
These operations are designed so that applying them in any order yields the same final result.

Examples of CRDTs in Practice

G-Counter (Grow-only counter): Each replica increments a local counter, merge = element-wise max.
PN-Counter (Positive-Negative counter): Like G-counter, but supports increment & decrement.
G-Set (Grow-only set): Only supports adding elements.
OR-Set (Observed-Remove set): Supports add & remove without ambiguity.
RGA (Replicated Growable Array) or WOOT or LSEQ: For collaborative text editing, where inserts/deletes happen at positions in a string.

These are the basis for how real-time editors like Google Docs or Figma handle concurrent text/graphic editing.

Below is a simplistic Java implementation of a CRDT:

https://github.com/mdkhanga/blog-code/tree/master/general/src/main/java/com/mj/crdt

The code above provides a simple implementation of a G-counter that supports insert, update, delete and merges replicas by taking the maximum value for each node. It is a starting point to understand how CRDTs ensure convergence in distributed systems.

CRDT vs. Centralized Coordination

If concurrent editing is rare → a simple centralized lock/version check may be enough (like your first idea).
If concurrent editing is common (e.g., Figma boards with dozens of people) → you want CRDTs to avoid merge conflicts.

In short:

A CRDT is a mathematically designed data structure that ensures all replicas in a distributed system converge to the same state without conflicts — perfect for real-time collaborative editing.

Note that this would be needed only for collaborative editing at scale in distributed systems. For anything else, it could be an overkill.

Cache in front of a slow database ?

2025-08-30T17:22:00.000-07:00

Should You Front a Slow Database with a Cache?

Most of us have been there: a slow database query is dragging down response times, dashboards are red, and someone says, “Let’s put Redis in front of it.”

I have done it myself for an advertising system that needed response times of less than 30 ms. It worked very well.

It’s a tried-and-true trick. Caching can take a query that costs hundreds of milliseconds and make it return in single-digit milliseconds. It reduces load on your database and makes your system feel “snappy.” But caching isn’t free — it introduces its own problems that engineers need to be very deliberate about.

Good Use Cases for Caching

Read-heavy workloads
When the same data is read far more often than it’s written. For example, product catalogs, user profiles, or static metadata.
Expensive computations
Search queries, aggregated analytics, or personalized recommendations where computing results on the fly is costly.
Burst traffic
Handling sudden spikes (sales events, sports highlights, viral posts) where the database alone cannot keep up.
Low latency requirements
Some systems have low latency requirements. Clients need a response is say less than 50 ms or client aborts.

The Catch: Cache Consistency

The hardest part of caching isn’t adding Redis or Memcached — it’s keeping the cache in sync with the database.

Here are the main consistency issues you’ll face:

Stale Data
If the cache isn’t updated when the database changes, users may see outdated results.
Example: A user updates their shipping address, but the checkout flow still shows the old one because it’s cached.
Cache Invalidation
The classic hard problem: When do you expire cache entries? Too soon → database load spikes. Too late → users see stale values.
Race Conditions
Writes may hit the database while another process is still serving old cache data. Without careful ordering, you risk “losing” updates.

Common Strategies

Cache Aside (Lazy Loading)
Application checks cache → if miss, fetch from DB → populate cache.
✅ Simple, common.
❌ Risk of stale data unless you also invalidate on updates.
Write-Through
Writes always go through the cache → cache updates DB.
✅ Consistency is better.
❌ Higher write latency, more complexity.
Write-Behind
Writes update the cache, and DB updates happen asynchronously.
✅ Fast writes.
❌ Risk of data loss if cache fails before DB is updated.
Time-to-Live (TTL)
Expire cache entries after a set period.
✅ Easy safety net.
❌ Not precise; stale reads possible until expiry.

So, Is It Worth It?

If your workload is read-heavy, latency-sensitive, and relatively tolerant of eventual consistency, caching is usually a big win.

But if your workload is write-heavy or requires strict consistency (think payments, inventory, or medical records), caching can create more problems than it solves.

The lesson: don’t add Redis or Memcached just because they’re shiny tools. Add them because you’ve carefully measured your system, know where the bottleneck is, and can live with the consistency trade-offs.

Takeaway:
Caching is like nitrous oxide for your system — it can make things blazing fast, but you need to handle it with care or you’ll blow the engine.

The Unsung Heroes Behind Your AI Coding Assistant

2025-08-28T04:45:00.000-07:00

While everyone's talking about ChatGPT and tools like Cursor, Windsurf, and GitHub Copilot transforming how we code, let's shine a light on the specialized models that actually power these coding experiences.

Meet the Code Generation Champions:

StarCoder - Trained on 80+ programming languages from GitHub repos, this open-source model excels at code completion and generation

CodeT5 - Google's encoder-decoder model that understands code structure and can translate between languages

InCoder - Meta's bidirectional model that can fill in code gaps, not just complete from left to right

CodeGen - Salesforce's autoregressive model trained on both natural language and code

Codex (OpenAI) - The foundation behind GitHub Copilot, though now evolved into GPT-4 variants

What makes these different from general LLMs?

Trained on massive code repositories (billions of lines)
Understand syntax, semantics, and programming patterns
Can maintain context across entire codebases
Specialized in code-specific tasks like debugging, refactoring, and documentation

The magic isn't just in having "AI that codes" - it's in having models that truly understand the intricacies of software development. They aren’t just regurgitating text—they’re tuned for the nuances of programming, which makes them invaluable for developers. These specialized architectures are why your AI assistant can suggest that perfect function name or catch that subtle bug you've been hunting for hours.

The real game-changer? Most of these models are open-source, democratizing access to powerful coding assistance beyond just the big tech companies.

JDK 21 Virtual threads: The end of regular threads ? Not quite.

2025-08-24T14:38:00.000-07:00

A question I get asked all the time: If JDK 21 supports virtual threads, do I ever need to use regular threads ?

Java 21 brought us virtual threads, a game-changer for writing highly concurrent applications. Their lightweight nature and massive scalability are incredibly appealing. It's natural to wonder: do we even need regular platform (OS) threads anymore?

While virtual threads are fantastic for many I/O-bound workloads, there are still scenarios where platform threads remain relevant. Here's why:

1. CPU-Bound Tasks:

Virtual threads yield the carrier thread when they perform blocking I/O operations. However, for purely CPU-bound tasks, they don't offer a significant advantage over platform threads in terms of raw processing power. In fact, the context switching involved might introduce a tiny bit of overhead.

Consider a computationally intensive task like calculating factorials:

Virtual threads example:


// A CPU-intensive task
Runnable cpuBoundTask = () -> {
    long result = 1;
    for (int i = 1; i <= 10000; i++) {
        result *= i;
    }
    System.out.println("Virtual thread task finished.");
};

// Start a virtual thread for the task
Thread.startVirtualThread(cpuBoundTask);

Platform threads example:


Runnable cpuBoundTask = () -> {
    long result = 1;
    for (int i = 1; i <= 10000; i++) {
        result *= i;
    }
    System.out.println("Platform thread task finished.");
};

// Start a regular platform thread
new Thread(cpuBoundTask).start();

For sustained CPU-bound work, managing a smaller pool of platform threads might still be a more efficient approach to leverage the underlying hardware.

2. Integration with Native Code and External Libraries:

Some native libraries or older Java APIs might have specific requirements or behaviors when used with threads. Virtual threads, being a newer abstraction, might not be fully compatible or optimally performant with all such integrations. Platform threads, being closer to the operating system's threading model, often provide better compatibility in these scenarios.

3. Thread-Local Variables with Care:

While virtual threads support thread-local variables, their potentially large number can lead to increased memory consumption if thread-locals are heavily used and store significant data. With platform threads, you typically have a smaller, more controlled number of threads, making it easier to reason about thread-local usage. However, it's crucial to manage thread-locals carefully in both models to avoid memory leaks.

4. Profiling and Debugging:

The tooling around thread analysis and debugging is more mature for platform threads. While support for virtual threads is rapidly improving, there might be cases where existing profiling tools offer more in-depth insights for platform threads.

5. Backward compatibility

If you want you library or server to be available to users who are on JDKs earlier than JDK21, then you have no choice but to use regular threads. Virtual threads are not just a new library; they are a fundamental change to the Java Virtual Machine's threading model (part of Project Loom). The underlying code that manages and schedules virtual threads on top of carrier threads is not present in older JVMs. This can be one of the most important reasons for using platform threads.

In Conclusion:

Virtual threads are a powerful addition to the Java concurrency landscape and will undoubtedly become the default choice for many concurrent applications, especially those with high I/O. However, platform threads still have their place, particularly for CPU-bound tasks, legacy integrations, and situations requiring fine-grained control over thread management.

Understanding the nuances of both models will allow you to make informed decisions and build more efficient and robust Java applications.

Understanding Isolation levels vs Consistency levels

2025-05-04T17:48:00.000-07:00

In databases, the terms isolation level and consistency level/model are sometimes used interchangeably. "Read repeatable" and "Serializable" are well known isolation levels. But "Strict Serializable" and "Linearizable" are consistency terms.

If you have used Mysql or Postgresql, you know probably know what an isolation levels like "Read repeatable" or "Serializable" means. But when you work on a distributed database you hear about consistency level much more.

The first time I heard about consistency level was when I worked with Apache Cassandra which claimed to only support "eventual consistency". A few years ago when my company was evaluating distributed databases, we had a few architects that insisted that we needed a database that support "strict serializability". CockroachDB was a database that supported this consistency level.

If you are confused, read long. I wrote this blog in attempt to clear up my confusion.

So far, the best explanations on this topic that I found are by Daniel J Abadi [2] [3]. Kyle Kingsbury @Jepson [1] has good descriptions of the topic as well.

But first, a clarification on what consistency means.

What is consistency ?

Consistency is an overloaded term and its meaning has changed in recent times.

ACID consistency

The database must preserve its internal correctness rules after every transaction.

Consider a banking database with a constraint account_balance > 0.

If the starting account_balance is 50 and a transaction tried to deduct 100, that is a violation of that constraint and should fail.

This is the C is ACID. Databases support constraints to ensure this. But it is mainly the responsibility of the application programmer. It is well understood and rarely discussed these days.

Distributed systems consistency

The system must ensure that all nodes (or clients) agree on the same view of data.

Make the distributed system feel like a single threaded single node system. Read of a value any where in the system produces the same result [2]. The result returned is the most recently written value no matter where it was written.

Consider a system with multiple nodes. X was 1. The value X=2 is written to one node and replicated to others. If clients read from the replicas. Do they all see X=2 immediately ? With strict serializable consistency level , the answer is yes. With weaker models, it is possible they read a an older value.

Most of us first heard of this description from the CAP theorem.

Why the difference ?

Both describe behavior under concurrency.

Isolation levels describe problems that occurs in single node databases when transactions execute concurrently. At the highest isolation level transactions execute in some order. Each transaction executes as if it were alone.

In distributed systems there is network latency, replication and partitioning, all contributing latency and timing issues to concurrency issues. Consistency approaches concurrency issues taking time and latency into account as well. At the highest consistency level, transactions execute in order of their order of completion (commit) in real time.

Serializable is the strictest isolation level. Strict serializability is the strictest consistency model. In a single node system, there is very little difference between the two because the time issues are small.

Isolation Levels vs. Consistency Models

To summarize the key differences.

Isolation Levels

Prevent read, writes of uncommitted data.
Prevent anomalies like read uncommitted, non repeatable reads, phantom reads
Focus on managing concurrent access to data while balancing performance and correctness.
Common isolation levels (from weakest to strongest):
- Read Uncommitted
- Read Committed
- Repeatable Read
- Serializable — the strictest standard defined by the ANSI SQL standard.
Old blog

Consistency Levels

Typically relevant distributed databases.
Time is a factor
They describe the guarantees about visibility and ordering of updates in a distributed, replicated data system.
They focus on the behavior perceived by clients across multiple nodes or replicas.
Examples include:

Strict serializability
Linearizability
causal consistency

Example to Illustrate the Difference:

Scenario:

Two accounts A and B initially have a balance of 100 each.
Two concurrent transactions:
- Tx1: Transfer 50 from A to B.
- Tx2: Reads balances of A and B and sums them

Isolation level Serializable:

Tx1 and Tx2 are serialized, and the sum read by Tx2 is 200.
(Tx1, Tx2) and (Tx2, Tx1) are valid orders irrespective of when each actually committed first.

Consistency level Strict Serializable

If Tx1 commits before Tx2 starts, Tx2 must see all effects of Tx1. The only valid order is (Tx1, Tx2)
However if there is some overlap like if Tx1 commits after Tx2 starts, then both orders (Tx1, Tx2) and (Tx2, Tx1) can be valid. Reason is that Tx2 cannot read the data committed by Tx1

A few descriptions

Let us briefly touch on some levels you will encounter often. For more detailed descriptions, I will refer you to https://jepsen.io/consistency [1]

Serializability

Transactions occur in some total order. Even though they may actually execute concurrently, it appears as if they execute one after another. While serializable will prevent non repeatable reads and phantom reads, It will allow "time travel" anomalies as shown in the example above. It can appear that Tx2 happened before Tx1, even though in reality it was the other way around.

Strict Serializability

Transactions occur in a strict order that is consistent with the real time (clock time) order in which transactions occur. It applies to the entire system encompassing multiple objects. A is before B in the order if A commits before B begins. So the only valid order is (A,B). However if A commits after B begin, then both orders (A, B) and (B, A) are valid.

Linearizable

Transactions occur in a strict order that is consistent with the real time (clock time) order in which transactions occur. But this applies to a single object not to the entire dataset. Definition of a single object varies. Could be a key or a table. [1]

Most of the time concurrency issues are important when multiple threads touch the same data and that why this model is also as important as strict serializability.

Causal Consistency

Transaction that are causally related are seen by all nodes in the same order, while concurrent (unrelated) operations may be seen in different orders. In a social media application, a user making a post and another user liking the post are causally related. The like must be seen only after the post is seen. However, it is ok for a unrelated post that happened after the previous post to be seen before that.

Conclusion

It is all about how systems behave under concurrency.

Isolation levels deal with how transactions behave when they run at the same time, while consistency models talk about how different nodes in a distributed system agree on data. And "consistency" itself has changed over time, from enforcing business rules in ACID databases to ensuring replicas don't drift apart in distributed ones.

Database vendors advertise the consistency level they support as a key feature. That is why it is important we understand what it means and ensure that we pick the right database the fits our needs.

References

1. https://jepsen.io/consistency

2. Introduction to consistency levels , Daniel J Abadi

3. Isolation levels vs Consistency levels, Daniel JAbadi

A non trivial concurrency example in Go language

2025-04-01T18:08:00.000-07:00

Overview

In this blog I describe a non trivial concurrency example in the Go programming language. The code is part of my Dynago project https://github.com/mdkhanga/dynago.

This is not a tutorial on Go or on writing concurrent programs. But if you know a little bit of Go and/or little bit of concurrent programing, then I am hoping this can be a useful example.

In Dynago, I have so far built a leaderless cluster of peer servers. The command for starting a server needs to only point to one other server. The only exception is the first server, which has nothing to point to. The servers exchange ping and gossip like messages to share the details on the members of the cluster. After a few messages every server has the cluster membership list. When servers join or leave the cluster, the membership list is updated.

Code

If you would like to skip reading the text and jump to the code, the relevant files are :

https://github.com/mdkhanga/dynago/blob/main/cluster/peer.go
https://github.com/mdkhanga/dynago/blob/main/cluster/ClusterService.go

The relevant tests are
https://github.com/mdkhanga/dynago/blob/main/e2e-tests/test_membership.py
https://github.com/mdkhanga/dynago/blob/main/e2e-tests/test_membership_chain.py

I show some screenshots of code snippets in this blog for the casual reader. But if you are interested in the code, I recommend directly looking at the code in github.

Go concurrency recap

In Go, you write code that executes concurrently by writing Goroutines.

A goroutine is like a lightweight thread. It is written as a regular function.

You run a goroutine using the go keyword.

Go provides channels for communicating between goroutines. Channels are a safer way to send and receive data. This is safer than using shared memory as is done in Java, C++. You do not have to use mutexes to avoid memory visibility and race conditions.

The select statement lets the goroutine wait on receiving data on a channels.

Though not recommended, the shared memory approach of data exchange is also supported. You will need to use primitives from the https://pkg.go.dev/sync package to synchronize access to data.

The Examples

There were two features in Dynago where concurrency is relevant.

1. Each server receives a message on a GRPC stream. It has to process the message and sometimes send a response back. This is the classic producer - consumer problem. Some goroutines produce. Other goroutines consume and do work. I use channels for sharing the messages between producers and consumers.

2. A map stores the list of cluster members.

The map is to be updated as servers join or leave the cluster.
Periodically we need to iterate over the map and send out our copy of the membership list to others.

The receiving server will merge the received list with it own list.
If the receiving servers list is more up to date, then it sends a response to and the original sender has to merge.
timestamps are used to determine who is more recent

This the case of multiple goroutines reading and writing a data structure concurrently.

Example 1: Channels

The peer struct shown in the code below defines a channel InMessagesChan for incoming messages and a channel OurMessgesChan for outgoing messages.

The receiveMessages goroutine method has for loop with following code. It reads a message from a Grpc stream and sends it to the InMessagesChannel.

The processMessageLoop goroutine method in peer.go has a loop that takes messages from the InMessagesChannel and processes them. When a response is needed, it processes the message and writes the response that needs to go out to the OutMessagesChannel. The image below is shortened version of the real code.

Lastly the sendLoop goroutine method has a for loop that takes messages from the outMessageChannel and writes them to the outbound Grpc stream.

As you can see, very simple. 3 goroutines working concurrently by exchange data over channels. No locking, no synchronization an fewer problems.

In my opinion, channels is one of the best features of Go.

Example 2 : Shared memory

I have this struct cluster which has the list of peers in the cluster

We need to add /update / remove entries from the map in a thread safe manner. The code below uses the mutex to synchronize access to the map. These methods are either called from both peer.go and cluster.go.

The code below shows a loop from the ClusterInfoGossip method that uses the same mutex to synchronize the map while iterating over it. Typically the calls from add, remove and gossip happen from different goroutines. If you do not synchronize, you will have memory visibility problems. Note that since peer is updated, we need to synchronize access to the peer as well.

You might be wondering, why can I not just do this using channels when it is safer and cleaner ? What you would do is to write a function with like an event loop. The function can accept commands on a channel and read , update or iterate over the map based on the command. The code would look something like below

Which approach you take is sometimes a personal choice. For the message processing, I preferred channels. But for CRUD on a map, I preferred shared memory.

Conclusion

In conclusion, what I have shown you here is a non trivial concurrency example in Go. Go makes it quite easy to write concurrent programs. The recommended way to share data between goroutines is by channels. By using channels, you can avoid race conditions and memory visibility issues. But the traditional approach of shared memory with synchronization is also supported.

As I build out Dynago, I plan to blog about the interesting pieces. If you like my blogs, please follow me on LinkedIn and/or X/twitter.

Multi Version Concurrency Control (MVCC) in databases

2025-03-01T08:16:00.000-08:00

Introduction

Multi version concurrency control (MVCC) is a popular optimistic technique used in modern databases for concurrency control.

MVCC does not use locking. In that regard it is an optimistic technique but distinct from what is know as the optimistic concurrency control (OCC).

MVCC is timestamp based.

MVCC does its concurrency control by keeping multiple copies or versions of each data item. Every transaction sees data only of a specific version , also known as snapshot. Changes made by a transaction will not be seen by others, until the changes are committed.

Concepts

Versioning

Versioning tuples is at the core of how MVCC does its concurrency control.

Consider the table

id, balance

A, 500

B, 1000

Now logically assume that under the hood, the database adds 2 hidden columns - start time (st) and end time (et)

id, balance, st. , et

A, 500 , 1 , -

B, 1000, 2, -

Let us assume that increasing numbers 1,2 ..... represent time and a - implies infinity or no end time. The version number can be a timestamp or something analogous to a timestamp.

In the above example, we are saying that the first tuple (A, 500) is valid from time 1 to infinity. The second tuple (B, 1000) is valid from 2 to infinity.

With versioning, when a tuple is updated, the original one is unchanged. Instead a new row is added.

Let us say at time 3, A is updated to 550. Logically the table would look like

A, 500 , 1 , 3

A, 550 , 3 , -

B, 1000, 2, -

The latest committed version has an end time of infinity. When a new update for a tuple is committed, the end timestamp of the previous latest is changed to the start timestamp of the new latest.

A transaction reads the latest committed version at the time it starts. A transaction that starts at time 2 and reads A will read 500. But a transaction that starts at 4 and reads A will read 550.

Snapshot isolation

Every transaction only reads committed values at the time it starts. It is a snapshot of the database at that time. This referred to as Snapshot isolation.

However it will read any changes that it makes within the scope of its transaction.

Use cases

It is easier to understand MVCC with some examples,

Lost update problem

Let us start with the same table again.

id, balance, st. , et

A, 500 , 1 , -

At time 2, transaction T1 reads A. The latest version with timestamp (1, -) has value 500. It reads 500.

At time 3, transaction T2 reads the same row. The latest version is still (1, -) . It too reads 500.

At time 3, transaction T2 updates the value to 550 and commits. We now have an addition version

A, 500 , 1 , 3

A, 550, 3 , -

At time 5, T1 updates the value to 600.

At time 6, T1 tries to commit.

T1s commit is disallowed and it is forced to abort.

The reason is that T1's snapshot is of time 1. After T1 started, another transaction came along , performed a WRITE on the same record and committed. If we allowed T1 to also commit, it would overwrite T2's update. This is the lost update problem.

The rule we can deduce here is : The write set of the committing transaction T1 should not intersect with the write set of any transactions that committed after T1 started and before T1 commits.

Write Skew

Let us start this time with the table having 2 tuples.

id, balance, st. , et

A, 500 , 1 , -

B, 1000, 2, -

At time 1, transaction T1 reads A. The latest version with timestamp (1, -) has value 500. It reads 500.

At time 3 , transaction T1 writes A to 600. But it is still uncommitted.

A, 500 , 1 , -,

A, 600 , 3 , - , uncommitted

B, 1000, 2, -

At time 4, transaction T2 reads A. The latest committed record is still at timestamp 1. It will read 500 ( not 600 ).

At time 5, T1 commits. The write set to T1 does not intersect with the write set of any other transaction. So it allowed to commit. So we have

A, 500 , 1 , 5,

A, 600 , 5 , - ,

B, 1000, 2, -,

At time 6, T2 updates B to 1200

B, 1200, 6, -, uncommitted

At time 7, T2 tries to commit. It is disallowed and T2 is forced to abort. The reason is that T2 read A which was committed after T2 started. Even though T2 does not update A. It might be using A (that is read earlier) to update say B and that update could be incorrect.This is the write skew problem.

We can thus deduce rule 2: The read set of a committing transaction T should not intersect with the write set of any transaction that has committed after T started.

Phantom reads

Let us look at a case where a new record is insert into the table.

id, balance, st. , et

A, 500 , 1 , -

B, 1000, 2, -

At time 3 , transaction T1 runs the query "select sum(balance) from table where balance >=500 ". It gets a result 1500.

At time 4, transaction T2 inserts a new row, (C, 600) and commits

id, balance, st. , et

A, 500 , 1 , -

B, 1000, 2, -

C, 600, 4 , -

At time 5, transaction T1 inserts the 1500 value it read into another table.

At time 6, T1 tries to commit. The transaction is not allowed to proceed and aborts.

The reason is that the 1500 value is no longer accurate. At time 4 , another transaction committed a new row and the value should be 2100.

Rule 3 : If the committing transaction T has predicate queries (where clause ) that depend to number of rows and affected by inserts/deletes, then just before committing the queries need to be run again. The commit is allowed only if the results are the same as before.

Obviously running the queries a second time can be expensive. But that is implementation detail and databases do various optimization.

Garbage collection

Maintaining versions for each and every record can take up a lot of disk space. Databases have background process or other method to remove old versions that have no active transaction depending on them.

Algorithm

Every tuple is versioned using timestamp or something analogous to it.

Every transaction only reads rows that are committed as of the start time of the transaction. This holds for the entire duration of the transaction. It cannot read any commits that happened after it starts. In other words it sees a "Snapshot" of the data, as of its start time. This is Snapshot isolation.

When a transaction tries to commit, we check if any other transaction has committed a new version for the rows that we read or write. If there are any such transactions, then we must abort.

Even if our read/write set does not intersect with any other transaction, if our read is impacted by inserts or deletes, we much abort.

To make this happen, the database needs to maintain start time, end time for every tuple. Not just the latest version, but older versions as well.

For every tuple that is read or writen, the database needs to know which transactions are active and which committed.

Older versions need to cleaned up when there are no active transactions depending on them.

For a transaction to be allowed to commit, there rules we discussed apply:

#1 : Write set of a committing transaction T should not intersect with Write set of any committed transaction that committed after T started.

#2: The Read set of a committing transaction T should not intersect with the write set of any transaction that has committed after T started.

#3: When the committing transaction T has predicate queries (WHERE clause, depend on number of rows meeting condition ), then before committing, the queries need to be rerun to ensure that they return the same result.

Some additional rules:

#4 Write set of a committing transaction T should not intersect with Read set of any transaction that committed after T started.

#5 If a transaction T updates a row, T should use its own updated version, not the one in older snapshot it started with.

Commercial Databases supporting MVCC

Almost all the modern databases we know off support MVCC.

PostgreSql

Mysql

Oracle

CockroachDB

SqlServer

.... and more

Advantages of MVCC

Readers do not block readers. Writers do not block readers.
Each transaction works under a consistent snapshot of data.
High concurrency is possible.
Databases can allow querying of old versions, which are known as time travel queries.

Disadvantages of MVCC

Increased storage requirement because we are storing older version.
Need to do garbage collection.
When conflicts are detect and transactions aborted, applications need to retry.

Conclusion

MVCC is a popular concurrency control technique used in many currently popular databases like Postgresql.

It does not use two phase locking. Unlike OCC, it does not use a staging area. The staging area is built in with versioning.

Like OCC, it works best for workloads where there are few conflicts.

In conclusion, Multi-Version Concurrency Control (MVCC) offers a robust approach to handling concurrent transactions in modern databases by maintaining multiple versions of data.

While it provides significant benefits such as high read throughput, reduced contention, and snapshot isolation, it also introduces complexity in garbage collection and storage management.

Modern distributed databases combine versioning with clock time to provide strict serializability even in distributed databases without the use of locking. But that is a topic for another blog.

References

Prof. Jen Dittrich Youtube videos

Andy Pavlov CMU Database Youtube video

Database Internals by Alex Petrov

Database Management Systems by Ramakrishnan & Gehrke

Optimistic Concurrency Control In Databases

2025-02-01T07:37:00.000-08:00

Introduction

Optimistic concurrency control refers to concurrency control in databases without the use of explicit locks. Traditionally it is done using locks. But locking reduces throughput and scalability. Modern databases therefore prefer not to use locking and use optimistic techniques. This is a brief description of the optimistic technique.

Why concurrency control ?

Concurrency control in databases refers to techniques used to execute transactions concurrently.

From a end users standpoint, every transaction needs to execute correctly and preserve the consistency of the database, irrespective of what other transactions are doing. For the user, it should appear that the transactions are executing in some serial order.

When multiple transactions read and/or write the same data several kinds of errors can happen such as

dirty reads - reading uncommitted data
non repeatable reads - read a second time and you get a different value
phantom reads - read a second time and you get additional rows
Lost update - one transaction overwrites others write
dirty write - a transaction reads an uncommitted value and updates it
write skew - the combination of two transactions breaks some invariant like minimum balance

At a user level, databases give the users a knob called the isolation level to protect against some of these errors. For an explanation of isolation levels, see the blog I wrote many a few years ago.

But under the hood databases implement concurrency control to ensure that users see a consistent view of the data at all times.

There are 3 main types of concurrency control:

Pessimistic concurrency control
Optimistic concurrency control (OCC)
Multi version concurrency control (MVCC)

The pessimistic approach involves locking resources - rows, tables , pages etc while they are used by one transactions. Other transaction that need that resource wait until the lock is released. This approach prevents conflicts but comes at the cost of throughput.

OCC and MVCC take the optimistic approach that conflicts rarely happen. They lets transactions proceed without acquiring locks. If a conflict is detect downstream, one of the transaction is forced to abort.

In this blog we discuss one such concurrency control technique - optimistic concurrency control. I will discuss the other two in subsequent blogs.

What is optimistic concurrency control ?

As the name suggests, the approach is optimistic. Optimistic implies a positive view of things that most transactions do not conflict with other and there is no need to slow things down by acquiring locks and holding on to them.

Assumptions are that

Transactions are short lived
They rarely conflict

The concurrency control here is done in 3 stages.

In the first stage (read) the transaction executes its operations in a private or staging area. Rows that are read known as the read set and rows that are updated known as the write set are identified.

In the second stage (validation) the operations are checked for conflicts. Rows that are being read by one transaction should not be be updated by others. If a conflict is detected, then the transaction is aborted and the staging area cleared

In the third stage, if there are no conflicts the changes are committed.

Detecting the conflicts

Given 2 transactions A and B,

A and B are given timestamps(TS) before the validation starts.

Let assume TS (A) < TS (B).

If A committed before B started read, then there are no issues and B is also allowed to commit.

If A committed after B started read but before B started write, then both can commit if the write set of A does not intersect with the read set of B.

If A read completes before B read completes, and the write set of B does not intersect with both the read set of A and write set of A, then both can commit.

In all the 3 cases, A and B are operating on different records and do not conflict.

In this example, we are looking "backward" from B. If there is a conflict B will asked to abort. Some databases look "forward" but the concept is the same.

When a validation is being done, no other transaction is allowed to commit. Only one transaction is validated at a time.

Timestamp based

In pessimistic concurrency control, transactions are ordered in the order in which locks are acquired.

In optimistic concurrency control, transactions are ordered based on timestamps given to transactions.

Commercial databases using OCC

Some example are:

Amazon Aurora DSQL

Yugabyte

CockroachDB

FoundationDB

Fauna

Google Spanner

Advantages of OCC

Higher scalability

Higher throughput

No deadlocks

In general this is suitable for low conflict data access.

Disadvantages

Transactions might need to be aborted.

Applications need to handle aborted transactions and retry.

This might not be suitable for cases when conflict is high - that is multiple transactions needs to access the same data.

Conclusion

Optimistic concurrency control is becoming increasing popular is modern databases that need to scale and can be distributed across multiple node. It really works well in low data contention scenarios. It is ideal for distributed databases where the locking approach becomes even more expensive when you need to lock across nodes - as would be needed in modern distributed databases. For reasons mentioned above, distributed , cloud native, high concurrency systems are moving towards OCC.

References:

1. Database Management Systems - Ramakrishnan and Gehrke

2. Database Internals - Petrov

3. Andy Pavlov CMU lecture on youtube

Dynago : A highly available distributed key value store

2025-01-04T15:02:00.000-08:00

Introduction

Dynago is my hobby project to implement a minimal implementation of the distributed system described the paper Dynamo : Amazons highly available key-value store.

This is the first in a series of blogs when I describe how I go about building this and post updates on the progress.

The is my way of "#buildinpublic".

The github repository is at https://github.com/mdkhanga/dynago

Why am I doing it ?

The primary goal is learning. Learning not just for me but also for others who may read this blog and review the code.

A secondary goal is build something useful. Perhaps I may use it somewhere or someone else might.

It is possible that parts of the code might be reusable in another project.

Why a Dynamo clone ?

The Dynamo paper is the first paper that got me interested in distributed systems. It has been by desire to build something like that for a long time.

In the real world, DynamoDB is Amazon highly available key/value database. Apache Cassandra is also a project inspired by this paper.

Tech stack

I pick Go as the programming language. Why Go ? I have already done a couple of complex projects in Java (B+tree, Raft protocol). So I was not interested in doing it in Java. I am not yet that familiar with Rust. And C/C++ felt like going back in time. So this is an opportunity to do a complex project in Go.

For network communication / RPC, I use GRPC instead of programming to sockets. My Raft implementation is based on sockets. I wanted to try Grpc. If it does not work out, I can switch to sockets.

Storage engine is TBD. Early versions will be in-memory. In the future Rocksdb or another embeddable store is a possibility.

Some technologies we might learn here:

Go programming
Distributed systems
Databases
Network programming
Programming servers from scratch
Concurrency
High availability

Implementation plan

I plan to implement in the following tentative order:

1. Cluster membership

Be able to start multiple servers connect and form a cluster. Gossip to detects servers joining and leaving the cluster.

2. Client API

Simple Get and Put API

Replicate to all nodes.

3. Partitioning using consistent hashing

4. Partitioning with replicas

5. Quorum based read / writes

6. Versioning

7. Hinted handoff

...... and so on

Conclusion

If you are interested in distributed systems and/or databases, I invite you to follow along.

I am open to suggestions and discussion. If you know more, I am happy to learn from you.

If you like what I am working on, please follow me on twitter/X or LinkedIn.

Replication in modern data systems

2024-07-20T16:27:00.000-07:00

Overview

Replication means making a copy of the data for future use in the case of failures or may be to scale.

Why is it a big deal ? We copy files for backup all the time. For static files, that do not change, making a copy is 1 copy command. But if the data is being updated by users all the time. How often do you run the command to copy. How do you keep the copy in sync with the source ?

That is the problem of replication in databases and data systems. All databases have replication built in that you can setup with a command or two. So why read or discuss it? If you are building a distributed systems that involves data, you will need to replicate data. The concepts from databases will be useful.

While replication is most well known for its use with databases, it is also a critical part of distributed systems where the data is unstructured such as distributed file systems (HDFS) or messaging systems (Apache Kafka)

This post covers replication in traditional single node systems as well as modern distributed systems.

Why do we need replication ?

There are several reasons why replication is needed. It is more than just taking a backup.

Redundancy

Make a copy of the data. When the main server becomes unavailable for any reason, switch to the copy. This is ensure that the data is always available.

Scalability

Your data becomes really popular and the database gets a lot of read requests and cannot keep up. So you make copies of the database and have a load balancer distribute the request across to the copies (replicas).

Geo distribution of data

Bring the data close of user. You have users in Americas, Europe and Asia. Data from americas is replicated to Europe and Asia, so users there can read data locally without making a round trip to the americas for every read.

Secondary use cases

These are lesser known and unconventional use cases. They might be done higher up in the stack at the application layer or middleware than in the database.

Mirroring

Mirroring involves replicating the requests to the application to a copy of the entire application stack. You can think of this as application level replication.

For example, for a REST service, this involves sending the http request, not just to the production service but also to a mirror service.

The mirror service reads and writes from the mirror database. Mirror database is a previous replica that was in sync with the leader. Just before starting mirroring, it is discontinued as a replica so it does not get duplicates.

Mirroring can be used for testing large complex changes against production traffic.

Data in the mirror database is then compared with data in the production database for accuracy.

Testing

A regular database replica is used as a test database. Various kinds of tests - feature tests, performance tests, concurrency tests, scalability tests can be run with services running with the replica. This is a different use case from mirroring.

Migration

This can be used to eliminate or reduce downtimes needed for migration.

Create additional replicas.

Run migration on them.

Rollover the application services to the new database replicas.

Replication strategies

Single leader

This is the most common pattern. It shown in Figure 1.

One server is designated as the leader. The others are followers. All writes go to the leader. The leader replicates the writes to the followers.

The advantages are :

Setting up is fairly easy.

Reads become scalable. You can put a load balancer in front and distribute read requests to followers.

High availability: If the leader fails, you fail over to one of the followers and let it become the leader.

The disadvantages are :

All the writes go to one server , the leader. So this can become a bottleneck. Writes are not scaled.

If you read from a replica that is behind on replication, you might read stale data.

Multi leader

Writes can go to more than one server.

Multi leader replication is needed when

(1) Writes and replication needs to happen across geographically distributed areas.

(2) Connectivity to single leader is not guaranteed. The is usually the case with mobile devices or laptops or when people want the ability to work offline and/or multiple devices.

In the geo distributed case, the writes go to a local local leader. The local leader not only replicates to local replicas but also to the distributed leader (who replicate to their replicas).

In the mobile case, the writes are store locally and the replicated periodically when connectivity is available.

Advantages:

Writes are also scaled.

Writes can done locally or close to clients. Better latency for writes.

Disadvantages:

Since writes happen at multiple leaders. There can be conflict. The conflicts need to be resolved.

Leaderless

In the leaderless model, all nodes are equal and no node is designated leader. Writes can go to any node and that node replicates the write to other nodes. This is the model made popular by AWS Dynamo and later adopted by Cassandra.

Consensus based replication

All the above methods have either write conflict or read consistency issues. Raft and Paxos are two well protocols for replicating log entries. Data to be replicated is modeled as a list of entries in log. The short story is that one server sends one entry or a sequence of entries to others and it is considered committed if a majority of servers acknowledge having received them. Raft has leader election but Paxos is leaderless. Raft protocol describes in detail leader election, replication, server crashes, recovery and consistency checks. The paper is a good read for anyone interested in distributed systems.

Replication Implementations

The first three techniques apply to databases which deal with structured data and are a little more complicated.

Statement based replication

In this approach, the SQL statements such as INSERT/UPDATE/DELETE etc are forwarded as they are from the leaders to the followers. While this can work in most cases, it does not work in certain cases such as timestamps or when you generate an id or a random number.

It is not efficient either. If you insert a record and then delete it, why replicate both commands ?

Write ahead log (WAL) replication

Databases first append every write to the WAL before doing anything else, before writing it to structured storage from where it will be read. WAL is used for recovery. If the database crashed, it state is reconstructed from the WAL. A recent slogan has been "The WAL is the database". Replication here involves replicating the WAL.

A disadvantage is that WAL entries contain where specific storage details like which byte in which block is to be updated. This can create compatibility issues if the leader and followers are on different versions.

Logical replication

A logical log on the other hand captures at a row, how the table was changed. You can view this as an approach somewhere between statement based and WAL replication.

Change data capture is a form of logical replication. It is used to replicate changes in a database to other third party systems. A popular use case is data warehousing where data from multiple sources is aggregated and summarized for analytics.

Unstructured data replication

For unstructured data as in distributed file systems the unit for replication is a block of data. Data is first partitioned into blocks and each block is replicated independently.

Potential issues with replication

Replication Lag

Most of the time replication is asynchronous. Client writes to the leader and returns before any acknowledgement that it has been replicated. Synchronous replication is not viable due both performance and availability issues. A single failure can hold up all replications.

Lost write

However, one problem this creates is that if you read immediately after a write, the replica you are reading from may not yet have your last write.

Inconsistent read

If you read multiple times in quick succession ( same read) , each read may get a different result depending on which replica services the read ( as the replicas may be in different stages of replication)

Cassandra addressed this issue using quorum. CockroachDb uses a consensus protocol like Raft.

Write Conflicts

Write conflict is an issue in multi leader replication. This happens when multiple clients update the same data while talking to a different master. The database does not know which update to accept and how they should be merged. This is similar to a merge conflict in git.

An approach to handle conflicts is to store both versions on write. But on read, send both versions to the client and let the the client resolve the conflict

Replication is real world systems

The product documentation for database on replication can be quite confusing. It best to follow a tutorial or blog in the internet.

Postgres

The documentation and blogs describe it in 2 ways.

You can set it up as synchronous, asynchronous , streaming , log file based etc

And it can be WAL based or logical replication. Statement based is rarely seen.

In snapshot replication, a snapshot of the database is taken and replicated to followers.

Instead of streaming, you can also setup the replication as file based, where the WAL files are periodically shipped to followers.

In WAL replication, replication slots lets the leader track how much of the WAL is replicated to each replica. This helps the leader not discard segments not yet replicated. But this consumes resources on the leader. Replication slots need to be managed and deleted when not needed.

Mysql

The traditional way in mysql was a logical replication based on their binlog file - a binary format for logical changes.

The newer way is based on global transaction identifier (GTID) which is built on top of the binlog. It can be either statement based or row based.

Dynamo / Cassandra

In this architecture, replication is fundamental to the architecture. All you need to do is to set the replication factor to greater than 1. All servers are equal - no leader and no follower. Writes can go to any server. Partitioning is also fundamental to the architecture. The server that receives the write redirects the write to appropriate server. From here it is replicated to other servers based on the replication factor.

Consistency issues are addressed using quorum based tunable consistency. Quorum mean a majority which is (RF/2+1) agree on something. If you have replication factor (RF) 3, quorum is 2. So on a write, at least 2 nodes need to acknowledge that the write was saved. On read, at least 2 nodes need to agree on the return value. In general, to avoid inconsistencies, you want Read quorum (R)+ Write quorum (W) > RF .

CockroachDb

CockroachDB uses the Raft distributed consensus protocol to ensure that a majority of replicas are in consensus before any change is committed. This is the safest approach to ensure consistency but comes at a cost.

Apache Kafka

In Kafka, messages are sent and received from topics. Topics are split into partition. Each partition has one leader and a configurable number of replicas. Writes go to the leader which replicates to the replicas. Reads can go to the replicas. Each broker is a leader for some partitions but a follower for other partitions. Like Cassandra and CockroachDb, replication is core to the architecture and easy to setup.

Apache Hadoop (HDFS)

This applies to any distributed file system. The file is a sequence of blocks of data. HDFS has a name node and data nodes. Name node maintains a map of which data nodes have the blocks of a file. Each block is replicated to a configurable number of data nodes.

Conclusion

Replication is a critical piece of any distributed data system. It has to be part of the core architecture. It cannot come after the fact like it did in the past. While redundancy and HA are well known benefits, there are other benefits such geo distribution of data as well. It can cause some effects such as read consistency. Care should be taken to address those. Different products use different strategies. You should be familiar with the replication strategies, configuration and side effects for your data product. If you are building a new system with data, understanding how existing systems replicate and the issues they face, can help you design your replication.

How modern distributed systems scale by partitioning ?

2024-07-03T17:12:00.000-07:00

1.0 Introduction

In the last 20 years, software systems moved to the internet and handled large volumes of data and millions of requests. Most people interact with these systems using a browser or a mobile device. At the back end, is not one powerful computer but generally a network of commodity computers. Both the processing and storage of data is spread across multiple computers. In this blog we discuss how large datasets can be stored using multiple commodity computers.

Partitioning is the process of breaking up a large dataset in parts so that each part can fit easily on the disk on one one node and be efficiently managed by each node. For very large data sets that cannot fit on 1 machine, data needs to be broken up into parts ( partition or shard). Each partition is stored on a different machine. This is just natural horizontal scaling. But most important is that, when it time to read the partitioned data, we need to be able to find (efficiently) which partition and node has the data we want to read.

Storage space is not the only benefit of partitioning. You are also spreading the compute required to read, write and process the data.

Partitioning is generally combined with replication to make the partitions highly available. But we do not discuss replication here. That is a topic for another blog.

2.0 Types of partitioning

There are 2 types of data that need to be considered: unstructured and structured.

Most discussions on partitioning discuss partitioning of data in databases ( structured data ) but not unstructured data which is outside databases in plain files. This blog discusses both unstructured data and structured data.

2.1 Structured data

The problem is more interesting for databases because it is not enough to break up the dataset into smaller parts. During reads you need to be able to find the data. And you need to do it fast. When the database receives a query - "Give me records for Customer X", How does it know which node hosts the data ? Does the database have to send the request to all the nodes ? That would be quite inefficient.

The goal is thus to partition data and query it efficiently. Another goal is to ensure that distribution of data between partitions is even. You do not want a situation where partition 1 has 70% of the data and the other 5 partitions has the remaining 30%. This will overload partition 1 ( known as a hot spot) and you lose the benefits of partitioning.

2 strategies are commonly used for database partitioning.

2.1.1 Range based partitioning

Database records are generally stored sorted based on the primary key.

Initially there is one partition with zero records.

As clients write to the database, the size of the partition increases. When it reaches a certain size say 10MB or 64MB, it is split into two partitions.

Each partition may be assigned to a different node.

This process is repeated as more data is added and partitions grow. If data is deleted and partitions shrink, then small partitions can be merged

To efficiently query data, the database needs to do some book keeping

-- which key range is in which partition

-- which partition is at which node

Starting with 1 partition and 1 node is not efficient for obvious reasons and databases typically start with configured number of partitions or a number of partitions proportional to the number of nodes.

To balance the load on nodes, partitions may need to be moved between nodes.

This is the strategy used by HBase, CockroachDb, MongoDb.

2.1.2 Hash based partitioning

The hash value calculated from the key is used to determine the location where the record can be stored.

The wrong way to determine the node is by using hash mod n, where n is the number of nodes. The problem with this approach is that when nodes are added or removed from the cluster, a very high percentage of the keys need to be removed.

A better approach is to start with a fixed number of partitions , way more that the number of the nodes the cluster will ever have , say for example 1000 or 10000. Partitions are logical. Hash ranges are assigned to partitions. Partitions are assigned to nodes either using numPartitions mod numNodes or other algorithms. This is shown in the top half of Figure 4. The bottom half of figure 4 visualizes the same as a ring as is done in many articles that to refers to this as consistent hashing. Think of partitions being placed on the ring. Each partition owns the key space from the position of the previous position to its position. The cluster needs to maintain a mapping of partitions to nodes. When a new node is added, the cluster can take a few partitions from existing nodes and assign to the new nodes. When a node is removed, the cluster assigns its partitions to other nodes. Looking up a key is a extra level of indirection. The hash of key maps to a partition. The partition node map tell you which node has the partition that has the key. Many studies has shown that this lead to less movement of keys between nodes as the cluster changes.

In popular press, this has the poorly understood name "consistent hashing". It is just hast hash based partitioning. Nothing consistent and nothing to do with consistency.

2.1.3 Secondary Indexes

So far we have been talking about partitioning by the primary key, also know as the primary index.

To speed up retrieval of records, databases also have secondary indexes which can be very large and might need to be partitioned.

One approach can be to keep the secondary indexes local to the node on which the primary index partition is. The advantage of this approach is that since all related rows are on the same node, inserts/updates/deletes are are all local. But queries on the secondary index requires sending queries to all nodes and aggregating the responses.

Another approach is to create a global secondary index and partition it as an independent entity. However since the secondary index partitions might be on a different partition from the primary partition, CUD operations are more expensive. Transactions might be distributed. However range queries on secondary indexes are more efficient since closer records (by sort) are on the same partition.

2.2 Unstructured data

Unstructured data refer to ordinary files that have text or binary data. Of course we are talking about large files or many large files. This is the use case for a distributed file system such as HDFS (hadoop file system) or GFS ( Amazon ). Logically the implementation of distributed file system is similar to say a linux filesystem. You view the file system as a list of blocks of fixed size. On a single node linux file system , all the disk blocks are on one node. In a distributed file system, the blocks are spread across multiple nodes. In HDFS, the name node maintains the metadata for the distributed filesystem -- given a file , which blocks make up the file and which nodes have the blocks. To create a file, the name node may assign a block on a particular node and the client talks directly to service called data node running on the target node to write to block. To read a block, the name node directs the client to the data node that hosts the block and the client reads directly from that block. But the basic algorithm is simple -- break up the file data into blocks and spread them across nodes.

Another example is partitioning of topic logs in Apache Kafka is a messaging system ( they like to call it event streaming) where producers write messages to a topic and consumers read messages from the topic. The storage for a topic is a log file. New messages are appended to the end of log file. They are read from the front. Obviously the logs can grow beyond what can fit on a node. So log is broken into partitions and distributed across multiple nodes. The broker serves producers and consumer. A Kafka cluster has multiple brokers with each broker running on a seprate node managing multiple partitions.

3.0 Rebalancing

Rebalancing is the process of moving partitions between nodes to make the distribution of load even across all the nodes. This is necessary when nodes join or leave the cluster or if the cluster starts receiving more data for certain keys. Either way rebalancing is an expensive operation that needs lot of CPU, memory and network bandwidth. It can have an impact on the performance of regular CRUD processing. In a ideal world, we would like rebalancing to happen automatically behind the scenes, without end users knowing about it. But for performance reasons listed above, that rarely works well in practice. Some databases require an admin to manually start a rebalance, which can be done during a period of low load and monitored.

4.0 Routing

How does a client know which partition to connect to ? The short answer is that the database has to maintain mapping of partition to node. In the case of hash based partitioning, hash maps to a partition which maps to a node. In the case of range based partitioning the key maps to a key range which maps to a partition which maps to a node. The partition node map is available to nodes. If a client can connect to any node, then if the node does not have the partition to handle the clients request, it can redirect the client to the appropriate node or it can get the data from the target node and return to client.

5.0 Use case

There are cases where you might not have a ready made database doing server side partitioning for you and you might need to do it yourself. Or even when the database does it for you, you still need to pick the right partition key for the partitioning to be optimal. Let us look at some large datasets and discuss how they might be logically partitioned.

Let us say you are building a twitter like system.

Say 100 million tweets of 140 character per day

100M * ( 280 bytes + 20 bytes for id, timestamp)

30 GB / day

10 TB / year

Need to store 5 years to data

Need to store and query 50 TB

How does twitter work ?

Users follow other users.

When a user connects, we need to show the most recent tweets from the users he follows.

So we need to store about 50 TB of tweets. Given a user, we need to query say the 50 most recent tweets from the users he follows.

Using commodity hardware, 50TB would need say 25 nodes. What key would you use to partition the data ?

Option 1 : hash based partitioning based on user.

To store tweets, a hash of the user is to used to locate the node where the tweet is stored. To query - for each user that the user follows, use the hash to query the node for that users tweets. A problem with this approach is that some users tweet way more than other users. Their nodes are going to be overloaded while others are idle. Unbalanced load.

Option 2 : hash based on randomly generated tweet id

The problem with this approach is the for every query, you have to query every server and aggregate the results. Inefficient for queries.

Option 3: hash based on timestamp

Timestamp is relevant because for each feed request we want the latest tweets. It would be good if tweets are sorted by timestamp. However with such as hash, at any given point in time , one server is overloaded as all the writes are going to the server.

Option 4:

Given the choices, inefficient query (option 2) is more tolerable that unbalanced load (option 1) which could crash some of the servers making the system unavailable. But we also want queries to return the most recent data (sorted by timestamp). So we can improve querying little bit by combining option 2 and 3. Assume a timestamp in epoch time in increments of 1 sec, the tweetid could be timestamp + auto incremented seq. The sequence gives randomness to the tweet id and will give uniform distribution across node.

So given a epoch 1692547708 you will have tweet ids like

1692547708 1

1692547708 2

1692547708 3

1692547708 n

Of course you are wondering how to generate unique tweet ids in a distributed system. That is a topic for another blog.

6.0 Summary

Partitioning data and spreading it across nodes is fundamental to distributed system. Special thought needs to be given to how the partitioned data can be queried efficiently. Hash based partitioning and range based partitioning are two popular strategies. Nodes can fail or additional nodes may need to added to scale. To ensure that load is even across nodes, partitions may be moved between nodes in a process called rebalancing. For best results design your partition keys so that load is distributed evenly and querying is efficient.

7.0 Related Content

CockroachDB Review

CockroachDb Review: Should I use CockroachDb ?

2024-06-30T16:26:00.000-07:00

Overview

CockroachDb is a modern distributed database that promises linear scalability with strict serializability.

Server side sharding is automatic. Nodes can be added easily as needed. Claims to provide the SERIALIZABLE isolation level.

Most distributed databases such as Cassandra, MongoDb, HBase etc sacrifice consistency to achieve high availability. CockroachDb distinguishes itself by claiming to be distributed and the same time offer strong consistency that even single node databases do not offer.

This falls into a database category called NewSql or DistributedSQL as opposed to NoSql (Cassandra, MongoDb)

When to choose CockroachDb ?

You should choose CockroachDb if

Your data is of a global scale.

As data size increases, you need to scale horizontally to several nodes.

You need data to be distributed and localized in specific geographical regions. For
example EU data resides in Europe while US data resides in US.

You need strong consistency. Serializable isolation level.

You need to keep the SQL / relational data model.

You need distributed transactions.

You may want to pass on it if

   You data size can easily fit on a node for the foreseeable future.
   You organization is more comfortable with a stable proven database. (CockroachDb is
   still maturing).
   You data model is heavily normalized and you do a lot of joins in your queries. While this
   database can support joins, it is still not recommended in a highly distributed
   environment.

Architecture

Architecture is based on Google's Spanner paper.

It is a key value store with a SQL interface on top of it.

Database is a cluster of nodes. All nodes are equal. Nodes may join and leave the cluster at any time.

Sorted map of key values pairs. Fully ordered monolithic key space. All tables/indexes go into the same key space by encoding tablename/indexname/key together.

Sharding

Key value pairs are broken up into contiguous ranges.

When range size reaches 512 Mib (2 power 20) It is split into 2 ranges.

Each range is assigned to a node and replicated.

If you have 1 node all the shards are in that node. To scale, you add more nodes and the shards get distributed across nodes. A minimum of 3 nodes is recommended.

Very easily spin up node(s) and add to cluster anywhere.

Btree like index structure used to locate shard that has a key.

Replication

Data in each range is replicated using the Raft consensus algorithm.

A minimum replication factor of 3 is needed.

This provides the high availability. Data is available as long as a majority of the nodes in the cluster are available.

Geo-partitioning

By adding a country or region to the primary key, you can limit storage to keys to a particular region. So European data can be make to reside in Europe, US data in US and so. This has 2 benefits
There is a performance benefit and data is local to its users.

It can satisfy legal requirements where data is not allowed to leave a country or region.

Read/Write

Reads

Any node can receive a request to read a key/value.

Request is forwarded to the node that is the raft leader for that table/range.

Leader returns the data to the node that requested it. Since leader returns the data, no consensus is required.

Node returns it to the client.

Writes

Any node can receive a request to write a key/value.

Request is forwarded to the node that is the raft leader for that table/range.

Leader writes the value to its log and initiates consensus with replicas for the range. When majority acknowledges, the key/value is considered committed and leader notifies the requesting node which notifies the client

Transactions

Supports transactions that spans multiple tables and rows.

Transactions can be distributed, that is span multiple nodes.

The supported isolation level is strict serializability which is the highest isolation level. Strict serializability means that not only are transactions ordered, but they are ordered as per wall clock time.
Transaction protocol is an improvement over two phase commit. In parallel, participants acquire locks and create write intents. The transaction is marked staged. When the client commits, if all locks are acquired and writes are replicated, the coordinator immediately returns success to client. In background the transaction is marked committed. This is one round trip between transaction coordinator and each participant - unlike two phase commit - which requires two round trips.

Hybrid logical clocks are used to timestamp each transaction. Timestamp is the version for MVCC.

Data Model

Clients see only the SQL row column relation model
Wire protocol is same as Postgresql wire protocol.

Performance

Efficient range scan.
Geo partitioning improves performance by locality.
Distributed SQL execution.
Distributed transactions will be slow.
Generally you do not want distributed transactions over large distances. If you build a 3 node CockroachDb cluster with 1 node in NewYork, 1 in London and 1 in San Francisco, the write latencies are going to be very high due to the round trips for RAFT and distributed transactions. The cluster topology needs to be designed appropriately to give you the lowest latency at the desired level of high availability.

Administration

Good command line tools and UI console make the the administration easy.
Since all nodes are equals, number of moving parts that need to be administered is low.

Summary

If you need a globally distributed database with strict serializability, this is definitely a database to look at. It has good pedigree. However remember that distributed databases are not drop in replacement for your traditional RDBMSs. Distributed queries especially joins and distributed transaction can be slow. So some application redesign, some denormalization is always required.

Note: Moved from heavydutysoftware.com

Quick Review: Mysql NDB cluster

2022-02-28T16:04:00.000-08:00

This is a quick 2 min overview of Mysql NDB Cluster. The goal is to help you decide within a minute or two, whether this is an appropriate solution for you.

Cluster of in-memory Mysql databases with a shared nothing architecture.

Consists of Mysql nodes and data nodes.

Mysql nodes are Mysql servers that get data from data nodes. Data nodes hold the data using the NDB storage engine. There are also admin nodes.

NDB nodes serve the data from memory. Data is persisted at checkpoints.

Data is partitioned and replicated.

Up to 48 data nodes and 2 replicas for each fragment of data.

ACID compliant.

READ_COMMITTED isolation level.

Sharding of data is done automatically. No involvement of user or application is required.

Data is replicated for high availability. Node failures are handled automatically.

Clients can access data using NDB Api. Both SQL and NOSQL styles are possible.

This is not a good general purpose database. It is suitable for certain specific use cases of telecom and game but not for general OLTP.

Feels like it has too many moving parts to manage.

High performance -- it is serving data from memory.

Summary

Not a general purpose distributed databases. Unless you are in telecom or gaming or know for sure why this meets your use case, don'nt even think about it.

If you are on Mysql and want high availability, try Mysql InnoDb Cluster, which is much easier to understand and use.

Quick Review: Mysql InnoDb Cluster

2022-02-14T16:27:00.001-08:00

This is a quick 2 min overview of Mysql InnoDb Cluster. The goal is to help you decide within a minute or two, whether this is an appropriate solution for you.

Simple HA solution for Mysql.

Built on top of MySql group replication.

It has 3 Components:

Replication: Uses existing mysql asynchronous replication capabilities. Default is Primary and secondary configuration. Writes go to master which replicates to slaves. Slaves can service reads

Mysql router: Provides routing between your application and the cluster. Supports automatic failover. If the primary dies. The router will redirect writes to the secondary that takes over.

Mysql shell: This is an advance shell that let you code and configure the cluster.

Works best over a local area network. Performance degrades over wide area networks

Easy to setup. Simple commands that are entered on the mysql shell.

var cluster = dba.createCluster('testCluster')

cluster.addInstance('server1@host1:3306')

cluster.addInstance('server2@host2:3306')

cluster.addInstance('server3@host3:3306')

Cluster elects the primary. If you want a particular server to be the primary, you can give it extra weight.

Client do not connect directly to the servers. Rather they connect to the Mysql router that provides the routing as well failover.

MySql InnoDB clusterSet provide additional resiliency by replicating data from a primary cluster to a cluster in another datacenter or location. If the primary cluster becomes available, one of the secondary cluster can become the primary.

Summary

Provides scalability for reads and some HA for Mysql deployments. Simple, easy to use solution. No sharding. Some consistency issues will there when you read from replicas that lag a little bit

References:

https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-innodb-cluster.html

Building Globally Distributed Applications

2020-11-01T06:26:00.000-08:00

A globally distributed application is one where the services and data for the application are partitioned and replicated across multiple regions over the globe. Popular distributed applications that everyone is familiar with are Facebook, Amazon.com, Gmail, Twitter, Instagram. However more and more enterprise applications are finding the need to become distributed because their user base is increasingly distributed around the globe. But not every company has the expertise of a Facebook or Amazon or Google. When going distributed, it is not enough to just spin up instances of your service on AWS or Google cloud on various regions. There are issues related to data that must be addressed for the application to work correctly. While consumer centric social media applications can tolerate some correctness issues or lags in data, the same might not be true for enterprise applications. This blog discusses the data and database issues related to a globally distributed application. Lastly, we discuss 2 research papers that been around since early part of this decade, but whose relevance is increasing in recent times.

Building globally distributed applications that are scalable, highly available and consistent can be challenging. Sharding has to be managed by the application. Keep it highly available requires non database tools. When you have been on a single node database whether it is Mysql or Postgresql etc, it is tempting to scale by manual sharding or one of the clustering solutions available for those databases. It might appear easy at the beginning but the cost of managing the system increases exponentially with scale. Additionally, sharding and replication lead to consistency issues and bugs that need to be addressed. Scaling with single node databases like Mysql beyond a certain point has extremely high operational overhead.

NoSql databases such as Cassandra, Riak, MongoDB etc offer scalability and high availability but at the expense of data consistency. That might be ok for some social media or consumer applications where the dollar value of individual transaction is very small. But not in enterprise applications where the correctness of each transaction is worth several thousands of dollars. In enterprise applications, we need distributed data to behave the same way that we are used to with single node databases.

Let us look at some common correctness issues that crop up with distributed data.

Example 1 : A distributed on line store with servers in San Francisco, New York and Paris.

Each server has 2 tables products and inventory with the following data.
Products:(product)
widget1
widget2
Inventory: (product, count):
widget1,6
widget2,1

Customer Jose connects to server in San Francisco and buys widget2 at time t1. At time t2, Customer Pierre connects to a server in Paris and also buys widget2. Assume t2 > t1 but t2-t1 is small.

Expected Behavior : Jose successfully completes transaction and gets the product. Since inventory of widget2 is now zero, Pierre’s transaction is aborted.
Actual Behavior (in an eventually consistent system): Both transactions complete. But only one of the customers gets the product. The other customer is later sent an apologetic email that widget2 is out of stock.

Example 2: A distributed document sharing system with servers in New York, London, Tokyo

Operation1: In London, User X creates a new empty document marked private.
Operation2. User X makes update 1 to document.
Operation3: User X deletes update 1.
Operation4: User X makes update 2.
Operation5: User X changes the document from private to public.
Due to network issues, only operations 1,2, 5 reach Tokyo. 3 and 4 do not.
In Tokyo, User Y tries to read the shared document.

Expected behavior: The document status is private and Y cannot read the document.
Actual behavior: Y is able to read the document but an incorrect version. The document has update1 which is deleted and is missing update2 which needs to be there.

The problems above are known as consistency issues. Different clients are seeing different views of the data. What is the correct view ?

Consistency here refers to C in the CAP theorem, not the C in ACID. Here Consistency means every thread in a concurrent application correctly reads the most recent write at that point in time.

How do you fix the above issues ? In a single node database, Example1 can be fixed by locking the row in the inventory table during update and Example2 is not even an issue because all the data is in one node. But in a distributed application data might be split across shards and shards replicated for high availability. User of the system might connect to any shard/server and read/write data. With NoSql databases, the application has to handle any in consistencies.

In traditional RDBMSs , database developers are given a knob called isolation level to control what concurrent threads can read. In this old blog I explain what isolation levels are. The safest isolation level is the SERIALIZABLE where the database behaves as if the transactions were executing in a serial order with no overlap, even though in reality they are executing concurrently. Most developers use the default isolation level which is generally READ_COMMITTED OR READ_REPEATABLE. In reality, these isolation levels are poorly documented and implemented differently by different vendors. The result is that in highly concurrent applications, there are consistency bugs even in traditional single node RDBMs. In a distributed database with data spread across shards and replicated for read scalability, the problem is compounded further. Most NoSql vendors punt the problem by claiming eventual consistency, meaning if there are no writes for a while, eventually all reads on all nodes will read the last write.

Consistency is often confused with isolation, which describes how the database behave under concurrent execution of the transactions. At the safest isolation level, the database behaves as if the transactions were executing in serial order, even though in reality they are executing concurrently. At the safest consistency level, every thread in a concurrent application correctly reads the most recent write. But most database documentations are not clear on how to achieve this in an application.

The problems in examples 1 and 2 would not occur if those applications/databases had the notion of a global transaction order with respect to real time. In example 1, Pierre’s transaction at t2 should see the inventory as 0 because a transaction at t1 <t2 set it to zero. In example 2, Y should only be able to read upto operation2 . It should not be able to read operation5 without operations 3,4 which occured before 5.

In database literature, the term for this requirement is called “Strict Serializability” or sometimes “external consistency”. Since this technical definitions can be confusing, it is often referred to as strong consistency.

2 research papers that have been around for a while provide answers on how this problems might be fixed. The papers are the Spanner paper and the Calvin paper.

Their approach is solving the problem can summarized as follows:
1. timestamp transactions with something that reflect their occurrence in real time
2. Order transactions based on timestamp
3. Commit transactions in the above order.

But the details of how they do it are significantly different. Let us look at how they do it.

Spanner paper from Google

Spanner is database built at Google and the paper describes the motivation and design of Spanner. Spanners approach involves
1. The use of atomic clocks and GPS to synchronize clocks across hosts in different regions and the true time API to give accurate time across nodes, regions or continent.
2. For a read/write transaction, spanner calls the true time API to get a timestamp. To address overlaps between transactions that are close to each other, the timestamp is assigned after locks are acquired and before they are released.
3. The commit order equals timestamp order.
4. Read for particular timestamp is sent to any shard/replica that has the data at that timestamp.
5. Read without timestamp (latest read) are serviced by assigning a timestamp.
6. Writes that cross multiple shards use two phase commit.
And of course,
7. It can scale horizontally to 1000s of nodes by sharding.
8. Each shard is replicated.
And most importantly,
9. Even though, it is a key value store, it provide SQL support to make it easy for application programmers.
CockroachDb and Yugabyte are 2 commercial databases based on spanner.

Calvin Paper

The Calvin paper addresses the above problem using distributed consensus protocols like Raft or Paxos.
1. Every transaction has to first go through distributed consensus and secure a spot in a linear replication log.
2. One can view the index in the log as the timestamp.
3. The committed entries in the replication log are then executed in the exact same serial order by every node in the distributed database.
4. Since the transaction log is replicated to every shard, it does not need or use two phase commit. In a transaction involving multiple shards, if a shard dies before committing a particular transaction, then on restart it just has to execute the uncommitted transaction from it replication log.
5. No dependency on wall clocks or time API.
6. No two phase commit.
7. No mention of SQL support.

FaunaDb is an example of a database based on Calvin.

This class of databases that offer horizontal scalability on a global scale without sacrificing consistency is also called NewSql.

In summary, if you are a building a globally distributed application that needs strong consistency, doing it on your own with SQL or NoSql database can be non trivial. Consistency is hard enough in a single node database. But on a distributed database, consistency bugs are harder to troubleshoot and even harder to fix. You might want to consider one of the NewSql databases to make life easier. Review the Spanner and Calvin papers to understand the architectural choices that are available. This will help you pick a database that is right for you. Spanner and Calvin papers have been around for almost a decade. But they have become more relevant now as real databases based on them become more popular. Most importantly understand what is consistency is and apply it, for lack of which can cause severe correctness bugs in your application.

References:

The Spanner paper

The Calvin paper

Consistency and Isolation

A Microservices Introduction

2019-01-20T18:31:00.000-08:00

Modern distributed applications are built as a suite of microservices. In this blog we discuss the characteristics of microservices. We will also compare microservices to its predecessors like SOA and monolithic applications. We point out the benefits and downsides of a microservices architecture.

1.0 Introduction

Let us start with a little bit of history and go back to late 90s or early 2000's. Web applications were monolithic. A single web container would serve the entire application. Even worse, a single web container would serve multiple applications. Not only was this not scalable, it was a development and maintenance nightmare. A single bug could bring multiple applications down. And there was an ownership issue. You had multiple teams/developers contributing code. When there was a bug, the ownership was not clear and bugs would bounce around among developers.

Around mid 2000's the new buzz word was service oriented architecture SOA. This was promoted by large web application server companies. See my blog on SOA written in 2010. SOA encouraged number of good design philosophies such as interface based programming, loosely coupled applications, asynchronous interaction. REST, XML,JSON and messaging platforms enabled SOA. SOA was a big improvement, but the tools and deployment technologies were still heavyweight.

The microservices architecture is the next step in evolution further improving the ideas from SOA.

Many dismiss microservices as another buzz word. But having developed real world applications using tools listed in section 3.0, I see real value and benefit in this architecture.

2.0 Description

The main idea around microservices is that large complex systems are easier to build, maintain and scale using independently built and owned smaller services that work together.

Each microservice is a modular fine grained application providing a specific service. Let us say you have an application that has a UI , authentication, Apis for customer info, Apis for uploading documents, Apis for analytics. You may have a microservice for the UI, a microservice for customer apis, a document upload microservice, an analytics microservice.

A microservice is fully functional.

A microservice performs one specific business or IT function.

The development of the microservice can be done independently.

A microservice runs as its own process.

A microservice communicates using common protocols such as REST/Http.

A microservice offers services via its Apis. It can communicate with other microservices using their APIs.

A microservice is deployable to production independently.

When your application has multiple microservices, each could be developed in a different or the most suitable programming language or framework suited for that service.

A microservice should scale horizontally by just running more instances of the microservice.

Testing, bug fixing, performance tuning etc on the microservice should happen independently without affecting other microservices.

The above listed characteristics make it easier to build large complex systems.

3.0 Enabling Technologies

A number of newer frameworks have made building microservices easier.

For Java programmers, Dropwizard and SpringBoot are very useful frameworks for building microservices. The old way was monolithic application servers like websphere, weblogic , jboss etc. Dropwizard and SpringBoot turn the table by embedding the web server within your java application. Development is much easier as you are developing a plain java application with a main method. The entire microservice is packaged in one jar and can be run with the java -jar command. For additional information, please read my blog comparing Dropwizard to Tomcat. For Javascript, python and other languages there are similar frameworks.

To start with microservices, a framework as mentioned above is all you need. Once you have developed and use several microservices, the following platforms may be useful.

Docker is a containerization technology that makes it easier to manage production deployments. This is of interest for a dev-ops person who has to roll out services to production.

Kubernetes is platform for automation, deployment and scaling of containerized applications.

4.0 Disadvantages

For smaller business and smaller applications, the overhead of many microservices could be a problem. If your infrastructure is one or two $20 per month VMs on AWS (or other cloud providers) you will not have enough memory/cpu/disk for multiple microservices.

The increased network communication is a cost.

Each microservice is its own process. The remote calls have a serialization/deserialization cost.

5.0 Conclusion

Microservices are a logical next step in the evolution of the development of complex applications.
They are a best practice. But they are not a silver bullet that solve every problem.

Apache kafka Streams

2018-12-02T08:23:00.000-08:00

Apache Kafka is a popular distributed messaging and streaming open source system. A key differentiator for Kafka is that its distributed broker architecture makes it highly scalable. Earlier versions of Kafka were more about messaging. I have a number of blogs on Kafka messaging some of which are listed below in the related blogs section.

This blog introduces Kafka streams which builds on messaging.

1.0 Introduction

In a traditional Kafka producer/consumer application, producers write messages to a topic and consumers consume the messages. The consumer may process the message and then write it to a database , filesystem or even discard it. For a consumer to write the message back to another topic, it has to create a producer.

Kafka streams is a higher level library that lets you build a processing pipeline on streams of messages where each stream processor reads a message, does some analytics such as counting, categorizing , aggregation etc and then potentially writes a result back to another topic.

2.0 Use Cases

Analytics from e-commerce site usage.

Analytics from any distributed application.

Distributed processing of any kind of event or data steams.

Transforming from monolithic to micro-services architecture.

Moving away from database intensive architectures.

3.0 When to use Kafka streams ?

If yours is a traditional messaging application, where you need the broker to hold on to messages till they get processed by a consumer, then the producer/consumer framework might be suitable. Here kafka competes with ActiveMQ, Websphere MQ and other traditional message brokers. Here the processing of each message is independent of other messages.

If yours is a analytics style application, where you have do different forms of counting, aggregation, slicing /dicing on a stream of data, then Kafka streams might be an appropriate library. Here the processing is for a set of messages in the stream. In this space. Kafka competes with analytics frameworks like Apache Spark, Storm, splunk etc.

If you were to use producers/consumers for an analytics style application, you would end up creating many producers/consumer, you would probably have to read and write a database several time, you would need to maintain in memory state and probably use a third party library for analytics primitives. Kafka streams library makes all this easier for you.

4.0 Features

Some key features of Kafka streams are:

Provides an API for stream processing primitives such counting, aggregation, categorization etc. API supports timing windows.

Message processing is one at the time. In producer/consumer , messages are generally processed in batches.

Fault tolerant local state is provided by the library. In consumers, any state has to be managed by the application.

Supports exactly once or once and only once message message delivery. In producer/consumer, it is at least once delivery.

No need to deal will lower level messaging concepts like partitions, producers, consumers, polling.

Self contained complete library that handles both messaging and processing. No need for other third party libraries like Spark.

5.0 Concepts

A stream is an unbounded sequence of Kafka messages on a topic.

A stream processor is a piece of code that gets a message, does some processing on it, perhaps stores some in memory state and then writes to another topic for processing by another processor. This is also known as a node.

A stream application is a set of processors where the output of one processor is further processed by one or more other processors. A stream application can depicted as graph with the processors as vertexes and streams/topics as edges.

A source processor is the first node in the topology. It has no upstream processors and gets messages from a topic. A sink processor has no downstream processors and will typically write a result somewhere.

The figure below shows a sample application topology

6.0 Programming model

2 core programming models. Below are some sample code snippets.

6.1 Streams DSL

This is a higher level API build on top of the processor API. Great for beginners.

KStream models the stream of messages. KTable is the in-memory story. You can convert from stream to table and vice versa.

Example: Simple analytics on a stream of pageviews from e-commerce site

// consume from a topic

StreamBuilder builder = new StreamBuilder() ;
KStream pageViewlines = builder.stream("someTopic") ;

// From each line extract productid and create a table key=productid,value=count
// We get page view count by product

KTable productCounts = pageViewlines.flatMapValues(value->getProduct(value))
.groupBy((key,value)->value)
.count() ;

// write the running counts to another topic or storage

productCounts.toStream.to("productCountsTopic",Produced.with(serdes.String(),serdes.Long()) ;

6.2 Processor API

same example using processor API

public class ProductFromPageViewProcessor implements Processor {

private KeyValueStore pcountStore ;

// Do any initialization here
// such as loading stores or scheduling punctuate
public void init(ProcessorContext context) {

// get the store that will store counts
pcountStore = (KeyValueStore)context.getStateStore("pcounts") ;

// schedule a punctuate to to periodically send the product counts to a downstream processor
// every 5 secs

context.schedule(5000,PunctuationType.STREAM_TIME,(timestamp)->{

// iterate over all values in the pcountStore
KeyValueIter iter = pCountStore.all()
while(iter.hasNext()) {
KeyValue val = iter.next() ;
context.forward(val.key,val.value) ;
}

context.commit() ;

} ;

// Called once for every line or message on the consumed topic
public void process(String k, String line) {
String productId = getProductId(line) ;
Long count = pCountStore.get(productId) ;
if (count == null) {
pCountStore.put(productId,1) ;
} else {
pCountStore.put(productId,count+1) ;
}
}

}
}

7.0 Conclusion

As you can see from both API examples, it is about processing streams of data, doing some analytics and producing results. No need to poll or deal with lower level details like partitions and consumers. The streams model moves you away from legacy database intensive architectures, where data is written to a database first and then slow inefficient queries try to do analytics.

Some disadvantages of Kafka Streams are:

You are tied to Kafka and have to go through a Kafka topic. Other Streaming libraries like Spark are more generic and might have more analytics features.

There are no ways to pause and resume a stream. If load suddenly increases or you want to pause the system for maintenance, there is no clean mechanism. In producer/consumer, there is explicit pause/resume API. Other streaming libraries also have some flow control mechanisms.

8.0 Related Blogs

8.1 Apache Kafka Introduction

8.2 Apache kafka Basic tutorial

8.3 Apache Kafka once and once delivery