My Public Notepad

How to install and setup Claude Code on MacOS + VS Code

2026-04-10T16:21:00.003+01:00

Let's follow steps from Quickstart - Claude Code Docs:

% curl -fsSL https://claude.ai/install.sh | bash

Setting up Claude Code...

✔ Claude Code successfully installed!

Version: 2.1.100

Location: ~/.local/bin/claude

Next: Run claude --help to get started

⚠ Setup notes:

• Native installation exists but ~/.local/bin is not in your PATH. Run:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc

✅ Installation complete!

Let's add path to bin to PATH, add it to zsh config and reload it:

% echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc

If you use Bash:

source ~/.bashrc

Verification:

$HOME/.local/bin is now in $PATH:

% echo $PATH

/Users/bojan/.local/bin:....

Let's check Claude version:

% claude --version

2.1.100 (Claude Code)

Let's also see its CLI arguments:

% claude --help

Usage: claude [options] [command] [prompt]

Claude Code - starts an interactive session by default, use -p/--print for non-interactive output

Arguments:

prompt Your prompt

Options:

--add-dir <directories...> Additional directories to allow tool access to

--agent <agent> Agent for the current session. Overrides the 'agent' setting.

--agents <json> JSON object defining custom agents (e.g. '{"reviewer": {"description": "Reviews code", "prompt": "You are a code

reviewer"}}')

--allow-dangerously-skip-permissions Enable bypassing all permission checks as an option, without it being enabled by default. Recommended only for

sandboxes with no internet access.

--allowedTools, --allowed-tools <tools...> Comma or space-separated list of tool names to allow (e.g. "Bash(git:*) Edit")

--append-system-prompt <prompt> Append a system prompt to the default system prompt

--bare Minimal mode: skip hooks, LSP, plugin sync, attribution, auto-memory, background prefetches, keychain reads, and

CLAUDE.md auto-discovery. Sets CLAUDE_CODE_SIMPLE=1. Anthropic auth is strictly ANTHROPIC_API_KEY or apiKeyHelper via

--settings (OAuth and keychain are never read). 3P providers (Bedrock/Vertex/Foundry) use their own credentials.

Skills still resolve via /skill-name. Explicitly provide context via: --system-prompt[-file],

--append-system-prompt[-file], --add-dir (CLAUDE.md dirs), --mcp-config, --settings, --agents, --plugin-dir.

--betas <betas...> Beta headers to include in API requests (API key users only)

--brief Enable SendUserMessage tool for agent-to-user communication

--chrome Enable Claude in Chrome integration

-c, --continue Continue the most recent conversation in the current directory

--dangerously-skip-permissions Bypass all permission checks. Recommended only for sandboxes with no internet access.

-d, --debug [filter] Enable debug mode with optional category filtering (e.g., "api,hooks" or "!1p,!file")

--debug-file <path> Write debug logs to a specific file path (implicitly enables debug mode)

--disable-slash-commands Disable all skills

--disallowedTools, --disallowed-tools <tools...> Comma or space-separated list of tool names to deny (e.g. "Bash(git:*) Edit")

--effort <level> Effort level for the current session (low, medium, high, max)

--exclude-dynamic-system-prompt-sections Move per-machine sections (cwd, env info, memory paths, git status) from the system prompt into the first user

message. Improves cross-user prompt-cache reuse. Only applies with the default system prompt (ignored with

--system-prompt). (default: false)

--fallback-model <model> Enable automatic fallback to specified model when default model is overloaded (only works with --print)

--file <specs...> File resources to download at startup. Format: file_id:relative_path (e.g., --file file_abc:doc.txt file_def:img.png)

--fork-session When resuming, create a new session ID instead of reusing the original (use with --resume or --continue)

--from-pr [value] Resume a session linked to a PR by PR number/URL, or open interactive picker with optional search term

-h, --help Display help for command

--ide Automatically connect to IDE on startup if exactly one valid IDE is available

--include-hook-events Include all hook lifecycle events in the output stream (only works with --output-format=stream-json)

--include-partial-messages Include partial message chunks as they arrive (only works with --print and --output-format=stream-json)

--input-format <format> Input format (only works with --print): "text" (default), or "stream-json" (realtime streaming input) (choices:

"text", "stream-json")

--json-schema <schema> JSON Schema for structured output validation. Example:

{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]}

--max-budget-usd <amount> Maximum dollar amount to spend on API calls (only works with --print)

--mcp-config <configs...> Load MCP servers from JSON files or strings (space-separated)

--mcp-debug [DEPRECATED. Use --debug instead] Enable MCP debug mode (shows MCP server errors)

--model <model> Model for the current session. Provide an alias for the latest model (e.g. 'sonnet' or 'opus') or a model's full name

(e.g. 'claude-sonnet-4-6').

-n, --name <name> Set a display name for this session (shown in /resume and terminal title)

--no-chrome Disable Claude in Chrome integration

--no-session-persistence Disable session persistence - sessions will not be saved to disk and cannot be resumed (only works with --print)

--output-format <format> Output format (only works with --print): "text" (default), "json" (single result), or "stream-json" (realtime

streaming) (choices: "text", "json", "stream-json")

--permission-mode <mode> Permission mode to use for the session (choices: "acceptEdits", "auto", "bypassPermissions", "default", "dontAsk",

"plan")

--plugin-dir <path> Load plugins from a directory for this session only (repeatable: --plugin-dir A --plugin-dir B) (default: [])

-p, --print Print response and exit (useful for pipes). Note: The workspace trust dialog is skipped when Claude is run with the

-p mode. Only use this flag in directories you trust.

--remote-control-session-name-prefix <prefix> Prefix for auto-generated Remote Control session names (default: hostname)

--replay-user-messages Re-emit user messages from stdin back on stdout for acknowledgment (only works with --input-format=stream-json and

--output-format=stream-json)

-r, --resume [value] Resume a conversation by session ID, or open interactive picker with optional search term

--session-id <uuid> Use a specific session ID for the conversation (must be a valid UUID)

--setting-sources <sources> Comma-separated list of setting sources to load (user, project, local).

--settings <file-or-json> Path to a settings JSON file or a JSON string to load additional settings from

--strict-mcp-config Only use MCP servers from --mcp-config, ignoring all other MCP configurations

--system-prompt <prompt> System prompt to use for the session

--tmux Create a tmux session for the worktree (requires --worktree). Uses iTerm2 native panes when available; use

--tmux=classic for traditional tmux.

--tools <tools...> Specify the list of available tools from the built-in set. Use "" to disable all tools, "default" to use all tools,

or specify tool names (e.g. "Bash,Edit,Read").

--verbose Override verbose mode setting from config

-v, --version Output the version number

-w, --worktree [name] Create a new git worktree for this session (optionally specify a name)

Commands:

agents [options] List configured agents

auth Manage authentication

auto-mode Inspect auto mode classifier configuration

doctor Check the health of your Claude Code auto-updater. Note: The workspace trust dialog is skipped and stdio servers from

.mcp.json are spawned for health checks. Only use this command in directories you trust.

install [options] [target] Install Claude Code native build. Use [target] to specify version (stable, latest, or specific version)

mcp Configure and manage MCP servers

plugin|plugins Manage Claude Code plugins

setup-token Set up a long-lived authentication token (requires Claude subscription)

update|upgrade Check for updates and install if available

And finally, let's launch it:

% claude

Welcome to Claude Code v2.1.100

…………………………………………………………………………………………………………………………………………………………

* █████▓▓░

* ███▓░ ░░

░░░░░░ ███▓░

░░░ ░░░░░░░░░░ ███▓░

░░░░░░░░░░░░░░░░░░░ * ██▓░░ ▓

░▓▓███▓▓░

* ░░░░

░░░░░░░░

░░░░░░░░░░░░░░░░

█████████ *

██▄█████▄██ *

█████████ *

…………………█ █ █ █………………………………………………………………………………………………………………

Let's get started.

Choose the text style that looks best with your terminal

To change this later, run /theme

❯ 1. Dark mode ✔

2. Light mode

3. Dark mode (colorblind-friendly)

4. Light mode (colorblind-friendly)

5. Dark mode (ANSI colors only)

6. Light mode (ANSI colors only)

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

1 function greet() {

2 - console.log("Hello, World!");

2 + console.log("Hello, Claude!");

3 }

Syntax theme: Monokai Extended (ctrl+t to disable)

After that we need to select a login method:

❯ 1. Claude account with subscription · Pro, Max, Team, or Enterprise

2. Anthropic Console account · API usage billing

3. 3rd-party platform · Amazon Bedrock, Microsoft Foundry, or Vertex AI

Option 1 - Claude Accounts are for the consumer/pro web interface (claude.ai) which is seat-based.

Option 2 - Anthropic Console account should be selected if your organization is on an API plan (pay-as-you-go billing based on token usage). Anthropic Console (platform.claude.com) is the hub for managing API keys, billing, and developer organizations.

Option 3 - 3rd-party platforms are only for when you want to route Claude's "brain" through your existing AWS (Bedrock) or Google Cloud (Vertex) bills.

After selecting Anthropic Console, you'll be taken to page which shows the following:

Claude Code would like to connect to your Anthropic organization MYORG

YOUR ACCOUNT WILL BE USED TO:

Generate API keys on your behalf
Access your Anthropic profile information
Upload files on your behalf

Logged in as user@myorg.com
Switch account

After clicking on Authorize button, you'll be redirected to a page which shows:

Build something great
You’re all set up for Claude Code.

You can now close this window.

Back in terminal, you'll see:

Logged in as user@myorg.com

After pressing Enter:

Security notes:

1. Claude can make mistakes

You should always review Claude's responses, especially when

running code.

2. Due to prompt injection risks, only use it with code you trust

For more details see:

https://code.claude.com/docs/en/security

Press Enter to continue…

After clicking on Enter:

Use Claude Code's terminal setup?

For the optimal coding experience, enable the recommended settings

for your terminal: Shift+Enter for newlines

❯ 1. Yes, use recommended settings

2. No, maybe later with /terminal-setup

Enter to confirm · Esc to skip

After choosing 1 - recommended settings:

Accessing workspace:

/Users/bojan/path/to/project

Quick safety check: Is this a project you created or one you trust? (Like your own code, a well-known open source project, or work from your team). If not, take a

moment to review what's in this folder first.

Claude Code'll be able to read, edit, and execute files here.

Security guide

❯ 1. Yes, I trust this folder

2. No, exit

Enter to confirm · Esc to cancel

After selecting 1:

╭─── Claude Code v2.1.100───────────────────────────────────────────────────────────────────────────────────────╮

│ │ Tips for getting started

│ Welcome back User! │ Run /init to create a CLAUDE.md file with instructions for Claude│

│ │ ─────────────────────────────────────────────────────────────────│

│ ▐▛███▜▌ │ Recent activity │

│ ▝▜█████▛▘ │ No recent activity │

│ ▘▘ ▝▝ │ |

│ │ │

│ Sonnet 4.6 · API Usage Billing · MYORG │

│ ~/…/path/to/project │

╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────

❯

? for shortcuts ● high · /effort

We can now run various commands, like:

────────────────────────────────────────────────────────────

❯ /stats

────────────────────────────────────────────────────────────

/stats Show your Claude Code usage statistics and activity

/status Show Claude Code status including version, model, account, API connectivity, and tool statuses

/statusline Set up Claude Code's status line UI

/ide Manage IDE integrations and show status

If we execute /stats at this point, the output will show:

❯ /stats

────────────────────────────────────────────────────────────

Status Config Usage Stats

No stats available yet. Start using Claude Code!

In my case Status tab showed, among other things:

IDE: ✘ Error installing VS Code extension: 1: Command failed with ERR_STREAM_PREMATURE_CLOSE: code --force --install-extension anthropic.claude-code

Premature close

Please restart your IDE and try again.

I restarted VS Code to no avail. I then manually installed Claude Code for VS Code plugin and restarted VD Code but the same error appeared again. There is a related bug, still with Open status: [BUG] Claude code VS Code extension error in MacOS · Issue #34639 · anthropics/claude-code

If we try /cost:

❯ /stats

⎿ Status dialog dismissed

❯ /cost

⎿ Total cost: $0.0000

Total duration (API): 0s

Total duration (wall): 1h 16m 21s

Total code changes: 0 lines added, 0 lines removed

Usage: 0 input, 0 output, 0 cache read, 0 cache write

Model Context Protocol (MCP)

2026-04-08T12:36:00.010+01:00

Model Context Protocol (MCP)

The Model Context Protocol (MCP):

Open-source standard
Enables AI models to seamlessly connect with external data sources, tools, and software systems
Acts as a universal "USB-C port" for AI, allowing LLMs to securely access local files, databases, and APIs to enhance context-aware responses.
Introduced by Anthropic in late 2024

Key Aspects of MCP:

Purpose: Replaces fragmented, custom integrations with a single, open standard, making it easier to connect AI assistants to enterprise data, tools, and development environments.
Components: Consists of:

MCP Clients
MCP Hosts - AI apps like Claude or coding agents
MCP Servers - programs that bridge specific data sources

Security: MCP supports secure, two-way connections, allowing developers to control exactly what data is exposed to the AI.
Functionality: Enables models to read files, query databases, use search engines, and call external APIs, providing live, relevant context for tasks.
Open Standard: Hosted by the Linux Foundation, the protocol is designed for broad industry adoption.

MCP differs from RAG (Retrieval-Augmented Generation) by focusing on active, two-way interaction with systems, whereas RAG is focused on retrieving documents for context.

For developers, it provides SDKs in Python and TypeScript.

MCP clients

MCP clients are the components within AI applications (AI Hosts) that manage one-to-one connections with MCP servers, translating AI requests into protocol-standardized messages.

Popular MCP Client Applications

Several major AI-powered platforms and editors have integrated MCP client support to allow users to pull in their own tools and context:

Claude Desktop: Anthropic’s flagship app provides a built-in interface for managing local and remote MCP servers (e.g., Google Drive, Slack, GitHub).
Cursor: An AI-native code editor that uses MCP to give its internal AI models direct access to project files, local databases, and custom developer tools.
Windsurf Editor: A developer environment that supports tool invocation through MCP servers, allowing it to seamlessly interact with external scripts and APIs during coding sessions.
Visual Studio Code (Agent Mode): Developers can use extensions to register MCP servers, enabling chat assistants to interact with internal enterprise tools directly within the editor.
JetBrains IDEs: Platforms like IntelliJ IDEA feature an MCP-client UI where users can paste server configurations to bring external tool catalogues into the AI Assistant pane.
BeeAI: An open-source desktop AI assistant from IBM that supports tool integration via built-in or custom MCP servers.

Core Client Features

In the MCP architecture, clients don't just consume data; they provide specific features that enable complex, "agentic" workflows:

Sampling: Allows a server to request that the client (and its LLM) generate a completion, enabling the AI to "ask back" for clarification or more information.
Elicitation: Provides a structured way for servers to ask users for specific information (like a password or preference) through the client’s UI.
Roots: Allows the client to define specific file system boundaries, telling servers which directories they are permitted to access for safety and context scoping.

Developer SDKs for Building Clients

If you are building your own AI application, you can implement an MCP client using official software development kits:

TypeScript/Node.js SDK: The most comprehensive SDK for building web-based or local Node.js clients using standard JSON-RPC 2.0.
Python SDK: Ideal for data science or automation workflows, allowing easy integration with Python-based AI agents.

MCP Hosts

Model Context Protocol (MCP) hosts are AI applications or environments that manage, connect, and run MCP clients to interact with external data sources (servers). Key examples include Claude Desktop, Claude Code, Zed, Replit, Cursor, and Sourcegraph.

Key Roles of MCP Hosts:

Orchestration: Act as the container for AI models to coordinate and communicate with multiple MCP servers.
Security & Permissions: Manage user consent and enforce security policies (e.g., in sandboxed iframes) to control what data servers can access.
Workflow Management: Aggregate user prompts, conversation state, and tool results from various sources to enable seamless interaction.
Session Control: Initialize, maintain, and terminate connections between LLMs and tools via the MCP client.

Primary MCP Host Examples:

Claude Desktop: Anthropic's desktop application is a primary implementation of an MCP host, allowing the Claude model to utilize local tools and data.
Claude Code: An AI agentic coding tool that acts as a host to interact with development-focused MCP servers.
AI IDEs (Zed, Cursor, Replit): Integrated Development Environments that use MCP to connect coding assistants with files, documentation, and databases.
Platforms & Agents: Other platforms like Workato or custom agents that orchestrate data access via the protocol.

What's the difference between mcp host and mcp client?

In the Model Context Protocol (MCP), the Host is the overall application you interact with, while the Client is the specific component inside that application that handles the technical connection to a server.

Key Differences at a Glance

What it is

MCP Host: The end-user application (e.g., Claude Desktop, Cursor).
MCP Client: A protocol handler or "adapter" inside the host.

Primary Role

MCP Host: Manages the user interface and coordinates multiple clients.
MCP Client: Maintains a 1:1 connection with a single MCP server.

Responsibility

MCP Host: Security policies, user consent, and aggregating data for the AI model
MCP Client: Translating protocol messages (JSON-RPC) between the host and server.

Hierarchy

MCP Host: A single Host can contain multiple Clients.
MCP Client: A Client is a subsidiary of the Host.

The "Restaurant" Analogy

To make it simpler, imagine a restaurant setting:

The Host is the Executive Chef: They decide what needs to be cooked and oversee everything, but they don't leave the kitchen to buy ingredients.
The Client is the Waiter: They take the Chef's specific order, run to the source (the Server), and bring back exactly what was requested in a format the Chef can use.

Why the distinction matters

While you will often hear people refer to applications like Claude Desktop as "the client," technically they are hosts. This architecture allows one app to connect to many different data sources (like Google Drive, Slack, and local files) simultaneously by instantiating a separate client for each one.

What is a MCP Client in Claude Desktop?

In Claude Desktop, the MCP client is the internal software layer that allows the app to "talk" to the tools you've added. While you might call the whole app "the client," it actually functions as a host that manages multiple individual client connections.

How it works in Claude Desktop

The Translator: When you ask Claude to "read a file," the client translates that human request into a technical JSON-RPC message that the Filesystem server understands.
The Connection Manager: Claude Desktop can run several clients at once. For example, one client might be connected to a GitHub server while another is connected to a Google Drive server.
Permission Gatekeeper: The client facilitates the security handshake. Before a tool executes, the client triggers the UI popup in Claude Desktop asking for your explicit permission.

How to see them

You can see your active MCP clients and their available tools by clicking the "hammer" or "plug" icon (the MCP server indicator) in the bottom-right corner of the chat input box.

Configuration

Claude Desktop's clients are configured via a local JSON file (claude_desktop_config.json). This file tells the internal clients exactly how to launch and communicate with your servers.

Config File Location:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

MCP Servers

Model Context Protocol (MCP) servers:

Lightweight programs that connect AI models (like Claude or ChatGPT) to external data sources and tools, such as local files, databases, GitHub, or Slack
Provide a standardized interface, enabling AI agents to securely access, read, and manipulate data beyond their training sets.

Key Aspects of MCP Servers:

Functionality: They expose specific capabilities—resources, prompts, and tools—to AI applications.
Use Cases: Common implementations include file system access for documentation, database querying, and API interactions for services like GitHub or Google Tasks.
Security: They provide controlled, authorized access to local or remote resources, with user permission required for actions.
Architecture: As part of the Model Context Protocol, they act as the "server" in a client-server model, connecting to "hosts" like desktop apps.

Common MCP Server Examples:

Local File System: Allows AI to read, write, and organize local documents.
GitHub/GitLab: Enables AI to manage repositories, create issues, and pull code.
Database/API Connectors: Connects AI to SQL databases, HubSpot CRM, or AWS services.
Developer Tools: Includes servers for Terraform, Angular CLI, and Home Assistant.

You can build your own MCP server using Python or TypeScript, often utilizing tools like uv for environment setup.

How to start using Agentic AI in DevOps and Platform Engineering

2026-04-08T12:18:00.001+01:00

The next frontier of DevOps and Platform Engineering is Agentic AI. We need to learn how autonomous agents reason and adapt to reduce cognitive load and accelerate the SDLC as we want to move beyond simple automation to build self-optimizing ecosystems that scale with confidence, innovation, and enterprise governance. 

We should be able to:

Explain the shift from automation to agentic AI and articulate what makes an AI system truly
“agentic”
Design agent-aware workflows in GitHub Actions, integrating LLMs with events, logs, APIs, and quality gates to create intelligent CI/CD pipelines
Build AI-powered diagnostic loops that ingest failure context, reason about root causes, and generate structured remediation proposals or self-healing fixes
Implement intelligent release decisions using multi-signal quality gates (test coverage, performance, security, cost) and generate auditable release rationale reports
Deploy our own end-to-end platform engineering agent, capable of diagnosing pipeline failures, evaluating release readiness, and autonomously opening a fix PR or escalating with structured context.

Learning while Doing

Identify Platform engineering pain points and the AI opportunity

How can we get from static scripts and CI/CD automation to agentic AI
Make a comparison of manual vs. AI-driven diagnosis
Understand how platform engineering is evolving from static automation toward AI-driven systems that proactively diagnose and resolve operational issues

Agentic AI fundamentals - how agents reason and act?

Learn about core agent components (LLMs, memory, and tools)
Compare event-driven vs. polling architectures
Balance autonomous actions with human oversight
Understand how agentic systems combine reasoning, memory, and tools to perceive events, make decisions, and act within engineering workflows

How to setup the environment and create our first agentic workflow

Set up an agentic runtime that responds to CI/CD events
Connect an AI agent to our pipeline's event stream and context
Trigger our first agent run and interpret its reasoning logs
Learn how to connect AI agents to CI/CD events and platform context to trigger automated reasoning and actions in real time

AI-powered diagnosis and remediation

Compare manual vs. AI-driven incident diagnosis
Build agents that read logs, reason about failures, and propose fixes
Define escalation boundaries: when the agent self-heals vs. asks a human
Understand how AI agents analyze logs, diagnose failures, and determine whether to self-heal or escalate issues to humans

Intelligent CI/CD & adaptive delivery

How to move beyond pass/fail pipelines to AI-driven release decision
Automate rollback decisions using AI quality gates
Query pipeline state and release history using natural language
How AI transforms CI/CD pipelines into adaptive systems that make context-aware release and rollback decisions

Operational intelligence & conversational observability

Replace complex dashboards with AI anomaly detection
Check platform health via chat interfaces
Shift from reactive alerts to predictive management
Understand how AI enables conversational access to platform health and detects anomalies to support proactive operations.

Multi-agent coordination & implementation strategy

Architect multi-agent systems for our platform workflows
Handle agent conflicts, failures, and graceful degradation
Design a phased enterprise rollout with guardrails and audit trails

Build our platform engineering agent

Learn how to design coordinated multi-agent systems that handle complex platform workflows with governance and reliability
Wire together diagnosis, quality gates, and observability into one agent pipeline
Implement self-healing PRs with confidence thresholds
Shift our role from platform operator to AI supervisor
Learn how to combine diagnosis, delivery intelligence, and observability into a unified agent that automates key platform workflows

---

Kubernetes StatefulSet

2026-04-02T11:08:00.006+01:00

In Kubernetes, a StatefulSet is a specialized workload API object designed to manage stateful applications. Unlike standard Deployments, where Pods are interchangeable "cattle," StatefulSets treat Pods as unique "pets" with a persistent identity that is maintained even if they are rescheduled or restarted.

Key Features

Stable Network Identity: Each Pod is assigned a unique, ordinal index (e.g., web-0, web-1) and a corresponding stable DNS name through a Headless Service.
Stable Storage: By using volumeClaimTemplates, each Pod is automatically paired with its own PersistentVolume. If a Pod dies, the replacement Pod with the same identity will automatically remount the same storage.
Ordered Deployment: Pods are created and scaled sequentially from 0 to N-1. Kubernetes ensures that the previous Pod is "Running and Ready" before starting the next one.
Ordered Termination: Scaling down or deleting the StatefulSet occurs in reverse order, starting from the highest ordinal (e.g., web-2 is deleted before web-1).

When to Use StatefulSets

StatefulSets are the standard choice for applications that require consistent data and unique identities, such as:

Databases: Systems like MySQL, PostgreSQL, MongoDB, and Cassandra.
Distributed Systems: Tools like ZooKeeper, Kafka, and Elasticsearch that need a quorum or master election.
Clustered Applications: Any software where instances need to know each other’s specific addresses to sync data.

Comparison: StatefulSet vs. Deployment

Feature StatefulSet Deployment

---------- -------------- ---------------

Pod Identity: Unique and stable (ordinal names) Randomly generated and ephemeral

Storage: Dedicated volume per Pod (via template) Typically shared or transient

Network: Fixed DNS per Pod (via Headless Service) Single Load Balancer for the whole set

Scaling: Sequential (0, then 1, then 2...) Parallel (multiple Pods at once)

Best Practices

Use Headless Services: Always pair our StatefulSet with a Service that has clusterIP: None to ensure Pods are individually addressable.
Persistent Storage: Ensure our StorageClass is correctly configured for dynamic provisioning so that each Pod gets its own disk automatically.
Manual Data Sync: Note that while Kubernetes manages the infrastructure, we are still responsible for configuring internal application logic like data replication or master/slave sync.

YAML manifest example for a basic MySQL StatefulSet

Below is a standard YAML manifest for a MySQL StatefulSet. It includes a Headless Service for network identity and a volumeClaimTemplate to automatically provision unique storage for each replica.

apiVersion: v1

kind: Service

metadata:

labels:

app: mysql

spec:

ports:

- port: 3306

clusterIP: None # Defines this as a Headless Service

selector:

app: mysql

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

spec:

selector:

matchLabels:

app: mysql

serviceName: "mysql"

replicas: 3

template:

metadata:

labels:

app: mysql

spec:

containers:

- name: mysql

image: mysql:8.0

env:

- name: MYSQL_ROOT_PASSWORD

value: "password" # Use Secrets in production!

ports:

- containerPort: 3306

volumeMounts:

- name: mysql-data

mountPath: /var/lib/mysql

volumeClaimTemplates:

- metadata:

spec:

accessModes: [ "ReadWriteOnce" ]

resources:

requests:

storage: 1Gi

Why this works:

Stable DNS: Each Pod gets a predictable name: mysql-0.mysql, mysql-1.mysql, etc.
Unique Storage: Kubernetes creates three separate PersistentVolumeClaims. mysql-0 will always mount the first disk, even after a reboot.
Ordered Startup: Pods launch one after another (0, then 1, then 2), which is critical for forming database clusters.

To use a Kubernetes Secret (like mysql-secret) instead of hardcoding passwords, we need to create a Secret object and then reference it in our StatefulSet. This is the standard practice for distributing credentials securely in Kubernetes.

1. Create the Secret

We can define our password in a YAML file. Note that values in the data field must be base64 encoded.

apiVersion: v1

kind: Secret

metadata:

type: Opaque

data:

# 'password' encoded in base64 is 'cGFzc3dvcmQ='

root-password: cGFzc3dvcmQ=

Alternatively, we can use stringData to provide the password in plain text; Kubernetes will handle the encoding for us when we apply it:

apiVersion: v1

kind: Secret

metadata:

type: Opaque

stringData:

root-password: "our-secure-password"

2. Update the StatefulSet

Modify the env section of our MySQL container to use valueFrom and secretKeyRef. This tells the Pod to pull the value of MYSQL_ROOT_PASSWORD from the secret we just created.

containers:

- name: mysql

image: mysql:8.0

env:

- name: MYSQL_ROOT_PASSWORD

valueFrom:

secretKeyRef:

key: root-password # The specific key inside the Secret

Key Considerations

Initialization Only: For MySQL, the MYSQL_ROOT_PASSWORD environment variable is typically only used during the first-time initialization of the data directory. Changing the Secret later will not automatically update the root password in an existing database.
Security: Ensure our cluster has encryption at rest enabled for Secrets to truly protect sensitive data.
Alternative for Multiple Variables: If we have many credentials (user, password, DB name), we can use envFrom to map all keys in a Secret to environment variables at once.

Changing Storage Spec

Changing spec.volumeClaimTemplate updates the StatefulSet template but will not resize already-created PVCs. If the goal is to fix an existing CrashLoopBackOff due to disk-full, we still need to expand the current PVC(s) (and ensure the general StorageClass allows volume expansion), or recreate the PVC/StatefulSet so the new size takes effect.

References:

StatefulSets | Kubernetes

Amazon EBS CSI Driver

2026-03-19T13:56:56.248+00:00

The Amazon EBS CSI Driver is a standard interface that allows Amazon Elastic Kubernetes Service (EKS) clusters to manage the full lifecycle of Amazon EBS volumes as persistent storage for containers. It replaces the older, deprecated "in-tree" Kubernetes storage plugin with a more flexible, decoupled model.

Key Features

Dynamic Provisioning: Automatically creates and attaches EBS volumes when a PersistentVolumeClaim (PVC) is made.
Volume Lifecycle Management: Handles the creation, attachment, mounting, and deletion of volumes.
Resizing & Snapshots: Supports online volume resizing (for gp3 and others) and taking volume snapshots for data backup.
EKS Auto Mode Support: In EKS Auto Mode, routine block storage tasks are automated, and you don't even need to manually install the driver.

Deployment Methods

You can install and manage the driver through several channels:

EKS Managed Add-on (Recommended): Simplifies installation and updates via the AWS Console, CLI, or Terraform.
Helm Chart: Provides highly customizable installation options.
Kustomize: Direct deployment using manifests from the official GitHub repository.

Core Requirements

IAM Permissions: The driver requires an IAM role with the AmazonEBSCSIDriverPolicy to interact with EBS resources.
Service Accounts: Typically uses IAM Roles for Service Accounts (IRSA) to securely provide AWS credentials to the driver pods.
Compatibility: Supports Linux and Windows worker nodes, as well as ARM64 architectures.

Driver Components

The driver is typically deployed into the kube-system namespace and consists of two main parts:

Controller Deployment: Runs as a set of replicas (ebs-csi-controller) to communicate with the AWS EC2 API and manage volume operations.
Node DaemonSet: Runs on every worker node (ebs-csi-node) to handle the actual mounting and unmounting of volumes to pods on that specific host.

In the Amazon EBS CSI driver architecture, the ebs-csi-controller and ebs-csi-node are the two primary components that work together to manage the lifecycle of EBS volumes in a Kubernetes cluster.

Core Feature Differences

ebs-csi-controller

Deployment Type:

Deployment (typically 2 replicas for HA)

Main Function:

Control Plane operations: Creating, deleting, attaching, and detaching volumes

AWS Interaction:

Calls the AWS EC2 API to manage EBS resources

IAM Permissions:

Requires an IAM role with permissions like ec2:CreateVolume and ec2:AttachVolume

ebs-csi-node

Deployment Type:

DaemonSet (runs on every worker node)

Main Function:

Node-level operations: Mounting and unmounting volumes to the local filesystem

AWS Interaction:

Interacts with the local OS (privileged system calls) to handle block devices

IAM Permissions:

Generally requires fewer/no AWS API permissions, as it mostly performs local mount actions

How They Work Together

Provisioning & Attachment: When you create a PersistentVolumeClaim (PVC), the ebs-csi-controller watches the request and calls the AWS API to create the EBS volume and attach it to the correct EC2 instance.
Mounting: Once the volume is physically attached to the EC2 instance, the ebs-csi-node pod running on that specific node detects the new block device and mounts it into the container’s path so your application can use it.

Key Considerations

Security: For better security, you can schedule the ebs-csi-controller on hardened management nodes, while the ebs-csi-node must run everywhere your workloads need storage.
Fargate: You can run the controller on Fargate nodes, but the ebs-csi-node (as a DaemonSet) only runs on EC2 instances.
Troubleshooting: If a volume fails to "attach," check the controller logs; if it fails to "mount" or "format," check the node logs.

Pods for both the ebs-csi-controller and ebs-csi-node typically share the same value for the app.kubernetes.io/name label.

In standard deployments (such as via the official Helm chart or EKS add-on), both components use this label to identify that they belong to the same overarching application: the Amazon EBS CSI Driver.

Label Comparisons

While they share the same application name, they use the app.kubernetes.io/component label to distinguish between their specific roles.

Label Key ebs-csi-controller Pods ebs-csi-node Pods

------------ ------------------------------ ----------------------

app.kubernetes.io/name aws-ebs-csi-driver aws-ebs-csi-driver

app.kubernetes.io/instance aws-ebs-csi-driver aws-ebs-csi-driver

app.kubernetes.io/component csi-driver (or controller) csi-driver (or node)

app (Legacy label) ebs-csi-controller ebs-csi-node

How to Verify in Your Cluster

You can check these labels yourself using kubectl. This is useful if you are writing Prometheus rules or network policies that need to target the entire driver or just one part of it.

To see labels for all EBS CSI pods:

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver --show-labels

To target only the controller:

kubectl get pods -n kube-system -l app=ebs-csi-controller

---

Monitoring and Observability

2026-03-19T10:37:52.290+00:00

Monitoring vs Observability

In the world of IT and DevOps, monitoring and observability are two related but distinct concepts used to manage system health and performance.

Core Difference

The simplest way to distinguish them is:

Monitoring tells you what is happening (and when). It is reactive and focuses on known problems using predefined metrics.
Observability tells you why it is happening. It is proactive and uses the system's outputs to understand its internal state, especially for "unknown unknowns".

Key Comparison Table

Feature Monitoring Observability

---------- --------------- -----------------

Purpose Detect known issues Diagnose root causes

Perspective External (symptoms) Internal (system state)

Question "Is the system healthy?" "Why is it behaving this way?"

Approach Reactive Proactive

Focus "Known knowns" "Unknown unknowns"

Data Types Metrics, logs Metrics, logs, and traces

Analogy: The Car

Monitoring is your dashboard. It has dials for speed and fuel, and a "check engine" light. It tells you if you are speeding or if something is broken.
Observability is the mechanic’s diagnostic tool. When the "check engine" light comes on, the mechanic plugs in a tool to see exactly which sensor failed and why, without having to take the entire engine apart.

Common Tools

Monitoring Tools: Nagios, Zabbix, Prometheus.
Observability Platforms: Datadog, New Relic, Honeycomb, Dynatrace.

Three Pillars of Observability

The three pillars of observability—metrics, logs, and traces—are essential telemetry data types used to understand the internal state of complex, distributed systems. They enable teams to detect, investigate, and resolve performance issues by providing high-level trends, granular event details, and full request-flow paths.

Metrics

Quantitative measurements over time (e.g., CPU usage, error rates).

Numerical measurements that describe the health, performance, and behavior of a system over time (e.g., CPU usage, error rates, throughput). They are ideal for alerting, capacity planning, and spotting trends or symptoms.

Logs

Granular, timestamped records of discrete events.

Timestamped, granular records of discrete events. They provide the detailed context (text or structured data) necessary to understand exactly what happened within an application or service.

Traces

End-to-end journeys of a single request through a distributed system, showing how different

Records showing the journey of a single request as it travels through a distributed system, encompassing multiple services. They are critical for pinpointing bottlenecks, latency, or failures in microservices architectures.

Why They Are Used Together

While metrics indicate that a problem exists, logs provide the context of why it happened, and traces show where it is occurring. Correlating these three data types provides actionable insights rather than just raw data.

---

How to architecture highly available and fault tolerant AWS EKS Kubernetes cluster

2026-03-16T17:29:00.001+00:00

We should follow a layered approach—from the network up to the application.

The "Highly Available EKS" Design Framework

1. Networking & Control Plane (The Foundation)

Multi-AZ VPC:

Design a VPC with at least 3 Private Subnets across 3 different Availability Zones (AZs).

EKS Endpoint (API server endpoint) Access:

Enable "Private Access" for the EKS control plane so worker nodes communicate with API sever within the VPC, reducing exposure and latency.

NAT Gateways:

Use one NAT Gateway per AZ (3 total) to ensure that if one AZ fails, the nodes in other zones still have outbound internet access for image pulls. This means that we need to have 3 public subnets, one per each AZ.

To use a NAT Gateway, you must place it in a public subnet (a subnet with a route to an Internet Gateway). If your goal is to have one NAT Gateway per Availability Zone (AZ) for high availability, you need a corresponding public subnet in each of those three AZs to host them.

Why this structure is necessary

The architecture follows a specific "dependency chain" to ensure that an issue in one data center doesn't take down your entire outbound connectivity:

AZ Independence: NAT Gateways are zone-redundant by design, but they physically reside in a specific AZ. If AZ-a goes down, the NAT Gateway inside it goes down too.
The Public Subnet Requirement: A NAT Gateway needs a Public IP (EIP) and a route to the Internet Gateway (IGW). Only subnets configured as "public" can provide this.
Cross-Zone Resilience: By having three public subnets (one in each AZ), you can place a NAT Gateway in each. Then, you point the private subnets in AZ-a to the NAT Gateway in AZ-a, the private subnets in AZ-b to the one in AZ-b, and so on.

The Standard Setup

If you are following the recommendation for a 3-AZ deployment, your VPC structure will typically look like this:

Component AZ-1 AZ-2 AZ-3

----------------- ---------------------- --------------------- ---------------------

Public Subnet Subnet-Pub-1 Subnet-Pub-2 Subnet-Pub-3

NAT Gateway NAT-GW-1 NAT-GW-2 NAT-GW-3

Private Subnet Nodes/Workloads Nodes/Workloads Nodes/Workloads

Note on Cost:

While this is the "Gold Standard" for reliability (preventing "cross-zone data charges" and ensuring 100% uptime during an AZ failure), keep in mind that AWS charges per hour for each NAT Gateway. Running three of them is significantly more expensive than running one.

A Common Misconception

You could technically have 3 private subnets and only 1 public subnet (with 1 NAT Gateway). In that case, all nodes in all 3 AZs would send their traffic to that single NAT Gateway.

The Risk: If the AZ containing that lone NAT Gateway fails, your nodes in the other two healthy AZs will lose their ability to pull images or talk to the internet, effectively "breaking" your cluster even though the nodes themselves are fine.

2. Compute & Data Plane (The Muscle)

Managed Node Groups:

Node groups implement basic compute scaling through EC2 Auto Scaling groups.
Use EKS Managed Node Groups spread across those 3 subnets. Select multiple subnets for a node group to provision nodes across multiple Availability Zones.
Amazon EKS managed node groups make it easy to provision compute capacity for your cluster. managed node groups consist of one or more Amazon EC2 instances running the latest EKS-optimized AMIs. All nodes are provisioned as part of an Amazon EC2 Auto Scaling group that is managed for you by Amazon EKS and all resources including EC2 instances and autoscaling groups run within your AWS account.

Auto Scaling Groups (ASG):

Set the min_size to 3. This ensures that even if a node fails, the ASG replaces it immediately.
This is NOT Kubernetes Cluster Autoscaler (or Karpenter), which, if we want to use it, needs to be installed separately.

CAS and ASG both control same node groups and can get in conflict. Solution: disable scaling policies on the ASG so CAS takes control
Karpenter and ASG might get in conflict only if they control the same nodes. But in a well-architected EKS cluster, we usually have two different "families" of nodes:

static: for small, fixed-size Managed Node Group To run "System" pods (CoreDNS, CNI, Karpenter itself)
dynamic: Karpenter-managed node group which runs actual workload, with dynamic number of nodes, depending on the current usage

Instance Diversity:

Mention using multiple instance types (e.g., m5.large and m6g.large) to avoid "insufficient capacity" errors in a specific AWS zone.

3. Traffic Management (The Entry Point)

AWS Load Balancer Controller:

Use the aws-load-balancer-controller to provision an Application Load Balancer (ALB).

Cross-Zone Load Balancing:

Ensure this is enabled so the ALB can route traffic to healthy pods in any AZ, even if the "local" node in its own zone is struggling.

4. Pod-Level Availability (The Brains)

Pod Anti-Affinity:

To ensure replicas don't land on the same node.

Topology Spread Constraints:

To force an equal distribution of pods across the 3 AZs.

Pod Disruption Budgets (PDB):

To prevent the Cluster Autoscaler or AWS maintenance from taking down too many replicas at once.

3 Pillars of High Availability

Component: Multi-AZ ASG

Layer: Infrastructure
Goal: Survive an entire AWS Data Center outage.

Component: PDBs & Rollouts

Layer: Orchestration
Goal: Survive maintenance and human error (updates).

Component: Spread Constraints

Layer: Application
Goal: Survive individual EC2 instance crashes.

---

Application Pod Disruption Budget (PDB) in Kubernetes

2026-03-09T18:24:00.132+00:00

A Pod Disruption Budget (PDB) in Kubernetes:

It's an API object
Sets for an application deployed in cluster:

Either

Minimum number of pods an application needs to function smoothly...

Maximum number of replicated pods that are down simultaneously...

...during voluntary disruptions (e.g., node upgrades, maintenance, draining)

Ensures high availability by guaranteeing a minimum number or percentage of pods remain active.

Key Aspects of PDBs:

In general, disruptions can be:

voluntary, such as maintenance operations or node scaling, or
involuntary, such as hardware failures or system crashes

Voluntary Focus: PDBs only protect against voluntary disruptions, such as kubectl drain or node repairs, not against involuntary, unavoidable failures.

Configuration: You define a PDB using either

either minAvailable

minimum pods that must run

or maxUnavailable

maximum pods allowed to be down
PDB configuration setting defining the maximum number of pods that can be voluntarily taken down simultaneously.

Use Case: Ideal for quorum-based applications (e.g., Elasticsearch, Zookeeper) to ensure quorum is never lost during node maintenance.

Mechanism: When a cluster administrator drains a node, the system checks the PDB. If removing a pod violates the budget, the action is delayed until enough replicas are available elsewhere.

Example PDB Configuration:

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

minAvailable: 2 # At least 2 pods must remain running

selector:

matchLabels:

app: web-app

Name is usually set to refer to the application for which PDB is created ("web" in the example above).

Best Practice:

Use PDBs in conjunction with pod anti-affinity rules to ensure pods are spread across nodes.

How to check PDB in cluster?

Example:

% kubectl get pdb -A

NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE

clickhouse chi-clickhouse-ch N/A 1 1 132d

kube-system ws-cluster-autoscaler N/A 1 1 133d

kube-system coredns N/A 1 1 140d

kube-system ebs-csi-controller N/A 1 1 140d

kube-system karpenter N/A 1 1 139d

ALLOWED DISRUPTIONS:

dynamic value, constantly re-calculated by the disruption controller
the real-time status indicator
showing how many pods can currently be evicted without violating the set maxUnavailable or minAvailable constraints
The non-zero value means that the disruption controller has seen the pods, counted the matching pods, and updated the status of the PDB

To see the number of current and desired healthy pods (and how ALLOWED DISRUPTIONS is actually calculated) run this command (in the example below it was for karpenter application):

% kubectl get poddisruptionbudgets karpenter -n kube-system -o yaml

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

annotations:

meta.helm.sh/release-name: karpenter

meta.helm.sh/release-namespace: kube-system

creationTimestamp: "2025-10-21T14:05:33Z"

generation: 1

labels:

app.kubernetes.io/instance: karpenter

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/name: karpenter

app.kubernetes.io/version: 1.3.2

helm.sh/chart: karpenter-1.3.2

namespace: kube-system

resourceVersion: "2664456"

uid: 2b58340a-fd07-4567-95a9-2a43b5dd4bca

spec:

maxUnavailable: 1

selector:

matchLabels:

app.kubernetes.io/instance: karpenter

app.kubernetes.io/name: karpenter

status:

conditions:

- lastTransitionTime: "2025-10-27T10:52:01Z"

message: ""

observedGeneration: 1

reason: SufficientPods

status: "True"

type: DisruptionAllowed

currentHealthy: 2

desiredHealthy: 1

disruptionsAllowed: 1

expectedPods: 2

observedGeneration: 1

PDB and Rolling Update of Node Group

ALLOWED DISRUPTIONS = 1 is generally the safest and most standard setting for a rolling node group update (which can be triggered if we e.g. upgrade k8s version of aws_eks_node_group), especially for high-availability workloads.

Key Considerations for maxUnavailable: 1

Safety First: This setting ensures only one node is updated at a time (node - if all pod replicas are distribute across all nodes). This is ideal for maintaining quorum in stateful applications like databases (e.g., Consul or ZooKeeper) where losing multiple nodes

simultaneously could cause data loss or service failure.

Default Behavior: In Amazon EKS managed node groups, maxUnavailable defaults to 1 if not specified.

Resource Availability: For this to work, your cluster must have enough spare capacity (CPU/Memory) on the remaining nodes to host the pods evicted from the node being updated.

Update Speed: While safe, updating one node at a time is the slowest method. For very large clusters, you might consider a higher absolute number or a percentage (e.g., 10%) to speed up the process.

When 1 is NOT Enough

Blocking Drains: If you have a Pod Disruption Budget (PDB) where minAvailable equals your total replicas, the node drain will be blocked, and the update will stall because no pods can be legally moved.

Timeouts: Amazon EKS has a 15-minute timeout for draining pods. If pods take too long to terminate, the update may fail unless you have configured pre-stop hooks or adjusted your PDBs.

In-Depth Explanation

In a Pod Disruption Budget (PDB), the setting ALLOWED DISRUPTIONS = 1 (which results from either maxUnavailable: 1 or a minAvailable value that leaves one "slot" free) acts as a safety valve that synchronizes infrastructure changes with application health.

Here is exactly how it makes a rolling node group update safe:

1. It Hooks into the "Eviction API"

When a node group update begins, the automation doesn't just "kill" pods; it calls the Kubernetes Eviction API.

The Check: Before a pod is removed from a node being updated, the Eviction API checks your PDB.

The Logic: If ALLOWED DISRUPTIONS is 1, the API allows exactly one pod to be terminated. Once that pod is gone, the "Allowed Disruptions" counter drops to 0.

2. It Forces a Sequential Wait

This is the most critical safety feature. If a second pod eviction is requested while the first one is still being replaced:

The Block: The API sees ALLOWED DISRUPTIONS = 0 and rejects the eviction request.
The Wait: The node "drain" process pauses and retries. It will stay paused until the first pod's replacement is scheduled on a different node and passes its Readiness Probe.
The Reset: Only when the new pod is "Ready" does the ALLOWED DISRUPTIONS count return to 1, allowing the next pod to be safely evicted.

3. It Prevents "Brain-Dead" Automation

Without this PDB setting, a node group update might try to drain a node that holds all replicas of your app.

Without PDB: The node drains, kills all pods at once, and you have a total outage.
With ALLOWED DISRUPTIONS = 1: The automation is physically unable to kill the second pod until the first one is safely back online elsewhere, ensuring your app always has at least some capacity.

Summary of the "Safety Loop"

Step Action PDB State (Allowed Disruptions)

1 Node update starts; first pod eviction requested 1 → Eviction granted

2 First pod is terminating; replacement is starting 0 → All further evictions blocked

3 Replacement pod passes Readiness Probe 0 → 1 → Block lifted

4 Next pod eviction requested 1 → Process repeats

Important Note: If your app only has 1 replica total, a PDB with minAvailable: 1 will block the node update forever because it can never safely evict that single pod. You generally need at least 2 replicas for this safety mechanism to work without stalling your updates.

Can pod disruption budget be defined in deployment manifest?

No, a Pod Disruption Budget (PDB) cannot be defined directly inside a Deployment manifest. In Kubernetes, a PDB is a separate API resource (Kind: PodDisruptionBudget) that exists independently of the Deployment.

While they are separate resources, they are linked by a Label Selector. You define the PDB to target specific pods by matching the labels specified in your Deployment's pod template.

Why they are separate?

Decoupled Lifecycle: You can create, update, or delete a PDB without needing to redeploy your application or trigger a rolling update of your pods.
Multiple Controllers: A single PDB can protect pods across multiple Deployments, ReplicaSets, or StatefulSets as long as they share the same labels.
Administrative Control: Cluster administrators may manage PDBs separately from application developers to ensure cluster-wide stability during maintenance like node drains.

How to link them?

To make them work together, ensure the spec.selector.matchLabels in your PDB YAML matches the spec.template.metadata.labels in your Deployment YAML.

Deployment snippet:

metadata:

labels:

app: my-app

spec:

template:

metadata:

labels:

app: my-app

PDB snippet:

spec:

selector:

matchLabels:

app: my-app

minAvailable: 2

Here is an example of a complete multi-resource YAML that includes both the Deployment and the PDB:

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: nginx

spec:

replicas: 3

selector:

matchLabels:

app: nginx

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx:latest

ports:

- containerPort: 80

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

# The PDB finds the pods using this selector

selector:

matchLabels:

app: nginx

# Ensures at least 2 pods stay up during voluntary disruptions

minAvailable: 2

Key Takeaways:

The Link: The spec.selector.matchLabels in the PDB must exactly match the spec.template.metadata.labels in the Deployment.
Voluntary Disruptions: This PDB will protect your pods during actions like node drains or cluster upgrades, but it won't prevent "involuntary" issues like a hardware crash.

Using maxUnavailable is often better for autoscaling environments because it scales with your replica count. Instead of saying "I need X pods alive," you’re saying "I can afford to lose X pods."

Here is the updated manifest:

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 5

selector:

matchLabels:

app: nginx

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx:latest

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

selector:

matchLabels:

app: nginx

# Only 1 pod can be taken down at a time during maintenance

maxUnavailable: 1

Pro Tips for maxUnavailable:

Absolute Number: Using 1 ensures that even if you have 100 replicas, Kubernetes will only drain one node at a time for this app.
Percentage: You can use a string like "25%" if you want the "allowed downtime" to grow or shrink as your Deployment scales.
The "One" Rule: You cannot use both minAvailable and maxUnavailable in the same PDB; you have to pick one strategy.

How to use a percentage to make the PDB more flexible for Horizontal Pod Autoscaling (HPA)?

Using percentages for a Pod Disruption Budget (PDB) is highly recommended when combined with a Horizontal Pod Autoscaler (HPA). This allows the disruption budget to scale dynamically as your application grows or shrinks based on traffic.

Dynamic YAML Example (Deployment + HPA + PDB)

This configuration ensures that no matter how many replicas the HPA creates, at least 80% of them will always remain available during maintenance.

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 2 # Initial count

selector:

matchLabels:

app: web

template:

metadata:

labels:

app: web

spec:

containers:

- name: nginx

image: nginx:latest

---

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

minReplicas: 2

maxReplicas: 10

metrics:

- type: Resource

resource:

target:

type: Utilization

averageUtilization: 50

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

selector:

matchLabels:

app: web

# Scales with HPA: if HPA scales to 10 pods, 8 must stay up.

minAvailable: 80%

Key Behaviours to Note:

Rounding Logic: Kubernetes always rounds up the result of the percentage to the nearest integer. For example, if you have 7 pods and set minAvailable: 50%, Kubernetes requires 4 pods to remain available.
Maintenance Blockers: Be careful with 100% or 0%. Setting minAvailable: 100% or maxUnavailable: 0% will completely block node drains, preventing cluster administrators from performing upgrades or maintenance.
Single Replica Limitation: If your HPA scales down to 1 replica, a PDB with minAvailable: 1 or 100% will prevent that single pod from being evicted, which can stall cluster maintenance.

How to troubleshoot a node drain that is stuck because of a PDB?

A kubectl drain typically gets stuck because the ALLOWED DISRUPTIONS for a specific PDB is 0. This means evicting even one more pod would violate your availability rules.

1. Identify the Blocking PDB

Run this command to see which PDB is preventing the drain:

kubectl get pdb -A

Look for any PDB where ALLOWED DISRUPTIONS is 0. If it’s 0, the eviction API will reject any attempt to move those pods, and the drain will wait indefinitely.

2. Common Reasons for "Allowed Disruptions: 0"

Too Few Replicas: You have minAvailable: 1 but only 1 pod is currently running.
Unhealthy Pods: Your PDB requires 3 pods to be healthy, but one is crashing or not yet "Ready." The PDB won't allow any more to be removed until that one is fixed.
Misconfiguration: You set minAvailable equal to your total replicas (e.g., 3 out of 3), meaning you've effectively told Kubernetes "never take a pod down".

3. How to Unstick the Drain

Depending on your situation, use one of these three strategies:

Strategy: Scale Up

Command: kubectl scale deployment <name> --replicas=<current+1>
Best For: Safest: Adds a new pod on a different node so the PDB is satisfied.

Strategy: Relax PDB

Command: kubectl edit pdb <pdb-name>
Best For: Cleanest: Temporarily lower minAvailable or increase maxUnavailable.

Strategy: Temporary Delete

Command: kubectl delete pdb <pdb-name>
Best For: Emergency: Delete the PDB, finish the drain, then re-apply the PDB later.

Warning: The "Force" Flag

Using kubectl drain <node> --force does not bypass PDBs. It only forces the deletion of pods not managed by a controller (like standalone pods) or pods using emptyDir. To truly bypass a PDB, you must either delete the PDB or manually kubectl delete pod the blocking pods

How to find all pods on a specific node that are currently protected by a PDB?

To find pods on a specific node that are protected by a Pod Disruption Budget (PDB), you need to cross-reference the pods on that node with the label selectors defined in your PDBs.

Option 1: The Quickest Way (Manual Check)

The most direct way to see all pods on a node and identify their labels is to use the describe node command.

kubectl describe node <node-name>

Look for the Non-terminated Pods section. You can then check if those pods' labels match your PDB selectors.

Option 2: Filtered Search (Best for Large Nodes)

If the node has many pods, use this command to list all pods on that node along with their labels:

kubectl get pods -A \

-o wide \

--field-selector spec.nodeName=<node-name> \

--show-labels

-A: Searches across all namespaces.

--field-selector spec.nodeName: Limits results to the specific node.

--show-labels: Displays the labels you need to compare against your PDB's spec.selector.

Option 3: Identifying the PDB Selector

To see exactly what labels a PDB is looking for, run:

kubectl get pdb <pdb-name> \

-o jsonpath='{.spec.selector.matchLabels}'

Any pod on your node that has these exact labels is protected by that PDB.

Summary of Useful Flags

Flag Purpose

----------------- ----------

-o wide Shows the NODE column to verify where pods are running.

--field-selector Filters the list server-side by the node name.

-l or --selector Filters pods by specific labels (e.g., kubectl get pods -l app=nginx).

---

How to install Clickhouse in AWS EKS cluster via Altinity Helm charts and Terraform

2026-03-05T11:50:00.006+00:00

Installing ClickHouse on an AWS EKS cluster using Terraform and the Altinity Helm charts typically involves two stages:

Installing the Altinity ClickHouse Operator
Deploying a ClickHouse Installation (CHI)

The Altinity Helm repository is located at https://helm.altinity.com.

Prerequisites

Ensure your Terraform environment is configured with the following providers:

aws: To manage EKS and underlying infrastructure.
kubernetes: To interact with the EKS cluster.
helm: To install the operator.

Terraform Configuration

The following example uses the helm_release resource to install the operator and the kubernetes_manifest resource to deploy the actual ClickHouse cluster.

Step A: Install the Altinity Operator

The operator is the "brain" that manages ClickHouse instances on Kubernetes.

resource "helm_release" "clickhouse_operator" {

name = "clickhouse-operator"

repository = "https://helm.altinity.com"

chart = "altinity-clickhouse-operator"

namespace = "clickhouse-operator"

create_namespace = true

# Optional: Enable metrics for Prometheus

set {

name = "metrics.enabled"

value = "true"

}

Step B: Deploy a ClickHouse Cluster (CHI)

Once the operator is running, you define your ClickHouse cluster using a Custom Resource (CRD). In Terraform, you use kubernetes_manifest.

resource "kubernetes_manifest" "clickhouse_cluster" {

depends_on = [helm_release.clickhouse_operator]

manifest = {

apiVersion = "clickhouse.altinity.com/v1"

kind = "ClickHouseInstallation"

metadata = {

name = "simple-clickhouse"

namespace = "clickhouse-operator"

}

spec = {

configuration = {

clusters = [

{

name = "cluster1"

layout = {

shardsCount = 1

replicasCount = 1

}

]

}

Production Considerations for EKS

When running ClickHouse on EKS, you should consider storage and networking:

Storage Class: Use AWS gp3 volumes for a good balance of price and performance. You can specify a volumeClaimTemplate in your kubernetes_manifest.
Node Affinity: It is recommended to run ClickHouse on specific node groups (e.g., using i3 or r5 instances) to ensure it doesn't compete with other workloads for IOPS.
Zookeeper/Keeper: For multi-node shards or replicas, you will need a Zookeeper cluster or the ClickHouse Keeper (also available via Altinity charts).

EKS Module

Altinity maintains a dedicated Terraform EKS ClickHouse module that automates the entire VPC, EKS, and ClickHouse setup if you prefer a pre-packaged solution.

How to view Clickhouse Installation Configuration?

% kubectl get chi -n clickhouse -o yaml

apiVersion: v1

items:

- apiVersion: clickhouse.altinity.com/v1

kind: ClickHouseInstallation

metadata:

annotations:

kubectl.kubernetes.io/last-applied-configuration: |

{"apiVersion":"clickhouse.altinity.com/v1","kind":"ClickHouseInstallation","metadata":....}

creationTimestamp: "2025-01-28T14:35:46Z"

finalizers:

- finalizer.clickhouseinstallation.altinity.com

generation: 12

namespace: clickhouse

resourceVersion: "67251031"

uid: 9fxxxx1-81e7-429b-9cf7-ffxxxxxxef

spec:

configuration:

clusters:

- layout:

replicasCount: 1

shardsCount: 1

templates:

dataVolumeClaimTemplate: ch-data

podTemplate: ch-pod

serviceTemplate: ch-svc

users:

admin/grants/query: GRANT ALL ON *.*

admin/networks/ip: 0.0.0.0/0

admin/password: my-admin-password

(or admin/password_sha256_hex: my-admin-password-in-sha256)

admin/profile: xxxx

admin/quota: xxxxx

admin/settings/enable_http_compression: 1

default/k8s_secret_password_sha256_hex: <namespace/secretName/key>

default/profile: default

default/quota: default

templates:

podTemplates:

- name: ch-pod

spec:

containers:

- image: altinity/clickhouse-server:24.8.14.10544.altinitystable

- args:

- server

env:

- name: LOG_LEVEL

value: info

- name: API_LISTEN

value: 0.0.0.0:7171

- name: API_CREATE_INTEGRATION_TABLES

value: "true"

- name: REMOTE_STORAGE

value: s3

- name: BACKUPS_TO_KEEP_REMOTE

value: "2"

- name: S3_BUCKET

value: my-clickhouse-backups

- name: S3_REGION

value: us-east-1

- name: CLICKHOUSE_HOST

value: localhost

- name: CLICKHOUSE_USERNAME

value: xxxxx

- name: CLICKHOUSE_PASSWORD

value: xxxx

image: altinity/clickhouse-backup:latest

imagePullPolicy: IfNotPresent

serviceAccountName: clickhouse-backup

tolerations:

- effect: NoSchedule

key: karpenter/clickhouse

operator: Exists

serviceTemplates:

- metadata:

annotations:

service.beta.kubernetes.io/aws-load-balancer-ip-address-type: ipv4

service.beta.kubernetes.io/aws-load-balancer-name: my-clickhouse-nlb

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

service.beta.kubernetes.io/aws-load-balancer-scheme: internal

service.beta.kubernetes.io/aws-load-balancer-type: nlb

spec:

ports:

- name: http

port: 8123

targetPort: 8123

- name: native

port: 9000

targetPort: 9000

type: LoadBalancer

volumeClaimTemplates:

- name: ch-data

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 100Gi

status:

chop-commit: 9abcd12

chop-date: 2025-01-24T08:40:12

chop-ip: 10.x.x.x

chop-version: 0.25.5

clusters: 1

endpoint: clickhouse-clickhouse.clickhouse.svc.cluster.local

fqdns:

- chi-clickhouse-ch-0-0.clickhouse.svc.cluster.local

hosts: 1

hostsWithTablesCreated:

- chi-clickhouse-ch-0-0.clickhouse.svc.cluster.local

pods:

- chi-clickhouse-ch-0-0-0

shards: 1

status: Completed

taskID: auto-1xxxxd2-5ba4-4c3a-9daa-baxxxxx850

taskIDsCompleted:

- auto-1fxxxxxd2-5ba4-4c3a-9daa-baxxxxxx50

...

- auto-bbxxxx6-31e3-4a4c-b04b-e5xxxxxx91

taskIDsStarted:

- auto-31xxxxx37-492f-4109-b515-4axxxxxx6c8

...

- auto-b8xxxxx7-0396-41e0-b5d1-95xxxxd48

kind: List

metadata:

resourceVersion: ""

Users section shows users config in form USER_NAME/ATTRIBUTE. In the example above we have two users: admin and default.

USER_NAME/password values is plain text password. This is very convenient for debugging (though usually a security "no-no" for production, especially if that's admin or default user!).

USER_NAME/password_sha256_hex is a SHA256 hashed password.

USER_NAME/k8s_secret_password_sha256_hex: <namespace/SECRET_NAME/KEY_NAME > shows that USER_NAME ClickHouse user is secured using a Kubernetes Secret. This maps the default user's password to a specific Kubernetes Secret.

USER_NAME/k8s_secret_password_sha256_hex: Specifies that for the user named USER_NAME, the password should be read from a Kubernetes Secret as a SHA256 hex string.
<namespace/SECRET_NAME/KEY_NAME >: This is the reference to the secret itself, structured as namespace/SECRET_NAME/KEY_NAME.
Purpose: This allows for secure, GitOps-friendly password management, preventing plain-text passwords from appearing in Kubernetes manifests.
Implementation: The ClickHouse Operator reads this secret and places the hashed password into the users.xml file for the ClickHouse server. Operator reads the secret, hashes the password (if necessary), and writes it into a file called /etc/clickhouse-server/users.d/chop-generated-users.xml inside your ClickHouse pod. If you have External Secrets installed, this secret is likely being pulled from AWS Secrets Manager.
Alternative: You can also use k8s_secret_env_password_sha256_hex to load the password via an environment variable.

In the Altinity Operator, the syntax USER_NAME/k8s_secret_password_sha256_hex is a pointer. It tells the operator to look into a specific secret to find the password hash for the USER_NAME user.

To get the password:

% kubectl get secret <SECRET_NAME> \

-n <NAMESPACE> \

-o jsonpath="{.data.<KEY_NAME>}" \

| base64 -d

NAMESPACE is usually clickhouse.

How to check Clickhouse health?

Since ClickHouse is running in our cluster, the best way to verify it's "working fine" is to move beyond just checking the Pod status and actually query the database engine itself.

Here is a 3-step approach to verify health, connectivity, and data integrity.

1. The "Internal" Health Check

The quickest way is to execute a command directly inside the pod using the clickhouse-client. This bypasses networking issues and tells you if the engine is responsive. Run this command:

kubectl exec -it chi-clickhouse-ch-0-0 \

-n clickhouse \

-- clickhouse-client --query "SELECT version(), uptime()"

chi-clickhouse-ch-0-0 is the name of the pod, it can also be like chi-clickhouse-ch-0-0-0.

If this returns data, it means ClickHouse is successfully reading from its system tables on the EBS volume.

What to look for: It should return the version string and the number of seconds the server has been up. If this fails, the DB engine itself is hung.

If you are using default user which has a password, or, default user was disabled, the output might show the error similar to this:

% kubectl exec -it chi-clickhouse-ch-0-0-0 -n clickhouse -- clickhouse-client --query "SELECT version(), uptime()"

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup

Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name.

If you have installed ClickHouse and forgot password you can reset it in the configuration file.

The password for default user is typically located at /etc/clickhouse-server/users.d/default-password.xml

and deleting this file will reset the password.

See also /etc/clickhouse-server/users.xml on the server where ClickHouse is installed.

. (AUTHENTICATION_FAILED)

command terminated with exit code 4

Seeing an AUTHENTICATION_FAILED error instead of a Connection Refused error is actually a positive result for this check:

Networking works: Your kubectl exec reached the pod.
Process is alive: The ClickHouse server is running and actively rejecting bad logins.
Storage is mounted: ClickHouse can't check credentials if it can't read its config files from disk.

If we know the Clickhouse credentials, we can perform the health check:

% kubectl exec -it chi-clickhouse-ch-0-0-0 -n clickhouse -- clickhouse-client --user USER --password PASS --query "SELECT version(), uptime(), name FROM system.clusters"

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup

24.8.14.10544.altinitystable 501389 all-clusters

24.8.14.10544.altinitystable 501389 all-replicated

24.8.14.10544.altinitystable 501389 all-sharded

24.8.14.10544.altinitystable 501389 ch

24.8.14.10544.altinitystable 501389 default

The output above is exactly what we wanted to see. The database is responsive, healthy, and has an uptime of ~5.8 days (501,389 seconds). The version 24.8.14.10544.altinitystable indicates we are on a very recent, stable Altinity build.

2. Check Replication and Disk Health

Since you are using the Altinity Operator, ClickHouse is likely managing data across disks. You want to ensure the "System" tables report no errors. Run this to check if the disks are mounted and have space:

% kubectl exec -it chi-clickhouse-ch-0-0-0 \

-n clickhouse \

-- clickhouse-client --user USER --password PASS \

--query "SELECT name, path, formatReadableSize(free_space) AS free, formatReadableSize(total_space) AS total FROM system.disks"

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup

default /var/lib/clickhouse/ 89.60 GiB 95.80 GiB

If you have multiple replicas (e.g., a ch-0-1 pod), check for replication lag:

% kubectl exec -it chi-clickhouse-ch-0-0-0 \

-n clickhouse \

-- clickhouse-client --user USER --password PASS \

--query "SELECT type, last_exception, num_tries FROM system.replication_queue WHERE last_exception != ''"

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup

Result: This should ideally be empty (or as above) . If you see exceptions here, your nodes aren't syncing correctly.

3. Verify the "Operator" View

The Altinity Operator provides a "Status" field in its Custom Resource that summarizes the health of the entire installation.

% kubectl get chi -n clickhouse

NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE

clickhouse 1 1 Completed 123d

What to look for: Look for the STATUS column. It should say Completed. If it says InProgress or Error, the Operator is struggling to configure the cluster.

4. Check the Backup (Safety Net)

Since you saw clickhouse-backup pods earlier, verify that the last backup actually succeeded. This is your "point of no return" check before the upgrade.

kubectl logs -n clickhouse -l job-name=clickhouse-backup-cron-<TIMESTAMP>

(Replace <TIMESTAMP> with one of the strings from your previous get all output, e.g., 29543400).

Look for: Done, Success, or Upload finished.

5. Check the Status of All Replicas

To be absolutely sure the cluster is "Green" before you start the EKS upgrade, run this to check the status of all replicas in the cluster:

% kubectl exec -it chi-clickhouse-ch-0-0-0 -n clickhouse -- clickhouse-client --user USER --password PASS --query "SELECT replica_path, is_leader, is_readonly, future_parts FROM system.replicas"

is_readonly: Should be 0. If it's 1, the node can't write data (usually a Zookeeper issue).

is_leader: One of your replicas should be 1.

Summary Checklist

Test Command Goal Good Result

Ping SELECT 1 1

Uptime SELECT uptime() >0

Storage system.disks Free space > 10%

Operator kubectl get chi Completed

Resources:

Altinity | Run Open Source ClickHouse® Better

Introduction to ArgoCD

2026-03-05T00:46:00.007+00:00

ArgoCD is a tool for deploying applications in Kubernetes cluster following the GitOps principles.

Argo CD is a declarative, GitOps-based continuous delivery (CD) tool designed specifically for Kubernetes. It acts as a controller that monitors running applications, compares their live state to the desired state defined in a Git repository, and automatically syncs them to ensure consistency.

Key Aspects of an Argo CD Application:

GitOps Source of Truth: Git repositories hold the desired state (manifests, Helm charts, Kustomize configs), which Argo CD pulls to apply to clusters.
Automated Synchronization: It automatically detects "OutOfSync" applications—where the live cluster state differs from Git—and can automatically or manually sync them to match.
Continuous Monitoring: It acts as a Kubernetes controller that continuously monitors applications.
Visualization & Management: It provides a web UI to visualize application structure, monitor status, and manage rollbacks.
Key Capabilities: Supports automated deployment, drift detection, and easy rollbacks.

Argo CD is often used to ensure that the actual state of a Kubernetes cluster matches the configuration stored in a Git repository, making the deployment process more reliable and transparent.

ArgoCD GitOps flow:

Commit and push infra code changes (Terraform, Helm values etc) to the main branch of your GitHub repository.
ArgoCD Sync:

Manual: Log in to our ArgoCD Dashboard and click the "Sync" or "Refresh" button on the psmdb-default-sharded application.
Automatic: If "Self-Heal" or "Auto-Sync" is enabled, ArgoCD will detect the Git change within ~3 minutes and apply it to the cluster automatically.

Monitor the Operator: Once ArgoCD syncs, infra will get changed e.g. operators will see the updated custom resources and then trigger the further actions.

Installation

Installing ArgoCD in a Kubernetes cluster is primarily done using manifests or Helm charts. The most common and recommended approach for beginners is using the Official Argo CD Manifests.

Prerequisites:

A running Kubernetes cluster (v1.22 or later).
kubectl command-line tool installed and configured to your cluster.
At least 2GB of available memory in your cluster.

Applications

ArgoCD defines custom Kubernetes objects like Application, AppProject, settings...which can be defined declaratively using Kubernetes manifests and deployed via kubectl apply to the ArgoCD namespace which is argocd by default.

To check ArgoCD applications:

% kubectl get applications -A

NAMESPACE NAME SYNC STATUS HEALTH STATUS

argocd my-app Synced Progressing

Dashboard

If you don't know the url of the ArgoCD dashboard, find the IP of the ArgoCD server and port-forward it:

% kubectl get svc argocd-server -n argocd

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

argocd-server ClusterIP 172.21.103.127 <none> 80/TCP,443/TCP 1d

By default, ArgoCD includes a built-in admin user with full super-user access. For better security practices, it is recommended that you use the admin account only for the initial configuration, then disable it once all required users have been added.

For this example, we can stick to this default admin user. Its password can be obtain via this command:

% kubectl get secret \

argocd-initial-admin-secret \

-n argocd \

-o jsonpath="{.data.password}" \

| base64 -d

Port forwarding:

% kubectl port-forward svc/argocd-server -n argocd 8080:443

Forwarding from 127.0.0.1:8080 -> 8080

Forwarding from [::1]:8080 -> 8080

Handling connection for 8080

...

We can now open the address http://localhost:8080/ in the browser to get the ArgoCD dashboard and log in with credentials above.

If there are no application registered with ArgoCD, we will see something like this:

---

Resources:

Argo CD - Declarative GitOps CD for Kubernetes

Continuous Deployment with Argo CD - Amazon EKS

Introduction to KubePug (Kubernetes tool)

2026-02-26T15:42:00.001+00:00

What is KubePug?

KubePug (Kubernetes PreUpGrade Checker) is an open-source kubectl plugin and CLI tool designed to identify deprecated or deleted APIs in your Kubernetes cluster or manifest files before you perform an upgrade.

Key Features

Deprecation Detection: Scans your live cluster or static YAML manifests to find resources using APIs that are slated for removal in future Kubernetes versions.
Replacement Guidance: Not only flags outdated APIs but also suggests the recommended replacement API and specifies the exact version where the deprecation or deletion occurs.
Version Targeting: Allows you to specify a target Kubernetes version (e.g., v1.31) to validate your current resources against that specific future release.
Flexible Data Source: It automatically downloads a frequently updated API definition file (every 30 minutes) to stay current with the latest Kubernetes releases.

Why Use It?

As Kubernetes evolves, APIs are moved from alpha/beta to stable (GA), and older versions are eventually removed. If you upgrade your cluster without updating your manifests, those resources will fail to deploy or operate. KubePug provides a "pre-flight" check to prevent these breaking changes from reaching production.

Installation & Usage

You can install KubePug via Krew (where it is listed under the name deprecations) or as a standalone binary.

Method Command

--------- -------------

Install via Krew kubectl krew install deprecations

Scan Live Cluster kubectl deprecations --k8s-version=v1.30

Scan Manifest File kubepug --input-file=./my-manifest.yaml

Installation via Krew

% kubectl krew install deprecations

Updated the local copy of plugin index.

Installing plugin: deprecations

Installed plugin: deprecations

| Use this plugin:

| kubectl deprecations

| Documentation:

| https://github.com/rikatz/kubepug

| Caveats:

| \

| | * By default, deprecations finds deprecated object relative to the current kubernetes

| | master branch. To target a different kubernetes release, use the --k8s-version

| | argument.

| |

| | * Deprecations needs permission to GET all objects in the Cluster

| /

WARNING: You installed plugin "deprecations" from the krew-index plugin repository.

These plugins are not audited for security by the Krew maintainers.

Run them at your own risk.

Execution

Once installed, the plugin is invoked using kubectl deprecations.

Scan Current Cluster

Check your live cluster for deprecated APIs against a specific target Kubernetes version:

% kubectl deprecations --k8s-version=v1.33

Error: failed to get apiservices: apiservices.apiregistration.k8s.io is forbidden: User "sso:user" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

time="2026-02-26T14:20:19Z" level=error msg="An error has occurred: failed to get apiservices: apiservices.apiregistration.k8s.io is forbidden: User \"sso:user\" cannot list resource \"apiservices\" in API group \"apiregistration.k8s.io\" at the cluster scope"

KubePug requires running user to have "list" permissions on resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope as otherwise the above error will appear.

If required permissions are in place:

% kubectl deprecations --k8s-version=v1.33

No deprecated or deleted APIs found

Kubepug validates the APIs using Kubernetes markers. To know what are the deprecated and deleted APIS it checks, please go to https://kubepug.xyz/status/

Scan Local Manifest Files

Validate static YAML files before applying them to a cluster:

kubectl deprecations --input-file=./my-manifests/

View Results in Different Formats

Output the findings in json or yaml for automated processing:

kubectl deprecations --format=json

Check for Help and Flags

See all available configuration options, such as using a custom database file or setting error codes:

kubectl deprecations --help

Key Parameters

--k8s-version: The Kubernetes release you intend to upgrade to (defaults to the latest stable).

--error-on-deprecated: Forces the command to exit with an error code if deprecated APIs are found, which is useful for CI/CD pipelines.

---

Introduction to Krew (Kubernetes tool)

2026-02-26T14:15:00.008+00:00

Krew is the official plugin manager for the kubectl command-line tool. Much like apt for Debian or Homebrew for macOS, it allows users to easily discover, install, and manage custom extensions that add new subcommands to Kubernetes.

Core Functionality

Discovery: Users can search a community-curated index of over 200 plugins designed for tasks like security auditing, resource visualization, and cluster management.
Lifecycle Management: It automates the process of installing, updating, and removing plugins across different operating systems (Linux, macOS, and Windows).
Unified Interface: Once a plugin is installed via Krew, it is invoked directly through kubectl (e.g., kubectl <plugin-name>).

Installation

MacOS example output:

% (

set -x; cd "$(mktemp -d)" &&

OS="$(uname | tr '[:upper:]' '[:lower:]')" &&

ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&

KREW="krew-${OS}_${ARCH}" &&

curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&

tar zxvf "${KREW}.tar.gz" &&

./"${KREW}" install krew

)

+-zsh:142> mktemp -d

+-zsh:142> cd /var/folders/8j/60m6_18j359_39ls0sr9ccvm0000gp/T/tmp.LKDSyQcO10

+-zsh:143> OS=+-zsh:143> uname

+-zsh:143> OS=+-zsh:143> tr '[:upper:]' '[:lower:]'

+-zsh:143> OS=darwin

+-zsh:144> ARCH=+-zsh:144> uname -m

+-zsh:144> ARCH=+-zsh:144> sed -e s/x86_64/amd64/ -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/'

+-zsh:144> ARCH=arm64

+-zsh:145> KREW=krew-darwin_arm64

+-zsh:146> curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-darwin_arm64.tar.gz

+-zsh:147> tar zxvf krew-darwin_arm64.tar.gz

x ./LICENSE

x ./krew-darwin_arm64

+-zsh:148> ./krew-darwin_arm64 install krew

Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git.

Updated the local copy of plugin index.

Installing plugin: krew

Installed plugin: krew

| Use this plugin:

| kubectl krew

| Documentation:

| https://krew.sigs.k8s.io/

| Caveats:

| \

| | krew is now installed! To start using kubectl plugins, you need to add

| | krew's installation directory to your PATH:

| |

| | * macOS/Linux:

| | - Add the following to your ~/.bashrc or ~/.zshrc:

| | export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

| | - Restart your shell.

| |

| | * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable

| |

| | To list krew commands and to get help, run:

| | $ kubectl krew

| | For a full list of available plugins, run:

| | $ kubectl krew search

| |

| | You can find documentation at

| | https://krew.sigs.k8s.io/docs/user-guide/quickstart/.

| /

Add Krew path to PATHs:

% vi ~/.zshrc

% cat ~/.zshrc

...

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" <-- added manually

Reload .zshrc configuration file within the currently running Zsh terminal (or just restart the terminal):

% source ~/.zshrc

Installation verification:

% kubectl krew

krew is the kubectl plugin manager.

You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:

kubectl krew [command]

Available Commands:

help Help about any command

index Manage custom plugin indexes

info Show information about an available plugin

install Install kubectl plugins

list List installed kubectl plugins

search Discover kubectl plugins

uninstall Uninstall plugins

update Update the local copy of the plugin index

upgrade Upgrade installed plugins to newer versions

version Show krew version and diagnostics

Flags:

-h, --help help for krew

-v, --v Level number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.

Common Commands

To use Krew, you must first install it as a kubectl plugin itself. Key commands include:

kubectl krew update: Updates the local list of available plugins.

kubectl krew search: Finds plugins in the official Krew index.

kubectl krew install <plugin>: Installs a specific plugin.

kubectl krew list: Displays all plugins currently installed through Krew.

kubectl krew upgrade: Updates all installed plugins to their latest versions.

Example:

% kubectl krew install deprecations

Updated the local copy of plugin index.

Installing plugin: deprecations

Installed plugin: deprecations

| Use this plugin:

| kubectl deprecations

| Documentation:

| https://github.com/rikatz/kubepug

| Caveats:

| \

| | * By default, deprecations finds deprecated object relative to the current kubernetes

| | master branch. To target a different kubernetes release, use the --k8s-version

| | argument.

| |

| | * Deprecations needs permission to GET all objects in the Cluster

| /

WARNING: You installed plugin "deprecations" from the krew-index plugin repository.

These plugins are not audited for security by the Krew maintainers.

Run them at your own risk.

Popular Plugins Managed by Krew

ctx / ns: Rapidly switch between Kubernetes contexts and namespaces.
tree: Visualizes the hierarchy of Kubernetes resources in a tree view.
access-matrix: Displays an RBAC (Role-Based Access Control) matrix for server resources.
get-all: Lists all resources in a namespace, including those often missed by kubectl get all.

Note on Security (!)

Plugins in the Krew index are community-contributed and are not audited for security by the Kubernetes maintainers; you should only install plugins from sources you trust.

---

Introduction to Pluto (Kubernetes tool)

2026-02-26T13:03:00.001+00:00

Pluto is:

CLI tool that helps users find deprecated Kubernetes API versions in your code repositories and Helm releases.
It's especially useful when upgrading Kubernetes clusters, as it identifies resources that need updating before the upgrade.
It works against:

Live clusters
Helm charts
Raw YAML

Pluto will show which APIs are deprecated or removed, what version they were deprecated in, and what the replacement API should be.

Installation on Mac (https://pluto.docs.fairwinds.com/installation/#homebrew-tap):

% brew install FairwindsOps/tap/pluto

Let's see its CLI arguments:

% pluto

You must specify a sub-command.

A tool to detect Kubernetes apiVersions

Usage:

pluto [flags]

pluto [command]

Available Commands:

completion Generate the autocompletion script for the specified shell

detect Checks a single file or stdin for deprecated apiVersions.

detect-all-in-cluster run all in-cluster detections

detect-api-resources detect-api-resources

detect-files detect-files

detect-helm detect-helm

help Help about any command

list-versions Outputs a JSON object of the versions that Pluto knows about.

version Prints the current version of the tool.

Flags:

-f, --additional-versions string Additional deprecated versions file to add to the list. Cannot contain any existing versions

--columns strings A list of columns to print. Mandatory when using --output custom, optional with --output markdown

--components strings A list of components to run checks for. If nil, will check for all found in versions.

-h, --help help for pluto

--ignore-deprecations Ignore the default behavior to exit 2 if deprecated apiVersions are found. (Only show removed APIs, not just deprecated ones)

--ignore-removals Ignore the default behavior to exit 3 if removed apiVersions are found. (Only show deprecated APIs, not removed ones)

--ignore-unavailable-replacements Ignore the default behavior to exit 4 if deprecated but unavailable apiVersions are found.

-H, --no-headers When using the default or custom-column output format, don't print headers (default print headers).

-r, --only-show-removed Only display the apiVersions that have been removed in the target version.

-t, --target-versions stringToString A map of targetVersions to use. This flag supersedes all defaults in version files. (default [])

-v, --v Level number for the log level verbosity

Use "pluto [command] --help" for more information about a command.

detect-files

If we want to scan local Helm charts or manifest files before they are deployed, we can use pluto detect-files or pluto detect, which do require us to be in the correct directory or provide a file path.

To scan and detect deprecated APIs in manifest files in a directory:

% pluto detect-files -d /path/to/your/manifests

detect-files is for checking YAML files in our repositories/filesystem before deploying them - that's separate from detect-helm and detect-all-in-cluster cluster commands.

To target particular k8s version:

% pluto detect-files -d . --target-versions k8s=v1.33.0

If we use Terraform to deploy Helm charts, we might want to keep chart values in separate files (.yaml or .yaml.tpl) as otherwise we won't be able to use Pluto directly (we'd need to extract values into files first). For more details, see Where to keep Helm chart values in Terraform projects | My Public Notepad.

detect-helm

To check Helm releases in the cluster (already deployed):

% pluto detect-helm -owide

To target particular k8s version:

% pluto detect-helm -owide --target-versions k8s=v1.33.0

There were no resources found with known deprecated apiVersions.

detect-helm specifically checks Helm release metadata stored in our cluster (in secrets or configmaps) after Helm chart have been deployed. It looks at the manifests that Helm used to install releases, which might contain deprecated APIs even if they haven't been applied yet or are stored in Helm's history.

This command can be run from any directory. This is because the detect-helm command scans live Helm releases currently deployed in our Kubernetes cluster, rather than looking for local files on our machine. Instead of relying on our current working directory, the command depends on our Kubernetes context (the cluster our CLI is currently pointed at) and our local Helm configuration to communicate with the cluster.

While the directory doesn't matter, the following must be true for the command to work:

Active Kubernetes Context: Your kubectl context must be set to the target cluster.
Cluster Permissions: You must have sufficient RBAC permissions to read Secrets in the namespaces you wish to scan, as Helm 3 stores release information in cluster secrets.
Target Versioning: The --target-versions k8s=v1.33.0 flag tells Pluto to check for APIs that are deprecated or removed specifically in Kubernetes version 1.33.0, regardless of what version the cluster is actually running

detect-all-in-cluster

To check all resources in the cluster:

% pluto detect-all-in-cluster -o wide

I0226 12:01:02.279788 47100 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"

There were no resources found with known deprecated apiVersions.

detect-all-in-cluster scans all live resources currently running in our cluster by querying the Kubernetes API directly. It checks deployments, services, pods, etc. that are actively deployed.

detect-all-in-cluster does NOT include detect-helm or detect-files. Here's why they're separate:

detect-all-in-cluster sees the current state of resources
detect-helm sees Helm's stored templates and history, which may include:

Templated manifests that haven't been rendered yet
Old release revisions
Chart templates with deprecated APIs

Run both to get complete coverage!

Target a specific Kubernetes version:

% pluto detect-all-in-cluster --target-versions k8s=v1.33.0

I0226 12:02:26.551401 47113 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"

There were no resources found with known deprecated apiVersions.

The warning message:

Warning: v1 ComponentStatus is deprecated in v1.19+

This is just Pluto itself triggering a Kubernetes API warning while scanning - it's not something wrong with our cluster resources.

The main result:

There were no resources found with known deprecated apiVersions.

This means all our cluster resources are using API versions that are still valid in Kubernetes v1.33.0 (our target version). This means:

Our cluster resources are already compatible with k8s v1.33.0
No manifests need updating before upgrading
No deprecated APIs that would be removed in v1.33.0

To be thorough before a k8s upgrade, we need to run all three commands:

detect-files
detect-helm
detect-all-in-cluster

---

Where to keep Helm chart values in Terraform projects

2026-02-26T13:02:00.001+00:00

If we use Terraform to deploy Helm charts, we might be using one of these strategies to keep chart values:

Values are in inline YAML string
Values in separate .yaml file
Values in separate YAML Template files (.yaml.tpl)
Use Helm's set for Dynamic Values
Multiple Values Files

(1) Values in inline YAML string

This is not ideal as problems with Inline YAML in Terraform include:

No syntax highlighting or validation - Easy to break YAML formatting
Hard to review in diffs - Changes are messy in PRs
Can't use standard tooling - No yamllint, Pluto, or other YAML tools
Mixing concerns - Infrastructure code mixed with application config
Escaping nightmares - Terraform string interpolation conflicts with Helm templating

Example:

resource "helm_release" "app" {

values = [<<-EOT

replicaCount: ${var.replicas}

image:

repository: myapp

tag: ${var.tag}

service:

type: LoadBalancer

EOT

]

}

(2) Separate Values Files

Keep values in YAML files, reference them in Terraform.

This is a better approach because:

Clean separation
Easy to validate with standard tools
Better diffs
Can use Pluto directly: pluto detect-files -d .

Example:

main.tf:

resource "helm_release" "my_app" {

name = "my-app"

chart = "my-chart"

repository = "https://charts.example.com"

values = [

file("${path.module}/helm-values.yaml")

]

}

(3) Templated Values Files

Use Terraform's templatefile() to inject dynamic values:

helm-values.yaml.tpl:

replicaCount: ${replica_count}

image:

repository: ${image_repo}

tag: ${image_tag}

ingress:

enabled: ${enable_ingress}

host: ${hostname}

main.tf:

resource "helm_release" "my_app" {

name = "my-app"

chart = "my-chart"

values = [

templatefile("${path.module}/helm-values.yaml.tpl", {

replica_count = var.replica_count

image_repo = var.image_repository

image_tag = var.image_tag

enable_ingress = var.enable_ingress

hostname = var.hostname

})

]

}

Pros:

Still gets variable injection
Can be validated as YAML (with placeholders)
Clean and readable

(4) Use Helm's set for Dynamic Values

Keep static config in files, override specific values:

resource "helm_release" "my_app" {

name = "my-app"

chart = "my-chart"

# Base values from file

values = [

file("${path.module}/helm-values.yaml")

]

# Override specific values dynamically

set {

name = "image.tag"

value = var.image_tag

}

set {

name = "replicaCount"

value = var.replica_count

}

set_sensitive {

name = "secret.password"

value = var.db_password

}

Pros:

Clear what's dynamic vs static
Base values file can be validated
Sensitive values handled properly

Here is the example how we can migrate inline YAML from the above to templated file:

helm-values.yaml:

image:

repository: myapp

service:

type: LoadBalancer

main.tf:

resource "helm_release" "app" {

values = [

file("${path.module}/helm-values.yaml")

]

set {

name = "replicaCount"

value = var.replicas

}

set {

name = "image.tag"

value = var.tag

}

Now we can run:

% pluto detect-files -f helm-values.yaml

(5) Multiple Values Files

We can layer our configuration:

resource "helm_release" "my_app" {

name = "my-app"

chart = "my-chart"

values = [

file("${path.module}/helm-values-base.yaml"),

file("${path.module}/helm-values-${var.environment}.yaml")

]

}

---

Introduction to Kubent (Kube No Trouble)

2026-02-26T11:07:00.002+00:00

Kubent (Kube No Trouble) [this link was the original repo, see comments below] is a tool which scans k8s cluster and reports resources that use deprecated or removed Kubernetes APIs, based on the target Kubernetes version. It’s especially useful before upgrading (e.g., EKS 1.32 → 1.33)

WARNING: Development at project's original repo (https://github.com/doitintl/kube-no-trouble) is not active anymore as the last commit was in January 2025. The original author announced here that they would be moving development to https://github.com/dark0dave/kube-no-trouble and that repo is ssemingly active as of today (last change was ) BUT https://github.com/dark0dave/kube-no-trouble/tree/301e5783904de5966f79b217a956651146630f50/pkg/rules/rego shows that rulesets only up to v1.32 were added (!).

Kube No Trouble relies on static Rego rule files in the repo. If new Kubernetes versions (e.g., >1.32) don’t have updated rules, then:

It won’t know about newly deprecated APIs
It won’t know about newly removed APIs
--target-version becomes unreliable for newer releases

For modern upgrades (especially 1.32 → 1.33+), kubent is no longer the safest tool.

To install it:

% sh -c "$(curl -sSL https://git.io/install-kubent)"

>>> kubent installation script <<<

> Detecting latest version

> Downloading version 0.7.3

Target directory (/usr/local/bin) is not writable, trying to use sudo

Password:

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

100 12.4M 100 12.4M 0 0 14.7M 0 --:--:-- --:--:-- --:--:-- 13.2M

> Done. kubent was installed to /usr/local/bin/.

To verify installation:

% kubent --version

7:48AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

To check CLI arguments:

% kubent --help

Usage of kubent:

-A, --additional-annotation strings additional annotations that should be checked to determine the last applied config

-a, --additional-kind strings additional kinds of resources to report in Kind.version.group.com format

-c, --cluster enable Cluster collector (default true)

-x, --context string kubeconfig context

-e, --exit-error exit with non-zero code when issues are found

-f, --filename strings manifests to check, use - for stdin

--helm3 enable Helm v3 collector (default true)

-k, --kubeconfig string path to the kubeconfig file

-l, --log-level string set log level (trace, debug, info, warn, error, fatal, panic, disabled) (default "info")

-o, --output string output format - [text|json|csv] (default "text")

-O, --output-file string output file, use - for stdout (default "-")

-t, --target-version string target K8s version in SemVer format (autodetected by default)

-v, --version prints the version of kubent and exits

pflag: help requested

It looks at default ~/.kube/config file in order to find the current context, otherwise use -k to specify kubeconfig at non-default location.

% kubent

7:59AM INF >>> Kube No Trouble `kubent` <<<

7:59AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

7:59AM INF Initializing collectors and retrieving data

7:59AM INF Target K8s version is 1.32.11-eks-ac2d5a0

7:59AM INF Retrieved 12 resources from collector name=Cluster

8:00AM INF Retrieved 361 resources from collector name="Helm v3"

8:00AM INF Loaded ruleset name=custom.rego.tmpl

8:00AM INF Loaded ruleset name=deprecated-1-16.rego

8:00AM INF Loaded ruleset name=deprecated-1-22.rego

8:00AM INF Loaded ruleset name=deprecated-1-25.rego

8:00AM INF Loaded ruleset name=deprecated-1-26.rego

8:00AM INF Loaded ruleset name=deprecated-1-27.rego

8:00AM INF Loaded ruleset name=deprecated-1-29.rego

8:00AM INF Loaded ruleset name=deprecated-1-32.rego

8:00AM INF Loaded ruleset name=deprecated-future.rego

Running kubent with no other arguments:

Connects to your current kube-context
Detects your cluster version automatically
Scans all namespaces
Compares resources against deprecations for that version

Before the upgrade to v1.33, we want kubent to scan the resources against that next k8s version so we need to specify it with --target-version:

% kubent --target-version=1.33

8:02AM INF >>> Kube No Trouble `kubent` <<<

8:02AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

8:02AM INF Initializing collectors and retrieving data

8:02AM INF Target K8s version is 1.33.0

8:02AM INF Retrieved 12 resources from collector name=Cluster

8:03AM INF Retrieved 361 resources from collector name="Helm v3"

8:03AM INF Loaded ruleset name=custom.rego.tmpl

8:03AM INF Loaded ruleset name=deprecated-1-16.rego

8:03AM INF Loaded ruleset name=deprecated-1-22.rego

8:03AM INF Loaded ruleset name=deprecated-1-25.rego

8:03AM INF Loaded ruleset name=deprecated-1-26.rego

8:03AM INF Loaded ruleset name=deprecated-1-27.rego

8:03AM INF Loaded ruleset name=deprecated-1-29.rego

8:03AM INF Loaded ruleset name=deprecated-1-32.rego

8:03AM INF Loaded ruleset name=deprecated-future.rego

---

Introduction to Grafana Loki

2026-02-23T11:59:00.001+00:00

Grafana Loki:

Log aggregation system. Like Prometheus, but for logs
Repo: https://github.com/grafana/loki

These are the notes from Loki Helm chart:

***********************************************************************

Welcome to Grafana Loki

Chart version: 6.31.0

Chart Name: loki

Loki version: 3.5.0

***********************************************************************

Tip:

Watch the deployment status using the command: kubectl get pods -w --namespace grafana-loki

If pods are taking too long to schedule make sure pod affinity can be fulfilled in the current cluster.

***********************************************************************

Installed components:

***********************************************************************

* gateway

* read

* write

* backend

***********************************************************************

Sending logs to Loki

***********************************************************************

Loki has been configured with a gateway (nginx) to support reads and writes from a single component.

You can send logs from inside the cluster using the cluster DNS:

http://loki-gateway.grafana-loki.svc.cluster.local/loki/api/v1/push

You can test to send data from outside the cluster by port-forwarding the gateway to your local machine:

kubectl port-forward --namespace grafana-loki svc/loki-gateway 3100:80 &

And then using http://127.0.0.1:3100/loki/api/v1/push URL as shown below:

curl \

-H "Content-Type: application/json" \

-XPOST \

-s "http://127.0.0.1:3100/loki/api/v1/push" \

--data-raw "{\"streams\": [{\"stream\": {\"job\": \"test\"}, \"values\": [[\"$(date +%s)000000000\", \"fizzbuzz\"]]}]}" \

-H X-Scope-OrgId:foo

Then verify that Loki did receive the data using the following command:

curl "http://127.0.0.1:3100/loki/api/v1/query_range" \

--data-urlencode 'query={job="test"}' \

-H X-Scope-OrgId:foo | jq .data.result

***********************************************************************

Connecting Grafana to Loki

***********************************************************************

If Grafana operates within the cluster, you'll set up a new Loki datasource by utilizing the following URL:

http://loki-gateway.grafana-loki.svc.cluster.local/

***********************************************************************

Multi-tenancy

***********************************************************************

Loki is configured with auth enabled (multi-tenancy) and expects tenant headers (`X-Scope-OrgID`) to be set for all API calls.

You must configure Grafana's Loki datasource using the `HTTP Headers` section with the `X-Scope-OrgID` to target a specific tenant.

For each tenant, you can create a different datasource.

The agent of your choice must also be configured to propagate this header.

For example, when using Promtail you can use the `tenant` stage. https://grafana.com/docs/loki/latest/send-data/promtail/stages/tenant/

When not provided with the `X-Scope-OrgID` while auth is enabled, Loki will reject reads and writes with a 404 status code `no org id`.

You can also use a reverse proxy, to automatically add the `X-Scope-OrgID` header as suggested by https://grafana.com/docs/loki/latest/operations/authentication/

For more information, read our documentation about multi-tenancy: https://grafana.com/docs/loki/latest/operations/multi-tenancy/

> When using curl you can pass `X-Scope-OrgId` header using `-H X-Scope-OrgId:foo` option, where foo can be replaced with the tenant of your choice.

EOT -> (known after apply)

---

Grafana Observability Stack

2026-02-20T14:13:00.003+00:00

Grafana uses these components together as an observability stack, but each has a clear role:

Loki – log database. It stores and indexes logs (especially from Kubernetes) in a cost‑efficient, label‑based way, similar to Prometheus but for logs.

Tempo – distributed tracing backend. It stores distributed traces (spans) from OpenTelemetry, Jaeger, Zipkin, etc., so you can see call flows across microservices and where latency comes from.

Mimir – Prometheus‑compatible metrics backend. It is a horizontally scalable, long‑term storage and query engine for Prometheus‑style metrics (time series).

Alloy – telemetry pipeline (collector). It is Grafana’s distribution of the OpenTelemetry Collector / Prometheus agent / Promtail ideas, used to collect, process, and forward metrics, logs, traces, profiles into Loki/Tempo/Mimir (or other backends).

How Grafana UI relates to them

Grafana UI itself is “just” the visualization and alerting layer:

It connects to Loki, Tempo, Mimir (and many others) as data sources.
For each backend you configure:

A Loki data source for logs.
A Tempo data source for traces.
A Prometheus/Mimir data source for metrics (Mimir exposes a Prometheus‑compatible API).

Grafana then lets you:

Build dashboards and alerts from Mimir metrics.
Explore logs from Loki.
Explore traces from Tempo and cross‑link them with logs/metrics (e.g., click from a log line to a trace, or from a metrics graph into logs/traces).

A useful mental model: Loki/Tempo/Mimir are databases, Alloy is the collector/router, and Grafana is the UI on top.

Are they deployed in the same Kubernetes cluster?

Common patterns:

Very common: deploy Loki, Tempo, Mimir, Alloy, and Grafana in the same Kubernetes cluster as your apps. This is the typical “in‑cluster LGTM” setup; all telemetry stays inside the cluster and traffic is simple.
Also common: run them in a separate observability cluster (or use Grafana Cloud backends), while Alloy/agents run in each workload cluster and ship data over the network. This improves isolation and makes it easier to share one observability stack across many clusters.
In smaller setups or dev environments, everything (apps + LGTM + Grafana) often lives in one cluster; in larger/regulated setups, people tend to separate “workload clusters” and an “observability cluster”.

So: they don’t have to be on the same cluster, but it’s perfectly normal (and often simplest) to run Grafana + Loki + Tempo + Mimir + Alloy together in a single Kubernetes cluster and point your apps’ telemetry to Alloy.

Why not using elasticsearch instead of loki, tempo and mimir?

Elasticsearch can replace part of what Loki, Tempo, and Mimir do, but not all of it, and usually with higher cost/complexity for cloud‑native observability.

1. Scope: logs vs full observability

Elasticsearch is a general search and analytics engine that’s great at full‑text search, aggregations, and analytics over documents (including logs).

The LGTM stack is explicitly split by signal:

Loki → logs
Tempo → traces
Mimir → metrics

Each is optimized only for its signal type and integrates tightly with Grafana and modern telemetry standards.

You could plausibly replace Loki with Elasticsearch for logs, but Elasticsearch does not natively replace Tempo (distributed tracing backend) or Mimir (Prometheus‑compatible metrics backend).

2. Logs: Loki vs Elasticsearch

Elasticsearch strengths:

Very powerful full‑text search, fuzzy matching, relevance scoring, complex aggregations.
Good when you need deep forensic search and advanced analytics on log text.

Loki strengths:

Stores logs as compressed chunks plus a small label index, so storage and compute are much cheaper than Elasticsearch for typical Kubernetes logs.
Very tight integration with Grafana and the rest of LGTM, and simple, label‑based querying.

Trade‑off: Elasticsearch gives richer search at a high infra + ops cost, Loki gives “good enough” search for operational troubleshooting with much lower cost and operational burden.

3. Traces and metrics: Tempo & Mimir vs “just ES”

Tempo:

Implements distributed tracing concepts (spans, traces, service graphs) and OpenTelemetry/Jaeger/Zipkin protocols; the data model and APIs are specialized for traces.
Elasticsearch can store trace‑like JSON documents, but you’d have to build/maintain all the trace stitching, UI navigation, and integrations yourself.

Mimir:

Is a horizontally scalable, Prometheus‑compatible time‑series database, with native remote‑write/read and PromQL semantics.
Elasticsearch can store time‑stamped metrics, but you lose Prometheus compatibility, PromQL semantics, and the whole ecosystem that expects a Prometheus‑style API.

So using only Elasticsearch means you’re giving up the standard metrics and tracing ecosystems and rebuilding a lot of tooling on top of a generic search engine.

4. Cost, complexity, and operational burden

Elasticsearch clusters generally need:

More RAM/CPU per node, careful shard and index management, and capacity planning.
Storage overhead from full‑text indexes (often 1.5–3× raw log size plus replicas).

Loki/Tempo/Mimir:

Are designed for object storage, compression, and label‑only indexing, which dramatically lowers storage and compute requirements for logs and metrics.
Have simpler, well‑documented reference architectures specifically for observability.

For a modern Kubernetes‑centric environment, that usually makes LGTM cheaper and easier to run than a single big Elasticsearch cluster for everything.

5. When Elasticsearch still makes sense

You might still choose Elasticsearch (often with Kibana/APM) if:

You already have a strong ELK stack and team expertise.
Your primary need is deep, flexible text search and analytics over logs, with less emphasis on Prometheus/OTel ecosystems.
You want Elasticsearch’s ML/anomaly‑detection features and are willing to pay the operational cost.

But if your goal is a Grafana‑centric, standards‑based (Prometheus + OpenTelemetry) observability platform, LGTM (Loki+Tempo+Mimir, plus Alloy as collector) is a better fit than trying to push everything into Elasticsearch.

---

Here document (heredoc)

2026-02-20T11:52:00.001+00:00

Here document (heredoc) redirects a multiline string literal to the preceding command while preserving line breaks. Unix syntax for it is:

[command] <<DELIMITER

First line.

Second line.

Third line.

Fourth line.

DELIMITER

<< is Redirection Operator

- is optional Tab Suppression

DELIMITER - an arbitrary string, Delimiter Token; must be the same at the beginning and at the end

Appending a minus sign to the redirection operator <<- causes all leading tab characters to be ignored. This allows you to use indentation when writing heredocs in shell scripts. We can then indent both the here-doc and the delimiter with tabs (not spaces!):

#! /bin/bash

cat <<-EOF

indented

EOF

echo Done

---

References:

Here document - Wikipedia

How to fix pods in Not Ready state?

2026-02-18T15:30:00.000+00:00

kubectl get pods might show that some of the pods have 0/N value in READY column.

What is the meaning of READY column value?

In the context of kubectl get pods, the READY column shows the number of containers in the pod that have passed their health checks and are ready to serve traffic.

The anatomy of R/T:

R (Left side): This is the number of containers currently Ready. A 0 means the application inside the container is not responding to its "Readiness Probe" or has not finished starting up.
/ (Separator): Separates ready containers from the total.
T (Right side): This is the Total number of user containers defined in that pod.

When we have a pod in 0/1 state with a status of Running, it means the container has started, but Kubernetes does not consider it "healthy" enough to handle requests. Our pods are technically "alive" (Running) but "unusable" (Not Ready).

Common Reasons for 0/1 Running:

Failed Readiness Probe: The application is running, but the health check URL (e.g., /ready) is returning an error or timing out.
Slow Startup: The application takes a long time to initialize, and the "Initial Delay" isn't long enough.
Dependency Issues: The pod is waiting for a database, a config file, or another service that isn't available.
CrashLoopBackOff (Transitions): Sometimes pods flicker between 0/1 Running and 0/1 CrashLoopBackOff as they try to start and immediately fail.

How to find out exactly what's wrong

To diagnose why our my-app pods are stuck, run these two commands:

(1) Check the Events (Why it's not ready):

kubectl describe pod my-app-0 -n my-app-namespace

Look at the "Events" section at the bottom for messages like "Readiness probe failed."

Output example:

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Warning Unhealthy 47s (x81205 over 8d) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503

(2) Check the Logs (What the app is saying):

kubectl logs my-app-0 -n my-app-namespace

Look for "Error," "Exception," or "Connection Refused" messages.

---

How to use terraform-docs automatically generate Terraform code documentation

2026-02-17T16:14:00.005+00:00

terraform-docs is a tool used to automatically generate Terraform code documentation.

To install it on Mac:

% brew install terraform-docs

To verify installation:

% terraform-docs --version

terraform-docs version v0.21.0 darwin/arm64

To generate a documentation for a module in the current directory and append it to the README file (which is in the same directory):

% terraform-docs markdown table --output-file README.md --output-mode inject ./

How to install Terraform on Mac

2026-02-17T14:23:00.007+00:00

First add Hashicorp's package repository:

% brew tap hashicorp/tap

Then install the Terraform:

% brew install hashicorp/tap/terraform

If Terraform was already installed, the command above will update it.

To verify installation, we can check its version:

% terraform --version

Terraform v1.14.5

on darwin_arm64

Amazon EKS Autoscaling with Karpenter

2026-02-06T12:18:00.015+00:00

Kubernetes autoscaling is a function that scales resources in and out depending on the current workload. AWS supports two autoscaling implementations:

Cluster Autoscaler

autoscaler/cluster-autoscaler/cloudprovider/aws/README.md at master · kubernetes/autoscaler
automatically adjusts the number of nodes in the cluster when pods fail or are rescheduled onto other nodes
uses Auto Scaling groups

Karpenter

Karpenter
flexible, high-performance Kubernetes cluster autoscaler and node provisioner
helps improve application availability and cluster efficiency
launches right-sized compute resources (for example, Amazon EC2 instances) in response to changing application load in under a minute
can provision just-in-time compute resources that precisely meet the requirements of our workload
automatically provisions new compute resources based on the specific requirements of cluster workloads. These include compute, storage, acceleration, and scheduling requirements.
creates Kubernetes nodes directly from EC2 instances
improves the efficiency and cost of running workloads on the cluster
open-source

Pod Scheduler

Kubernetes cluster component responsible for determining which node Pods get assigned to
default Pod scheduler for Kubernetes is kube-scheduler

logs the reasons Pods can't be scheduled

Unschedulable Pods

From How to troubleshoot unschedulable Pods in Kubernetes:

A Pod is unschedulable when it's been put into Kubernetes' scheduling queue, but can't be deployed to a node. This can be for a number of reasons, including:
The cluster not having enough CPU or RAM available to meet the Pod's requirements.
Pod affinity or anti-affinity rules preventing it from being deployed to available nodes.
Nodes being cordoned due to updates or restarts.
The Pod requiring a persistent volume that's unavailable, or bound to an unavailable node.

How to detect unschedulable Pods?
Pods waiting to be scheduled are held in the "Pending" status, but if the Pod can't be scheduled, it will remain in this state. However, Pods that are being deployed normally are also marked as "Pending." The difference comes down to how long a Pod remains in "Pending."

How to fix unschedulable Pods?

There is no single solution for unschedulable Pods as they have many different causes. However, there are a few things we can try depending on the cause.

Enable cluster autoscaling
If we're using a managed Kubernetes service like Amazon EKS or Google Kubernetes Engine (GKE), we can very easily take advantage of autoscaling to increase and decrease cluster capacity on-demand. With autoscaling enabled, Kubernetes' Cluster Autoscaler will trigger our provider to add nodes when needed. As long as we've configured our cluster node pool and it hasn't reached its max node limit, our provider will automatically provision a new node and add it to the pool, making it available to the cluster and to our Pods.
Increase our node capacity
Check our Pod requests
Check our affinity and anti-affinity rules

In this article we'll show how to enable cluster autoscaling with Karpenter.

How does the regular Kubernetes Autoscaler work in AWS?

When we create a regular Kubernetes cluster in AWS, each node group is managed by the AWS Auto-scaling group [Auto Scaling groups - Amazon EC2 Auto Scaling]. Cluster native autoscaler adjusts the desired size based on the load in the cluster to fit all unscheduled pods.

HorizontalPodAutoscaler (HPA) [Horizontal Pod Autoscaling | Kubernetes] is built into Kubernetes and it uses metrics like CPU usage, memory usage or custom metrics we can write to decide when to spin up or down additional pods in the node of the cluster. If our app is receiving more traffic, HPA will kick in and provision additional pods.

VerticalPodAutoscaler (VPA) can also be installed in cluster where it manages the resource (like CPU and memory) allocation to pods that are already running.

What about when there's not enough capacity to schedule any more pods in the node? That's when we'll need an additional node. So we have a pod that needs to be scheduled but we don't know where to put it. We could call AWS API, spin up an additional EC2 node, get added it to our cluster or if we're using managed groups we can use Managed Node Group API, bump up the desired size but easier approach is to use cluster auto-scaler. There is a mature open-source solution called Cluster Auto-Scaler (CAS).

CAS was built to handle hundreds of different combinations of nodes types, zones, purchase options available in AWS. CAS works directly with managed node groups or self-managed managed nodes and auto-scaling groups which are AWS constructs to help us manage nodes.

What are the issues with the regular Kubernetes Autoscaler?

Let's say CAS is installed on node, in cluster and manages one managed node group (MNG). It's filling up and we have an additional pod that needs to be provisioned so CAS tells MNG to bump up the number of nodes so it spins up another one so pod can now be scheduled. But this is not ideal. We have a single pod in a node, we don't need such a big node.

This can be solved by creating a different MNG with a smaller instance type and now CAS recognizes that instance and provisions pod on a more appropriately-sized node.

Unfortunately, we might end up with many MNGs, based on requirements which might be a challenge to manage especially when looking best practices in terms of cost efficiency and high availability.

How does Karpenter work?

Karpenter works differently, It doesn't use MNG or ASGs and manages each node directly. Let's say we have different pods, of different sizes. Let's say that HPA says that we need more of the smaller pods. Karpenter will intelligently pick the right instance type for that workload. If we need to spin up a larger pod it will again pick the right instance type.

Karpenter picks exactly the right type of node for our workload.

If we're using spot instances and spot capacity is not available, Karpenter does retries more quickly. Karpenter offers, faster, dynamic, more intelligent compute, using best practices without operational overhead of managing nodes ourselves.

How to control how Karpenter operates?

There are many dimensions here. We can set constraints on Karpenter to limit the instances type, we can set up taints to isolate workloads to specific types of nodes. Different teams can have isolated access to different pods, one team can access billing pods, another GPU-based instances.

Workload Consolidation feature: Pods are consolidated into fewer nodes.. let's say we have 3 nodes, two at 70% and one at 20% utilization. Karpenter detects this and will move pods from underutilized node to those two and shut down this now empty node (instances are terminated). This leads to lower costs.

Karpenter is making it easier to use spot and graviton instances which can also lead to lower costs.

A feature to keep our nodes up to date. ttlSecondsUntilExpired parameter tells Karpenter to terminate nodes after a set amount of time. These nodes will automatically be replaced with new nodes, running the latest AMIs.

Karpenter:

1) lower costs

2) higher application availability

3) lower operation overhead

Karpenter needs permissions to create EC2 instances in AWS.

If we use a self-hosted (on bare metal boxes or EC2 instances), self-managed (we have full control over all aspects of Kubernetes) Kubernetes cluster, for example by using kOps (see also Is k8s Kops preferable than eks? : r/kubernetes), we can add additional IAM policies to the existing IAM role attached to Kubernetes nodes.

If using EKS, the best way to grant access to internal service is with IAM roles for service accounts (IRSA).

Karpenter's Kubernetes Custom Resources

NodePool

NodePool is the primary Custom Resource (CR) in Karpenter that defines scheduling constraints, how nodes are provisioned and managed (node management policies). It is the successor to the older Provisioner API and acts as the brain that tells Karpenter which nodes to create and how to handle them over time. It acts as the "brain" for scheduling decisions by evaluating the requirements of pending pods and matching them to infrastructure constraints.

Core Role of NodePool

Scheduling Authority: It defines the constraints (instance types, zones, architectures) that determine which nodes can be created.
Successor to Provisioner: It replaced the older Provisioner API to provide a more scalable and configuration-based approach.
Management Hub: It handles node lifecycle settings, including disruption policies (consolidation and expiration) and aggregate resource limits (CPU/Memory).

Core Functions

A NodePool manages three primary aspects of our cluster's compute capacity:

Scheduling Constraints: Restricts which nodes can be provisioned using requirements for instance types, zones, architectures (e.g., x86 vs. ARM), and capacity types (Spot vs. On-Demand).
Disruption Policies: Governs how Karpenter optimizes the cluster by defining when nodes should be expired or consolidated to save costs.
Resource Limits: Sets a cap on the total CPU and memory that the NodePool can provision, preventing runaway costs.

Key Components of a NodePool

The specification is divided into several functional areas:

template: Defines the configuration for the nodes that will be created.
requirements: Uses well-known Kubernetes labels (e.g., karpenter.sh/capacity-type) to select hardware.
nodeClassRef: Points to an EC2NodeClass for cloud-provider-specific settings like subnets and security groups.
disruption: Replaces older TTL settings with a unified policy for consolidationPolicy (e.g., WhenUnderutilized) and expireAfter.
limits: Defines the maximum aggregate resources (e.g., cpu: 1000) allowed for this pool.

Example v1 Configuration

This example demonstrates a production-ready NodePool that prioritises Spot instances but allows for On-

Demand fallback.

apiVersion: karpenter.sh/v1

kind: NodePool

metadata:

spec:

template:

spec:

requirements:

- key: "karpenter.sh/capacity-type"

operator: In

values: ["spot", "on-demand"]

- key: "karpenter.k8s.aws/instance-category"

operator: In

values: ["c", "m", "r"]

- key: "kubernetes.io/arch"

operator: In

values: ["amd64", "arm64"]

nodeClassRef:

group: karpenter.k8s.aws

kind: EC2NodeClass

disruption:

consolidationPolicy: WhenUnderutilized

expireAfter: 720h # 30 days

limits:

cpu: "500"

memory: 1000Gi

Comparison with Other Objects

While the NodePool is the central configuration object, it works in a hierarchy with two other key resources:

NodePool

Purpose: The Logic: Defines what nodes should look like and how they should behave.

EC2NodeClass

Purpose: The Infrastructure: Defines where and with what AWS-specific settings (subnets, AMIs, security groups) nodes launch.

NodeClaim

Purpose: The Instance: Represents an individual node currently being managed or provisioned by Karpenter.

Every NodePool must reference at least one EC2NodeClass to successfully provision capacity on AWS.

Useful Commands:

To see all node pools:

% kubectl get nodepools

NAME NODECLASS NODES READY AGE

clickhouse clickhouse 0 True 140d

clickhouse-backup clickhouse-backup 0 True 140d

Cluster user needs to have permission to list resource "nodepools" in API group "karpenter.sh" at the cluster scope.

To debug a specific node pool:

kubectl describe nodepool <nodepool-name>

Cluster user needs to have permission to get resource "nodepools" in API group "karpenter.sh" at the cluster scope.

% kubectl describe nodepool clickhouse

Name: clickhouse

Namespace:

Labels: <none>

Annotations: karpenter.sh/nodepool-hash: 12671849087427876759

karpenter.sh/nodepool-hash-version: v3

API Version: karpenter.sh/v1

Kind: NodePool

Metadata:

Creation Timestamp: 2025-10-22T15:02:58Z

Generation: 2

Resource Version: 1073678

UID: f7869dd3-ac24-4600-98a6-059073645769

Spec:

Disruption:

Budgets:

Nodes: 10%

Consolidate After: 0s

Consolidation Policy: WhenEmptyOrUnderutilized

Template:

Metadata:

Labels:

Karpenter - Node - Pool: clickhouse

Spec:

Expire After: 720h

Node Class Ref:

Group: karpenter.k8s.aws

Kind: EC2NodeClass

Name: clickhouse

Requirements:

Key: node.kubernetes.io/instance-type

Operator: In

Values:

r8g.xlarge

r8g.2xlarge

r8g.4xlarge

r8g.8xlarge

Key: karpenter.sh/capacity-type

Operator: In

Values:

on-demand

spot

Status:

Conditions:

Last Transition Time: 2025-10-22T15:02:59Z

Message:

Observed Generation: 2

Reason: ValidationSucceeded

Status: True

Type: ValidationSucceeded

Last Transition Time: 2025-10-22T15:03:07Z

Message:

Observed Generation: 2

Reason: NodeClassReady

Status: True

Type: NodeClassReady

Last Transition Time: 2025-10-23T17:24:01Z

Message:

Observed Generation: 2

Reason: Ready

Status: True

Type: Ready

Resources:

Cpu: 0

Ephemeral - Storage: 0

Memory: 0

Nodes: 0

Pods: 0

Events: <none>

EC2NodeClass

EC2NodeClass is a Custom Resource (CR) used to define AWS-specific infrastructure configurations for the nodes Karpenter provisions.

While a NodePool handles high-level scheduling constraints (like instance types or taints), the EC2NodeClass dictates the underlying Amazon EC2 settings.

Key Responsibilities

The EC2NodeClass abstracts cloud provider-specific details, including:

Networking: Selects subnets using subnetSelectorTerms.
Security: Identifies security groups via securityGroupSelectorTerms.
Identity: Assigns the IAM role or instance profile for the nodes.
Storage: Configures blockDeviceMappings for EBS volumes.
Images: Specifies the Amazon Machine Image (AMI) family (e.g., AL2, Bottlerocket) or selects specific AMIs.
Customisation: Includes userData for custom bootstrap scripts.

Relationship with NodePools

A NodePool must reference an EC2NodeClass using the nodeClassRef field. Multiple NodePools can point to the same EC2NodeClass if they share the same infrastructure requirements (e.g., same VPC and IAM role).

Example Configuration

A basic EC2NodeClass manifest typically looks like this:

apiVersion: karpenter.k8s.aws/v1

kind: EC2NodeClass

metadata:

spec:

amiFamily: AL2

role: "KarpenterNodeRole-my-cluster"

subnetSelectorTerms:

- tags:

karpenter.sh/discovery: my-cluster

securityGroupSelectorTerms:

- tags:

karpenter.sh/discovery: my-cluster

Useful Commands:

To see all EC2NodeClasses

kubectl get ec2nodeclasses

Cluster user needs to have permission to list resource "ec2nodeclasses" in API group "karpenter.k8s.aws" at the cluster scope.

Example:

% kubectl get ec2nodeclasses

NAME READY AGE

clickhouse True 140d

clickhouse-backup True 140d

To debug a specific node that isn't coming online:

kubectl describe ec2nodeclasses <ec2nodeclass-name>

Cluster user needs to have permission to get resource "ec2nodeclasses" in API group "karpenter.k8s.aws" at the cluster scope.

Example:

% kubectl describe ec2nodeclass clickhouse

Name: clickhouse

Namespace:

Labels: <none>

Annotations: karpenter.k8s.aws/ec2nodeclass-hash: 358699366951558737

karpenter.k8s.aws/ec2nodeclass-hash-version: v4

API Version: karpenter.k8s.aws/v1

Kind: EC2NodeClass

Metadata:

Creation Timestamp: 2025-10-22T15:02:58Z

Finalizers:

karpenter.k8s.aws/termination

Generation: 1

Resource Version: 73323969

UID: 25c663e7-cc29-47b2-8a97-937fb5f39825

Spec:

Ami Family: AL2023

Ami Selector Terms:

Alias: al2023@latest

Detailed Monitoring: true

Metadata Options:

Http Endpoint: enabled

httpProtocolIPv6: disabled

Http Put Response Hop Limit: 1

Http Tokens: required

Role: KarpenterNodeRole-mycorp-prod-clickhouse-k8s

Security Group Selector Terms:

Tags:

karpenter.sh/discovery/mycorp-prod-clickhouse-k8s: true

Subnet Selector Terms:

Tags:

karpenter.sh/discovery: true

private_subnet: true

Tags:

Name: mycorp-prod-clickhouse-k8s-karpenter-clickhouse

karpenter.sh/discovery/mycorp-prod-clickhouse-k8s: true

Status:

Amis:

Id: ami-06ab427136b8ffa61

Name: amazon-eks-node-al2023-x86_64-nvidia-1.33-v20260304

Requirements:

Key: kubernetes.io/arch

Operator: In

Values:

amd64

Key: karpenter.k8s.aws/instance-gpu-count

Operator: Exists

Id: ami-08f492a005f7b8703

Name: amazon-eks-node-al2023-x86_64-neuron-1.33-v20260304

Requirements:

Key: kubernetes.io/arch

Operator: In

Values:

amd64

Key: karpenter.k8s.aws/instance-accelerator-count

Operator: Exists

Id: ami-0023c4931d42779e6

Name: amazon-eks-node-al2023-x86_64-standard-1.33-v20260304

Requirements:

Key: kubernetes.io/arch

Operator: In

Values:

amd64

Key: karpenter.k8s.aws/instance-gpu-count

Operator: DoesNotExist

Key: karpenter.k8s.aws/instance-accelerator-count

Operator: DoesNotExist

Id: ami-061bed77c8a6d03cd

Name: amazon-eks-node-al2023-arm64-standard-1.33-v20260304

Requirements:

Key: kubernetes.io/arch

Operator: In

Values:

arm64

Key: karpenter.k8s.aws/instance-gpu-count

Operator: DoesNotExist

Key: karpenter.k8s.aws/instance-accelerator-count

Operator: DoesNotExist

Conditions:

Last Transition Time: 2025-10-22T15:02:59Z

Message:

Observed Generation: 1

Reason: AMIsReady

Status: True

Type: AMIsReady

Last Transition Time: 2025-10-22T15:02:59Z

Message:

Observed Generation: 1

Reason: SubnetsReady

Status: True

Type: SubnetsReady

Last Transition Time: 2025-10-22T15:02:59Z

Message:

Observed Generation: 1

Reason: SecurityGroupsReady

Status: True

Type: SecurityGroupsReady

Last Transition Time: 2025-10-22T15:02:59Z

Message:

Observed Generation: 1

Reason: InstanceProfileReady

Status: True

Type: InstanceProfileReady

Last Transition Time: 2025-10-22T15:03:07Z

Message:

Observed Generation: 1

Reason: ValidationSucceeded

Status: True

Type: ValidationSucceeded

Last Transition Time: 2025-10-22T15:03:07Z

Message:

Observed Generation: 1

Reason: Ready

Status: True

Type: Ready

Instance Profile: mycorp-prod-clickhouse-k8s_15693974848685646064

Security Groups:

Id: sg-09f3cd41bcef827c0

Name: mycorp-prod-clickhouse-k8s-node-20251020164545608400000006

Subnets:

Id: subnet-04xxxxxxxxxx5d30b

Zone: us-east-1b

Zone ID: use1-az2

Id: subnet-00xxxxxxxxxx08cef

Zone: us-east-1c

Zone ID: use1-az3

Id: subnet-02xxxxxxxxxxx8711

Zone: us-east-1a

Zone ID: use1-az1

Events: <none>

NodeClaim

In Karpenter, a NodeClaim is the Custom Resource (CR) that represents a single, specific instance of compute capacity.

While a NodePool is the template and a NodeClass is the blueprint, the NodeClaim is the actual request sent to the cloud provider to launch a specific node.

Key Characteristics

1:1 Relationship: Each NodeClaim typically corresponds to exactly one EC2 instance and its associated Kubernetes Node.
Immutable: Once created, a NodeClaim cannot be changed. If the requirements for a node change (e.g., due to "drift"), Karpenter deletes the existing NodeClaim and creates a new one.
Lifecycle Management: It tracks the instance from its initial "launch" request through "registration" with the cluster until it is fully "initialized" and ready to run pods.

Why We Should Monitor NodeClaims

NodeClaims are the best place to look when debugging provisioning failures. We can use them to identify why a node failed to join the cluster:

Status Conditions: A NodeClaim status will show if an instance failed to launch (e.g., "LaunchFailed" due to AWS service limits) or if it's stuck during initialization.
Provider Information: It stores the specific providerID (e.g., EC2 instance ID) and the nodeName once the instance successfully joins the cluster.
Drift Detection: Karpenter adds a Drifted status to a NodeClaim if the underlying instance no longer matches the current NodePool or EC2NodeClass configuration.

How it Fits the Hierarchy:

NodePoolDefines

Function: scheduling rules and resource limits.
Analogy: The Foreman

EC2NodeClass

Function: Defines AWS-specific infrastructure (subnets, AMIs).
Analogy: The Blueprint

NodeClaim

Function: Represents the actual instance being provisioned.
Analogy: The Building Material

Useful Commands:

To see all active node claims and their status:

kubectl get nodeclaims

Cluster user needs to have permission to list resource "nodeclaims" in API group "karpenter.sh" at the cluster scope.

To debug a specific node that isn't coming online:

kubectl describe nodeclaim <nodeclaim-name>

Cluster user needs to have permission to get resource "nodeclaims" in API group "karpenter.sh" at the cluster scope.

Example:

% kubectl get nodeclaims

No resources found

How to Debug Misfunctioning Karpenter

Based on the output we've provided above, our NodePool and EC2NodeClass are both in a Ready state and have correctly discovered subnets, security groups, and AMIs. However, we have 0 nodes and 0 NodeClaims. The most common reason for this behavior—where the infrastructure is ready but Karpenter isn't scaling—is a scheduling mismatch between our pending Pods and the NodePool's requirements. Here is a breakdown of why Karpenter might be sitting idle:

1. Missing Architecture Requirement

Our clickhouse NodePool specifies r8g instance types. These are AWS Graviton (ARM64) instances. In our NodePool spec, we haven't explicitly defined the architecture requirement. While Karpenter is smart, if our Pods do not have a nodeSelector or toleration for kubernetes.io/arch: arm64, or if the Pods are explicitly asking for amd64, Karpenter will not use this NodePool to satisfy them.Check our Pods:

kubectl get pods -A | grep -i pending

kubectl describe pod <pending-pod-name>

Look for NodeSelectors or Affinity rules in the Pod spec. If the Pod doesn't allow arm64, Karpenter won't spin up an r8g node.

2. Missing Taints/Tolerations

ClickHouse workloads often use specialized nodes. If our NodePool (or the nodes it creates) has a Taint that we haven't shown in the snippet, or if our Pods have a Toleration for a taint that doesn't exist on this NodePool, Karpenter will ignore it.

Check if our pending Pods have specific nodeSelector labels that aren't present in the NodePool template.metadata.labels. Our NodePool only has one label: Karpenter - Node - Pool: clickhouse.

3. The "Karpenter Controller" Logs

If the logic seems correct but nothing is happening, the answer is always in the controller logs. Karpenter will explicitly tell us why it is passing over a Pod.Run this to see the scheduling decisions:

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter | grep -i "scheduling"

Look for messages like:no reachable nodeclassesno possible pod inventoryunschedulable, ... did not match requirements

4. Service Linked Role / Permissions

Since our EC2NodeClass is Ready, our basic AWS tags are likely fine. However, double-check that the KarpenterNodeRole-geeiq-prod-clickhouse-k8s actually exists in IAM and has the AmazonEKSWorkerNodePolicy and AmazonEC2ContainerRegistryReadOnly attached. If the role is missing or misconfigured, the EC2 instance might start but fail to join the cluster, causing Karpenter to terminate it immediately.

Summary Checklist

Potential Issue Fix/Action

Arch Mismatch => Add kubernetes.io/arch with arm64 to NodePool requirements or Pod nodeSelector.

Pending Pods => Ensure there are actually Pods in Pending state. Karpenter only scales in response to unschedulable pods.

Instance Availability => r8g instances are relatively new. Ensure they are available in us-east-2 for the capacity type (Spot/On-Demand) we requested.

How to install Karpenter in the cluster?

Namespace

Current Karpenter best practices recommend deploying it in the kube-system namespace rather than its own dedicated namespace.

While many early adopters used a separate karpenter namespace, the project shifted toward kube-system starting with version v0.33.0.

Why kube-system is preferred:

API Priority & Fairness: By default, Kubernetes grants higher priority to requests coming from the kube-system namespace. This ensures the Karpenter controller can still communicate with the API server to provision nodes even during periods of heavy cluster congestion.
Critical Component Status: Placing Karpenter in kube-system denotes it as a critical cluster component, aligning it with other essential services like kube-proxy or the VPC CNI.
Reduced Complexity: Using a standard system namespace avoids the need to manually configure custom FlowSchemas or PriorityLevelConfigurations that would otherwise be required to give a custom namespace the same level of reliability.

When to use a separate namespace:

Legacy Installations: If we installed Karpenter before v0.33.0, it likely lives in a karpenter namespace. Migrating is recommended but requires updating our IAM Roles for Service Accounts (IRSA) trust policy to reflect the new namespace.
Fargate Isolation: If we run Karpenter on AWS Fargate, we must create a Fargate profile for the specific namespace where Karpenter is deployed.

Labels

While most standard Karpenter installations use the label app.kubernetes.io/name=karpenter for the controller pods, it is not guaranteed for every pod in every environment.

Why it might differ:

Helm Chart Customisation: If we (or our platform team) overrode the podLabels or nameOverride values during the Karpenter Helm installation, this label will be different.
Version Variance: Very old versions of Karpenter sometimes used different labelling conventions (e.g., just app=karpenter), though modern versions follow Kubernetes recommended labels.
Webhook Pods: In some configurations, Karpenter may run separate pods for webhooks that might carry slightly different descriptive labels depending on the deployment strategy.

How to configure Karpenter?

We can configure specific Karpenter NodePools or Provisioners.

How to set up weighted NodePools for multi-tenant isolation?

In Karpenter, Weighted NodePools allow us to control which NodePool is selected when a pod's requirements match multiple pools. This is a powerful tool for multi-tenant isolation, enabling us to prioritize specific hardware or cost models for certain teams while providing a fallback mechanism.

How Weighting Works

Precedence: Higher weight values indicate higher priority.
Default: If no weight is specified, it defaults to 0.
Selection: If a pending pod matches the requirements of multiple NodePools, Karpenter will always select the one with the highest weight first.

Multi-Tenant Strategy: Isolation & Priority

For multi-tenant environments, we can use weights to enforce distinct tiers of service or cost:

Reserved/Savings Plan Tier (Highest Weight):

Create a NodePool that specifically includes instance types covered by our Savings Plans or Reserved Instances. By giving this pool a high weight (e.g., 100), Karpenter will prioritize using this pre-paid capacity before launching new nodes.

Spot Instance Tier (Medium Weight):

A general-purpose pool for non-critical workloads or "Team A" can be set with a medium weight (e.g., 50) and restricted to spot capacity.

On-Demand Fallback (Lowest Weight):

A "catch-all" NodePool with a low weight (e.g., 10) that allows on-demand instances. This ensures that if Spot capacity is unavailable or Savings Plans are exhausted, workloads still have a place to land.

Implementation Example

Below is an example of two overlapping NodePools where the "Premium" pool is prioritized for any workload that could run on it.

# NodePool 1: High Priority (e.g., Reserved Capacity)

apiVersion: karpenter.sh/v1

kind: NodePool

metadata:

spec:

weight: 100 # Higher weight = Higher priority

template:

spec:

requirements:

- key: "node.kubernetes.io/instance-type"

operator: In

values: ["m5.large", "m5.xlarge"] # Specific reserved types

nodeClassRef:

---

# NodePool 2: Standard Priority (e.g., Spot)

apiVersion: karpenter.sh/v1

kind: NodePool

metadata:

spec:

weight: 50

template:

spec:

requirements:

- key: "karpenter.sh/capacity-type"

operator: In

values: ["spot"]

nodeClassRef:

Best Practices for Isolation

Mutual Exclusivity: While weights handle overlaps, the official Karpenter guidance suggests making NodePools mutually exclusive whenever possible (using taints/tolerations or unique labels) to simplify debugging.
Resource Limits: Always set spec.limits on tenant-specific pools to prevent one team from consuming the entire cluster's budget.
Billing Attribution: Use the spec.template.metadata.labels field in each NodePool to add "Team" or "Project" tags. These labels propagate to the EC2 instances, making it easy to track costs per tenant

How to implement Taints and Tolerations alongside weights for stricter tenant "hard" isolation?

While weights allow Karpenter to prefer one NodePool over another, Taints and Tolerations are required for hard isolation. They ensure that nodes provisioned for one tenant "repel" pods from all other tenants.

The Isolation Strategy

To achieve strict tenant separation, we combine three elements:

Taints: Applied to the NodePool to prevent unauthorized pods from scheduling on its nodes.
Tolerations: Applied to the tenant's pods so they can "bypass" the taint.
Node Affinity: Applied to the tenant's pods to "attract" them specifically to their dedicated nodes.

1. Dedicated Tenant NodePool

In the NodePool spec, add a taint. Any node Karpenter creates from this pool will automatically carry this "keep out" sign.

apiVersion: karpenter.sh/v1

kind: NodePool

metadata:

spec:

weight: 50

template:

spec:

taints:

- key: "tenant"

value: "team-a"

effect: "NoSchedule" # Only pods with matching toleration can land here

labels:

tenant: "team-a" # Used for affinity

nodeClassRef:

2. Tenant Pod Configuration

For Team A's workloads to run, their pods must explicitly tolerate the taint and prefer (or require) the tenant label.

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- name: app

image: nginx

tolerations:

- key: "tenant"

operator: "Equal"

value: "team-a"

effect: "NoSchedule"

affinity:

nodeAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

nodeSelectorTerms:

- matchExpressions:

- key: "tenant"

operator: In

values: ["team-a"]

Why use both?

Taint + Toleration alone stops other pods from accidentally using Team A's nodes, but it doesn't stop Team A's pods from accidentally landing on "General" nodes.
Node Affinity ensures Team A's pods only go to their dedicated nodes.
Weights (e.g., weight: 100) can still be used within a tenant's pool to prioritize Spot vs. On-Demand specifically for that tenant.

Best Practices

Mutually Exclusive Pools: It is recommended to design NodePools so they do not overlap. If a pod matches multiple pools, Karpenter uses the one with the highest weight.
NoExecute for Critical Changes: Use the NoExecute effect if we need to evict existing pods immediately when a node becomes inappropriate for them.
Limit Resources: Always set spec.limits on each tenant pool to prevent a single team's auto-scaling from exhausting the entire AWS account's resources.

How to ensure our cluster has at least 3 nodes spread across 3 different Availability Zones (AZs)?

This is important if we want to implement highly available architecture. We want nodes to be spread across multiple data centres and with them, pod which belong to our application.

We can define a NodePool that forces a spread across zones using topology:

apiVersion: karpenter.sh/v1beta1

kind: NodePool

metadata:

spec:

template:

spec:

requirements:

# Force the nodes to be spread across these zones

- key: "topology.kubernetes.io/zone"

operator: In

values: ["us-east-1a", "us-east-1b", "us-east-1c"]

- key: "karpenter.sh/capacity-type"

operator: In

values: ["on-demand"]

# Ensure the autoscaler keeps a minimum of 3 nodes

limits:

cpu: 1000

BONUS: Forcing Pods to use all 3 Zones

Even if we have 3 nodes in 3 zones, Kubernetes might try to put all our pods on just one of those nodes to be "efficient."

To prevent this, we use Topology Spread Constraints. This is the modern, more powerful version of Anti-Affinity. It ensures our pods are distributed evenly across the zones we just created.

spec:

topologySpreadConstraints:

- maxSkew: 1

topologyKey: "topology.kubernetes.io/zone"

whenUnsatisfiable: DoNotSchedule # Or ScheduleAnyway

labelSelector:

matchLabels:

app: my-app

maxSkew: 1: This means the difference in the number of pods between any two zones can't be more than 1. (e.g., 1-1-1 is fine, 2-1-0 is not).

How to check if Karpenter is deployed and operational in the cluster?

To verify that Karpenter is correctly configured and operational in our EKS cluster, we should follow validation steps described below.

1. Check Controller Health

a) Check Pod Status

Ensure the Karpenter controller pods are running without errors in the dedicated namespace (usually kube-system or karpenter).

We know that its pods should be installed in kube-system namespace and that they should have label app.kubernetes.io/name=karpenter so we can filter pods by these two criterias:

% kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter

NAME READY STATUS RESTARTS AGE

karpenter-598976645b-96dps 1/1 Running 0 11h

karpenter-598976645b-nxm24 1/1 Running 0 12h

b) Inspect Logs

To watch for successful discovery of our cluster endpoint and region use:

% kubectl logs -f -n kube-system -l app.kubernetes.io/name=karpenter -c controller

-f = follow (command does not return)

- l = logs from objects with specified label

-c = only logs from specified container

To verify successful discovery of our EKS cluster endpoint and region, we should look for specific initialisation and informer messages in the Karpenter controller logs.

Key Success Indicators

When Karpenter starts, it must connect to the AWS EKS API to "describe" the cluster. Look for these signs in the output of kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -c controller:

"Starting informers...": This indicates Karpenter has successfully authenticated with the Kubernetes API server and is beginning to watch for unschedulable pods.
Absence of "DescribeCluster" Errors: If discovery is working, we will not see errors like failed to detect the cluster CIDR or AccessDeniedException: ... eks:DescribeCluster.
Region and Cluster Verification: In newer versions, Karpenter logs its configuration during startup. Look for a log entry mentioning the cluster name and AWS region we provided in our Helm values.

Example log:

{"level":"DEBUG","time":"2026-03-11T01:10:28.203Z","logger":"controller","caller":"operator/operator.go:132","message":"discovered karpenter version","commit":"1c39126","version":"1.3.2"}

{"level":"DEBUG","time":"2026-03-11T01:10:28.461Z","logger":"controller","caller":"operator/operator.go:124","message":"discovered region","commit":"1c39126","region":"us-east-1"}

{"level":"DEBUG","time":"2026-03-11T01:10:28.749Z","logger":"controller","caller":"operator/operator.go:129","message":"discovered region","commit":"1c39126","region":"us-east-1"}

{"level":"DEBUG","time":"2026-03-11T01:10:28.909Z","logger":"controller","caller":"operator/operator.go:135","message":"discovered cluster endpoint","commit":"1c39126","cluster-endpoint":"https://CA0xxxxxxx5FDD.yxx.us-east-1.eks.amazonaws.com"}

{"level":"DEBUG","time":"2026-03-11T01:10:28.914Z","logger":"controller","caller":"operator/operator.go:143","message":"discovered kube dns","commit":"1c39126","kube-dns-ip":"172.20.0.10"}

{"level":"INFO","time":"2026-03-11T01:10:28.948Z","logger":"controller.controller-runtime.metrics","caller":"server/server.go:208","message":"Starting metrics server","commit":"1c39126"}

{"level":"INFO","time":"2026-03-11T01:10:28.948Z","logger":"controller","caller":"manager/runnable_group.go:226","message":"starting server","commit":"1c39126","name":"health probe","addr":"[::]:8081"}

{"level":"INFO","time":"2026-03-11T01:10:28.950Z","logger":"controller.controller-runtime.metrics","caller":"server/server.go:247","message":"Serving metrics server","commit":"1c39126","bindAddress":":8080","secure":true}

{"level":"INFO","time":"2026-03-11T01:10:29.052Z","logger":"controller","caller":"leaderelection/leaderelection.go:215","message":"attempting to acquire leader lease kube-system/karpenter-leader-election...","commit":"1c39126"}

{"level":"DEBUG","time":"2026-03-11T06:00:19.215Z","logger":"controller","caller":"provisioning/provisioner.go:128","message":"computing scheduling decision for provisionable pod(s)","commit":"1c39126","controller":"provisioner","namespace":"","name":"","reconcileID":"921af0a4-f057-4041-bff5-d1861d9f72d1","pending-pods":1,"deleting-pods":0}

{"level":"DEBUG","time":"2026-03-11T06:00:21.223Z","logger":"controller","caller":"provisioning/provisioner.go:128","message":"computing scheduling decision for provisionable pod(s)","commit":"1c39126","controller":"provisioner","namespace":"","name":"","reconcileID":"9f0c7833-8e01-4661-8728-890f0001a634","pending-pods":1,"deleting-pods":0}

{"level":"INFO","time":"2026-03-11T06:00:29.230Z","logger":"controller","caller":"lifecycle/controller.go:148","message":"initialized nodeclaim","commit":"1c39126","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"xxxx-ms587"},"namespace":"","name":"xxxxx","reconcileID":"35624d4f-833a-4939-9785-24df4c975e0e","provider-id":"aws:///us-east-1c/i-0123456df20484e26","Node":

{"name":"ip-10-1-46-231.us-east-1.compute.internal"},"allocatable":{"cpu":"3920m","ephemeral-storage":"192128045146","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"15147932Ki","pods":"58"}}

{"level":"DEBUG","time":"2026-03-11T06:00:29.741Z","logger":"controller","caller":"disruption/controller.go:99","message":"marking consolidatable","commit":"1c39126","controller":"nodeclaim.disruption","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"xxxx-ms587"},"namespace":"","name":"xxxx-ms587","reconcileID":"8c5c3d20-36eb-4a78-b0e8-792532db530d","Node":{"name":"ip-10-2-45-230.us-east-1.compute.internal"}}

{"level":"INFO","time":"2026-03-11T06:01:46.399Z","logger":"controller","caller":"disruption/controller.go:193","message":"disrupting node(s)","commit":"1c39126","controller":"disruption","namespace":"","name":"","reconcileID":"acc96c52-0cda-475f-b8a9-1251e7a98dc1","command-id":"3fa9d95e-8f45-48a9-b524-94786e1ac91a","reason":"empty","decision":"delete","disrupted-node-count":1,"replacement-node-count":0,"pod-count":0,"disrupted-nodes":[{"Node":{"name":"ip-10-2-45-230.us-east-1.compute.internal"},"NodeClaim":{"name":"xxxx-ms587"},"capacity-type":"on-demand","instance-type":"m5.xlarge"}],"replacement-nodes":[]}

Common Error Patterns to Watch For

If discovery fails, the logs will explicitly mention connectivity or permission issues:

DNS/Endpoint Issues: Look for i/o timeout or lookup sts.<region>.amazonaws.com. This often means Karpenter can't reach the AWS STS endpoint to get credentials.
IAM Permission Issues: Messages stating is not authorized to perform: eks:DescribeCluster mean the controller's IAM role (IRSA) is missing the necessary permissions to discover the cluster details.
Controller Crash/Restart: If the logs show repeated restarts right after "Starting informers", it often points to a mismatch between the provided clusterName and the actual cluster.

Tip: Enable Debug Logging

If we don't see enough detail, we can increase the log verbosity. Update our Helm deployment with --set logLevel=debug or change the LOG_LEVEL environment variable in the deployment to debug

2. Verify CRD Configurations

Karpenter requires specific Custom Resource Definitions (CRDs) to know how to provision nodes.

(1) List NodePools: Run kubectl get nodepools to ensure our provisioning logic is active.

(2) List EC2NodeClasses: Run kubectl get ec2nodeclasses to confirm AWS-specific settings (like subnets and security groups) are defined.

3. Perform a Scaling Test ("Inflate" Test)

The standard way to test Karpenter is by deploying a "dummy" workload that exceeds current cluster capacity.

(1) Deploy a test app: Apply a deployment (often called inflate) with high CPU/Memory requests.

(2) Scale it up: Run:

% kubectl scale deployment inflate --replicas=5

(3) Watch for new nodes: Monitor:

% kubectl get nodes -w

If configured correctly, Karpenter will detect the pending pods and provision a new EC2 instance within about a minute.

During the inflate scaling test, how to know that a new node was provisioned by karpenter and not cluster autoscaler?

During an inflate scaling test, we can distinguish between nodes provisioned by Karpenter and those from Cluster Autoscaler (CAS) by checking for specific labels, console status, and controller logs.

1. Check for Specific Kubernetes Labels

Karpenter automatically injects unique labels into every node it creates. CAS nodes usually belong to an Auto Scaling Group (ASG) and do not have these specific Karpenter markers.

Run this command to see the labels on our nodes:

kubectl get nodes --show-labels

Look for these Karpenter-exclusive labels:

karpenter.sh/nodepool: The name of the NodePool that provisioned the node.
karpenter.sh/capacity-type: Set to spot or on-demand.
karpenter.k8s.aws/instance-category: (e.g., c, m, r).

Nodes provisioned by Cluster Autoscaler (CAS) don't have a unique "CAS" label. Instead, they carry labels that identify them as members of an Auto Scaling Group (ASG) or an EKS Managed Node Group (MNG).

If we are looking at a node and trying to confirm if it came from CAS, look for these specific markers:

1. Managed Node Group Labels (Most Common)

If we use EKS Managed Node Groups with CAS, the nodes will always have:

eks.amazonaws.com/nodegroup: The name of the MNG.
eks.amazonaws.com/nodegroup-image: The AMI ID used.
eks.amazonaws.com/capacityType: Usually ON_DEMAND or SPOT
eks.amazonaws.com/sourceLaunchTemplateId:
eks.amazonaws.com/sourceLaunchTemplateVersion:

2. Auto Scaling Group Labels

Since CAS works by increasing the "Desired Capacity" of an ASG, the underlying EC2 instance is tagged by AWS. Kubernetes reflects these as:

alpha.eksctl.io/nodegroup-name: (If using eksctl)
node.kubernetes.io/instance-type: (Standard, but CAS uses this to match ASG definitions)

3. The "Missing" Labels

The easiest way to identify a CAS node during a Karpenter test is by what it doesn't have. A CAS node will NEVER have:

❌ karpenter.sh/nodepool
❌ karpenter.sh/provisioner-name (deprecated)
❌ karpenter.k8s.aws/instance-category

Quick Check Command

Run this to see which nodes belong to Karpenter vs. CAS/MNG:

kubectl get nodes -L karpenter.sh/nodepool,://eks.amazonaws.com

If the nodepool column is populated, it's Karpenter.

If the nodegroup column is populated, it's CAS/MNG.

2. Identify via "Self-Managed" Status in EKS Console

In the AWS EKS Console under the Compute tab:

Karpenter Nodes: Appear as "Self-managed" because Karpenter bypasses Auto Scaling Groups to launch instances directly via the EC2 Fleet API.
Cluster Autoscaler Nodes: Appear as part of a "Managed Node Group" or are tied to a specific ASG.

3. Check for the NodeClaim Object

Karpenter creates a NodeClaim for every node it provisions. Cluster Autoscaler does not use this resource. Run

kubectl get nodeclaims

...during the test. If we see new entries appearing that correspond to our inflate pods, Karpenter is doing the work.

4. Monitor Controller Logs

We can watch Karpenter’s real-time decision-making process by tailing its logs. It will explicitly state when it discovers unschedulable pods and which instance type it is launching.

kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f

CAS logs, by contrast, will show it interacting with ASGs and increasing the "desired capacity" of a group

How to disable Cluster Autoscaler temporarily to ensure Karpenter is the only one responding to our tests?

To ensure Karpenter is the only controller responding to our scaling tests, we can temporarily disable the Cluster Autoscaler (CAS) by scaling its deployment to zero replicas.

1. Identify the CAS Deployment

The Cluster Autoscaler typically runs in the kube-system namespace. Verify its name first:

kubectl get deployments -n kube-system | grep cluster-autoscaler

2. Scale to Zero

Run the following command to stop the CAS from running. This will terminate the pod responsible for monitoring the cluster and scaling our Auto Scaling Groups (ASGs):

kubectl scale deployment cluster-autoscaler -n kube-system --replicas=0

3. Verify the Shutdown

Ensure no CAS pods are running to prevent them from interfering with our inflate test:

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler

4. (Optional) Remove ASG Tags

If we want a more permanent "hard" disable without deleting the deployment, we can remove the specific AWS tags from our Auto Scaling Groups that the CAS uses for auto-discovery:

k8s.io/cluster-autoscaler/enabled
k8s.io/cluster-autoscaler/<cluster-name>

Without these tags, the CAS will ignore those node groups even if the deployment is scaled back up.

To Re-enable

Once our tests are complete, we can restore the Cluster Autoscaler by scaling it back to its original replica count:

kubectl scale deployment cluster-autoscaler -n kube-system --replicas=1

To confirm which instance types Karpenter chose during our inflate test, we can watch the controller logs in real-time. Karpenter will log exactly how it batches our pods and which instances it requests from AWS.

1. Tail Karpenter Logs

Run the following command while our inflate pods are in a Pending state:

kubectl logs -f -n kube-system -l app.kubernetes.io/name=karpenter

Note: Some installations use the karpenter namespace instead of kube-system.

2. What to Look For

Karpenter logs its decisions in JSON or text format. Look for INFO messages containing found provisionable pod(s) or created nodeclaim.

A typical log entry looks like this:

2024-03-12T10:00:00.000Z INFO controller.provisioner created nodeclaim {"commit": "...", "nodeclaim": "default-abc12", "nodepool": "general-purpose", "requests": {"cpu":"4","memory":"8Gi"}, "instance-types": "m5.xlarge, m6i.xlarge, c5.2xlarge..."}

requests: Shows the total CPU/Memory requested by our inflate pods.
instance-types: Lists the candidates Karpenter passed to the EC2 Fleet API. Karpenter usually sends a diversified list (up to 60 types) to ensure high availability and best pricing.

3. Identify the Winning Instance

Once the node is live, we can confirm the final selected type by checking the NodeClaim status:

kubectl get nodeclaims -o custom-columns=NAME:.metadata.name,TYPE:.status.instanceType,ZONE:.status.zone

This command explicitly shows the specific instance type (e.g., m5.large) that AWS actually provisioned for that claim.

4. Enable Debug Logging (Optional)

If we don't see enough detail, we can increase the log verbosity to debug. This will reveal exactly how Karpenter "binpacks" our pods onto different instance options.

kubectl patch deployment karpenter -n kube-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/env/0/value", "value": "debug"}]'

Tip: After our test, remember to set the LOG_LEVEL back to info to avoid excessive log storage costs.

4. Monitor NodeClaims

Karpenter uses NodeClaims to manage the lifecycle of the nodes it creates.

Check Status: Run kubectl get nodeclaims to see if requests for new nodes are being fulfilled.
Debug Failures: If a node fails to join, use kubectl describe nodeclaim <name> to check for errors like LaunchFailed or issues with IAM permissions.

5. Verify Tagging

Karpenter discovers infrastructure via tags. Ensure our Subnets and Security Groups have the following tag:

Key: karpenter.sh/discovery

Value: <our-cluster-name>

How to know if node was provisioned by Karpenter?

Karpenter applies labels on nodes it provisions so let's check labels:

% kubectl get nodes --show-labels

If labels like karpenter.sh/nodepool or karpenter.sh/provisioner-name exist, Karpenter launched the node.

When setting up EKS Managed Node Group, we set desired, min and max number of nodes for ASG. Does that enable Cluster Autoscaler automatically? How does ASG play with Karpenter?

Setting the desired, min, and max size on an EKS Managed Node Group only configures the underlying AWS Auto Scaling Group (ASG).

What AWS does: If a node crashes, the ASG will see that the "current" count is less than the "min" (or "desired") and spin up a new node to replace it.
What AWS does NOT do: It will not look at your pending Kubernetes pods and say, "Oh, we need more space, let's increase the count from 3 to 4."

To get that "intelligent" scaling based on pod demand, we must install a separate controller.

Is Cluster Autoscaler (CAS) enabled by default?

No. Kubernetes Cluster Autoscaler is not enabled by default on EKS.

If we want to use it, we must:

Deploy the Cluster Autoscaler as a Pod in our cluster (usually via Helm).
Give that Pod an IAM Role (IRSA) that has permission to update your ASG's desired_capacity.
Add specific tags to our Node Group so the Autoscaler knows which ASG to "manage."

Do we need to disable CAS to use Karpenter?

Yes, absolutely. We should not run Cluster Autoscaler and Karpenter simultaneously on the same nodes.

The Conflict: CAS tries to scale nodes by changing the "desired capacity" of an ASG. Karpenter works differently—it bypasses ASGs entirely and talks directly to the EC2 Fleet API to launch specific instances.
The Result of Running Both: They will fight over the cluster. CAS might try to shrink a group while Karpenter is trying to add capacity, leading to "flapping" nodes and unpredictable costs.

If we switch to Karpenter:

Uninstall/Scale down the Cluster Autoscaler deployment.
Set our Node Group sizes to fixed values (or migrate to "headless" node groups where Karpenter manages the entire lifecycle).
Karpenter is the "New Way": Most AWS users are moving toward Karpenter because it is faster (seconds vs minutes) and more efficient at picking the right instance sizes.

Summary Comparison

Feature ASG (Default) Cluster Autoscaler (CAS) Karpenter

--------- ------------------ ------------------------------- ------------

Logic "Keep X nodes alive" "Add nodes if Pods are Pending" "Provision exactly what Pods need"

Speed Slow (Health-based) Medium (Polling ASG) Fast (Direct EC2 API)

Setup Built-in to EKS Manual Install + IAM Manual Install + IAM

Best for Fixed capacity Traditional scaling Cost-optimization & high speed

Updating Kubernetes version on nodes managed by Karpenter

References:

Autoscaling - Amazon EKS

Karpenter for Kubernetes | Karpenter vs Cluster Autoscaler - YouTube

Cluster-Autoscaler - EKS Best Practices Guides

Karpenter Vs Cluster Autoscaler: The Essential Guide

Kubernetes Cluster Autoscaler

2026-02-06T12:16:00.018+00:00

Kubernetes Cluster Autoscaler (CAS):

Designed to automatically adjust the number of nodes (EC2 instances) in our cluster based on the resource requests of the workloads running in the cluster
Kubernetes project, supported on EKS: https://github.com/kubernetes/autoscaler

Key Features:

Node Scaling: It adds or removes nodes based on the pending pods that cannot be scheduled due to insufficient resources.
Pod Scheduling: Ensures that all pending pods are scheduled by scaling the cluster up.

It works with EKS Managed Node Groups backed by AWS Auto Scaling Groups. In node group, if we provide specific settings (like custom block_device_mappings), EKS creates an EC2 Launch Template under the hood.

Cluster Autoscaler and kube-scheduler

kube-scheduler is the default control plane component in Kubernetes responsible for deciding which Node a newly created or unscheduled Pod should run on. It essentially matches pods to the most suitable available machines based on resource requirements and specific constraints.

Cluster Autoscaler and kube-scheduler components DO NOT directly communicate with the other. Instead, the Cluster Autoscaler (CA) watches the kube-scheduler by monitoring the state of pods in the cluster.

They work in an indirect loop via the Kubernetes API server:

Kube-scheduler: Attempts to place pods on existing nodes. If it cannot find a node with sufficient capacity, it marks the pod as Pending with an Unschedulable status.
Cluster Autoscaler: Monitors the cluster for these Unschedulable pods.
Action: When CA detects a pending pod, it triggers a scale-up by adding a node.
Completion: Once the new node joins, the kube-scheduler notices the new capacity and schedules the pending pod.

Key Takeaways:

The Autoscaler watches the Scheduler: The autoscaler reacts to the decisions (or failed attempts) of the scheduler.
No Direct Connection: They are "blissfully unaware" of each other and interact only through Kubernetes API objects.
Not Resource Based: The Cluster Autoscaler does not directly monitor node CPU/memory usage; it only cares if the scheduler cannot place a pod.

This indirect workflow ensures that new nodes are only provisioned when necessary to satisfy pod scheduling constraints.

How to check if it's installed and enabled?

(1) Look for its deployment

Cluster Autoscaler usually runs as a Deployment in kube-system namespace so we can look for that deployment:

% kubectl get deployments -n kube-system | grep -i cluster-autoscaler

cluster-autoscaler-aws-cluster-autoscaler 2/2 2 2 296d

We can also list pods directly:

% kubectl get pods -n kube-system | grep -i cluster-autoscaler

cluster-autoscaler-aws-cluster-autoscaler-7cbb844455-q2lxv 1/1 Running 0 206d

cluster-autoscaler-aws-cluster-autoscaler-7cbb844455-vhbsw 1/1 Running 0 206d

If we see a pod running, it’s installed.

Typical names:

cluster-autoscaler
cluster-autoscaler-aws-clustername
cluster-autoscaler-eks-...

(2) Inspect the Deployment

Confirm it’s enabled & configured.

% kubectl describe deployment cluster-autoscaler -n kube-system

Name: cluster-autoscaler-aws-cluster-autoscaler

Namespace: kube-system

CreationTimestamp: Wed, 16 Apr 2025 12:25:38 +0100

Labels: app.kubernetes.io/instance=cluster-autoscaler

app.kubernetes.io/managed-by=Helm

app.kubernetes.io/name=aws-cluster-autoscaler

helm.sh/chart=cluster-autoscaler-9.46.6

Annotations: deployment.kubernetes.io/revision: 1

meta.helm.sh/release-name: cluster-autoscaler

meta.helm.sh/release-namespace: kube-system

Selector: app.kubernetes.io/instance=cluster-autoscaler,app.kubernetes.io/name=aws-cluster-autoscaler

Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable

StrategyType: RollingUpdate

MinReadySeconds: 0

RollingUpdateStrategy: 25% max unavailable, 25% max surge

Pod Template:

Labels: app.kubernetes.io/instance=cluster-autoscaler

app.kubernetes.io/name=aws-cluster-autoscaler

Service Account: cluster-autoscaler-aws-cluster-autoscaler

Containers:

aws-cluster-autoscaler:

Image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.32.0

Port: 8085/TCP

Host Port: 0/TCP

Command:

./cluster-autoscaler

--cloud-provider=aws

--namespace=kube-system

--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/mycorp-prod-mycluster

--logtostderr=true

--stderrthreshold=info

--v=4

Liveness: http-get http://:8085/health-check delay=0s timeout=1s period=10s #success=1 #failure=3

Environment:

POD_NAMESPACE: (v1:metadata.namespace)

SERVICE_ACCOUNT: (v1:spec.serviceAccountName)

AWS_REGION: us-east-1

Mounts: <none>

Volumes: <none>

Priority Class Name: system-cluster-critical

Node-Selectors: <none>

Tolerations: <none>

Conditions:

Type Status Reason

---- ------ ------

Progressing True NewReplicaSetAvailable

Available True MinimumReplicasAvailable

OldReplicaSets: <none>

NewReplicaSet: cluster-autoscaler-aws-cluster-autoscaler-7cbb844455 (2/2 replicas created)

Events: <none>

Key things to look for:

Replicas ≥ 1
No crash loops
Command args like:

--cloud-provider=aws
--nodes=1:10:nodegroup-name
--balance-similar-node-groups

If replicas are 0, it’s installed but effectively disabled.

(3) Check logs

Is it actively scaling?

This confirms it’s working, not just running.

kubectl logs deployment/cluster-autoscaler -n kube-system

or find pods:

kubectl get pods \

-l app.kubernetes.io/name=cluster-autoscaler \

-n kube-system

Then check logs:

kubectl logs \

-l app.kubernetes.io/name=cluster-autoscaler \

-n kube-system \

| grep "Standard-Autoscaler"

Healthy / active signs:

scale up
scale down
Unschedulable pods
Node group ... increase size
If you see messages like Refresher: resolving ASGs, it will list the names of the ASGs it is currently monitoring.

Red flags:

AccessDenied
no node groups found
failed to get ASG

(4) Check for unschedulable pods trigger

If CA is working, it reacts to pods stuck in Pending.

% kubectl get pods -A | grep Pending

If pods are pending and CA logs mention them → CA is enabled and reacting.

(5) AWS EKS-specific checks (very common)

a) Check IAM permissions (classic failure mode)

Cluster Autoscaler must run with an IAM role that can talk to ASGs.

% kubectl -n kube-system get sa | grep autoscaler

cluster-autoscaler-aws-cluster-autoscaler 0 296d

horizontal-pod-autoscaler 0 296d

Let's inspect cluster-autoscaler-aws-cluster-autoscaler service accont:

% kubectl -n kube-system get sa cluster-autoscaler-aws-cluster-autoscaler -o yaml

apiVersion: v1

automountServiceAccountToken: true

kind: ServiceAccount

metadata:

annotations:

eks.amazonaws.com/role-arn: arn:aws:iam::xxxxx:role/mycorp-prod-mycluster-cluster-autoscaler

meta.helm.sh/release-name: cluster-autoscaler

meta.helm.sh/release-namespace: kube-system

creationTimestamp: "2026-04-16T11:25:37Z"

labels:

app.kubernetes.io/instance: cluster-autoscaler

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/name: aws-cluster-autoscaler

helm.sh/chart: cluster-autoscaler-9.46.6

namespace: kube-system

resourceVersion: "15768"

uid: 0a7da521-1bf5-5a5f-a155-8801e876ea7b

Look for:

eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ClusterAutoscalerRole

If missing → CA may exist but cannot scale.

b) Check Auto Scaling Group tags

Our node group ASGs must be tagged:

k8s.io/cluster-autoscaler/enabled = true

k8s.io/cluster-autoscaler/<cluster-name> = owned

Without these → CA runs but does nothing. If those tags are missing, Cluster Autoscaler will ignore that ASG entirely.

(6) Check Helm (if installed via Helm)

Let's list all Helm releases across every namespace in a Kubernetes cluster and look for cluster autoscaler:

% helm list -A

NAME NAMESPACE REVISION UPDATED

cluster-autoscaler kube-system 1 2025-04-16 12:25:30.389073326 +0100BST

STATUS CHART APP VERSION

deployed cluster-autoscaler-9.46.6 1.32.0

Then:

helm status cluster-autoscaler -n kube-system

The command helm list -A (or its alias helm ls -A) is used to list all Helm releases across every namespace in a Kubernetes cluster. Helm identifies your cluster and authenticates through the same mechanism as kubectl: the kubeconfig file. It uses the standard Kubernetes configuration file, typically located at ~/.kube/config, to determine which cluster to target.

(7) Double-check it’s not replaced by Karpenter

Many newer EKS clusters don’t use Cluster Autoscaler anymore.

% kubectl get pods -A | grep -i karpenter

kube-system karpenter-6f67b8c97b-lbq8p 1/1 Running 0 206d

kube-system karpenter-6f67b8c97b-wmprj 1/1 Running 0 206d

If Karpenter is installed, Cluster Autoscaler usually isn’t (or shouldn’t be).

Quick decision table

-----------------------------------------------------------------

Symptom Meaning

-----------------------------------------------------------------

No CA pod Not installed

Pod running, replicas=0 Installed but disabled

Logs show AccessDenied Broken IAM

Pods Pending, no scale-up ASG tags / config issue

Karpenter present CA likely not used

-----------------------------------------------------------------

(8) Check the "Status" ConfigMap

Cluster Autoscaler maintains a ConfigMap that shows which groups it is managing and if they are at their max/min size:

% kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

apiVersion: v1

data:

status: |

time: 2026-03-21 08:52:07.308206626 +0000 UTC

autoscalerStatus: Running

clusterWide:

health:

status: Healthy

nodeCounts:

registered:

total: 6

ready: 6

notStarted: 0

longUnregistered: 0

unregistered: 0

lastProbeTime: "2026-03-21T08:52:07.308206626Z"

lastTransitionTime: "2026-03-20T16:30:07.460032826Z"

scaleUp:

status: NoActivity

lastProbeTime: "2026-03-21T08:52:07.308206626Z"

lastTransitionTime: "2026-03-20T16:30:07.460032826Z"

scaleDown:

status: NoCandidates

lastProbeTime: "2026-03-21T08:52:07.308206626Z"

lastTransitionTime: "2026-03-20T16:30:07.460032826Z"

nodeGroups:

- name: eks-mycorp-env-app-k8s-v1_33-202603...04-2cc...dc8

health:

status: Healthy

nodeCounts:

registered:

total: 2

ready: 2

notStarted: 0

longUnregistered: 0

unregistered: 0

cloudProviderTarget: 2

minSize: 2

maxSize: 10

lastProbeTime: "2026-03-21T08:52:07.308206626Z"

lastTransitionTime: "2026-03-20T16:30:07.460032826Z"

scaleUp:

status: NoActivity

lastProbeTime: "2026-03-21T08:52:07.308206626Z"

lastTransitionTime: "2026-03-20T16:30:07.460032826Z"

scaleDown:

status: NoCandidates

lastProbeTime: "2026-03-21T08:52:07.308206626Z"

lastTransitionTime: "2026-03-20T16:30:07.460032826Z"

kind: ConfigMap

metadata:

annotations:

cluster-autoscaler.kubernetes.io/last-updated: 2026-03-21 08:52:07.308206626 +0000

UTC

creationTimestamp: "2026-03-20T16:29:56Z"

namespace: kube-system

resourceVersion: "18...78"

uid: 17b...0af

Installation and Setup:

To use the Cluster Autoscaler in the EKS cluster we need to deploy it using a Helm chart or a pre-configured YAML manifest.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

In Terraform:

resource "helm_release" "cluster_autoscaler" {

name = "cluster-autoscaler"

repository = "https://kubernetes.github.io/autoscaler"

chart = "cluster-autoscaler"

version = "9.46.6"

namespace = "kube-system"

set {

name = "autoDiscovery.clusterName"

value = local.cluster_name

}

set {

name = "awsRegion"

value = local.aws_region

}

set {

name = "rbac.serviceAccount.create"

value = "false"

}

set {

name = "rbac.serviceAccount.name"

value = local.service_account_name

}

Configuration:

Ensure the --nodes flag in the deployment specifies the min and max nodes for your node group.
Annotate your node groups with the k8s.io/cluster-autoscaler tags to enable autoscaler to manage them.

How to know if node was provisioned by Cluster Autoscaler?

Cluster Autoscaler applies labels on nodes it provisions so let's check labels:

% kubectl get nodes --show-labels

If label like eks.amazonaws.com/nodegroup exists, node was launched by and belongs to EKS Managed Node Group as Cluster Autoscaler launched the node.

Example:

% kubectl get nodes --show-labels

NAME STATUS ROLES AGE VERSION

ip-10-2-1-244.us-east-1.compute.internal Ready <none> 206d v1.32.3-eks-473151a

LABELS

Environment=prod,

beta.kubernetes.io/arch=amd64,

beta.kubernetes.io/instance-type=m5.xlarge,

beta.kubernetes.io/os=linux,

eks.amazonaws.com/capacityType=ON_DEMAND,

eks.amazonaws.com/nodegroup-image=ami-07fa6c030f5802c74,

eks.amazonaws.com/nodegroup=mycorp-prod-mycluster-20260714151819635800000002,

eks.amazonaws.com/sourceLaunchTemplateId=lt-0edc7a2b08ea82a28,

eks.amazonaws.com/sourceLaunchTemplateVersion=1,

failure-domain.beta.kubernetes.io/region=us-east-1,

failure-domain.beta.kubernetes.io/zone=us-east-1a,

mycorp;/node-type=default,

k8s.io/cloud-provider-aws=12b0e11196b7091c737cf66015f19720,

kubernetes.io/arch=amd64,

kubernetes.io/hostname=ip-10-2-1-244.us-east-1.compute.internal,

kubernetes.io/os=linux,

node.kubernetes.io/instance-type=m5.xlarge,

topology.ebs.csi.aws.com/zone=us-east-1a,

topology.k8s.aws/zone-id=use1-az1,

topology.kubernetes.io/region=us-east-1,

topology.kubernetes.io/zone=us-east-1a

If we list all nodegroups in the cluster, the one above is listed:

% aws eks list-nodegroups \

--cluster-name mycorp-env-app-k8s \

   --profile my_profile
{
   "nodegroups": [
       "mycorp-env-app-k8s-20260714151819635800000002"
   ]
}

To inspect the nodegroup, including its labels, use:

% aws eks describe-nodegroup \

--cluster-name mycorp-env-app-k8s \

--nodegroup-name mycorp-env-app-k8s-v1_33-20260...03 \

--region us-east-2 \

--profile my_profile \

--output json

{

"nodegroup": {

"nodegroupName": "mycorp-env-app-k8s-v1_33-20260...003",

"nodegroupArn": "arn:aws:eks:us-east-2:xxxx:nodegroup/mycorp-env-app-k8s/mycorp-env-app-k8s-v1_33-202....03/2cce8....c7",

"clusterName": "mycorp-env-app-k8s",

"version": "1.33",

"releaseVersion": "1.33.8-20260317",

"createdAt": "2026-03-20T12:41:07.961000+00:00",

"modifiedAt": "2026-03-22T09:21:58.892000+00:00",

"status": "ACTIVE",

"capacityType": "ON_DEMAND",

"scalingConfig": {

"minSize": 2,

"maxSize": 10,

"desiredSize": 2

"instanceTypes": [

"m5.large"

"subnets": [

"subnet-02xxx",

"subnet-00xxx",

"subnet-04xxx"

"amiType": "AL2023_x86_64_STANDARD",

"nodeRole": "arn:aws:iam::xxx:role/mycorp-env-app-k8s-v1_33-eks-node-group",

"labels": {

"Environment": "prod",

"mycorp/node-type": "v1.33"

"resources": {

"autoScalingGroups": [

{

"name": "mycorp-env-app-k8s-v1_33-202603...dc7"

}

]

"health": {

"issues": []

"updateConfig": {

"maxUnavailablePercentage": 33

"launchTemplate": {

"name": "mycorp-env-app-k8s-v1_33-202...0001",

"version": "1",

"id": "lt-xxx"

"tags": {

"ClusterName": "mycorp-env-app-k8s",

"Environment": "prod",

"terraform-aws-modules": "eks",

"Terraform": "true",

"Name": "mycorp-env-app-k8s-v1_33"

}

If we are using terraform-aws-modules/eks/aws to provision EKS cluster and within it we define EKS-managed node groups, this module will create AWS Autoscaling Group (ASG) for each of them. Their names can be read from module's output variable eks_managed_node_groups_autoscaling_group_names.

If we inspect the labels on one such ASG, we can see that this Terraform module attached labels on it, so CAS can discover it and manage its parameters (usually just desired_size which is used for increasing or decreasing the number of current EC2 instances):

% aws autoscaling describe-auto-scaling-groups \

--auto-scaling-group-names eks-mycorp-env-app-k8s-v1_33-20260320...4-2c...8 \

--query "AutoScalingGroups[].Tags[?starts_with(Key, 'k8s.io/cluster-autoscaler')]" \

--region us-east-2 \

--profile my_profile

[

{

"ResourceId": "eks-mycorp-env-app-k8s-v1_33-20260320...4-2c...8",

"ResourceType": "auto-scaling-group",

"Key": "k8s.io/cluster-autoscaler/enabled",

"Value": "true",

"PropagateAtLaunch": true

{

"ResourceId": "eks-mycorp-env-app-k8s-v1_33-20260320...4-2c...8",

"ResourceType": "auto-scaling-group",

"Key": "k8s.io/cluster-autoscaler/mycorp-env-app-k8s",

"Value": "owned",

"PropagateAtLaunch": true

}

]

In the example above, mycorp-env-app-k8s is the name of the cluster.

If cluster is overprovisioned, why Cluster Autoscaler doesn't scale nodes down automatically?

If Cluster Autoscaler is running but not shrinking the cluster, it's usually because:

System Pods: Pods like kube-dns or metrics-server don't have PDBs (Pod Disruption Budgets) and CA is afraid to move them.
Local Storage: A pod is using emptyDir or local storage.
Annotation: A pod has the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation.
Manual Overrides: Check if someone manually updated the Auto Scaling Group (ASG) or the EKS Managed Node Group settings in the AWS Console. Terraform won't automatically "downgrade" those nodes until the next terraform apply or a node recycle.
If nodes are very old, they are "frozen" in time. Even if you changed your Terraform to smaller EC2 instances recently, EKS Managed Node Groups do not automatically replace existing nodes just because the configuration changed. They wait for a triggered update or a manual recycling of the nodes.

How to fix this overprovisioning?

Since your current Terraform state says you want e.g. 2 nodes of m5.large, but the reality is e.g. 4 nodes of m5.xlarge, you need to force a sync.

Step 1: Check for Drift

Run a terraform plan. It will likely show that it wants to update the Launch Template or the Node Group version to switch from xlarge back to large.

Step 2: Trigger a Rolling Update

If you apply the Terraform and nothing happens to the existing nodes, you need to tell EKS to recycle them. You can do this via the AWS CLI:

aws eks update-nodegroup-version \

--cluster-name <your-cluster-name> \

--nodegroup-name <your-nodegroup-name> \

--force

Note: This will gracefully terminate nodes one by one and replace them with the new m5.large type defined in your TF.

Cluster Autoscaler VS Karpenter

CAS (Cluster Autoscaler) and Karpenter are Kubernetes tools for adjusting node capacity based on workload, with CAS relying on fixed node groups and slow, infrastructure-driven scaling. Karpenter is a faster, modern, open-source, workload-driven node provisioner that directly interacts with cloud APIs, improving efficiency and cost-optimization.

Cluster Autoscaler (CAS): Operates by adjusting the size of specific, pre-defined node groups (e.g., autoscaling groups). It is generally better suited for smaller, predictable, or steady-state workloads where strict node group management is preferred.

Karpenter: Evaluates pending pods and launches optimally sized nodes directly, bypassing the need for manual node group management. It is ideal for high-churn, highly dynamic, and cost-sensitive, large-scale production environments.

While both tools scale Kubernetes nodes to meet pod demand, they use fundamentally different approaches. Cluster Autoscaler (CA) is the traditional, "group-based" tool that adds nodes to existing pools, whereas Karpenter is a "provisioning" tool that directly creates the specific instances your applications need.

Quick Feature Comparison Table

Scaling Logic

Cluster Autoscaler (CA): Scales pre-defined node groups (ASGs)
Karpenter: Directly provisions individual EC2 instances.

Speed

Cluster Autoscaler (CA): Slower; waits for cloud provider group updates
Karpenter: Faster; provisions nodes in seconds via direct APIs; better for rapid, "spiky" traffic.

Cost Control

Cluster Autoscaler (CA): Limited; uses fixed node sizes in groups.
Karpenter: High; picks the cheapest/optimal instance for the pod. It has built-in node consolidation, which intelligently reduces costs by binpacking, or packing, pods onto fewer, more efficient nodes.

Complexity

Cluster Autoscaler (CA): Higher; must manage multiple node groups.
Karpenter: Lower; one provisioner can handle many pod types.

Flexibility

Karpenter: supports diverse instance types and, while commonly used with AWS, it can be used with other providers.

Configuration

Karpenter uses Kubernetes-native YAML for defining node pools and node classes.

Key Differences

Infrastructure Model:

CA asks, "How many more of these pre-configured nodes do I need?".
Karpenter asks, "What specific resources (CPU, RAM, GPU) does this pending pod need right now?" and builds a node to match.

Node Groups:

CA requires you to manually define and maintain Auto Scaling Groups (ASGs) for different instance types or zones.
Karpenter bypasses ASGs entirely, allowing it to "mix and match" instance types dynamically in a single cluster.

Consolidation:

Karpenter actively monitors the cluster to see if it can move pods to fewer or cheaper nodes to save money (bin-packing).
While CA has a "scale-down" feature, it is less aggressive at optimizing for cost.

Spot Instance Management:

Karpenter handles Spot interruptions and price changes more natively, selecting the most stable and cost-efficient Spot instances in real-time.

Which should you choose?

Use Cluster Autoscaler if you need a stable, battle-tested solution that works across multiple cloud providers (GCP, Azure) or if your workloads are very predictable and don't require rapid scaling.

Use Karpenter if you are on AWS EKS, need to scale up hundreds of nodes quickly, want to heavily use Spot instances, or want to reduce the operational burden of managing dozens of node groups.

Disable Cluster Autoscaler if you plan to use Karpenter. Having both leads to race conditions and wasted cost.

When to Run Both Together

It's generally not recommended to run Cluster Autoscaler and Karpenter together in the same cluster. However, there are specific scenarios where it might be acceptable:

Valid use cases for running both:

Migration period: Transitioning from Cluster Autoscaler to Karpenter, where you temporarily run both while gradually moving workloads
Hybrid node management: Managing distinct, non-overlapping node groups where Cluster Autoscaler handles some node groups and Karpenter handles others (though this adds complexity)

When It's Not Recommended (and Why)

Primary reasons to avoid running both:

Conflicting decisions: Both tools make independent scaling decisions, which can lead to:

Race conditions where both try to provision nodes simultaneously
Inefficient resource allocation
Unpredictable scaling behavior
One tool removing nodes the other just provisioned

Increased operational complexity:

Two systems to monitor, troubleshoot, and maintain
Doubled configuration overhead
More difficult to understand which tool made which scaling decision

Resource contention: Both tools consume cluster resources and API server capacity, adding unnecessary load.

No significant benefits: Karpenter can handle everything Cluster Autoscaler does, often more efficiently, so there's rarely a technical need for both.

EKS-Specific Considerations

The same principles apply to AWS EKS clusters, with some additional context:

EKS particularities:

Karpenter was designed specifically for AWS/EKS and integrates deeply with EC2 APIs
Karpenter typically provides better performance on EKS (faster provisioning, better bin-packing)
If you're on EKS, the general recommendation is to choose Karpenter over Cluster Autoscaler for new deployments

Migration best practice for EKS: If migrating from Cluster Autoscaler to Karpenter on EKS, ensure they manage completely separate node groups, and complete the migration as quickly as feasible to minimize the period of running both.

How to migrate pods from nodes deployed by Cluster Autoscaler to those deployed by Karpenter?

If you'd rather use Karpenter for everything, you should eventually set your min_size, max_size, and desired_size to 0 in this node group and let Karpenter handle the provisioning instead.

---

Kubernetes Metrics Server

2026-02-06T12:04:00.004+00:00

Kubernetes Metrics Server is a foundational component required by several other critical cluster modules and tools:

1. Horizontal Pod Autoscaler (HPA)

2. Vertical Pod Autoscaler (VPA)

Purpose: While HPA adds more pods, the Vertical Pod Autoscaler (VPA) adjusts the CPU and memory requests/limits of existing pods.
Dependency: VPA relies on Metrics Server for the real-time resource data it uses to recommend or apply these resource changes.

2. Native CLI Observability (kubectl top)

Purpose: Commands used for ad-hoc debugging and performance monitoring.
Dependency: Both kubectl top pods and kubectl top nodes query the Metrics API directly. Without the server, these commands will return an error.

3. Kubernetes Dashboard

Purpose: A web-based UI for managing and troubleshooting clusters.
Dependency: The Kubernetes Dashboard uses Metrics Server to display resource usage graphs and live statistics for nodes and pods.

4. Third-Party Monitoring Tools & Adapters

Custom Metrics Adapters: Some adapters that bridge external sources (like CloudWatch or Datadog) to Kubernetes may use the standard Metrics API for fallback or basic resource data.
Resource Management Tools: Operational tools such as Goldilocks, which suggests "just right" resource requests, often depend on the baseline metrics provided by this server.

Key Distinction

While the Metrics Server is essential for these control loops (HPA, VPA), it is not a replacement for a full observability stack like Prometheus. It only stores a short-term, in-memory snapshot and does not provide historical data

How to to install the Metrics Server as an EKS Community Add-on to enable these features?

In March 2025, AWS introduced a new catalog of community add-ons that includes the Metrics Server. This allows you to manage it directly through EKS-native tools like any other AWS-managed add-on (e.g., VPC CNI or CoreDNS).

Method 1: Using the AWS Management Console

The easiest way to install it is through the EKS console:

Navigate to your EKS cluster in the AWS Console.
Select the Add-ons tab and click Get more add-ons.
Scroll down to the Community add-ons section.
Find Metrics Server, select it, and click Next.
Choose the desired version (usually the latest recommended) and click Create.

Method 2: Using the AWS CLI

You can also install the community add-on via the command line:

aws eks create-addon \

--cluster-name <YOUR_CLUSTER_NAME> \

--addon-name metrics-server

Verification

Once the installation status moves to Active, verify that the pods are running in the kube-system namespace:

kubectl get deployment metrics-server -n kube-system

Finally, test that the Metrics API is responding:

kubectl top nodes

Note: If you are using AWS Fargate, you may need to update the containerPort from 10250 to 10251 in the deployment configuration to ensure compatibility with Fargate's networking constraints.

Metrics Server Configuration

To configure custom resource limits for the Metrics Server EKS community add-on, you can use Configuration Values during installation or update. This is essential for high-pod-count clusters where the default allocation may lead to OOMKilled errors.

1. Scaling Recommendations

The Metrics Server's resource consumption scales linearly with your cluster's size. Baseline recommendations include:

CPU: Approximately 1 millicore per node in the cluster.
Memory: Approximately 2 MB of memory per node.
Large Clusters: If your cluster exceeds 100 nodes, it is recommended to double these defaults and monitor performance.

2. How to Apply Custom Limits

You can provide a JSON or YAML configuration block via the AWS EKS Add-ons API.

Via AWS CLI

Use the configuration-values flag to pass your resource overrides:

aws eks create-addon \

--cluster-name <YOUR_CLUSTER_NAME> \

--addon-name metrics-server \

--configuration-values '{

"resources": {

"requests": { "cpu": "100m", "memory": "200Mi" },

"limits": { "cpu": "200m", "memory": "500Mi" }

}

Via AWS Console

Go to the Add-ons tab in your EKS cluster.
Click Edit on the metrics-server add-on.
Expand the Optional configuration settings.
Paste the JSON configuration into the Configuration values text box.

3. Critical Configuration for High Traffic

In addition to resource limits, you may want to adjust the scraping frequency to make HPA more responsive.

Metric Resolution: The default is 60s. For faster scaling, add --metric-resolution=15s to the container arguments via the same configuration block.
High Availability: The community add-on defaults to 2 replicas to prevent downtime during scaling events.

Horizontal Pod Autoscaler (HPA)

2026-02-05T12:26:00.021+00:00

The Horizontal Pod Autoscaler (HPA) serves to automatically align your application's capacity with its real-time demand by adjusting the number of pod replicas. Its operation depends on several critical components and configurations within an EKS or Kubernetes cluster.

Kubernetes Horizontal Pod Autoscaler (HPA) is a built-in Kubernetes controller

Standard Kubernetes autoscaling mechanism
HPA API is available out of the box. It is a part of the Core Kubernetes and does not need installing third-party controllers or addons.

But it is NOT fully operational "by default" in a standard Amazon EKS cluster. API definitions for HPA resources exist within Kubernetes, but they require a Metrics Server to function—which AWS does not install for you automatically during cluster creation.

It's "standard" in the sense that the feature is built into the Kubernetes control plane, but it isn't "automatic" in the sense that it guesses which of your apps need scaling.

Think of it like a Thermostat: The thermostat (HPA Controller) is already installed on the wall (EKS Control Plane), but it won't turn on the AC until you tell it what the Target Temperature (CPU/Memory threshold) is and which room (Deployment) to monitor.

Here is why a manifest is required for every app:

1. The Controller vs. The Resource

The Controller (The "How"): This is a loop running inside the EKS Control Plane. It is always active, waiting for instructions. Kubernetes HPA Documentation explains this loop.

The Resource (The "What"): The HPA Manifest is that instruction. It tells the controller: "Watch Deployment X, keep CPU at 50%, and don't go above 10 pods."

2. Manual Intent

Kubernetes follows a Declarative Model. It never assumes you want to scale. If it scaled every pod automatically, a single bug in your code (like an infinite loop) could scale your cluster to 1,000 nodes and drain your AWS budget instantly. You must explicitly opt-in by creating the HPA resource.

3. Unique Criteria for Every App

Not all apps scale the same way:

Web API: Might scale when CPU hits 70%.
Background Worker: Might scale based on Memory usage.
Data Processor: Might scale based on a Custom Metric like SQS queue depth.

Summary: What is "Standard"?

What is standard is the API definition and the Controller. What is not standard is your specific application's scaling logic.

To see what the HPA Controller is looking for, you can check your Deployment's resource requests via kubectl:

kubectl get deployment <name> -o yaml | grep resources -A 5

Key Features:

Pod Scaling: Adjusts the number of pod replicas to match the demand.
Automatically scales up/down the number of pods in a deployment, replication controller, or replica set based on observed CPU utilization, memory or other selected custom/external metrics.

Purpose

Dynamic Scalability: Automatically adds pods during traffic surges to maintain performance and removes them during low-traffic periods to reduce waste.
Cost Optimisation: Ensures you only pay for the compute resources currently needed rather than over-provisioning for peak loads.
Resilience & Availability: Prevents application crashes and outages by proactively scaling out before resources are fully exhausted.
Operational Efficiency: Replaces manual intervention with "architectural definition," allowing infrastructure to manage itself based on predefined performance rules.

Dependencies

HPA cannot function on its own; it requires the following "links" and infrastructure:

Metrics Server (The Aggregator): This is the most critical infrastructure dependency. The HPA controller queries the Metrics API (typically provided by the Metrics Server) to get real-time CPU and memory usage data.

Both HPA and VPA rely on the metrics.k8s.io API to retrieve CPU and memory data. Because EKS is a managed control plane, AWS keeps it "lean" by leaving the choice of metrics provider to you.
HPA: Without Metrics Server, HPA will show a status of <unknown> for its targets.
VPA: Without Metrics Server, the VPA Recommender cannot analyze resource usage to suggest changes

Resource Requests (The Baseline): For the HPA to calculate percentage-based utilization (e.g., "scale at 50% CPU"), the target Deployment must have resources.requests defined. Without these, the HPA has no 100% baseline to measure against and will show an unknown status.
Controller Manager: The HPA logic runs as a control loop within the Kubernetes kube-controller-manager, which periodically (every 15 seconds by default) evaluates the metrics and updates the desired replica count.
Scalable Target: The HPA must be linked to a resource that supports scaling, such as a Deployment, ReplicaSet, or StatefulSet.
Cluster Capacity (Node Scaling): While HPA scales pods, it depends on an underlying node scaler (like Karpenter or Cluster Autoscaler) to provide new EC2 instances if the cluster runs out of physical space for the additional pods.

Installation and Configuration

To use HPA ensure the Metrics Server is installed in your cluster to provide resource metrics.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Once the Metrics Server is installed, HPA is ready to go. It is a core Kubernetes controller, so you don't need to install any additional software beyond the metrics provider. We can simply create an HPA resource for our deployment:

kubectl autoscale deployment our-deployment \

--cpu=50% \

--min=1 \

--max=10

We can also use the manifest:

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

minReplicas: 2

maxReplicas: 10

metrics:

- type: Resource

resource:

target:

type: Utilization

averageUtilization: 50

How to check if HPA is enabled in the cluster?

% kubectl api-resources -o wide | grep autoscaling

NAME SHORTNAMES APIVERSION NAMESPACED KIND

VERBS CATEGORIES

...

horizontalpodautoscalers hpa autoscaling/v2 true HorizontalPodAutoscaler create,delete,deletecollection,get,list,patch,update,watch all

In which namespace do HorizontalPodAutoscalers reside in?

In AWS EKS, HorizontalPodAutoscalers (HPA) are namespaced resources, meaning they belong in the same namespace as the workload (e.g., Deployment or StatefulSet) they are intended to scale.

While there is no single "HPA namespace," here is how they are distributed and where related components live:

1. The HPA Resource Namespace

Application Namespace: When you create an HPA, you define it within the specific namespace where your application is running (e.g., default, production, or demo).
Constraint: An HPA can only scale a target resource (like a Deployment) that exists in that same namespace.

2. Infrastructure & Metrics Namespaces

While the HPA resource lives with your app, the supporting infrastructure often resides in system namespaces:

Metrics Server: This is a mandatory prerequisite for HPA on EKS. It is typically deployed in the kube-system namespace.
Custom Metrics Adapters: If you are scaling based on custom metrics (like Prometheus or CloudWatch), components like the prometheus-adapter or k8s-cloudwatch-adapter may be installed in kube-system or a dedicated namespace like custom-metrics.
Cluster Autoscaler: Often confused with HPA, the Cluster Autoscaler (which scales EC2 nodes rather than pods) also typically resides in the kube-system namespace.

To find all HPAs across your entire EKS cluster, you can run:

kubectl get hpa -A

We might have an output like this:

% kubectl get horizontalpodautoscalers -A

No resources found

It is possible to get "No resources found" for several reasons, despite the resource being namespaced. This usually means that while the API type exists, no actual instances of that resource have been created in your EKS cluster yet.

Why you see "No resources found":

HPA is not yet created: By default, EKS clusters do not come with any HorizontalPodAutoscalers pre-configured. You must explicitly create one for your application.
Metrics Server Missing: HPAs rely on the Kubernetes Metrics Server to function. While the HPA object can be created without it, it will show a status of <unknown> and may not appear if you are looking for active scaling.
Namespace Context: Even with -A (all namespaces), if no user or system service has defined an HPA resource, the list will be empty.

How to Verify and Fix:

Check if Metrics Server is running:

Run kubectl get deployment metrics-server -n kube-system. If it’s missing, you can install it via the AWS EKS Add-ons in the console or via kubectl apply.

kubectl get all -A | grep metrics-server

Check API availability:

Run kubectl api-resources | grep hpa to confirm the cluster recognizes the resource type.

Create a test HPA:

If you have a deployment named my-app, try creating one:

kubectl autoscale deployment my-app \

--cpu=50% \

--min=1 \

--max=10

Note: If you are using a newer version of EKS (like 1.31) with Auto Mode, some autoscaling is handled automatically by the control plane, but standard HPAs still need to be manually defined if you want pod-level scaling based on custom metrics.

% kubectl get all -A | grep metrics-server

default pod/metrics-server-5db5f64c66-sjd2p 1/1 Running 0 205d

default service/metrics-server ClusterIP 172.21.76.224 <none> 443/TCP 95d

default deployment.apps/metrics-server 1/1 1 1 295d

default replicaset.apps/metrics-server-5db5f64c66 1 1 1 295d

This behavior occurs because no instances of HorizontalPodAutoscaler (HPA) have been created yet, even though the supporting infrastructure (Metrics Server) and API are active.

In Kubernetes, the presence of the metrics-server and the autoscaling/v2 API resource does not mean an HPA is automatically running for your apps. You must manually define an HPA for each deployment you want to scale.

Why kubectl get hpa -A is empty

Workloads are not yet auto-scaled: By default, EKS (and Kubernetes) does not apply HPAs to your deployments. You must explicitly create an HPA object that references your target Deployment or StatefulSet.
kubectl get all exclusion: Standard kubectl get all does not include HPAs in its output, which is why your previous command didn't show them even if they existed.
Namespace Location: While your metrics-server is in the default namespace (though typically it's in kube-system), HPAs must be created in the same namespace as the app they are scaling.

How to create your first HPA

If you have a deployment (e.g., named my-deployment) in the default namespace, you can create an HPA for it using this command:

kubectl autoscale deployment my-deployment \

--cpu=60%

--min=1 \

--max=10

--cpu string

Target CPU utilization over all the pods. When specified as a percentage (e.g."70%" for 70% of requested CPU) it will target average utilization. When specified as quantity (e.g."500m" for 500 milliCPU) it will target average value. Value without units is treated as a quantity with miliCPU being the unit (e.g."500" is "500m").

--memory string

Target memory utilization over all the pods. When specified as a percentage (e.g."60%" for 60% of requested memory) it will target average utilization. When specified as quantity (e.g."200Mi" for 200 MiB, "1Gi" for 1 GiB) it will target average value. Value without units is treated as a quantity with mebibytes being the unit (e.g."200" is "200Mi").

To interpret --cpu=60%, you have to look at it through the lens of your Pod's Resource Requests, not the actual capacity of the physical Node (EC2 instance).

1. The Reference Point: "Requests"

The 60% is a percentage of the CPU Request defined in your Deployment manifest.

If your Pod has request: 100m (0.1 CPU), the HPA will try to scale when average usage hits 60m.
If your Pod has request: 1000m (1 CPU), the HPA will try to scale when average usage hits 600m.

Crucial Requirement: If you do not define resources.requests.cpu in your Deployment, the HPA will have no "base" to calculate from. It will show a status of <unknown> and will not scale.

2. The Scaling Math

The HPA uses a specific ratio to decide how many replicas you need. It isn't instantaneous; it calculates the "desired" number of pods using this formula:

DesiredReplicas = ceil(CurrentReplicas x CurrentMetricValue/TargetMetricValue)

Example Scenario:

Target: 60%
Current Replicas: 2
Current Average Usage: 90%
Calculation: 2 x (90 / 60) = 3
Result: HPA will scale your deployment up to 3 replicas.

3. What happens if usage exceeds 100%?

How can usage go above 100%? If your CPU Limit is higher than your CPU Request (or if you have no limit), a pod can "burst" and use more than its requested amount.

If your request is 100m and your pod is using 120m, your utilization is 120%.
In this case, the HPA will aggressively scale up because you are significantly over your 60% target.

4. Summary of your Command

When you run kubectl autoscale deployment my-deployment --cpu=60% --min=1 --max=10:

The Floor: You will never have fewer than 1 pod, even if there is zero traffic.
The Ceiling: You will never have more than 10 pods, even if the CPU is screaming at 100% on all of them.
The Trigger: As soon as the average CPU usage across all running pods exceeds 60% of their requested value, the HPA will ask the Deployment to add more pods.

A Common Pitfall

The "Flapping" Effect: If you set your CPU target too low (like 10%) or your min and max too close together, your cluster might "flap"—constantly adding and removing pods. Kubernetes has a built-in stabilization window (usually 5 minutes for downscaling) to prevent this "yo-yo" behavior.

Verification Steps

Once created, verify it with the following:

List all HPAs: kubectl get hpa -A
Check Metrics Flow: Since your metrics-server is running, ensure it is actually collecting data by running kubectl top pods -A. If this returns usage data, your HPA will be able to scale correctly.

Note: For HPA to function, your Deployment must have resource requests (specifically cpu) defined in its container spec, or the HPA will show <unknown> targets.

How to configure metrics it needs to observe?

In AWS EKS, you set criteria for scaling in the spec section of a HorizontalPodAutoscaler (HPA) resource. You define thresholds through two primary blocks: metrics (to trigger scaling) and behavior (to control the rate and stability of scaling).

1. Setting Thresholds (metrics)

The HPA calculates the required number of replicas based on the gap between current usage and your target.

Target Utilization: Typically set as a percentage of a pod's requested CPU or memory.
Where to define: Inside the metrics list in your HPA manifest.

metrics:

- type: Resource

resource:

target:

type: Utilization

averageUtilization: 60 # Scale when average CPU exceeds 60% of requests

2. Setting Scaling Speed (behavior)

Advanced scaling logic is set in the behavior block, allowing you to fine-tune how fast the cluster grows or shrinks.

Stabilization Window: Prevents "flapping" by making the HPA wait and look at past recommendations before acting.

Scale-Up: Default is 0 seconds (instant growth).
Scale-Down: Default is 300 seconds (5 minutes) to ensure a spike is truly over before killing pods.

Policies: Restrict the absolute number or percentage of pods changed within a specific timeframe.

behavior:

scaleUp:

stabilizationWindowSeconds: 0

policies:

- type: Percent

value: 100

periodSeconds: 15 # Double replicas every 15 seconds if needed

scaleDown:

stabilizationWindowSeconds: 300

policies:

- type: Pods

value: 1

periodSeconds: 60 # Remove only 1 pod per minute for stability

3. Critical Prerequisites

Resource Requests: You must define resources.requests in your Deployment manifest. HPA cannot calculate utilization percentages without this baseline.
Metrics Server: Must be running in your cluster (usually in kube-system or default) to provide the data HPA needs.

To link an HPA to your Deployment, the HPA uses a scaleTargetRef. This acts like a pointer, telling the HPA controller exactly which resource to watch and resize.

1. Ensure your Deployment has "Requests"

The HPA cannot calculate percentages (like "50% CPU") unless the Deployment defines what 100% looks like. Check your Deployment for a resources.requests block:

# Inside your Deployment manifest

spec:

containers:

- name: my-app

image: my-image

resources:

requests:

cpu: "250m" # HPA uses this as the 100% baseline

memory: "512Mi"

Create a file named hpa.yaml. The scaleTargetRef is the "link" that connects it to your app.

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

namespace: default # MUST be the same as your deployment

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

minReplicas: 2

maxReplicas: 10

metrics:

- type: Resource

resource:

target:

type: Utilization

averageUtilization: 60 # Target 60% of requested CPU

3. Apply and Verify

Run the following commands to put the link into action:

Apply the manifest:

kubectl apply -f hpa.yaml

Check the link:

kubectl get hpa

Wait about 30-60 seconds. If you see TARGETS: <unknown>/60%, the Metrics Server is still calculating.

If you see TARGETS: 0%/60% (or a specific number), the link is successful!

Pro Tip - The Quick Link: If you don't want to write YAML, you can create this link instantly via the CLI:

kubectl autoscale deployment my-deployment-name \

--cpu=60% \

--min=2 \

--max=10

Replicas number settings in HPA vs Deployment

The Deployment manifest and the HPA manifest handle pod counts in fundamentally different ways. Here is the breakdown of how they interact:

1. The Deployment Manifest

In a standard Deployment, you define a static number of pods using the replicas field.

Static Nature: If you set replicas: 3, the Deployment controller will always try to keep exactly 3 pods running.

No Min/Max: There is no concept of "range" inside a Deployment. It doesn't know how to look at CPU usage or scale itself up or down based on traffic.

# Inside a Deployment

spec:

replicas: 3 # This is a fixed instruction

2. The HPA Manifest

The Horizontal Pod Autoscaler (HPA) is a separate resource that "takes control" of the Deployment's replica count. This is where your min and max boundaries live.

Dynamic Nature: The HPA monitors your pods (via Metrics Server).
Override Power: When you link an HPA to a Deployment, the HPA dynamically updates the replicas field in the Deployment for you. If traffic is low, it sets it to your minReplicas; if traffic spikes, it pushes it toward maxReplicas.

# Inside an HPA

spec:

minReplicas: 2 # The floor

maxReplicas: 10 # The ceiling

scaleTargetRef:

How They Work Together

Think of the Deployment as the "Engine" and the HPA as the "Cruise Control."

The "Deployment Conflict" Warning

One mistake many engineers make is leaving a hardcoded replicas: 3 in their Deployment manifest while also using an HPA.

What happens:

You apply your Deployment with replicas: 3.
The HPA sees low traffic and scales the Deployment down to minReplicas: 1.
Later, you run kubectl apply on your Deployment file again.
The cluster briefly resets the pods to 3 because that's what's in your file, before the HPA notices and fights back to scale it back down to 1.

The Solution: Once you have an HPA configured, it is best practice to remove the replicas line from your Deployment manifest entirely. This ensures the HPA has sole authority over the pod count.

---

HPA does not modify the replicas field in your Deployment manifest (the YAML file). Here's how it actually works:

What HPA does: HPA dynamically updates the spec.replicas field of the live Deployment object in the Kubernetes API server (i.e., the in-cluster resource), not your source manifest file on disk. It does this by issuing a PATCH request to the API server whenever it decides to scale.

What this means in practice:

Your deployment.yaml file stays unchanged — the replicas: 3 (or whatever you set) remains as-is on disk.
The actual running replica count in the cluster is controlled by HPA at runtime.
If you kubectl apply your manifest again, Kubernetes is smart enough (since k8s 1.21+ with server-side apply) to not override HPA's current value — but with client-side apply, re-applying can reset replicas back to what's in the file, which is a common footgun.

Recommended practice:

If you're using HPA, you should either:

Omit replicas from your Deployment manifest entirely — Kubernetes defaults to 1, and HPA takes over immediately.
Or set it to a value within your HPA's minReplicas/maxReplicas range, and be aware of the re-apply caveat.

The control loop in short:

HPA Controller (runs every 15s by default)

→ reads metrics (CPU, memory, custom)

→ calculates desired replicas

→ PATCHes the Deployment object in etcd

→ Deployment controller reconciles actual pods

So your manifest is the initial desired state, and HPA is a runtime override that continuously manages the live object — it never touches your files.

---

HPA vs Karpenter

In AWS EKS, it is perfectly normal to have Karpenter running without any HorizontalPodAutoscaler (HPA) manifests. This happens because they solve two entirely different scaling problems:

1. Karpenter vs. HPA: The "Supply" vs. "Demand" Gap

HPA manages Pods (The Demand): It decides how many pods you need (e.g., "CPU is high, let's go from 2 pods to 5 pods").
Karpenter manages Nodes (The Supply): It provides the underlying infrastructure for those pods. It watches for pods that are "Pending" because there is no room for them, then quickly spins up a new EC2 instance that fits them perfectly.

If you have no HPAs, it means your application replica counts are currently static (e.g., always 3 pods). Karpenter is only "scaling" when you manually change that number or when you deploy a new app that needs more room.

2. Can pods be adjusted automatically without HPA?

Yes, there are a few other ways pod counts or resources can be adjusted:

Vertical Pod Autoscaler (VPA): Instead of adding more pods, VPA adjusts the CPU and Memory limits of your existing pods based on actual usage.
KEDA (Kubernetes Event-driven Autoscaling): Often used instead of standard HPA for complex triggers. It can scale pods to zero and back up based on external events like AWS SQS queue depth, Kafka lag, or Cron schedules.
GitOps/CD Pipelines: Sometimes scaling is "automated" via external CI/CD tools (like ArgoCD) that update the replica count in your git repo based on specific triggers or schedules rather than in-cluster metrics.

3. Why you might want to add HPA

Without HPA, Karpenter is essentially a "just-in-time" provisioning tool for a static workload. If your traffic spikes, your pods might crash from resource exhaustion before Karpenter has a reason to act. Adding HPA allows your app to "request" more pods, which then triggers Karpenter to "supply" more nodes.

To handle traffic spikes, HPA and Karpenter work as a two-stage relay:

HPA (The Demand): Triggers when CPU/Memory usage spikes, creating "Pending" pods that cannot fit on current nodes.
Karpenter (The Supply): Sees those "Pending" pods and immediately provisions new EC2 instances to accommodate them.

The Combined YAML Example

This configuration sets up an application to scale up during spikes and ensures Karpenter has the right "instructions" to provide nodes.

Part A: The Workload (Deployment)

You must define resource requests so HPA has a baseline and Karpenter knows what size node to buy.

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 2

template:

spec:

containers:

- name: web-server

image: nginx

resources:

requests:

cpu: "500m" # Crucial: HPA uses this for % calculation

memory: "512Mi" # Crucial: Karpenter uses this to select EC2 size

Part B: The Scaling Rule (HPA)

This tells Kubernetes to add pods when the existing ones are busy.

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

minReplicas: 2

maxReplicas: 20

metrics:

- type: Resource

resource:

target:

type: Utilization

averageUtilization: 60 # Scale up at 60% to give Karpenter time to boot nodes

Part C: The Node Provisioner (Karpenter NodePool)

This tells Karpenter which AWS instances are "allowed" for your scaling pods.

apiVersion: karpenter.sh/v1beta1

kind: NodePool

metadata:

spec:

template:

spec:

requirements:

- key: "karpenter.sh/capacity-type"

operator: In

values: ["spot", "on-demand"] # Use Spot to save money during spikes

- key: "karpenter.k8s.aws/instance-category"

operator: In

values: ["c", "m", "r"]

nodeClassRef:

disruption:

consolidationPolicy: WhenUnderutilized # Automatically kill nodes when pods scale down

Why this works for spikes

Buffer Time: Setting the HPA to 60% (instead of 90%) ensures you start scaling before pods are overwhelmed, giving Karpenter ~60 seconds to join new nodes to the cluster.
Just-in-Time Nodes: Unlike the old Cluster Autoscaler, Karpenter doesn't wait for "Node Groups" to update; it calls the EC2 Fleet API directly to get exactly what your pending pods need.
Automatic Cleanup: When the spike ends, HPA reduces pod counts. Karpenter's consolidationPolicy then notices the nodes are empty and terminates them to stop AWS billing.

In Kubernetes, "700m" stands for 700 millicores.

It is a unit of measurement for CPU processing power, where 1000m is equal to 1 vCPU (or 1 Core). Therefore, 700m is 0.7 of a vCPU.

How it works in your HPA:

Since your HPA is using type: AverageValue (instead of Utilization), it is looking at the raw CPU usage rather than a percentage:

The Trigger: The HPA controller calculates the average CPU usage across all currently running pods in that deployment.
The Action: If the average usage exceeds 700m, the HPA will add more pods to spread the load and bring that average back down to 700m.

Why use AverageValue instead of AverageUtilization?

AverageUtilization (Percentage): Requires you to have resources.requests defined. It scales based on "percentage of what I asked for."
AverageValue (Raw Number): Does not technically require a request baseline to function. It scales based on "absolute CPU consumed." This is useful if your app has a hard performance limit (e.g., "This app starts lagging if it hits 0.7 cores") regardless of what the pod's requested limit is.

Pro-Tip for Karpenter users:

When using AverageValue, ensure your Deployment's CPU request is set to something sensible (like 800m or 1000m). If your request is lower than your HPA target (e.g., request is 500m but target is 700m), your pods will constantly throttle before they ever trigger a scale-up! Kubernetes Resource Management provides more details on these units.

If you are using Karpenter for node scaling, it works beautifully with HPA. Just remember that HPA and VPA generally shouldn't be used on the same metric (like CPU) for the same deployment, as they might fight each other.

Horizontal Pod Autoscaler and Upgrading Kubernetes version of the cluster

When upgrading your Kubernetes cluster version, the most critical Horizontal Pod Autoscaler (HPA) considerations involve API version deprecations, metrics server compatibility, and newly introduced scaling configuration fields.

1. API Version Deprecations & Removals

Kubernetes frequently matures its APIs, meaning older HPA versions are deprecated and eventually removed.

autoscaling/v2 is now GA (General Availability): As of Kubernetes v1.23, the autoscaling/v2 API version is stable and generally available.
Removal of v2beta2: The autoscaling/v2beta2 version was removed in v1.26. If your manifests still use this version, they will fail to apply or update in clusters v1.26 and newer.
Manifest Updates: You must update the apiVersion in your YAML files. Note that fields like targetAverageUtilization in beta versions were replaced by a more structured target block in the stable v2 API.

2. Metrics Server & Infrastructure

The HPA depends on external components that may also require updates during a cluster upgrade.

Metrics Server Compatibility: Ensure your Metrics Server version is compatible with your new Kubernetes version. Without it, HPA cannot fetch CPU or memory data, and scaling will fail.
Custom Metrics Adapters: if you use custom metrics (e.g., via Prometheus), ensure your Prometheus Adapter supports the new Kubernetes API version, as some older adapters may still attempt to call removed API endpoints.

3. New Features and Behaviors

Upgrading allows you to leverage newer scaling controls that improve stability:

Configurable Scaling Behavior: Introduced in v1.18 and matured in later versions, the behavior field allows you to set a stabilizationWindowSeconds for scale-up and scale-down independently. This is essential for preventing "flapping" (rapidly scaling up and then down).
Configurable Tolerance: In very recent versions (e.g., v1.33), you can now fine-tune the 10% default tolerance. Previously, HPA would only act if the metric differed by more than 10%; you can now adjust this for more sensitive or coarser scaling needs.

4. Best Practices for the Upgrade Process

Audit Before Upgrading: Use tools like Kube-no-trouble (kubent) or Pluto to find resources using deprecated HPA APIs.
Dry Runs: Run kubectl apply --dry-run=client on your HPA manifests against the target cluster version to catch schema errors before they impact production.
Monitor Events: After upgrading, watch HPA events using kubectl get events --field-selector involvedObject.kind=HorizontalPodAutoscaler to ensure it is still successfully fetching metrics and making decisions.

When moving from the deprecated autoscaling/v2beta2 (removed in v1.26) to the stable autoscaling/v2 (available since v1.23), the primary change is the unification of target fields.

YAML Comparison

The stable v2 API replaces direct target fields (like averageUtilization) with a nested target block.

Feature Deprecated autoscaling/v2beta2 Stable autoscaling/v2

------- ------------------------------- ----------------------

API Version apiVersion: autoscaling/v2beta2 apiVersion: autoscaling/v2

CPU Target averageUtilization: 50 target:type:Utilization averageUtilization: 50

Custom Target averageValue: 100 target:type:AverageValue averageValue: 100

Comparison Example:

A comparison of the YAML structure shows how the apiVersion changes and the resource target is nested within a target block in the v2 version. You can see the full YAML example in the referenced documents.

Key Migration Notes

Seamless Conversion: The Kubernetes API server can convert between these versions, allowing you to use kubectl get hpa <name> -o yaml --output-version=autoscaling/v2 to view HPAs in the new format.
Manifest Updates: While conversion is possible, you must update your CI/CD pipelines and YAML manifests to use autoscaling/v2 before upgrading to v1.26 to prevent errors.
Behavior Block: The behavior block remains the same structurally, but using the v2 API is required for long-term support.

To identify which HPAs in your cluster are using deprecated API versions, you can use built-in kubectl commands or specialized open-source tools.

1. Using Built-in kubectl Commands

While kubectl doesn't have a single "find-deprecated" flag, you can use these methods to audit your resources:

Audit via API Server Warnings (v1.19+):

The API server automatically sends a warning header when you access a deprecated endpoint. Simply listing them often triggers a warning in the console if they use deprecated APIs:

kubectl get hpa -A

Dry-Run Manifest Validation:

Before applying an update, use a client-side dry-run to see if the manifest will be accepted by the new cluster version's schema.

kubectl apply -f your-hpa.yaml --dry-run=client

Check Metrics for Requested Deprecated APIs:

You can query the API server's raw metrics to see if any client (like an old CI/CD script) is still requesting deprecated HPA versions.

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

2. Using Specialized Audit Tools (Recommended)

Specialized tools are the most reliable way to find exactly which resources are affected before an upgrade.

Kube-no-trouble (kubent):

This tool scans your live cluster and lists exactly which resources are using APIs scheduled for removal.

# Install and run (requires no cluster installation)

sh -c "$(curl -sSL https://git.io/install-kubent)"

kubent

Pluto:

While kubent scans the live cluster, Pluto is best for scanning your Helm charts and static YAML files in your git repository to catch issues before they are deployed.

# Scan local directory

pluto detect-files -d .

3. Quick Check of Supported Versions

To see which API versions your cluster currently supports for horizontal scaling, use the following command:

kubectl api-versions | grep autoscaling

Note: If you are upgrading to v1.26 or newer, any HPA using autoscaling/v2beta2 must be updated to autoscaling/v2, as the older version will no longer be served

How to use kubectl convert to automatically upgrade your existing YAML manifests to the latest API version?

The kubectl convert command is no longer part of the standard kubectl binary; it is now a standalone plugin. You must install it to automatically upgrade your HPA manifests from v2beta2 to v2.

1. Install the kubectl-convert Plugin

Choose the method that matches your operating system:

macOS (via Homebrew):

brew install kubectl-convert

Manual Download (Linux/macOS/Windows):

Download the binary for your architecture from the official Kubernetes release page and move it to your system path (e.g., /usr/local/bin/kubectl-convert).

Verification:

Run kubectl convert --help to confirm the plugin is active.

2. Convert Your HPA Manifests

Once installed, you can use the command to rewrite your old YAML files to the stable autoscaling/v2 version.

Convert a Specific File:

This command reads your v2beta2 file and outputs a clean v2 version to your terminal.

kubectl convert -f old-hpa.yaml --output-version autoscaling/v2

Save the Converted File:

kubectl convert -f old-hpa.yaml --output-version autoscaling/v2 > new-hpa.yaml

Bulk Conversion (Directory):

You can point it to a directory containing multiple manifests to update them all at once.

kubectl convert -f ./my-hpa-folder/ --output-version autoscaling/v2

3. Alternative: Direct Export from the Cluster

Because the Kubernetes API server internally handles conversion between versions, you can "live-convert" an existing HPA by explicitly requesting the target version during a get command:

kubectl get hpa.v2.autoscaling <hpa-name> -o yaml > upgraded-hpa.yaml

This method is often faster if the HPA is already running in your cluster, as it bypasses the need for the convert plugin.

How to use Kustomize to handle these API version changes across multiple environments?

Kustomize allows you to manage the transition from v2beta2 to v2 across multiple environments (e.g., Dev, Staging, Prod) by layering environment-specific changes over a common base configuration.

1. Structure Your Directory

Keep your primary HPA manifest in a base folder and create overlays for each environment.

├── base

│ ├── hpa.yaml

│ └── kustomization.yaml

└── overlays

├── dev

│ └── kustomization.yaml

└── prod

├── hpa-v2-patch.yaml

└── kustomization.yaml

2. Strategy for Phased Upgrades

If you are upgrading clusters one by one, you can use Kustomize patches to change the apiVersion only for specific environments while keeping others on the older version.

Example: Upgrading 'Prod' to v2

If your base/hpa.yaml still uses v2beta2, you can create a patch in your Prod overlay to upgrade it to v2 without touching the base file used by other environments.

overlays/prod/hpa-v2-patch.yaml:

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

spec:

metrics:

- type: Resource

resource:

target:

type: Utilization

averageUtilization: 60

overlays/prod/kustomization.yaml:

resources:

- ../../base

patches:

- path: hpa-v2-patch.yaml

target:

kind: HorizontalPodAutoscaler

3. Validating the Conversion

Before applying changes to a live cluster, use the following commands to ensure Kustomize has correctly merged the new apiVersion and schema:

View Rendered YAML:

kubectl kustomize overlays/prod

Diff Against Live Cluster:

Use kubectl diff to see exactly what will change in the API server.

kubectl diff -k overlays/prod

4. Best Practices

Keep Base "Newest": Once all clusters are upgraded, move the v2 configuration into the base and remove the patches from your overlays to keep your code DRY.

CI/CD Integration: Use Pluto in your CI pipeline to scan the output of kustomize build for any remaining deprecated API versions.