My Public Notepad

Elastic Fleet

2026-06-25T13:36:56.911+01:00

Elastic Agents do not strictly require Fleet and Fleet Server. You can deploy them in standalone mode, which allows you to manually configure and manage them using local configuration files.

However, running Elastic Agents in Fleet-managed mode with a Fleet Server is the recommended best practice for most enterprise environments.

Here is how the two approaches compare:

1. Fleet-Managed Mode (Recommended)

In this setup, you use the Fleet UI in Kibana to centrally manage agent policies, roll out upgrades, and apply integrations.

How it works: Agents connect to a Fleet Server (which is just a specialized Elastic Agent process), which then receives policies from Elasticsearch and pushes them to your endpoints.
Best for: Large-scale deployments, continuous monitoring, and Elastic Security/Endpoint integrations.
Action: To set this up, refer to the official Elastic Agent Installation Guide.

2. Standalone Mode

In this setup, you manually install the agent and define its inputs, outputs, and integrations directly in a local YAML configuration file.

How it works: The agent connects directly to outputs like Elasticsearch or Logstash without a Fleet Server intermediary.
Limitations: Central management, automated upgrades, and certain advanced Endpoint Security features are disabled.
Best for: Edge cases, highly air-gapped networks, or evaluating specific integrations.
Action: To configure this, use the steps outlined in the Standalone Elastic Agent Tutorial.

Example of how an Elastic Fleet can actually be wired

Components & where they live

Everything is in cluster company-prod-elastic-eks (EKS, us-east-2, now on v1.36.2), namespace elastic-system, across 5 nodes in 3 AZs:

Node AZ Node Group Runs

==== == ========== ====

ip-10-99-44-1 2a es MNG (m7g.2xlarge) agent

ip-10-99-55-34 2b default MNG (m5.large) agent + Fleet Server pod

ip-10-99-55-89 2b es MNG (m7g.2xlarge) agent

ip-10-99-66-51 2c es MNG (m7g.2xlarge) agent

ip-10-99-66-78 2c default MNG (m5.large) agent

Fleet Server — Deployment fleet-server-prod, replicas: 1 (single pod, currently on the 2b node). Listens on :8220 HTTPS. Exposed by a NodePort service fleet-server-prod-agent-http (8220 --> 32202 on every node).
Elastic Agents — DaemonSet agent-prod-eck-agent, one pod per node (5 total), spread across all 3 AZs. hostNetwork: true, mode: fleet, FLEET_INSECURE=true. They collect node/pod logs + metrics (hostPath mounts of /var/log/...).
ALB k8s-elastics-eckfleet-a1b2c3d4e5 — internal (not internet-facing), spanning subnets in 2a / 2b / 2c. Created by the AWS LB controller from Ingress eck-fleet-server-prod-ingress. Targets = all 5 nodes' NodePort 32202 (instance mode), backend-protocol: HTTPS, idle_timeout: 300s.
DNS — eck-fleet-server.internal-domain.local (Route53 via external-dns) --> the internal ALB. Elasticsearch is a separate endpoint, elasticsearch.internal-domain.local:443.

Who listens, who initiates

The key thing: every connection is initiated by the agent (outbound). Nothing is ever pushed to the agents — that's why this works with agents behind hostNetwork and no inbound rules of their own.

┌───────────────────────────────────────────────┐

│ cluster company-prod-elastic-eks / elastic-sys│

┌──────────┐ │ │

│ Agent │ check-in │ internal ALB Fleet Server (1 pod) │

│ DaemonSet│─────────►│ eckfleet-a1b2... ┌──────► :8220 (HTTPS) │

│ (5 pods, │ :443 │ :443 TLS ─────────┘ NodePort 32202→8220 │

│ 3 AZs) │ HTTPS │ (internal cert) kube-proxy → the 1 pod │

└────┬─────┘ │ ▲ re-encrypt HTTPS to a node's :32202 │

│ │ │ │ │

│ ship data └───────┼─────────────────────┼─────────────────┘

│ (logs/metrics) │ DNS │ writes .fleet-* / agent

▼ eck-fleet-server. ▼ metadata, reads policy

elasticsearch. internal-domain.local Elasticsearch + Kibana (Fleet app)

internal-domain.local:443 ◄──────────────────────

(NOT via the fleet ALB)

Traffic Flows: Breakdown

1. Agent --> Fleet Server (Control Plane: Enrollment & Policy Check-in)

The agent acts as the client, and Fleet Server listens on :8220. The agent dials https://eck-fleet-server.internal-domain.local:443:

Traffic Path:

agent --> DNS --> internal ALB :443 (TLS termination, internal cert) --> re-encrypt --> a node's NodePort :32202 --> kube-proxy --> the single Fleet Server pod :8220

Cross-AZ Routing:

Because the agent always targets the ALB DNS name (not the pod directly), an agent can land on any node's NodePort. It is then forwarded by kube-proxy to whichever node is hosting the single Fleet Server pod—making cross-AZ hops standard behavior.

The Long Poll Mechanism: The check-in is a long poll. The agent opens a connection, and Fleet Server holds it open for up to ~5 minutes until a policy change occurs or the poll times out.

The Latency "False Positive": This held-open duration is exactly what the ALB records as a TargetResponseTime of ~300s, which triggered the false-positive alert. The ALB's idle_timeout is intentionally set to 300s to support these long polls. Conversely, the health check (/health, expects 404, 300s interval) is a separate lightweight probe, explaining why targets report as perfectly healthy while "latency" metrics look highly skewed.

2. Agent --> Elasticsearch (Data Plane: Telemetry Shipping)

This data path does not traverse the Fleet ALB. The Fleet output configuration points agents directly at https://elasticsearch.internal-domain.local:443.

Consequently, the eckfleet ALB only handles control-plane check-in traffic. This explains why its request volume is incredibly low (~3–5 requests/min) while individual request durations hover around 300s. They are two entirely separate paths utilizing two different load balancers.

3. Fleet Server --> Elasticsearch + Kibana

Fleet Server (via elasticsearchRefs / kibanaRef) acts as the client here. It writes agent and policy metadata directly into the .fleet-* indices and coordinates with the Kibana Fleet application, where the eck-fleet-server and eck-agent policies are managed.

One-Line Summary

Agents (DaemonSet, 5 pods across 2a/2b/2c) long-poll the single Fleet Server pod for policy updates via the internal ALB on :443 --> NodePort 32202 --> :8220, but ship their actual telemetry data directly to Elasticsearch, completely bypassing that ALB. The ALB only processes the slow, held-open control traffic—which is the root cause of the alarm.

Why does fleet server holds connection open for so long? Why it does not reply immediately?

It's a deliberate design choice called long polling, and it's the opposite of a bug. Here's the reasoning.

The problem Fleet is solving

Fleet Server's job is to push policy changes out to agents promptly — a new integration, a changed log path, an output credential rotation. Agents need to find out "has my policy changed?" with low latency.

There are three ways an agent could learn about changes:

1. Short polling — agent asks "anything new?" every N seconds, server replies instantly "no", agent sleeps, repeats.

2. Server push — server opens a connection to each agent when something changes.

3. Long polling — agent asks "anything new?", and the server holds the request open until either something actually changes or a timeout fires.

Fleet uses #3, and the connection sitting open for ~5 minutes is that hold.

Why not reply immediately (short polling)?

If Fleet answered every check-in instantly with "nothing changed", then to get fast reaction to a policy change, agents would have to poll very frequently — say every few seconds. With your 5 agents that's tolerable, but Fleet is built to manage thousands to tens of thousands of agents from one server. At that scale:

- Frequent short polls = a constant storm of requests, almost all of which return "no change". Huge wasted CPU/network on both ends.

- To keep reaction time low you'd poll more often, which makes the storm worse. To cut the storm you'd poll less often, which makes policy changes take longer to land. There's no good setting.

Long polling breaks that trade-off: the agent gets a near-instant reaction to a real change (the server responds the moment policy updates) and there's almost no idle chatter (one held-open connection per agent instead of hundreds of empty round-trips).

Why not server push (#2)?

Pushing would mean the server initiating connections inward to every agent. Agents are all over the place — behind NAT, firewalls, in private subnets, on laptops, on hostNetwork pods like yours. The server usually can't reach them, and you'd need inbound rules everywhere. Long polling flips it: the agent always dials out to the server, the connection is already established and held open, and the server pushes the change down that existing agent-initiated connection the instant it happens. You get push-like latency with poll-like (outbound-only) connectivity. That's exactly why your agents work behind hostNetwork with no inbound exposure.

So what's actually happening in those ~300s

A check-in is essentially: "Here's my current state; tell me the moment my policy differs." The server parks that request. Two things can end it:

- A policy change occurs → server responds immediately with the new policy (could be 2 seconds in).

- Nothing changes → the server lets the request time out at its poll ceiling (~5 min), responds "no change", and the agent immediately opens a fresh one.

Since your policies rarely change, almost every check-in runs the full clock and returns at ~300s. That's the held connection the ALB measures as TargetResponseTime.

Why this specifically fools the ALB

From the ALB's point of view, "request received → response sent" took 300 seconds, so it reports TargetResponseTime ≈ 300s. The ALB can't tell the difference between "the backend was slow for 300s" (bad) and "the backend intentionally held an idle long-poll for 300s" (normal). That ambiguity is the whole reason your generic 0.8s threshold misfires — and why the right fix (DOP-833) is to exempt this endpoint rather than treat it as latency. It's also why the Ingress sets idle_timeout.timeout_seconds=300: the ALB has to be told to tolerate the held connection, otherwise it would sever the long poll before the agent's poll cycle completes.

In short: Fleet holds the connection open so it can deliver policy changes near-instantly without either hammering the server with empty polls or needing to reach inward to firewalled agents. The ~300s is just an idle long-poll waiting for a change that usually never comes during that window — efficient by design, and only a problem for a latency metric that doesn't know to expect it.

----

AWS EC2: Application Load Balancer

2026-06-25T13:11:38.955+01:00

An Application Load Balancer (ALB) is a fully managed AWS service that automatically distributes incoming HTTP and HTTPS traffic across multiple backend targets.

It operates at the Application Layer (Layer 7) of the Open Systems Interconnection (OSI) model.

Key Features:

Content-Based Routing: Routes traffic based on URL paths (/api vs /images) or hostnames (://example.com).
Container Support: Integrates directly with Amazon ECS and EKS using dynamic port mapping.
Advanced Protocols: Native support for modern protocols like HTTP/2, gRPC, and WebSockets.
Security Integration: Features built-in HTTPS/TLS termination and integrates directly with AWS WAF for web security.

How Components Work Together

Listener: Evaluates connection requests from clients using protocols and ports you configure.
Rules: Determines how the load balancer routes requests to its registered targets.
Target Group: Groups backend resources (like EC2 instances, containers, or IP addresses) that receive the traffic

How ALB health checks keep applications online?

An Application Load Balancer (ALB) keeps your application online by continuously monitoring the health of your backend targets and dynamically redirecting traffic away from failing nodes.

1. Automatic Traffic Redirection

The ALB sends periodic ping requests (health checks) to every registered target. If a target fails to respond correctly, the ALB marks it as unhealthy and immediately stops sending user traffic to it. Traffic is rerouted to the remaining healthy nodes with zero downtime for the user.

2. Auto Scaling Integration

When paired with an Auto Scaling Group (ASG), ALB health checks can trigger the automatic replacement of broken instances.

The Problem: An EC2 instance might be running (healthy at the hardware level), but the web server inside it has crashed (unhealthy at the application level).
The Solution: The ALB tells the ASG that the instance is failing application health checks. The ASG terminates that specific broken instance and launches a fresh, working one.

3. Graceful Recovery

When an unhealthy instance recovers, or when a new instance is launched, the ALB does not send traffic to it immediately. It enters an initial state and undergoes consecutive successful health checks. Only when it passes the threshold does the ALB safely introduce it back into the traffic rotation.

How to Configure an ALB Health Check

You configure health checks inside the Target Group settings using these parameters:

Parameter What it does Recommended Setting

======== ========= =================

Health Check Path The URL endpoint the ALB hits

(e.g., /health or /index.html). /health

Healthy Threshold Consecutive successes needed to mark a target as healthy. 3

Unhealthy Threshold Consecutive failures needed to mark a target as unhealthy. 2

Timeout How long the ALB waits for a response before failing. 5 seconds

Interval The time between individual health check pings. 30 seconds

Success Codes The HTTP status codes that prove the app is working. 200 (or 200-399)

When you configure an ALB, you do not select an Availability Zone (AZ) directly; instead, you must select at least two subnets in different Availability Zones to ensure high availability.

How it works: AWS places a load balancer node in each of the specified subnets.
The AZ link: Because each subnet belongs to exactly one AZ, this fundamentally binds the ALB's nodes to those corresponding Availability Zones.
Custom routing: You can modify the subnets via the Update Availability Zones settings in the EC2 Console at any time.

Public ALB

Binding an ALB to public subnets makes it a public (internet-facing) load balancer.

When you create an internet-facing ALB, AWS requires you to select public subnets so the ALB nodes can receive a public IP address and route traffic from the internet.

Key Characteristics:

Public DNS: The ALB receives a public DNS name that resolves to public IP addresses.
Internet Gateway: The selected public subnets must have a route to an Internet Gateway (IGW) in their route tables.
Target Routing: Even though the ALB is public, it can still route traffic to EC2 instances living in private subnets

An internet-facing ALB routes traffic directly to the individual backend targets (such as EC2 instances or IP addresses), not to the private subnets themselves.

How Routing Works

Target Group Config: You configure the ALB to route traffic to a Target Group.
Direct Node Communication: The ALB nodes in the public subnets communicate directly with the private IP addresses of your backend nodes.
Cross-Subnet Traffic: AWS handles this routing internally via the VPC router, allowing the public ALB to securely traverse into private subnets.

Configuration Checklist

VPC: Both the public subnets (where the ALB lives) and the private subnets (where the nodes live) must be in the same VPC.
Security Groups: The private instances must have a security group that allows inbound traffic from the ALB's security group

Private ALB

An internal (private) ALB routes traffic in the exact same way as a public ALB, but it is only accessible within your VPC or connected networks.

It routes traffic directly to individual backend targets, not to subnets.

Key Characteristics

Private Subnets: You deploy the ALB nodes into private subnets.
Private DNS: The ALB receives a public DNS name, but it resolves exclusively to private IP addresses.
No Internet Access: It cannot receive any traffic from the public internet because it lacks a public IP.

Common Use Cases

Internal Microservices: Routing traffic from a public-facing web tier to a private backend API tier.
Hybrid Networks: Routing traffic coming from an on-premises data centre via AWS Direct Connect or a VPN

Setting Up ALB in AWS Console

AWS Elastic Load Balancing shows basic building blocks of AWS Load Balancer which include listeners and target groups.

To create Application Load Balancer go to EC2 >> Load balancers >> Create Load balancer >> Select load balancer type (click on Create under Application Load Balancer)

Here we can set:

Basic configuration

Name
Scheme (cannot be changed after the load balancer is created)

Internet-facing. An internet-facing load balancer routes requests from clients over the internet to targets. Requires a public subnet.
Internal. An internal load balancer routes requests from clients to targets using private IP addresses.

IP address type. Select the type of IP addresses that your subnets use.

IPv4. Recommended for internal load balancers.
Dualstack. Includes IPv4 and IPv6 addresses.

Network mapping. The load balancer routes traffic to targets in the selected subnets, and in accordance with your IP address settings.

VPC. Virtual private cloud for your targets. If balancer is internet-facing, only VPCs with an internet gateway are enabled for selection. The selected VPC cannot be changed after the load balancer is created. As VPC is region-specific so is Application Load Balancer.
Mappings. Once VPC is selected, its availability zones are listed here and are selectable. Select at least two Availability Zones and one subnet per zone. The load balancer routes traffic to targets in these Availability Zones only. Availability Zones that are not supported by the load balancer or the VPC are not available for selection. We should select all AZs that we listed in the Auto scaling group (if we used it).

Security groups. A security group is a set of firewall rules that control the traffic to your load balancer. We can select up to 10 security groups.

If our application is listening for HTTP requests on port 80 we should select a security group with:

Inbound rule: accept HTTP/TCP traffic on port 80 with source Anywhere-IPv4
Outbound rule: allow all traffic for all protocols and port ranges to custom destination 0.0.0.0/0

Listeners and routing. A listener is a process that checks for connection requests using the port and protocol you configure. The rules that you define for a listener determine how the load balancer routes requests to its registered targets.

Add listener

Protocol e.g. HTTP
Port e.g. 80. This is a public facing port and it does not need to be the same as the port from the attached target group. E.g. LB can listen on port 80 and forward traffic to target group port 8080.
Default action: Forward to (select a target group)
Add listener tags

Add-on services - optional

AWS Global Accelerator

Tags - optional

More info on Scheme, from AWS documentation:

When you create a load balancer, you must choose whether to make it an internal load balancer or an internet-facing load balancer.

The nodes of an internet-facing load balancer have public IP addresses.

The nodes of an internal load balancer have only private IP addresses.

Both internet-facing and internal load balancers route requests to your targets using private IP addresses. Therefore, your targets do not need public IP addresses to receive requests from an internal or an internet-facing load balancer.

More info on how ALB routes traffic to multiple Availability Zones (and about what Load Balancer Nodes are):

When you enable an Availability Zone for your load balancer, Elastic Load Balancing creates a load balancer node in the Availability Zone.
The nodes for your load balancer distribute requests from clients to registered targets. When cross-zone load balancing is enabled, each load balancer node distributes traffic across the registered targets in all enabled Availability Zones. When cross-zone load balancing is disabled, each load balancer node distributes traffic only across the registered targets in its Availability Zone.
Before a client sends a request to your load balancer, it resolves the load balancer's domain name using a Domain Name System (DNS) server. The DNS entry is controlled by Amazon, because your load balancers are in the amazonaws.com domain. The Amazon DNS servers return one or more IP addresses to the client. These are the IP addresses of the load balancer nodes for your load balancer.
As traffic to your application changes over time, Elastic Load Balancing scales your load balancer and updates the DNS entry. The DNS entry also specifies the time-to-live (TTL) of 60 seconds. This helps ensure that the IP addresses can be remapped quickly in response to changing traffic.

The client determines which IP address to use to send requests to the load balancer. The load balancer node that receives the request selects a healthy registered target and sends the request to the target using its private IP address.

With Application Load Balancers, the load balancer node that receives the request uses the following process:

1) Evaluates the listener rules in priority order to determine which rule to apply.

2) Selects a target from the target group for the rule action, using the routing algorithm configured for the target group. The default routing algorithm is round robin. Routing is performed independently for each target group, even when a target is registered with multiple target groups.

For further info: How Elastic Load Balancing works - Elastic Load Balancing

ALB nodes use Elastic Network Interface (Elastic network interfaces - Amazon Elastic Compute Cloud) which has public IP address:

At least one ENI is created and attached to the balancer in each availability zone where the balancer is deployed (except NLB, which should only have one per AZ). Over the life of the balancer, new ENIs will appear and old ones will disappear, as the balancer scales horizontally (number of nodes) and/or vertically (capacity of underlying hardware), all of which is handled transparently by the infrastructure. Even though you can tag them, the tagging will become stale over time.

Source: amazon web services - AWS - Affect Load Balancer's tags to its Network Interfaces (ENI) - Stack Overflow

You can determine the IP addresses associated with an internal load balancer or an internet-facing load balancer by resolving the DNS name of the load balancer. These are the IP addresses where the clients should send the requests that are destined for the load balancer. However, Classic Load Balancers and Application Load Balancers use the private IP addresses associated with their elastic network interfaces as the source IP address for requests forwarded to your web servers.

Source: Find the IP address used by a load balancer to forward traffic to web servers

Load balancer routes requests to the targets in a target group and performs health checks on the targets. Target group is accepting requests from the load balancer and forwards them to targets. These targets can be e.g. EC2 instances created either manually or through auto scaling group.

How to create a Target Group used by Load Balancer listeners? (This applies for any type of Load Balancer)

EC2 >> Target groups >> Create target group

Step 1: Specify group details

Here we can set:

Basic configuration. Settings in this section cannot be changed after the target group is created.

Target type

Instances

Supports load balancing to instances within a specific VPC.
Facilitates the use of Amazon EC2 Auto Scaling to manage and scale your EC2 capacity.

IP addresses

Supports load balancing to VPC and on-premises resources.
Facilitates routing to multiple IP addresses and network interfaces on the same instance.
Offers flexibility with microservice based architectures, simplifying inter-application communication.
Supports IPv6 targets, enabling end-to-end IPv6 communication, and IPv4-to-IPv6 NAT.

Lambda function

Facilitates routing to a single Lambda function.
Accessible to Application Load Balancers only.

Application Load Balancer

Offers the flexibility for a Network Load Balancer to accept and route TCP requests within a specific VPC
Facilitates using static IP addresses and PrivateLink with an Application Load Balancer.

Target group name
Protocol:Port e.g. If our application is accepting HTTP requests on port 8080 this would be HTTP:8080
VPC - VPC with the instances that you want to include in the target group.
Protocol version

HTTP1. Send requests to targets using HTTP/1.1. Supported when the request protocol is HTTP/1.1 or HTTP/2.
HTTP2. Send requests to targets using HTTP/2. Supported when the request protocol is HTTP/2 or gRPC, but gRPC-specific features are not available.
gRPC. Send requests to targets using gRPC. Supported when the request protocol is gRPC.

Health checks. The associated load balancer periodically sends requests, per the settings below, to the registered targets to test their status.

Health check protocol

HTTP
HTTPS

Health check path. Use the default path of “/“ to ping the root, or specify a custom path if preferred.
Advanced health check settings

Port. The port the load balancer uses when performing health checks on targets. The default is the port on which each target receives traffic from the load balancer, but you can specify a different port.

Traffic port
Override

Healthy threshold. The number of consecutive health checks successes required before considering an unhealthy target healthy.
Unhealthy threshold. The number of consecutive health check failures required before considering a target unhealthy.
Timeout. The amount of time, in seconds, during which no response means a failed health check.
Interval. The approximate amount of time between health checks of an individual target
Success codes. The HTTP codes to use when checking for a successful response from a target. You can specify multiple values (for example, "200,202") or a range of values (for example, "200-299").

Attributes
Tags - optional

Step 2: Register targets

This is an optional step to create a target group. However, to ensure that your load balancer routes traffic to this target group you must register your targets.

After load balancer is created it takes several minutes while it's in provisioning state and get into active state. After this, we can use its DNS name in order to see what it's doing.

If we copy its DNS name and paste it to our browser, if we haven't registered any targets in the target group associated with the load balancer, we'll get error 503 - Service Temporary Unavailable.

If we've registered targets and are getting error 504 Gateway time-out, we should check first if security groups (firewalls) for our EC2 instances (inbound rule - source IP range) are set up correctly as this error usually indicates that inbound traffic is not allowed.

AWS Terraform provider offers provisioning all these resources:

Application Load Balancer: aws_lb | Resources | hashicorp/aws | Terraform Registry
Listener: aws_lb_listener | Resources | hashicorp/aws | Terraform Registry
Target Group: aws_lb_target_group | Resources | hashicorp/aws | Terraform Registry

How is AWS Application Load Balancing usually implemented?

Let's say we have our application running on 3 EC2 instances where 2 are in the same region e.g. us-west-2 but in separate availability zones e.g. us-west-2a and us-west-2b. Third EC2 instance is in eu-central-1, in availability zone eu-central-1a.

VPC is region-specific but can span multiple availability zones (AZ).

Subnet is an IP address range within VPC.

VPC can have public and private subnets.

VPC can be divided into multiple subnets but each subnet is AZ-specific.

AZ can have multiple subnets.

So, all EC2 instances belong to the same VPC but, as they are in different AZs, each of them belongs to different subnet.

Load balancer must be in the public subnet of VPC as clients communicate with load balancer via internet (public network).

Load balancer does not get associated directly with EC2 instances but subnets:

resource "aws_lb" "test" {

subnets = ["subnet-0001", "subnet-0002"]

...

}

Target group is associated with VPC:

resource "aws_alb_target_group" "test" {

vpc_id = var.vpc_id

...

}

Difference between ALB and NLB (Network Load Balancer)

An Application Load Balancer (ALB) and a Network Load Balancer (NLB) serve different purposes based on the layer of the network they operate on and the type of traffic they handle.

The core difference is that an ALB understands application-level traffic (Layer 7) like HTTP/HTTPS headers, while an NLB handles low-level network traffic (Layer 4) like TCP/UDP packets at extreme speeds.

Direct Comparison Matrix

Feature Application Load Balancer (ALB) Network Load Balancer (NLB)

====== ========================= =======================

OSI Layer Layer 7 (Application) Layer 4 (Transport)

Protocols HTTP, HTTPS, HTTP/2, gRPC, WebSockets TCP, UDP, TLS

IP Addresses

Dynamic IPs (Changes automatically; requires a DNS name)

Static IPs (Can assign an Elastic IP per AZ)

Routing Features

Advanced (Path, Host, Query parameters, Headers)

Basic (Port and IP protocol routing only)

Performance

Optimized for complex web apps (Millions of requests/sec)

Optimized for ultra-low latency (Billions of requests/sec)

Key Technical Differences

1. Smart Routing vs. Raw Speed

ALB (Smart): Can read the contents of your HTTP requests. It can route traffic bound for ://example.com to an API server cluster, and traffic for ://example.com to a storage cluster.
NLB (Fast): Does not look inside the data packet. It simply looks at the target port and forwards the packet instantly. This results in ultra-low latency (measured in milliseconds).

2. IP Addresses and DNS

ALB: Scale out dynamically by adding or removing nodes. This causes its underlying IP addresses to change frequently. You must always point your domain name to the ALB's DNS Name, never to a static IP.
NLB: Gives you a Static IP address per Availability Zone. You can also assign your own Elastic IP addresses. This is critical if your corporate clients need to whitelist specific, unchanging IPs in their firewalls.

3. Client IP Preservation

ALB: Terminates the connection and makes a new one to your backend instances. The backend see the ALB's private IP. To find the real user's IP, your code must read the X-Forwarded-For HTTP header.
NLB: Passes the original TCP packet straight through to your backend server. Your backend instances see the original source IP address of the client natively, without needing extra headers.

When to Choose Which?

Choose an ALB if you are building:

Standard web applications and microservices.
Containerized apps (ECS/EKS) requiring path-based or host-based routing.
Applications requiring tight integration with AWS Web Application Firewall (WAF).

Choose an NLB if you are building:

Non-HTTP applications (e.g., gaming servers, SFTP, MQTT, database clusters).
Architectures requiring fixed, static IP addresses or Elastic IPs.
High-frequency financial applications where sub-millisecond network latency is a hard requirement

Which alerts should typically be set for AWS ALB?

To keep your applications highly available, you should set up Amazon CloudWatch alarms for a mix of availability, performance, and target health metrics.

The most critical metrics to monitor for an AWS ALB are grouped by priority below:

1. High Priority (Critical Infrastructure Impact)

UnHealthyHostCount (Per Target Group)

What it means: The number of backend instances failing health checks Target Group Metrics.

Alert Threshold: > 0 (or > 1 for larger clusters).

Why it matters: Signals that your servers are crashing or cannot handle traffic.

HTTPCode_Target_5XX_Count

What it means: The number of 5xx server error codes generated by your backend application ALB Metrics.

Alert Threshold: Depends on baseline traffic, typically > 5 failures within a 1-minute to 5-minute window.

Why it matters: Indicates server crashes, database connection timeouts, or unhandled exceptions in your application code.

HTTPCode_ELB_5XX_Count

What it means: The number of 5xx errors generated directly by the ALB itself (not your servers) ALB Metrics.

Alert Threshold: > 0.

Why it matters: Usually means the ALB cannot find any healthy hosts, or it is experiencing a configuration mismatch (e.g., bad TLS handshake with the target).

2. Medium Priority (Performance & User Experience)

TargetResponseTime

What it means: The time elapsed (in seconds) from when the ALB sent the request to the target until the target started responding ALB Metrics.

Alert Threshold: Use the p95 or p99 statistic. Alert if it exceeds your application’s maximum acceptable latency (e.g., > 2.0 seconds).

Why it matters: Users are experiencing severe application slowdowns, likely due to high CPU/memory usage on your instances.

RejectedConnectionCount

What it means: The load balancer is rejecting connections because it has reached its maximum capacity ALB Metrics.

Alert Threshold: > 0.

Why it matters: Your application is getting sudden traffic spikes and the ALB cannot scale fast enough, or backend targets are failing to keep up.

3. Low Priority (Anomalies & Security)

HTTPCode_Target_4XX_Count

What it means: The number of 4xx client errors (like 404 Not Found or 401 Unauthorized) returned by backend targets ALB Metrics.

Alert Threshold: A significant spike above your standard baseline.

Why it matters: A sudden surge might indicate a broken frontend deployment, a bad API update, or a malicious entity scanning your network for vulnerabilities.

Summary Checklist for CloudWatch Alarms

Metric Name Statistic Recommended Suggested Action

Period Threshold

========== ====== =========== ======== =====

UnHealthyHostCount Maximum 1 Minute > 0 Page/On-Call

HTTPCode_ELB_5XX_Count Sum 1 Minute > 0 Page/On-Call

HTTPCode_Target_5XX_Count Sum 5 Minutes > 10 (or > 1% of traffic) Ticket/Slack

TargetResponseTime p95 5 Minutes> [Your Limit] Ticket/Slack

Resources:

How Elastic Load Balancing works - Elastic Load Balancing

Some notes on LLMs

2026-06-03T11:20:16.303+01:00

What does e.g. 1M context mean in a model description?

"1M context" means the model can handle up to 1 million tokens in its context window — the combined input (your messages, files, tool results, system prompt) plus output it can consider at once. For reference, that's roughly 750,000 words or a few thousand pages of text, letting it work with very large codebases or long conversations without losing earlier content.

What is a LLM's tool-use loop?

An LLM tool-use loop is the pattern where you let a language model drive an investigation by repeatedly choosing tools to call, rather than answering in one shot.

The shape

1. Send: system prompt + user request + list of available tools (with JSON schemas)

2. Model responds with either:

(a) a final text answer -> exit loop

(b) a "tool_use" block: { name: "run_aws_cli", input: { args: [...] } }

3. Your code executes that tool, captures the result

4. Append the tool result to the conversation as a "tool_result" message

5. Send the whole conversation back to the model

6. Goto 2

The model never executes anything itself — it just emits requests to call tools. Your code is the runtime that actually runs them and feeds the output back.

Why it's a loop

Each turn the model sees everything it has learned so far (prior tool calls + their outputs) and decides the next step based on that. So a real run looks like:

- Turn 1: model calls cloudwatch describe-alarms --state-value ALARM

- Turn 2: sees 3 alarms, picks the noisiest, calls logs filter-log-events for that log group around the alarm time

- Turn 3: sees an error pattern, calls kubectl describe pod on the affected workload

- Turn 4: emits final Markdown report, no tool call → loop exits

The model is doing the planning; your code is the dispatcher.

Why you need a budget

Without limits the loop can spin forever — the model keeps finding "one more thing to check." Hence in agent.run():

- max_iterations=30 — hard cap on turns

- max_tokens_per_turn=12288 — cap on a single response

- Per-tool wall-clock timeouts (60 s for CLI, 30 s for HTTP)

- Output truncation (50 000 char stdout) so a giant tool result doesn't blow the context window

How it ends

The loop terminates when the model returns a response with no tool_use block — that's the "I'm done, here's the answer" signal (stop_reason: end_turn). Or when you hit a budget limit and force-stop it.

Where the safety lives

Because the model can ask for arbitrary tool calls, the loop is only as safe as the tool implementations. That's why when implementing agents we should have the allowlists (services, verbs, paths) - the model can request aws s3 rm, but the validator rejects it before subprocess.run ever sees it.

The "two-pass" design in agent is a refinement: pass 1 is a tool-use loop (gather), pass 2 is a single non-loop call (synthesize). Splitting them lets each prompt focus on one job.

What are those .md files used by AI Agents?

There isn't a universally agreed official name, but people commonly refer to files like CLAUDE.md, GEMINI.md, AGENTS.md, COPILOT_INSTRUCTIONS.md, and .cursorrules as:

AI agent instruction files (most generic)
Agent configuration files
Agent context files
LLM instruction files
Repository AI instructions
Project AI guidelines

In the developer tooling community, "agent instructions" or "agent context files" are probably the most widely understood umbrella terms.

For example:

Tool File

---- -----

Claude Code CLAUDE.md

Gemini CLI GEMINI.md

GitHub Copilot .github/copilot-instructions.md

OpenAI Codex CLI AGENTS.md

Cursor .cursorrules / project rules

Windsurf Rules files

Collectively, you could describe them as:

"Repository-level AI agent instruction files that provide persistent context and operating rules for coding assistants."

If you're building tooling around them (e.g., in your DevOps work), I'd recommend using "agent instructions" as the generic term because it's vendor-neutral and easily understood across Claude, Gemini, Copilot, Cursor, Codex, and similar tools.

Introduction to Claude by Anthropic

2026-06-03T11:19:38.085+01:00

How to give Claude an instruction to apply label "DevOps" whenever it creates a new Linear ticket?

It depends on whether you're creating Linear tickets via:

Linear MCP,
a custom /create-ticket command,
or just asking Claude in chat to create tickets

The best location depends on which of those you're using.

If you're just asking Claude in chat to create tickets: if you want this behavior for all projects, put it in:

~/.claude/CLAUDE.md

If you want it only for a specific repository/project, put it in:

<repo-root>/CLAUDE.md

If you want it only for yourself in a specific project (without committing it to git), put it in:

<repo-root>/CLAUDE.local.md

CLAUDE.local.md is usually the best place for personal workflow preferences like Linear labels. Claude loads it after the project-level file, so your preference is read later and tends to have more influence.

For example:

## Linear Ticket Creation

When creating new Linear issues:

- Always apply the "DevOps" label.

- If multiple labels are appropriate, include "DevOps" in addition to the others.

- Verify the label exists before creating the issue.

If you're creating tickets through a custom slash command (e.g. /create-ticket) or a Linear MCP workflow, an even more reliable approach is to put the instruction directly in the command definition under:

~/.claude/commands/create-ticket.md

or in a dedicated skill, because the instruction is then attached specifically to the ticket-creation workflow rather than relying on general memory. Many Claude Code users find workflow-specific commands more reliable than large global CLAUDE.md files for things like Linear automation.

How to instruct Claude to use feature branch name from a Linear ticket when implementing it?

Add a specific workflow rule to CLAUDE.md rather than a vague instruction. Claude follows procedural instructions much more reliably.

For example:

## Linear Ticket Workflow

When implementing work from a Linear ticket:

1. Retrieve the ticket details from Linear.

2. Check whether the ticket has a suggested branch name.

3. Before making any code changes, switch to or create a git branch using the Linear branch name.

4. Do not create an alternative branch name unless the Linear branch name is unavailable.

5. If the branch already exists remotely, check it out instead of creating a new branch.

6. Confirm the branch name being used before starting implementation.

Example:

Linear ticket: DEV-123 Add deployment health checks

Branch: bojan/dev-123-add-deployment-health-checks

Commands:

git fetch origin

git checkout bojan/dev-123-add-deployment-health-checks || \

git checkout -b bojan/dev-123-add-deployment-health-checks

If you're using the Linear MCP server, I'd make it even more explicit:

## Branch Naming

For any task originating from Linear:

- Always obtain the branch name from the Linear ticket metadata.

- Never invent a branch name if Linear provides one.

- Use the Linear-provided branch name exactly as written.

- Checkout the branch before planning or implementing changes.

An even better approach is to create a dedicated command such as .claude/commands/implement-ticket.md:

Given a Linear ticket ID:

1. Load the ticket.

2. Read the ticket's branch name.

3. Checkout the branch.

4. Implement the work.

5. Commit using the ticket ID in the commit message.

Workflow-specific commands tend to be more reliable than relying on global instructions in CLAUDE.md, especially for things like branch creation, ticket labeling, and PR naming.

If I update ~/.claude/CLAUDE.md, do I need to start the new Claude session?

Usually no restart is strictly required, but it depends on how you’re using Claude:

If you’re using Claude Code / CLI (most common case)

~/.claude/CLAUDE.md is typically read when a new session starts

So:

✅ New terminal session / new Claude run → picks up changes

⚠️ Already-running interactive session → may not reliably reload it

Best practice: restart the Claude session (or re-open the CLI/chat) after changing it if you want to be sure the new rules apply.

If you’re using a long-running agent session

(e.g. an ongoing “Claude is working in the repo” process)

It often caches the instructions in memory at session start

So updates won’t consistently apply mid-task

If you’re using repo-level CLAUDE.md

Same rule: loaded at session start or project initialization

Changes apply reliably on next session or re-attach to repo

Practical rule of thumb

Changed CLAUDE.md → start a new Claude session to guarantee it applies

Minor edits while actively working → assume they won’t fully apply until restart

How to add auto-generated instructions to global context?

During Claude Code session, it created two memories but they were bound to the current project only:

Write(/Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/gh-token-unset.md)

⎿ Wrote 10 lines to /Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/gh-token-unset.md

1 ---

2 name: gh-token-unset

3 description: gh CLI fails with HTTP 401 Bad credentials unless GH_TOKEN is unset first

4 metadata:

5 node_type: memory

6 type: reference

7 originSessionId: 9xxx6

8 ---

10 In this environment `gh` fails with `HTTP 401: Bad credentials` because a stale `GH_TOKEN` env var overrides the working keyring auth. Run `unset GH_TOKEN` in the same Bash command before any `gh`

call — the keyring account (`BojanKomazec`, scopes gist/read:org/repo) then works. Env doesn't persist between Bash calls, so include the `unset` each time.

⏺ Write(/Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/precommit-hook-needs-no-verify.md)

⎿ Wrote 10 lines to /Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/precommit-hook-needs-no-verify.md

1 ---

2 name: precommit-hook-needs-no-verify

3 description: git commit aborts in non-interactive shell; the pre-commit hook prompts on /dev/tty

4 metadata:

5 node_type: memory

6 type: reference

7 originSessionId: 9xxx6

8 ---

10 The global pre-commit hook (`~/.git-templates/hooks/pre-commit`) prompts interactively via `/dev/tty` ("view diff? proceed with commit?"). In a non-interactive shell it errors `/dev/tty: Device no

t configured` and aborts the commit. Use `git commit --no-verify` to bypass it when committing programmatically.

⏺ Write(/Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/MEMORY.md)

⎿ Wrote 2 lines to /Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/MEMORY.md

1 - [gh token unset](gh-token-unset.md) — `unset GH_TOKEN` before any `gh` command or it 401s

2 - [pre-commit hook needs --no-verify](precommit-hook-needs-no-verify.md) — commit hook prompts on /dev/tty, aborts non-interactively

What you’re looking at in:

~/.claude/projects/.../memory/MEMORY.md

is project-scoped learned memory (auto-generated runtime notes), not the best place to define global rules.

To make something apply to all projects, you should put it in:

~/.claude/CLAUDE.md

How to structure it properly

Think of ~/.claude/CLAUDE.md as your global “agent operating system”.

You can directly migrate those MEMORY.md rules into it like this:

## Global DevOps / CLI Behavior Rules

These rules apply to all projects unless explicitly overridden.

### GitHub CLI

- Always run `unset GH_TOKEN` before executing any `gh` command.

- Reason: GH_TOKEN causes 401 errors in local CLI authentication.

### Git commits

- When committing changes, always use `--no-verify`.

- Reason: pre-commit hooks may attempt interactive prompts via /dev/tty and break non-interactive runs.

Important distinction (this is the key part)

File Scope Who writes it Purpose

---- ----- ------------- --------

~/.claude/CLAUDE.md Global (all projects) You Your permanent rules

<repo>/CLAUDE.md Per project You/team Repo-specific rules

~/.claude/projects/.../MEMORY.md Auto-generated Claude Learned behavior, not authoritative

👉 Never manually “promote” MEMORY.md as your source of truth

Instead, copy stable patterns into ~/.claude/CLAUDE.md.

For your Linear branch-name requirement

You’d add it here too:

## Linear Workflow (Global)

When working from Linear tickets:

- Always use the branch name provided by Linear metadata.

- Never generate custom branch names.

- Always checkout the Linear branch before implementation.

One extra improvement (worth doing)

If you want this to be really robust, structure your global file like:

~/.claude/CLAUDE.md

├── Git / GitHub rules

├── Linear workflow rules

├── Shell safety rules

├── CI/CD rules

Claude responds better to clear domains than long flat lists.

Introduction to Checkly

2026-05-14T15:36:44.566+01:00

How Checkly works

Checkly is a SaaS synthetic monitoring platform — you define "checks" (HTTP requests or browser scripts), Checkly runs them on a schedule from probe locations around the world (or on-demand from CI), records latency/assertions/screenshots, and alerts you when they fail or get slow.

Two main check types:

- API checks — a single HTTP request with assertions on status, headers, body, response time.

- Browser checks — a Playwright script run in a real headless Chromium against your deployed app.

There's also multi-step API checks (chain requests, e.g. login → use token → logout) and heartbeat checks (your job pings Checkly; alert if it stops).

Hearbeat vs Ping

Heartbeats and pings are both vital network failure-detection mechanisms, but they differ in purpose: Heartbeats are proactive, periodic "I am alive" messages sent by an application to signal it is healthy, while Pings are reactive requests to check if a server is reachable. Heartbeats detect application crashes, while pings detect network downtime.

Checks are typically authored as code (Checkly CLI, TypeScript) and checkly deploy'd to the cloud. You can tag them (tags: ["auth"]), parametrise them with env vars like ENVIRONMENT_URL, and trigger them on-demand from CI — which is exactly what this PR does with npx checkly trigger --tags=auth.

Runtime model:

- Scheduled: every N minutes from chosen regions (e.g. us-east-2, eu-west-1) — catches regressions/outages between deploys.

- Triggered from CI: post-deploy smoke test, results gate (or just annotate) the deploy.

- Alerts: Slack/PagerDuty/email on failure, with retry/degraded thresholds to avoid flap.

---

What it would check for this auth API

Given the auth API's surface (login, OAuth, JWT issuance, admin endpoints), realistic auth-tagged checks:

1. Health endpoint — basic liveness

new ApiCheck("auth-health", {

tags: ["auth"],

frequency: 1, // minute

locations: ["us-east-2", "eu-west-1"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [

AssertionBuilder.statusCode().equals(200),

AssertionBuilder.responseTime().lessThan(500),

AssertionBuilder.jsonBody("$.status").equals("ok"),

});

2. Login flow — happy path, returns a JWT

new ApiCheck("auth-login", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/auth/login`,

method: "POST",

headers: [{ key: "Content-Type", value: "application/json" }],

body: JSON.stringify({

email: process.env.SYNTHETIC_USER_EMAIL,

password: process.env.SYNTHETIC_USER_PASSWORD,

}),

assertions: [

AssertionBuilder.statusCode().equals(200),

AssertionBuilder.responseTime().lessThan(1500),

AssertionBuilder.jsonBody("$.token").isNotNull(),

// structural check on JWT shape

AssertionBuilder.jsonBody("$.token").matches("^eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+$"),

});

3. Login — wrong password returns 401 (negative path)

Catches the "accidentally accepts anything" class of regression.

new ApiCheck("auth-login-bad-pw", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/auth/login`,

method: "POST",

headers: [{ key: "Content-Type", value: "application/json" }],

body: JSON.stringify({ email: process.env.SYNTHETIC_USER_EMAIL, password: "wrong" }),

assertions: [AssertionBuilder.statusCode().equals(401)],

});

4. Multi-step — login then call protected endpoint

This is the most useful kind for an auth API, because it proves the token actually works.

new MultiStepCheck("auth-token-roundtrip", {

tags: ["auth"],

code: { entrypoint: path.join(__dirname, "token-roundtrip.spec.ts") },

});

// token-roundtrip.spec.ts

import { test, expect } from "@playwright/test";

test("login then /me", async ({ request }) => {

const login = await request.post(`${process.env.ENVIRONMENT_URL}/auth/login`, {

data: { email: process.env.SYNTHETIC_USER_EMAIL, password: process.env.SYNTHETIC_USER_PASSWORD },

});

expect(login.ok()).toBeTruthy();

const { token } = await login.json();

const me = await request.get(`${process.env.ENVIRONMENT_URL}/me`, {

headers: { Authorization: `Bearer ${token}` },

});

expect(me.status()).toBe(200);

const body = await me.json();

expect(body.email).toBe(process.env.SYNTHETIC_USER_EMAIL);

});

5. TLS & cert expiry

A pure config check — useful because cert rotation is a classic outage cause.

new ApiCheck("auth-tls", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [AssertionBuilder.statusCode().equals(200)],

// Checkly surfaces cert expiry on the run; you set a threshold per check

});

6. Browser check — full login UX

expect(login.ok()).toBeTruthy();

const { token } = await login.json();

const me = await request.get(`${process.env.ENVIRONMENT_URL}/me`, {

headers: { Authorization: `Bearer ${token}` },

});

expect(me.status()).toBe(200);

const body = await me.json();

expect(body.email).toBe(process.env.SYNTHETIC_USER_EMAIL);

});

5. TLS & cert expiry

A pure config check — useful because cert rotation is a classic outage cause.

new ApiCheck("auth-tls", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [AssertionBuilder.statusCode().equals(200)],

// Checkly surfaces cert expiry on the run; you set a threshold per check

5. TLS & cert expiry

A pure config check — useful because cert rotation is a classic outage cause.

new ApiCheck("auth-tls", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [AssertionBuilder.statusCode().equals(200)],

// Checkly surfaces cert expiry on the run; you set a threshold per check

});

new ApiCheck("auth-tls", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [AssertionBuilder.statusCode().equals(200)],

// Checkly surfaces cert expiry on the run; you set a threshold per check

});

6. Browser check — full login UX

Runs against the front-end but exercises the auth API end-to-end including redirects, cookies, CSRF.

new BrowserCheck("auth-ui-login", {

tags: ["auth"],

code: { entrypoint: path.join(__dirname, "login.spec.ts") },

});

import { test, expect } from "@playwright/test";

test("user can sign in", async ({ page }) => {

await page.goto(process.env.ENVIRONMENT_URL!);

await page.getByLabel("Email").fill(process.env.SYNTHETIC_USER_EMAIL!);

await page.getByLabel("Password").fill(process.env.SYNTHETIC_USER_PASSWORD!);

await page.getByRole("button", { name: "Sign in" }).click();

await expect(page.getByText("Dashboard")).toBeVisible({ timeout: 10_000 });

});

7. OAuth callback reachability

Doesn't fully exercise the Google/Microsoft flow (those need real consent), but checks the callback

endpoint responds correctly to a missing-code request — confirms route + handler are wired.

new ApiCheck("auth-oauth-google-callback-shape", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/auth/google/callback`,

method: "GET",

assertions: [

// 400 for missing `code`, not 404/500 — proves handler is mounted

AssertionBuilder.statusCode().equals(400),

});

Intro to QA with Headless Browsers

2026-05-14T11:14:54.403+01:00

Headless browsers are used in QA to execute automated browser tests faster and more efficiently by eliminating the graphical user interface (GUI). Because they don't render visuals, they consume fewer resources, enabling rapid, parallel testing in CI/CD pipelines, making them ideal for high-volume functional and regression testing.

Key Reasons for Using Headless Browsers in QA:

Faster Execution: Without the need to render CSS, images, or layout, tests run significantly faster.
CI/CD Integration: They are ideal for server-side environments where a GUI is unavailable, allowing automated tests to run after every code commit.
Lower Resource Usage: They consume significantly less RAM and CPU, allowing for higher parallelization (running many tests simultaneously) without overloading hardware.
Automated Functional Testing: They can accurately simulate user actions such as clicking buttons, submitting forms, and navigating pages.
Regression Testing: Due to speed and efficiency, they are perfect for running large suites of regression tests to ensure new changes haven't broken existing functionality.

Common tools for headless testing include headless Chrome, Firefox, Puppeteer, and Playwright

Headless browsers parse, compile, and execute the exact same underlying code as standard browsers, but they skip the final step of painting pixels to a physical screen.

What Headless Browsers Still Do

Construct the DOM: They parse HTML into a full Document Object Model tree.
Apply Styling: They process CSS and calculate layout, element positions, and visibility.
Execute JavaScript: They run a full JS engine (like V8 in Chrome) to handle AJAX, animations, and frontend logic.
Manage Network Traffic: They make real HTTP requests, download cookies, and handle API responses.

How QA Verifies Visuals Without a Display

Layout Queries: Code checks if elements are present, hidden, or overlapping by querying their coordinates.
Computed Styles: Scripts verify specific CSS properties, like checking if a button color is exactly rgb(0, 0, 255).
Virtual Screenshots: The browser renders the page into an in-memory buffer, allowing QA tools to save PNGs or perform pixel-by-pixel visual regression comparisons.

To help tailor using headless browser to our workflow, we need to know:

Which testing framework we are using (e.g., Playwright, Selenium, Cypress)?
Are we trying to catch functional bugs or visual layout glitches?
Do our tests run on a local machine or a CI/CD server (e.g., GitHub Actions, Jenkins)?

---

Introduction to Amazon Simple Notification Service (SNS)

2026-04-29T14:24:23.579+01:00

Amazon Simple Notification Service (SNS) is a fully managed messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Here's how SNS works:

Key Concepts

Topics:

A topic is a logical access point and communication channel. Publishers send messages to a topic, and subscribers receive these messages by subscribing to the topic.

Publishers:

Publishers are entities that send messages to an SNS topic. They could be applications, services, or even other AWS services like Lambda or CloudWatch.

Subscribers:

Subscribers are endpoints that receive messages from an SNS topic. These can include Amazon SQS queues, AWS Lambda functions, HTTP/S endpoints, email addresses, and SMS numbers.

Messages:

Messages are the payload sent by publishers to SNS topics. They can include a variety of data formats, typically JSON.

How It Works

Creating a Topic:

First, you create a topic using the AWS Management Console, AWS CLI, or AWS SDKs. This topic acts as a communication channel.

Subscribing to a Topic:

You then subscribe one or more endpoints to the topic. These endpoints can be other AWS services or external services capable of receiving notifications.

When subscribing, you specify the protocol (such as HTTP, SQS, Lambda, etc.) and the endpoint (like the URL or ARN of the SQS queue).

Publishing a Message:

Publishers send messages to the SNS topic using the Publish API. The message can include a subject, a message body, and optional attributes.

SNS stores multiple copies of the message for redundancy and high availability.

Message Delivery:

SNS distributes the message to all subscribed endpoints.

Each endpoint processes the message according to its protocol. For example, an HTTP endpoint receives a POST request with the message content, and an SQS queue receives the message as a new queue entry.

Use Cases

Fan-out Scenarios:

When a message published to an SNS topic needs to be sent to multiple endpoints, SNS acts as a fan-out service. For example, updating various microservices or notifying multiple systems about an event.

Push Notifications:

SNS can be used to send push notifications to mobile devices through services like Amazon Device Messaging (ADM), Apple Push Notification Service (APNS), and Google Cloud Messaging (GCM).

Application Alerts:

Sending notifications or alerts to administrators or users via SMS, email, or other endpoints.

Event-Driven Computing:

Integrating with AWS Lambda to trigger functions in response to incoming messages, enabling event-driven processing.

Benefits

Scalability:

SNS is designed to handle high-throughput and large volumes of messages, making it suitable for large-scale applications.

Flexibility:

It supports multiple protocols, allowing for versatile messaging architectures.

Reliability:

With built-in redundancy and fault tolerance, SNS ensures reliable message delivery.

Ease of Use:

The service is fully managed, reducing the operational overhead associated with maintaining messaging infrastructure.

Security

Access Control:

Using AWS IAM policies, you can control who can publish or subscribe to topics.

Encryption:

SNS supports server-side encryption (SSE) for messages at rest and SSL/TLS for messages in transit.

Compliance:

SNS is compliant with various industry standards and regulations, ensuring it meets security and compliance requirements.

Amazon SNS is a powerful and flexible tool for building loosely coupled, scalable, and robust messaging solutions in the cloud.

How does Amazon SNS differ from Amazon SQS?

Amazon SNS (Simple Notification Service) and Amazon SQS (Simple Queue Service) are both messaging services provided by AWS, but they serve different purposes and have distinct features. Here's a comparison highlighting their differences:

Purpose and Use Cases

Amazon SNS:

Publish/Subscribe Messaging:

SNS is designed for sending notifications or messages to multiple subscribers.
It's ideal for broadcasting messages to multiple endpoints, such as sending alerts, notifications, or updates to different systems or users.

Use Cases:

Real-time notifications
push notifications
fan-out scenarios (where a message needs to be sent to multiple recipients simultaneously)
application alerts
event-driven architectures

Amazon SQS:

Message Queuing:

SQS is designed for decoupling and scaling distributed systems.
It allows you to send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

Use Cases:

Task queues
asynchronous processing
decoupling microservices
job dispatching
buffering messages between producer and consumer systems

Messaging Patterns

Amazon SNS:

Push-Based: SNS pushes messages to subscribers. Subscribers can be other AWS services (like Lambda, SQS), HTTP/S endpoints, email addresses, SMS numbers, and mobile push notifications.
Fan-Out: One message can be sent to multiple subscribers.

Amazon SQS:

Pull-Based: Consumers pull messages from the queue. A consumer explicitly retrieves messages from the queue.
Point-to-Point: Each message is delivered to and processed by one consumer.

Message Handling

Amazon SNS:

Real-Time Delivery: Messages are delivered immediately to all subscribers.
No Message Persistence: Messages are not stored after delivery; if a subscriber is unavailable, the message is lost unless it's sent to an SQS queue or some other durable store.

Amazon SQS:

Message Persistence: Messages are stored in the queue until they are processed and deleted by a consumer, or until they expire.
Delivery Guarantees: Ensures at least once delivery. With FIFO queues, SQS provides exactly-once processing and message ordering.

Scalability and Performance

Amazon SNS:

Scalable: Designed to handle massive numbers of messages and deliver them to large numbers of subscribers.
Latency: Typically has very low latency for message delivery.

Amazon SQS:

Scalable: Automatically scales to handle large volumes of messages. Suitable for high-throughput applications.
Latency: Slightly higher latency compared to SNS due to the nature of pull-based consumption.

Features and Capabilities

Amazon SNS:

Multiple Protocols: Supports multiple delivery protocols including HTTP/S, email, SMS, SQS, Lambda, and mobile push notifications.
Filtering: Allows message filtering, enabling subscribers to receive only the messages that match their filter policies.

Amazon SQS:

Visibility Timeout: Temporarily hides a message from other consumers while it is being processed.
Dead-Letter Queues (DLQ): Allows you to handle messages that can't be processed successfully.
FIFO Queues: Ensures the order of messages and exactly-once processing.
Delay Queues: Postpones the delivery of new messages to consumers for a specified amount of time.

Pricing

Amazon SNS:

Pricing Model: Based on the number of requests (publishes, deliveries, and notifications) and data transfer.
Cost Efficiency: More cost-effective for scenarios requiring a high number of subscribers and real-time notifications.

Amazon SQS:

Pricing Model: Based on the number of requests (send, receive, delete) and data transfer.
Cost Efficiency: More cost-effective for decoupling microservices and scenarios requiring message persistence and complex message handling.

Integration and Interoperability

Amazon SNS:

Integration: Easily integrates with a wide range of AWS services (e.g., Lambda, SQS, HTTP/S endpoints, etc.).
Interoperability: Often used in conjunction with SQS for fan-out scenarios where messages need to be processed asynchronously and stored reliably.

Amazon SQS:

Integration: Commonly used to decouple systems and provide reliable message delivery. Often used with other AWS services like Lambda, ECS, and EC2.
Interoperability: Can be subscribed to SNS topics to receive messages that need persistent storage or further processing.

Summary

In summary, Amazon SNS is a pub/sub messaging service optimized for real-time notifications and broadcasting messages to multiple subscribers, while Amazon SQS is a message queuing service designed for decoupling distributed systems and ensuring reliable message delivery through persistence and processing guarantees. They are often used together to build scalable, resilient, and flexible messaging architectures in AWS.

Push Notification Service - Amazon Simple Notification Service - AWS

Provisioning AWS EKS Cluster with terraform-aws-modules/eks/aws

2026-04-21T13:07:00.002+01:00

terraform-aws-modules/terraform-aws-eks: Terraform module to create Amazon Elastic Kubernetes (EKS) resources is a popular Terraform module for provisioning AWS EKS cluster.

In this article we want to explore and breakdown its key components and their purposes.

We'd typically use this module like here:

module "eks" {

source = "terraform-aws-modules/eks/aws"

version = "21.15.1"

...

}

Let's explore this module's attributes.

1. Cluster Configuration

name, version

Sets the name and Kubernetes version for the EKS cluster. Use local and variable values for flexibility.

endpoint_public_access

Set public (Internet) access to the Kubernetes API endpoint (via kubectl). Disable it for enhanced security.

endpoint_private_access

Set private access to the API endpoint, whether only resources within the VPC can access it. If enabled, it is only reachable from within the VPC (Virtual Private Cloud) where your EKS cluster is deployed. There are few ways to access it:

How to Access the Kubernetes API from VPC

1. Use a Bastion Host or EC2 Instance in the VPC

Launch an EC2 instance (bastion host or jump box) in a subnet within the same VPC as your EKS cluster.
SSH into this instance, and from there, use kubectl to access the cluster.
Alternatively, use SSH port forwarding or a VPN to proxy kubectl commands from your local machine through the bastion.

2. Use AWS Systems Manager (SSM) Session Manager

If your EC2 instances have the SSM agent and the necessary IAM permissions, you can use AWS SSM Session Manager to start a shell session on an instance in the VPC, then run kubectl from there.

3. Use a VPN Connection

Set up a VPN (such as AWS Client VPN or OpenVPN, or Site-to-site VPN for office LAN) that connects your local network to the VPC. Once connected, your local machine will be able to reach the private endpoint.

4. Use AWS PrivateLink (Interface VPC Endpoints)

For advanced scenarios, you can use AWS PrivateLink to expose the Kubernetes API endpoint privately to other VPCs or on-premises networks.

enable_cluster_creator_admin_permissions

If enabled, grants admin permissions to the user who creates the cluster.

2. Logging and Add-ons

enabled_log_types

Enables logging for various Kubernetes components (API, audit, authenticator, controllerManager, scheduler) for monitoring and troubleshooting.

Example:

enabled_log_types = [

"api",

"audit",

"authenticator",

"controllerManager",

"scheduler"

]

addons

A dictionary-type attribute which installs and configures essential Kubernetes add-ons. Dictionary keys are addon names like:

coredns
kube-proxy
aws-ebs-csi-driver
vpc-cni

Dictionary values are objects which attributes are:

most_recent - to set using the latest version (set it to false for version pinning)
version - addon version (use it for version pinning)
before_compute - set it to true if addon should be installed and set before nodes (compute layer)
service_account_role_arn - to configure addon with IAM roles for service accounts, enabling secure integration with AWS services.

Example:

addons = {

...

vpc-cni = {

most_recent = false

version = "v1.21.1-eksbuild.7"

before_compute = true

service_account_role_arn = module.k8s_default_vpc_cni_irsa.iam_role_arn

}

...

}

VPC CNI (Container networking interface) is responsible for allocating IP addresses to the Kubernetes nodes and provides networking to pods. The plugin manage network interfaces (ENIs) on the nodes and uses it to assign IP addresses to pods.

3. Networking

We need to integrate the EKS cluster with existing VPC and subnets:

vpc_id

VPC ID

subnet_ids

Subnets in which nodes (EC2 instances) will be created.

Where your worker nodes (EC2 instances) run.

control_plane_subnet_ids

Where the EKS control plane ENIs (network interfaces) are placed

Defines where the EKS control plane creates its Elastic Network Interfaces (ENIs)

What it controls:

The EKS control plane runs in an AWS-managed VPC (you don't see it)
To communicate with your worker nodes, it creates ENIs in your VPC
These ENIs are placed in the subnets you specify here

Typical configuration:

module "eks" {

source = "terraform-aws-modules/eks/aws"

name = "my-cluster"

# Control plane ENIs go here

control_plane_subnet_ids = [

"subnet-private-1a",

"subnet-private-1b",

"subnet-private-1c"

]

}

Best practices:

Usually private subnets
Should span multiple AZs for high availability (AWS requires at least 2)
Minimum of 2 subnets, maximum of 16
Each subnet needs at least 5 available IP addresses

What these ENIs do:

Allow the control plane to communicate with worker nodes
Allow worker nodes to communicate with the API server
Handle API server endpoint traffic

security_group_additional_rules

Adds custom security group rules for the cluster, such as allowing node-to-node communication and VPN access for kubectl.

node_security_group_additional_rules

Further customizes node security groups, allowing all node-to-node traffic and all outbound traffic.

Understanding EKS Architecture

An EKS cluster has two main components:

┌─────────────────────────────────────────────────────────┐

│ EKS Cluster │

│ │

│ ┌───────────────────────────────────────┐ │

│ │ Control Plane (AWS Managed) │ │

│ │ - API Server │ │

│ │ - etcd │ │

│ │ - Scheduler │ │

│ │ - Controller Manager │ │

│ │ │ │

│ │ Runs in AWS-managed account │ │

│ └──────────────┬────────────────────────┘ │

│ │ │

│ │ ENIs in your VPC │

│ │ (control_plane_subnet_ids) │

│ ┌──────────────▼────────────────────────┐ │

│ │ Your VPC │ │

│ │ ┌─────────────────────────────┐ │ │

│ │ │ Worker Nodes (subnet_ids) │ │ │

│ │ │ - EC2 instances │ │ │

│ │ │ - Your pods run here │ │ │

│ │ └─────────────────────────────┘ │ │

│ └───────────────────────────────────────┘ │

└─────────────────────────────────────────────────────────┘

ENI: elastic network interface. It is a logical networking component in a VPC that represents a virtual network card.

4. Node Group Configuration

node_security_group_tags

Adds a tag for Karpenter (an open-source Kubernetes node autoscaler) discovery.

eks_managed_node_group_defaults

Sets default properties for all managed node groups, including:

Attaching the CNI policy for networking.
Using a specific SSH key.
Associating additional security groups.
Defining block device mappings for EBS volumes.
Attaching the AmazonSSMManagedInstanceCore policy for SSM access.

eks_managed_node_groups

Defines a default managed node group with:

A specific AMI type.
Desired, minimum, and maximum node counts.
Instance types from a variable.
On-demand capacity, EBS optimization, and disk size.
Custom labels for node identification and environment.

The gold standard for production environments is explicit pinning. This ensures that our infrastructure only changes when we decide to change the code. In order to pin AMI version used in node groups we need to set two attributes:

ami_release_version needs to be set. This prevents nodes from cycling unexpectedly during a routine deployment.
use_latest_ami_release_version needs to be set to false (without this, terraform plan will still show that it wants to upgrade AMI version, even if we've set ami_release_version)

Example:

eks_managed_node_groups = {

"${local.cluster_name}-v1_33" = {

...

ami_release_version = "1.33.8-20260224"

use_latest_ami_release_version = false

...

5. Tagging

Summary

Our configuration sets up a secure, private, and production-ready EKS cluster with managed node groups, essential add-ons, robust logging, and fine-grained network and IAM controls. It leverages best practices for security (private endpoints, IAM roles for service accounts), scalability (managed node groups, Karpenter tags), and maintainability (modular, versioned, and tagged infrastructure).

---

terraform-aws-iam/modules/iam-role-for-service-accounts-eks

Core Security Practices in DevSecOps & Software Engineering

2026-04-15T12:31:00.001+01:00

Integrating security into DevOps and software engineering, often called DevSecOps, is a critical shift from treating security as a final checkpoint to embedding it throughout the entire development lifecycle.

Here are the best security practices, with a specific focus on secrets management and key rotation.

Core Security Practices in DevSecOps & Software Engineering

1. Shift Left

This is the foundational principle of DevSecOps. It means introducing security testing and considerations as early as possible in the software development life cycle (SDLC).

Why: It is significantly cheaper and faster to fix a security flaw during the design or coding phase than it is after deployment.
Action: Conduct threat modeling during design, use secure coding standards, and run security scans on every code commit.

2. Automate Security Testing

Manual security reviews cannot keep up with the speed of DevOps. Automation is essential.

Static Application Security Testing (SAST): Scans your source code for known vulnerabilities (like SQL injection or cross-site scripting) without running the application. Tools: SonarQube, CodeQL.
Dynamic Application Security Testing (DAST): Tests the running application from the outside, mimicking an attacker to find runtime vulnerabilities. Tools: OWASP ZAP, Burp Suite.
Software Composition Analysis (SCA): Analyzes your application’s dependencies (open-source libraries) for known vulnerabilities. Tools: Snyk, Dependabot, OWASP Dependency-Check.

3. Implement the Principle of Least Privilege (PoLP)

Every user, process, and system should have only the minimum permissions necessary to perform its function.

Action:

Developers should not have administrative access to production environments.
CI/CD pipelines should use dedicated service accounts with tightly scoped permissions (e.g., a pipeline deploying to a specific AWS S3 bucket should only have s3:PutObject permissions on that bucket).
Use Role-Based Access Control (RBAC) to manage permissions.

4. Secure the CI/CD Pipeline

The pipeline itself is a high-value target for attackers. If they compromise the pipeline, they can inject malicious code into your production application.

Action:

Lock down pipeline configurations: Require code reviews for any changes to pipeline definition files (e.g., .github/workflows/*.yml).
Use code signing: Digitally sign your build artifacts (containers, binaries) to ensure their integrity and origin.
Monitor pipeline logs: Look for unauthorized changes or suspicious activity.

Best Practices for Secrets Management

"Secrets" are non-human privileged credentials, including API keys, database passwords, SSH keys, and cryptographic tokens. The golden rule of secrets management is: Never hardcode secrets in source code. If secret was hardcoded in the code and then it was moved to external secrets vault (like GitHub Secrets or AWS Secrets Manager), rotate (re-generate) that secret as the old value is present in git history!

Using GitHub Secrets

GitHub Secrets allows you to store sensitive information directly in your GitHub repository, organization, or environment. These secrets are encrypted and only exposed to GitHub Actions workflows.

Best Practices:

Scope Secrets Properly

Define secrets at the environment level (e.g., production, staging) rather than the repository level whenever possible. This ensures that only workflows deploying to production can access production secrets.

Use OIDC for Cloud Access

Instead of storing long-lived cloud credentials (like AWS Access Keys) as GitHub Secrets, use OpenID Connect (OIDC). This allows GitHub Actions to request short-lived, temporary credentials directly from your cloud provider, eliminating the need to manage long-lived secrets.

Minimize GITHUB_TOKEN Permissions

The automatic GITHUB_TOKEN provided to workflows has broad permissions by default. In your workflow YAML, explicitly set the permissions to the minimum required.

Avoid Printing Secrets to Logs

Although GitHub attempts to mask secrets in logs, avoid writing code that explicitly prints them. Be careful with debugging commands.

Using AWS Secrets Manager

For applications running on AWS, Secrets Manager provides a more robust, centralized solution for lifecycle management.

Best Practices:

Centralized Storage

Store all application secrets (database passwords, third-party API keys) in Secrets Manager, not in application configuration files or environment variables.

IAM Integration

Use IAM roles and policies to grant specific EC2 instances, Lambda functions, or ECS tasks permission to retrieve only the secrets they need. This perfectly implements Least Privilege.

Encrypt with Customer Master Keys (CMKs)

Use AWS Key Management Service (KMS) with a customer-managed key to encrypt your secrets, giving you full control over who can use the key to decrypt the secret.

Configure Automatic Rotation

This is a premier feature of AWS Secrets Manager. You can configure it to automatically rotate secrets (like database credentials) on a schedule, often without any application downtime.

Key and Secret Rotation

Rotation is the practice of regularly changing a secret or cryptographic key to limit the timeframe an attacker can use it if it is compromised.

Why Rotation is Essential

Limits Exposure Window: If a database password is stolen, but rotated every 30 days, the attacker's window of opportunity is limited.

Enforces Hygiene: Regular rotation ensures that you have a documented, tested process for changing credentials. This is vital during an active security incident.

Best Practices for Rotation

Automate Rotation: Manual rotation is error-prone and often neglected. Use tools like AWS Secrets Manager to handle the complexity.
Define Rotation Schedules based on Risk:

Critical Secrets (e.g., Production DB): Every 30-60 days.
High-Risk Secrets (e.g., Third-party API keys with write access): Every 60-90 days.
Lower-Risk Secrets: Every 90-180 days.

Use a "Database/Application First" Approach: When rotating a credential used by an application to connect to a service (like a database):

Step 1: Create a new credential version in the Secrets Manager.
Step 2: Update the service (database) to accept both the old and new credentials.
Step 3: Update the application to use the new credential.
Step 4: Verify the application is working.
Step 5: Revoke the old credential from the service (database).

Test the Rotation Process: Periodically perform manual, "out-of-band" rotations to ensure the automated process works as expected and doesn't cause outages.

How to install and setup Claude Code on MacOS + VS Code

2026-04-10T16:21:00.003+01:00

Let's follow steps from Quickstart - Claude Code Docs:

% curl -fsSL https://claude.ai/install.sh | bash

Setting up Claude Code...

✔ Claude Code successfully installed!

Version: 2.1.100

Location: ~/.local/bin/claude

Next: Run claude --help to get started

⚠ Setup notes:

• Native installation exists but ~/.local/bin is not in your PATH. Run:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc

✅ Installation complete!

Let's add path to bin to PATH, add it to zsh config and reload it:

% echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc

If you use Bash:

source ~/.bashrc

Verification:

$HOME/.local/bin is now in $PATH:

% echo $PATH

/Users/bojan/.local/bin:....

Let's check Claude version:

% claude --version

2.1.100 (Claude Code)

Let's also see its CLI arguments:

% claude --help

Usage: claude [options] [command] [prompt]

Claude Code - starts an interactive session by default, use -p/--print for non-interactive output

Arguments:

prompt Your prompt

Options:

--add-dir <directories...> Additional directories to allow tool access to

--agent <agent> Agent for the current session. Overrides the 'agent' setting.

--agents <json> JSON object defining custom agents (e.g. '{"reviewer": {"description": "Reviews code", "prompt": "You are a code

reviewer"}}')

--allow-dangerously-skip-permissions Enable bypassing all permission checks as an option, without it being enabled by default. Recommended only for

sandboxes with no internet access.

--allowedTools, --allowed-tools <tools...> Comma or space-separated list of tool names to allow (e.g. "Bash(git:*) Edit")

--append-system-prompt <prompt> Append a system prompt to the default system prompt

--bare Minimal mode: skip hooks, LSP, plugin sync, attribution, auto-memory, background prefetches, keychain reads, and

CLAUDE.md auto-discovery. Sets CLAUDE_CODE_SIMPLE=1. Anthropic auth is strictly ANTHROPIC_API_KEY or apiKeyHelper via

--settings (OAuth and keychain are never read). 3P providers (Bedrock/Vertex/Foundry) use their own credentials.

Skills still resolve via /skill-name. Explicitly provide context via: --system-prompt[-file],

--append-system-prompt[-file], --add-dir (CLAUDE.md dirs), --mcp-config, --settings, --agents, --plugin-dir.

--betas <betas...> Beta headers to include in API requests (API key users only)

--brief Enable SendUserMessage tool for agent-to-user communication

--chrome Enable Claude in Chrome integration

-c, --continue Continue the most recent conversation in the current directory

--dangerously-skip-permissions Bypass all permission checks. Recommended only for sandboxes with no internet access.

-d, --debug [filter] Enable debug mode with optional category filtering (e.g., "api,hooks" or "!1p,!file")

--debug-file <path> Write debug logs to a specific file path (implicitly enables debug mode)

--disable-slash-commands Disable all skills

--disallowedTools, --disallowed-tools <tools...> Comma or space-separated list of tool names to deny (e.g. "Bash(git:*) Edit")

--effort <level> Effort level for the current session (low, medium, high, max)

--exclude-dynamic-system-prompt-sections Move per-machine sections (cwd, env info, memory paths, git status) from the system prompt into the first user

message. Improves cross-user prompt-cache reuse. Only applies with the default system prompt (ignored with

--system-prompt). (default: false)

--fallback-model <model> Enable automatic fallback to specified model when default model is overloaded (only works with --print)

--file <specs...> File resources to download at startup. Format: file_id:relative_path (e.g., --file file_abc:doc.txt file_def:img.png)

--fork-session When resuming, create a new session ID instead of reusing the original (use with --resume or --continue)

--from-pr [value] Resume a session linked to a PR by PR number/URL, or open interactive picker with optional search term

-h, --help Display help for command

--ide Automatically connect to IDE on startup if exactly one valid IDE is available

--include-hook-events Include all hook lifecycle events in the output stream (only works with --output-format=stream-json)

--include-partial-messages Include partial message chunks as they arrive (only works with --print and --output-format=stream-json)

--input-format <format> Input format (only works with --print): "text" (default), or "stream-json" (realtime streaming input) (choices:

"text", "stream-json")

--json-schema <schema> JSON Schema for structured output validation. Example:

{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]}

--max-budget-usd <amount> Maximum dollar amount to spend on API calls (only works with --print)

--mcp-config <configs...> Load MCP servers from JSON files or strings (space-separated)

--mcp-debug [DEPRECATED. Use --debug instead] Enable MCP debug mode (shows MCP server errors)

--model <model> Model for the current session. Provide an alias for the latest model (e.g. 'sonnet' or 'opus') or a model's full name

(e.g. 'claude-sonnet-4-6').

-n, --name <name> Set a display name for this session (shown in /resume and terminal title)

--no-chrome Disable Claude in Chrome integration

--no-session-persistence Disable session persistence - sessions will not be saved to disk and cannot be resumed (only works with --print)

--output-format <format> Output format (only works with --print): "text" (default), "json" (single result), or "stream-json" (realtime

streaming) (choices: "text", "json", "stream-json")

--permission-mode <mode> Permission mode to use for the session (choices: "acceptEdits", "auto", "bypassPermissions", "default", "dontAsk",

"plan")

--plugin-dir <path> Load plugins from a directory for this session only (repeatable: --plugin-dir A --plugin-dir B) (default: [])

-p, --print Print response and exit (useful for pipes). Note: The workspace trust dialog is skipped when Claude is run with the

-p mode. Only use this flag in directories you trust.

--remote-control-session-name-prefix <prefix> Prefix for auto-generated Remote Control session names (default: hostname)

--replay-user-messages Re-emit user messages from stdin back on stdout for acknowledgment (only works with --input-format=stream-json and

--output-format=stream-json)

-r, --resume [value] Resume a conversation by session ID, or open interactive picker with optional search term

--session-id <uuid> Use a specific session ID for the conversation (must be a valid UUID)

--setting-sources <sources> Comma-separated list of setting sources to load (user, project, local).

--settings <file-or-json> Path to a settings JSON file or a JSON string to load additional settings from

--strict-mcp-config Only use MCP servers from --mcp-config, ignoring all other MCP configurations

--system-prompt <prompt> System prompt to use for the session

--tmux Create a tmux session for the worktree (requires --worktree). Uses iTerm2 native panes when available; use

--tmux=classic for traditional tmux.

--tools <tools...> Specify the list of available tools from the built-in set. Use "" to disable all tools, "default" to use all tools,

or specify tool names (e.g. "Bash,Edit,Read").

--verbose Override verbose mode setting from config

-v, --version Output the version number

-w, --worktree [name] Create a new git worktree for this session (optionally specify a name)

Commands:

agents [options] List configured agents

auth Manage authentication

auto-mode Inspect auto mode classifier configuration

doctor Check the health of your Claude Code auto-updater. Note: The workspace trust dialog is skipped and stdio servers from

.mcp.json are spawned for health checks. Only use this command in directories you trust.

install [options] [target] Install Claude Code native build. Use [target] to specify version (stable, latest, or specific version)

mcp Configure and manage MCP servers

plugin|plugins Manage Claude Code plugins

setup-token Set up a long-lived authentication token (requires Claude subscription)

update|upgrade Check for updates and install if available

And finally, let's launch it:

% claude

Welcome to Claude Code v2.1.100

…………………………………………………………………………………………………………………………………………………………

* █████▓▓░

* ███▓░ ░░

░░░░░░ ███▓░

░░░ ░░░░░░░░░░ ███▓░

░░░░░░░░░░░░░░░░░░░ * ██▓░░ ▓

░▓▓███▓▓░

* ░░░░

░░░░░░░░

░░░░░░░░░░░░░░░░

█████████ *

██▄█████▄██ *

█████████ *

…………………█ █ █ █………………………………………………………………………………………………………………

Let's get started.

Choose the text style that looks best with your terminal

To change this later, run /theme

❯ 1. Dark mode ✔

2. Light mode

3. Dark mode (colorblind-friendly)

4. Light mode (colorblind-friendly)

5. Dark mode (ANSI colors only)

6. Light mode (ANSI colors only)

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

1 function greet() {

2 - console.log("Hello, World!");

2 + console.log("Hello, Claude!");

3 }

Syntax theme: Monokai Extended (ctrl+t to disable)

After that we need to select a login method:

❯ 1. Claude account with subscription · Pro, Max, Team, or Enterprise

2. Anthropic Console account · API usage billing

3. 3rd-party platform · Amazon Bedrock, Microsoft Foundry, or Vertex AI

Option 1 - Claude Accounts are for the consumer/pro web interface (claude.ai) which is seat-based.

Option 2 - Anthropic Console account should be selected if your organization is on an API plan (pay-as-you-go billing based on token usage). Anthropic Console (platform.claude.com) is the hub for managing API keys, billing, and developer organizations.

Option 3 - 3rd-party platforms are only for when you want to route Claude's "brain" through your existing AWS (Bedrock) or Google Cloud (Vertex) bills.

After selecting Anthropic Console, you'll be taken to page which shows the following:

Claude Code would like to connect to your Anthropic organization MYORG

YOUR ACCOUNT WILL BE USED TO:

Generate API keys on your behalf
Access your Anthropic profile information
Upload files on your behalf

Logged in as user@myorg.com
Switch account

After clicking on Authorize button, you'll be redirected to a page which shows:

Build something great
You’re all set up for Claude Code.

You can now close this window.

Back in terminal, you'll see:

Logged in as user@myorg.com

After pressing Enter:

Security notes:

1. Claude can make mistakes

You should always review Claude's responses, especially when

running code.

2. Due to prompt injection risks, only use it with code you trust

For more details see:

https://code.claude.com/docs/en/security

Press Enter to continue…

After clicking on Enter:

Use Claude Code's terminal setup?

For the optimal coding experience, enable the recommended settings

for your terminal: Shift+Enter for newlines

❯ 1. Yes, use recommended settings

2. No, maybe later with /terminal-setup

Enter to confirm · Esc to skip

After choosing 1 - recommended settings:

Accessing workspace:

/Users/bojan/path/to/project

Quick safety check: Is this a project you created or one you trust? (Like your own code, a well-known open source project, or work from your team). If not, take a

moment to review what's in this folder first.

Claude Code'll be able to read, edit, and execute files here.

Security guide

❯ 1. Yes, I trust this folder

2. No, exit

Enter to confirm · Esc to cancel

After selecting 1:

╭─── Claude Code v2.1.100───────────────────────────────────────────────────────────────────────────────────────╮

│ │ Tips for getting started

│ Welcome back User! │ Run /init to create a CLAUDE.md file with instructions for Claude│

│ │ ─────────────────────────────────────────────────────────────────│

│ ▐▛███▜▌ │ Recent activity │

│ ▝▜█████▛▘ │ No recent activity │

│ ▘▘ ▝▝ │ |

│ │ │

│ Sonnet 4.6 · API Usage Billing · MYORG │

│ ~/…/path/to/project │

╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────

❯

? for shortcuts ● high · /effort

We can now run various commands, like:

────────────────────────────────────────────────────────────

❯ /stats

────────────────────────────────────────────────────────────

/stats Show your Claude Code usage statistics and activity

/status Show Claude Code status including version, model, account, API connectivity, and tool statuses

/statusline Set up Claude Code's status line UI

/ide Manage IDE integrations and show status

If we execute /stats at this point, the output will show:

❯ /stats

────────────────────────────────────────────────────────────

Status Config Usage Stats

No stats available yet. Start using Claude Code!

In my case Status tab showed, among other things:

IDE: ✘ Error installing VS Code extension: 1: Command failed with ERR_STREAM_PREMATURE_CLOSE: code --force --install-extension anthropic.claude-code

Premature close

Please restart your IDE and try again.

I restarted VS Code to no avail. I then manually installed Claude Code for VS Code plugin and restarted VD Code but the same error appeared again. There is a related bug, still with Open status: [BUG] Claude code VS Code extension error in MacOS · Issue #34639 · anthropics/claude-code

If we try /cost:

❯ /stats

⎿ Status dialog dismissed

❯ /cost

⎿ Total cost: $0.0000

Total duration (API): 0s

Total duration (wall): 1h 16m 21s

Total code changes: 0 lines added, 0 lines removed

Usage: 0 input, 0 output, 0 cache read, 0 cache write

Model Context Protocol (MCP)

2026-04-08T12:36:00.010+01:00

Model Context Protocol (MCP)

The Model Context Protocol (MCP):

Open-source standard
Enables AI models to seamlessly connect with external data sources, tools, and software systems
Acts as a universal "USB-C port" for AI, allowing LLMs to securely access local files, databases, and APIs to enhance context-aware responses.
Introduced by Anthropic in late 2024

Key Aspects of MCP:

Purpose: Replaces fragmented, custom integrations with a single, open standard, making it easier to connect AI assistants to enterprise data, tools, and development environments.
Components: Consists of:

MCP Clients
MCP Hosts - AI apps like Claude or coding agents
MCP Servers - programs that bridge specific data sources

Security: MCP supports secure, two-way connections, allowing developers to control exactly what data is exposed to the AI.
Functionality: Enables models to read files, query databases, use search engines, and call external APIs, providing live, relevant context for tasks.
Open Standard: Hosted by the Linux Foundation, the protocol is designed for broad industry adoption.

MCP differs from RAG (Retrieval-Augmented Generation) by focusing on active, two-way interaction with systems, whereas RAG is focused on retrieving documents for context.

For developers, it provides SDKs in Python and TypeScript.

MCP clients

MCP clients are the components within AI applications (AI Hosts) that manage one-to-one connections with MCP servers, translating AI requests into protocol-standardized messages.

Popular MCP Client Applications

Several major AI-powered platforms and editors have integrated MCP client support to allow users to pull in their own tools and context:

Claude Desktop: Anthropic’s flagship app provides a built-in interface for managing local and remote MCP servers (e.g., Google Drive, Slack, GitHub).
Cursor: An AI-native code editor that uses MCP to give its internal AI models direct access to project files, local databases, and custom developer tools.
Windsurf Editor: A developer environment that supports tool invocation through MCP servers, allowing it to seamlessly interact with external scripts and APIs during coding sessions.
Visual Studio Code (Agent Mode): Developers can use extensions to register MCP servers, enabling chat assistants to interact with internal enterprise tools directly within the editor.
JetBrains IDEs: Platforms like IntelliJ IDEA feature an MCP-client UI where users can paste server configurations to bring external tool catalogues into the AI Assistant pane.
BeeAI: An open-source desktop AI assistant from IBM that supports tool integration via built-in or custom MCP servers.

Core Client Features

In the MCP architecture, clients don't just consume data; they provide specific features that enable complex, "agentic" workflows:

Sampling: Allows a server to request that the client (and its LLM) generate a completion, enabling the AI to "ask back" for clarification or more information.
Elicitation: Provides a structured way for servers to ask users for specific information (like a password or preference) through the client’s UI.
Roots: Allows the client to define specific file system boundaries, telling servers which directories they are permitted to access for safety and context scoping.

Developer SDKs for Building Clients

If you are building your own AI application, you can implement an MCP client using official software development kits:

TypeScript/Node.js SDK: The most comprehensive SDK for building web-based or local Node.js clients using standard JSON-RPC 2.0.
Python SDK: Ideal for data science or automation workflows, allowing easy integration with Python-based AI agents.

MCP Hosts

Model Context Protocol (MCP) hosts are AI applications or environments that manage, connect, and run MCP clients to interact with external data sources (servers). Key examples include Claude Desktop, Claude Code, Zed, Replit, Cursor, and Sourcegraph.

Key Roles of MCP Hosts:

Orchestration: Act as the container for AI models to coordinate and communicate with multiple MCP servers.
Security & Permissions: Manage user consent and enforce security policies (e.g., in sandboxed iframes) to control what data servers can access.
Workflow Management: Aggregate user prompts, conversation state, and tool results from various sources to enable seamless interaction.
Session Control: Initialize, maintain, and terminate connections between LLMs and tools via the MCP client.

Primary MCP Host Examples:

Claude Desktop: Anthropic's desktop application is a primary implementation of an MCP host, allowing the Claude model to utilize local tools and data.
Claude Code: An AI agentic coding tool that acts as a host to interact with development-focused MCP servers.
AI IDEs (Zed, Cursor, Replit): Integrated Development Environments that use MCP to connect coding assistants with files, documentation, and databases.
Platforms & Agents: Other platforms like Workato or custom agents that orchestrate data access via the protocol.

What's the difference between mcp host and mcp client?

In the Model Context Protocol (MCP), the Host is the overall application you interact with, while the Client is the specific component inside that application that handles the technical connection to a server.

Key Differences at a Glance

What it is

MCP Host: The end-user application (e.g., Claude Desktop, Cursor).
MCP Client: A protocol handler or "adapter" inside the host.

Primary Role

MCP Host: Manages the user interface and coordinates multiple clients.
MCP Client: Maintains a 1:1 connection with a single MCP server.

Responsibility

MCP Host: Security policies, user consent, and aggregating data for the AI model
MCP Client: Translating protocol messages (JSON-RPC) between the host and server.

Hierarchy

MCP Host: A single Host can contain multiple Clients.
MCP Client: A Client is a subsidiary of the Host.

The "Restaurant" Analogy

To make it simpler, imagine a restaurant setting:

The Host is the Executive Chef: They decide what needs to be cooked and oversee everything, but they don't leave the kitchen to buy ingredients.
The Client is the Waiter: They take the Chef's specific order, run to the source (the Server), and bring back exactly what was requested in a format the Chef can use.

Why the distinction matters

While you will often hear people refer to applications like Claude Desktop as "the client," technically they are hosts. This architecture allows one app to connect to many different data sources (like Google Drive, Slack, and local files) simultaneously by instantiating a separate client for each one.

What is a MCP Client in Claude Desktop?

In Claude Desktop, the MCP client is the internal software layer that allows the app to "talk" to the tools you've added. While you might call the whole app "the client," it actually functions as a host that manages multiple individual client connections.

How it works in Claude Desktop

The Translator: When you ask Claude to "read a file," the client translates that human request into a technical JSON-RPC message that the Filesystem server understands.
The Connection Manager: Claude Desktop can run several clients at once. For example, one client might be connected to a GitHub server while another is connected to a Google Drive server.
Permission Gatekeeper: The client facilitates the security handshake. Before a tool executes, the client triggers the UI popup in Claude Desktop asking for your explicit permission.

How to see them

You can see your active MCP clients and their available tools by clicking the "hammer" or "plug" icon (the MCP server indicator) in the bottom-right corner of the chat input box.

Configuration

Claude Desktop's clients are configured via a local JSON file (claude_desktop_config.json). This file tells the internal clients exactly how to launch and communicate with your servers.

Config File Location:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

MCP Servers

Model Context Protocol (MCP) servers:

Lightweight programs that connect AI models (like Claude or ChatGPT) to external data sources and tools, such as local files, databases, GitHub, or Slack
Provide a standardized interface, enabling AI agents to securely access, read, and manipulate data beyond their training sets.

Key Aspects of MCP Servers:

Functionality: They expose specific capabilities—resources, prompts, and tools—to AI applications.
Use Cases: Common implementations include file system access for documentation, database querying, and API interactions for services like GitHub or Google Tasks.
Security: They provide controlled, authorized access to local or remote resources, with user permission required for actions.
Architecture: As part of the Model Context Protocol, they act as the "server" in a client-server model, connecting to "hosts" like desktop apps.

Common MCP Server Examples:

Local File System: Allows AI to read, write, and organize local documents.
GitHub/GitLab: Enables AI to manage repositories, create issues, and pull code.
Database/API Connectors: Connects AI to SQL databases, HubSpot CRM, or AWS services.
Developer Tools: Includes servers for Terraform, Angular CLI, and Home Assistant.

You can build your own MCP server using Python or TypeScript, often utilizing tools like uv for environment setup.

How to start using Agentic AI in DevOps and Platform Engineering

2026-04-08T12:18:00.001+01:00

The next frontier of DevOps and Platform Engineering is Agentic AI. We need to learn how autonomous agents reason and adapt to reduce cognitive load and accelerate the SDLC as we want to move beyond simple automation to build self-optimizing ecosystems that scale with confidence, innovation, and enterprise governance. 

We should be able to:

Explain the shift from automation to agentic AI and articulate what makes an AI system truly
“agentic”
Design agent-aware workflows in GitHub Actions, integrating LLMs with events, logs, APIs, and quality gates to create intelligent CI/CD pipelines
Build AI-powered diagnostic loops that ingest failure context, reason about root causes, and generate structured remediation proposals or self-healing fixes
Implement intelligent release decisions using multi-signal quality gates (test coverage, performance, security, cost) and generate auditable release rationale reports
Deploy our own end-to-end platform engineering agent, capable of diagnosing pipeline failures, evaluating release readiness, and autonomously opening a fix PR or escalating with structured context.

Learning while Doing

Identify Platform engineering pain points and the AI opportunity

How can we get from static scripts and CI/CD automation to agentic AI
Make a comparison of manual vs. AI-driven diagnosis
Understand how platform engineering is evolving from static automation toward AI-driven systems that proactively diagnose and resolve operational issues

Agentic AI fundamentals - how agents reason and act?

Learn about core agent components (LLMs, memory, and tools)
Compare event-driven vs. polling architectures
Balance autonomous actions with human oversight
Understand how agentic systems combine reasoning, memory, and tools to perceive events, make decisions, and act within engineering workflows

How to setup the environment and create our first agentic workflow

Set up an agentic runtime that responds to CI/CD events
Connect an AI agent to our pipeline's event stream and context
Trigger our first agent run and interpret its reasoning logs
Learn how to connect AI agents to CI/CD events and platform context to trigger automated reasoning and actions in real time

AI-powered diagnosis and remediation

Compare manual vs. AI-driven incident diagnosis
Build agents that read logs, reason about failures, and propose fixes
Define escalation boundaries: when the agent self-heals vs. asks a human
Understand how AI agents analyze logs, diagnose failures, and determine whether to self-heal or escalate issues to humans

Intelligent CI/CD & adaptive delivery

How to move beyond pass/fail pipelines to AI-driven release decision
Automate rollback decisions using AI quality gates
Query pipeline state and release history using natural language
How AI transforms CI/CD pipelines into adaptive systems that make context-aware release and rollback decisions

Operational intelligence & conversational observability

Replace complex dashboards with AI anomaly detection
Check platform health via chat interfaces
Shift from reactive alerts to predictive management
Understand how AI enables conversational access to platform health and detects anomalies to support proactive operations.

Multi-agent coordination & implementation strategy

Architect multi-agent systems for our platform workflows
Handle agent conflicts, failures, and graceful degradation
Design a phased enterprise rollout with guardrails and audit trails

Build our platform engineering agent

Learn how to design coordinated multi-agent systems that handle complex platform workflows with governance and reliability
Wire together diagnosis, quality gates, and observability into one agent pipeline
Implement self-healing PRs with confidence thresholds
Shift our role from platform operator to AI supervisor
Learn how to combine diagnosis, delivery intelligence, and observability into a unified agent that automates key platform workflows

---

Kubernetes StatefulSet

2026-04-02T11:08:00.006+01:00

In Kubernetes, a StatefulSet is a specialized workload API object designed to manage stateful applications. Unlike standard Deployments, where Pods are interchangeable "cattle," StatefulSets treat Pods as unique "pets" with a persistent identity that is maintained even if they are rescheduled or restarted.

Key Features

Stable Network Identity: Each Pod is assigned a unique, ordinal index (e.g., web-0, web-1) and a corresponding stable DNS name through a Headless Service.
Stable Storage: By using volumeClaimTemplates, each Pod is automatically paired with its own PersistentVolume. If a Pod dies, the replacement Pod with the same identity will automatically remount the same storage.
Ordered Deployment: Pods are created and scaled sequentially from 0 to N-1. Kubernetes ensures that the previous Pod is "Running and Ready" before starting the next one.
Ordered Termination: Scaling down or deleting the StatefulSet occurs in reverse order, starting from the highest ordinal (e.g., web-2 is deleted before web-1).

When to Use StatefulSets

StatefulSets are the standard choice for applications that require consistent data and unique identities, such as:

Databases: Systems like MySQL, PostgreSQL, MongoDB, and Cassandra.
Distributed Systems: Tools like ZooKeeper, Kafka, and Elasticsearch that need a quorum or master election.
Clustered Applications: Any software where instances need to know each other’s specific addresses to sync data.

Comparison: StatefulSet vs. Deployment

Feature StatefulSet Deployment

---------- -------------- ---------------

Pod Identity: Unique and stable (ordinal names) Randomly generated and ephemeral

Storage: Dedicated volume per Pod (via template) Typically shared or transient

Network: Fixed DNS per Pod (via Headless Service) Single Load Balancer for the whole set

Scaling: Sequential (0, then 1, then 2...) Parallel (multiple Pods at once)

Best Practices

Use Headless Services: Always pair our StatefulSet with a Service that has clusterIP: None to ensure Pods are individually addressable.
Persistent Storage: Ensure our StorageClass is correctly configured for dynamic provisioning so that each Pod gets its own disk automatically.
Manual Data Sync: Note that while Kubernetes manages the infrastructure, we are still responsible for configuring internal application logic like data replication or master/slave sync.

YAML manifest example for a basic MySQL StatefulSet

Below is a standard YAML manifest for a MySQL StatefulSet. It includes a Headless Service for network identity and a volumeClaimTemplate to automatically provision unique storage for each replica.

apiVersion: v1

kind: Service

metadata:

labels:

app: mysql

spec:

ports:

- port: 3306

clusterIP: None # Defines this as a Headless Service

selector:

app: mysql

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

spec:

selector:

matchLabels:

app: mysql

serviceName: "mysql"

replicas: 3

template:

metadata:

labels:

app: mysql

spec:

containers:

- name: mysql

image: mysql:8.0

env:

- name: MYSQL_ROOT_PASSWORD

value: "password" # Use Secrets in production!

ports:

- containerPort: 3306

volumeMounts:

- name: mysql-data

mountPath: /var/lib/mysql

volumeClaimTemplates:

- metadata:

spec:

accessModes: [ "ReadWriteOnce" ]

resources:

requests:

storage: 1Gi

Why this works:

Stable DNS: Each Pod gets a predictable name: mysql-0.mysql, mysql-1.mysql, etc.
Unique Storage: Kubernetes creates three separate PersistentVolumeClaims. mysql-0 will always mount the first disk, even after a reboot.
Ordered Startup: Pods launch one after another (0, then 1, then 2), which is critical for forming database clusters.

To use a Kubernetes Secret (like mysql-secret) instead of hardcoding passwords, we need to create a Secret object and then reference it in our StatefulSet. This is the standard practice for distributing credentials securely in Kubernetes.

1. Create the Secret

We can define our password in a YAML file. Note that values in the data field must be base64 encoded.

apiVersion: v1

kind: Secret

metadata:

type: Opaque

data:

# 'password' encoded in base64 is 'cGFzc3dvcmQ='

root-password: cGFzc3dvcmQ=

Alternatively, we can use stringData to provide the password in plain text; Kubernetes will handle the encoding for us when we apply it:

apiVersion: v1

kind: Secret

metadata:

type: Opaque

stringData:

root-password: "our-secure-password"

2. Update the StatefulSet

Modify the env section of our MySQL container to use valueFrom and secretKeyRef. This tells the Pod to pull the value of MYSQL_ROOT_PASSWORD from the secret we just created.

containers:

- name: mysql

image: mysql:8.0

env:

- name: MYSQL_ROOT_PASSWORD

valueFrom:

secretKeyRef:

key: root-password # The specific key inside the Secret

Key Considerations

Initialization Only: For MySQL, the MYSQL_ROOT_PASSWORD environment variable is typically only used during the first-time initialization of the data directory. Changing the Secret later will not automatically update the root password in an existing database.
Security: Ensure our cluster has encryption at rest enabled for Secrets to truly protect sensitive data.
Alternative for Multiple Variables: If we have many credentials (user, password, DB name), we can use envFrom to map all keys in a Secret to environment variables at once.

Changing Storage Spec

Changing spec.volumeClaimTemplate updates the StatefulSet template but will not resize already-created PVCs. If the goal is to fix an existing CrashLoopBackOff due to disk-full, we still need to expand the current PVC(s) (and ensure the general StorageClass allows volume expansion), or recreate the PVC/StatefulSet so the new size takes effect.

References:

StatefulSets | Kubernetes

Amazon EBS CSI Driver

2026-03-19T13:56:56.248+00:00

The Amazon EBS CSI Driver is a standard interface that allows Amazon Elastic Kubernetes Service (EKS) clusters to manage the full lifecycle of Amazon EBS volumes as persistent storage for containers. It replaces the older, deprecated "in-tree" Kubernetes storage plugin with a more flexible, decoupled model.

Key Features

Dynamic Provisioning: Automatically creates and attaches EBS volumes when a PersistentVolumeClaim (PVC) is made.
Volume Lifecycle Management: Handles the creation, attachment, mounting, and deletion of volumes.
Resizing & Snapshots: Supports online volume resizing (for gp3 and others) and taking volume snapshots for data backup.
EKS Auto Mode Support: In EKS Auto Mode, routine block storage tasks are automated, and you don't even need to manually install the driver.

Deployment Methods

You can install and manage the driver through several channels:

EKS Managed Add-on (Recommended): Simplifies installation and updates via the AWS Console, CLI, or Terraform.
Helm Chart: Provides highly customizable installation options.
Kustomize: Direct deployment using manifests from the official GitHub repository.

Core Requirements

IAM Permissions: The driver requires an IAM role with the AmazonEBSCSIDriverPolicy to interact with EBS resources.
Service Accounts: Typically uses IAM Roles for Service Accounts (IRSA) to securely provide AWS credentials to the driver pods.
Compatibility: Supports Linux and Windows worker nodes, as well as ARM64 architectures.

Driver Components

The driver is typically deployed into the kube-system namespace and consists of two main parts:

Controller Deployment: Runs as a set of replicas (ebs-csi-controller) to communicate with the AWS EC2 API and manage volume operations.
Node DaemonSet: Runs on every worker node (ebs-csi-node) to handle the actual mounting and unmounting of volumes to pods on that specific host.

In the Amazon EBS CSI driver architecture, the ebs-csi-controller and ebs-csi-node are the two primary components that work together to manage the lifecycle of EBS volumes in a Kubernetes cluster.

Core Feature Differences

ebs-csi-controller

Deployment Type:

Deployment (typically 2 replicas for HA)

Main Function:

Control Plane operations: Creating, deleting, attaching, and detaching volumes

AWS Interaction:

Calls the AWS EC2 API to manage EBS resources

IAM Permissions:

Requires an IAM role with permissions like ec2:CreateVolume and ec2:AttachVolume

ebs-csi-node

Deployment Type:

DaemonSet (runs on every worker node)

Main Function:

Node-level operations: Mounting and unmounting volumes to the local filesystem

AWS Interaction:

Interacts with the local OS (privileged system calls) to handle block devices

IAM Permissions:

Generally requires fewer/no AWS API permissions, as it mostly performs local mount actions

How They Work Together

Provisioning & Attachment: When you create a PersistentVolumeClaim (PVC), the ebs-csi-controller watches the request and calls the AWS API to create the EBS volume and attach it to the correct EC2 instance.
Mounting: Once the volume is physically attached to the EC2 instance, the ebs-csi-node pod running on that specific node detects the new block device and mounts it into the container’s path so your application can use it.

Key Considerations

Security: For better security, you can schedule the ebs-csi-controller on hardened management nodes, while the ebs-csi-node must run everywhere your workloads need storage.
Fargate: You can run the controller on Fargate nodes, but the ebs-csi-node (as a DaemonSet) only runs on EC2 instances.
Troubleshooting: If a volume fails to "attach," check the controller logs; if it fails to "mount" or "format," check the node logs.

Pods for both the ebs-csi-controller and ebs-csi-node typically share the same value for the app.kubernetes.io/name label.

In standard deployments (such as via the official Helm chart or EKS add-on), both components use this label to identify that they belong to the same overarching application: the Amazon EBS CSI Driver.

Label Comparisons

While they share the same application name, they use the app.kubernetes.io/component label to distinguish between their specific roles.

Label Key ebs-csi-controller Pods ebs-csi-node Pods

------------ ------------------------------ ----------------------

app.kubernetes.io/name aws-ebs-csi-driver aws-ebs-csi-driver

app.kubernetes.io/instance aws-ebs-csi-driver aws-ebs-csi-driver

app.kubernetes.io/component csi-driver (or controller) csi-driver (or node)

app (Legacy label) ebs-csi-controller ebs-csi-node

How to Verify in Your Cluster

You can check these labels yourself using kubectl. This is useful if you are writing Prometheus rules or network policies that need to target the entire driver or just one part of it.

To see labels for all EBS CSI pods:

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver --show-labels

To target only the controller:

kubectl get pods -n kube-system -l app=ebs-csi-controller

---

Monitoring and Observability

2026-03-19T10:37:52.290+00:00

Monitoring vs Observability

In the world of IT and DevOps, monitoring and observability are two related but distinct concepts used to manage system health and performance.

Core Difference

The simplest way to distinguish them is:

Monitoring tells you what is happening (and when). It is reactive and focuses on known problems using predefined metrics.
Observability tells you why it is happening. It is proactive and uses the system's outputs to understand its internal state, especially for "unknown unknowns".

Key Comparison Table

Feature Monitoring Observability

---------- --------------- -----------------

Purpose Detect known issues Diagnose root causes

Perspective External (symptoms) Internal (system state)

Question "Is the system healthy?" "Why is it behaving this way?"

Approach Reactive Proactive

Focus "Known knowns" "Unknown unknowns"

Data Types Metrics, logs Metrics, logs, and traces

Analogy: The Car

Monitoring is your dashboard. It has dials for speed and fuel, and a "check engine" light. It tells you if you are speeding or if something is broken.
Observability is the mechanic’s diagnostic tool. When the "check engine" light comes on, the mechanic plugs in a tool to see exactly which sensor failed and why, without having to take the entire engine apart.

Common Tools

Monitoring Tools: Nagios, Zabbix, Prometheus.
Observability Platforms: Datadog, New Relic, Honeycomb, Dynatrace.

Three Pillars of Observability

The three pillars of observability—metrics, logs, and traces—are essential telemetry data types used to understand the internal state of complex, distributed systems. They enable teams to detect, investigate, and resolve performance issues by providing high-level trends, granular event details, and full request-flow paths.

Metrics

Quantitative measurements over time (e.g., CPU usage, error rates).

Numerical measurements that describe the health, performance, and behavior of a system over time (e.g., CPU usage, error rates, throughput). They are ideal for alerting, capacity planning, and spotting trends or symptoms.

Logs

Granular, timestamped records of discrete events.

Timestamped, granular records of discrete events. They provide the detailed context (text or structured data) necessary to understand exactly what happened within an application or service.

Traces

End-to-end journeys of a single request through a distributed system, showing how different

Records showing the journey of a single request as it travels through a distributed system, encompassing multiple services. They are critical for pinpointing bottlenecks, latency, or failures in microservices architectures.

Why They Are Used Together

While metrics indicate that a problem exists, logs provide the context of why it happened, and traces show where it is occurring. Correlating these three data types provides actionable insights rather than just raw data.

Are Logs concern of Monitoring or Observabilty?

Both monitoring and observability deal with logs, but they do so in fundamentally different ways, representing a shift from simply knowing something is broken to understanding why.

Monitoring is generally used to detect known issues using logs. It is reactive and focuses on pre-defined metrics or alert thresholds, such as alerting when error logs spike or when a specific error code appears.

Observability is used to investigate and understand the "why" behind issues by exploring logs, metrics, and traces together. It is proactive, allowing you to debug complex, distributed systems without needing to know every question ahead of time.

Comparison: Logs in Monitoring vs. Observability

Feature Log Monitoring Log Observability

---------- --------------------- -------------------------

Primary Question What went wrong? Why did it go wrong?

Approach Reactive: Alerts when logs meet criteria Proactive: Explores data to find root causes

Log Handling Searchable, indexed logs for active alerts Contextualized, correlated logs (with traces)

Data Usage Simple monitoring and basic dashboards Deep, ad-hoc, and exploratory analysis

Typical Usage "Error rate > 5%" "Why did this transaction fail?"

How They Work Together

Logs are one of the "three pillars" of observability—alongside metrics and traces—that provide the detailed, granular context necessary for troubleshooting, notes Grafana.

Monitoring tells you the system is unhealthy (e.g., an alert fires because of high error rates in log files).

Observability allows you to use tools like Splunk or Datadog to dive into the logs and traces to find the specific line of code or database failure causing the issue.

In short, monitoring is a component of observability—you cannot have true observability without comprehensive logging.

Monitoring vs Observability on the example of AWS Lambda

For AWS Lambda, monitoring identifies what is wrong (e.g., an execution failed), while observability reveals why it happened by connecting logs, metrics, and traces across your entire serverless architecture.

Monitoring AWS Lambda: Detecting the Known

Monitoring focuses on pre-defined health indicators. You use it to track "known-knowns" and trigger reactive alerts when thresholds are breached.

Key Tool: Amazon CloudWatch collects standard metrics automatically.
Monitored Metrics:

Invocations: The total number of times your function runs.
Errors: The count of failed executions.
Duration: How long your function takes to run.
Throttles: Occurrences where invocations are blocked due to concurrency limits.

Example Scenario: You set a CloudWatch Alarm to notify you if your Lambda's error rate exceeds 5%. Monitoring tells you there is a problem, but not the specific line of code that caused it.

Observability in AWS Lambda: Investigating the Unknown

Observability is a property of the system that allows you to understand its internal state from external outputs. It uses high-cardinality telemetry to investigate complex, distributed issues.

Key Tool: AWS X-Ray provides distributed tracing to visualize the request path across multiple services.
Key Elements:

Distributed Traces: Seeing a "waterfall" view of a request as it moves from API Gateway to Lambda, then to DynamoDB.
Log Insights: Using CloudWatch Logs Insights to run ad-hoc queries across massive log volumes to find specific patterns.
Enhanced Instrumentation: Using libraries like AWS Lambda Powertools to add structured logging and context to your telemetry.

Example Scenario: You notice high latency in a specific user's request. Using X-Ray, you see that the Lambda function itself is fast, but it is waiting 2 seconds for a downstream third-party API call. Observability provided the "why".

Comparison Summary for Lambda

Feature Monitoring (CloudWatch) Observability (X-Ray + Logs + Metrics)

---------- --------------------------------- ---------------------------------------------------

Goal Track health against thresholds. Understand root causes and behavior.

Questions "Is my function failing?" "Why is this specific request slow?"

Visibility Isolated metrics for one function. Request paths across multiple services.

Action Reactive (Alarms/Notifications). Proactive (Debugging/Optimisation).

---

How to architecture highly available and fault tolerant AWS EKS Kubernetes cluster

2026-03-16T17:29:00.001+00:00

We should follow a layered approach—from the network up to the application.

The "Highly Available EKS" Design Framework

1. Networking & Control Plane (The Foundation)

Multi-AZ VPC:

Design a VPC with at least 3 Private Subnets across 3 different Availability Zones (AZs).

EKS Endpoint (API server endpoint) Access:

Enable "Private Access" for the EKS control plane so worker nodes communicate with API sever within the VPC, reducing exposure and latency.

NAT Gateways:

Use one NAT Gateway per AZ (3 total) to ensure that if one AZ fails, the nodes in other zones still have outbound internet access for image pulls. This means that we need to have 3 public subnets, one per each AZ.

To use a NAT Gateway, you must place it in a public subnet (a subnet with a route to an Internet Gateway). If your goal is to have one NAT Gateway per Availability Zone (AZ) for high availability, you need a corresponding public subnet in each of those three AZs to host them.

Why this structure is necessary

The architecture follows a specific "dependency chain" to ensure that an issue in one data center doesn't take down your entire outbound connectivity:

AZ Independence: NAT Gateways are zone-redundant by design, but they physically reside in a specific AZ. If AZ-a goes down, the NAT Gateway inside it goes down too.
The Public Subnet Requirement: A NAT Gateway needs a Public IP (EIP) and a route to the Internet Gateway (IGW). Only subnets configured as "public" can provide this.
Cross-Zone Resilience: By having three public subnets (one in each AZ), you can place a NAT Gateway in each. Then, you point the private subnets in AZ-a to the NAT Gateway in AZ-a, the private subnets in AZ-b to the one in AZ-b, and so on.

The Standard Setup

If you are following the recommendation for a 3-AZ deployment, your VPC structure will typically look like this:

Component AZ-1 AZ-2 AZ-3

----------------- ---------------------- --------------------- ---------------------

Public Subnet Subnet-Pub-1 Subnet-Pub-2 Subnet-Pub-3

NAT Gateway NAT-GW-1 NAT-GW-2 NAT-GW-3

Private Subnet Nodes/Workloads Nodes/Workloads Nodes/Workloads

Note on Cost:

While this is the "Gold Standard" for reliability (preventing "cross-zone data charges" and ensuring 100% uptime during an AZ failure), keep in mind that AWS charges per hour for each NAT Gateway. Running three of them is significantly more expensive than running one.

A Common Misconception

You could technically have 3 private subnets and only 1 public subnet (with 1 NAT Gateway). In that case, all nodes in all 3 AZs would send their traffic to that single NAT Gateway.

The Risk: If the AZ containing that lone NAT Gateway fails, your nodes in the other two healthy AZs will lose their ability to pull images or talk to the internet, effectively "breaking" your cluster even though the nodes themselves are fine.

2. Compute & Data Plane (The Muscle)

Managed Node Groups:

Node groups implement basic compute scaling through EC2 Auto Scaling groups.
Use EKS Managed Node Groups spread across those 3 subnets. Select multiple subnets for a node group to provision nodes across multiple Availability Zones.
Amazon EKS managed node groups make it easy to provision compute capacity for your cluster. managed node groups consist of one or more Amazon EC2 instances running the latest EKS-optimized AMIs. All nodes are provisioned as part of an Amazon EC2 Auto Scaling group that is managed for you by Amazon EKS and all resources including EC2 instances and autoscaling groups run within your AWS account.

Auto Scaling Groups (ASG):

Set the min_size to 3. This ensures that even if a node fails, the ASG replaces it immediately.
This is NOT Kubernetes Cluster Autoscaler (or Karpenter), which, if we want to use it, needs to be installed separately.

CAS and ASG both control same node groups and can get in conflict. Solution: disable scaling policies on the ASG so CAS takes control
Karpenter and ASG might get in conflict only if they control the same nodes. But in a well-architected EKS cluster, we usually have two different "families" of nodes:

static: for small, fixed-size Managed Node Group To run "System" pods (CoreDNS, CNI, Karpenter itself)
dynamic: Karpenter-managed node group which runs actual workload, with dynamic number of nodes, depending on the current usage

Instance Diversity:

Mention using multiple instance types (e.g., m5.large and m6g.large) to avoid "insufficient capacity" errors in a specific AWS zone.

3. Traffic Management (The Entry Point)

AWS Load Balancer Controller:

Use the aws-load-balancer-controller to provision an Application Load Balancer (ALB).

Cross-Zone Load Balancing:

Ensure this is enabled so the ALB can route traffic to healthy pods in any AZ, even if the "local" node in its own zone is struggling.

4. Pod-Level Availability (The Brains)

Pod Anti-Affinity:

To ensure replicas don't land on the same node.

Topology Spread Constraints:

To force an equal distribution of pods across the 3 AZs.

Pod Disruption Budgets (PDB):

To prevent the Cluster Autoscaler or AWS maintenance from taking down too many replicas at once.

3 Pillars of High Availability

Component: Multi-AZ ASG

Layer: Infrastructure
Goal: Survive an entire AWS Data Center outage.

Component: PDBs & Rollouts

Layer: Orchestration
Goal: Survive maintenance and human error (updates).

Component: Spread Constraints

Layer: Application
Goal: Survive individual EC2 instance crashes.

---

Application Pod Disruption Budget (PDB) in Kubernetes

2026-03-09T18:24:00.132+00:00

A Pod Disruption Budget (PDB) in Kubernetes:

It's an API object
Sets for an application deployed in cluster:

Either

Minimum number of pods an application needs to function smoothly...

Maximum number of replicated pods that are down simultaneously...

...during voluntary disruptions (e.g., node upgrades, maintenance, draining)

Ensures high availability by guaranteeing a minimum number or percentage of pods remain active.

Key Aspects of PDBs:

In general, disruptions can be:

voluntary, such as maintenance operations or node scaling, or
involuntary, such as hardware failures or system crashes

Voluntary Focus: PDBs only protect against voluntary disruptions, such as kubectl drain or node repairs, not against involuntary, unavoidable failures.

Configuration: You define a PDB using either

either minAvailable

minimum pods that must run

or maxUnavailable

maximum pods allowed to be down
PDB configuration setting defining the maximum number of pods that can be voluntarily taken down simultaneously.

Use Case: Ideal for quorum-based applications (e.g., Elasticsearch, Zookeeper) to ensure quorum is never lost during node maintenance.

Mechanism: When a cluster administrator drains a node, the system checks the PDB. If removing a pod violates the budget, the action is delayed until enough replicas are available elsewhere.

Example PDB Configuration:

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

minAvailable: 2 # At least 2 pods must remain running

selector:

matchLabels:

app: web-app

Name is usually set to refer to the application for which PDB is created ("web" in the example above).

Best Practice:

Use PDBs in conjunction with pod anti-affinity rules to ensure pods are spread across nodes.

How to check PDB in cluster?

Example:

% kubectl get pdb -A

NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE

clickhouse chi-clickhouse-ch N/A 1 1 132d

kube-system ws-cluster-autoscaler N/A 1 1 133d

kube-system coredns N/A 1 1 140d

kube-system ebs-csi-controller N/A 1 1 140d

kube-system karpenter N/A 1 1 139d

ALLOWED DISRUPTIONS:

dynamic value, constantly re-calculated by the disruption controller
the real-time status indicator
showing how many pods can currently be evicted without violating the set maxUnavailable or minAvailable constraints
The non-zero value means that the disruption controller has seen the pods, counted the matching pods, and updated the status of the PDB

To see the number of current and desired healthy pods (and how ALLOWED DISRUPTIONS is actually calculated) run this command (in the example below it was for karpenter application):

% kubectl get poddisruptionbudgets karpenter -n kube-system -o yaml

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

annotations:

meta.helm.sh/release-name: karpenter

meta.helm.sh/release-namespace: kube-system

creationTimestamp: "2025-10-21T14:05:33Z"

generation: 1

labels:

app.kubernetes.io/instance: karpenter

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/name: karpenter

app.kubernetes.io/version: 1.3.2

helm.sh/chart: karpenter-1.3.2

namespace: kube-system

resourceVersion: "2664456"

uid: 2b58340a-fd07-4567-95a9-2a43b5dd4bca

spec:

maxUnavailable: 1

selector:

matchLabels:

app.kubernetes.io/instance: karpenter

app.kubernetes.io/name: karpenter

status:

conditions:

- lastTransitionTime: "2025-10-27T10:52:01Z"

message: ""

observedGeneration: 1

reason: SufficientPods

status: "True"

type: DisruptionAllowed

currentHealthy: 2

desiredHealthy: 1

disruptionsAllowed: 1

expectedPods: 2

observedGeneration: 1

PDB and Rolling Update of Node Group

ALLOWED DISRUPTIONS = 1 is generally the safest and most standard setting for a rolling node group update (which can be triggered if we e.g. upgrade k8s version of aws_eks_node_group), especially for high-availability workloads.

Key Considerations for maxUnavailable: 1

Safety First: This setting ensures only one node is updated at a time (node - if all pod replicas are distribute across all nodes). This is ideal for maintaining quorum in stateful applications like databases (e.g., Consul or ZooKeeper) where losing multiple nodes

simultaneously could cause data loss or service failure.

Default Behavior: In Amazon EKS managed node groups, maxUnavailable defaults to 1 if not specified.

Resource Availability: For this to work, your cluster must have enough spare capacity (CPU/Memory) on the remaining nodes to host the pods evicted from the node being updated.

Update Speed: While safe, updating one node at a time is the slowest method. For very large clusters, you might consider a higher absolute number or a percentage (e.g., 10%) to speed up the process.

When 1 is NOT Enough

Blocking Drains: If you have a Pod Disruption Budget (PDB) where minAvailable equals your total replicas, the node drain will be blocked, and the update will stall because no pods can be legally moved.

Timeouts: Amazon EKS has a 15-minute timeout for draining pods. If pods take too long to terminate, the update may fail unless you have configured pre-stop hooks or adjusted your PDBs.

In-Depth Explanation

In a Pod Disruption Budget (PDB), the setting ALLOWED DISRUPTIONS = 1 (which results from either maxUnavailable: 1 or a minAvailable value that leaves one "slot" free) acts as a safety valve that synchronizes infrastructure changes with application health.

Here is exactly how it makes a rolling node group update safe:

1. It Hooks into the "Eviction API"

When a node group update begins, the automation doesn't just "kill" pods; it calls the Kubernetes Eviction API.

The Check: Before a pod is removed from a node being updated, the Eviction API checks your PDB.

The Logic: If ALLOWED DISRUPTIONS is 1, the API allows exactly one pod to be terminated. Once that pod is gone, the "Allowed Disruptions" counter drops to 0.

2. It Forces a Sequential Wait

This is the most critical safety feature. If a second pod eviction is requested while the first one is still being replaced:

The Block: The API sees ALLOWED DISRUPTIONS = 0 and rejects the eviction request.
The Wait: The node "drain" process pauses and retries. It will stay paused until the first pod's replacement is scheduled on a different node and passes its Readiness Probe.
The Reset: Only when the new pod is "Ready" does the ALLOWED DISRUPTIONS count return to 1, allowing the next pod to be safely evicted.

3. It Prevents "Brain-Dead" Automation

Without this PDB setting, a node group update might try to drain a node that holds all replicas of your app.

Without PDB: The node drains, kills all pods at once, and you have a total outage.
With ALLOWED DISRUPTIONS = 1: The automation is physically unable to kill the second pod until the first one is safely back online elsewhere, ensuring your app always has at least some capacity.

Summary of the "Safety Loop"

Step Action PDB State (Allowed Disruptions)

1 Node update starts; first pod eviction requested 1 → Eviction granted

2 First pod is terminating; replacement is starting 0 → All further evictions blocked

3 Replacement pod passes Readiness Probe 0 → 1 → Block lifted

4 Next pod eviction requested 1 → Process repeats

Important Note: If your app only has 1 replica total, a PDB with minAvailable: 1 will block the node update forever because it can never safely evict that single pod. You generally need at least 2 replicas for this safety mechanism to work without stalling your updates.

Can pod disruption budget be defined in deployment manifest?

No, a Pod Disruption Budget (PDB) cannot be defined directly inside a Deployment manifest. In Kubernetes, a PDB is a separate API resource (Kind: PodDisruptionBudget) that exists independently of the Deployment.

While they are separate resources, they are linked by a Label Selector. You define the PDB to target specific pods by matching the labels specified in your Deployment's pod template.

Why they are separate?

Decoupled Lifecycle: You can create, update, or delete a PDB without needing to redeploy your application or trigger a rolling update of your pods.
Multiple Controllers: A single PDB can protect pods across multiple Deployments, ReplicaSets, or StatefulSets as long as they share the same labels.
Administrative Control: Cluster administrators may manage PDBs separately from application developers to ensure cluster-wide stability during maintenance like node drains.

How to link them?

To make them work together, ensure the spec.selector.matchLabels in your PDB YAML matches the spec.template.metadata.labels in your Deployment YAML.

Deployment snippet:

metadata:

labels:

app: my-app

spec:

template:

metadata:

labels:

app: my-app

PDB snippet:

spec:

selector:

matchLabels:

app: my-app

minAvailable: 2

Here is an example of a complete multi-resource YAML that includes both the Deployment and the PDB:

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: nginx

spec:

replicas: 3

selector:

matchLabels:

app: nginx

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx:latest

ports:

- containerPort: 80

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

# The PDB finds the pods using this selector

selector:

matchLabels:

app: nginx

# Ensures at least 2 pods stay up during voluntary disruptions

minAvailable: 2

Key Takeaways:

The Link: The spec.selector.matchLabels in the PDB must exactly match the spec.template.metadata.labels in the Deployment.
Voluntary Disruptions: This PDB will protect your pods during actions like node drains or cluster upgrades, but it won't prevent "involuntary" issues like a hardware crash.

Using maxUnavailable is often better for autoscaling environments because it scales with your replica count. Instead of saying "I need X pods alive," you’re saying "I can afford to lose X pods."

Here is the updated manifest:

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 5

selector:

matchLabels:

app: nginx

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx:latest

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

selector:

matchLabels:

app: nginx

# Only 1 pod can be taken down at a time during maintenance

maxUnavailable: 1

Pro Tips for maxUnavailable:

Absolute Number: Using 1 ensures that even if you have 100 replicas, Kubernetes will only drain one node at a time for this app.
Percentage: You can use a string like "25%" if you want the "allowed downtime" to grow or shrink as your Deployment scales.
The "One" Rule: You cannot use both minAvailable and maxUnavailable in the same PDB; you have to pick one strategy.

How to use a percentage to make the PDB more flexible for Horizontal Pod Autoscaling (HPA)?

Using percentages for a Pod Disruption Budget (PDB) is highly recommended when combined with a Horizontal Pod Autoscaler (HPA). This allows the disruption budget to scale dynamically as your application grows or shrinks based on traffic.

Dynamic YAML Example (Deployment + HPA + PDB)

This configuration ensures that no matter how many replicas the HPA creates, at least 80% of them will always remain available during maintenance.

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 2 # Initial count

selector:

matchLabels:

app: web

template:

metadata:

labels:

app: web

spec:

containers:

- name: nginx

image: nginx:latest

---

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

minReplicas: 2

maxReplicas: 10

metrics:

- type: Resource

resource:

target:

type: Utilization

averageUtilization: 50

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

spec:

selector:

matchLabels:

app: web

# Scales with HPA: if HPA scales to 10 pods, 8 must stay up.

minAvailable: 80%

Key Behaviours to Note:

Rounding Logic: Kubernetes always rounds up the result of the percentage to the nearest integer. For example, if you have 7 pods and set minAvailable: 50%, Kubernetes requires 4 pods to remain available.
Maintenance Blockers: Be careful with 100% or 0%. Setting minAvailable: 100% or maxUnavailable: 0% will completely block node drains, preventing cluster administrators from performing upgrades or maintenance.
Single Replica Limitation: If your HPA scales down to 1 replica, a PDB with minAvailable: 1 or 100% will prevent that single pod from being evicted, which can stall cluster maintenance.

How to troubleshoot a node drain that is stuck because of a PDB?

A kubectl drain typically gets stuck because the ALLOWED DISRUPTIONS for a specific PDB is 0. This means evicting even one more pod would violate your availability rules.

1. Identify the Blocking PDB

Run this command to see which PDB is preventing the drain:

kubectl get pdb -A

Look for any PDB where ALLOWED DISRUPTIONS is 0. If it’s 0, the eviction API will reject any attempt to move those pods, and the drain will wait indefinitely.

2. Common Reasons for "Allowed Disruptions: 0"

Too Few Replicas: You have minAvailable: 1 but only 1 pod is currently running.
Unhealthy Pods: Your PDB requires 3 pods to be healthy, but one is crashing or not yet "Ready." The PDB won't allow any more to be removed until that one is fixed.
Misconfiguration: You set minAvailable equal to your total replicas (e.g., 3 out of 3), meaning you've effectively told Kubernetes "never take a pod down".

3. How to Unstick the Drain

Depending on your situation, use one of these three strategies:

Strategy: Scale Up

Command: kubectl scale deployment <name> --replicas=<current+1>
Best For: Safest: Adds a new pod on a different node so the PDB is satisfied.

Strategy: Relax PDB

Command: kubectl edit pdb <pdb-name>
Best For: Cleanest: Temporarily lower minAvailable or increase maxUnavailable.

Strategy: Temporary Delete

Command: kubectl delete pdb <pdb-name>
Best For: Emergency: Delete the PDB, finish the drain, then re-apply the PDB later.

Warning: The "Force" Flag

Using kubectl drain <node> --force does not bypass PDBs. It only forces the deletion of pods not managed by a controller (like standalone pods) or pods using emptyDir. To truly bypass a PDB, you must either delete the PDB or manually kubectl delete pod the blocking pods

How to find all pods on a specific node that are currently protected by a PDB?

To find pods on a specific node that are protected by a Pod Disruption Budget (PDB), you need to cross-reference the pods on that node with the label selectors defined in your PDBs.

Option 1: The Quickest Way (Manual Check)

The most direct way to see all pods on a node and identify their labels is to use the describe node command.

kubectl describe node <node-name>

Look for the Non-terminated Pods section. You can then check if those pods' labels match your PDB selectors.

Option 2: Filtered Search (Best for Large Nodes)

If the node has many pods, use this command to list all pods on that node along with their labels:

kubectl get pods -A \

-o wide \

--field-selector spec.nodeName=<node-name> \

--show-labels

-A: Searches across all namespaces.

--field-selector spec.nodeName: Limits results to the specific node.

--show-labels: Displays the labels you need to compare against your PDB's spec.selector.

Option 3: Identifying the PDB Selector

To see exactly what labels a PDB is looking for, run:

kubectl get pdb <pdb-name> \

-o jsonpath='{.spec.selector.matchLabels}'

Any pod on your node that has these exact labels is protected by that PDB.

Summary of Useful Flags

Flag Purpose

----------------- ----------

-o wide Shows the NODE column to verify where pods are running.

--field-selector Filters the list server-side by the node name.

-l or --selector Filters pods by specific labels (e.g., kubectl get pods -l app=nginx).

---

How to install Clickhouse in AWS EKS cluster via Altinity Helm charts and Terraform

2026-03-05T11:50:00.006+00:00

Installing ClickHouse on an AWS EKS cluster using Terraform and the Altinity Helm charts typically involves two stages:

Installing the Altinity ClickHouse Operator
Deploying a ClickHouse Installation (CHI)

The Altinity Helm repository is located at https://helm.altinity.com.

Prerequisites

Ensure your Terraform environment is configured with the following providers:

aws: To manage EKS and underlying infrastructure.
kubernetes: To interact with the EKS cluster.
helm: To install the operator.

Terraform Configuration

The following example uses the helm_release resource to install the operator and the kubernetes_manifest resource to deploy the actual ClickHouse cluster.

Step A: Install the Altinity Operator

The operator is the "brain" that manages ClickHouse instances on Kubernetes.

resource "helm_release" "clickhouse_operator" {

name = "clickhouse-operator"

repository = "https://helm.altinity.com"

chart = "altinity-clickhouse-operator"

namespace = "clickhouse-operator"

create_namespace = true

# Optional: Enable metrics for Prometheus

set {

name = "metrics.enabled"

value = "true"

}

Step B: Deploy a ClickHouse Cluster (CHI)

Once the operator is running, you define your ClickHouse cluster using a Custom Resource (CRD). In Terraform, you use kubernetes_manifest.

resource "kubernetes_manifest" "clickhouse_cluster" {

depends_on = [helm_release.clickhouse_operator]

manifest = {

apiVersion = "clickhouse.altinity.com/v1"

kind = "ClickHouseInstallation"

metadata = {

name = "simple-clickhouse"

namespace = "clickhouse-operator"

}

spec = {

configuration = {

clusters = [

{

name = "cluster1"

layout = {

shardsCount = 1

replicasCount = 1

}

]

}

Production Considerations for EKS

When running ClickHouse on EKS, you should consider storage and networking:

Storage Class: Use AWS gp3 volumes for a good balance of price and performance. You can specify a volumeClaimTemplate in your kubernetes_manifest.
Node Affinity: It is recommended to run ClickHouse on specific node groups (e.g., using i3 or r5 instances) to ensure it doesn't compete with other workloads for IOPS.
Zookeeper/Keeper: For multi-node shards or replicas, you will need a Zookeeper cluster or the ClickHouse Keeper (also available via Altinity charts).

EKS Module

Altinity maintains a dedicated Terraform EKS ClickHouse module that automates the entire VPC, EKS, and ClickHouse setup if you prefer a pre-packaged solution.

How to view Clickhouse Installation Configuration?

% kubectl get chi -n clickhouse -o yaml

apiVersion: v1

items:

- apiVersion: clickhouse.altinity.com/v1

kind: ClickHouseInstallation

metadata:

annotations:

kubectl.kubernetes.io/last-applied-configuration: |

{"apiVersion":"clickhouse.altinity.com/v1","kind":"ClickHouseInstallation","metadata":....}

creationTimestamp: "2025-01-28T14:35:46Z"

finalizers:

- finalizer.clickhouseinstallation.altinity.com

generation: 12

namespace: clickhouse

resourceVersion: "67251031"

uid: 9fxxxx1-81e7-429b-9cf7-ffxxxxxxef

spec:

configuration:

clusters:

- layout:

replicasCount: 1

shardsCount: 1

templates:

dataVolumeClaimTemplate: ch-data

podTemplate: ch-pod

serviceTemplate: ch-svc

users:

admin/grants/query: GRANT ALL ON *.*

admin/networks/ip: 0.0.0.0/0

admin/password: my-admin-password

(or admin/password_sha256_hex: my-admin-password-in-sha256)

admin/profile: xxxx

admin/quota: xxxxx

admin/settings/enable_http_compression: 1

default/k8s_secret_password_sha256_hex: <namespace/secretName/key>

default/profile: default

default/quota: default

templates:

podTemplates:

- name: ch-pod

spec:

containers:

- image: altinity/clickhouse-server:24.8.14.10544.altinitystable

- args:

- server

env:

- name: LOG_LEVEL

value: info

- name: API_LISTEN

value: 0.0.0.0:7171

- name: API_CREATE_INTEGRATION_TABLES

value: "true"

- name: REMOTE_STORAGE

value: s3

- name: BACKUPS_TO_KEEP_REMOTE

value: "2"

- name: S3_BUCKET

value: my-clickhouse-backups

- name: S3_REGION

value: us-east-1

- name: CLICKHOUSE_HOST

value: localhost

- name: CLICKHOUSE_USERNAME

value: xxxxx

- name: CLICKHOUSE_PASSWORD

value: xxxx

image: altinity/clickhouse-backup:latest

imagePullPolicy: IfNotPresent

serviceAccountName: clickhouse-backup

tolerations:

- effect: NoSchedule

key: karpenter/clickhouse

operator: Exists

serviceTemplates:

- metadata:

annotations:

service.beta.kubernetes.io/aws-load-balancer-ip-address-type: ipv4

service.beta.kubernetes.io/aws-load-balancer-name: my-clickhouse-nlb

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

service.beta.kubernetes.io/aws-load-balancer-scheme: internal

service.beta.kubernetes.io/aws-load-balancer-type: nlb

spec:

ports:

- name: http

port: 8123

targetPort: 8123

- name: native

port: 9000

targetPort: 9000

type: LoadBalancer

volumeClaimTemplates:

- name: ch-data

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 100Gi

status:

chop-commit: 9abcd12

chop-date: 2025-01-24T08:40:12

chop-ip: 10.x.x.x

chop-version: 0.25.5

clusters: 1

endpoint: clickhouse-clickhouse.clickhouse.svc.cluster.local

fqdns:

- chi-clickhouse-ch-0-0.clickhouse.svc.cluster.local

hosts: 1

hostsWithTablesCreated:

- chi-clickhouse-ch-0-0.clickhouse.svc.cluster.local

pods:

- chi-clickhouse-ch-0-0-0

shards: 1

status: Completed

taskID: auto-1xxxxd2-5ba4-4c3a-9daa-baxxxxx850

taskIDsCompleted:

- auto-1fxxxxxd2-5ba4-4c3a-9daa-baxxxxxx50

...

- auto-bbxxxx6-31e3-4a4c-b04b-e5xxxxxx91

taskIDsStarted:

- auto-31xxxxx37-492f-4109-b515-4axxxxxx6c8

...

- auto-b8xxxxx7-0396-41e0-b5d1-95xxxxd48

kind: List

metadata:

resourceVersion: ""

Users section shows users config in form USER_NAME/ATTRIBUTE. In the example above we have two users: admin and default.

USER_NAME/password values is plain text password. This is very convenient for debugging (though usually a security "no-no" for production, especially if that's admin or default user!).

USER_NAME/password_sha256_hex is a SHA256 hashed password.

USER_NAME/k8s_secret_password_sha256_hex: <namespace/SECRET_NAME/KEY_NAME > shows that USER_NAME ClickHouse user is secured using a Kubernetes Secret. This maps the default user's password to a specific Kubernetes Secret.

USER_NAME/k8s_secret_password_sha256_hex: Specifies that for the user named USER_NAME, the password should be read from a Kubernetes Secret as a SHA256 hex string.
<namespace/SECRET_NAME/KEY_NAME >: This is the reference to the secret itself, structured as namespace/SECRET_NAME/KEY_NAME.
Purpose: This allows for secure, GitOps-friendly password management, preventing plain-text passwords from appearing in Kubernetes manifests.
Implementation: The ClickHouse Operator reads this secret and places the hashed password into the users.xml file for the ClickHouse server. Operator reads the secret, hashes the password (if necessary), and writes it into a file called /etc/clickhouse-server/users.d/chop-generated-users.xml inside your ClickHouse pod. If you have External Secrets installed, this secret is likely being pulled from AWS Secrets Manager.
Alternative: You can also use k8s_secret_env_password_sha256_hex to load the password via an environment variable.

In the Altinity Operator, the syntax USER_NAME/k8s_secret_password_sha256_hex is a pointer. It tells the operator to look into a specific secret to find the password hash for the USER_NAME user.

To get the password:

% kubectl get secret <SECRET_NAME> \

-n <NAMESPACE> \

-o jsonpath="{.data.<KEY_NAME>}" \

| base64 -d

NAMESPACE is usually clickhouse.

How to check Clickhouse health?

Since ClickHouse is running in our cluster, the best way to verify it's "working fine" is to move beyond just checking the Pod status and actually query the database engine itself.

Here is a 3-step approach to verify health, connectivity, and data integrity.

1. The "Internal" Health Check

The quickest way is to execute a command directly inside the pod using the clickhouse-client. This bypasses networking issues and tells you if the engine is responsive. Run this command:

kubectl exec -it chi-clickhouse-ch-0-0 \

-n clickhouse \

-- clickhouse-client --query "SELECT version(), uptime()"

chi-clickhouse-ch-0-0 is the name of the pod, it can also be like chi-clickhouse-ch-0-0-0.

If this returns data, it means ClickHouse is successfully reading from its system tables on the EBS volume.

What to look for: It should return the version string and the number of seconds the server has been up. If this fails, the DB engine itself is hung.

If you are using default user which has a password, or, default user was disabled, the output might show the error similar to this:

% kubectl exec -it chi-clickhouse-ch-0-0-0 -n clickhouse -- clickhouse-client --query "SELECT version(), uptime()"

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup

Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name.

If you have installed ClickHouse and forgot password you can reset it in the configuration file.

The password for default user is typically located at /etc/clickhouse-server/users.d/default-password.xml

and deleting this file will reset the password.

See also /etc/clickhouse-server/users.xml on the server where ClickHouse is installed.

. (AUTHENTICATION_FAILED)

command terminated with exit code 4

Seeing an AUTHENTICATION_FAILED error instead of a Connection Refused error is actually a positive result for this check:

Networking works: Your kubectl exec reached the pod.
Process is alive: The ClickHouse server is running and actively rejecting bad logins.
Storage is mounted: ClickHouse can't check credentials if it can't read its config files from disk.

If we know the Clickhouse credentials, we can perform the health check:

% kubectl exec -it chi-clickhouse-ch-0-0-0 -n clickhouse -- clickhouse-client --user USER --password PASS --query "SELECT version(), uptime(), name FROM system.clusters"

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup

24.8.14.10544.altinitystable 501389 all-clusters

24.8.14.10544.altinitystable 501389 all-replicated

24.8.14.10544.altinitystable 501389 all-sharded

24.8.14.10544.altinitystable 501389 ch

24.8.14.10544.altinitystable 501389 default

The output above is exactly what we wanted to see. The database is responsive, healthy, and has an uptime of ~5.8 days (501,389 seconds). The version 24.8.14.10544.altinitystable indicates we are on a very recent, stable Altinity build.

2. Check Replication and Disk Health

Since you are using the Altinity Operator, ClickHouse is likely managing data across disks. You want to ensure the "System" tables report no errors. Run this to check if the disks are mounted and have space:

% kubectl exec -it chi-clickhouse-ch-0-0-0 \

-n clickhouse \

-- clickhouse-client --user USER --password PASS \

--query "SELECT name, path, formatReadableSize(free_space) AS free, formatReadableSize(total_space) AS total FROM system.disks"

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup

default /var/lib/clickhouse/ 89.60 GiB 95.80 GiB

If you have multiple replicas (e.g., a ch-0-1 pod), check for replication lag:

% kubectl exec -it chi-clickhouse-ch-0-0-0 \

-n clickhouse \

-- clickhouse-client --user USER --password PASS \

--query "SELECT type, last_exception, num_tries FROM system.replication_queue WHERE last_exception != ''"

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup

Result: This should ideally be empty (or as above) . If you see exceptions here, your nodes aren't syncing correctly.

3. Verify the "Operator" View

The Altinity Operator provides a "Status" field in its Custom Resource that summarizes the health of the entire installation.

% kubectl get chi -n clickhouse

NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE

clickhouse 1 1 Completed 123d

What to look for: Look for the STATUS column. It should say Completed. If it says InProgress or Error, the Operator is struggling to configure the cluster.

4. Check the Backup (Safety Net)

Since you saw clickhouse-backup pods earlier, verify that the last backup actually succeeded. This is your "point of no return" check before the upgrade.

kubectl logs -n clickhouse -l job-name=clickhouse-backup-cron-<TIMESTAMP>

(Replace <TIMESTAMP> with one of the strings from your previous get all output, e.g., 29543400).

Look for: Done, Success, or Upload finished.

5. Check the Status of All Replicas

To be absolutely sure the cluster is "Green" before you start the EKS upgrade, run this to check the status of all replicas in the cluster:

% kubectl exec -it chi-clickhouse-ch-0-0-0 -n clickhouse -- clickhouse-client --user USER --password PASS --query "SELECT replica_path, is_leader, is_readonly, future_parts FROM system.replicas"

is_readonly: Should be 0. If it's 1, the node can't write data (usually a Zookeeper issue).

is_leader: One of your replicas should be 1.

Summary Checklist

Test Command Goal Good Result

Ping SELECT 1 1

Uptime SELECT uptime() >0

Storage system.disks Free space > 10%

Operator kubectl get chi Completed

Resources:

Altinity | Run Open Source ClickHouse® Better

Introduction to ArgoCD

2026-03-05T00:46:00.007+00:00

ArgoCD is a tool for deploying applications in Kubernetes cluster following the GitOps principles.

Argo CD is a declarative, GitOps-based continuous delivery (CD) tool designed specifically for Kubernetes. It acts as a controller that monitors running applications, compares their live state to the desired state defined in a Git repository, and automatically syncs them to ensure consistency.

Key Aspects of an Argo CD Application:

GitOps Source of Truth: Git repositories hold the desired state (manifests, Helm charts, Kustomize configs), which Argo CD pulls to apply to clusters.
Automated Synchronization: It automatically detects "OutOfSync" applications—where the live cluster state differs from Git—and can automatically or manually sync them to match.
Continuous Monitoring: It acts as a Kubernetes controller that continuously monitors applications.
Visualization & Management: It provides a web UI to visualize application structure, monitor status, and manage rollbacks.
Key Capabilities: Supports automated deployment, drift detection, and easy rollbacks.

Argo CD is often used to ensure that the actual state of a Kubernetes cluster matches the configuration stored in a Git repository, making the deployment process more reliable and transparent.

ArgoCD GitOps flow:

Commit and push infra code changes (Terraform, Helm values etc) to the main branch of your GitHub repository.
ArgoCD Sync:

Manual: Log in to our ArgoCD Dashboard and click the "Sync" or "Refresh" button on the psmdb-default-sharded application.
Automatic: If "Self-Heal" or "Auto-Sync" is enabled, ArgoCD will detect the Git change within ~3 minutes and apply it to the cluster automatically.

Monitor the Operator: Once ArgoCD syncs, infra will get changed e.g. operators will see the updated custom resources and then trigger the further actions.

Installation

Installing ArgoCD in a Kubernetes cluster is primarily done using manifests or Helm charts. The most common and recommended approach for beginners is using the Official Argo CD Manifests.

Prerequisites:

A running Kubernetes cluster (v1.22 or later).
kubectl command-line tool installed and configured to your cluster.
At least 2GB of available memory in your cluster.

Applications

ArgoCD defines custom Kubernetes objects like Application, AppProject, settings...which can be defined declaratively using Kubernetes manifests and deployed via kubectl apply to the ArgoCD namespace which is argocd by default.

To check ArgoCD applications:

% kubectl get applications -A

NAMESPACE NAME SYNC STATUS HEALTH STATUS

argocd my-app Synced Progressing

Dashboard

If you don't know the url of the ArgoCD dashboard, find the IP of the ArgoCD server and port-forward it:

% kubectl get svc argocd-server -n argocd

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

argocd-server ClusterIP 172.21.103.127 <none> 80/TCP,443/TCP 1d

By default, ArgoCD includes a built-in admin user with full super-user access. For better security practices, it is recommended that you use the admin account only for the initial configuration, then disable it once all required users have been added.

For this example, we can stick to this default admin user. Its password can be obtain via this command:

% kubectl get secret \

argocd-initial-admin-secret \

-n argocd \

-o jsonpath="{.data.password}" \

| base64 -d

Port forwarding:

% kubectl port-forward svc/argocd-server -n argocd 8080:443

Forwarding from 127.0.0.1:8080 -> 8080

Forwarding from [::1]:8080 -> 8080

Handling connection for 8080

...

We can now open the address http://localhost:8080/ in the browser to get the ArgoCD dashboard and log in with credentials above.

If there are no application registered with ArgoCD, we will see something like this:

application.yaml vs applicationset.yaml

In the world of Argo CD and Kubernetes GitOps, these two files represent different levels of automation.

1. application.yaml

An Application is the basic unit of Argo CD. This file defines a single link between a specific Git repository (the source) and a specific Kubernetes cluster/namespace (the destination).

The Goal: To tell Argo CD: "Take the code from this folder and make sure it looks like that in my cluster."
Key Components:

Project: The logical group it belongs to.
Source: Argo CD needs three pieces of information to find our files:

repoURL: Git URL. Which "building" (repository) do I go to?
targetRevision: revision (branch/tag), Which "floor" (branch/tag/commit) am I looking at?
path (mandatory): path to the manifests. Which "room" (folder) contains the manifests?

Destination: The target cluster URL and namespace.
Sync Policy: Whether to automatically sync changes or require a manual button press.

Example Snippet:

apiVersion: argoproj.io/v1alpha1

kind: Application

metadata:

spec:

project: default

source:

repoURL: https://github.com/argoproj/argocd-example-apps.git

targetRevision: main

path: guestbook

destination:

server: https://kubernetes.default.svc

namespace: default

syncPolicy:

automated:

selfHeal: true

prune: true

Argo CD is completely "tool-agnostic." It doesn't care if you use Helm, Kustomize, or just a folder full of .yaml files; it just needs to know where they are.

How Argo CD Decides

When Argo CD looks at the path you provided, it follows this logic:

Does it see Chart.yaml? -> It uses Helm.

Argo CD uses values.yaml by default if it detects that the directory specified in the path is a Helm chart.

Does it see kustomization.yaml? -> It uses Kustomize.
Does it see only .yaml or .json files? -> It uses Plain Manifests.

Custom Helm Charts

At an absolute minimum, a Helm chart requires a directory containing these three things:

my-app/ # The "path" referenced in Argo CD

├── Chart.yaml # Metadata about the chart (Required)

├── values.yaml # Default configuration variables (Required)

└── templates/ # The folder for your Kubernetes manifests (Required)

└── deployment.yaml # At least one manifest to actually deploy something

2. applicationset.yaml

The ApplicationSet is a controller that acts as a factory for Applications. Instead of manually creating 50 application.yaml files for 50 different microservices or 10 different clusters, you write one ApplicationSet.

The Goal: To automate the creation, update, and deletion of multiple Argo CD Applications at scale.
How it works: It uses Generators to decide how many Applications to create.

List Generator: Hardcode a list of clusters or environments.
Git Generator: Automatically create an App for every folder found in a Git repo.
Cluster Generator: Automatically deploy an App to every cluster registered in Argo CD.

Key Advantage: It follows the "DRY" (Don't Repeat Yourself) principle. If you add a new cluster to your fleet, the ApplicationSet detects it and automatically deploys your apps there.

Example Snippet:

apiVersion: argoproj.io/v1alpha1

kind: ApplicationSet

metadata:

spec:

generators:

- list:

elements:

- cluster: engineering-dev

- cluster: engineering-prod

metadata:

spec:

project: default

source:

repoURL: https://github.com/argoproj/argocd-example-apps.git

path: guestbook

destination:

server: https://{{cluster}}-api.com

namespace: guestbook

Example with Pull Request Generator

Using the Pull Request Generator is the "gold standard" for enabling ephemeral (preview) environments. It allows Argo CD to watch your GitHub repository for open PRs and automatically spin up a dedicated version of your app for testing.

The following parameters are dynamic variables provided by the generator based on the metadata of the PR:

{{ number }} - PR number from GitHub API

Role: Unique Identifier.
Usage: Since PR numbers are unique and immutable in GitHub, this is the safest way to name your Application and Namespace.
Result: Prevents naming collisions when multiple developers are testing different features simultaneously.

{{ branch }} - Head branch name of the PR

Role: Git Reference.
Usage: This tells Argo CD exactly which commit to look at. Without this, Argo CD would try to deploy from main.
Result: Your ephemeral environment stays in sync with every push to that specific feature branch.

{{ branch_slug }} - URL-safe version of branch name
{{ head_sha }} - Commit SHA at the tip of the branch
{{ labels }} - List of all labels on the PR e.g. ["preview", "staging"]

Role: Filter & Logic.
Usage: In the example above, we use the filters block. The generator will ignore any PR that doesn't have the feature-test-deployment label.
Advanced Tip: You can also use labels in the template. For example, if you have labels like size:large or db:postgres, you could map those to Helm values to provision different resources for that specific PR.

The Lifecycle Workflow

Developer opens a PR in GitHub.
Developer adds the label feature-test-deployment.
Argo CD detects the label, triggers the ApplicationSet, and creates a new Application named my-app-pr-42.
Argo CD creates the namespace preview-pr-42 and deploys the code from the feature branch.
Developer closes the PR or removes the label.
Argo CD automatically deletes the Application and cleans up the resources (garbage collection).

The ApplicationSet Configuration

This ApplicationSet will scan your repository, filter for the specific label, and then use the PR details to name the application and target the correct branch.

apiVersion: argoproj.io/v1alpha1

kind: ApplicationSet

metadata:

namespace: argocd

spec:

generators:

- pullRequest:

github:

owner: my-organization

repo: my-app-repo

# Use a secret for your GitHub Token to avoid rate limits

tokenRef:

secretName: github-token

key: token

# This filters the PRs based on the label

filters:

- labels:

- feature-test-deployment

requeueAfterSeconds: 60

template:

metadata:

# Use .number to ensure a unique app name per PR (e.g., my-app-pr-42)

spec:

project: default

source:

repoURL: https://github.com/my-organization/my-app-repo.git

# Use .branch to pull code from the specific feature branch

targetRevision: '{{branch}}'

path: kubernetes/manifests

helm:

parameters:

# You can pass the PR number or labels into your app's code/config

- name: "envName"

value: "pr-{{number}}"

destination:

server: https://kubernetes.default.svc

# Create a unique namespace for the ephemeral environment

namespace: 'preview-pr-{{number}}'

syncPolicy:

automated:

prune: true

selfHeal: true

syncOptions:

- CreateNamespace=true

The requeueAfterSeconds: 60 means ArgoCD polls GitHub every 60 seconds - when a PR is opened with the feature-test-deployment label, within a minute ArgoCD picks it up, injects these variables into the template, and creates the child Application. When the PR is closed or the label is removed, ArgoCD deletes the child Application (and prune: true tears down the K8s resources).

Go template engine

By enabling the Go template engine (goTemplate: true), Argo CD switches from simple string replacement to a full Go templating context. In this mode, all parameters from the generator are nested under a root object, meaning you must use the leading dot to access them.

Here is the rewritten example using goTemplate: true.

apiVersion: argoproj.io/v1alpha1

kind: ApplicationSet

metadata:

spec:

# This enables the standard Go template engine

goTemplate: true

goTemplateOptions: ["missingkey=error"]

generators:

- pullRequest:

github:

owner: my-organization

repo: my-app-repo

tokenRef:

secretName: github-token

key: token

filters:

- labels:

- feature-test-deployment

template:

metadata:

# Now using the leading dot because goTemplate is true

spec:

project: default

source:

repoURL: https://github.com/my-organization/my-app-repo.git

targetRevision: '{{.branch}}'

path: kubernetes/manifests

helm:

parameters:

- name: "envName"

value: "pr-{{.number}}"

# goTemplate allows us to use range to iterate over labels

- name: "appliedLabels"

value: "{{range .labels}}{{.}},{{end}}"

destination:

server: https://kubernetes.default.svc

namespace: 'preview-pr-{{.number}}'

syncPolicy:

automated:

prune: true

selfHeal: true

syncOptions:

- CreateNamespace=true

Key Changes with goTemplate: true

The Dot Prefix: Since the parameters are now passed as a data structure to the Go template engine, you access them as fields of the root object: {{.number}}, {{.branch}}, and {{.labels}}.
Missing Key Error: With goTemplateOptions: ["missingkey=error"], if you make a typo (e.g., {{.numbr}}), the ApplicationSet controller will explicitly error out rather than just rendering an empty string. This is much safer for production.
Iteration/Logic: You can now use standard Go template functions. For instance, in the example above, I used a range loop to iterate through the .labels list:

{{range .labels}}{{.}},{{end}} loops through each label and prints it.

Why use this over the default?

While it requires the extra . syntax, goTemplate: true is much more powerful. It allows you to use if/else logic (e.g., "if label is 'large', request more CPU") and complex string manipulations that the default "fast replacement" engine cannot handle.

If you omit the dot while goTemplate: true is active, the template will fail to render because it will look for a global variable instead of a field in the provided data object.

Using goTemplate: true opens up a world of "smart" deployments where your PR labels can act as configuration switches. This prevents you from needing a separate ApplicationSet for every minor variation.

Here is how you can use conditional logic to adjust resources or features based on the labels found on a PR.

Example: Advanced Conditional ApplicationSet

In this scenario, if a PR has the label env:large, we increase the resource limits. If it has feature:postgres, we enable a database flag.

apiVersion: argoproj.io/v1alpha1

kind: ApplicationSet

metadata:

spec:

goTemplate: true

goTemplateOptions: ["missingkey=error"]

generators:

- pullRequest:

github:

owner: my-org

repo: my-repo

tokenRef:

secretName: github-token

key: token

filters:

- labels:

- feature-test-deployment

template:

metadata:

spec:

project: default

source:

repoURL: https://github.com/my-org/my-repo.git

targetRevision: '{{.branch}}'

path: charts/my-app

helm:

values: |

replicaCount: {{ if (contains "env:high-availability" (join "," .labels)) }}3{{ else }}1{{ end }}

resources:

limits:

cpu: "1000m"

memory: "2Gi"

limits:

cpu: "200m"

memory: "512Mi"

experimentalFeatures:

enabled: {{ contains "feature:beta" (join "," .labels) }}

destination:

server: https://kubernetes.default.svc

namespace: 'pr-{{.number}}'

Pro-Tips for this Setup

The "Contains" Trick: In Go templating, searching a list directly can be clunky. By using (join "," .labels), we turn the list into a single string (e.g., "feature-test-deployment,size:large") and then use contains to check for our specific trigger labels.
The Root App Advantage: Because this logic is inside your infrastructure/ folder (managed by your Root App), you can test changes to this logic on a branch. You can actually have a "Development Root App" that points to your branch to see if your new if/else logic works before merging it to main.
Cleanup: When you merge the PR and delete the branch in GitHub, the PR Generator sees the PR is closed. It removes the generated Application, and Argo CD's prune logic deletes the namespace and all associated resources automatically.

Summary of the Flow

Management: The Root App (application.yaml) monitors the repo for automation changes.
Automation: The ApplicationSet (applicationset.yaml) monitors GitHub for new PRs with specific labels.
Deployment: The Ephemeral App (the child application.yaml) is created with specific CPU/RAM settings dictated by those PR labels.

How do they work together?

In many production GitOps setups, they must exist together to create a functional hierarchy. In Argo CD, this is often referred to as the "App of Apps" pattern or a "Management Repo" structure.

How They Interact

Think of the relationship as Infrastructure vs. Implementation:

The ApplicationSet (applicationset.yaml): Acts as the automation engine. It lives in your management folder and its job is to watch the repository and "spawn" Applications.
The Application (application.yaml): These are the individual instances created by the ApplicationSet. You might also have a manual application.yaml for a unique service that doesn't fit the "factory" mold of the ApplicationSet.

Recommended Repository Structure

To keep things clean, teams usually organize their repository like this:

my-gitops-repo/

├── bootstrap/

│ ├── root-app.yaml # A manual Application that points to this folder

│ └── cluster-addons-set.yaml # An ApplicationSet for things like ingress, monitoring

├── projects/

│ └── microservices-set.yaml # Your PR Generator ApplicationSet

├── apps/

│ ├── my-unique-legacy-app/

│ │ └── application.yaml # A one-off manual Application

│ └── templates/ # The base manifests the ApplicationSet points to

│ ├── deployment.yaml

│ └── service.yaml

Common Scenarios for Co-existence

1. The "Bootstrap" Application

You often have a single, manually created application.yaml (sometimes called the Root App) whose only job is to deploy the applicationset.yaml.

Why? This allows you to manage your automation via GitOps. If you update the ApplicationSet logic in Git, the Root App syncs it, and the factory settings update automatically.

2. The "Hybrid" Approach

You might use an ApplicationSet to handle 50 identical microservices across 3 clusters, but keep a standard Application for a third-party tool (like a database or a legacy monolith) that requires very specific, manual configuration that doesn't fit a template.

Important Collision Warning ⚠️

While they can exist together, you should avoid a "Circular Dependency" or "Ownership Fight":

Don't have an ApplicationSet generate an Application with the exact same name as a manually created application.yaml in the same namespace.
Management: If an ApplicationSet creates an app, it "owns" it. If you try to manually edit that child app in the Argo CD UI, the ApplicationSet controller will immediately revert your changes to match its template.

Summary: They are complementary tools. Use application.yaml for specific, static deployments and applicationset.yaml for templated, dynamic, or multi-environment deployments. Using them together is the standard way to achieve full "Cluster as Code."

The "Root App" (or App-of-Apps) pattern is the ultimate way to manage your Argo CD instance. It allows you to manage your automation through the same GitOps workflow as your code.

The "Root" Application Structure

The Root App is a standard application.yaml that points to a folder in your Git repo containing your ApplicationSet files. Once you apply this one file manually, Argo CD takes over and manages all your "factories" automatically.

So there are two separate layers here that do different jobs, and it's easy to conflate them.

1. Layer 1 — Bootstrap Application (The Root Application) (root-app.yaml)

This is the only file you apply with kubectl apply -f. It tells Argo CD to watch the infrastructure/ directory for any ApplicationSets or other Applications. It manages the ArgoCD-level resources: ApplicationSet, other Applications, ExternalSecrets.... Its only job is to keep the e.g. ApplicationSet definition in sync with the repo. This is what the targetRevision in root-app.yaml controls.

apiVersion: argoproj.io/v1alpha1

kind: Application

metadata:

namespace: argocd

spec:

project: default

source:

repoURL: https://github.com/your-org/gitops-control-plane.git

targetRevision: HEAD

path: infrastructure # Folder where your ApplicationSets live

destination:

server: https://kubernetes.default.svc

namespace: argocd

syncPolicy:

automated:

prune: true

selfHeal: true

2. Layer 2 — ApplicationSet (pr-generator-set.yaml)

This is the pull-request generator. It watches GitHub for open PRs with the preview label and creates a child Application per PR. Each child Application it creates points at {{ .branch }} — the PR's own
branch:

targetRevision: "{{ .branch }}"
path: deploy/helm-chart

The Directory Layout

To make this work seamlessly, organize your repository so the Root App can find its "children" without getting confused.

gitops-control-plane/

├── root-app.yaml # Managed manually (once)

└── infrastructure/ # Managed by Root App

├── pr-generator-set.yaml # Your PR Generator with goTemplate: true

├── production-cluster-set.yaml

└── global-monitoring-app.yaml # A standard Application for Prometheus

Why this is powerful:

Self-Healing Infrastructure: If someone accidentally deletes your ApplicationSet from the cluster, the Root App will see the discrepancy and recreate it instantly.
Version Controlled Automation: If you want to update your goTemplate logic or change a github-token reference, you just submit a PR to the infrastructure/ folder. Once merged, the Root App updates the ApplicationSet, which then updates all your ephemeral environments.
Single Source of Truth: You can look at one Git folder and see exactly what automation logic is governing your entire cluster fleet.

So the two layers are independent:

root-app.yaml (tracks main)
└─ manages: ApplicationSet (in cluster)
                       └─ polls GitHub PRs with specified label
                               ├─ PR #1 → Application → branch: feat1-branch → deploy/helm-chart
                               └─ PR #2 → Application → branch: feat2-branch → deploy/helm-chart

Changing root-app.yaml's targetRevision to main only affects whether the ApplicationSet definition itself stays in sync with main. It has zero effect on how individual PR previews are deployed —
those are handled entirely by the generator using each PR's own branch.

The hardcoded branch in root-app.yaml is only necessary during development of this feature branch so that iterative changes to applicationset.yaml were picked up live. Once merged to main, main is
the correct target.

One Final Pro-Tip for PR Generators

When using the PR Generator inside a Root App structure, ensure your github-token Secret is already created in the argocd namespace. The ApplicationSet needs that token to "talk" to the GitHub API to see which PRs have the feature-test-deployment label.

Summary Comparison

Feature Application (.yaml) ApplicationSet (.yaml)

---------- ------------------------- -----------------------------

Scope Manages one app in one environment. Manages many apps across many environments.

Manual Effort High (must create a file for every app). Low (one template handles everything).

Use Case Simple setups or unique, one-off apps. Microservices, multi-tenant, or multi-cluster environments.

Relationship The "Child" object. The "Parent" (Factory) that creates children.

---

Resources:

Argo CD - Declarative GitOps CD for Kubernetes

Continuous Deployment with Argo CD - Amazon EKS

Introduction to KubePug (Kubernetes tool)

2026-02-26T15:42:00.001+00:00

What is KubePug?

KubePug (Kubernetes PreUpGrade Checker) is an open-source kubectl plugin and CLI tool designed to identify deprecated or deleted APIs in your Kubernetes cluster or manifest files before you perform an upgrade.

Key Features

Deprecation Detection: Scans your live cluster or static YAML manifests to find resources using APIs that are slated for removal in future Kubernetes versions.
Replacement Guidance: Not only flags outdated APIs but also suggests the recommended replacement API and specifies the exact version where the deprecation or deletion occurs.
Version Targeting: Allows you to specify a target Kubernetes version (e.g., v1.31) to validate your current resources against that specific future release.
Flexible Data Source: It automatically downloads a frequently updated API definition file (every 30 minutes) to stay current with the latest Kubernetes releases.

Why Use It?

As Kubernetes evolves, APIs are moved from alpha/beta to stable (GA), and older versions are eventually removed. If you upgrade your cluster without updating your manifests, those resources will fail to deploy or operate. KubePug provides a "pre-flight" check to prevent these breaking changes from reaching production.

Installation & Usage

You can install KubePug via Krew (where it is listed under the name deprecations) or as a standalone binary.

Method Command

--------- -------------

Install via Krew kubectl krew install deprecations

Scan Live Cluster kubectl deprecations --k8s-version=v1.30

Scan Manifest File kubepug --input-file=./my-manifest.yaml

Installation via Krew

% kubectl krew install deprecations

Updated the local copy of plugin index.

Installing plugin: deprecations

Installed plugin: deprecations

| Use this plugin:

| kubectl deprecations

| Documentation:

| https://github.com/rikatz/kubepug

| Caveats:

| \

| | * By default, deprecations finds deprecated object relative to the current kubernetes

| | master branch. To target a different kubernetes release, use the --k8s-version

| | argument.

| |

| | * Deprecations needs permission to GET all objects in the Cluster

| /

WARNING: You installed plugin "deprecations" from the krew-index plugin repository.

These plugins are not audited for security by the Krew maintainers.

Run them at your own risk.

Execution

Once installed, the plugin is invoked using kubectl deprecations.

Scan Current Cluster

Check your live cluster for deprecated APIs against a specific target Kubernetes version:

% kubectl deprecations --k8s-version=v1.33

Error: failed to get apiservices: apiservices.apiregistration.k8s.io is forbidden: User "sso:user" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

time="2026-02-26T14:20:19Z" level=error msg="An error has occurred: failed to get apiservices: apiservices.apiregistration.k8s.io is forbidden: User \"sso:user\" cannot list resource \"apiservices\" in API group \"apiregistration.k8s.io\" at the cluster scope"

KubePug requires running user to have "list" permissions on resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope as otherwise the above error will appear.

If required permissions are in place:

% kubectl deprecations --k8s-version=v1.33

No deprecated or deleted APIs found

Kubepug validates the APIs using Kubernetes markers. To know what are the deprecated and deleted APIS it checks, please go to https://kubepug.xyz/status/

Scan Local Manifest Files

Validate static YAML files before applying them to a cluster:

kubectl deprecations --input-file=./my-manifests/

View Results in Different Formats

Output the findings in json or yaml for automated processing:

kubectl deprecations --format=json

Check for Help and Flags

See all available configuration options, such as using a custom database file or setting error codes:

kubectl deprecations --help

Key Parameters

--k8s-version: The Kubernetes release you intend to upgrade to (defaults to the latest stable).

--error-on-deprecated: Forces the command to exit with an error code if deprecated APIs are found, which is useful for CI/CD pipelines.

---

Introduction to Krew (Kubernetes tool)

2026-02-26T14:15:00.008+00:00

Krew is the official plugin manager for the kubectl command-line tool. Much like apt for Debian or Homebrew for macOS, it allows users to easily discover, install, and manage custom extensions that add new subcommands to Kubernetes.

Core Functionality

Discovery: Users can search a community-curated index of over 200 plugins designed for tasks like security auditing, resource visualization, and cluster management.
Lifecycle Management: It automates the process of installing, updating, and removing plugins across different operating systems (Linux, macOS, and Windows).
Unified Interface: Once a plugin is installed via Krew, it is invoked directly through kubectl (e.g., kubectl <plugin-name>).

Installation

MacOS example output:

% (

set -x; cd "$(mktemp -d)" &&

OS="$(uname | tr '[:upper:]' '[:lower:]')" &&

ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&

KREW="krew-${OS}_${ARCH}" &&

curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&

tar zxvf "${KREW}.tar.gz" &&

./"${KREW}" install krew

)

+-zsh:142> mktemp -d

+-zsh:142> cd /var/folders/8j/60m6_18j359_39ls0sr9ccvm0000gp/T/tmp.LKDSyQcO10

+-zsh:143> OS=+-zsh:143> uname

+-zsh:143> OS=+-zsh:143> tr '[:upper:]' '[:lower:]'

+-zsh:143> OS=darwin

+-zsh:144> ARCH=+-zsh:144> uname -m

+-zsh:144> ARCH=+-zsh:144> sed -e s/x86_64/amd64/ -e 's/$arm$$64$\?.*/\1\2/' -e 's/aarch64$/arm64/'

+-zsh:144> ARCH=arm64

+-zsh:145> KREW=krew-darwin_arm64

+-zsh:146> curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-darwin_arm64.tar.gz

+-zsh:147> tar zxvf krew-darwin_arm64.tar.gz

x ./LICENSE

x ./krew-darwin_arm64

+-zsh:148> ./krew-darwin_arm64 install krew

Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git.

Updated the local copy of plugin index.

Installing plugin: krew

Installed plugin: krew

| Use this plugin:

| kubectl krew

| Documentation:

| https://krew.sigs.k8s.io/

| Caveats:

| \

| | krew is now installed! To start using kubectl plugins, you need to add

| | krew's installation directory to your PATH:

| |

| | * macOS/Linux:

| | - Add the following to your ~/.bashrc or ~/.zshrc:

| | export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

| | - Restart your shell.

| |

| | * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable

| |

| | To list krew commands and to get help, run:

| | $ kubectl krew

| | For a full list of available plugins, run:

| | $ kubectl krew search

| |

| | You can find documentation at

| | https://krew.sigs.k8s.io/docs/user-guide/quickstart/.

| /

Add Krew path to PATHs:

% vi ~/.zshrc

% cat ~/.zshrc

...

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" <-- added manually

Reload .zshrc configuration file within the currently running Zsh terminal (or just restart the terminal):

% source ~/.zshrc

Installation verification:

% kubectl krew

krew is the kubectl plugin manager.

You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:

kubectl krew [command]

Available Commands:

help Help about any command

index Manage custom plugin indexes

info Show information about an available plugin

install Install kubectl plugins

list List installed kubectl plugins

search Discover kubectl plugins

uninstall Uninstall plugins

update Update the local copy of the plugin index

upgrade Upgrade installed plugins to newer versions

version Show krew version and diagnostics

Flags:

-h, --help help for krew

-v, --v Level number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.

Common Commands

To use Krew, you must first install it as a kubectl plugin itself. Key commands include:

kubectl krew update: Updates the local list of available plugins.

kubectl krew search: Finds plugins in the official Krew index.

kubectl krew install <plugin>: Installs a specific plugin.

kubectl krew list: Displays all plugins currently installed through Krew.

kubectl krew upgrade: Updates all installed plugins to their latest versions.

Example:

% kubectl krew install deprecations

Updated the local copy of plugin index.

Installing plugin: deprecations

Installed plugin: deprecations

| Use this plugin:

| kubectl deprecations

| Documentation:

| https://github.com/rikatz/kubepug

| Caveats:

| \

| | * By default, deprecations finds deprecated object relative to the current kubernetes

| | master branch. To target a different kubernetes release, use the --k8s-version

| | argument.

| |

| | * Deprecations needs permission to GET all objects in the Cluster

| /

WARNING: You installed plugin "deprecations" from the krew-index plugin repository.

These plugins are not audited for security by the Krew maintainers.

Run them at your own risk.

Popular Plugins Managed by Krew

ctx / ns: Rapidly switch between Kubernetes contexts and namespaces.
tree: Visualizes the hierarchy of Kubernetes resources in a tree view.
access-matrix: Displays an RBAC (Role-Based Access Control) matrix for server resources.
get-all: Lists all resources in a namespace, including those often missed by kubectl get all.

Note on Security (!)

Plugins in the Krew index are community-contributed and are not audited for security by the Kubernetes maintainers; you should only install plugins from sources you trust.

---

Introduction to Pluto (Kubernetes tool)

2026-02-26T13:03:00.001+00:00

Pluto is:

CLI tool that helps users find deprecated Kubernetes API versions in your code repositories and Helm releases.
It's especially useful when upgrading Kubernetes clusters, as it identifies resources that need updating before the upgrade.
It works against:

Live clusters
Helm charts
Raw YAML

Pluto will show which APIs are deprecated or removed, what version they were deprecated in, and what the replacement API should be.

Installation on Mac (https://pluto.docs.fairwinds.com/installation/#homebrew-tap):

% brew install FairwindsOps/tap/pluto

Let's see its CLI arguments:

% pluto

You must specify a sub-command.

A tool to detect Kubernetes apiVersions

Usage:

pluto [flags]

pluto [command]

Available Commands:

completion Generate the autocompletion script for the specified shell

detect Checks a single file or stdin for deprecated apiVersions.

detect-all-in-cluster run all in-cluster detections

detect-api-resources detect-api-resources

detect-files detect-files

detect-helm detect-helm

help Help about any command

list-versions Outputs a JSON object of the versions that Pluto knows about.

version Prints the current version of the tool.

Flags:

-f, --additional-versions string Additional deprecated versions file to add to the list. Cannot contain any existing versions

--columns strings A list of columns to print. Mandatory when using --output custom, optional with --output markdown

--components strings A list of components to run checks for. If nil, will check for all found in versions.

-h, --help help for pluto

--ignore-deprecations Ignore the default behavior to exit 2 if deprecated apiVersions are found. (Only show removed APIs, not just deprecated ones)

--ignore-removals Ignore the default behavior to exit 3 if removed apiVersions are found. (Only show deprecated APIs, not removed ones)

--ignore-unavailable-replacements Ignore the default behavior to exit 4 if deprecated but unavailable apiVersions are found.

-H, --no-headers When using the default or custom-column output format, don't print headers (default print headers).

-r, --only-show-removed Only display the apiVersions that have been removed in the target version.

-t, --target-versions stringToString A map of targetVersions to use. This flag supersedes all defaults in version files. (default [])

-v, --v Level number for the log level verbosity

Use "pluto [command] --help" for more information about a command.

detect-files

If we want to scan local Helm charts or manifest files before they are deployed, we can use pluto detect-files or pluto detect, which do require us to be in the correct directory or provide a file path.

To scan and detect deprecated APIs in manifest files in a directory:

% pluto detect-files -d /path/to/your/manifests

detect-files is for checking YAML files in our repositories/filesystem before deploying them - that's separate from detect-helm and detect-all-in-cluster cluster commands.

To target particular k8s version:

% pluto detect-files -d . --target-versions k8s=v1.33.0

If we use Terraform to deploy Helm charts, we might want to keep chart values in separate files (.yaml or .yaml.tpl) as otherwise we won't be able to use Pluto directly (we'd need to extract values into files first). For more details, see Where to keep Helm chart values in Terraform projects | My Public Notepad.

detect-helm

To check Helm releases in the cluster (already deployed):

% pluto detect-helm -owide

To target particular k8s version:

% pluto detect-helm -owide --target-versions k8s=v1.33.0

There were no resources found with known deprecated apiVersions.

detect-helm specifically checks Helm release metadata stored in our cluster (in secrets or configmaps) after Helm chart have been deployed. It looks at the manifests that Helm used to install releases, which might contain deprecated APIs even if they haven't been applied yet or are stored in Helm's history.

This command can be run from any directory. This is because the detect-helm command scans live Helm releases currently deployed in our Kubernetes cluster, rather than looking for local files on our machine. Instead of relying on our current working directory, the command depends on our Kubernetes context (the cluster our CLI is currently pointed at) and our local Helm configuration to communicate with the cluster.

While the directory doesn't matter, the following must be true for the command to work:

Active Kubernetes Context: Your kubectl context must be set to the target cluster.
Cluster Permissions: You must have sufficient RBAC permissions to read Secrets in the namespaces you wish to scan, as Helm 3 stores release information in cluster secrets.
Target Versioning: The --target-versions k8s=v1.33.0 flag tells Pluto to check for APIs that are deprecated or removed specifically in Kubernetes version 1.33.0, regardless of what version the cluster is actually running

detect-all-in-cluster

To check all resources in the cluster:

% pluto detect-all-in-cluster -o wide

I0226 12:01:02.279788 47100 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"

There were no resources found with known deprecated apiVersions.

detect-all-in-cluster scans all live resources currently running in our cluster by querying the Kubernetes API directly. It checks deployments, services, pods, etc. that are actively deployed.

detect-all-in-cluster does NOT include detect-helm or detect-files. Here's why they're separate:

detect-all-in-cluster sees the current state of resources
detect-helm sees Helm's stored templates and history, which may include:

Templated manifests that haven't been rendered yet
Old release revisions
Chart templates with deprecated APIs

Run both to get complete coverage!

Target a specific Kubernetes version:

% pluto detect-all-in-cluster --target-versions k8s=v1.33.0

I0226 12:02:26.551401 47113 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"

There were no resources found with known deprecated apiVersions.

The warning message:

Warning: v1 ComponentStatus is deprecated in v1.19+

This is just Pluto itself triggering a Kubernetes API warning while scanning - it's not something wrong with our cluster resources.

The main result:

There were no resources found with known deprecated apiVersions.

This means all our cluster resources are using API versions that are still valid in Kubernetes v1.33.0 (our target version). This means:

Our cluster resources are already compatible with k8s v1.33.0
No manifests need updating before upgrading
No deprecated APIs that would be removed in v1.33.0

To be thorough before a k8s upgrade, we need to run all three commands:

detect-files
detect-helm
detect-all-in-cluster

---

Where to keep Helm chart values in Terraform projects

2026-02-26T13:02:00.001+00:00

If we use Terraform to deploy Helm charts, we might be using one of these strategies to keep chart values:

Values are in inline YAML string
Values in separate .yaml file
Values in separate YAML Template files (.yaml.tpl)
Use Helm's set for Dynamic Values
Multiple Values Files

(1) Values in inline YAML string

This is not ideal as problems with Inline YAML in Terraform include:

No syntax highlighting or validation - Easy to break YAML formatting
Hard to review in diffs - Changes are messy in PRs
Can't use standard tooling - No yamllint, Pluto, or other YAML tools
Mixing concerns - Infrastructure code mixed with application config
Escaping nightmares - Terraform string interpolation conflicts with Helm templating

Example:

resource "helm_release" "app" {

values = [<<-EOT

replicaCount: ${var.replicas}

image:

repository: myapp

tag: ${var.tag}

service:

type: LoadBalancer

EOT

]

}

(2) Separate Values Files

Keep values in YAML files, reference them in Terraform.

This is a better approach because:

Clean separation
Easy to validate with standard tools
Better diffs
Can use Pluto directly: pluto detect-files -d .

Example:

main.tf:

resource "helm_release" "my_app" {

name = "my-app"

chart = "my-chart"

repository = "https://charts.example.com"

values = [

file("${path.module}/helm-values.yaml")

]

}

(3) Templated Values Files

Use Terraform's templatefile() to inject dynamic values:

helm-values.yaml.tpl:

replicaCount: ${replica_count}

image:

repository: ${image_repo}

tag: ${image_tag}

ingress:

enabled: ${enable_ingress}

host: ${hostname}

main.tf:

resource "helm_release" "my_app" {

name = "my-app"

chart = "my-chart"

values = [

templatefile("${path.module}/helm-values.yaml.tpl", {

replica_count = var.replica_count

image_repo = var.image_repository

image_tag = var.image_tag

enable_ingress = var.enable_ingress

hostname = var.hostname

})

]

}

Pros:

Still gets variable injection
Can be validated as YAML (with placeholders)
Clean and readable

(4) Use Helm's set for Dynamic Values

Keep static config in files, override specific values:

resource "helm_release" "my_app" {

name = "my-app"

chart = "my-chart"

# Base values from file

values = [

file("${path.module}/helm-values.yaml")

]

# Override specific values dynamically

set {

name = "image.tag"

value = var.image_tag

}

set {

name = "replicaCount"

value = var.replica_count

}

set_sensitive {

name = "secret.password"

value = var.db_password

}

Pros:

Clear what's dynamic vs static
Base values file can be validated
Sensitive values handled properly

Here is the example how we can migrate inline YAML from the above to templated file:

helm-values.yaml:

image:

repository: myapp

service:

type: LoadBalancer

main.tf:

resource "helm_release" "app" {

values = [

file("${path.module}/helm-values.yaml")

]

set {

name = "replicaCount"

value = var.replicas

}

set {

name = "image.tag"

value = var.tag

}

Now we can run:

% pluto detect-files -f helm-values.yaml

(5) Multiple Values Files

We can layer our configuration:

resource "helm_release" "my_app" {

name = "my-app"

chart = "my-chart"

values = [

file("${path.module}/helm-values-base.yaml"),

file("${path.module}/helm-values-${var.environment}.yaml")

]

}

---

Introduction to Kubent (Kube No Trouble)

2026-02-26T11:07:00.002+00:00

Kubent (Kube No Trouble) [this link was the original repo, see comments below] is a tool which scans k8s cluster and reports resources that use deprecated or removed Kubernetes APIs, based on the target Kubernetes version. It’s especially useful before upgrading (e.g., EKS 1.32 → 1.33)

WARNING: Development at project's original repo (https://github.com/doitintl/kube-no-trouble) is not active anymore as the last commit was in January 2025. The original author announced here that they would be moving development to https://github.com/dark0dave/kube-no-trouble and that repo is ssemingly active as of today (last change was ) BUT https://github.com/dark0dave/kube-no-trouble/tree/301e5783904de5966f79b217a956651146630f50/pkg/rules/rego shows that rulesets only up to v1.32 were added (!).

Kube No Trouble relies on static Rego rule files in the repo. If new Kubernetes versions (e.g., >1.32) don’t have updated rules, then:

It won’t know about newly deprecated APIs
It won’t know about newly removed APIs
--target-version becomes unreliable for newer releases

For modern upgrades (especially 1.32 → 1.33+), kubent is no longer the safest tool.

To install it:

% sh -c "$(curl -sSL https://git.io/install-kubent)"

>>> kubent installation script <<<

> Detecting latest version

> Downloading version 0.7.3

Target directory (/usr/local/bin) is not writable, trying to use sudo

Password:

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

100 12.4M 100 12.4M 0 0 14.7M 0 --:--:-- --:--:-- --:--:-- 13.2M

> Done. kubent was installed to /usr/local/bin/.

To verify installation:

% kubent --version

7:48AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

To check CLI arguments:

% kubent --help

Usage of kubent:

-A, --additional-annotation strings additional annotations that should be checked to determine the last applied config

-a, --additional-kind strings additional kinds of resources to report in Kind.version.group.com format

-c, --cluster enable Cluster collector (default true)

-x, --context string kubeconfig context

-e, --exit-error exit with non-zero code when issues are found

-f, --filename strings manifests to check, use - for stdin

--helm3 enable Helm v3 collector (default true)

-k, --kubeconfig string path to the kubeconfig file

-l, --log-level string set log level (trace, debug, info, warn, error, fatal, panic, disabled) (default "info")

-o, --output string output format - [text|json|csv] (default "text")

-O, --output-file string output file, use - for stdout (default "-")

-t, --target-version string target K8s version in SemVer format (autodetected by default)

-v, --version prints the version of kubent and exits

pflag: help requested

It looks at default ~/.kube/config file in order to find the current context, otherwise use -k to specify kubeconfig at non-default location.

% kubent

7:59AM INF >>> Kube No Trouble `kubent` <<<

7:59AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

7:59AM INF Initializing collectors and retrieving data

7:59AM INF Target K8s version is 1.32.11-eks-ac2d5a0

7:59AM INF Retrieved 12 resources from collector name=Cluster

8:00AM INF Retrieved 361 resources from collector name="Helm v3"

8:00AM INF Loaded ruleset name=custom.rego.tmpl

8:00AM INF Loaded ruleset name=deprecated-1-16.rego

8:00AM INF Loaded ruleset name=deprecated-1-22.rego

8:00AM INF Loaded ruleset name=deprecated-1-25.rego

8:00AM INF Loaded ruleset name=deprecated-1-26.rego

8:00AM INF Loaded ruleset name=deprecated-1-27.rego

8:00AM INF Loaded ruleset name=deprecated-1-29.rego

8:00AM INF Loaded ruleset name=deprecated-1-32.rego

8:00AM INF Loaded ruleset name=deprecated-future.rego

Running kubent with no other arguments:

Connects to your current kube-context
Detects your cluster version automatically
Scans all namespaces
Compares resources against deprecations for that version

Before the upgrade to v1.33, we want kubent to scan the resources against that next k8s version so we need to specify it with --target-version:

% kubent --target-version=1.33

8:02AM INF >>> Kube No Trouble `kubent` <<<

8:02AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

8:02AM INF Initializing collectors and retrieving data

8:02AM INF Target K8s version is 1.33.0

8:02AM INF Retrieved 12 resources from collector name=Cluster

8:03AM INF Retrieved 361 resources from collector name="Helm v3"

8:03AM INF Loaded ruleset name=custom.rego.tmpl

8:03AM INF Loaded ruleset name=deprecated-1-16.rego

8:03AM INF Loaded ruleset name=deprecated-1-22.rego

8:03AM INF Loaded ruleset name=deprecated-1-25.rego

8:03AM INF Loaded ruleset name=deprecated-1-26.rego

8:03AM INF Loaded ruleset name=deprecated-1-27.rego

8:03AM INF Loaded ruleset name=deprecated-1-29.rego

8:03AM INF Loaded ruleset name=deprecated-1-32.rego

8:03AM INF Loaded ruleset name=deprecated-future.rego

---

Introduction to Grafana Loki

2026-02-23T11:59:00.001+00:00

Grafana Loki:

Log aggregation system. Like Prometheus, but for logs
Repo: https://github.com/grafana/loki

These are the notes from Loki Helm chart:

***********************************************************************

Welcome to Grafana Loki

Chart version: 6.31.0

Chart Name: loki

Loki version: 3.5.0

***********************************************************************

Tip:

Watch the deployment status using the command: kubectl get pods -w --namespace grafana-loki

If pods are taking too long to schedule make sure pod affinity can be fulfilled in the current cluster.

***********************************************************************

Installed components:

***********************************************************************

* gateway

* read

* write

* backend

***********************************************************************

Sending logs to Loki

***********************************************************************

Loki has been configured with a gateway (nginx) to support reads and writes from a single component.

You can send logs from inside the cluster using the cluster DNS:

http://loki-gateway.grafana-loki.svc.cluster.local/loki/api/v1/push

You can test to send data from outside the cluster by port-forwarding the gateway to your local machine:

kubectl port-forward --namespace grafana-loki svc/loki-gateway 3100:80 &

And then using http://127.0.0.1:3100/loki/api/v1/push URL as shown below:

curl \

-H "Content-Type: application/json" \

-XPOST \

-s "http://127.0.0.1:3100/loki/api/v1/push" \

--data-raw "{\"streams\": [{\"stream\": {\"job\": \"test\"}, \"values\": [[\"$(date +%s)000000000\", \"fizzbuzz\"]]}]}" \

-H X-Scope-OrgId:foo

Then verify that Loki did receive the data using the following command:

curl "http://127.0.0.1:3100/loki/api/v1/query_range" \

--data-urlencode 'query={job="test"}' \

-H X-Scope-OrgId:foo | jq .data.result

***********************************************************************

Connecting Grafana to Loki

***********************************************************************

If Grafana operates within the cluster, you'll set up a new Loki datasource by utilizing the following URL:

http://loki-gateway.grafana-loki.svc.cluster.local/

***********************************************************************

Multi-tenancy

***********************************************************************

Loki is configured with auth enabled (multi-tenancy) and expects tenant headers (`X-Scope-OrgID`) to be set for all API calls.

You must configure Grafana's Loki datasource using the `HTTP Headers` section with the `X-Scope-OrgID` to target a specific tenant.

For each tenant, you can create a different datasource.

The agent of your choice must also be configured to propagate this header.

For example, when using Promtail you can use the `tenant` stage. https://grafana.com/docs/loki/latest/send-data/promtail/stages/tenant/

When not provided with the `X-Scope-OrgID` while auth is enabled, Loki will reject reads and writes with a 404 status code `no org id`.

You can also use a reverse proxy, to automatically add the `X-Scope-OrgID` header as suggested by https://grafana.com/docs/loki/latest/operations/authentication/

For more information, read our documentation about multi-tenancy: https://grafana.com/docs/loki/latest/operations/multi-tenancy/

> When using curl you can pass `X-Scope-OrgId` header using `-H X-Scope-OrgId:foo` option, where foo can be replaced with the tenant of your choice.

EOT -> (known after apply)

---