My Public Notepad

Splitting One Domain Across GitHub Pages and Blogger: A Complete DNS Setup Guide

2026-08-07T12:53:08.305+01:00

If you own a single domain but want to run two separate things on it — say, a static landing page on GitHub Pages at the root domain, and a blog on Blogger at a subdomain — you'll quickly discover that the actual mechanics of DNS, custom domain verification, and HTTPS certificate provisioning are scattered across three different systems that don't talk to each other and don't always explain their errors clearly.

This post documents the full setup, end to end: what records you need, what each provider actually requires, and the specific errors you're likely to hit along the way (all of which I hit myself while setting this up).

The goal, concretely:

example.com (the apex/root domain) → a static site hosted on GitHub Pages
blog.example.com (a subdomain) → a Blogger blog, with its original xxxx.blogspot.com address redirecting to it
www.example.com → optionally also works, redirecting to the apex

I'll use bojankomazec.com as the running example, since that's the domain I set this up on.

1. The Big Picture: Three Systems, One Domain

Before touching any settings, it helps to understand the three layers involved and what each one owns:

Layer	What it controls	Where you configure it
DNS registrar (e.g. GoDaddy, Namecheap, Cloudflare)	Which records exist for your domain — A, CNAME, MX, etc. This is the actual "phone book" that tells the internet where to send traffic.	Your registrar's DNS management panel
GitHub Pages	Which domain(s) it will accept and serve your static site for, and whether it issues an HTTPS certificate for them	Repository → Settings → Pages
Blogger	Which domain(s) it will accept and serve your blog for, and whether it issues an HTTPS certificate for them	Blogger → Settings → Publishing

The critical thing to understand: DNS records alone are not enough. Both GitHub Pages and Blogger require you to also tell them, inside their own settings, which custom domain you're using. If DNS points to them but their own settings don't know about your domain, you'll get connection errors — not because DNS is wrong, but because the receiving service doesn't recognize the incoming hostname and refuses it.

This is the single most common source of confusion in this kind of setup: DNS can be 100% correct and the site can still fail to load, because DNS and application-level domain configuration are two separate steps that both have to be done.

2. DNS Records You'll Need

Here's the full set, using bojankomazec.com as the example domain, blog as the subdomain, and bojankomazec as the GitHub username (so the Pages repo is bojankomazec.github.io).

Apex domain → GitHub Pages

DNS doesn't allow a CNAME record at the apex/root of a domain (technically, because the apex must also hold other record types like SOA and often MX, and a hostname can't mix CNAME with anything else). So GitHub Pages publishes a fixed set of IP addresses, and you point the apex at those directly with A records:

Type: A    Host: @    Value: 185.199.108.153
Type: A    Host: @    Value: 185.199.109.153
Type: A    Host: @    Value: 185.199.110.153
Type: A    Host: @    Value: 185.199.111.153

(@ is the conventional way most DNS panels represent "the apex/root domain itself.") If your provider supports IPv6 and you want dual-stack support, GitHub also publishes AAAA records:

Type: AAAA    Host: @    Value: 2606:50c0:8000::153
Type: AAAA    Host: @    Value: 2606:50c0:8001::153
Type: AAAA    Host: @    Value: 2606:50c0:8002::153
Type: AAAA    Host: @    Value: 2606:50c0:8003::153

Important: check whether your registrar auto-created a placeholder A record for @ when the domain was registered (many do, often pointing to a parking page). If so, delete it — a leftover default record will conflict with the four you're adding.

www subdomain → GitHub Pages (optional)

If you also want www.example.com to work (redirecting to the apex once GitHub verifies both), add:

Type: CNAME    Host: www    Value: bojankomazec.github.io.

Note this points to your GitHub Pages hostname (<username>.github.io), not to a URL, and not to the repo's github.com page. A CNAME value is always a hostname, never a full URL with https:// or a path.

Blog subdomain → Blogger

Type: CNAME    Host: blog    Value: ghs.google.com.

This is the one that trips people up: it's tempting to point this directly at your xxxx.blogspot.com address, since that's "where the blog lives." Don't do that. Blogger's custom-domain routing works through a shared Google endpoint (ghs.google.com) that inspects the incoming hostname and routes it to the correct blog based on what's configured in your Blogger account. Pointing the CNAME straight at your blogspot address sends the request to blogspot-specific infrastructure that doesn't know how to handle a request arriving under a different hostname — the connection gets abruptly closed rather than serving anything (ERR_CONNECTION_CLOSED in Chrome).

Blogger domain verification CNAME

Separately from the routing CNAME above, Blogger needs to confirm you actually own the subdomain before it'll accept it. When you enter your custom domain into Blogger's settings, it will generate a unique verification CNAME — something like:

Type: CNAME    Host: <random-string>    Value: gv-<random-string>.dv.googlehosted.com.

This value is generated per-domain and per-account — you can't predict or reuse it, you have to copy it exactly from Blogger's settings screen after entering your custom domain. Add it as its own DNS record alongside the routing CNAME.

Gotcha: if you ever change which subdomain Blogger is using (e.g. you originally set it up as www.example.com and later switch to blog.example.com), the old verification CNAME becomes stale and should be deleted — Blogger will issue a new one specific to the new subdomain when you re-save.

Full example record set

A       @        185.199.108.153
A       @        185.199.109.153
A       @        185.199.110.153
A       @        185.199.111.153
CNAME   www      bojankomazec.github.io.
CNAME   blog     ghs.google.com.
CNAME   <verify> gv-xxxxxxxxxxxx.dv.googlehosted.com.

3. The GitHub Pages Side

In your <username>.github.io repository, go to Settings → Pages.
Under Custom domain, enter your apex domain (e.g. bojankomazec.com — no www, no https://).
Save. GitHub will write a file literally named CNAME (no extension) into the root of your repository, containing just the domain name. This file is how GitHub Pages knows which domain to serve your content for — DNS alone doesn't tell it this.
GitHub will attempt to verify your DNS. If your A records are already correct, this usually succeeds quickly. If you see a DNS valid for primary message alongside a separate www.example.com is improperly configured warning — that's normal. The www check is a secondary, automatic check GitHub runs to see if it can also offer www support and a unified certificate; it doesn't block your primary domain from working. If your primary domain error clears and only the www one remains, your site is live.
Once verified, tick Enforce HTTPS (this option is often greyed out until GitHub finishes issuing a certificate — see the HTTPS section below).

4. The Blogger Side

In Blogger, go to Settings → Publishing.
Under Custom domain, enter your blog subdomain (e.g. blog.bojankomazec.com).
Save. Blogger will likely reject the save the first time and show you the verification CNAME described above. Copy it exactly.
Add both the routing CNAME (ghs.google.com) and the verification CNAME to your DNS.
Wait for propagation (see below), then go back and save again in Blogger. It should now accept the domain.
Check the Redirect domain setting on the same page carefully — see the dedicated warning about this below, it's a common trap.
Look for an HTTPS availability toggle further down the page. It usually reads "Status: Unknown" for a while after you first set up the domain — this is Blogger's certificate issuance still in progress, not an error (see HTTPS section).

The "Redirect domain" trap

Blogger's Publishing settings include a "Redirect domain" option, something like "Redirect example.com to blog.example.com." This setting, if enabled, makes Blogger redirect your apex domain to your blog. If your apex domain is meant to serve something else entirely (like a GitHub Pages landing page, as in this setup), this option will silently hijack it — visitors to example.com will get redirected to your blog instead of seeing your landing page, even though your DNS and GitHub Pages configuration are both completely correct.

This setting can also change target without obviously telling you: if you initially set up Blogger's custom domain as www.example.com and later switch it to blog.example.com, the "Redirect domain" checkbox may still be checked, just silently re-pointed at whatever your current custom domain is. If you don't want your apex domain touched by Blogger at all, make sure this is switched off, and re-check it any time you change Blogger's custom domain setting.

5. HTTPS and Certificates: Why It Takes a While

Both GitHub Pages and Blogger use automated certificate issuance (GitHub uses Let's Encrypt; Google's infrastructure handles Blogger's) that kicks in after your domain passes DNS verification — not before, and not instantly.

What this means in practice:

Your site can be fully reachable over plain http:// while https:// still fails, sometimes with a confusing certificate error, for anywhere from a few minutes to around 24–48 hours after you first set up or change a custom domain.
The specific error you'll likely see in Chrome during this window is something like:

  Your connection is not private
  net::ERR_CERT_COMMON_NAME_INVALID
  This server could not prove that it is www.example.com;
  its security certificate is from *.github.io

This happens because the connection is reaching GitHub's infrastructure, but GitHub hasn't yet issued a certificate specifically covering your hostname, so it falls back to serving its generic wildcard certificate — which your browser correctly flags as invalid for your domain.

Don't repeatedly remove and re-add the custom domain while waiting — each change can reset the certificate issuance queue and make the wait longer, not shorter.
Once issuance completes, https:// starts working with no further action needed on your part.

A red herring worth knowing about: antivirus HTTPS scanning

If you inspect your site's certificate in Chrome and see something unexpected — like the certificate authority being your antivirus software (Avast, Kaspersky, and others do this) rather than Let's Encrypt or Google — this is not a sign of a compromised or misconfigured site. Many antivirus suites include an "HTTPS scanning" or "Web Shield" feature that intercepts encrypted connections locally, decrypting and re-encrypting them with a certificate the antivirus generated and installed into your system's trust store, in order to scan traffic for threats.

This substitution happens for every HTTPS site you visit, not just yours, so it's not diagnostic of anything wrong with your setup. To see the actual certificate your host issued, either temporarily disable the antivirus's HTTPS scanning feature, or check from an external tool that doesn't go through your local machine at all — such as SSL Labs' SSL Test, which queries your server directly.

6. Troubleshooting Toolkit

A handful of tools make this whole process far less guess-and-check:

dig <domain> <record-type> (Mac/Linux, or Windows via WSL) — queries DNS directly and shows you exactly what's published, bypassing any local caching. E.g. dig bojankomazec.com A or dig blog.bojankomazec.com CNAME.
nslookup <domain> — similar, built into Windows natively.
dnschecker.org — checks DNS propagation across many resolvers worldwide simultaneously, which is invaluable for figuring out whether a record change has actually rolled out everywhere yet, or is still mid-propagation.
SSL Labs SSL Test — inspects a site's actual certificate chain independent of your local browser/antivirus interference.

General debugging approach when something doesn't load:

Confirm the DNS record itself is correct and propagated (dnschecker.org).
Confirm the receiving service (GitHub Pages / Blogger) has the domain configured in its own settings, not just DNS.
Try plain http:// before https:// — this isolates whether the problem is routing/configuration (affects both) or just certificate issuance (only affects https).
If a certificate error shows an unexpected issuer (like an antivirus name instead of Let's Encrypt/Google), suspect local HTTPS interception before suspecting your DNS setup.

7. Common Errors and What They Actually Mean

Error	What it usually means
`InvalidCNAMEError` (GitHub)	The named subdomain's CNAME doesn't point where GitHub expects, or DNS hasn't propagated to the resolver GitHub's checker used yet. Verify with dnschecker.org before assuming misconfiguration.
`NotServedByPagesError` (GitHub)	DNS looks fine, but the domain isn't yet resolving to GitHub Pages from GitHub's own check — often transient/propagation-related.
`ERR_CERT_COMMON_NAME_INVALID`	The connection reached the right server, but no certificate has been issued for this specific hostname yet — a provisioning delay, not a misconfiguration.
`ERR_CONNECTION_CLOSED` (on a Blogger subdomain)	Usually means the CNAME points somewhere that doesn't recognize the incoming hostname — most often, pointing directly at a `.blogspot.com` address instead of `ghs.google.com`.
A domain unexpectedly redirecting somewhere else entirely	Check Blogger's "Redirect domain" setting — it may be silently redirecting your apex domain to your blog.
Certificate issuer shown as an antivirus company	Local HTTPS scanning/interception by your antivirus software — check via an external tool instead.

8. Summary Checklist

Four A records for the apex domain, pointing at GitHub Pages' published IPs
(Optional) CNAME for www pointing at <username>.github.io
CNAME for your blog subdomain pointing at ghs.google.com (not directly at your .blogspot.com address)
Blogger's unique verification CNAME, copied exactly from its settings screen
Custom domain entered in GitHub Pages settings (creates the repo's CNAME file)
Custom domain entered in Blogger's Publishing settings
Blogger's "Redirect domain" setting switched off, unless you deliberately want your apex redirected to your blog
Patience for HTTPS certificate issuance (minutes to ~48 hours) on both platforms
Verify using http:// first, then https://, and cross-check unexpected certificate issuers against local antivirus software before assuming a server-side problem

Once all of this settles, you end up with a clean split: your apex domain serving whatever static site or landing page you want full control over, and a subdomain serving your blog — both fully on their own custom domains, both secured with HTTPS, and both surviving independently of where the underlying content is actually hosted.

kube-state-metrics

2026-08-05T17:17:41.286+01:00

kube-state-metrics (KSM) is an official Kubernetes add-on agent that listens to the Kubernetes API server and generates Prometheus-format metrics about the state of Kubernetes objects (such as Pods, Deployments, Nodes, StatefulSets, and CronJobs).

Unlike agent metrics collectors like cAdvisor or node_exporter, KSM does not measure resource usage (like CPU usage, memory consumption, or disk I/O). Instead, it translates the raw state of objects in the Kubernetes API into structured metrics.

Core Concept: Usage vs. State

cAdvisor / Node Exporter (Usage): "How much CPU is Pod X consuming right now?"

kube-state-metrics (State): "How many replicas are ready in Deployment Y?", "When did CronJob Z last succeed?", "What phase is Pod A in?"

How kube-state-metrics Exposes Metrics

┌─────────────────────────┐

│ Kubernetes API Server │

└────────────┬────────────┘

│ Watch / List (Informer Cache)

▼

┌─────────────────────────┐

│ kube-state-metrics │ ─── (Generates OpenMetrics in-memory)

└────────────┬────────────┘

│ HTTP GET /metrics (Port 8080)

▼

┌─────────────────────────┐

│ Prometheus Server │

└─────────────────────────┘

1. Consuming the API via Kubernetes Informers

KSM connects directly to the Kubernetes API server using standard Client-GO Informers.

Instead of constantly polling the API server with heavy requests, KSM maintains an in-memory cache updated in real-time via long-polling watch streams.
When a resource changes (e.g., a Deployment scales down or a Pod enters CrashLoopBackOff), the local cache updates instantly.

2. Generating Prometheus OpenMetrics In-Memory

KSM parses the fields of the Kubernetes object specs and status subresources (such as .status.phase, .spec.replicas, .status.conditions) and converts them directly into Prometheus gauge/counter metrics.

3. Serving via HTTP Endpoint

KSM hosts a lightweight HTTP server (default port :8080 at path /metrics).

It does not store metrics over time, nor does it push metrics out.

When a Prometheus server scrapes KSM's /metrics endpoint, KSM reads its in-memory snapshot, formats the raw text payload, and streams it back.

Key Technical Properties

Property

Details

Port & Path

Port 8080 (/metrics for cluster state), Port 8081 (/metrics for KSM internal performance telemetry)

Data Format

Prometheus text format / OpenMetrics standard

Metric Types

Predominantly Gauge metrics representing instantaneous state (1 or 0 state representations)

Scaling

Supports Horizontal Sharding and resource filtering (--resources, --namespaces) for large clusters

Example Metric Output

A scrape request to kube-state-metrics for a Deployment produces plain-text metrics like this:

# HELP kube_deployment_spec_replicas The desired number of pods declared in the deployment spec.

# TYPE kube_deployment_spec_replicas gauge

kube_deployment_spec_replicas{namespace="default",deployment="api-server"} 3

# HELP kube_deployment_status_replicas_available The number of available pods created by the deployment.

# TYPE kube_deployment_status_replicas_available gauge

kube_deployment_status_replicas_available{namespace="default",deployment="api-server"} 2

Do we need kube-state-metrics if we have Prometheus deployed in the cluster?

Yes, you still need kube-state-metrics (KSM) even if Prometheus is deployed in your cluster.

Prometheus is the engine that collects, stores, and queries time-series data, while kube-state-metrics is the agent that generates metrics about the state of Kubernetes API objects.

Prometheus does not natively inspect Kubernetes API objects on its own—it relies on exporters like KSM to expose that data.

What Prometheus collects out-of-the-box vs. with KSM?

Metric Source

Responsible Agent

Examples of Metrics Provided

Node / OS Metrics

node_exporter

Node CPU utilization, RAM usage, disk space, network traffic.

Container Usage

cAdvisor (built into kubelet)

Container CPU throttling, memory usage (RSS), network bytes per container.

Control Plane

kube-apiserver, etcd, coredns

API latency, request counts, etcd commit durations.

Kubernetes State

kube-state-metrics

Deployment replica counts, CronJob last success times, Pod restart counts, pending PVCs, ingress statuses, TLS certificate secret expiration.

What happens without kube-state-metrics?

If you run Prometheus without KSM, you lose visibility into the high-level health and configuration of your cluster resources. You will not be able to:

Alert on Deployment Health: You won't know if spec.replicas (desired) doesn't match status.available_replicas.
Track Job / CronJob Success: Metrics like kube_cronjob_status_last_successful_time or kube_job_status_failed won't exist.
Monitor Pod Lifecycle States: You won't have metrics for Pods stuck in Pending, CrashLoopBackOff status codes, or ImagePullBackOff.
Track Resource Requests vs. Limits: You won't be able to compare requested CPU/memory (kube_pod_container_resource_requests) against node capacity to measure cluster overcommit.

How They Work Together

┌─────────────────────────────────────────────────────────┐

│ Kubernetes API │

└────────────────────────────┬────────────────────────────┘

│ Watch API Objects

▼

┌───────────────────────┐

│ kube-state-metrics │

└───────────┬───────────┘

│ Exposes /metrics

▼

┌───────────────────────┐

│ Prometheus Server │ ◄── (Scrapes KSM)

└───────────────────────┘

KSM listens to the API server and generates metrics representing resource states.
Prometheus scrapes KSM's /metrics endpoint along with cAdvisor, node-exporter, and your application endpoints.
Prometheus evaluates Alertmanager rules and stores the metrics for Grafana dashboards.

Standard Stack Setup

In standard Kubernetes monitoring deployments (such as the kube-prometheus-stack Helm chart or Prometheus Operator), kube-state-metrics is included by default as a core component alongside Prometheus and Alertmanager.

How to install kube-state-metrics?

kube-state-metrics (KSM) is typically deployed using one of four common approaches:

1. As Part of the kube-prometheus-stack (Most Common)

If you are setting up cluster monitoring with Prometheus and Grafana, kube-state-metrics is usually installed automatically as a bundled sub-chart.

Using the kube-prometheus-stack Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

helm install prometheus-stack prometheus-community/kube-prometheus-stack

kube-state-metrics runs out of the box with ServiceMonitors already configured.

2. Standalone Helm Chart

If you already have a Prometheus instance or another monitoring agent (like Datadog, New Relic, or Grafana Alloy) and just need KSM running, you can deploy the official standalone Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

helm install kube-state-metrics prometheus-community/kube-state-metrics --namespace kube-system

3. Native Kustomize / Standard Manifests (kubectl apply)

For gitops workflows or lightweight clusters where Helm isn't used, you can apply the official manifests maintained directly in the upstream Git repository using Kustomize:

git clone https://github.com/kubernetes/kube-state-metrics.git

cd kube-state-metrics

kubectl apply -k examples/standard

Note: This creates the necessary ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, and Service in the kube-system namespace.

4. Cloud Managed Kubernetes Add-ons

On managed platforms (like GKE, EKS, or AKS), kube-state-metrics is often enabled as a one-click add-on or managed component integrated into cloud-native observability services (e.g., Google Cloud Observability, AWS CloudWatch Container Insights, or Azure Monitor).

What Gets Created in the Cluster?

Regardless of how you deploy it, KSM will create:

ServiceAccount & ClusterRole/Binding: Grants read-only access (list, watch, get) to cluster API resources.
Deployment: Runs the kube-state-metrics container.
Service: Exposes the HTTP endpoint (usually on port 8080 at /metrics).

How to inspect which default metrics KSM exposes or how to disable unused ones?

Understanding which metrics kube-state-metrics (KSM) emits and how to prune unwanted metrics is key to controlling Prometheus ingestion volume and storage overhead.

Part 1: Inspecting Exposed Metrics

You can inspect the metrics exposed by KSM using three different methods:

1. Official Documentation Reference

The KSM GitHub repository contains generated documentation for every supported Kubernetes API group.

Standard Docs: kube-state-metrics/docs contains dedicated files for each resource (e.g., pod-metrics.md, cronjob-metrics.md).
Resource Status: The docs categorize metrics as STABLE (default), EXPERIMENTAL (alpha/beta features), or DEPRECATED.

2. Direct In-Cluster Scraping (curl / port-forward)

To see the exact payload KSM generates in your active cluster, port-forward to the KSM service and fetch /metrics:

# Port-forward the KSM service locally

kubectl port-forward svc/kube-state-metrics -n monitoring 8080:8080

# In a separate terminal, curl the metrics endpoint

curl http://localhost:8080/metrics

Filter for specific resource types using grep:

# View all Pod-related metrics exposed by KSM

curl -s http://localhost:8080/metrics | grep "^kube_pod_"

# View unique metric family names

curl -s http://localhost:8080/metrics | grep -v "^#" | cut -d'{' -f1 | sort -u

3. Prometheus Metric Explorer

In Prometheus, run the following query in the Expression Browser to list all distinct metric names harvested from the KSM job:

count by (__name__) ({job="kube-state-metrics"})

Part 2: Disabling or Filtering Unused Metrics

There are two primary ways to reduce metric volume: at the source (KSM Flags) or at the scraper (Prometheus Relabeling).

Method 1: Filtering at Source via KSM Flags (Recommended)

You can instruct KSM to only collect or expose specific resources or metric families using command-line arguments. This reduces both KSM memory utilization and network output.

--resources: Restricts KSM to watch specific Kubernetes objects only (e.g., ignore configmaps or secrets).

--metric-allowlist: Explicitly lists metric families to expose (all others are dropped).

--metric-denylist: Drops specific metric families while keeping the rest.

Setting via Helm (values.yaml):

# Helm configuration for prometheus-community/kube-state-metrics

resources:

# Watch only specific K8s API resources

resources:

- pods

- deployments

- statefulsets

- cronjobs

- nodes

# Disable specific metric families across enabled resources

extraArgs:

- --metric-denylist=kube_configmap_info,kube_secret_info,kube_pod_labels

Setting via Argo CD Application Manifest:YAMLapiVersion: argoproj.io/v1alpha1

kind: Application

metadata:

namespace: argocd

spec:

# ... (repo configuration omitted)

source:

helm:

extraArgs:

- --resources=pods,deployments,cronjobs,nodes

- --metric-denylist=kube_pod_created,kube_pod_completion_time

Method 2: Dropping Metrics at Prometheus Ingestion

If you do not manage KSM CLI flags directly, configure Prometheus or Grafana Agent/Alloy to drop metrics during the scrape phase using metric_relabel_configs:

Prometheus CustomResource (Prometheus / ServiceMonitor):

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

namespace: monitoring

spec:

endpoints:

- port: http-metrics

metricRelabelings:

# Drop entire metric families matching a regex pattern

- action: drop

sourceLabels: [__name__]

regex: "(kube_configmap_.*|kube_secret_.*|kube_lease_.*)"

# Drop high-cardinality container state metrics if unused

- action: drop

sourceLabels: [__name__]

regex: "kube_pod_container_state_started"

kube_job_status_failed Source Filtering vs. Scraper DroppingFeatureKSM CLI Flags (--resources, --metric-denylist)Prometheus metric_relabel_configsKSM Memory UsageLower (KSM does not store or process ignored objects in memory)Unchanged (KSM processes objects regardless)Network TrafficLower (HTTP payload size is smaller)Unchanged (KSM transmits full payload; Prometheus discards post-fetch)Prometheus DB StorageLowerLowerConfig PlacementKSM Deployment manifestPrometheus scrape configuration

Prometheus Metrics Exposed by kube-state-metrics

kube_job_status_succeeded

kube_job_status_succeeded is a Prometheus metric exposed by kube-state-metrics that tracks the status of Kubernetes Jobs.

It returns a gauge value representing whether a Job execution succeeded, partitioned by label selectors.

Metric Breakdown

Metric Name: kube_job_status_succeeded
Type: Gauge
Value:

1: The Job has completed successfully.
0: The Job is in progress, failed, or has not succeeded.

Key Labels

Label Description

==== =========

job_name The name of the Kubernetes Job resource

namespace The namespace where the Job resides

condition Condition status (typically true when checking successful completion)

Common PromQL Queries

1. List All Currently Succeeded Jobs

kube_job_status_succeeded{condition="true"} == 1

2. Detect Jobs That Failed or Did Not Succeed

kube_job_status_succeeded{condition="true"} == 0

3. Alerting Rule: Job Failure

Alert when a Job has completed its execution attempt without succeeding:

- alert: KubernetesJobFailed

expr: kube_job_status_failed{condition="true"} == 1

for: 5m

labels:

severity: warning

annotations:

summary: "Kubernetes Job {{ $labels.namespace }}/{{ $labels.job_name }} failed"

4. Track Job Completion Rate Over Time

Calculate the rate of successful Job completions across a cluster:

sum(increase(kube_job_status_succeeded{condition="true"}[1h]))

kube_job_status_failed

kube_job_status_failed is a Prometheus metric exposed by kube-state-metrics that tracks whether a Kubernetes Job has failed to execute successfully.

It returns a gauge value representing the failure status of a Job, partitioned by label selectors.

Metric Breakdown

Metric Name: kube_job_status_failed
Type: Gauge
Value:

1: The Job reached its failure condition (e.g., exceeded backoffLimit or failed execution).
0: The Job has not failed (it is currently running, pending, or succeeded).

Key Labels

Label Description

==== =========

job_name The name of the Kubernetes Job resource

namespace The namespace where the Job resides

condition Condition status (typically true when evaluating an active failure state)

reason The reason for failure if populated (e.g., BackoffLimitExceeded, DeadlineExceeded)

Common PromQL Queries

1. List All Currently Failed Jobs

kube_job_status_failed{condition="true"} == 1

2. Count Failed Jobs by Namespace

sum by (namespace) (kube_job_status_failed{condition="true"} == 1)

3. Prometheus Alerting Rule for Job Failures

Alert when a Job has reached a failed condition and remained failed for 5 minutes:

- alert: KubernetesJobFailed

expr: kube_job_status_failed{condition="true"} == 1

for: 5m

labels:

severity: warning

annotations:

summary: "Kubernetes Job {{ $labels.namespace }}/{{ $labels.job_name }} failed"

description: "Job {{ $labels.job_name }} in namespace {{ $labels.namespace }} failed to complete."

4. Filter CronJobs / Generated Jobs by Prefix

If Jobs are generated dynamically by CronJobs, group or filter using regular expressions:

kube_job_status_failed{condition="true", job_name=~"nightly-backup-.*"} == 1

kube_cronjob_status_last_successful_time

kube_cronjob_status_last_successful_time is a gauge metric exposed by kube-state-metrics.

It records the Unix timestamp (in seconds) of when a Kubernetes CronJob last completed execution successfully.

Metric Details

Attribute:Value

Exporter: kube-state-metrics
Metric Type:Gauge
Value: Unix timestamp (seconds) or omitted if the CronJob has never completed
Labels: cronjob, namespace

Common PromQL Use Cases

1. Time Since Last Successful Run

Calculates how many seconds have elapsed since the CronJob last succeeded:

time() - kube_cronjob_status_last_successful_time

2. Alert on CronJob Failure or Missed Schedule

Alerts if the time since the last successful execution exceeds a target threshold (e.g., 1 day / 86,400 seconds):

(time() - kube_cronjob_status_last_successful_time) > 86400

3. Alerting when Schedule Run Failed

Compares the last scheduled time (kube_cronjob_status_last_schedule_time) against the last successful time. If last_schedule_time is greater than last_successful_time, the most recent run failed or is taking abnormally long:

kube_cronjob_status_last_schedule_time - kube_cronjob_status_last_successful_time > 0

4. Filter out Suspended CronJobs

Combine with kube_cronjob_spec_suspend to suppress false alerts on intentionally paused jobs:

(

(time() - kube_cronjob_status_last_successful_time) > 86400

) and on (cronjob, namespace) (

kube_cronjob_spec_suspend == 0

)

Monitoring term: Dead-man's switch

2026-08-04T15:00:41.676+01:00

It's monitoring inverted: instead of alerting when you observe something bad, you alert when you stop observing something good.

Example: ping on a successful run. If run is unsuccessful, monitoring catches missing ping and triggers alert.

The name comes from industrial safety — the lever on a train's throttle or a chainsaw that has to be actively held down. If the operator dies or lets go, the machine stops. Safety is the default state; it takes continuous positive action to keep running.

Normal alerting is presence-based. Something goes wrong, it emits a signal, you alert on the signal: error rate spikes, latency crosses a threshold, a pod enters CrashLoopBackOff. It works well when failures are noisy.

A dead-man's switch is absence-based. The healthy system periodically says "still fine." You alert when that message doesn't arrive on time.

Kubernetes Debugging Scenario: Node.JS CronJob dies with a V8 JavaScript heap OOM

2026-08-04T13:33:06.648+01:00

Problem Scenario

A Node.js batch job running as a Kubernetes CronJob aborts with FATAL ERROR: Ineffective mark-compacts near heap limit at ~4 GB. No NODE_OPTIONS, no resources block. Each scheduled run leaves several failed pods behind.

Knowledge required to fix the problem (Q&A)

Detailed Q&A

1. Node.js / V8 memory model

Q: What does --max-old-space-size actually control, and what does it not control?

It caps V8's old space — the long-lived generation of the JS heap. It does not cap new space (--max-semi-space-size), code space, large object space, or external/off-heap memory such as Buffer and ArrayBuffer allocations, native addon memory, thread-pool stacks, or glibc malloc arenas. So a process with a 6 GB old-space ceiling can easily have an RSS well above 6 GB.

V8 is Google's open source high-performance JavaScript and WebAssembly engine, written in C++. It is used in Chrome and in Node.js, among others.

--max-old-space-size sets the maximum memory limit (in megabytes) allocated to the Old Generation heap space inside V8, the JavaScript engine powering Node.js.

When V8 allocates memory for your application, it divides the JavaScript heap into distinct regions based on object lifecycle. This flag configures the largest region where long-lived objects reside.

What It Measures & Controls

--max-old-space-size explicitly caps memory allocated for:

Old Generation JavaScript Objects: Objects, arrays, functions, closures, and strings that have survived initial garbage collection cycles in the Young Generation space and were promoted to the Old Generation.
Old Pointer Space & Old Data Space: Regions holding objects that contain pointers to other objects and raw data (like numbers or unboxed scalars).

What It Does NOT Control

A common misconception is that --max-old-space-size caps the entire Resident Set Size (RSS) or system memory footprint of your Node.js process. It does not limit:
Node.js Buffers (ArrayBuffers): Since Node.js v8.0+, binary Buffer allocations use off-heap C++ memory backing stores (ArrayBuffer). While the JavaScript wrapper object lives on the V8 heap, the underlying raw bytes do not count toward the old space limit.
Native C++ Allocations: Memory used by native C++ add-ons, libuv threads, or external libraries compiled into Node.
Other V8 Heap Spaces:

New Space (Nursery/Young Generation): Where new allocations land (--max-semi-space-size).
Code Space: JIT-compiled bytecode and machine code.
Map/Cell Spaces: V8 internal hidden classes and metadata.

Call Stack Memory: Memory used by execution contexts and local variables on the stack.

Because of off-heap memory, a Node.js process with --max-old-space-size=2048 (2 GB) can easily consume 3 GB or more of total system RAM (RSS).

What Happens When the Limit Is Reached

Aggressive Garbage Collection: As old space usage approaches the limit, V8 triggers blocking, high-overhead Mark-Sweep-Compact garbage collection cycles to reclaim dead objects.
Process Crash: If V8 cannot free enough memory to fit the next allocation below the configured threshold, Node.js crashes with a fatal error:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Default Values & Usage

Default Behavior: In modern Node.js versions, V8 dynamically sets the limit based on total available system RAM—typically around 2 GB to 4 GB on 64-bit systems if unspecified.

Command Line Flag:

node --max-old-space-size=4096 app.js

Environment Variable:

export NODE_OPTIONS="--max-old-space-size=4096"

Q: Why did the process die at ~4064 MB when nobody configured a heap limit?

V8 picks a default heap ceiling from the memory it believes is available, and on 64-bit builds that lands at roughly 4 GB. The Mark-Compact 4064.3 MB line in the log is the giveaway that it hit that default ceiling rather than any limit you set.

Q: Why doesn't Node just size its heap to the container's memory limit?

Historically V8 read host RAM, not the cgroup limit, so a Node process in a 512 Mi container would happily set a multi-gigabyte heap and get OOM-killed. Newer Node versions do consult cgroup constraints, but the behaviour varies by version — which is why the defensive answer is always to set the flag explicitly rather than rely on auto-detection.

Q: The workload starts with npm run start. Does setting NODE_OPTIONS in the container env actually reach the Node process?

Yes — NODE_OPTIONS is an environment variable, so it's inherited by every child process npm spawns. Two caveats worth naming: it also applies to the npm wrapper process itself (harmless, just an extra reservation), and not every V8 flag is permitted inside NODE_OPTIONS. The alternative is passing the flag in the npm script itself, which is more surgical but easier to lose.

2. Diagnosis: which kind of OOM is this?

Q: How do you tell a V8 heap OOM from a kernel OOMKill from a kubelet eviction?

Signal	V8 heap OOM	OOMKilled	Evicted
Log line	`FATAL ERROR: Ineffective mark-compacts near heap limit`	none from the app — killed mid-flight	none from the app
Signal / exit	`SIGABRT`, exit 134	`SIGKILL`, exit 137	pod deleted
Pod status	`Error`	`OOMKilled` in `lastState.terminated.reason`	`Failed`, reason `Evicted`
Fix direction	raise heap ceiling or reduce allocation	raise container limit	set requests so you aren't the first target

The ticket's evidence — the mark-compact message plus signal SIGABRT — puts it firmly in column one. That matters because raising the container limit alone would have changed nothing: V8 would still have aborted at 4 GB.

Q: How would you size the flag rather than guessing?

Instrument before you tune. --trace-gc shows the heap trajectory over the run; process.memoryUsage() sampled periodically distinguishes heapUsed from external; --heapsnapshot-near-heap-limit=1 writes a snapshot right before the abort that you can open in Chrome DevTools to find the retaining structure. That tells you whether the working set is genuinely ~6 GB or whether one unbounded array is the whole problem.

Q: Is raising the heap the right fix at all?

Usually it's a mitigation, not a fix. A benchmark job whose memory scales with input size will hit any ceiling you pick — the durable fix is streaming, batching, or paginating so peak memory is bounded by chunk size rather than dataset size. Raising the flag is defensible as a stopgap; the honest version says so in the ticket and files the follow-up. Note that in this case the real numbers came out at 10 GB heap with a 5-hour runtime, which is a fairly loud hint that the algorithm is the underlying issue.

3. Kubernetes resource management

Q: What's the difference between a memory request and a memory limit?

The request is what the scheduler reserves — it decides which node the pod fits on and is the baseline the kubelet uses when deciding who to evict. The limit is enforced at runtime by the cgroup; exceed it and the kernel OOM-kills the container. Memory, unlike CPU, is incompressible: there's no throttling, only killing.

Q: What QoS class does a pod with no resources block get, and why does that matter here?

BestEffort — the first thing evicted under node memory pressure, and it contributes nothing to the scheduler's accounting so the node can be oversubscribed into pressure in the first place. Setting requests equal to limits gives Guaranteed; requests below limits gives Burstable.

Q: How do you choose the relationship between the heap flag and the container limit?

Limit strictly above heap ceiling, with headroom for everything --max-old-space-size doesn't cover — off-heap buffers, native memory, the npm and node process overhead, plus GC working room. The ticket proposed 6 GB heap under a 7 Gi limit; what actually shipped was 10 GB heap under a 12 Gi limit. Too tight and you convert a clean SIGABRT into a much harder-to-debug OOMKill.

Q: What's the risk of setting limits.memory well above requests.memory?

You're overcommitting the node. It schedules against the request but can consume up to the limit, so several such pods on one node can drive it into memory pressure and trigger evictions of unrelated workloads. Matching them costs you scheduling flexibility but makes the blast radius predictable.

Q: You set requests.memory: 8Gi and the pod never starts. What's your first check?

Whether any node has 8 Gi of allocatable memory free — allocatable is capacity minus kube-reserved, system-reserved, and eviction thresholds. The pod sits Pending with an Insufficient memory scheduling event. Big-request batch jobs are a classic case for a dedicated or autoscaling node pool.

4. CronJobs and Jobs

Q: What produced seven Error pods from one scheduled run?

backoffLimit retries on failure, and a deterministic OOM fails identically every time — so the Job burned through its retries producing one dead pod each. The fix applied was a podFailurePolicy that fails the Job on the application's exit code instead of retrying, plus ttlSecondsAfterFinished so finished Jobs get garbage-collected rather than accumulating.

Q: What does podFailurePolicy require to work?

restartPolicy: Never on the pod template, and rules matching on either container exit codes (onExitCodes) or pod conditions (onPodConditions, e.g. DisruptionTarget). Actions are FailJob, Ignore, Count, and FailIndex. The point is distinguishing retryable infrastructure failures from deterministic application failures — retrying a heap OOM six times is pure waste.

Q: Which CronJob fields govern history and overlap?

successfulJobsHistoryLimit / failedJobsHistoryLimit for retained Job objects, ttlSecondsAfterFinished on the Job for automatic cleanup, concurrencyPolicy (Allow / Forbid / Replace) for overlapping runs, startingDeadlineSeconds for missed schedules, and activeDeadlineSeconds as a wall-clock kill switch. For a job that runs five hours, concurrencyPolicy: Forbid deserves a hard look.

Q: The schedule is 30 10 */14 * *. Does that run every 14 days?

No — and this is the trap. Step values in day-of-month are evaluated within each month, so it fires on the 1st, 15th, and 29th, then resets. The gap between the 29th and the following 1st is two or three days, not fourteen. Genuine "every N days" needs an external scheduler or a daily run that no-ops based on a stored timestamp.

5. Container memory accounting

Q: When you read a container's memory usage, what are you actually seeing?

Under cgroup v2 the kubelet reports working set derived from memory.current minus inactive file cache; memory.max is the hard limit. Crucially memory.current includes page cache, so a process doing heavy file I/O can look alarming without any anonymous-memory problem. RSS is anonymous plus mapped pages for the process specifically, and glibc often doesn't return freed memory to the OS — so RSS is sticky and lags real usage downward.

Q: Why is container_memory_rss a poor alerting signal for some workloads?

Because it only captures what lives in RSS. For a JVM or Node process the heap is anonymous memory and RSS tracks it reasonably; for something like Percona MongoDB, where WiredTiger's cache sits in the OS page cache rather than RSS, the metric is structurally blind to the thing you care about — you want cache fill percentage instead. Matching the metric to the workload's memory architecture is the actual skill.

6. Verification

Q: How do you prove the fix worked?

Trigger a manual run (kubectl create job --from=cronjob/experience-benchmarks) and confirm the Job reaches Complete with no SIGABRT and no Error pods. Then compare peak usage against the limit — completing at 95% of the ceiling is luck, not a fix. Verification model: live CronJob spec matches main, last three runs all Complete, runtimes recorded, zero Error pods.

Q: How do you confirm what's actually running in prod matches what's in the repo?

Diff the live object against the manifest — kubectl get cronjob experience-benchmarks -o yaml against deploy/prod.yml. Drift between a merged PR and the running cluster is exactly the kind of gap that lets a "fixed" ticket keep failing, and it's the check that would have surfaced the tickets overlap before any code was written.

Q: What should you have checked before writing a single line for this ticket?

Whether the problem still existed. The ticket sat in Backlog for four days, a ticket shipped a superset of the fix during that window, and the work that followed would have lowered the heap from 10 GB to 6 GB — reintroducing the OOM. Reading main and the live spec before implementing is the cheapest step in the whole process and the one that was skipped.

Brief Q&A

V8 / Node

Q: What does --max-old-space-size cap? Only V8's old space. Not new space, code space, or off-heap memory (Buffer, ArrayBuffer, native addons). RSS can exceed it substantially.

Q: Why die at ~4 GB with no flag set? That's V8's default ceiling on 64-bit. Node has historically sized it from host RAM, not the cgroup limit — so always set it explicitly.

Q: Does NODE_OPTIONS reach a process started via npm run start? Yes, it's inherited by child processes. It also applies to the npm wrapper itself.

Diagnosis

Q: Distinguish the three OOM flavours. V8 heap OOM → mark-compact message, SIGABRT, exit 134, pod Error. Kernel kill → no app log, SIGKILL, exit 137, OOMKilled. Eviction → pod Failed, reason Evicted. Only the first is fixed by the heap flag.

Q: How do you size the flag instead of guessing? --trace-gc for the trajectory, process.memoryUsage() for heap vs. external, --heapsnapshot-near-heap-limit=1 for a snapshot at the abort.

Q: Is raising the heap the real fix? Usually a stopgap. If memory scales with input size, any ceiling eventually fails — stream or batch so peak is bounded by chunk size.

Kubernetes resources

Q: Request vs. limit? Request drives scheduling and eviction ranking; limit is cgroup-enforced. Memory is incompressible — no throttling, only killing.

Q: No resources block means what QoS? BestEffort — first evicted under node pressure, and invisible to scheduler accounting. Requests == limits gives Guaranteed.

Q: How do heap ceiling and container limit relate? Limit strictly above the heap, with headroom for off-heap and process overhead. Too tight converts a clean SIGABRT into a harder-to-debug OOMKill.

Q: Request set high and the pod won't schedule? Check node allocatable (capacity minus reserved and eviction thresholds). Expect Pending with Insufficient memory.

Jobs / CronJobs

Q: Why several failed pods per run? backoffLimit retries, and a deterministic OOM fails identically each time. Use podFailurePolicy with onExitCodes (requires restartPolicy: Never) to fail fast, plus ttlSecondsAfterFinished for cleanup.

Q: Does 30 10 */14 * * run every 14 days? No. Day-of-month steps reset monthly → the 1st, 15th, and 29th. True "every N days" needs external scheduling.

Verification

Q: How do you prove it's fixed? Trigger a manual run from the CronJob, confirm Complete with no failed pods, and compare peak usage to the limit — finishing at 95% of the ceiling is luck.

Q: What do you check before writing any code? That the problem still exists. Diff the live object against the repo manifest; a stale ticket can lead you to lower limits that a since-merged fix raised.

What happened: V8 hit its own default ~4 GB old-space ceiling and aborted at 4064 MB. The container had no memory limit at all (no resources block), so nothing external was squeezing it — Node self-limited while there was ~10 Gi of node memory sitting unused. Not a regression either: no app code change since Day/Month, so the dataset had simply grown past 4 GB for a job that aggregates benchmarks over a 5–10 hour run.

The RAM figures:

	Value
Node	`m5.xlarge` — 16 GiB total, 14.4 Gi allocatable, 4 vCPU
Container limit	none — no `resources` block, so effectively bounded only by node allocatable, QoS `BestEffort`
Effective ceiling	~4 GB, imposed by V8's default, not by Kubernetes

Why the fix landed where it did: heap raised to 10240 MB, with requests 8Gi and limit 12Gi — the limit chosen to sit under 14.4 Gi allocatable so the pod still schedules on the same cron node, and the heap chosen well under the limit to leave off-heap headroom.

One nuance worth noting against what I wrote earlier: the containers exited with code 1, not 134. npm run start wraps the Node process, so npm catches the child's SIGABRT and exits 1 itself — the abort signal never reaches the container's exit status. That's why the podFailurePolicy rule had to match on exit code 1, and it's a good reminder that a wrapper process masks the signal you'd normally use to identify a V8 abort. The mark-compact line in the logs was the real diagnostic.

There are workable rules of thumb, but they're ordering constraints plus margins rather than fixed numbers. The three values have to be set in a specific order because each one bounds the next.

The ordering

working set  <  heap ceiling  <  limit
                                 ↑
                            requests (== limit, or below it)

1. Heap ceiling ≈ working set × 1.25–1.4. V8 needs slack above your live set to do mark-compact efficiently. Set the ceiling at the observed peak and you get GC thrash — the process burns CPU in near-limit collections for a long time before it finally aborts. Slack is what makes the difference between "slow" and "dead."

2. Heap ceiling ≈ 70–80% of the container limit. The remaining 20–30% covers everything --max-old-space-size doesn't: new space, code space, Buffer/ArrayBuffer external memory, native addon allocations, thread-pool stacks, and the wrapper process. Equivalently: limit ≈ heap × 1.3.

Adjust that ratio to the workload's off-heap profile:

Workload shape	Heap as % of limit
Plain JSON/object crunching, little I/O buffering	75–80%
Heavy streaming, large `Buffer` use, many worker threads	55–65%
Native addons (image processing, compression, crypto)	50–60%

3. Requests: for a batch or cron job with a known peak, set requests == limits. You get Guaranteed QoS, no eviction risk, no overcommit surprises — and a batch job's cost is dominated by the run, not by bin-packing efficiency. For long-running services with rare spikes, requests at the p95 steady state and limits at peak plus margin is reasonable, but keep the ratio under ~2× since memory can't be reclaimed under pressure the way CPU can.

How to get the working-set number

Run it once with generous headroom and --trace-gc, then read the plateau of heapUsed from process.memoryUsage() sampled through the run. Separately note external — that's your off-heap figure and it tells you which row of the table above applies. Container-level peak from container_memory_working_set_bytes gives you the total to size the limit against. One clean run gives you all three numbers; anything else is guessing.

Failure modes to avoid

Heap ceiling == container limit. Converts a clean V8 abort with a diagnosable log line into a SIGKILL OOMKill with no application output.
Limit set, heap flag omitted. Depending on Node version, V8 may not read the cgroup limit and will pick its own default — possibly far above or below what you provisioned.
Request larger than ~50% of node allocatable. You've effectively dedicated a whole node to one pod. Fine if intentional (which it was here: 8Gi request on 14.4Gi allocatable), wasteful if not.
Raising numbers on a repeating schedule. If memory scales with input size, every ceiling has an expiry date. Sizing buys time; streaming or batching is the fix.

elasticsearch-exporter

2026-08-03T23:18:08.073+01:00

Elasticsearch doesn't speak Prometheus. It exposes its internals over its own REST API — GET /_nodes/stats, /_cluster/health, /_cat/indices — as JSON, in Elastic's own shape. Prometheus can't scrape that.

The elasticsearch-exporter is a small sidecar-or-Deployment-shaped translator that sits between the two:

it polls those ES REST endpoints on an interval,
flattens the JSON into Prometheus text-format metrics,
and serves them on /metrics (conventionally :9114) for Prometheus to scrape.

The canonical implementation is prometheus-community/elasticsearch_exporter (formerly justwatchcom/elasticsearch_exporter), packaged as the prometheus-elasticsearch-exporter Helm chart. Elastic also ships a first-party alternative path — Metricbeat's elasticsearch module, or the newer Elastic Agent integration — but those ship into Elasticsearch/Kibana's own monitoring cluster, not into Prometheus, so they don't help a Grafana-alerting-on-Prometheus setup.

The metrics it produces are for example:

metric

what it gives you

search_jvm_memory_used_bytes{area="heap"}

actual JVM heap in use — the thing that actually predicts an ES OOM

elasticsearch_jvm_memory_max_bytes{area="heap"}

configured heap ceiling (-Xmx), so you can take a real ratio

elasticsearch_jvm_gc_collection_seconds_*

GC pressure; sustained old-gen GC is the pre-OOM tell

elasticsearch_breakers_tripped

circuit breakers firing — ES rejecting work to avoid OOM

elasticsearch_cluster_health_status

green/yellow/red, unassigned shards │

The exporter is a separate deployable — ECK does not install it.

Prometheus Metrics

2026-08-03T23:17:59.220+01:00

The 4 Core Prometheus Metric Types

Prometheus defines four primary metric types. Choosing the right type depends on how the value behaves over time.

┌───────────────────────────┐

│ Prometheus Metric Types │

└─────────────┬─────────────┘

│

┌───────────────────┬──────────────┴───────────────┬────────────────────┐

▼ ▼ ▼ ▼

┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐

│ Gauge │ │ Counter │ │ Histogram │ │ Summary │

├──────────────┤ ├──────────────┤ ├──────────────┤ ├──────────────┤

│ Value goes │ │ Value ONLY │ │ Groups values│ │ Calculates │

│ UP and DOWN │ │ increases │ │ into buckets │ │ quantiles │

│ (Snapshot) │ │ (Cumulative) │ │ (Client-side)│ │ (Client-side)│

└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘

1. Gauge

Behavior: Can increase, decrease, or stay the same.
Use Case: Current state, temperatures, memory usage, concurrent connections, replica counts.
PromQL Functions: avg_over_time(), max_over_time(), direct evaluation.

In Prometheus (and OpenMetrics standards), a Gauge is a metric that represents a single numerical value that can arbitrarily go up and down.

It acts like a digital display or a vehicle's speedometer—it gives you a point-in-time snapshot of a current state.

Understanding the Gauge Type

Because a Gauge can fluctuate freely, it is used to measure snapshot values, current state levels, and instantaneous statuses.

Example Real-World Analogy: A car's speedometer (can go from 0 to 70 mph, back to 30 mph), ambient temperature, or fuel tank level.
Kubernetes/KSM Examples:

kube_cronjob_status_last_successful_time: Holds a Unix timestamp (moves forward or stays fixed).
kube_deployment_status_replicas_available: Number of ready pods (e.g., 3 → 5 → 2).
node_memory_MemAvailable_bytes: Available system memory in bytes.

Key Feature: Instantaneous Operations

Because gauges fluctuate, you do not run rate or increase functions like rate() or increase() on them in PromQL. Instead, you query their current value directly or perform operations like running averages (avg_over_time()), min/max checks, or scalar math (e.g., time() - gauge).

2. Counter

Behavior: A cumulative metric that only increases (or resets to 0 upon process restart). It never decreases naturally.
Use Case: Counting total occurrences of events (e.g., total HTTP requests served, total pipeline errors, network bytes transmitted).
PromQL Functions: rate(), irate(), increase().

Why use a Counter instead of a Gauge? A counter allows PromQL's rate() function to accurately calculate "per-second rates" while automatically handling process restarts (resets).

3. Histogram

Behavior: Samples observations (usually things like request durations or payload sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
Exposed Data:

<basename>_bucket{le="<upper_bound>"}: Counter of observations with value <= upper bound.
<basename>_count: Total number of observations (Counter).
<basename>_sum: Sum of all observed values (Counter).

Use Case: Measuring latencies, response times, or request sizes where you want to calculate percentiles (e.g., p95, p99) on the server side using PromQL (histogram_quantile()).

4. Summary

Behavior: Similar to a histogram, but calculates configurable quantiles (e.g., 0.50, 0.90, 0.99) directly on the client application side over a sliding time window.
Exposed Data:

<basename>{quantile="0.95"}: The 95th percentile value.
<basename>_count: Total observations.
<basename>_sum: Sum of observations.

Use Case: When client-side percentile calculation is required and you don't need to aggregate quantiles across multiple instances in Prometheus.

Summary Comparison Matrix

Metric Type

Can Decrease?
Typical PromQL Functions
Best Used For

Gauge

Yes
Direct value, avg_over_time()
Current state, memory usage, counts of current objects

Counter

No (only on restart)
rate(), increase()
Total count of events over time, request volume

Histogram

No (bucket counts increase)
histogram_quantile(), rate()
Request latencies, payload sizes (aggregatable)

Summary

No (counts/sums increase)
Direct quantile query
Request latencies (pre-calculated on client)

Container Metrics

container_memory_rss

container_memory_rss is a Prometheus metric (exposed by cAdvisor) that measures a container's Resident Set Size—specifically, the amount of physical RAM allocated to non-reclaimable, non-file-backed memory.

Key Technical Breakdown

Under the hood in Linux cgroups, container_memory_rss tracks:

Anonymous Memory: Process heap allocations, execution stack, and memory allocated via malloc or mmap(MAP_ANONYMOUS).
Swap Cache: Memory swapped out to disk that is being brought back into RAM.

What it EXCLUDES:

Unlike standard Linux host RSS metrics, container_memory_rss in cAdvisor excludes file-backed page caches (memory used to cache files read from disk). Because of this, it only measures memory that cannot be automatically freed by the Linux kernel under memory pressure.

container_memory_rss vs. Other Container Metrics

To understand its role, it helps to see how it fits into cAdvisor's other core memory metrics:

Metric

Includes
Purpose / Key Characteristic

container_memory_rss

Heap + Stack (Anonymous memory)
Stable indicator of process memory footprint. Does not fluctuate with disk I/O.

container_memory_working_set_bytes

Heap + Stack + Active Page Cache
What Kubernetes actually monitors. Fluctuates with file reads/writes.

container_memory_usage_bytes

Heap + Stack + Active Cache + Inactive Cache
Raw total RAM usage. Can be misleading because inactive cache is easily reclaimed by the OS.

Why is container_memory_rss Important?

Memory Leak Detection: Because it excludes cached file reads, container_memory_rss provides a much cleaner signal for application-level memory leaks. If this metric steadily climbs over time without dropping, your application (e.g., Go heap, JVM heap, Node process) is holding onto unmanaged memory.
Debugging OOM Kills: While Kubernetes triggers OOMKills based on container_memory_working_set_bytes hitting resource limits, container_memory_rss helps you determine why it happened:

High RSS + High Working Set --> Application memory leak or underestimated heap limit.
Low RSS + High Working Set --> Heavy disk I/O / file caching (e.g., reading massive log files or database indexes into memory).

When container_memory_rss is NOT a relevant metric?

Example:

WiredTiger is the default storage engine that MongoDB (here, the Percona Server for MongoDB / PSMDB cluster) uses to actually read and write data to disk. In <ticketID> it matters specifically because of how it uses memory, which is what breaks those four Grafana alert rules.

The relevant behavior:

WiredTiger keeps a large in-memory cache. It maintains its own cache of frequently-accessed data and indexes, and it deliberately sizes that cache to roughly half the container's memory limit (the issue cites ~10 GiB against a 21Gi limit on rs0/rs2, and it's tunable per <ticketID>). WiredTiger runs its own eviction, targeting about 80% cache fill under normal operation and triggering aggressive eviction around 95%.

That cached data shows up as page cache, not RSS. This is the crux of the ticket. container_memory_rss counts anonymous/resident process memory but excludes the OS page cache — and for a WiredTiger workload the file-backed pages (the cache) are the bulk of the real footprint. So rss / limit sits structurally pinned around 50-58% no matter how much memory pressure the pod is actually under. An alert keyed on container_memory_rss > 0.90 can therefore never fire for a WiredTiger container. It's dead. (alert is defined as: container_memory_rss{container="mongod"} / limit > 0.90)

But you can't just switch to working-set either. container_memory_working_set_bytes does include those active file pages, so on the busy replicas it reads 95-96%. That looks alarming but is expected and healthy: it's the WiredTiger cache sitting at its designed ~half-of-limit size and WT's own eviction watermark. The kernel reclaims those pages under real pressure, and the evidence is that nothing in the mongodb namespace has ever been OOMKilled. Flipping the metric would just move the rule from never-firing to always-firing.

The signal that actually matters is WiredTiger cache fill. Because WT stalls user operations when its cache can't evict fast enough, the meaningful early-warning metric is cache utilization (the existing MongoDB WiredTiger cache fill — above 95% for 30m rule from <ticketID>), not container RSS or working set. That's why the ticket argues the RSS rules are redundant, not just broken, and leans toward deleting them.

So in this issue's context, "WiredTiger" is essentially shorthand for "a workload whose memory lives mostly in a large, self-managed, page-cache-backed database cache" — which is exactly the profile that makes RSS-based memory alerting meaningless and makes cache-fill the correct signal instead. (The same logic applies to the Elasticsearch rules, where the JVM heap plays the analogous role.)

Example PromQL Query

To track application heap growth without the noise of filesystem caching, query RSS like this:

# Rate of container RSS growth over 5-minute intervals

rate(container_memory_rss{namespace="production", container!=""}[5m])

container_memory_working_set_bytes

...

container_memory_usage_bytes

...

Introduction to cAdvisor (Container Advisor)

2026-08-03T16:28:55.615+01:00

cAdvisor (short for Container Advisor) is an open-source tool created by Google to collect, aggregate, process, and export resource usage and performance metrics for running containers.

It acts as a daemon that monitors resource isolation parameters, historical resource usage, and network statistics directly from the host node.

How It Works

cAdvisor doesn't require instrumenting code inside containers. Instead, it inspects the node environment where containers run:

Queries Linux Kernel Structures: It pulls raw performance data directly from kernel mechanisms—primarily cgroups (control groups) for CPU, memory, and disk utilization, and network interfaces for throughput metrics.
Discovers Running Containers: It automatically detects running containers across multiple runtimes (Docker, containerd, CRI-O, systemd containers).
Exposes Metrics: It formats gathered data and exposes it over a /metrics HTTP endpoint (primarily in Prometheus format) for scrapers to ingest.

Role in Kubernetes

In Kubernetes, cAdvisor is not deployed as a standalone pod. Instead, it is built directly into the kubelet binary that runs on every node.

The kubelet uses cAdvisor internally to monitor local container resource usage.
It exposes cAdvisor metrics under the /metrics/cadvisor endpoint on the kubelet API port (typically 10250).
Tools like Prometheus scrape this endpoint to collect system-wide container metrics (container_cpu_usage_seconds_total, container_memory_rss, container_network_transmit_bytes_total, etc.).

Core Capabilities

Resource Usage Monitoring: Tracks real-time CPU utilization, memory breakdown (RSS, cache, swap), network I/O, and disk space usage per container.
Historical Trend Aggregation: Keeps a small buffer of historical telemetry in memory for local inspection.
Multi-Runtime Support: Works out of the box with containerd, Docker, and CRI-compliant container engines.
Built-in Web UI: When run as a standalone binary or Docker container outside Kubernetes, it provides a lightweight built-in dashboard for quick visual inspection of container stats.

Summary of Metric Flow

Linux Kernel / cgroups --> cAdvisor --> Prometheus --> Grafana

cAdvisor sits right at the boundary between the underlying host/kernel layer and the monitoring stack, turning low-level kernel counters into structured metrics for alerting and visualization.

Kubernetes Job object

2026-08-03T14:03:43.217+01:00

A Kubernetes Job is a controller object designed to run a batch task to completion.

Unlike Deployments or ReplicaSets (which keep applications running indefinitely) or CronJobs (which trigger tasks on a schedule), a Job creates one or more Pods, executes the workload, and ensures they terminate cleanly. Once the specified number of Pods complete successfully, the Job itself is marked as complete and stops.

Standard Job Manifest

apiVersion: batch/v1

kind: Job

metadata:

spec:

backoffLimit: 4 # Number of retries before marking the job failed

completions: 1 # Number of successful pod completions required

parallelism: 1 # How many pods run concurrently

ttlSecondsAfterFinished: 600 # Clean up job & pods 10 minutes after completion

template:

spec:

containers:

- name: migration-task

image: python:3.11-slim

command: ["python", "-c", "print('Running database migration...'); import time; time.sleep(10); print('Done!')"]

restartPolicy: OnFailure # Required: OnFailure or Never (Always is invalid)

Core Execution Patterns

Kubernetes Jobs support three primary workload execution models:

1. Non-Parallel Jobs

Behavior: Starts a single Pod and waits for it to complete successfully.
Use Case: One-off database schema migrations, report generation, or administrative scripts.

2. Parallel Jobs with Fixed Completions

Behavior: Runs multiple Pods in parallel until a total number of successful completions (spec.completions) is reached.
Use Case: Batch processing where $N$ independent tasks need to be completed.

3. Parallel Jobs with Work Queue

Behavior: Pods coordinate via an external message queue (e.g., RabbitMQ, Redis, SQS). Each Pod pulls work until the queue is empty, then exits.
Use Case: High-throughput task processing, media transcoding, or distributed data transformation.

Key Configuration Fields

Field

Default
Description

restartPolicy

Required
Must be OnFailure (restarts container inside same Pod) or Never (spawns a new Pod on failure).

backoffLimit

6
Maximum number of retries before marking the Job as failed.

completions

1
Total number of successful Pod terminations needed for Job completion.

parallelism

1
Max number of Pods allowed to run concurrently at any given moment.

activeDeadlineSeconds

Unlimited
Max time allowed for the entire Job (including retries) before terminating all running Pods.

completionMode

NonIndexed
Set to Indexed to assign each Pod a unique completion index ($0$ to $\text{completions}-1$) via environment variables.

ttlSecondsAfterFinished

Disabled
Automatically deletes the Job and its underlying Pods after $N$ seconds of finishing.

ttlSecondsAfterFinished is a field in a Kubernetes Job specification (spec.ttlSecondsAfterFinished) that controls the automatic cleanup of completed or failed Jobs and their associated Pods via the TTL-after-finished controller.

It is a good practice to set sensible ttlSecondsAfterFinished / history limits on Jobs / CronJobs so dead pods don't accumulate.

How It Works

Automatic Cleanup: Once a Job reaches a terminal state (Complete or Failed), the TTL controller starts a clock using the completion timestamp. When the specified number of seconds elapses, the Job resource is deleted.
Cascading Garbage Collection: Deleting the Job also automatically cleans up its dependent Pods.
Immediate Deletion (0): Setting ttlSecondsAfterFinished: 0 makes the Job eligible for immediate removal upon completion or failure.
Unset / Omitted: If left undefined or set to null, the Job and its Pods remain in the cluster indefinitely until manually deleted.

Manifest Example

apiVersion: batch/v1

kind: Job

metadata:

spec:

ttlSecondsAfterFinished: 300 # Deletes Job and Pods 5 minutes after finishing

template:

spec:

containers:

- name: migration-runner

image: my-app-migrations:v1.2.0

command: ["npm", "run", "db:migrate"]

restartPolicy: Never

Key Considerations

Clock Trigger: For successful Jobs, the timer counts down from .status.completionTime. For failed Jobs, it uses the transition timestamp of the Failed condition.
Live Modifications: You can update or extend ttlSecondsAfterFinished while a Job is running or even after it finishes (as long as the original TTL hasn't already expired).
Log Loss: Because Pods are destroyed during garbage collection, application logs will be lost unless forwarded to an external logging system (e.g., Loki, OpenSearch, Datadog).

Essential kubectl Commands

Operation Command

================== ===========

Create a Job imperatively kubectl create job my-job --image=busybox -- echo "Hello World"

Get Job status kubectl get jobs

Inspect job details kubectl describe job my-job

List Pods associated with Job kubectl get pods --selector=batch.kubernetes.io/job-name=my-job

View logs of Job Pods kubectl logs job/my-job

Delete Job and its Pods kubectl delete job my-job

Jobs vs Deployments vs CronJobs

┌─────────────────────────┐

│ Workload Type │

└────────────┬────────────┘

│

┌─────────────────────┴─────────────────────┐

▼ ▼

Long-Running Services Batch Tasks / One-Off

(Deployments, StatefulSets) (Jobs & CronJobs)

│ │

Maintains target Pod Executes task, then

count indefinitely. terminates cleanly.

│

┌────────────────┴────────────────┐

▼ ▼

Single Run Scheduled Run

(Job) (CronJob)

Kubernetes CronJob

2026-08-03T12:27:07.766+01:00

A Kubernetes CronJob creates and manages short-lived Jobs on a scheduled, repeating basis. It is the Kubernetes equivalent of a standard Unix crontab file, making it ideal for periodic tasks like database backups, report generation, or maintenance scripts.

Minimal Example Manifest

apiVersion: batch/v1

kind: CronJob

metadata:

spec:

schedule: "0 2 * * *" # Runs every day at 02:00 UTC

timeZone: "Etc/UTC" # Optional: set preferred timezone (Kubernetes 1.27+)

concurrencyPolicy: Forbid

startingDeadlineSeconds: 100

successfulJobsHistoryLimit: 3

failedJobsHistoryLimit: 1

jobTemplate:

spec:

template:

spec:

containers:

- name: backup-task

image: alpine:latest

command:

- /bin/sh

- -c

- echo "Running database backup..."; sleep 5

restartPolicy: OnFailure

Schedule Syntax Quick Reference

The schedule field uses standard cron syntax with 5 fields:

minute hour day-of-month month day-of-week

Schedule Format Interpretation

* * * * * Every minute

*/15 * * * * Every 15 minutes

0 0 * * * Every day at midnight

0 9 * * 1 Every Monday at 9:00 AM

Critical Settings

concurrencyPolicy: Controls how overlapping executions are handled when a previous run hasn't finished:

Allow (default): Runs concurrent jobs simultaneously.
Forbid: Skips the new job if the previous one is still running.
Replace: Cancels the currently running job and starts the new one.

startingDeadlineSeconds: The deadline (in seconds) for starting a job if it missed its scheduled time (e.g., cluster was temporarily down).
successfulJobsHistoryLimit / failedJobsHistoryLimit: Number of completed or failed Job/Pod records to keep for auditing before automatic cleanup.
restartPolicy: Must be set on the pod template spec to either OnFailure or Never (Always is invalid for Jobs).

Helpful kubectl Commands

Task Command

==== ========

List CronJobs kubectl get cronjobs

Inspect configuration kubectl describe cronjob <name>

Manually trigger immediately kubectl create job --from=cronjob/<cronjob-name> <manual-job-name>

Pause schedule kubectl patch cronjob <name> -p '{"spec": {"suspend":true}}'

View logs of latest run kubectl logs job/<job-name>

CronJobs Inner Mechanism

Under the hood, Kubernetes CronJobs rely on a decentralized control loop pattern. They are not handled by a traditional Linux cron daemon running on a single server, but rather by the Kubernetes Control Plane through cascading controllers.

How CronJobs Are Implemented

The implementation follows a 3-tier hierarchical model:

CronJob Object --> Job Object --> Pod(s)

Rather than running code directly, a CronJob acts as a factory for Job objects, which in turn manage the Pods where your container actually executes.

┌─────────────────────────────────────────────────────────┐

│ kube-controller-manager │

│ │

│ ┌───────────────────┐ Creates ┌─────────┐ │

│ │ CronJob Controller│ ─────────────────> │ Job │ │

│ └───────────────────┘ └─────┬───┘ │

└──────────────────────────────────────────────────┼──────┘

│

Creates

│

▼

┌─────────┐

│ Pod │

└─────────┘

The Control Loop Mechanism

Synchronization Loop: The CronJob Controller runs inside kube-controller-manager. Every ~10 seconds, it iterates through all CronJob objects defined in the cluster.
Schedule Checking: The controller parses the schedule field (e.g., 0 * * * *) and compares the current time against the last time the job was executed.
Job Spawning: If a run is due, the CronJob controller reads the embedded jobTemplate and creates an actual Job resource.
Execution: The cluster's separate Job Controller detects the newly created Job resource and spawns one or more Pods to execute your container workload to completion.
Garbage Collection: Depending on successfulJobsHistoryLimit and failedJobsHistoryLimit, the CronJob controller periodically deletes old completed Job objects (and their associated logs/pods).

Who Controls Them?

Control over CronJobs is split between system components (automation) and users/roles (permissions).

System Component Control

kube-controller-manager: The core control plane component where the CronJob controller code actually executes. If this component is down, scheduled triggers will pause until it recovers.
kube-apiserver: Stores the desired state in etcd and validates user manifests.
kube-scheduler: Assigns the individual Pods spawned by the resulting Jobs to healthy worker nodes.

User & Permission Control (RBAC)

Human administrators and automated service accounts control CronJobs via Kubernetes Role-Based Access Control (RBAC):

Role / Action Required API Permissions (batch/v1)

============= ==============================

Manage Schedules create, update, patch, delete on cronjobs

View Status get, list, watch on cronjobs

Manual Trigger create permissions on jobs (to invoke kubectl create job --from=cronjob/...)

Example RBAC Role for CronJob Operators:

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

namespace: prod

rules:

- apiGroups: ["batch"]

resources: ["cronjobs", "jobs"]

verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Technical Considerations

At-Least-Once Execution: Kubernetes schedules are designed around at-least-once execution semantics. Due to control loop timing or network hiccups, a scheduled job might occasionally run twice or run slightly late. Workloads should always be designed to be idempotent.
Timezones: Controller clocks default to UTC or the local time of kube-controller-manager unless explicit timezones are passed via spec.timeZone (supported in K8s 1.27+).

CronJob is a controller object

In Kubernetes, a controller is a control loop that watches the state of your cluster through the API server and makes changes attempting to move the current state toward the desired state.

Here is how a CronJob fits into the controller pattern:

Why CronJob is a Controller

Custom Resource / Spec & Status Model: Like Deployment, ReplicaSet, and Job, a CronJob has an API object schema (spec defining desired behavior, status tracking execution state).
Control Loop Execution: The CronJob implementation runs as a control loop inside the kube-controller-manager component.
Cascading Controller Pattern: CronJob sits at the top of a controller hierarchy:

CronJob Controller --[creates/manages]--> Job Controller --[creates/manages]--> Pods

The CronJob Controller reconciles the CronJob spec: it checks the schedule, creates Job resources when a execution is due, cleans up old jobs based on history limits, and handles concurrency policies.
The Job Controller reconciles those created Job resources to manage individual Pods to completion.

Summary Table

Controller API Group What it Watches What it Creates/Manages

======== ========= ============= ===================

CronJob Controller batch/v1 CronJob specs Job objects

Job Controller batch/v1 Job specs Pod objects

Deployment Controller apps/v1 Deployment specs ReplicaSet objects

Grafana Alerting

2026-08-03T11:46:28.850+01:00

How it works

Grafana alerting periodically queries data sources and evaluates the condition defined in the alert rule
If the condition is breached, an alert instance fires
Firing instances are routed to notification policies based on matching labels
Notifications are sent out to the contact points specified in the notification policy

How to set alerts

Alert rules: Create an alert rule to query a data source and evaluate the condition defined in the alert rule.

There are two types of alerts rules:

Grafana-managed. Examples:

APM
AWS
Data Quality
Kubernetes
MongoDB
Storage
Synthetics

Data source-managed. Data sources containing configured alerts rules are for example Mimir or Loki data sources where alert rules are stored and evaluated in the data sources itself. In these data sources you can select Manage alerts via Alerting UI to be able to manage these alerts rules in the Grafana UI as well as in the data source where they were configured.

Prometheus
Mimir
Loki

Define the condition that must be met before an alert rule fires

Route alert notifications either directly to a contact point or through notification policies for more flexibility

Contact points: Configure who receives notifications and how they are sent
Notification policies: Configure how firing alert instances are routed to contact points

Monitor your alert rules using dashboards and visualizations

Kubernetes Custom Resource Definition (CRD)

2026-07-21T16:49:50.806+01:00

Why are Custom Resource Definitions (CRDs) stored in etcd?

Custom Resource Definitions (CRDs) are stored in etcd because etcd serves as the single source of truth and persistent data store for the entire Kubernetes control plane. Storing CRDs in etcd allows the kube-apiserver to dynamically register, validate, and manage custom resources exactly like native objects without requiring a separate database.

Why Kubernetes Stores CRDs in etcd

Native API Integration: The kube-apiserver inherently reads from and writes to etcd. Storing CRDs here allows custom objects to instantly leverage native features like kubectl support, RBAC security, and namespaces.
Consistent State: Like Pods or Deployments, custom resources represent the "desired state" of a system. etcd provides the strong consistency and distributed consensus needed to safely store this state across cluster nodes.
Watch Mechanism: Custom controllers and operators rely on the Kubernetes watch API to listen for resource changes. Because etcd natively supports watch events, controllers can instantly react when a custom resource is created, updated, or deleted.

How custom controllers interact with the stored CRD data using the reconciliation loop?

Custom controllers interact with etcd through a continuous synchronization mechanism called the reconciliation loop. This loop constantly drives the actual state of the cluster toward your desired state.

Here is exactly how custom controllers interact with stored CRD data:

1. The Relationship to CRDs

A CRD (Custom Resource Definition) acts as the database schema. It defines the structure and validation rules for your custom data. When you create a CR (Custom Resource) based on that CRD, its data is stored directly in etcd.

The custom controller is the active software component (the brain) that watches that specific CR data and takes action based on what is stored.

2. The Reconciliation Loop Step-by-Step

The controller executes a continuous loop, often formalized as a Reconcile() function, which follows a Observe, Analyze, Act workflow:

+--------------------------------------------+
| |
| 1. OBSERVE |
| (Watch etcd via API Server) |
| |
+---------------------+----------------------+
|
v
+---------------------+----------------------+
| |
| 2. ANALYZE |
| (Compare Desired vs. Actual State) |
| |
+---------------------+----------------------+
|
v
+---------------------+----------------------+
| |
| 3. ACT |
| (Create/Update/Delete Infrastructure) |
| |
+--------------------------------------------+

Step 1: Observe (The Watch Mechanism): Instead of constantly polling etcd, the controller establishes a long-lived HTTP connection (a watch) to the kube-apiserver. When a user creates, updates, or deletes a CR in etcd, the API server instantly broadcasts an event. The controller captures this event and queues the resource's key for processing.
Step 2: Analyze (Diffing the State): The controller fetches the full CR data from etcd. It reads the .spec section, which represents the user's desired state (e.g., "replicas": 3). It then looks at the real-world cluster infrastructure to see the actual state (e.g., only 2 pods are running).
Step 3: Act (Fixing the Drift): The controller executes the necessary logic to close the gap. In this case, it talks to the API server to spin up a 3rd pod.
Step 4: Update Status: Once the action is complete, the controller updates the .status subresource of the CR. The API server writes this new status back to etcd, confirming the actual state now matches the desired state.

Key Characteristics

Level-Triggered: The loop reacts not just to isolated events, but to the current state of the system. If the controller crashes and restarts, it reads etcd to immediately understand the current state and resume fixing any drift.
Asynchronous: The user gets an immediate "Success" response from the API server when saving a CR to etcd. The controller processes the request in the background asynchronously.

Accessing AWS EKS Cluster from EC2

2026-07-17T13:10:13.307+01:00

When an EC2 instance (acting as a CI/CD runner, bastion, or deployment node) runs Terraform or Helm to interact with a private EKS cluster, it needs two things:

network visibility (which it gets by being in the same VPC or connected network) and
proper IAM-to-Kubernetes mappings

AWS modern access management feature uses EKS Access Entries. This native API feature entirely replaces the messy, deprecated aws-auth ConfigMap.

Here is how to configure the IAM Role and wire it into the cluster's auth configuration using Terraform.

1. The EC2 IAM Role (AWS Side)

The EC2 instance does not need any specific EKS admin permissions attached directly to its IAM role policies. It just needs a standard IAM Role that it can assume via an EC2 Instance Profile. The actual Kubernetes cluster access is granted inside EKS by referencing this role's Amazon Resource Name (ARN).

# 1. Create the IAM Role for the EC2 Instance

resource "aws_iam_role" "cicd_runner" {

name = "eks-cicd-runner-role"

assume_role_policy = jsonencode({

Version = "2012-10-17"

Statement = [

{

Action = "sts:AssumeRole"

Effect = "Allow"

Principal = { Service = "ec2.amazonaws.com" }

}

]

})

# 2. Create the Instance Profile so the EC2 can use the role

resource "aws_iam_instance_profile" "cicd_runner_profile" {

name = "eks-cicd-runner-profile"

role = aws_iam_role.cicd_runner.name

}

2. Wiring it into the Cluster (EKS Side)

To authorize this IAM role to run Helm deployments or manage Kubernetes resources via Terraform, you use EKS Access Entries (aws_eks_access_entry) combined with Access Policy Associations (aws_eks_access_policy_association).

⚠️ Prerequisite: Ensure your aws_eks_cluster resource has authentication_mode = "API_AND_CONFIG_MAP" or "API" enabled so it accepts Access Entries.

The Terraform Configuration

# 1. Define the Access Entry mapping the IAM Role ARN to EKS

resource "aws_eks_access_entry" "cicd_runner_entry" {

cluster_name = "my-cluster-name"

principal_arn = aws_iam_role.cicd_runner.arn

type = "STANDARD" # Standard workflow for users/roles

}

# 2. Attach an AWS-managed access policy to the Access Entry

resource "aws_eks_access_policy_association" "cicd_admin_mapping" {

cluster_name = "my-cluster-name"

principal_arn = aws_iam_role.cicd_runner.arn

policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"

access_scope {

type = "cluster"

}

Another example (in the cluster's Terraform source code):

# read-only access entry for the in-VPC runner's instance role, so operators can kubectl the

# private cluster via SSM-on-the-runner for diagnostics without an out-of-band `aws eks create-access-entry` grant each time. AmazonEKSAdminViewPolicy = read all resources incl.

# CRDs, excluding Secrets — safe for diagnostics. The runner role is defined in another repo

# (applications/test/github-runner); it also gets eks:DescribeCluster there so `aws eks update-kubeconfig`

# works. Writes still require github-actions-role (the CD/harness OIDC principal above).

resource "aws_eks_access_entry" "runner_readonly" {

cluster_name = local.cluster_name

principal_arn = "arn:aws:iam::1234567890123:role/github-runner-role"

type = "STANDARD"

depends_on = [module.eks]

}

resource "aws_eks_access_policy_association" "runner_readonly" {

cluster_name = local.cluster_name

policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminViewPolicy"

principal_arn = aws_eks_access_entry.runner_readonly.principal_arn

access_scope {

type = "cluster"

}

and in EC2 Terraform repo:

# eks:DescribeCluster so `aws eks update-kubeconfig` works from the runner. Paired with the

# read-only EKS access entry for this role on test-default-eks (infra-k8s), it lets operators

# kubectl the private cluster via SSM for diagnostics without a hand-built kubeconfig or an ad-hoc grant.

# IAM (this) authorises the DescribeCluster call; the access entry (RBAC) authorises what kubectl can read.

data "aws_iam_policy_document" "eks_describe" {

statement {

sid = "EksDescribeCluster"

actions = ["eks:DescribeCluster"]

resources = ["arn:aws:eks:${var.aws_region}:${data.aws_caller_identity.current.account_id}:cluster/test-default-eks"]

}

resource "aws_iam_role_policy" "eks_describe" {

name = "${local.name}-eks-describe"

role = aws_iam_role.runner.id

policy = data.aws_iam_policy_document.eks_describe.json

}

Alternative: Custom Kubernetes Groups

If you don't want to grant full AmazonEKSClusterAdminPolicy and prefer mapping the EC2 instance to custom Kubernetes RBAC roles, you can pass target groups directly via the access entry:

resource "aws_eks_access_entry" "cicd_runner_entry" {

cluster_name = "my-cluster-name"

principal_arn = aws_iam_role.cicd_runner.arn

kubernetes_groups = ["my-custom-helm-deployers-group"]

type = "STANDARD"

}

(You would then manage a native Kubernetes ClusterRoleBinding pointing to my-custom-helm-deployers-group using the kubernetes provider.)

3. How the EC2 Instance Connects

Once the IAM and EKS access configurations are applied, you don't need to pass raw AWS access keys to the EC2 instance. The tools leverage the Instance Profile automatically.

For Helm & kubectl:Log onto the EC2 instance (or configure your shell script/user-data) to update the local kubeconfig using the AWS CLI:

aws eks update-kubeconfig --region us-east-1 --name my-cluster-name

When Helm or kubectl is invoked, it calls the aws eks get-token command under the hood, uses the EC2's IAM profile to generate a signed token, routes over the private VPC endpoint network, and authenticates flawlessly.

For the Terraform Kubernetes/Helm Providers:

If Terraform itself is running on that EC2 instance and deploying helm charts into EKS, configure your Terraform provider block to pull tokens dynamically using the instance profile credentials:

data "aws_eks_cluster_auth" "cluster" {

name = "my-cluster-name"

}

provider "helm" {

kubernetes {

host = "https://ABC123XYZ.gr7.us-east-1.eks.amazonaws.com"

cluster_ca_certificate = base64decode("CLUSTER_CA_DATA")

token = data.aws_eks_cluster_auth.cluster.token

}

Just-In-Time (JIT) Access (Least Privilege Workflow)

Another approach is a Just-In-Time (JIT) access or Least Privilege workflow. It is highly secure, but it can quickly become an operational headache for a DevOps engineer if it isn't completely automated.

Instead of giving your GitHub Actions runner continuous, permanent admin access to your private EKS cluster, you only grant access for the exact duration of an ad-hoc task, and then immediately rip it away.

Here is exactly how that breakdown works mechanically step-by-step, why someone would build it, and the hidden risks you should look out for.

The Workflow Cycle Breakdown

When you trigger a workflow or step that needs to run an ad-hoc kubectl command, an automation tool (like a pipeline script, a step function, or a privileged security wrapper) executes this lifecycle:

[Pipeline Starts]

│

▼

1. CREATE ACCESS ENTRY ──► (Allows 'github-runner-role' network identity into EKS)

│

▼

2. ASSOCIATE POLICY ──► (Binds ClusterAdmin or specific RBAC permissions)

│

▼

3. RUN KUBECTL COMMAND ──► (The runner executes its ad-hoc tasks safely)

│

▼

4. DISASSOCIATE POLICY ──► (Strip away the RBAC permissions)

│

▼

5. DELETE ACCESS ENTRY ──► (Completely remove the IAM role mapping from EKS)

│

▼

[Pipeline Finishes]

1. The Gate: create-access-entry

By default, your runner's IAM role (github-runner-role) is completely blocked at the EKS front door. Even if it can physically route to the API endpoint, EKS will return a 401 Unauthorized.

The setup script calls the AWS API to create an EKS Access Entry. This tells EKS: "Be ready to recognize this specific IAM role ARN."

2. The Permission: associate-access-policy

Creating the entry just gets the runner past the front door; it still has zero privileges inside the cluster. The next API call binds an access policy (like AmazonEKSClusterAdminPolicy or a custom namespace policy) to that entry. Now kubectl commands will actually work.

3. The Execution: Ad-hoc Commands

The runner runs aws eks update-kubeconfig, grabs its temporary token, and runs the necessary kubectl or helm actions.

4. The Clean-up: Revoking Access

Once the kubectl step finishes, the pipeline runs a cleanup block (usually wrapped in a always() or finally clause to ensure it runs even if the deployment fails). It reverses the process by disassociating the policy and deleting the access entry entirely. The runner is now unauthorized again.

Why would someone architecture it this way?

Blast Radius Reduction: If that specific GitHub runner instance or the AWS role credentials are ever compromised, the attacker cannot access your Kubernetes cluster because the role has no standing permissions.
Audit Trails: Every single time access is granted, used, and revoked, a permanent trail is logged in AWS CloudTrail. It makes passing compliance checks exceptionally easy because you can prove no system has persistent, unchallenged access.

The Catch: Why this pattern can trip you up

While it sounds bulletproof on paper, this setup has a few major operational trade-offs that you have to manage closely:

Pipeline Failures Leave Backdoors: If the runner gets abruptly killed (e.g., the GitHub Actions job is manually canceled mid-run, or the runner machine loses power), the cleanup step might never run. The role will remain an admin of your cluster until someone goes in and manually deletes the entry.
Race Conditions on Concurrent Runs: If you have multiple workflows trying to use that same runner role at the same time, Run A might delete the access entry while Run B is right in the middle of executing a kubectl apply.
AWS API Throttling: If you scale up your CI/CD pipelines and are constantly creating and destroying access entries every few minutes, you can easily hit AWS API rate limits (throttled EKS control plane requests).

For the same EC2 instance (github-runner-role) to grant itself access via the JIT loop, it must possess two entirely separate sets of permissions:

Control Plane Permissions (AWS IAM): It needs permission to call the AWS EKS APIs (CreateAccessEntry, AssociateAccessPolicy) to tell AWS, "Hey, let me in."
Data Plane Permissions (Kubernetes RBAC): This is what it actually gives itself inside the cluster to view pods, run Helm charts, or apply manifests.

Here is how this works in practice, how to configure the IAM policy so the runner can self-authorize, and why you must be extremely cautious with this pattern.

1. The Setup: The Instance's Base IAM Policy

To make this happen, the IAM role attached to the EC2 instance must be pre-provisioned with standard AWS permissions allowing it to modify the cluster's access configuration.

You must attach an IAM policy like this to the github-runner-role:

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "AllowEKSAccessEntrySelfManagement",

"Effect": "Allow",

"Action": [

"eks:CreateAccessEntry",

"eks:DeleteAccessEntry",

"eks:AssociateAccessPolicy",

"eks:DisassociateAccessPolicy",

"eks:DescribeCluster"

"Resource": "arn:aws:eks:us-east-1:123456789012:cluster/my-cluster"

{

"Sid": "AllowSTSAssumeRoleSelf",

"Effect": "Allow",

"Action": "sts:AssumeRole",

"Resource": "arn:aws:iam::123456789012:role/github-runner-role"

}

]

}

💡 Note on the STS block: This allows the runner role to assume itself to generate those unique, concurrent session ARNs (.../github-runner-role/gh-run-123) we discussed earlier.

2. The Execution Flow

When your automation script starts on the EC2 instance, it uses its own local AWS credentials (provided by the EC2 Instance Profile) to run the exact commands that grant it access:

# 1. The EC2 instance talks to AWS IAM & STS to get a unique session ARN

# (This isolates concurrent runs from each other)

ASSUMED_ROLE_ARN="arn:aws:sts::123456789012:assumed-role/github-runner-role/gh-run-${GITHUB_RUN_ID}"

# 2. The EC2 instance calls the AWS EKS API using its base IAM permissions

aws eks create-access-entry \

--cluster-name my-cluster \

--principal-arn "$ASSUMED_ROLE_ARN" \

--type STANDARD

# 3. The EC2 instance elevates its own session's privileges inside Kubernetes

aws eks associate-access-policy \

--cluster-name my-cluster \

--principal-arn "$ASSUMED_ROLE_ARN" \

--policy-arn "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy" \

--access-scope type=cluster

We could also assign a cluster read-only policy, AmazonEKSAdminViewPolicy.

At this point, the instance has successfully leveraged its permanent AWS-level permissions to dynamically grant itself Kubernetes-level permissions.

3. The Security Catch: "The Illusion of Security"

While this satisfies the requirement of keeping the Kubernetes cluster clean of permanent admin entries, you need to look closely at the security trade-off here.

If an attacker compromises this EC2 instance, they inherit the base IAM role. Because that base role has permission to call CreateAccessEntry and AssociateAccessPolicy, the attacker can simply run the exact same script to grant themselves cluster admin privileges whenever they want.

How to make it actually secure:

To make this pattern truly robust, the EC2 instance should not have direct permission to grant itself access. Instead, a external "Gatekeeper" should handle it:

Lambda Gatekeeper: The runner sends a request to an internal AWS Lambda function (or an internal API gateway). The Lambda verifies the job context (e.g., checks if there is a valid, active GitHub Action job running right now), and if verified, the Lambda creates the EKS Access Entry on behalf of the runner.
OIDC/IRSA Approach: If this runner is executing jobs from GitHub, skip the EC2 instance profile entirely. Use GitHub's OIDC token to exchange directly for a short-lived AWS session that EKS recognizes natively. This removes the "self-granting" risk entirely.

AWS EKS Cluster API

2026-07-17T12:18:40.038+01:00

In AWS EKS, managing the cluster's API endpoint correctly is one of the most critical steps for balancing day-to-day DevOps convenience with robust security.

Here is exactly how the EKS API works, what public vs. private means, how subnets factor into the equation, and how authentication is handled.

1. What is the Cluster's API?

The cluster API is the Kubernetes API Server control plane hosted and managed by AWS. It is the single entry point for all administrative actions. Every time you run a command via kubectl, deploy a Helm chart, or when worker nodes send heartbeats, they are talking to this API server endpoint. AWS gives you a unique HTTPS URL for this endpoint (e.g., https://ABC123XYZ.gr7.us-east-1.eks.amazonaws.com).

2. Public vs. Private Endpoints

This setting dictates network-level visibility—essentially, who can physically route traffic to that HTTPS URL.

EKS supports three access modes:

Public Only (Default): The API URL resolves to a public IP address. Anyone on the internet can physically reach the login page, but they still must authenticate to get in. Worker nodes inside your VPC must route out to the internet (often via a NAT Gateway) to talk back to the control plane.
Public and Private (Recommended for most production setups): The URL resolves to a public IP for external users, but within your VPC, it resolves directly to private IP addresses via internal Cross-Account ENIs (Elastic Network Interfaces). Your worker nodes communicate completely internally, while you can still run kubectl from your local machine without a VPN. You can also allowlist specific CIDRs/IPs on the public side to lock it down to your office or home IP.
Private Only: The public internet endpoint is completely shut off. The API URL resolves only to private IPs inside your VPC. To run a kubectl command, you must be inside the VPC network (e.g., via AWS Client VPN, Transit Gateway, or a bastion host).

How to change it:

You can switch this at any time with no downtime via the AWS Console (Cluster -> Networking -> Manage Endpoint Access) or via the AWS CLI:

Example: Switch to Public + Private recommended mode

aws eks update-cluster-config \

--name my-cluster \

--resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true

3. Does it matter which Subnets the cluster runs on?

Yes, but do not confuse Cluster Subnets with the API Endpoint setting. They are two distinct layers.

When you create an EKS cluster, you must provide a set of subnets (usually at least two in different Availability Zones). EKS drops its managed network interfaces (X-ENIs) into these subnets so the control plane can communicate with your worker nodes.

Best Practice: Always provide private subnets (subnets that route outbound via a NAT gateway and don't assign public IPs to instances) for your actual worker nodes and EKS ENIs. This keeps your EC2 nodes completely hidden from the internet.
The Disconnect: You can have an EKS cluster running on entirely private subnets, but still have a Public API endpoint. The API endpoint is hosted by AWS's managed infrastructure, not directly inside your subnets.

4. Does the API require authentication? Who can access it?

Yes, absolutely. Network visibility (Public/Private) is just the perimeter fence. Even if your API endpoint is wide-open public, nobody can do anything without explicit authentication and authorization.

EKS uses a dual-layer system to handle access:

Layer 1: Authentication (Who are you?) via AWS IAM

EKS natively leverages AWS IAM for authentication. When you run kubectl, your local setup uses the AWS CLI to generate a temporary, signed token based on your current AWS IAM identity (User or Role). It passes this token in the HTTPS header to the EKS API. EKS verifies this token with AWS IAM to prove you are who you say you are.

Layer 2: Authorization (What can you do?) via Kubernetes RBAC

Just having a valid IAM identity isn't enough. Once EKS knows your identity, it hands the request off to the internal Kubernetes Role-Based Access Control (RBAC) engine.

By default, the IAM identity that originally created the EKS cluster is automatically granted full cluster administrator (system:masters) privileges.
For anyone else (other team members, CI/CD tools), you must map their AWS IAM ARN to specific Kubernetes Groups using the EKS Access Entries feature (or the legacy aws-auth ConfigMap) and assign them RBAC Roles/ClusterRoles.

Summary of the flow:

kubectl get pods -->

Local AWS CLI generates an IAM token -->

Request hits EKS API (routes via public internet or private VPC depending on endpoint settings) -->

EKS checks token validity via IAM -->

Kubernetes RBAC verifies if your user profile has permission to view pods -->

Success

Helm and Infrastructure

2026-07-16T15:39:06.615+01:00

Does Helm provision aws infrastructure?

The short answer is no, Helm does not natively provision AWS cloud infrastructure.

Helm is a package manager designed specifically for Kubernetes. Its job is to template, package, and deploy Kubernetes-native YAML manifests (like Deployments, Services, ConfigMaps, and Ingresses) onto an already-running cluster. It does not natively understand or manage AWS resources like VPCs, EC2 instances, RDS databases, or S3 buckets. However, because the cloud native ecosystem is highly integrated, the line can sometimes look blurry. Here is how that boundary gets crossed and how the two worlds interact.

The "Grey Area": When Helm Indirectly Triggers AWS Provisioning

While Helm itself only talks to the Kubernetes API, certain Kubernetes operators and controllers can watch for Helm-deployed resources and provision AWS infrastructure in response:

1. Cloud-Provider Load Balancers

If your Helm chart deploys a Kubernetes Service of type: LoadBalancer:

Helm simply submits that Service YAML to the EKS control plane.
The cloud-controller-manager inside EKS sees this and automatically provisions an AWS Classic Load Balancer or Network Load Balancer (NLB) in your AWS account.
Similarly, deploying an Ingress resource alongside the AWS Load Balancer Controller (often installed via Helm itself) triggers the creation of an AWS Application Load Balancer (ALB).

2. Custom Resource Definitions (CRDs) & Operators

You can use Kubernetes operators that are explicitly designed to manage cloud infrastructure through Kubernetes. If you install these operators (often using Helm), you can then use Helm to deploy manifests that provision AWS resources:

AWS Controllers for Kubernetes (ACK): This framework allows you to define AWS resources (like an S3 bucket or an RDS instance) directly as Kubernetes custom resources. If you bundle an ACK custom resource inside a Helm chart, running helm install will tell the ACK operator to provision that infrastructure in AWS.
Crossplane: Similar to ACK, Crossplane turns your cluster into a control plane. You can package Crossplane claims inside Helm charts to spin up databases, networks, and queues.

The Standard Best Practice: "Separation of Concerns"

Because managing cloud infrastructure via Kubernetes manifests can get complex quickly, the industry-standard pattern is to keep these layers strictly separated:

┌────────────────────────────────────────────────────────┐

│ App Layer (Helm) │

│ - Deployments, Services, Ingress, ConfigMaps, etc. |

└───────────────────────────┬────────────────────────────┘

│ Runs inside

┌───────────────────────────▼────────────────────────────┐

│ Kubernetes Cluster (EKS) │

└───────────────────────────┬────────────────────────────┘

│ Provisioned by

┌───────────────────────────▼────────────────────────────┐

│ Infra Layer (IaC - Terraform) │

│ - VPC, Subnets, EKS Cluster, IAM, RDS, S3, etc. │

└────────────────────────────────────────────────────────┘

Infrastructure as Code (Terraform, OpenTofu, CloudFormation, Pulumi): Used to provision the underlying AWS infrastructure—the VPC, subnets, IAM roles, EKS cluster, and managed node groups.
Helm: Used once the cluster is up and running to deploy and configure applications, ingress controllers, logging agents, and service meshes on top of that infrastructure.

Note: If you want a single tool to bridge the gap, Terraform actually has a Helm Provider. This allows you to use Terraform to provision your EKS cluster first, and then immediately call Helm charts within the same Terraform code to bootstrap your cluster applications.

How to use Terraform to install a Helm chart?

To bridge the gap between infrastructure and application deployment, you can use the HashiCorp Helm Provider directly inside your Terraform configuration.

This approach ensures strict ordering: Terraform first provisions the AWS network and EKS cluster, and only after the cluster is fully active does it initialize Helm to install your charts.

Here is a clean, production-ready example showing how to connect your AWS provider, configure the Kubernetes/Helm providers dynamically using the cluster's output token, and deploy a chart.

# 1. PROVIDERS CONFIGURATION

# We configure Kubernetes and Helm providers dynamically using tokens

# generated from the newly created AWS EKS cluster.

provider "aws" {

region = "eu-west-2" # London

}

provider "kubernetes" {

host = aws_eks_cluster.main.endpoint

cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)

token = data.aws_eks_cluster_auth.main.token

}

provider "helm" {

kubernetes {

host = aws_eks_cluster.main.endpoint

cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)

token = data.aws_eks_cluster_auth.main.token

}

# Data source to fetch authentication tokens for EKS dynamically

data "aws_eks_cluster_auth" "main" {

name = aws_eks_cluster.main.name

}

# 2. AWS EKS CLUSTER PROVISIONING (Simplified)

resource "aws_eks_cluster" "main" {

name = "production-core-cluster"

role_arn = aws_iam_role.eks_cluster.arn

vpc_config {

subnet_ids = ["subnet-12345678", "subnet-87654321"] # Replace with your VPC subnet IDs

}

# 3. HELM RELEASE PROVISIONING

# Terraform treats this Helm chart as a resource managed within its state file.

resource "helm_release" "ingress_nginx" {

name = "ingress-nginx"

repository = "https://kubernetes.github.io/ingress-nginx"

chart = "ingress-nginx"

version = "4.10.0"

namespace = "ingress-system"

create_namespace = true

# Explicit dependency guarantees the cluster exists before Helm tries to talk to it

depends_on = [aws_eks_cluster.main]

# Passing configuration values into the Helm chart

set {

name = "controller.replicaCount"

value = "3"

}

set {

name = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-type"

value = "nlb"

}

Why this pattern works so well:

Token-Based Auth Lifecycle: By using the aws_eks_cluster_auth data source, Terraform fetches a short-lived administrative token on the fly during the apply phase, ensuring Helm can authenticate without storing static kubeconfig files on disk or in version control.
Escaping Helm Keys: Note the double backslash (\\) in the annotation key. Helm uses dot notation to traverse nested values, so if an AWS annotation key contains a dot (like aws-load-balancer-type), you must escape it in Terraform so Helm interprets it as a single key string rather than a nested map.
State Management: When you run terraform destroy, Terraform is smart enough to reverse the order: it calls Helm to cleanly uninstall the charts, deletes any cloud-provider resources (like NLBs) created by those charts, and then tears down the EKS cluster.

Mitigating Load Balancer Routing Clamping

2026-07-02T11:48:23.892+01:00

In computing and networking, routing clumping (also known as traffic clumping or clustering) refers to the phenomenon where a load balancer or router unevenly sends bursts of traffic to the same backend servers instead of distributing requests evenly across the entire available server pool.

While a load balancer is designed to act as a traffic cop, routing clumping defeats this purpose, momentarily overloading specific servers while leaving others idle.

Primary Causes of Routing Clumping

Hashing Bias: Algorithms like 5-tuple hashing (which use source IP, destination IP, ports, and protocols) calculate a fixed value to select a routing path. If thousands of users share a proxy or exit node, the load balancer may calculate the same hash and clump all their traffic onto a single backend server.
Sticky Sessions: If you use session persistence (sticky sessions) based on a client's IP, multiple requests from the same office or network range will clump to a single server to maintain session continuity.
Connection Multiplexing: Load balancers maintain long-lived backend connections to save resources. When a new batch of requests comes in, the load balancer routes them through existing open connections, causing requests to cluster on specific nodes rather than opening new connections to idle servers.
Inconsistent Pod or Instance Count: In environments like Kubernetes, if traffic is processed node-by-node, unequal distribution of application pods across nodes or restrictive node affinities can cause traffic to clump in specific zones or servers.

How to Prevent and Mitigate Clumping

Use Round Robin or Least Connections: Switch from IP-hashing or fixed-hash methods to AWS Least Connections or round-robin algorithms, which evaluate server availability in real-time rather than relying on mathematical routing properties.
Enable Cross-Zone Load Balancing: For cloud architectures (like AWS Elastic Load Balancing), enabling cross-zone load balancing forces your load balancers to distribute traffic evenly across all registered targets in all enabled Availability Zones.
Use Layer 7 Intelligent Routing: Instead of Layer 4 packet forwarding, use a Layer 7 Application Load Balancer to inspect HTTP headers and distribute traffic based on request context, allowing for finer-grained distribution.
Deploy Hierarchical Splitting: For massive, hyperscale systems, implementing a hierarchical L4 + L7 structure (ultra-fast L4 hardware balancers routing to a software-based L7 pool) helps prevent clumping while maintaining high raw throughput.

Why round robin routing didn't work well in this scenario so we need to use least connections routing?

Good question — the answer is about the kind of workload service3 is. Round robin isn't broken in general; it's the wrong fit for this specific traffic shape.

What each algorithm does

Round robin: hands each new request to the next target in rotation (pod A, B, C, A, B, C…), purely by count — blind to how busy each target currently is.
Least outstanding requests (LOR): sends each new request to the target with the fewest in-flight requests right now — i.e., the least-busy one.

Why round robin fails here

Round robin distributes by count, which only equals distributing by load when requests are uniform and short. service3 is the opposite — it has the toxic combination:

1. Highly variable, long service times — a spaCy extraction is anywhere from ~0.3s to multi-second depending on text length/entity count. One heavy request ties a pod up for seconds.

2. Low per-pod concurrency — each pod does ~1–2 concurrent extractions (2 workers, GIL-bound). It saturates almost immediately.

Under those conditions, round robin keeps handing new requests to a pod that's still grinding a heavy multi-second extraction, just because it's "next in rotation" — while a pod that just finished sits idle. The new request then queues head-of-line behind the heavy one for seconds, even though free capacity exists elsewhere. Over a burst this compounds: unlucky pods accumulate a backlog while others drain and idle → exactly the clumping / 33%-CPU-but-3s-latency pattern we measured.

The analogy: round robin is a supermarket sending shoppers to checkout lanes strictly 1‑2‑3‑1‑2‑3, ignoring that lane 2 has someone with a giant cart. You get stuck behind the big cart while lane 3 is empty. LOR is "go to the lane with the fewest items in it right now."

Why LOR fixes it

LOR tracks in-flight requests per target and skips the busy pod — a pod mid-heavy-extraction has a high outstanding count, so new requests go to the idle/draining pods instead. It self-corrects for variance (whichever pod is busy, for any reason, automatically gets fewer new requests until it catches up) and it reaches the idle capacity round robin was stranding.

The catch — why we also needed IP mode

LOR only helps if the ALB is choosing among the right units. In instance mode, the ALB's targets are nodes, so it'd balance outstanding requests across nodes — then kube-proxy randomly picks a pod within the node, re-scrambling the per-pod balance. LOR literally can't see per-pod busyness in instance mode.

IP mode makes the targets pods, so LOR's "least outstanding" is measured per pod → it can actually steer to the least-busy pod. That's why the fix is ip mode + LOR together: ip mode gives LOR the right granularity; LOR gives ip mode the right decision rule. Either alone wouldn't do it.

Two footnotes

- The other 5 services keep round robin — deliberately. auth/data/game/webhook are fast (<0.5s), uniform, high-concurrency — round robin distributes them fine, and there's no reason to touch them. The algorithm choice is workload-specific: LOR for

slow/variable/expensive backends, round robin for fast/uniform ones.

- Terminology: you said "least connections" — the ALB (L7) equivalent is least outstanding requests, which counts in-flight HTTP requests, not TCP connections. That distinction matters here: the ALB reuses keep-alive connections (one connection carries many sequential requests), so "least connections" would be misleading — LOR counts the actual requests, which is the right signal for this.

In which cases round-robin doesn't help so we need to use least outstanding requests routing?

While Round-Robin works perfectly for uniform traffic and identical servers, it fails when the system experiences variations in request complexity or server capacity. You must switch from Round-Robin to Least Outstanding Requests (LOR) (also known as Least Connections) in the following four specific engineering scenarios:

1. Varying Request Processing Times (Asymmetric Workloads)

The Problem: In many applications, some API calls take 5 milliseconds (e.g., fetching a profile cached in memory), while others take 5 seconds (e.g., generating a heavy PDF report or running a complex database query).Why Round-Robin Fails: Round-Robin blindly hands out requests in a strict, alternating sequence. If Server A randomly receives a consecutive string of heavy PDF requests while Server B receives fast cache requests, Server A's queue will spike, causing high latency or timeouts, while Server B sits mostly idle.
How LOR Helps: LOR actively tracks the active connection count. It will notice Server A is backed up with pending work and will divert all new incoming traffic to Server B until Server A finishes its heavy processing.

2. Heterogeneous Server Capacities (Mixed Server Sizes)

The Problem: Production clusters often use mixed hardware. For example, during an auto-scaling event, you might temporarily mix older 4-core virtual machines with newer, high-performance 16-core instances.Why Round-Robin Fails: Round-Robin treats every backend target as equal. It sends exactly 1,000 requests to the weak 4-core machine and exactly 1,000 requests to the powerful 16-core machine. The weaker machine will quickly choke, run out of memory, or drop packets, while the stronger machine remains underutilised.
How LOR Helps: The faster, more powerful server processes and closes connections much quicker than the weaker server. Because its outstanding connection count drops rapidly, LOR naturally funnels a significantly higher volume of traffic to the stronger hardware without needing manual weight configurations.

3. Persistent and Long-Lived Connections (WebSocket & gRPC)

The Problem: Modern applications rely heavily on persistent connections like WebSockets, HTTP/2 multiplexing, server-sent events (SSE), or gRPC streams. These connections stay open for minutes or hours.Why Round-Robin Fails: Round-Robin only counts the initial connection establishment. If Server A hosts 50 clients who stay connected for 3 hours chatting, and Server B hosts 50 clients who disconnect after 30 seconds, Round-Robin will continue to feed new connections to both servers equally. Server A will slowly crush under the cumulative weight of long-lived active sessions.
How LOR Helps: LOR continuously monitors active, open connections rather than connection arrival rates. It will see that Server A has 50 active outstanding connections and Server B has 0, immediately routing all new clients to Server B.

4. Unpredictable Backend "Cold Starts" and Drifts

The Problem: When a new application container or server boots up (e.g., during a Kubernetes deployment or AWS auto-scaling event), it often suffers from a "cold start" where it runs slowly while warming up caches or compiling code just-in-time (JIT).Why Round-Robin Fails: Round-Robin immediately floods the newly booted server with its full share of production traffic. Because the server is not yet performing at 100% capacity, this sudden burst easily overwhelms it, causing it to crash immediately after launching.

How LOR Helps: LOR naturally throttles traffic to the warming server. Since the cold server processes its first few requests slowly, its outstanding request count will naturally rise, signaling the load balancer to back off and route traffic elsewhere until the server catches up.

Algorithm Comparison Table

Scenario
Round-Robin Behavior
Least Outstanding Requests Behavior

All requests take equal time
Perfect, completely equal distribution.
Excellent, identical result to Round-Robin.

Mix of short & long requests
Overloads random servers (creates traffic clumps).
Dynamically balances the processing queue.

Mixed server capacities
Overwhelms smaller, weaker backend servers.
Directs more traffic to faster instances automatically.

Long-lived WebSockets / gRPC
Ignores session duration; causes massive load imbalance.
Tracks real-time active sessions perfectly.

If you are using a cloud platform, you can learn more about implementing this via the AWS Least Outstanding Requests Documentation or the NGINX Least Connections Guide.

k8s ingress - AWS ALB

To implement Least Outstanding Requests (LOR) routing with an AWS Application Load Balancer (ALB) inside a Kubernetes cluster, you must configure it using the AWS Load Balancer Controller.

By default, the AWS ALB uses Round-Robin routing. You can change this behavior by applying a specific routing annotation to your Kubernetes TargetGroupBinding or Ingress resource.

1. Ingress Configuration Example

Apply the alb.ingress.kubernetes.io/target-group-attributes annotation to your Ingress manifest. This tells the AWS controller to configure the underlying target groups to use the load_balancing.algorithm.type=least_outstanding_requests setting.

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

namespace: production

annotations:

# Essential ALB Controller configurations

kubernetes.io/ingress.class: alb

alb.ingress.kubernetes.io/scheme: internet-facing

alb.ingress.kubernetes.io/target-type: ip

# Enable Least Outstanding Requests Routing

alb.ingress.kubernetes.io/target-group-attributes: load_balancing.algorithm.type=least_outstanding_requests

spec:

rules:

- http:

paths:

- path: /

pathType: Prefix

backend:

service:

port:

number: 80

2. Crucial Step: Use "IP" Target Type

When running AWS ALB in Kubernetes, you should always set alb.ingress.kubernetes.io/target-type: ip instead of instance mode.

Why? In instance mode, the ALB routes traffic to NodePorts on the EC2 worker nodes. The node then uses kube-proxy (iptables or IPVS) to randomly route the packet to a pod. This completely breaks the math behind Least Outstanding Requests because the ALB can only see outstanding connections to the node, not the actual pods.
The Fix: Using ip mode configures the ALB to bypass the node network entirely and route traffic directly to the Pod IPs. This gives the ALB true visibility into the exact outstanding request count for each application container.

3. Verify the Configuration

After applying the manifest via kubectl apply -f ingress.yaml, you can verify the changes are active in AWS:

Open the AWS EC2 Console and navigate to Target Groups.
Select the target group automatically generated by your Kubernetes Ingress.
Click on the Attributes tab.
Verify that Routing algorithm is set to Least outstanding requests.

Alternatively, check the AWS Load Balancer Controller logs to ensure the modification was successfully reconciled:

kubectl logs -n kube-system deployment/aws-load-balancer-controller

---

Introduction to Velero

2026-07-01T14:51:07.912+01:00

Velero is an open-source disaster recovery, Kubernetes-native backup, restore and migration tool for Kubernetes. It allows you to back up and restore both your Kubernetes cluster resources and, optionally, the persistent volumes (PVs) that hold application data.

It was originally created by Heptio (the company founded by two of Kubernetes' creators) and is now maintained by VMware and the open-source community.

What does Velero back up?

Velero can back up:

Kubernetes resources (Cluster API objects):

Deployments
StatefulSets
Services
Ingresses
ConfigMaps
Secrets
CRDs
Namespaces
RBAC resources
Custom Resources

They are stored as tarballs in an S3 bucket.

Optionally, it can also back up:

Persistent Volumes (application data)
- via storage snapshots (AWS EBS, Azure Disk, GCP Persistent Disk, etc.)
- or via a file-level backup tool called Node Agent (formerly Restic)

How it works

A typical Velero deployment consists of:


                    +----------------+
                    | Kubernetes API |
                    +--------+-------+
                             |
                        Velero Server
                             |
         +-------------------+-------------------+
         |                                       |
 Metadata Backup                        Volume Backup
         |                                       |
         v                                       v
   Object Storage                    Snapshot or File Backup
   (S3, Azure Blob,                  (EBS, CSI Snapshot,
    GCS, MinIO...)                    Node Agent/Restic)

For example:

Cluster metadata → stored in an S3 bucket
PV data → stored as EBS snapshots or uploaded to object storage

Typical use cases

1. Disaster recovery

Your EKS cluster is accidentally deleted.

With Velero you can:

recreate the cluster
install Velero
restore all workloads
restore persistent data

2. Accidental deletion

Someone runs:


kubectl delete namespace production

Instead of recreating everything manually:


velero restore create \
  --from-backup production-backup

3. Cluster migration

Move workloads from:

EKS → EKS
EKS → AKS
EKS → GKE
On-prem → cloud

Velero restores Kubernetes objects into the new cluster.

4. Scheduled backups

Example:


Every night at 2 AM

↓

Backup namespaces:
- production
- monitoring
- logging

Retention can be configured, for example:


Keep 30 daily backups
Delete older ones automatically

What it does NOT back up

Velero does not automatically back up:

etcd directly
cloud infrastructure (VPCs, Load Balancers, IAM, Security Groups)
managed databases like RDS
container images (they remain in your registry)
external services

Those require separate backup strategies.

Storage providers

Velero supports many object storage backends:

Amazon S3
MinIO
Azure Blob Storage
Google Cloud Storage
OCI Object Storage
many S3-compatible systems

Persistent Volume backup methods

There are two main approaches.

1. CSI snapshots (preferred)

If your storage class supports the Container Storage Interface (CSI) snapshot API:


PVC
 ↓
VolumeSnapshot
 ↓
Cloud snapshot

Advantages:

very fast
incremental (depending on storage backend)
cloud-native
recommended

2. Node Agent (formerly Restic)

If snapshots aren't available:


PVC
 ↓
Read filesystem
 ↓
Compress
 ↓
Upload to object storage

Advantages:

works almost everywhere
storage-independent

Disadvantages:

slower
consumes CPU and network bandwidth

Example architecture in AWS


                 Amazon EKS
                      |
          +-----------+-----------+
          |                       |
     Kubernetes API          Persistent Volumes
          |                       |
      Velero Server         EBS Snapshots
          |
          |
      S3 Bucket
      backups/

Example installation

Install the CLI:


brew install velero

Velero CLI operates on the current kubectl context by default. It automatically inherits your active cluster credentials and configurations from your default kubeconfig file. You can override this behavior using the --kubecontext or --kubeconfig flags

Deploy into an EKS cluster:


velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws \
  --bucket my-backups \
  --backup-location-config region=eu-west-2

Example backup

Back up an entire cluster:


velero backup create full-cluster

Back up a namespace:


velero backup create production \
    --include-namespaces production

Example restore


velero restore create \
    --from-backup full-cluster

Scheduling

Create a nightly backup:


velero schedule create nightly \
    --schedule="0 2 * * *"

When should you use Velero?

Velero is a good fit if you want to:

Recover Kubernetes workloads after accidental deletion or cluster failure.
Back up application manifests and, optionally, persistent data.
Migrate workloads between Kubernetes clusters or cloud providers.
Automate recurring backups with retention policies.
Protect stateful applications running on Kubernetes.

If your applications also depend on external systems (for example, managed databases, message brokers, or cloud resources), Velero should be part of a broader disaster recovery strategy rather than the only backup solution.

Velero vs. etcd backup

Feature	Velero	etcd backup
Kubernetes resources	✅	✅
Persistent volume data	✅	❌
Restore individual namespaces	✅	❌
Restore individual applications	✅	❌
Migrate between clusters	✅	Limited
Cloud agnostic	✅	Mostly
Disaster recovery for applications	✅	Partial

For managed Kubernetes services such as Amazon Elastic Kubernetes Service (EKS), Velero is generally the preferred backup tool because it focuses on application-level recovery rather than restoring the control plane itself. In contrast, direct etcd backups are more common in self-managed Kubernetes clusters where you control the control plane.

Would we benefit from Velero if we already have everything in Terraform?

Yes, because Terraform and Velero back up different layers. Terraform captures the desired infrastructure (the cluster, node groups, add-ons) and our GitOps/Helm captures workload definitions. Neither captures the runtime state that lives inside the cluster:

SecretsConfigMaps created in-cluster rather than from Git

If source of truth is AWS Secrets Manager and the definitions are in Git, we recover every secret by redeploying external-secrets and waiting ~1 minute for re-sync. Velero adds little here.

ConfigMaps created in-cluster rather than from Git

If they are100% in Git: they can be recovered either via kubernetes_config_map Terraform resources (e.g. the kube-prometheus-stack one with computed IAM ARNs) or static YAML applied via kubectl_manifest (e.g. MSK scrape targets). No imperative creation.

Dynamically provisioned PVCs and the data on their volumes
Operator-managed CRs and other objects mutated after (terraform) apply

So Terraform lets us rebuild an empty, correctly-shaped cluster; it does not restore what was running and stored in it at a point in time. Velero fills exactly that gap — plus it gives granular (namespace/PVC-level) and fast restores that a Terraform rebuild can't.

Do we actually need cluster (k8s object definitions + storage) backups?

Yes — most acutely on the stateful clusters e.g. those running MongoDB and Clickhouse, where large EBS-backed PVs hold real data. In non-stateful clusters the main value is namespace/object-level DR (Disaster Recovery) and fast granular restore (e.g. an accidentally deleted namespace or PVC). The EBS CSI driver is already installed on all clusters, so PV snapshots are straightforward to enable.

How does Velero's PVC backup differ from taking snapshots of all volumes associated with all PVCs in cluster? What's the value that Velero brings here?

The short version: snapshotting every volume gives you a pile of disk images; Velero gives you disk images that are still wired into the Kubernetes objects that make them usable. The bytes are the easy part — the value is everything around them.

Concretely, here's what a "loop aws ec2 create-snapshot over all PVC-backed volumes" script leaves you to build yourself, and Velero does for free:

1. The snapshot↔object binding (the big one)

A raw EBS snapshot is an orphan — a snap-0abc… with no memory of which PVC, PV, namespace, StorageClass, or pod it belonged to. To restore it you must, per volume, by hand:

1. create-volume from the snapshot (right AZ, right type),

2. hand-craft a PV pointing at the new vol-… ID,

3. craft a matching PVC with the correct size/StorageClass/claimRef,

4. fix the binding so they actually pair up,

5. restart the workload so it mounts it.

For one volume that's annoying; for a whole cluster it's an error-prone afternoon during an incident. Velero records the full PVC→PV→StorageClass→namespace mapping in its backup manifest and reconstructs all of it on restore — you get a bound PVC attached to a running pod, not a disk image.

2. Consistency & point-in-time grouping

A create-snapshot loop is staggered — volume 1 snapped at T, volume 50 at T+30s, no coordination. For a multi-volume app (say data + WAL on separate PVs) that's an inconsistent set. Velero treats a backup as one logical operation and supports pre/post hooks (fsfreeze, db.fsyncLock()) to quiesce the app first (to quiesce = to pause or alter a device or application to achieve a consistent state, usually in preparation for a backup or other maintenance), so you can get application-consistent — not just crash-consistent — snapshots. This is exactly the nuance behind your earlier "do we still need the mongodump if we snapped the PVCs" question: a bare snapshot is crash-consistent; Velero + hooks tightens that.

3. Discovery & scoping via the K8s API

Velero enumerates PVCs through the Kubernetes API, so it knows which volumes are PVCs (vs node root disks vs unrelated EBS) and can scope by namespace or label — "back up just the mongodb namespace." A raw script has to reconstruct the volume→PVC mapping from EBS tags and risks grabbing node OS disks or missing dynamically-provisioned ones. There's no namespace concept at the EBS layer at all.

4. Selective & cross-cluster restore

Because Velero holds the object graph, a restore can remap — into a different namespace, a different StorageClass (gp2→gp3, or a renamed SC [Storage Class] in the target cluster), or an entirely different cluster/region. This is what makes the blue/green K8s-upgrade path work. Raw snapshots have none of this; you'd script every remap yourself.

5. Lifecycle, catalog, portability

- Retention/GC: Velero backups carry a TTL and expired snapshots are garbage-collected. A snapshot script means you own tagging, retention, and GC — and snapshot sprawl (cost) if you get it wrong.

- Catalog: velero backup describe / restore gives you a queryable history. Raw snapshots are just IDs in EC2 you have to correlate.

- Portability (DataMover): Velero's CSI snapshot data mover can push the snapshot data into S3 via Kopia, so backups aren't locked to regional, account-bound EBS snapshots — enabling cheaper long retention and cross-region DR. It also covers non-EBS volumes (EFS, etc.) nthat create-snapshot can't touch.

In short: Raw volume snapshots back up the data; Velero backs up the data plus everything needed to turn it back into a running, bound, correctly-classed volume in the right namespace — atomically, with consistency hooks, retention, and cross-cluster remapping. The reattachment/orchestration layer is the value, not the snapshot itself.

Can Velero be used in cluster K8s version upgrades?

Yes, in two distinct ways:

Pre-upgrade safety net (in place): take a full Velero backup immediately before an EKS control-plane/version bump. If the upgrade goes wrong, we have an instant restore target for workloads + PV data, independent of the control-plane change.
Blue/green upgrade (migration): stand up a new cluster on the target version and velero restore the workloads into it, then cut traffic over — instead of an in-place upgrade. Velero remaps storage classes and strips node-specific fields on restore, which makes this cross-cluster move practical. This is the cleaner pattern for risky major jumps.

Caveat: on version upgrades, watch for deprecated/removed APIs — objects backed up under an old apiVersion may need conversion on restore into a newer cluster; test-restore before relying on it.

Show Case: 4 Kubernetes Clusters in AWS EKS

For e.g. a 4-EKS-cluster estate on AWS, the concrete wins:

1. Disaster recovery

If a cluster gets corrupted, mis-applied, or accidentally deleted, you can stand up a fresh EKS cluster and Velero restore the whole thing — workloads, configs, PVs — instead of hoping every namespace is reproducible from Git. Even with GitOps, Velero captures the runtime state (secrets created in-cluster, dynamically provisioned PVCs, operator-managed CRs) that Git doesn't.

2. Namespace / PVC-level granular restore

"Someone deleted the prod PVC / dropped a namespace" — restore just that namespace or volume from last night's backup. Much faster than rebuilding.

3. Cluster migration & cloning

Lift workloads from one cluster to another — e.g. clone prod into a staging cluster for testing, or migrate to a new cluster during an EKS version upgrade or AZ/region move. Velero remaps storage classes and strips node-specific fields on restore.

4. Scheduled backups with retention

Cron-style Schedule CRs (e.g. nightly full, hourly for critical namespaces) with TTL-based expiry. One bucket per cluster (or prefixes within one), lifecycle-policied in S3.

5. Pre-upgrade safety net

Take a backup right before an EKS control-plane bump, Helm release, or risky terraform apply that touches cluster resources — instant rollback target.

How it fits this AWS setup

Plugin: velero-plugin-for-aws:

object store = S3
volume snapshotter = EBS

Auth: IRSA (IAM Roles for Service Accounts) per cluster — no static keys. This should ideally match how the rest of the stack is wired.
CSI snapshots: since you're on EBS, enable the CSI snapshot data mover so PV backups are real EBS snapshots, not just file copies.
One S3 bucket, per-cluster prefix (or one bucket each) with versioning + a lifecycle rule for expiry.

Honest caveats

It is not an etcd backup — EKS already manages the control plane / etcd for you, so that's fine, but don't think of Velero as "backing up the cluster brain."
It is not a substitute for application-level DB backups (RDS snapshots, Mongo dumps). Treat Velero as the Kubernetes object + PV layer; keep your existing DB backup story separate.
Restores of stateful apps need testing — ordering, hooks, and re-attach behaviour matter. A backup you've never test-restored isn't a backup.

A sensible rollout for 4 clusters: install via the Velero Helm chart through your existing Terraform/Helm pipeline, IRSA per cluster, nightly scheduled backups to S3 with a 30-day TTL, CSI snapshot data mover on, and one documented test-restore into a throwaway cluster to prove the loop.

How does Velero k8s objects backup differentiate from kubectl get all --all-namespaces -o yaml > "$backup_file_name"?

kubectl get all dump is a far weaker thing than it looks, and the gap is exactly where Velero earns its keep. Breaking it down:

1. kubectl get all doesn't actually get "all"

all is a small, hardcoded category — roughly pods, services, deployments, replicasets, daemonsets, statefulsets, jobs, cronjobs. It silently omits most of what you'd need to reconstruct a namespace:

- Secrets and ConfigMaps ← this alone makes the dump useless for DR

- PVCs, Ingresses, NetworkPolicies, ResourceQuotas, LimitRanges, HPAs

- ServiceAccounts, Roles, RoleBindings (all RBAC)

- Namespaces themselves (you're dumping resources inside namespaces)

- Cluster-scoped objects entirely — CRDs, ClusterRoles, ClusterRoleBindings, PVs, StorageClasses, webhooks

- CRD instances (your custom resources — cert-manager Certificates, external-secrets, Karpenter NodePools, etc.)

So --all-namespaces is a misnomer twice over. Velero, by contrast, enumerates the API discovery list and backs up every resource type (namespaced + cluster-scoped), with include/exclude filters if you want to trim.

2. The biggest gap: no volume data

A YAML dump captures the PVC object — a few hundred bytes of metadata saying "I want 300Gi". It does not capture the 300Gi of bytes on the EBS volume. Velero snapshots the actual disk (EBS snapshot via CSI, or file-level via Kopia). For e.g. Mongo/ClickHouse clusters this is the entire point — the dump backs up the description of your data, Velero backs up your data.

3. The dump isn't restorable as-is

kubectl get -o yaml includes live runtime cruft that a blind kubectl apply chokes on:

- status: blocks, resourceVersion, uid, creationTimestamp, managedFields

- Cluster-assigned values: Service clusterIP/nodePort, PV volumeName bindings, node-specific nodeName on pods

- Objects that shouldn't be recreated at all (running pods owned by a ReplicaSet, endpoint objects)

Re-applying that fails on immutable fields and conflicts. Velero strips and regenerates these on restore (drops status/managedFields, re-allocates ClusterIPs, rebinds PVCs to freshly-restored PVs, lets controllers re-adopt pods).

4. Restore ordering & dependencies

A single giant file applied top-to-bottom breaks: you can't create a CR before its CRD is Established, a PVC before its PV, a RoleBinding before its namespace/ServiceAccount. Velero restores in a dependency-aware order (namespaces → CRDs → then the rest, PVs before PVCs) and waits for prerequisites.

5. Selective / cross-cluster restore

Velero restores are first-class operations:

restore one namespace
remap namespace names
filter by label
restore into a different cluster (this is what makes the blue/green K8s-upgrade story work)
skip-existing-resources
run restore hooks

The dump is all-or-nothing text with none of that.

6. Lifecycle, not a loose file

Velero stores backups to S3 as tracked objects with TTL/retention, checksums, and a queryable history (velero backup get), plus scheduling, metrics, and consistency hooks (fsfreeze / db.fsyncLock() pre-snapshot). The kubectl dump is one file you now have to name, rotate, secure (it contains plaintext secrets!), and hope is internally consistent since it was assembled by looping over the API.

---

Where the dump is fine: a quick "let me eyeball/grep the current state of a namespace" or copying one manifest between environments. As a disaster-recovery mechanism it fails on the two things that matter most — it doesn't capture secrets or volume data, and what it does capture isn't cleanly restorable.

BackupStorageLocation (BSL)

BackupStorageLocation (BSL) is a Velero custom resource (CRD, backupstoragelocations.velero.io) that tells Velero where in object storage to read/write backups.

Each BSL defines:

provider (e.g. aws),
bucket (e.g. test-velero-backups),
prefix (a path within the bucket, e.g. default-k8s, so multiple clusters can share one bucket),
region, and
an access mode — ReadWrite (default) or ReadOnly.

When you run velero backup create, Velero writes the backup (metadata + object YAML + Kopia file data) to the default BSL's bucket. When you velero restore, it reads from the BSL the backup lives in.

To list backup locations:

velero backup-location get

NAME PHASE ... DEFAULT

default Available true

Phase Available means that Velero validated it can reach the bucket.

Restoring namespace from one cluster into another cluster needs a second BSL on the target cluster — a ReadOnly one pointing at the source bucket (e.g. prod-velero-backups) — so the target cluster's Velero can read source's backups and restore them locally, without any risk of writing to source's backup store. ReadOnly also means Velero won't run garbage collection / TTL deletion against that bucket — it can only restore from it, never modify it.

Example:

% velero describe schedules

Name: daily-full

Namespace: velero

Labels: <none>

Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"velero.io/v1","kind":"Schedule","metadata":{"annotations":{},"name":"daily-full","namespace":"velero"},"spec":{"schedule":"0 1 * * *","template":{"defaultVolumesToFsBackup":false,"includedNamespaces":["*"],"snapshotVolumes":false,"storageLocation":"default","ttl":"720h0m0s"}}}

Phase: Enabled

Paused: false

Schedule: 0 1 * * *

Backup Template:

Namespaces:

Included: *

Excluded: <none>

Resources:

Included cluster-scoped: <none>

Excluded cluster-scoped: <none>

Included namespace-scoped: *

Excluded namespace-scoped: <none>

Label selector: <none>

Or label selector: <none>

Storage Location: default

Velero-Native Snapshot PVs: false

File System Backup (Default): false

Snapshot Move Data: auto

Data Mover: velero

TTL: 720h0m0s

CSISnapshotTimeout: 0s

ItemOperationTimeout: 0s

Hooks: <none>

Last Backup: <never>

DevOps Interview Questions - k8s ALB alarm

2026-06-26T15:36:17.077+01:00

Post-incident review / technical interview questions.

These questions and answers were generated by Claude, upon it analyzed and fixed one CloudWatch alarm which have been flipping. Prompt I used:

I was following your analysis and resolution of this issue in order to acquire your knowledge. I would like to test my knowledge now. Can you compile a list of questions which cover every aspect of the issue and solution here? Don't be shy of creating a really long list of questions. If I am able to answer them all that means I am able to fix the issue on my own next time. Also prepare the answer key.

A. The alarm itself (CloudWatch / ALB fundamentals)

What does the metric TargetResponseTime actually measure, and from whose perspective (client → ALB → target)?
The alarm name was k8s-api-prod-core-7db0ccf2c3-target-response-time. What do the k8s- prefix and the hash portion tell you about how it was created?
The alarm config was: threshold 0.8, GreaterThanThreshold, period 60s, 2 evaluation periods, statistic Average. In plain English, what condition makes it fire?
Why does the alarm use the LoadBalancer dimension only, and not a TargetGroup dimension? What consequence did that have for our investigation?
What is "flapping," and why does this particular threshold/period combination make flapping likely for a bursty workload?
How do you pull an alarm's state-transition history, and what did 9 OK→ALARM cycles in 5 hours tell you?
The alarm fires on Average. Why is that distinction (vs Maximum/p99) absolutely central to both the diagnosis and the fix?
What is the "low-traffic statistical artifact" pattern, where a handful of slow requests inflate the average on a near-idle target — and what evidence did we use to rule it out here?

B. Narrowing from ALB to one service

The ALB fronted six target groups. Name the technique we used to find which one was responsible, and the AWS CLI call behind it.
Why can a single ALB serve six different Kubernetes services? What AWS-LB-Controller concept ties them together (hint: group.name)?
Given the target group name k8s-default-dataservice-88e28b4b6c, how do you map it back to a Kubernetes Service and namespace?
During the burst, data-service showed 5.83s avg / 29.7s max while every other service was <0.5s. Why did that immediately exonerate the shared ALB/ingress as the cause?

C. First look at the workload

What does kubectl top pods show, and why is a single snapshot from it dangerous as evidence?
The first snapshot showed one pod at 997m and three near-idle. What two different explanations are consistent with that, and why can't a snapshot distinguish them?
What's the difference between a CPU request and a CPU limit in Kubernetes?
How did we check whether the pod was being CPU-throttled at its limit, and what file did we read inside the container? What did nr_throttled / throttled_usec tell us?
A deploy to v1.14.3-prod had happened ~15 min earlier. Why was it a red herring, and what evidence dated the flapping as pre-existing?
Distinguish the three probe types (liveness, readiness, startup). What does each one control?
The deployment had a startupProbe and livenessProbe but no readinessProbe. Operationally, what can't the system do without a readiness probe?

D. The load-balancing theory (and why it was wrong)

Explain why round-robin load balancing degrades when request durations are highly variable. Use the "request count vs. total work" framing.
What is the feedback-loop difference between round-robin and least-outstanding-requests (LOR)?
What older, well-known algorithm is LOR equivalent to in nginx/HAProxy terms, and what does AWS's own guidance say about when to use LOR vs round-robin?
"Head-of-line blocking" appeared twice in this incident at two different layers. Name both layers and how each one blocks.
Why is a single kubectl top snapshot insufficient to prove round-robin is causing imbalance, and what data would actually prove or refute it?
We initially called round-robin the "smoking gun," then withdrew it. What specifically made that conclusion wrong?

E. Target mode: instance vs IP (the topology that broke the theory)

What's the difference between an ALB target group in instance mode vs ip mode? How can you tell which one you have from the registered targets and the target-group port?
The target group registered 10 EC2 instances on port 31892. What does that tell you about the request path from ALB to pod?
What is a NodePort service, and what is externalTrafficPolicy: Cluster vs Local?
With instance mode + Cluster policy, describe the full path of a request from client to the backend process, including every hop and who load-balances at each hop.
Given that topology, explain precisely why switching the ALB to least_outstanding_requests would not fix per-pod imbalance.
What two changes together would enable load-aware per-pod routing, and why is that a much bigger change than a one-line annotation?
Why were ALB access logs not useful for confirming per-pod distribution in this setup?

F. Getting the proof (Prometheus / time series)

What is kube-prometheus-stack, and where did it live in the cluster?
Why is kubectl port-forward an acceptable read-only way to query Prometheus, and what does it actually do?
Write (conceptually) the PromQL that gives per-pod CPU usage over time. Why rate(container_cpu_usage_seconds_total[...]) rather than the raw counter?
The time series showed all four pods evenly at 0.5–0.95 cores during the burst. Why did that refute the imbalance theory in one stroke?
Each pod plateaued at ~0.9 cores despite a 1.5-core limit. What does that plateau strongly imply about the process model inside the pod?

G. The real root cause (app server / GIL / async)

What is an application server worker (e.g., Gunicorn), and what's the difference between the master process and a worker process?
What is a Global Interpreter Lock (GIL) or similar single-threaded runtime constraint, and why does it mean one worker process ≈ one core of CPU-bound throughput?
The config had workers = 1 and an asynchronous event-loop worker class specified. Explain what each line does.
What is the difference between a synchronous worker and an asynchronous worker in an application server? When does each shine?
Why is an async (event-loop) worker the wrong model for heavy, synchronous CPU-bound data processing? What does "blocking the event loop" mean concretely?
So with one async worker doing CPU-bound work, what is the per-pod concurrency for heavy requests — and how did that produce the fleet-wide ceiling of ~4 concurrent requests?
How did we confirm the worker count and the CPU quota from inside a running pod (what command, what does cpu.max = 150000 100000 mean, what does worker count = 2 mean)?
Why was the 1.5-core CPU limit effectively unusable given workers = 1?
Tie it together: explain the full causal chain from "traffic burst" to "alarm fires," in one paragraph, using the confirmed root cause.

H. Designing the fix

We considered vertical (more workers/CPU per pod), horizontal (more pods via HPA), and both. What's the trade-off, and why did "both" win?
Why couldn't we just change the worker class to synchronous (or offload to a process pool) as part of this immediate infrastructure fix? What kind of change would that be?
We set workers = 2. Why did the CPU limit have to go up to 2000m at the same time? What would happen if we'd set workers = 2 but left the limit at 1500m?
The pods used ~2.5Gi RSS each at one worker. Why did we expect ~5Gi with two workers, and why wouldn't worker-fork preloading save us here? (What did we learn about when the application memory caches are built?)
There's one import-time load we found (e.g., heavy model loading in a utility file). Why is that one shareable-via-fork but the bulk of the runtime memory is not?
The HPA was autoscaling/v1, target 80%, min 4 / max 7. We measured bursts peaking at ~75% of the 1200m request. Explain mechanically why the HPA never scaled.
targetCPUUtilizationPercentage is a percentage of what? Recompute: at the new 1500m request and a pod using ~0.9 cores, what utilization does the HPA see?
Why did we lower the target to 50% and raise max to 10, rather than just one of those?
What does a topologySpreadConstraints with maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway do — and why soft (ScheduleAnyway) rather than hard (DoNotSchedule)?
We deliberately did not add a readiness probe. Explain the failure mode that a naive /health readiness probe would cause in this specific app under heavy load. Why is "no readiness probe" temporarily safer than a bad one here?
What was the container port vs Service target port mismatch situation? Why was alignment low-risk, and why didn't it affect routing?
An orphaned HPA manifest file was deleted. Why was it safe to delete, and how did we confirm it was no longer active?

I. Where the config lives & deploy mechanics

How did we determine the workload was not managed by GitOps tool deployments (e.g., Argo CD), despite the tool being installed? What metadata annotation was the fingerprint?
Which repository and file holds the deployment/HPA, and which separate repository/file holds the ingress? Why do they deploy through different mechanisms?
Describe the deployment pipeline flow end to end. What event triggers it, and what are the key build/test/deploy jobs?
The deploy step does a string substitution (sed 's#$TAG#...#') then kubectl apply. What's the role of the $TAG placeholder, and where does the tag value come from?
Why does merging a PR to the main branch not deploy anything, while pushing a specific environment release tag does?
The pipeline runs on private self-hosted runners. Why does that matter for a private-endpoint cluster?

J. Capacity analysis

What instance types/sizes back the general-purpose compute tier, and how much allocatable CPU/memory does each have?
What is Cluster Autoscaler, and how does it differ from node lifecycle managers like Karpenter? Which one is active in this cluster?
There are two node groups feeding the tier (spot instances and on-demand instances). What are their min/max sizes, and what's the combined node ceiling?
Do the packing math: given ~3920m allocatable CPU and ~400m daemonset overhead, how many pods at 1500m request fit per node? Why is CPU, not memory, the binding constraint?
At HPA max (10 pods), how many nodes are needed, and is that within the ceiling? What's left for other tenants?
Why is horizontal scale-out slow relative to instant traffic bursts? List every contributor to a cold pod's total time-to-serve.
Given that scale-out lag, which part of our fix delivers immediate relief, and which acts as the slower "second line of defense"?
Why did we bump the memory request to 5Gi even though it doesn't change node packing density?

K. Staging-first deploy & verification

Why deploy to the staging environment first when the staging manifest file wasn't even changed by the PR?
Precisely what does the staging deploy validate, and what does it not validate?
Staging runs on the same cluster as production. How is it isolated, and what was the risk we flagged about a tiny 0.5-core staging pod suddenly running workers=2?
List the verification steps we ran on staging, and the pass criteria for each.
Why did we test both GET /health and a heavy POST data processing route, rather than just the health check?
The startup probe warms up by hitting an internal pre-cache endpoint. What does that endpoint do, and why is it the most likely place for a deploy to fail (especially on a CPU-starved staging pod)?

L. Production rollout & confirmation

What version did we tag, and why a patch bump (v1.14.4) rather than a minor/major version shift?
During the production rollout, two new pods went Pending, one with no available node. What happened next, and which log event confirmed the cluster autoscaler reacting?
Why did the rollout take ~6 minutes and stay safe (no dropped traffic) the whole time? What deployment configuration controls the surge/unavailable behavior?
Post-deploy, the new pods showed only ~2.6Gi memory usage, not ~5Gi. Why — and why is that expected rather than a contradiction of our sizing?
After deploy, individual requests still hit ~3s max, but the alarm stayed OK. Why is that consistent with a successful fix? (Connect it back to question 7.)
We monitored for 45 minutes and saw no flapping, yet we kept the incident ticket open. What's the honest gap in that evidence, and what would definitively close it?
The HPA stayed at 4 pods during the entire monitoring window. Why is that a good sign rather than a sign the HPA fix did nothing?

M. Operational / process & gotchas

The cluster API is private-only. What's the practical consequence for local engineers using kubectl, and why does aws sts get-caller-identity succeed while kubectl times out? How do you distinguish an auth problem from a network/VPN problem?
Why did the CI/CD deployment job still work even when our local machine's kubectl couldn't reach the cluster?
The standard code security checks failed on automated image-scanning and vulnerability gates. Diagnose how to isolate the root cause. Were they caused by our configuration change? How do we prove that?
The repository management system showed a BLOCKED merge status, but branch protection rules returned a 404. What's the resolution of that apparent contradiction (e.g., legacy branch protection vs. modern repository rulesets)?
What was the only thing actually gating the code merge, and why were the red security check flags irrelevant to it?
Which actions in this whole deployment flow required explicit manual operator confirmation before execution, and why those specifically?
What's the commit-authorship convention in this engineering environment, and what must never appear in a commit message or PR description?

N. Synthesis & transfer (test of true mastery)

If you were paged for this exact alarm tomorrow with zero prior context, list the first five commands/queries you'd run, in order, and what each one would tell you.
Name three plausible-but-wrong hypotheses for a flapping TargetResponseTime alarm, and the single piece of evidence that kills each.
Suppose the per-pod CPU time series had shown one pod pinned at 100% and three idle (the load imbalance pattern we originally expected). Given the instance-mode topology, what would the real fix have been then — and why is it different from the LOR annotation?
The fix here was vertical + HPA scaling. Under what circumstances would the correct long-term fix instead be an application architecture change, and what would that change look like?
Generalize the core lesson: what specific property of an application workload makes "adding more replicas behind a standard round-robin/random balancer" fail to resolve response time spikes, and what is the class of fixes that does help?
If the morning peak traffic burst still trips the alarm after this infrastructure fix, what are your next two remediation levers (in order), and what data would you collect to choose between them?

Incident Study: core-service-api · prod-default-k8s

The Flapping Latency Alarm, From Page to Fix

A complete answer key to the 98-question self-test — covering triage, the two false trails, the confirmed root cause, the fix, capacity, deployment, and the operational mechanics.

How to use: Answer from memory first, then check here. If you can reconstruct section N (synthesis) unaided, you can run this incident solo. Answers are deliberately tight; the why-not-the-other-thing reasoning matters more than the recall.

Sections Overview

A: The Alarm Itself
B: ALB → One Service
C: First Look at the Workload
D: Load-Balancing Theory (False Trail)
E: Instance vs IP Mode (False Trail)
F: Getting the Proof
G: The Real Root Cause (Confirmed)
H: Designing the Fix
I: Where Config Lives
J: Capacity Analysis
K: Staging & Verification
L: Prod Rollout
M: Operational Gotchas
N: Synthesis & Transfer

A. The Alarm Itself

CloudWatch & ALB fundamentals — what the metric is and what makes it fire.

A1. What does `TargetResponseTime` measure, and from whose perspective?

The elapsed seconds from when the ALB forwards a request to a target until the target starts returning the response headers — i.e., backend processing time as measured by the ALB. It excludes the ALB's own request/response processing and the client ↔ ALB network time. The vantage point is the load balancer, not the client or the pod.

A2. What do the `k8s-` prefix and the hash in the alarm name tell you?

It's machine-generated, not hand-authored. The ALB k8s-coreproddefault-7db0ccf2c3 is one that the AWS Load Balancer Controller created for an ingress group; the hash is derived from it, and the alarm is named after that LB by monitoring automation. Practical takeaway: it tracks an ALB resource, so you investigate at the ALB, and hand-edits to the alarm may be reverted by whatever automated system manages it.

A3. Threshold 0.8, GreaterThanThreshold, period 60s, 2 evaluation periods, statistic Average — what fires it?

When the 60-second average TargetResponseTime is greater than 0.8s for two consecutive 1-minute periods.

A4. Why only the `LoadBalancer` dimension, and what did that cost us?

With only the LoadBalancer dimension, the metric is aggregated across all six target groups behind the ALB. Consequence: the alarm can't tell you which specific service is slow, so we had to break latency down per-target-group ourselves to localize it.

A5. What is flapping, and why is it likely here?

Flapping is the alarm rapidly toggling OK ↔ ALARM. A modest threshold (0.8s) on a short 2 × 60s average, against a workload that spikes for a few minutes then falls to near-zero, means every burst crosses the line and every recovery clears it — resulting in repeated state transitions rather than a sustained ALARM.

A6. How do you pull state-transition history, and what did 9 cycles / 5h say?

aws cloudwatch describe-alarm-history --history-item-type StateUpdate. Nine OK → ALARM cycles in five hours signaled a genuine, recurring degradation — a pattern worth real investigation, not a one-off network blip.

A7. The alarm fires on `Average`. Why is that central to both diagnosis and fix?

Because the trigger is the average, the fix doesn't need to eliminate slow individual requests (tail max can stay ~3s) — it needs to keep the average under 0.8s. Adding concurrency pulls the average down even while tail latency persists. This framing drives the diagnosis (bursts raise the average once capacity saturates) and the fix (more concurrency → average drops below threshold). Keep this in mind for Q83.

A8. What is the low-traffic statistical artifact, and how did we rule it out?

On a near-idle target, a handful of slow requests dominate a tiny sample and inflate the average, tripping the alarm with no real systemic problem. We ruled it out because request count rose during the spikes (~100 → 130–320/min) — the latency was load-correlated, meaning real degradation under load, not a sparse-sample artifact.

B. Narrowing from ALB to One Service

B9. How did we find the responsible target group?

A per-target-group breakdown of TargetResponseTime using the TargetGroup dimension, via aws cloudwatch get-metric-data with one query per target group (avg, max, request count) over the burst window.

B10. Why can one ALB serve six services?

The AWS Load Balancer Controller's ingress grouping feature: ingresses sharing the alb.ingress.kubernetes.io/group.name annotation are merged onto a single ALB, with each backend service getting its own target group.

B11. From `k8s-default-dataapi-88e28b4b6c`, how do you get back to a Service?

The target-group name encodes namespace + service: namespace default, service truncated to dataapi... → core-service-api. Confirm via the ingress backend mapping (host → service) or by matching the target-group port to the service's NodePort.

B12. Why did `data-api` being slow exonerate the shared ALB/ingress?

Only data-api showed 5.83s avg / 29.7s max; the other five services on the same ALB and ingress were <0.5s. A shared-path fault (ALB, ingress, global TLS) would hurt all of them. Isolation to one backend points squarely at that backend, not the shared infrastructure.

C. First Look at the Workload

C13. What does `kubectl top pods` show, and why is one snapshot dangerous?

Current per-pod CPU/memory from metrics-server — a point-in-time sample. It has no time axis, so it can't distinguish a persistent hotspot from a transient one or reveal behavior across an entire traffic burst. Conclusions drawn from a single sample are easy to get wrong (we did, briefly).

C14. One pod at 997m, three idle — what two explanations fit, and why can't a snapshot decide?

(a) Routing keeps hammering one pod (a persistent imbalance); or
(b) That pod merely happens to be mid-way through one heavy request while the others are momentarily between requests (a transient, rotating hotspot).

With no time dimension, a snapshot can't tell a fixed hotspot from a moving one.

C15. CPU request vs limit?

Request: Reserved amount used for scheduling (and the denominator for HPA CPU %).
Limit: Hard ceiling; exceed CPU and you're throttled, exceed memory and you're OOMKilled.

C16. How did we check CPU-limit throttling, and what did we find?

Read the cgroup stats inside the container: cat /sys/fs/cgroup/cpu.stat. nr_throttled / throttled_usec were near-zero, so the pod was not being throttled at its CPU limit — limit-throttling wasn't the cause; the ceiling was elsewhere.

C17. Why was the recent `v1.14.3` deploy a red herring?

The deploy created the young pods (~12:00), but the alarm history showed flapping starting ~04:34 and running ~8h — well before the deploy. A cause can't post-date its effect, so the deploy was purely coincidental.

C18. Distinguish liveness, readiness, startup probes.

Liveness: Is the container alive — fail → restart container.
Readiness: Is the pod fit to receive traffic — fail → removed from Service/LB endpoints (no restart).
Startup: Has the app finished initializing — gates the other two until it passes; fail → restart container.

C19. What can't the system do without a readiness probe?

It can't shed load: a pod has no way to signal "I'm overloaded, stop sending me traffic" and drop out of the ALB target pool. As long as the container runs, it stays in the routing rotation.

D. The Load-Balancing Theory

False Trail

A well-reasoned hypothesis that the evidence later refuted — kept here because the reasoning is correct in general, just not the cause here.

D20. Why does round-robin degrade with variable request durations?

Round-Robin (RR) equalizes request count, but actual load is count × duration. With uniform durations, equal count ≈ equal load. With high variance, a target that draws a long request still receives its turn-by-turn share of new requests (count-fair) while it's busy, so its backlog grows while targets handling short requests finish quickly and idle. RR is blind to how busy each target actually is.

D21. The feedback-loop difference vs least-outstanding-requests?

RR has no feedback — it rotates regardless of backend state. Least Outstanding Requests (LOR) keeps a per-target in-flight counter and sends each new request to the least-busy target, so a stuck target stops receiving work until it drains. LOR closes the feedback loop that RR lacks.

D22. What is LOR equivalent to, and what does AWS advise?

It's AWS's name for classic least-connections (nginx least_conn, HAProxy leastconn). AWS guidance: round-robin when requests and targets are similar; least-outstanding-requests when request complexity varies or targets differ in capacity — exactly the 0.01–30s spread we saw.

D23. "Head-of-line blocking" appeared at two layers — name both.

ALB/target layer: Under RR, requests queue behind a slow request on a busy target instance.
Inside the pod: The single async worker's event loop is blocked by a CPU-bound request, freezing other in-flight requests on that same worker.

Same phenomenon, two different layers — and the second turned out to be the real root cause.

D24. Why can't a snapshot prove RR is the cause, and what would?

A snapshot lacks a time axis, so it can't show whether the hot pod persists or rotates, nor whether requests spread evenly by count while load concentrates. Proof needs a time series: per-pod CPU through a burst, or per-target request-count and latency over time.

D25. What made the round-robin conclusion wrong?

Two things: the per-pod CPU time series showed all pods evenly loaded (no pinned pod), and the target group is in instance mode, where the ALB balances across nodes — not pods — so its algorithm can't even produce a per-pod imbalance. Both facts independently killed the theory.

E. Target Mode: Instance vs IP

False Trail

The topology detail that demolished the LB-algorithm fix and reframed the problem.

E26. `instance` vs `ip` target mode — how do you tell which you have?

instance: Targets are EC2 nodes (ALB → node:NodePort → kube-proxy → pod).
ip: Targets are pod IPs directly.

Tell from the registered targets (instance IDs i-... vs pod IPs) and the target-group port (a NodePort like 31892 ⇒ instance; a container port like 8080 ⇒ ip).

E27. 10 instances registered on 31892 — what does that imply about the path?

The ALB sends to nodes on the NodePort, then each node's kube-proxy forwards to a pod. So the ALB's balancing decision is across nodes, and there's a second hop (kube-proxy) that actually picks the destination pod.

E28. NodePort, and `externalTrafficPolicy: Cluster` vs `Local`?

NodePort exposes the service on a static port on every node.

Cluster: Any node may forward to any pod cluster-wide (extra network hop, source IP is SNAT'd).
Local: A node only forwards to pods running on itself (preserves client IP, no second hop, but a node with no local pod fails LB health checks).

E29. Trace the full request path in this topology.

client → ALB (round-robins across 10 nodes) → node:31892 → kube-proxy (iptables/IPVS, effectively random across all 4 pod endpoints cluster-wide, possibly a second network hop to a pod on a different node) → application master → a worker process.

E30. Why wouldn't LOR fix per-pod imbalance here?

LOR would balance outstanding requests across EC2 nodes, but kube-proxy then picks the destination pod randomly afterward. The ALB neither sees nor controls which pod serves a request, so per-pod imbalance is completely untouched by altering the ALB algorithm.

E31. What two changes together would enable load-aware per-pod routing?

Switch target-type to ip (ALB targets pods directly, removing the kube-proxy hop) and set least_outstanding_requests. It's a much bigger change: ip mode alters the data path, requires specific VPC CNI / security-group wiring for ALB-to-pod traffic, and the algorithm annotation applies ingress-wide.

E32. Why were ALB access logs useless for the per-pod question?

In instance mode, the access log records the EC2 node as the target (node IP:31892), not the individual pod — so it cannot reveal per-pod traffic distribution.

F. Getting the Proof

Replacing inference with a time series — the turning point.

F33. What is `kube-prometheus-stack` and where did it live?

The Prometheus Operator bundle — Prometheus, Alertmanager, Grafana, kube-state-metrics, and node-exporter. It ran in the dedicated monitoring namespace.

F34. Why is `kubectl port-forward` an acceptable read-only way to query it?

It opens a local secure tunnel (localhost → pod/service port) through the Kubernetes API server. It proxies a connection without mutating cluster state — so it's completely safe for querying metrics.

F35. What PromQL gives per-pod CPU over time, and why `rate(...)`?

rate(container_cpu_usage_seconds_total{pod=~"core-service-api.*"}[1m]). The metric is a monotonic CPU-seconds counter; rate(...[1m]) turns it into cores-used-per-second (i.e., CPU cores used) over the window. The raw counter alone is meaningless for "how busy is it right now".

F36. All four pods even at 0.5–0.95 cores — why did that refute imbalance?

The imbalance theory predicted one pinned pod at 100% and the rest idle. The time series showed the exact opposite — every pod working evenly and hard, none starved — so load was already naturally balanced. There was nothing to rebalance; the LB algorithm was irrelevant.

F37. Why does a ~0.9-core plateau under a 1.5-core limit point at the process model?

A CPU-bound workload that flatlines near exactly one core despite a 1.5-core allowance is the classic fingerprint of a single-process / single-threaded execution ceiling — the pod cannot push past ~1 core, which points straight at having a single execution worker.

G. The Real Root Cause

Confirmed

Application server workers, runtime resource locks, and an async worker doing synchronous CPU work.

G38. What is an application server worker vs the master?

A worker is an OS process running a full copy of the application code and handling incoming HTTP requests. The master process binds the network socket, spawns/monitors/restarts workers, and handles reloads — but serves no requests itself.

G39. What is a Global Interpreter Lock (GIL) or single-threaded runtime constraint and its consequence?

It prevents multiple threads from executing native bytecode at the same time per process. A single process yields about one core of CPU-bound throughput regardless of how many threads are created or how many cores are allocated to the container.

G40. What do `workers = 1` and `worker_class = "async_class_name"` do?

workers = 1 creates exactly one worker process. The worker class sets each worker to run an asynchronous event loop, designed to juggle many concurrent connections via async non-blocking I/O multiplexing.

G41. Sync vs async worker — when does each shine?

Sync: Handles one request at a time to completion; concurrency comes from adding more worker processes — ideal for CPU-bound tasks.
Async: Handles many concurrent requests on a single loop — but is only an advantage when requests spend most of their time waiting on I/O.

G42. Why is async wrong for CPU-bound data processing work?

An event loop can only switch tasks at await points (non-blocking I/O waits). A heavy synchronous CPU computation holds the thread and the runtime lock tightly and never yields, meaning the entire event loop is blocked for the duration of the calculation — freezing every other concurrent request on that worker. Async buys nothing for CPU-heavy tasks and introduces massive head-of-line blocking.

G43. So what's the per-pod and fleet concurrency for heavy requests?

Per pod: Effectively one heavy request in flight at a time (one worker, loop blocked during CPU work).
Fleet: 4 pods × 1 worker ⇒ a strict fleet ceiling of about 4 concurrent heavy data extractions.

G44. How did we confirm worker count and CPU quota from inside the pod?

cat /sys/fs/cgroup/cpu.max → 150000 100000 = quota/period = 1.5 cores.
ps | grep -c app_server → confirmed 1 master + 1 worker, meaning workers=1.

The Prometheus plateau and the running configuration now perfectly agreed.

G45. Why was the 1.5-core limit largely unusable?

Because one worker ≈ one core max due to the process lock, the extra ~0.5 core of the 1.5 limit could never be consumed by CPU-bound work. The resource allocation was there on paper, but the process model couldn't physically use it.

G46. State the full causal chain, burst → alarm.

A traffic burst arrives → with only ~4 concurrent-request fleet capacity, excess requests queue up immediately. On each pod, the single event loop is completely blocked by the in-progress CPU-bound processing task → per-request latency climbs dramatically into seconds (max ~30s) → the 60s average crosses 0.8s for two consecutive periods → the CloudWatch alarm fires. Once the burst passes, the queue drains, the average falls, the alarm clears, resulting in a flapping pattern.

H. Designing the Fix

H47. Vertical, horizontal, or both — and why?

Vertical (more workers/CPU per pod) adds concurrency instantly to existing running pods.
Horizontal (more pods via HPA) adds capacity but provisions slowly due to node provisioning.

Both won: vertical gives immediate relief on the real bottleneck; HPA adds elastic burst headroom. Horizontal alone is too slow for sudden short bursts; vertical alone has no safety ceiling.

H48. Why not just switch to sync workers / offload to a pool?

The application code is fundamentally built on top of an async framework with coroutine request handlers. Switching to sync workers or offloading heavy work to a background process pool requires rewriting structural parts of the app — an application-code change, which is out of scope for an emergency infrastructure hotfix.

H49. Why raise the CPU limit to 2000m alongside `workers=2`?

Two workers need up to ~2 full cores. Leaving the limit at 1500m would CPU-throttle them (two processes fighting over 1.5 cores), meaning neither could run a full core and defeating the change. The limit rose to 2000m to back two full workers.

H50. Why expect ~5Gi at two workers, and why won't application preloading help?

Each worker builds its own heavy data models/caches in memory at runtime (via a warmup endpoint or on initial requests), so memory scales near-linearly with workers (~2.5Gi → ~5Gi). Preloading only shares state loaded at import time before the process forks (copy-on-write); caches built post-fork are private to each worker process.

H51. Why is the import-time model load shareable but the runtime caches aren't?

The base library imports run at module load time — with preloading, this data lives in the master process before the fork and is shared copy-on-write. The actual execution caches are built after the fork, per worker, making them private anonymous memory.

H52. Why did the HPA never scale originally?

The targetCPUUtilizationPercentage is a percentage of the CPU request (1200m). Traffic bursts pushed pods to ~0.9 cores ≈ 75% — just under the 80% trigger — and the bursts were too short for the time-averaged utilization metric to sustain above 80%. So the cluster sat flat at 4 replicas.

H53. Percent of what? Recompute at the new request.

It is a percentage of the CPU request. At the new 1500m request, 0.9 cores ≈ 60%. Under real heavy load, two workers can use up to ~2 cores ≈ 133% of the request — far over the new 50% target, which easily triggers scaling.

H54. Why lower the target to 50% and raise max to 10?

Lowering the target makes it scale much earlier during short bursts; raising max gives the cluster room to scale out. A lower target without a higher max caps out too early; a higher max without a lower target would still fail to trigger.

H55. What does the topology spread constraint do, and why soft?

maxSkew: 1 / topologyKey: kubernetes.io/hostname ensures the per-node count of these pods stays within 1 of each other, spreading them evenly across nodes. Soft (ScheduleAnyway) makes it a preference: if node capacity is scarce, it will still schedule pods (just less evenly) rather than leaving pods stuck in Pending. Hard (DoNotSchedule) could block scheduling entirely and cause a self-inflicted outage.

H56. Why deliberately omit the readiness probe?

A readiness probe shares the exact same event loop that CPU-bound requests block. Under sustained load, the probe would time out on every busy pod → all pods would flip to NotReady at once → all would be removed from the ALB → causing wide-scale 503 errors, which is worse than slow responses. Without a readiness probe, pods stay in rotation (slow but serving). Revisit this once loop-blocking is fixed in code.

E57. The container port vs Service target port situation?

The application server binds to port 8080 and the Kubernetes Service targetPort was set to 8080 (numeric), so traffic already routed correctly; containerPort: 80 in the manifest was just an incorrect documentation declaration. Changing it to 8080 aligns the documentation with no risk to production routing.

H58. Why was deleting the separate HPA file safe?

The deployment pipeline only applies k8s.prod.yml (which has its own inline HPA definition); the old hpa-prod.yml was never referenced anywhere — confirmed by searching the repo and workflows. It was a confusing duplicate file, safe to remove.

I. Where the Config Lives & Deploy Mechanics

I59. How did we know it wasn't GitOps-managed despite the tool being installed?

The live Deployment carried a kubectl.kubernetes.io/last-applied-configuration annotation — the clear fingerprint of a standard kubectl apply execution. GitOps-managed objects (like Argo CD) carry distinct tracking annotations instead, and querying the tool showed it only managed specific preview apps.

I60. Which repo/file holds the Deployment vs the ingress, and why different mechanisms?

Deployment/HPA: org/core-service → deploy/k8s.prod.yml, shipped by the repo's tag-triggered GitHub Actions (kubectl apply).
Ingress: org/infra-k8s → .../platform/router/main.tf, shipped via terraform apply.

They have different lifecycles — application config moves with the app's release tags, while cluster routing is foundational infrastructure managed in Terraform.

I61. Describe the production pipeline end to end.

Trigger: Pushing a git tag matching v*-prod.
Jobs: test (unit tests) → build (docker build + push to AWS ECR) → deploy-prod (sed string replacement on the tag, then kubectl apply executed on an in-VPC self-hosted runner) → release-prod (creates GitHub release).

I62. Role of the `$TAG` placeholder?

It acts as a placeholder string in the manifest's image declaration line; the deploy step uses sed to substitute it with the exact pushed tag (e.g., v1.14.4-prod) so the applied manifest pulls the freshly built image. The value is sourced from the pipeline context (GITHUB_REF).

I63. Why does merging to main branch not deploy, but a tag does?

The automation workflows do not trigger deployments on main branch pushes; the CD workflow is configured to trigger only on a v*-prod tag. Deployment is therefore a deliberate, separate engineering act — explicitly cutting and pushing the tag.

I64. Why does the self-hosted runner matter for a private cluster?

The runner instance is situated inside the VPC, meaning it can natively reach the Kubernetes cluster's private API endpoint to run kubectl commands — even when an engineer's local machine is off the VPN.

J. Capacity Analysis

J65. What instances back the general-purpose tier, and how big?

t3.xlarge / t3a.xlarge (and similar compute variants in the node group) — 4 vCPU, ~3920m allocatable CPU, ~14.4Gi allocatable memory each.

J66. Cluster Autoscaler vs Karpenter — which is here?

Cluster Autoscaler scales fixed-shape Auto Scaling Groups (ASGs) based on pods being marked as unschedulable (Pending); Karpenter provisions right-sized nodes directly on demand from pending pod resource definitions without predefined groups. This cluster uses Cluster Autoscaler (the alternative tool was removed during a node group migration).

J67. The two node groups and the combined ceiling?

gen-purpose (Spot instances): min 2 / max 5.
gp-od (On-Demand fallback): min 0 / max 5.
Combined ceiling: 10 total nodes.

J68. Do the packing math — why is CPU the binding constraint?

(3920m allocatable − ~400m daemonset overhead) ÷ 1500m request ≈ 2 pods per node. Memory utilization at 2 pods would be 2 × 5Gi + ~1Gi system overhead ≈ 11Gi, well under the 14.4Gi allocatable. A 3rd pod would require more CPU than the node possesses (4500m > 3920m). Therefore, CPU exhaustion prevents further packing before memory runs out.

J69. At HPA max (10 pods), how many nodes, and does it fit?

10 pods ÷ 2 pods per node = 5 nodes. This is completely within the 10-node group ceiling, leaving 5 nodes available for other workloads in the tier. It fits safely.

J70. Why is scale-out slow relative to the bursts?

Spot instance capacity acquisition wait + node OS boot and cluster join + 726MB docker image download pull time (~48s) + startup probe warmup execution (up to 300s) ≈ 3–6 minutes total before a brand-new pod can actively serve traffic. This latency struggles against brief 5–8 minute traffic spikes.

J71. Which part of the fix gives immediate relief?

The vertical workers=2 change — every single existing pod immediately doubles its concurrent capacity for handling heavy requests. HPA horizontal scale-out serves as a slower second line of defense and may not even complete processing within a short burst window.

J72. Why bump the memory request to 5Gi if density is unchanged?

5Gi matches the actual steady-state memory utilization of the dual-worker process model, making the scheduler's reservation honest and avoiding over-packing followed by sudden runtime evictions. Density remains unchanged because the CPU request (1500m) already limits placement to 2 pods per node.

K. Staging-First Deploy & Verification

K73. Why deploy to staging first when the staging manifest file wasn't changed?

The workers=2 code and environment configuration is baked directly into the Docker image artifact, which staging runs. Thus, deploying to staging smoke-tests the internal behavior of the new container version before hitting production — even though the staging infra manifest itself remains unchanged.

K74. What does staging validate, and what does it not validate?

Validates: Image build/push mechanics, container runtime startup with multiple workers, startup probe caching execution, and basic API routing correctness (/health and live requests).
Does NOT validate: Production resource sizing under load, HPA auto-scaling dynamics, or topology spread rules (as those are production-only manifest keys). Those configurations were verified via client-side dry-runs and theoretical capacity analysis.

K75. How is staging isolated, and what risk did we flag?

Staging runs as a distinct Deployment and Service (core-service-api-staging) inside the staging namespace on the same shared cluster. Risk: Running workers=2 inside a tiny 0.5-core staging allocation means both workers are severely CPU-throttled. The startup warmup steps could slow down and exceed the 300s threshold, triggering an artifact crashloop that is a byproduct of staging constraints rather than an application defect.

K76. List the staging verification steps and pass criteria.

Pipeline checks pass green → Rollout completes successfully with zero container restarts → Verify process table via container exec (1 master + 2 workers) → HTTP GET /health returns 200 → Sample heavy request returns 200 with correct output payloads → Logs are clean. All must be green to pass.

K77. Why test both `/health` and a real request route?

The /health endpoint only verifies that the web server socket is open and listening (it is trivial and lightweight). Sending a real processing request confirms that the heavy data models have loaded correctly into active memory and that the application can execute computations under the new worker configuration.

K78. What does the warmup endpoint do, and why is it the likely failure point?

It builds and primes the heavy in-memory entity models for data extraction. It is the most intensive step during application boot and is gated by the startup probe; on a resource-starved container, this is the most likely location for a timeout failure to occur.

L. Production Rollout & Confirmation

L79. Why a patch bump (`v1.14.4`)?

It represents an operational configuration and infrastructure hotfix with absolutely no changes to the external API surface or core application behavior, following v1.14.3 — making it a patch version under semver guidelines.

L80. Two new pods went Pending — what happened next?

The larger 1500m CPU / 5Gi memory pods could not fit on the existing allocated nodes. A TriggeredScaleUp event was successfully logged by the Cluster Autoscaler, a new node was provisioned and joined the cluster, and the pending pods scheduled onto it — behaving exactly as predicted in the capacity analysis.

L81. Why did the ~6-minute rollout drop no traffic?

A RollingUpdate deployment replaces pods incrementally based on maxSurge and maxUnavailable definitions. Each new pod is required to completely pass its lengthy startup probe before an older pod is decommissioned out of the active routing pool, ensuring continuous handling capacity.

L82. Why only ~2.6Gi post-deploy, not ~5Gi?

Only the initial worker that processed the startup warmup request had fully populated its internal cache; the second worker process was still cold (having received no user traffic yet). Memory usage scales up toward ~5Gi as the second worker processes requests under load, which our sizing footprint anticipates.

L83. ~3s max requests persisted, yet the alarm stayed OK — why is that consistent with success?

The CloudWatch alarm tracks the mathematical average. Individual 3-second maximum latencies are expected due to the variable nature of heavy data workloads, but because 2 workers × 4 pods are now absorbing the volume, the aggregate average stays low (0.1s - 0.3s). Tail latency maxes can persist while keeping the alarm green (confirming the point in Q7).

L84. What's the honest gap in the 45-minute monitoring evidence?

The monitoring window captured only moderate off-peak traffic patterns, not a worst-case peak morning burst. The fix is not yet fully proven against maximum production volume; observing a morning traffic peak stay entirely under the latency threshold is required to definitively close out the incident ticket.

L85. HPA stayed at 4 pods — why is that good, not a non-event?

The vertical optimization change alone provided enough per-pod concurrency to keep individual utilization comfortably below the scaling threshold under the observed load. The horizontal scaling capacity is a safety cushion held in reserve, not a failure of the fix.

M. Operational & Process Gotchas

M86. Why does `sts get-caller-identity` succeed while `kubectl` times out?

The Kubernetes cluster control plane API endpoint is configured as strictly private. kubectl requires a path through the internal VPN network to resolve and route traffic. The AWS STS command calls public AWS API endpoints, succeeding regardless of local network routing. Takeaway: AWS SSO success combined with a kubectl I/O timeout or context deadline error indicates a local network/VPN connectivity issue, not an authentication problem.

M87. Why did the deploy still work when local `kubectl` couldn't?

The CI/CD pipeline's deployment step executes directly on a self-hosted runner instance stationed natively inside the project VPC, giving it a direct internal network path to the private cluster endpoint independent of local engineer connectivity.

M88. Diagnose the two CI failures — were they ours?

security-check: The internal package scanner attempted to run an interactive prompt setup in an environment with no standard input (stdin), resulting in an EOFError crash (a regression in the testing utility version).
image-scan: The automated vulnerability scan flagged pre-existing CVE thresholds within upstream base dependencies.

Neither issue was introduced by our configuration change (the Dockerfile and application dependencies were completely untouched), and historical runs across unrelated branches confirmed identical global failures.

M89. `BLOCKED` but branch protection returned a 404 — resolve the contradiction.

The repository does not use legacy branch protection rules (hence the API returning 404), but instead implements modern repository compliance rulesets. Rulesets represent a distinct platform feature and must be queried via specific ruleset endpoints. The merge block was enforced by an active ruleset requirement.

M90. What actually gated the merge?

A ruleset requirement mandating exactly 1 peer-approved review. The automated status checks were not configured as blocking gates within the ruleset, which allowed the branch to be merged despite the red CI scanner flags once a manual review was submitted.

M91. Which actions needed explicit confirmation, and why?

Any operation writing state changes to production infrastructure or mutating master source tracking (e.g., executing infrastructure applications, pushing release tags, merging to master, modifying tracking tickets). These actions are highly visible and difficult to cleanly reverse, necessitating a strict "confirm-before-acting" protocol.

M92. Commit-authorship convention here?

Commits must enforce verified corporate identity configurations using strict compliance controls, explicitly avoiding automated local environment placeholders or any explicit AI author attribution footers in either commit histories or pull request descriptions.

N. Synthesis & Transfer

The Real Mastery Test

N93. Paged cold — first five moves, in order?

describe-alarms → Check current metric state, threshold definitions, and exact configurations.
describe-alarm-history → Determine if the issue is a flapping pattern and correlate the start time against the cluster deployment log.
Break down TargetResponseTime and RequestCount by Target Group → Identify the specific failing service and check if the degradation is load-correlated.
kubectl get/describe pods and top → Check pod health, restart loops, resource tracking, and probe status.
Analyze Prometheus per-pod CPU over time series → Evaluate if workload distribution is balanced and look for single-core saturation plateaus.

Each progressive step narrows the operational scope: Alarm → Service → Pod → Process.

N94. Name three plausible-but-wrong hypotheses and the one fact that kills each.

Low-traffic statistical artifact: Contradicted by clear increases in total request volume coinciding exactly with the latency spikes.
Load balancer round-robin routing imbalance: Disproven by historical Prometheus time series showing even CPU utilization across all pods, alongside an instance-mode infrastructure topology that prevents pod-level load steering.
The recent application deployment: Refuted by historical alarm logs demonstrating that the latency flapping pattern was active hours prior to the deployment timestamp.

N95. If the CPU series had shown one pinned pod, what's the real fix in instance mode?

Applying a load balancer annotation wouldn't work because an ALB operating in instance mode cannot steer individual pod traffic behind a NodePort hop. The structural resolution would require changing the target mode configuration to ip (paired with LOR) to give the ALB direct visibility into individual pods, or rectifying the balancing mechanics at the internal proxy layer. Most importantly, per-pod concurrency limits would need to be raised to prevent single long-running requests from starving concurrent traffic.

N96. When is the correct long-term fix an application change?

When the core performance bottleneck stems directly from the internal concurrency model of the runtime — such as running compute-intensive, synchronous operations inside a single asynchronous event loop thread. The robust architectural solution requires modifying the application code: transitioning to synchronous worker processes mapped to available cores, or offloading compute tasks onto background worker queues so the main request loop remains fully responsive. Infrastructure configuration tweaks merely act as mitigation.

N97. Generalize: what makes "more replicas behind round-robin" fail, and what helps?

The Failure Property: Significant variance in per-request computation times, coupled with a routing layer that distributes traffic based purely on request count without visibility into backend state, running on replicas with restricted internal concurrency. Adding replicas fails because the actual load (not request count) is severely unbalanced, leaving individual instances susceptible to head-of-line blocking.
What Helps: Implementing load-aware routing metrics (e.g., Least Connections / Least Outstanding Requests), increasing true parallel processing capability within each discrete replica, or refactoring the code to handle heavy calculations asynchronously outside the primary request path. You must provide capacity where the work actually serializes.

N98. If the morning peak still trips it — next two levers, and how to choose?

Optimize Horizontal Scaling Responsiveness: Increase minimum running replicas ahead of time or adjust the auto-scaler to be more aggressive. Evaluate per-pod CPU limits, HPA state machine transitions, and raw request velocity during spikes to verify if additional pre-warmed instances can absorb the peak.
Refactor Application Concurrency: Rewrite the request handling layer to run synchronous workers or execute compute tasks via isolated worker pools. Check detailed profiling data to confirm if event loop starvation continues to limit per-pod throughput.

Choose based on whether the exhaustion is caused by total cluster capacity constraints (→ scale out) or localized thread blocking (→ refactor application).

Elastic Fleet

2026-06-25T13:36:56.911+01:00

Elastic Agents do not strictly require Fleet and Fleet Server. You can deploy them in standalone mode, which allows you to manually configure and manage them using local configuration files.

However, running Elastic Agents in Fleet-managed mode with a Fleet Server is the recommended best practice for most enterprise environments.

Here is how the two approaches compare:

1. Fleet-Managed Mode (Recommended)

In this setup, you use the Fleet UI in Kibana to centrally manage agent policies, roll out upgrades, and apply integrations.

How it works: Agents connect to a Fleet Server (which is just a specialized Elastic Agent process), which then receives policies from Elasticsearch and pushes them to your endpoints.
Best for: Large-scale deployments, continuous monitoring, and Elastic Security/Endpoint integrations.
Action: To set this up, refer to the official Elastic Agent Installation Guide.

2. Standalone Mode

In this setup, you manually install the agent and define its inputs, outputs, and integrations directly in a local YAML configuration file.

How it works: The agent connects directly to outputs like Elasticsearch or Logstash without a Fleet Server intermediary.
Limitations: Central management, automated upgrades, and certain advanced Endpoint Security features are disabled.
Best for: Edge cases, highly air-gapped networks, or evaluating specific integrations.
Action: To configure this, use the steps outlined in the Standalone Elastic Agent Tutorial.

Example of how an Elastic Fleet can actually be wired

Components & where they live

Everything is in cluster company-prod-elastic-eks (EKS, us-east-2, now on v1.36.2), namespace elastic-system, across 5 nodes in 3 AZs:

Node AZ Node Group Runs

==== == ========== ====

ip-10-99-44-1 2a es MNG (m7g.2xlarge) agent

ip-10-99-55-34 2b default MNG (m5.large) agent + Fleet Server pod

ip-10-99-55-89 2b es MNG (m7g.2xlarge) agent

ip-10-99-66-51 2c es MNG (m7g.2xlarge) agent

ip-10-99-66-78 2c default MNG (m5.large) agent

Fleet Server — Deployment fleet-server-prod, replicas: 1 (single pod, currently on the 2b node). Listens on :8220 HTTPS. Exposed by a NodePort service fleet-server-prod-agent-http (8220 --> 32202 on every node).
Elastic Agents — DaemonSet agent-prod-eck-agent, one pod per node (5 total), spread across all 3 AZs. hostNetwork: true, mode: fleet, FLEET_INSECURE=true. They collect node/pod logs + metrics (hostPath mounts of /var/log/...).
ALB k8s-elastics-eckfleet-a1b2c3d4e5 — internal (not internet-facing), spanning subnets in 2a / 2b / 2c. Created by the AWS LB controller from Ingress eck-fleet-server-prod-ingress. Targets = all 5 nodes' NodePort 32202 (instance mode), backend-protocol: HTTPS, idle_timeout: 300s.
DNS — eck-fleet-server.internal-domain.local (Route53 via external-dns) --> the internal ALB. Elasticsearch is a separate endpoint, elasticsearch.internal-domain.local:443.

Who listens, who initiates

The key thing: every connection is initiated by the agent (outbound). Nothing is ever pushed to the agents — that's why this works with agents behind hostNetwork and no inbound rules of their own.

┌───────────────────────────────────────────────┐

│ cluster company-prod-elastic-eks / elastic-sys│

┌──────────┐ │ │

│ Agent │ check-in │ internal ALB Fleet Server (1 pod) │

│ DaemonSet│─────────►│ eckfleet-a1b2... ┌──────► :8220 (HTTPS) │

│ (5 pods, │ :443 │ :443 TLS ─────────┘ NodePort 32202→8220 │

│ 3 AZs) │ HTTPS │ (internal cert) kube-proxy → the 1 pod │

└────┬─────┘ │ ▲ re-encrypt HTTPS to a node's :32202 │

│ │ │ │ │

│ ship data └───────┼─────────────────────┼─────────────────┘

│ (logs/metrics) │ DNS │ writes .fleet-* / agent

▼ eck-fleet-server. ▼ metadata, reads policy

elasticsearch. internal-domain.local Elasticsearch + Kibana (Fleet app)

internal-domain.local:443 ◄──────────────────────

(NOT via the fleet ALB)

Traffic Flows: Breakdown

1. Agent --> Fleet Server (Control Plane: Enrollment & Policy Check-in)

The agent acts as the client, and Fleet Server listens on :8220. The agent dials https://eck-fleet-server.internal-domain.local:443:

Traffic Path:

agent --> DNS --> internal ALB :443 (TLS termination, internal cert) --> re-encrypt --> a node's NodePort :32202 --> kube-proxy --> the single Fleet Server pod :8220

Cross-AZ Routing:

Because the agent always targets the ALB DNS name (not the pod directly), an agent can land on any node's NodePort. It is then forwarded by kube-proxy to whichever node is hosting the single Fleet Server pod—making cross-AZ hops standard behavior.

The Long Poll Mechanism: The check-in is a long poll. The agent opens a connection, and Fleet Server holds it open for up to ~5 minutes until a policy change occurs or the poll times out.

The Latency "False Positive": This held-open duration is exactly what the ALB records as a TargetResponseTime of ~300s, which triggered the false-positive alert. The ALB's idle_timeout is intentionally set to 300s to support these long polls. Conversely, the health check (/health, expects 404, 300s interval) is a separate lightweight probe, explaining why targets report as perfectly healthy while "latency" metrics look highly skewed.

2. Agent --> Elasticsearch (Data Plane: Telemetry Shipping)

This data path does not traverse the Fleet ALB. The Fleet output configuration points agents directly at https://elasticsearch.internal-domain.local:443.

Consequently, the eckfleet ALB only handles control-plane check-in traffic. This explains why its request volume is incredibly low (~3–5 requests/min) while individual request durations hover around 300s. They are two entirely separate paths utilizing two different load balancers.

3. Fleet Server --> Elasticsearch + Kibana

Fleet Server (via elasticsearchRefs / kibanaRef) acts as the client here. It writes agent and policy metadata directly into the .fleet-* indices and coordinates with the Kibana Fleet application, where the eck-fleet-server and eck-agent policies are managed.

One-Line Summary

Agents (DaemonSet, 5 pods across 2a/2b/2c) long-poll the single Fleet Server pod for policy updates via the internal ALB on :443 --> NodePort 32202 --> :8220, but ship their actual telemetry data directly to Elasticsearch, completely bypassing that ALB. The ALB only processes the slow, held-open control traffic—which is the root cause of the alarm.

Why does fleet server holds connection open for so long? Why it does not reply immediately?

It's a deliberate design choice called long polling, and it's the opposite of a bug. Here's the reasoning.

The problem Fleet is solving

Fleet Server's job is to push policy changes out to agents promptly — a new integration, a changed log path, an output credential rotation. Agents need to find out "has my policy changed?" with low latency.

There are three ways an agent could learn about changes:

1. Short polling — agent asks "anything new?" every N seconds, server replies instantly "no", agent sleeps, repeats.

2. Server push — server opens a connection to each agent when something changes.

3. Long polling — agent asks "anything new?", and the server holds the request open until either something actually changes or a timeout fires.

Fleet uses #3, and the connection sitting open for ~5 minutes is that hold.

Why not reply immediately (short polling)?

If Fleet answered every check-in instantly with "nothing changed", then to get fast reaction to a policy change, agents would have to poll very frequently — say every few seconds. With your 5 agents that's tolerable, but Fleet is built to manage thousands to tens of thousands of agents from one server. At that scale:

- Frequent short polls = a constant storm of requests, almost all of which return "no change". Huge wasted CPU/network on both ends.

- To keep reaction time low you'd poll more often, which makes the storm worse. To cut the storm you'd poll less often, which makes policy changes take longer to land. There's no good setting.

Long polling breaks that trade-off: the agent gets a near-instant reaction to a real change (the server responds the moment policy updates) and there's almost no idle chatter (one held-open connection per agent instead of hundreds of empty round-trips).

Why not server push (#2)?

Pushing would mean the server initiating connections inward to every agent. Agents are all over the place — behind NAT, firewalls, in private subnets, on laptops, on hostNetwork pods like yours. The server usually can't reach them, and you'd need inbound rules everywhere. Long polling flips it: the agent always dials out to the server, the connection is already established and held open, and the server pushes the change down that existing agent-initiated connection the instant it happens. You get push-like latency with poll-like (outbound-only) connectivity. That's exactly why your agents work behind hostNetwork with no inbound exposure.

So what's actually happening in those ~300s

A check-in is essentially: "Here's my current state; tell me the moment my policy differs." The server parks that request. Two things can end it:

- A policy change occurs → server responds immediately with the new policy (could be 2 seconds in).

- Nothing changes → the server lets the request time out at its poll ceiling (~5 min), responds "no change", and the agent immediately opens a fresh one.

Since your policies rarely change, almost every check-in runs the full clock and returns at ~300s. That's the held connection the ALB measures as TargetResponseTime.

Why this specifically fools the ALB

From the ALB's point of view, "request received → response sent" took 300 seconds, so it reports TargetResponseTime ≈ 300s. The ALB can't tell the difference between "the backend was slow for 300s" (bad) and "the backend intentionally held an idle long-poll for 300s" (normal). That ambiguity is the whole reason your generic 0.8s threshold misfires — and why the right fix (DOP-833) is to exempt this endpoint rather than treat it as latency. It's also why the Ingress sets idle_timeout.timeout_seconds=300: the ALB has to be told to tolerate the held connection, otherwise it would sever the long poll before the agent's poll cycle completes.

In short: Fleet holds the connection open so it can deliver policy changes near-instantly without either hammering the server with empty polls or needing to reach inward to firewalled agents. The ~300s is just an idle long-poll waiting for a change that usually never comes during that window — efficient by design, and only a problem for a latency metric that doesn't know to expect it.

----

AWS EC2: Application Load Balancer

2026-06-25T13:11:38.955+01:00

An Application Load Balancer (ALB) is a fully managed AWS service that automatically distributes incoming HTTP and HTTPS traffic across multiple backend targets.

It operates at the Application Layer (Layer 7) of the Open Systems Interconnection (OSI) model.

Key Features:

Content-Based Routing: Routes traffic based on URL paths (/api vs /images) or hostnames (://example.com).
Container Support: Integrates directly with Amazon ECS and EKS using dynamic port mapping.
Advanced Protocols: Native support for modern protocols like HTTP/2, gRPC, and WebSockets.
Security Integration: Features built-in HTTPS/TLS termination and integrates directly with AWS WAF for web security.

How Components Work Together

Listener: Evaluates connection requests from clients using protocols and ports you configure.
Rules: Determines how the load balancer routes requests to its registered targets.
Target Group: Groups backend resources (like EC2 instances, containers, or IP addresses) that receive the traffic

How ALB health checks keep applications online?

An Application Load Balancer (ALB) keeps your application online by continuously monitoring the health of your backend targets and dynamically redirecting traffic away from failing nodes.

1. Automatic Traffic Redirection

The ALB sends periodic ping requests (health checks) to every registered target. If a target fails to respond correctly, the ALB marks it as unhealthy and immediately stops sending user traffic to it. Traffic is rerouted to the remaining healthy nodes with zero downtime for the user.

2. Auto Scaling Integration

When paired with an Auto Scaling Group (ASG), ALB health checks can trigger the automatic replacement of broken instances.

The Problem: An EC2 instance might be running (healthy at the hardware level), but the web server inside it has crashed (unhealthy at the application level).
The Solution: The ALB tells the ASG that the instance is failing application health checks. The ASG terminates that specific broken instance and launches a fresh, working one.

3. Graceful Recovery

When an unhealthy instance recovers, or when a new instance is launched, the ALB does not send traffic to it immediately. It enters an initial state and undergoes consecutive successful health checks. Only when it passes the threshold does the ALB safely introduce it back into the traffic rotation.

How to Configure an ALB Health Check

You configure health checks inside the Target Group settings using these parameters:

Parameter What it does Recommended Setting

======== ========= =================

Health Check Path The URL endpoint the ALB hits

(e.g., /health or /index.html). /health

Healthy Threshold Consecutive successes needed to mark a target as healthy. 3

Unhealthy Threshold Consecutive failures needed to mark a target as unhealthy. 2

Timeout How long the ALB waits for a response before failing. 5 seconds

Interval The time between individual health check pings. 30 seconds

Success Codes The HTTP status codes that prove the app is working. 200 (or 200-399)

When you configure an ALB, you do not select an Availability Zone (AZ) directly; instead, you must select at least two subnets in different Availability Zones to ensure high availability.

How it works: AWS places a load balancer node in each of the specified subnets.
The AZ link: Because each subnet belongs to exactly one AZ, this fundamentally binds the ALB's nodes to those corresponding Availability Zones.
Custom routing: You can modify the subnets via the Update Availability Zones settings in the EC2 Console at any time.

Public ALB

Binding an ALB to public subnets makes it a public (internet-facing) load balancer.

When you create an internet-facing ALB, AWS requires you to select public subnets so the ALB nodes can receive a public IP address and route traffic from the internet.

Key Characteristics:

Public DNS: The ALB receives a public DNS name that resolves to public IP addresses.
Internet Gateway: The selected public subnets must have a route to an Internet Gateway (IGW) in their route tables.
Target Routing: Even though the ALB is public, it can still route traffic to EC2 instances living in private subnets

An internet-facing ALB routes traffic directly to the individual backend targets (such as EC2 instances or IP addresses), not to the private subnets themselves.

How Routing Works

Target Group Config: You configure the ALB to route traffic to a Target Group.
Direct Node Communication: The ALB nodes in the public subnets communicate directly with the private IP addresses of your backend nodes.
Cross-Subnet Traffic: AWS handles this routing internally via the VPC router, allowing the public ALB to securely traverse into private subnets.

Configuration Checklist

VPC: Both the public subnets (where the ALB lives) and the private subnets (where the nodes live) must be in the same VPC.
Security Groups: The private instances must have a security group that allows inbound traffic from the ALB's security group

Private ALB

An internal (private) ALB routes traffic in the exact same way as a public ALB, but it is only accessible within your VPC or connected networks.

It routes traffic directly to individual backend targets, not to subnets.

Key Characteristics

Private Subnets: You deploy the ALB nodes into private subnets.
Private DNS: The ALB receives a public DNS name, but it resolves exclusively to private IP addresses.
No Internet Access: It cannot receive any traffic from the public internet because it lacks a public IP.

Common Use Cases

Internal Microservices: Routing traffic from a public-facing web tier to a private backend API tier.
Hybrid Networks: Routing traffic coming from an on-premises data centre via AWS Direct Connect or a VPN

Setting Up ALB in AWS Console

AWS Elastic Load Balancing shows basic building blocks of AWS Load Balancer which include listeners and target groups.

To create Application Load Balancer go to EC2 >> Load balancers >> Create Load balancer >> Select load balancer type (click on Create under Application Load Balancer)

Here we can set:

Basic configuration

Name
Scheme (cannot be changed after the load balancer is created)

Internet-facing. An internet-facing load balancer routes requests from clients over the internet to targets. Requires a public subnet.
Internal. An internal load balancer routes requests from clients to targets using private IP addresses.

IP address type. Select the type of IP addresses that your subnets use.

IPv4. Recommended for internal load balancers.
Dualstack. Includes IPv4 and IPv6 addresses.

Network mapping. The load balancer routes traffic to targets in the selected subnets, and in accordance with your IP address settings.

VPC. Virtual private cloud for your targets. If balancer is internet-facing, only VPCs with an internet gateway are enabled for selection. The selected VPC cannot be changed after the load balancer is created. As VPC is region-specific so is Application Load Balancer.
Mappings. Once VPC is selected, its availability zones are listed here and are selectable. Select at least two Availability Zones and one subnet per zone. The load balancer routes traffic to targets in these Availability Zones only. Availability Zones that are not supported by the load balancer or the VPC are not available for selection. We should select all AZs that we listed in the Auto scaling group (if we used it).

Security groups. A security group is a set of firewall rules that control the traffic to your load balancer. We can select up to 10 security groups.

If our application is listening for HTTP requests on port 80 we should select a security group with:

Inbound rule: accept HTTP/TCP traffic on port 80 with source Anywhere-IPv4
Outbound rule: allow all traffic for all protocols and port ranges to custom destination 0.0.0.0/0

Listeners and routing. A listener is a process that checks for connection requests using the port and protocol you configure. The rules that you define for a listener determine how the load balancer routes requests to its registered targets.

Add listener

Protocol e.g. HTTP
Port e.g. 80. This is a public facing port and it does not need to be the same as the port from the attached target group. E.g. LB can listen on port 80 and forward traffic to target group port 8080.
Default action: Forward to (select a target group)
Add listener tags

Add-on services - optional

AWS Global Accelerator

Tags - optional

More info on Scheme, from AWS documentation:

When you create a load balancer, you must choose whether to make it an internal load balancer or an internet-facing load balancer.

The nodes of an internet-facing load balancer have public IP addresses.

The nodes of an internal load balancer have only private IP addresses.

Both internet-facing and internal load balancers route requests to your targets using private IP addresses. Therefore, your targets do not need public IP addresses to receive requests from an internal or an internet-facing load balancer.

More info on how ALB routes traffic to multiple Availability Zones (and about what Load Balancer Nodes are):

When you enable an Availability Zone for your load balancer, Elastic Load Balancing creates a load balancer node in the Availability Zone.
The nodes for your load balancer distribute requests from clients to registered targets. When cross-zone load balancing is enabled, each load balancer node distributes traffic across the registered targets in all enabled Availability Zones. When cross-zone load balancing is disabled, each load balancer node distributes traffic only across the registered targets in its Availability Zone.
Before a client sends a request to your load balancer, it resolves the load balancer's domain name using a Domain Name System (DNS) server. The DNS entry is controlled by Amazon, because your load balancers are in the amazonaws.com domain. The Amazon DNS servers return one or more IP addresses to the client. These are the IP addresses of the load balancer nodes for your load balancer.
As traffic to your application changes over time, Elastic Load Balancing scales your load balancer and updates the DNS entry. The DNS entry also specifies the time-to-live (TTL) of 60 seconds. This helps ensure that the IP addresses can be remapped quickly in response to changing traffic.

The client determines which IP address to use to send requests to the load balancer. The load balancer node that receives the request selects a healthy registered target and sends the request to the target using its private IP address.

With Application Load Balancers, the load balancer node that receives the request uses the following process:

1) Evaluates the listener rules in priority order to determine which rule to apply.

2) Selects a target from the target group for the rule action, using the routing algorithm configured for the target group. The default routing algorithm is round robin. Routing is performed independently for each target group, even when a target is registered with multiple target groups.

For further info: How Elastic Load Balancing works - Elastic Load Balancing

ALB nodes use Elastic Network Interface (Elastic network interfaces - Amazon Elastic Compute Cloud) which has public IP address:

At least one ENI is created and attached to the balancer in each availability zone where the balancer is deployed (except NLB, which should only have one per AZ). Over the life of the balancer, new ENIs will appear and old ones will disappear, as the balancer scales horizontally (number of nodes) and/or vertically (capacity of underlying hardware), all of which is handled transparently by the infrastructure. Even though you can tag them, the tagging will become stale over time.

Source: amazon web services - AWS - Affect Load Balancer's tags to its Network Interfaces (ENI) - Stack Overflow

You can determine the IP addresses associated with an internal load balancer or an internet-facing load balancer by resolving the DNS name of the load balancer. These are the IP addresses where the clients should send the requests that are destined for the load balancer. However, Classic Load Balancers and Application Load Balancers use the private IP addresses associated with their elastic network interfaces as the source IP address for requests forwarded to your web servers.

Source: Find the IP address used by a load balancer to forward traffic to web servers

Load balancer routes requests to the targets in a target group and performs health checks on the targets. Target group is accepting requests from the load balancer and forwards them to targets. These targets can be e.g. EC2 instances created either manually or through auto scaling group.

How to create a Target Group used by Load Balancer listeners? (This applies for any type of Load Balancer)

EC2 >> Target groups >> Create target group

Step 1: Specify group details

Here we can set:

Basic configuration. Settings in this section cannot be changed after the target group is created.

Target type

Instances

Supports load balancing to instances within a specific VPC.
Facilitates the use of Amazon EC2 Auto Scaling to manage and scale your EC2 capacity.

IP addresses

Supports load balancing to VPC and on-premises resources.
Facilitates routing to multiple IP addresses and network interfaces on the same instance.
Offers flexibility with microservice based architectures, simplifying inter-application communication.
Supports IPv6 targets, enabling end-to-end IPv6 communication, and IPv4-to-IPv6 NAT.

Lambda function

Facilitates routing to a single Lambda function.
Accessible to Application Load Balancers only.

Application Load Balancer

Offers the flexibility for a Network Load Balancer to accept and route TCP requests within a specific VPC
Facilitates using static IP addresses and PrivateLink with an Application Load Balancer.

Target group name
Protocol:Port e.g. If our application is accepting HTTP requests on port 8080 this would be HTTP:8080
VPC - VPC with the instances that you want to include in the target group.
Protocol version

HTTP1. Send requests to targets using HTTP/1.1. Supported when the request protocol is HTTP/1.1 or HTTP/2.
HTTP2. Send requests to targets using HTTP/2. Supported when the request protocol is HTTP/2 or gRPC, but gRPC-specific features are not available.
gRPC. Send requests to targets using gRPC. Supported when the request protocol is gRPC.

Health checks. The associated load balancer periodically sends requests, per the settings below, to the registered targets to test their status.

Health check protocol

HTTP
HTTPS

Health check path. Use the default path of “/“ to ping the root, or specify a custom path if preferred.
Advanced health check settings

Port. The port the load balancer uses when performing health checks on targets. The default is the port on which each target receives traffic from the load balancer, but you can specify a different port.

Traffic port
Override

Healthy threshold. The number of consecutive health checks successes required before considering an unhealthy target healthy.
Unhealthy threshold. The number of consecutive health check failures required before considering a target unhealthy.
Timeout. The amount of time, in seconds, during which no response means a failed health check.
Interval. The approximate amount of time between health checks of an individual target
Success codes. The HTTP codes to use when checking for a successful response from a target. You can specify multiple values (for example, "200,202") or a range of values (for example, "200-299").

Attributes
Tags - optional

Step 2: Register targets

This is an optional step to create a target group. However, to ensure that your load balancer routes traffic to this target group you must register your targets.

After load balancer is created it takes several minutes while it's in provisioning state and get into active state. After this, we can use its DNS name in order to see what it's doing.

If we copy its DNS name and paste it to our browser, if we haven't registered any targets in the target group associated with the load balancer, we'll get error 503 - Service Temporary Unavailable.

If we've registered targets and are getting error 504 Gateway time-out, we should check first if security groups (firewalls) for our EC2 instances (inbound rule - source IP range) are set up correctly as this error usually indicates that inbound traffic is not allowed.

AWS Terraform provider offers provisioning all these resources:

Application Load Balancer: aws_lb | Resources | hashicorp/aws | Terraform Registry
Listener: aws_lb_listener | Resources | hashicorp/aws | Terraform Registry
Target Group: aws_lb_target_group | Resources | hashicorp/aws | Terraform Registry

How is AWS Application Load Balancing usually implemented?

Let's say we have our application running on 3 EC2 instances where 2 are in the same region e.g. us-west-2 but in separate availability zones e.g. us-west-2a and us-west-2b. Third EC2 instance is in eu-central-1, in availability zone eu-central-1a.

VPC is region-specific but can span multiple availability zones (AZ).

Subnet is an IP address range within VPC.

VPC can have public and private subnets.

VPC can be divided into multiple subnets but each subnet is AZ-specific.

AZ can have multiple subnets.

So, all EC2 instances belong to the same VPC but, as they are in different AZs, each of them belongs to different subnet.

Load balancer must be in the public subnet of VPC as clients communicate with load balancer via internet (public network).

Load balancer does not get associated directly with EC2 instances but subnets:

resource "aws_lb" "test" {

subnets = ["subnet-0001", "subnet-0002"]

...

}

Target group is associated with VPC:

resource "aws_alb_target_group" "test" {

vpc_id = var.vpc_id

...

}

Difference between ALB and NLB (Network Load Balancer)

An Application Load Balancer (ALB) and a Network Load Balancer (NLB) serve different purposes based on the layer of the network they operate on and the type of traffic they handle.

The core difference is that an ALB understands application-level traffic (Layer 7) like HTTP/HTTPS headers, while an NLB handles low-level network traffic (Layer 4) like TCP/UDP packets at extreme speeds.

Direct Comparison Matrix

Feature Application Load Balancer (ALB) Network Load Balancer (NLB)

====== ========================= =======================

OSI Layer Layer 7 (Application) Layer 4 (Transport)

Protocols HTTP, HTTPS, HTTP/2, gRPC, WebSockets TCP, UDP, TLS

IP Addresses

Dynamic IPs (Changes automatically; requires a DNS name)

Static IPs (Can assign an Elastic IP per AZ)

Routing Features

Advanced (Path, Host, Query parameters, Headers)

Basic (Port and IP protocol routing only)

Performance

Optimized for complex web apps (Millions of requests/sec)

Optimized for ultra-low latency (Billions of requests/sec)

Key Technical Differences

1. Smart Routing vs. Raw Speed

ALB (Smart): Can read the contents of your HTTP requests. It can route traffic bound for ://example.com to an API server cluster, and traffic for ://example.com to a storage cluster.
NLB (Fast): Does not look inside the data packet. It simply looks at the target port and forwards the packet instantly. This results in ultra-low latency (measured in milliseconds).

2. IP Addresses and DNS

ALB: Scale out dynamically by adding or removing nodes. This causes its underlying IP addresses to change frequently. You must always point your domain name to the ALB's DNS Name, never to a static IP.
NLB: Gives you a Static IP address per Availability Zone. You can also assign your own Elastic IP addresses. This is critical if your corporate clients need to whitelist specific, unchanging IPs in their firewalls.

3. Client IP Preservation

ALB: Terminates the connection and makes a new one to your backend instances. The backend see the ALB's private IP. To find the real user's IP, your code must read the X-Forwarded-For HTTP header.
NLB: Passes the original TCP packet straight through to your backend server. Your backend instances see the original source IP address of the client natively, without needing extra headers.

When to Choose Which?

Choose an ALB if you are building:

Standard web applications and microservices.
Containerized apps (ECS/EKS) requiring path-based or host-based routing.
Applications requiring tight integration with AWS Web Application Firewall (WAF).

Choose an NLB if you are building:

Non-HTTP applications (e.g., gaming servers, SFTP, MQTT, database clusters).
Architectures requiring fixed, static IP addresses or Elastic IPs.
High-frequency financial applications where sub-millisecond network latency is a hard requirement

Which alerts should typically be set for AWS ALB?

To keep your applications highly available, you should set up Amazon CloudWatch alarms for a mix of availability, performance, and target health metrics.

The most critical metrics to monitor for an AWS ALB are grouped by priority below:

1. High Priority (Critical Infrastructure Impact)

UnHealthyHostCount (Per Target Group)

What it means: The number of backend instances failing health checks Target Group Metrics.

Alert Threshold: > 0 (or > 1 for larger clusters).

Why it matters: Signals that your servers are crashing or cannot handle traffic.

HTTPCode_Target_5XX_Count

What it means: The number of 5xx server error codes generated by your backend application ALB Metrics.

Alert Threshold: Depends on baseline traffic, typically > 5 failures within a 1-minute to 5-minute window.

Why it matters: Indicates server crashes, database connection timeouts, or unhandled exceptions in your application code.

HTTPCode_ELB_5XX_Count

What it means: The number of 5xx errors generated directly by the ALB itself (not your servers) ALB Metrics.

Alert Threshold: > 0.

Why it matters: Usually means the ALB cannot find any healthy hosts, or it is experiencing a configuration mismatch (e.g., bad TLS handshake with the target).

2. Medium Priority (Performance & User Experience)

TargetResponseTime

What it means: The time elapsed (in seconds) from when the ALB sent the request to the target until the target started responding ALB Metrics.

Alert Threshold: Use the p95 or p99 statistic. Alert if it exceeds your application’s maximum acceptable latency (e.g., > 2.0 seconds).

Why it matters: Users are experiencing severe application slowdowns, likely due to high CPU/memory usage on your instances.

RejectedConnectionCount

What it means: The load balancer is rejecting connections because it has reached its maximum capacity ALB Metrics.

Alert Threshold: > 0.

Why it matters: Your application is getting sudden traffic spikes and the ALB cannot scale fast enough, or backend targets are failing to keep up.

3. Low Priority (Anomalies & Security)

HTTPCode_Target_4XX_Count

What it means: The number of 4xx client errors (like 404 Not Found or 401 Unauthorized) returned by backend targets ALB Metrics.

Alert Threshold: A significant spike above your standard baseline.

Why it matters: A sudden surge might indicate a broken frontend deployment, a bad API update, or a malicious entity scanning your network for vulnerabilities.

Summary Checklist for CloudWatch Alarms

Metric Name Statistic Recommended Suggested Action

Period Threshold

========== ====== =========== ======== =====

UnHealthyHostCount Maximum 1 Minute > 0 Page/On-Call

HTTPCode_ELB_5XX_Count Sum 1 Minute > 0 Page/On-Call

HTTPCode_Target_5XX_Count Sum 5 Minutes > 10 (or > 1% of traffic) Ticket/Slack

TargetResponseTime p95 5 Minutes> [Your Limit] Ticket/Slack

Resources:

How Elastic Load Balancing works - Elastic Load Balancing

Some notes on LLMs

2026-06-03T11:20:16.303+01:00

What does e.g. 1M context mean in a model description?

"1M context" means the model can handle up to 1 million tokens in its context window — the combined input (your messages, files, tool results, system prompt) plus output it can consider at once. For reference, that's roughly 750,000 words or a few thousand pages of text, letting it work with very large codebases or long conversations without losing earlier content.

What is a LLM's tool-use loop?

An LLM tool-use loop is the pattern where you let a language model drive an investigation by repeatedly choosing tools to call, rather than answering in one shot.

The shape

1. Send: system prompt + user request + list of available tools (with JSON schemas)

2. Model responds with either:

(a) a final text answer -> exit loop

(b) a "tool_use" block: { name: "run_aws_cli", input: { args: [...] } }

3. Your code executes that tool, captures the result

4. Append the tool result to the conversation as a "tool_result" message

5. Send the whole conversation back to the model

6. Goto 2

The model never executes anything itself — it just emits requests to call tools. Your code is the runtime that actually runs them and feeds the output back.

Why it's a loop

Each turn the model sees everything it has learned so far (prior tool calls + their outputs) and decides the next step based on that. So a real run looks like:

- Turn 1: model calls cloudwatch describe-alarms --state-value ALARM

- Turn 2: sees 3 alarms, picks the noisiest, calls logs filter-log-events for that log group around the alarm time

- Turn 3: sees an error pattern, calls kubectl describe pod on the affected workload

- Turn 4: emits final Markdown report, no tool call → loop exits

The model is doing the planning; your code is the dispatcher.

Why you need a budget

Without limits the loop can spin forever — the model keeps finding "one more thing to check." Hence in agent.run():

- max_iterations=30 — hard cap on turns

- max_tokens_per_turn=12288 — cap on a single response

- Per-tool wall-clock timeouts (60 s for CLI, 30 s for HTTP)

- Output truncation (50 000 char stdout) so a giant tool result doesn't blow the context window

How it ends

The loop terminates when the model returns a response with no tool_use block — that's the "I'm done, here's the answer" signal (stop_reason: end_turn). Or when you hit a budget limit and force-stop it.

Where the safety lives

Because the model can ask for arbitrary tool calls, the loop is only as safe as the tool implementations. That's why when implementing agents we should have the allowlists (services, verbs, paths) - the model can request aws s3 rm, but the validator rejects it before subprocess.run ever sees it.

The "two-pass" design in agent is a refinement: pass 1 is a tool-use loop (gather), pass 2 is a single non-loop call (synthesize). Splitting them lets each prompt focus on one job.

What are those .md files used by AI Agents?

There isn't a universally agreed official name, but people commonly refer to files like CLAUDE.md, GEMINI.md, AGENTS.md, COPILOT_INSTRUCTIONS.md, and .cursorrules as:

AI agent instruction files (most generic)
Agent configuration files
Agent context files
LLM instruction files
Repository AI instructions
Project AI guidelines

In the developer tooling community, "agent instructions" or "agent context files" are probably the most widely understood umbrella terms.

For example:

Tool File

---- -----

Claude Code CLAUDE.md

Gemini CLI GEMINI.md

GitHub Copilot .github/copilot-instructions.md

OpenAI Codex CLI AGENTS.md

Cursor .cursorrules / project rules

Windsurf Rules files

Collectively, you could describe them as:

"Repository-level AI agent instruction files that provide persistent context and operating rules for coding assistants."

If you're building tooling around them (e.g., in your DevOps work), I'd recommend using "agent instructions" as the generic term because it's vendor-neutral and easily understood across Claude, Gemini, Copilot, Cursor, Codex, and similar tools.

Introduction to Claude by Anthropic

2026-06-03T11:19:38.085+01:00

How to give Claude an instruction to apply label "DevOps" whenever it creates a new Linear ticket?

It depends on whether you're creating Linear tickets via:

Linear MCP,
a custom /create-ticket command,
or just asking Claude in chat to create tickets

The best location depends on which of those you're using.

If you're just asking Claude in chat to create tickets: if you want this behavior for all projects, put it in:

~/.claude/CLAUDE.md

If you want it only for a specific repository/project, put it in:

<repo-root>/CLAUDE.md

If you want it only for yourself in a specific project (without committing it to git), put it in:

<repo-root>/CLAUDE.local.md

CLAUDE.local.md is usually the best place for personal workflow preferences like Linear labels. Claude loads it after the project-level file, so your preference is read later and tends to have more influence.

For example:

## Linear Ticket Creation

When creating new Linear issues:

- Always apply the "DevOps" label.

- If multiple labels are appropriate, include "DevOps" in addition to the others.

- Verify the label exists before creating the issue.

If you're creating tickets through a custom slash command (e.g. /create-ticket) or a Linear MCP workflow, an even more reliable approach is to put the instruction directly in the command definition under:

~/.claude/commands/create-ticket.md

or in a dedicated skill, because the instruction is then attached specifically to the ticket-creation workflow rather than relying on general memory. Many Claude Code users find workflow-specific commands more reliable than large global CLAUDE.md files for things like Linear automation.

How to instruct Claude to use feature branch name from a Linear ticket when implementing it?

Add a specific workflow rule to CLAUDE.md rather than a vague instruction. Claude follows procedural instructions much more reliably.

For example:

## Linear Ticket Workflow

When implementing work from a Linear ticket:

1. Retrieve the ticket details from Linear.

2. Check whether the ticket has a suggested branch name.

3. Before making any code changes, switch to or create a git branch using the Linear branch name.

4. Do not create an alternative branch name unless the Linear branch name is unavailable.

5. If the branch already exists remotely, check it out instead of creating a new branch.

6. Confirm the branch name being used before starting implementation.

Example:

Linear ticket: DEV-123 Add deployment health checks

Branch: bojan/dev-123-add-deployment-health-checks

Commands:

git fetch origin

git checkout bojan/dev-123-add-deployment-health-checks || \

git checkout -b bojan/dev-123-add-deployment-health-checks

If you're using the Linear MCP server, I'd make it even more explicit:

## Branch Naming

For any task originating from Linear:

- Always obtain the branch name from the Linear ticket metadata.

- Never invent a branch name if Linear provides one.

- Use the Linear-provided branch name exactly as written.

- Checkout the branch before planning or implementing changes.

An even better approach is to create a dedicated command such as .claude/commands/implement-ticket.md:

Given a Linear ticket ID:

1. Load the ticket.

2. Read the ticket's branch name.

3. Checkout the branch.

4. Implement the work.

5. Commit using the ticket ID in the commit message.

Workflow-specific commands tend to be more reliable than relying on global instructions in CLAUDE.md, especially for things like branch creation, ticket labeling, and PR naming.

If I update ~/.claude/CLAUDE.md, do I need to start the new Claude session?

Usually no restart is strictly required, but it depends on how you’re using Claude:

If you’re using Claude Code / CLI (most common case)

~/.claude/CLAUDE.md is typically read when a new session starts

So:

✅ New terminal session / new Claude run → picks up changes

⚠️ Already-running interactive session → may not reliably reload it

Best practice: restart the Claude session (or re-open the CLI/chat) after changing it if you want to be sure the new rules apply.

If you’re using a long-running agent session

(e.g. an ongoing “Claude is working in the repo” process)

It often caches the instructions in memory at session start

So updates won’t consistently apply mid-task

If you’re using repo-level CLAUDE.md

Same rule: loaded at session start or project initialization

Changes apply reliably on next session or re-attach to repo

Practical rule of thumb

Changed CLAUDE.md → start a new Claude session to guarantee it applies

Minor edits while actively working → assume they won’t fully apply until restart

How to add auto-generated instructions to global context?

During Claude Code session, it created two memories but they were bound to the current project only:

Write(/Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/gh-token-unset.md)

⎿ Wrote 10 lines to /Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/gh-token-unset.md

1 ---

2 name: gh-token-unset

3 description: gh CLI fails with HTTP 401 Bad credentials unless GH_TOKEN is unset first

4 metadata:

5 node_type: memory

6 type: reference

7 originSessionId: 9xxx6

8 ---

10 In this environment `gh` fails with `HTTP 401: Bad credentials` because a stale `GH_TOKEN` env var overrides the working keyring auth. Run `unset GH_TOKEN` in the same Bash command before any `gh`

call — the keyring account (`BojanKomazec`, scopes gist/read:org/repo) then works. Env doesn't persist between Bash calls, so include the `unset` each time.

⏺ Write(/Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/precommit-hook-needs-no-verify.md)

⎿ Wrote 10 lines to /Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/precommit-hook-needs-no-verify.md

1 ---

2 name: precommit-hook-needs-no-verify

3 description: git commit aborts in non-interactive shell; the pre-commit hook prompts on /dev/tty

4 metadata:

5 node_type: memory

6 type: reference

7 originSessionId: 9xxx6

8 ---

10 The global pre-commit hook (`~/.git-templates/hooks/pre-commit`) prompts interactively via `/dev/tty` ("view diff? proceed with commit?"). In a non-interactive shell it errors `/dev/tty: Device no

t configured` and aborts the commit. Use `git commit --no-verify` to bypass it when committing programmatically.

⏺ Write(/Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/MEMORY.md)

⎿ Wrote 2 lines to /Users/bojan/.claude/projects/-Users-bojan-repos-Company-Project/memory/MEMORY.md

1 - [gh token unset](gh-token-unset.md) — `unset GH_TOKEN` before any `gh` command or it 401s

2 - [pre-commit hook needs --no-verify](precommit-hook-needs-no-verify.md) — commit hook prompts on /dev/tty, aborts non-interactively

What you’re looking at in:

~/.claude/projects/.../memory/MEMORY.md

is project-scoped learned memory (auto-generated runtime notes), not the best place to define global rules.

To make something apply to all projects, you should put it in:

~/.claude/CLAUDE.md

How to structure it properly

Think of ~/.claude/CLAUDE.md as your global “agent operating system”.

You can directly migrate those MEMORY.md rules into it like this:

## Global DevOps / CLI Behavior Rules

These rules apply to all projects unless explicitly overridden.

### GitHub CLI

- Always run `unset GH_TOKEN` before executing any `gh` command.

- Reason: GH_TOKEN causes 401 errors in local CLI authentication.

### Git commits

- When committing changes, always use `--no-verify`.

- Reason: pre-commit hooks may attempt interactive prompts via /dev/tty and break non-interactive runs.

Important distinction (this is the key part)

File Scope Who writes it Purpose

---- ----- ------------- --------

~/.claude/CLAUDE.md Global (all projects) You Your permanent rules

<repo>/CLAUDE.md Per project You/team Repo-specific rules

~/.claude/projects/.../MEMORY.md Auto-generated Claude Learned behavior, not authoritative

👉 Never manually “promote” MEMORY.md as your source of truth

Instead, copy stable patterns into ~/.claude/CLAUDE.md.

For your Linear branch-name requirement

You’d add it here too:

## Linear Workflow (Global)

When working from Linear tickets:

- Always use the branch name provided by Linear metadata.

- Never generate custom branch names.

- Always checkout the Linear branch before implementation.

One extra improvement (worth doing)

If you want this to be really robust, structure your global file like:

~/.claude/CLAUDE.md

├── Git / GitHub rules

├── Linear workflow rules

├── Shell safety rules

├── CI/CD rules

Claude responds better to clear domains than long flat lists.

Introduction to Checkly

2026-05-14T15:36:44.566+01:00

How Checkly works

Checkly is a SaaS synthetic monitoring platform — you define "checks" (HTTP requests or browser scripts), Checkly runs them on a schedule from probe locations around the world (or on-demand from CI), records latency/assertions/screenshots, and alerts you when they fail or get slow.

Two main check types:

- API checks — a single HTTP request with assertions on status, headers, body, response time.

- Browser checks — a Playwright script run in a real headless Chromium against your deployed app.

There's also multi-step API checks (chain requests, e.g. login → use token → logout) and heartbeat checks (your job pings Checkly; alert if it stops).

Hearbeat vs Ping

Heartbeats and pings are both vital network failure-detection mechanisms, but they differ in purpose: Heartbeats are proactive, periodic "I am alive" messages sent by an application to signal it is healthy, while Pings are reactive requests to check if a server is reachable. Heartbeats detect application crashes, while pings detect network downtime.

Checks are typically authored as code (Checkly CLI, TypeScript) and checkly deploy'd to the cloud. You can tag them (tags: ["auth"]), parametrise them with env vars like ENVIRONMENT_URL, and trigger them on-demand from CI — which is exactly what this PR does with npx checkly trigger --tags=auth.

Runtime model:

- Scheduled: every N minutes from chosen regions (e.g. us-east-2, eu-west-1) — catches regressions/outages between deploys.

- Triggered from CI: post-deploy smoke test, results gate (or just annotate) the deploy.

- Alerts: Slack/PagerDuty/email on failure, with retry/degraded thresholds to avoid flap.

---

What it would check for this auth API

Given the auth API's surface (login, OAuth, JWT issuance, admin endpoints), realistic auth-tagged checks:

1. Health endpoint — basic liveness

new ApiCheck("auth-health", {

tags: ["auth"],

frequency: 1, // minute

locations: ["us-east-2", "eu-west-1"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [

AssertionBuilder.statusCode().equals(200),

AssertionBuilder.responseTime().lessThan(500),

AssertionBuilder.jsonBody("$.status").equals("ok"),

});

2. Login flow — happy path, returns a JWT

new ApiCheck("auth-login", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/auth/login`,

method: "POST",

headers: [{ key: "Content-Type", value: "application/json" }],

body: JSON.stringify({

email: process.env.SYNTHETIC_USER_EMAIL,

password: process.env.SYNTHETIC_USER_PASSWORD,

}),

assertions: [

AssertionBuilder.statusCode().equals(200),

AssertionBuilder.responseTime().lessThan(1500),

AssertionBuilder.jsonBody("$.token").isNotNull(),

// structural check on JWT shape

AssertionBuilder.jsonBody("$.token").matches("^eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+$"),

});

3. Login — wrong password returns 401 (negative path)

Catches the "accidentally accepts anything" class of regression.

new ApiCheck("auth-login-bad-pw", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/auth/login`,

method: "POST",

headers: [{ key: "Content-Type", value: "application/json" }],

body: JSON.stringify({ email: process.env.SYNTHETIC_USER_EMAIL, password: "wrong" }),

assertions: [AssertionBuilder.statusCode().equals(401)],

});

4. Multi-step — login then call protected endpoint

This is the most useful kind for an auth API, because it proves the token actually works.

new MultiStepCheck("auth-token-roundtrip", {

tags: ["auth"],

code: { entrypoint: path.join(__dirname, "token-roundtrip.spec.ts") },

});

// token-roundtrip.spec.ts

import { test, expect } from "@playwright/test";

test("login then /me", async ({ request }) => {

const login = await request.post(`${process.env.ENVIRONMENT_URL}/auth/login`, {

data: { email: process.env.SYNTHETIC_USER_EMAIL, password: process.env.SYNTHETIC_USER_PASSWORD },

});

expect(login.ok()).toBeTruthy();

const { token } = await login.json();

const me = await request.get(`${process.env.ENVIRONMENT_URL}/me`, {

headers: { Authorization: `Bearer ${token}` },

});

expect(me.status()).toBe(200);

const body = await me.json();

expect(body.email).toBe(process.env.SYNTHETIC_USER_EMAIL);

});

5. TLS & cert expiry

A pure config check — useful because cert rotation is a classic outage cause.

new ApiCheck("auth-tls", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [AssertionBuilder.statusCode().equals(200)],

// Checkly surfaces cert expiry on the run; you set a threshold per check

});

6. Browser check — full login UX

expect(login.ok()).toBeTruthy();

const { token } = await login.json();

const me = await request.get(`${process.env.ENVIRONMENT_URL}/me`, {

headers: { Authorization: `Bearer ${token}` },

});

expect(me.status()).toBe(200);

const body = await me.json();

expect(body.email).toBe(process.env.SYNTHETIC_USER_EMAIL);

});

5. TLS & cert expiry

A pure config check — useful because cert rotation is a classic outage cause.

new ApiCheck("auth-tls", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [AssertionBuilder.statusCode().equals(200)],

// Checkly surfaces cert expiry on the run; you set a threshold per check

5. TLS & cert expiry

A pure config check — useful because cert rotation is a classic outage cause.

new ApiCheck("auth-tls", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [AssertionBuilder.statusCode().equals(200)],

// Checkly surfaces cert expiry on the run; you set a threshold per check

});

new ApiCheck("auth-tls", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/health`,

method: "GET",

assertions: [AssertionBuilder.statusCode().equals(200)],

// Checkly surfaces cert expiry on the run; you set a threshold per check

});

6. Browser check — full login UX

Runs against the front-end but exercises the auth API end-to-end including redirects, cookies, CSRF.

new BrowserCheck("auth-ui-login", {

tags: ["auth"],

code: { entrypoint: path.join(__dirname, "login.spec.ts") },

});

import { test, expect } from "@playwright/test";

test("user can sign in", async ({ page }) => {

await page.goto(process.env.ENVIRONMENT_URL!);

await page.getByLabel("Email").fill(process.env.SYNTHETIC_USER_EMAIL!);

await page.getByLabel("Password").fill(process.env.SYNTHETIC_USER_PASSWORD!);

await page.getByRole("button", { name: "Sign in" }).click();

await expect(page.getByText("Dashboard")).toBeVisible({ timeout: 10_000 });

});

7. OAuth callback reachability

Doesn't fully exercise the Google/Microsoft flow (those need real consent), but checks the callback

endpoint responds correctly to a missing-code request — confirms route + handler are wired.

new ApiCheck("auth-oauth-google-callback-shape", {

tags: ["auth"],

request: {

url: `${process.env.ENVIRONMENT_URL}/auth/google/callback`,

method: "GET",

assertions: [

// 400 for missing `code`, not 404/500 — proves handler is mounted

AssertionBuilder.statusCode().equals(400),

});

Intro to QA with Headless Browsers

2026-05-14T11:14:54.403+01:00

Headless browsers are used in QA to execute automated browser tests faster and more efficiently by eliminating the graphical user interface (GUI). Because they don't render visuals, they consume fewer resources, enabling rapid, parallel testing in CI/CD pipelines, making them ideal for high-volume functional and regression testing.

Key Reasons for Using Headless Browsers in QA:

Faster Execution: Without the need to render CSS, images, or layout, tests run significantly faster.
CI/CD Integration: They are ideal for server-side environments where a GUI is unavailable, allowing automated tests to run after every code commit.
Lower Resource Usage: They consume significantly less RAM and CPU, allowing for higher parallelization (running many tests simultaneously) without overloading hardware.
Automated Functional Testing: They can accurately simulate user actions such as clicking buttons, submitting forms, and navigating pages.
Regression Testing: Due to speed and efficiency, they are perfect for running large suites of regression tests to ensure new changes haven't broken existing functionality.

Common tools for headless testing include headless Chrome, Firefox, Puppeteer, and Playwright

Headless browsers parse, compile, and execute the exact same underlying code as standard browsers, but they skip the final step of painting pixels to a physical screen.

What Headless Browsers Still Do

Construct the DOM: They parse HTML into a full Document Object Model tree.
Apply Styling: They process CSS and calculate layout, element positions, and visibility.
Execute JavaScript: They run a full JS engine (like V8 in Chrome) to handle AJAX, animations, and frontend logic.
Manage Network Traffic: They make real HTTP requests, download cookies, and handle API responses.

How QA Verifies Visuals Without a Display

Layout Queries: Code checks if elements are present, hidden, or overlapping by querying their coordinates.
Computed Styles: Scripts verify specific CSS properties, like checking if a button color is exactly rgb(0, 0, 255).
Virtual Screenshots: The browser renders the page into an in-memory buffer, allowing QA tools to save PNGs or perform pixel-by-pixel visual regression comparisons.

To help tailor using headless browser to our workflow, we need to know:

Which testing framework we are using (e.g., Playwright, Selenium, Cypress)?
Are we trying to catch functional bugs or visual layout glitches?
Do our tests run on a local machine or a CI/CD server (e.g., GitHub Actions, Jenkins)?

---

Introduction to Amazon Simple Notification Service (SNS)

2026-04-29T14:24:23.579+01:00

Amazon Simple Notification Service (SNS) is a fully managed messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Here's how SNS works:

Key Concepts

Topics:

A topic is a logical access point and communication channel. Publishers send messages to a topic, and subscribers receive these messages by subscribing to the topic.

Publishers:

Publishers are entities that send messages to an SNS topic. They could be applications, services, or even other AWS services like Lambda or CloudWatch.

Subscribers:

Subscribers are endpoints that receive messages from an SNS topic. These can include Amazon SQS queues, AWS Lambda functions, HTTP/S endpoints, email addresses, and SMS numbers.

Messages:

Messages are the payload sent by publishers to SNS topics. They can include a variety of data formats, typically JSON.

How It Works

Creating a Topic:

First, you create a topic using the AWS Management Console, AWS CLI, or AWS SDKs. This topic acts as a communication channel.

Subscribing to a Topic:

You then subscribe one or more endpoints to the topic. These endpoints can be other AWS services or external services capable of receiving notifications.

When subscribing, you specify the protocol (such as HTTP, SQS, Lambda, etc.) and the endpoint (like the URL or ARN of the SQS queue).

Publishing a Message:

Publishers send messages to the SNS topic using the Publish API. The message can include a subject, a message body, and optional attributes.

SNS stores multiple copies of the message for redundancy and high availability.

Message Delivery:

SNS distributes the message to all subscribed endpoints.

Each endpoint processes the message according to its protocol. For example, an HTTP endpoint receives a POST request with the message content, and an SQS queue receives the message as a new queue entry.

Use Cases

Fan-out Scenarios:

When a message published to an SNS topic needs to be sent to multiple endpoints, SNS acts as a fan-out service. For example, updating various microservices or notifying multiple systems about an event.

Push Notifications:

SNS can be used to send push notifications to mobile devices through services like Amazon Device Messaging (ADM), Apple Push Notification Service (APNS), and Google Cloud Messaging (GCM).

Application Alerts:

Sending notifications or alerts to administrators or users via SMS, email, or other endpoints.

Event-Driven Computing:

Integrating with AWS Lambda to trigger functions in response to incoming messages, enabling event-driven processing.

Benefits

Scalability:

SNS is designed to handle high-throughput and large volumes of messages, making it suitable for large-scale applications.

Flexibility:

It supports multiple protocols, allowing for versatile messaging architectures.

Reliability:

With built-in redundancy and fault tolerance, SNS ensures reliable message delivery.

Ease of Use:

The service is fully managed, reducing the operational overhead associated with maintaining messaging infrastructure.

Security

Access Control:

Using AWS IAM policies, you can control who can publish or subscribe to topics.

Encryption:

SNS supports server-side encryption (SSE) for messages at rest and SSL/TLS for messages in transit.

Compliance:

SNS is compliant with various industry standards and regulations, ensuring it meets security and compliance requirements.

Amazon SNS is a powerful and flexible tool for building loosely coupled, scalable, and robust messaging solutions in the cloud.

How does Amazon SNS differ from Amazon SQS?

Amazon SNS (Simple Notification Service) and Amazon SQS (Simple Queue Service) are both messaging services provided by AWS, but they serve different purposes and have distinct features. Here's a comparison highlighting their differences:

Purpose and Use Cases

Amazon SNS:

Publish/Subscribe Messaging:

SNS is designed for sending notifications or messages to multiple subscribers.
It's ideal for broadcasting messages to multiple endpoints, such as sending alerts, notifications, or updates to different systems or users.

Use Cases:

Real-time notifications
push notifications
fan-out scenarios (where a message needs to be sent to multiple recipients simultaneously)
application alerts
event-driven architectures

Amazon SQS:

Message Queuing:

SQS is designed for decoupling and scaling distributed systems.
It allows you to send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

Use Cases:

Task queues
asynchronous processing
decoupling microservices
job dispatching
buffering messages between producer and consumer systems

Messaging Patterns

Amazon SNS:

Push-Based: SNS pushes messages to subscribers. Subscribers can be other AWS services (like Lambda, SQS), HTTP/S endpoints, email addresses, SMS numbers, and mobile push notifications.
Fan-Out: One message can be sent to multiple subscribers.

Amazon SQS:

Pull-Based: Consumers pull messages from the queue. A consumer explicitly retrieves messages from the queue.
Point-to-Point: Each message is delivered to and processed by one consumer.

Message Handling

Amazon SNS:

Real-Time Delivery: Messages are delivered immediately to all subscribers.
No Message Persistence: Messages are not stored after delivery; if a subscriber is unavailable, the message is lost unless it's sent to an SQS queue or some other durable store.

Amazon SQS:

Message Persistence: Messages are stored in the queue until they are processed and deleted by a consumer, or until they expire.
Delivery Guarantees: Ensures at least once delivery. With FIFO queues, SQS provides exactly-once processing and message ordering.

Scalability and Performance

Amazon SNS:

Scalable: Designed to handle massive numbers of messages and deliver them to large numbers of subscribers.
Latency: Typically has very low latency for message delivery.

Amazon SQS:

Scalable: Automatically scales to handle large volumes of messages. Suitable for high-throughput applications.
Latency: Slightly higher latency compared to SNS due to the nature of pull-based consumption.

Features and Capabilities

Amazon SNS:

Multiple Protocols: Supports multiple delivery protocols including HTTP/S, email, SMS, SQS, Lambda, and mobile push notifications.
Filtering: Allows message filtering, enabling subscribers to receive only the messages that match their filter policies.

Amazon SQS:

Visibility Timeout: Temporarily hides a message from other consumers while it is being processed.
Dead-Letter Queues (DLQ): Allows you to handle messages that can't be processed successfully.
FIFO Queues: Ensures the order of messages and exactly-once processing.
Delay Queues: Postpones the delivery of new messages to consumers for a specified amount of time.

Pricing

Amazon SNS:

Pricing Model: Based on the number of requests (publishes, deliveries, and notifications) and data transfer.
Cost Efficiency: More cost-effective for scenarios requiring a high number of subscribers and real-time notifications.

Amazon SQS:

Pricing Model: Based on the number of requests (send, receive, delete) and data transfer.
Cost Efficiency: More cost-effective for decoupling microservices and scenarios requiring message persistence and complex message handling.

Integration and Interoperability

Amazon SNS:

Integration: Easily integrates with a wide range of AWS services (e.g., Lambda, SQS, HTTP/S endpoints, etc.).
Interoperability: Often used in conjunction with SQS for fan-out scenarios where messages need to be processed asynchronously and stored reliably.

Amazon SQS:

Integration: Commonly used to decouple systems and provide reliable message delivery. Often used with other AWS services like Lambda, ECS, and EC2.
Interoperability: Can be subscribed to SNS topics to receive messages that need persistent storage or further processing.

Summary

In summary, Amazon SNS is a pub/sub messaging service optimized for real-time notifications and broadcasting messages to multiple subscribers, while Amazon SQS is a message queuing service designed for decoupling distributed systems and ensuring reliable message delivery through persistence and processing guarantees. They are often used together to build scalable, resilient, and flexible messaging architectures in AWS.

Push Notification Service - Amazon Simple Notification Service - AWS

Provisioning AWS EKS Cluster with terraform-aws-modules/eks/aws

2026-04-21T13:07:00.002+01:00

terraform-aws-modules/terraform-aws-eks: Terraform module to create Amazon Elastic Kubernetes (EKS) resources is a popular Terraform module for provisioning AWS EKS cluster.

In this article we want to explore and breakdown its key components and their purposes.

We'd typically use this module like here:

module "eks" {

source = "terraform-aws-modules/eks/aws"

version = "21.15.1"

...

}

Let's explore this module's attributes.

1. Cluster Configuration

name, version

Sets the name and Kubernetes version for the EKS cluster. Use local and variable values for flexibility.

endpoint_public_access

Set public (Internet) access to the Kubernetes API endpoint (via kubectl). Disable it for enhanced security.

endpoint_private_access

Set private access to the API endpoint, whether only resources within the VPC can access it. If enabled, it is only reachable from within the VPC (Virtual Private Cloud) where your EKS cluster is deployed. There are few ways to access it:

How to Access the Kubernetes API from VPC

1. Use a Bastion Host or EC2 Instance in the VPC

Launch an EC2 instance (bastion host or jump box) in a subnet within the same VPC as your EKS cluster.
SSH into this instance, and from there, use kubectl to access the cluster.
Alternatively, use SSH port forwarding or a VPN to proxy kubectl commands from your local machine through the bastion.

2. Use AWS Systems Manager (SSM) Session Manager

If your EC2 instances have the SSM agent and the necessary IAM permissions, you can use AWS SSM Session Manager to start a shell session on an instance in the VPC, then run kubectl from there.

3. Use a VPN Connection

Set up a VPN (such as AWS Client VPN or OpenVPN, or Site-to-site VPN for office LAN) that connects your local network to the VPC. Once connected, your local machine will be able to reach the private endpoint.

4. Use AWS PrivateLink (Interface VPC Endpoints)

For advanced scenarios, you can use AWS PrivateLink to expose the Kubernetes API endpoint privately to other VPCs or on-premises networks.

enable_cluster_creator_admin_permissions

If enabled, grants admin permissions to the user who creates the cluster.

2. Logging and Add-ons

enabled_log_types

Enables logging for various Kubernetes components (API, audit, authenticator, controllerManager, scheduler) for monitoring and troubleshooting.

Example:

enabled_log_types = [

"api",

"audit",

"authenticator",

"controllerManager",

"scheduler"

]

addons

A dictionary-type attribute which installs and configures essential Kubernetes add-ons. Dictionary keys are addon names like:

coredns
kube-proxy
aws-ebs-csi-driver
vpc-cni

Dictionary values are objects which attributes are:

most_recent - to set using the latest version (set it to false for version pinning)
version - addon version (use it for version pinning)
before_compute - set it to true if addon should be installed and set before nodes (compute layer)
service_account_role_arn - to configure addon with IAM roles for service accounts, enabling secure integration with AWS services.

Example:

addons = {

...

vpc-cni = {

most_recent = false

version = "v1.21.1-eksbuild.7"

before_compute = true

service_account_role_arn = module.k8s_default_vpc_cni_irsa.iam_role_arn

}

...

}

VPC CNI (Container networking interface) is responsible for allocating IP addresses to the Kubernetes nodes and provides networking to pods. The plugin manage network interfaces (ENIs) on the nodes and uses it to assign IP addresses to pods.

3. Networking

We need to integrate the EKS cluster with existing VPC and subnets:

vpc_id

VPC ID

subnet_ids

Subnets in which nodes (EC2 instances) will be created.

Where your worker nodes (EC2 instances) run.

control_plane_subnet_ids

Where the EKS control plane ENIs (network interfaces) are placed

Defines where the EKS control plane creates its Elastic Network Interfaces (ENIs)

What it controls:

The EKS control plane runs in an AWS-managed VPC (you don't see it)
To communicate with your worker nodes, it creates ENIs in your VPC
These ENIs are placed in the subnets you specify here

Typical configuration:

module "eks" {

source = "terraform-aws-modules/eks/aws"

name = "my-cluster"

# Control plane ENIs go here

control_plane_subnet_ids = [

"subnet-private-1a",

"subnet-private-1b",

"subnet-private-1c"

]

}

Best practices:

Usually private subnets
Should span multiple AZs for high availability (AWS requires at least 2)
Minimum of 2 subnets, maximum of 16
Each subnet needs at least 5 available IP addresses

What these ENIs do:

Allow the control plane to communicate with worker nodes
Allow worker nodes to communicate with the API server
Handle API server endpoint traffic

security_group_additional_rules

Adds custom security group rules for the cluster, such as allowing node-to-node communication and VPN access for kubectl.

node_security_group_additional_rules

Further customizes node security groups, allowing all node-to-node traffic and all outbound traffic.

Understanding EKS Architecture

An EKS cluster has two main components:

┌─────────────────────────────────────────────────────────┐

│ EKS Cluster │

│ │

│ ┌───────────────────────────────────────┐ │

│ │ Control Plane (AWS Managed) │ │

│ │ - API Server │ │

│ │ - etcd │ │

│ │ - Scheduler │ │

│ │ - Controller Manager │ │

│ │ │ │

│ │ Runs in AWS-managed account │ │

│ └──────────────┬────────────────────────┘ │

│ │ │

│ │ ENIs in your VPC │

│ │ (control_plane_subnet_ids) │

│ ┌──────────────▼────────────────────────┐ │

│ │ Your VPC │ │

│ │ ┌─────────────────────────────┐ │ │

│ │ │ Worker Nodes (subnet_ids) │ │ │

│ │ │ - EC2 instances │ │ │

│ │ │ - Your pods run here │ │ │

│ │ └─────────────────────────────┘ │ │

│ └───────────────────────────────────────┘ │

└─────────────────────────────────────────────────────────┘

ENI: elastic network interface. It is a logical networking component in a VPC that represents a virtual network card.

4. Node Group Configuration

node_security_group_tags

Adds a tag for Karpenter (an open-source Kubernetes node autoscaler) discovery.

eks_managed_node_group_defaults

Sets default properties for all managed node groups, including:

Attaching the CNI policy for networking.
Using a specific SSH key.
Associating additional security groups.
Defining block device mappings for EBS volumes.
Attaching the AmazonSSMManagedInstanceCore policy for SSM access.

eks_managed_node_groups

Defines a default managed node group with:

A specific AMI type.
Desired, minimum, and maximum node counts.
Instance types from a variable.
On-demand capacity, EBS optimization, and disk size.
Custom labels for node identification and environment.

The gold standard for production environments is explicit pinning. This ensures that our infrastructure only changes when we decide to change the code. In order to pin AMI version used in node groups we need to set two attributes:

ami_release_version needs to be set. This prevents nodes from cycling unexpectedly during a routine deployment.
use_latest_ami_release_version needs to be set to false (without this, terraform plan will still show that it wants to upgrade AMI version, even if we've set ami_release_version)

Example:

eks_managed_node_groups = {

"${local.cluster_name}-v1_33" = {

...

ami_release_version = "1.33.8-20260224"

use_latest_ami_release_version = false

...

5. Tagging

Summary

Our configuration sets up a secure, private, and production-ready EKS cluster with managed node groups, essential add-ons, robust logging, and fine-grained network and IAM controls. It leverages best practices for security (private endpoints, IAM roles for service accounts), scalability (managed node groups, Karpenter tags), and maintainability (modular, versioned, and tagged infrastructure).

---

terraform-aws-iam/modules/iam-role-for-service-accounts-eks