MisLinux

How I Manage AI Agents Like a New Hire (Detailed Briefs, Reviewed Output, CIR Notes)

2026-06-16T12:14:41.750+02:00

By Vicente Arteaga Gomez

MisLinux

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It describes how I actually *manage* AI coding agents day to day — not the tools themselves, but the operating model that keeps them useful in production work.

Most teams experiment with AI in one of two broken modes:

Magic button — "fix it" with no context, hope for the best.
Copilot as autocomplete — fast typing, no durable output.

The model that works for business and operations automation is a third one:

> Treat the agent like a new hire who is fast, literal, and forgetful — brilliant at execution when the brief is clear, dangerous when the brief is vague, and unable to remember yesterday unless you wrote it down.

The contract I use with every agent session

New-hire parallel	What I ask the AI to do
Onboarding packet	Point it at `AGENTS.md`, relevant `SKILL.md`, and proof artifacts from the last run
Written task brief	State goal, constraints, what must not change, and how success is verified
Deliverable, not chat	Require a script/program/test/CIR entry — not "here is what you could run"
Code review	Read the diff myself; rerun dry-run/smoke tests before any production path
Manager notes	Append Context / Intent / Rationale when behavior or tooling changes

If the session ends with only prose, I failed the brief.

Step 1: Route manual work through the agent

When I catch myself doing something twice — spreadsheet reconciliation, onboarding checks, config diff, report export — I stop and rephrase:

> "Do this for me, but implement it as a repeatable script with tests and a CIR entry. Dry-run first."

That single sentence forces three outputs:

Automation — PHP/Python/ shell the agent can rerun without the conversation
Guardrails — PHPUnit or smoke tests on the invariant logic
Memory — AGENTS.md bullet explaining *why* the approach exists

The agent is the implementer. I remain the approver.

Step 2: Write briefs the way you would for a junior engineer

Vague briefs produce confident wrong code. I include:

Scope boundary — read-only vs production mutation; which namespace/network/sheet
Inputs — file paths, env vars, credentials *location* (never the secret itself)
Expected output — JSON shape, exit codes, artifact directory layout
Verification — exact command I will run to prove it worked
Discarded options — "do not patch production CronJob inline; fix the generator"

Example weak brief:

> Clean up the onboarding mess.

Example strong brief:

> Add onboard.php --plan read-only mode for YAML manifests. Must label EXTERNAL steps, emit JSON for monitoring, PHPUnit for the diff engine, CIR in operations/adsystem/AGENTS.md, and save proof under history/<timestamp>/. No live DB writes in this slice.

The second brief is longer because ambiguity is expensive.

Step 3: Review output like a manager, not a spectator

I assume the first pass is wrong somewhere. My review checklist:

[ ] Did it touch only the files the brief allowed?
[ ] Does dry-run default to safe?
[ ] Are tests asserting behavior, not implementation trivia?
[ ] Does failure output say expected vs actual?
[ ] Is there a CIR entry with Rationale (not just Context)?
[ ] Would another operator know how to rerun this in six months?

When something is off, I nudge, not restart:

> "Keep the planner, but EXTERNAL must not count as WRONG. Add a test. Update CIR with why."

That preserves context already loaded in the session.

Step 4: Demand CIR annotations for every non-obvious choice

Models forget. Repositories should not.

Good CIR captures why, not what:

- Context: onboarding checks were duplicated in chat and cron.
  Intent: one manifest drives plan, apply, and readiness monitoring.
  Rationale: without a single contract, AI regenerated slightly different
  checks each session; manifest + `--plan` makes drift visible before apply.

I explicitly ask:

> "Add a CIR entry to the nearest AGENTS.md explaining what you tried, what failed, and what would re-break if reverted."

Without that, the next agent (or me in three weeks) "cleans up" the guardrail that prevented an outage.

Step 5: Reduce manual touchpoints every iteration

Each manual step is randomness:

Wrong network selected
Stale token
Spreadsheet column renamed overnight
"I thought we already ran that"

My metric: count human decisions per run. If a process still needs ten, the next session removes two — with tests.

Automation does not mean zero humans. It means humans approve gates, not re-type data.

What this looks like in practice

Recent classes of work where the new-hire model helped:

Chore	Agent built	I approved
Publisher onboarding diff	YAML manifest + `--plan` library	Read-only proof JSON before any apply
Control panel ad-request filter	Second-pass PHP + PHPUnit	One-off kubectl job logs
Blogger publish recovery	CDP publish helper fixes + crawl gate	Public HTML verification
Finance month fill	Mapping libraries + failure email contract	Dry-run Job in cluster

In every case, the valuable artifact was not the chat. It was the script + test + CIR triad.

Anti-patterns I stop early

Anti-pattern	Why it fails
"Just do it live"	No replay, no audit trail
Accepting code without tests	Regresses silently on next model pass
Letting AI edit production YAML by hand	Drifts from generator source of truth
Skipping CIR "because it is obvious"	Obvious fades in two weeks
Treating refusal as failure	Good agent should block unsafe mutations

Sibling posts on this blog

Together they describe one strategy: delegate execution to the agent, keep judgment and memory in the repo.

---

Independence note: AI tools mentioned (Copilot, Claude, Codex, Cursor) reflect my stack. No vendor sponsorship implied.

How I Fixed Cursor Getting Slower Every Message (Desktop + CLI, Any OS)

2026-06-16T12:13:02.642+02:00

By Vicente Arteaga Gomez

MisLinux

This is a standalone MisLinux article about a real performance problem I hit while working in large infrastructure repositories with AI coding agents. It reflects my own experience only; I am not affiliated with, sponsored by, or endorsed by Cursor or Anysphere.

The symptom was subtle at first and painful later:

Open a repo. Start a chat. Everything feels snappy.
After a few turns, typing in the chat box gets slower.
After more turns, even single-character input lags noticeably.
Activity Monitor / Task Manager / top shows Cursor helper or indexer processes burning CPU.

That pattern looked like model slowness. It was not. It was indexer churn.

What was actually happening

Cursor (IDE and CLI) indexes the workspace so agents can search and reason about files. That is useful when the indexed files are source code you edit.

It is harmful when the indexer keeps re-walking directories that never help coding:

Noise directory	Why it hurts
`history/` and `**/history/`	Timestamped run artifacts, JSON dumps, screenshots
`__pycache__/`, `.pytest_cache/`	Regenerated on every test run
`node_modules/`, `vendor/` (when huge)	Third-party trees
`.sqlite`, large `.json`, `*.csv`	Operational databases and exports
Media (`.png`, `.mp4`, …)	Useless for code intelligence, expensive to scan

In my ops repo, those paths added up to gigabytes of churn. The chat got slower because the indexer kept working, not because the model suddenly became dumb.

The fix: a root `.cursorignore`

Git has .gitignore. Cursor has .cursorignore at the workspace root (same glob semantics).

The goal is not to hide code. The goal is to starve the indexer of directories that agents should not treat as editable source.

Here is the template I use on large monorepos:

# Dependencies and build output
node_modules/
vendor/
dist/
build/
.next/
out/
target/
coverage/
.turbo/
.cache/

# Python/runtime caches
__pycache__/
*.pyc
venv/
.venv/
.pytest_cache/

# Generated / minified assets
*.min.js
*.min.css
*.bundle.js
*.d.ts.map
*.tsbuildinfo

# Operational data (not source)
*.csv
*.json.gz
*.parquet
*.sqlite
*.sql

# Lock files (large, low signal)
package-lock.json
yarn.lock
pnpm-lock.yaml
Cargo.lock

# Media
*.png
*.jpg
*.gif
*.svg
*.mp4
*.woff2

# Repo-specific noise — adapt these
history/
**/history/
operations/**/history/

Three rules that mattered most for me:

Put .cursorignore at each repo root you open in Cursor (not only a parent folder).
If you use a monorepo, consider opening the smallest subdirectory that contains the code you edit that day.
After the first ignore file, restart Cursor once (or trigger reindex) so the exclusion actually applies.

How I verify the fix worked

I use a simple before/after checklist:

Check	Pass criteria
Character latency	Type in chat for 2 minutes — lag should stay flat, not climb
CPU	Cursor indexer/helper CPU should drop after reindex settles
Agent usefulness	Agent can still read the files you care about when explicitly referenced

If chat is still slow after a good .cursorignore, the next suspects are: opening too large a folder, an extension fighting the indexer, or a workspace with thousands of small files outside ignore rules.

Cursor CLI uses the same contract

If you run Cursor Agent from the CLI against a checkout, the same .cursorignore at that checkout root applies. CLI and IDE share the indexing model.

Practical CLI habit: cd into the service directory you are changing, not the entire 50 GB infrastructure tree, unless you truly need cross-repo context.

Windows, Linux, and macOS

The file name and location are the same on all platforms: .cursorignore in the project root.

Path separators inside patterns use forward slashes (Cursor/glob style), even on Windows.

Typical locations I open:

OS	Example workspace root
Linux	`/var/www/my-service/`
macOS	`~/Projects/my-service/`
Windows	`C:\Users\me\Projects\my-service\`

When `.cursorignore` is not enough

It fixes indexer noise. It does not fix:

A chat session that already accumulated a huge context window — start a fresh chat after large investigations.
Needing files that you ignored — narrow ignores, or @-mention specific paths.
Multi-root workspaces where only one root has ignore rules — add .cursorignore per root.

My decision rule now

Before blaming the model or switching tools, I ask:

> Is the indexer eating directories that are not source code?

If yes, I fix .cursorignore first. That single file has saved me more wall-clock time than any prompt tweak.

Business Automation Through AI: How I Turn Manual Chores Into Tested Scripts

2026-06-16T11:31:27.997+02:00

By Vicente Arteaga Gomez

MisLinux

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It describes how I actually use AI coding agents for business and operations work — not as a chatbot, but as a compiler from messy intent into durable automation.

Most "AI for business" content stops at drafting emails or summarizing meetings. That is fine. It is also ephemeral.

The pattern that changed my operations work is different:

> Every time I catch myself doing a manual chore twice, I ask the AI to implement a scripted, tested procedure — and I capture the nuance in AGENTS.md or SKILL.md.

Manual intervention becomes the exception. Randomness shrinks.

The loop I use

1. Do the task once with AI assistance (read-only first)
2. Demand a script/program that encodes the steps
3. Add tests at the strongest safe level (unit → smoke → integration)
4. Record Context / Intent / Rationale in AGENTS.md
5. Save proof artifacts from the first real run
6. Next time: rerun the script, not the conversation

Step 2 is the discipline. Without it, AI becomes a very fast way to recreate tribal knowledge in chat logs.

Example: from spreadsheet panic to finance fill automation

Finance month-close used to fail loudly but opaquely: a missing title row, a renamed column, a wrong rev-share cell — and the operator rebuilt context from memory under time pressure.

The automated path now:

Layer	Purpose
PHP fill libraries	Deterministic column mapping + rev-share math
CronJob in cluster	Scheduled run with read-only staging inputs
PHPUnit	Locks mapping rules offline
CIR in AGENTS.md	Explains why Spanish locale uses comma decimals
Failure email	Includes sheet URL, stage, excerpt, relaunch command

The AI did not "run finance." It helped me extract the invariant rules and encode them where regressions are caught before production.

Example: publisher onboarding as manifest-driven automation

Another class of business work: onboarding a publisher across ad server, dashboards, tags, and internal CMS records.

The AI helped build:

onboard.php --plan / --apply over YAML manifests
Readiness checks wired to Telegram on failure only
Funnel diagnostics (wrapper traffic → ad server → dashboard revenue)

Each piece has tests. Each non-automatable seam is labeled EXTERNAL in plan output instead of silently skipped.

That is business automation: the contract is in git, not in someone's notebook.

Where AGENTS.md and SKILL.md fit

Models forget. Repositories should not.

File	What it captures
`AGENTS.md`	Context / Intent / Rationale ledger — why the system looks this way
`SKILL.md`	Operator runbook — ordered steps, inputs, safety gates
Top-of-file CIR in scripts	Why this script exists and what breaks if reverted

When a future agent (or human) asks "why is this weird?", the answer is searchable.

Good CIR entry shape:

- Context: what was happening.
  Intent: what we wanted.
  Rationale: why this approach; what we discarded; what re-breaks if reverted.

Without Rationale, the next person "cleans up" the guardrail that prevented yesterday's outage.

Tests are the guardrails, not decoration

I default to PHPUnit in this repo. Python appears where browser CDP is genuinely better — not because a one-off was faster in Python.

Test layers I aim for on operational code:

Layer	Catches
Unit	Pure logic regressions (parsers, planners, diff engines)
Smoke	CLI wiring, `--help`, required args
Fuzzy / property	Random inputs that must not crash or violate invariants
Integration	Fixture pipelines end-to-end offline
Gated e2e	Live API/browser only when explicitly enabled

If a layer is impractical, the blocker goes in AGENTS.md with the nearest safe substitute. "No tests" is not silent.

How I prompt AI so the output is durable

Weak prompt:

> Fix the control panel numbers for one publisher row.

Strong prompt:

> Read the existing control-panel updater. Add a read-only second pass that filters ad requests using intranet flags. Must be idempotent, PHPUnit-tested, CIR in AGENTS.md, dry-run default, and include exact kubectl job command for one-off proof.

The second prompt forces artifacts — the things that survive the session.

Reducing manual points reduces randomness

Every manual step is a slot for:

Wrong network selected
Stale OAuth token
Spreadsheet column drift
"I thought we already ran that"

Automation does not mean zero humans. It means humans approve gates, not re-type data.

My practical metric: count manual decisions per run. If a process still needs ten, the next AI session should remove two — with tests.

What AI is bad at in business automation

Be explicit about limits:

Economically loaded ad server UI changes without human confirm
Irreversible production mutations without dry-run proof
Legal/commercial judgment on contract terms
Pretending read-only credentials can write because "the script should"

The manifest/onboard pattern works because it separates check from apply and labels EXTERNAL work honestly.

Checklist: is this real automation?

Before I merge operational tooling, I ask:

[ ] Can a new operator rerun it from --help output alone?
[ ] Does dry-run exist and default to safe?
[ ] Is there at least one automated test locking the core logic?
[ ] Is there a CIR entry explaining discarded alternatives?
[ ] Does failure output say what was expected vs what was found?
[ ] Is there a saved artifact from a successful run?

If most boxes are empty, I have a chat transcript — not automation.

Tie-in to AI-assisted development hygiene

Two sibling posts on this blog:

How I Use CIR Notes to Make AI Coding Agents More Useful
How I Fixed Cursor Getting Slower Every Message — keep indexer noise out of the repo so agents stay fast

Business automation and dev environment hygiene are the same strategy: make the machine repeat the boring truth, and document why the truth looks odd.

---

Independence note: Tools mentioned (Copilot, Claude, Codex, Cursor, Google Ad Manager, Kubernetes) reflect my stack. No vendor sponsorship implied.

Publisher Onboarding as Manifests: How I Automate Ad Server, ads.txt, and Intranet Setup with YAML

2026-06-16T11:28:34.626+02:00

By Vicente Arteaga Gomez

MisLinux

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It comes from building a real publisher-onboarding pipeline in production, not from a vendor playbook.

Onboarding a new publisher used to mean a long checklist in someone's head:

Create ad units in the ad server
Wire demand and pricing floors
Register the child network and seller ID
Stand up a revenue dashboard spreadsheet
Confirm ads.txt and sellers.json
Create intranet publisher, sites, wrappers, and contracts
Send tags to the publisher

Each step touched a different system. Each step had a different tool. And "done" meant different things to different people.

I fixed that by treating onboarding as declarative desired state.

The core idea

One YAML file describes what "fully onboarded" means for a publisher:

publisher: Example Media Co
gam:
  parent_network: "12345678901"
  child_network: "98765432"
  seller_id: pub-example-001
sites:
  - domain: example-news.com
    display_use_domain: true
tags:
  parent_network: "12345678901"
  communication_language: en
stats:
  expect_adunits:
    - Example_news.com-320x50_Display
    - Example_news.com-728x90_Display
    - Example_news.com-PR1_Video
readiness:
  wrappers:
    - uuid: abc123def456-d
  lineitems:
    - name: AdExchange Example Video

Then one orchestrator command compares live state against that manifest:

php onboard.php --config=manifests/example-publisher.onboard.yaml --plan
php onboard.php --config=manifests/example-publisher.onboard.yaml --apply=pcontrol --confirm
php onboard.php --config=manifests/example-publisher.onboard.yaml --apply=all --confirm

--plan is read-only. It prints OK, MISSING, WRONG, or EXTERNAL for every step.

--apply mutates only the steps that are safe to automate. Everything else stays a explicit human gate.

That split is the whole safety model.

Why manifests beat ad-hoc scripts

Ad-hoc approach	Manifest approach
"Run these twelve commands in order"	One file defines end state
Hard to see what is still missing	`--plan` is a live diff
Every new publisher copies paste errors	New publisher = new YAML
Monitoring is an afterthought	Checks reuse the same manifest

The manifest is not magic. It is the contract that scripts, tests, and monitoring all read.

What one onboarding manifest covers

In my pipeline, a full publisher touches roughly nine areas:

Area	Manifest expresses	Automation level
GAM inventory	Ad unit names, sizes, tiers	Apply where API is safe
AdExchange demand	Line item names + targeted units	Partial — creative activation may be UI-only
Placement membership	Union into shared placement	Apply with idempotent sync
UPR floors	Expected floor levels vs wrapper	Check against daily export
MCM child / seller ID	Network codes + seller id	Read-only verify
Revenue dashboard	Report IDs + sheet templates	Seed + management rows
ads.txt / sellers.json	Exact lines expected per domain	HTTP fetch + match
Intranet objects	Company, network, sites, wrappers	DB read checks + controlled writes
Publisher comms	Tag email EN/ES artifacts	Generated from manifest inputs

Some steps are fully automated. Some are assisted (human in GAM UI, machine verifies after). The manifest makes that honest instead of pretending everything is one button.

Checks that can hook into monitoring

The same readiness config powers a daily cron:

php publisher-readiness-check.php --config=manifests/example-publisher.onboard.yaml --json

Gates include:

Wrapper serves VAST with expected ad unit in the tag
Line items are active and target the right inventory
Control panel rows exist with correct metric flags
Live ads.txt contains the expected DIRECT/RESELLER lines
GAM inventory count matches expectation

When a gate fails, Telegram gets an incident message. When everything is green, silence.

That is the monitoring story the operator asked for: detect activity and detect missing setup, not spam on success.

Detecting "onboarding is working" via traffic

Separate mode: day-by-day ad request stats per ad unit:

php onboard.php --config=manifests/example-publisher.onboard.yaml --stats=14

This answers: "We onboarded ten ad units — which ones are actually receiving requests?"

A roster unit with zero requests is not always failure (publisher has not deployed yet). But once wrapper traffic exists and GAM requests stay zero, the funnel check localizes the break:

php onboard.php --config=manifests/example-publisher.onboard.yaml --funnel --days=7

Wrapper requests → GAM ad unit requests → dashboard revenue. The hop where upstream is non-zero and downstream is zero names the broken layer.

AI's role: build the procedure, not wing the production click

I use AI agents to author and harden the onboarding tooling:

Generate check libraries and PHPUnit tests
Record CIR notes when GAM API quirks block automation
Produce diff output (--diff) that shows field-level expected vs actual

I do not use AI to "just click through GAM" on economically loaded pricing rules without guardrails.

The manifest makes AI useful because the desired state is explicit. The agent is not guessing what "done" means.

What I would not put in a manifest

Keep human gates visible:

Creating UPR rules that set global floors (no public API — browser workflow with confirm tokens)
First-time publisher legal review
Publisher-side ads.txt edits (we verify, they publish)

Trying to hide those behind --apply=all would create false confidence.

Reusable decision rules

If it is read-only, automate the check first. Production mutation comes later.
If two systems must agree (intranet DB vs UI), verify both. One path lies sometimes.
If a step is EXTERNAL, say so in `--plan`. Do not mark WRONG when waiting on the publisher.
Store proof artifacts next to the manifest run (history/onboarding-emails/..., activation proof JSON).

Closing thought

Publisher onboarding is a business process disguised as nine technical chores.

YAML manifests do not remove the chores. They make desired state portable, testable, and monitorable — which is what lets AI help without turning production into a roulette wheel.

---

Scope note: Examples use fictional publisher, domain, and network identifiers. The pattern reflects MCM-style ad-ops workflows from my production environment; it is not a case study of any live customer.

How I Turn Read-Only Analysis Into Safer Production Automation

2026-06-04T11:17:00.000+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It is about building safer automation from real operations, not about promising autonomous production.

The automation path I trust most rarely starts with a mutating script.

It usually starts with read-only analysis:

inspect the live state
preserve artifacts
learn which fields really matter
discover where the false signals are

Only after that do I want to automate the mutation.

Why I prefer this order

Because the read-only phase tells me:

What I learn	Why it matters for automation
true runtime contract	prevents automating the wrong layer
trustworthy proof signal	tells me what success should look like
rejected paths	stops the next tool from repeating them
safe scope boundary	keeps the mutation narrow

That means the later automation is grounded in reality instead of optimism.

The transition I want to see

The important thing there is that proof is not an afterthought. Proof is what bridges the investigation into the automation contract.

A small workflow example

# 1. Read-only inspection first
php inspect-current-state.php --out history/20260505-proof/

# 2. Build a proposed mutation from that evidence
php build-change-plan.php --from history/20260505-proof/

# 3. Dry-run the mutation
php apply-change.php --plan history/20260505-proof/plan.json --dry-run

# 4. Apply only once proof + rollback are explicit
php apply-change.php --plan history/20260505-proof/plan.json

This is much safer than starting from step 4 and improvising the rest around it.

What I watch for before promoting read-only work into automation

Is the input shape stable enough?
Is the output verifiable?
Can I preserve a before-state?
Is rollback explicit?
Do I understand what *not* to automate yet?

The last question matters a lot. Some workflows should stay partially manual.

Failure case: skipping the read-only learning phase

If I skip the read-only phase, the first mutation script often makes one of these mistakes:

patches the wrong representation
treats a derived metric as the source of truth
assumes a browser/UI state that was never actually reached
writes a partial output where a full-file rewrite is required

Those are not abstract risks. They are exactly the kinds of misunderstandings that production tooling accumulates.

Why this matters for AI agents

This is probably the most useful pattern I have found for AI-assisted infrastructure work:

let the agent investigate read-only first
force it to preserve artifacts and explain the contract
then allow the mutating helper to exist
keep proof and rollback attached to the same workflow

That is a much better path than starting with "write the deployment fix."

What I'd do differently now

I used to think read-only analysis was just preparation before the "real" automation work. What I'd do differently now is treat the read-only phase as the place where the automation earns its right to exist.

Why Live Follow-Up Files Beat Forgetting the Last Incident

2026-06-01T15:41:00.000+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It reflects the follow-up system I now prefer for ongoing operational questions.

The incident is not really over just because the page stopped firing.

Some of the most important work happens after the immediate pressure is gone:

verify whether the fix actually held
confirm second-order metrics recovered
watch for regressions a few days later
record which questions are still open

If I do not keep those as live follow-up items, they turn into vague intentions that disappear under the next urgent task.

What I want a follow-up file to contain

Section	Purpose
Original issue	why this item exists
Analysis	what the investigation actually found
Actions taken	what changed already
Future checks	what still needs to be watched
Artifact links	where the evidence lives

That is enough to turn a lingering concern into an inspectable queue item.

Why one markdown file is often enough

I do not need a huge tracker to get value here.

One file per follow-up works because it is:

easy to diff
easy to link from a registry page
easy to archive later
easy to extend when the story changes

What matters is that the follow-up is durable and evidence-backed, not that it looks like an enterprise ticket system.

The real advantage

The biggest benefit is not organization. It is time.

When the same topic resurfaces, I want to answer these questions quickly:

what exactly happened last time?
what did we change already?
what metric were we supposed to keep watching?
which artifact folder contains the supporting data?

That is much easier with a live follow-up file than with memory plus old shell output.

Failure case: the unresolved item that looks resolved

One of the most common mistakes in operational work is treating "the immediate symptom improved" as equivalent to "the issue is closed."

That hides important follow-up questions like:

did total demand recover or only one slice?
did the workaround create a new imbalance?
did a monitoring threshold now become misleading?

Those are exactly the questions I want to keep alive explicitly.

What I'd do differently now

I used to write big postmortems and almost no living follow-up notes. What I'd do differently now is create the follow-up file as soon as I know the issue needs days or weeks of observation, even if the initial analysis is not perfect yet.

How I Decide When an Operator-Facing UI Is Good Enough to Operate From

2026-05-30T10:05:00.000+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It is about operator-facing tools in general: spreadsheets, intranet dashboards, web pages, status panels, or any other UI that asks people to make decisions from what it shows.

I like operator-facing UIs when they are honest.

By honest, I mean the sheet tells me two things clearly:

how an operator should use it
how the important fields were computed

Without those two pieces, a dashboard can still look polished while quietly training people to trust unexplained numbers or unexplained statuses.

The minimum questions I ask

Question	Why it matters
What should I look at first?	reduces wasted scanning time
What does the status label actually mean?	prevents false confidence
What is a shortcut metric and what is the real classifier?	stops shallow reading
Where can I drill into one row?	makes investigation possible
Where are the thresholds and caveats?	avoids operator folklore

If the UI cannot answer those questions near the place where people actually use it, I do not think it is ready.

The two sections I now prefer

I like a dedicated docs panel, inline help surface, nearby runbook, or equivalent split into:

Operator guide

what the main tab is for
what to compare first
where the notes/help live
where the drill-down tab is

Inner workings

how status is computed
what each important field means
what the trailing windows are
what caveats still exist

That split matters because most people do not need the formula explanation first thing in the morning, but they do need it during disputes, investigations, or audits.

Why comments/notes matter too

A good operator-facing UI often needs two layers of explanation:

stable documentation close to the tool
row, field, or state-specific detail near the exact number/status the user is reading

For example, averages become much more useful when the UI can expose the daily components behind them, and a status badge becomes much more trustworthy when the expert user can still reproduce the exact calculation externally.

That lets me answer:

> Was this driven by one spike day or by a sustained shift?

without leaving the sheet.

A simple readiness checklist

[ ] operator-facing usage guidance exists in the workbook
[ ] status logic is documented
[ ] key columns are defined
[ ] drill-down links/tabs work
[ ] filters/frozen headers support real operator use
[ ] caveats are written down explicitly

If two or three of those are missing, I treat the UI as provisional, not operational.

Failure case: polished but misleading

The failure mode I distrust most is the polished UI that has:

a strong-looking status column
nice formatting
no explanation of the thresholds
no path to the underlying detail
no explanation an expert user can use to reproduce the result externally and validate that the premises still hold

That kind of tool can create faster wrong decisions than a rough but transparent one.

What I'd do differently now

I used to think operator documentation was a nice-to-have once the formulas or backend logic were correct. What I'd do differently now is design the inexpert-user guide and the expert-user calculation notes as part of the same deliverable as the UI itself, so people can both use the tool and independently verify when it stops deserving trust.

How I Use Proof Artifacts to Keep One-Off Investigations Repeatable

2026-05-27T13:41:00.000+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It reflects my own incident and reporting workflow.

I no longer believe in the phrase "I already checked that."

If I checked it and did not preserve the result, what I really have is memory.

Memory is useful. It is not an artifact.

The artifact types I reach for first

Artifact	What it proves
JSON snapshot	machine-readable state at a point in time
Screenshot	the real UI path or rendered state
Before/after manifest	configuration delta
Job log	exact runtime behavior
Run note	why this artifact mattered

The run note is important because raw artifacts without framing can become archaeology instead of evidence.

Why I save them even for small tasks

Because small tasks become big tasks later.

The one-off investigation often becomes:

the baseline for a later regression
the model for a new helper script
the evidence behind a follow-up item
the reason a CIR note was added

That only works if the evidence still exists.

A folder layout I like

history/20260505-example-run/
  summary.json
  before-state.json
  after-state.json
  browser-proof.jpg
  command-log.txt
  run-notes.md

Nothing about that layout is magical. It is just easy to inspect later.

The mistake I am trying to avoid

The mistake is saving only the "pretty" output.

For example:

saving the chart but not the raw values
saving the screenshot but not the route or state summary
saving the report but not the query window

That creates a result I can cite but not re-derive.

Why this matters with AI agents

AI agents are much more useful when the artifact contract is explicit.

If the output folder has known files, the next run can:

diff the new result against the old one
explain what changed
reuse the same helper path
avoid re-grepping the whole repo

Without that structure, every investigation starts wide again.

A lightweight command trail

mkdir -p history/20260505-example-run
php collect-state.php > history/20260505-example-run/before-state.json
python3 take-proof-screenshot.py --out history/20260505-example-run/browser-proof.jpg
printf '%s\n' "Why this run exists" > history/20260505-example-run/run-notes.md

That is enough to make the work reusable.

Failure case: artifact-only without explanation

Artifacts alone are not enough if they do not say:

what the operator was trying to prove
whether the run was read-only or mutating
why this snapshot matters

That is why I still like a small summary file or run note next to the raw files.

What I'd do differently now

I used to save artifacts only after a task felt important. What I'd do differently now is decide the artifact contract at the start. The earlier I know what proof I want, the less likely I am to finish with missing evidence.

What Registry Garbage Collection Actually Costs in Operator Time

2026-05-22T17:53:00.000+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It is an operator-focused post about self-hosted registry maintenance, not a product review.

When people talk about registry garbage collection, they usually talk about storage.

I think the more painful cost is operator time.

Not because cleanup commands are long, but because safe cleanup turns into a proof exercise:

which tags are actually still referenced?
which ones only *look* referenced?
are the manifest children still readable?
which tags are part of failover or rollback paths?

The delete is fast. The confidence is expensive.

The real work around cleanup

This is the table I wish more registry discussions started with:

Phase	Operator work
Before cleanup	classify runtime vs proof tags, inspect manifests, check active deployments
During cleanup	keep the destructive path narrow and explicit
After cleanup	prove that kept tags still resolve, pull, and match the intended runtime

That is why I now think of registry GC as a maintenance workflow, not a command.

Why tag existence is not enough

One of the nastiest cases in self-hosted registries is the "present but broken" tag:

the tag still resolves
the manifest still exists
one child descriptor or blob is gone
the next pull fails where nobody expected it

That means post-cleanup verification has to be deeper than "the tag name still appears."

The diagram I keep in mind

The chart uses the cleanup workflow phase on the X axis and a relative operator-effort score on the Y axis. It is not a benchmark. It is a way to show where the real time goes: keep-set classification, manifest/blob proof, and post-cleanup validation.

The key lesson there is that the operator cost grows with uncertainty about reachability and rollback, not with the number of characters in the command line.

The command trail I actually trust

# 1. List what production and failover still reference
kubectl get deploy,ds,cronjob -A -o json | jq '.items[].spec?'

# 2. Inspect manifest structure, not only tag names
./inspect-registry-manifest.sh runtime-tag --check-blobs

# 3. Run the cleanup only after the keep-set is explicit
registry garbage-collect /etc/distribution/config.yml

# 4. Re-run manifest and pull proof on the kept tags
./inspect-registry-manifest.sh runtime-tag --check-blobs
docker pull registry.example.invalid/app:runtime-tag

If I skip step 4, I have not actually finished the maintenance.

Here is what those commands are actually doing:

kubectl get ... asks Kubernetes which images the live workloads reference right now, so the cleanup keep-set starts from reality instead of memory
inspect-registry-manifest.sh --check-blobs walks the manifest and child blobs so a tag that merely looks present does not get mistaken for a healthy runtime artifact
registry garbage-collect ... is the destructive phase itself
the final inspect-registry-manifest.sh plus docker pull are the proof that the kept runtime path still resolves from the registry all the way down to pullable content

One additional check matters a lot in practice: monitor the exact images and tags the Kubernetes workloads are using, and verify that those tags are still resolvable in the registry. That catches registry drift early, before the next restart turns a hidden integrity problem into a live outage.

Failure case: cleanup without path awareness

The most dangerous cleanup is not "too aggressive" in the abstract. It is cleanup that does not know:

which tags the live cluster uses
which tags the standby/failover path uses
which per-architecture descriptors need direct protection

That is how storage maintenance becomes runtime breakage.

Why operator time dominates

The expensive part is not CPU. It is:

reading the current runtime references
classifying proof tags versus active tags
checking manifest reachability
confirming rollback paths were not silently broken

That is real maintenance work even if the registry disk frees space quickly.

What I'd do differently now

I used to think of registry GC as a storage task with a verification step. What I'd do differently now is describe it the other way around: it is a verification-heavy operational workflow that happens to reclaim storage at the end.

Why Monitoring Needs Host, Hosting, and Global Views at the Same Time

2026-05-17T17:05:00.000+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It is based on real operational monitoring design work, not a generic observability checklist.

One of the easiest ways to build a misleading dashboard is to choose only one monitoring scope.

If I watch only the global service, I can miss a saturated origin.

If I watch only the individual host, I can page on a healthy rebalance.

If I watch only the provider slice, I can miss a cross-provider traffic cliff.

That is why I now want three views at the same time:

View	Main question
Host	Is one concrete machine or origin sick, stale, or overloaded?
Hosting	Is one provider/location carrying the wrong share or drifting operationally?
Global	Is the public service really up, healthy, and serving normal demand overall?

What each level catches

Host view

The host view is where I want:

latency
CPU or saturation signals
config freshness
health endpoint state
reload or restart anomalies

This is how I catch "one box is bad" before it becomes "the system feels random."

Hosting view

The hosting view groups hosts by provider or operational domain.

That is where I can see:

traffic imbalance
one provider falling behind on config freshness
one provider serving the wrong share of requests
one provider drifting in latency

This matters even if there is only one host in that hosting today, because the grouping is an operational concept, not just a host count.

Global view

The global view answers the question that users care about:

> Is the public service healthy overall?

That is the view where I want:

total public traffic
total error rate
canary health across all active public origins
global per-UUID or per-surface demand cliffs

Without this layer, it is too easy to confuse a redistribution event with a real outage.

The architecture I prefer

The point of that diagram is that each level aggregates differently. They are not interchangeable charts with different legends.

A practical example

Imagine public traffic is split across two providers:

provider A carries less traffic than usual
provider B carries more traffic than usual
the total public request rate is stable

If I alert only on provider A, I page on a healthy balance shift.

If I alert only on global traffic, I miss the provider imbalance.

If I alert only on per-host CPU, I may not notice that one provider is now becoming the hidden bottleneck.

The right answer is to keep all three views and interpret them together.

Metrics I now want at each level

Scope	Metrics I care about first
Host	health, latency, CPU, config age, recent reload status
Hosting	total requests, mean latency, error rate, freshness spread, host count
Global	total public requests, total errors, demand cliffs, public canary result

That table looks obvious after the fact, but it took me longer than it should have to stop flattening these scopes into one graph.

Failure case: the wrong dashboard story

The wrong dashboard story sounds like this:

> "Traffic is down on one origin, so the public service is down."

That sentence mixes scopes. It takes a host observation and turns it into a global conclusion.

Once I noticed that pattern, a lot of monitoring noise suddenly made more sense.

Command trail I want during validation

# Host-level health
curl -s https://origin-a.example.invalid/health

# Hosting-level request split
python3 inspect-request-split.py --group-by hosting

# Global service trend
python3 inspect-request-split.py --group-by global

The exact implementation can vary. The important part is that the queries do not collapse distinct operational questions into one series.

What I'd do differently now

I used to think multiple scopes were a nice refinement for later. I now think they are the minimum design for any public service that can move traffic across origins or providers. If I were rebuilding the monitoring from scratch, I would model host, hosting, and global identity from day one.

What a Web AI Automation Blocker Really Costs During an Investigation

2026-05-14T16:05:00.001+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It is based on real operational browser work for web AI automation and adjacent browser-driven investigation paths, not on demo-only automation.

When a browser automation task gets blocked by login friction, bot protection, or a challenge page, the obvious cost is time.

The less obvious cost is decision quality.

Once the clean automated path is blocked, people often start improvising:

one manual browser for the challenge
another browser for the actual task
partial screenshots
copied HTML from the wrong frame
assumptions about what would have happened after the challenge

That is how proof quality degrades.

Why this matters

During an investigation, a browser path is often being used to answer one of these questions:

what does the real UI show right now?
which exact button or dialog state matters?
is the saved HTML the same as the rendered state?
which route fragment is still valid?

If the automation path is blocked, the investigation has to account for that blocker explicitly instead of pretending the missing evidence is equivalent to a negative result.

The three states I distinguish now

State	What it means
Automated success	The browser reached the intended DOM/runtime path and the evidence is complete
Manual bridge needed	A human has to clear a gate, then automation may resume
Hard blocker	The session cannot proceed safely or repeatably right now

That middle state matters a lot. Many investigations are not "blocked forever." They are "blocked until a human clears exactly one step."

What I do instead of brute forcing it

My preferred sequence is:

preserve the blocker evidence
launch a separate, clearly scoped headed session if needed
let the human clear the gate
resume automation from the now-authenticated session
save artifacts from the real post-gate state

This keeps the proof chain cleaner than mixing one-off manual browsing with later guesswork.

A command trail that reflects that model

# 1. Launch a separate headed session for the blocked surface
python3 ensure-browser-session.py --port 9230 --headed --target-url https://example.invalid

# 2. Record the blocker artifact first
python3 capture-blocked-state.py --port 9230 --output history/blocked-proof/

# 3. After the manual gate is cleared, resume the structured automation
python3 collect-real-ui-proof.py --port 9230 --output history/post-gate-proof/

The important part is not the exact script names. The important part is the explicit handoff.

In plain language, those example commands do three different jobs:

ensure-browser-session.py starts a separate browser profile on its own debugging port so the blocked flow does not contaminate the main session
capture-blocked-state.py saves the proof that the current stop point is the challenge page rather than the real application DOM
collect-real-ui-proof.py resumes structured inspection after the human clears the blocker, so the later screenshots/HTML/runtime notes all come from the real target state

The filenames here are illustrative. The reusable rule is what matters: separate the blocked session, preserve evidence of the blocker, and only then resume the automated collection path.

Failure case: what goes wrong if I skip the handoff model

If I skip that structure, I get mixed evidence:

screenshots from one session
HTML from another
cookies in a third
conclusions that assume all of them refer to the same state

That is especially dangerous when the system under investigation has expiring auth or anti-bot checks.

Why this matters for AI-assisted work

AI agents are very good at continuing confidently after a blocker page.

That is exactly why I want the blocker to be first-class in the workflow.

The right conclusion is often:

> "I do not have the actual runtime DOM yet. I only have the blocker state."

That is a useful result, but it is not the same result as a successful DOM inspection.

What I'd do differently now

I used to treat browser blockers as an inconvenience outside the "real" work. What I'd do differently now is design the browser workflow around them from the start: separate session, explicit handoff, and artifact-backed resumption. It turns a messy interruption into a real operator state.

Why I Treat Ad-Hoc Scripts Like Real Software

2026-05-12T15:05:00.001+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It comes from repeated operator work, not from a vendor playbook.

I used to use the phrase "just a quick script" as if it reduced the risk.

It did not.

The script still:

touched real data
encoded assumptions
created artifacts someone would rely on later
became copy-paste material for the next incident

At some point I stopped calling these scripts disposable. I started treating them as source code that merely had a short original deadline.

The rule I use now

If a script is important enough to run once against real operational data, it is important enough to deserve:

Minimum standard	Why it matters
DRY helpers	avoid re-deriving parsing and transport logic later
tests	catch the obvious contract breaks before the next rerun
CIR notes	preserve why the workflow exists in this shape
saved artifacts	prove what happened on that run

That does not mean every script needs a huge framework. It means the script should stop pretending it is exempt from engineering standards.

What changed my mind

The real problem was not the first run. The first run usually works well enough.

The problem was the second and third run:

slightly different data
slightly different date window
slightly different production state
slightly different operator

That is where "ad hoc" turns into "mysterious legacy."

A script can be short and still be real code

I do not need a large package structure before I apply discipline.

Even a small automation helper can be structured like this:

script entrypoint
  -> parse arguments
  -> call a shared library/helper
  -> write artifacts
  -> exit non-zero on real failure

That shape makes testing possible and keeps the second script from re-implementing the same logic a week later.

A concrete command trail I prefer

# Good: deterministic rerun with explicit output folder
php operations/example/report.php \
  --input artifacts/source.json \
  --output history/20260505-example-run/

# Good: test the shared logic separately
vendor/bin/phpunit tests/example/ReportBuilderTest.php

# Better: use the exact same helper from the next automation layer
php operations/example/next-stage-proof.php --from history/20260505-example-run/

This is much safer than three unrelated shell fragments that happen to work today.

The subtle benefit: better failure messages

Once I treat the script like real software, I also stop accepting vague operator output.

I want errors that explain:

what input was missing
what upstream state was unexpected
which artifact path contains the evidence

That matters more than people think. A one-off script with useless errors becomes a slow manual process again the moment it fails.

Failure case: the fake one-off

The fake one-off script usually looks like this:

one file
mixed fetching, parsing, mutation, and reporting
hardcoded ids or paths
no test seam
no artifact contract

Then somebody asks for the same run with a new date range or a new target, and now it has to be reverse-engineered from scratch.

Why this matters more with AI-assisted workflows

AI agents are extremely good at making a one-off script "work."

They are much less useful if the script:

hides its assumptions
cannot be re-run deterministically
has no tests
has no saved evidence

That is why I now want the script to be reusable before I want it to be clever.

My current checklist for a so-called ad-hoc script

Before I accept it, I want:

a small shared library if parsing or transport logic may recur
at least one unit test around the risky part
an output directory or artifact contract
one CIR note explaining why the script exists and what would go wrong if someone "simplified" it blindly

If it touches production directly, I want even more proof than that.

What I'd do differently now

I used to optimize for speed on the first run and structure on the second run. What I would do differently now is merge those decisions: build the smallest reusable version first. It is usually only slightly slower at the beginning and much faster after that.

How I Decide Whether an AI Agent Is Allowed to Touch Production

2026-05-08T17:53:00.011+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 20, 2026

This is a standalone MisLinux article about production safety and AI-assisted operations. It reflects my own working rules and mistakes. I am not affiliated with GitHub, OpenAI, Anthropic, or any other AI vendor mentioned here.

I like AI coding agents, but I do not think "the agent can do it" is a deployment policy.

The question I care about is more specific:

> Under what conditions is an AI agent allowed to inspect, modify, or deploy something that touches production?

That answer should not depend on how impressive the last demo was. It should depend on the shape of the task, the reversibility of the action, the quality of the documentation, and the proof I expect before the change becomes real.

My default assumption

My default assumption is that an AI agent is good at speed, pattern matching, and synthesis, but dangerous when context is ambiguous and the blast radius is large.

That means I am comfortable letting an agent do more when:

the task is well-bounded
the workflow is reproducible
the expected output can be validated clearly
rollback is easy

I get much more conservative when any of those stop being true.

The first filter: read-only or mutating?

The first question I ask is whether the task is read-only or mutating.

Read-only work is the easiest place to gain value safely:

investigate logs
compare manifests
trace where a metric comes from
generate a report
audit what changed

Mutation is a different category:

editing a production manifest
publishing a new image
updating an ad server or video player config
changing a live Blogger theme
switching a mail relay

I do not treat those categories as morally different. I treat them as risk-different.

The second filter: can the agent explain the exact runtime contract?

One of my strongest rules is that I want the system to be documented well enough that the agent can explain what the change actually does, not merely what line it changed.

That is one reason I use Context / Intent / Rationale notes near the code and configuration. Good CIR is not there to make the repo look organized. It is there so the next change is grounded in the reason the current behavior exists.

If an agent cannot explain:

why the current behavior was chosen
what would break if it is reverted
which constraints shaped the implementation

then I do not think it is ready to mutate that area safely.

The third filter: can I demand proof, not confidence?

This is the rule that matters most in practice.

I do not care whether the agent sounds confident. I care whether it can produce the right proof for the task.

The proof depends on the task:

Task type	Minimum proof I want
Code-only local change	tests and a reproducible output or diff
Build/publish change	successful image build plus inspection of what was published
Production config mutation	before/after state, live status proof, and rollback artifact
Browser/UI automation	saved screenshots or structured artifacts from the real UI path
Report generation	saved raw inputs plus derived summary output

This is where a lot of AI-agent workflows fail. They substitute explanation for proof. I want both, but if I have to choose, proof matters more.

The fourth filter: how reversible is the action?

The easiest production mutations to trust are the ones with:

explicit backups
narrow scope
rollback instructions
separate staging or test path

The hardest ones are the ones where:

the live system is the only place to validate behavior
the previous state was not captured
rollback depends on memory or luck
the action affects several systems at once

That is why I am much happier letting an agent update a test DaemonSet than directly change the only public hostPort DaemonSet serving real traffic.

The fifth filter: can the task be serialized?

One subtle production risk with AI agents is concurrency.

An agent can work very fast and in parallel, but some systems should not be mutated in parallel even if parallelism is technically possible. I care about that especially for:

live browser editors
production registry maintenance
ad server and video player config changes on the same object
migration flows where one save can overwrite another

So another rule I use is: the agent is allowed to move fast only where the workflow is safe to parallelize. Otherwise, the correct behavior is serialization.

What I let agents do comfortably today

I am fairly comfortable letting an agent:

investigate a live issue in read-only mode
prepare a deterministic config change with backups and proof steps
update automation or tests in the repo
generate reports, charts, or artifacts from saved data
perform staged mutations when the rollback and validation path is already documented

That is a lot of useful work. It is also very different from saying "the agent can just operate production."

What I still treat cautiously

I get much more cautious when the task involves:

ambiguous business logic
undocumented production behavior
irreversible live changes
auth/session-sensitive browser flows
destructive cleanup where the real state graph is not fully known

Those are the areas where the agent can still help, but usually as a prepared operator rather than an unsupervised one.

The working rule I come back to

My practical rule is this:

> An AI agent is allowed to touch production only when the task is documented well enough, bounded enough, and verifiable enough that the agent is not replacing judgment — it is executing a judgment-backed workflow.

That distinction matters a lot.

The goal is not to create a magical autonomous operator. The goal is to turn documented operational knowledge into a faster, more repeatable execution path without lowering the safety bar.

Final thought

The most dangerous mistake with AI agents is not overestimating their code skills. It is underestimating how much production safety depends on context, reversibility, and proof.

That is why my trust model is not based on whether the agent "feels smart." It is based on whether the workflow itself deserves automation.

If the system is well-documented, the action is bounded, the evidence is explicit, and rollback is real, I am happy to let the agent do a lot. If those things are missing, the right answer is not "trust the model more." The right answer is to improve the workflow first.

Why I Log Context Before I Touch Anything Live

2026-05-08T12:17:00.001+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This is part of my Kubernetes-on-Hetzner-and-operations series on MisLinux. It is not a vendor tutorial. It is a working rule I adopted after too many live investigations started with a terminal and ended with fuzzy memory.

The mistake is easy to make:

something looks wrong
I jump into the shell
I start reading logs, manifests, dashboards, or browser state
two hours later I remember the fix, but not the full reasoning path

That is exactly how the same class of incident keeps returning in slightly different forms.

What I write down before I start

I do not try to write a novel. I write three things:

Field	What I want
Context	What is happening right now, and what changed recently
Intent	What I am trying to prove or change
Rationale	Why this path is safer than the alternatives

That is the smallest version of a change log that still survives a bad week.

Why I do this even for read-only work

People often treat documentation as a mutation-only habit. I think that is backwards.

Read-only investigations still shape later decisions:

which graphs I trusted
which logs were misleading
which browser path was blocked by auth or anti-bot checks
which saved artifact became the source of truth later

If I do not log that context while the investigation is fresh, the next run starts from folklore.

The exact failure pattern I am trying to avoid

The failure is not "I forgot everything." It is more subtle:

I remember the conclusion
I forget the rejected paths
I forget the external constraint
I forget which proof actually convinced me

That is dangerous because a future me can look at the final state and decide some "unnecessary" workaround should be removed, when in reality it was the part keeping the system safe.

My practical template

This is the kind of note I want near the work:

Context: public traffic looked down on one origin, but the service was partially balanced elsewhere.
Intent: verify whether this is a real global drop or only a per-origin distribution change.
Rationale: checking the summed public path first avoids "fixing" a healthy balance event as if it were an outage.

That template works for code, dashboards, browser automation, one-off reports, and production runbooks.

A small diagram of the decision path

The point of that flow is simple: I want the note *before* I accumulate terminal tabs, ad hoc commands, and screenshots I will not be able to explain later.

The command trail I usually keep

I do not dump every command. I keep the commands that define the reasoning path:

# 1. Confirm the public symptom
curl -I https://example-service.invalid/health

# 2. Check whether the issue is global or isolated
kubectl get pods -n example-namespace -o wide

# 3. Preserve the before-state artifact
kubectl get deployment example -n example-namespace -o yaml > before.yaml

# 4. Check the data source behind the alert
python3 inspect-metric-source.py --metric service_request_rate

This is enough to reconstruct the investigation later without pretending the shell history is documentation.

Failure case: when I skip the note

When I skip the note, the same bad pattern appears:

the artifact folder contains files but not their meaning
the dashboard screenshot exists but not why it mattered
the final fix is remembered as "obvious"
the next operator repeats the discarded approach first

That is how harmless-looking cleanup becomes accidental regression.

What this changes when AI agents are involved

AI agents amplify the value of explicit context because they can move faster than humans through the same investigation surface.

That is useful only if the task is bounded clearly enough that the agent is not inventing the contract while it works.

Without a context note:

the agent may optimize the wrong thing
a saved artifact may be treated as authoritative when it was only a probe
a fallback path may be removed because it looks redundant

With a context note, the agent is executing inside a known frame.

A simple operator checklist

Before touching a live-adjacent system, I now want:

one sentence for the symptom
one sentence for the intended outcome
one sentence for why this path is the least risky
one artifact that captures the before-state

That is a very low bar, but it removes a surprising amount of confusion.

What I'd do differently now

I used to think I needed "full documentation" before any of this would help. I no longer believe that.

What I would do differently now is start with a tiny Context / Intent / Rationale note first, then let the richer artifacts accumulate around it. Small, explicit notes beat larger undocumented piles of output every time.

The Hidden Maintenance Cost of a Small Kubernetes Cluster

2026-05-05T09:53:00.003+02:00

This article is part 15 of my MisLinux series on Kubernetes on Hetzner. It reflects my own operating experience and cost tradeoffs, not a vendor benchmark. I am not affiliated with Hetzner or any other vendor mentioned here.

When people compare the cost of a small Kubernetes cluster against a managed-cloud alternative, they usually compare invoices first. That is sensible, but incomplete.

The invoice matters. It just is not the whole bill.

There is a second cost layer that shows up in operator time, debugging attention, and the number of low-level details you have to keep honest as the cluster ages. That is the hidden maintenance cost, and it is one of the main reasons a small cluster can be either a smart decision or a bad bargain depending on who is running it.

The hidden cost is not "Kubernetes is hard"

I do not find the phrase "Kubernetes is hard" especially useful. The more practical truth is:

> Kubernetes is a multiplier on the quality of your operational habits.

If your cluster is small and focused, the visible infrastructure bill can be very attractive. What grows in parallel is the maintenance surface around it:

image publishing discipline
registry storage hygiene
node disk pressure and log growth
monitoring retention tuning
backup and restore confidence
failover preparation
architecture-specific build handling

None of these are shocking individually. Together, they become a real recurring cost center.

What the invoice misses

On the invoice, I can usually see:

compute
volumes
IPv4 addresses
backups
bandwidth when relevant

What the invoice does not show is the human maintenance needed to keep the cluster from drifting into fragility.

That hidden work often includes:

Hidden cost area	What it really means
Registry maintenance	not just storing images, but protecting pullability, multi-arch correctness, and safe cleanup
Monitoring maintenance	keeping alerts meaningful, storage right-sized, and exporter scope honest
Node hygiene	log routing, image GC, ephemeral storage checks, and "why is this disk filling?" investigations
Recovery readiness	validating snapshots, standby paths, credentials, and DNS cutover steps before a crisis
Automation safety	making sure the things that save time do not create silent production risk

This is why a cheap cluster can still be expensive if the operating discipline is weak.

Why small clusters are especially tricky

A large organization often spreads these responsibilities across roles. A small cluster usually does not get that luxury.

The same person or small team ends up owning:

platform design
incident response
registry maintenance
monitoring hygiene
build troubleshooting
application rollout validation

That concentration can be efficient. It can also mean that every unresolved low-level maintenance issue becomes future debt for the same operator.

The maintenance that surprised me most

The areas that changed my thinking the most were not the ones that felt dramatic initially.

Registry correctness

I expected the registry to need disk space and backup attention. What surprised me was how much care is needed around tag semantics, cleanup heuristics, and multi-architecture manifest safety. That is not just storage administration. It is deployment correctness.

Monitoring retention

I expected Prometheus and Grafana to need resources. What surprised me was how quickly retention strategy becomes a design problem rather than a configuration detail once you care about restart behavior, WAL replay, and realistic history windows.

Node disk pressure

I expected workloads to use disk. What surprised me was how much slow, indirect growth comes from side effects:

unrotated logs
stale image layers
abandoned temp files
orphaned overlay snapshots

Those are the kinds of costs that do not show up in architectural diagrams but absolutely show up in late-night debugging.

Why this does not invalidate the small-cluster choice

The hidden maintenance cost is real, but it does not automatically mean "do not run the cluster yourself."

It means you need to compare two honest totals:

vendor invoice plus managed-service convenience
lower infrastructure invoice plus self-managed maintenance burden

For some workloads, the second option is still clearly better. That has often been my experience. But only if the operator is willing to count the maintenance burden as part of the decision instead of pretending the invoice tells the whole story.

The question I ask now before adding anything

Whenever I add infrastructure to a small cluster, I ask:

> What new maintenance loop does this create?

Not "can I deploy it?" Not even "is it useful?"

The most important question is whether it creates a new recurring responsibility around:

cleanup
rotation
recovery
monitoring
correctness validation

That question helps me avoid systems that look cheap to start and expensive to keep honest.

What I think small-cluster operators should optimize for

I do not think the right goal is "run as much as possible yourself." I think the right goal is:

keep the architecture understandable
automate the repeatable pain
document the non-obvious failure modes
avoid adding components whose maintenance cost is larger than the problem they solve

That is what makes a small Kubernetes cluster sustainable rather than merely affordable.

Final thought

The hidden maintenance cost of a small Kubernetes cluster is not a reason to dismiss the model. It is a reason to evaluate it honestly.

If you only compare the provider invoice, you will underestimate the real price. If you only compare the maintenance burden, you will miss how much control and cost-efficiency a small cluster can still offer.

The useful middle ground is to treat maintenance time as part of the platform budget. Once I started doing that, my infrastructure decisions got less romantic and more reliable.

Why 'Latest' Is Not a Deployment Strategy in a Self-Hosted Registry

2026-05-01T17:41:00.005+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 20, 2026

This article is part 14 of my MisLinux series on Kubernetes on Hetzner. It reflects my own production experience with a self-hosted registry. I am not affiliated with Docker or any vendor mentioned here.

There is a version of latest that is harmless: the local development convenience tag you use while building or testing. That is not the version I am talking about.

The version I mean is latest as an operational contract in a real self-hosted registry, where storage pressure, cleanup policy, multi-architecture manifests, cached images, and historical provenance all matter.

In that world, latest is not a deployment strategy. It is a label sitting on top of a set of assumptions. If those assumptions are weak, the label becomes misleading very quickly.

What `latest` hides

The problem with latest is not that it points to a recent image. The problem is that it hides the answer to the question operators actually need:

> Which exact image, built from which exact inputs, is this workload supposed to be running?

That matters even more in a self-hosted registry because the registry itself is part of your operational surface. A tag is not an abstract truth. It is a mutable pointer living inside a system you also maintain.

When I depend too heavily on latest, I lose clarity around:

provenance
promotion order
rollback path
multi-arch descriptor protection
what is actually safe to clean up

Why self-hosting makes this worse

Managed registries already make tag discipline important. Self-hosting raises the stakes because the registry can have its own incidents:

storage pressure
cleanup mistakes
missing child manifests under multi-arch tags
cached-vs-pullable mismatches
broken rebuilds that do not become visible until the next node restart

If all of that sits behind a single mutable latest, the operator story gets weak fast.

What I use `latest` for now

I still use latest, but I treat it as a convenience pointer, not as the authoritative identity of the image.

The authoritative identity is one of:

an immutable commit-derived tag
a specific digest
a clearly named staged/runtime promotion tag

latest can exist on top of that, but it should not be the only thing you know.

Why immutable tags matter more than pretty tags

In my environment, some build tags are ugly. They encode the source commits that went into the image. Operationally, that ugliness is a feature.

An immutable provenance tag answers questions latest cannot:

what source state produced this image?
is this the same thing I built yesterday or just something also called latest?
can I rebuild or compare it safely?
is this the tag I should preserve because a live workload still depends on its digest?

Pretty tags are easier to read. Immutable tags are easier to trust.

The promotion model I find safer

I prefer a layered tagging model:

Tag type	Role
Immutable build tag	Exact provenance of one build
Stage/runtime tag	Operational promotion step with a concrete meaning
`latest`	Convenience alias, never the only truth

That structure lets me separate build identity from human-friendly promotion state.

It also makes rollback better. If a runtime tag is bad, I can see what it used to point to. If latest is the only visible identity, rollback becomes a memory exercise.

Why this matters even when Kubernetes pins digests

It is fair to say: if workloads are pinned by digest, why care about latest at all?

Because the operator still has to do all of this:

inspect what new builds produced
compare live digests to current tags
rebuild missing pull paths
clean up old manifests safely
explain to another human what "the right image" means

Digest pinning helps. It does not replace a sane promotion vocabulary.

The cleanup angle most people miss

The registry cleanup story is another reason latest is not enough.

If a cleanup policy says "keep latest and the newest N tags," that sounds reasonable until you realize:

latest may point to a multi-arch manifest list
child manifests may still need direct protection
live workloads may still be running older digests
the immutable provenance tag may be the only safe direct reference left

That is why I prefer cleanup logic based on reachability and deployment usage instead of only on visible tag names.

What changed in how I deploy

I think of image deployment in three questions now:

what exact immutable build did I produce?
what exact build did I promote to the intended runtime tag?
what exact digest is the workload actually using?

If I cannot answer all three, then I am not really operating the image path deliberately. I am mostly trusting conventions to hold.

Final thought

latest is convenient, and convenience has its place. But convenience is not the same thing as a deployment strategy.

A deployment strategy needs provenance, promotion, rollback, and cleanup safety. In a self-hosted registry, those requirements are too important to hide behind one mutable tag and a hopeful mental model.

That is why I still keep latest around, but I no longer let it pretend to be more than it is.

My Cold-Standby Disaster Recovery Plan for a Single Critical Service

2026-04-29T12:41:00.004+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 20, 2026

This article is part 13 of my MisLinux series on Kubernetes on Hetzner. It is based on a real production service and the recovery planning around it. I am not affiliated with Hetzner, Cloudflare, Netcup, or any vendor mentioned here.

If you run a small platform, "disaster recovery" can become a vague comfort word very quickly. It is easy to say you can rebuild from source, recreate the cluster, restore the database, and recover the service. It is much harder to define what you will actually do if the main environment is gone or unreachable and a critical public endpoint needs to come back fast.

That difference is why I ended up designing a very explicit cold-standby plan for one service instead of pretending the whole platform needed full hot-failover sophistication.

Why I focused on one service first

Not every service deserves the same recovery model.

Some services are annoying to lose. Some are expensive to lose. A smaller number are central enough that their absence immediately becomes a business problem.

For me, the right first DR target was a single critical serving path. That let me answer the recovery question honestly:

what exact artifact does this service need to start?
where can I store that artifact outside the cluster?
how would I run the same runtime elsewhere if Hetzner were unavailable?
what manual step am I willing to take for cutover?

That is a much healthier question than "how do I make the whole cluster magically highly available?"

Why I chose cold standby instead of active-active

I did not want to build a complicated active-active or automatic failover design for a small environment unless the business case clearly justified it.

The cost of more advanced failover is not just extra infrastructure. It is operational ambiguity:

which side is authoritative?
how do you keep config perfectly in sync?
what happens during split-brain conditions?
how do you test the failover path without risking production?

For one critical service, a cold standby with good artifacts and a documented cutover path was a better fit.

My design goal became:

> keep one externally runnable copy of the service ready enough that I can start it from a known-good snapshot and then cut DNS over manually if the primary environment is gone.

What the standby actually needs

Once I stripped away the vague language, the required ingredients were simpler than I expected:

the exact runtime image or equivalent runnable artifact
the last known-good serving configuration
a minimal host or VM where that runtime can start
a TLS front door
a manual DNS cutover path

That is the core of the plan. Everything else is support structure.

Why the config snapshot matters so much

The most important insight was that disaster recovery for a service often depends more on the last known-good runtime state than on the source code.

Source code is necessary, but it is not the immediate recovery substrate in a crisis. If the live service depends on a generated configuration that reflects database-backed objects, rules, or mappings, then the practical recovery question is:

> Do I have the actual config the service was serving from?

If the answer is no, then "we can rebuild it from the database" only works if the whole control plane and upstream data path are still available during the incident. That is not a disaster assumption I want to build around.

So I prefer taking validated off-cluster snapshots of the real serving config instead.

Why the cutover stays manual

A small platform can talk itself into dangerous automation very easily.

Automatic failover sounds mature, but it only helps if the automation is more reliable than the operator judgment it replaces. For my environment, I do not think that is true yet.

That is why DNS cutover stays manual.

Manual cutover buys me two things:

I decide based on evidence, not on one possibly misleading health signal
I avoid accidental flapping between origins

In practice, that means a documented runbook is more valuable than a half-trusted automatic failover controller.

What the standby is meant to prove

The cold standby is not meant to be a second production platform in permanent parallel service. It is meant to prove four things:

Proof goal	Why it matters
The service can start elsewhere from the saved config	Prevents DR from depending on cluster-local state or wishful re-generation
The runtime image is available outside the primary pull path	Avoids turning registry or network dependency into the next blocker
TLS and reverse proxying work on the standby host	Makes cutover realistic instead of theoretical
DNS can be pointed at the standby origin intentionally	Turns the recovery path into an operator action, not a design diagram

That is enough for a credible first DR layer on a small platform.

What I do not want from DR

I do not want the disaster-recovery system to become a second fragile platform with its own hidden dependencies.

That means I try to avoid:

rebuilding from source during the incident
assuming private registries are reachable
depending on interactive browser-only recovery steps
distributing the failover state across too many places

The more moving parts the DR path has, the less likely it is to be trustworthy under actual pressure.

The tradeoff I accept

Cold standby is not instant recovery. I accept that.

The real tradeoff is:

slower recovery
in exchange for
a simpler, more explicit, more testable path

For a small team, that is often the right bargain. I would rather have a cold standby that I can explain and rehearse than an elaborate failover story I never fully trust.

Final thought

The most useful change in how I think about DR is this: disaster recovery starts becoming real only when you can name the exact artifact, the exact host, and the exact operator action that would bring the service back.

Until then, it is mostly optimism dressed up as architecture.

For me, the right first step was not platform-wide high availability. It was proving that one critical service could start elsewhere from a real snapshot and be put back behind its public hostname deliberately. That is not the final state. But it is a recovery plan I actually believe.

How I Built an External Watchdog So My Cluster Is Monitored From the Outside Too

2026-04-27T09:41:00.004+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 20, 2026

This article is part 12 of my MisLinux series on Kubernetes on Hetzner. It is based on a real operational setup I use. I am not affiliated with Synology, Grafana Labs, Hetzner, or any vendor mentioned here.

When people say "the cluster is monitored," they often mean "the cluster is monitoring itself." That is useful, but it is not the same thing.

An in-cluster monitoring stack can tell you a great deal about node pressure, pod restarts, internal health endpoints, and application metrics. What it cannot prove by itself is that the cluster is still reachable from the outside world under the same conditions your users see.

That gap matters more than it sounds.

Why internal monitoring is not enough on its own

If all of your monitoring, alerting, and dashboards live inside the system being monitored, you inherit its blind spots:

a networking break can isolate the cluster and the monitoring at the same time
DNS or ingress issues can make public services fail while internal health checks still pass
a bad change can break external reachability without breaking pod-level liveness

In other words, the cluster can be healthy in its own opinion while being unavailable in the way that actually matters.

That is why I added a small external watchdog running outside the cluster.

What I wanted the watchdog to do

I did not want a second full observability stack. I wanted a narrow, opinionated external view:

can the public URL resolve?
does TLS still work?
does the service answer quickly enough?
if it fails, can I get a notification even if the cluster itself is confused?

That is a very different job from Prometheus inside the cluster. The external watchdog is not for rich diagnosis. It is for independent truth.

The design I ended up with

I run a lightweight watchdog process outside the cluster on a small always-on system. Its responsibilities are intentionally boring:

poll the critical public endpoints
enforce explicit connection and response timeouts
send a Telegram message when a check fails
emit a periodic heartbeat back toward the cluster so the cluster can notice if the watchdog itself disappears

That last step matters. An external watchdog that silently dies is not much of a watchdog.

So the design becomes two-way:

Direction	What it proves
External watchdog -> public services	The internet-facing path still works from outside the cluster
Cluster monitoring -> watchdog heartbeat	The watchdog itself is still alive and checking in

Together, those signals are much more trustworthy than either one alone.

What the watchdog checks

I keep the checks simple and explicit:

public service health endpoints or equivalent minimal URLs
DNS resolution for important hostnames
TLS validity and impending certificate expiry
response-time budget, not just up/down state

This is deliberately narrower than full application monitoring. If I want latency histograms or per-route error rates, Prometheus is the right place. If I want to know whether the service is still reachable from outside the cluster without trusting the cluster to tell me, the watchdog is the right place.

Why I like Telegram for this role

I wanted something easy to receive on a phone, easy to automate, and not dependent on setting up yet another mail or paging system. Telegram is a good fit for that:

straightforward bot API
fast delivery
readable messages on mobile
low friction for a small operational setup

The important part is not the specific chat platform. The important part is that the alert message explains itself clearly enough that I do not have to mentally reconstruct the system at the moment I receive it.

What makes the alerts useful instead of noisy

The external watchdog helped me reinforce a monitoring principle I already believed: alerts should explain what the failure probably means and what to check first.

A useful alert says something like:

which public service failed
how it failed (timeout, DNS failure, bad status, TLS issue)
what the first operator check should be

That is much better than a generic "service down" message, especially when the point of the watchdog is to be the first signal you receive.

Why I did not replace the in-cluster monitoring

The external watchdog is not a substitute for Prometheus, Grafana, node metrics, or per-service health instrumentation. It is a complement.

Prometheus answers questions like:

which node is filling up?
which pod restarted?
did the request rate collapse gradually or instantly?
is this a localized workload issue or a broader cluster issue?

The watchdog answers a much simpler question:

if I were outside the cluster right now, would the service look alive?

That division of labor is exactly why the pairing works.

The real value during incidents

The biggest benefit is not prettier monitoring architecture. It is incident classification.

When the external watchdog reports a failure and the internal cluster metrics are also sick, I know I am dealing with something broad.

When the external watchdog reports a failure but the internal metrics still look normal, I immediately suspect:

DNS
ingress
certificate
external routing
origin reachability from outside the cluster

That narrows the response path quickly.

Likewise, if the cluster starts alerting that the external watchdog heartbeat has stopped, I know the monitoring blind spot may have moved to the watchdog side rather than the service side.

What I would recommend to small-cluster operators

If you self-manage a small production cluster, my recommendation is not "build a second observability platform." It is "give yourself one independent external truth source."

The minimum useful version can be very small:

a handful of critical URL checks
explicit timeouts
one alert channel
one heartbeat back into the cluster

That is enough to cover a whole class of failures that purely internal monitoring does not see clearly.

Final thought

The reason I like an external watchdog is not that it is sophisticated. It is that it is independent.

Operationally, independence is what gives the signal value. A cluster that says "I am healthy" is useful information. A cluster that is independently reachable from the outside is better information. When both views agree, I trust them more. When they disagree, I know where to investigate next.

That is the real job of the watchdog: not to replace the internal monitoring stack, but to stop it from being the only voice in the room.

The Problem With hostPort DaemonSets in Production, and How I Test Them Safely

2026-04-24T11:17:00.004+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 20, 2026

This article is part 11 of my MisLinux series on Kubernetes on Hetzner. It reflects my own operational experience. I am not affiliated with Kubernetes, CNCF, or any vendor mentioned here.

There are Kubernetes patterns that are perfectly reasonable until the day they have to be updated under pressure. A public-facing DaemonSet with hostPort is one of them.

I use this kind of pattern for a latency-sensitive service that I want reachable directly on each selected node. That is a valid design choice. The mistake would be pretending it rolls like an ordinary Deployment behind a Service. It does not.

Why `hostPort` changes the rollout story

When a pod binds a hostPort, it is competing for a real port on the node itself. That means a replacement pod cannot simply appear next to the old one if the old one is still holding the port.

That single detail collapses a lot of the usual Kubernetes comfort:

no meaningful same-node blue/green on the same port
no maxSurge rescue if only one node serves that traffic path
no assumption that a new pod can come up fully before the old one disappears

If you only have one node serving that workload class, a bad rollout is not a degraded rollout. It is an outage.

Why this is easy to underestimate

Kubernetes is so good at making ordinary workloads boring that it is easy to project the same confidence onto special cases.

With a normal Deployment behind a Service, the rollout safety story is familiar:

new pods come up
readiness gates traffic
old pods drain away

With a hostPort DaemonSet on one eligible node, the real story is closer to this:

old pod is removed or becomes unavailable
new pod has to win the host port
new pod still has to complete its own initialization
if anything in that chain is wrong, public traffic drops immediately

That is not a reason to avoid the pattern entirely. It is a reason to stop lying to yourself about the rollout risk.

The specific failure mode I care about most

The worst version of this pattern is not "the pod crashes instantly." That is obvious. The more dangerous case is:

the YAML looks correct
the image tag looks correct
the process starts
but the service is not actually ready to answer the public request path yet

That can happen because of a slow startup sequence, a missing generated config, a probe that fires too early, or a runtime process that behaves differently after reload than after cold boot.

In other words, the main risk is not syntax. It is false confidence.

The test pattern I trust now

I do not apply new behavior directly to the public DaemonSet anymore if I can avoid it. I first prove the runtime path in a parallel test DaemonSet that stays away from the live ports.

The exact details depend on the workload, but the pattern is:

Step	Purpose
Keep the public DaemonSet unchanged	Avoid accidental real traffic impact while validating the new image or command path
Run a parallel test DaemonSet or equivalent pod on alternate ports	Prove startup, health checks, generated config, and runtime behavior without binding the production port
Check the real readiness surface, not only container liveness	Make sure the service answers the same path the live system depends on
Promote only after the parallel proof is clean	Treat the public rollout as the final step, not the experiment

This is less elegant than a normal Deployment rollout, but it is honest about the shape of the risk.

What I specifically verify before promotion

My minimum checklist for a hostPort DaemonSet change now includes:

image pulls correctly on the target architecture
startup script finishes the full initialization path
generated config files actually exist and are valid
readiness only turns green after the real serving path works
process supervision survives reloads, not only cold start
the test instance answers a real request, not just a dummy socket check

If I cannot prove those on the parallel test path, the production rollout is not ready.

Why readiness probes matter even more here

I used to think of readiness primarily as a traffic-quality control. In this pattern, it is also a self-defense mechanism against human optimism.

If a service takes 90 to 180 seconds to become truly ready because it generates config from a database and only then launches the serving process, a 15-second readiness probe is not "aggressive." It is wrong.

That kind of mismatch is exactly how a Kubernetes manifest can look disciplined while still causing a live outage on rollout.

The broader point is that readiness has to model the real application timeline, not the timeline you wish the application had.

When I still think this pattern is worth using

Even with the rollout risk, I still think hostPort DaemonSets can be the right answer when:

each selected node is meant to serve traffic directly
you want to avoid an extra hop for the workload
node-local identity actually matters
the service is operationally narrow and worth the explicit handling

But the price of that pattern is operational ceremony. If you choose it, you are choosing:

stricter promotion discipline
stronger rollback expectations
more careful probe tuning
explicit non-production proof before live mutation

That is the actual cost, and it should be counted up front.

What I would not do again

I would not treat "the manifest change looks small" as a reason to skip the test path.

Small changes can break this kind of workload in large ways:

a slightly different startup command
a different image tag
a config generator timing change
a probe that turns green too early

The blast radius comes from the serving topology, not from how many lines changed.

Final thought

hostPort DaemonSets are not bad. They are just less forgiving than the default Kubernetes workload patterns people get used to.

If you use one for a public service, the safe mental model is not "Kubernetes will roll this safely for me." The safe mental model is "Kubernetes will do exactly what I asked, so I need to make sure I asked for a promotion flow that deserves production traffic."

That is why I now treat the test DaemonSet as part of the design, not a nice-to-have around the design.

How Docker Registry Garbage Collection Broke My Multi-Arch Images

2026-04-22T09:05:00.002+02:00

This article is part 10 of my MisLinux series on Kubernetes on Hetzner. It is based on a real failure in a self-hosted production registry I operate. I am not affiliated with Docker, CNCF, or any vendor mentioned here.

Multi-architecture images are one of those things that feel clean in theory and treacherous in practice.

In theory, you publish one image tag, Kubernetes pulls the right platform-specific manifest for each node, and everything is elegant. In practice, that elegance depends on the registry handling manifest lists, referenced sub-manifests, and garbage collection exactly the way you think it does.

Mine did not.

The incident in one sentence

I ran garbage collection on a self-hosted Docker registry and ended up with a perfectly normal-looking multi-arch tag whose platform-specific image manifests had been swept out from under it.

That is the kind of breakage that destroys confidence because it does not look broken at first glance. The tag can still exist. The manifest list can still exist. But a real pull on one architecture fails because the child manifest or referenced blobs are gone.

Why this was not just "I deleted the wrong thing"

The important nuance is that this was not simply an operator typo. It was a mismatch between what I assumed garbage collection would protect and what the registry implementation actually considered reachable.

I had a manifest list like this in conceptual terms:

latest
linux/amd64 child manifest
linux/arm64 child manifest

The mistake was assuming the top-level tag was enough protection for the architecture-specific manifests beneath it. That assumption is intuitive. It is also operationally dangerous if your registry version or GC behavior does not traverse those references the way you expect.

In my case, GC reclaimed the platform-specific manifests even though the higher-level multi-arch tag still existed.

Why this hurt more than a normal image mistake

If a single-architecture tag breaks, the blast radius is often obvious. The affected workload on the affected architecture fails quickly. You investigate the exact tag.

Multi-arch breakage is more deceptive because different nodes can see different realities:

one architecture may still pull successfully because it already has the image cached
another architecture may fail on the next real pull
a running deployment may look healthy until a rollout, restart, or node replacement happens

That delay is what makes the failure expensive. The registry problem is created earlier than it is discovered.

The cleanup assumption I had to unlearn

I had been treating some older immutable build tags too casually because of their naming style. In my environment, the Rocker-derived tags look like this:

0_<commit>_<commit>_<timestamp>_--no-wait

At a glance, they look disposable. They are not.

Those tags encode source-commit identity for the build inputs. The stale --no-wait suffix is historical baggage from an old Satis-related flag, not evidence that the tag is transient. Once I re-centered on that fact, the cleanup policy had to change. The question stopped being "which ugly tags look safe to prune?" and became "which tags are still protecting manifests that matter?"

What I changed in the cleanup rules

My cleanup rule is stricter now:

never assume a weird-looking immutable tag is disposable
never assume a latest manifest list fully protects its child manifests
keep direct architecture-specific tags for manifests that back a live multi-arch tag
prefer reachability- and deployment-aware cleanup over name-based cleanup

In practice, that means a safe cleanup process should understand at least three things:

Question	Why it matters
Is this tag the current `latest`?	`latest` still matters operationally in many environments, even when I prefer digest pinning
Is this digest currently referenced by a live Deployment, DaemonSet, CronJob, or Job?	Deleting its tags may not break the running pod immediately, but it removes the safe pull path for restart or scale-out
Is this immutable tag the only direct reference protecting a child manifest under a multi-arch tag?	Removing it can silently corrupt the apparent safety of the multi-arch tag

This is more complex than "keep the N newest tags," but complexity is better than pretending registry cleanup is harmless.

Why self-hosting a registry changes the standard advice

On a managed registry, a lot of this implementation detail disappears behind a service boundary. On a self-hosted registry, especially one that has already seen aggressive cleanup or storage pressure, you own the consequences directly:

disk pressure
GC semantics
manifest reachability assumptions
downtime risk from re-pulls
differences between cached images and actually pullable images

That does not make self-hosting wrong. It just means registry maintenance is not janitorial work. It is production reliability work.

The verification step I now treat as mandatory

The most important new habit is manifest inspection before and after cleanup.

I want to know:

what the top-level tag points to
which architecture descriptors exist under it
whether those per-arch manifests also have direct tags
whether the live cluster is still referencing an older digest

If I cannot answer those questions quickly, I do not trust the cleanup operation.

The broader lesson is that registries are not just blob stores. They are graph stores with operational consequences. When GC runs, it is traversing a reference graph. If your mental model of that graph is wrong, the cleanup can be "successful" and still destroy a production-safe pull path.

Why this changed my view of image hygiene

Before this incident, I thought of registry cleanup mostly as a disk-usage discipline problem. Afterward, I think of it as a correctness problem first and a storage problem second.

Of course the registry needs cleanup. A self-hosted registry cannot grow forever. But correctness has to come first:

live tags must remain pullable
immutable provenance tags must not be treated as trash
multi-arch child manifests must stay directly protected
cleanup heuristics must reflect deployment reality, not naming aesthetics

If that means the registry stays a little fatter until the logic is right, that is a better trade than recovering from a broken production image path.

Final thought

The phrase "garbage collection" sounds like maintenance. In a production registry, it is really a graph-deletion operation with platform-specific consequences.

That is the real lesson I took from this failure: a multi-arch tag is only as safe as the child manifests and tags that still exist behind it. If your cleanup policy cannot prove those relationships before it deletes anything, it is not conservative enough yet.

If you want the broader platform context for why I care so much about mixed ARM64 and AMD64 pull safety, read my earlier post on handling multi-architecture images in a mixed-architecture Kubernetes cluster. This post is the production failure mode that taught me what that architecture really costs in maintenance discipline.

Why My Prometheus History Suddenly Disappeared, and What I Changed Afterward

2026-04-20T08:41:00.004+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 20, 2026

This article is part 9 of my MisLinux series on Kubernetes on Hetzner. Everything here comes from a real cluster I operate. I am not affiliated with or sponsored by Hetzner, Prometheus, Grafana Labs, or any other vendor mentioned in this post.

When monitoring feels stable for a while, it becomes easy to think of it as solved infrastructure. That is exactly when it tends to surprise you.

I had a moment recently where Grafana suddenly stopped showing the historical Prometheus data I expected. The dashboards still loaded. Fresh data was still arriving. But the multi-day view that had been visible earlier was effectively gone. What made this more dangerous was that nothing had "failed" in the obvious sense: Prometheus was up, Grafana was up, and queries still returned current values.

That kind of failure is worse than a full outage. A full outage is noisy. Silent history loss makes you think the system is working while the evidence you need for trend analysis, incident review, or storage growth comparisons has already disappeared.

What I saw first

The symptom was simple: a Grafana dashboard that had shown several days of data earlier in the morning suddenly only had fresh points. It would have been easy to blame Grafana, because Grafana is where the problem became visible. But Grafana was just the messenger.

The first useful clue was storage. The Prometheus PVC had dropped from a level that looked normal for a populated time-series database to something dramatically smaller. That told me the data had not merely become hard to query. It had actually been removed from the TSDB.

The second clue was timing. This happened after a Prometheus recovery path that involved WAL replay and compaction pressure. In other words, this was not random data drift. It was a storage and retention interaction.

The actual cause

The root cause was the combination of two settings that looked individually reasonable:

a small on-disk size cap for the Prometheus TSDB
a retention target that implied more history than that size cap could safely hold after replay and compaction

In practice, Prometheus recovered, replayed WAL state, rebuilt blocks, and then immediately had to delete older blocks because the effective database size was now above the configured limit. From Prometheus's point of view, this was correct behavior. From an operator's point of view, it was a trap.

The lesson is that retention by time and retention by size are not interchangeable. If you say "keep N days" but the size budget cannot physically hold N days of your scrape volume, the size cap wins. That is especially easy to underestimate on a cluster where monitoring scope gradually expands: more exporters, more labels, more synthetic metrics, more per-node detail, more per-service probes.

Why this is easy to miss on a small cluster

On a small Kubernetes cluster, it is tempting to be aggressive with Prometheus storage because you do not want the monitoring stack competing with production workloads. That instinct is correct. The mistake is assuming a low storage cap is safe just because the cluster itself is small.

Small clusters often collect a surprisingly wide mix of data:

host metrics from every node
Kubernetes object state
ingress and application health checks
Pushgateway-fed metrics from cron jobs and watchdogs
custom operational counters that answer "did this thing actually happen?"

Individually, none of these feel large. Together, they create a TSDB shape that grows in bursts and compacts in ways that are hard to reason about from intuition alone.

What I changed afterward

I changed two things.

First, I stopped treating the storage limit as a rough guess. I treated it as a capacity contract. If I wanted multi-day history for the metrics that matter operationally, the size cap had to reflect real ingestion and compaction behavior, not a convenient round number.

Second, I separated the question of how much history I want from how much history I can afford. Those are related, but they are not the same design decision.

My post-incident rule now looks like this:

keep enough retention to support real incident comparison, not aspirational observability
size the TSDB for the real metric set plus recovery headroom
verify what happens after restart and replay, not only during steady-state uptime
watch disk trends for Prometheus specifically instead of assuming Grafana will expose the problem early enough

The operational mistake I would avoid next time

The mistake was not "using a small disk." The mistake was assuming a one-time retention choice would stay valid while the monitoring system kept evolving.

Prometheus configuration often grows in operationally healthy ways:

you add a better exporter
you add a new per-service alert
you add a custom metric for a problem that used to be invisible
you add a standby or external watchdog path

Each one makes the monitoring better. Each one also changes the storage profile. If you do not revisit the retention budget after those improvements, you eventually get a misleadingly "healthy" monitoring stack that cannot retain the history your dashboards imply.

What I now consider the minimum safe review loop

If I were setting this up again on a small cluster, I would review these four things together instead of separately:

Check	Why it matters
TSDB size cap vs. actual usage	Prevents size-based retention from silently defeating the intended history window
WAL replay behavior after restart	Reveals whether recovery itself can trigger aggressive block deletion
Per-target scrape scope	Keeps metric growth tied to actual operational value
Dashboard time ranges people rely on	Prevents the UI from implying more retained history than Prometheus can really provide

This is not glamorous work, but it is the kind of detail that decides whether your monitoring helps during an incident or merely looks professional between incidents.

What this changed in how I think about monitoring

Before this, I already believed a small cluster should run lean monitoring. After this, I think lean monitoring has to include lean retention design, not only lean scrape configuration.

It is easy to spend time tuning exporters and alert rules while leaving storage behavior as a default plus one arbitrary disk number. That is not enough. The retention model is part of the monitoring design, not a background implementation detail.

That is also why I prefer practical monitoring over "collect everything." The more honest question is not "can I scrape this?" It is "will I still trust this system after restart, replay, compaction, and a week of real production behavior?"

Final thought

Losing Prometheus history was frustrating, but it was also clarifying. It forced me to stop thinking of monitoring data as something that passively accumulates and start treating it like any other production storage system with real capacity and lifecycle constraints.

If you run your own small cluster, my recommendation is simple: test the recovery path, inspect the size-based retention interaction, and do not wait for a missing graph to tell you your monitoring history was more fragile than you thought.

If you want the broader monitoring architecture around this story, read my earlier post on lean monitoring for a small Kubernetes cluster on Hetzner. This article is the more specific follow-up: what happened once that monitoring stack had to survive real operational pressure rather than just look correct on day one.

Handling Multi-Architecture Images in a Kubernetes Cluster Where Nodes Run Different CPU Architectures

2026-04-07T14:38:00.002+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 7, 2026

This article is part of my MisLinux series on building and operating a small Kubernetes cluster on Hetzner. It reflects my own hands-on experience and opinions, and I am not affiliated with, sponsored by, or endorsed by Hetzner, Docker, or any tool mentioned here.

When I started building my Kubernetes cluster on Hetzner, I chose the ARM64-based cax instances because they offered a better price-to-performance ratio than comparable AMD64 options. That seemed straightforward at the time. A few months later, it became complicated when I needed to rebuild container images from my AMD64 development laptop and push them to a private registry used by an all-ARM64 cluster.

This post is about how I handle multi-architecture container images in that environment, including a workflow for pushing a single-architecture update without destroying the other architecture in the registry.

Why architecture matters for container images

A container image is not architecture-neutral. When you build an image on an AMD64 machine with docker build, you produce an AMD64 image. If a Kubernetes node running ARM64 tries to pull that image, the pull fails or the container crashes at startup.

For a homogeneous cluster, this is not a concern. Every node is the same architecture, every image is built for that architecture, and nothing collides. But the moment you mix architectures, or build from a machine that does not match your cluster, you have a problem.

The solution is a multi-architecture manifest. Instead of one image pointing to one set of layers, you create an OCI image index (often called a "fat manifest" or multi-arch manifest) that points to two or more architecture-specific images under the same tag. When a node pulls that tag, the registry serves the right variant for the node's architecture automatically.

How Docker buildx creates multi-arch images

Docker buildx is the standard tool for building multi-architecture images. With a properly configured builder, you can target multiple platforms in a single build command:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag registry.example.com/myimage:latest \
  --push \
  .

This produces an image index at registry.example.com/myimage:latest that contains two descriptors: one for linux/amd64 and one for linux/arm64. Any node pulling that tag gets the right image for its architecture.

There are two ways buildx produces the non-native architecture:

QEMU emulation: slow, but works on any machine. The non-native architecture is emulated in software.
Native cross-compilation: fast, but requires a builder node of the target architecture.

For production-quality builds, QEMU emulation is often too slow to be practical for large images. The right long-term answer is a native builder for each target architecture.

The current setup: AMD64 laptop, ARM64 cluster

In my environment today:

The Kubernetes cluster runs exclusively on Hetzner cax ARM64 nodes.
I build images from an AMD64 development machine.
There is no ARM64 build agent inside the cluster yet.

Building linux/arm64 images from an AMD64 machine via QEMU is feasible for small images but impractically slow for images that compile Rust code or run a full test suite during the build. For those, I need a native ARM64 build.

Until a native ARM64 builder is set up, the pragmatic solution is to build each architecture separately and merge the results.

Building for a single architecture and preserving the other

The scenario I face regularly is this: I have made a change that needs to be deployed now. The linux/arm64 image is already in the registry and working correctly. I want to build a new linux/amd64 image from my laptop and push it without removing the existing linux/arm64 descriptor from the registry.

If I do a naive docker buildx build --platform linux/amd64 --push, the tag is replaced with a single-arch manifest pointing only to AMD64. The ARM64 image is gone from that tag. Any ARM64 node that pulls the tag will fail.

The fix is to use docker buildx imagetools create to compose a new multi-arch index from the individual descriptors. Here is the workflow:

Step 1: Build and push the AMD64 image under a staging tag.

docker buildx build \
  --platform linux/amd64 \
  --tag registry.example.com/myimage:amd64-staging \
  --push \
  .

Step 2: Inspect the current multi-arch manifest to confirm the ARM64 descriptor is still there.

docker buildx imagetools inspect registry.example.com/myimage:latest

This shows the image index with each architecture's digest. If the ARM64 descriptor is present, proceed.

Step 3: Merge the new AMD64 staging image with the existing ARM64 image into the target tag.

docker buildx imagetools create \
  --tag registry.example.com/myimage:latest \
  registry.example.com/myimage:amd64-staging \
  registry.example.com/myimage:latest@sha256:<arm64-digest>

The imagetools create command reads the listed sources and produces a new image index containing all their descriptors. Referencing the ARM64 image by its digest guarantees you get exactly the version you inspected, not whatever might be at the tag at the moment the command runs.

Step 4: Verify the result.

docker buildx imagetools inspect registry.example.com/myimage:latest

Confirm both linux/amd64 and linux/arm64 descriptors are present. Then delete the staging tag to keep the registry clean.

Why the registry itself can cause problems

One thing I learned the hard way is that not every registry handles multi-arch manifests cleanly. In my environment, I run a private registry backed by the standard Docker registry:2 image. Under aggressive garbage collection, OCI image indexes can be partially invalidated even when the blobs they reference still exist.

The symptom is a tag that appears in the registry but returns a manifest parsing error or a missing-layer error when you try to pull it. The root cause is usually that GC removed a blob referenced by the manifest, or that a tag was rewritten in a format the registry did not store cleanly.

A few things that help:

Run garbage collection during low-traffic periods.
Validate multi-arch tags immediately after a push by running docker manifest inspect from a separate machine.
Keep a backup of the digest for each architecture before doing any manifest manipulation. Digests are content-addressed and immutable; you can always reconstruct a tag from them.

The long-term goal: native ARM64 builds in CI

Building ARM64 images from QEMU on an AMD64 laptop is a workaround, not a permanent solution. The right answer is a native ARM64 build node.

The Hetzner cax instances I already use in the cluster are ARM64. A cax21 instance costs roughly a few euros per month and can function as a docker buildx builder node using the remote or kubernetes buildx driver. Once that is in place, a CI pipeline can build both architectures natively and push a proper multi-arch manifest in a single step, without any manual digest tracking.

Until then, the staging-and-merge workflow described above is the practical path for keeping both architectures available under the same tag.

Summary

Container images are architecture-specific; a multi-arch manifest lets one tag serve both ARM64 and AMD64 nodes.
docker buildx build --platform linux/amd64,linux/arm64 builds both at once, but requires either QEMU or a native builder for the non-native architecture.
When updating only one architecture, use docker buildx imagetools create to merge the new descriptor with the preserved descriptor from the other architecture.
Always reference the other architecture by digest, not by tag, during the merge.
Validate the result immediately after pushing; registry GC and format mismatches can leave a tag broken without obvious errors.
The permanent solution is a native builder for each target architecture.

Lean Monitoring for a Small Kubernetes Cluster on Hetzner

2026-04-06T14:18:00.002+02:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 6, 2026

This article is part 8 of my MisLinux series on Kubernetes on Hetzner. All configurations described here are based on my own setup and real operational experience. I am not affiliated with or sponsored by Hetzner, Grafana Labs, or any other tool mentioned.

A cluster that runs without visibility is just a time bomb with a pleasant uptime. After getting the basics working on a small Hetzner k3s cluster, the next practical step is knowing whether the services running on it are actually healthy, and getting a notification before a problem becomes an incident.

This post covers how I built a lean monitoring stack that fits comfortably on modest virtual machines without consuming the CPU or disk budget that the actual workloads need.

What I wanted from monitoring

Before choosing tools I wrote down a short list of practical questions I needed answered without manual inspection:

Is each workload responding? Do the health checks pass?
Are pods crashing or restarting unexpectedly?
Is any node running low on disk space?
Are requests being served, or has traffic dropped to zero?
Did a new node join the cluster correctly?
Is the external watchdog alive and checking in?

I also wanted a notification channel I could use from a phone without installing anything complicated. Telegram works well for this.

The stack: Prometheus, Grafana, and node_exporter

For a small cluster, the standard combination of Prometheus (metrics collection), node_exporter (host-level metrics per node), and Grafana (dashboards and alerting) is hard to beat. It is widely documented, has reasonable resource consumption when tuned carefully, and integrates directly with Kubernetes through service discovery.

Resource discipline

On modest hardware, a few settings matter:

Prometheus scrape interval: I use 60 seconds rather than the default 15 seconds. For a services-focused cluster this is more than adequate. Disk write load drops sharply.

Retention: I keep 15 days of raw data. Beyond that, Grafana's visualization has diminishing returns for the day-to-day questions I actually ask. Prometheus compacts data automatically; I just make sure the retention flag is set explicitly rather than relying on the default.

node_exporter collectors: I disable the collectors I do not use: bcache, drbd, infiniband, and several others that generate irrelevant metrics for a plain virtual machine. The --collector.disable-defaults flag and selective re-enables keep the cardinality low.

Grafana data retention: Grafana itself stores dashboards and alert state but not time-series data. The dashboards are kept in a ConfigMap so they survive pod restarts without a separate persistent volume.

What gets scraped

The standard target set for a k3s cluster:

node_exporter on each node (CPU, memory, disk, network)
k3s itself exposes kubelet and API server metrics on standard ports
Any application that exposes a /metrics endpoint gets added to a ServiceMonitor or static scrape config
A custom Pushgateway for metrics that cannot be scraped directly (batch jobs, external health checks)

For CPU usage per node, node_cpu_seconds_total with rate() over 5 minutes gives a smooth enough signal to detect saturation without false alarms on short spikes.

Dashboards: one unified view

I keep one main dashboard with panels for each concern:

Node health: CPU, memory, disk per node side by side
Pod restarts: a table showing any pod with more than 0 restarts in the last hour
Ingress error rate: HTTP 4xx and 5xx rates from the nginx ingress controller
Health endpoint status: custom panels that poll each service's /health endpoint and show a green/red state
Request rate: the requests-per-second metric from the main production service, shown as a time-series so drops are immediately visible
Bandwidth: inbound and outbound bytes per node, useful for estimating how close you are to the included monthly traffic allocation

I also keep individual per-service dashboards linked from the main one. This keeps the top-level view clean while allowing a drill-down when something looks wrong.

Bandwidth and cost tracking

Hetzner includes a monthly traffic limit per server. Exceeding it has cost implications. I track node_network_transmit_bytes_total on the public interface for each node, compute a monthly rate using Prometheus's increase() function, and display it as a panel alongside a threshold line.

For cost, I pull the daily Hetzner billing data through their API and push it via Pushgateway. This gives a simple bar chart of daily spend, which is useful for catching unexpected charges early.

Alerting: Grafana rules with Telegram delivery

Grafana's built-in alerting can evaluate PromQL expressions and send notifications. I use it instead of the Prometheus Alertmanager to keep the stack simpler.

Telegram is a convenient delivery channel: no email server required, messages arrive on mobile immediately, and the Bot API is straightforward to configure. In Grafana: add a Contact Point of type Telegram, paste the bot token and chat ID, and notifications start flowing.

What alerts I actually run

Keeping the alert list short and meaningful avoids alarm fatigue:

Disk above 80% on any node — gives time to act before a full disk causes silent failures
No requests in the last 30 minutes on any production service that should have traffic — this catches outages that would otherwise go unnoticed overnight
Pod restart count increasing — anything more than 2 restarts in an hour gets flagged
Health endpoint not returning 200 — a simple up/down alert for each public service
External watchdog not checking in — see the next section

Writing alert messages that explain themselves

The most useful discipline I adopted was writing alert messages as if the reader has never seen the system before. Each message includes:

Where it comes from (which cluster, which service)
What it means in plain terms (the service is down / latency is elevated / traffic fell unexpectedly)
What to check first (a specific command, a specific URL, or a specific Kubernetes namespace)

A message that just says "alert firing" is not useful at 2am. A message that says "Production VAST service on the Hetzner cluster is receiving zero requests — likely the nginx ingress pod has restarted. Check: kubectl get pods -n production" is actionable immediately.

The external watchdog: a check from outside the cluster

A monitoring stack that lives entirely inside the cluster it monitors has a structural blind spot: if the cluster loses connectivity, or if a misconfiguration blocks all traffic, the alerting fails silently too.

I run a small Docker container on a home NAS device as an external watchdog. Its job is simple: poll each critical service URL every minute, and send a Telegram message if any check fails or times out. If the NAS container itself stops sending its periodic heartbeat to the cluster, the cluster alerts that the external watchdog is gone.

This gives a two-way health check:

NAS → Cluster: the NAS polls the cluster from outside and alerts if service URLs stop responding
Cluster → NAS: the cluster checks whether the NAS watchdog has recently posted its heartbeat, and alerts if it has not

Neither direction alone is sufficient. Together they cover the main failure scenarios.

What the watchdog checks

Each public service URL (health endpoint, or a minimal response that confirms the service is alive)
TLS certificate validity (flagging within 14 days of expiry)
DNS resolution for production hostnames
Response time — a timeout on connection or response is treated as a failure

All checks include explicit timeouts: connection timeout separate from response timeout. A service that connects but hangs indefinitely is a different kind of problem than one that refuses the connection, and both need to be caught.

What I learned building this

A few things that took more than one attempt to get right:

Alert thresholds need tuning based on real traffic patterns. A "no requests in 30 minutes" alert needs to account for legitimate low-traffic periods (nights, weekends). I adjusted the alert to trigger only when traffic drops significantly below the rolling average for the same time window the previous day. This eliminates false alarms without hiding real outages.

Disk alerts should fire early. An 80% threshold sounds conservative but on a busy registry node with several large container images, it is easy to cross from 80% to 95% in hours. I also added a separate alert for fast-growing directories.

The Pushgateway is useful but needs housekeeping. Metrics pushed via Pushgateway persist until explicitly deleted. A job that fails silently and stops pushing will show stale metrics rather than absence. I treat any Pushgateway metric older than 10 minutes for a frequently-running job as a signal of failure, not as the last known good value.

Resource limits on Prometheus and Grafana pods matter. Without explicit limits, Prometheus will happily consume several gigabytes of memory on a small node if scrape interval and retention are not tuned. I set memory requests and limits explicitly and check them after any scrape configuration change.

The result

After tuning, the monitoring stack runs comfortably alongside production workloads on modest Hetzner virtual machines. Dashboards load quickly. Alerts arrive within 2 minutes of an actual failure. The external watchdog has already caught two incidents where the internal cluster monitoring would have been silent.

The key constraint I kept in mind throughout was that monitoring should cost less than what it protects. On a small cluster, that means being deliberate about scrape intervals, retention, and the number of metrics collected — not maximizing observability for its own sake.

For a small self-managed cluster, this is enough. It answers the questions that matter, and it does not add a maintenance burden that competes with the actual work.

---

*Next in the series: Handling multi-architecture images in a Kubernetes cluster where nodes run different CPU architectures.*

How I Use CIR Notes to Make AI Coding Agents More Useful

2026-03-28T12:42:00.003+01:00

By Vicente Arteaga Gomez

MisLinux · Last updated: April 4, 2026

This is a standalone MisLinux article about how I work with AI coding agents such as GitHub Copilot, Claude, and OpenAI Codex. It reflects my own workflow and opinions only; I am not affiliated with, sponsored by, or endorsed by Microsoft, Anthropic, or OpenAI.

The most useful change I have made for AI-assisted development is not a newer model or a longer prompt. It is documenting Context, Intent, and Rationale close to the work itself.

I usually shorten that to CIR:

Context: what environment, system state, history, or constraint already exists
Intent: what I am trying to achieve right now
Rationale: why this approach was chosen instead of another one

That sounds simple, but it closes one of the biggest gaps in AI-generated code: the model often sees what the code does without clearly seeing why it was shaped that way.

Why tools like Copilot, Claude, and Codex still miss important things

Modern coding agents are genuinely useful. They can trace files, suggest patches, explain APIs, and move through a repository faster than a human can when the work is mechanical.

But they still struggle in a few recurring situations:

a configuration looks strange for a good historical reason
a seemingly redundant check exists because production failed without it
an unpopular dependency was chosen to match another system
a simple-looking refactor would quietly break deployment, compliance, or recovery workflows

When that reasoning only lives in a developer's head, an AI agent has to guess. Sometimes it guesses well. Sometimes it confidently "improves" the exact thing that was protecting the system.

The missing piece is rarely context alone

Many teams already write some context down. They might have a README, a ticket, or a comment saying what a script is for.

That helps, but it is incomplete.

The missing piece is usually the combination of:

what already exists
what change is intended
why that change was implemented that way

Without the third point, AI tooling tends to optimize for local neatness instead of system truth.

A guard clause looks ugly, so it gets removed. A duplicate-looking config path seems unnecessary, so it gets unified. A staging-only publish flow seems too specialized, so it gets generalized and breaks production.

Rationale is what tells the agent, "This is not an accident. Preserve the tradeoff unless you are deliberately changing it."

Why I layer CIR instead of keeping it in one place

I have found that AI agents work best when CIR exists in more than one layer.

1. Repo-level guidance

At the repository level, I like a file such as .github/copilot-instructions.md or another top-level instructions document.

That is where I keep things like:

the big-picture architecture
expected deployment paths
important external systems
how changes are normally validated
repo-wide CIR notes for recent design decisions

This helps the agent avoid making changes that are technically tidy but operationally wrong.

2. Directory-level `AGENTS.md`

Inside important directories, I like an AGENTS.md that narrows the scope.

That file is where I describe:

what this directory is responsible for
what kinds of changes are normal or risky here
local naming rules and workflow expectations
the Context / Intent / Rationale behind the current shape of that area

This is the layer that helps an agent stop treating every folder like generic code. A Blogger-content directory, a Terraform directory, and a Docker build directory may all be in the same repo, but they need different instincts.

3. CIR near the code itself

For important scripts, modules, classes, or tests, I like a short CIR block close to the code.

That is where the most specific reasoning belongs:

why this function exists
which edge case it protects against
why a weird branch is intentional
which operational failure led to this implementation

This is often the difference between an AI agent preserving a critical safeguard and deleting it because it looks redundant.

What this looks like in practice

If I only write one top-level instruction file, the agent gets broad awareness but misses local tradeoffs.

If I only write comments in code, the agent may understand a function but miss the system around it.

If I only write an AGENTS.md in one folder, the agent may still miss the repo-wide deployment or compliance model.

Layering the guidance works better because each level answers a different question:

repo instructions explain the operating environment
directory `AGENTS.md` explains local expectations
code-level CIR explains why this implementation looks the way it does

That combination reduces accidental "helpful" changes.

The benefit is not only for AI

The nice side effect is that humans benefit too.

A future maintainer can skim the repo-level notes, move into the directory-level guidance, and then understand the code-level tradeoff without hunting through chat logs or old tickets.

That matters because AI-assisted work creates a new failure mode: important design reasoning can end up trapped in one-off conversations with the tool instead of becoming part of the project memory.

If the reasoning does not land in the repository, it might as well not exist.

How CIR changes the quality of AI edits

When I work this way, the output from coding agents improves in very practical ways:

they are less likely to remove deliberate safety checks
they make fewer wrong assumptions about deployment or publishing flow
they can explain their own changes in terms of the existing system
they are more likely to preserve edge-case handling
they require less corrective prompting from me afterward

In other words, CIR turns the agent from a fast code generator into a better repository participant.

My rule of thumb

If a future engineer or AI agent could reasonably ask, "Why on earth is it done this way?", I try to record the answer where the question will naturally arise.

That usually means:

repo-wide context near the repo root
local intent in the relevant directory
implementation rationale in or near the code itself

I do not try to document everything. I try to document the decisions that would otherwise be silently lost.

Final thought

Copilot, Claude, and Codex are all more useful when they are not forced to reverse-engineer intent from raw files alone. The better I capture Context, Intent, and Rationale in the repository, the more often the agent makes a change that is not only syntactically correct, but operationally correct too.

For me, that is the real productivity gain: not just faster edits, but fewer wrong edits.

What Kubernetes on Hetzner Costs, and When I Wouldn't Use It

2026-03-18T10:58:09.458+01:00

By Vicente Arteaga Gomez

MisLinux · Last updated: May 5, 2026

This article is part 6 of my MisLinux series on Kubernetes on Hetzner. It reflects my personal experience and tradeoff analysis only; I am not affiliated with, sponsored by, or endorsed by Hetzner.

The strongest argument for running Kubernetes on Hetzner is not that it is the most feature-rich platform. It is that the cost-to-control ratio can be excellent for the right workload profile.

The question that matters more than the cheapest invoice

When I revisit the economics now, I ask a slightly different question than I did at the beginning:

> What is the cheapest platform that still gives me a boring recovery story?

That framing matters because infrastructure cost is not only node price. It is also the number of future incidents that force a skilled operator to stop everything else and debug the platform under pressure.

What the platform saves you

Compared with larger managed-cloud setups, Hetzner can keep infrastructure spend far more predictable. That matters when the cluster supports a real product but the team is still sensitive to recurring cost.

The savings come from straightforward compute and infrastructure pricing, not from some hidden optimization trick. If your workloads are stable and your team can operate the cluster responsibly, the economics can be compelling.

Real Hetzner costs: what I actually pay

The table below reflects real Hetzner pricing (Falkenstein datacenter, April 2026) for a production cluster with one control-plane node, one general worker, and one dedicated workload node. These are list prices in EUR excluding VAT; your exact invoice will vary by datacenter, usage, and any discounts.

Compute (ARM64 `cax` series — best price/performance for most workloads)

Node role	Machine type	vCPU	RAM	Monthly price
Master / control plane	cax21	4 vCPU (ARM64)	8 GB	~€7.49
General worker	cax41	16 vCPU (ARM64)	32 GB	~€22.49
Dedicated workload node	cax21	4 vCPU (ARM64)	8 GB	~€7.49
Compute subtotal				~€37.47

Storage and network add-ons

Item	Spec	Monthly price
Block volumes (7 PVCs, ~170 GB total)	€0.0571/GB/month	~€9.72
Primary IPv4 addresses (3 nodes)	€0.50/IP/month	~€1.50
Server backup for master node	20% of server price	~€1.60
Add-ons subtotal		~€12.82

Total: ~€50–60/month for a real three-node production cluster

The full monthly cost including compute, storage, IPs, and backups is approximately €50–60/month depending on storage usage. This compares directly against:

Platform	Comparable setup	Approximate monthly cost
Hetzner (self-managed k3s)	3 nodes, 7 PVCs, 1 load balancer	~€50–60
GKE Autopilot	3-node equivalent	~€150–250
EKS (AWS)	3 × t3.large + EBS + ELB	~€200–300
DigitalOcean DOKS	3 × basic-2vCPU-4GB + volumes	~€90–120
Fly.io / Render	Comparable compute	~€80–150

The Hetzner number is real; the others are estimates based on list prices for comparable node sizes. All managed Kubernetes providers include control-plane costs that Hetzner does not charge for directly — you pay for that with operator time instead.

The command trail behind the monthly numbers

The reason I trust cost notes more when they are written down is that the verification path is concrete:

# Inventory the live Hetzner shape
hcloud server list
hcloud volume list

# Check what the cluster is actually using
kubectl get nodes -o wide
kubectl get pvc -A

The kind of output I want from that first command is something I can reason about quickly:

master-node      cax21
worker-node-0    cax41
vpaidd-worker-0  cax21

Even when I later graph or summarize the costs, those are still the commands that keep the cost discussion grounded in real infrastructure instead of in generic provider marketing.

What the platform asks from you in return

The lower infrastructure cost is balanced by higher operational ownership.

You are taking responsibility for things such as:

node configuration and cloud-init templates
cluster networking assumptions (Flannel VXLAN, firewall rules)
firewall correctness (both Hetzner Cloud Firewall and in-cluster network policies)
storage strategy (PVCs, backup, restore)
upgrade execution for k3s, OS packages, and cluster components
incident recovery when a node or service fails

That trade can be worth it. But it is only worth it if the team actually has the time and skill to own those areas.

The hidden cost: operator hours

A rough estimate for a team that is comfortable with Linux and Kubernetes:

Initial cluster setup: 8–16 hours
Routine monthly maintenance (patches, cert rotation checks, backup verification): 2–4 hours/month
Incident response per year (a bad node, a full disk, a certificate miss): 4–12 hours/year

At a conservative rate of €50/hour for skilled operator time, the annual non-infrastructure cost is roughly €1,500–3,000. That still compares favorably with managed Kubernetes if your cluster would otherwise cost €100+/month more. But it narrows the gap significantly.

A failure case that changed how I price "cheap"

I think about cost differently after incidents where one operational problem cancels out a lot of monthly savings:

registry cleanup that needs blob-level validation before it is safe
disk pressure caused by slow log/layer drift
certificate / routing issues that depend on remembering too many special cases

Those are not arguments *against* Hetzner. They are arguments for pricing operator clarity into the platform choice.

The costs I think about beyond the invoice

Cloud cost discussions get distorted when they only compare monthly server prices. For me, the real question is broader:

how many operator hours will this platform consume each month?
how expensive is a networking mistake or a slow incident response?
how much complexity is the team adding just to save money on compute?

Sometimes Hetzner still wins easily. Sometimes the operational overhead narrows the gap more than people expect.

Decision matrix: when Hetzner makes sense vs when it does not

Situation	Hetzner self-managed k8s	Managed k8s (GKE/EKS/DOKS)
Small team, cost-sensitive, strong Linux skills	✅ Strong fit	❌ Likely overpaying
First-time Kubernetes operator	❌ High risk	✅ Better starting point
Need hyperscaler-native services (Lambda, RDS, BigQuery)	❌ Wrong provider	✅ Use that provider
Compliance requires specific certifications (FedRAMP, etc.)	❌ Not available	✅ Choose certified provider
Moderate, stable workloads with predictable scaling	✅ Excellent fit	⚠ Fine but more expensive
Rapid unpredictable scaling (viral traffic)	⚠ Manual scale-up lag	✅ Autoscaling built in
Global multi-region with sub-50ms latency	⚠ Limited datacenters	✅ Global POP coverage
Long-running stateful batch jobs with ARM64 compute	✅ cax series is fast	❌ ARM64 on AWS/GCP is limited

When I would use Hetzner

I would seriously consider Hetzner when:

the workloads are moderate in scale
the team values cost efficiency and is willing to earn it with operational discipline
the architecture does not depend heavily on hyperscaler-native services
the operators are comfortable with Linux and networking
the platform can stay relatively simple

In that situation, Hetzner can feel refreshingly direct and honest.

When I would not use Hetzner

I would choose a different path if:

the organization needs extensive managed services immediately
compliance or governance requirements strongly favor a different provider
the team is not ready to own cluster operations
the workload would be better served by a simpler platform than Kubernetes
global multi-region complexity is a day-one requirement

Sometimes the smartest infrastructure decision is not to use Kubernetes at all. A simpler deployment model is often the better business choice.

My quick decision checklist now

If I had to choose again quickly, I would ask these in order:

Do I actually need Kubernetes, or just a reliable deployment target?
Do I have someone who can explain the node-IP, firewall, backup, and upgrade story without hand-waving?
Is the workload stable enough that Hetzner's lower invoice will really matter over a year?
Will one avoided managed-cloud feature later cost me more in custom ops work than I saved?

My practical decision rule

If a small team can clearly explain its node strategy, network model, restore path, and upgrade routine, Hetzner is a serious option. If those answers are fuzzy, then the cheap invoice may hide a platform the team is not actually ready to operate.

That is why I do not treat Hetzner as a universal recommendation. I treat it as a strong option when the team wants control and is honest about responsibility.

The honest conclusion

Kubernetes on Hetzner is powerful when you want control, reasonable performance, and lower recurring cost. It is a poor fit if you want to avoid operational responsibility.

That is the tradeoff in one sentence: you save money and keep flexibility, but you must earn reliability through disciplined configuration and maintenance.

For me, that trade has been absolutely worth it. My cluster costs ~€55/month and handles real production traffic, including latency-sensitive ad-serving workloads that need sub-100ms response times. That would cost three to five times more on a managed platform with comparable performance.

The key is being honest about what the platform gives you, what it does not give you, and what your team is realistically prepared to operate.

What I'd do differently now

If I were writing the first cost memo again, I would include two explicit footnotes:

"cheap compute does not mean cheap incidents"
"platform fit is a people/skills question as much as a pricing question"

Those two clarifications would have made the recommendation more honest from day one.

Series note

This closes the first six-part Kubernetes-on-Hetzner series on MisLinux. I wanted the final article to be deliberately honest rather than promotional, because platform choice only becomes useful when the tradeoffs are written down as plainly as the benefits.

The series continues with two later articles covering monitoring and multi-architecture image management. The full series index is available on the Series Index page.

MisLinux

How I Manage AI Agents Like a New Hire (Detailed Briefs, Reviewed Output, CIR Notes)

The contract I use with every agent session

Step 1: Route manual work through the agent

Step 2: Write briefs the way you would for a junior engineer

Step 3: Review output like a manager, not a spectator

Step 4: Demand CIR annotations for every non-obvious choice

Step 5: Reduce manual touchpoints every iteration

What this looks like in practice

Anti-patterns I stop early

Sibling posts on this blog

How I Fixed Cursor Getting Slower Every Message (Desktop + CLI, Any OS)

What was actually happening

The fix: a root .cursorignore

How I verify the fix worked

Cursor CLI uses the same contract

Windows, Linux, and macOS

When .cursorignore is not enough

My decision rule now

Related reading on MisLinux

Business Automation Through AI: How I Turn Manual Chores Into Tested Scripts

The loop I use

Example: from spreadsheet panic to finance fill automation

Example: publisher onboarding as manifest-driven automation

Where AGENTS.md and SKILL.md fit

Tests are the guardrails, not decoration

How I prompt AI so the output is durable

Reducing manual points reduces randomness

What AI is bad at in business automation

Checklist: is this real automation?

Tie-in to AI-assisted development hygiene

Publisher Onboarding as Manifests: How I Automate Ad Server, ads.txt, and Intranet Setup with YAML

The core idea

Why manifests beat ad-hoc scripts

What one onboarding manifest covers

Checks that can hook into monitoring

Detecting "onboarding is working" via traffic

AI's role: build the procedure, not wing the production click

What I would not put in a manifest

Reusable decision rules

Closing thought

How I Turn Read-Only Analysis Into Safer Production Automation

Why I prefer this order

The transition I want to see

A small workflow example

What I watch for before promoting read-only work into automation

Failure case: skipping the read-only learning phase

Why this matters for AI agents

What I'd do differently now

Why Live Follow-Up Files Beat Forgetting the Last Incident

What I want a follow-up file to contain

Why one markdown file is often enough

The real advantage

Failure case: the unresolved item that looks resolved

What I'd do differently now

How I Decide When an Operator-Facing UI Is Good Enough to Operate From

The minimum questions I ask

The two sections I now prefer

Operator guide

Inner workings

Why comments/notes matter too

A simple readiness checklist

Failure case: polished but misleading

What I'd do differently now

How I Use Proof Artifacts to Keep One-Off Investigations Repeatable

The artifact types I reach for first

Why I save them even for small tasks

A folder layout I like

The mistake I am trying to avoid

Why this matters with AI agents

A lightweight command trail

Failure case: artifact-only without explanation

What I'd do differently now

What Registry Garbage Collection Actually Costs in Operator Time

The real work around cleanup

Why tag existence is not enough

The diagram I keep in mind

The command trail I actually trust

Failure case: cleanup without path awareness

Why operator time dominates

The fix: a root `.cursorignore`

When `.cursorignore` is not enough

What `latest` hides

What I use `latest` for now