WangLu's Notes

Solving LVM Detection Failures in GRUB After a Force Shutdown

2026-03-26T00:13:00.003+01:00

After a routine system update and an unfortunate hang that required a hard reset, my Linux machine refused to boot. Instead of the familiar login prompt, I was greeted by a cryptic GRUB error: error: no cryptodisk module can handle this device.

My setup uses LUKS2 + LVM. From the GRUB rescue shell, I could actually decrypt the LUKS container. But once decrypted, GRUB completely failed to detect any LVM volumes. It simply acted as if the LVM structure didn't exist.

Meanwhile, if I boot it from a Live Rescue USB, everything worked perfectly. I could open the LUKS container, and the volume group appeared immediately. Tools like vgck and pvck reported no issues.

After a length discussion with AI, eventually I found the magical commands:

vgcfgbackup -f lvm_backup.txt <vg_name>
vgcfgrestore -f lvm_backup.txt <vg_name>

After running these commands and rebooting, GRUB recognized the LVM volumes immediately, and I was back in my system.

Supposedly this forces rewriting the LVM metadata. Perhaps there were issues with the metadata, which were caused by the forced shutdown. The issues can be handled by LVM parser in Linux, but not by the limited implementation in GRUB.

Notes on a Tricky Linux Installation: Qubes OS and Windows

2026-03-12T20:17:00.006+01:00

I recently tried to install Qubes OS alongside an existing Windows installation. It turned out to be surprisingly difficult—way harder than my last attempt—likely due to a combination of my encrypted /boot setup and older hardware. Here are some notes from the process.

Shrinking an NTFS Volume

I needed to free up some space from a Windows NTFS volume. Normally, this just takes a few clicks in Disk Management. This time, however, Windows reported a "shrinkable volume" that was suspiciously small.

Following this answer, I tried the standard fixes:

Disabled hibernation (powercfg /h off)
Disabled the pagefile
Disabled system protection

This increased the shrinkable volume a bit, but nowhere near the actual free space left on the partition.

Digging into the Windows Application logs in Event Viewer, I finally found the culprit: The last unmovable file appears to be: \$Mft::$DATA. It turns out $Mft is a special block in NTFS that cannot be easily moved, and a simple defrag wasn't going to cut it. I tried a few third-party partition managers, but they all failed initially. Following a hint from one of the tools, I temporarily disabled BitLocker. That did the trick—AOMEI Partition Assistant was finally able to shrink the volume. Once it was done, I just had to re-enable everything.

Configuring the Display

I use a dual-monitor setup (let's call them A and B). The Linux console and GUI installer assumed monitor A was the primary display, leaving monitor B either completely blank (in the terminal) or showing an empty desktop (in the GUI). I wanted everything on B, and the usual Super+Arrow Key shortcut wasn't working.

Here is how I forced the display:

Turn Off the Display in the Terminal

Note: This forces the GUI installer to the correct screen, but doesn't change the terminal itself.

Switch to the terminal (Ctrl + Alt + F2).
Find the "bad" display in /sys/class/drm/.
Run echo off > /sys/class/drm/card0-<DEVICE_NAME>/status.

Example: echo off > /sys/class/drm/card0-HDMI-A-1/status

Switch back to the GUI (Ctrl + Alt + F6). The installer should now be forced onto the only "active" screen.

Turn Off the Display via Kernel Parameters

First, find the device/port name by running: ls -d /sys/class/drm/card*-*. Examples:

/sys/class/drm/card0-DP-1 (Integrated)
/sys/class/drm/card1-HDMI-A-1 (Discrete)

Add something like video=eDP-1:d to the kernel parameters.
This gets more complicated when the ports are the same but the cards are different, though I haven't tested that scenario.

GRUB and Encrypted `/boot`

GRUB 2.12 supports LUKS2, but it doesn't support the Argon2 hashing algorithm—that didn't arrive until GRUB 2.14.

While GRUB 2.14 works perfectly on my newer machine, it refused to boot properly on this older one. After hours of troubleshooting, I realized that while GRUB 2.14 could boot directly into Linux, but it failed completely when trying to boot through Xen. Suspecting the issue lay somewhere between GRUB and Xen, I eventually gave up and downgraded to GRUB 2.12, changing my LUKS partition to use PBKDF2 instead of Argon2.

Another quirk: GRUB's decryption implementation feels about 100 times slower than cryptsetup. It likely lacks hardware acceleration for decryption, and the -A parameter isn't available for the cryptomount command in my version of GRUB. To keep boot times reasonable, I had to decrease the number of PBKDF2 iterations in LUKS.

LVM Issues

Xen and Linux finally booted, but the celebration was cut short. The boot process stalled, complaining that the /dev/mapper/boot device timed out, and it wouldn't even let me enter the rescue shell.

I fixed the rescue shell issue by booting from a live USB and giving root a password. The timeout issue, however, was much weirder. It turned out that only the LVM logical volumes specifically listed in the rd.lvm.lv kernel parameters were being unlocked; the rest were completely invisible. Even running vgs and lvs returned empty results.

I was eventually able to recover everything using the vgimportdevices -a command. Oddly, this created a duplicate LVM entry in the system.devices file, and manually removing the duplicate broke everything again. I ended up just deleting the file entirely and letting vgimportdevices recreate it from scratch. It’s been running smoothly ever since.

Secure Boot

Just as I got Qubes OS working with Secure Boot enabled, Windows broke. The UEFI complained that the boot signature wasn't recognized.

After some debugging, I figured out what happened: I had reset the Secure Boot keys on my motherboard to their factory defaults. Because the laptop is old, the factory defaults only contained the 2011 Microsoft keys. However, my Windows boot loader had been updated and was signed with the newer 2023 key.

I tried copying the Secure Boot database from another machine, but that failed (likely due to PK/KEK mismatch issues). After hours of trial and error, I solved it with a combination of the following actions:

Set Secure Boot to Setup mode.
Reset to factory keys
Update Secure Boot variables via Windows registry and scheduled tasks (source)

reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Secureboot /v AvailableUpdates /t REG_DWORD /d 0x5944 /f

Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

# Manually reboot the system when the AvailableUpdates becomes 0x4100

Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

Boot SecureBootRecovery.efi from the Windows EFI partition.

I suspect running SecureBootRecovery.efi was the magic bullet, though the other steps likely set the stage. Surprisingly, this file never came up in my online troubleshooting; I just stumbled across it by accident while browsing the EFI partition.

Final Thoughts

Looking back, I definitely stacked too many tricky components together—older hardware, dual-booting, encrypted /boot, LVM, and Secure Boot—and hit a bit of bad luck along the way. Fortunately, it all worked out in the end. What a journey!

Refined Boot for Qubes OS: Minimal USB Key, Dual Boot, Secure Boot

2026-03-08T23:56:00.003+01:00

I've been running Qubes OS on Machine A alongside Windows for a while. My setup involved storing the unencrypted /boot partition and the LUKS header on an external USB drive.

Recently, I planned to install Qubes on Machine B, also in a dual-boot configuration. However, the complexity jumped significantly:

Machine B has Secure Boot enabled because BitLocker requires it. On previous installs, I grew tired of toggling Secure Boot in the BIOS every time I switched operating systems.
I only have one USB drive. Managing separate /boot partitions for two different Qubes installations on a single thumb drive is messy.

After some experimentation, I found a way to solve both problems.

Sharing One USB Drive for Multiple Qubes Installations

The solution is elegant: Don't store /boot on the USB drive. Instead, move /boot to the encrypted internal disk partition. The USB drive's only job is to unlock that partition and hand over control to the system. Once I grasped this concept, implementation was relatively straightforward using the Arch Wiki and AI.

Some notes:

The USB drive only needs an EFI partition containing the GRUB binary and the LUKS header files. In my case, these total less than 30MB.
While recent GRUB versions support LUKS2, but only very recent commits supports Argon2. Fedora's version was too old, but the packages in Debian Sid worked. Arch shoud also work.
I wrote a simple grub.cfg that loads the necessary modules (GPT, LUKS2), unlocks the partition via cryptomount using the header file on the USB, and then passes control using configfile.
For this setup, usually, /boot can just be a folder in the root directory/filesystem. However, because my root is on a thin-provisioned LVM logical volume (which GRUB does not yet support), I had to create a separate, standard LVM logical volume specifically for /boot.
I used grub-mkstandalone to create a bundled GRUB binary and efibootmgr to register it with the UEFI firmware.
When the kernel/initramfs boots from the now-unlocked /boot, it doesn't "inherit" the unlocked state. To avoid typing the password twice, I embedded a LUKS keyfile into the initramfs to automate the second unlock.
To support both machines, I copied both LUKS headers to the USB. I then modified grub.cfg to detect the machine's UUID via smbios, allowing it to automatically select the correct header and partition.

The Benefits

Compared to the standard "detached USB" method, this stores significantly less data on the unencrypted drive, making backups easier and security tighter.

It also solved a major headache: updates. Previously, I had to manually mount /boot before updating dom0 and remember to unmount it before rebooting. If I forgot and triggered a reboot of sys-usb, the system would often hang. Now, those days are over. I can update the system without the USB drive even being plugged in; the drive itself rarely needs updates.

Make Qubes OS Play Nice with Secure Boot

It turned out I misunderstood "Qubes OS doesn't support Secure Boot." What that actually means is that Qubes doesn't provide an out-of-the-box way to verify the entire chain (Xen, kernels, etc.).

However, if our goal is simply to allow Qubes to boot without disabling Secure Boot in the BIOS, it’s actually quite easy:

Set up shim, register the entry with UEFI, and point it to my GRUB EFI binary.
Ensure my GRUB binary contains an SBAT section. I used objcopy to extract this from Debian GRUB binary, then included it via grub-mkstandalone.
Disable MOK validation mokutil --disable-validation. The MOK Manager actually refers to it by "Disable Secure Boot", but it doesn't actually affect UEFI or Windows.
Enroll my GRUB binary hash during the first boot.

That's it!

I was surprised by the simplicity. I expected to be wrestling with PK/KEK keys and accidentally bricking my Windows bootloader. Instead, learning how shim and MOK interact made the process painless.

Security Implications

To be clear: this method allows Qubes to run alongside Secure Boot, but it does not (yet) cryptographically verify the Xen image or Linux kernels.

However, since that data resides on an encrypted partition and the bootloader is on a detached USB, it meets my personal threat model. In the future, I may look into creating my own keys to sign the images properly.

I’m also considering moving the GRUB binary to an internal partition. If I embed the LUKS header directly into the GRUB binary and enroll its hash, the shim would theoretically notify me if the binary was tampered with. It’s not a perfect defense against physical access, but it's a significant step up from a standard unencrypted boot.

Writing Sudoku Solvers

2026-02-05T21:58:00.005+01:00

After writing a Nonogram solver, I decided to tackle a Sudoku solver to practice Rust. My goal wasn't just to support classic Sudoku rules, but also to handle variants like Thermometer, Arrow, and Cage etc.

1. Brute Force

It is fairly easy to write a brute force or backtracking algorithm. This approach is sufficient for most classic Sudoku puzzles, but it becomes unbearably slow as soon as variant rules are introduced.

I considered this step a warmup—a baseline to improve upon.

2. Constraint Propagation

Here, I tried to introduce "logical thinking" to the algorithm. I used u16 as a bitmask to represent the possible values of a cell. Whenever a cell's state changes (due to guessing, backtracking, or propagation), the algorithm consults all constraints to eliminate impossible candidates.

While Nonogram is technically an NP-Complete problem, in practice, my constraint-propagation solver (without guessing) can solve almost all puzzles found online. I’ve only seen one exception where I had to guess a few cells. This proves that puzzles designed for human players are meant to be solved via logical deduction, making them computationally "easy."

It turns out Sudoku is similar. Although some backtracking is still needed, classic puzzles are typically solved within 100~200 µs (microseconds), while variants might take a few milliseconds. I did see one puzzle take ~20 seconds, but overall, I was happy with the result.

So, can it go faster? Two optimization options came to mind:

When a cell changes, only consult relevant constraints rather than re-checking all of them.
Instead of copying the entire board state for every guess, I could carefully track changes and undo them during backtracking. However, since the board state is already quite small, I doubted this would yield a significant performance boost in practice.

Right before I decided to optimize further, I learned something new.

3. Dancing Links (DLX)

I discovered "Dancing Links" while asking an AI for the best Sudoku algorithms. This is a technique invented by Donald Knuth to efficiently implement Algorithm X, which solves the "exact cover" problem.

This is perfect for classic Sudoku, where the goal is to find an exact cover between "putting a digit in a cell" and "satisfying every row, column, and box constraint."

This is perfect for the classic Sudoku algorithm, where we essentially try to find the exact cover between "putting a digit into each cell" and "each row/column/box must contain exactly one digit X (where X is 1 ~ 9)".

The magical part: We can precisely undo this process by restoring the covered rows and columns. This means we don't need to copy the state during backtracking! The algorithm only needs to remember the latest guess; the data structure itself holds the information required to reverse it.

However, there's a catch: things get complicated quickly with variant rules. Only a few rules (like distinct values) can be easily encoded into the DLX structure. For most others, I had to implement them as "external observers" that eliminate impossible candidates after a guess. This forced me to maintain an undo stack again, essentially plugging a constraint propagation engine back into DLX.

Another issue: DLX wasn't actually faster than my custom constraint propagation solver. Since it consistently took ~200 µs for classic puzzles, I didn't bother implementing the complex variant rules for it.

4. Generic Solvers

I talked to a colleague about my progress, and he asked, "Why are you writing a custom solver? Why not just use a generic SAT/SMT solver like Z3?"

That was a good point. I did some quick research and picked three candidates to test:

The results are shown below.

5. Results

Here is how the solvers compared. (cp = my constraint propagation implementation, dlx = my dancing links implementation)

Classic Sudoku 1 (Easy)

cp: ~100 µs
dlx* ~200 µs
OR-Tools: ~10 ms
Z3: ~20 ms
cvc5: ~70 ms

Classic Sudoku 2 (Harder)

cp: ~200 µs
dlx: ~200 µs
OR-Tools: ~10 ms
Z3: ~80 ms
cvc5: ~200 ms

Hard: Empty Board + Thermometer Rules

cp: ~3 ms
OR-Tools: ~30 ms
Z3: ~20 s
cvc5: ~23 s

Hard: Almost Empty Board + Arrow Rules

OR-Tools: ~200 ms (slightly faster than cp)
cp: ~200 ms
Z3: ~20 s
cvc5: ~100 s

6. Discussion

I found these results both surprising and reasonable.

OR-Tools is significantly faster than Z3 and cvc5. I believe this is because OR-Tools uses a CP-SAT (Constraint Programming) solver, whereas Z3 and cvc5 are primarily SMT (Satisfiability Modulo Theories) solvers. Since the Sudoku puzzles I used are designed for human logic, they align better with the constraint propagation techniques used by OR-Tools.

My custom solver vs. OR-Tools. For easy puzzles, OR-Tools is slower than my implementation. This is likely due to initialization overhead; my code is hyper-optimized specifically for Sudoku. However, OR-Tools catches up quickly on complex puzzles. My constraint propagation logic is naturally inferior to the sophisticated heuristics inside OR-Tools, so as complexity rises, the generic solver wins.

7. Conclusion

I had a lot of fun and learned a great deal during this process. My next step is to explore OR-Tools further. Perhaps I'll write solvers for even more complex puzzles without reinventing the wheel!

An Adventure with Qubes OS2025-10-29T01:06:00.006+01:00
I've been experimenting with Qubes OS on my new laptop and wanted to share some notes on the experience.
HardwareOverall, Qubes OS works quite well on my hardware. Aside from typical issues like deep sleep, speaker performance, and touchpad scroll speed, the experience has been smooth. I particularly like that I can boot directly from a microSD card. This allowed me to move the /boot partition to the card while completely disabling USB access in dom0 for better security.
Detached /boot and LUKS HeaderMoving /boot and the LUKS header to a microSD card is a fun project, but it has some drawbacks:
I have to remember to mount /boot before updating dom0.
The system won't shut down properly if I forget to unmount /boot.
Testing Qubes OS 4.3 rc3I decided to test the Qubes OS 4.3 rc3 release by performing an in-place upgrade. Unfortunately, the system failed to boot afterward.
dracut IssuesAfter the upgrade, the system would hang before prompting me for my LUKS password. Eventually, the watchdog timer would kick in and drop me into an emergency shell.
Using the emergency shell and the installation media, I was able to investigate. I realized something was wrong with dracut, as it seemed unable to detect the encrypted disk. I tried including more files in the dracut configuration based on information from various sources, but that didn't help. So I gave up. Thankfully, the upgrade tool created an LVM snapshot, which made it easy to revert the changes. I did have to manually downgrade the kernel and Xen using the installation media to get my Xen domains working again.
After more research, I found a bug (1, 2) in the version of dracut included in Qubes OS 4.3. Essentially, the crypt module stops working when systemd is available. I also believe the systemd-cryptsetup module wasn't automatically included because the LUKS header was on a separate device, leading dracut to assume that LUKS decryption wasn't needed.
The fix was simple: manually enable the systemd-cryptsetup module in the dracut configuration.
amdgpu IssuesAfter the upgrade, only the old kernel worked, that's how I could fix the dracut issue. With the new kernels, the system would boot to a blank screen. Removing the rhgb and quiet kernel parameters revealed some log messages, but the blank screen remained.
Adding the nomodeset parameter disabled the graphics driver, which allowed me to enter the LUKS password and log in. However, this caused Xorg and LightDM to fail, with Xorg repeatedly crashing.
It was clear this was an issue with the amdgpu module. I tried several kernel parameters without success:
amdgpu.modeset=1
amdgpu.dc=0
amdgpu.dpm=0
amdgpu.ppfeaturemask=0xffffb
Eventually, I found that amdgpu.dcdebugmask=0x10 resolved the problem.
qubesd IssuesAfter finally booting into the system, I couldn't attach any block devices, including my /boot partition, to dom0. This turned out to be a bug in qubesd, which I reported.
Backup StrategyBacking up data in Qubes OS can be tricky due to its design. Here is the high-level strategy I've been planning:
Create as few templates as possible.
Only use Salt to configure templates. So I only need to back up Salt files in dom0.
Changes to dom0 are either managed by Salt, or the relevant files are included in the backup. ExamplesXfce settings.
Qube settings/features, /etc/qubes
/etc/default/grub
/etc/dracut.conf.d
My backup process is as follows:
I use a dedicated disposable VM with restic installed. The actual backup scripts and SSH keys remain in dom0.
Data from dom0 and other qubes is archived using tar. Use --transform to prepend the VM name to the path.
Use qvm-run --pass-io --no-gui to pass the scripts, SSH keys an data to the disposable VM, which then runs the scripts to execute the backup.
There are a few things to consider:
Using --pass-io could have security implications. It might be possible to mitigate this by limiting the number of bytes passed and saving output to a file.
The tar archives are currently extracted in the disposable VM. If the data isn't trusted, this could be a security risk. In such cases, the extraction step could be skipped.
Accessing the backed-up data requires starting a VM. An alternative could be to create an LVM snapshot and back that up directly.
Thought: Manging Qubes like ContainersThere are many similarities between Qubes VMS and containers (docker, podman):
Template VMs are similar to building containers.
AppVMs are like running containers with persistent volumes.
Disposable VMs are like running containers without persistent volumes.
There are some gaps:
One container can be based on others, the shared part are often stored only once as layers. While cloning a template VM can use reflink for initial efficienty, changes will evetually cause the data to diverge. bootc might help bridge this gap.
Containers support bind mounts, which is very convenient for backup. While virtiofsd could work for Xen, but I guess there are security concerns. An alternative could be to centralize all data in one qube and share it with others over the network, but again, there could be security concerns.
ConclusionIt's been a fun and challenging journey exploring Qubes OS. While there were some hurdles with hardware and system upgrades, working through them has been a valuable learning experience. The security architecture is powerful, and I'm excited to continue finding new ways to make it work for my setup.

A Rocky Migration: Moving from docker-compose to Podman and gVisor2025-10-09T00:26:00.005+02:00
I've been running a few containers for several years. They were all running under rootless Docker with a single user.
Initially, I planned to migrate the containers to VMs, but I couldn't get a stable workflow after about two months of effort. Later, gVisor caught my attention, and I decided to migrate to Podman with gVisor instead.
The new plan is to run each container with --userns=auto and use Quadlet for systemd integration. This approach provides better isolation and makes writing firewall rules easier.
I'm now close to migrating all my containers. Here are a couple of rough edges I'd like to share.
Network LayoutI compared various networking options and spent a few hours trying the one-interface-per-group approach before giving up. I settled on a single macvlan network and decided to use static IP addresses for my containers.
To prevent a randomly assigned IP address from conflicting with a predefined one, I allocated a large IP range for my containers and assigned random addresses from that range.
Routing IssuesI ran into a tricky routing problem. Let's say my host has a network interface eth0 and a veth pair where veth0-host is on the host and veth0-ctr is in the container.
Here are the IP addresses:
eth0: 192.168.0.1
veth0-host: 192.168.13.1
veth0-ctr: 192.168.13.100
To allow an external client to talk to a service on 192.168.13.100:1234, I set up a prerouting DNAT rule in nftables to forward traffic from 192.168.0.1:1234 to 192.168.13.100:1234. To my surprise, the host itself couldn't access 192.168.0.1:1234.
It turned out there were two issues:
DNAT in the prerouting chain doesn't apply to loopback traffic from the host itself.
Masquerade and SNAT also didn't work, likely because the kernel has a short-circuit mechanism for local traffic, so the transport happens at Layer 2.
Because of this, the service at 192.168.13.100 would send reply packets to 192.168.0.1 instead of 192.168.13.1. The packet would still be received on veth0-host, but my firewall would complain that this violates the "strong host model".
I didn't have this issue before, probably because port forwarding was handled at Layer 3 by slirp4netns. This seems to be a "hairpin NAT" issue, but it's more complicated with a veth pair.
In the end, I just configured the host to talk to the container at 192.168.13.100 directly.
File PermissionsBecause I'm using --userns=auto, the :U flag is almost a must when mounting volumes. Surprisingly, this didn't cause too many problems as long as I set up the correct user and group.
Sometimes a container needs to access files on the host. If a group HOST_GID has access to a file, we can grant access to the container's primary user with --userns=auto:gidmapping=$CONTAINER_GID:$HOST_GID:1 --group-add $CONTAINER_GID. Here, CONTAINER_GID is an unused GID inside the container.
However, this only works well with the default crun runtime. With gVisor, I found two problems:
First, the permission only works if CONTAINER_GID is the primary group of the container's main user. It doesn't work if it's a supplementary group.
Second, gVisor does not seem to support POSIX ACL. This means the dac_override capability is needed if the CONTAINER_GID doesn't appear to have permission according to standard Unix permissions.
It's not too bad in practice, but it was surprising until I figured out what was going on.
Default DNS ServerDocker provides a DNS server at 127.0.0.11 for each container. Podman, however, creates a dedicated DNS server for each bridge network. Some of my containers relied on the Docker behavior and had this IP address hard-coded, which caused quite a bit of trouble.
DNS Servers for Multiple NetworksIf a container joins both an internal bridge network and an external macvlan network, the container only sees the DNS server from the bridge network. IP routing still works, meaning the container can access the internet by IP, but it can't resolve external domains.
This is a bug in Podman that has been fixed in the latest version, but it still exists in the version I'm using from Debian. Below are the hacks I considered for this old version.
Option 1: Override DNSIf the container doesn't need to resolve internal container names, we can force it to use an external DNS server. However, the --dns flag didn't work as I expected.
According to Issue #17500, the server from --dns is added into other DNS servers as upstream, such that internal container names can be resolved first. This doesn't work in my case because the internal DNS server can't access the internet, so it can't forward the query.
A simple workaround is to override /etc/resolv.conf using a bind mount.
Option 2: HTTP ProxyIf the container supports HTTP proxy, we can remove it from the external network. Instead, we can add a proxy container to both the internal and external networks. The proxy can use its own external DNS server, and the original container can use this proxy via its internal IP.
This should work in theory, but it felt like too much effort. It also surprised me that nginx doesn't support proxying HTTPS traffic (CONNECT method) without extra efforts. If I had to go this route, I would probably use mitmproxy.
Option 3: Transparent HTTP ProxyI have set up a transparent HTTP proxy in my network. The container thinks it is talking to an external server, but my firewall redirects the traffic to an nginx server, which can forward or reject the traffic. Unlike a normal HTTP proxy, this is easy to do with nginx and is completely transparent to the client.
Now, if a container only needs to access a few known HTTP/S domains, I just add entries for those domains to the container's /etc/hosts file with an arbitrary IP (like 1.1.1.1), using AddHost=. Nginx completely ignores this IP, it reads the domain from the request and resolves it on its own. It's very hacky but also very practical.
NamespacesSome containers assume they share the same user namespace, which is common when they are running under docker-compose.
Podman has --userns=container:id to join an existing container's user namespace, but this doesn't work with gVisor. From what I've learned, this is related to gVisor's sandbox model.
The solution is to put containers into a pod. However, with gVisor, containers will not join the network namespace of the pod due to the same security model. This wasn't a big deal for my use case, but it was unexpected.
The same thing happens with the UTS namespace. When running with gVisor, a container's hostname becomes empty if it joins a pod. Issue #7995 is relevant here. Apparently, some binaries (like busybox's sendmail) don't like an empty hostname. The solution is to give the container a private UTS namespace: --uts=private.
Shared VolumeSome of my containers use flock on a shared volume to communicate. I think this is a bad design. And guess what? It doesn't work with gVisor.
I thought I might need to set up some mount hints annotations, but that didn't help, so I guess inotify wasn't the issue. Turning on shared file access didn't work either.
Other NotesI had planned to use socket activation extensively, but it turned out I didn't. Most containers need networking anyway, and it's much easier to manage port forwarding with simple firewall rules.
It is possible to mount an empty volume, which acts as an upper overlay on existing files at the mount destination. This is very useful for read-only containers.
Final ThoughtsThis wasn't a trivial migration, but I guess that was expected since I was changing so many variables at once. In any case, I'm quite happy with the new setup.

Hardening Container Network Security: Filtering Outgoing Traffic2025-09-25T22:54:00.009+02:00
I want to filter the outgoing network traffic for all of my containers based on a set of rules. For example:
Some containers should be blocked from accessing the internet entirely.
Some containers should have unrestricted internet access.
Some containers should be able to access the internet, but not a specific list of URLs.
Some containers should only be allowed to access a specific list of URLs.
To manage this, I will define logical policy groups and assign each container to one. As a general rule, only DNS and HTTP/HTTPS traffic will be permitted.
Option 1: A Proxy for Each Policy Group
Imagine Container A is only allowed to access www.google.com. Here’s how this approach would work:
Create an Nginx (or socat) container that listens on port 443 and acts as a reverse proxy for www.google.com.
Place both the Nginx proxy and Container A into an internal container network.
Within this network, add www.google.com as a network alias for the Nginx container.
Connect the Nginx container to a second network that has internet access.
ThoughtsThis is my current solution using docker-compose, and I believe it should also work with Podman.
It is possible to use a single Nginx container to proxy multiple domains, even for HTTPS traffic. By using the ngx_stream_ssl_preread_module, Nginx can inspect the requested domain from the TLS handshake and forward the traffic accordingly without needing to decrypt it.
This option is straightforward to implement, and a key advantage is that I don't need to set up a custom DNS server. It is also relatively easier to write firewall rules.
On the other hand, configuring and managing a separate proxy container for each rule can become tedious. I think using Quadlet files, especially with templates and drop-in overrides, could simplify this process.
Another significant downside is the inability to log blocked traffic. If a container tries to access a domain that isn't explicitly proxied, the connection will simply fail without a log entry, making troubleshooting difficult.
Option 2: Central Proxy on a Single Network
In this design, we set up a central proxy for both HTTP/S and DNS traffic and then perform the following steps:
Intercept and redirect all traffic from containers to the central proxy using nftables rules. For DNS, this is simpler, as I can configure the container network to use my custom DNS server.
The proxy must identify the source container to determine which policy group it belongs to.
The proxy must identify the requested destination. This is easy for HTTP (from the URL) and DNS (from the query). For HTTPS, we can again use the SSL preread technique to find the domain in the TLS handshake.
The proxy applies the policy, then either blocks or forwards the traffic.
NetworkingFirst, I would create a veth pair. On one end, I would create a macvlan network in "private" mode and connect the containers to it. The other end would be assigned an IP address on the host to allow routing. This essentially creates a bridge where connected containers are isolated from each other but can reach the gateway.
Podman doesn't seem to support configuring a standard bridge with a mix of isolated and non-isolated ports. Note that the --isolate option in podman network isolates the entire network from other container networks, not individual ports on the bridge.
In the diagram, the proxies are shown on a separate bridge connected to the internet, mainly for illustration. In practice, it might be easier to connect all containers to the same macvlan network and use a firewall to control traffic flow. Although the macvlan network is in private mode, the firewall may allow "hairpin" packets to allow traffic between specific containers.
Identifying ContainersWe can identify containers by their IP addresses. The tricky part is ensuring these IP addresses are trustworthy and that the setup isn't prone to errors.
Let's review the IPAM drivers supported by podman network:
dhcp: For each container, we can assign a fixed MAC address and create a static reservation in the DHCP server. The firewall can then reliably use the container's IP address to identify it. This assumes that containers are unprivileged and cannot change their own MAC or IP addresses. Ideally, the default address pool of the DHCP server should be disabled to prevent unassigned containers from getting an IP.
host-local: With this driver, we assign a static IP address during podman run. While this sounds simple, it's easy to forget to provide an IP when running a container manually. If that happens, Podman will assign an IP automatically. This could accidentally grant a container internet access or cause an IP conflict. I haven't found a way to disable this automatic IP address allocation.
none: This driver does not assign an IP address, and you cannot manually provide one either.
In theory, using "dhcp" for the IP configuration should work. However, a practical issue emerges because systemd-networkd respects the client ID in DHCP requests, and Podman sends the container ID as this client identifier. Furthermore, since podman-systemd utilizes podman run --rm, restarting a container.service generates a new container with a new ID.
This combination means the DHCP server sees the restarted container as a new machine. It then refuses to offer the configured static IP address, believing the lease is still valid and held by the original container. I have not yet found a way to override this client ID, so I may need to evaluate a different DHCP server or abandon this approach.
Deciding the Policy GroupOnce the container is identified, applying the policy is relatively easy:
CoreDNS has the view plugin, which can apply different rules based on the client's IP address.
Nginx has the geo module, which can be used to map a client's IP address to a variable for use in access rules. You can also use map $remote_addr.
Option 3: One Network Per Policy Group

This approach extends the "veth+macvlan" technique by creating a separate network for each policy group. We then use nftables rules to forward traffic from all networks to a central proxy. This is similar to Option 2, but this time nftables can identify the source policy group by the network interface the traffic arrives on.
This approach is more secure if you are concerned about IP or MAC spoofing since the network interface is a more reliable identifier than an IP address alone.
Identifying the Policy GroupBy IP Address: We can configure a DHCP server for each network with a non-overlapping IP range. The proxies can then identify containers by their IP address, just like in Option 2, but with greater trust since the IP is tied to a specific network. We still need to be cautious to ensure IP ranges don't overlap.
By Interface: We can identify traffic by the interface it comes from.CoreDNS has the bind plugin, which allows it to listen on specific host interfaces. However, this requires CoreDNS to run in the host network, and the proxy would need to be restarted every time a new policy group (and thus a new interface) is added. It's also unclear how this would work with Nginx.
A variation is to run CoreDNS and use port forwarding (or maybe socket activation) to listen on all interfaces, then I can use redirect in nftables. This way the traffic within each policy group should be redirected to the corresponding gateway. However, this setup sounds complicated, and similar to above, I'm not sure if it'll work for nginx.
A more complex option is to use nftables to map each incoming interface to a different port on the host. We could then run a proxy instance for each policy group, listening on its assigned port. This essentially moves the identification logic into nftables and is useful if a proxy doesn't support IP-based policies, but the rules would be complicated and fragile. For example, we would need rules to prevent a container from accessing a proxy port it's not authorized for.
My PlanUltimately, I need to find a balance between two goals:
Maximum Security: Resisting vulnerabilities and malicious actors.
Ease of Maintenance: Requiring minimal effort and not being error-prone.
I will most likely implement Option 2, if I can make it work. It offers a good blend of centralized control and flexibility without the complexity of managing dozens of networks or proxy containers.

A Journey into Podman: Notes on My First Adventure

2025-09-20T01:45:00.006+02:00

For the last few days, I've been experimenting with Podman. My goal was to get a feel for the setup, create a minimal yet scalable environment for a few containers, and identify potential problems early on.

Here are my notes from this experience.

[Updates 2025-09-21] Added more networking options and other information.

Quadlet

Quadlet allows you to define containers, networks and more using a syntax similar to systemd. This includes helpful features like drop-in overrides and templates.

The framework is tightly integrated with systemd, and Quadlet actually generates real systemd units. This means I can directly write systemd options in my Quadlet files.

One of the biggest benefits I've found is how easy Quadlet makes it to set up socket activation. This allows me to place some containers in an internal network or even without a network at all.

Hardening Defaults

Let's say I have a group of Systemd and Quadlet units, all named in the format of xyz-*. My goal is to define some secure, hardened default values for these units that can still be overridden by individual units.

For example, I want to change the default for ProtectSystem= from false to true.

Simply creating a xyz-.service.d/00-override.conf doesn't work, because an individual unit cannot override this setting. While I could create a xyz-foo.service.d/10-override.conf for each specific service, this would split the definition of xyz-foo into two separate files, which isn't ideal.

To solve this, I created a script that moves the main xyz-foo.service file to xyz-foo.service.d/10-override.conf and creates a nearly empty xyz-foo.service as a placeholder. This facilitates the process of setting and overriding defaults.

gVisor

Setting up gVisor's runsc was straightforward, and I haven't encountered any compatibility issues so far.

Unfortunately, the version of gVisor in the Debian repositories is quite old, and Debian is unable to provide prompt security updates for it. This meant I had to add the official gVisor apt repository. It's not the ideal solution, but it works.

Credentials

Systemd-Credentials is a very handy tool for managing sensitive information and can be used directly in Quadlet files.

However, I ran into an issue when using it with containers that have --userns=auto. Because Podman still runs as root, the credentials are only readable by the root user and not by the containers.

Podman offers its own solution for this called podman-secret. This feature allows you to either have Podman store the secrets or use drivers to connect to your own storage solution.

I prefer to keep all my secrets in a dedicated directory. To accommodate this, I wrote a simple script that registers a file as a Podman secret:

podman secret create \
  --driver=shell \
  --driver-opts=list="echo $1" \
  --driver-opts=lookup="cat $2" \
  --driver-opts=store=/bin/true \
  --driver-opts=delete=/bin/true \
  --replace=true \
  "$1" - <<<SECRET

For any container that needs access to secrets, I simply call this script in ExecStartPre=.

I found a few issues, and create an issue on GitHub. Fortunately workarounds are available.

Networking

My plan is to run a few groups of containers with the following requirements:

Containers within the same group can communicate with each other, but containers from different groups cannot.
I want to avoid manually managing IP addresses for each group or container.
It should be easy to write firewall rules to intercept container traffic.

I explored a few options to achieve this:

Plain Bridges: The simplest approach is to create a default bridge for each container group, which is the default behavior in Podman. Everything works out of the box, and I don't need to explicitly specify IP addresses. However, writing firewall rules is tricky because the IP addresses and bridge names are not predetermined. To make this work, I would need to carefully define an IP range and/or bridge name for each bridge.
Single Bridge: Another option is to create a bridge using systemd-networkd and then, for each container group, create a Podman bridge network using the existing bridge with VLANs and/or isolation. Since the bridge is unmanaged, I will need to manually set up DHCP, DNS, and the firewall.
- Simply setting up a DHCP server on the bridge (i.e. in bridge.netweork) doesn't work. I think it is because podman, or more precisely, netavark-dhcp-proxy, will use the interface (bridge in this case) as DHCP client rather than server.
- It works if I add an external DHCP server to the bridge. E.g. create a veth pair, put one end under the bridge and add a DHCP server on the other end.
VRF: I tried creating a VRF with systemd-networkd and then creating a Podman bridge network for each container group, specifying the VRF. DHCP works with this setup, but DNS server doesn't. While I was able to force aardvark-dns to use the VRF by using a wrapper script (exec ip vrf exec MY-VRF aardvark-dns "$@"), this solution felt too fragile for long-term use. Also, I wasn't able to easily isolate containers within the same network.
veth + macvlan: This setup involves creating a veth pair for each container group, putting DHCP server on one end, and then using the other end to create a Podman macvlan network. This works, and it is easy to isolate containers within one podman network, by using the private mode and setting up necessary firewall rules. But note that the DNS server is not supported in macvlan mode.

My Networking Plan

All these options more-or-less work, with different trade-offs. I plan to use 1 and 4 for different use cases:

When a group of containers need to find each other by name, I put them into a bridge network (option 1) with Internal=true. Both DCHP and DNS work, and I don't need to write much firewall rules.
When a container needs to access Internet, I create a private macvlan network (option 4) and put the container into it. Since I can easily isolate containers within the same network, it is possible to put containers from different groups into the same bridge network. Then I can just use the bridge network to group containers by policies, e.g. some containers can access everything, but some are only allows to visit specific URLs. I just need to write firewall rules for each bridge network.

Final Thoughts

So far, the combination of Podman, Quadlet, and gVisor has been a positive experience. Not everything has worked perfectly, but I'm quite happy with the setup. If things continue to go well, I might be able to migrate my docker-compose setup in the near future.

gVisor: A Fresh Look at Container Security2025-09-17T22:08:00.009+02:00
My original plan was to stabilize my VM pipeline before deploying containers using a hardened stack of Podman, QEMU, SELinux, and user namespaces (--userns=auto). However, the pipeline's complexity grew, requiring script rewrites and schema redesigns, and the process took much longer than anticipated.
In the meantime, an interesting alternative has captured my attention: gVisor. It occupies a unique space between traditional SELinux policies and full-blown virtual machines, offering a compelling set of trade-offs.
What is gVisor?At its core, gVisor is an application kernel, written in the memory-safe language Go, that provides an additional layer of isolation between containerized applications and the host operating system. It's essentially a user-space implementation of the Linux kernel's system call interface.
The security model is explained here.
gVisor in PracticegVisor provides an OCI-compliant runtime called runsc, which can be almost transparently integrated with container tools like Docker and Podman.
And that's it! Unlike SELinux, here we don't need to write any policies. This is the most attractive feature for me.
However, it comes with notable downsides:
SELinux is not supported, I cannot use both gVisor and SELinux at the same time.
--ignore-cgroups must be used for rootless podman, this mean cgroups won't work. Maybe it can be fixed later.
There can be potential compatibily issues, because gVisor implements its own version of syscalls.
The performance overhead is higher, especially for IO-related syscalls. It is well explain here.
My PlanI plan to evaluate gVisor with a few of my simple containers. Its promise of "secure-by-default" sandboxing without complex configuration is very appealing, especially for running applications where trust is a concern but the overhead of a full VM is undesirable.
I also believe that I don't really need the fine-grained control offered by SELinux. Bind mounts (read-only, read-write) should be enough for me. Eventually I might even drop the VM pipeline and just use gVisor.
We'll see.

A Declarative Approach to Config File Management

2025-08-24T23:40:00.010+02:00

Configuration files for different services are rarely independent. For example, in nftables, I might tag traffic with a firewall mark, and that mark is then used by systemd-networkd or in ip routes. Similarly, when the name of the primary network interface changes, multiple services like nftables, postfix, and samba need to be updated.

Requirements

I want to define core data in one place, then update all config files with a simple command.
If a configuration file is modified by an external process (for example, a package update from a vendor or distribution), the changes must be handled gracefully. Either the merge should be automatic and permanent, or I should be notified to easily resolve any conflicts.
It should be obvious within the config file itself what changes I have made.

Existing Solutions

I did some quick survey and found a few options.

1. Templates

These tools render a template using provided data sources. To manage /etc/config.txt, I would create a /etc/config.txt.template with all the moving parts marked using the required syntax.

Examples include:

The biggest issue is that the generated config file is no longer the source of truth. This means if the generated file is modified by other tools, those changes will be lost the next time I render the template.

Perhaps these tools are better suited for scenarios like building images with bootc or mkosi.

2. Patching

These tools record and apply the diff between a default state and the desired state.

Examples include:

diff and patch
Ansible's lineinfile and blockinfile
Augeas
crudini

There are two issues with these tools:

The diff is stored separately from the config file, which is hard to read and maintain. I might also need to keep a copy of the original, unpatched file for reference.
The patch might not be reliable if there isn't enough context to locate the exact area for patching. For example, consider a config file like this:
```
[Config for user A]
// many lines
use_https = true

[Config for user B]
// many lines
use_https = false
```
We want to modify the use_https setting for user A and generate a diff. Later, if the vendor's config file swaps the order of user A and B, the patch might still apply without error, but it will modify the wrong section!

Note that while Ansible can place markers around managed blocks, it must first insert them. For the initial insertion, it relies on regular expressions (insertafter and insertbefore) to find the location, which can be brittle.

3. Generators

NixOS allows you to generate all config files using custom data and functions in the same language.

The biggest issues with this approach are:

You are forced to commit to a specific ecosystem like NixOS or another tool that fully manages your system's configuration.
Merge conflicts almost never happen because your own NixOS configuration is just an override of the default values. This means you aren't notified of potential semantic conflicts. For example, if a default value you were referencing changes upstream, your configuration will adapt silently, which may not be the desired behavior without a manual review.

My Plan

The existing solutions I found almost solve my problem, but not 100%.

The closest approach I found is a combination of:

Adding a custom, unique anchor comment in the config file.
Using Ansible's blockinfile with the anchor comment for insertafter or insertbefore.

But I still don't like that the diff is stored separately from the config file. To solve this, my plan is to embed the template directly inside the configuration file, like this:

### BEGIN MANAGED BLOCK
### binds_to = {{ config.permanent_lan_interface.name }}
### END OF TEMPLATE
### END MANAGED BLOCK

I'll then write a script that:

Deletes all text after END OF TEMPLATE.
Parses the template before END OF TEMPLATE.
Renders the template using libraries like Jinja.
Puts the rendered template after END OF TEMPLATE.

Final Thoughts

My current plan may not be elegant, but it seems to meet my requirements more effectively than the other solutions.

Meanwhile, I'm still looking for new options. Please let me know if you know any.

VM Networking From Scratch

2025-08-22T01:01:00.005+02:00

Now that I've settled on my VM image pipeline, the next logical step is to tackle networking.

My Requirements

So far, I've been using QEMU's default user-mode networking. It's convenient for quick tasks, allowing for easy port forwarding, Samba shares, and DNS with just a few flags. However, this setup is ultimately insufficient for my needs for a couple of key reasons:

Security and Isolation: In the default user-mode setup, a VM can access the host's services via localhost. Worse, because it uses NAT, the VM can also access the host's entire LAN using the host's IP address. Ideally, VMs should have their own identifiable IP addresses, and more importantly, there should be strong network isolation between the host and the VMs.
Centralized Auditing: I want to audit all network traffic from my VMs through a centralized solution. This means I need a way to route all VM traffic through a single point of control.

Choosing the Right Tool

For most people, tools like libvirt or Incus are the best choice for this task. They are well-maintained, thoroughly tested, and have well-designed command-line interfaces that are less error-prone. I should probably just choose one of them and be done with it.

...except that I'm genuinely interested in learning the underlying building blocks and terminology. This is the main reason I chose to write QEMU scripts manually in the first place. Meanwhile, I find myself constantly referring to the documentation for these tools anyway when I'm studying security options.

Maybe one day, when I'm satisfied with my knowledge, I'll migrate my scripts to one of these excellent tools. But for now, let's learn by ~~suffering~~ doing.

Bridge + TAP

As many guides suggest, for anything beyond basic networking, the place to start is with a Linux bridge and TAP devices. A bridge acts like a virtual network switch, and a TAP device acts like a virtual network port for a VM to connect to that switch.

Thankfully, systemd-networkd makes this setup fairly easy. In the .network file for my bridge, setting IPv4Forwarding=yes and IPMasquerade=ipv4 saves me from writing custom nftables rules for NAT, which is a huge time-saver. QEMU also makes it simple to attach a VM's network interface to an existing TAP device.

To keep things tidy, I decided to automatically generate the systemd-networkd configuration files (e.g., .netdev and .network) directly from my VM configuration files. I save these generated files to /run/systemd/network/. This ensures I don't have to manually keep two sets of configurations in sync.

IP Addresses

The easiest way to assign IP addresses to VMs is to run a DHCP server on the bridge. Most standard cloud images, including bootc images, are configured to use DHCP by default.

However, I ultimately decided to use static IP addresses. Setting up a DHCP server securely, whether on the host or in a dedicated VM, takes some effort. Even with a DHCP server, I would likely configure static reservations to make it easier to write firewall rules to prevent IP address spoofing.

So, my process for each VM looks like this:

Generate a unique MAC address and a static IP address, and store them in the VM's configuration file.
Before starting the VM, generate a temporary systemd-networkd .network file that matches the VM's MAC address and configures its static IP, gateway, and DNS settings.
Pass these configuration files into the VM at boot time using systemd's network.* credentials feature.

This should work perfectly... right?

Wrong! I quickly discovered that CentOS does not ship systemd-networkd (1, 2, 3).

After looking through the official options for bootc images, I settled on using NetworkManager. This requires me to generate a NetworkManager keyfile and embed it into the container image. This isn't ideal because updating the network configuration requires rebuilding the image, which is slow. In the future, I might explore better options, such as:

Separating the Linux kernel from the image and booting it directly with QEMU, allowing me to pass network configuration via kernel parameters.
Using a different base image that includes systemd-networkd.

Inter-VM Traffic

By default, all VMs connected to the same bridge can communicate with each other freely. This isn't what I want; my goal is to enforce a "default deny" policy and only allow traffic that is explicitly permitted.

After some research with some help from AI, I learned a few key terms: port isolation, private VLAN, and proxy ARP. It turns out these concepts are perfect for my use case.

Here’s what I discovered when I put it into practice:

I started with a standard bridge and TAP setup, with host firewall rules in nftables that block all traffic. As expected, the VMs could not connect to the internet. However, they could still talk to each other. Why?

A quick debugging session with nft monitor revealed that packets traveling between VMs on the same bridge never hit my inet family firewall rules. This is because the bridge was forwarding the traffic at Layer 2 (like a real switch), so the host's Layer 3 IP-level firewall was never consulted. nftables has a bridge family specifically for filtering this kind of traffic.

Next, I enabled port isolation on the bridge. Now, even the bridge family rules couldn't see the packets between VMs. This confirmed that port isolation operates at an even lower level, preventing the bridge from forwarding frames between isolated ports altogether.

This gave me the perfect foundation. Now, if I want to allow two VMs to communicate, I have to do it explicitly. I have two main options:

Force Gateway Routing: I can remove the local subnet route inside each VM, forcing them to send all packets (even to other VMs on the same subnet) to the bridge's gateway IP address. The host's routing stack will then receive the packets, which can be filtered by my standard inet family nftables rules.
Use Proxy ARP: I can enable IPv4ProxyARPPrivateVLAN=yes on the bridge's network configuration. The host will then respond to ARP requests on behalf of the VMs. This tricks the VMs into sending all their packets to the host's MAC address.

Ultimately, both options achieve the same goal: they force Layer 2 traffic up to Layer 3, where it can be inspected by a central firewall. Option #2 is more elegant because it doesn't require custom network configuration inside the VMs. Option #3 seems less hacky.

Notes:

My initial assumption was that with Proxy ARP (Option #2), the traffic would be captured by the bridge family in nftables. This is incorrect. The ARP resolution happens at Layer 2, but the subsequent IP packets are routed, so they are captured by the ip or inet families.
Proxy ARP doesn't remove the need for a Layer 3 firewall. A malicious VM could simply add its own static routes (as in Option #1) to try and communicate directly. The key is to have a firewall at the gateway that inspects all traffic, ensuring that even if a VM tries to bypass the intended path, the traffic is still filtered. The main benefit of port isolation is preventing direct, unfiltered Layer 2 communication.

Outgoing Traffic

For controlling traffic leaving the host, I have a draft plan that provides strong isolation. The idea is to create a dedicated firewall VM.

On the host, I'll set up two bridges: bridge-internal and bridge-external.
All my regular VMs will be connected to bridge-internal. The host itself will not have an IP address on this bridge. This ensures the VMs cannot directly talk to the host. If needed I can set up SSH connection over vsock.
I will set up a special firewall VM that has two network interfaces: one connected to bridge-internal and the other to bridge-external.
The host's physical network interface will be connected to bridge-external.

With this setup, all outgoing traffic from the VMs must pass through the firewall VM, giving me a single place to manage all rules. It also isolates the host's network stack from the VMs by default.

For services, I can configure the firewall VM:

For non-HTTP services like NTP, I can set up forwarding or proxy rules.
For HTTP/HTTPS traffic, I can set up a transparent proxy using Nginx. Previously, I thought this would require a separate proxy configuration for each domain, e.g. dedicated proxy and DNS entry, but AI showed me a much better way:
- Nginx's ngx_stream_ssl_preread_module allows it to inspect the SNI (Server Name Indication) in the TLS handshake without decrypting the traffic.
- I can use firewall rules to redirect all outgoing HTTPS traffic from bridge-internal to this Nginx stream proxy.
- In the proxy, I can maintain a simple allowlist of domains and block everything else.

I plan to explore this design further. For example, is it better to use the host as the firewall? Or split the firewall services into multiple VMs? Could macvlan be useful here? These are questions for a future post.

Conclusion

In the end, I've replaced QEMU's basic networking with a much more secure, custom setup. Using a Linux bridge and port isolation, I can now force all VM traffic through a central firewall for inspection.

While it was more work than using a tool like libvirt, building this from scratch was a fantastic way to learn the fundamentals of VM networking and gain complete control over my environment.

Sending Emails with Curl: A Nifty Systemd Workaround

2025-08-17T20:02:00.003+02:00

I recently tried to create a simple systemd service to send an email notification, but my initial approach with mail and sendmail failed with a strange permission error.

My original service file looked like this:

[Service]
ExecStart=mail --subject=Subject recipient@example.com

The error message was a bit of a head-scratcher: warning: mail_queue_enter: create file maildrop/....: Permission denied.

A quick search pointed me to the cause: the postdrop binary has setgid. However, the systemd setting NoNewPrivileges=true prevents this.

While I hadn't explicitly used that setting, I was using DynamicUser=true, which implies and enforces NoNewPrivileges=true. This meant my service, running as a temporary user, couldn't get the permissions it needed to interact with the mail queue. Note that this implication cannot be disabled/overriden.

I wanted to avoid creating a new, dedicated user for this task. I realized that the problem was how mail and sendmail directly interact with the mail queue. The solution was to bypass that entire process and talk directly to the local SMTP server.

I didn't want to install another dedicated SMTP client. Fortunately, I learned that the curl can also act as an SMTP client! This command worked perfectly, sending the email by directly:

curl --url smtp://localhost:25 --mail-rcpt recipient@example.com --upload-file body.txt

mkosi: First Impressions

2025-08-14T22:06:00.001+02:00

I stumbled upon the Gentoo wiki page for systemd-nspawn, which in turn led me to nspawn.org, mkosi, and later systemd-sysupdate.

mkosi quickly caught my eye because it's almost exactly what I wanted to build myself, as mentioned in a previous post. So, I decided to spend my "sysadmin fun quota" on it.

Overview

mkosi is similar to docker build or podman build, but it's designed for creating full OS images. It focuses on development and testing. For example, much like nix-shell, mkosi can quickly launch a sandboxed shell with a specific distribution and selected packages installed. The systemd project itself uses mkosi for testing across different distros.

The re-introduction article is a great read.

Speed

Note that this is by no means a rigid benchmark.

My setup is an SSD with LUKS and an ext4 filesystem (without reflink support).

Building Container Images

mkosi is pretty fast. A simple mkosi command creates a fresh Debian image. I used the --incremental flag for subsequent builds.

First run: ~30s
Second run (after trivial changes): ~5s

Using mkosi -p systemd allows the container to boot (via systemd-nspawn -b), which adds only a few seconds to the build time.

Building VM Images

Building a VM image with mkosi --include mkosi-vm is a bit slower, likely due to the extra steps for installing a bootloader and kernel.

First run: ~1m 30s
Second run (after minor changes): ~30s

Comparison with bootc

I tried to build a fresh CentOS image using both tools.

mkosi --include mkosi-vm -d centos

Duration: ~1m 30s
Output disk size: 1.2 GB

podman pull quay.io/centos-bootc/bootc-image-builder:latest && \
podman run ... \
    quay.io/centos-bootc/bootc-image-builder:latest \
    --type raw ... \
    quay.io/centos-bootc/centos-bootc:stream9

Duration: ~4m 30s
Output disk size: 1.9 GB

Notes:

The bootc-image-builder was pre-pulled, and this time isn't included in the measurement.
The time to pull the base CentOS image is included.
I'm generating a raw image here instead of QCOW2.

Again, these numbers aren't directly comparable outside of my specific setup.

bootc-image-builder runs in a VM, while mkosi runs directly on the host.
centos and centos-bootc are different distributions, and their configurations (like installed packages) are also very different. This is obvious from the difference in their final image sizes.

Running Images with systemd-nspawn

I attempted to get unprivileged systemd-nspawn working but failed:

systemd-nsresourced.socket and systemd-mountfsd.socket must be running.
systemd-mountfsd complains that the image is untrusted unless it's signed or located in a trusted location.
I got stuck on another error.

Eventually, I resorted to using sudo systemd-nspawn -U, which worked well. The -b flag "boots" the image by running systemd/init as PID 1.

Running Images with QEMU

mkosi --kvm vm works nicely.

Notes:

Credentials are visible in the command-line arguments
I'm not a fan of all the default flags for QEMU, but mkosi provides many options for customization.

Observations, Thoughts and Concerns

mkosi is deeply integrated with systemd. Its configuration files are also following the systemd style: e.g. declarative, ini, drop-in overrides.
I wasn't able to test the performance benefits of reflink, because my filesystem doesn't support it and the disk images were small anyway.
I also wasn't able to test if SELinux works. Supposedly, it needs an extra flag in mkosi.conf and might be slow. On the other hand, it works out-of-the-box in bootc images.
I don't really miss Containerfiles much. I usually just need to copy files, and for my use case, a Containerfile would essentially just be running my scripts with bind mounts. Plus, I don't use many layers. But I might miss having an immutable Linux setup.
mkosi supports many popular distributions. while bootc only support Fedora/CentOS.
mkosi may add surprising modifications to the image:

mkosi doesn't use debootstrap. It actually used to depend on it, but that dependency was removed. Not sure if this approach is hacky.
mkosi may inject its own SSH server unit.

zvol doesn't seem very reliable, so I'll probably avoid using it for another few years.

Conclusion

mkosi is a very interesting tool. While I'm not ready to migrate my entire image-building pipeline yet, I might consider replacing my current LXC setup with it.

Rethinking My VM Image Pipeline

2025-08-12T21:33:00.009+02:00

Today, my pipeline regularly builds images for my disposable VMs. Here's the current process:

A dedicated builder VM reads Containerfiles for all VMs, including itself.
The builder VM uses podman build to create container images for all VMs.
The builder VM then uses bootc-image-builder to create disk images for all VMs.

This process works well, but it has a significant issue: the disk images aren't built efficiently. Unlike container images, which benefit from reusable, cacheable layers, disk images are always built from scratch. This leads to long build times and limited opportunities for data deduplication.

To address this, I've been exploring alternative options to improve the pipeline.

Disk Image Formats and Deduplication

My Current Format: QCOW2

I currently use QCOW2 with compression enabled. This format offers several features like snapshots, compression, and sparse files, which are useful when the underlying filesystem doesn't support them. However, if the filesystem does provide these features, QCOW2 doesn't offer many additional benefits over a simple raw disk image, at least for my use case.

Some notes:

Raw disk images are more transparent and widely supported by various tools. It's also much easier to deduplicate raw image files than compressed QCOW2 images. A QCOW2 image without compression should theoretically be similar to a raw image, but I haven't verified this.
The compression in QCOW2 is "read-only," meaning new writes aren't compressed. This isn't a problem for me because my VMs are immutable, so the images are rarely written to after creation.
bootc-image-builder actually builds the raw image first, before converting it into the QCOW2 format.

The Power of Deduplication

I expect deduplication to be highly effective in my setup because most of my disk images are very similar. There are a few ways to achieve this:

Filesystem Deduplication: This approach can be either online (e.g., ZFS) or offline (e.g., btrfs). The filesystem finds duplicate data blocks within files and removes redundant data from the disk. This is a general solution but doesn't necessarily speed up the initial build process.
Proactive Deduplication: This method is about building new images by applying small changes to an existing one. For example, you can "fork" an image using cp --reflink a.img b.img or qemu-img create -b a.qcow2 b.qcow2. Only the differences between the two images are stored on disk. This approach can significantly speed up the build process because you are not building from scratch, but it requires images to be built incrementally, not from a clean slate.

Exploring New Approaches

Bootc and In-Place Updates

I'm not currently using bootc images in their intended way. bootc is designed so you build a single disk image once and then update it in-place via a container registry.

I've considered two ways of leveraging this:

I could trust the VMs to update themselves.
I could maintain a "trusted base image" and follow this process:
- Create a base disk image using bootc. This image is only used for building other images and never for running services.
- To create the disk image for a specific VM, say VM X, I would first fork the base image using cp --reflink or qemu-img create -b to create X.img.
- I would then boot a VM using X.img and have it upgrade itself using VM X's specific container image. This container image could either be served from the builder VM via a server or a mounted directory, or it could be built locally within the forked VM, potentially using shared layers from a mounted cache.

This process seems workable, but it's overly complex for my taste. It involves running VMs during the build process, which would require a significant amount of scripting.

Plain Disk Images and In-Place Updates

This is similar to the bootc approach but uses standard raw disk images. Again, I could set up temporary VMs for the build process, but instead of relying on bootc's update mechanism, I would need custom scripts. This starts to resemble tools like cloud-init or Ansible.

A key benefit here is that a VM isn't a strict dependency. I could use something like systemd-nspawn to directly modify the disk images in-place, which would simplify scripting and make the process more reliable. I did attempt this with bootc images, but they don't work well with systemd-nspawn out of the box because the partitions lack the UUIDs that systemd-nspawn requires.

Final Thoughts

Ultimately, I haven't found a truly satisfying improvement to my current build process. While some of these approaches could theoretically improve build times and reduce disk usage, they also make the build pipeline more complicated and less reliable. At this moment, I don't think the trade-off is worth it.

For now, I'll probably just experiment with deduplication on ZFS and reflink on XFS. I noted that ZFS doesn't support reflink (zfs_bclone_enabled) by default, so that's a small hurdle.

This exploration has been an interesting learning experience. I've revisited/discovered some relevant tools:

libvirt
incus
systemd-nspawn
cloud-init
ansible
systemd-volatile-root.service

Sometimes, when I'm writing my own scripts, I feel like I'm building a slimmed-down version of these tools myself. However, I'm not yet convinced that it's the right time to fully switch to them.

[UPDATE]: I learned that this process is called Golden Image and Phoenix Server.

A Practical Guide to Passing Secrets to VMs

2025-08-06T22:09:00.007+02:00

The central question is: how do you manage secrets like SSH keys, API keys, and passwords for disposable VMs? 🤷‍♂️

Let's establish some ground rules for this scenario. Suppose I want to pass an API key to the VM chimera, which is run by the chimera-runner user on the host. My security requirements are:

On the host, only root and chimera-runner should have access to the secrets.
In VM chimera, only root and relevant service users should have access to the secrets.
No one from other VMs, including their root users, should have access to VM chimera's secrets.
The guest VMs themselves are not trusted.

The bootc documentation on this topic is very informative.

On a high level, there are a few ways to achieve this.

1. OEM Strings / Firmware

QEMU can pass data to a VM via SMBIOS OEM strings (-smbios) or firmware configuration (-fw_cfg). Notably, both methods are supported by systemd-creds using special keys.

This approach is practical for small pieces of data, like an individual password or an encryption key. It's not a new idea and was discussed years ago.

However, there are a few caveats:

Size Limits: The QEMU manpage states that the total size of all SMBIOS tables is limited to 65535 bytes. While not explicitly defined, fw_cfg is also intended for small amounts of data.
Key Length: The maximum length of a fw_cfg key name is 55 characters. If you use it with systemd-creds, a special prefix is required, making the available space for your key name even shorter.
Security Risk: If you pass data as a string (e.g. -fw_cfg string=secrets), the secret becomes part of the QEMU process's command line, which is visible to all users on the host!
Bugs: You can provide a file to SMBIOS using -smbios path=filename, which avoids exposing the secret on the command line. However, this feature is affected by a bug that is still present in Debian Bookworm.
Accessibility: Data passed via firmware appears in the guest's /sys/firmware directory, which cannot be mounted as a typical block device.

For these reasons, I generally use -fw_cfg file=filepath for passing small secrets and leverage systemd-creds within the guest whenever possible.

I might switch to -smbios path=filename later, when the bug is fixed.

2. Network Share

This is probably the most common way of sharing files between a host and its guests. The host sets up a file-sharing service, and the guest connects to it to access the files.

Pros: Changes on the host are reflected in the guest immediately, although this might not be important for static secrets.
Cons: The server on the host needs to authenticate clients (the VMs) to ensure one VM cannot access another's secrets. The guest VM also needs to set proper file permissions internally.

Common options include:

QEMU's built-in SMB server: Easy to set up, but the guest share is accessible to everyone in the guest. A non-root user can access the content using userspace tools.
SMB Server: For each VM, you must create a dedicated user/password pair. This password must then be passed securely to the VM (using another method).
NFS Server: NFS authenticates clients by IP address and trusts the client's root user. This is risky because a compromised VM could spoof its IP address. An extra authentication layer, like WireGuard, might be necessary.
SSHFS: Each VM needs a dedicated SSH key pair stored securely. The host can use a standard SSH server configuration.

While these options are viable, I find them less than ideal. They require significant effort to set up correctly, and the servers add extra maintenance overhead. Furthermore, SSHFS appears to be no longer actively maintained.

3. Filesystem Passthrough

This method is similar to a network share but is optimized for VM environments. Instead of the network stack, it uses a more direct channel to expose a host filesystem to the guest, which is generally faster.

Common options are:

9pfs: Easy to set up, but my guest OS (centos-bootc:stream9) lacks the necessary kernel support, and I prefer not to compile custom kernels.
virtiofsd: This is a popular and high-performance method, but it requires shared memory between the host and guest, plus an extra daemon running on the host.

Unfortunately, neither of these options worked for my specific setup.

4. Credential Fetcher

With this approach, the guest fetches credentials as needed, typically during boot. This can be implemented easily in the guest, for example, by using scp to copy secrets into a ramfs mount.

However, this requires setting up a server (e.g. SSH) on the host, which I wanted to avoid due to the added complexity and maintenance.

5. Embed in Container Image

It's possible to generate and embed secrets directly into the bootc container image during the build process.

An interesting variation is to encrypt the secrets, embed the encrypted data in the image, and pass the decryption key to the VM using another method (like an OEM string). You could use systemd-creds for this and even bind decryption to a virtual TPM simulated by QEMU. However, as noted in this discussion, this might not be the intended use case for systemd-creds.

While this works in theory, it doesn't offer significant benefits for my workflow and feels tedious to implement.

6. Disk Images

Any file can be used as a raw disk image for the VM, the guest just directly read the device for the data, without mounting a filesystem. Note that there may be issues if the file size is not aligned with the block size.

A more practical approach is to create a proper disk image containing the secrets and attach it to the VM. Any filesystem supported by the guest OS will work, but the ideal choice would have these characteristics:

The disk image can be created without root privileges on the host.
The filesystem is optimized for read-only data.

I found three practical options:

EROFS: A modern, read-only filesystem that supports volume labels.
SquashFS: mkfs.squashfs is very flexible; its "pseudo file" feature lets you create files directly from command output.
ISO 9660 (CD/DVD images): Universally supported but less flexible.

I plan to use EROFS mainly because it supports volume labels. I cannot guarantee that the disk order will be consistent across all VMs. Therefore, the guest needs a reliable way to identify the secrets disk, and mounting by label is the easiest solution.

Conclusion

After evaluating the options, I settled on the following combination:

OEM strings for small, individual secrets (like a decryption key).
Read-only EROFS disk images for larger sets of secret files.
QEMU's built-in SMB server for sharing encrypted data blobs, where the decryption key is passed separately via an OEM string.

Keep in mind that this solution is tailored to my specific use case, which has the following constraints:

The guest VMs are not fully trusted.
I want to minimize setting up and maintaining complex services on the host.
Automating the setup via scripting before a VM starts is acceptable and even preferable.
The solution must scale easily to many different VMs.

Backing Up VM Disk Images

2025-08-01T23:23:00.002+02:00

[Update] I managed to work out the AppArmor profile, and decided to go with guestmount for now.

I am setting up a maintenance pipeline for my virtual machines.

The pipeline has two main routines:

The BACKUP routine: Every day, this routine shuts down each VM, backs up the data in /var, updates the VM's disk image if a new version is available, and then restarts it.
The BUILD routine: Every week, this routine uses a special builder VM to create new disk images for all VMs.

There is a scheduling conflict with the builder VM: the BACKUP routine needs to shut it down, while the BUILD routine needs it running. To resolve this, I merged both into a single set of systemd services that runs daily. The BUILD routine starts automatically when the builder VM starts, at the end of the BACKUP routine. The builder VM's systemd unit has an ExecCondition= property, which is skipped 6 days a week.

Surprisingly, the most difficult part of this pipeline was not the scheduling, but the backup process itself.

There are two general approaches to backing up a VM: backing up the entire disk image or backing up the files directly from the filesystem.

Backing Up Disk Images

Backing up a disk image is straightforward because disk images are just regular files. However, this method is often inefficient:

Deduplication may not work well if the disk image is compressed.
Incremental backups are not natively supported.
The entire disk image must be backed up, including unused and deleted data.
You cannot easily choose which specific files to include or exclude from the backup.

There are some ways to improve this approach:

For deduplication: I can decompress the disk image and pipe the data stream directly to the backup software without saving the decompressed file.
For incremental backups: I can create snapshots and back up only the differences. I would also need to regularly merge the snapshots.

QEMU supports incremental backup for a running VM. Relavent article: 1, 2.

To reduce backup size: I can defragment, shrink, wipe, and sparsify the disk image (for example, with virt-sparsify) before backing it up.
To exclude specific files: I can put the files or directories that I don't want to back up on a separate disk image.

Backing Up Filesystems

One can back up files from inside the VM. It is also possible to mount the disk image from the host system when the VM is shut down.

Raw disk images can be mounted directly using loop devices.

Qcow2 images have several options:

`qemu-nbd` can expose an image as a block device (e.g., /dev/nbd*), which can then be mounted. This requires the nbd kernel module and root access. The qemu-nbd man page warns that this may not be suitable for untrusted guests. To back up multiple VMs, I would also need a way to find an available /dev/nbd* device.

The block device can be exported without root, there are tools like `qemu-nbd`, `nbdfuse` and `qemu-storage-daemon`. This article is worth reading.
`qemu-nbd` also supports exposing an internal snapshot of a qcow2 image.

`guestmount` can mount a disk image without needing root. It uses QEMU and FUSE. While it works, it can be slow, and creating a secure AppArmor profile for it is difficult due to its complexity.

My Thoughts

All of these options have trade-offs between security, performance, complexity, and flexibility.

In my case, my priorities are:

Security: I do not trust the guest VMs. This means the guest should not connect to the host, and the host should not load the guest's disk image using a kernel module. While I could move the backup logic to a separate VM, this would add a lot of complexity.
Simplicity: I want a simple workflow that is easy to maintain and secure. I prefer to avoid writing complicated AppArmor profiles that are difficult to update.
Performance: I don't need the backup to be super fast, meanwhile I don't want to keep a VM shut down for too long.
Space Efficiency: I don't have a large amount of data, so disk space is not a major concern.

Considering these factors, I prefer backing up the entire disk image. Recall that my root filesystem is created from a Containerfile, and /etc is transient, so I primarily need to back up /var.

For now, I plan to use qcow2 images with compression. I will decompress the disk image and pipe the data to the backup software.

In the future, I might explore some optimizations:

Using a raw disk image on a ZFS host filesystem with compression and possibly deduplication enabled.
Taking a snapshot (either qcow2 or ZFS) and backing it up. This would allow me to restart the VM without waiting for the backup to finish.

GNU Stow

2025-07-28T22:00:00.001+02:00

Just learned about GNU Stow, which is a tool for managing symlink farm.

Basically the idea is to store all files in one place, then create symlink all around the system pointing to your files.

There are various use cases, like dot files and installing/uninstalling packages. But I mostly use it for tracking system config files, similar to how NixOS works. In fact I wrote my own scripts with "cp -rs", but GNU Stow works much better.

Disposable VMs for Home Lab Security and Reproducibility

2025-07-27T20:54:00.010+02:00

Today, various services (native, LXC, Docker) are running on my server. I'm mostly happy with the setup, but I decided to revisit my server's defenses under the assumption that a remote attacker or malicious code could compromise my services. A service might break out of its container or even gain root privilege.

VMs are a better security boundary than containers; they can limit the damage if an attacker gains root privilege. I cannot afford to run a dedicated VM for each service, so I will need to carefully group the services and run a dedicated VM for each group. Each group should be carefully designed based on the data accessed and the features/capabilities required. For example, some VMs may have access to my photos, while others may not have network access.

The Goal

There are two particular issues I want to address:

First, I want VM images to be easily reproducible, which makes backup and restore trivial. NixOS and GNU Guix System are great examples, where you only need to back up the configuration file. However, I don't really like them because of their domain-specific language/design.

Second, I want to seal the system as much as possible. Even a compromised root user inside a VM should not be able to permanently infect the VM. Many so-called "immutable" Linux distributions are not truly 100% immutable. Often, they just mean a read-only /usr. Some can be easily broken via `mount -o remount,rw`, and most of them allow self-upgrade, meaning a malicious root user can still inject code via "upgrade and reboot."

The Approach

I use bootc containers. This allows me to build the whole system with standard scripts, and it offers the standard "immutability."

Furthermore, I run QEMU with `--no-reboot --snapshot`, which means the system cannot update itself even with root privilege.

Lastly, I'll regularly build new images and restart the VM to pick up the latest security fixes.

This approach is essentially managing VMs like containers. It's not a new idea; frood and gokrazy are good examples of this principle.

On a side note, I also plan to learn more about KubeVirt and Nix VMs. Especially, I like the idea (from NixOS) that the guest can directly use the store from the host.

Notes about QEMU

Permanent machine-local data is stored in /var, which is put into a separate disk image.

Secrets are sent to QEMU via systemd credentials.

I tried virtiofsd, but didn't like it. I ended up with Samba anyway. Maybe I'll revisit virtiofsd later.

To shut down the VM (e.g., via systemd), I created a special admin user with special privilege defined in the sudoers file, so that I can run `ssh admin@vm sudo poweroff`. The SSH key pair is regenerated before each VM boot. Related: In a systemd unit, ExecStop= does not have access to LoadCredential.

I use `-chardev socket,logfile=...` and `-serial` so that the systemd logs are not filled with console output, and I can view or attach to the serial console later.

I plan to learn more about virtio-balloon and pmem later.

Conclusion

I find it very beneficial to deploy VMs. It allows me to shrink and harden the host OS (e.g., disable unprivileged user namespaces), and it allows me to design fine-grained access control.

Next, I'll start investigating how to organize the containers inside VMs.

SELinux and useful systemd components

2025-06-14T08:47:00.006+02:00

Just learned about a few interesting and useful stuff, when playing with bootc:

systemd Components

systemd-tmpfiles and systemd-sysusers allows managing files and users in a declarative way. Originally I learned about this for building bootc images, but later I realized that they are also very useful on Debian.

I learned systemd-credential as a way of passing ssh authorized keys to a QEMU VM, but after reading more, I realized it can be used in other interesting ways. My favorite one is with LoadCredential=, I can run a script with DynamicUser=yes and the script can access some root-only secrets.

I finally decided to migrate from cron to systemd-timer. systemd-timer is more interesting and handy than expected, and the migration process is less painful than expected.

SELinux

Actually I heared about SELinux many years ago. Over the time I just know SELinux as "something about security, similar but more complicated to AppArmor".

Recently I got to learn more about it:

- https://people.redhat.com/duffy/selinux/selinux-coloring-book_A4-Stapled.pdf

- https://www.youtube.com/watch?v=_WOKRaM-HI4

- https://developers.redhat.com/articles/2025/04/11/my-advice-selinux-container-labeling#

- https://docs.podman.io/en/v5.0.3/markdown/podmansh.1.html

- https://reintech.io/blog/securing-debian-12-with-selinux

- https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/using_selinux/assembly_using-multi-category-security-mcs-for-data-confidentiality_using-selinux

- https://www.redhat.com/en/blog/how-selinux-separates-containers-using-multi-level-security

- https://www.redhat.com/en/blog/why-you-should-be-using-multi-category-security-your-linux-containers

- https://wiki.gentoo.org/wiki/SELinux/User-based_access_control

With more knowledge about it, I feel that I like it much better than before:

- I actually have written my own tool to verify file permissions, in a similar fashion (regex -> permissions)

- rootless docker and apparmor didn't work very well together in my case. rootless podman and selinux might work better together.

- :Z is pretty nice for containers: https://github.com/containers/container-selinux

I probably will start trying SELinux. With more confidence I might eventually enable it on Debian and replace AppArmor.

First 3 Days with bootc

2025-05-10T22:23:00.004+02:00

I decided to spend some time playing with bootc. Mostly I'm inspired by the following articles:

CoreOS + native container
Hand-on demo (the last video), build bootc and auto update from registry
bootc desktop
bootc for homelab

Day 1

To install bootc in a VM I need an image. bootc-image-builder requires root and I don't want to run this on the host. So I chose CoreOS as the inital system and installed it to QEMU.

I thought it is a great idea to share a folder from host to guest as podman container storage. However, it was not as smooth as I had expected:

virtiofsd on Debian is too old, so I set up NFS.
rootless podman doesn't work well with NFS.
rootfull podman complains upstream fs of overlayfs missing features, the performance was terrible.

I gave up. I guess I'll just use the CoreOS disk, whose size is 10G, not enough.

Day 2

I didn't find a way of resizing a qcow2 image online. On the other hand I figured maybe I don't need build a disk image after all. CoreOS is already based on ostree, maybe I can use `bootc switch`. This is essentially the same approach as in the first blog post.

`bootc switch` just works, it can reboot, but I cannot login (ssh or local). Fortunately (and quite nicely), I can rollback ostree even without logging in, because I can do that with grub.

I suspect it is because some files are overriding the files in the image.

Day 3

I learned that QEMU has builtin samba support, which is much easier to use than NFS.

Eventually I found that it was SELinux that has been messing up. With `restorecon -R` I could login from QEMU terminal, but not ssh. After logging in, `bootc status` threw an error about /boot, so I guess I needed `bootc install` afte rall.

So I just went back to the original solution, just resize the CoreOS image and build bootc image inside CoreOS. It worked this time.

Now I need to complete the loop, the new bootc OS should build itself and automatically update itself. And a few more things to fix:

- /etc/fstab does not work if modified when building the container, I need to create systemd mount files for mounts

- /etc/hostname does not work if modified when building the container, I need to set it after each boot

- Unlike `bootc install`, `bootc-image-builder` does not provide flags to override the bootc repository, so I need to set it after boot

- Transient etc sounds like a good idea, but I'll need to manually configure /boot. I'd like to enable it when the official doc explains more details.

Conclusion:

bootc works and its quite fun. But unfortunately I didn't find a way of actually using it in production yet:

I could put it into a VM, but sharing files between host and VM is not pretty at the moment (on Debian stable)
I don't trust it as my main server yet, and I don't have other machine to which I can install bootc bare-metal.

I think later I'll spend more time trying to tinker with it.

UID and GID: The New Order

2025-05-05T22:50:00.003+02:00

When I have important data on a device, I back it up to my server using dedicated user accounts. The other day, I checked /etc/passwd on my server and found entries like this:

some-backup-user1:x:1003:1004:...

some-backup-user2:x:1004:1007:...

A few inconsistencies immediately bothered me:

UID/GID Mismatches: Many users have UIDs that don't match their primary GIDs. While this technically works and might seem like just an aesthetic concern, I realized that UIDs and GIDs are crucial metadata. I need to preserve them accurately for future system migrations to maintain correct file ownership.ID Ambiguity:
ID Ambiguity: The same number (e.g., 1004) could represent a User ID for one account and a Group ID for a completely different group. This overlap is a recipe for mistakes during administration tasks if I'm not paying close attention.
Lack of Structure: User and group accounts created for very different purposes – regular logins, backup processes, container management, specific ACLs – were all jumbled together in the same ID range. This made management and auditing more cumbersome than necessary.

So, I decided it was time for a cleanup. My goal was to reorganize these user and group IDs to achieve clarity and predictablity, aiming for a system where I could infer the purpose of an ID even without direct access to /etc/passwd or /etc/group.

My Strategy

To achieve this, I established the following strategy:

Purpose-Based ID Ranges: Users and groups serving similar functions will be grouped together by assigning them IDs within dedicated numerical ranges. This makes it easier to understand the role of an account at a glance. For exampe:

1000-1999: Backup-related users and groups.
2000-2999: Groups specifically for managing ACLs.

UID/GID Correspondence: If a number X is used as both a UID and a GID, then GID X must be the primary group for the user with UID X. Unrelated users and groups never share the same ID number.
Allocation Within Ranges: Within each designated range, users and their primary groups start from the lower end, secondary groups starts from the higher end.

Example: backup-user1 has 1000:1000, while a group backup-users has GID 1999.

Options for Managing IDs

I considered several approaches to implement this strategy:

Manual Scripting: This would involve carefully crafting scripts using useradd, usermod, groupadd, and groupmod. However, this approach is fraught with risk – a single typo could cause significant problems. It's also labor-intensive to get right and tedious to maintain. I quickly ruled this out.
Virtual Machine with cloud-init: Using a VM combined with cloud-init offered a more structured way to script user/group creation via its built-in directives, executing reliably during boot. This reduces some risks compared to updating an existing system but it introduces the overhead of managing a VM (file sharing, updates, resource consumption), which I also wanted to avoid.
Virtual Machine with NixOS/Guix System: These operating systems offer truly declarative user management, which is very appealing. While I run simple NixOS instances, fully embracing either OS just for UID/GID management felt like overkill and required a significant learning investment. Plus, this option still carried the VM overhead.
Using sysusers.d: While searching for "declarative user management" solutions, I discovered sysusers.d. This systemd mechanism uses simple configuration files (like fstab) to declare users and groups and their properties (UIDs/GIDs). Systemd ensures these users/groups exist on boot. Crucially, it was already supported and installed on my Debian server. This offered a declarative, script-free, VM-free solution integrated directly into my existing OS – making it the clear choice.

The Migration

For each user and group, I added a corresponding line to a configuration file within /etc/sysusers.d. All IDs are manually allocated. Then I deleted the old users one by one and restarted the systemd-sysusers service. Note that the service does not touch existing users or groups.

After that I needed to migrate the permission of existing files. This proved trickier than I had expected:

chown has the nice `--from=OWNER:GROUP` parameter. The man page says "Either may be omitted, in which case a match is not required for the omitted attribute." However:

`chown --from :group` works
`chown --from user` works
`chown --from user:` doesn't work.

`chown user:` actually means `chown user:user`, i.e. it also updates group, but `chown user` updates only the owner
`setfacl` by default updates the mask, so `setfacl -m g:group:rx` actually made all my files executable! I had to rollback a ZFS snapshot and use `setfacl --no-mask` instead.

Finally, I also learned about `pwck -s` and `grpck -s` to improve the quality of my life.

What's Next

I plan to apply similar principles to my container setup. Specifically, I want to create dedicated users for each rootless container. And I will have to figure out:

How to manage subuid and subgid ranges in a structured, preferably declarative, way.
Choosing between docker and podman.
Whether to use VMs. If so, which OS to use, e.g. NixOS, GuixSystem, CoreOS.

Exploring Immutable Distros and Declarative Management

2025-04-18T12:17:00.008+02:00

My current server setup, based on Debian Stable and Docker, has served me reliably for years. It's stable, familiar, and gets the job done. However, an intriguing article I revisited recently about Fedora CoreOS, rpm-ostree, and OSTree native containers sparked my curiosity and sent me down a rabbit hole exploring alternative approaches to system management. Could there be a better way?

Core Goals & Requirements

Before diving into new technologies, I wanted to define what "better" means for my use case:

The base operating system must update automatically and reliably.
Hosted services (applications) should be updatable either automatically or manually, depending on the service.
Configuration and data files need to be easy to modify, and crucially, automatically tracked and backed up.

Current Setup: Debian Stable + Docker

My current infrastructure consists of several servers, all running Debian Stable.

System Updates are andled automatically via unattended-upgrades.
Services consist of a mix of native Debian packages and applications running in rootless Docker containers. Docker images are updated either periodically via scripts or manually when needed.
Config and data files are backed up automatically using rsync, but tracking which files need backing up is manual. This is my main pain point. Every time I add or modify a service or configuration file, I have to remember to update my backup scripts.

While this setup works, it's not perfect:

The manual tracking of configuration files is error-prone and tedious. I like how systems like NixOS manage the entire system declaratively from configuration files, automatically tracking changes. However, I have reservations about NixOS's learning curve and ecosystem, and I generally prefer sticking to more "traditional" distributions like Debian if possible.

Some services are Internet-facing. While rootless Docker improves security over rootful Docker, it's not a full security boundary. Moving these services into dedicated VMs could offer better isolation.

Fedora CoreOS + OSTree Native Container

Fedora CoreOS is an automatically updating, minimal, monolithic, container-focused operating system. Key concepts here include:

The core OS is largely read-only, making updates safer and more predictable (atomic rollbacks).
Configuration is applied on the first boot using Butane/Ignition. This is great for initial setup but less ideal for ongoing configuration changes.
rpm-ostree allows layering additional RPM packages onto the immutable base image. While OSTree tracks file changes, rpm-ostree itself doesn't inherently provide a declarative way to manage the list of installed packages or track arbitrary configuration file modifications in a user-friendly, declarative manifest like NixOS does. It knows files changed, but not necessarily why in a structured way (e.g., "/usr/bin/vim was added" vs "package vim was installed").
OSTree Native Containers: The article that inspired me highlighted using OSTree's ability to pull container images as system updates. The workflow looks like this:

Define a system image using a Containerfile/Dockerfile, starting from a CoreOS base.
Build this definition into an OCI container image.
Install a standard CoreOS, using a minimal Butane config that tells rpm-ostree to "rebase" onto your custom container image.
To update the system or add packages, modify the Containerfile, rebuild the image, push it to a registry, and the CoreOS systems will eventually pull and apply it as their next OS update.

This approach essentially creates a custom, version-controlled OS image. Variations include building the image via CI/CD (like quay.io), building locally, or even using coreos-assembler with a lower-level "treefile" manifest.

Bootable Containers

Emerging from the CoreOS world is Fedora Bootc. This seems specifically designed for use cases requiring more customization than standard CoreOS allows. Instead of Ignition and rpm-ostree, bootc directly manages bootable container images derived from a Containerfile. You essentially build your entire OS, including customizations, as a container image, and the system boots directly into it and updates by pulling new image versions. This feels conceptually cleaner and more aligned with the goal of a declaratively built system image.

Cloud-init

cloud-init is the de facto standard for bootstrapping cloud instances across various Linux distributions. Like Butane/Ignition, it runs on first boot, but critically, it can be re-run on subsequent boots. This makes testing configuration changes much easier, as you don't necessarily need to re-image the entire machine for every small tweak. It's widely supported but focuses more on initial setup and less on managing the entire OS lifecycle declaratively like the OSTree/Bootc approaches.

cloud-init is a popular tool to bootstrap linux machines for cloud. So it is similar to butane for CoreOS. However, cloud-init can be re-run, which means testing a simple change does not require re-imaging the machine.

Guix

Guix System is the GNU project's take on a declarative operating system, similar in principle to NixOS. It uses Guile/Scheme for its configuration language and boasts a clean command-line interface. While appealing from a purity perspective, its smaller community and adoption compared to NixOS make it a less pragmatic choice for me currently. Like NixOS, it can build VM images and mange VMs.

Conclusion and Thoughts

Conceptually, Fedora Bootc is the most compelling alternative I found. It directly addresses the desire to build a customized, yet manageable and updatable, system image using familiar container tooling. The main drawback? It's very new, lacks widespread adoption, and crucially, there's no "Debian Bootc" yet. If a stable, Debian-based bootc implementation existed, I'd likely jump on it.
Cloud-init is mature, widely adopted, and works across many distributions (including Debian). It could improve my bootstrapping process, but it doesn't solve the core desire for managing the entire system state declaratively post-install.
CoreOS/OSTree: While powerful, the standard CoreOS workflow with rpm-ostree layering feels slightly less integrated than bootc for building a heavily customized system declaratively. The native container approach is interesting but adds complexity.
NixOS/Guix/Ansible: These are powerful, but represent a significant shift in tooling and philosophy. They feel like a larger commitment than I'm ready for, especially given my preference for sticking closer to traditional distribution paradigms if possible.

For the time being, I'll likely stick with my current Debian + Docker setup. On the other hand, I might as well start a VM and try some options for better understanding.

Qubes OS: First Impressions

2025-04-12T21:57:00.000+02:00

A few days ago, while browsing security topics online, Qubes OS surfaced—whether via YouTube recommendations or search results, I can't recall precisely. Intrigued by its unique approach to security through compartmentalization, I delved into the documentation and watched some demos. My interest was piqued enough that I felt compelled to install it and give it a try firsthand.

My overall first impression of Qubes OS is highly positive. Had I discovered it earlier, I might have reconsidered starting my hardware password manager project.

Conceptually, Qubes OS is not much different from running a bunch of virtual machines simultaneously. However, its brilliance lies in the seamless desktop integration and the well-designed template system, making it far more user-friendly than a manual VM setup. I was particularly impressed by the concept of disposable VMs for temporary tasks and the clear separation of critical functions like networking (sys-net) and USB handling (sys-usb) into their own isolated VMs.

Although the default Qubes OS environment is Fedora-based, it doesn't feel drastically different from my experiences with Arch or Debian. This reinforces my feeling that the core differences between many Linux distributions are shrinking. It's also a plus that Qubes officially supports templates for Debian and other systems, offering valuable flexibility. (While exploring Qubes, I noted other compartmentalization-focused OS projects exist, like Spectrum OS and RancherOS, though I haven't investigated them yet.)

After using the system for just a few days, I can already see how its compartmentalized approach could be incredibly beneficial for managing different aspects of digital life securely. However, despite my enthusiasm, I'm hesitant to adopt Qubes OS as my daily driver just yet. I encountered a few practical hurdles:

Secure Boot Support: Qubes OS doesn't currently support Secure Boot. This is inconvenient for my dual-boot setup with Windows, as it requires toggling the setting in the BIOS every time I switch operating systems. According to a recent talk, support might arrive with version 4.3, which is encouraging.
Anti Evil Maid (AEM): While AEM is an excellent security concept, its current implementation has significant restrictions that make it difficult to use easily on many systems. My current workaround is just keeping the boot partition on a separate USB drive.
Backup Workflow: The built-in backup tool enforces encryption. While I understand the security rationale, this complicates my preferred strategy of frequent (e.g., hourly) incremental backups to a trusted location. Although tutorials exist for bypassing this, native support for optionally disabling encryption (perhaps with strong warnings) would be a welcome convenience.
Bluetooth: This seems to be a minor point, but Bluetooth isn't supported out-of-the-box and requires additional setup to get working with peripherals.
GPU Passthrough: Setting up GPU passthrough for performance-intensive applications (like gaming) appears non-trivial. This limitation means I'll likely need to keep my Windows installation as a dual boot for gaming purposes.

In conclusion, Qubes OS is a fascinating and powerful operating system with a unique and compelling security architecture. I'm genuinely impressed with its design and potential. However, the current practical limitations prevent me from making it my primary OS at this time. I'll definitely be keeping a close eye on its development, and may revisit it as these areas mature.

Cardputer as a Hardware Password Manager

2025-03-18T23:49:00.002+01:00

For the past month, I've been prototyping my own hardware password manager using the Cardputer, a compact device that's surprisingly perfect for the task.

I learned about the Cardputer while searching for the ideal hardware to build a password manager. It's essentially a microcontroller (the ESP32-S3) packed with a screen and a keyboard. What really sold me on it were these features:

Direct Interaction: I can interact directly with the device – typing my master password, searching for credentials, and confirming password entry, all on the Cardputer itself.
USB Keyboard Emulation: The device can seamlessly emulate a USB keyboard, allowing for both manual and automatic password entry into other devices.
Hardware-Accelerated Crypto: The ESP32-S3 boasts hardware acceleration for AES and SHA operations, crucial for secure password management.
Secure Element Integration: The Cardputer supports an optional ATECC608B secure element (available as an official accessory), providing additional cryptographic capabilities, secure key storage, and true random number generation.

Taming the Cardputer

The official development framework is ESP-IDF, but after some research I decide to use esp-hal. I took it as a nice opportunity to learn Rust and embedded systems.

My journey with esp-hal quickly turned into a hands-on crash course in embedded systems development. I wasn't just writing high-level application code; I was crafting drivers from the ground up.

This meant diving into the specifics of the Cardputer's hardware. I spent most of my time implementing drivers, not quite at the level of manipulating memory addresses directly, but still very much involved in the low-level details. This included tasks like scanning the keyboard's button matrix and translating physical presses into logical key events, as well as working with communication protocols like I2C and SPI to interact with peripherals.

Simultaneously, I delved into the world of cryptography, deepening my understanding of common algorithms and concepts like AES modes of operation, key derivation functions (KDFs), and secure hashing. This project became a fantastic opportunity to connect the theoretical aspects of cryptography with practical implementation.

This project also sparked an exhilarating, and at times challenging, adventure in learning and applying Rust. I deliberately chose the no_std and no_alloc path, embracing a truly bare-metal approach. This decision plunged me into the deep end of embedded Rust, where every byte of memory matters. It was both a blessing and a curse: while it demanded careful resource management, with everything either stack-allocated or statically allocated, it also provided unparalleled control and efficiency. Along the way, I encountered some of Rust's more… idiosyncratic features, wrestling with async function traits, navigating the intricacies of generic types, and occasionally resorting to seemingly superfluous .map(|x| x) closures to appease the borrow checker and its lifetime requirements.

Securing the Secrets

The prototype is functional. While the source code isn't ready for release yet, I'd like to share some of the key design decisions.

For data serialization, I chose FlatBuffers. Its zero-copy nature is particularly well-suited to the no_alloc environment of this project, minimizing memory overhead. The main database structure is inspired by KeePass. Note that the database is read-only on the device; entries cannot be added, modified, or removed.

The database is encrypted using AES-GCM-SIV. The encryption key is derived via HKDF, with the input keying material (IKM) composed of three distinct components:

User's Master Password: The output of PBKDF2 applied to the user's master password, using a randomly generated salt.
eFuse Key in ESP32S3: The result of HKDF-Expand using a pre-defined message, keyed by random bytes securely burned into an ESP32-S3 eFuse. This ties the encryption to the specific hardware.
ATECC608B Private Key: The output of HKDF using a pre-defined message, keyed by a shared secret. This secret is established through ECDH key exchange between the ECC key stored securely within the ATECC608B and an ephemeral key pair generated by the encryptor (e.g., a host computer or another device).

Furthermore, when either or both of the hardware-based secrets (eFuse and ATECC608B) are enabled, they also contribute to individual password encryption. A separate key is derived from these secrets, using the password's sequence number within the database as the IKM for HKDF. This approach, inspired by KeePass's in-memory protection, provides an additional layer of security, though its benefit in this specific embedded context might be less pronounced than in a traditional software environment.

Okay, I might have gone a little overboard with the encryption. The eFuse and ATECC608B key derivation steps are probably not strictly necessary. But hey, it was fun to build! And even though secure elements aren't magic security shields (they do have known vulnerabilities), they make it way harder for someone to steal your passwords.

Final Thoughts

The Cardputer has proven to be a surprisingly capable platform for this secure password manager project, though not without its limitations.

What I liked:

Affordability: The Cardputer's low cost is a major advantage, making this project accessible.
Well-Suited Hardware: The hardware feature set is almost ideal – providing all the necessary functionality (display, keyboard, secure element, connectivity) without unnecessary extras.
Small Attack Surface: The firmware footprint is relatively small (~300KB), minimizing the potential attack surface.
Fast Boot Time: The device boots up quickly, making it convenient to use.
Rust Ecosystem: The Rust ecosystem, with its package manager (cargo) and readily available crates, made development fast, easy, and enjoyable.

Areas for Improvement:

Performance Limitations: The ESP32-S3's processing speed is a bottleneck, particularly for computationally intensive operations like PBKDF2-SHA512, which takes approximately 5 seconds for 65536 rounds. On the other hand, the performance is more than enough after the unlocking
Documentation Gaps: The documentation for both the ESP32-S3 and ATECC608B could be improved. I encountered significant challenges setting up flash encryption, secure boot, and the undocumented KDF command in the ATECC608B.
ATECC608B-TNGTLS Limitations: The Cardputer uses the ATECC608B-TNGTLS variant, which, unfortunately, does not enforce I/O encryption. This means communication between the ESP32-S3 and the secure element is not as secure as it could be.
Known Vulnerabilities: Both the ESP32-S3 and ATECC608B have known vulnerabilities. While these don't necessarily render the device insecure, they highlight the importance of a strong master password as the primary defense.
Missing RTIC Support: The lack of RTIC support on the ESP32-S3 is a missed opportunity for learning and exploration.
RustCrypto Hardware Acceleration Limitations: RustCrypto's AES implementation can effectively leverage hardware acceleration when available, but that does not seems true for the HMAC, HKDF, and PBKDF2 implementations. I believe it's related to the dependency on the Clone trait.
ECC Algorithm Support: The ATECC608B focuses on ECDSA curves and algorithms. Support for Ed25519 would be nice.

ESP32S3: Flash Encryption and Secure Boot

2025-03-07T21:39:00.001+01:00

Flash encryption and secure boot are useful security features for ESP32S3 chip. While not perfect, they definitely make it harder to extract the secrets in the chip.

However, it is tricky to enable both features at the same time. The topic is actually discussed in the official documentation:

Especially, the second one mentioned it is recommended to enable flash encryption before secure boot. But I still find the documentation confusing.

In the end I was able to successfully enable both, here's my findings.

My Understanding

After my adventure, here's what I think could have worked. WARNING, this is untested.

Follow Security Features Enablement Workflows:

Burn all the keys, as long as their purpose eFuses and read/write protections
Burn other security eFuses, but DO NOT burn ENABLE_SECURITY_DOWNLOAD in the middle, which is mentined at the end of the instruction for both flash encryption and secure boot.
Burn SPI_BOOT_CRYPT_CNT (all 3 bits) and SECURE_BOOT_EN

Sign bootloader and application
Encrypt signed bootloader, partition table and signed application
Flash three encrypted pieces into correct offsets.

My Actual Adventure

I had thought the official bootloader was necessary to enable the features, so I actually downloaded ESP-IDF and flashed the bootloader as the last step of enabling flash encryption. Later I found that ENABLE_SECURITY_DOWNLOAD had been automatically burned by the bootloader, because I used the recommended "secure download mode" option in `idf.py menuconfig`.

This is not the end of the world, it just meant I could no longer use `espefuse.py`. I could actually use the C API to burn the eFuses. The API is actually quite user-friendly, and the high-level functions (e.g. burn key, purpose and read/write protection at the same time) do check common errors. I just had to read the documentation to understand the differences between blocks, bits and registers.

One more thing about the bootloader: A recent version (5.3-ish) of ESP-IDF changed something about application image header, which is no longer compatible with esp-hal before this change, this took me quite a while to figure out.

WangLu's Notes

Solving LVM Detection Failures in GRUB After a Force Shutdown

Notes on a Tricky Linux Installation: Qubes OS and Windows

Shrinking an NTFS Volume

Configuring the Display

Turn Off the Display in the Terminal

Turn Off the Display via Kernel Parameters

GRUB and Encrypted /boot

LVM Issues

Secure Boot

Final Thoughts

Refined Boot for Qubes OS: Minimal USB Key, Dual Boot, Secure Boot

Sharing One USB Drive for Multiple Qubes Installations

The Benefits

Make Qubes OS Play Nice with Secure Boot

Security Implications

Writing Sudoku Solvers

1. Brute Force

2. Constraint Propagation

3. Dancing Links (DLX)

4. Generic Solvers

5. Results

Classic Sudoku 1 (Easy)

Classic Sudoku 2 (Harder)

Hard: Empty Board + Thermometer Rules

Hard: Almost Empty Board + Arrow Rules

6. Discussion

7. Conclusion

An Adventure with Qubes OS

Hardware

Detached /boot and LUKS Header

Testing Qubes OS 4.3 rc3

dracut Issues

amdgpu Issues

qubesd Issues

Backup Strategy

Thought: Manging Qubes like Containers

Conclusion

A Rocky Migration: Moving from docker-compose to Podman and gVisor

Network Layout

Routing Issues

File Permissions

Default DNS Server

DNS Servers for Multiple Networks

Option 1: Override DNS

Option 2: HTTP Proxy

Option 3: Transparent HTTP Proxy

Namespaces

Shared Volume

Other Notes

Final Thoughts

Hardening Container Network Security: Filtering Outgoing Traffic

Option 1: A Proxy for Each Policy Group

Thoughts

Option 2: Central Proxy on a Single Network

Networking

Identifying Containers

Deciding the Policy Group

Option 3: One Network Per Policy Group

Identifying the Policy Group

My Plan

A Journey into Podman: Notes on My First Adventure

Quadlet

Hardening Defaults

gVisor

Credentials

Networking

My Networking Plan

Final Thoughts

gVisor: A Fresh Look at Container Security

What is gVisor?

gVisor in Practice

My Plan

A Declarative Approach to Config File Management

Requirements

Existing Solutions

1. Templates

2. Patching

3. Generators

My Plan

GRUB and Encrypted `/boot`

Detached `/boot` and LUKS Header

`dracut` Issues

`amdgpu` Issues

`qubesd` Issues