To give you an example on how effective it was in freeing up some bandwidth, I implemented it as a DNS server in a students home as a ‘last effort’ to fight against the weak line available at that location. As soon as the clients picked up the new DNS server on the network, I was able to use a remote desktop connection again without going completely insane – mostly because of the vast amount of video ads back then just playing. A big win in my books.

Now there is this endless discussion going on about if AdGuard Home or PiHole is the better solution. I mean, AdGuard does a fairly good job at poking at piHole and I decided to give it a fair shot in my environment. A side by side comparison in my homelab, which sports an AMD EPYC 7282 16-Core Processor where I spun up coreOS as a docker host and pulled both containers and set them up on the same host – side by side.

At the first glance both candidates are open-source with a community, a lot of (additional) blocklists to implement and that’s basically all you need on your local network. AdGuard Home also incorporates DoH as well as DoT which are not really needed on a local network to be honest, as long as I trust my own environment. On top of that, AdGuard may also enforce safe-search which could be interesting if you have kids in your house – which I haven’t. But to even the playingfield regarding encryption I set up traefik on piHole – and yes, there may be some more knowledge required. Adguard takes the price home for an easier setup whereas I could configure piHole using a docker-compose file without having to run through a wizard asking me questions.

Setting up the DNS blocker – at least to me – is something I tend to do only once. That’s why it doesn’t really matter much to me for this test. So let’s get down to usage.

I added the same blocklists to PiHole as well as AdGuard Home which I fetched from firehol.net – and adding them in bulk is where piHole clearly takes the lead whereas AdGuard Home, sporting the more pleasing looking WebUI, lacks import functionality. So that’s definitely a win on PiHole if you change your lists a lot, which I do at least once a month with some scripting.

Resource-wise I let adGuard Home warm up a bit and I’ve seen its RAM usage increase quite a bit to 250 MB

PiHole was already running on my system for a while, so it’s fair to use the end value of 100 MB of RAM

Despite rumors of people telling me that piHole needs more RAM, I could not recreate that on my test.

The next thing I wanted to test is latency – that’s how fast I can get a reply from the DNS to my client. I decided to ‘dig’ for ‘google.de’ twice and took the ‘time’ (command). To get started, I took a baseline measurement on the infamous 8.8.8.8 DNS from Google. Asking twice is for getting the result into the cache, eliminating the need to ask an external DNS. Not that it would matter on the google dns, but I wanted to make sure it’s the same procedure for every candidate.

vbauer@ts ~ $ time dig google.de @8.8.8.8

; <<>> DiG 9.16.41 <<>> google.de @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55953
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.de.                     IN      A

;; ANSWER SECTION:
google.de.              299     IN      A       142.250.180.227

;; Query time: 64 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Aug 02 15:00:37 CEST 2023
;; MSG SIZE  rcvd: 54


real    0m0.078s
user    0m0.004s
sys     0m0.004s

Now the interesting part here is the 'real' time which is the actual stopwatch here, clocking in at 78ms which is still quite impressive. Next up was my internal bind9 server clocking in at 43ms for my internal domain. Not really a slowpoke considering that it runs with an LDAP backend. Clearly the faster response time is due to it being in my local network. But it doesn't do DNS blocking at all.
PiHole came in straight with 18ms while AdGuard Home went for the 21ms mark. So performance wise, those 3ms don't really matter at all. What matters is how you like it. So my tip is setting up both and keeping the one you like more and provides the features you need.

Long story short, I set up my homeserver and started to expand it – even by adding my own push service as I didn’t want to mess with building my own client app to register with the Google Push Service and everything looked fine – except for the push service never actually waking the matrix client but working fine with me pushing alerts to my phone. At first I didn’t notice and enjoyed the silence of not being bugged by messages, but it also got me the feeling that ‘something is off’ and so I started looking at the logfiles where I was greeted with a message that was confusing as hell to me:


Mar 18 23:30:14 matrix matrix-synapse[2213]: 2023-03-18 22:30:14,267 - synapse.push.httppusher - 432 - WARNING - httppush.process-17 - Failed to push event $bLGuzRj3t4p6Z6h16aTmfw9YA5gEhZZ0fZOebFToL04 to @user:domain/im.vector.app.android/https://ntfy.domain/upv5aFQWe5uAHe?up=1: DNS lookup failed: no results for hostname lookup: ntfy.domain.

Glancing over the message, I thought my DNS was done for – or something in the docker network went haywire. My first step of debugging this mystery was attaching a shell to the synapse container, who was responsible for this message. To add salt to injury – or should I say ‘for security reasons’, the container was kept pretty slim and didn’t include ping or any nslookup alike. Luckily there was a curl binary in it, so I could test push messages and… success?

Alrighty: It was not the systems DNS messing up. But it looks like, synapse did something pretty wrong – or did it?

The synapse server is written in python, using the twisted framework. So I quickly grabbed a DNS testing script written in that framework to verify that and nuke my theory of twisted being the issue. Oh boy! This is getting fun as it turns out the error message isn’t really about DNS at all. I felt defeated, grabbed some tea and started digging into the synapse federation service and config where I finally – after hours of reading code – found this:


# Prevent outgoing requests from being sent to the following blacklisted IP address
# CIDR ranges. If this option is not specified then it defaults to private IP
# address ranges (see the example below).
#
# The blacklist applies to the outbound requests for federation, identity servers,
# push servers, and for checking key validity for third-party invite events.
#
# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
# listed here, since they correspond to unroutable addresses.)
#
# This option replaces federation_ip_range_blacklist in Synapse v1.25.0.
#
# Note: The value is ignored when an HTTP proxy is in use
#
#ip_range_blacklist:
# - '127.0.0.0/8'
# - '10.0.0.0/8'
# - '172.16.0.0/12'
# - '192.168.0.0/16'
# - '100.64.0.0/10'
# - '192.0.0.0/24'
# - '169.254.0.0/16'
# - '192.88.99.0/24'
# - '198.18.0.0/15'
# - '192.0.2.0/24'
# - '198.51.100.0/24'
# - '203.0.113.0/24'
# - '224.0.0.0/4'
# - '::1/128'
# - 'fe80::/10'
# - 'fc00::/7'
# - '2001:db8::/32'
# - 'ff00::/8'
# - 'fec0::/10'

# List of IP address CIDR ranges that should be allowed for federation,
# identity servers, push servers, and for checking key validity for
# third-party invite events. This is useful for specifying exceptions to
# wide-ranging blacklisted target IP ranges - e.g. for communication with
# a push server only visible in your network.
#
# This whitelist overrides ip_range_blacklist and defaults to an empty
# list.
#
#ip_range_whitelist:
# - '192.168.1.1'


As my ntfy service was exposed in my DMZ network as it was used by my other services as well, it was being blocked by those default configurations. Simply whitelisting it did the trick and made me feel like a boss as I solved that mystery. Did I regret enabling push messages in the end? Sometimes.

# CONFIG_STACKPROTECTOR is not set
CONFIG_LTO=y
CONFIG_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
CONFIG_HAS_LTO_CLANG=y
# CONFIG_LTO_NONE is not set
CONFIG_LTO_CLANG_FULL=y
# CONFIG_LTO_CLANG_THIN is not set

Compiling itself is pretty much straight forward as soon as you get the clang-11 binaries (default debian 11) linked to their base name without the version. Update-alternatives takes care of that

update-alternatives \
        --verbose \
        --install /usr/bin/llvm-config       llvm-config      /usr/bin/llvm-config-11 100 \
        --slave   /usr/bin/llvm-ar           llvm-ar          /usr/bin/llvm-ar-11 \
        --slave   /usr/bin/llvm-as           llvm-as          /usr/bin/llvm-as-11 \
        --slave   /usr/bin/llvm-bcanalyzer   llvm-bcanalyzer  /usr/bin/llvm-bcanalyzer-11 \
        --slave   /usr/bin/llvm-cov          llvm-cov         /usr/bin/llvm-cov-11 \
        --slave   /usr/bin/llvm-diff         llvm-diff        /usr/bin/llvm-diff-11 \
        --slave   /usr/bin/llvm-dis          llvm-dis         /usr/bin/llvm-dis-11 \
        --slave   /usr/bin/llvm-dwarfdump    llvm-dwarfdump   /usr/bin/llvm-dwarfdump-11 \
        --slave   /usr/bin/llvm-extract      llvm-extract     /usr/bin/llvm-extract-11 \
        --slave   /usr/bin/llvm-link         llvm-link        /usr/bin/llvm-link-11 \
        --slave   /usr/bin/llvm-mc           llvm-mc          /usr/bin/llvm-mc-11 \
        --slave   /usr/bin/llvm-nm           llvm-nm          /usr/bin/llvm-nm-11 \
        --slave   /usr/bin/llvm-objdump      llvm-objdump     /usr/bin/llvm-objdump-11 \
        --slave   /usr/bin/llvm-ranlib       llvm-ranlib      /usr/bin/llvm-ranlib-11 \
        --slave   /usr/bin/llvm-readobj      llvm-readobj     /usr/bin/llvm-readobj-11 \
        --slave   /usr/bin/llvm-rtdyld       llvm-rtdyld      /usr/bin/llvm-rtdyld-11 \
        --slave   /usr/bin/llvm-size         llvm-size        /usr/bin/llvm-size-11 \
        --slave   /usr/bin/llvm-stress       llvm-stress      /usr/bin/llvm-stress-11 \
        --slave   /usr/bin/llvm-symbolizer   llvm-symbolizer  /usr/bin/llvm-symbolizer-11 \
        --slave   /usr/bin/llvm-tblgen       llvm-tblgen      /usr/bin/llvm-tblgen-11 \
        --slave   /usr/bin/llvm-objcopy      llvm-objcopy     /usr/bin/llvm-objcopy-11 \
        --slave   /usr/bin/llvm-strip        llvm-strip       /usr/bin/llvm-strip-11

update-alternatives \
        --verbose \
        --install /usr/bin/clang                 clang                 /usr/bin/clang-11 100 \
        --slave   /usr/bin/clang++               clang++               /usr/bin/clang++-11  \
        --slave   /usr/bin/asan_symbolize        asan_symbolize        /usr/bin/asan_symbolize-11 \
        --slave   /usr/bin/clang-cpp             clang-cpp             /usr/bin/clang-cpp-11 \
        --slave   /usr/bin/ld.lld                ld.lld                /usr/bin/ld.lld-11

As this is finally taken care of, we can finally try compiling with make LLVM=1 LLVM_IAS=1 CC=clang HOSTCC=clang LD=ld.lld -j5 deb-pkg – but be aware: LTO needs quite some RAM to get its stuff done. About the results and the speed, I still need to do some tests to see how those kernels act – especially if I can

Second, I created a script to build the kernel I wanted to, to shave off some time as I didn’t want the system to wait for my input during the whole build process. It wasn’t magic at first, but it was sufficient to run all the commands I needed. This way of doing things works well as long as there are no errors and all goes like plan as in a perfect world which we sadly don’t have and it went well for one way of building it.

Now it was time to improve the script:

The first thing I did was introducing some kind of config management to keep the configs consistent over my builds without dropping important modules or any unwanted changes. Git did a great job and the configs are even backed up on github and everywhere I mirrored my repo. Sweet.

Next I had to do a ‘one for all’ solution – merging my four “flavor” scripts into one and break it up into a lot of functions for every step I want the script to perform. While I was at it, I threw in a couple of IFs to act differently per kernel as required. Still not perfect, but another big step into the right direction.

Still, the script itself wasn’t what I needed. For me, the biggest issue was a broken patch or anything that messes my builds up. As I had it all broken down into functions, it was easy to grab them one by one and add failure conditions like patches not applying cleanly or the compiler suddenly dying on which we just exit without saving the config back into the repo. This step was crucial for my build as it saved me a lot of rollbacks on broken builds.

But the script was still a bit too clunky for me. I wanted some more comfort in it as I do not want to look up the exact version number of the kernel as it’s sufficient to know that it is ‘new’. A little bit of grep magic did that for me – which allowed me to be more relaxed about that point too.

KERNEL_VERSION=`curl -s https://www.kernel.org | grep -A1 latest_link | tail -n1 | egrep -o '>[^<]+' | egrep -o '[^>]+'`

A quick reminder here: This script is pure bash for the sake of keeping it simple and easy to maintain. Compiling it into C code wouldn’t give me any benefit here as I would have to run a lot of shell-exec calls anyways and recompiling it for testing and fixing bugs is all way too much for the CPU cycles saved.

The script itself now worked for now, but it led me to some issues when I wanted to give it to a friend for looking over it to check for stuff I could improve on. My credentials were still in there and I do not want to pass them on – even to another trusted person. So I had to deal with an external config file which is part of another blog post here to keep this post slim and on topic.

First of all: Configuration on some scripts may be needed. If it’s for uploading stuff onto a remote repository, FTP server or whatever – you need to have at least the remote host specified somewhere. In case of you’re making your script public, it’s for sure no good idea to share that information as other folks may try their luck and snoop around using their findings.

So what’s the solution to that? At first I decided to move all of my config stuff into variables at the beginning of the file to save me some headache on changing configuration to solve the mistakes we described in the previous step. Nothing big, but it works.

Next would be reading the configuration from an external file. In bash it can be as simple as just including another script like

source /home/myuser/test/config

But that’s where I want to think about what this line does. Sourcing another script is executing it. In other words, source is not secure as it will execute arbitrary code. This may not be a concern for you, but if file permissions are incorrect, it may be possible for an attacker with filesystem access to execute code as a privileged user by injecting code into a config file loaded by an otherwise-secured script such as an init script.

But what’s the solution? Mine was parsing the configuration instead of executing it. As json is simple, I decided to put it to good use here as the program ‘jq’ is a nifty little shell parser for that. If you’re more into xml, I’m sure the XML starlet has you covered.

Now let’s do an example – shall we? Let’s try with a very simple config that I would use:

{ 
"username": "username-or-email",
"password": "the-password" 
}

Now for our main script I can fetch the data using jq:

USERNAME="$( jq -r '.username' "$config_file" )"

The moral of the story: parse your config, do not execute it.

System Authentication is pretty easy on Linux system if we stick to PAM, which allows us to mix and match authentication methods as we want to. In my case I wanted to tackle ‘su’ or ‘sudo’ to save me entering my passwords over and over again. Our savior is the U2F module named pam_u2f.so which may be in your distro’s repository – I was lucky for Debian and Gentoo.

Configuration is shockingly simple:

In /etc/pam.d/ are the config files for every service – some of those are symlinks, but that doesn’t change things. You should be able to see which config goes for what service. I did go for ‘su’ as I need that on my local workstation way too often.

Let’s have a look at the gentoo default config for ‘su’:

auth            sufficient      pam_rootok.so 
auth            required        pam_wheel.so use_uid 
auth            include         system-auth 
account         include         system-auth 
password        include         system-auth 
session         include         system-auth 
session         required        pam_env.so 
session         optional        pam_xauth.so

In this file you find entries from ‘sufficient’ up to ‘required’ for defining the rules for the PAM modules noted in the 3rd column. So to spice up my config, I added this to my ‘auth’ section:

auth            sufficient      pam_u2f.so cue

By using this line, I tell PAM that it’s ‘sufficient’ if I authenticate using my Yubikey and its PIN. If I would use ‘required’ instead of ‘sufficient’ I’d make it a requirement to authenticate successfully using the Yubikey on top of the password. I seriously do not advise for using ‘required’ on the first test run to make sure you do not lock yourself out of the system!

The parameter ‘cue’ tells the PAM module to merrily print a message to ‘touch the device’. All other parameters are documented on https://developers.yubico.com/pam-u2f/

Sadly we’re not finished yet as we just told PAM that it should use the Yubikey. The system also needs some identification of each Yubikey used and some mapping to their users. That’s done either globally in /etc/u2f_mappings or per user in ~/.ssh/u2f_keys

No matter which way you choose, those files look the same and are set up like this:

<username1>:<KeyHandle1>,<UserKey1>,<Options1>:<KeyHandle2>,<UserKey2>,<Options2>:...
<username2>:<KeyHandle1>,<UserKey1>,<Options1>:<KeyHandle2>,<UserKey2>,<Options2>:...

The key handles are acquired using the following command:

pamu2fcfg -uusername -opam://myorigin -ipam://myappid

Now you can assign the keys to the user.

For testing out the system, there’s no need to reboot or reload anything as PAM reads those files on the fly.

I do run a central syslog using graylog and I tend to ship my logs to that device for easier searching and poking around in those. No big deal until I looked at my DNS traffic which was stupidly high. To be exact, more than 3000 DNS requests in less than 10 minutes. Turns out it was the log server being addressed using a dns name instead of its IP address which lead to that problem of my Access Points resolving their log target on every line.

I would have hoped for them to have at least some caching or anything like that implemented – but that wasn’t the case. Lesson learned: Use the IP for syslog servers.

If you check back on my post about ccache on gentoo you might remember that a cache is only good for if you build the same thing over and over again as we do in this case compared to a regular gentoo system. Implementation itself was pretty cheap as I just had to add CC=”ccache gcc” to my make command. To give you some numbers on how this works, here are the stats with a cold cache after a build time of about an hour:

cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                         15556
cache hit rate                      0.00 %
called for link                       29
called for preprocessing            2857
unsupported code directive             4
no input file                        682
cleanups performed                     0
files in cache                     46654
cache size                           7.2 GB
max cache size                      10.0 GB

As you see – no cache hits. After building a second kernel, you can see ccache shaving off some compile time:

cache hit (direct)                    34
cache hit (preprocessed)             774
cache miss                          7318
cache hit rate                      9.94 %
called for link                       28
called for preprocessing            1964
unsupported code directive             4
no input file                        682
cleanups performed                    17
files in cache                     53763
cache size                           8.8 GB
max cache size                      10.0 GB

The next big thing to get things going faster would be using distcc by extending our CC variable to CC=’ccache distcc gcc’ although I would consider this unpractical due to the toolchain having to be the exact same version on the machines involved plus some advanced networking which goes beyond of what I have at my disposal.

To put an end to this manual installation I had to do a meta-package – a debian package that is just dependencies. To do so, I used a program called ‘equivs’, which is installed by the following command:

apt install equivs

As the package is there, it’s time to generate the initial config for our package

equivs-control my-kernel-package

This command gives us a template file that’s to be edited to fit our needs. Pay attention to the line saying “Depends:” as that’s where we add our Debian packages. As of writing this, the current config looks like that:

### Commented entries have reasonable defaults.
### Uncomment to edit them.
# Source: <source package name; defaults to package name>
Section: misc
Priority: optional
# Homepage: <enter URL here; no default>
Standards-Version: 5.9.10

Package: vanilla-kernel
Version: 5.9.10
Maintainer: Viktoria Rei Bauer <debian@example.com>
# Pre-Depends: <comma-separated list of packages>
Depends: linux-image-5.9.10,linux-headers-5.9.10,linux-libc-dev
# Recommends: <comma-separated list of packages>
# Suggests: <comma-separated list of packages>
# Provides: <comma-separated list of packages>
# Replaces: <comma-separated list of packages>
Architecture: amd64
# Multi-Arch: <one of: foreign|same|allowed>
# Copyright: <copyright file; defaults to GPL2>
# Changelog: <changelog file; defaults to a generic changelog>
# Readme: <README.Debian file; defaults to a generic one>
# Extra-Files: <comma-separated list of additional files for the doc directory>
# Links: <pair of space-separated paths; First is path symlink points at, second is filename of link>
# Files: <pair of space-separated paths; First is file to include, second is destination>
# <more pairs, if there's more than one file to include. Notice the starting space>
Description: <short description; defaults to some wise words>
Meta-Package for vanilla kernel built based on debian .config

Now it’s time to build the actual package by calling

equivs-build my-kernel-package

The result is a slick deb-file that can be imported in your repository. I use Sonatype Nexus.

Clients just need to add the repository to their sources.list and add the PGP key and we are ready to roll.