First, I should point out that the procedure outlined in yesterday’s post has no guardrails and should not be attempted unless you are very comfortable with FreeBSD system administration and willing and able to fix things if they go awry.

Second, perhaps I should elaborate on why I (mostly) skipped 15.0. We (the FreeBSD project collectively) have been working on packaged base for over a decade. At some point we thought 14.0 was when it would finally land, but that did not pan out. Then it was going to be 15.0, and there was a massive push to get everything ready, both in base and in pkg… and we still failed: packaged base in FreeBSD 15 is officially a technology preview, and FreeBSD 16 is when we will finally transition. As a result, the pre-release churn on the stable/15 branch was much higher than for a normal major release and a number of things fell through the cracks (including some that had nothing to do with packaged base). The bottom line is that I did not feel that 15.0 was production-ready when it came out, but I worked hard to get most of the issues I was aware of sorted out for 15.1, and I feel much better about 15.1 than I did about 15.0.

Third, I left out a couple of steps for brevity. Most importantly, the systems I converted were all on ZFS (eight jails and their host, as previously mentioned) which allowed me to take a snapshot of the root filesystem (which in a standard FreeBSD install includes /var and /usr/local) so if something had gone wrong I could simply have reverted to that, or rebooted to the boot environment that freebsd-update created when it upgraded my system to 15.1.

My case is also atypical because I have my own package repositories, including for packaged base. I converted the host to packaged base using the official FreeBSD-base repository first, then switched to my own before running pkg upgrade -fy; I then converted the jails directly to my own repository without passing through FreeBSD-base. This is somewhat risky because my base packages are built with non-standard options and do not match FreeBSD distribution sets perfectly, but I know exactly where they differ and how to deal with it.

Assuming your system meets the prerequisites (recently updated binary install of 15.0 or 15.1), the worst thing that can happen is that you forget to install a set and end up with unregistered files (e.g. you had tests.txz installed on your system but left out FreeBSD-set-tests from your pkg install --register-only command line). This is easily remedied, even after the fact, by registering that set and then removing it:

# pkg install --register-only -y FreeBSD-set-tests
# pkg remove -fy FreeBSD-set-tests
# pkg autoremove -y

But watch out! FreeBSD-set-tests depends on FreeBSD-set-base which depends on FreeBSD-set-minimal and FreeBSD-set-optional, so if you had already trimmed your system down to e.g. FreeBSD-set-minimal-jail you will have to remove those extra sets again:

# pkg remove -fy FreeBSD-set-{base,minimal,optional}
# pkg autoremove -y

Since we used --register-only, one thing that can’t happen is that pkg replaces a modified configuration file with its stock version, saving the modified version with a .pkgsave extension. On the other hand, it is possible that pkg upgrade -fy fails to install a file because a directory of the same name exists. I’ve had that happen exactly once due to a discrepancy in /usr/src (I’m not sure why but freebsd-update(8) has never been entirely reliable when it came to patching the source tree). So you may want to run the following after converting:

# find -xs / -name '*.pkgnew'

If you get any hits, there will usually be an empty directory of the same name (without the .pkgnew suffix) which you can remove before manually renaming the file.

Another useful post-conversion command is:

# find -xs / \( -name var -prune \) -or -not -type d -print0 |
    xargs -0 pkg which 2>&1 | grep 'not found' | less

This will give you a list of files on the root filesystem which pkg does not know about. You should be able to identify most of them (configuration files, the trust store, miscellaneous generated files such as /etc/spwd.db).

Finally, look for empty directories on the root filesystem:

# find -xs / -type d -empty

There will usually be a few in /usr/lib/debug and /usr/tests which you can remove if you did not choose to install debugging symbols or tests.

And my final piece of advice, if you want minimum hassle at the cost of having the machine do some pointless work that immediately gets reverted, just register all the sets, then remove the ones you don’t have or don’t want. You’ll get errors from autoremove about missing files, but that (and the few dozen extra seconds the process will take) is about the only downside.

After a brief look at the FreeBSD Foundation’s pkgbasify script I came to the conclusion that I do not trust it to work correctly, so I developed my own method for converting a system from distribution sets to packaged base.

Assuming a full install (including debug symbols, source tree, and tests) of FreeBSD 15.0-RELEASE or 15.1-RELEASE with pkg 2.7.5 recently updated to the latest patch level (if any), you can convert to packaged base with a single command:

# pkg install -r FreeBSD-base --register-only FreeBSD-set-{base,lib32,kernels}{,-dbg} FreeBSD-set-{src,tests}

Note 1: I’m assuming that you are using bash or zsh or some other shell that supports curly brace notation; FreeBSD’s /bin/sh does not.

Note 2: the pkg-install(8) manual page claims that you can abbreviate --register-only to -X, but you can’t, although that will be fixed in the next version.

There may be minor discrepancies between what the installer and freebsd-update(8) produced and what the packages contain, which you can usually fix with a forced upgrade:

# pkg upgrade -r FreeBSD-base -fy

If you do not have a full install, you should trim down the pkg install command line to only the sets that correspond to what you have; otherwise, the subsequent pkg upgrade will install what was missing. Particularly, you should not install FreeBSD-set-src if /usr/src is already managed by Git.

Once you’re done, you can start trimming down your system by removing sets you don’t need, then autoremoving their contents. For instance, if you’ve decided that you no longer need tests or debugging symbols:

# pkg remove -fg FreeBSD-set-\*-dbg FreeBSD-set-tests
# pkg autoremove

Note that package sets are marked vital so you must use -f when removing them.

Now is the time to enable the FreeBSD-base repository, by the way. Manually specifying it with -r FreeBSD-base temporarily enables it for the current command, but you’ll want it permanently enabled:

# mkdir -p /usr/local/etc/pkg/repos
# echo 'FreeBSD-base: { enabled: yes }' >/usr/local/etc/pkg/repos/FreeBSD-base.conf

Finally, if on bare metal or a VM, create a boot environment of your converted system:

# bectl create $(pkg query %v FreeBSD-runtime)

The only issue I’ve noticed after converting one bare-metal server and the eight jails it hosts is that pkg leaf does not take shared library dependencies into account:

# sudo pkg -j myjail leaf
FreeBSD-bluetooth-lib-15.1
FreeBSD-set-base-jail-15.1
FreeBSD-ufs-lib-15.1
FreeBSD-zfs-lib-15.1
pkg-2.7.5

Those three -lib packages provide libraries which are used by other base packages, but the dependencies are all implicit, so they show up as leaves despite not actually being removable.

In closing, I would like to thank Baptiste Daroussin and Lexi Winter for their suggestions while I was working out how I was going to do this, and to Emmanuel Vadot for implementing the --register-only option.

Addendum: follow-up

This afternoon, I graduated from “has not yet owned an Alfa Romeo” to “currently owns an Alfa Romeo”. Specifically, a 2007 Alfa Romeo 147 1.6 TS. At some point in the future, I will move on from “currently owns an Alfa Romeo” to “has owned an Alfa Romeo”, thereby completing my car guy journey; but that will take a lot of work and a bit of money, although in theory I should be able to flip it for a decent profit.

Yes, it’s red, and yes, it has little Italian flags all over.

Currently planned work:

• Install “24h Le Mans” sticker on trunk lid for immediate 5 bhp boost
• Change all fluids and filters
• Replace rear wiper blade
• Polish headlights
• Investigate lifter tick
• Replace clutch pump and / or follower and / or hydraulic line
• Replace passenger side window regulator
• Replace driver side window and mirror controls
• Replace trunk lid lock actuator
• Fix driver’s seat release
• Fix or replace various bits of interior trim
• Replace gear shift lever knob, possibly with a shorter one
• Replace whip antenna with either bee sting or shark fin

Maybe I’ll even remember to post updates.

Update: 5 bhp boost unlocked

Update: of course my Italian car has electrical gremlins.

Update 2026-03-16: I had low washer nozzle pressure and assumed the pump might need changing. Only after ordering a replacement pump from a breaker did I realize that the real issue was that the washer hose was getting pinched when I closed the hood. Cutting the cable ties that tied it to the hood strut solved the issue.

Update 2026-03-19: It passed inspection today, with a slew of minor defects but little I wasn’t already aware of and nothing I can’t fix.

One thing I found odd was that NUUG’s own event calendar (archive) has no information about the event beyond a time and a place. I get that the list of speakers is not yet finalized, even though there’s less than a month to go, but there is literally nothing beyond “more information to come”. All the links just point back to the same page.

This morning, someone pointed out to me that there is a registration page (archive) for the event (which, again, NUUG’s own website does not link to or even mention!) with extensive information about the event. And that information… is not good.

tl;dr: the event is collocated with the Miss Norway 2024 finals and attendees are promised opportunities to have their pictures taken with the winners and runners-up.

WHAT THE EVERLOVING FUCK, NUUG?

It goes without saying that I will not be speaking at this event. I will try to turn my presentation (which I’d only outlined, not written yet) into a blog post, but no promises. It would have been a reflection on what “Unix” actually means (beyond the simple fact of the trademark) and how that has changed over the years.

If you’re a member (and perhaps even if you’re not), I encourage you to contact NUUG and let them know your thoughts on this matter.

EDIT: I have written to NUUG to ask them to either change their plans for the event, in which case I will be happy to attend and speak, or cancel my membership. I will update this post again when (if) they respond.

EDIT 2024-08-07: I have yet to receive an official response from NUUG. I’ve been told out-of-band that my email has been received and my membership canceled, but I appear to still be on the mailing list. In the meantime, Norwegian magazine kode24 has written a piece on the topic, citing my tweet and this blog post, which kicked up a bit of a fuss on Twitter. Knut Yrvin chose to respond directly to my month-old tweet with a strawman and insults while Malin Bruland rode to his defense with the same strawman. I am not impressed.

# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="yes"
local_unbound_tls="yes"
local_unbound_forwarders="9.9.9.9@853#dns.quad9.net 149.112.112.112@853#dns.quad9.net 2620:fe::fe@853#dns.quad9.net 2620:fe::9@853#dns.quad9.net"
EOF
# service local_unbound setup
# service local_unbound restart

No need to reboot.

Note that if you only have IPv4, you may experience slightly degraded performance unless you leave out the IPv6 addresses from the local_unbound_forwarders line (and vice versa in the unlikely scenario where you only have IPv6).

Edit: the author of mod_md, Stefan Eissing, got in touch and pointed me to his own documentation, which is far superior to the one available from Apache.

My starting point is a freshly installed FreeBSD 13.0 server with Apache 2.4, but this isn’t really OS dependent.

First, you will need mod_ssl (of course) and a session cache, and you will need to tweak the TLS parameters, as the defaults are far from fine.

LoadModule ssl_module libexec/apache24/mod_ssl.so
SSLProtocol +TLSv1.3 +TLSv1.2
SSLCipherSuite TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCompression off

LoadModule socache_dbm_module libexec/apache24/mod_socache_dbm.so
SSLSessionCache dbm:/var/db/httpd_ssl_cache.db

You will also need to load mod_md, of course, and mod_watchdog, which mod_md needs to function.

LoadModule watchdog_module libexec/apache24/mod_watchdog.so
LoadModule md_module libexec/apache24/mod_md.so
MDCertificateAgreement accepted
MDContactEmail acme@example.com

The MDCertificateAgreement directive indicates that you have read and accepted Let’s Encrypt’s subscriber agreement, while MDContactEmail is the email address that you used to sign up to Let’s Encrypt.

You will also need mod_rewrite to redirect HTTP requests to HTTPS and mod_headers for HSTS.

LoadModule rewrite_module libexec/apache24/mod_rewrite.so
LoadModule headers_module libexec/apache24/mod_headers.so

By default, Apache only listens on port 80, so you’ll need an extra Listen directive for port 443.

Listen 443

And as always with Apache, you should probably set ServerName and ServerAdmin to sensible values.

ServerName server.example.com
ServerAdmin www@example.com

Next, set up an HTTP-only virtual host that you can use to check the status of mod_md.

<VirtualHost *:80>
  ServerName localhost
  <Location />
    Require ip 127.0.0.1/8 ::1
  </Location>
  <Location "/md-status">
    SetHandler md-status
  </Location>
</VirtualHost>

(Once Apache is running, you will be able to query it at any time as http://localhost/md-status.)

On to the actual website. First, you need to tell mod_md to manage certificates for it.

MDomain site.example.com

Next, set up a redirect from HTTP to HTTPS for everything except ACME challenge tokens.

<VirtualHost localhost:80>
  ServerName site.example.com
  RewriteEngine on
  RewriteRule "^/(?!.well-known/acme-challenge)(.*)" https://site.example.com/$1 [R=301,L]
  ErrorLog /www/site.example.com/logs/http-error.log
  CustomLog /www/site.example.com/logs/http-access.log combined
</VirtualHost>

And finally, the site itself, including HSTS and strict SNI:

<VirtualHost *:443>
  ServerName site.example.com
  SSLEngine on
  SSLStrictSNIVHostCheck On
  Header always set Strict-Transport-Security "max-age=15552000; includeSubdomains;"
  DocumentRoot /www/site.example.com/data
  IncludeOptional /www/site.example.com/etc/*.conf
  ErrorLog /www/site.example.com/logs/https-error.log
  CustomLog /www/site.example.com/logs/https-access.log combined
</VirtualHost>

Now start Apache and monitor the error log. You should see something like this pretty quickly:

[Sun Oct 10 16:15:27.450401 2021] [md:notice] [pid 12345] AH10059: The Managed Domain site.example.com has been setup and changes will be activated on next (graceful) server restart.

Once you do as it says (apachectl graceful), your site will be up and running and you can head over to the Qualys SSL Server Test and admire your solid A+.

Download the sample configuration and try it out yourself.

1. Your web server hosts both www.example.com and www.example.net using Apache httpd with name-based virtual hosts, and the configuration is structured such that httpd sees the former before the latter. Each vhost has its own non-wildcard certificate.
2. A user tries to access www.example.net using an old browser which does not support SNI.
3. Since httpd cannot see the Host header without first performing a TLS handshake, and cannot perform a TLS handshake without a key and certificate, it needs to pick one blindly. In accordance with its virtual host matching algorithm, it defaults to the first configured vhost, i.e. www.example.com.
4. The browser receives a certificate wit a CN which does not match the name of the host it is trying to access, and either throws up a scary warning or simply refuses to connect.

You might say that this acceptable — after all, it only affects a minuscule proportion of users using very old clients — but TLS validators such as Qualys SSL labs have started testing non-SNI requests, and while they don’t yet penalize servers for this issue, they might very well start doing so soon.

Apache httpd has an option called SSLStrictSNIVHostCheck which, in theory, should take care of this. But I haven’t had much luck with it (to put it bluntly: it does not seem to have any effect at all), and it appears from various discussions on Reddit, StackExchange, etc. that I am not alone. The solution I have found is to create a dummy vhost with a dummy, self-signed certificate, and ensure that it is loaded first.

The first step is to create a dummy certificate for localhost with a 100-year lifespan (unless you feel like setting up a cron job that regenerates a 90-day certificate every 30 days or so):

$ openssl genrsa -out localhost.key 2048
$ openssl req -new -key localhost.key -subj /CN=localhost -out localhost.csr
$ echo subjectAltName=DNS:localhost | openssl x509 \
    -in localhost.csr -out localhost.crt -req -signkey localhost.key -days 36525 -extfile -

Next, create a vhost configuration which uses that certificate:

<VirtualHost *:443>
  ServerName localhost
  SSLEngine On
  SSLStrictSNIVHostCheck On
  SSLCertificateKeyFile /path/to/localhost.key
  SSLCertificateFile /path/to/localhost.crt
  <Location />
    Require all denied
  </Location>
</VirtualHost>

This must be placed in a location where it will be read before any other vhost configuration, e.g. 00localhost.conf in your server’s Includes directory.

TLS connections without a server name will now be directed to your dummy vhost and fail, and Qualys will report that any site hosted on your server “works only in browsers with SNI support”.

As a footnote, my TLS configuration (derived from the Mozilla project’s recommendations) looks like this:

SSLProtocol +TLSv1.3 +TLSv1.2
SSLCipherSuite TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder off
SSLCompression off

Between that, strict SNI, HSTS, and CAA, you will easily achieve an A+ score.

% uname -sr
FreeBSD 12.1-RELEASE-p10
% for sh in sh csh bash zsh ; do printf "%-8s" $sh ; $sh -c 'echo \\x21' ; done 
sh      \x21
csh     \x21
bash    \x21
zsh     !
% cowsay wtf, zsh       
 __________ 
< wtf, zsh >
 ---------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

I mean. Bruh. I know it’s intentional & documented & can be turned off, but every other shell defaults to POSIX semantics…

BTW:

% ln -s =zsh /tmp/sh
% /tmp/sh -c 'echo \x21'
\x21

Let us start with what Netlink is, for readers who aren’t very familiar with Linux: it is a mechanism for communicating directly with kernel subsystems using the BSD socket API, rather than by opening device nodes or files in a synthetic filesystem such as /proc. It has pros and cons, but mostly pros, especially as a replacement for ioctl(2), since Netlink sockets are buffered, can be poll(2)ed, and can more easily accommodate variable-length messages and partial reads.

Note: all links to source code in this post point to the versions used in Ubuntu 18.04 as of 2020-08-21: kernel 5.4, userspace 2.8.2.

Netlink messages start with a 16-byte header which looks like this: ^{(source, man page)}

struct nlmsghdr {
	__u32		nlmsg_len;	/* Length of message including header */
	__u16		nlmsg_type;	/* Message content */
	__u16		nlmsg_flags;	/* Additional flags */
	__u32		nlmsg_seq;	/* Sequence number */
	__u32		nlmsg_pid;	/* Sending process port ID */
};

The same header also provides a few macros to help populate and interpret Netlink messages: ^{(source, man page)}

#define NLMSG_ALIGNTO	4U
#define NLMSG_ALIGN(len) ( ((len)+NLMSG_ALIGNTO-1) & ~(NLMSG_ALIGNTO-1) )
#define NLMSG_HDRLEN	 ((int) NLMSG_ALIGN(sizeof(struct nlmsghdr)))
#define NLMSG_LENGTH(len) ((len) + NLMSG_HDRLEN)
#define NLMSG_SPACE(len) NLMSG_ALIGN(NLMSG_LENGTH(len))
#define NLMSG_DATA(nlh)  ((void*)(((char*)nlh) + NLMSG_LENGTH(0)))
#define NLMSG_NEXT(nlh,len)	 ((len) -= NLMSG_ALIGN((nlh)->nlmsg_len), \
				  (struct nlmsghdr*)(((char*)(nlh)) + NLMSG_ALIGN((nlh)->nlmsg_len)))
#define NLMSG_OK(nlh,len) ((len) >= (int)sizeof(struct nlmsghdr) && \
			   (nlh)->nlmsg_len >= sizeof(struct nlmsghdr) && \
			   (nlh)->nlmsg_len <= (len))
#define NLMSG_PAYLOAD(nlh,len) ((nlh)->nlmsg_len - NLMSG_SPACE((len)))

Going by these definitions and the documentation, it is clear that the length field of the message header reflects the total length of the message, header included. What is somewhat less clear is that Netlink messages are supposed to be padded out to a multiple of four bytes before transmission or storage.

The Linux audit subsystem not only breaks these rules, but does not even agree with itself on precisely how to break them.

The userspace tools (auditctl(8), auditd(8)…) all use libaudit to communicate with the kernel audit subsystem. When passing a message of length size to the kernel, libaudit copies the payload into a large pre-zeroed buffer, sets the type, flags, and sequence number fields to the appropriate values, sets the pid field to zero (which is probably a bad idea but, strictly speaking, permitted), and finally sets the length field to NLMSG_SPACE(size), which evaluates to sizeof(struct nlmsghdr) + size rounded up to a multiple of four. It then writes that exact number of bytes to the socket.

Bug #1: The length field should not be rounded up; the purpose of the NLMSG_SPACE() and NLMSG_NEXT() macros is to ensure proper alignment of subsequent message headers when multiple messages are stored or transmitted consecutively. The length field should be computed using NLMSG_LENGTH(), which simply adds the length of the header to its single argument.

Note: to my understanding, Netlink supports sending multiple messages in a single send / receive provided that they are correctly aligned, that they all have the NLM_F_MULTI flag set, and that the last message in the sequence is a zero-length message of type NLMSG_DONE. The audit subsystem does not use this feature.

Moving on: NETLINK_AUDIT messages essentially fall into one of four categories:

1. Requests from userspace; for instance, an AUDIT_GET message which requests the current status of the audit subsystem, an AUDIT_SET message which changes parameters, or an AUDIT_LIST_RULES message which requests a list of currently active auditing rules.
2. Responses from the kernel; these usually have the same type as the request. For instance, the kernel will respond to an AUDIT_GET request with a message of the same type containing a struct audit_status, and to an AUDIT_LIST_RULES request with a sequence of messages of the same type each containing a single struct audit_rule_data.
3. Errors and acknowledgements. These use standard Netlink message types: NLMSG_ERROR in response to an invalid request (or a valid request with the NLM_F_ACK flag set), or NLMSG_DONE at the end of a multi-part response.
4. Audit data. Every event that matches an auditing rule will trigger a series of messages with varying types: usually one which describes the system call that triggered the event, one each for every file or directory affected by the call, one or more describing the process, etc. Each message consists of a header of the form audit(timestamp:serial):␣ which uniquely identifies the event, followed by a space-separated list of key-value pairs. The final message has the type AUDIT_EOE and has the same header, trailing space included, but no data.

The kernel pads responses, errors and acknowledgements, but does not include that padding in the length reported in the message header. So far, so good. However…

Bug #2: Audit data messages are sent from the kernel without padding.

This is not critical, but it does mean that an implementation that batches up incoming messages and stores them consecutively must take extra care to keep them properly aligned.

Bug #3: The length field on audit data messages does not include the length of the header.

This is jaw-dropping. It is so fundamentally wrong. It means that anyone who wants to talk to the audit subsystem using their own code instead of libaudit will have to add a workaround to the Netlink layer of their stack to either fix or ignore the error, and apply that workaround only for certain message types.

How has this gone unnoticed? Well, libaudit doesn’t do much input validation. It relies on the NLMSG_OK() macro, which checks only three things:

1. That the length of the buffer (as returned by recvfrom(2), for instance) is no less than the length of a Netlink message header.
2. That the length field in the message header is no less than the length of a Netlink message header.
3. That the length field in the message header is less than or equal to the length of the buffer.

Since every audit data message, even the empty AUDIT_EOE message, begins with a timestamp and serial number, the length of the payload is never less than 25-30 bytes, and NLMSG_OK() is always satisfied. And since the audit subsystem never sends multiple messages in a single send / receive, it does not matter that NLMSG_NEXT() will be off by 16 bytes.

Consumers of libaudit don’t notice either because they never look at the header; libaudit wraps the message in its own struct audit_reply with its own length and type fields and pointers of the appropriate types for messages that contain binary data (this is a bad idea for entirely different reasons which we won’t go into here). The only case in which the caller needs to know the length of the message is for audit events, when the length field just happens to be the length of the payload, just like the caller expects.

The odds of these bugs getting fixed is approximately zero, because existing applications will break in interesting ways if the kernel starts setting the length field correctly.

Turing wept.

THIS IS WHY WE CAN’T HAVE NICE THINGS

Some of you may know that the 2020 Hugo Award ceremony was held last night¹ and that it was hosted by George R. R. Martin. Some of you may have heard that it did not go well. Some of you may already know what happened, more or less. I watched it live, and unsurprisingly, I have opinions.

This post is not a blow-by-blow account of events or any sort of clever analysis or deep thoughts on how to move forward. Better minds than mine have already taken care of that; see for instance Natalie Luhrs’s take on the affair. Instead, I would like to offer a little bit of context for those who heard what happened (or watched it happen) and have a vague idea that it was bad but do not understand why everybody is so upset and do not want to jump down the rabbit hole of SFF fandom drama.

So, here is my best attempt at putting the controversy in context.

1. This year’s Hugo award administrators and / or WorldCon organizing committee selected George R. R. Martin to host the award ceremony. He, in turn, selected Robert Silverberg to present two of the awards (Best Editor Short Form / Long Form).
2. There is a Hugo-adjacent award for new authors, which used to be called the John W. Campbell Award for Best New Writer. It is not technically a Hugo award, but in practice, the only difference is that instead of a pin and a silver rocket, you get a different pin and a plaque.
3. In her acceptance speech for the 2019 Campbell award, Jeannette Ng referenced the fact that Campbell was, shall we say, Not Very Nice.² As she points out herself, she was neither the first nor the most prominent person to bring that up, or even the first or most prominent to do so in front of a large audience. But we live in interesting times, and her speech resonated as never before. This had two major consequences: the award was renamed to the Astounding Award for Best New Writer, and Ng’s acceptance speech was nominated for the 2020 Hugo Award for Best Related Work.
4. As editor of the Astounding Science Fiction magazine (later renamed to Analog Science Fiction and Fact) for thirty-odd years, Campbell was credited with discovering, mentoring, and promoting many new authors, some of whom went on to become the most recognizable names in the genre. In that light, it seems natural that an award given to new writers should be named after him. However, we cannot mention the voices which Campbell elevated without also mentioning the ones he suppressed. Because Campbell had a very clear idea of who was worth listening to (white men) and who wasn’t (everybody else). If Campbell was still alive and in a position of power, it is doubtful that Jeannette Ng would be a published author. Or Rebecca F. Kuang, who won this year’s Astounding Award. Or Rebecca Roanhorse (2018). Or Wesley Chu (2015). Or Sofia Samatar (2014). Or any but five or six of the last twenty Best New Writer laureates. And certainly not N. K. Jemisin, whose Broken Earth trilogy netted her three consecutive Best Novel Hugos (2016-2018), a Nebula (2018), and a Locus (2018), but whose skin is of a shade which Campbell particularly detested.

When Martin (and Silverberg, who was granted — or granted himself — a disproportionate amount of airtime compared to other presenters) brought up Campbell again, and again, and again, during the ceremony; and when Martin repeatedly pronounced “Astounding” in the most theatrical way imaginable when referring to the award; and when Martin gave a list of examples of what constitutes a “related work”³ which covered every nominee except Ng; this has to be viewed in the context of Ng’s speech last year, of her nomination this year, and of the renaming of the award. And it does not look pretty.

Finally,

¹ It was to have been held in Wellington, New Zealand as part of the 2020 World Science Fiction Convention, but was entirely virtual due to the ongoing pandemic. It started at 11 am on Saturday in New Zealand time, which was the middle of the night in Europe and Friday evening in the US.

² Her exact words were “a fucking fascist”, which may have been impolitic but was certainly not inaccurate. Also, if there is one thing we have learned from the history of civil rights, it is that politeness gets you nowhere; when the ruling class asks protesters to “be civil”, what they actually mean is “be invisible”.

³ Starting at 2:46:15 in the archived livestream: “biographies are related works, autobiographies, works of criticism, art books, collections of reviews, memoirs, histories, and… other things”

In the F/OSS community, there are two different philosophies when it comes to applying copyright statements to a project. If the code base consists exclusively (or almost exclusively) of code developed for that specific project by the project’s author or co-authors, many projects will have a single file (usually named LICENSE) containing the license, a list of copyright holders, and the copyright dates or ranges. However, if the code base incorporates a significant body of code taken from other projects or contributed by parties outside the project, it is customary to include the copyright statements and either the complete license or a reference to it in each individual file. In my experience, projects that use the BSD, ISC, MIT, adjacent licenses tend to use the latter model regardless.

The advantage of the second model is that it’s hard to get wrong. You might forget to add a name to a central list, but you’re far less likely to forget to state the name of the author when you add a new file. The disadvantage is that it’s really, really easy to forget to update the copyright year when you commit a change to an existing file that hasn’t been touched in a while.

So, how can we automate this?

One possibility is to have a pre-commit hook that updates it for you (generally a bad idea), or one that rejects the commit if it thinks you forgot (better, but not perfect; what if you’re adding a file from an outside source?), or one that prints a big fat warning if it thinks you forgot (much better, especially with Git since you can commit --amend once you’ve fixed it, before pushing).

But how do you fix the mistake retroactively, without poring over commit logs to figure out what was modified when?

Let’s start by assuming that you have a list of files that were modified in 2017, and that each file only has one copyright statement that needs to be updated to reflect that fact. The following Perl one-liner should do the trick:

perl -p -i -e 'if (/Copyright/) { s/ ([0-9]{4})-20(?:0[0-9]|1[0-6]) / $1-2017 /; s/ (20(?:0[0-9]|1[0-6])) / $1-2017 /; }'

It should be fairly self-explanatory if you know regular expressions. The first substitution handles the case where the existing statement contains a range, in which case we extend it to include 2017, and the second substitution handles the case where the existing statement contains a single year, which we replace with a range starting with the original year and ending with 2017. The complexity stems mostly from having to take care not to replace 2018 (or later) with 2017; our regexes only match years in the range 2000-2016.

OK, so now we know how to fix the files, but how do we figure out which ones need fixing?

With Git, we could try something like this:

git diff --name-only 'HEAD@{2017-01-01}..HEAD@{2018-01-01}'

This is… imperfect, though. The first problem is that it will list every file that was touched, including files that were added, moved, renamed, or deleted. Files that were added should be assumed to have had a correct copyright statement at the time they were added; files that were only moved or renamed should not be updated, since their contents did not change; and files that were deleted are no longer there to be updated.¹ We should restrict our search to files that were actually modified:

git diff --name-only --diff-filter M 'HEAD@{2017-01-01}..HEAD@{2018-01-01}'

Some of those changes might be too trivial to copyright, though. This is a fairly complex legal matter, but to simplify, if the change was inevitable and there was no room for creative expression — for instance, a function in a third-party library you are using was renamed, so both the reason for the change and the nature of the change are external to the work itself — then it is not protected. So perhaps you should remove --name-only and review the diff, which is when you realize that half those files were only modified to update their copyright statements because you forgot to do so in 2016. Let’s try to exclude them mechanically, rather than manually. Unfortunately, git diff does not have anything that resembles diff -I, so we have to write our own diff command which does that, and ask git to use it:

$ echo 'exec diff -u -ICopyright "$@"' >diff-no-copyright
$ chmod a+rx diff-no-copyright
$ git difftool --diff-filter M --extcmd $PWD/diff-no-copyright 'HEAD@{2017-01-01}..HEAD@{2018-01-01}'

This gives us a diff, though, not a list of files. We can try to extract the names as follows:

$ git difftool --diff-filter M --no-prompt --extcmd $PWD/diff-no-copyright 'HEAD@{2017-01-01}..HEAD@{2018-01-01}' | awk '/^---/ { print $2 }'

Wait… no, that’s just garbage. The thing is, git difftool works by checking out both versions of a file and diffing them, so what we get is a list of the names of the temporary files it created. We have to be a little more creative:

$ echo '/usr/bin/diff -q -ICopyright "$@" >/dev/null || echo "$BASE"' >list-no-copyright
$ chmod a+rx list-no-copyright
$ git difftool --diff-filter M --no-prompt --extcmd $PWD/list-no-copyright 'HEAD@{2017-01-01}..HEAD@{2018-01-01}'

Much better. We can glue this together with our Perl one-liner using xargs, then repeat the process for 2018.

Finally, how about Subversion? On the one hand, Subversion is far simpler than Git, so we can get 90% of the way much more easily. On the other hand, Subversion is far less flexible than Git, so we can’t go the last 10% of the way. Here’s the best I could do:

$ echo 'exec diff -u -ICopyright "$@"' >diff-no-copyright
$ chmod a+rx diff-no-copyright
$ svn diff --ignore-properties --diff-cmd $PWD/diff-no-copyright -r'{2017-01-01}:{2018-01-01}' | awk '/^---/ { print $2 }'

This will not work properly if you have files with names that contain whitespace; you’ll have to use sed with a much more complicated regex, which I leave as an exercise.

¹ I will leave the issue of move-and-modify being incorrectly recorded as delete-and-add to the reader. One possibility is to include added files in the list by using --diff-filter AM, and review them manually before committing.

DNS over TLS is just what it sounds like: DNS over TCP, but wrapped in a TLS session. It encrypts your requests and the server’s replies, and optionally allows you to verify the identity of the server. The advantages are protection against eavesdropping and manipulation of your DNS traffic; the drawbacks are a slight performance degradation and potential firewall traversal issues, as it runs over a non-standard port (TCP port 853) which may be blocked on some networks. Let’s take a look at how to set it up.

Basic setup

As a simple test case, let’s set up our 12.0-ALPHA10 VM to use Cloudflare’s DNS service:

# uname -r
12.0-ALPHA10
# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="YES"
local_unbound_tls="YES"
local_unbound_forwarders="1.1.1.1@853 1.0.0.1@853"
EOF
# service local_unbound start
Performing initial setup.
destination: 
/var/unbound/forward.conf created
/var/unbound/lan-zones.conf created
/var/unbound/control.conf created
/var/unbound/unbound.conf created
/etc/resolvconf.conf not modified
Original /etc/resolv.conf saved as /var/backups/resolv.conf.20181021.192629
Starting local_unbound.
Waiting for nameserver to start... good
# host www.freebsd.org
www.freebsd.org is an alias for wfe0.nyi.freebsd.org.
wfe0.nyi.freebsd.org has address 96.47.72.84
wfe0.nyi.freebsd.org has IPv6 address 2610:1c1:1:606c::50:15
wfe0.nyi.freebsd.org mail is handled by 0 .

Note that this is not a configuration you want to run in production—we will come back to this later.

Performance

The downside of DNS over TLS is the performance hit of the TCP and TLS session setup and teardown. We demonstrate this by flushing our cache and (rather crudely) measuring a cache miss and a cache hit:

# local-unbound-control reload
ok
# time host www.freebsd.org >x
host www.freebsd.org > x  0.00s user 0.00s system 0% cpu 0.553 total
# time host www.freebsd.org >x
host www.freebsd.org > x  0.00s user 0.00s system 0% cpu 0.005 total

Compare this to querying our router, a puny Soekris net5501 running Unbound 1.8.1 on FreeBSD 11.1-RELEASE:

# time host www.freebsd.org gw >x
host www.freebsd.org gw > x  0.00s user 0.00s system 0% cpu 0.232 total
# time host www.freebsd.org 192.168.144.1 >x
host www.freebsd.org gw > x  0.00s user 0.00s system 0% cpu 0.008 total

or to querying Cloudflare directly over UDP:

# time host www.freebsd.org 1.1.1.1 >x      
host www.freebsd.org 1.1.1.1 > x  0.00s user 0.00s system 0% cpu 0.272 total
# time host www.freebsd.org 1.1.1.1 >x
host www.freebsd.org 1.1.1.1 > x  0.00s user 0.00s system 0% cpu 0.013 total

(Cloudflare uses anycast routing, so it is not so unreasonable to see a cache miss during off-peak hours.)

This clearly shows the advantage of running a local caching resolver—it absorbs the cost of DNSSEC and TLS. And speaking of DNSSEC, we can separate that cost from that of TLS by reconfiguring our server without the latter:

# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="YES"
local_unbound_tls="NO"
local_unbound_forwarders="1.1.1.1 1.0.0.1"
EOF
# service local_unbound setup
Performing initial setup.
destination: 
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.205328
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
Original /var/unbound/unbound.conf saved as /var/backups/unbound.conf.20181021.205328
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound start
Starting local_unbound.
Waiting for nameserver to start... good
# time host www.freebsd.org >x
host www.freebsd.org > x  0.00s user 0.00s system 0% cpu 0.080 total
# time host www.freebsd.org >x
host www.freebsd.org > x  0.00s user 0.00s system 0% cpu 0.004 total

So does TLS add nearly half a second to every cache miss? Not quite, fortunately—in our previous tests, our first query was not only a cache miss but also the first query after a restart or a cache flush, resulting in a complete load and validation of the entire path from the name we queried to the root. The difference between a first and second cache miss is quite noticeable:

# time host www.freebsd.org >x 
host www.freebsd.org > x  0.00s user 0.00s system 0% cpu 0.546 total
# time host www.freebsd.org >x
host www.freebsd.org > x  0.00s user 0.00s system 0% cpu 0.004 total
# time host repo.freebsd.org >x
host repo.freebsd.org > x  0.00s user 0.00s system 0% cpu 0.168 total
# time host repo.freebsd.org >x
host repo.freebsd.org > x  0.00s user 0.00s system 0% cpu 0.004 total

Revisiting our configuration

Remember when I said that you shouldn’t run the sample configuration in production, and that I’d get back to it later? This is later.

The problem with our first configuration is that while it encrypts our DNS traffic, it does not verify the identity of the server. Our ISP could be routing all traffic to 1.1.1.1 to its own servers, logging it, and selling the information to the highest bidder. We need to tell Unbound to validate the server certificate, but there’s a catch: Unbound only knows the IP addresses of its forwarders, not their names. We have to provide it with names that will match the x509 certificates used by the servers we want to use. Let’s double-check the certificate:

# :| openssl s_client -connect 1.1.1.1:853 |& openssl x509 -noout -text |& grep DNS
            DNS:*.cloudflare-dns.com, IP Address:1.1.1.1, IP Address:1.0.0.1, DNS:cloudflare-dns.com, IP Address:2606:4700:4700:0:0:0:0:1111, IP Address:2606:4700:4700:0:0:0:0:1001

This matches Cloudflare’s documentation, so let’s update our configuration:

# cat >/etc/rc.conf.d/local_unbound <<EOF
local_unbound_enable="YES"
local_unbound_tls="YES"
local_unbound_forwarders="1.1.1.1@853#cloudflare-dns.com 1.0.0.1@853#cloudflare-dns.com"
EOF
# service local_unbound setup            
Performing initial setup.
destination: 
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.212519
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
/var/unbound/unbound.conf not modified
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound restart
Stopping local_unbound.
Starting local_unbound.
Waiting for nameserver to start... good
# host www.freebsd.org
www.freebsd.org is an alias for wfe0.nyi.freebsd.org.
wfe0.nyi.freebsd.org has address 96.47.72.84
wfe0.nyi.freebsd.org has IPv6 address 2610:1c1:1:606c::50:15
wfe0.nyi.freebsd.org mail is handled by 0 .

How can we confirm that Unbound actually validates the certificate? Well, we can run Unbound in debug mode (/usr/sbin/local-unbound -dd -vvv) and read the debugging output… or we can confirm that it fails when given a name that does not match the certificate:

# perl -p -i -e 's/cloudflare/cloudfire/g' /etc/rc.conf.d/local_unbound
# service local_unbound setup
Performing initial setup.
destination: 
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.215808
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
/var/unbound/unbound.conf not modified
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound restart
Stopping local_unbound.
Waiting for PIDS: 33977.
Starting local_unbound.
Waiting for nameserver to start... good
# host www.freebsd.org
Host www.freebsd.org not found: 2(SERVFAIL)

But is this really a failure to validate the certificate? Actually, no. When provided with a server name, Unbound will pass it to the server during the TLS handshake, and the server will reject the handshake if that name does not match any of its certificates. To truly verify that Unbound validates the server certificate, we have to confirm that it fails when it cannot do so. For instance, we can remove the root certificate used to sign the DNS server’s certificate from the test system’s trust store. Note that we cannot simply remove the trust store entirely, as Unbound will refuse to start if the trust store is missing or empty.

While we’re talking about trust stores, I should point out that you currently must have ca_root_nss installed for DNS over TLS to work. However, 12.0-RELEASE will ship with a pre-installed copy.

Conclusion

We’ve seen how to set up Unbound—specifically, the local_unbound service in FreeBSD 12.0—to use DNS over TLS instead of plain UDP or TCP, using Cloudflare’s public DNS service as an example. We’ve looked at the performance impact, and at how to ensure (and verify) that Unbound validates the server certificate to prevent man-in-the-middle attacks.

The question that remains is whether it is all worth it. There is undeniably a performance hit, though this may improve with TLS 1.3. More importantly, there are currently very few DNS-over-TLS providers—only one, really, since Quad9 filter their responses—and you have to weigh the advantage of encrypting your DNS traffic against the disadvantage of sending it all to a single organization. I can’t answer that question for you, but I can tell you that the parameters are evolving quickly, and if your answer is negative today, it may not remain so for long. More providers will appear. Performance will improve with TLS 1.3 and QUIC. Within a year or two, running DNS over TLS may very well become the rule rather than the experimental exception.

My level of engagement with the FreeBSD project has varied greatly over the twenty years I’ve been a committer. There have been times when I worked on it full-time, and times when I did not touch it for months. The last few years, health issues and life events have consumed my time and sapped my energy, and my contributions have come in bursts. Commit statistics do not tell the whole story, though: even when not working on FreeBSD directly, I have worked on side projects which, like OpenPAM, may one day find their way into FreeBSD.

My contributions have not been limited to code. I was the project’s first Bugmeister; I’ve served on the Security Team for a long time, and have been both Security Officer and Deputy Security Officer; I managed the last four Core Team elections and am doing so again this year.

In return, the project has taught me much about programming and software engineering. It taught me code hygiene and the importance of clarity over cleverness; it taught me the ins and outs of revision control; it taught me the importance of good documentation, and how to write it; and it taught me good release engineering practices.

Last but not least, it has provided me with the opportunity to work with some of the best people in the field. I have the privilege today to count several of them among my friends.

For better or worse, the FreeBSD project has shaped my career and my life. It set me on the path to information security in general and IAA in particular, and opened many a door for me. I would not be where I am now without it.

I won’t pretend to be able to tell the future. I don’t know how long I will remain active in the FreeBSD project and community. It could be another twenty years; or it could be ten, or five, or less. All I know is that FreeBSD and I still have things to teach each other, and I don’t intend to call it quits any time soon.

Previously

Then came the apologies. Let me tell you about the apologies. The average response from a manager or venture capitalist accused of sexism went something like this:

I apologized unreservedly for my treatment of X. I realize now that my innocent jokes may have been misinterpreted. I’m actually a pretty nice guy, and X’s refusal to sleep with me had no impact whatsoever on my decision not to invest in her startup.

Then came the White Knights:

As a VC, I’m appalled to hear about my colleagues’ behavior towards women. I would like to reassure you all that Not All Men are like that. I myself am actually a pretty nice guy and completely innocent in all this.

Guys, it’s time to face the music. We have all been That Guy. We have all made sexist jokes, or laughed when others made them, or stood by silently while our male bosses, coworkers and colleagues ignored or patronized or belittled or humiliated women. We have all benefited from a system that eliminates close to 50% of our competition before the race has even started.

We are all complicit. We are all guilty.

So what do we do? Where do we go next?

First, take a deep breath, do a little soul-searching, and re-read that paragraph until any impulse, however minor, to say to yourself “OK, but not me” is gone. Yes, you too.

Next, if you’ve ever acted inappropriately towards a female coworker or friend or acquaintance, or stood by silently while others did, consider apologizing.

Third, vow to never do so again, and work hard to keep that vow. Respect the women around you as much as you respect the men. If someone around you acts or speaks inappropriately, speak up, even if there are no women present. Be proactive: make sure that women are given equal opportunity to join those career-building projects, and are included in those backstage chats where decisions are made. If you are hiring, seek out female candidates, keeping in mind that women have a tendency to underestimate their abilities and experience just as men have a tendency to overestimate them. If you are teaching, encourage and mentor female students. Reach out to them if they seem discouraged. Don’t wait until they drop out.

Open your eyes. Open your ears. Listen to the women around you. Believe them. Respect them. Be someone they can vent to and someone they can count on for support when push comes to shove.

You will slip up. When you do, apologize and vow to do better.

As a man, you are, and always have been, part of the problem. Accept it, and start being part of the solution.

Scene 1

Florestan: Shit, it’s really dark in here. It’s a good thing I’m really, really righteous and brave! I just hope Leonore is OK.

Scene 2

Fidelionore: Brr, it’s cold as balls down here.

Rocco: Sorry, Pizarro must have forgotten to pay the electricity bill. Anyway, here we are.

Fidelionore: He’s not moving!

Rocco: You think he’s dead? Nah, just asleep. Help me dig. You scared?

Fidelionore: Just cold. Sorry.

Rocco: Start digging, it’ll keep you warm.

They dig.

Fidelionore: I think he’s waking up!

Rocco: Get out of here, I need to talk to him.

Florestan: I have been imprisoned here for over two years, and surely I must know where I am and what has happened, but the audience wasn’t here, so please pretend I don’t know, and tell me who is keeping me here.

Rocco: Pizarro, and believe me, I like him about as much as you do.

Florestan: Pizarro? Shit. Send word to Sevilla, let my wife know where I am!

Rocco: Sorry, bud, no can do. I brought some wine to dull the pain of digging another man’s grave, want a drop?

Florestan: Why the hell not.

Rocco: Fidelio, bring the wine! Hey, you don’t look too good.

Florestan: Poor kid!

Fidelionore: I don’t feel too good.

Rocco: Look, it sucks that he’s going to die, but I’m only following orders.

Fidelionore: Want some stale bread? I’ve been carrying this around for days instead of grabbing a fresh piece at the breakfast table every morning.

Rocco: I am about to become complicit in an innocent man’s death, but it’s all good because I’m only following orders, plus I gave him some wine.

Florestan: I am sorry that I cannot repay you for bringing me a stale piece of bread and the dregs of your wine after you were done digging my grave.

Rocco: Okiedokie, off to tell Pizarro everything is ready.

Fidelionore: Don’t worry, it’ll all work out. Somehow. Maybe. I hope.

Scene 3

Pizarro: All done?

Rocco: All done.

Pizarro: Send the kid away and untie the prisoner while I gloat over his impending doom and make sure to let him know at whose hand it will come.

Florestan: Murderer!

Fidelionore: Murderer!

Rocco: Just following orders.

Fidelionore throws herself between Pizarro and Florestan

Fidelionore: You’ll have to kill me first!

Pizarro: Wut?

Fidelionore: I’m his wife, dumbass. Did nobody notice the hips and the tits and the fact that I’m a ducking soprano?

Pizarro: You’re his wife?

Rocco: You’re his wife?

Florestan: You’re my wife?

Pizarro: Wow, you’re really brave.

Rocco: Wow, you’re really brave.

Florestan: Wow, you’re really brave.

Pizarro: Shit, now I have to kill them both.

Fidelionore: Well, sucks to be you.

The alarm sounds.

Fidelionore: You’re saved!

Florestan: I’m saved?

Pizarro: Shit, Fernando!

Rocco: We’re saved!

Scene 4

Jaquino: Hey boss, Secretary Fernando has arrived.

Rocco: Send the guards down to, eh, ensure Pizarro gets safely up the stairs.

Fidelionore & Florestan: We’re saved!

Pizarro: I’m doomed!

Rocco: Remember, I was only following orders!

Scene 5

Fidelionore: You’re saved!

Florestan: We gonna frick!

Scene 6

Prisoners: We’re free!

Fernando: The King has sent me to free you!

Prisoners: Bonus heart-wrenching chorus!

Scene 7

Rocco: Help! Help!

Pizarro: Shut up!

Fernando: What’s all this then?

Rocco: Have mercy on Florestan—

Fernando: Florestan? Isn’t he dead?

Rocco: Only mostly dead. There’s a big difference between mostly dead and all dead. Mostly dead is slightly alive.

Fernando: Florestan is slightly alive?

Fidelionore: HE’S RIGHT IN FRONT OF YOU, YOU DUMBASS!

Rocco: …with his wife Leonore! Dressed as a man!

Marzelline: Oh shit.

Rocco: Pizarro was going to murder him!

Pizarro: And I would have gotten away with it too, if it weren’t for that meddling kid! Besides, you were helping.

Rocco: For the last time, I was only following orders!

Pizarro is arrested and taken away

Fernando: Here, Leonore, unshackle your husband.

Everybody: WE’RE SO HAPPY!

Florestan: Let this be a lesson to you all, the righteous always prevail in the end!

Fidelionore: Not to mention Tru Wuv.

Everybody: Yeah, yeah, we’re all very impressed.

Fidelionore: We gonna frick!

TEH END

Thank you, you’ve been a lovely audience.

Background

Two years ago, Florestan uncovered evidence of his rival Pizarro’s crimes. Since then, he has been illegally detained by the latter in the prison he governs. Florestan’s wife Leonore has tracked Florestan down and obtained employment at the prison, disguised as a young man named Fidelio (because opera, that’s why). She has gradually gained warden Rocco’s trust. Prison guard Jaquino is infatuated with Rocco’s daughter Marzelline, who is infatuated with Fidelio (because opera, that’s why).

Scene 1

Jaquino: Finally got you cornered! I need to talk to you.

Marzelline: Dude, I got work to do.

Jaquino: Come on, give a guy a break!

Marzelline: OK, spit it out. Just don’t expect me to like it.

Jaquino: I’ve decided that you are to be my wife. We can have the wedding in a few weeks.

Marzelline: LOL WUT

someone knocks

Jaquino: PISS OFF I’M TRYING TO PROPOSE HERE

Marzelline: Dude, I don’t even like you. I’m in love with Fidelio.

Jaquino: What can I do to convince you?

knocking intensifies

Marzelline: Good, maybe he’ll let me go now.

Rocco (off-stage): Jaquino, you lazy good-for-nothing, get back to work!

Marzelline: You heard the man, now scram!

Jaquino leaves

Marzelline: Poor guy. I used to like him until I met Fidelio. Now there’s a man in touch with his feminine side!

Scene 2

Marzelline pines for Fidelio

Scene 3

Rocco: Where the f— is Fidelio?

Marzelline: Hell if I know. Oh, wait, there he is!

Scene 4

Fidelionore: Sorry, boss. Dude took forever. Here’s the receipt.

Rocco: Wow, how’d you talk the price down so much?

Fidelionore: I try my best, boss.

Rocco: Good man. Don’t worry, you’ll get what you’re after.

Fidelionore: Say what now?

Rocco: Oh come on, I know you like Marzelline.

Marzelline: OMG OMG OMG FIDELIO LIKES ME OMG!

Fidelionore: Oh shit.

Rocco: I’m so happy for them!

Jaquino: Oh shit.

Rocco: Good, then it’s settled! We can have the wedding as soon as that ass Pizarro leaves for Sevilla.

Marzelline: Squee!

Rocco: One thing though, Fidelio. Promise me you’ll provide for my daughter. Love means nothing if you can’t put food on the table. Gold, gold, gold, gold, gold, gold, gold!

Fidelionore: Well, I still maintain that true love… but there’s one thing that bothers me. Why don’t you trust me to accompany you down to the lower cells?

Rocco: It’s not that I don’t trust you, it’s that I’m not allowed to let anyone near them.

Fidelionore: But you work your fingers to the bone! Let me help.

Rocco: Well, maybe. But I won’t let you near the oubliette. It’s too gruesome.

Marzelline: Is that where the secret prisoner is kept?

Fidelionore: Has he been there long?

Rocco: Two years. And now Pizarro has ordered me to let him starve in the dark.

Marzelline: Please don’t take Fidelio there, it’ll break his heart!

Fidelionore: Are you calling me a wuss?

Rocco: You need guts to get ahead in life, boy.

Fidelionore: I got plenty. Try me.

Marzelline: Your determination to see a man being starved to death makes me inexplicably proud.

Rocco: Fine! I’ll ask the governor to allow you to assist me. I’m working myself to death as it is.

Marzelline: I am so turned on right now.

Scene 5

Pizarro storms in from who knows where

Pizarro: Man the walls! Let no-one in without my express permission. Rocco, bring me my mail!

Rocco: Here, sir.

Pizarro: Bill—bill—advertising—overdue bill—Sears catalogue—final notice—pre-approved credit card—oh shit, I know that letterhead. “It has come to my attention that you are illegally keeping political prisoners, and I am therefore conducting a surprise inspection. You have a few hours to hide the evidence. XOXO Fernando.” Holy shit, he and Florestan were like besties. Captain! Post guards and sound the alarm the moment you see Secretary Fernando’s limo arrive!

Captain: Sir, yes, sir!

Pizarro: The only thing that can save me now is an act of unsurpassed bravery. And I will finally have my revenge, which I could have had at any time in the two years this man has been my prisoner, but inexplicably postponed! Oh, I can’t wait to see the knife twist in his heart! Rocco!

Rocco: Sir?

Pizarro: Rocco, I want you to prove my bravery, courage and high moral conviction by murdering this man in my place.

Rocco: Let’s not, and say we did.

Pizarro: Wuss. Never mind, I’ll do it myself. Go dig his grave while I put on an unconvincing disguise so I can tell myself it wasn’t really me who did it. And God help you if I get blood on my shoes!

Rocco: Oh well. At least he won’t starve any more.

Scene 6

Fidelionore: WHAT THE F— JUST HAPPENED? I have to stop this.

Scene 7

Jaquino: Marzelline! You used to love me, but ever since this Fidelio…

Marzelline: Leave me alone!

Scene 8

Rocco: Let it go, son. She doesn’t want you.

Fidelionore: Rocco, won’t you please let the prisoners out into the sun? Pizarro doesn’t need to know.

Rocco: Oh, why the hell not. Jaquino, Fidelio, open the upper cells!

Scene 9

Prisoners: Watch as we march dramatically out of our cells and sing a heart-wrenching chorus about how bad it is to imprison people for their political opinions! But not too loud, someone might hear us.

Scene 10

Fidelionore: How did it go?

Rocco: He agreed to the wedding and to let you accompany me to the lower cells.

Fidelionore: O frabjous day! Callooh! Callay!

Rocco: What are you going on about? Anyway, we have to go down to the secret prisoner.

Fidelionore: Is he being released?

Rocco: Released? Oh, no, we are to bury him.

Fidelionore: What?! He’s dead?

Rocco: Well, not quite yet…

Fidelionore: YOU’RE GOING TO KILL HIM?

Rocco: No, just dig his grave and wash my hands of his murder.

Fidelionore starts crying

Rocco: Oh, grow up. Let’s go.

Scene 11

Marzelline: Dad! Dad! Pizarro is looking for you, and let me tell you, he is PISSED. The guard captain told him we let the prisoners out. You know how mad he gets…

Rocco: Oh shit. Quick, get everybody back inside.

Scene 12

Pizarro: Rocco! ROCCO! Who the FRACK gave you permission to let the prisoners out?

Rocco: Well, uh, it’s, uh, spring, and also the King’s birthday or something? Also, I only let the regular prisoners out, not the secret one.

Pizarro: Oh, shut up, and go dig that grave.

Prisoners: We’re really sad to have to go back inside.

Marzelline: I’m really sad to see the prisoners go back inside.

Fidelionore: I’m really sad to see the prisoners go back inside.

Jaquino: Oh, screw them.

Pizarro: Off you go, Rocco, and stay there until the deed is done.

End of Act One

I’ll post Act Two as soon as that ass Pizarro leaves for Sevilla.

Update: Act Two

(Aside: the reasons why this has taken years, when I initially expected to publish the first release in the spring or summer of 2014, are too complex to explain here; I may write about them at a later date. Keywords are health, family and world events.)

Two of the major features of that collection are the OATH Authentication Methods (which includes the algorithm used by Google Authenticator and a number of commercial one-time code fobs) and the Common Platform Enumeration, part of the Security Content Automation Protocol. I implemented the former years ago for my employer, and it has languished in the OpenPAM repository since 2012. The latter, however, has proven particularly elusive and frustrating, to the point where it has existed for two years as merely a header file and a set of mostly empty functions, just to sketch out the API. I decided to have another go at it yesterday, and actually made quite a bit of progress, only to hit the wall again. And this morning, I realized why.

The CPE standard exists as a set of NIST Interagency reports: NISTIR 7695 (naming), NISTIR 7696 (name matching), NISTIR 7697 (dictionary) and NISTIR 7698 (applicability language). The one I’ve been struggling with is 7695—it is the foundation for the other three, so I can’t get started on them until I’m done with 7695.

It should have been a breeze. On the surface, the specification seems quite thorough: basic concepts, representations, conversion between representations (including pseudocode). You know the kind of specification that you can read through once, then sit down at the computer, start from the top, and code your way down to the bottom? RFC 4226 and RFC 6238, which describe OATH event-based and time-based one-time passwords respectively, are like that. NISTIR 7695 looks like it should be. But it isn’t. And I’ve been treating it like it was, with my nose so close to the code that I couldn’t see the big picture and realize that it is actually not very well written at all, and that the best way to implement it is to read it, understand it, and then set it aside before coding.

One sign that NISTIR 7695 is a bad specification is the pseudocode. It is common for specifications to describe algorithms, protocols and / or interfaces in the normative text and provide examples, pseudocode and / or a reference implementation (sometimes of dubious quality, as is the case for RFC 4226 and RFC 6238) as non-normative appendices. NISTIR 7695, however, eschews natural-language descriptions and includes pseudocode and examples in the normative text. By way of example, here is the description of the algorithm used to convert (“bind”, in their terminology) a well-formed name to a formatted string, in its entirety:

6.2.2.1 Summary of algorithm

The procedure iterates over the eleven allowed attributes in a fixed order. Corresponding attribute values are obtained from the input WFN and conversions of logical values are applied. A result string is formed by concatenating the attribute values separated by colons.

This is followed by one page of pseudocode and two pages of examples. But the examples are far from exhaustive; as unit tests, they wouldn’t even cover all of the common path, let alone any of the error handling paths. And the pseudocode looks like it was written by someone who learned Pascal in college thirty years ago and hasn’t programmed since.

The description of the reverse operation, converting a formatted string to a well-formed name, is slightly better in some respects and much worse in others. There is more pseudocode, and the examples include one—one!—instance of invalid input… but the pseudocode includes two functions—about one third of the total—which consist almost entirely of comments describing what the functions should do, rather than actual code.

You think I’m joking? Here is one of them:

function get_comp_fs(fs,i)
  ;; Return the i’th field of the formatted string. If i=0,
  ;; return the string to the left of the first forward slash.
  ;; The colon is the field delimiter unless prefixed by a
  ;; backslash.
  ;; For example, given the formatted string:
  ;; cpe:2.3:a:foo:bar\:mumble:1.0:*:...
  ;; get_comp_fs(fs,0) = "cpe"
  ;; get_comp_fs(fs,1) = "2.3"
  ;; get_comp_fs(fs,2) = "a"
  ;; get_comp_fs(fs,3) = "foo"
  ;; get_comp_fs(fs,4) = "bar\:mumble"
  ;; get_comp_fs(fs,5) = "1.0"
  ;; etc.
end.

This function shouldn’t even exist. It should just be a lookup in an associative array, or a call to an accessor if the pseudocode was object-oriented. So why does it exist? Because the main problem with NISTIR 7695, which I should have identified on my first read-through but stupidly didn’t, is that it assumes that implementations would use well-formed names—a textual representation of a CPE name—as their internal representation. The bind and unbind functions, which should be described in terms of how to format and parse URIs and formatted strings, are instead described in terms of how to convert to and from WFNs. I cannot overstate how wrong this is. A specification should never describe a particular internal representation, except in a non-normative reference implementation, because it prevents conforming implementations from choosing more efficient representations, or representations which are better suited to a particular language and environment, and because it leads to this sort of nonsense.

So, is the CPE naming specification salvageable? Well, it includes complete ABNF grammars for URIs and formatted strings, which is good, and a partial ABNF grammar for well-formed names, which is… less good, but fixable. It also explains the meanings of the different fields; it would be useless otherwise. But apart from that, and the boilerplate at the top and bottom, it should be completely rewritten, including the pseudocode and examples, which should reference fictional, rather than real, vendors and products. Here is how I would structure it (text in italic is carried over from the original):

1. Introduction
   1.1. Purpose and scope
   1.2. Document structure
   1.3. Document conventions
   1.4. Relationship to existing specifications and standards
2. Definitions and abbreviations
3. Conformance
4. CPE data model
   4.1 Required attributes
   4.2 Optional attributes
   4.3 Special attribute values
5. Textual representations
   5.1. Well-formed name
   5.2. URI
   5.3. Formatted string
6. API
   6.1. Creating and destroying names
   6.2. Setting and getting attributes
   6.3. Binding and unbinding
7. Non-normative examples
   7.1. Valid and invalid attribute values
   7.2. Valid and invalid well-formed names
   7.3. Valid and invalid URIs
   7.4. Valid and invalid formatted strings
8. Non-normative pseudo-code
9. References
10. Change log

I’m still going to implement CPE naming, but I’m going to implement it the way I think the standard should have been written, not the way it actually was written. Amusingly, the conformance chapter is so vague that I can do this and still claim conformance with the Terrible, Horrible, No Good, Very Bad specification. And it should only take a few hours.

By the way, if anybody from MITRE or NIST reads this and genuinely wants to improve the specification, I’ll be happy to help.

PS: possibly my favorite feature of NISTIR 7695, and additional proof that the authors are not programmers: the specification mandates that WFNs are UTF-8 strings, which are fine for storage and transmission but horrible to work with in memory. But in the next sentence, it notes that only characters with hexadecimal values between x00 and x7F may be used, and subsequent sections further restrict the set of allowable characters. In case you didn’t know, the normalized UTF-8 representation of a sequence of characters with hexadecimal values between x00 and x7F is identical, bit by bit, to the ASCII representation of the same sequence.

Naturally, people have started asking whether FreeBSD is affected. The FreeBSD Security Officer has not yet released an official statement, but in the meantime, here is a brief look at the issue as far as FreeBSD is concerned.

First of all: neither FreeBSD itself nor native FreeBSD applications are affected. While the resolver in FreeBSD’s libc and GNU libc share a common ancestry, the bug was introduced when the latter was rewritten to send A and AAAA queries in parallel rather than sequentially when the application requests both.

However, Linux applications running under emulation on a FreeBSD system use the GNU libc and are therefore vulnerable unless patched. I believe, but have not verified, that the linux_base-c6 port contains a vulnerable version of GNU libc while the older linux_base-f10 port does not. For now, it is safest to assume that they are both vulnerable.

UPDATE 2016-02-17 18:40 UTC: the linux_base-c6 port has been updated in ports-head, 2016Q1 branch to follow, no word on linux_base-f10

UPDATE 2016-02-18 00:15 UTC: the quarterly branch has been updated

The issue can be mitigated by only using resolvers you trust, and configuring them to avoid sending responses which can trigger the bug.

If you already have your own resolvers, you can configure them to avoid sending UDP responses larger than 2048 bytes. If the response does not fit in 2048 bytes, the server will send a truncated response, and the client should retry using TCP. While a similar bug exists in the code path for TCP requests, I believe that it can only be exploited by a malicious resolver, and interposing your own resolver will protect affected Linux systems and applications.

• If your resolvers run BIND, add the following line to the options section in named.conf (note the semicolon!):

  max-udp-size 2048;
  

• If your resolvers run Unbound, add the following line to the server section in unbound.conf (no semicolon!):

  max-udp-size: 2048
  

• If your resolvers run something else, please consult the documentation and look for a way to restrict the size of UDP responses to 2048 bytes.

If you do not already have your own resolvers, you can set one up in a couple of minutes as follows:

# pkg install -y unbound
# fetch -o /usr/local/etc/unbound https://blog.des.no/wp-content/uploads/2016/02/unbound.conf
# vi /usr/local/etc/unbound/unbound.conf
  ... fix access-control statements as mentioned in comments ...
# echo 'unbound_enable="yes"' >>/etc/rc.conf
# service unbound start

(link to the config file)

You must then edit /etc/resolv.conf on all affected systems to point to your new resolver. If you are running Linux applications under emulation, make sure that there is no /compat/linux/etc/resolv.conf, as it would override the system-wide /etc/resolv.conf.

Note that unlike FreeBSD libc, GNU libc does not automatically pick up changes to /etc/resolv.conf, so you will have to restart all affected Linux services and applications.

Bottom of the barrel here.

Take a look at good health products

checks backpack Lamotrigine, Rizatriptan, Diclofenac and Zopiclone (prescription), Paracetamol and Ibuprofen (over-the-counter) and three sets of single-use contact lenses. Oh, does WD-40 count?

Time for great nights with your female partner

Not bloody likely, what with my chronic insomnia.

Your girlfriend wants you to be the best

That’s the great thing about a healthy relationship, we both want each other to succeed and support each other’s endeavors.

Here you will find some photos

Hey, I’ve been looking for those! Thanks for finding that memory card.

The easiest way to gain more health

Look, I know I could stand to lose a few pounds, but I’m already eating relatively healthy and there’s nothing easy about working out regularly, especially with a busted knee. Now get off my case!