There is a lot I could say in this post about the different approaches I took and the interesting rabbit holes I went down in trying to extract valid base64 data from the images included in the PDF; however, I am somewhat exhausted from all this and don’t have the energy to go into it all in as much detail as it deserves, so I’ll just post some bullet points with the main takeaways:

Properly extracting the PDF images

I had made a mistake in how I converted the original PDF into images for processing. u/BCMM on r/netsec reminded me of a different tool from the poppler-utils package that could extract the images as originally embedded in the PDF, whereas my approach with pdftoppm was unnecessarily re-rasterizing the PDF and possibly reducing the quality (as well as increasing the size).

It turns out I had rasterized the PDF at a high enough resolution/density that it made little to no visual difference, but it did get rid of the Bates numbering/identification at the bottom of each page and was definitely the correct approach I should have used from the beginning. Funnily enough, when I started typing out pdfima— in my shell, a history completion from the last time I used it (admittedly long ago) popped up, so I can’t even claim to have been unaware of its existence.

I re-uploaded the images (of the redacted emails) from EFTA00400459 to a new archive and you can download it here.

OCR is a no-go for base64 text

I learned much more than I ever wanted to¹ about OCR. Some takeaways that I plan on turning into a proper post sometime soon (famous last words, I know):

• I had assumed OCR would be less prone to “hallucinations” than an LLM because the traditional OCR tools are all heuristic/algorithm-driven. This is true, and they do not hallucinate in the same sense that an LLM does, but they are not meant to be used for faithful reconstructions where byte-for-byte accuracy is desired. It turns out they all (more or less) work by trying to make sensible words out of the characters they recognize at a lower level (this is why OCR needs to understand your language, not just its script/alphabet, for best results). This obviously doesn’t work for base64, or anything else that requires context-free reconstruction of the original.
• Tesseract supposedly has a way to turn off at least some of these heuristics, but that did not improve its performance in any measurable way, though there’s always the chance that I was holding it wrong:
  

  The tesseract(1) tunables I tried using to get it to work better with images containing base64 data.

• The vast, vast majority of OCR usecases and training corpora are done with proportional fonts (the opposite of a monospace font). In my experience, all the engines ironically performed better at recognizing images with the much less regularly spaced (and, therefore, harder to algorithmically separate) proportional fonts than they did with monospace fonts.
• Adobe Acrobat OCR is terrible. Just terrible. Don’t pay for it.

Trying a data science approach

My first idea for tackling this without OCR was to identify the bounding boxes for each character on the page (in order, of course) and cluster them using kmeans with k set to 64 (or 65 if you want to consider the = padding). Theoretically, there’s no reason this shouldn’t work. In practice, it just didn’t.

I’m sure the approach itself is sound, but the problem was trying to get regular enough inputs. However hard I tried, I was unable to get the characters sliced regularly and perfectly enough to get scikit-learn to give me even remotely sane results with the KMeans module. I tried feeding it the images thresholded to black-and-white, greyscaled, and even as three-dimensional color inputs (essentially the lossless data). Every time it would create repeated buckets for some letters while sticking the actually different characters into other buckets.

Instead of letting kmeans decide the buckets for me, I decided to try seeding the buckets myself, rendering each letter of the base64 alphabet in Courier New at the right scale/point size to match the scans, then using different methods to bucket the letters into the right bins. But despite these being monospaced fonts, perhaps as an artifact of the resolution the images were captured at, I couldn’t get some glyphs to include a hint of the letter that came before/after, and tweaking it one way (adding a border on one side) would solve it for some inputs but break others (e.g. if you clip even one pixel from the righthand side of the box surrounding the letter O at the font size and DPI the images were captured/rendered in, it turns into a C).

I turned to an LLM for help, but they are really bad at deriving things from first principles even when you give them all the information and constraints beforehand. Gemini 3 kept insisting I try HOG to handle the inputs and would repeatedly insist on trying a connected components approach to weed out noise (some on r/netsec were insisting that these images were 100% noise-free and without any encoding artifacts – this is patently not true), despite the fact that that absolutely destroys the recognition even if you apply the same algorithm to your seed buckets.

This approach with seeding the buckets yielded ~fair results, but it was completely stymied by the l vs 1 conundrum discussed in the previous post. Subpixel hinting differences between the ClearType algorithm used on the PCs the DoJ was using to originally render these emails and the font rendering engine used by OpenCV might look like nothing, but when the difference between an l and a 1 is 3 pixels at most, it adds up. Also, it didn’t help that my bounding box algorithm to draw a grid around the text and isolate each letter wasn’t perfect – I think there was a fractional-pixel mismatch everywhere in the 2x (losslessly) upscaled sources I was using, and that threw everything off (while using the 1x sources made it hard to slice individual characters, though I know the algorithm could have been tweaked to fix this).

Image kernels can’t solve bad typography

A number of people in the comments and on social media kept suggesting applying various kernels to the input in order to get better OCR results. But OCR wasn’t just getting 1 vs l wrong, it was making up or omitting letters altogether, giving me 75-81 characters per line instead of the very-much-fixed 78 monospaced characters I was feeding it. And using them to try and improve the situation with the classifier I wrote turned out to also be futile: if you darken pixels above a certain threshold, you make the difference between the 1 and the l more noticeable, but at that threshold you also darken the subpixel hinting around the lowercase w turning into a capital N.² The kernels to make one pair of letters distinguishable from another would ultimately hurt the recognition of another pair, unless you knew in advance which to apply them to and which not to… which would imply you had already figured out which letter was which!

Except that’s not true: of course you could make a multi-level classifier to first just identify the ambiguous pairs like l and 1 and then feed those into a secondary classifier that applied the kernels, then tried bucketing these separately. But you know what? I didn’t think of that at the time. Mea culpa!

CNNs really are powerful dark magic

With the traditional image processing/data science classification approach not working too well, I decided to try another approach and see if I could train a CNN to handle both the discrepancies with the imperfect slicing around each character/cell, noise from adjacent cells abutting into the cell being analyzed, and the difference between 1 vs l. And you know what? I think I’m going to have to start using CNNs more!

They really are a powerful answer to a lot of this stuff. After just typing out two lines of base64 data from the first page of the PDF³ and training the CNN on those as ground truth, it was able to correctly identify the vast majority of characters in the input corpus, or at least the ones I could visibly tell it had gotten right, meaning I couldn’t be sure if it had handled l vs 1 correctly or not.

It turns out the alignment grid was off by a vertical pixel or two by the time it reached the end of the page, so tweaking the training algorithm to train against the top x lines and the bottom x lines of the page separately got rid of the remaining recognition errors… or at least, it did for the first page.

It’s really hard to see, but while the grid is almost identically placed around the characters at the top-left vs bottom-right, the subtle difference is there.

But alignment issues with how the bounding box was drawn on subsequent pages pushed the error rate well above the necessary zero, and it wasn’t until I realized that since these pages weren’t scanned but rather rendered into images digitally, I could just have the training/inference script memorize the grid from the first page and reuse it on subsequent pages was I able to get rid of all the recognition errors on all pages. (As an interesting sidebar, despite augmenting the training data with artificially generated inputs introducing randomized subtle (-2px to +2px) vertical/horizontal shifts, that wasn’t enough to address the grid drift.)

Well, all except the same pesky 1 vs l which still plagued the outputs and led to decode errors.

I admit I wasted a lot of time here barking up the wrong tree: I spent hours tweaking the CNN’s layout, increasing the kernel size of the first layer, getting rid of the second MaxPool, playing with further augmenting the input samples, trying denoising and alignment techniques, and much more trying to figure out how to induce the CNN into correctly recognizing the two pixel difference between the ones and the ells, all to no avail. I kept trying to increase the training data, meticulously typing out line after line of base64 (becoming thoroughly sick and tired of zooming and panning) to see if that’s what it would take to nudge the error rate down further.

A debug view I added to the training script after I mistyped a character one time too many. It displays the greatest deviation between the average shape of all characters in each training-provided “ground truth” bucket and the max outlier in the same.

It was only after I took a break for my sanity and came back to doggedly tackle it again that I realized one of the errors qpdf reported for the recovered PDF never changed no matter how much I tweaked the CNN or how much additional training data I supplied that I realized the problem: despite zooming in and taking a good 5 or 10 seconds on each 1 vs l I was entering into the training data, I had still gotten some wrong! A second email forward/reply chain with the same base64 content was included in the DoJ archives (EFTA02154109) and while it wasn’t at a much better resolution or quality than the first, it was still different and the 1s and ls were distinguished with different quirks. After I spotted one mistake I had made, I quickly found a few more (and this was even with zooming in on the training corpus debug view pictured above and verifying that the ones and ells had the expected differences!), and lo and behold, the recovered PDF validated and opened!

I posted the code that solved it to GitHub, but apologies in advance: I didn’t take the time to make the harness scripts properly portable. Everything works and they should run on your machine as well as they run on mine, except they’re using a very idiosyncratic mix of tooling. The scripts are written in fish instead of sh or even bash, use parallel (which I actually hate), and have some WSL-isms baked into them for (my) practicality. The README was rushed, and some of the (relative) paths to input files are baked in instead of being configurable. Maybe I’ll get to it. We’ll see.

As to what’s next, well, there are more base64-encoded attachments where this one came from. But unfortunately I cannot reuse this approach as the interesting-looking ones I’ve found are all in proportional fonts this time around!

Follow me at @mqudsi or @neosmart for more updates or to have a look at the progress posts I’ve shared over the past 48 hours while trying to get to the bottom of this.

Thanks for tuning in!

1. Okay, that’s not true. I love learning about random stuff and I don’t know why people say this about anything. ↩

2. If I remember correctly! I wasn’t taking notes (I wish I had, and more screenshots, too) and it could have been two different characters. ↩

3. Well, second page, actually. The first page is mostly just the correspondence with one line of base64, but the second page onwards are 100% base64 content. ↩

There have been a lot of complaints about both the competency and the logic behind the latest Epstein archive release by the DoJ: from censoring the names of co-conspirators to censoring pictures of random women in a way that makes individuals look guiltier than they really are, forgetting to redact credentials that made it possible for all of Reddit to log into Epstein’s account and trample over all the evidence, and the complete ineptitude that resulted in most of the latest batch being corrupted thanks to incorrectly converted Quoted-Printable encoding artifacts, it’s safe to say that Pam Bondi’s DoJ did not put its best and brightest on this (admittedly gargantuan) undertaking. But the most damning evidence has all been thoroughly redacted… hasn’t it? Well, maybe not.

I was thinking of writing an article on the mangled quoted-printable encoding the day this latest dump came out in response to all the misinformed musings and conjectures that were littering social media (and my dilly-dallying cost me, as someone beat me to the punch), and spent some time searching through the latest archives looking  for some SMTP headers that I could use in the article when I came across a curious artifact: not only were the emails badly transcoded into plain text, but also some binary attachments were actually included in the dumps in their over-the-wire Content-Transfer-Encoding: base64 format, and the unlucky intern that was assigned to the documents in question didn’t realize the significance of what they were looking at and didn’t see the point in censoring seemingly meaningless page after page of hex content!

Just take a look at EFTA00400459, an email from correspondence between (presumably) one of Epstein’s assistants and Epstein lackey/co-conspirator Boris Nikolic and his friend, Sam Jaradeh, inviting them to a ████████ benefit:

Those hex characters go on for 76 pages, and represent the file DBC12 One Page Invite with Reply.pdf encoded as base64 so that it can be included in the email without breaking the SMTP protocol. And converting it back to the original PDF is, theoretically, as easy as copy-and-pasting those 76 pages into a text editor, stripping the leading > bytes, and piping all that into base64 -d > output.pdf… or it would be, if we had the original (badly converted) email and not a partially redacted scan of a printout of said email with some shoddy OCR applied.

If you tried to actually copy that text as digitized by the DoJ from the PDF into a text editor, here’s what you’d see:

You can ignore the EFTA00400459 on the second line; that (or some variant thereof) will be interspersed into the base64 text since it’s stamped at the bottom of every page to identify the piece of evidence it came from. But what else do you notice? Here’s a hint: this is what proper base64 looks like:

Notice how in this sample everything lines up perfectly (when using a monospaced font) at the right margin? And how that’s not the case when we copied-and-pasted from the OCR’d PDF? That’s because it wasn’t a great OCR job: extra characters have been hallucinated into the output, some of them not even legal base64 characters such as the , and [, while other characters have been omitted altogether, giving us content we can’t use:¹

> pbpaste \
     | string match -rv 'EFTA' \
     | string trim -c " >" \
     | string join "" \
     | base64 -d >/dev/null
base64: invalid input

I tried the easiest alternative I had at hand: I loaded up the PDF in Adobe Acrobat Pro and re-ran an OCR process on the document, but came up with even worse results, with spaces injected in the middle of the base64 content (easily fixable) in addition to other characters being completely misread and butchered – it really didn’t like the cramped monospace text at all. So I thought to do it manually with tesseract, which, while very far from state-of-the-art, can still be useful because it lets you do things like limit its output to a certain subset of characters, constraining the field of valid results and hopefully coercing it into producing better results.

Only one problem: tesseract can’t read PDF input (or not by default, anyway). No problem, I’ll just use imagemagick/ghostscript to convert the PDF into individual PNG images (to avoid further generational loss) and provide those to tesseract, right? But that didn’t quite work out, they seem (?) to try to load and perform the conversion of all 76 separate pages/png files all at once, and then naturally crash on too-large inputs (but only after taking forever and generating the 76 (invalid) output files that you’re forced to subsequently clean up, of course):

> convert -density 300 EFTA00400459.pdf \
        -background white -alpha remove \
        -alpha off out.png
convert-im6.q16: cache resources exhausted `/tmp/magick-QqXVSOZutVsiRcs7pLwwG2FYQnTsoAmX47' @ error/cache.c/OpenPixelCache/4119.
convert-im6.q16: cache resources exhausted `out.png' @ error/cache.c/OpenPixelCache/4119.
convert-im6.q16: No IDATs written into file `out-0.png' @ error/png.c/MagickPNGErrorHandler/1643.

So we turn to pdftoppm from the poppler-utils package instead, which does indeed handle each page of the source PDF separately and turned out to be up to the task, though incredibly slow:

> pdftoppm -png -r 300 EFTA00400459.pdf out.png

After waiting the requisite amount of time (and then some), I had files out-01.png through out-76.png, and was ready to try them with tesseract:

for n in (printf "%02d\n" (seq 1 76))
    tesseract out-$n.png output-$n \
        --psm 6 \
        -c tessedit_char_whitelist='>'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/= \
        -c load_system_dawg=0 \
        -c load_freq_dawg=0
end

The above fish-shell command instructs tesseract(1) to assume the input is a single block of text (the --psm 6 argument) and limit itself to decoding only legal base64 characters (and the leading > so we can properly strip it out thereafter). My original attempt included a literal space in the valid char whitelist, but that gave me worse results: the very badly kerned base64 has significant apparent spacing between some adjacent characters (more on this later) and that caused tesseract to both incorrectly inject spaces (bad but fixable) and also possibly affect how it handled the character after the space (worse).

Unfortunately, while tesseract gave me slightly better output than either the original OCR’d DoJ text or the (terrible) Adobe Acrobat Pro OCR results, it too suffered from poor recognition and gave me very inconsistent line lengths… but it also suffered from something that I didn’t really think a heuristic-based, algorithm-driven tool like tesseract would succumb to, as it was more reminiscent of how first-generation LLMs would behave: in a few places, it would only read the first dozen or so characters on a line then leave the rest of the line blank, then pick up (correctly enough) at the start of the next line. Before I saw how generally useless the OCR results were and gave up on tesseract, I figured I’d just manually type out the rest of the line (the aborted lines were easy enough to find, thanks to the monospaced output), and that was when I ran into the real issue that took this from an interesting challenge to being almost mission impossible.

I mentioned earlier the bad kerning, which tricked the OCR tools into injecting spaces where there were supposed to be none, but that was far from being the worst issue plaguing the PDF content. The real problem is that the text is rendered in possibly the worst typeface for the job at hand: Courier New.

If you’re a font enthusiast, I certainly don’t need to say any more – you’re probably already shaking with a mix of PTSD and rage. But for the benefit of everyone else, let’s just say that Courier New is… not a great font. It was a digitization of the venerable (though certainly primitive) Courier fontface, commissioned by IBM in the 1950s. Courier was used (with some tweaks) for IBM typewriters, including the IBM Selectric, and in the 1990s it was “digitized directly from the golf ball of the IBM Selectric” by Monotype, and shipped with Windows 3.1, where it remained the default monospace font on Windows until Consolas shipped with Windows Vista. Among the many issues with Courier New is that it was digitized from the Selectric golf ball “without accounting for the visual weight normally added by the typewriter’s ink ribbon”, which gives its characteristic “thin” look. Microsoft ClearType, which was only enabled by default with Windows Vista, addressed this major shortcoming to some extent, but Courier New has always struggled with general readability… and more importantly, with its poor distinction between characters.

You can clearly see how downright anemic Courier New is when compared to the original Courier.

While not as bad as some typewriter-era typefaces that actually reused the same symbol for 1 (one) and l (ell), Courier New came pretty close. Here is a comparison between the two fonts when rendering these two characters, only considerably enlarged:

Comparing Courier and Courier New when it comes to differentiating between 1 (one) and l (ell).

The combination of the two faults (the anemic weights and the even less distinction between 1 and l as compared to Courier) makes Courier New a terrible choice as a programming font. But as a font used for base64 output you want to OCR? You really couldn’t pick a worse option! To add fuel to the fire, you’re looking at SVG outlines of the fonts, meticulously converted and preserving all the fine details. But in the Epstein PDFs released by the DoJ, we only have low-quality JPEG scans at a fairly small point size. Here’s an actual (losslessly encoded) screenshot of the DoJ text at 100% – I challenge you to tell me which is a 1 and which is an l in the excerpt below:

It’s not that there isn’t any difference between the two, because there is. And sometimes you get a clear gut feeling which is which – I was midway through manually typing out one line of base64 text when I got stuck on identifying a one vs ell… only to realize that, at the same time, I had confidently transcribed one of them earlier that same line without even pausing to think about which it was. Here’s a zoomed-in view of the scanned PDF: you can clearly see all the JPEG DCT artifacts, the color fringing, and the smearing of character shapes, all of which make it hard to properly identify the characters. But at the same time, at least in this particular sample, you can see which of the highlighted characters have a straight serif leading out the top-left (the middle, presumably an ell) and which of those have the slightest of strokes/feet extending from them (the first and last, presumably ones). But whether that’s because that’s how the original glyph appeared or it’s because of how the image was compressed, it’s tough to say:

But that’s getting ahead of myself: at this point, none of the OCR tools had actually given me usable results, even ignoring the very important question of l vs 1. After having been let down by one open source offering (tesseract) and two commercial ones (Adobe Acrobat Pro and, presumably, whatever the DoJ used), I made the very questionable choice of writing a script to use yet another commercial offering, this time Amazon/AWS Textract, to process the PDF. Unfortunately, using it directly via the first-party tooling was (somewhat) of a no-go as it only supports smaller/shorter inputs for direct use; longer PDFs like this one need to be uploaded to S3 and then use the async workflow to start the recognition and poll for completion.

Amazon Textract did possibly the best out of all the tools I tried, but its output still had obvious line length discrepancies – albeit only one to two characters or so off on average. I decided to try again, this time blowing up the input 2x (using nearest neighbor sampling to preserve sharp edges) as a workaround for Textract not having a tunable I could set to configure the DPI the document is processed at, though I worried all inputs could possibly be prescaled to a fixed size prior to processing once more:²

> for n in (printf "%02d\n" (seq 01 76))
      convert EFTA00400459-$n.png -scale 200% \
              EFTA00400459-$n"_2x".png; or break
  end
> parallel -j 16 ./textract.sh {} ::: EFTA00400459-*_2x.png

These results were notably better, and I’ve included them in an archive, but some of the pages scanned better than others. Textract doesn’t seem to be 100% deterministic from my brief experience with it, and their features page does make vague or unclear mentions to “ML”, though it’s not obvious when and where it kicks in or what it exactly refers to, but that could explain why a couple of the pages (like EFTA00400459-62_2x.txt) are considerably worse than others, even while the source images don’t show a good reason for that divergence.

With the Textract 2x output cleaned up and piped into base64 -i (which ignores garbage data, generating invalid results that can still be usable for forensic analysis), I can get far enough to see that the PDF within the PDF (i.e. the actual PDF attachment originally sent) was at least partially (de)flate-encoded. Unfortunately, PDFs are binary files with different forms of compression applied; you can’t just use something like strings to extract any usable content. qpdf(1) can be (ab)used to decompress a PDF (while leaving it a PDF) via qpdf --qdf --object-streams=disable input.pdf decompressed.pdf, but, predictably, this doesn’t work when your input is garbled and corrupted:

> qpdf --qdf --object-streams=disable recovered.pdf decompressed.pdf
WARNING: recovered.pdf: file is damaged
WARNING: recovered.pdf: can't find startxref
WARNING: recovered.pdf: Attempting to reconstruct cross-reference table
WARNING: recovered.pdf (object 34 0, offset 52): unknown token while reading object; treating as string
WARNING: recovered.pdf (object 34 0, offset 70): unknown token while reading object; treating as string
WARNING: recovered.pdf (object 34 0, offset 85): unknown token while reading object; treating as string
WARNING: recovered.pdf (object 34 0, offset 90): unexpected >
WARNING: recovered.pdf (object 34 0, offset 92): unknown token while reading object; treating as string
WARNING: recovered.pdf (object 34 0, offset 116): unknown token while reading object; treating as string
WARNING: recovered.pdf (object 34 0, offset 121): unknown token while reading object; treating as string
WARNING: recovered.pdf (object 34 0, offset 121): too many errors; giving up on reading object
WARNING: recovered.pdf (object 34 0, offset 125): expected endobj
WARNING: recovered.pdf (object 41 0, offset 9562): expected endstream
WARNING: recovered.pdf (object 41 0, offset 8010): attempting to recover stream length
WARNING: recovered.pdf (object 41 0, offset 8010): unable to recover stream data; treating stream as empty
WARNING: recovered.pdf (object 41 0, offset 9616): expected endobj
WARNING: recovered.pdf (object 41 0, offset 9616): EOF after endobj
qpdf: recovered.pdf: unable to find trailer dictionary while recovering damaged file

Between the inconsistent OCR results and the problem with the l vs 1, it’s not a very encouraging situation. To me, this is a problem begging for a (traditional, non-LLM) ML solution, specifically leveraging the fact that we know the font in question and, roughly, the compression applied. Alas, I don’t have more time to lend to this challenge at the moment, as there are a number of things I set aside just in order to publish this article.

So here’s the challenge for anyone I can successfully nerdsnipe:

• Can you manage to recreate the original PDF from the Content-Transfer-Encoding: base64 output included in the dump? It can’t be that hard, can it?
• Can you find other attachments included in the latest Epstein dumps that might also be possible to reconstruct? Unfortunately, the contractor that developed the full-text search for the Department of Justice did a pretty crappy job and full-text search is practically broken even accounting for the bad OCR and wrangled quoted-printable decoding (malicious compliance??); nevertheless, searching for Content-Transfer-Encoding and base64 returns a number of results – it’s just that, unfortunately, most are uselessly truncated or only the SMTP headers from Apple Mail curiously extracted.

I have uploaded the original EFTA00400459.pdf from Epstein Dataset 9 as downloaded from the DoJ website to the Internet Archive, as well as the individual pages losslessly encoded to WebP images to save you the time and trouble of converting them yourself. If it’s of any use to anyone, I’ve also uploaded the very-much-invalid Amazon Textract OCR text (from the losslessly 2x’d images), which you can download here.

Oh, and one final hint: when trying to figure out 1 vs l, I was able to do this with 100% accuracy only via trial-and-error, decoding one line of base64 text at-a-time, but this only works for the plain-text portions of the PDF (headers, etc). For example, I started with my best guess for one line that I had to type out myself when trying with tesseract, and then was able to (in this case) deduce which particular 1s or ls were flipped:

> pbpaste
SW5mbzw8L01sbHVzdHJhdG9yIDgxIDAgUj4+L1Jlc29lcmNlczw8L0NvbG9yU3BhY2U8PC9DUzAG
> pbpaste | base64 -d
Info<</Mllustrator 81 0 R>>/Resoerces<</ColorSpace<</CS0
> 
> # which I was able to correct:
>
> pbpaste
SW5mbzw8L0lsbHVzdHJhdG9yIDgxIDAgUj4+L1Jlc291cmNlczw8L0NvbG9yU3BhY2U8PC9DUzAG
> pbpaste | base64 -d
Info<</Illustrator 81 0 R>>/Resources<</ColorSpace<</CS0

…but good luck getting that to work once you get to the flate-compressed sections of the PDF.

I’ll be posting updates on Twitter @mqudsi, and you can reach out to me on Signal at mqudsi.42 if you have anything sensitive you would like to share. You can join in the discussion on Hacker News or on r/netsec. Leave a comment below if you have any ideas/questions, or if you think I missed something!

UPDATE

This was solved last night (February 6, 2026). You can read about it in the follow-up to this article.

1. In case you’re wondering, the shell session excerpts in this article are all in fish, which I think is a good fit for string wrangling because of its string builtin with extensive operations with a very human-readable syntax (at least compared to perl or awk, the usual go-tos for string manipulation), and it lets you compose multiple operations as separate commands while not devolving to performing pathologically because no external commands are fork/exec‘d because of its builtin nature. And I’m not just saying that because of the blood, sweat, and tears I’ve contributed to the project. ↩

2. I didn’t want to convert the PNGs back to a single PDF as I didn’t want any further loss in quality. ↩

Let’s say you want to shard based off a UUIDv7 key: maybe to assign data across a horizontally-distributed database, or to avoid having a couple of million files in the same directory. You can’t use the timestamp portion to derive a shard key, as not even the microseconds are necessarily uniformly distributed, so you need to shard based off the random portion of the UUID.

Lucky for us, the last eight bytes of the UUID contain random data not just in the case of a UUIDv7 – those bytes also contain random data in the case of UUID v3 (MD5-hashed), UUID v4 (randomized), and UUID v5 (like v3 but using SHA-1), letting us use one approach to hash all four of these UUID types.

With some judicious use of the stabilized subset of const generics, we can generate a shard key N-chars long without any allocation, and provide a wrapper ShardKey type to simplify both creating the shard key and passing it around in a way that lets it decompose to a &str in a way that wouldn’t have been possible if we had returned a [char; N] instead.²

/// Derives a deterministic N-char sharding key from a UUID.
///
/// It extracts entropy starting from the last byte (the most random 
/// part of a UUIDv7) moving backwards, converting nibbles to 
/// hexadecimal characters. It uses only the last 8 bytes (the `rand_b`
/// section) to ensure the timestamp (first 6 bytes) is never touched, 
/// preventing "hot spot" sharding issues based on time.
///
/// # Panics
/// * Panics if the provided UUID is not shardable based off the last 
///   8 bytes (i.e. if the UUID is not one a v3, v4, v5, or v7 UUID).
/// * Panics if the requested shard key length `N` exceeds the 
///   available entropy.
pub fn shard<const N: usize>(uuid: &uuid::Uuid) -> ShardKey<N> {
    // Ensure UUID random bytes assumption
    if !matches!(uuid.get_version_num(), 3 | 4 | 5 | 7) {
        panic!("Provided UUID cannot be sharded with this interface!");
    }
    // Validate shard key length
    if N > 15 {
        panic!("Requested shard key length exceeds available entropy!");
    }

    const HEX: &[u8; 16] = b"0123456789abcdef";
    let bytes = uuid.as_bytes();
    let mut result = [b'\0'; N];

    // Generate N-char shard key
    for i in 0..N {
        // We extract data from the tail (byte 15) backwards to byte 8.
        // Bytes 8-15 in UUIDv7 contain 62 bits of randomness + 2 bits 
        // to indicate variant. We avoid bytes 0-7 (Timestamp + Version) 
        // to ensure uniform distribution.

        // Calculate byte index, consuming two nibbles per byte.
        let inverse_offset = i / 2;
        let byte_index = 15 - inverse_offset;

        let byte = bytes[byte_index];

        // Even indices take the low nibble; odd indices take the high nibble.
        let nibble = if i % 2 == 0 {
            byte & 0x0F
        } else {
            (byte >> 4) & 0x0F
        };

        result[i] = HEX[nibble as usize];
    }

    ShardKey { key: result }
}

#[repr(transparent)]
#[derive(Copy, Clone, PartialOrd, Ord, PartialEq, Eq, Hash)]
pub struct ShardKey<const N: usize> {
    key: [u8; N],
}

impl<const N: usize> ShardKey<N> {
    pub fn new(uuid: &uuid::Uuid) -> Self {
        shard(uuid)
    }

    pub fn as_str(&self) -> &str {
        // Safe because only we have access to this field and we always
        // initialize it fully with ASCII-only characters.
        unsafe { std::str::from_utf8_unchecked(&self.key) }
    }
}

The shard() function can be used standalone, or the ShardKey wrapper type can be created from a reference to a &Uuid and then passed around until the point where you need to retrieve the key with a quick .as_str() call. As promised, the length of the shard key itself is variable and parameterized, by providing different values for N you can generate shard keys from one to fifteen (or sixteen, if you prefer) characters long (past which point the available uniformly-distributed entropy has been exhausted and the function will panic to ensure contract is upheld).

1. Of course, non-standardized solutions abound and UUIDv7 itself takes a lot of inspiration from predecessors like Ulid and others. ↩

2. As rust is famously one of the very few languages that uses different-sized units/types to represent a standalone char versus the smallest type used to compose a String; in rust, char is 4-bytes long and has UTF-32 encoding while a String or &str uses an underlying 1-byte u8 type and are UTF8-encoded. This means you can’t go from a String to a [char] without allocating, nor go from a [N; char] to an &str without looping and allocating, either. ↩

You could just do a quick :saveas! file1 and :saveas! file2 in the respective buffers to set things right, but of course you have autoread enabled, so correcting the file in one buffer will cause the other copy in the second buffer to be immediately lost!¹

The solution is a quick :0f away: it’s a shortcut for :filename "" that clears the association between the buffer and the underlying file, leaving the contents in the buffer unchanged but severing the link with the underlying inode, meaning after you run :0f in one buffer, you can swap back to the other and save it to the correct path without risking the other buffer’s contents from being lost, as it’s been dissociated from the underlying file. Then you can go back to the now-unnamed buffer and save it to correct path too, and all will be well.

1. Yes, of course you can undo the autoread and then :saveas again, but do you really want to be taking that risk with your hours of work? ↩

Take for example this snippet from a Dockerfile that sets up permissions on certain directories:

mkdir -p /var/log/php /var/log/unitd /var/log/mysql
chown -R user:user /var/log/php /var/log/unitd /var/log/mysql
chmod ug+rw /var/log/php /var/log/unitd /var/log/mysql

You can immediately see how the /var/log/ prefix is a source of repetition: wasting valuable horizontal screen real estate, extra room for typos or mistakes, and diluting the signal:noise ratio of the lines in question.

In most shells,¹ you can use braces ({})² to accomplish the same with much less repetition:

mkdir -p /var/log/{php,unitd,mysql}
chown -R user:user /var/log/{php,unitd,mysql}
chmod ug+rw /var/log/{php,unitd,mysql}

..which expands to the same, effectively operating as a declarative “for loop” running for each of the items within the braces; e.g. echo {hello,goodbye}_world prints hello_world goodbye_world (mxn complexity with each braced/non-braced component being one set) while echo {hello,goodbye}_world{.com,.exe} prints hello_world.com goodbye_world.com hello_world.exe goodbye_world.exe, or mxnxo complexity with m equal to two, n equal to one, and o equal to two for four results.

Depending on the shell, this functionality can be taken a step further and used to do questionable things, like combining with glob expansion to get a list of all files with a certain extension that appear in one of n different directories with an expression like foo/{bar,baz}/*.{c,h} or anything else your heart desires.

Alas, the one issue with Cartesian/brace expansion is that it isn’t formalized in the posix sh spec, meaning you can’t use it in truly portable shell scripts or with a #!/bin/sh shebang. But fret not, because you can still accomplish the same thing with another feature: subshell output interpolation.

To jump right to it, here is portable syntax that will accomplish the same as our original demonstration snippet, except this is truly portable and you can use it in your Alpine Linux Dockerfile running sh:

mkdir -p $(printf "/var/log/%s\n" php unitd mysql)
chown -R user:user /var/log/$(echo php unitd mysql)
chmod ug+rw /var/log/$(printf "%s\n" php unitd mysql)

As you can see, there are several ways it can be used, relying on printf to perform the expansion for you directly (taking advantage of the fact that printf is spec’d to repeat the input when it is provided with more parameters than placeholders), use echo instead of printf if your $IFS configuration splits on spaces, or use printf %s\n if you want to support spaces in paths and have $IFS set to split only on new lines (or you’re smart and sane and use fish which defaults to splitting only on new lines anyway) – in which case the second example with echo instead of printf %s\n won’t work for you.

So keep it DRY³ and embrace the power of brace expansion (or Cartesian expansion in general, if you don’t have access to brace expansion in your shell of choice) the next time you’re banging out a shell script!

1. Including, but not limited to, fish (the one true shell), bash, zsh, ksh (as of ksh88), zsh, and csh/tcsh (where this feature originated). But notably not supported by dash and not part of the posix sh spec. ↩

2. These are called braces, or sometimes curly braces. They are not brackets ([]) and there is no such thing as a “curly bracket”. ↩

3. Don’t Repeat Yourself ↩

In a New Year’s tweet earlier today, the maintainer of the site going by the alias of Zionism Observer on Twitter detailed the suspension (with the registrar placing the domain under a clientHold suspension) of the Genocide.live domain name, under the seemingly ridiculous claim of its hosting material that “promotes, encourages, engages or displays cruelty to humans or animals:”

It’s true that the site’s content did indeed depict gross violations of human rights (along with 70+ videos of animal rights violations), but it could not be honestly (or even mistakenly) described as being posted for purposes of “promoting, encouraging,” or glorifying such violations – an accusation much akin to doggedly accusing Wikipedia of being a pornography site because it contains medical and anatomical depictions of the human body. The maintainer has also clarified that all sensitive videos and imagery were placed behind a click-to-view overlay with a warning about the violent or triggering nature of the content.

While the site has been reportedly mirrored and backed up to multiple locations, there remain obvious problems with this seizure. As various links to individual videos from the archive – all now broken – have been entered into evidence with various international and supranational courts in multiple ongoing proceedings opened against the State of Israel (and, separately, individual Israeli soldiers) over the past couple of years, the takedown of the domain could have severe legal ramifications, if not rectified posthaste, – and could potentially open Namecheap to accusations of obstruction of justice, which their General Counsel might wish to consider.¹

A citation of the Genocide.live archives in legal proceedings accusing Israel of war crimes and genocide in Gaza, provided by the site’s maintainer.

An instance of Genocide.live, née TikTokGenocide.com, in UN Security Council proceedings from February 2025.

The archive had already come into the legal spotlight earlier last year, when it was forced to change from its original name of TikTokGenocide to Genocide.live after cease-and-desist requests from the new owners of the popular social media platform targeted its use of the domain name – despite its arguably fair-use claim to the name.²

Namecheap is no stranger to accusations of frivolous or unjustified domain name suspensions – a search on Reddit reveals hundreds of such complaints in recent years – but neither are any of the other large domain name registrars exempt from reports of such behavior. The site’s maintainer suggests that the decision to suspend the domain over the New Years holiday might not have been coincidental, as it robbed them of a 48-hour notice of impending revocation (with, apparently, no option to appeal or contest the charges), during which time they might have been able to transfer the domain to a different registrar.

The seizure of the domain has gone somewhat viral on Twitter, where it has become a trending topic with users speculating on the motivation and suggesting that the domain registrar might have been the target of an email campaign by those unhappy with the archive’s existence or growing prominence.

In addition to hosting over 16,000 videos of evidence documenting evidence of war crimes by Israeli soldiers and examples of intent of genocide from Israeli military and civil leaders, the Genocide.live archive also included an interactive map of Gaza detailing IDF violations against the populace in each area, a geolocated index of the videos for which location data was positively determined, a categorized listing of videos detailing the nature of violations, an extensive index of the different types of victims of Israeli agression, a cross-indexed reference of various weapons of war used, and, perhaps most sensitively of all, a cross-indexed list of individual IDF brigades and battalions tied to each of the hosted pieces of evidence, where that information was available.

The site’s maintainer noted with concern an anomaly in the traffic the archive was receiving from Israel, and had earlier reported recent attempts to probe their infrastructure from the same.

Genocide.live is a part of the Databases for Palestine project, a collective founded in December of 2023 using tech to shed light on the terrible situation in Gaza and the acts of the Israeli government and army that contributed to Israel being credibly accused of committing genocide in Palestine by prominent human rights organizations including Amnesty International, Médecins Sans Frontières, and Human Rights Watch, among others.

Requests for comment from Namecheap have not yet been met with a response.

Update January 5, 2026
Following approximately 96 hours of downtime, the Genocide.live video archive is now accessible once more. Namecheap agreed to un-suspend the domain in response to significant social media backlash, but refused to continue acting as the archive’s registrar. After some extended back-and-forth, the registrar-enforced clientTransferProhibited was ultimately removed, allowing the site’s maintainers to move the domain to Trustname, their new registrar.

1. Interestingly, Namecheap does not publicly name their Chief Legal Officer/General Counsel, leading even the Office of the Attorney General of the State of New York to leave them unnamed in previous legal petitions. ↩

2. The previous name was a not unsubtle jab at the public broadcasting of war crimes by IDF soldiers as they boasted about the destruction they wrought in Gaza. ↩

This post is less of a deep dive into a bug I ran into upgrading an x86_64 machine from FreeBSD 14.3 to FreeBSD 15 and more of a PSA: I have a possible workaround for anyone that runs into the same, but I don’t have a full root analysis or proper diagnosis of what the underlying issue was.

FreeBSD 15.0 was released a week ago, and I decided to try to upgrade one of my ZFS appliance servers (running nothing more than ZFS and some scripts) to it as a possible low-stakes trial run. The machine in question is a fairly old (but very reliable) Dell PowerEdge R720, running FreeBSD 14.3p2 and booting in UEFI mode from a ZFS zroot pool at the time.

As always, I started my FreeBSD upgrade with the usual sudo zfs snap -r zroot@freebsd-14.3p2 prior to anything else (yes, I know about ZFS boot environments, but I also know that ZFS snapshots are fast and free). The first part of the upgrade went swimmingly after installing the newest version of freebsd-rustdate from the pkg repos and executing freebsd-rustdate upgrade -r 15.0-RELEASE followed by freebsd-rustdate install; the initial upgrade of the kernel components to 15.0-RELEASE went well, and I was prompted to restart the system… and that’s when the troubles began.

This was a headless machine racked elsewhere, and to my dismay I found that I wasn’t able to SSH into it after waiting the requisite amount of time. After discovering I wasn’t even able to ping the machine, I resigned myself to attaching a display and keyboard to the machine and went to see what was going on. Despite it being at least ten minutes (and probably more) since the initial part of the upgrade had succeeded, connecting the display caught the final moments of the machine as it was syncing disks in prep for a reboot. Had some service really taken that long to shut down after I had executed sudo reboot?

Despite seeing what finally looked like progress, I figured I might as well monitor the system come back online since I was already there with the display and keyboard hooked up. After the agonizingly long POST process followed by Dell’s inventory manager startup and the HBA additional boot-time payload execution (and Intel’s iSCSI firmware, and, and, and), the machine began to boot FreeBSD. I looked away for a second.. only to see the last vestiges of a shutdown notice as the system began to reboot just after coming up! Ten minutes later, this time with my iPhone recording video in slow motion,¹ I was able to unfortunately verify that it wasn’t a fluke, and the machine was indeed stuck in a reboot loop that kicked off just after all the services were brought up successfully and the kernel’s writes to the vtty quiesced. It didn’t look like anything had gone wrong: there was no kernel panic, services like sshd were being shut down in an orderly fashion, and the disks were synced before the machine would completely power cycle.

I naturally tried to boot up in single-user mode, both by pressing 2 at the UEFI bootloader menu and by manually executing boot -s at the bootloader prompt: they both failed in the same, exact manner. Stymied and really frustrated with how long each attempt took thanks to all the boot-time firmware applications and processes that had to happen before the system began to actually boot, I shut down the machine, yanked out the OS drive, and connected it to a spare desktop to be able to iterate more quickly.

And imagine my surprise when the FreeBSD 15 install booted up just fine with the disk in the test rig, with no hint of a reboot loop even in normal/multi-user mode!

I took the opportunity of having the machine fully boot up in normal mode to investigate the situation from the comfort of my regular environment, with my preferred keyboard layout, tools, and editors at my service (much better than a rescue shell!), and was surprised to see absolutely nothing out of the ordinary. The logs didn’t reveal any errors, no watchdog timers kicking in, no kernel panics (as I’d already surmised), no filesystem errors, no hardware failures (that were logged, at any rate). At this point, my best guess was that there was possibly something amiss with the ACPI support for this machine in the FreeBSD 15 release.²

Before trying to disable ACPI in the bootloader configuration, I figured I would try one last thing: finish the upgrade by updating the FreeBSD userland from 14.3 to 15.0, so I ran the requisite commands to bootstrap pkg, upgrade all installed packages, and then finish the freebsd-rustdate/freebsd-update install process. With the update fully complete, I ejected the root disk from the test machine and connected it back to the R720 server, though not with much hope of success.

But my skepticism was unwarranted: the fully upgraded userland didn’t exhibit the same symptoms I’d seen earlier and booted right into multiuser mode without a hint of a reboot loop! I had been prepared to really dive in to the issue, disabling ACPI to see if that fixed the problem, and bisecting my way through the system services to see if any were responsible (even though this happened in single user mode as well).

So if anyone finds themselves stuck with a reboot loop after upgrading to FreeBSD 15, try sticking the disk in another machine and completely upgrading the userland to FreeBSD 15 to see if that takes care of your problem. It did for me – though it certainly left me feeling particularly dissatisfied that I wasn’t able to figure out what actually caused the issue, and of course, always wondering if it would start happening again.³

1. A really dumb – yet undeniably useful – trick I learned to capture everything that takes place prior to a kernel panic or other sudden reboot, if the machine isn’t connected over serial. Don’t knock it until you’ve tried it, the iPhone camera is fast enough to capture everything that gets dumped to screen and you can scrub through it as slowly as you like. ↩

2. Although an errant interpretation of the ACPI data would normally result in a full shutdown, not a system reboot. ↩

3. Which is the only healthy reaction to heisenbugs if you’re not able to determine what’s causing them! ↩

A fully functional ZFS setup following ZFS best practices and Linux/Ubuntu idiomatic approaches

This guide will focus mainly on the Linux sysadmin side of things; note that basic knowledge and understanding of ZFS concepts and principles is assumed, but I’ll do my best to provide a succinct summary of what we’re doing and why we’re doing it at each point.

Step 1: Installing ZFS

Unlike on FreeBSD, on Linux you need to manually install the ZFS kernel modules and userland tooling to bring in ZFS filesystem support and install the venerable zfs and zpool utils used to manage a ZFS installation. Canonical’s Ubuntu was, to my knowledge, the first to offer a pre-packaged ZFS option for Linux users (after gambling Oracle wouldn’t sue them for violating the CDDL license if they included ZFS support in their repos), and I believe it’s still the most popular Linux distribution for ZFS users, so the specific command line incantations below are for Ubuntu:

sudo apt install zfs-dkms

This will download, build, and install the ZFS kernel modules to match the version of the Linux kernel you’re currently running. Unlike most kernel modules,¹ ZFS support isn’t built or distributed as part of the base kernel that Canonical maintains for its distributions, instead you have to manually build and load the kernel module that provides ZFS support (but this is automated by the .deb installed with apt) – and this needs to be done each time you upgrade the kernel.² All this really means is that installing zfs-dkms will take longer than installing most packages – expect the installation process to look like it’s stuck and be extra patient. Installing zfs-dkms will also pull in an automatic dependency on the userspace tools, zfsutils-linux, as well as other ZFS-related libraries and dependencies.

Step 2: Setting up your zpool

This part of the process is largely going to be the same regardless of which operating system you are using and is standard ZFS fare. You’ll need to identify the drives you wish to use in your zpool (the ZFS abstraction over the physical disks, arranged in the hierarchy/topography you desire) and use sudo zpool create to create your first zpool (traditionally named tank). The only thing of note here is that you should use a stable path available to identify your disks, so instead of doing something like sudo zpool create tank mirror /dev/sda /dev/sdb to create a two-disk mirror zpool comprised of the two disks /dev/sda and /dev/sdb, you should instead use a different path to the same device, such as via /dev/disk/by-id/ or /dev/disk/by-uuid/ (going with by-id/ might make it easier to figure out which disk to use, as the contents of by-uuid/ are all GUIDs).³ On Linux, lsblk is your friend here to list disks attached to the system.

# to create a mirror of the two volumes:
sudo zpool create -o ashift=12 tank mirror /dev/disk/by-id/scsi-0VOLUME_NAME_01 /dev/disk/by-id/scsi-0VOLUME_NAME_02

And you can verify that the operation has succeeded by using `zpool list` to see the list of zpools live on the system.

Most ZFS properties and features are configurable, can be set at any time,⁴ and are inherited from parent datasets. Let’s set some default properties that are good starting values (we can always change them or override them for specific child datasets at any time):

sudo zfs set compression=lz4 tank # or =zstd if you're on the very newest versions
sudo zfs set recordsize=1m tank # i/o optimization for storing content that rarely changes

Step 3: Creating your ZFS datasets

The “zpool” is, as mentioned, the abstraction over the physical disks in your PC. It’s closest analogue is a “smart disk” comprised of multiple physical disks arranged in some specific topography with certain striping/redundancy/parity managed by the lower-level zpool command. Just as a zpool is a virtual disk, a dataset is a “smart partition” used to break up your “disk” into multiple logical storage units. Unlike real partitions, zfs datasets aren’t fixed in size, they rather straddle the line between a partition and a folder. They can be nested (like a folder), but you can’t rename a file across datasets (like a separate filesystem/partition). You can snapshot them individually (or altogether, atomically) for backup and cloning purposes (see below), and its the finest-grained level of control you have for turning on/off or re-configuring ZFS features and properties like the record size (only available to the time of creation), automatic block-level compression, etc.

While ZFS automatically creates a dataset for the root zpool (in this case we now have /tank/ mounted and ready) but it’s generally not a good practice to write directly to this dataset. Instead, you should create one or more child datasets where most content will go. We’ll just create one dataset for now:

sudo zfs create tank/data

and we can see all our datasets with `zfs list`, which shows them in their hierarchy/tree as configured.

Step 4: Configuring automatic snapshots

One of the coolest and most important ZFS features is undoubtedly its instant, zero-cost snapshotting (enabled by its copy-on-write design). This lets you freeze an image of any dataset (or all of them) as it exists at any point of time, then restore back to it (or selectively copy files/data back, as needed) at any point in the future, regardless of any changes you’ve made. (You only pay the storage cost of data added or deleted thereafter.) ZFS snapshots can be made manually with sudo zfs snap -r tank@snapshot_name_or_date (which snapshots tank and all its child datasets, instantly) or sudo zfs snap tank/data@snapshot_name (which snapshots only the one tank/data dataset). But since they’re virtually free, why not go a step further and automatically take snapshots of the data on a schedule? That way you’re protected in case of inadvertent data loss, not just when you take a snapshot before manually performing a known potentially destructive action.

On Linux, the best way to automate these snapshots is with zfs-auto-snapshot, which we’ll install with sudo apt install zfs-auto-snapshot. It’ll automatically create new snapshots every month/week/day/hour of designated zfs datasets, and delete the oldest ones too so you’re not paying the storage price forever.

After installing zfs-auto-snapshot, it’s time to choose which datasets we want to protect and how often we want to take the snapshots. Instead of using a configuration file, zfs-auto-snapshot uses zfs properties to determine which zfs datasets to include in each snapshot interval, and since zfs properties are inherited by default, if we set up snapshots for the root dataset, it’ll automatically do the same for all child datasets.

Let’s enable a daily snapshot of the root volume (and all child datasets):

sudo zfs set com.sun:auto-snapshot:daily=true tank

You can repeat this but replace daily with one of daily, monthly, weekly, or hourly to (additionally) opt-into that frequency of snapshots. To exclude a dataset (and its children) from being included in a particular schedule, you can e.g. use sudo zfs set com.sun:auto-snapshot:daily=false tank/no_backups to turn off daily snapshots for the tank/no_backups dataset (assuming it exists).

You can check if this is working (after waiting the prescribed amount of time) by checking to see what snapshots you have listed:

zfs list -t snap

Step 5: Automatic monthly ZFS scrubs on Linux with systemd

One thing that makes ZFS stand out compared to other operating systems like ext4 or even xfs is that it calculates the hash of each block of data you store on it. In the event of bitrot (the silent corruption of data already stored to disk), zfs can a) flag that a file has been silently corrupted, b) automatically restore a good copy from another disk or parity (assuming your zpool topography provides redunancy).⁵ It does this automatically every time you read a file, but what about if you have terabytes of data just sitting there, silently rotting away? How do you catch that corruption in time to fix it from a second copy on the zpool? The zpool scrub tank operation runs a low-priority scan in the background to detect and (hopefully) repair just that, but it needs to be scheduled (or run manually).

On FreeBSD, this would be accomplished with the help of a simple monthly periodic script, but on Linux (Ubuntu particularly) it’s not as simple. The idiomatic way of setting up monthly work on Ubuntu is via the use of systemd units (aka services) and timers, unfortunately this requires setting up two separate files, but the good news is that you can just copy and paste what I’ve provided below, only modifying the zpool name from tank to whatever you are using, as needed.

The first file we need to create is the actual systemd service, which is what is tasked with running running the zfs scrub operation. Copy the following to /etc/systemd/system/zfs-scrub.service:

[Unit]
Description=ZFS scrub of all pools

[Service]
Type=oneshot
ExecStart=/usr/sbin/zpool scrub tank

And copy this timer file (which specifies when zfs-scrub.service is automatically run) to /etc/systemd/system/zfs-scrub.timer:

[Unit]
Description=Run ZFS scrub service monthly

[Timer]
OnCalendar=monthly
# Run if missed while machine was off
Persistent=true
# Add some randomization to start time to prevent thundering herd
RandomizedDelaySec=30m
AccuracySec=1h

[Install]
WantedBy=timers.target

Then execute the following to get systemd to see and activate the monthly scrub service:

systemctl daemon-reload
systemctl enable --now zfs-scrub.timer

and verify that the timer has been started with the following:

systemctl list-timers zfs-scrub.timer

which should show you output along the lines of the following:

$ systemctl list-timers zfs-scrub.timer
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2025-10-01 00:23:31 UTC 2 weeks 6 days - - zfs-scrub.timer zfs-scrub.service

1 timers listed.

You can see if this is working by checking when the last scrub took place with zpool status:

$ zpool status
pool: tank
state: ONLINE
scan: scrub repaired 0B in 00:00:00 with 0 errors on Wed Sep 10 18:43:22 2025

And with that, you’re all set!

1. Largely due to licensing restriction workarounds ↩

2. This too is *normally* taken care of by the package manager, provided all the packages have been correctly built and uploaded to the repository by the time you try to install a newer kernel version. There’s very little you have to do manually. ↩

3. You could use /dev/disk/by-path/ but that means if you physically swap disks around in their cages the references would become switched around, so it’s best not to. ↩

4. The most notable exception to this is the ashift property set above with -o ashift=12, which is a decent value for any ssd or 4k/512e hdd. ↩

5. Or assuming you are using zfs set copies=2 tank (or greater). ↩

I’ve been an sccache user almost from the very start, when the Mozilla team first introduced it in the rust discussions on GitHub maybe seven years back or so, probably because of my die-hard belief in the one-and-only ccache earned over long years of saving my considerable time and effort on C++ development projects (life pro-tip: git bisect on large C or C++ projects is a night-and-day difference with versus without ccache). At the same time, I was always painfully aware of just how little sccache actually cached compared to its C++ progenitor, and I was left feeling perpetually discontented ever since learning to query its hit rate stats with sccache -s (something I never needed to do for ccache).

But my blind belief in the value of build accelerators led me to complacency, and I confess that with sccache mostly chugging away without issue in the background, I kind of forgot that I had RUSTC_WRAPPER set at all. But I recently remembered it and in a bout of procrastination, decided to benchmark how much time sccache was actually saving me… and the results were decidedly not great.

The bulk of my rust use at $work is done on a workstation under WSLv1, while a lot of of my open source rust work is done on a “mobile workstation” running a Debian derivative. I began my investigation on the WSL desktop, originally spurred by a need desire to see what kind of speedups different values for the ~recently added -Z threads flag to parallelize the rustc frontend would net me, then remembered that I had sccache enabled and should disable that for deterministic results… leading me to include the impact of sccache in my benchmarks.

On a (well-cooled, not thermally throttled) 16-core (32-thread) AMD Ryzen ThreadRipper 1950X machine with 64 GB of DDR4 RAM and NVMe boot and (code) storage disks, using rustc 1.81.0-nightly to compile a relatively modest-sized rust project from scratch in debug mode (followed by exactly one clean rebuild in the case of sccache to see how much of a speedup it gives), I obtained the following results:

Shockingly enough, there was not a single configuration where the usage of sccache ended up speeding up compilation over the baseline without it — not even on a subsequent clean build of the same code with the same RUSTFLAGS value.

After inspecting the results of sccache -s and with some experimentation, it turned out that this pathological performance appeared to be caused – in part – by having a full sccache cache folder.² Running rm -rf ~/.cache/sccache/* then re-running the benchmark (a few project commits later, so not to be directly compared to the above) revealed a significant improvement to the baseline build times with sccache… but probably not enough to justify its use:

Looking at the chart above, using sccache slowed down the clean build by anywhere from 23% (in the case of no -Zthreads) to 153% (in the case of -Zthreads set to 8 or 16), while providing a speed up to subsequent builds with identical flags and unchanged code/toolchain only in the case of no -Zthreads (6% speedup) and by 19% in the case of -Zthreads set to 0 (its default value), but it still managed to slow down even subsequent, clean builds with a fully primed cache as compared to the no-sccache baseline by from 25% (-Zthreads=2) to 51% (-Zthreads=32).

Analyzing the benefits of -Zthreads is much harder. Looking at the cases with -Zthreads but no sccache, it appears that with a heavily saturated pipeline (building all project dependencies from scratch in non-incremental mode affords a lot of opportunities for parallelized codegen), the use of -Zthreads can provide at best a very modest 5% speed up to build times (in the case of -Zthreads=4) while actually slowing down compile times by 14% (the soon-to-be-default -Zthreads=0) and 12% (with -Zthreads=16).³

It’s interesting to note that the rust developers were rather more ebullient when introducing this feature to the world, claiming wins of up to 50% with -Zthreads=8 and suggesting that lower levels of parallelism would see lower speedups (the opposite of what I saw, where using 8 threads provided about half the benefit of using 4, and going any higher caused slow-downs rather than speed-ups). Note that I was compiling in the default dev/debug profile above, so maybe I should try and see what happens in release mode, though I would think the architectural limitations would persist.

Back to sccache, though.

One of the open source projects I contribute to most is fish-shell, which recently underwent a complete port from C++ to Rust (piece-by-piece while still passing all tests every step of the way). Some day I want to write at length about that experience, but the reason I’m bringing this up is because my fish experience has taught me that some things that are normally very fast under Linux can run much slower than expected under WSL, primarily due to I/O constraints caused by the filesystem virtualization layer. I haven’t dug into it yet, but going with the working theory that reading/writing some 550-800 MiB⁴ was slowing down my builds (even with the code and the cache located on two separate NVMe drives), I moved on to my other machine (where I’m running Linux natively).

Running the same benchmark with the same versions of rustc and sccache on the 4-core/8-thread Intel Xeon E3-1545M v5 (Skylake) with 32GiB of DDR3 RAM gave the following results, which were much more in line with my expectations for sccache (though even more disappointing when it came to the frontend parallelization flag):

Here at last are the sccache results I was expecting! A maximum slowdown of about 8% for uncached builds and speedups of about 80% across the board for a (clean) rebuild of the same project immediately after caching the artifacts.⁵

As for -Zthreads, the results here are consistent with what we saw above, at least once you take into account the fact that there are significantly fewer cores/threads to distribute the parallelized frontend work across. But we’re left with the same conclusion that at times when the CPU is already handling a high degree of concurrency with jobserver already saturated with work from the compilation pipeline across multiple compilation units from independent crates, adding further threads to the mix ends up hurting overall performance (to the tune of 14% in the worst case, when -Zthreads is set to total number of cores). The good news is that adding just a slight degree of parallelization with -Zthreads=2 doesn’t hurt build times in this worst-case scenario and likely helps when the available threads aren’t already saturated with more work than they can handle, so that at least seems to be a safe value for the option for now.

I would have expected that -Zthreads wouldn’t “unilaterally” dictate the number of chunks frontend work was being broken up into. While I’m sure it integrates nicely with jobserver to prevent an insane number of threads from being spawned and overwhelming the machine, it would seem that dividing the frontend work into n chunks when there aren’t n threads immediately available ends up hurting overall build performance. So in that sense, I suppose it would be better if -Zthreads were a hint of sorts, treated as a “max chunks” limit, and if introspection of available threads happened before the decision was made to chunk the available work (and to what extent) so that the behavior of -Zthreads, even with a hard-coded number, would hopefully only ever improve build times and never hurt.

1. sccache does not cache nor speed up incremental builds, and recent versions try to more or less bypass the caching pipeline altogether in an attempt to avoid slowing down incremental builds. ↩

2. A similar (but not identical) issue was reported in the project’s GitHub repo in December of 2019. ↩

3. I assumed -Zthreads=0 would mean “default to the available concurrency” (i.e. 32 threads, in my case) but that doesn’t appear to be the case just by looking at the numbers. ↩

4. The size of the target/ folder varies depending on the RUSTFLAGS used. ↩

5. This was, of course, the same version of sccache that was tested under WSL above. ↩

rsconf itself is a newer crate that was born out of the need (in the fish-shell transition from C++ to rust) for a replacement for some work that’s traditionally been relegated to the build system (e.g. CMake or autoconf) in order to “feature detect” various native system capabilities in the kernel, selected runtime (e.g. libc), or installed libraries. It (optionally) integrates with the popular cc crate so you can test and configure the build toolchain for various underlying features or behavior, and then unlock conditional compilation of native rust code that interops with the system or external libraries accordingly.

While Cargo is an impressive build tool and normally more than sufficient for the needs of the majority of rust crates shipping standalone, contained libraries or packages, for those of us transitioning “more brittle” system software or libraries that rely on functionality of the native kernel, libc, or external libraries – and often have to support older versions thereof, lacking in features or enhancements – it doesn’t offer feature parity with some of the build tools we’ve been traditionally using in the C and C++ world.

Fortunately, the language and tooling devs behind rust recognized early on the need for a more flexible and involved approach to building certain crates and came up with the build.rs approach that lets developers run what is effectively a rust script prior to the traditional build steps invoked by Cargo, influencing how Cargo works, what flags are passed to rustc (the actual rust compiler), and what libraries Cargo asks your linker to include when generating the final binary. Importantly, at this stage devs can inspect the host/target systems and tell the compiler which Cargo features and rust cfgs should be enabled, letting you conditionally compile (or not) various bits and pieces of regular rust code to take advantage of functionality discovered to be present at build time or work around capabilities found to be missing or lacking.

In addition to supporting external libraries (like gettext and, once upon a time, curses), it also runs on quite a number of different unixy systems, starting with Linux, macOS, and the BSDs, and with a long tail of support for various other fun posix-compatible operating systems and projects.¹ As you can imagine, as a shell, fish does a lot of stuff outside the purview of the rust standard library and a lot of the codebase deals with low-level integration with the operating system – and the details of this change quite a bit just from one kernel version to the other, let alone across different OS kernels and distributions altogether.

A lot of the crates in the rust ecosystem seem to rely on OS detection to determine what feature should or shouldn’t be available, but the C/C++ world moved from that to feature detection a long time ago. Fish uses build.rs to determine what low-level operating system features are available (regardless of what OS we are targeting) and enables or disables the compilation of rust code sitting behind #[cfg(key = "value")] depending on the results.

Cargo exposes a very bare-bones mechanism for build.rs to influence which cfg or feature will be enabled/disabled at build-time by means of intercepting specially prefixed stdout messages. Generally, this would look like

1. In build.rs, check for the presence/absence of some feature somehow,
2. In response, execute println!("cargo:rustc-cfg=foo");

Which prints to stdout a line of text Cargo will intercept when it is running the compiled build.rs binary and, based off of that, tell rustc to enable the cfg foo. This “unlocks” code behind a #[cfg(foo)], allowing rustc to see and compile it (normally it would be as if it weren’t there at all).

This is all well and good, but it has a few obvious drawbacks. The first is that somehow that glares at us from point number one above. How exactly does one check if a system feature is present or not? Why doesn’t Cargo help us in this regard? In the world of legacy build systems, this is part and parcel of what a build system does and, in fact, a raison d’être for their existence in a world that predates package managers, semantic versioning, and all the other nice stuff we can now take for granted in a rust-native ecosystem.

It’s from this need that rsconf was born. The crate offers some of the functionality typically made available by “legacy” build systems, wrapped in an easy-to-use and rust-friendly api. Some examples of the available functionality:

• system.has_library(libname)
• system.has_header(name)
• system.has_symbol(symbol)
• system.has_symbol_in(symbol, &[libname])
• system.has_type(type_name)
• system.get_macro_value(name)
• system.get_{u32,u64,i32,i64}_value(ident)
• system.ifdef(define)
• system.if(expr)
• add_library_search_path(path)
• link_library(libname, LinkType::Static/Dynamic)

The names of these methods should make what they do quite clear, and there are various convenience functions to simplify some common patterns around these same principles. The methods themselves are largely implemented as build-time attempts to compile or link minimal C source code to determine the truthiness of the expressions, while the latter two direct Cargo as to how it should attempt find and use external libraries pre-installed on the build host/target.²

In addition, there are methods that make it easier to perform “regular” Cargo stuff in a build.rs script, offering a “strongly typed” api instead of the usual println!() stuff that is prone to typos, mangled types, and more:

• enable_cfg(cfg_name)
• enable_feature(feature_name)
• set_cfg_value(cfg_name, cfg_value)
• rebuild_if_env_changed(env_var_name)
• rebuild_if_path_changed(path)

New to the 0.2.0 rsconf release (as hinted at above) are variations on enable_cfg() and set_cfg_value(), necessitated by changes to the rust compiler that will land in 1.80.³ The compiler will begin checking expressions such as #[cfg(foo)], #[cfg(foo = "bar")], and #[cfg(feature = "baz")], to make sure that all of foo, bar, and baz are valid constraints (not typos or hallucinations). As you can imagine, if the compiler comes across the attribute cfg!(hello) while the hello cfg is enabled, it knows that it’s a valid cfg name. But what about when it’s not enabled? So now we have to let rustc know up front not only which cfg or feature names/values are valid and active, we also have to let it know which are valid even when they’re inactive. To that end, rsconf 0.2.0 introduces the following:

• declare_cfg(cfg_name: &str, enabled: bool)
• declare_cfg_values(cfg_name: &str, valid_values: &[&str])
• declare_feature(feature: &str, enabled: bool)

The first can be used directly in lieu of the old enable_cfg() to both declare a cfg and specify that it is to be enabled (or disabled), but, for now, the second needs to be used in conjunction with the existing set_cfg_value(cfg_name, value) to let the compiler know in advance all the valid values, regardless of whether they’re defined for the current compilation or not.⁴

If you’re curious, you can take a quick look at fish-shell’s current build.rs to get an idea of what real-world rsconf usage in the wild looks like. It’s a short build script, and quite easy to understand.

The rsconf crate itself is still in its early days and will, DV, continue to evolve and see new features. If there are build-related tests or tasks that you feel would fall under its purview, do open an issue in the repository and let us know. Feedback about the proposed builder api for declaring cfg values is also welcome! Otherwise, please give it a try and see if it can help you make your build.rs sane and easier to understand. It’s intentionally written to be fast and lite, with only a dependency on the cc-rs crate (which you’ll almost certainly already be taking a dependency on if you’re compiling against system libraries or headers), so you only stand to benefit from making the switch!

Looking for something else to read or learn? Take a look at my other rust posts, especially this one about designing truly safe semaphores in rust or learn about using simd to speed up rust applications significantly. Sign up below to get emails about open source rust stuff or add my blog to your RSS reader!

1. Unfortunately, as a part of the transition from C++ to rust, we have currently lost support for the vast majority of the legacy OSes and other non-tier-1 unix platforms we used to support due to issues with availability of the rust toolchain and incompatible dependencies like the libc crate, but we are keen to get them back. ↩

2. One thing I really appreciate about Cargo is that it goes to great lengths to support cross-compilation out-of-the-box, clarifying operations and configurations taken from/applying to the machine you are building on (the host) vs the machine you are building for (the target). rsconf is written in a way that similarly respects this divide. ↩

3. You can also refer to this rustc documentation page for more on how this new feature works. ↩

4. A separate “builder-style” api will be added at some point, but there are decisions to make about its shape. For example, it could look like add_cfg("cfg_name").with_values(&["value1", "value2"]).set_value("value1") or along the lines of add_cfg("cfg_name").set_values("active_value", &["other value 1", "other value 2"]) ↩

In the screenshot above, Sarah is sharing a video that was originally shared by Luc, but she’s not embedding/quoting Luc’s tweet itself – only the video. This post will cover how to do that yourself, both on the desktop/web and in the iOS Twitter app on iPhone.

All of Twitter’s features are really just special-cased handling of URLs, and video embedding is no different. If you want to quote-reply, you are actually just posting something followed by the URL of the original tweet you want to quote. For example,

Look at the size of this crowd!
https://twitter.com/LucAuffret/status/1716085946016252251

ends up with the following quote reply:

And similarly, embedding just the video from a tweet is as simple as appending /video/1 to the URL of the source tweet. In this case:

Look at the size of that crowd! #LibérezPalestine
https://twitter.com/LucAuffret/status/1716085946016252251/video/1

becomes

On iOS, in the Twitter/X app, this is all done for you automatically. If you just long-press on a video, you can use the “Post Video” option in the menu that pops up to have twitter copy-and-paste the full tweet URL with the /video/1 already appended for you:

If the source tweet/post contains more than one video, you can change the 1 in /video/1 to a different number in order to embed a video other than the first one in the post.

Liked this post? Follow me on twitter @mqudsi or subscribe to new posts via email from the sidebar to the right!

For example, here’s a video that shows what happens when you select some text in vim and then use CTRL + A to increment the values:

(It’s also a fact that a lot of vim users learned about this functionality the terribly hard way: accidentally pressing CTRL + A then later realizing that all the numbers in their document were off-by-one for some unknown reason.)

But in all honesty, this isn’t a very useful mapping because it’s rare (at least in the programming world) to have numeric and text content completely separate: you usually have numbers in certain, key places while also possibly having numbers intermixed with the remainder of the text in the document. And we’ll often need to only increment certain numeric values but not the rest.

Here’s how you can use increment only numbers matching against a regular expression (including multiple numbers on the same line) while leaving the rest intact:

1. Write a regex that matches against only the numbers you want to change. By doing this in normal mode, we can get vim to highlight the matches as we edit the regular expression, allowing us to visually confirm that the regex matches the numbers we want to increment. To do this, just use our trusty, old friend :s/foo (you can match against numeric content by using \d\+ to select all consecutive digits)
2. For the second half of the /s/foo/bar/ expression (bar, the replacement value), we’ll use the magic of a vim expression to increment (or otherwise manipulate) the matching value. Remember that regex capture groups are made with (match here), match group 0 is the entirety of the match, and our manually captured groups (via the parentheses) are then counted from left-to-right from number 1 onward. The magic bits are \=submatch(n)+1 which replaces the nth match group with the incremented value.

Here’s an example where we want to insert some text in the middle of a numbered/indexed structured body of text then update all the indexes afterwards by bumping them up by one:

We have some subtitles in the SRT format and we want to insert a new caption in the middle, then update all the caption numbers but not the timestamps to reflect the insertion in the middle of the list. We have this text to start with:

1
00:00:03,400 --> 00:00:06,177
In this episode, we'll be talking about
the importance of strong typing in programming.

2
00:00:010,000 --> 00:00:11,200
Strongly-typed languages have many benefits over
their loosely-typed counterparts.

3
00:00:11,500 --> 00:00:13,655
Using strongly-typed languages can actually make
you more productive.

And we want to insert the following subtitles between 1 and 2, but not have to increment all the indexes that come after one-by-one by hand, which is time-consuming, error-prone, and a chore:

00:00:06,600 --> 00:00:09,220
Hang on to your hats because this is going to be fun!

We’ll do this by pasting the text where we want it to go, selecting the remainder of the text (where we need to increment the subtitle index number), and then using the vim expression :s/^\d\+$/\=submatch(0)+1/g to match a line that contains only numeric content (so it’ll match the subtitle index number but not the timestamps, which we absolutely don't want to inadvertently increment in the process):

As you can see, it’s simply a matter of selecting the text you want to replace in (in our case, everything past the captions we just entered) and then coming up with a regex that matches only the numbers we want to replace but not the numbers we don’t. If we had used CTRL + A here instead, we would have ended up with the first timestamp in each caption in our selection incremented, in addition to incrementing the index.

I think the syntax for this one is easy enough to remember that you probably don’t need a plugin or a custom key mapping to do this for you. The trickiest part is just the regex, and odds are in most cases judicious application of ^ (start of line), $ (end of line), and whitespace will probably suffice to get you a regex that matches only the values you need. Unlike some other vim expressions that have really inscrutable names or incantations, using \=submatch(0) (or \=submatch(4) or whatever) a few times is probably all it will take for you to memorize the syntax and soon enough it’ll be second nature.

If you enjoyed this tip, consider subscribing to blog posts via RSS or via email from the sidebar to the right and follow me on twitter @mqudsi for more fun hacking or programming stuff! (If you’re an emacs user, it’s highly unlikely I’ll have any text editing hacks for you at any time, unfortunately!)

1. To be pedantic, only the first group-of-digits/number on each line gets incremented; like many vim commands this only works on the first match per line of text unless some sort /g global modifier is used. ↩

Image courtesy of Hack A Day

This blog post was a bit delayed in the pipeline, but a new release of tcproxy, our educational async (tokio) rust command line proxy project, is now available for download (precompiled binaries or install via cargo).

I was actually surprised to find that we haven’t written about tcpproxy before (you can see our other rust-related posts here), but it’s a command line tcp proxy “server” written with two purposes in mind: a) serving as a real-world example of an async (tokio-based) rust networking project, and b) serving as a minimal but-still-useful tcp proxy you can run and use directly from the command line, without needing complex installation or configuration procedures. (You can think of it as being like Minix, but for rust and async networking.)

The tcpproxy project has been around for quite some time, originally published in 2017 before rust’s async support was even stabilized. At the time, it manually chained futures to achieve scalability without relying on the thread-per-connection model – but today its codebase is a lot easier to follow and understand thanks to rust’s first-class async/await support.

That doesn’t mean that there aren’t “gotchas” that rust devs need to be aware of when developing long-lived async-powered applications, and tcpproxy’s purpose here is to serve as a real-world illustration of the correct way to handle some of the thornier issues such as tying the lifetime of various connections (or halves of connections) to one-another and aborting all remaining tasks when the first terminates (without blocking or polling).

The 0.4.0 release doesn’t contain any major changes but tweaks a number of things to improve both the usability of the application and to model the correct way of handling a few things (such as not using an Arc<T> to share state that remains alive (and static) for the duration of the program’s execution¹)

One of the user-visible changes in this release is that ECONNRESET and ECONNABORT are no longer treated as exceptional, meaning that tcpproxy proceeds as if the connection in question were closed normally and uneventfully. While a compliant TCP client shouldn’t just abort a tcp connection (and a server shouldn’t reset one), these things happen quite often in the real world, and since all tcpproxy connections are stateless, there’s really no reason to handle these any differently than a normal, compliant tcp connection tear-down. Since we don’t report a connection error in these cases, tcpproxy prints (when executed in debug -d mode, that is) the normal messages about the number of bytes proxied in each direction, hopefully leading to less confusion.

For those of you hearing about the tcpproxy project for the first time, I invite you to look over the core event loop which remains fairly small even when correctly handling all the cases we need to account for and synchronizing lifetimes the way we like. If you spot something that’s wrong, not quite right, or could be done in a more idiomatic way, please do leave a comment, send an email, or open an issue – tcpproxy is an open source project and it takes a village to raise and nurture even the smallest of projects to a healthy state!

You can follow me on twitter @mqudsi or sign up below for our rust-only mailing list to receive a heads-up when new rust educational content or rust open source crates are released. If you’re in a position to do so, I am also experimenting with accepting sponsors on my Patreon page and would greatly appreciate your patronage and support!

1. In cases like this, the recommendation is to actually just leak the memory instead to reduce cache coherency traffic in the MESI or MOESI protocols that is caused when each new task increments or decrements the shared reference count bits in the Arc<T>. If you know the value is going to live until the end of the application’s lifetime anyway, there’s no need to incur that cost and any future (read-only) access to the shared variable from any thread on any core will be ~free. ↩

A quick recap: [CallerArgumentExpression] landed as part of the C# 10.0 language update and helps to reduce the (often brittle!) boilerplate involved in, among other uses, creating useful error messages capturing the names of variables or the text of expressions. You tag an optional string method parameter with [CallerArgumentExpression("argName")] where argName is the name of the method argument you want stringified, and the compiler does the rest.

Here’s a quick demo of how [CallerArgumentExpression] works:

using System;
using System.Runtime.CompilerServices;

public class Program
{
    static string Stringify(object obj,
        [CallerArgumentExpression("obj")] string expr = "")
    {
        return expr;
    }

    public static class Foo
    {
        public string Bar = "bar";
    }

    public static void Main()
    {
        var expr = Stringify(Foo.Bar);
        Console.WriteLine(expr); // prints "Foo.Bar"
        expr = Stringify(Foo.Bar + Foo.Bar);
        Console.WriteLine(expr); // prints "Foo.Bar + Foo.Bar"
    }
}

And you can try it online yourself in this .NET Fiddle.

It’s really cool and it opens the door to a lot of possibilities (though I’m still stuck trying to figure some of them out, such as reliably setting/clearing model binding errors that involve array expressions).

As mentioned, this shipped with C# 10. And of course, C# 8 shipped “the big one:” nullable reference types. Since then, the following pattern has become familiar in many a codebase while devs figure out where variables actually can or can’t be null:

using System;
using System.Diagnostics.CodeAnalysis;
using System.Runtime.CompilerServices;

static class Extensions
{
    public static T ThrowIfNull<T>([NotNull] T? value, string expr)
    {
        if (value is null) {
            throw new ArgumentNullException(expr);
        }
        return value;
    }
}

This does exactly what you think it does: it verifies that a value isn’t null or throws an exception if it is. And it lets the compiler know that downstream of this call, the passed-in value is non-null. To make it useful, it’s common enough to extend it with more caller attribute magic:

using System;
using System.Diagnostics.CodeAnalysis;
using System.Runtime.CompilerServices;

static class Extensions
{
    public static T ThrowIfNull<T>(
        [NotNull] T? value,
        string expr,
        [CallerMemberName] string callerName = "",
        [CallerFilePath] string filePath = "",
        [CallerLineNumber] int lineNumber = 0)
    {
        if (value is null) {
            throw new InvalidOperationException(
                $"{expr} unexpectedly null in {callerName} "
                + $"at {filePath}:{lineNumber}");
        }
        return value;
    }
}

Now we get useful exceptions that we’ll hopefully log and revisit to help us find any places in our codebase where we are assuming a value can’t be null but it turns out that, in fact, it can be.

But what about if we try to add our new best buddy [CallerArgumentExpression] here, to get rid of the need to manually specify the text of the argument via argName in our ThrowIfNull()?

using System;
using System.Diagnostics.CodeAnalysis;
using System.Runtime.CompilerServices;

static class Extensions
{
    public static T ThrowIfNull<T>(
        [NotNull] T? value,
        [CallerArgumentExpression("value")] string expr = "",
        [CallerMemberName] string callerName = "",
        [CallerFilePath] string filePath = "",
        [CallerLineNumber] int lineNumber = 0)
    {
        if (value is null) {
            throw new InvalidOperationException(
                $"{expr} unexpectedly null in {callerName} "
                + $" at {filePath}:{lineNumber}");
        }
        return value;
    }
}

At first blush, this works great. Use it with a single variable directly, as in foo.ThrowIfNull(), and everything will work swimmingly and it’ll do exactly what it says on the tin. But try using it in a more-complicated setting, say like foo?.bar?.ThrowIfNull(), and you’ll see what I mean: here, argName will only capture the last token in the chain and you’ll see that argName is only bar and not foo.bar!

It’s actually not particularly surprising behavior. Even without knowing what Roslyn desugars the above code to, you could logically think of it as being an expression (conditionally) invoked on/with the final variable bar itself – after all, T here would have been bar.GetType(), so it’s not a huge stretch of the imagination to guess that expr might only span bar as well.¹

Indeed, when you look at what the code compiles to, you’ll see why. For the following code fragment:

public class Foo {
    public string? Bar;
}

public class C {
    public void M(Foo? foo) {
        foo?.Bar.ThrowIfNull();
    }
}

We get

public class Foo
{
    [System.Runtime.CompilerServices.Nullable(2)]
    public string Bar;
}

public class C
{
    [System.Runtime.CompilerServices.NullableContext(2)]
    public void M(Foo foo)
    {
        if (foo != null)
        {
            Extensions.ThrowIfNull(foo.Bar, ".Bar");
        }
    }
}

Which, while still helpful, is not exactly what we want. Although as C# developers we are somewhat allergic to calling static helper utilities directly instead of cleverly turning them into their more ergonomic extension method counterparts, in this we don’t have any other choice.

When we change ThrowIfNull() from an extension method to a regular static method though, we get the result we really wanted:

public static class Utils
{
    public static T ThrowIfNull<T>(
        [NotNull] T? value,
        [CallerArgumentExpression("value")] string? expr = null) 
    {
        if (value is null) {
            throw new ArgumentNullException(expr);
        }
        return value;
    }
}

public class Foo
{
    public string? Bar;
}

public class C
{
    public void M(Foo? foo)
    {
        Utils.ThrowIfNull(foo?.Bar);
    }
}

Desugaring to:

public class C
{
    [System.Runtime.CompilerServices.NullableContext(2)]
    public void M(Foo foo)
    {
        Utils.ThrowIfNull((foo != null) ? foo.Bar : null, "foo?.Bar");
    }
}

Liked this post? Follow me on twitter @mqudsi and like this tweet for more .NET awesomeness!

Do you know how to effectively use [CallerArgumentExpression] to supercharge C# @dotnet codebase? Here's the number one mistake to look out for. https://t.co/sReLDSS3iw

— Mahmoud Al-Qudsi (@mqudsi) September 11, 2023

1. Except expr is actually not bar but rather .bar! ↩

Low-level or systems programming languages generally strive to provide libraries and interfaces that enable developers, boost productivity, enhance safety, provide resistance to misuse, and more — all while trying to reduce the runtime cost of such initiatives. Strong type systems turn runtime safety/sanity checks into compile-time errors, optimizing compilers try to reduce an enforced sequence of api calls into a single instruction, and library developers think up of clever hacks to even completely erase any trace of an abstraction from the resulting binaries. And as anyone that’s familiar with them can tell you, the rust programming language and its developers/community have truly embraced this ethos of zero-cost abstractions, perhaps more so than any others.

I’m not going to go into detail about what the rust language and standard library do to enable zero-cost abstractions or spend a lot of time going over some of the many examples of zero-cost interfaces available to rust programmers, though I’ll just quickly mention a few of my favorites: iterators and all the methods the Iterator trait exposes have to be at the top of every list given the amount of black magic voodoo the compiler has to do to turn these into their loop-based equivalents, zero-sized types make developing embedded firmware in rust a dream and it’s really crazy to see how all the various peripheral abstractions can be completely erased giving you small firmware blobs despite all the safety abstractions, and no list is complete the newest member of the team, async/await and how rust manages to turn an entire web server api into a single state machine and event loop. (And to think this can be used even on embedded without a relatively heavy async framework like tokio and with even zero allocations to boot!)

But the tricky thing with abstractions is that the relative price you pay scales rather unfairly with the size of the interface you are abstracting over. While a byte here and a byte there may mean nothing when we’re talking framework-scale interfaces, when you are modeling smaller and finer-grained abstractions, every byte and every instruction begin to count.

A couple of weeks ago, we released an update to rsevents, our crate that contains a rusty cross-platform equivalent to WIN32 events for signaling between threads and writing your own synchronization primitives, and rsevents-extra a companion crate that provides a few handy synchronization types built on top of the manual- and auto-reset events from the rsevents crate. Aside from the usual awesome helpings of performance improvements, ergonomics enhancements, and more, this latest version of rsevents-extra includes a Semaphore synchronization primitive – something that the rust standard library surprisingly lacks… but not without good reason.

What makes a semaphore a semaphore?

Semaphores are well-documented and fairly well-understood underpinnings of any concurrency library or framework and essential Computer Science knowledge. So why doesn’t the rust standard library have a semaphore type?

Unlike the synchronization types that the rust standard library currently provides (such as Mutex<T> and RwLock<T>, a semaphore is somewhat harder to model as it doesn’t so much restrict concurrent access to a single object or variable so much as it does limit concurrency within a region of code.

Of course it can be argued that in traditional programming a semaphore is just a more general case of a mutex, and just like mutexes traditionally protected a region of code from concurrent access¹ but were converted into synchronization primitives owning the data they protect and marshalling access to it, there’s no reason a rust semaphore couldn’t do the same. But therein lies the problem: a mutex and a read-write lock can both be understood in terms of readers and writers,² a semaphore makes no such guarantees. And rust is quite fundamentally built on the concept of read ^ write: it needs to know if a thread/scope is reading or writing from a variable or memory location in order to uphold its most basic memory safety guarantee: there can either be multiple “live” read-only references to an object or a single write-enabled (&mut) reference to the same — but a semaphore doesn’t make that distinction!

While a strictly binary semaphore (max concurrency == 1) can guarantee that there will never be multiple writers accessing a memory region, there’s not much theoretical benefit to such a binary semaphore over a mutex – in fact, they’re interchangeable. What makes a semaphore truly special is that it can be created (or even dynamically modified) with a concurrency limit n and then uphold its core precondition, guaranteeing that at any given time there will never be more than n threads/stacks³ accessing a semaphore-protected region at any given time.

The problem is that with n > 1, there’s no concept of a “privileged” owning thread and all threads that have “obtained” the semaphore do so equally. Therefore, a rust semaphore can only ever provide read-only (&T) access to an underlying resource, limiting the usefulness of such a semaphore almost to the point of having no utility. As such, the only safe “owning” semaphore with read-write access that can exist in the rust world would be  Semaphore<()>,⁴ or one that actually owns no data and can only be used for its side effect of limiting concurrency while the semaphore is “owned,” so to speak.⁵ (Actual mutation of accessed resources within the concurrency-limited region, if needed, would continue to be marshalled via Mutex<T> or RwLock<T> on a fine-grained level.)

Ok, so this explains why the rust standard library doesn’t contain a Semaphore<T> type to mirror Mutex<T> and its friends, but then what’s so hard about shipping a non-owning std::sync::Semaphore instead?

Designing a safe Semaphore for rust

To answer this, we need to look at what a semaphore API generally looks like in other languages. While the names and calling semantics differ, a semaphore is generally found as a type that provides the following, starting with the most fundamental to de facto properties:

• It is a type that can be used to limit concurrency to a resource or region of code, up to a dev-defined limit n.
• It is a type that has a concept of “currently available concurrency,” which represents and tracks the remaining number of threads/stacks that can “obtain” the semaphore, thereby reducing its available concurrency and generally giving the calling thread access to the concurrency-limited region,
• A semaphore can be created/declared with an “initially available concurrency” and a “maximum possible concurrency,” which may differ (indeed, “initially available concurrency” is often zero),
• Semaphores don’t generally have a concept of ownership, meaning a thread (or any thread) can increment (up to the pre-defined limit) or decrement (down to zero) the available concurrency for a semaphore without having “obtained” or “created” it. (This is necessary otherwise it’d be impossible to initialize a semaphore with a lower initial concurrency limit than its maximum, because no thread could then increase it.)

It’s the last of these points that makes a semaphore so tricky to model in any language that prides itself on safety. While a semaphore that acts strictly as a variable-occupancy mutex (i.e. initial concurrency equals the max concurrency and each time it is obtained, it must be subsequently released by the same thread that obtained it), that’s not generally a requirement that semaphores impose, and such a requirement would considerably limit the utility that a semaphore could offer.

Let’s look at some ways we might design such a semaphore in rust, some of which we actually tried while prototyping rsevents_extra::Semaphore.

Before anything else, let’s get the hard part out of the way by introducing you to rsevents::AutoResetEvent, a one-byte⁶ synchronization primitive that takes care of putting threads to sleep when the event isn’t signalled/available and allowing one-and-only-one waiting thread to either consume the event (if it’s not already asleep) or to wake up (if it’s asleep waiting for the event) when the event is signalled (after which the event is atomically reset to a “not signalled” state). It doesn’t even have any spurious waits, making it really nice and easy to work with in a safe fashion. All of our Semaphore implementations will use this auto-reset event to take care of the synchronization and we’ll omit the details of when and where to call AutoResetEvent::set() and AutoResetEvent::reset() for now.

So here’s what our initial semaphore skeleton looks like. We know we need an internal count of some integral type to keep track of the current concurrency (since we already established that it’s going to be variable and not just zero or one), and we know that at minimum a semaphore’s interface needs to provide a wait to “obtain” the semaphore (decrementing the available concurrency for future callers) and a way to “release” the semaphore (at least to be used by a thread that has already obtained a “concurrency token” to re-increment the count after it is done and wants to give up its access to the concurrency-restricted region for some other caller to take).

struct Semaphore {
    event: AutoResetEvent,
    count: AtomicU32,
    // TODO: Fill in the rest
}

impl Semaphore {
    /// Create a new `Semaphore`
    pub fn new(/* TODO */) -> Semaphore { /* TODO */ }

    /// Obtain the `Semaphore`, gaining access to the protected concurrency
    /// region and reducing the semaphore's internal count. If the 
    /// `Semaphore` is not currently available, blocks sleeping until it
    /// becomes available and is successfully obtained.
    pub fn wait(&self) -> ??? { ... }

    /// Increment the semaphore's internal count, increasing the available
    /// concurrency limit.
    pub fn release(&self, ...) { ... }
}

Our goal now is to fill in the blanks, attempting to make a semaphore that’s maximally useful and as safe as possible, while limiting its size and runtime checks (the costs of said safety).

A constant maximum concurrency?

We’ve already established that a semaphore needs to offer a tunable “maximum concurrency” parameter that decides the maximum number of threads that can have access to the concurrency-limited region at any given time. This number is typically supplied at the time the semaphore is instantiated, and it’s quite normal for it to be fixed thereafter: while the current available concurrency may be artificially constrained beyond the number of threads that have actually borrowed/obtained the semaphore, it’s OK for the absolute maximum to be unchangeable after a semaphore is created.

We really have only two choices here: we either add a max_count integral struct member to our Semaphore or we take advantage of rust’s const generics (another brilliant zero-cost abstraction!) to eliminate this field from our struct altogether… but at a considerable cost.

Let’s consider some constraints that might determine the maximum concurrency a semaphore can have:

• We’ve experimentally determined that our backup application works best with up to eight simultaneous threads in the file read part of the pipeline and up to two simultaneous threads in the file write part of the pipeline, to balance throughput against maxing out the disk’s available IOPS. Here we can use two separate semaphores, each with a different hard-coded maximum concurrency.
• Almost all modern web browsers limit the maximum number of live TCP connections per network address to a hard-coded limit, typically 16. Internally, the browser has a semaphore for each domain name with an active TCP connection (perhaps in a HashMap<DomainName, Semaphore>), and blocks before opening a new connection if the maximum concurrency is exceeded.

In these cases, we could get away with changing our Semaphore declaration to Semaphore<const MAX: usize> and using const generics when initializing a semaphore to specify the hard-coded maximum concurrency limit, e.g. let sem = Semaphore::<16>::new(...). But const generics doesn’t just lock us into a hard-coded value specified at the time of object initialization, it locks us into a hard-coded value specified at the time of writing the source code for the object’s initialization. That means we can’t use a const generic parameter in lieu of a max_concurrency field in cases like the following, which we unfortunately can’t just ignore:

• We want to limit the number of threads accessing a code section or running some operation to the number (or a ratio of the number) of CPU cores the code is running on (not compiled on).
• We want to let the user select the max concurrency at runtime, perhaps by parsing a -j THREADS argument or letting the user choose from a drop-down menu or numselect widget in a GUI application.
• Going back to our backup application example, we want to use different read/write concurrency limits in the case of an SSD vs in the case of an old-school HDD.

This means that we’re going to have to approximately double the size of our Semaphore object, mirroring whatever type we’re using for Semaphore::count to add a Semaphore::max_concurrency member (they have to be the same size because count can vary from zero to max_concurrency), giving us the following, fairly complete, struct declaration.

A functionally complete semaphore

As we’ve determined, we unfortunately can’t use rust’s const generics to roughly halve the space a Semaphore object takes in memory, giving us the following declaration:

struct Semaphore {
    event: AutoResetEvent,
    count: AtomicU32,
    max_count: AtomicU32,
}

While we were forced to add a max_count field to store the maximum allowed concurrency for the semaphore, this wasn’t at all a violation of the “zero-cost abstraction” principle: if we want to allow the user to specify something at runtime and match against it later, this is the cost we invariably pay (whether in rust or in assembly).

As bare as our Semaphore structure is, this is actually enough to implement a completely functional and – for the most part – safe semaphore. Thanks to how AutoResetEvent is implemented internally and its own safety guarantees, we can get away with just some extremely careful use of atomics, without a lock or mutex of any sort:

impl Semaphore {
    /// Create a new `Semaphore`
    pub fn new(initial_count: u32, max_count: u32) -> Semaphore {
        Semaphore { 
            event: AutoResetEvent::new(EventState::Unset),
            count: initial_count,
            max_count,
        }
    }

    /// Obtain the `Semaphore`, gaining access to the protected concurrency
    /// region and reducing the semaphore's internal count. If the 
    /// `Semaphore` is not currently available, blocks sleeping until it
    /// becomes available and is successfully obtained.
    pub fn wait(&self) {
        let mut count = self.count.load(Ordering::Relaxed);

        loop {
            count = if self.count == 0 {
                // Semaphore isn't available, sleep until it is.
                self.event.wait();
                // Refresh the count after waking up
                self.count.load(Ordering::Relaxed)
            } else {
                // We can't just fetch_sub(1) because it might underflow in a race
                match self.count.compare_exchange(count, count - 1, 
                    Ordering::Acquire, Ordering::Relaxed)
                {
                    Ok(_) => {
                        // We officially obtained the semaphore.
                        // If the (now perhaps stale) new `count` value is non-zero, 
                        // it's our job to wake someone up (or let the next caller in).
                        if count - 1 > 0 {
                            self.event.set();
                        }
                        break;
                    },
                    Err(count) => count, // Refresh the cached value and try again
                }
            }
        }

        // This must hold true at all times!
        assert!(count <= self.max_count); 
        return ();
    }

    /// Increment the semaphore's internal count, increasing the available
    /// concurrency limit. 
    pub fn release(&self) { 
        // Try to increment the current count, but don't exceed max_count
        let old_count = self.count.fetch_add(1, Ordering::Release);
        if old_count + 1 > self.max_count {
            panic!("Attempt to release past max concurrency!");
        }
        // Check if we need to wake a sleeping waiter
        if old_count == 0 {
            self.event.set();
        }
    }
}

If you’re familiar with atomic CAS, the annotated source code above should be fairly self-explanatory. In case you’re not, briefly, AtomicXXX::compare_and_exchange() is how most lock-free data structures work: you first read the old value (via self.count.load()) and decide based on it what the new value should be, then use compare_and_exchange() to change $old to $new, if and only if the value is still $old and hasn’t been changed by another thread in the meantime (if it has changed, we re-read to get the new value and try all over again until we succeed).

With the Semaphore skeleton filled out, we now have a functionally complete semaphore with features comparable to those available in other languages. At the moment, Semaphore::wait() returns nothing and any thread that’s obtained the semaphore must ensure that Semaphore::release() is called before it returns, but that’s just an ergonomic issue that we can easily work around by returning a concurrency token that calls Semaphore::release() when it’s dropped instead.⁷

But can we make it safer?

The problem with Semaphore::release()

For the remainder of this article, we’ll be focusing on the ugly part of a semaphore, the part that makes writing a truly safe semaphore so challenging: Semaphore::release().

In rust, the standard library concurrency primitives all return “scope guards” that automatically release (or poison) their associated synchronization object at the end of the scope or in case of panic. This actually isn’t just a question of ergonomics (as we mentioned before), it’s a core part of their safety. On Windows, you can create a mutex/critical section with InitializeCriticalSection() and obtain it with EnterCriticalSection() but any thread can call LeaveCriticalSection(), even if it invokes undefined behavior that may cause a deadlock. On Linux and its pals, pthread_mutex_init() can be used to create a mutex and pthread_mutex_lock() can be used to enter the mutex. While pthread_mutex_unlock() is documented as may return an EPERM error if the current thread doesn’t own the mutex, the notes clarify that the current owning thread id isn’t stored and deadlocks are allowed in order to avoid overhead – meaning its implementation-defined whether or not pthread_mutex_unlock() actually protects against unlocking from a different thread. In practice, it doesn’t.⁸

Rust, as a partially object-oriented language, sidesteps all this via our good friend RAII by simply making the equivalent of Mutex<T>::unlock(&self) unavailable except via the scope guard (MutexGuard) returned by Mutex<T>::lock(&self) (and the same for RwLock<T>). The type system prevents you from calling the equivalent of pthread_mutex_unlock() unless you already own a MutexGuard, which can only be acquired as a result of a successful call to Mutex::lock() – without needing any runtime code to check whether or not the calling thread is the owning thread, because the type system provides that safety guarantee at zero cost.

Unfortunately, as I mentioned earlier in passing, even if we did make our Semaphore::wait() function return some sort of concurrency token/scope guard, it would at best by an ergonomic improvement to make sure one doesn’t forget to call Semaphore::release() before exiting the scope, but it wouldn’t allow us to eliminate a publicly callable Semaphore::release() function that any thread could call at any time, at least not without making it impossible to create a semaphore with an initial count that doesn’t equal the maximum count, and not without making the “current maximum concurrency” limit non-adjustable at runtime.

Do we even need Semaphore::release() anyway?

At this point, you might be tempted to ask if we really truly absolutely need these options and questioning what purpose they serve – which is a good and very valid question, given the cost we have to pay to support it and especially because in all the cases we mentioned above (those with a fixed max_count and those without), you’d always create a semaphore with initial_count == max_count. Here are some hopefully convincing reasons:

• Semaphores aren’t just used to limit concurrency up to a maximum pre-defined limit, they’re also used to temporarily artificially constrain available concurrency to a value in the range of [0, max_count] – in fact, this is where you might find their greatest utility.
• CS principle: Preventing a “thundering herd” when a number of threads are blocked waiting on a semaphore to allow them access to a resource or code region. You can have up to n threads performing some task at the same time, but you don’t want them to all start synchronously because it’ll cause unwanted contention on some shared state, e.g.
  • Perhaps an atomic that needs to be incremented, and you don’t want to waste cycles attempting to satisfy compare_exchange() calls under contention or you otherwise have a traditional fine-grained lock or even lock-free data structure where uncontended access is ~free but contention wastes CPU cycles busy-waiting in a spinlock or even puts threads to sleep;
  • Starting operations at ~the same time can run into real-world physical limitations, e.g. it takes n IO threads to achieve maximum saturation of available bulk bandwidth but the initial seeks can’t satisfy as many IOPS without unduly increasing latency and decreasing response times.
  • The semaphore controls the number of threads downloading in parallel but you don’t want to DOS the DNS or HTTP servers by flooding them with simultaneous requests, though you do ultimately want n threads handling the response, downloading the files, etc. in parallel.
• CS principle: Making available concurrency accrue over time, up to a maximum saturation limit (represented by max_count), for example:
  • A family of AWS EC2 virtual machine instances have what is referred to as “burstable performance” where for every m minutes of low CPU usage, you get t time slices of “more than what you paid for” instantaneous CPU performance, up to n maximum time slices accrued. As a simplified example, you “rent” a virtual machine with a lower nominal CPU speed of 2.0 GHz and for every 5 minutes with a p95 per-5-second-time-slice CPU utilization below 50%, you get a “free” 5-second-time-slice of 3.0 GHz compute. If you launch a burstable instance and immediately try running Prime95, you’ll be capped to 2.0 GHz, but if you are running nginx and serving web requests under relatively no load, then after 10 minutes you’ll have accrued 10 seconds of “free” 3.0 GHz performance. When you suddenly get a burst of traffic because a link to your site was just shared on an iMessage or WhatsApp group and 200 phones are hitting your server to generate a web preview, you can “cash in” those accrued 3.0 GHz time slices to satisfy those requests quickly, after which you’re constrained to the baseline 2.0 GHz performance until the requests settle down and you begin to accrue 3.0 GHz time slices once again.
  • While you can implement a rate limiter by giving each connected user/IP address a maximum number of requests they can make per 5 minutes and resetting that limit every 5 minutes, that still means your server can be DDOS’d by n clients saturating their limit of 100-requests-per-5-minutes in the first few seconds after establishing a connection. You can instead give each client a semaphore⁹ with an initial available count of, say, 2 and then every three seconds increment the available concurrency by 3; meaning clients would have to be connected for a full 5 minutes before they can hit you with 100 requests in one go (making it more expensive to mount an attack and giving your firewall’s heuristics a chance to disconnect them).

Hopefully the above reasons demonstrate the utility in being able to create a Semaphore with an initial count lower than the maximum count, and therefore, the need to have a free-standing Semaphore::release() function that, at the very least, the creator of the semaphore can call even without having previously made a corresponding call to Semaphore::wait().

Can we make Semaphore::release() safer?

There’s an immediately obvious way to make Semaphore::release() safer to call, and that’s to replace it with a Semaphore::try_release() method that first checks to ensure that the semaphore’s core integrity predicate (self.count <= self.max_count) is upheld instead of blindly incrementing self.count and then panicking if it then exceeds self.max_count, returning true or false instead to denote whether the operation completed successfully or not.

It’s actually not that hard of a change to make, provided you’re familiar with compare-and-exchange atomics (which we used above in safely implementing Semaphore::wait() to prevent two threads from racing to increment self.count after checking that it’s less than self.max_count), which we’ll use to write our try_release() method now:

impl Semaphore {
    /// Attempts to increment the `Semaphore`'s internal count,
    /// returning `false` if it would exceed the maximum allowed.
    pub fn try_release(&self) -> bool {
        let mut prev_count = self.count.load(Ordering::Relaxed);
        loop {
            if prev_count == self.max_count {
                // Incrementing self.count would violate our core precondition
                return false;
            }

            match self.count.compare_exchange_weak(
                prev_count, // only exchange `count` if it's still `prev_count`
                prev_count + 1, // what to exchange `count` with
                Ordering::Release, Ordering::Relaxed)
            {
                // The CAS succeeded
                Ok(_) => {
                    if prev_count == 0 {
                        // Wake a waiting thread
                        self.event.set();
                    }
                    return true;
                }
                // If it failed, refresh `prev_count` and retry, failing if 
                // another thread has caused `prev_count` to equal `max_count`.
                Err(new_count) => prev_count = new_count,
            }
        }
    }
}

This is much better. We only increment count if we can guarantee that it won’t cause it to exceed max_count, and relay our results to the caller. We can now guarantee that call to Semaphore::try_release() that wasn’t paired with a previous Semaphore::wait() won’t inadvertently violate the absolute limit on the allowed concurrency.

But can you spot the problem? Look at the code sample carefully, and consider it not in the context of a single call to Semaphore::try_release() but as a whole.

If you think you’ve figured it out or you just want to skip to the answers, read on to find out where the problem still lies. (Yes, this is just a filler paragraph to avoid your eyes immediately seeing the answer while you think this over. Hopefully you haven’t already scrolled down too much so that the answer is already in view. Sorry, I can’t really do better than this, my typesetting system doesn’t let me insert empty vertical whitespace very elegantly.)

The issue is that we’ve prevented this call to Semaphore::try_release() from incrementing count past max_count, but we might already have extant threads chugging away in parallel that have already obtained the semaphore, oblivious to the fact that count has changed from under them. The problem is easy to spot if we keep a copy of our old Semaphore::release() around and mix-and-match between it and try_release(), with calls we expect to always succeed using the former and calls that may overflow the current count using the latter:

fn main() {
    // Create a semaphore with a count of 1 and max count of 2
    let semaphore = Semaphore::new(1, 2); 

    // Create a number of threads that will use the semaphore 
    // to make sure concurrency limits are observed.
    
    for n in 0..NUM_CPUS {
        std::thread::spawn(|| {
            // Make sure to obtain a concurrency token before beginning work
            semaphore.wait();

            // TODO: Replace `sleep()` with actual work!
            std::thread::sleep(Duration::from_secs(5));

            // Release the concurrency token when we're done so another 
            // thread may enter this code block and do its thing.
            semaphore.release();
        });
    }

    while !work_finished {
        // <Read user input from keyboard here>
        
        // In response to a user command, increase concurrency 
        if !semaphore.try_release() {
            eprintln!("Cannot raise limit, max already reached!");
        }
    }
}

We’re using our semaphore to limit the number of simultaneous worker threads, originally operating at 50% of our maximum concurrency limit (count == 1). To start, one of NUM_CPUS¹⁰ worker threads obtains the semaphore (count == 0) to access the concurrency limited region and the rest are locked out.

In the main thread’s event loop, we wait for work to finish or a user command to be entered at the keyboard (omitted from the example).¹¹ The user keys in an input that’s interpreted as “try to speed things up by increasing the concurrency limit,” and in response the main thread calls Semaphore::try_release() and succeeds in incrementing its internal self.count (now count == 1 again), thereby allowing a second thread to enter the semaphore-protected region (giving count == 0) and contribute to progress.

But what happens now that two threads are already in the semaphore-protected scope but the user triggers a second call to Semaphore::try_release()? As far as the semaphore knows, count (equal to 0) is less than max_count, and can be safely incremented, yielding count == 1 and thereby unblocking yet another worker thread sleeping in semaphore::wait() (reducing count to zero).

But what does that mean? Even though our core precondition hasn’t been violated  in this instant (Semaphore::count <= Semaphore::max_count), we now have three worker threads in the concurrency-limited region, exceeding the hard-coded max_count provided during the semaphore initialization! In our example above where each worker thread, having first obtained a concurrency token via Semaphore::wait(), assumes that it can call the infallible Semaphore::release() method safely when it’s done, but when the last worker thread finishes its work, it’ll end up incrementing count past max_count and panicking in the process.

Of course, the panic isn’t the problem – it’s just reporting the violation when it sees it. We could replace all Semaphore::release() calls with Semaphore::try_release() and we’d still have a problem: there are more worker threads in the semaphore-protected region than allowed, and one of the “shouldn’t fail” calls to Semaphore::try_release() will eventually return false, whether that triggers a panic or not.

The crux of the matter is that we borrow from Semaphore::count but don’t have a way to track the total “live” concurrency count available (the remaining concurrency tokens in self.count plus any temporarily-unavailable-but-soon-to-be-returned concurrency tokens borrowed by the various worker threads). And, importantly, we don’t need this for any core semaphore functionality but rather only to protect against a user/dev calling Semaphore::release() more times than they’re allowed to. In a perfect world, it would be the developer’s responsibility to make sure that there are never more outstanding concurrency tokens than allowed, and we could omit these safety checks altogether. She could statically ensure it’s the case by only ever pairing together wait() and release() calls (perhaps after issuing a fixed, verifiable number of release() calls upon init), or by tracking when and where needed the number of free-standing calls to release() in an external variable to make sure the limit is never surpassed.

At last, a truly safe semaphore?

While we could offload the cost of verifying that there will never be more than max_count calls to Semaphore::release() to the developer, we’re rust library authors and we hold ourselves to a higher standard. We can most certainly do just that, but we would have to mark Semaphore::release() as an unsafe function to warn users that there are preconditions to calling this and dangers to be had if calling it willy-nilly without taking them into account. But what if we want a synchronization type that’s easier to use and worthy of including in the standard library, being both safe and easy-to-use? What then?

There are actually several approaches we can take to solving this gnarly problem. The easiest and safest solution would be to simply change our Semaphore definition, adding another field of the same integral count type, perhaps called total_count or live_count that would essentially track initial_count plus the number of successful free-standing calls to Semaphore::release(), but not decrement it when a call to Semaphore::wait() is made.¹² This way each time try_release() is called, we can check if total_count (and not count) would exceed max_count — and continue to use just self.count for the core semaphore functionality in Semaphore::wait():

struct Semaphore {
	event: AutoResetEvent,
	count: AtomicU32,
	total_count: AtomicU32,
    max_count: AtomicU32,
} 

impl Semaphore {
    /// Attempts to increment the `Semaphore`'s internal count,
    /// returning `false` if it would exceed the maximum allowed.
    pub fn try_release(&self) -> bool {
        // First try incrementing `total_count`:
        let mut prev_count = self.total_count.load(Ordering::Relaxed);
        loop {
            if prev_count == self.max_count {
                // Incrementing self.total_count would violate our core precondition
                return false;
            }

            match self.total_count.compare_exchange_weak(
                prev_count, // only exchange `total_count` if it's still `prev_count`
                prev_count + 1, // what to exchange `total_count` with
                Ordering::Relaxed, Ordering::Relaxed)
            {
                // If the CAS succeeded, continue to the next phase
                Ok(_) => break,
                // If it failed, refresh `prev_count` and retry, failing if 
                // another thread has caused `prev_count` to equal `max_count`.
                Err(new_count) => prev_count = new_count,
            }
        }

        // Now increment the actual available count:
        let prev_count = self.count.fetch_add(1, Ordering::Release);
        if prev_count == 0 {
            // Wake up a sleeping waiter, if any
            self.event.set();
        }

        return true;
    }
}

But now, something else breaks!

Let’s go back to our last example with many worker threads attempting to access a concurrency-restricted region and a main event loop that reads user commands that can affect the available concurrency limit. To make this more relatable, let’s use a real-world example: we’re developing a command-line BitTorrent client and we want the user to control the number of simultaneous uploads or downloads, up to some limit (that might very well be u32::MAX). In the last example we were focused on user commands that increased the available concurrency limit, in our real-world example perhaps reflecting a user trying to speed up ongoing downloads or seeds by allowing more concurrent files or connections.

But this isn’t a one-way street! Just as a user may wish to unthrottle BitTorrent downloads, they might very well wish to throttle them to make sure they’re just seeding in the background without saturating their Top Tier Cable Provider’s pathetic upload speed and killing their internet connection in the process. How do we do that safely?

One way would be to introduce a method to directly decrement our semaphore’s self.total_count and self.count fields (down to zero), but what do we do if total_count was non-zero but count was zero (i.e. all available concurrency was currently in use)? Apart from the fact that we are using unsigned atomic integers to store the counts, we could decrement count (but not total_count) past zero, for example to -1, and let the “live” concurrency eventually settle at the now-reduced total_count after a sufficient number of threads finish their work and leave the concurrency-limited region.

But we don’t actually need to do any of that: by its nature a semaphore already provides a way to artificially reduce the available concurrency by means of a free-standing call to Semaphore::wait(), i.e. calling wait() without calling release() afterwards. It ensures that the count isn’t reduced until it’s non-zero and that count never exceeds max_count or total_count at any time, not even temporarily.

Unfortunately, herein lies a subtle problem. With our revised semaphore implementation, we increase both count and total_count when the free-standing release() is called and assume that each call to Semaphore::wait() will have a matching call to some Semaphore::special_release() that increases count without touching total_count. This way, total_count tracks the “total available” concurrency, assuming that it’s equal to “remaining concurrency limit” plus “outstanding calls to wait() that haven’t yet called xxx_release().

While free-standing calls to Semaphore::release() were our problem before, here we’ve shifted that to an issue with free-standing calls to Semaphore::wait() – an admittedly less hairy of a situation but, as we have seen, still not one that we can afford to ignore!

More importantly, even if we weren’t using free-standing calls to Semaphore::wait() to artificially reduce the available concurrency, we actually can’t guarantee that release() is always called after wait(): it’s a form of the halting problem, and even if we ignore panics and have wait() return a scope guard that automatically calls release() when it’s dropped, it’s still completely safe for a user to call std::mem::forget(scope_guard) thereby preventing release() from being called!¹³

Fundamentally, we can’t really solve this problem. We either err on the side of potentially allowing too many free-standing calls to release() to be made, with safety checks delaying the overflow of max_count until the last borrowed concurrency token is returned after a call to wait(); or we err on the side of prudence and incorrectly prevent a free-standing call to release() from going through because we don’t know (and can’t know) that a thread which previously call wait() and took one of our precious concurrency tokens has decided it’s not going to ever return it.

But don’t despair! Do you remember the old schoolyard riddle? You’re silently passed a pen and notebook, on which you see the following:

Like with the riddle,¹⁴ we can implement scope guards to make it more likely that every call to wait() is matched by call to release() but we can’t actually stop the user from calling std::mem::forget(sem.wait()) – and we don’t have to. Without trying to think of ways to cause a compiler error when a scope guard is leaked and not dropped, we can still make it, if not hard then at least harder for the user to leak a scope guard and throw our internal count off. How? Not by hiding the ability to forget a scope guard but by highlighting it front and center!

Let’s fast forward to our semaphore from above, but modified to return a scope guard instead to encourage returning concurrency tokens back to the semaphore when a thread has finished with a concurrency-limited operation:

/// This concurrency token is returned from a call to `Semaphore::wait()`.
/// It's automatically returned to the semaphore upon drop, incrementing
/// the semaphore's internal available concurrency counter once more.
struct ConcurrencyToken<'a> {
    sem: &'a Semaphore
}

impl Semaphore {
    pub fn wait<'a>(&'a self) -> ConcurrencyToken<'a> {
        include!("earlier definition of Semaphore::wait() here");

        // Now instead of returning () we return a ConcurrencyToken

        return ConcurrencyToken {
            sem: self,
        }
    }

    /// Directly increments the internal concurrency count without touching 
    /// `total_count` and without checking if it would exceed `max_count`.
    unsafe fn release_internal() {
        let prev_count = self.count.fetch_add(1, Ordering::Release);

        // We only need to wake a sleeping waiter if the previous count
        // was zero. In all other cases, no one will be asleep.
        if prev_count == 0 {
            self.event.set();
        }
    }
}

impl Drop for ConcurrencyToken<'_> {
    fn drop (&mut self) {
        unsafe { self.sem.release_internal(); }
    }
}

This was our initial attempt at “strongly encouraging” calls to Semaphore::wait() to always be paired with calls to Semaphore::internal_release() (called by the ConcurrencyToken on drop), which decrements count without touching total_count so our logic in Semaphore::try_release() can continue to work.

As we said though, if one were to call std::mem::forget(sem.wait()) the ConcurrencyToken would be forgotten without internal_release() ever being called, and the count we track in total_count would be off by one, preventing a free-standing call to Semaphore::release() that should have been allowed.

So what if we just directly add a new method to our concurrency token? A  ConcurrencyToken::forget() makes it harder to call std::mem::forget() on our concurrency token than it is to just call Semaphore::wait().forget() directly! (See, I really was going somewhere with that riddle!)

impl ConcurrencyToken<'_> {
    /// It is a violation of this crate's contract to call `std::mem::forget()`
    /// on the result of `Semaphore::wait()`. To forget a `ConcurrencyToken`, 
    /// use this method instead.
    pub fn forget(self) {
        // We're keeping `count` permanently reduced, but we need to decrement
        // `total_count` to reflect this as well before forgetting ourselves.
        self.sem.total_count.fetch_sub(1, Ordering::Relaxed);
        std::mem::forget(self);
    }
}

And just like that, we now have something we can reasonably call a “safe” semaphore, worthy of rust!

The price we pay for safety

While I can’t say with complete confidence that this is the optimal implementation of a safe semaphore (exposing the same functionality), our journey above is still representative of the constant tug-of-war that takes place when trying to build an API as you juggle performance, the desire for zero-cost abstractions, and the imperative of surfacing a (within the boundaries of reasonable use) safe and misuse-resistant interface.

We started with something completely simple: two words for the count and a single byte auto-reset event we could use to impose strict ordering and optimized waiting/sleeping in cases of contention. Correctness (which, if you squint at it in a particular way, is just another kind of safety) mandated the use of atomics from the very start, preventing us from playing fast and loose with our integer math and imposing heavy penalties when it came to ensuring cache coherence and cross-core integrity. Then, just when we thought we had everything figured out, we needed to completely change our approach and even add a new struct member to boot (raising the size of the Semaphore struct by 33-45% depending on the integer width, which sounds really scary until you realize it’s still just a handful of bytes).

There are of course other possible solutions to the same problem, all of which potentially have their own drawbacks.¹⁵ And even if there are cost-free solutions here, the general picture isn’t unique to semaphores or even concurrency primitives in general: it’s a story that’s played on repeat and comes up every time an interface needs to be added that has some caveats the caller needs to keep in mind. Writing correct code is hard, writing safe and correct code is harder. But, in my opinion, this is what actually makes rust special.

Rust’s concepts of ownership and exclusive RO/RW semantics play a huge role in making it such a popular language with low-level developers, but I would argue that it’s this attention that’s paid when writing libraries that deal with intricate or abstract concepts that can’t be reduced to simple &foo vs &mut foo semantics that make rust truly unique. As an old hat at C and C++ development, I’ve already worn my “low-level library developer” mantle thin, and it’s absolutely awesome to be able to write an abstraction that other developers can use without having to dive into syscalls and kernel headers as the only source of literature. But with rust, I’m experiencing a completely different kind of challenge in writing libraries and APIs: here there’s a bar even higher than just writing clean abstractions, and it’s being able to write these low-level abstractions in a way that not only can clever developers that have previously dealt with these system internals appreciate and use our libraries to their advantage, but that even others new to the scene can just read your documentation (or maybe not even that) and then let the shape of your API guide them to the rust, figuring out the right way of using it.

In any language, a savvy enough developer can usually figure their way around even a completely new library dealing with concepts and mechanisms completely alien to them. But in the process of figuring it out, they’re bound to make mistakes, bump into leaky abstractions (“but why shouldn’t I call pthread_mutex_unlock from another thread if I need to access the mutex and its currently locked? What is it there for, then?” – whether they’re asking it on SO or mulling it over quietly in their head as they figure out the black-box internals by poking and prodding away at the API surface), pull out what’s left of their hair, and bang their head against the wall some before arriving at a generally correct and workable solution.

But it doesn’t have to be that way, and the burden is on us as developers of these libraries and crates to give our fellow devs a better option. Runtime errors (like the ones the pthread API doesn’t even bother returning!) are good and other languages¹⁶ have demonstrated how it can be used with non-zero but still fairly minimal overhead, but with the benefits of a strongly typed language, powerful type abstractions, and perhaps a smattering of generics and ZSTs, we can and should do better.

The truly safe semaphore, for your benefit and review

The semaphore we iteratively designed in this article is available for you to use, study, or review as part of the 0.2 release of the rsevents-extra crate. This is the current API exposed by our Semaphore type (source code here), incorporating some of the ideas discussed above.¹⁷

The Semaphore type in rsevents-extra actually includes even more safety than we’ve demonstrated above, but it’s of the bog-standard variety (checking for integer overflows, making sure we’re not decrementing past zero, etc) and not something unique to the challenges presented by semaphores in particular. The Semaphore docs example shows a more fleshed-out version of the “listen for user events to artificially throttle/unthrottle the semaphore,” if you want to check it out.

If you have an itch to try your own hand at writing concurrency primitives, I cannot encourage you enough: it’s all kinds of challenging and rewarding, and really opens your mind to what goes on behind-the-scenes with synchronization types. The rsevents crate was written to make doing just that a breeze, and I recommend at least starting off with either manual- or auto-reset events to take care of the intricacies of sleeping and the correctness of waking one and only one past/future awaiter at a time. Rust generally uses channels and mutexes to take care of synchronization, but there’s always a time and place for lower level thread signalling constructs!

Show some love and be the first to get new rust articles!

Last but not least: a request for you, dear reader. I put a lot of effort into these rust writeups (and into the open source libraries and crates I author) for nothing but love. I’ve heard good things about Patreon, and have literally just now put up a page to see if anyone would be interested in sponsoring my open source work. If you can’t spare some change to support me and my work on Patreon, please consider following me and my work on twitter, and starring rsevents and rsevents-extra on GitHub.

I’m currently looking for work opportunities as a senior engineer (rust is preferable, but I’m a polyglot). If you or your team is hiring, let me know!

If you liked this writeup, please share it with others on your favorite nerdy social media platform. I also set up a small mailing list that only sends out an email when I post about rust here on this blog, you can sign up below to join (no spam, double opt-in required, one-click unsubscribe, I never share your email, etc. etc.):

Just posted a ~longer writeup on what it takes to implement a truly safe Semaphore type in #rust. Feedback welcome. https://t.co/QZdeCACpnH

— Mahmoud Al-Qudsi (@mqudsi) October 4, 2022

Update (10/5/2022): The examples in this post have been updated to use Ordering::Acquire or Ordering::Release when reading/writing self.count in the wait() and release() family of functions to synchronize memory/cache coherence between threads and ensure correct instruction ordering. In the original version of this article, the examples all used Ordering::Relaxed and relied on self.event to take care of ordering, but as self.event is skipped as an optimization in cases where the semaphore has available concurrency, this was insufficient.

1. In fact, in some languages/apis a “critical section” is another name for a mutex. ↩

2. For a mutex, they are one and the same as it mandates exclusive access to a region. ↩

3. Semaphores are generally non-reentrant so recursively obtaining a semaphore will count against its limit! ↩

4. The empty parentheses/tuple () is rust-speak for void, for those of you not familiar with the language. ↩

5. In rust, there’s actually (out of necessity) an entire class of  types or values that can be changed even through read-only references, a concept known as interior mutability and exposed via types like AtomicXXX and the various std::cell Cell types — but those are generally to be used on a fine-grained level and in general you wouldn’t be using them to make entire objects writeable via read-only references. ↩

6. Actually, the AutoResetEvent implementation only takes all of two bits, but let’s just call it a byte to make everything nice and easy. ↩

7. We would still have to keep Semaphore::release() around and make sure it can be publicly called so that a semaphore initialized with { count: m, max_count: n, ... } with m ≥ 0 and n > m can be used. ↩

8. You can see a sample application testing for pthread_mutex_unlock() safety here and try it out for yourself online here. ↩

9. After all, if we use an AtomicU8 to represent the max/initial count, they can be as small as three bytes each! ↩

10. For the sake of this example, let’s assume NUM_CPUS is a sufficiently large number like 4 or 8, so that enough worker threads will try to enter the semaphore-protected region. ↩

11. rsevent-extra‘s CountdownEvent might just be the perfect tool for this job, btw! ↩

12. Another way would be to pack count and max_count into a single double-width atomic (assuming such an atomic of this size exists) and to decrement both count and max_count when a call to Semaphore::wait() is made. This way, any calls to Semaphore::release() would compare the potential increase in count against a similarly decremented max_count and can catch any violations of our core precept. The issues described in the remainder of this article persist regardless of which method was chosen. ↩

13. In rust parlance, memory leaks do not fall under the safety guarantees of the language and it’s perfectly “safe” if not exactly cromulent to write code that doesn’t drop() RAII resources. ↩

14. Give up? Just draw another, longer line next to it! ↩

15. I still need to sit down and experiment with packing count and max_count into one atomic double-width word and see how it works to decrement both count and max_count after each call to wait() instead of tracking with an additional total_count field, but even there we’d have a price to pay. We can no longer use AtomicXXX::fetch_add() and we’d have to use compare_exchange_weak() in a loop, after fetching the initial value, separating it into its component fields, incrementing/decrementing, then combining the fields into a single word again – although a quick godbolt attempt shows the compiler actually does a rather nice job. ↩

16. C# is a great example here, with extensive input and sanity checking for most APIs but almost all of it in the form of runtime exceptions – despite it being a strongly typed language with powerful and extensible compiler integration. ↩

17. It currently still uses atomic unsigned integers in the implementation and so does not implement the wait-free, eventually consistent API to artificially reduce the currently available concurrency without making a wait() call, waiting for it to return, and then forgetting the result. At the time of its initial development, I had started off with signed integers then realized I didn’t need them for the core functionality and switched to unsigned atomic values instead. I may need to revisit that decision in another release if it can give us either a wait-free reduce() corollary to release() instead of the Semaphore::wait().forget() or the current modify() method which allows wait-free direct manipulation of the internal count, but only with an &mut Semaphore reference (to guarantee that it isn’t in use, eschewing eventual consistency for correctness), but feedback on whether a wait-free reduce() at the cost of eventual consistency is a win or a draw/loss would be appreciated from anyone nerdy enough to read these footnotes! ↩