A stunning day to head out and play with a new wide lens.

This churchyard tree had such vibrant contrast with the blue sky:

Parramatta has seen a lot of construction. I love the scale here of the tree, the building, the cirrus.

Let's mix in St John's Anglican Cathedral:

The wonderful clock glows bright gold in the noon sun. I learned how to un-distort this image to get the spire roughly rectilinear. Very happy with the result.

Do you ever want to profile some Ubuntu/Debian apps installed with apt-get, to see why they're slow, but all your symbols come out as [unknown], or there are stack frames missing? I'll explain how to fix these issues.

As an example, have you ever run top and seen that one of the top users of CPU is... top itself? It seems suboptimal, that the CPU usage debugger, uses so much CPU itself.

I'd like to profile top to find out why it's slow. There's lots of programs like top: prebuilt programs from apt. I've run into many roadblocks trying to profile these prebuilt binaries many times, and I'm not sure anyone's written up a guide yet.

My examples are from Ubuntu 22.04, but will probably work on Debian.

TL;DR

# Install perf
$ sudo apt install --yes linux-tools-common

# Install debug symbols for package you are profiling
$ sudo apt install --yes $FOO-dbgsym

# Allow profiling by non-root users, and visibility to kernel stacks.
$ sudo sh -c 'echo 1 >/proc/sys/kernel/perf_event_paranoid'

# Allow visibility of kernel stack traces
$ sudo sh -c 'echo 0 > /proc/sys/kernel/kptr_restrict'

# Profile an app, using the -dbgsym dwarf information.
$ perf record --call-graph dwarf $(pidof $FOO)

# Convert to a text format
$ perf script --input perf.data -F +pid > perf.txt

Then drag and drop perf.txt into https://profiler.firefox.com.

Install Perf

We'll use perf to profile Linux applications. Install it:

$ sudo apt install --yes linux-tools-common
$ perf version
perf version 5.15.148

Install Debug Symbols for the Profiled App

Debian/Ubuntu packages don't come with debug symbols by default, but we'll need them. They come in debug symbols packages, which conventionally have the same name as the base package, but ending with -dbgsym.

First, we have to find which package top is in, there is no top package. What package installs top? We can see with dpkg --search <path>.

What's the path we should be searching for? I don't know if top is in /bin or somewhere else. which will tell us:

$ which top
/usr/bin/top

Putting it together, we find the package:

$ dkpg --search /usr/bin/top
procps: /usr/bin/top

So top is installed from package procps, therefore the debug symbols will be in procps-dbgsym. Let's install that:

$ sudo apt install --yes procps-dbgsym
The following NEW packages will be installed:
  procps-dbgsym
After this operation, 664 kB of additional disk space will be used.
Setting up procps-dbgsym (2:3.3.17-6ubuntu2.1) ...

Less Paranoid Perf

We could profile as root, but:

• I'm often profiling short-lived apps I need the profiler to run, and I don't trust them to run as root.
• It's a faff having the output files be owned by root.

If you try profiling on Ubuntu as a non-privileged user, you get this long and incorrect error:

$ perf record -p $(pidof top) --call-graph dwarf
Error:
Access to performance monitoring and observability operations is limited.
Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
access to performance monitoring and observability operations for processes
without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
More information can be found at 'Perf events and tool security' document:
https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
perf_event_paranoid setting is 4:
  -1: Allow use of (almost) all events by all users
      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>= 0: Disallow raw and ftrace function tracepoint access
>= 1: Disallow CPU event access
>= 2: Disallow kernel profiling
To make the adjusted perf_event_paranoid setting permanent preserve it
in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>)

The docs talk about -1, 0, 1, and 2. But the perf_event_paranoid setting is 4 ? Huh? The kernel documentation only describes up to level 2. What is 4? Well, Debian patched in an extra level 3, and Ubuntu changed it to level 4, which means: "disallow all unpriv perf event use". See AskUbuntu, and the commit adding this.

Let's lower (open) this to 1, which is the highest level that allows kernel profiling.

$ sudo sh -c 'echo 1 >/proc/sys/kernel/perf_event_paranoid'

I don't really understand the security ramifications here. The LKML thread, where an Android developer tries to upstream it, talks about information leaks and local privilege escalations via perf-events. Maybe reset it once you're done?

Allow Profiling Kernel Symbols

The next hurdle is seeing what functions we are calling in kernel space. You may get this warning:

$ perf record -p $(pidof top) --call-graph dwarf
WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted,
check /proc/sys/kernel/kptr_restrict and /proc/sys/kernel/perf_event_paranoid.

Samples in kernel functions may not be resolved if a suitable vmlinux
file is not found in the buildid cache or in the vmlinux path.

Samples in kernel modules won't be resolved at all.

If some relocation was applied (e.g. kexec) symbols may be misresolved
even with a suitable vmlinux or kallsyms file.

Couldn't record kernel reference relocation symbol
Symbol resolution may be skewed if relocation was used (e.g. kexec).
Check /proc/kallsyms permission or run as root.

By default, Linux disallows unprivileged users from seeing the locations of kernel function symbols. Symbol locations are randomised to make attacks on these structures harder.

But I just want to profile, and this is a system that only I'm running code on. Disable this with:

$ sudo sh -c 'echo 0 > /proc/sys/kernel/kptr_restrict'

Set it to 1 once you're done if you like.

Profile the Program

Finally! Let's run top in one terminal, then in another, profile it with perf record. Ctrl-C when done:

$ perf record --call-graph dwarf $(pidof top)
^C[ perf record: Woken up 6 times to write data ]
[ perf record: Captured and wrote 1.322 MB perf.data (163 samples) ]

Visualise the Output

My favourite way to look at the output of perf is with Firefox Profiler. Despite being named after the browser, it's a tremendous general-purpose profile analysis UI.

Follow their instructions for loading perf profiles:

$ perf script --input perf.data -F +pid > perf.txt

Then drag and drop perf.txt into https://profiler.firefox.com. All going well, you should see a profile like this:

As it turns out, top is using so much CPU because it's spending most of its time inside close, open, fstat, opendir, and getdirents system calls reading thousands of files in /proc.

Resolving [unknown] Stack Frames

I still have some missing [unknown] symbol stack frames. Hovering over the frame, Firefox Profiler tells me these are in file /usr/lib/x86_64-linux-gnu/libprocps.so.8.0.3. Let's install debug symbols for those, too. We can find what package with the same dpkg command:

$ dpkg --search /usr/lib/x86_64-linux-gnu/libprocps.so.8.0.3
dpkg-query: no path found matching pattern /usr/lib/x86_64-linux-gnu/libprocps.so.8.0.3

Huh, I don't know why that doesn't work. Let's try without the path:

$ dpkg --search libprocps.so.8.0.3
libprocps8:amd64: /lib/x86_64-linux-gnu/libprocps.so.8.0.3

OK, weird, dpkg is reporting the file as in /lib, and perf is reporting it's in /usr/lib. Both files exist and have the same hash.

$ sha1sum {/lib,/usr/lib}/x86_64-linux-gnu/libprocps.so.8.0.3
a2a2cd0dc5c0d88282a15e27742bac42a1e550d5  /lib/x86_64-linux-gnu/libprocps.so.8.0.3
a2a2cd0dc5c0d88282a15e27742bac42a1e550d5  /usr/lib/x86_64-linux-gnu/libprocps.so.8.0.3

Maybe it's a bug that dpkg can't find this? If anyone knows, leave a comment?

Anyway, let's guess that libprocps8's debug symbols are in libprocps8-dbgsym:

$ sudo apt install --yes libprocps8-dbgsym
Setting up libprocps8-dbgsym:amd64 (2:3.3.17-6ubuntu2.1) ...

Excellent. Re-profiling, the profile looks complete. We can see the previously-unknown symbols in the libprocps.so.8.0.3 frames. Here, simple_readtask:

Common Problems: No Kernel Stack Frames

If your profile is only yellow (userland) frames with no orange (kernel) frames, you may be missing permission to profile the kernel. Check the "Less Paranoid Perf" section above.

Common Problems: No Kernel Stack Symbols

If you have kernel stack frames, but they all say [unknown], check the "Allow Profiling Kernel Symbols" section above.

Conclusion

Well, this is a bit of a faff! Can't we have nice things? No wonder hardly anybody bothers to profile, and so much of our software is still so slow.

Maybe one day, perf can be security-hardened enough that these settings could be enabled by default?

Until then, I hope this checklist can help lower the bar to understanding software performance. Go, profile, and make code faster!

How we built a low bed, for co-sleeping with baby.

We sleep in the same bed as our baby, and we like it. To reduce risk with a young baby, we slept with baby on the side of the bed, next to mum.

But on the side of the bed, particularly once baby starts rolling, they can fall out of bed. Higher falls hurt more. You can put bed rails up, but rails have their own risks: it seemed simpler to lower the bed.

Pre-Baby: A very high mattress

We had a queen mattress, on a box spring base, elevated on legs. Probably over 50cm over the ground. We put a camping mattress next to the bed to dampen falls.

What do people do? We found lots of opinions online:

• Put a mattress on the floor
• Place the mattress on the bedslats on the floor (or just planks of wood)
• Sleep on a camping mattress / tatami mat
• Saw the legs off the bed
• Buy a short bedframe
• Build a custom pallet bed

For how popular co-sleeping is, this is surprisingly unclear! Many of these options are uncomfortable, annoying, or require you to be handy.

First Try: Mattress on the floor

Our first attempt was to just put the mattress on the floor. Height: 25cm. The obvious problem is mould growth. Reddit threads said it'd be fine.

You’ll want to prop up your mattress against a wall every now and then to air it out and prevent mold from growing underneath. – cosleepy

So we dutifully propped up our mattress on the wall. A huge mattress, this was very annoying, so we didn’t do it often. Perhaps predictably, we found mould on the bottom after a few months. Perhaps Sydney is more humid? Maybe we didn't do it enough? I don't think we'd have the energy to do this after every naptime.

We bought a new mattress, and researched what to put under it.

Option: Wood slats under the mattress

A common opinion:

go to the hardware store and ask them to cut some boards to put under the mattress so it’s not right on the ground

So we just took the bed slats from our bed frame and put them underneath the mattress

But I'm pretty sure the slats will move under the bed and scratch the floor. Slats need a frame to hold/stretch them in place!

Option: Buy a short bed frame

I tried to search for a short bed frame. They aren't very short! Most are at least 20cm high:

• 21cm legs: $69: GRIMSBU Bed frame, grey, Queen - IKEA
• 20cm legs: $199: VEVELSTAD Bed frame, white, Queen - IKEA. same style as single bed we have
• 20cm legs: $229: TARVA Bed frame, pine, Queen - IKEA  + $70 slats
• 20cm legs: $279: ASKVOLL Bed frame, white, Queen - IKEA
• 20cm legs: $299: KLEPPSTAD Bed frame, white/Vissle beige, Queen - IKEA
• 15cm high: $179: Studio Home Harmony Low Rise H15cm Bed | Temple & Webster. has slats. 
• 15cm high: $189: Tori Low Queen Bed | Fantastic Furniture
• 24cm high: $1,445: Eva Timber Bed Frame

Some people advised to "get a wood ikea bed frame and cut the leg off with 1cm clearance" which I guess would work, but I'm not that handy.

Solution: IKEA LÖNSET Slatted bed base

IKEA's LÖNSET bed slats are just 9cm high: truly short. The slats bend, offering more flexibility vs mattress directly on the floor.

LÖNSET Slatted bed base, Queen - IKEA

LÖNSET Slatted bed base, Queen. 30 slats adjust to your body weight and increase the suppleness of the mattress. Comfort zones adjust to your body.

IKEAIKEA

They come with their own rectangular frame to keep them in place. IKEA intends for them to be used nested inside another bed base, but they work perfectly fine standalone.

To make queen-size, they come in two units. We tied them together with some gardening ties so they don't skew apart:

We were a little concerned about the wood slats scratching the wood floor, so we bought some cheap non-slip pads:

I kept stubbing my toe on the ends so we added babyproofing (well, daddy-proofing) foam edge/corner protectors. See bottom right:

The end result is about 32cm high, and almost all of that height is the mattress itself. The slats flex, giving significantly more comfort than having the mattress directly on the floor. At the start, it's annoying bending down to the floor, but your muscles adapt and get used to it.

An unexpected benefit: the baby can climb in and out of bed by himself. Saves us getting out of bed.

I'm pretty happy with how this turned out!

P.S. (2024-12-27): A reader asked what happens when our child falls out of bed.

We had carpet next to the bed in old last apartment and that was OK for him to fall onto: a few cries, but no injuries. We are a bit more worried about these wood floors, so we put our camping mattress next to the bed.

Partially inflated, not fully inflated, so it cushions falls better. In our small bedroom, it wedges in between the wall, bed and drawers.

Our child rolls out of bed onto the mat often. And, astonishingly, almost always stays asleep! It's a short drop and he loves sleeping down there. Maybe there's more room and less disturbance from us.

Before having a baby, I canvassed all the dads I know for "new dad tips". The most common advice was "get a good camera, because they grow up so fast". I wasn't sure I'd be into this whole photography thing, so I picked up a second-hand entry-level mirrorless Sony camera off eBay. It's fun, and yes: it's better than the phone camera.

After the baby arrived, we had lots of family visits, leading to family photos. I wanted us all to be in the family photos, so something needs to hold the camera.

The normal answer is "get a tripod", but don't we already have a rigid, stable, movable frame: our pram! Could I use that?

I first tried zipping up the pram top and resting the camera on that. But the camera was too low; the crotch-height angle was unflattering.

Could I attach to the pram handle instead, a bit higher? With our Uppababy Cruz V2 pram, the handle telescopes, allowing some height control.

Researching camera mounts (definition: "things that attach cameras to other things") was more complicated than I expected. There exist mounts for attaching cameras to all sorts of objects.

But even with all the upselling accessories of modern prams ($100 cup-holders, anyone?), I couldn't find specifically-marketed pram/stroller mounts.

I worried about screw incompatibility, but cameras seem to have standardised on the 1/4" threaded screw. Thank goodness.

Mounting the Pram-Camera (Pramera)

I bought the SmallRig model 1124: "Super Clamp Mount with Ball Head Mount" for $21AUD, and found I can clamp it to the top of the pram: it particularly grips well on the pram's hard plastic handle, not the soft leather.

You must clamp the narrow axis of the handle, otherwise it rotates around. The narrow axis points diagonally, so the clamp points diagonally, but the ball joint lets you correct for this and angle the camera flat.

I worried about the camera falling off and smashing the lens. The camera's centre of mass is lopsided with the heavy lens putting torque on the screw.

And the diagonal axis puts a bit of torque on the clamp. You can help this a little by aiming the camera sideways, or backwards toward the mount, as shown in the picture above. You have to aim the camera backwards anyway to avoid getting the pram in your photos.

I probably wouldn't keep the camera screwed there while moving over uneven terrain; but it's been perfectly stable stationary while taking photos.

It's a little slow to screw on the camera and adjust the ball to get things level; but no more difficult than a tripod. The pram wheels help positioning for framing.

Baby can't reach the handle, so I'm not worried about small parts/choke hazards. Even so, I think you'd struggle to disassemble this to small-enough parts to worry.

Conclusion

Overall, the pram-mounted camera has been good for our family, and I'd recommend it. The pram is a natural tripod. The height is good. We keep the clamp on the pram handle long-term, and dangle toys from it. The clamp doesn't get in the way. And we get nice family photos, with all of us in them.

Walking around, I don't see anyone else doing this. I think the Pramera should be more of a thing!

People are flocking to Mastodon. The cool users are "verified": the links in their profile are ticked and green, so you can be more confident they aren't imposters. e.g. here's Scott Hanselman's profile:

In contrast, my profile looks decidedly less "legit":

I would like my profile to be clear that it's the real me, not some spammer pretending to be me. You achieve this with Mastodon verification:

Document-based verification and blue ticks are not possible without a central authority. However, Mastodon can cross-reference the links you put on your profile to prove that you are the real owner of those links. In case one of those links is your personal homepage that is known and trusted, it can serve as the next-best-thing to identity document verification.

If you put an HTTPS link in your profile metadata, Mastodon checks if that link resolves to a web page that links back to your Mastodon profile with a special rel=me attribute. If so, you get a verification checkmark next to that link, since you are confirmed as the owner.

So I need to change this site. I host this site, markhansen.co.nz, on the Ghost blog engine. How do you add such a link with rel=me? There's this forum thread, proposing two solutions and ultimately floundering:

1. Update the theme's HTML: I don't want to fork the theme and add custom code, and constantly rebase vs upstream. I pay Ghost to avoid that maintenance pain.
2. Add an <a rel="me" href="..."> link. Where? My theme doesn't support extra links with custom rel. Dead-end.

I took a different approach, using the <link ref="me" href="..."> syntax in the site-wide header with code injection. Visit your blog, Settings, Code Injection, and add:

<link href="https://hachyderm.io/@markhansen" rel="me">

Save, reload and view-source on your blog to check the source is now just before your </head> tag closes.

Editing your Mastodon profile and saving will trigger validation:

I blogged this because I couldn't find a quick answer searching [ghost blog validate mastodon]. Hope this helps.

I had a familiar conversation at work recently. A colleague asked: "There's a feature I only want to show to users if some hardware's present. The Android API for hardware presence says the API must be called on a background thread. But I need it from the UI thread, and on application startup, at the first frame. How can I do this?"

I asked:

• Where does the API get the data from? From some driver implemented by the device manufacturer.
• How long could the API call take? Up to 2 seconds, with retries.
• Does the data ever change? No, the same device will always return the same value.

If we call this API on the UI thread, we might block user interaction for two seconds. That’s probably long enough to cause "Application Not Responding" timeouts (crashes!) on some devices. And it's definitely long enough to annoy users.

It’s a shame that this logically static data is fetched from a general-purpose asynchronous API, backed by the manufacturer's hardware-abstraction-layer drivers, which could be arbitrary code.

The call is at least interprocess communication, which can be slow even if the other side is implemented well. But this app was running was in a car (Android Automotive), so it's possible the driver might even fetch the data from another vehicle computer via the in-vehicle Ethernet. This might be even slower; the other computer might not even reply at all!

What can we do?

So we have a choice, either:

1. Fetch the data synchronously at app startup, and risk app downtime while the app waits for the response, or:
2. Fetch the data asynchronously. This may need a reworking of the code to support async loading. The code will have to handle the case where the data isn’t ready yet, then update later on once the data is ready; even if the user's in the middle of doing something with the UI. A bunch more complexity.

Please don't block the app

I told my colleague, you have to change your code to be asynchronous; we can't risk app downtime if the API is slow, because of your one feature.

You have to show something to the user while you're waiting for this data; perhaps you should hide your feature until API responds? "Pessimistic UI" and "Optimistic UI" are opposite ways of how to handle this.

CAP Theorem

This is an everyday example of the CAP Theorem. The CAP theorem (loosely) states that out of:

• Consistency (serving the right answer)
• Availability (always serving data)
• Partition-Tolerance (robustness in face of network disconnections)

You can choose max 2 of 3.

Our Android system can experience network partitions (between computers in the car), and we some data we want to read across the network... should we either:

1. Block waiting for the data: choosing consistency, waiting, possibly for a long time, and blocking the user from doing their work while you wait? or
2. Code asynchronously, 'guessing' until the data arrives with a default answer (maybe: "the hardware doesn't exist, disable the feature"). We lose consistency, but maintain availability.

Most modern applications are networked, so have to deal with network partitions. Usually, availability correlates with business outcomes: users being able to do the work they need to, and ultimately pay the business. So, for many applications, consistency is given up first.

What's Acceptable to Block On?

There's always a choice. Often, your app needs to make many async calls: over the network; or inter-processes; or to disk. These connections can fail. In our app, we generally accept blocking on resources where there's no other way:

• Blocking on disk reads to load our application's code.
• Making inter-process calls to known-mostly-reliable components, e.g. calling SurfaceFlinger to render frames to the screen.

We generally don't accept blocking on:

• Network calls.
• Inter-process calls backed by arbitrary unknown code.

Spotting CAP Problems

It's useful to spot when you're facing a CAP problem. CAP is stereotyped as mostly applying to server-side databases, but it comes up a lot in networked client applications.

Ask: when making a request that could take a long time, should our app wait a long time for the response (choosing consistency)?, or assume an answer until we get the response (choosing availability)?

TL;DR: No.

Android code can be slow. Profiling will often reveal the cause as"View inflation", which just means "constructing a View", e.g. calling new TextView(Context), or new ImageView(Context), or new LinearLayout(Context). These constructors often do a lot of work, slowly (for reasons out of scope to this post), and they block the UI thread.

When people find UI-thread-blocking jank, they naturally want to speed it up. In Android, the usual advice is to move janky slow stuff to a background thread. But it's unsafe to move View inflation to a background thread. You may make your code faster, but you risk undefined behaviour: very annoying, impossible-to-debug failures that will waste your team's time.

Android is pretty clear on this:

the Android UI toolkit is not thread-safe. So, you must not manipulate your UI from a worker thread—you must do all manipulation to your user interface from the UI thread.

The Problems

View constructors access their Context, their parent View, and static variables. All of this data is mutable, and written and read without synchronization. And if you do these reads and writes from different threads, you have data races, you may see both old and new data. You'll get undefined behaviour.

Here is a (non-exhaustive) list of examples:

1. Mutable Context. Context (really, ContextImpl behind the scenes) is very mutable. Some data is protected by mutexes, but not all. For example, ContextImpl.mDisplay is non-final and mutated. Without synchronization, there's no safe publication across threads. Views may access the Display to decide what UI to show.
2. Parent LayoutParams. Views often access their parent's LayoutParams to decide how to render. The parent could be mutated from the UI thread (perhaps in the middle of a layout pass) while you're reading their params.
3. Static Variables. Android often puts data in mutable static datastructures, to save allocations. If you're read these concurrently with a write, you'll see corrupted data.

For example, every View subclass must eventually call the super-constructor android.view.View(Context), which calls ViewConfiguration.get(Context), which reads and writes a static SparseMap cache variable:

static final SparseArray<ViewConfiguration> sConfigurations =
    new SparseArray<ViewConfiguration>(2);

public static ViewConfiguration get(@NonNull @UiContext Context context) {
  ...
  ViewConfiguration configuration = sConfigurations.get(density);
  if (configuration == null) {
    configuration = new ViewConfiguration(context);
    sConfigurations.put(density, configuration);
  }
  ...
}

SparseArray is a compound data structure, backed by an Object[] and an int length. Interleaved calls to this method from different threads are a data race, and could (for example) result in an int length longer than the Object[]'s size.

This is just one data race, which affects all Views. There are probably many more races. Custom Views can have arbitrary logic.

What about AsyncLayoutInflater?

What about AsyncLayoutInflater, the official Android support library which "Triggers view inflation on background thread"?

AsyncLayoutInflater notes some requirements about where it should be used:

For a layout to be inflated asynchronously it needs to have a parent whose generateLayoutParams is thread-safe and all the Views being constructed as part of inflation must not create any Handlers or otherwise call myLooper.

This addresses problem #2 above... but not #1 and #3.

So is AsyncLayoutInflater safe? I don't think so.

My partner & I recently had a baby. As part of having a baby, the state of New South Wales demands you name them, with (at least) a first name and a surname. First names are mostly anything-goes, but surnames present difficult tradeoffs around gender equality, recognition of history, convention, social ease, verbosity, convenience, spelling, stigma, future-proofing, and the politics of parents and grandparents.

We eventually decided on a portmanteau surname for our child. Here's how.

Background: Surname History

It's complex. Surnames were usually mandated by state bureaucrats, to help disambiguate citizens, making it easier to tax and conscript them.

My surname (Hansen) started as a patronym (son-of-Hans); when my great-great-great-grandfather Hans Peterson had a child Hans Roby Hansen, sometime in the 1800s. This became a permanent/fixed patrilineal name sometime around Denmark's mid-1800s naming laws, passing down to children who's father was not Hans. European nobility had long used surnames to indicate legitimacy of children, but mandatory European commoner surnames are relatively recent: many countries only required surnames from the 1800s.

My partner's surname "Lowe" is an Anglicised version of the Cantonese "Lau", which traces back at least hundreds of years. Her parents' generation chose the Anglo-Saxon-sounding "Lowe" when immigrating to Australia, likely to reduce the racism their family would face. China, with its strong bureaucratic central government, has had widespread surnames for thousands of years.

Both names have significant history for us. But, so do our mother's maiden names, and we don't carry those. We can recognise history without carrying a surname. Genealogy is getting easier with better record-keeping, even without matching surnames.

Surname Options

We weren't sure what to do. We wanted to be intentional about choosing a name, not just "go with the default".

I didn't really like the patriarchy of choosing my surname over my partner's. I was a little worried about "giving up the history" of my name and choosing my partner's surname. Double-barrelled names are getting more popular. "Lowe-Hansen" rolls off the tongue nicely (like Johansen), but both our names have spelling difficulty, with "e"s that you always have to spell out over the phone; would our child resent spelling out "Lowe-Hansen" for the rest of their life?

I read some interviews on the Feminist Last Naming Practices Project. And I was grateful to find a book that explores the topic, by an Australia-based author:

The book explores the tradeoffs by interviewing parents who have chosen from the following options:

1. Father's surname
2. Mother's surname
3. Hyphenation, or a double-barrelled surname (without a hyphen)
4. Alternating the two parents' surnames between siblings
5. Combining the two surnames into a portmanteau or blended surname
6. Making up a completely new surname

All present tradeoffs, there's no right or wrong answer.

A Portmanteau/Blend?

It was tempting. We'd heard rumours of portmanteau names. But didn't know anyone that had actually done it. Lowe + Hansen... "Hanslow"? "Lohan"?

We initially discounted it as too "out there". But we kept thinking about it. The interviews in the book with happy parents who'd chosen it made us more confident we'd be happy with one.

The book helped mitigate our main concerns: trouble at borders? Not really, and if there is ever any, show the birth certificate. Are there other bureaucratic challenges? Not really, apparently. It's pretty common for children to have different names from their parents anyway, with divorces and combined families. Systems generally already have to handle this.

A Tool for Finding Portmanteau Surnames

There are surname-history websites, and portmanteau-search websites. But no surname-portmanteau-search websites. So I made my own.

I looked for a dataset of surnames – although we could choose any word, probably limiting it to 'actual surnames' seemed like a reasonable place to start. The most detailed surname dataset I could find was the US Census' list of surnames with over 100 occurrences: 162,254 surnames.

I loaded this into a Jupyter Notebook. I put the code online into a Colab Notebook so you can copy and edit the code in-browser.

import io
import pandas as pd
import requests
import zipfile

r = requests.get("https://www2.census.gov/topics/genealogy/2010surnames/names.zip")
with zipfile.ZipFile(io.BytesIO(r.content)) as z:
  with z.open("Names_2010Census.csv") as f:
    df = pd.read_csv(f, na_filter=False)
df

First looking at anagrams: with all characters drawn from our names:

def anagram(s):
    return all([c in "HANSEN LOWE" for c in s])
df[df["name"].map(anagram)]

This didn't work well. It kept giving options with letters drawn from just one of our names (e.g. LEE, HESS). So I constrained it to require at least two characters from each person's name:

def anagram(s):
    return (all([c in "HANSENLOWE " for c in s])
            and sum([c in "HANSEN " for c in s]) > 2
            and sum([c in "LOWE " for c in s]) > 2)
df[df["name"].map(anagram)]

The results weren't great: ALLEN, NELSON, OWENS, LAWSON, OLSEN... these didn't really feel like a blend of our names.

So I tried another approach, looking for names that had some part of each of our names:

def bits(s):
    return ("LO" in s or "LAU" in s) and ("HAN" in s or "SEN" in s)
df[df["name"].map(bits)]

This turned up much better options with a blended sound: HARLOW (fun?), HANLON (of razor fame?), HALLOWELL (that's a lot of L's), HALLOWAY (if you think of my partner's name as pronounced Lowé it almost makes sense), SHALLOW (hmm, not sure of the implications), HALLOW/HALLOWS (fun, but sounds like a greeting), LOHAN, HANSELL.

Copy and edit the notebook online.

Using the code, we didn't find any names we liked more than the initial guesses of Lohan and Hanslow. But the process reassured me that we weren't missing a better option.

Lock it in

We tried-out the name Lohan for a week. It felt good. Easy to spell, easy to say.

It feels egalitarian: half of my name, half of my partner's name. While I get more letters of my name in (3 letters to 2), my partner gets the whole pronounced syllable of her name in.

It removes our surname's hard-to-spell suffixes. It's a mix, just like the baby is. It has Celtic history, just like I do.

We put Lohan on the birth registration, and the state of New South Wales was happy to ink it on the birth certificate.

The parent's surnames haven't changed: we didn't think "having the same surname" is very important for family cohesion.

I'm glad that we have this choice. We found a name that's true to our values. Blended surnames are a real option.

After I lost a hard drive to (possibly) a lightning storm, I re-evaluated my backup strategy. I chose which cloud backup software I'd use (winner: Duplicacy), but where should I upload my backups to? There are lots of blob storage providers.

The pricing is fairly tricky to work out. There's usually storage fees (per GB per month), sometimes 'restore fees' if it's in deep storage, and sometimes network egress fees when you want to get your data back out onto the public internet. Some providers have 'minimum retentions' where you have to pay for at least 1, 3, 6, or 12 months of storage.

What I care about: I have about 1TB to back up. I want backups to be quick, so they should probably have an Australian region. I'm happy to keep the backups for at least 90 days. But I probably want to start thinning backups out after 90 days, and delete altogether after a year. I don't really want minimum retention period of a year because then I'll feel very locked-in.

Egress fees: I don't care so much about network egress fees because I think the risk of me losing this data locally is pretty low, about <10% each year. If my local data fails, I'll probably be happy to pay a bit to restore. Or maybe I can poke around the data on a VM in their cloud, and only egress the really interesting stuff. So my expected value of egress fees is maybe one tenth what's quoted. Egress fees are generally high, so you really don't want to use a backup software that has to read back data as part of normal operation.

Your mileage may vary: This is a pretty rough, not very rigorous, somewhat "vibes-based" comparison. I didn't bother considering restore fees. You should probably read Anthony Agius' Cloud Archive Storage Comparison in addition to mine, because he goes deeper into niche players that I couldn't be bothered evaluating, and considers restore fees. Also this great comparison on the Duplicacy Forum.

I chose Google Cloud Storage Coldline. This was frankly a bit disappointing, as I was hoping to find something better than what I was already using!

Options

Here's the spreadsheet I made to evaluate (original in Google Sheets):

Google Cloud Storage

Disclosure: I'm employed by Google, and use Google Cloud Storage at work.

I was already using Google Cloud Storage Coldline for Arq backups from my laptop. It's a good fit for backups, with decent storage pricing ($72/TB/year) and still instant retrieval when needed. There's a high-speed Sydney region, the security posture is great, I'm very familiar with the platform, the tool ecosystem is large, but the egress pricing, particularly in Australia, is extremely expensive.

Amazon Web Services S3

The market leader and inventor of this segment. Amazon has 5(!) classes of blob storage, from "Standard" to "Glacier Deep Archive".

Amazon has a Sydney region, which is probably fast to upload to. Their security posture is great. But their instant-access 90-day retention offering, "Glacier Instant Retrieval" was much more expensive ($166/TB/year storage) than Google's. And I didn't really feel like wrestling with the tradeoffs of "Glacier Flexible Retrieval" ($43/TB/year storage). Would each API request take an hour to return? What if I need to restore many files, do I need to rewrite my backup client to work in parallel? Or do I copy to other storage first? There's probably good answers to these questions but I don't have them.

AWS' egress fees are famously expensive and opaque, though it looks like it's cheaper than Google in Australia. It was pretty hard to find the egress pricing and I'm not totally sure I'm reading the right price?

Microsoft Azure Blob Storage

Azure is the #2 cloud. They have a Sydney region, and 30-day ("Cool") and 180-day retention options ("Archive"). They have lots of different pricing depending how much level of replication you want. I just looked at their cheapest option. Their 30-day retention plan "Cold" costs $132/TB/year storage. The 180-day "Archive" storage is cheaper than Google, but might take hours to restore, and I don't want to figure that out.

Backblaze B2

A lot of people on backup forums recommend Backblaze B2. The price is great: $60/TB/year storage, the egress fees are <10x cheaper than Google/Amazon/Microsoft, and it's backed by a big backup company. I thought it was promising, so I tried it.

B2 was slow. I'm not sure exactly why; maybe it's the latency from Australia to their closest datacenter (US-West)? Or maybe their computers or disks just can't sink the data fast enough. My home internet is 40Mbps up, but with Duplicacy backing up to Backblaze B2, I struggled to maintain 10Mbps, even when I cranked it up to 20 parallel upload threads.

This made the backup take around 3x as long as it should (around a week, rather than two-ish days), so I cancelled the upload midway and moved to Google Cloud Storage. It's a shame! I would have loved a cheaper option with no egress/retention fees.

Wasabi

Backup forum people also recommended Wasabi. Their price is the same as Google Cloud Storage Coldline ($72/TB/year), and they have no egress fees. I signed up to try them.

I like that they have a Sydney datacenter, which might make them faster. But it turned out you can only list buckets by sending requests to the US?

What? So I couldn't set up Duplicacy to use the local datacenter. This seems to have been broken since 2019, so I don't think it'll be fixed anytime soon.

I find it shady that Wasabi don't explain on their pricing page that they will charge you for minimum 90 days storage, while all other competitors I surveyed were up-front about this, Wasabi buries this information in their FAQ. I only found out about this because of people warning about it on backup forums! It would have been very annoying if I'd deleted some data and charged for 90 days of storage.

On Wasabi's pricing page, they directly compare their offering (90day min retention) against Google Cloud Storage Standard (0 day retention), when Coldline would be the comparable offering, and cost the same price.

This seems, at best, misinformed about their competitors, and dishonest at worst.

With data storage, trust is so important! You're paying for the trust that you'll be able to read your data back. Underhanded-at-best price comparisons, and hiding retention fees, hardly inspires trust. So I closed my Wasabi account and moved on.

Synology C2

I was backing up my Synology Network-Attached-Storage device. And Synology has an integrated cloud storage offering. They have no retention fees, but the closest datacenter they have is Taiwan, which is still a while away. And the price was pretty similar to Google Cloud Storage Coldline. In the end, I was pretty unimpressed with their software's speed, so I moved on.

Cloudflare R2

Cloudflare R2 made a big splash recently with object storage with $0 egress fees from a large CDN. It's unclear whether they have storage close to Sydney: they only support the auto zone, where they decide for you where your data lives. The pricing didn't really make sense for me though, at $180/TB/year. So I didn't look further into it.

iDrive e2

iDrive e2 has rock-bottom pricing ($48/TB/year), no egress fees. Their storage is "S3-compatible". No Australian region however, the closest is San Jose, CA.

But the vibes seem off – their pricing lights up the "too good to be true" scam-detection part of my brain. iDrive offer 90% off the first year, which seems... steeper of a discount than a reputable company would do.

Obviously it's a customer acquisition strategy and they hope to lock you in. But I'm skeptical of storage businesses that sell unsustainable products.

There are a lot of bargain-basement storage providers that offer "lifetime deals" or "unlimited storage". These are unsustainable and smell like pyramid schemes. iDrive isn't quite doing this, but it's close. I want to pay them a sustainable price to store my data. If not, why wouldn't they cut corners, and lose my data?

Storj

Some backup forum people talked of Storj, which claims to "Store data securely on over 16k nodes worldwide instead of just a few vulnerable data centers with privacy and CDN-like performance by design". The price is really low: $48/TB/year, with very cheap egress: $7/TB/year.

Random people get paid for their disk space on Ethereum. Regular audits are carried out to ensure nodes are actually storing the data.

It's interesting. But, again! Backup storage is about trust, and I don't trust blockchain grifters with my backups.

Conclusion

I went with Google Cloud Storage Coldline. The price is transparent and decent, the data is stored in Sydney, the upload speed is good, and the security posture is strong. I will pay a bunch in egress fees if I have to restore, but there's low probability of my disks failing, so I'm discounting that.

Hope this helps if you're trying to set up your own backups.

I'm setting up Duplicacy Backup after a drive failed during a lightning storm, and after evaluating backup software options. Backups are no good if they fail without telling you!

Here's how I set it up Mailgun to send the notifications of backup success and failure.

Duplicacy lets you send SMTP notifications after scheduled backups, by ticking the 'Send email after completion' checkbox.

Usually I'd register a throwaway GMail account for sending email from automated bots, but saspus@ on the Duplicacy forums suggested:

It’s best not to use Gmail for sending automated email; especially if it involves exporting credentials that provide full access to your google account and storing them pretty much plaintext within the service that has no business accessing your gmail account in the first place. This not only defeats any MFA you might have there but is also likely against googles TOS.

Especially when dedicated services for sending this type of transactional emails are readily available (E.g. mail gun, Amazon SES, etc).

They're right, it's annoying registering throwaway GMail accounts: you have to set up security questions and backup accounts and the account might get closed as spam. So let's try Mailgun instead.

Mailgun has a free indefinitely-long trial tier, which looks sufficient:

Sign up, verify your email, validate your phone number by SMS.

Then you have to add yourself as an 'authorized recipient'. I had trouble finding this, it's under "Sending > Overview > Authorized Recipients".

Free trial accounts can only send to these verified recipients. You get 5 free recipients; I only need one.

Then click 'SMTP', as Duplicacy only does SMTP sending, not HTTP-with-API-key sending:

Then create credentials for Duplicacy. You could just use the default sender credentials, but just in case I use this Mailgun account for more than one thing, I don't want Duplicacy to use the default (postmaster?) credentials. Click "Manage SMTP credentials", and "Add new SMTP user". I called mine "duplicacy":

The password isn't shown on screen, instead you copy it to clipboard:

Now go back to Duplicacy, and click "Send email after completion" and set it up like this, pasting in the password:

There were a few ports you can use, but only Server: smtp.mailgun.org:587 worked for me. Trying a different port actually hung the Duplicacy Web UI bugreport; the email sending blocked forever while holding a mutex.

Use the same 'From' and Username` field as your SMTP credentials. Put your email in 'To'.

Put {{Status}} into the Subject line which will be replaced with whether the backup succeeded or failed.

I then clicked 'Test'. My first few tests failed, I looked in the duplicacy_web.log file to see why. My test email ended up in Spam, and I clicked 'Not Spam'. Hopefully that will train GMail not to put this in Spam? But to be sure, in the message kebab-menu, select 'Filter messages like this':

Then "Never send it to Spam":

And you should be getting backup messages! Now, if only Duplicacy would allow you to only send messages on failure, I'd be very happy.

After my last blogpost, where one of my hard drives was maybe taken out by lightning, I was reminded to reevaluate my backup strategy.

I have data on my macbook, and on my Synology NAS. My macbook's backups are all good, having revisioned daily backups on Arq.

But my big files that live only on the Synology NAS (mostly disk images taken from disks inherited from a deceased estate) were only "synced" to Google Cloud Storage, by Synology Cloud Sync. Cloud Sync is not revisioned – only the latest version is kept, so a corrupted source would overwrite the sync. So, not really backups! Time to re-evaluate cloud backup software. I'm looking for software that will run on the NAS (Linux, Synology), at a reasonable cost for 1TB of backups, client-side encrypted, maybe with a year's worth of version retention.

Spoiler: Ultimately, I use Arq to backup my Macs, and Duplicacy for my Synology Network-Attached-Storage (NAS) device. See below.

Arq: It's OK

Arq is decent, and I use it for my Macbook. It uses a modern, content-addressed storage format. It mostly works, although I do worry when I hear about the developer breaking backwards compatibility between versions. And the UX is a little awkward to find things. And I worry a little that it's closed-source, but I'm reassured by them documenting the storage format. Overall, I would weakly recommend it as the best of a bad bunch.

But Arq doesn't run on Linux. So I can't use it on the Synology.

Crashplan: The Good Old Days

Crashplan used to be good. I used to use it. The price point was great and the software mostly just worked. But then they shut down their consumer offering, leaving only a too-expensive small business offering. A real shame.

Backblaze: The Market Leader

Backblaze is the best left after Crashplan exited. It's the Wirecutter recommendation. But I don't like Backblaze's software offering:

• Backblaze has no option to 'restore in place'. Your only restore options are downloading the data as .zip from a web UI, or requesting a delivery of a disk. Why not? All the other backup software offers in-place restores. Is it the challenge of a 3-way merge? Data is more than just the bytes; it's the metadata, the permissions, this is just not captured in the archives you download from the web. A dedicated restore software could do this. I don't know why they don't. Update, Aug 2023: Backblaze have announced a "new local restore experience". Good!
• By default, Backblaze will only keep 30 days of data. If it takes you longer than a month to notice data corruption, sorry! You're screwed. This is, frankly, a bonkers default to have. To their credit, they now allow longer retention (1 year for +$2/month, or pay-per-GB for >1yr). Update, Aug 2023: Backblaze raised prices by $2 and announced "all Computer Backup licenses may add One Year Extended Version History, previously a $2 per month expense, for free".
• When I last used Backblaze a few years ago, they immediately started uploading my backups in plaintext, before I had a chance to say "no, I want mine encrypted". While this probably makes their software easier for unsophisticated users (some growth hacker probably got promoted for it), it's a breach of trust to hoover up my personal files when I want them encrypted. I have no idea if they've fixed this now.

With these limitations, Backblaze being Wirecutter's annointed 'best' is an indictment of the backup software industry.

Synology Hyper Backup: Slow

I thought, Synology's built-in backup software will probably be decent, right? Commercial company behind it that sells lots of data backup boxes, hopefully they put some effort into this? A lot of people on /r/synology recommended it.

In positives, it's well-integrated with Synology, with emails alerts on failure, and can backup Synology software configuration. It has compression, and scheduling. It has an active user base. And it's built-in.

But unfortunately, I was disappointed, and gave up on Hyper Backup.

Synology Hyper Backup's Format is Secret

Having a secret data format is, I think, such an own-goal for backup software. It helps trust so much to be able to say "it doesn't matter if we disappear, here's how to read the data". You have to document it anyway for your own engineers, just make the docs public!

The closest we get to an explanation of the data format is this bullet-point-heavy marketing page.

To their credit, Synology have a software package that can read the backup files, Synology Hyper Backup Explorer, but this is still cold comfort if the backups get corrupted, as some people report happening in forums.

Synology Hyper Backup is Slow

It's extremely slow. Here's a representative progress bar: 0.00 B/s. It would jump between 0 and 1MB/s, mostly staying on 0.

I don't know why it was so slow. If you google [synology hyper backup slow], as you might expect, there's a lot of laypeople on the forum confidently being wrong about what makes it slow. Compression? Network bandwidth? Disk I/O are all given as explanations. Let's consider these.

Many people say compression will slow down the backup. I had compression enabled. But compression is usually very fast, and CPU-bound, and the CPU stayed at <10%. Even if it was 'nice'd, you'd expect it to be able to burst to use idle CPU. So we can immediately discount compression as a cause.

And htop confirms, the HyperBackup jobs are using 0-1% CPU.

My hard drive can read at 10MB/s, if sequential. Seeks will reduce this, but not to zero.

And my home internet upload pipe is 40Mbps (or about 4MB/s, after overhead). And I can regularly hit that 40Mbps, whenever Synology Cloud Sync runs.

So I wonder, what the heck is Synology Hyper Backup doing? I don't have the linux-perf-tools profiler installed, but I have "poor man's perf", looking at the kernel stack of the threads: cat /proc/$pid/stack. First I find the pid using pstree, with -p to show pids, we see that synobackupd is the root job, spawning img_backup which spawns img_worker which spawns img_backup and php.

You could expect the leaf nodes to be doing work, but here the img_backup tool is sleeping, calling the nanosleep system call. I can't tell exactly how long it's sleeping for (though I tried), but I'm guessing it's a long time, and that that's probably why my backup is slow.

The PHP process is at least polling a file descriptor for work to do, rather than sleeping. I looked at the PHP source – it's the plugin for uploading to S3. I found about 400 lines of uncommented PHP.

It seemed kind of a bizarre choice to use PHP as a backup software adapter. AWS S3 has bindings for all languages. Is it language-chauvinist to say it didn't inspire confidence? Well, backup software is all about trust, and uncommented PHP doesn't inspire trust.

Between the slowness (it would have taken weeks to upload), the lack of transparency into what it's doing, and the closed-sourceness, I wasn't keen. So I looked into other stuff.

Duplicity: Obsolete

Duplicity bills itself as "encrypted bandwidth-efficient backup using the rsync algorithm". It's webpage quaintly declares a "A standards (CSS) compliant web browser will display this page correctly". It's command-line, it's mature.

It's format is open: full backups are tarballs, and incremental backups are tarfiles containing new files and diffs. It relies on librsync to find diffs.

Managing full and incremental backups are a pain. Long incremental chains make it slow to recover data, and prevent garbage-collection of any file in the full backup (costing you extra storage).

To work around this, their advice is:

Keep the number of incrementals for a maximum of one month, then do a full backup.

But if I want a year's worth of backups, this means I'm redundantly storing 12x my dataset. That's going to cost me real money.

And rsync is fast if you have ssh access to the remote to run a rolling checksum there, but not so fast with modern cloud object storage where you don't have that. I'm not sure exactly if Duplicity downloads the full blob, but it certainly has to download the checksum header in the tar file. And with cloud object storage, that'll cost you exorbitant egress fees.

The full/incremental backup strategy is obsolete in 2023. Really, I'd argue it's been obsolete ever since Git popularised content-addressable-storage and immutable data structures plus garbage collection.

Duplicati: Unmaintained?

Duplicati is "Free backup software to store encrypted backups online". It's open source, written in C#, with a nice web interface. It looks really nice!

The system is completely open-source, even the web UI. C# is a language that inspires confidence. And the data format is modern, playing well with cloud object storage: garbage-collected content-addressed chunk storage.

But the last commit was merged over 6 months ago. It sounds like the maintainer doesn't have time, and there are insufficient volunteers. While I could volunteer, I already have open source commitments. I'd like a backup software that's actively developed, if only to get security fixes.

Duplicacy: Good enough?

There are too many backup tools that start with "Dupli-". It's very confusing.

Duplicacy says it "backs up your files to many cloud storages with client-side encryption and the highest level of deduplication". It's open-core, with an open-source free-for-personal-use command line tool, and a paid web UI to manage it. It's written in Go, which is probably the language that inspires the most confidence for me (they're probably not ignoring errors!).

The data format is very modern: content-addressed chunks. Just like Git. There's no need to read the chunk content back from the cloud to see what's inside it. And they have a clever two-step garbage collection process, which was neat enough that they got an IEEE Transactions on Cloud Computing paper out of it.

Duplicacy was able to max out my upload bandwidth. There's a way to install it on Synology. There are things I'm worried about: I was able to wedge the web UI behind a lock on first use, and leak goroutines (bug report), the developer has limited time between this and other projects, and email alerting seems limited or unimplemented. But it's seeing regular development (MR merged 2 weeks ago) and releases (last Dec 2022).

I'm impressed with the data format, and the open core means I'll be able to recover or move my data easily even if I don't pay for the web UI.

Duplicacy is the winner, for now. I forked out the money for a personal license for the web UI.

Software I haven't looked into

• Borg Backup looks like it has a modern data format, and is seeing active development. Although it doesn't natively back up to cloud, so would double my local disk space.
• Restic backup was recommended by a few people, and seems to see active development.
• Kopia backup is open-source, backs up to clouds, deduplicates, and has a GUI. Looks like a good option, but I found out about it too late.

Conclusion

If you're on a Mac or Windows, I recommend Arq. I don't recommend Backblaze.

If you're on Synology, I recommend Duplicacy. I don't recommend Synology Hyper Backup. Duplicity is obsolete. Duplicati's development has stalled.

All backup software has tradeoffs and bugs. I don't feel great about anything I've tried, really. I wonder if backup software is a Market for Lemons, where most people don't have to restore, and so they don't even know if their backups work, and the market sinks to dodgy players. That's another post, though.

Next in this series: What location should I backup to? I evaluate where cloud backup storage providers, and how to setup Duplicacy email notifications with Mailgun.

Reddit discussion of this post.

A few mornings ago, we woke up to the power out. The fuse had tripped, we flicked the switch to re-set it. We weren't sure what had tripped the fuse. My home electricity monitoring logged its last event at 02:38am, so the power must have failed just after.

Later, I tried to read some data off the Western Digital My Book external hard disk I'd plugged in the night before.

The drive had some data from a relative who passed away, so I was keen to read it. But the disk wasn't showing up on the computer any more. Why?

Is the power supply broken?

I couldn't hear the disk spin up, unless I plugged in another power supply. So I tested the power supply with a multimeter. It should output 12V but was outputting 0V:

Plug it into other machines?

I was hoping it was 'just the power supply' that shorted, that the disk was undamaged. However, with an alternative power supply, the disk still didn't mount. macOS said the disk was 'uninitialized' (unformatted), but further tries to read the first few bytes of the disk (where the partition table is stored) failed:

$ sudo cat /dev/disk4 | xxd | less
error: resource busy

That seems to say we can't even read any bytes off the disk at all! Plugging the disk into my NAS and running sudo dmesg -w, I see the kernel having errors reading the disk:

I hoped: maybe it's just the enclosure that's fried, and maybe the disk inside is still good? This was a last-resort because you have to snap the disk enclosure to open it up. So I tried plugging it into one more Linux machine, but I got much the same errors. This time, so many that buffer_io_error said 103 callbacks suppressed. I guess there were over 100 errors?

Bypass the USB controller?

I prised the case off with a screwdriver, breaking the clips:

And I found the internal disk. A cheap Western Digital disk inside an expensive Western Digital case, that's vertical integration!

I popped the raw disk into my Synology NAS. Its spare disk slot has been handy for reading many disks as I've been processing these inherited hard drives. Now Linux is talking directly to the disk, without a USB controller. Maybe it'll work?

The drive is throwing exceptions when we send read ATA commands to it, and Linux tries to reset the drive to fix it, but the reset doesn't fix it. Presumably the USB controller was trying to reset the drive too, and returning resource busy.

Conclusion

Unfortunately, I think this drive is toast. I e-wasted it, labelling it "shorted" with a sharpie, just in case any recycler tries to use it.

I failed to recover the data, but I hope that sharing the steps I followed might help others in this situation.

I'm not sure why the disk failed. The disk was maybe 10 years old. I'm not sure if drives can just 'short out' themselves? I hadn't heard of this failure mode.

The most fun explanation is: there were a few lightning storms in Sydney around the time this failed. Could it have been a lightning surge that ruined this? But no other electronics in the house seem to be broken. Is it really likely that lightning hit our power, and only one hard drive fails?

Next in this series: I evaluate cloud backup software options.

The Snoo is a fancy baby bassinet, that rocks babies to sleep, with motors that respond to baby's stirring by rocking the more. It's neat, although expensive. Unfortunately, the moving parts fail, and, shamefully, the manufacturer doesn't offer repairs, and encourages you to buy a new device instead.

To save money, we bought a third-hand Snoo online. It worked great, but just three weeks in, the motor started making grinding noises, and a few days later stopped moving altogether.

I was really worried, as our newborn was used to rocking to sleep. I searched for [Snoo Repair Sydney] and nothing came up. My options were pretty awful:

1. Replace it. Should I pay $2000 new to get a warranty? Or $900 second-hand and risk it breaking again? Either way: ouch.
2. Try to repair it myself: watch long YouTube videos and do multiple trips to Bunnings to get the right parts (all the while trying to take care of a newborn and fighting sleep deprivation). I would probably fail at this.

My partner (being much smarter than me) jumped on social media and found Ashleigh, aka "The Snoo Lady".

The very next day, Ashleigh visited us in our home, with a huge collection of Snoo parts (well beyond what I'd be able to source), and stripped, repaired, dusted, and thoroughly cleaned our bassinet. Unsurprisingly, the O-rings had failed (they crumbled to dust as she removed them). Ashleigh not only replaced the broken O-rings, but replaced all the other worn parts. She'd done this a lot of times before and did it very quickly.

I was very impressed. I'd strongly recommend her to anyone staring at a broken Snoo. She does a much more thorough job than you'll probably do yourself, will save you precious newborn time, and her repair price was extremely reasonable – if anything, I think she should raise her prices.

I'm writing this post because, today, when I Google for [Snoo Repair Sydney], nothing useful comes up. I didn't get any discount or anything for writing this recommendation. It's a disgrace that Snoo's manufacturer sells expensive devices with these known failure modes but doesn't offer repair. I hope this can help other parents can find repair help, and keep their devices out of landfill.