Paul Mucur

Weeknotes 71

2021-03-07T11:55:00+00:00

My Joy-Con diagnosis came back from Nintendo: “liquid damage main board”. Turns out C dunking it in a glass of water might not have been the best for its delicate electronics.
C surprised us one morning by saying “go go” when “Go Jetters” came on the TV during breakfast (after family favourite “Hey Duggee” but before family least-liked “Peter Rabbit”).

E and I now sing the theme song on demand.
I’ve been working on a brand new Rails application at work and it has been a long time since I had to build something customer-facing from scratch.

What test frameworks should I use? Should I still use Cucumber? What front-end framework should I be using? Should I theme Bootstrap? What Rubocop configuration should I use? Should I use Standard?

In the past, I’ve tried to reduce the amount of code I had to write (e.g. perhaps unthinkingly in the name of “Don’t repeat yourself”), extracting as much as possible so I could whip up resourceful controllers with the fewest lines of code. Now though, I’m less concerned about optimising for that initial writing and more concerned about making the code easier to read, change and delete.

The gamble I’m taking is that it might take a little longer to write at first but it’ll be much easier to understand and change in future.

Perhaps I’m getting more cynical about my ability to predict the right abstraction. As Sandi Metz said:

Prefer duplication over the wrong abstraction.

I’m reminded of Jay Fields’ “Working Effectively with Unit Tests”:

[Test] code and production code is written, maintained, and reviewed in drastically different ways. Production code collaborates to provide a single running application, and it’s generally wise to avoid duplicating concepts within that application. Tests do not, or at least should not collaborate; it’s universally accepted that inter-test dependency is an anti-pattern. If we think of tests as tiny, independent universes, then code that appears in one test should not necessarily be considered inadvisable duplication if it appears in another test as well.

I wonder if it is also useful to think of separate parts of your application as “tiny, independent universes” too.
I had to sign up for a Facebook account in order to fix a problem with Instagram embeds and, despite providing only my name and a work email address (which no one else would have in their contacts), Facebook suggested I add people I know in real life.

I was spooked until I realised Facebook use the phone number you give for 2-factor authentication to look you up in others’ contacts.

Weeknotes 70

2021-02-28T14:37:00+00:00

After having a drifting left Joy-Con for over a year, I finally sent it to Nintendo for repair.

With the Switch out of action, I signed up for a free trial of Apple Arcade and have so far downloaded “What the Golf?, “A Monster’s Expedition (Through Puzzling Exhibitions)” and “Sayonara Wild Hearts”.

I’ve only played “What the Golf?” so far and it is wonderful.
My mum and I took C to a nearby park to feed the ducks (oats, to avoid controversy) and saw a raft of glorious Mandarin ducks.

Along with the red kites we’ve seen in the garden, they certainly make a change from London’s feral parakeets.
I had to parse a CSV file sent as a Data URL this week and converting a Request for Comments into a regular expression is a great joy in life:
```
REGEXP = %r{
  data:
  (?<mediatype>
    (?<mimetype> .+? / .+? )?
    (?<parameters> ( ; .+? = .+? )* )
  )?
  (?<extension>;base64)?
  ,
  (?<data>.*)
}x.freeze
```
You can find a full Ruby implementation on GitHub.

I learnt about Active Record’s ability to rollback migrations using the revert method which came in very handy when amending a unique index:

require_relative '20210211104247_add_unique_index_to_product_links'

class ChangeUniqueIndexOnProductLinks < ActiveRecord::Migration[6.1]
  def change
    revert AddUniqueIndexToProductLinks

    add_index(:product_links, [:product_id, :reseller_id, :country_id], unique: true, name: 'by_product_reseller_country')
  end
end

An overgrown bush had begun to crowd the patio so my mum and I decided to cut it back. She disappeared into the garage and returned with a truly horrifying but brilliantly effective single-purpose tool: a telescopic branch cutter.
In the footsteps of other power couples, E and I went for our first run together yesterday. My legs now ache and, about 5 minutes into the run, my shorts nearly fell off.

Weeknotes 69

2021-02-21T10:21:00+00:00

I spent a long time rewriting re2’s CI configuration to test against Ruby versions 1.8 through to 3.0 and all six ABI versions of the underlying Google re2 library using GitHub Actions.

My previous setup, based on using the Brightbox Ruby packages for Ubuntu and the official Rubinius binary releases, stopped working when I ran it for the first time in ten months.

Thankfully, RSpec’s rspec-core also uses GitHub Actions to test against legacy Ruby versions and their custom Docker images were the basis for my own.

I hope that by switching to the official Ruby setup-ruby GitHub Action, it will prevent further bit rot but I suspect the necessary workarounds for legacy Ruby versions will only get worse over time (see Joshua Stein’s “Plaintext HTTP in a Modern World”).

I hoped this work would mean I could add TruffleRuby as a supported version of Ruby but alas, it doesn’t seem to work with Ubuntu on GitHub Actions.
We celebrated Pancake Day which doubles as the anniversary of C first eating solid food. While he demonstrated what difference a year makes, we introduced the concept of savoury pancakes to my sceptical parents.
I dropped the ball from my trackball onto the sheet of smoked glass covering my desk, accidentally recreating the Tesla Cybertruck bulletproof window demo.
I enjoyed sending my mum to the hardware shop to buy a specific adhesive to fix a broken window blind: “Sticks Like Sh*t”.

Weeknotes 68

2021-02-14T13:35:00+00:00

The kitchen tap has been dripping for a week.

The replacement cartridge valves we ordered arrived quickly but the 14mm socket needed to remove the existing cartridge was delayed by a week.

Impatient and sick of the constant drip-drip-drip of the tap, my mum and E separately went in search of the missing part, returning home with various bits and pieces from local hardware shops.

We have ended up with:
- A 14mm ½ inch drive socket.
- A 14mm ¼ inch drive socket.
- A set of hexagonal things I do not recognise but now think might be Allen keys for a socket set.
- A whole new socket set that goes up to 14mm but has sockets too shallow to remove a cartridge valve.
- A ½ inch drive ratchet handle.
The tap is now fixed.
Kent Beck famously said:

for each desired change, make the change easy (warning: this may be hard), then make the easy change

I would add the corollary:

for each desired DIY job, find the tools to make the job easy (warning: this may be hard), then do the easy job.
While hoarding the aforementioned tools, I was also attempting to complete multiple lengthy client questionnaires from our conveyancer, finish applying for probate, mend C’s bedroom door (which doesn’t fully close) and retrieve two table football balls from my dad’s home cinema subwoofer.

While completing the probate application with my mum, I realised I had fallen into a familiar trap: trying to be useful to those around by busying myself with odd jobs (both DIY and administrative).

In reality:
- I was stressing myself out.
- I was absent from the people I was trying (and failing) to help.
I’ve given up on the table football balls in the subwoofer.
Euan’s latest creation is absolutely wonderful.
Having taken responsibility for some projects written using Gatsby at work, you will now find me using the term “jamstack” in emails.

Weeknotes 67

2021-02-07T18:14:00+00:00

Following our decision, E and I signed terms with an estate agent, booked in a photographer and opened a file with a conveyancer. We left our flat in a hurry so we decided to return to prepare it for sale and bid it fare well.

With E as a temporary driver on my dad’s oversized car, we drove down and set to work removing all the clutter essential for living (particularly with a toddler) but unsightly to prospective buyers. E filled two suitcases with belongings as I pulled up weeds and removed an old planter full of dead flowers from our patio. It was a task that was literally years overdue.

I cannot unsee what I found on the underside of long-neglected bags of compost.

E ordered what may well be our last Yard Sale Pizza and I slept poorly, my mind racing with remaining jobs to do. By late morning, we’d done all we could, leaving a gooseneck kettle and a toaster on the kitchen worktop.
I’ve been helping apply for probate and, my goodness, for the sake of your surviving relatives: please maintain a single, up-to-date list of all your assets. Every time we think we might be getting somewhere, a slip of undated paper is found that throws everything into doubt.
During the latest attempt, I received a notification on my phone that my beverage had reached my “perfect temperature”. That’s right, I indulged in an Ember mug.

This was mostly due to Noah Kalina and Adam Lisagor’s review. As you can tell from James Hoffmann’s comments, this is obviously an absurd luxury item (let’s not even mention Rishi Sunak’s unfortunate photo op):

I’ll be honest, owning one of these will mean that your colleagues, friends and loved ones will mock you mercilessly for owning an incredibly expensive smart mug that holds coffee at a particular temperature of your choosing.

[…]

You should absolutely tease anyone who spent that much money on one of these… but they’re also having a kind of a good time with it.

Weeknotes 66

2021-01-31T15:35:00+00:00

My sister and I baked Liam Charles’ “Mega-Choccy Stack” from the latest series of Junior Bake Off. We didn’t have any double cream so I used a sachet of Bird’s Dream Topping instead.
I’ve created a Trello board called “Leaving London”.
I don’t know how to write about our decision to leave a city I have called home for 12 years.

Others have written about the peculiar feeling that your life is on pause but I hadn’t really thought about it until we started listing all the things we need to do to sell our flat.

At the time of writing my life is currently happening!

Perhaps I haven’t really believed this. That I could live five minutes at a time until “real life” could resume.

That, yes, we have outgrown our flat and, yes, want to be closer to family and, yes, we’re both working remotely and it seems to be going well. None of that ever connected with: right, let’s sell our flat, move our belongings and redirect our mail.

I originally moved to London on little more than a whim so perhaps I shouldn’t be surprised that leaving would be unceremonious too.

Weeknotes 65

2021-01-24T15:10:00+00:00

Taking inspiration from Rowan and Alice, I’ve separated my weeknotes (and solitary yearnotes) from my other, less frequent posts (e.g. “Using a Raspberry Pi for Time Machine”) on the homepage.

As well as the existing combined Atom feed, there are now separate feeds for posts and weeknotes if you only care to subscribe to one.
In preparation for a launch and an expected burst of traffic this week, I worked on automatically scaling a Rails application hosted on Heroku.

Heroku provide their own autoscaling but only for their “performance” tier of dynos which are 10 times the price of their standard ones.

Thankfully, they recommend third-party add-ons such as Rails Autoscale if you’re not using performance dynos or if your app has variable response times (as most do). I hadn’t heard of Rails Autoscale before but the demo was very compelling. Its use of Heroku “request queueing” time rather than response time makes it especially useful when your app might have the occasional spike in response time without impacting its ability to serve requests to other clients.

Adam McCrea’s “How Many Heroku Dynos Do You Need, and Which Size—An Opinionated Guide” is a fantastic guide and led me to look into “in-dyno concurrency” with multiple Puma workers rather than relying solely on threads.

However, as soon as I bumped the number of workers up for my application, a slow memory leak became much more pronounced with my app peaking at 131.5% of its memory quota.
Fortunately, I stumbled across Frederick Cheung’s “Debugging a Memory Leak in a Rails App” which not only described how to find memory leaks in running Rails applications using rbtrace and Sam Saffron’s excellent guide but described a potential memory leak in Rails’ prepend_view_path.

What did I find in a before_action in my Application Controller? Something like the following:
```
def prepend_subdomain_specific_templates
  prepend_view_path(Rails.root.join('app', 'views', request.subdomain))
end
```
As Frederick points out, prepending a String or Pathname to the view path causes Rails to initialize a new Action View Resolver for every request. This would be fine as Ruby should garbage collect the Resolvers when they are no longer used but Rails’ template caching means that each Resolver defines methods whose names are never collected, causing a slow but steady leak. If you’re calling prepend_view_path on a particularly heavily trafficked route, this leak will become more pronounced.

The fix in my situation was to initialize a Resolver per subdomain once and then re-use the same Resolver whenever possible, e.g.
```
SUBDOMAIN_SPECIFIC_TEMPLATES = {
  'subdomain1' => ActionView::OptimizedFileSystemResolver.new('app/views/subdomain1'),
  'subdomain2' => ActionView::OptimizedFileSystemResolver.new('app/views/subdomain2'),
  'subdomain3' => ActionView::OptimizedFileSystemResolver.new('app/views/subdomain3')
}.freeze

def prepend_subdomain_specific_templates
  prepend_view_path(SUBDOMAIN_SPECIFIC_TEMPLATES.fetch(request.subdomain))
end
```
With the fix deployed, the resulting memory usage graph made me very happy indeed.
I’ve been rightly chastised for using the trope of interpreting children’s TV shows through the eyes of a jaded adult before but I’m genuinely enjoying “In the Night Garden…”.

Watching C’s excitement build as we watch Igglepiggle lie down in his boat, covering himself with its sail as we pan up from the rolling ocean to see the stars of the night sky bloom into flowers is a delight.

The creators’ intentions behind the series is interesting too:

“We became very aware of the anxiety surrounding the care of young children which manifested itself in all kind of directions – but the one big subject that came up again and again was bedtime. It’s the classic time for tension between children who want to stay up and parents who want them to go to bed.

“We wanted to explore the difference between being asleep and being awake from a child’s point of view: the difference between closing your eyes and pretending to be asleep and closing your eyes and sleeping.

“So this is a programme about calming things down whereas most children’s TV is about gee-ing everything up!”

Weeknotes 64

2021-01-17T11:29:00+00:00

After reading Duncan Jones’ script for “Full Throttle”, I replayed the original 1995 LucasArts adventure game using ScummVM having copied the files from my original CD-ROMs.

I wanted to do the same with Broken Sword 1 and 2 but struggled to find a computer in the house with a disc drive. I ended up using my grandad’s abandoned Asus laptop and SMB file sharing to get the files onto my Mac.
I’ve been thoroughly enjoying Brian David Gilbert’s “Unraveled” series for Polygon (now complete), e.g. “Which Dark Souls boss is the best manager?”
Kushal’s tweet introduced me to the term “doomscrolling” and, perhaps it is the Baader-Meinhof phenomenon but I find myself doing it a lot.

I needed to extract a custom WordPress theme from a subdirectory in a git repository while preserving its git history. I first tried using git filter-branch but it told me to use git filter-repo instead.

I was able to extract the subdirectory as well as some top-level files and rewrite some hard-coded paths all with a single command:

$ git filter-repo --path .stylelintrc --path .stylelintignore --path .phpcs.xml --path package.json --path package-lock.json --path gulpfile.js --path .browserslistrc --path .eslintrc --path .prettierrc --path wp-content/themes/custom-theme/ --path assets/ --path-rename wp-content/themes/custom-theme/: --path-rename assets/: --replace-text ../replacements.txt

We got our very first robot vacuum cleaner and watching it navigate takes me back to 2004 and my adventures programming LEGO MINDSTORMS with Not Quite C.
I finally managed to solve my photo book and iCloud Photos syncing problems:
1. I restored my working (but outdated) photo library from a backup.
2. I logged out of iCloud altogether on my Mac.
3. I logged back into iCloud.
4. I tried to enable iCloud Photos.
5. I encountered a baffling error that “iCloud Photos is only available on macOS Standard, macOS Extended and APFS formatted volumes”.
6. I restarted into macOS Recovery.
7. I ran First Aid on my Big Sur volumes.
8. I restarted.
9. I successfully enabled iCloud Photos, accepting its warning that I might run out of storage and watched as it began to re-upload all ~37,000 photos and videos in my library.
10. Days passed.
11. The update completed!
I now have my latest photos and my Mimeo Photos project is intact.
I’ve been reading Philip Pullman’s “Northern Lights” (after enjoying the BBC TV adaptation) and this quote from Roger felt a bit close to home:

There en’t been nothing good more than about five minutes ahead. Like I can see now, this bath’s nice, and there’s a nice warm towel there, about five minutes away. And once I’m dry, maybe I’ll think of summing nice to eat, but no further ahead than that. And when I’ve eaten, maybe I’ll look forward to a kip in a comfortable bed. But after that, I dunno, Lyra. There’s been terrible things we seen, en’t there? And more a coming, more’n likely. So I think I’d rather not know what’s in the future. I’ll stick to the present.

Weeknotes 63

2021-01-10T15:16:00+00:00

I haven’t seen snow like this in years.
I’m going to try to keep this quick as I want to try out my new running shoes before C wakes up. The feeling of having eaten too much at every meal is becoming far too familiar.
C has various place mats, posters and books that all feature cupcakes for the letter C. In an act of pointless defiance, I am insisting on calling them “buns” instead.
My parents’ house is shaped like one long corridor so I bought them a BT Whole Home Wi-Fi system a few years ago. Now there are three of us working from home (not to mention my dad’s binge-watching of any Netflix show that has more than three seasons), we’re especially vulnerable to any problems with the internet.

The day after upgrading their router to the hub BT insisted on sending them, we started to have problems. I spent that evening meticulously re-arranging the various nodes in the mesh network, trying to ensure a reading of “Excellent Connection” for each one (with mixed success).

The very next day at 8:30 am, all of our internet cut out unexpectedly.

C had pushed the router off a side table, turning the whole thing off.
For Christmas, I put together and printed a photo album of C’s 2020 using Mimeo Photos as they used to be the provider of Apple’s official photo books.

I’ve been very happy with the result but during my upgrade to Big Sur, I downloaded my photo library afresh from iCloud Photo Library after experiencing syncing problems with a restored backup. I clicked on my Mimeo project before all my original photos had finished downloading and it threw a “Some Photos Are Missing” warning, immediately gutting the project of all its photos.

Thankfully, I still had a backup of my entire photo library including the full project and, sure enough, loading that into Photos.app worked fine but there was seemingly no way to transfer that project over to my new library.

This led me to take the following steps:
1. Disable iCloud Photos.
2. Drag my entire photo library to the Bin.
3. Restore my working (but now out-of-date) photo library from my backup.
4. Without opening my Mimeo project, delete the entire contents of my library.
5. Re-enable iCloud Photos so that my library is re-downloaded, hopefully fixing my project.
Step 4 took over three hours to complete.

It didn’t work.

Weeknotes 62

2021-01-03T16:55:00+00:00

I built a snowman.
It’s been great to read people’s yearnotes.

This time last year I was still on “parental leave” and anxious about finding a new client. By the beginning of March, I was ready to spend most of the year commuting over two hours a day to west London. In the end, I only spent one week doing that commute before everyone was working remotely. It meant I’ve been able to spend much more time with C than I might have hoped but I still can’t reconcile “C the baby” with “C the little boy who loves to climb stairs”.
It’s the end of my second year working for myself and I’ve been exceptionally lucky enough to work with some great clients.

I have even less of an idea than usual about what the future might hold so I’m extremely grateful to have the flexibility to work from home. It’s even more appealing now that I finally have a desk with a window after working in an unlit hallway for nine months.
On top of an unusually sedentary year, Christmas and the Accidental Tech Podcast review of Apple Fitness+ have pushed me further into reconsidering my exercise regime (which currently consists of nothing).

I’ve ordered Ring Fit Adventure and a new pair of running shoes in an attempt to cajole me into some form of rigorous movement at least once a day.
For a friend’s birthday on New Year’s Day, E and I took part in an online escape room over Jitsi.

There were four of us playing and a single “games master” (no, not that one) who acted as a sort of human text adventure game: they described the scene and responded to our commands to look at things, use objects, etc. As we slowly solved puzzles and revealed things in the scene, they presented an updating top-down view of the room.

It was a surprising amount of fun though it was difficult to coordinate our decision what to do next. At one point, we freed a character trapped in a cupboard and my friend instructed the games master to remove all of their teeth with dental instruments.
While working with Create React App, I noticed it uses the prefers-reduced-motion CSS media feature.

Some may argue the rotating eyes on my company logo are vital but I thought it best to disable them if someone has requested that non-essential motion is disabled.
```
@media (prefers-reduced-motion: no-preference) {
    .eye {
        animation: playing 23s linear infinite;
    }
}
```
This way, the eyes only rotate if you haven’t explicitly asked for reduced motion, i.e. if prefers-reduced-motion is set to reduce, the animation property isn’t set at all.
As it is a new year and I recently formatted my laptop, it’s time for a new Terminal colour scheme! Having used Solarized and Nord in the past, Leo introduced me to Base16.

It took me a while to understand that I had to combine both Base16 Shell (to set my shell colours) and Base16 Vim (to use those colours in Vim).

With both installed and configured, I could run the following in the shell and have both the Terminal and vim update:
```
$ base16_tomorrow
```
However, after having pointlessly sped up my zsh startup time over a year ago, I am now avoiding Base16 Shell’s profile_helper.sh and instead use the following in my .zshrc (where .base16_theme is a symlink to ~/.config/base16-shell/scripts/base16-tomorrow.sh):
```
source ~/.base16_theme
```
And the following in my .vimrc:
```
let base16colorspace=256
colorscheme base16-tomorrow
```
If I want to choose a different theme, I need only update the .base16_theme symlink and the colorscheme in my .vimrc.
Happy new year!

Weeknotes 61

2020-12-28T13:33:00+00:00

As 2020 draws to a close, I performed the annual cleansing ritual of formatting my laptop and upgrading to macOS Big Sur.

Following Tom’s recommendation, I bought a Samsung T5 500GB portable SSD and used SuperDuper! to take a copy of my entire hard drive. I then upgraded to Big Sur, restarted my Mac into macOS Recovery, used Disk Utility to erase my internal hard drive and reinstall macOS from scratch.

While I don’t have an automation tool to my name, almost everything I need to set up my computer can be found in my dotfiles (aside from my private SSH and GPG keys). After copying over my music and photo libraries, I was back up and running within an hour.
We enjoyed Christmas day with the family with its equal parts of rich food and busying myself with the construction of various toys given to C (e.g. a tricycle and a repair garage with 32 pieces of track).
While we’ve been staying with my parents, C has started saying words. His first was “apple” and his vocabulary has now expanded to include “ball” and “dada” which, confusingly, is his attempt at “grandad”.
C was particularly fond of a Hey Duggee wooden puzzle from my parents but managed to break it after dropping it only once.

From the outside, it seemed the circuit board attached to the inside of the puzzle had simply come loose but, as the entire unit was glued shut, I had to take a handsaw to it to investigate further.

After much sawing and clumsy prying with various screwdrivers, I managed to get inside to discover the integrated circuit that powered the device had detached as it was stuck in a blob of glue.

After asking Leo for advice, he encouraged me to look up surface-mount technology (SMT) soldering and give it a go. Calling upon my father-in-law’s soldering iron once more, we gave it our best shot and got it (mostly) working! Only one of the characters (sorry, Tag) no longer speaks when you fit their piece in the puzzle but five out of six isn’t bad.
Leo has also been teaching me Go as I’ve been working on a custom HTTP reverse proxy to complete requests on behalf of a desktop application whose HTTP client has not been configured to follow redirects.

Despite having done the excellent A Tour of Go, I find that new languages only really stick when I try to ship my own project.

Writing automated tests using the httptest package was great fun and Leo’s recommendation to watch Mitchell Hashimoto’s “Advanced Testing with Go”, focussing on subtests and table driven tests was invaluable.

It was also useful to learn about the standard Go project layout even though my project is so small I’ve stuck everything in main.go.
I’ve been working my way through various Christmas specials: Mortimer & Whitehouse: Gone Christmas Fishing, Motherland: Christmas Special, The Great British Sewing Bee: Celebrity Christmas Special and, of course, The Great Christmas Bake Off 2020.

I’ve still got The Repair Shop at Christmas and the new episode of Worzel Gummidge to go.

Weeknotes 59 and 60

2020-12-21T12:10:00+00:00

After last week, we decided to decamp to my parents’ house for the foreseeable future.

The day after we made the decision, C came home from nursery with a cold and E caught it soon after. We waited for it to pass but, by Friday, our anxiety led to a last minute coronavirus test and a nervous two day wait for the result. Just as we were about to give up travelling that weekend, the text message arrived: a negative result.

Within seconds, E ordered our first (and perhaps only) Pret of 2020 while we loaded up the car. Complaining about the inexplicable popularity of cranberries in every festive sandwich, we drove north without stopping, listening to Adam Buxton interview Sir Paul McCartney along the way.

I’m so glad we did.
With my parents available to look after C, I spent the day I usually look after him alone constructing a climbing frame.

The instructions said it would take two people four hours to complete. It took me around seven hours to screw, bolt, drill and hammer together the finished product.

It’s no 1960s cocktail cabinet but it is the most complicated thing I have built.
During C’s giddy exploration of his new surroundings, he threw my parents’ 23 year old telephone base station on their kitchen tiles. It immediately stopped working, rendering all of their handsets useless.

Upon inspection, we realised the station had landed on its own cable, severing two of four wires within.

After speaking with my father-in-law, he lent me his soldering iron and some solder sleeves and I attempted to reattach each of the four wires.

I have never successfully repaired anything by soldering but this was my very first success!

I used Rake’s multitask for the first time:

Same as a regular task, but the immediate prerequisites are done in parallel using Ruby threads.

desc 'Run all migrations in parallel'
multitask migrate: %i[migrate_issues migrate_books migrate_magazines]

desc 'Migrate issues'
task migrate_issues: :environment do
  # ...
end

# ...

My adventures with Rails’ Active Storage continue and the use of direct routes stands out while reading the source code. This allows you to create powerful routing helpers that take arbitrary objects (rather than relying on overriding ActiveRecord::Base#to_param):

get '/downloads/:signed_id/*filename' => 'downloads#show', as: :download_blob

direct :download do |object, options|
  signed_id = object.some.custom.signed_id
  filename = object.another.custom.filename

  route_for(:download_blob, signed_id, filename, options)
end

# So now you can call..

download_path(object)
download_url(object)

While I’ve since given up on Active Storage’s public access mode as it doesn’t let you set a Content-Disposition with S3, trying to make it work led me to discover Aws::S3::Object#public_url doesn’t respect the AWS client’s :use_accelerate_endpoint option to enable Amazon S3 Transfer Acceleration.

You can work around this by instead setting a custom :endpoint in your client configuration:
```
amazon:
  service: S3
  endpoint: "https://s3-accelerate.amazonaws.com"
```
I also disappeared down a bit of a rabbit-hole trying to debug why an Active Storage Variant in my tests wasn’t being detected as processed.

After much use of Pry, I discovered the bug was being caused by the fact Variant keys are generated by ActiveSupport::MessageVerifier#generate which uses Marshal.dump to serialize a Variant’s transformations and, while my test variant seemed identical to the expected variant, there was a very subtle difference: the filename of the test variant was encoded with ASCII, not UTF-8:
```
> Marshal.dump(variant1.variation.transformations)
"\x04\b{\b:\vformatI\"\bpng\x06:\x06ET:\x12resize_to_fit[\a0i\x01\x80:\fconvertI\"\bjpg\x06;\x06T"
> Marshal.dump(variant2.variation.transformations)
"\x04\b{\b:\vformatI\"\bpng\x06:\x06EF:\x12resize_to_fit[\a0i\x01\x80:\fconvertI\"\bjpg\x06;\x06T"
```
The difference can be hard to spot: it’s the \x06ET after png instead of \x06EF.

This stems from the encoding of Rails.root being ASCII (which is due to the underlying behaviour of Pathname) and Rack::Test::UploadedFile using that filename to determine the original_filename.

Passing an explicit :original_filename fixes this:
```
Rack::Test::UploadedFile.new(
  Rails.root.join('spec/fixtures/image.png'),
  original_filename: 'image.png'
)
```
I warned you it was a rabbit-hole.

Weeknotes 58

2020-12-06T14:22:00+00:00

While preparing some chocolate chip cookies, E asked, “there’s one left from the last batch, shall I KitKat it?”
I completed my first successful escape attempt from Hades thanks to repeatedly watching Haelian’s videos, particularly the top 6 easiest builds.
After receiving some bad news, I went to make bread and discovered our kitchen scales didn’t work. Sitting on a flat worktop with nothing on the pan, the reading jumped around of its own accord.

With a ratchet screwdriver in one hand and some WD-40® Specialist® Fast Drying Contact Cleaner in the other, I took the whole thing apart.

Prying up the circuit board with a plectrum and discovering water pooled underneath the screen, I thought about my 92 year old grandfather who was the subject of the morning’s bad news. Taken ill in hospital, my occasionally overzealous sense of DIY comes entirely from him.

Always one to be fixing or building something throughout my childhood (or, on one occasion, inexplicably sawing the base off a beloved bin of my mother’s), he would invite me to help with whatever contraption he was working on. Even in his eighties, he drilled a hole in his walking stick so he could thread a loop through and secure it to his wrist, preventing it from ever falling from his grasp.
After rain broke our car gate again, I bought myself a four-way utility key which could lock and unlock the gate arms properly (rather than struggling with a pair of needle nose pliers). There’s something strangely satisfying about having a very specific tool for a job.

This might be why I own a spanner designed specifically for removing bath taps.
I finally had justification to migrate from Thoughtbot’s long-deprecated Paperclip to Rails’ Active Storage for storing uploads in a web application.

While Thoughtbot maintain a migration guide, I wanted to make sure I really understood the changes required before making them. Especially since I’m using Amazon S3 and Paperclip has a bug that can truncate your uploads when using S3.

While Paperclip concerns itself with processing uploads and storing them on disk or in a service like S3, it expects you to handle serving them by storing them somewhere publicly accessible (e.g. by storing files with a public-read ACL by default). As such, all my current uploads are stored in a public S3 bucket and I serve them via a CloudFront distribution.

This notion of both storage and serving are entwined: uploads are stored with relatively human-readable paths, e.g. /system/books/full_pdfs/00/00/01/original/MyPDF.pdf, and are used to render any links, images, etc.

Looking at Active Storage, I was initially confused not to see any support for serving uploads via a CDN. When doing some test uploads to my existing S3 bucket, I was even more confused to see that a file like MyPDF.pdf was uploaded to the top-level of the bucket with a base58 filename like 77TMHrHJFvFDwodq8w7Ev2m7.

This is because I was still thinking of storage and serving as being one and the same. Diving into how Active Storage handles linking to uploads, things started making a little more sense.

Instead of serving uploads using their “real” location in your storage service, e.g. a random key in an S3 bucket, Active Storage serves all uploads through its own controller with its own URLs. This way, the URL can contain the original filename, e.g. /rails/blobs/lengthy-signed-id-here/MyPDF.pdf and generate short-lived, signed URLs to uploads in an otherwise private S3 bucket. What’s more, the implementation of this controller for Rails 6.0 (and its replacement in the upcoming Rails 6.1) is fantastically simple, effectively boiling down to:
```
def show
  @blob = ActiveStorage::Blob.find_signed!(params[:signed_id])

  expires_in ActiveStorage.service_urls_expire_in
  redirect_to @blob.service_url(disposition: params[:disposition]) # or @blob.url(...) for Rails 6.1
end
```
The official documentation explains this design decision:

Upon access, a redirect to the actual service endpoint is returned. This indirection decouples the public URL from the actual one, and allows, for example, mirroring attachments in different services for high-availability.

This decoupling means you rely on Rails to maintain your public URLs rather than using the underlying storage (as they rely on signed IDs to records in your database) but affords you the flexibility of moving storage services.

The difficult bit now is figuring out how to migrate all my existing Paperclip attachments to Active Storage without anyone noticing.

One useful thing I’ve discovered is that it is relatively easy to get a reference to a Paperclip attachment for a record even once you’ve replaced your call to has_attached_file with has_one_attached:
```
class Book < ApplicationRecord
  has_one_attached :full_pdf # used to be has_attached_file :full_pdf
end

book = Book.first
book.full_pdf
# => #<ActiveStorage::Attached::One:0xfeedface>

paperclip_attachment = Paperclip::Attachment.new(:full_pdf, book)
# => #<Paperclip::Attachment:0xdecafbad>
```
You can then convert from a Paperclip attachment to an Active Storage one (albeit by downloading and re-uploading the original file) like so:
```
book.full_pdf.attach(
  io: URI.open(paperclip_attachment.url),
  filename: paperclip_attachment.original_filename,
  content_type: paperclip_attachment.content_type
)
```
My hope is that I can write a custom Rake task that will copy all existing attachments to Active Storage before actually switching the application code over to using has_one_attached so that the database and S3 bucket are fully populated before the switch, e.g. something like the following:
```
# The real implementation still uses Paperclip so override for the purposes of this task
Book = Class.new(ApplicationRecord) { has_one_attached :full_pdf }

Book.find_each do |book|
  paperclip_attachment = Paperclip::Attachment.new(:full_pdf, book)
  next unless paperclip_attachment.exists?

  book.full_pdf.attach(
    io: URI.open(paperclip_attachment.url),
    filename: paperclip_attachment.original_filename,
    content_type: paperclip_attachment.content_type
  )
end
```
On a related note, I was wondering how easy it would be to run your own S3-compatible storage service given that quite a few competing services offer interoperability. This led me to MinIO which is an open source tool you can point at a location on disk and then interact with it via the AWS CLI.
Despite writing weeknotes for over a year, I only just discovered that Jekyll has “Live Reload” and “Open URL” serve command options. No more will I manually open my browser to localhost:4000 or repeatedly mash ⌘R:
```
$ jekyll s -lo
```

Weeknotes 56 and 57

2020-11-28T13:47:00+00:00

It has been a trying couple of weeks so we have been in need of uplifting television shows.

Our current rotation is “Schitt’s Creek”, “Ted Lasso”, “Nadiya Bakes” and “Nigella’s Cook, Eat, Repeat”.

The latter two are a bit heavy-handed on the “everything’s alright, everything’s alright, it’s OK, it’s fine” but I don’t mind.
In an attempt to break the Groundhog Day of cooking at home and re-introduce some semblance of a date night, E and I ordered a meal kit from Perilla. It met our hopes of providing a change from something we would typically make ourselves without being a chore to prepare: the main course came in two vacuum sealed bags and only needed boiling in water for 12 minutes.

We sat at our dining table for the first time in months and realised how much we needed a break from our daily routine.
I got a Raspberry Pi 400 and was pleasantly surprised to find a print edition of the Raspberry Pi Beginner’s Guide in the box. Leafing through it while booting the Pi 400 for the first time, I pictured myself at 8 years old opening this on Christmas Day.
I’ve had my Twitter account for 14 years, initially only using it to send free group text messages. I am not the most famous “mudge” and am constantly mistaken for Peiter “Mudge” Zatko, so much so that I decided to “protect” my account a few weeks ago.

I did so in the nick of time as Zatko is the new Head of Security for Twitter.

Alas, now I’m receiving a whole new type of misdirected tweet: support and feature requests for Twitter itself.
When looking after C, one of my favourite things to do is build him the most elaborate tower out of DUPLO possible. I’ll incorporate bogies, animal heads and even the occasional submarine.

As I know he will eventually destroy it, it has become a sort of mandala.
In versions of Mozilla Firefox prior to 61.0, requests made with the Fetch API won’t send cookies unless you specify the credentials option.

I ran into this when Rails’ cross-site request forgery countermeasures prevented my Stripe Checkout integration from working properly in Firefox 52. Switching to the new default value of same-origin for credentials meant the Rails session cookie was correctly sent and the CSRF forgery protection would permit the request:
```
fetch("/some-endpoint", {
  method: "POST",
  credentials: "same-origin",
  headers: {
    "Content-Type": "application/json",
    "X-CSRF-Token": csrfToken,
  },
  // ...
})
```
Concerned there were other bugs I had missed, I started using React’s Strict Mode to perform stricter checks in development. In particular, I wanted to see how my components would behave when intentionally double-rendered.
I added JavaScript error monitoring with Sentry which caused a 493,803% increase in “events” since the previous week when I was only monitoring Ruby errors.

Specifically, I started to see a lot of errors from a window.fetch polyfill bundled with the Shopify BuyButton.js but almost all of them came from a download page.

This page shows a simple “Thank you” message to the user and immediately redirects them to a PDF using a <meta http-equiv="refresh">. When trying to reproduce the error in Safari, I noticed a bunch of different errors all due to the fact the redirect effectively cancels any JavaScript in progress.

To work around this, I copied a technique used by SourceForge and Audible and used an invisible <iframe> with a src pointing to a file with a Content-Disposition of attachment, e.g.
```
<iframe src="https://example.com/some.pdf" class="d-none"></iframe>
```
This immediately causes the browser to start downloading the PDF without cancelling any JavaScript running on the page.
I’ve been working on a web service to record anonymous usage statistics of a desktop application. A key goal of this project is to avoid storing personal data and exemplify data minimisation.

In looking for other applications that do a similar thing, I dug into the anonymous aggregate user behaviour analytics collected by Homebrew which uses curl to report events to the Google Analytics Measurement Protocol.

Reporting events to a third party over HTTP opens a whole can of worms, particularly as IP addresses alone can be considered personal data. As our reporting needs are very simple, I decided to implement my own service using Rack and Redis.

Specifically, requests to the service are stored as daily counts in Sorted Sets that expire after 90 days like so:
```
ZADD choice:2020-11-28 INCR 1 "Some choice"
EXPIRE choice:2020-11-28 7776000
```
This way, we can see what the top ten most popular choices for a particular day are with ZREVRANGE like so:
```
ZREVRANGE choice:2020-11-28 0 10 WITHSCORES
```
If we want to look at choices over a longer time period, we use ZUNIONSTORE to aggregate each of the dates in the range:
```
ZUNIONSTORE choice:past_week 8 choice:2020-11-20 choice:2020-11-21 choice:2020-11-22 choice:2020-11-23 choice:2020-11-24 choice:2020-11-25 choice:2020-11-26 choice:2020-11-27
```
Then we use ZREVRANGE again to get the results:
```
ZREVRANGE choice:past_week 0 10 WITHSCORES
```
Despite best intentions, it is very easy to accidentally store personal data without realising it. For example, even if you don’t log requests to your service in your application code, Heroku will log the IP addresses of every request made to your app, retaining the most recent 1,500 lines of logs for one week.
I enjoyed Rowan Manning’s “Yeehaw!” about attempting side projects and tip #5 “When the Fun Stops, Stop” feels especially relevant when working alone. I’ve had several situations recently where I find myself down some blind alley of refactoring and have to force myself to git restore . and refocus my efforts.
C is unhappily teething but please try not to think about how every child’s jaws are packed with teeth.

Weeknotes 55

2020-11-15T10:52:00+00:00

This week, I’ve been trying to make Shopify BuyButton.js play nicely with Turbolinks.

Integration with Turbolinks is often tricky because it retains the state of window and document across page changes and any other JavaScript objects will stay in memory. In the case of BuyButton.js, this means any UI state and bound event listeners will persist as a user clicks around your site. This is further complicated by Turbolinks’ caching which means any transformations may be applied multiple times.

If you don’t take Turbolinks into consideration, you might see broken behaviour such as clicks on “Add to cart” not working, the entire UI failing to appear or multiple buy buttons appearing every time you use the browser’s back and forward buttons.

While I’m still working on this, I’ve had some success with the following strategy:
1. Create a Shop client once to be re-used across all pages.
```
const client = ShopifyBuy.buildClient({
  domain: "my-shop.myshopify.com",
  storefrontAccessToken: "your-storefront-access-token",
});
```
2. Initialize the library once to be re-used across all pages.
```
const ui = ShopifyBuy.UI.init(client);
```
  This will append a <style> tag to the <head> of your document and add various event listeners to document. As the <head> of your page will be merged when a user navigates between pages this will retain the styles and it is best practice to only bind event listeners once on document.
3. On turbolinks:load, reset the inner state of the library using destroyComponent as you can no longer guarantee the various DOM elements the UI will have injected into your <body> are still present.
```
ui.destroyComponent("product");
ui.destroyComponent("cart");
ui.destroyComponent("collection");
ui.destroyComponent("productSet");
ui.destroyComponent("modal");
ui.destroyComponent("toggle");
```
  Update: In writing this, I discovered destroyComponent doesn’t work if you don’t pass the ID of a component’s model. It uses Array.prototype.splice() to mutate an array while looping over it. This means the position of each component in the array is changing as the loop advances so some components will not be destroyed, e.g.
```
> const foo = [1, 2, 3, 4];
> foo.forEach((e, index) => foo.splice(index, 1));
> foo
[ 2, 4 ]
```
  We can fix this by destroying and resetting the components ourselves:
```
ui.components.product.forEach((component) => component.destroy());
ui.components.cart.forEach((component) => component.destroy());
ui.components.collection.forEach((component) => component.destroy());
ui.components.productSet.forEach((component) => component.destroy());
ui.components.modal.forEach((component) => component.destroy());
ui.components.toggle.forEach((component) => component.destroy());
ui.components.product = [];
ui.components.cart = [];
ui.components.collection = [];
ui.components.productSet = [];
ui.components.modal = [];
ui.components.toggle = [];
```
  Or, more concisely:
```
Object.keys(ui.components).forEach((type) => {
  ui.components[type].forEach((component) => component.destroy());
  ui.components[type] = [];
});
```
  We can do this when the turbolinks:before-cache event fires as recommended in the documentation:
  
  You can use this event to reset forms, collapse expanded UI elements, or tear down any third-party widgets so the page is ready to be displayed again.
  
  This will ensure any DOM elements injected by BuyButton.js won’t be cached so you don’t end up with duplicate “Add to cart” buttons that don’t do anything when clicked.
```
 document.addEventListener("turbolinks:before-cache", () => {
   Object.keys(ui.components).forEach((type) => {
     ui.components[type].forEach((component) => component.destroy());
     ui.components[type] = [];
   });
 });
```
4. Following that, create your components as normal.
```
ui.createComponent("product", {
  handle: "some-product-handle",
  node: element,
});
```
  To make it easy for Turbolinks to create the appropriate components based on the <body> contents for every turbolinks:load, I embedded the necessary information in data attributes.
```
document.querySelectorAll("[data-shopify-product-handle]").forEach((element) => {
  const { dataset: { shopifyProductHandle } } = element;
  const temporaryWrapper = document.createElement("div");

  element.appendChild(temporaryWrapper);

  ui.createComponent("product", {
    handle: shopifyProductHandle,
    node: temporaryWrapper,
  });
});
```
```
<div data-shopify-product-handle="some-product-handle"></div>
```
  I create a new <div> for the component to use as calling destroy will remove the node from the DOM.
~~I still need to iron out some issues with Turbolinks caching but I’m optimistic.~~ Update: with the update to use turbolinks:before-cache above, all my caching issues are now resolved.
On Tuesday, Apple announced the M1 system on a chip. While I’m not in a hurry to replace my beloved refurbished 2015 15-inch MacBook Pro, the pandemic and my musculoskeletal woes have made my working setup decidedly stationary. I’m seriously considering the Mac mini for the first time.

I’ve also been working on an integration with Stripe and have been amazed how good their developer tools are. From the ability to test webhooks with their CLI, their go-live checklist and extensive support for testing, it has been surprisingly pleasant to work with.

If you need to test handling their webhooks but are verifying their signatures, I found the following useful in my controller specs (assuming you’re storing your signing secret in Rails.configuration.stripe_webhook_signing_secret):

def sign_stripe_webhook(payload)
  timestamp = Time.zone.now
  signature = Stripe::Webhook::Signature.compute_signature(
    timestamp,
    payload.to_json,
    Rails.configuration.stripe_webhook_signing_secret
  )

  request.headers['Stripe-Signature'] = Stripe::Webhook::Signature.generate_header(timestamp, signature)
end

Then I can use it like so:

it 'returns a 200 OK with a valid Stripe event' do
  event = {
    'id' => 'evt_1',
    'object' => 'event',
    'type' => 'checkout.session.completed',
    'data' => {}
  }

  sign_stripe_webhook(event)

  post :create, params: event, as: :json

  expect(response).to have_http_status(:ok)
end

Following last week’s adventures with brew bundle and Scott’s tip about Homebrew Bundle being integrated with mas, I committed my Brewfile to my dotfiles.
The days of spring will surely bring the birds and bees cavorting.

But since I am a gentleman, I’d much rather be jorting.

Weeknotes 54

2020-11-08T14:12:00+00:00

I worked on two projects with Chris for Go Free Range, the first of which involved working with a React form to record numbers given by the user. We already had some <select> elements to do this, e.g.
```
const Form = () => {
  const [unit, setUnit] = useState(1);

  const handleUnitChange = ({ target: { value }}) => {
    setUnit(parseFloat(value));
  };

  return (
    <select value={unit} onChange={handleUnitChange}>
      <option value="1">m</option>
      <option value="100">cm</option>
      <option value="1000">mm</option>
    </select>
  );
};
```
This works fine. React is able to interpret our numeric value for unit and select the appropriate <option> in our dropdown. When the user changes their selection, the corresponding value is passed to handleUnitChange and stored as a floating point number. This way, we keep our unit as a number throughout so it is ready to be passed to other functions without any coercion.

However, if we try to do the same thing with an <input type="text">, we quickly run into trouble:
```
const Form = () => {
  const [unit, setUnit] = useState(1);

  const handleUnitChange = ({ target: { value }}) => {
    setUnit(parseFloat(value));
  };

  return (
    <input type="text" value={unit} onChange={handleUnitChange} />
  );
};
```
What happens when the user tries to empty the field (e.g. perhaps to enter a new value from scratch)? handleUnitChange will fire with a value of an empty string:
```
> parseFloat("")
NaN
```
Perhaps then we try to avoid storing NaN and instead set unit to be undefined or null if it is falsy?
```
const handleUnitChange = ({ target: { value }}) => {
  setUnit(value ? parseFloat(value) : undefined);
  // or setUnit(value ? parseFloat(value) : null);
}
```
If we do this, when the user clears the field, React will log a warning that we’re switching a component from being controlled to being uncontrolled because setting the value of a field to undefined or null has a special meaning: it tells React this field is now uncontrolled.

Given it is totally valid for the user to enter an empty value, we realised we needed to separate our concerns: use strings for the input value but convert the value to a number when we pass it elsewhere in our application.
```
const Form = () => {
  const [unit, setUnit] = useState("1");

  const handleUnitChange = ({ target: { value }}) => {
    setUnit(value);
  };

  doSomeCalculation(parseFloat(unit));

  return (
    <input type="text" value={unit} onChange={handleUnitChange} />
  );
};

const doSomeCalculation = (unit) => {
  if (Number.isNaN(unit)) {
    return;
  }

  // ...
};
```
Using an <input type="number"> helps a little with this as it prevents non-numeric inputs firing the onChange event but it seemed cleaner to deal with the problem in our logic than rely on browser behaviour.
The second project Chris and I worked on involved a Python API written with Flask. Getting things running locally was quite a pain with my attempts to use pip to install NumPy ending in compilation errors. In the end, switching to the previous stable Python release of 3.8 (rather than the latest of 3.9) using pyenv solved my problems.

However, I was left with a bit of a mess of installed Homebrew formulae and wanted to clean things up. Thankfully, I found an answer on the macOS Super User with a great tip for removing unused Homebrew dependencies:
```
$ brew bundle dump
$ brew bundle --force cleanup
```
This will first dump all explicitly installed formulae and casks into a Brewfile (which you could then edit) and then uninstall anything no longer required.
I ended up in a similar mess after installing Xcode and brew doctor reported that I had out-of-date Command Line Tools.

Attempting to use xcode-select --install failed with a strange error about the software update not being found and Software Update itself couldn’t install the update properly.

It turns out that Xcode includes the Command Lines Tools itself even though I had a previous installation in /Library/Developers/CommandLineTools installed via xcode-select. I ended up completely removing both Xcode and the Command Line Tools and reinstalling the latter before Software Update could successfully pick up the right update.
Simon wrote a truly excellent blog post about Rails autoloading with Zeitwerk titled “Rails autoloading — now it works, and how!”. This is a follow-up to his 2013 post “Rails autoloading — how it works, and when it doesn’t” to which I often found myself referring.
Following last week’s need for earplugs, I lost some hearing in my left ear. I returned from a trip to the pharmacy with some urea hydrogen peroxide and watched the Great British Bake Off sideways with the deeply unsettling sensation of fizzing in my ear canal.

I woke in the small hours of Thursday morning to find the room uncontrollably spinning around me. I staggered to the bathroom where I discovered I could no longer get my eyes to focus straight ahead without them constantly drifting to my right. With E’s help, I managed to speak to a doctor on the phone and go in to our local GP’s surgery to see someone in person that morning.

After waiting outside, the doctor greeted me at the door wrapped in plastic, said that they needed to “fire [their] gun” (a thermometer) at me and instructed me not to touch anything as we walked to a consultation room.

Thankfully, it’s a viral ear infection that’ll clear up by itself in a matter of days.
Now that C is confident enough walking, he happily roams our flat looking for things to prod, pull or combine in some way. We’ve already lost three bowls to his curiosity and I’ve had to rewire one plug. It also means that I took the step of lowering the mattress in his cot now that he more closely resembles the IKEA cartoon for “toddler” than the one for “baby”.

Weeknotes 53

2020-11-01T14:35:00+00:00

A year ago—one week before C was born—wearing rubber gloves and a face mask, I pulled out all of our kitchen appliances and stuffed every crevice with steel wool. I covered the wool with electrical tape before carefully arranging boxes of mouse poison given to me by the council along the walls.

Last night, just as E and I were falling asleep, there was the sound of scratching mere metres from our heads. As soon as we moved to investigate, it stopped and I spent the rest of the night waiting to hear something else other than the pounding of my own heart.
With that and the new national restrictions in the UK from 5th November, my level of “cheersy-cheers” spirit is at an all-time low.
Brian David Gilbert’s “Earn $20K EVERY MONTH by being your own boss” helped a little though.
I’m working on a publishing platform that has a lot of special cases for a single publication. That publication is now being handled by another codebase so I have the opportunity to remove it and simplify a lot of the existing software. However, removing it cleanly without risking breaking other functionality has proven very difficult.

I tried a few different strategies: trying to remove all of the view-related code first then trying to come at it by removing a whole feature and all its related dependencies at a time. However, I kept coming unstuck: finding myself having deleted a lot of code but with far too many tests failing and not enough confidence I was on the right track so I’d git reset --hard and abandon the approach.

On Friday, I finally succeeded by being far more conservative in the amount I deleted at any one time. Rather than deleting an entire feature, I’d pick a small aspect of it (say, an admin-only view) and remove that, ensuring all the tests still passed before removing the next slice.

The result?

39 additions and 2,786 deletions.
Time to find some ear plugs.

Weeknotes 52

2020-10-29T10:32:00+00:00

It has been a year since my first weeknotes.

I still haven’t finished that blog post on React Hooks.
Since reading Caroline Crampton’s “The Problem with the Inconsequential Quest”, I’ve found it hard to listen to recommended podcasts that fit the description, e.g. Underunderstood’s “Amazon’s Next Top Model”:

I’m sure readers will be very familiar with the form: a host chooses something that is of seemingly little importance and investigates it with a thoroughness and journalistic rigour that seems completely out of proportion to the original question. The resulting episode or series documents this journey in detail and ultimately reveals a conclusion that surprises and delights listeners.
Instead of writing these notes on Sunday while C napped, I attempted to remap the extra buttons on my trackball to go back and forwards in my browser. I tried USB Overdrive, SteerMouse and Sensible Side Buttons but balked at the need for kernel extensions (and sadly Sensible Side Buttons doesn’t work because one of the buttons is interpreted as a middle-click).

I spent an hour trying to clean up the various bits of software I’d tried as kextstat revealed USB Overdrive’s kernel extension lingered after uninstallation. In the end, I had to use kextcache to clear “staged” extensions in Recovery Mode.

I’ve actually made things worse than when I installed the official Kensington TrackballWorks driver.
I’ve been doing quite a lot of work with Amazon Web Services for a client and was very grateful to have automated it with Hashicorp Terraform as I needed to quickly destroy and recreate S3 buckets connected to CloudFront distributions with ACM certificates.

I’ve found it useful to manage both our production and staging environment using a single Terraform repository by leveraging AWS Organizations and the ability for an administrator in the root organization to temporarily assume the role of an administrator in any of the member accounts. By default, AWS will create a role called OrganizationAccountAccessRole to do exactly this.

To manage Terraform resources in different AWS accounts we can then use an “Assume Role” block in an AWS provider like so:
```
provider "aws" {
  alias  = "root"
  region = "eu-west-1"
}

provider "aws" {
  alias  = "production"
  region = "eu-west-1"
  assume_role {
    role_arn = "arn:aws:iam::PRODUCTION-ID:role/OrganizationAccountAccessRole"
  }
}

provider "aws" {
  alias  = "staging"
  region = "eu-west-1"
  assume_role {
    role_arn = "arn:aws:iam::STAGING-ID:role/OrganizationAccountAccessRole"
  }
}
```
Then you specify the appropriate provider alias in each resource, e.g.
```
resource "aws_s3_bucket" "staging" {
  provider = aws.staging
  bucket   = "bucket-staging"
}

resource "aws_s3_bucket" "production" {
  provider = aws.production
  bucket   = "bucket-production"
}
```
You can use something like AWS Vault to run your Terraform commands as an administrator in your root account and it’ll assume the appropriate role for each resource automatically:
```
$ aws-vault exec root-account-administrator -- terraform plan
```

I’ve been working on a new feature in a Rails application and needed to hide it behind a feature flag so I wasn’t blocking other features that needed deploying. Now that I have two possible sets of behaviour in various parts of the application, I wanted to test them both with RSpec.

The feature flag lives in a bit of custom Rails configuration, e.g.

Rails.configuration.cool_new_feature = ENV['COOL_NEW_FEATURE']

Then various parts of the application switch on the value of that flag:

if Rails.configuration.cool_new_feature
  # do something cool
else
  # do something less cool
end

I wanted to change the value of that configuration before running some examples and reset it back to its original value after the test, e.g.

context 'with the cool new feature enabled' do
  around do |example|
    original_value = Rails.configuration.cool_new_feature
    Rails.configuration.cool_new_feature = true

    example.run

    Rails.configuration.cool_new_feature = original_value
  end

  # Examples here...
end

However, adding this across several places in the test suite seemed rather noisy but we can leverage user-defined metadata with hook filtering to clean this up:

# in spec/rails_helper.rb
RSpec.configure do |config|
  # ...

  config.around(:each, :cool_new_feature) do |example|
    original_value = Rails.configuration.cool_new_feature
    Rails.configuration.cool_new_feature = example.metadata[:cool_new_feature]

    example.run

    Rails.configuration.cool_new_feature = original_value
  end
end

Then in our specs, we simply tag examples with the appropriate metadata:

context 'with cool new feature enabled', cool_new_feature: true do
  # ...
end

context 'with cool new feature disabled', cool_new_feature: false do
  # ...
end

I haven’t had any epiphanies after writing these notes every week for a year but I still think about the Paul Murray quote I shared in weeknotes 15:

Although technology has the capability now to record entire lifetimes, meaning that every moment may be pulled from the foaming sea of oblivion to the dry land of perfect recall, the mythic power of the photograph nevertheless relates to the future, and not to the past. Every recording conceals the secret fantasy of a future self who will observe it; this future self is himself the simulacrum, the persona ficta. He exists beyond time, beyond action, beyond need; his only function is to witness the continuum of the past, as he might observe the steps that brought him to godhood. Through this fantasy, time is transformed from the condition of loss into a commodity that may be acquired and stockpiled; rather than disappear ceaselessly into the past, life accumulates, each moment becoming a unit of a total self that is the culmination of our experiences in a way that we—biological composites who profligately shed our cells, our memories and our possessions—can never be.

Weeknotes 51

2020-10-18T09:33:00+00:00

Two weeks ago I wrote about choosing between two 12” pizzas and one 18” pizza by comparing their surface area but Tom pointed out this only works if you care about sheer quantity of pizza.

If you care more about pizza crust (especially when studded with burgers), you should instead compare the circumference:
- “circumference of 12 inch diameter circle” = 37.7 inches * 2 = 75.4 inches
- “circumference of 18 inch diameter circle” = 56.6 inches
I added a page about Leo’s astounding “iCE2Tetris” project to the London Computation Club wiki this week.

According to the git history, I have made 637 changes to the wiki since 2015, adding 8,228 lines and removing 1,911.

While I’d go years between posts on this site, you could often find me writing up club meetings every few weeks.

The write-up I remember most is “Types and Programming Languages Chapter 15: Subtyping” as I left the meeting defeated and burnt out, ready to give up on the club altogether but the act of writing it up clarified my understanding and restored my energy for returning to the club.
I recently inherited a codebase and am the sole developer working on it. This means I’m in the rare position to make whatever change I see fit to make it easier to work on and so I decided to start with dependency updates.

Having seen it used before, I decided to set up GitHub Dependabot. While finding the right documentation was tricky given that Dependabot is being integrated into GitHub itself, once I had it set up I was pleasantly surprised how well it worked: raising PRs for upgrades, automatically rebasing them and even detecting when multiple dependencies need to be updated in tandem.

I have mixed feelings about updating dependencies now being an essential part of being a web developer but if you’re going to have to do it anyway, you might as well automate it.
The other change I’d like to make is to the code style itself: the formatting, linting, etc. Running RuboCop with the project’s current configuration produces over 1,000 violations so I needed some advice on how to approach this in a manageable way.

Thankfully, Scott Matthewman’s “Calling in the cops” is an excellent talk about this exact topic: how to sensibly approach linting a legacy codebase.
This led me to discover the Rails/FilePath cop prefers using forward slashes in paths and share in the collective disbelief that Ruby’s File.join does not change separators based on platform.

Weeknotes 50

2020-10-11T12:02:00+00:00

After realising our DNS sinkhole was causing the BBC iPlayer app for the PlayStation 4 to crash (presumably because some essential telemetry was being blocked), I switched it over to use Cloudflare’s 1.1.1.1 and discovered two whole series of “Hey Duggee” not on Netflix. That’s my viewing for the foreseeable sorted.
As C grows more confident with his toddling, his love of opening cupboards and drawers to explore their contents has also increased. I began securing certain cupboards with “child safety locks”: plastic straps secured at either end with adhesive tape.

Having placed a lock on our fridge the wrong way round, I followed the instructions to remove it: heat up the adhesive with a hair-dryer and peel it off. While that did the trick, it also melted the thermofoil on our fridge door causing it to warp and peel away from the wood.

It turns out the lock is no match for C’s strength as he can easily pull the ruined door open.
My mum refers to C’s teeth as “tooshy pegs”.
Our evening schedule is dictated entirely by Richard Osman.

First, there’s “Pointless” from 5:15 pm to 6 pm when C has his dinner. We switch over to BBC Two for “Richard Osman’s House of Games”, the end of which marks the beginning of C’s bedtime routine.

After a hard day, I long for the “Answer Smash” siren.
Following Maciej’s recommendation, I’ve been reading Cixin Liu’s “The Dark Forest”. I read the previous book back in December last year but didn’t continue with the series. In looking for something new to read, I’m not sure why I overlooked this as I am thoroughly enjoying it so far.
Our flat has an automatic car gate that stops working whenever it rains. As this week has been particularly wet, one of our neighbours discovered you can unlock the gate arms (so they can be opened and closed manually) by turning a small bolt.

You can now find a pair of needle-nose pliers in my coat pocket wherever I go.
E and I watched “Onward” last night. While she wasn’t especially moved, I cried the most I have in years.
I enjoyed the first two episodes of Noah Kalina and Adam Lisagor’s “All Consuming” podcast reviewing new direct-to-consumer products commonly advertised on the likes of Instagram, etc.

The subject of their first episode, “the Tesla of chicken” NUGGS’ release notes are quite the read:
- Enables a close to indistinguishable chicken-like substrate (utilizing soy and wheat proteins)
Ben Schott’s “Welcome to Your Bland New World” (via Alice Bartlett) is another exploration of this “zeitgeist marketing”:

All startups seek to disrupt and disintermediate a smug status quo, or originate and dominate an entirely new niche. But what makes a brand a bland is duality: claiming simultaneously to be unique in product, groundbreaking in purpose, and singular in delivery, while slavishly obeying an identikit formula of business model, look and feel, and tone of voice.
In need of a fun game to pass the time, I read Eurogamer.net’s review of “Hades” and bought it.

Having previously tried and failed to get into “Dead Cells”, I was nervous about another “roguelike” but something about it works for me. Perhaps the sense of progress with every failed run as the story and world unfolds breaks the monotony of dying over and over.

It has also inspired me to listen to Anaïs Mitchell’s “Hadestown” again.
While not going quite so far down the ergonomic keyboard rabbit hole as Steven Waterman, I did buy the wired version of Kensington’s Expert Mouse Trackball. It is comically large with a 55 mm diameter trackball but if it helps my ailing skeleton, I’m willing to try it.

Weeknotes 49

2020-10-04T09:34:00+00:00

C had his 1 year vaccinations which meant we had to top him up with a cherry-flavoured paracetamol suspension to keep his fever down. At the same time, he was fighting off his first cold caught from nursery which made for a grumpy, ruddy-cheeked boy.

I caught C’s cold so my Lemsip intake has sky-rocketed.
E and I began watching HBO’s “Watchmen” TV series after its success at the Emmy Awards. E has neither read the comic nor seen the 2009 Zack Snyder film so I attempted to summarise the key plot points.

It got tricky when I had to explain the squid.
We celebrated our third wedding anniversary by exchanging gifts, making an extravagant dinner and indulging in various indie classics from the 90s including Garbage’s “Version 2.0”, Placebo’s “Without You I’m Nothing” and Belle and Sebastian’s “If You’re Feeling Sinister”.
I worked on a project with Chris all week, mostly pair-programming via Tuple. As my other project work is quite solitary, this was a welcome change. It was great to get a chance to work with Chris for the first time after years of being involved with London Computation Club and London Ruby User Group together.
Despite being put off Netflix’s “The Chef Show” by its first episode with Gwyneth Paltrow, I watched the episode with Chad Robertson and Chris Bianco at Tartine Bakery and am warming to it a little. I assumed it was yet another entry in the genre of “famous chef and their celebrity friends” but, in this recent episode, Favreau seems genuinely humble and keen to learn from others.

It also gave me a detailed look at Robertson’s final shaping technique for his country loaf recipe which prompted me to give it a try:

And here’s the result:

Weeknotes 48

2020-09-27T12:45:00+00:00

I thoroughly enjoyed Anna Shipman’s “Meeting everyone on a new team”:

When I joined the Financial Times as Technical Director for FT.com, I inherited a team of around 50 engineers. One of the first things I did was meet each of them for a one-to-one.

Though it did make me envious of Anna’s mentors:

The idea was suggested to me by a mentor, who’d been advised to do it by his mentor, a Rear Admiral [emphasis added], who said this was something you should do whenever you have a team of fewer than 150 people.
Torn between sharing two takeaway 12 inch pizzas or a single 18 inch one, E and I turned to Wolfram Alpha:
- “area of circle with diameter 12 inches” = 36 square inches * 2 = 72 square inches
- “area of circle with diameter 18 inches” = 81 square inches
I dropped C off at nursery for the first time. Wearing a face mask, I had to slip on blue, disposable shoe covers while waiting for him to settle.

At a certain point, one of the key workers said to the other “let’s do the separation now” and it was time for me to leave.
In order to test parts of a new project, I have to send an extra HTTP header with my requests. If you use Chrome or Firefox, ModHeader seems a popular choice for doing this but, as a Safari user, I needed an alternative.

Instead of using a browser extension, I settled on using mitmproxy, a “free and open source interactive HTTPS proxy”. I use its “Set Headers” feature to add my extra header to requests matching a particular domain, e.g. to add X-Special-Header: header-value to any request to a Heroku review app:
```
$ mitmdump --modify-headers '/herokuapp.com/X-Special-Header/header-value'
```
This works regardless of browser and with tools such as HTTPie.
“The Great British Bake Off” is back.

After finding last year’s competition disappointing and upsetting, my faith was somewhat restored by the much more joyful “Junior Bake Off” and “Bake Off: The Professionals”.

After the first episode and Matt Lucas’ surprisingly accurate impression of Boris Johnson, I’m cautiously optimistic.
I’m a big fan of mint chocolate and have come to the conclusion the only mint chocolate worth buying is “Lindt Excellence Dark Mint Intense”. Anything else just doesn’t cut it.
C has been developing a special move he deploys whenever you prevent him doing something he wants (e.g. pulling things out of the fridge, grabbing bottles of gel for unclogging drains) where he’ll suddenly stiffen his entire body, arching backwards as hard as he can. We’ve tried explaining this usually results in an outcome no one desires: flying headfirst backwards.

We call it the “protest banana”.

Weeknotes 47

2020-09-21T11:55:00+00:00

It was C’s “settling in” week at nursery this week. E took him in for a few hours every day to get used to the unfamiliar surroundings and his key worker. This meant ironing name labels into all of his clothes and that he was in someone else’s care for an hour or two for the first time.

I capitalised on this strange, guilty sense of freedom by going for a run for the first time in months. I didn’t get very far and spent a lot of the time trying to keep my running tights up. I also indulged in some science fiction.
Five months ago, I planted a clove of garlic. Today, I harvested it!

It certainly looks like a bulb of garlic but we now need to dry it out for a few days. I’m only slightly alarmed by the RHS Gardening website’s warning that growing garlic from a supermarket “may carry disease”.
The official GitHub CLI 1.0 was released. I’ve long used hub but as its maintainer Mislav was hired by GitHub to build GitHub CLI, I’m going to try switching over.

The GitHub CLI that we are building is not exactly what I would have chosen to create if I was the only person in charge of making it, but this is A Good Thing. Before, I never really made an effort to understand who the audience of hub was, but with an actual team we finally get to explore that and hopefully build something that’s ambitious not in terms of the number of features it offers or how much of the GitHub API it covers, but in how well it helps people be productive with their daily work.
With an increasingly mobile C in nursery, E going back to work and me working with multiple clients for the first time, I’m at a bit of a loss for notes this week.

Weeknotes 46

2020-09-13T10:23:00+00:00

We’re back in London after two weeks up in t’north of England.

It had been six months since our families had properly seen C which is basically half his life so far. I got to rummage through old toys, uncovering an old DUPLO aeroplane and train set of mine. We discovered a local Yorkshire gelateria and revisited the bakery that provided bread for our wedding, giving C his first taste of a Yorkshire Curd tart.

It was the break we all needed before everything changes in the next few weeks as E returns to work and C starts nursery.
Part of the ritual of leaving guest accommodation is logging out of one’s Netflix account. Depending on the device, this may involve an arcane sequence of button presses to reveal a secret Settings screen:

From within the Netflix app, use the arrows on your remote to enter the following sequence: Up, Up, Down, Down, Left, Right, Left, Right, Up, Up, Up, Up.

It is easier to remember if you think of it as a variation of the Konami Code.
Speaking of Netflix: during the trip, we signed my dad up for his own account so he could stop filling my viewing history with endless shows related to the drug trade. I’ve been tempted to delete my profile and start again but am trying to hide the things he has watched instead.

However, it can “take up to 24 hours for a hidden title to be removed from all your devices” and the interface seems remarkably flaky. I’m currently having to delete each and every episode of “Suits” and “Homeland” individually (I blame eventual consistency).

Hopefully I’ll soon be rewarded for my meticulous clicking with fewer recommendations for documentaries about Pablo Escobar and El Chapo.
One thing we’ve missed while being away was a damn fine cup of coffee. Leo recommended James Hoffmann’s “The Ultimate V60 Technique” which has immediately sent me down something of a “coffee brewing theory” rabbit-hole.

Weeknotes 45

2020-09-06T11:02:00+00:00

“We’re going to pull the alarm now and a lot of people will come into the room but don’t worry.”

A year ago, I held an exhausted E’s hand as consultants, doctors and midwives gathered around C on a Resuscitaire. It’d been two days since E’s waters had broken and what little sleep we’d had was punctuated by my clumsy timing of contractions. It was a time of being offered choices, desperately trying to understand Cochrane systematic reviews and ultimately making decisions I was in no way equipped to make.

I listened to the mundane counting as they performed CPR, each new round of “1, 2, 3, 4” steadily filling me with dread.

Eventually, a mewling and a hubbub as C urinated on several members of staff. They began bickering among themselves as to what had happened to him with the most commanding person in the room repeatedly insisting “there was no meconium on the vocal cords”. As they finally beckoned me over to properly see C for the first time, the lead midwife jostled to show me a clear tube with something cloudy within, “see, look at the mucus!”

Bewildered, I did as I was told and followed as he was wheeled to the neonatal intensive care unit (the “NICU”). He was soon covered in dressings with teddy bears and the 3M logo on them. I quickly began misspelling the word “cannula”.

As I watched him through the plastic, I turned to the nurses and, masking my panic, asked “erm, is it OK he’s pulling his breathing tubes out?”

I returned to the labour ward to find E eating a bowl of bran flakes.

Later, after being told parents shouldn’t be present for the procedure, E and I would sit in a small room off the NICU as C had a lumbar puncture. I stared at a noticeboard, trying not to think about what I’d read about “bacterial meningitis” the night before.

Eventually, we’d make it home after the most careful drive of my life. There’d be a trip to the children’s ambulatory care unit to deal with C’s cannula which we were told had “tissued”. We’d watch as two people struggled to fit another and I distracted myself by staring at some morning TV show about how a teenager had gone blind from a diet that consisted exclusively of chips.

There’d be a red book and a course of benzylpenicillin and gentamicin. I forget which one stings.

There’d be tearful debriefs with our community midwives. I’d grow resentful and then angry at how polarised the NHS and NCT were and how that might have affected what happened. I’d reluctantly share our “birth story” with our NCT class and immediately regret it.

I’d think about sharing what had happened more publicly in an attempt to order my own thoughts and feelings but nothing would come of it.

Instead, after nine weeks, I’d start writing weeknotes.
Happy 1st birthday, C.

Weeknotes 44

2020-08-30T09:39:00+00:00

In the past two weeks, C has begun insisting we hold his hands so he can toddle around. Having previously shown little interest in movement, this is quite a change of pace and has led to the discovery there is no greater fun in the world than pulling magnets off a radiator or mashing the buttons on a washing machine.
This newfound mobility has meant I routinely find tiny, greasy hand-prints in new and exciting places. John Siracusa would not approve of C’s love of pressing his sticky palms against our TV screen.
Now that I have been able to help with childcare in these last few weeks before E returns to work, she has begun a methodical deep clean of our flat starting with our kitchen (grubby hand-prints notwithstanding). It has been a great opportunity to KonMari the strange, useless things one accumulates over time, e.g. an expansive collection of reusable wine bottles that can only be filled at a shop that no longer offers refills.
I write this looking out over a rolling Yorkshire dale.

We’re staying in a self-catering lodge which was apparently thoroughly disinfected and—as if to really shock-and-awe any virus—had some kind of antibacterial fog bomb detonated within.

This isn’t our first time here as this was our wedding venue nearly three years ago. Upon arrival, we were greeted by a member of staff who had looked after us back then. She told us they stopped doing weddings shortly after ours so we were one of the last.
This is the furthest we’ve travelled since the pandemic began. Halfway through the journey, we stopped at a motorway service station and enjoyed a picnic of cold pizza and peanut butter sandwiches in our car.

Weeknotes 43

2020-08-23T15:23:00+00:00

I met a former colleague at a local pub and we sat out front to catch up. After taking my order, a member of staff asked to scan my wrist with an infrared thermometer. This was an entirely new experience and I was quite giddy at having my faculties critically assessed by a beige handheld device.
I finished my latest contract on Friday and, due to last week’s preparation, I write these notes on a freshly formatted laptop.

There’s something cathartic about starting afresh, clearing out my old 1Password vaults and asking whether each piece of software sparks joy or not.
I went down a bit of a rabbit-hole installing Backblaze via Homebrew.
Now that I’ll be helping out with childcare more, I inevitably found myself watching Peppa Pig this morning and was reminded of its damning satire of knowledge work at Daddy Pig’s office.
Once more, I’m not sure what’s next but I’m looking forward to a break and to celebrating C’s first birthday.

Weeknotes 42

2020-08-16T09:42:00+00:00

I made no notes this week.
Andrew Helwer’s “Taking my home work setup seriously” is making me reconsider the suitability of my Magic Trackpad especially given that I’ve developed some sort of callus between my palm and my wrist.

Then again:

Inside us all there is a void. People want to complete themselves and fill this void with spirituality, or hedonistic pursuits, or material things. If you’ll indulge a metaphor, this is not a void that can be filled - its nature is more akin to a black hole of the cosmic variety. Feeding it things - for example, expensive ergonomic equipment - will simply add to its mass and pull. Only if left alone might it slowly evaporate. You must learn to live with it.
stephanie’s “tech brain” provoked discussion among myself and some fellow software developers.

what is tech brain? there are lots of things to point to, but if i had to come up with a thesis it would be that tech brain is a sort of constant willful reductionism: an addiction to easy answers combined with a wholesale cultural resistance to any kind of complexity.

Like others, I found myself nodding along but Maciej asked an interesting question: how much of this is Twitter brain rather than being specific to the tech industry?
I’m currently running a full backup of my laptop with SuperDuper!. Next week, in a ritual to mark moving on from a client I’ve been working with since March, I’ll wipe my computer clean and start afresh.
C’s current favourite activity is to prop himself up on a bookshelf and systematically pull all of the books onto the floor.

Sometimes he wants to read them (or, in the case of certain pop-up books, attempt to decapitate as many cardboard dogs as possible) and sometimes he just wants to see ~~the world burn~~ the shelf emptied.

Weeknotes 41

2020-08-09T10:12:00+00:00

C has a set of “First Word” books that he often wants to read. They came in a little paper box so they can be kept together and each one is a different colour. Every time I put them away, I recite the mnemonic “Richard Of York Gave Battle In” for there are only six.
As it has been unusually hot and humid, I had to forego wearing socks and even resort to flip-flops on Friday. This marks my first slipper-free day working from home since March.
As the word “flip-flop” is not quite onomatopoeic (or is that echomimetic?) enough for this household, we like to use the much more accurate phrase “snicker-snackers”.
Listening to Adam Buxton interview Radiohead guitarist Ed O’Brien reminded me of my deep and perhaps-in-retrospect-unhealthy love of the band in my teens.
Buxton and O’Brien’s discussion of Christopher Nolan’s 2014 film “Interstellar” in relation to parenting reminded me of something I mentioned in my second weeknotes:

It’s like the experience of being a parent is an exercise in relativity in a way because your sense of time is so different to theirs and it’s so cruel how fast it is for you as an adult and it’s so torturously slow for them.
On a more joyous (but still poignant note), they also mentioned Tom Petty, Jeff Lynne, Steve Winwood, Dhani Harrison and Prince’s performance of “While My Guitar Gently Weeps” during A Tribute to George Harrison in 2004. Where does that guitar go?
We continue to watch “Parts Unknown” and, despite owning one of his books and seeing him on countless other cookery shows, I found the episode about Noma’s René Redzepi fascinating.

It’s one thing to hear the phrase “New Nordic Cuisine” over and over but another to hear of Redzepi’s childhood in Macedonia foraging with his father and watching chefs present their dishes for the scrutiny of their peers:

And that might well end up on the menu?

No. This is not about putting things on the menu. No, I mean, if somebody makes a masterpiece, its their masterpiece.

Really?

Yeah, yeah, of course.

Isn’t it your historical imperative as the chef to take his good work and innovation and put it on the menu and take credit for it as your own? I mean, that’s the way it’s been done for centuries.

This is not the point here.

The pursuit of enlightenment and knowledge is its own reward?

To me, yes.
I did a technical test this week for a potential client and, in doing so, thought about the difficulty of trying to condense your experience and philosophy of programming into the solution of a canned exercise.

There’s often an expectation from interviewers this is possible: that a candidate can concisely convey their approach and a lot of what you need to know about them with these exercises which are typically blind of context.

Last year, I was doing a lot of recruitment for a client and saw many solutions to the same problem. It’s a variation on one I dare say a lot of Ruby programmers in London have seen: a checkout with various discount rules.

I grew to love the problem after seeing so many people focus on different aspects of it. With each new submission, I got a clearer and clearer idea of what I thought was interesting and what was unimportant. In the end, I was driven to write my own minimal solution: with nothing spare, focussed purely on my view of what the beating heart of the problem was.

In the end though, it was just my opinion. I believe code should be optimised for change (or deletion) but predicting exactly which changes it needs to accommodate is extremely hard. In the case of a tech test which contains only business rules and no context, perhaps it borders on the impossible.

Weeknotes 40

2020-08-02T10:20:00+00:00

Following the success of our trip north, we met up with my parents for the first time since lockdown began, having a picnic together so they could see their grandson in person after months of only FaceTime calls.

C enjoyed the extra attention and I got to tell my dad off for ignoring social distancing advice.

We came back from the trip with my parents’ old microwave. I’d tell you this is the first time I’ve had a microwave in 11 years but I fear it’s like people who are at pains to tell you they don’t even own a TV.
On the radio one morning, someone described tourism as:

The industry of human [emphasis added] happiness.

Tough luck, other species.
Continuing the theme of uncanny speech, we caught a TV interview with a politician talking about easing lockdown for our country’s beloved “sandwich bars”.
E and I began watching “Anthony Bourdain: Parts Unknown” after multiple recommendations.

We’ve only watched the first few episodes (originally aired in 2013) but I’m enjoying it so far. I expected a food tourism show with a larger-than-life host (not unlike David Chang in “Ugly Delicious”) but I was pleasantly surprised that Bourdain doesn’t dominate the show, instead preferring to let others speak in places like Myanmar and Koreatown.

The amount of truffle and foie gras on display in Quebec was a bit much though.

Seeing him travel the world and listening to his narration, it’s hard not to think about his death in 2018.
This week was dominated by some stressful decisions so I don’t have too much to share. I spent a lot of it agonising over details and going for walks around the block (mostly to the sound of Taylor Swift’s “Folklore”) in an attempt to clear my head.

Weeknotes 39

2020-07-25T10:48:00+00:00

Working from home with C in the house can be a strange experience. After a morning of calls about tree shaking, QR codes and gas fees, I’ll leave my desk and upon seeing him for the first time in a few hours, I’ll ask E, “whose baby is that?”
As E will be going back to work soon, we’ve been trying to find a nursery for C to attend three days a week. We were lucky enough to get more than one place for him but hadn’t seen one of the options due to the lockdown. This week, we arranged to have a look around (wearing masks, of course) and, after much deliberation, made our decision and paid our deposit.

(It still doesn’t feel real.)
After a stressful day, there’s no better way to unwind than by meticulously organising the wooden blocks in C’s walker.
During one of my lunchtime walks with C, he was fascinated by a passing bin lorry. One of the dustmen noticed him watching and waved emphatically.
During another stroll, a man crossing the street cooed at C in his baby carrier and gleefully called for all to hear, “look at his chubby cheeks!”
On a call with a potential vendor this week, when discussing the user experience of a particular system, their representative said:

Sure, if you want to cater to Joe Schmoe… from Kokomo.
It was the third anniversary of the London Computation Club trip to Bletchley Park.

Just look at us all: bright-eyed, bushy-tailed and sitting so close to one another.

Weeknotes 38

2020-07-19T10:10:00+00:00

After feeling “out of sorts” last week due to the monotony of life during the pandemic, I decided to go for a walk with C every lunchtime this week. I get a change of scenery and he gets to babble at dog walkers.

Now that Alice Bartlett’s “Week 98: Trees” has introduced me to TreeTalk, I might alter my route to investigate one of only two Portuguese laurel trees in London.
E and I decided to make our first trip outside London since lockdown began. We drove north for two hours and E’s family drove south for two hours and we all met for a picnic in the grounds of a country house.

As it seems unlikely we’ll be able to spend more time together soon, it felt good to exercise some small amount of freedom after feeling hopelessly trapped.

It has given us the confidence to get away for C’s first birthday and be closer to family so we won’t have to celebrate it purely virtually as we did for both E’s and my birthday this year.
This week saw the introduction of “Arctic Code Vault Badges” on GitHub, meaning open source software I wrote made it into the GitHub Archive Program.

More specifically, it is stored on “3,500-foot film reels” and has been written “into quartz glass platters using a femtosecond laser” with the intent of being archived for “over 10,000 years”.

Let future historians judge the value of adding Solaris support to rbenv or passing blocks to ActiveSupport::OrderedHash#merge.
I’m currently avoiding tweeting my weeknotes as my mentions are full of people mistaking me for Peiter Zatko.
After adding bamtech.sc.omtrdc.net to my DNS sinkhole’s safelist, we watched “Hamilton” on Disney+ having previously seen it on stage in London a year ago. It’s still rather good.

Weeknotes 37

2020-07-12T13:55:00+00:00

The Double Fine Productions remasters of classic LucasArts’ adventure games “Day of the Tentacle”, “Full Throttle” and “Grim Fandango” (as well as Double Fine’s own, more recent adventure game “Broken Age”) are all free on the Mac App Store this weekend.

When I was a child, my mum came back from PC World with a copy of “The Lion King” video game on 3½-inch floppy disks. Sadly, I had broken my disk drive when the sliding metal shutter from a diskette got stuck inside. My mum returned to the shop and swapped it for “Day of the Tentacle” on CD-ROM and my love of adventure games began.

In the words of the mechanics from one of my personal favourites:

We shoot you now like an arrow into the wind. May you pierce the heart of the wind itself, and drink the blood of flight.
The latest episode of Adam Buxton’s podcast where he discusses the recent death of his mother with Joe Cornish is worth your time.
Reading Una Kravets’ “Ten modern layouts in one line of CSS” drives home how woefully out-of-date my CSS knowledge is.
I caught C as he toppled backwards in surprise while sat on his mat and, without pausing, E said: “tell your mum I saved your life.”
Hillel Wayne’s “The Frink Is Good, the Unit Is Evil” introduced me to Frink, a “special purpose [language] for dealing with units”.

There’s a web-based interface to Frink available and it supports a dizzying number of units. As someone who often has to convert American recipe quantities, it will come in very handy indeed.

Of course, it was named after Professor John Frink:

I predict that within 100 years, computers will be twice as powerful, ten thousand times larger, and so expensive that only the five richest kings of Europe will own them.
After an intense week at work, I feel somewhat out of sorts and so decided to take a long, meandering walk and catch up on some podcasts while E took C to visit her aunt.

I made it as far as the local reservoir before a group of people gathered in my path made me turn back. Turns out I’m neither good at meandering nor staying outside the house for longer than half an hour.

Weeknotes 36

2020-07-05T09:19:00+00:00

Since getting “Animal Crossing” over a month ago, I’ve gone through phases with it: at first, learning how to play then relentlessly plundering other islands for resources then checking turnip prices twice a day and having a bad time on the Stalk Market. After feeling a little burnt out, I let go of all material concerns and simply fished to my heart’s content.

This week I visited some friends’ islands for the very first time and hosted my first guest. This social aspect is entirely new to me and, having seen how beautiful others’ islands are, you will now find me happily planting imaginary flowers.
We had occasion to use our car for the first time in months and weren’t surprised when it failed to start. Thankfully, I’ve had to charge our car battery several times before and so painstakingly removed it from the car after reading the order in which to disconnect the two terminals at least six times (negative then positive and then the reverse order to reconnect).

There’s nothing like bringing a metal socket spanner toward a car battery terminal for focussing the mind.
Last week I credited Murray with describing modern web development as “putting strings into databases and getting strings out of databases” but it turns out he was paraphrasing James Darling who, in turn, was quoting Ben Griffiths.
There’s a new series of “Everybody Loves Raymond” creator Phil Rosenthal’s travel documentary “Somebody Feed Phil”. I find all food-based TV shows comforting but Rosenthal’s genuine, unbridled glee makes it especially optimistic viewing. I forgot how catchy the theme song is.
Speaking of food TV, a recent episode of “Bake Off: The Professionals” featured host Tom Allen posing as a market trader. When I lived in south-east London, I used to walk past a market stall selling fruit every day and the trader would bellow a single phrase:

Tray of bananas, sayp!

For literal years, I could never decode what this last word was.

Finally, after nearly four years, I happened to hear the trader’s bark one more time:

Box of strawberries a pound, tray of bananas same!

The mystery was solved.
I decided to get a repeat prescription for my occasional migraines and struggled to navigate the different websites and iOS apps required to talk to a doctor.
- At first, I tried the myGP app which I’ve used to book appointments before but nothing was visible there.
- I then tried to use the Patient Access website but still nothing was available.
- I then followed instructions to set up the NHS app but, again, nothing.
- I rang my local surgery and they told me I needed to use a fourth service: WebGP.
After filling out a long questionnaire (and tweaking my answers so it didn’t tell me to immediately ring 999 for an ambulance), I got a call from a GP that same morning who happily created a repeat prescription for me.

However, it then became obvious I have no idea how repeat prescriptions work (a familiar fact given I was once shouted out over the phone by a receptionist for not understanding the difference between the prescription and the medication itself). The doctor said they had created the prescription but what did that mean? Looking in the NHS app, I could see the prescription but there was a separate section for “orders”; did I need to order it? After finding no answers online, I went ahead and ordered it.

The next day, my order was marked as “rejected” in the NHS app and I was instructed to ring my GP. After a hilarious exchange as my local surgery attempted to verify my email address and I ran through the four separate addresses I used for each of their services, it turns out I didn’t need to order anything: my prescription was waiting for me at the local pharmacy.

Utterly baffling.
Whenever E and I are discussing a topic that requires some disclaimer (e.g. a relatively petty matter given the current pandemic), we’ll preface what we say with the single word “Rigby”, e.g.

Rigby, I wish there was more variety to our day.

This incomprehensive phrase comes from an episode of “Silicon Valley”:

“I know we keep saying this but even though Richard is a great guy—”

“And a brilliant coder.”

“Wonderful. Nevertheless, he is lacking in certain managerial capacities.”

“Totally fair.”

“It’s fair, right?”

“He’s still a great coder.”

“And an amazing human being!”

“OK, we have a lot of shit we want to say about him, do we have to keep prefacing it with all this nice guy stuff? I mean, if so, we’re going to be here all night. […] What if we use like a dictionary patch to compress all the nice guy stuff?”

“Like an acronym?”

“Exactly. Richard Is Great But, Y’know. RIGBY.”
Despite being put off by the trailer, E and I watched “Eurovision Song Contest: The Story of Fire Saga” last night after hearing recommendations from several people.

It reduced us both to actual tears.

Weeknotes 35

2020-06-28T09:40:00+00:00

Nikita Prokopov’s “Time to upgrade your monitor” finally convinced me to upgrade to a 4K display after over a decade of staring at less than 100 pixels per inch on an old BenQ monitor.

In the words of Alex, I’ve been punching myself in the eyes with squares for too long.
I was reminded that “Celery Man” exists and it brought me great joy.
For the past few weeks, my laptop has been increasingly struggling when running Zoom calls with multiple video streams, screen sharing, etc. with its performance dragging to a halt and the fan noise approaching that of a jet engine.

Receiving the new monitor and seeing photos of Tom’s setup inspired me to remove the protective plastic case from my computer and stand it on its end with the help of an old, wooden iPad stand. The roaring fan noise is now gone and my performance woes are no more.

In retrospect, it probably wasn’t a great idea covering up the few vents on the underside of my MacBook Pro for several years.
A former colleague asked me for a copy of a presentation I had given at Altmetric about using Dijkstra’s algorithm to exhaustively map scholarly identifiers to one another (e.g. DOIs to PubMed IDs to Handles to ISBNs to arXiv identifiers).

That project was so much fun to work on and led me to write a Fibonnaci Heap implementation.

To paraphrase Murray: so much of modern software development on the web is putting strings in to databases and getting strings out of databases so it is a rare treat to work on a problem that requires genuine Computer Science to solve.
Apologies for the brevity of this week’s notes: I have started working on my unfinished article about React Hooks again.
I enjoyed my very first Fathers’ Day.

Weeknotes 34

2020-06-21T10:55:00+00:00

While making coffee with C in my arms one morning, he suddenly lunged for the Chemex, spilling a slurry of hot water and coffee grounds as I shouted “no no no no no!” A stunned E declared she would now call me Shia LaBeouf but thankfully that has not happened.
I can’t mention Shia LaBeouf without recommending Rob Cantor’s song about the true story of an actual cannibal.
I’ve been making sourdough bread since March 2012 long before it became cool, even encouraging others to get into it but my recent bakes have not been good.

My starter has been underactive and overactive, the bulk fermentation has been both too long and too short. More often than not, my bread is what Murray would call “a tasty cowpat”.

As others got into baking, measuring their dough temperature with infrared thermometers and making their own blend of flour based on exacting protein levels, I was dismissive, believing only in some woolly notion of technique. Those bakers now produce consistently great loaves; each of Leo’s bakes is more beautiful than the last.

Perhaps I should get with the times and be less “son-mat” and more molecular gastronomy.
After correcting some typos in last week’s notes, I enabled Vim’s spell checking. The following bit of configuration now highlights any misspelt words in Markdown files:
```
autocmd BufRead,BufNewFile *.markdown,*.md setlocal spell spelllang=en_gb
```
E tells me I should look into something called a “word processor”.
I changed both this site and the Ghost Cassette website to have a more consistent vertical rhythm. I found this little snippet of CSS to display a baseline grid extremely useful when trying to debug spacing issues:
```
body {
  background: linear-gradient(rgba(0, 0, 0, 0.3) 1px, transparent 1px);
  background-size: 100% 24px;
}
```
C’s latest discovery is clapping. I recommend everyone indulge in a round of applause every meal and at random times throughout the day.
I’ve been listening to a lot of Phoebe Bridgers as her new album “Punisher” came out this week. I dearly hope her 2018 album “Stranger in the Alps” is named after the censored TV edit of “The Big Lebowski”.
I’ve been thinking about the streetlight effect and problem solving. There are many versions of the story but here’s “Nasreddin’s ring”:

Mulla had lost his ring in the living room. He searched for it for a while, but since he could not find it, he went out into the yard and began to look there. His wife, who saw what he was doing, asked: “Mulla, you lost your ring in the room, why are you looking for it in the yard?” Mulla stroked his beard and said: “The room is too dark and I can’t see very well. I came out to the courtyard to look for my ring because there is much more light out here.”

As a software developer, I wonder if there is a trap as you solve similar problems over and over again during the course of your career. Do you become more likely to take any problem and turn it into a familiar one even if it’s not appropriate?

Weeknotes 33

2020-06-14T11:03:54+00:00

I’ve been researching how best to generate unique, human-friendly codes for a new project.

Scott recommended we take a look at Douglas Crockford’s “Base 32” system:
The encoding scheme is required to
- Be human readable and machine readable.
- Be compact. Humans have difficulty in manipulating long strings of arbitrary symbols.
- Be error resistant. Entering the symbols must not require keyboarding gymnastics.
- Be pronounceable. Humans should be able to accurately transmit the symbols to other humans using a telephone.
Crockford excludes the letters I, L and O to avoid ambiguity with the numbers 1 and 0. Less intuitively, he also excludes the letter U as this helps avoid accidentally generating codes that contain English obscenities: a problem I would have missed entirely.

We looked into the implementation of Rails’ SecureRandom.base58:
```
def self.base58(n = 16)
  SecureRandom.random_bytes(n).unpack("C*").map do |byte|
    idx = byte % 64
    idx = SecureRandom.random_number(58) if idx >= 58
    BASE58_ALPHABET[idx]
  end.join
end
```
I wondered why they didn’t simply modulo byte with 58 like so:
```
def self.base58(n = 16)
  SecureRandom.random_bytes(n).unpack("C*").map do |byte|
    idx = byte % 58
    BASE58_ALPHABET[idx]
  end.join
end
```
Scott found the explanation in a four-year old pull request to simplify the implementation: modulo bias. Naïvely simplifying the implementation would mean that the “random” choice would actually be biased towards picking some characters more than others.

Coincidentally, someone else suggested the same change again 12 days ago.
After last week’s big migration to Fargate, I wanted to double check the new Docker version of our application wasn’t writing to disk (in order to remain stateless).

The docker diff command handily told me exactly what a container was changing on its file system when running and revealed that Phusion Passenger was logging to a file when I didn’t expect it to.
```
$ docker diff 1fdfd1f54c1b
A /var/log/nginx/error.log
```
E has been binging the French spy drama “The Bureau” but I can’t help but think of The Day Today’s BBC drama of the same name every time I see her watching it:

This is supposed to be a high class bureau de change, not some two-bit Punch and Judy show down on the seafront at Margate!
Jeff Geerling’s “The best way to keep your cool running a Raspberry Pi 4” convinced me to buy the FLIRC case from The Pi Hut and he’s right: my Pi runs much cooler than my previous setup with a large heatsink inside the official case:
```
mudge@raspberrypi:~ $ cat /sys/class/thermal/thermal_zone0/temp
36998
```
In a move that threatens my marriage, I have spent far too much time this weekend trying to generate a custom Raspbian image that already has my Time Machine backups, Unbound DNS sinkhole and AirPlay receiver set up on it.

Thankfully, the tool used to create the raspberrypi.org images is open-source and you can find the results of my indulgent noodling in my fork of pi-gen.
Last week’s investment paid off.

Weeknotes 32

2020-06-07T14:18:00+00:00

Thom May’s “Sinkholing with PowerDNS Recursor” made me wonder why I run both a Pi-hole and our own Unbound DNS resolver. Perhaps I could implement blocklists and recommended exceptions using only Unbound configuration?

Our Pi-hole currently serves two purposes:
1. It runs its built-in DHCP server so every device connecting to our network uses the Pi-hole’s DNS server rather than my ISP’s DNS servers (as this isn’t configurable on my router).
2. It runs its own FTLDNS server to block access to domains.
As Thom points out in his post, given only those two roles and the fact I’m already running Unbound, the required tech stack for the Pi-hole (PHP, Lighttpd, Dnsmasq, etc.) seems excessive.

I decided to replace the Pi-hole with only two pieces of software:
1. Use ISC DHCP as a standalone DHCP server, replacing the Pi-hole’s Dnsmasq-based DHCP server.
2. Use the existing Unbound DNS server to block unwanted domains.
The Firebog’s “The Big Blocklist Collection” contains a curated list of domains to block, all in the following format:
```
0.0.0.0 some.bad.domain.here
0.0.0.0 another.bad.domain.here
```
In order to block these domains with Unbound, we can use the local-zone and local-data configuration options. The documentation for local-zone’s redirect type contains the following example:

It can be used to redirect a domain to return a different address record to the end user, with local-zone: "example.com." redirect and local-data: "example.com. A 127.0.0.1" queries for www.example.com and www.foo.example.com are redirected, so that users with web browsers cannot access sites with suffix example.com.

We’ll copy Pi-hole’s unspecified IP blocking strategy and return 0.0.0.0 for any A record lookups and :: for any AAAA record lookups:

Following RFC 3513, Internet Protocol Version 6 (IPv6) Addressing Architecture, section 2.5.2, the address 0:0:0:0:0:0:0:0 (or :: for short) is the unspecified address. It must never be assigned to any node and indicates the absence of an address. Following RFC1122, section 3.2, the address 0.0.0.0 can be understood as the IPv4 equivalent of ::.

So to block the two example domains above, we would need the following Unbound configuration:
```
local-zone: "some.bad.domain.here." redirect
local-data: "some.bad.domain.here. IN A 0.0.0.0"
local-data: "some.bad.domain.here. IN AAAA ::"
local-zone: "another.bad.domain.here." redirect
local-data: "another.bad.domain.here. IN A 0.0.0.0"
local-data: "another.bad.domain.here. IN AAAA ::"
```
We can do this by stitching together some basic command-line tools:
```
curl -sSf "https://v.firebog.net/hosts/static/w3kbl.txt" | # Download a blocklist
  grep '^0\.0\.0\.0' | # Filter out any comments, etc. that aren't rules
  tr -d '\r' |         # Normalize line endings by removing carriage returns
  sort -u |            # Remove any duplicates
  awk '{print "local-zone: \""$2".\" redirect\nlocal-data: \""$2". IN A 0.0.0.0\"\nlocal-data: \""$2". IN AAAA ::\""}' | # Convert to Unbound configuration
```
(You can find my full script to download and re-format all ticked blocklists on GitHub.)

As the blocklists contain domains which break essential functionality when blocked, we want to explicitly allow any domains on Anudeep’s list of exceptions.

To do this, we can use the always_transparent type of local-zone:

Like transparent, but ignores local data and resolves normally.

So, if we wanted to allow traffic to some.bad.domain.here even though it is already in our configuration, we need to add the following to our Unbound configuration before our blocklist:
```
local-zone: "some.bad.domain.here." always_transparent
```
This will ensure we ignore our redirect to local-data and look the domain up normally.

Again, we can do some basic scripting to convert the list into configuration for us:
```
curl -sSf "https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt" |
  sort -u | # Remove any duplicates
  awk '{print "local-zone: \""$1".\" always_transparent"}' # Convert to Unbound configuration
```
If we save those two lists into separate configuration files called 01-safelist.conf and 02-blocklist.conf under /etc/unbound/lists.d, we can include them in our Unbound server definition (e.g. under /etc/unbound/unbound.conf.d):
```
server:
  # ...other configuration here

  # Include blocklist and safelist
  include: /etc/unbound/lists.d/*.conf
```
We can use unbound-checkconf to ensure we don’t have any syntax errors and service unbound reload to load our new configuration if you’re using control-enable, otherwise service unbound restart.

We can test this by doing a lookup of a blocked domain with dig:
```
~> dig @127.0.0.1 doubleclick.net +short
0.0.0.0
~> dig @127.0.0.1 doubleclick.net AAAA +short
::
```
And check that any exceptions still return results:
```
~> dig @127.0.0.1 cdn.optimizely.com +short
cdn.o6.edgekey.net.
e5048.dsca.akamaiedge.net.
23.59.68.248
```
And with that, I could run pihole uninstall.
I spent a good part of the week working with The Scale Factory to migrate a project to AWS Fargate using the CircleCI Amazon Elastic Container Registry and Elastic Container Service Orbs.

It all felt rather familiar given how similar Fargate is to Hashicorp Nomad and my experience kicking off a migration to Nomad at Altmetric.

Matt wrote Rocksteady to handle orchestrating jobspec updates when deploying but thankfully, this is provided out-of-the-box by the AWS ECS Orb’s deploy-service-update job.

After all these changes, I’m hoping I will have cut down the total time to deploy (meaning from the start of our CI build to running our test suite, deploying to staging and then deploying to production) from over 34 minutes to 13 minutes.
After a bad week on the Stalk Market, I have now spent my entire life savings in bells on turnips.

Weeknotes 31

2020-05-31T10:12:00+00:00

The stress of buying turnips may have impacted the quality of last week’s notes.
After putting C to bed, E said “I wish we could just sit outside and have a glass of wine.” Thanks to the range of our baby monitor, we did exactly that.

While setting up, our upstairs neighbour walked by and invited us to join them further up our driveway. We ended up chatting the evening away from socially distanced camping chairs, contactlessly sharing our last four After Eight mints and giving directions to lost Deliveroo riders.

I didn’t realise how much I miss socialising in person with people outside our household.
I don’t know why but E and I have been making our way through the “Mission: Impossible” films.

So far, we’ve watched Brian De Palma’s “Mission: Impossible”, John Woo’s “Mission: Impossible II”, J.J. Abrams’ “Mission: Impossible III” and Brad Bird’s “Mission: Impossible - Ghost Protocol”.

The second one has not aged well.
I made bread this weekend after a short break from baking.

While our large quantity of flour means I can’t deviate from my usual recipe, Leo sharing a video of a baker shaping sourdough has inspired me to try different shaping techniques.
I have a file called budget.rb that contains all our household expenses and calculates how much money we spend every month. I often tinker with it and decided to switch to using the geometric mean instead of the typical arithmetic mean for some averages. (Despite E dismissing anything other than the arithmetic mean, median and mode as a “computer mean”.)

This led me to write a Ruby refinement to add all three Pythagorean means to Enumerable objects, e.g.
```
using PythagoreanMeans

[1, 2, 3, 4, 5].arithmetic_mean
# => 3

[1, 2, 3, 4, 5].geometric_mean
# => ~2.61

[1, 2, 3, 4, 5].harmonic_mean
# => ~2.18
```
I’ve never written a refinement before and immediately encountered a bug in Ruby 2.7 meaning you can’t refine the Enumerable module. Despite being four years old, James Adam’s “Why is nobody using Refinements?” is still as relevant as ever.
C got his first teeth.

Despite months of asking ourselves “could he be teething?” whenever he was irrationally grumpy, it was still a surprise to see them. This means we have to brush his teeth twice daily. So begins a lifelong wretched task.

Weeknotes 30

2020-05-24T14:06:00+00:00

I successfully completed another full orbit around the sun. To celebrate, E, C and I went for our first “stay alert” picnic in the park. While it was surprisingly busy for a weekday, it was great to be outside for the longest time since lockdown began.
As an antidote to the stress of Doom Eternal, Will and Andrew gave me the gift of “Animal Crossing: New Horizons”. I don’t think I’ve played a game that requires waiting real days for progression before but the enforced slow pace is a welcome change. My island is called Custard and I now understand more of Tom’s tweets.
As a birthday treat, E made Felicity Cloake’s perfect miso ramen and it was fantastic.
Following last week’s adventures with GPG, Maciej recommended drduh’s guide to using YubiKey for GPG and SSH. Even if you don’t have a YubiKey, it is a thorough guide to setting up a single GPG master key and single-purpose sub-keys.
I went on a related adventure to upgrade my SSH keys from a sole 2048-bit RSA key to a 4096-bit RSA key and a Ed25519 key.

Leo pointed out you can get the public SSH keys for any GitHub user by appending .keys to their profile URL, e.g. https://github.com/mudge.keys. You can also use .gpg to get their public GPG keys, e.g. https://github.com/mudge.gpg.
Shane told me off for not simply tilting my head in order to better locate last week’s infernal whining noise.
We watched (or is that played?) Netflix’s interactive “Unbreakable Kimmy Schmidt: Kimmy vs. the Reverend” and enjoyed it more than I expected. I’m not sure we’ll go back to watch all the alternate paths we could have taken though.
Despite being indifferent about the show, E will not stop singing The Mandalorian theme tune. It may well unseat the Outer Wilds theme and XCOM 2’s “Squad Loadout” music as the most annoying earworm in this house.
Paul Ford has started writing a newsletter called “i absolutely am going to bail on this in a month” (there’s an RSS feed too). As a fan of Ford’s Ftrain.com back in the 2000s, this makes me happy.

Weeknotes 29

2020-05-17T09:59:00+00:00

After publishing last week’s notes, E and I noticed a high pitched whining noise coming from our kitchen.

Unable to locate the source of the noise, I switched off the power to our oven, fridge and freezer but still the whining continued. I resorted to holding the cardboard tube from a toilet roll to one ear while plugging my other ear with a headphone, crawling on all fours trying to pinpoint this infernal whine.

With a growing headache, I climbed onto the kitchen counter and checked our fire alarm when suddenly I realised: the noise was coming from C’s baby bouncer. He’d accidentally turned on one of its soothing sounds and the manufacturers had seemingly chosen the sound of a single, agitated mosquito to represent the rainforest.
In my search, I did discover our fire alarm had a sticker on its underside that read “Replace by 2014”.
My changes to RubyGems.org stats were deployed so I can tell you on Friday, 15th May 2020, out of 54,933,639 gem install and bundle installs:
- 19,634,055 (35.7%) were from Ruby 2.3 (end-of-life since April 2019)
- 13,631,483 (24.8%) were from Ruby 2.5
- 8,370,929 (15.2%) were from Ruby 2.4 (end-of-life since April 2020)
- 5,952,688 (10.8%) were from Ruby 2.6
- 3,882,306 (7.1%) were from Ruby 2.1 (end-of-life since April 2017)
- 1,090,639 (2.0%) were from Ruby 1.9 (end-of-life since February 2015)
- 782,778 (1.4%) were from Ruby 2.0 (end-of-life since February 2016)
- 726,975 (1.3%) were from Ruby 2.2 (end-of-life since April 2018)
- 720,261 (1.3%) were from Ruby 2.7
- 122,573 (0.2%) were from Ruby 1.8 (end-of-life since August 2014)
This means that 34,610,255 (63%) of all requests were from versions of Ruby that are technically “end-of-life”.
Following Facebook’s description of their newly rebuilt tech stack for Facebook.com, I enjoyed reading Tom MacWright’s “Second-guessing the modern web”.

I was particularly drawn to the idea of the pit of success:

And it should be easy to do a good job.

Frameworks should lure people into the pit of success, where following the normal rules and using normal techniques is the winning approach.

I’ve mentioned my interest in the design concept of “affordance” before and my main consideration when designing software is what kind of behaviours does a decision encourage and discourage. “Pit of success” is a pithy way to communicate this idea: how can we construct a system such that the easiest possible thing is also the right thing to do?
In one of our recent trips to The Baby Club, the answer to the regular question of “what’s in the bag?” was “a bag”. The recursive nature of this answer reminded me of the filling inside KitKats and made me secretly wish the show would take a more surreal turn. I suppose I’ll just have to rewatch “Don’t Hug Me I’m Scared” instead.
This week at work, after successfully launching a major new feature, we took a few days to clean up some technical debt we’d accrued to meet our deadline. I took to calling it a “firebreak” week, having stolen the term from GOV.UK.

Having overheard this, E wished me well “turning off the fire engine”.
After seeing John Siracusa recommend using pmset to diagnose problems with a Mac that refuses to sleep, I used it myself when E’s laptop kept draining its battery with the lid closed.

pmset -g and pmset -g assertions revealed that sharingd was preventing UserIdleSystemSleep and, sure enough, toggling Screen Sharing (and a reboot) fixed the problem.
I went down a bit of a GPG rabbithole after discovering that my 12 year old 1024 bit DSA key is threatened by weaknesses found in SHA-1. Thankfully, The Apache Software Foundation’s “How to transition to a new PGP key” was an excellent guide to generating a new 4096 bit RSA key and transitioning to using it.
After reading many recommendations, we started watching “The Mandalorian”. It’s fine.
A recurring section of C’s Sing and Sign lessons is a cat named Jessie hidden in a box. I assume this is to teach children the problem of the Copenhagen interpretation of quantum mechanics.

Weeknotes 28

2020-05-10T10:16:00+00:00

Following radical candor and radical sensibleness, I propose a new thought technology: radical competency.

It can be summarised in three steps:
1. Think about a thing.
2. Say you’ll do something related to step 1.
3. Do the thing you said you’d do in step 2.
This is inspired by recent experience and by Maciej sharing Chelsea Troy’s “Reviewing Pull Requests”. While I’m not sure I agree about pushing commits to someone else’s branch, I agree with the goal of saving time and frustration by placing more burden on the reviewer.

In any exchange, you should be trying to reduce effort for the other party.
E and I watched “Star Wars: Episode IX - The Rise of Skywalker”. It was not good.
Following last week’s debugging adventures, I switched re2 from using Travis CI to GitHub Actions.

Most of my effort went into writing a new, custom GitHub Action called “re2 Test Action” which creates a consistent environment using Docker for running the re2 test suite against different versions of Ruby and the underlying re2 library.

It was a little tricky because the recommended way to setup Ruby versions with GitHub Actions only supports Ruby 2.4, 2.5, 2.6 and 2.7. However, the recently released RubyGems.org stats shows that the most popular Ruby version is Ruby 2.3 (and over 7% of people still use Ruby 2.1).

I’m attempting to get a more detailed breakdown of older Ruby versions as I’d love to know how many people are still using long end-of-lifed versions.

People have often been surprised, confused and occasionally frustrated that I support Ruby 1.8 in my libraries such as re2, Fieldhand and Embiggen. Having wasted so much time in my career struggling to upgrade dependencies that have, in turn, caused other breakages, I strongly believe libraries should go out of their way to be as compatible and frictionless as possible for the user to use. I never want one of my libraries to cause you incidental issues because I made a change internally.

While Mike Perham called for us to “Kill Your Dependencies”, I agree more with James Coglan who talked about this in Why Are Computers’ “Episode 1: A Fairly Deep Yak Shave”:

The thing that most concerns me is stuff that gets changed because someone thought that the new way is how it should’ve been done in the first place and it’s obviously better, but it doesn’t give any real new capabilities or power and doesn’t really fix any mistakes, and it breaks their existing software.

To me the canonical example of that is the Ruby 1.9 hash syntax, which a lot of people are like “oh, it’s obviously better”, but it doesn’t let you write programs that you couldn’t write before, it doesn’t fix any mistakes that anyone was making before, and it means that if someone uses that syntax, that program now won’t run on an older thing. It’s purely an aesthetic change. The aesthetics of code are important, and it’s important to have stuff that’s readable, but if you have a thing that’s already been shipped, making those tiny little fussy aesthetic changes to it, to me, never seems really worth it.
Never one to let a bank holiday weekend go to waste, I spent a large portion of yesterday on administrivia mostly informed by Martin Lewis’ “Money Saving Expert”.
On a bittersweet note, we finished “The Great Pottery Throw Down”. I am attempting to fill the hole it has left in my life by encouraging C to watch “Hey Duggee” instead.

Weeknotes 27

2020-05-03T09:40:00+00:00

E and I started watching series 3 of “The Great Pottery Throw Down”. I was drawn in by the sight of expert judge Keith Brymer Jones in dungarees throwing an amphora before asking contestants to do the same. We were hooked when we saw him moved to tears while judging a breakfast set:

That’s fantastic, mate. Fantastic!

After writing about my disappointment with the Great British Bake Off in my early weeknotes, this is the feel-good show I’ve been looking for.
In Coronavirus lockdown news: E cut my hair with a beard trimmer. After watching a few YouTube videos and trying to convert UK haircut “grades” into millimetres, we went for 6 mm on the side, 12 mm on top and 9 mm in between.
After my adoption of Shortcuts a few weeks ago, Tom asked why I didn’t use “Hey Siri” to activate my shortcuts instead of fumbling with my phone’s screen in the dark. I had tried but was frustrated pausing between saying “Hey Siri” and waiting for the telltale “hmm?”.

Tom told me you don’t need to pause at all: Siri will buffer everything you say after being activated. You will now find me occasionally saying “Hey Siri Bounce”, “Hey Siri Fold” and “Hey Siri Autolyse”.
I haven’t been enjoying the increased number of emotionally manipulative Coronavirus-related advertisements from commercial companies and seeing Uber’s “Thank You For Not Riding” made me particularly cynical. As Sean Haney demonstrated, “Every Covid-19 Commercial is Exactly the Same”.
There should only be two types of advert: the straightforward Cillit Bang approach or the utterly sublime Halo Top.
Someone in this household accidentally cooked our kitchen scales, reducing them to molten plastic so we had to invest in a replacement. As Neven Mrgan mentions in his pizza dough recipe, having gram-precision scales that have an extra long timeout is ideal for baking:

My favorite feature: extra long timeout (before it turns off) so I can forget to get the flour and run downstairs and hunt for it and when I come back, my measurement is still up on the screen.

We got ourselves some Salter Pro Stainless Steel Digital Kitchen Scales as a replacement and, while it isn’t listed anywhere, they have a very generous timeout.
Siguza’s write-up of their “Psychic Paper” security exploit for iOS is a great and extremely accessible read on how the incidental complexity of XML parsing resulted in a very serious bug.
We finished DEVS and, like Tom, I think it is well worth watching even if it wasn’t perfect.

It made me want to re-read Ted Chiang’s “Exhalation”, “Arrival” (formerly published as “Stories of Your Life and Others”) and Greg Egan’s “Permutation City”.
I investigated and fixed an issue related to re2, my most popular open source project, which led me on a journey to better understand shared library names and how they communicate ABI compatibility. This then led me to discover and fix a bug in a Homebrew formula to resolve the original issue.

I do love the twists and turns of debugging.
The jury is in: Doom Eternal is far too stressful.

Weeknotes 26

2020-04-26T11:00:00+00:00

I waffle.

My first draft of that sentence was:

When writing, I have a tendency to waffle.

Last week’s notes were a perfect example of this. When I first published them, E proofread and found so much to cut and simplify. So I did, over and over and over and over again.

In the words of Blaise Pascal (not Mark Twain):

I have made this longer than usual because I have not had time to make it shorter.
I’m experimenting with writing these notes on my phone during the week. The main issue is how inept I am at editing text on iOS, especially when trying to place the insertion pointer to paste text. It turns out there are three finger gestures for cut, copy and paste which make things much easier. I doubt I would have ever discovered them on my own.
Piotr flagged that my unminified JavaScript from last week fits into a single TCP packet when gzipped. As gzip compression has been supported in browsers for the past 20 years and I’m using Cloudflare to compress my JavaScript with Brotli, there really isn’t any need to minify it.

Still, I couldn’t resist playing a little code golf.
I ported the original distance_of_time_in_words test suite from Rails to JavaScript and used it as an excuse to set up my first workflow with GitHub Actions.

Running the test suite revealed JavaScript’s Date.prototype.getYear() doesn’t return what I expected:

A number representing the year of the given date, according to local time, minus 1900.

Sure enough:
```
> (new Date("2020-04-26")).getYear()
120
```
Thankfully, you can use Date.prototype.getFullYear() instead:
```
> (new Date("2020-04-26")).getFullYear()
2020
```
I disappeared down a bit of a rabbit hole trying to add a Content Security Policy to this site in pursuit of a perfect score from Mozilla Observatory.
Following both Nat and Tom mentioning the 2017 BBC News interview with Professor Robert Kelly that was famously interrupted by his children, Leo told me about a colleague using the footage as a Zoom virtual background. This was too good an idea not to copy, so I downloaded a version of the video with Professor Kelly removed and set it as my background. Sadly, I had to quickly turn it off when the meeting I was hoping to brighten with that entrance dance turned unexpectedly gloomy.
On an impulse, I bought Doom Eternal from the PlayStation Network. The jury is still out on whether blasting fiends from hell at an intentionally unrelenting pace is an activity that can be considered “relaxing”.
C is fascinated by anyone drinking. When he gets chance to drink water himself, he goes into a kind of joyful frenzy, finding every aspect of the experience wonderful. His enthusiasm for mundane things is an inspiration.

Weeknotes 25

2020-04-19T13:28:00+00:00

I spent part of the bank holiday weekend adding a new section to every post’s byline showing how long it has been since it was published, e.g.

By Paul Mucur, 7 days ago on Sunday 12th April 2020

As every post is a static page generated by Jekyll, these new relative dates are calculated entirely in your browser so they are always up-to-date, regardless of when I last generated the site. As a bonus, they update every minute so it should still be accurate even if you leave a post open for a while.

I was too lazy to come up with my own algorithm for turning lengths of time into words so I decided to port Rails’ time_ago_in_words to JavaScript.

You can view the full, unminified source code on GitHub (or enjoy the minified version).
Since clearing out our planter a few weeks ago, the sight of a rectangle of barren soil with a single, surviving chard plant has not been a cheery one. To improve matters and following Matt’s lead, I planted a clove of garlic in the hope it will grow.
E and I watched Bo Burnham’s “Eighth Grade” and loved it.

The movie doesn’t seem to be that well known despite receiving a lot of critical praise.

As Molly Ringwald put it:

I just saw [Eighth Grade] and thought it was the best film about adolescence I’ve seen in a long time. Maybe ever.
After watching the chefs of Bon Appétit make their favourite coffee, E suggested we buy a coffee grinder.

Asking for recommendations led me to the Wilfa Svart Aroma Coffee Grinder. Seeing Tim Wendelboe’s name in association with Wilfa made the decision easy given that E and I had visited his Espresso Bar in Oslo back in September 2015.

During our visit, E asked whether there were any pastries but was told they served no food as it might detract from the coffee drinking experience. While we disagree about the importance of pastries, I trust his judgement on coffee-making equipment.
We ended up buying the grinder from Square Mile Coffee Roasters along with a bag of beans.

The next day, I received the following email:

Dear Paul,

As you might know, we’ve recently launched our 25% GRINDATHOME discount code on all our grinders. We know, you just missed it, and to make up for it we would like to extend the discount code to your order by refunding the 25% on your recently purchased Wilfa grinder.

In a week of extremely stressful emails, this was an unexpected and very welcome kindness.
To celebrate the arrival of the grinder, I followed Blue Bottle Coffee’s Chemex Brewing Guide and, reader, it was a glorious cup of coffee.
Despite hearing about it on podcasts for a while, I hadn’t found a use for iOS Shortcuts.

When C wakes up unexpectedly in the night, I get up to reassure him and set a timer for 10 minutes to see if he will put himself back to sleep. After struggling to do this in the dark for over a month, I realised this might be the perfect use case.

I created a new shortcut to set a 10 minute timer with a single button press but it is still awkward swiping over to the widget in the Today View.
E and I started watching Alex Garland’s “DEVS” on BBC iPlayer. I’ll join others in highly recommending it.

While the title makes it sound like a comedy in the vein of “The IT Crowd” or “Silicon Valley”, it is a drama more akin to Garland’s previous work: “Ex Machina” and “Annihilation”.
It has been a trying week but as we retired for bed one evening, I accidentally kicked C’s walker. In its cheery voice, it said “keep going!”

Weeknotes 24

2020-04-12T15:24:00+00:00

In the spirit of repairing things, I took apart C’s broken toy arch, cursing to myself as I pried open the stubborn end caps with a palette knife. The telltale cracking sound of glue gave way to reveal critical bits of broken plastic, which I’ve now mended.

I was watching “The Repair Shop” throughout, imagining myself as a regular Will Kirk.
One of Phil Gyford’s weeknotes last week really made me rethink my own discomfort with “performative whimsy”. I won’t quote it here as it is worth reading it in its entirety.
After over a year of running our Pi-hole, my DNS over HTTPS proxy “Homer”, an AirPlay audio player and using it for Time Machine backups, my Raspberry Pi 1 Model B started failing under load.

I decided to treat myself to a brand new Raspberry Pi 4 Model B, going for a starter kit from The Pi Hut and a heatsink following Matt’s recommendation to keep it from overheating and throttling.

Everything is now up and running again and I took the opportunity to replace Homer with running our own Unbound DNS resolver after reading criticism of DNS over HTTPS.
I use Homebrew Cask to install software and noticed that running brew cask upgrade rarely does anything. However, forcibly calling upgrade on each installed Cask does:
```
$ brew cask list | xargs brew cask upgrade
# or, if you're running bash/zsh:
$ brew cask upgrade $(brew cask list)
```
This is because Homebrew Cask ignores applications that update themselves when upgrading but you can override this with --greedy:
```
$ brew cask upgrade --greedy
```
An update on our ant situation: while washing the dishes, I spotted a sole ant going for some bait on our window sill. Within seconds, an entire procession appeared so I could finally trace back where they have been coming from.

I discovered they were coming from a broken bit of sealant around a kitchen window and promptly filled it up with more silicone. After checking outside and seeing that our wedding rose was doubling as an ant walkway, I returned to check on the window sill and what I saw next is not easily forgotten.

Previously unseen ants found their exit blocked and started crawling out through the imperceptible gap between the glass windowpane and its frame.

I find that thought particularly enjoyable to turn over in your mind just before bed.
It was E’s birthday and our first during the Coronavirus outbreak.

Alongside a pack of Ocado’s “Happy Birthday” balloons, I decided to make Felicity Cloake’s perfect sticky orange cake (substituting Seville oranges with blood oranges) and a slap-up evening meal.

I decided to significantly complicate otherwise simple foods, e.g. instead of regular chips, I attempted Heston’s Triple Cooked chips, boiling and frying chipped potatoes the night before. While doing that, I was infusing white wine vinegar with tarragon and shallots for The Hawkmoor’s Stilton hollandaise, freeing time to simmer oranges for two hours in the morning while my sourdough bread baked in its Dutch oven.

Am I a Sous Vide Bro?
My new webcam arrived earlier than expected and we used it to host a birthday video conference in our living room. While I wasn’t sure how it would turn out (after all, people can’t have side conversations), it was surprisingly fun and it lifted E’s spirits to see friends in Aberdeen, Madrid and Dallas all in one place on an unusual birthday.

Weeknotes 23

2020-04-05T00:00:00+00:00

I was somewhat amazed to see Jamie Oliver’s “Keep Cooking and Carry On”, a cookery show made with Coronavirus (COVID-19) in mind. As if there was any doubt how quickly he put this out, he even makes a joke about people stockpiling pasta in the first episode.
Don’t worry, everyone: there is a new series of “The Repair Shop”. I suspect it was held in reserve for times of crisis.
Thanks to meticulous planning around C’s bed time, E and I managed to watch Yorgos Lanthimos’ “The Favourite” and loved it. Like Euan, I choose to believe the ballroom dance is real.
I haven’t really been taking advantage of being able to leave the house for one form of exercise a day but did venture out for a stroll with C. While it was a nice change to get fresh air and focus on something more than a few metres from my face, I found the silent negotiation of pavement space with other pedestrians strangely stressful.
Last week’s ant situation led me to fill conspicuous gaps in our kitchen worktops with silicone and—touch faux marble—we haven’t seen any more curious insectoid invaders.
My mum sent us a copy of “My First Signs”, a book of British Sign Language (BSL) signs for children. While there is debate about the effects of baby sign language, my sister is profoundly deaf so I’m keen to teach C some sign language regardless.

Reading the signs in the book makes it clear the sign language I use with my sister has morphed from BSL into our own private language. After sending her photos of the book, she immediately proposed her own sign for “toilet” as she is not a fan of how it might be confused with a gesture of defiance.
In a home baking first, E made a batch of 14 croissants over three days using a recipe from a Bread Ahead vienoisserie course she did three years ago.

We now start our day eating more than our daily recommended amount of butter.
For the past few weeks, I have been seeing bizarre advertisements on Instagram declaring “Bear Grylls [sic] TV Career is over!” because the “sources of his wealth became known”. Despite reporting these ads as scams to Instagram, they continue to flood in from various accounts with different images but always the same message.

I asked E what his real name is and she replied “I don’t know; Michael Microwaves?”

Weeknotes 22

2020-03-29T16:12:00+00:00

Now we are staying at home and have little outside space of our own, we’re having to get creative when it comes to getting fresh air.

Our solution? Lunchtime driveway picnics!

Not only do we get to enjoy some sunshine while remaining on private property but we also get to use our rarely-seen VonShef picnic backpack.
Following on from last week, I have been using Krisp to mute background noise on calls and my colleagues report that it works well.

If you’re interested in trying it out, here’s an invite from me that’ll give you an extra free month (and I’ll get two back in return).
After awkwardly wrangling my laptop lid a few too many times, I finally ordered a Logitech C920 HD PRO Webcam on several people’s recommendation. Sadly, it seems I am not the only person who had this idea and they’re out of stock on Amazon until the second week of May.
E has started doing yoga classes over Zoom. It is reassuring to hear that every class begins by negotiating volume levels and asking people to mute themselves.
As we continue to seek ways to entertain C at home, E discovered CBeebies’ “The Baby Club”. You can now find us occasionally singing its theme song or, its other earworm, “what’s in the bag?”
Skills Matter are back and have restored their videos. After previously lamenting their disappearance, I fired up youtube-dl as fast as I could to archive my talks.
I struggled with migraines at the start of the year but, thankfully, wearing glasses has helped a lot. Unfortunately, it also means I have developed some new, unconscious facial expressions. E is not a fan of these new tics and has taken to wordlessly scrunching up her own face as a gentle reminder.
During the seven years in our flat, I have done battle with various unwanted house pests: first, moths and then mice. This week we met a new enemy: ants.

My hands are now lacerated and stinging with vinegar solution thanks to pulling out our kitchen appliances and spraying potential entry points.

My true nemesis is not the ants themselves but our cursed dishwasher, propped up on three—not four—flimsy plastic legs, ready to buckle at the slightest provocation. I spent more time yesterday trying to re-level the blasted thing than I did creating barriers out of washing up liquid.
E and I caught the end of an episode of “The £100k House: Tricks of the Trade” and looked at each other as we contemplated Piers Taylor’s parting line:

But will DIY turn to DI-don’t?

Weeknotes 21

2020-03-22T00:00:00+00:00

I would typically euphemistically refer to “current events” but let’s be plain: COVID-19, the disease caused by the coronavirus SARS-CoV-2 has dominated our lives this week.
We’ve been rallying together with our neighbours and are sharing a precious online supermarket order. I intend to repay their kindness in bread made from my fortuitous flour stockpile.
With UK schools now closed, more people need to juggle childcare with remote working and I really want to make this work in my current team. I’m hoping “The Basecamp Guide to Internal Communication” and Hailley Griffis’ “Asynchronous Communication and Why It Matters For Remote Work” will come in handy as we move to work more asynchronously.
For those of you joining remote calls when you can’t really control the level of background noise (e.g. because your six-month old is really enthusiastic about eating his porridge), Alex recommended Krisp App to me. I haven’t tried it yet but might give it a go this week.
Now that I am working from home for the foreseeable future, I was surprised to discover that the daily decision whether to wear socks or not has become much more significant.
NetNewsWire, an RSS reader for iOS is now out. I admit I’m still struggling to get back into the habit of using an RSS reader versus mindlessly checking individual websites in turn.
Adrianna Chang’s “Refactoring Legacy Code with the Strangler Fig Pattern” looks like another great dive into tackling legacy code in a considered, incremental way.
Happy Mother’s Day! Or is that Mothering Sunday?
Cabel Sasser has been putting up Twitter polls for MIDI songs to play on his Roland Music Player. I was proud to contribute to Friday’s vote, resulting in Smash Mouth’s “All Star” and, as I enjoyed the tinny beats, E stunned me by singing along, revealing that she knew all of the lyrics.
Keeping the weekend spirit alive, yesterday I indulged in a little flower arranging, adding a little homemade flower food made from a squirt of bleach and a sprinkling of granulated sugar.
To celebrate the end of winter, I removed the Sellotape from our patio door and stepped out onto our tiny spit of outside space. Our planter resembled something out of “The Last of Us” so I cleared it of dead salad plants and tilled the soil. It wasn’t quite forest bathing but was surprisingly restorative.
I learnt that the strange outcrop of salt on the outside of our water softener is known as “salt creep” and tackled it with a combination of a sturdy brush and one of my three trusty cans of WD-40 (WD-40® Specialist® Water Resistant Silicone Lubricant, for those of you keeping track).
One night, as I put my Kindle down, E asked me what I was reading and I replied “Children of Ruin”. She scoffed, “of course; that’s a very Paul book title.”

Weeknotes 20

2020-03-15T00:00:00+00:00

Following recent events, I’ve been working from home all week.

Thankfully, I recently cobbled together a standing desk out of an old BenQ monitor perched atop Donald Knuth’s “The Art of Computer Programming” and Harold McGee’s “McGee on Food & Cooking”. Combining that with Tuple for pair-programming and Miro as a virtual whiteboard has made the switch from being co-located to being fully remote easier.

If you’re new to remote working, Alice Goldfuss’ “Work in the Time of Corona” has some good, practical advice about maintaining good mental health when working from home.
Somewhat late to the party, we finally watched—and were moved by—Hannah Gadsby’s “Nanette”.
As someone who often uses git commit --fixup to tell stories through my commits before code review, git absorb could save me a lot of time (via Andrew Gallant).
I baked a loaf of bread with less salt than usual for C to try (although he’d have to eat 120g of my usual recipe to meet his daily limit). Compared to his reaction to the kale and butternut squash on his plate, I’d say it was a rousing success.
While I refuse to stockpile, I do have 41 kg of bread flour in my kitchen from a spontaneous purchase back in January.
Tom and I were discussing the workplace superpower of “saying you’ll do a thing and then doing it” and he linked me to a Twitter thread by Lucy Prebble arguing competence is culturally undervalued.
I started playing “The Legend of Zelda: Link’s Awakening” and it is wonderful so far. Despite owning a SNES, the first Zelda game I really played was 1998’s “The Legend of Zelda: Ocarina of Time” on a Nintendo 64 so this is the first top-down perspective Zelda I’ve ever played.
I put up a mirror in our bedroom today. Another day, another Ian’s secure shoelace knot.

Weeknotes 19

2020-03-08T00:00:00+00:00

I followed Andy Baio’s recommendation and enjoyed listening to Reply All’s “The Case of the Missing Hit”. To avoid spoiling anything, I won’t say any more but it’s a good one.
Despite the current climate of hand washing and sneezing into elbows, I’ve managed to catch a cough. NHS 111 reassures me it is nothing more and I have been consuming a record number of lemons in the form of hot drinks. The timing is unfortunate but at least I don’t need to worry about scurvy.
Jean reminded me that the second season of “Ugly Delicious” is now on Netflix.

I loved the first season for exploring the complicated notion of “authenticity” in food and its related sociopolitical baggage. While waiting for more, I tried to watch “Breakfast, Lunch & Dinner” (also starring David Chang) but the first episode—where Chang gets high with his celebrity friend Seth Rogen—put me off. Thankfully, the first episode of the new season centres on Chang as a nervous dad-to-be so I’m hooked once more.

From Chang’s Instagram, I discovered there is ritual called “Zhuazhou” where a child picks an object that forecasts their future. I’m now planning to perform this ritual with C, placing before him a keyboard, a sword and a plastic dough scraper.
We successfully completed the 2,000 year old problem of moving C into his own room and so were finally in the market for some sort of baby monitor. As a real Internet of Things sceptic, I was adamant that we buy no device with the word “Smart” in its name. Thankfully, we bought the decidedly dumb BT Audio Baby Monitor 400 and it is wonderful in its simplicity.

Weeknotes 18

2020-03-02T00:00:00+00:00

Thanks to all who helped with my job search. I started work with a new client today, commuting on the Tube for the first time in eight years.
Speaking of the Tube, I hate that all news coverage of Coronavirus features a photo of someone wearing a face mask on public transport even when the news isn’t about the UK.
Every time I mention C to someone, I think of Euan’s mastery of undersharing. I remember first hearing he had a cat after working with him for four years.
Speaking of dad chat, I can’t wait to invest in a Father-Son Podcasting Microphone:

Well, it’s tough, you know? I mean, there’s no book you can read about being a dad.

Actually, there are several.
I’ve always had a soft spot for macOS indie developers. I have been a customer of LaunchBar, NetNewsWire (and its recent update), DropDMG, Airfoil, SuperDuper!, Interarchy, OmniGraffle, DiskWarrior, BBEdit, Transmit, SpamSieve and TextMate. This is partly why I enjoyed hearing about John Siracusa’s new apps and why I’m now trying Craig Hockenberry’s new notepad app—Tot—after reading John Gruber’s summary of it.

I secretly wish I had an idea for an app of my own.
I read Ryan Singer’s “Shape Up: Stop Running in Circles and Ship Work that Matters” and—despite unnecessarily dismissing existing agile practices in the foreword—found a lot I agreed with.

I’m increasingly convinced of the need for something like shaping before something is picked up by a development team. The struggle, as Singer writes, is getting the right level of detail: neither too abstract nor too concrete.

I was especially intrigued by the idea of appetite, not estimation as I’ve often found people unwilling to reveal truly how much they want a particular feature or project and drawing this out up front is extremely appealing.

The idea of pitch documents that can be reviewed asynchronously by members of the team sounds very similar to a Request For Comments (RFC) process (e.g. RFCs for changes to Rust, Yarn and Ember; see also Gergely Orosz’ “Scaling Engineering Teams via Writing Things Down and Sharing - aka RFCs”): something I am a huge fan of if you’re looking to introduce both rigour and equitability when proposing significant changes to teams and systems.
We took a trip to IKEA to get C a new cot and a Billy bookcase (one of which is sold every 5 seconds). I spent a lot of time in the kids’ section, fawning over the train sets, toy kitchen and its many, many accessories. Luckily for my wallet, C is still too young for them.
As one of my many “jobs to complete before returning to work”, I lovingly re-seasoned our wooden chopping board following J. Kenji López-Alt’s advice. There is something therapeutic about rubbing a wooden board with coarse salt and half a lemon.
The major thing E and I have been working on is getting C out of our bedroom and into his own room in the hope we can all improve our quality of sleep. Not wanting to confuse him with too much change at once, we first moved ourselves out and onto our sofa bed. When contemplating the next step, I was reminded of the 2,000 year old problem of the wolf, goat and cabbage.

In turn, this reminded me of a great London Computation Club meeting back in 2015: “The New Turing Omnibus, Chapter 58: Predicate Calculus” when Tom showed us how to use first-order logic to mechanically resolve the problem.

Bonus trivia: that was the same meeting that Chris announced his work on his programming language, Sentient.
E and I have many phrases that would be incomprehensible to others but a recent favourite is the following emoji-only message:

🤡⬇️

Of course, this translates to “the clown is down” which is a quote from The Simpsons’ “The Parent Rap” we’ve distorted to mean “our baby is asleep”.

Weeknotes 17

2020-02-23T00:00:00+00:00

My father-in-law was on The One Show discussing why a multi-storey car park in Liverpool burnt down, destroying 1,300 cars.
In an effort to be a better son-in-law, I extracted an animated GIF of his heroic introduction and, thanks to Collin Burger’s “How to make GIFs with FFMPEG”, ended up using this one-liner:
```
$ ffmpeg -ss 1499.5 -t 2.5 -i oneshow.mp4 -filter_complex "[0:v] fps=12,scale=480:-1,split [a][b];[a] palettegen [p];[b][p] paletteuse" hero.gif
```
Following a recommendation from Leo, I listened to an episode of “On the Metal” interviewing Kenneth Finnegan about starting his own internet exchange. Despite working on the web for 20 years, it quickly became clear I had no idea how the internet really worked.
I enjoyed Nicolas Carlo’s “Avoid rewriting a legacy system from scratch, by strangling it” about the Strangler Fig Application pattern. It reminded me of the time I failed to migrate the main database in a legacy system from MongoDB to PostgreSQL while trying very hard not to rewrite it from scratch.

When we finally cancelled the project, I wrote a long explanation to the whole company (like an Etsy PSA) called “The State of Custard”. Sadly, having left the company, that is now locked inside someone else’s Basecamp account.
As part of my job search, I have spent a lot of time editing my LinkedIn profile. In particular, I have changed my “headline” many times in an attempt to concisely explain what I do. At the time of writing it is “Consultant Technical/Engineering Lead”, shamelessly inspired by Andrew McDonough but I’m sure I will continue to agonise about it.
Having tempted fate by writing that our white noise snafus were over last week, our Bluetooth speaker simply stopped charging. Sadly, no amount of WD40 could fix the problem.

Weeknotes 16

2020-02-16T00:00:00+00:00

The job search continues.
I’m slowly getting used to my split keyboard thanks to some deliberate typing practice recommended by Maciej. I still managed to send an unfinished email instead of pressing the backspace key though.
I was intrigued by Oliver Burkeman’s “How to stop the spread of fake news? Pause for a moment” about continuous partial attention:

We tend to think of distraction as an all-or-nothing affair: either you’re concentrating successfully on something, or else you’ve been distracted by Twitter or Netflix yet again. But this is more like an erosion of attention, consistent with at least nominally remaining focused on the task at hand.

The article’s conclusion reminded me of the xkcd comic, “Listen to Yourself” (which was actually implemented).
Shane’s mention that he was trying to write a solver for the “Snake Egg Puzzle” in his weeknotes led me to recommend Chris Patuzzo’s Sentient programming language for the task and mention the problem to Chris. A few days later and Chris successfully wrote a program to solve the puzzle and generate new ones.
We’ve managed to drastically reduce the number of white noise snafus by using Guided Access on an old iPad. This way we can still enable and disable the susurration of a mild downpour but have the device ignore all accidental button presses. Thanks to Tom for drawing my attention to this feature of iOS.
I enjoyed Robin Sloan’s story of building a messaging app for his family, “An app can be a home-cooked meal”. For similar reasons, I’ve been enjoying hearing about John Siracusa’s new macOS apps written to meet his own needs on Accidental Tech Podcast (“Big Hole In the Middle” and “Day-One Cowboy”).
On a slightly less optimistic note, I also enjoyed Daniel Bogan’s interview with the creator of the Rust programming language, Graydon Hoare:

Software-wise I really wish we’d done almost everything differently. The focus has been on eyeballs and engagement, not like “does it work” or “can it resist attack by a bad actor”. Almost no modern software works well or is safe in any serious sense. It’s always on fire. A sinking ship with everyone frantically bailing. Swiss cheese. Pick your metaphor. So my dream setup would involve “software that has any sort of reliability”. Which probably requires rewinding time to before the personal computing era and enacting liability legislation or something.

Weeknotes 15

2020-02-10T00:00:00+00:00

After thoroughly enjoying my parental leave, I am now looking for contract work so if you know anyone who could make use of a former CTO/Technical Lead with 14 years’ experience developing Ruby and JavaScript in teams, please get in touch.
It took me a little while but I am loving “Outer Wilds” (and I am not the only one). I’m looking forward to watching Noclip’s “The Making of Outer Wilds - Documentary” when I’m done.
I nearly nodded my head clean off my neck while reading Basecamp’s “Until the End of the Internet” policy for their products.
For that reason, I was very excited to read the teaser for Basecamp’s new email service, “Hey” coming in April 2020.
The second episode of the Panic Podcast, “Pantscast”, is funny for admittedly puerile reasons but it made me laugh all the same. I’m not sure how I missed the app when it was first released.
Speaking of laughter, let me join many others by recommending “Frankie Boyle’s Tour of Scotland”. There are far too many lines to quote here but let’s just say I have a soft spot for his monologue about the beautiful Glen Coe.
I didn’t take any notes this week as I spent a large part of it blowing my nose and stinging my nostrils with a mixture of steam and Olbas Oil.
Just as blogging about blogging or podcasting about podcasting is a real faux pas, thinking about writing this week’s notes reminded me of a passage from Paul Murray’s “The Mark and the Void”:

Although technology has the capability now to record entire lifetimes, meaning that every moment may be pulled from the foaming sea of oblivion to the dry land of perfect recall, the mythic power of the photograph nevertheless relates to the future, and not to the past. Every recording conceals the secret fantasy of a future self who will observe it; this future self is himself the simulacrum, the persona ficta. He exists beyond time, beyond action, beyond need; his only function is to witness the continuum of the past, as he might observe the steps that brought him to godhood. Through this fantasy, time is transformed from the condition of loss into a commodity that may be acquired and stockpiled; rather than disappear ceaselessly into the past, life accumulates, each moment becoming a unit of a total self that is the culmination of our experiences in a way that we—biological composites who profligately shed our cells, our memories and our possessions—can never be.

Weeknotes 14

2020-02-03T00:00:00+00:00

Three days before attending a “starting solids” workshop at a local health centre, E made several batches of soup as I pressed a whole clove against the back of my mouth.
After two days of alternating doses of paracetamol and ibuprofen as rapidly as enclosed leaflets allowed, my dentist confirmed I had pericoronitis (do not click unless you have a stomach for pink, glistening medical photography) and sent me away with a prescription for antibiotics.
Brushing your teeth and doing a warm saline rinse every time you eat really makes one reconsider snacking. (Apologies to Nat and Tom for mentioning the wretched task.)
The elation felt when you can once again eat your morning porridge without first thoroughly mashing banana with a fork is exquisite.
I am clumsily typing this on a Microsoft Sculpt Ergonomic keyboard. The split keys make me realise the homegrown touch typing I’ve been practicing for the past two decades might not be technically correct.
I heartily second Tom’s recommendation for “Booksmart” especially as it is currently free for Prime Video subscribers in the UK.
I took Ze Frank’s “Human Test For People Who Work In An Office”:

Has a co-worker ever said—in the nicest way—“you look tired” when, in fact, you weren’t tired at all but now you feel bad because you know you look like crap for some unknown reason and somehow your co-worker gets credit for caring?
I thought of Ben Rosen being taught to use Snapchat by his 13-year-old sister as I posted my first Instagram story despite having previously railed at them. I hadn’t understood that people’s responses would be both private and much more personal than a shallow “like”.

Maybe I’ve been yelling at clouds all along.

Weeknotes 13

2020-01-26T00:00:00+00:00

Shoulders back and down.
E, C and I went to see “Uncut Gems” at baby cinema. As the stress and tension in the film grew relentlessly, I thought of my last two trips to the dentist about the burgeoning “vertical cracks” on my teeth from grinding.
For that reason, I found myself curled up in bed, a neoprene wrist support velcroed to one arm and a remouldable mouth guard separating my long-suffering molars. Not so dissimilar to Bart in “The Last Temptation of Homer”, laden with orthopaedic shoes, hair matted with medicated salve and voice hoarse with throat spray.
A friend caught me describing a room as “roasty toasty” and successfully deduced I spend most of my time in conversation with a four-month old baby.
Following Tom Stuart’s recommendation, I started watching Chris Morocco’s “Reverse Engineering”. In it, Morocco attempts to recreate dishes in two days after only tasting them while blindfolded.

From Gordon Ramsay’s Beef Wellington to Guy Fieri’s “Trash Can Nachos”, he does consistently well despite never having made these dishes before. There’s a mastery in his ability to quickly recognise various components and techniques in dishes, e.g. a Mornay sauce or a roux. Just don’t ask him to identify a specific cheese.
In an attempt to improve my posture, I’ve been doing yoga every morning under E’s tutelage. Naturally, I made the ritual my own by struggling to use Spotify to find appropriately soothing muzak on our aging iPad and ended up replacing our entire music setup with a Raspberry Pi running Shairport Sync. We can now stream any audio of our choice over AirPlay and I’m no longer allowed to procrastinate.
Speaking of which: shoulders back and down.
Among our desperate attempts to improve C’s sleep, we use “Rain Rain” and a small Bluetooth speaker to play the sound of running water throughout the night. When we’re not accidentally increasing the volume or fast-forwarding to the sound of a raging thunderstorm, I hear the strangest things in the white noise: snatches of a distant news broadcast, the wailing of some audacious post-rock and the hiss of Icelandic death metal on an AM radio.
The self check-in at my GP was out of order this week. After speaking with the receptionist, I settled in the waiting room and looked up to see a display showing a “Your Windows 7 PC is out of support” screen.
Watching Claire Saffitz soak toast and popcorn in water to flavour jelly beans has inspired me to get back into making ice cream. I enjoyed making Christina Tosi’s “Cereal Milk ice cream” but now I’m wondering what else I could steep.
People are never impressed when I talk about my collection of not one, not two but three types of WD40. Only I will know the deep satisfaction of spraying WD40 electrical contact cleaner into sticky game controllers.
One more time: shoulders back and down.

Weeknotes 12

2020-01-19T00:00:00+00:00

Since returning to London after our stay in a Glasgow tenement last week, I’ve yearned for the high ceilings and sheer space we enjoyed there. You can often find me standing in each room of our relatively pokey flat, a look of concentration on my face as I desperately try to rearrange furniture in my mind to somehow give us a single morsel more space. So far, I’ve only managed to do this successfully in our bedroom; disappointingly, everywhere else requires the removal of at least one major piece of furniture.
E asked me who Henry Cavill plays in his new Netflix show and I replied, “the titular Witcher.”
Tom Stuart’s video making Anna Jones’ cassoulet inspired me to make a similar video showing how I bake bread. Not having made a video since I was a teenager, I planned out the various parts I’d need to capture:
1. Making a starter
2. Making a leaven
3. Making the initial dough
4. Autolyse
5. Adding salt and water
6. Folding and bulk fermentation
7. Initial shaping
8. Final shaping
9. Proving overnight
10. Preparing the oven
11. Scoring
12. Baking in a Dutch oven
On reflection, I think I might have to pick something simpler for my first foray into moving pictures.
In an attempt to curb my tissue habit, E gave me a set of handkerchiefs for Christmas. I decided to clean them this morning and may have squirted bleach all over them before really thinking through my actions resulting in an accidental and not entirely pleasant tie-dye.
I’ve been going back through episodes of Adam Buxton’s podcast and especially enjoyed his interview with David Sedaris. It might even have convinced me to purchase my very first audiobook just so that I can hear more of Sedaris’ wonderful storytelling.
We had our electricity meter replaced with a “smart” one which automatically reports readings to our supplier every month. It came with an “in-home display” but my creeping sense of paranoia has:
1. Precluded me from connecting it to our WiFi network
2. Encouraged me to consign the thing to a dark cupboard/possible future in the bin
I suspect my reticence to have any form of “smart” device in our household is increasingly antiquated and quaint.
In a surprise to no one but myself, using my laptop on our sofa seems to have given me an exciting new repetitive strain injury. Sadly, wearing a neoprene splint, moving the computer to our 70s dining table and perching atop a pile of sofa cushions has not improved matters. In fact, I’m writing this week’s notes on my phone thanks to GitHub’s web interface though even this is making one of my hands feel suspiciously cold. Time for a bit of a break from musculoskeletal stress perhaps.

Weeknotes 11

2020-01-14T00:00:00+00:00

I redesigned this site in response to feedback from a former colleague that the old inline lists made it look like the styling had failed to load (never what you want to hear).

It doesn’t look too wildly different but I wanted it to resemble Safari’s Reader Mode and look good on mobile devices so I’ve cranked both the font size and the contrast throughout.

I was most thrilled to replace the old border-based link underlines so modern browsers can render better ones that respect descenders.

I’ve added an admittedly rather drab personal introduction to the home page after seeing much better introductions on Nat Buckley and Alice Bartlett’s sites. I might tweak it some more but perfect is the enemy of the good.
I felt rather sheepish after reading Basecamp removed Google Analytics so I removed all analytics from this site too. To be honest, I never checked the results and it feels increasingly hypocritical to run a Pi-hole at home and yet thoughtlessly track visitors on my own sites.

For similar reasons, I’ve also ditched Disqus comments from all posts. I’d disabled it for new posts but it felt a good time to remove them altogether.
C had his very first swimming lesson this week. It consisted mostly of E walking up and down the shallow end of the swimming pool singing nursery rhymes while pumping his fat little legs but did involve submerging him completely under water twice. My own father (who was a competitive swimmer in his youth) is particularly excited to hear C is taking to the water as he recently revealed how disappointed he was that I never truly mastered the front crawl or butterfly. Keep those mildly distressing revelations coming, people!
I was overjoyed to see two people publish their first weeknotes: Tom Stuart and my former colleague Shane. I continue my campaign to convince others to join in: perhaps you’ll be next?
We all went up to Glasgow over the weekend for E’s grandmother’s 94th birthday. This meant C met three grand uncles and three first cousins once removed (you could argue a second cousin was also present but is still in development). One of his grand uncles described C as someone who “perceives life as an adventure, not a threat” which I thought was rather lovely.

I can never remember how cousins work (first? second? how many times removed?) so I am very grateful for this excellent family tree diagram from Wikipedia.
On the train up to Scotland, I enjoyed Adam Buxton’s podcast with Joe Cornish where he mentioned Brian Eno’s concept of “screwdrivering”:

When I had Brian Eno on the podcast he was saying the impulse is always to add when you’re making music or making anything creative. You always feel—he calls it “screwdrivering”—you always feel as if you should be putting more in and that’ll be good but actually what you should be doing is taking things out.
Speaking of creative endeavours, a good friend of mine is currently making his first full-length feature film and I keep thinking about the sheer enormity of such a project. Sadly, I won’t be reprising my beloved character of policeman in the background.

Weeknotes 10

2020-01-05T00:00:00+00:00

Happy New Year! And double digit weeknotes too!

It’s amazing to me that it has been over a year since I nervously posted my 2018 Yearnotes having just left a company after eight years. At the time, I was deeply conflicted about my career and didn’t really know what to do. As I wrote then:

I’m honestly not sure what is next. After working continuously for a decade in London, I plan to take a break from keyboards and screens for a short while before seeing how I can best be of help to others, be that as a contractor or as a full-time employee.

My “break from keyboards and screens” ended up being a month of not enjoying playing “Red Dead Redemption 2” and then, after interviewing for a few full-time roles, landing my first contract as a freelance software developer through a friend of a friend. It was only for two weeks but it took a mere three days to convince me to commit and set up my own limited company, Ghost Cassette.

I was extremely fortunate and managed to work with two other companies before C’s turbulent arrival, getting the opportunity to work with technologies I had long wanted to use in production: e.g. HashiCorp’s Terraform and Facebook’s React.

Conspicuously missing from my yearnotes is that we knew C was on his way but it was far too early to share with anyone. During E’s pregnancy, we referred to him as “Tujol” due to a vivid dream I had where I found myself in a hospital, watching Turkish soap operas with a young boy of that name dressed in corduroy and with a small black felt triangle covering the end of his nose. It would later prove extremely difficult to stop affectionately referring to him as “Tuj” but it now seems a distant memory.

Against all odds, I did manage to run again though.
On New Year’s Eve, E and I continued our tradition of asking each other to reflect on our highlights and lowlights of the past year (and there was no shortage of either) before thinking about what we’d like to change in 2020.

While I had a few ideas, a common theme in my resolutions was my inability to do nothing or, more specifically, how often I squander the present worrying about something else (e.g. DIY jobs to be done, being a better person, comparing insurance providers, etc).
I’m a little late to the party but have been blown away by George the Poet’s “Have You Heard George’s Podcast?”.
Continuing my accidental “things you can do with AirPods” column, E and I listened to “A Grenfell Story” together while walking through the park thanks to iOS 13’s audio sharing which I’m sure made us look extremely antisocial. (I also can’t help but be reminded of the Microsoft Zune’s awfully-named “squirting”.)
On E’s mum’s recommendation, we listened to the director of the Centre for Neuroscience in Education at St. John’s College, Cambridge, Professor Usha Goswami discuss children’s language development on BBC Radio 4’s Woman’s Hour:

Rhythm is absolutely fundamental to the way the speech signal is structured. So when we’re talking, we’re creating a pressure wave in the air, it’s like a sound wave. It comes to the brain, the brain has its own intrinsic brain rhythms or brainwaves and effectively what happens is that, through automatic synchronisation processes, the brainwaves at different speeds align themselves with these energy changes in the speech signal that are also occurring at different speeds and that’s how you process the speech signal. It’s actually the most complex processing that your brain does, it’s more complex than vision.

Her explanation of the importance of “baby talk” (characterised by the strong-weak rhythm of words such as “mummy”, “daddy”, “baby”, “doggy”, etc.) was fascinating.

As a result, we’re singing “ye cannae shove your granny off a bus” with even more gusto (if such a thing were possible) and I’m trying to break my habit of referring to C solely as “pud” (short for “pudding”, of course).
As with my love of knot-related web sites, I thoroughly enjoyed a detailed analysis of which emoji scissors close. I file it alongside a Twitter thread rating every horse emoji and an entomologist rating ant emojis.

Weeknotes 9

2019-12-30T00:00:00+00:00

This past week of relative inactivity is probably best summed up by Sam Delaney’s “‘No tricks. No mantras. I just want to learn how to do nothing’: my quest to stay still”:

“Doing nothing won’t be comfortable at first, however you try,” he says. “You must deliberately put the time aside to do nothing, and have the discipline to get through it, even when it is boring or creates anxiety. You have to sweat it out and gnash your teeth doing nothing for a while, until you learn to cope. The only person you have to live with 24/7 is yourself, so you might as well learn how to do it.”

Though I did manage to let go of de-duplicating my sister’s photographs.
I was struck by the poetry of Samantha Harvey’s “‘It’s as if I’m falling from a 50-storey building’: a year without sleep”:

Why is it – she’s thinking – that she has to sit there day after day listening to patients who refuse to take responsibility for their own wellbeing? People in Syria can sleep with bombs falling, why can’t you sleep on your king-size mattress with your winter-togged duvet and your kelp-scented hair on a fake-down pillow under a bomb-free sky? What pea disrupts your sleep, princess? A passing Audi? What paucity and fragility of spirit has left you relying on drugs to do that which is the natural inheritance of all animals everywhere and for ever?
My guide to function composition in Ruby made Ruby Inside’s “Top 6 Ruby Items of 2019” which was an unexpected and very pleasant early Christmas present.
Speaking of gifts, we were very spoilt by others’ generosity including C receiving Julia Donaldson’s “The Gruffalo” translated into Scots by James Robertson so he can really embrace his heritage:

A moose took a dauner through the deep, mirk widd.
A tod saw the moose and the moose looked guid.
I received a brand new pair of shoes and what was the first thing I did with them? Why re-lace them with Ian’s favourite “Over Under” lacing, of course!
Having had a whale of a time playing it with colleagues earlier in the year, I finally managed to introduce Codenames to another group of people—my in-laws—via a free online version and tried to dazzle them with my extremely obtuse code-words (with mixed success).
There’s far too much Christmas television to cover but I was particularly pleased to see Mackenzie Crook’s “Worzel Gummidge” being well-received after how much we enjoyed his last series, “Detectorists”.
I managed to put on an astonishing amount of weight over the festive period so I’ve already started thinking about having some extremely predictable New Year’s Resolutions. The amount of junk food we’ve returned home with (Jaffa Cakes, After Eight Mints, chocolate Santa Clauses, Oreos, a slab of Christmas cake, mince pies, a tube of Smarties, two bags of chocolate coins, a box of Cadbury “Festive Friends”, etc.) only makes that all the more thrilling.
Finally, thanks to another gift for C, we’ve taken to regularly singing the Scottish classic “Ye Canny Shove Yer Grannie” and he loves it. My mum (and therefore his ill-fated “daddy’s mammy” in the rhyme) is less keen.

Ye cannae shove yer granny off a bus (Push push!)
No, ye cannae shove yer granny off a bus (Push push!)
No, ye cannae shove yer granny
‘Cause she’s yer mammy’s mammy
Ye cannae shove yer granny off a bus

Weeknotes 8

2019-12-24T00:00:00+00:00

My constant low-level consumption of gingerbread and “Homes Under the Hammer” over the past seven days has made it feel like the most sedentary week in recent memory (and this is coming from someone who was living off freezer food with a newborn not too long ago). Despite my only exercise being moving between sofa and dining chair, it’s been remarkably difficult to find time to write this week’s notes so I’ll be brief!
We spent the week at my in-laws and I wanted to repay their hospitality by whipping up a loaf of bread or two (as one does). This was yet another adventure in breadmaking away from all my various paraphernalia so I had to cobble together something in which to prove the loaves. Thankfully, a couple of bowls lined with a tea towel in a plastic shopping bag did the trick though you really do need to thoroughly dust the towels with some sort of rice or semolina flour. If not, you’re in for the nerve-wracking and delicate task of gingerly teasing apart cloth and cold dough.
A home luxury I’ve been surprised to miss as I idly scroll through my phone on relatives’ WiFi is my Pi-hole: the amount of internet advertising it has been dutifully hiding from me (particularly in iOS apps) is quite alarming.
We’re now at my family’s home and every year I fail to simply enjoy my time here as a guest and, instead, find some inane job with which to busy myself.

Previously, I’ve found myself precariously balancing on wooden beams in the attic, coordinating with my dad by barking questions down the hatch in order to painstaking identify and label various coaxial cables connected to the TV aerial.

This year, I have somehow found myself feeling responsible for ensuring all my sister’s photographs are stored in iCloud Photo Library and—crucially—are free of duplicates.

Presumably because she used to be an avid Picasa user, for some periods of time she has up to four copies of the exact same photo (sometimes in different sizes). I’m not sure how long this has been the case but I’ve been unable to let this rest and so, based on my recent adventures scripting Photos for macOS, I’ve been tinkering with cleaning them up automatically.

I’ve got as far as the following script but, in my heart of hearts, I know I should just let it be and move on with the holidays.
```
const Photos = Application("Photos");

const duplicatesAlbum = Photos.albums().byName("Duplicates");
const seen = new Map();

for (const photo of Photos.mediaItems()) {
  const [lat, lng] = photo.location();
  const date = photo.date();
  const description = photo.description();
  const fingerprint = `date:${date} lat:${lat} lng:${lng} desc:${description}`;

  if (!lat || !lng || !date) {
    continue;
  }

  if (seen.has(fingerprint)) {
    seen.get(fingerprint).push(photo);
  } else {
    seen.set(fingerprint, [photo]);
  }
}

for (let duplicates of seen) {
  const [biggest, ...rest] = duplicates.sort((a, b) => b.width() - a.width());

  if (rest.length) {
    Photos.add(rest, {to: duplicatesAlbum});
  }
}
```
I’m hoping that tomorrow’s traditional morning sherry will render me blissfully free of caring any more about this particular bit of busywork.
We finally managed to watch “Avengers: Endgame” but it took around six sittings and a lot of rewinding. It turns out 3 hour 1 minute action epics are trickier to enjoy with a three-month old.

Weeknotes 7

2019-12-16T00:00:00+00:00

We’re in Yorkshire spending time with our family in the run up to Christmas so please forgive the lateness of this week’s notes.
I was very excited to consult “Animated Knots by Grog” to fix a new loop of string onto a duster (a lovely bowline knot did the trick). I’d be remiss if I didn’t take this opportunity to mention my favourite web site: “Ian’s Shoelace Site”. It embodies exactly what I love about the internet: a true labour of love that is narrow in scope but extremely deep. I am yet to convert E (much to my annoyance as her laces come undone when we’re out walking) but I never leave home without tying a couple of trusty Ian’s secure shoelace knots.
Travelling north meant our very first significant road trip with C so we kitted out the car with a backseat baby mirror and tentatively set off up the M1. While we were nervous about C getting upset along the way, I got to indulge in one of my great, guilty pleasures in life: spending time in motorway services. I don’t just mean fancy, peaceful ones like Tebay but ones with appalling men’s health warnings on toilet stall doors presumably targetted at long haul drivers. I’m not sure what it is about them but I find them curiously relaxing places to be; there’s something about their transitory, incidental nature that is strangely freeing.
C met his great-grandfather and great-grand-aunt for the very first time who are 91 and 95 years his senior respectively (the latter worked at the Avro “shadow” factory in Leeds during the Second World War building Lancaster bombers). I’m mostly sharing this to use the term grand-aunt instead of great-aunt.
As we’re away from home in houses that actually have more than three rooms, we’ve been putting C down to sleep farther from us than ever before. We don’t have any sort of baby monitor but I’ve been using Live Listen with AirPods to turn my phone into a microphone and listen in. It has the added benefit of meaning I get to sit with a single earpiece in like some sort of cool dude.
Continuing my streak of recommending Felicity Cloake recipes in the past few weeks: we made a large batch of her perfect mince pies. To be honest, I’m just excited whenever we get an excuse to use our increasingly dusty bottle of orange blossom water.
I tried to make a loaf of bread with my sourdough starter (“Colin”) at my mum’s house but got a cracking migraine early in the process which meant a 40 minute autolyse unexpectedly became a 24 hour autolyse. Seeing as I had already used the ingredients, I decided to go ahead with the rest of the recipe and, while the dough was extremely difficult to handle, I was pleasantly surprised to end up with a decent loaf!

I’m increasingly inclined to agree with James Adam:

I bet if you just chuck all the ingredients in a bowl, stir it once, put it in an oven whose thermostat is triggered on and off by radioactive decay, you’ll get a normal loaf at the end of the process.
Spending time with my dad means catching glimpses of the two hour long miniature epics that are Turkish soap operas, or “dizi”: I never watch enough to understand the plot but you can guarantee there’ll be some smouldering looks and some fantastic facial hair.

Weeknotes 6

2019-12-08T00:00:00+00:00

Following last week’s beginnings of a cold for the newest member of our family, both E and I also succumbed, all three of us sniffling and coughing though none more pitifully than C. I took to using a large muslin square as a handkerchief which gave E great amusement every time I would unfurl it from a pocket while she relied on the restorative power of Felicity Cloake’s perfect hot toddies.
With little desire to leave the house while recuperating, we found ourselves drawn to the neverending stream of made-for-TV Christmas films shown during the day on Channel 5. An early highlight was 2017’s “The Sweetest Christmas”, which featured the following classic line:

I have to tell him how I feel the best way I know how: with gingerbread.

A bonus mention has to go out to 2017’s “Four Christmases and a Wedding” for its title.

All of this reminded me of Alonso Duralde’s “El Mero Mero De Navidad Christmas Minute” on “Who Shot Ya?” where the film critic would review as many Hallmark Channel Christmas movies as possible in a minute.
E baked a large batch of Helen Goh’s black treacle gingerbread with the treacle left over from our Christmas pudding adventures and, what’s more, even managed to use up the blight on our household that is greaseproof paper to wrap them up as gifts. The combination of large quantities of biscuits and a daily ritual of tea in the afternoon is proving disastrous for my waistline.
We voted by post for the very first time which I found rather exciting having only ever been to polling stations in the past. I was particularly intrigued by the design problem of explaining exactly how to fill out a postal ballot; not least because it involves two envelopes and not tearing off a slip of paper. Perforated paper seems like a classic affordance and the temptation to tear it, even having read the instructions multiple times, was huge.
I’ve been using FaceTime more and more so that distant family members can see C and have been startled when calls have been answered with unfamiliar faces present. It’s not quite David Foster Wallace’s predictions about video telephony from “Infinite Jest” but there’s something odd when you expect to see a loved one at the other end of a call and instead find yourself suddenly showcasing your child for a stranger.
Continuing the theme of things for which I lack mental preparation: we visited two nurseries for C as there are waiting lists of up to a year in our area. I was a little caught in the headlights as we were given tours and heard about the various facilities on offer (even being reassured that any information technology used in the nursery was divorced from social media).
Finally, our year-long nightmare of not opening a tiny cardboard door every day is over: it’s advent calendar season!

A former colleague of mine really made me question what is sacred when he took up happily consuming entire advent calendars as a snack in the run up to Christmas a few years ago. As I pleaded with him that such a thing was just not done, I realised that I had become a character in a Mitchell and Webb Look sketch about starving Antarctic explorers:

Lieutenant, there are some things that a true Englishman will never countenance. That is a Christmas pudding and we are saving it for Christmas, is that understood?

Weeknotes 5

2019-12-01T00:00:00+00:00

For a few years now, cookery shows on TV have been a guilty pleasure of mine: when your day has been a little too trying for the latest nail-biting thriller or existential drama, I find there’s no better way to unwind than to watch Jamie “glug” olive oil or “click off” lettuce leaves. This week I indulged in Rick Stein’s new series “Secret France” which was expertly summed up by Joel Golby in his review for The Guardian:

Every episode of every TV show Rick Stein has ever made is essentially like watching your dad have a particularly joyful Christmas Day, rubbing his hands with glee over a long-aged sherry, sitting upright to eat a large plate of continental sausages, regaling you with tales of childhoods gone by as soon as someone rolls a braised rabbit out in front of him, saying he’s too full for pudding but eating it anyway.
Having waited 12 years since the last episode, watching the announcement trailer for “Half-Life: Alyx” was a strange experience having reached a kind of heavy-hearted acceptance I would never step foot in City 17 again.

I last built a PC 15 years ago, specifically to play “Half-Life 2” and it is tempting to repeat that effort to meet the new game’s lofty VR requirements. That said, the Valve Index VR Kit alone is over £900 so I might just have to rely on the kindness of any VR early adopters I know.
I reluctantly attended my very first Parent and Baby Screening at the Barbican (or, as we like to call it, “baby cinema”) to see Noah Baumbach’s “Marriage Story”. I was mostly dreading trying to keep C happy and quiet throughout the film’s two and a quarter hours of running time but, with E’s help, we actually got to enjoy it. It’s not exactly light viewing for a married couple who have just had a child together but it’s well worth it.

Before the film started, I took C for a tactical change and was unexpectedly grilled by a member of staff when trying to re-enter the screen. I was slightly incredulous as I stood there coatless, holding C in my arms as I recited my row and seat number, imagining someone going to great lengths to smuggle their baby into a screening but then I thought of David Foster Wallace’s “This Is Water”.

The point is that petty, frustrating crap like this is exactly where the work of choosing is gonna come in. Because the traffic jams and crowded aisles and long checkout lines give me time to think, and if I don’t make a conscious decision about how to think and what to pay attention to, I’m gonna be pissed and miserable every time I have to shop. Because my natural default setting is the certainty that situations like this are really all about me. About my hungriness and my fatigue and my desire to just get home, and it’s going to seem for all the world like everybody else is just in my way. And who are all these people in my way? And look at how repulsive most of them are, and how stupid and cow-like and dead-eyed and nonhuman they seem in the checkout line, or at how annoying and rude it is that people are talking loudly on cell phones in the middle of the line. And look at how deeply and personally unfair this is.

As Wallace warned, it’s difficult to catch yourself in a moment where you feel personally inconvenienced but it’s good to consider—if only for a second—maybe this isn’t all about you.
Having said that, it is spectacularly annoying how many different, inconsistent screens one has to find the opt-out link on (no, I don’t want an Amazon Prime trial; no, I don’t want to try Kindle Unlimited) when resetting an Amazon Kindle these days.
In this week’s mostly pointless computer noodling corner: I was keen to get my zsh startup time down as much as possible and so used the “faster way to measure the total zsh startup time” from Benjamin Esham’s “How to profile your zsh startup time” to gradually whittle it down under 30 milliseconds.
```
$ time zsh -i -c echo
zsh -i -c echo  0.01s user 0.01s system 89% cpu 0.028 total
```
You can find my full .zshrc for macOS Catalina on GitHub but the main trick is to avoid spawning subprocesses where possible. For me, this meant switching my prompt to use the vcs_info function for showing git branch names and setting up my own alias for hub rather than using hub alias -s.

(I told you it was mostly pointless.)
As it is Christmas shopping season, I was roped in to do some technical support as a family member was struggling with an online retailer’s checkout. Fortunately, after listening to John Siracusa describe his latest bout of family tech support on Accidental Tech Podcast, I used Messages on macOS’ little known “Ask to Share Screen” functionality to guide them through the process (and subsequently got angry that an email address ending in .name was considered invalid 18 years after the domain was first introduced).
A member of our NCT group told us about an excellent episode of Adam Buxton’s BBC Radio 4 show “You’re Doing It Wrong” about parenting:

In this programme, we’re taking a look at parenting and proving that no matter how hard you try, you’re probably going to screw it up and raise a psychopath who hates you.

This quickly led to us listening to all other episodes covering topics such as work, diet and the environment.
I started reading Cixin Liu’s award-winning “The Three-Body Problem” and was immediately struck by how much of what I read is based in USA or Western Europe and what a refreshing change it is to read something set elsewhere (in this case, China).
I finally made a batch of Bon Appétit’s Best Chocolate Chip Cookies following Chris Zetter’s recommendation. I was warned they would spread when baking but I was still taken aback by how much! Luckily, they are delicious and even better the next day when they’ve firmed up a bit (eating them shortly after baking was basically like eating warm caramel-flavoured melted chocolate).
Speaking of Chris, we have now formed a daily bedtime routine of using “Several People are Solving” to do The Guardian quick crossword together. Chris wrote up how he made it in “Multiplayer Crosswords” and I highly recommend giving it a go with a fellow crossword enthusiast.
C has his very first cold so I’ll be spending today mostly worrying about him and audibly aching at every pitiful sneeze and cough.

Weeknotes 4

2019-11-24T00:00:00+00:00

I read Blake Hyde’s “The Danger of ‘Simplicity’” this week and wished I had written it. I’ve wanted to write about everything being a trade-off for years (I even have a draft from 2014 but sadly it only contains a bulleted list of technology buzzwords) but this is a fantastic, concise piece on the difficulty of pursuing simplicity in software.

In other words: every decision made in order to simplify a program will cost something.
I also enjoyed Dylan Araps’ “I’ve gone to great lengths for this silence” as relentlessly optimising web page markup is also a bit of a guilty pleasure of mine. Dylan goes much further than I have but I thoroughly appreciated their efforts.
On the topic of markup, I did not have fun trying to switch to using Google’s recommended structured data (based on JSON-LD) for this site. The Google Structured Data Testing Tool does not approve of these blog posts (marked up as BlogPostings) not having a publisher that is an Organization with a mandatory logo. Frustrated, I added only the markup that seemed to make sense but I risk the ire of Googlebot.
E made our very first batch of Christmas puddings using Felicity Cloake’s “perfect” recipe (prefixing search terms with “Felicity Cloake perfect” is the first thing I do before cooking or baking anything) and we each took it in turns (including C) to stir the mixture. Unexpected bonus baked good: we used up the rest of the recipe’s Guinness in Nigella Lawson’s Chocolate Guinness Cake and let’s just say that our daily regimen of tea and cake in the afternoon isn’t doing wonders for my weight. I’d be remiss if I didn’t also mention my former colleague Matt MacLeod’s Christmas pudding recipe which features both IRN-BRU and Buckfast and is always in great demand.
There’s a new series of BBC’s “The Repair Shop” and it truly warms my heart. Having already been made a fan of mechanical restoration videos following Simon Coffey’s recommendation, this takes the calming effect of watching a true expert methodically repair something and combines it with heartwrenching human emotion. E still teases me about being in floods of tears hearing the tale of “wheelie ted”.
We began watching Apple TV+’s “The Morning Show” as E has a free trial for a year and I was overjoyed to be reminded of the existence of Benjamin Clementine’s wonderful music in its theme song, “Nemesis”.
We’ve taken to singing nursery rhymes to C but I’m having to make alterations to the version of “The Wheels on the Bus” we found on Spotify as we’re not sure “the mummies on the bus go ‘chatter, chatter, chatter’” and “the daddies on the bus go ‘nod, nod, nod’” is a great depiction of our family life.
I’m not exercising great restraint in sharing pictures, videos and stories of C with friends (have I told you he’s taking to shouting “goo”?) but a poignant moment in a recent episode of BBC’s “Mortimer & Whitehouse: Gone Fishing” as Bob talked about the loss of his mother gave me pause:

When my mum died, so I had zero parents, do you know the thing I really miss? And I miss it nearly every Sunday, because I used to phone her every Sunday, so it always passes through my mind even now on Sundays is… It was lovely having someone to tell your kids’ little achievements. Someone who really was interested. No-one else cares less, do they, really? But you could phone up your mum and she loved hearing what they’d said or what they’d done. Now I’ve got no-one to tell, you know?

Weeknotes 3

2019-11-16T00:00:00+00:00

While we initially recoiled at the adverts for the Junior Bake Off, after reading that it is “the Bake Off we deserve” after this year’s disappointing series, we gave it a go and are now fully converted. Watching the contestants recover from baking disasters and help each other without hesitation is truly joyful in a way the regular series totally failed to be.
After my computer noodling last week, I decided to write up in detail how to use a Raspberry Pi for Time Machine and how to script Photos for macOS with JavaScript. It was good to write something in-depth again, especially for the first post as I wanted to ensure every step was both necessary and justified which was especially tricky given the rather opaque configuration necessary for Avahi.
Long-time host of the London Ruby User Group Skills Matter recently went into administration and their recordings of previous talks have sadly been disappearing from the internet. Among them, the video of my March 2015 talk “Exploring #to_proc” seems to have been been removed. Another day, another reason to think about the 21-year old “Cool URIs don’t change”.
Having said that, I changed the format of permalinks on this site to drop the .html extension from the URLs. Thankfully, there’s a redirection plugin for Jekyll to keep old links working.

Tom Stuart shared the following logic puzzle on the London Computation Club Slack:

Which answer in this list is the correct answer to this question?

All of the below.

None of the below.

All of the above.

One of the above.

None of the above.

None of the above.

I thought this would be an excellent opportunity to use Chris Patuzzo’s Sentient programming language to solve it:

bool answer1, answer2, answer3, answer4, answer5, answer6;

# 1. All of the below.
invariant answer1 == [answer2, answer3, answer4, answer5, answer6].all?(*self);

# 2. None of the below.
invariant answer2 == [answer3, answer4, answer5, answer6].none?(*self);

# 3. All of the above.
invariant answer3 == [answer1, answer2].all?(*self);

# 4. One of the above.
invariant answer4 == [answer1, answer2, answer3].one?(*self);

# 5. None of the above.
invariant answer5 == [answer1, answer2, answer3, answer4].none?(*self);

# 6. None of the above.
invariant answer6 == [answer1, answer2, answer3, answer4, answer5].none?(*self);

expose answer1, answer2, answer3, answer4, answer5, answer6;

Which returned a single answer: number 5!

$ sentient --number 0 answers.snt
{"answer1":false,"answer2":false,"answer3":false,"answer4":false,"answer5":true,"answer6":false}
{}

The story of Sentient is a great one and is told excellently by Chris and Tom in episode 4 of “Why Are Computers”.

A blog post circulated this week about solving puzzles with Amb so I thought I’d try using an ambiguous function/operator implementation in Ruby to solve the logic puzzle as well:

require 'amb'

A = Class.new { include Amb }.new

answer1 = A.choose(true, false)
answer2 = A.choose(true, false)
answer3 = A.choose(true, false)
answer4 = A.choose(true, false)
answer5 = A.choose(true, false)
answer6 = A.choose(true, false)

# 1. All of the below.
A.assert answer1 == [answer2, answer3, answer4, answer5, answer6].all?

# 2. None of the below.
A.assert answer2 == [answer3, answer4, answer5, answer6].none?

# 3. All of the above.
A.assert answer3 == [answer1, answer2].all?

# 4. One of the above.
A.assert answer4 == [answer1, answer2, answer3].one?

# 5. None of the above.
A.assert answer5 == [answer1, answer2, answer3, answer4].none?

# 6. None of the above.
A.assert answer6 == [answer1, answer2, answer3, answer4, answer5].none?

puts "Answer 1: #{answer1}"
puts "Answer 2: #{answer2}"
puts "Answer 3: #{answer3}"
puts "Answer 4: #{answer4}"
puts "Answer 5: #{answer5}"
puts "Answer 6: #{answer6}"

And it returned the same answer!

$ ruby answers.rb
Answer 1: false
Answer 2: false
Answer 3: false
Answer 4: false
Answer 5: true
Answer 6: false

A friend bought C a lovely toy firefly which constantly reminds me of the butterfly from Don’t Hug Me I’m Scared 3 (we’d even taken to calling it “Bertie the Butterfly”). Pesky bee.

Scripting Photos for macOS with JavaScript

2019-11-13T00:00:00+00:00

I recently imported hundreds of home videos from the past 20 years into Photos for macOS only to discover that none of the videos had the correct date and time. Thankfully, the correct date and time were encoded into each video’s filename, e.g. CLIP-2003-08-02 21;40;28.MOV but I didn’t fancy updating them all by hand.

As I use iCloud Photos to back up and store all my videos and photos, I needed to find a way to update individual items’ metadata that would also register with iCloud. As editing the underlying files stored inside the rather opaque Photos Library.photoslibrary wasn’t guaranteed to propagate changes to other devices, I decided the safest approach was to use Photos for macOS directly and rely on the JavaScript for Automation framework introduced in OS X 10.10. This gives us the ability to use JavaScript in place of the rather more arcane AppleScript to automate actions in macOS applications.

Running scripts
Using the Photos Dictionary
The final script

Running scripts

There are several ways to write these scripts but let’s start by using the Script Editor application found in /Applications/Utilities, starting a new script and choosing “JavaScript” from the “Script language” dropdown.

If we open up Photos and select a few items, we can run the following JavaScript in our Script Editor window to print out the selected item filenames:

const Photos = Application("Photos");

for (const photo of Photos.selection()) {
    console.log(photo.filename());
}

Note we’re using relatively modern JavaScript features here from ECMAScript 6 such as const and for...of, this is because JavaScript for Automation uses the JavaScriptCore from the version of Safari bundled with macOS and I’m running macOS Catalina 10.15 and Safari 13.

Running the script will print its output in the bottom pane of the Script Editor, hopefully listing the filenames of your selected items.

Alternatively, if you’re more comfortable on the command line, we can start a REPL using osascript:

$ osascript -il JavaScript
>> Application("Photos").selection()
=> [Application("Photos").mediaItems.byId("16AE4D88-E094-466E-B726-74CE4E876DE5/L0/001"), Application("Photos").mediaItems.byId("D11B78C8-82CD-472F-9039-DA148FEF8092/L0/001")]

Using the Photos Dictionary

While it is all well and good printing filenames, what else can we do with our items? To find out, we can open the Photos “Dictionary” (AppleScript terminology for its API documentation) using Script Editor by going to File > Open Dictionary... and choosing Photos.app from the list.

You can then choose JavaScript from the language dropdown and browse the Photos Suite to see the various objects and properties available to you. It’s useful to know that the root object you’ll be using is the Application which is described as follows:

Application Object [see also Standard Suite] : The top level scripting object for Photos.

ELEMENTS
contains containers, albums, folders, mediaItems.

PROPERTIES
selection (list of MediaItem, r/o) : The currently selected media items in the application
favoritesAlbum (Album, r/o) : Favorited media items album.
slideshowRunning (boolean, r/o) : Returns true if a slideshow is currently running.
recentlyDeletedAlbum (Album, r/o) : The set of recently deleted media items

“Elements” here refer to properties on the Application("Photos") object so you could print the filename of the first item in your whole library like so:

console.log(Application("Photos").mediaItems[0].filename());

However, while it appears that mediaItems here is an array, it is actually a function:

typeof Application("Photos").mediaItems
=> "function"

It just happens that these “elements” functions allow you to access specific elements by their index but if you want a true JavaScript array, you’ll need to explicitly call the property as a function:

// Missing out the () on albums causes an error
for (const album of Application("Photos").albums()) {
  console.log(album.name());
}

“Properties” are also functions: to read them, you need to call them. Note that, unlike “elements”, they do not support indexing and must always be called as functions:

Application("Photos").selection()
=> [Application("Photos").mediaItems.byId("16AE4D88-E094-466E-B726-74CE4E876DE5/L0/001"), Application("Photos").mediaItems.byId("D11B78C8-82CD-472F-9039-DA148FEF8092/L0/001")]

Read-only properties are documented as r/o (such as selection above) but properties without this mean they can be updated.

If we take a look at the documentation for a MediaItem, we can see the two properties we’re interested in:

MediaItem Object : A media item, such as a photo or video.

PROPERTIES
date (date) : The date of the media item
filename (text, r/o) : The name of the file on disk.

In order to read the filename and date, we know to call them as functions:

Application("Photos").selection()[0].filename()
=> "IMG_9295.HEIC"

Application("Photos").selection()[0].date()
=> Tue Nov 12 2019 08:28:34 GMT+0000 (GMT)

However, to set a property, we set it as we would any property in JavaScript (even though there is already a function with the same name):

Application("Photos").selection()[0].date = new Date(2001, 0, 1, 12, 0, 0)
=> Mon Jan 01 2001 12:00:00 GMT+0000 (GMT)

The final script

With this and a little help from JavaScript Regular Expressions, we’re now able to write a script to update my videos:

// Regular expression to extract date and times from filenames
const pattern = /CLIP-(\d{4})-(\d{2})-(\d{2}) (\d{2});(\d{2});(\d{2})\.MOV/;

// Pull out all the items from my current selection that match the above pattern
const clips = Application("Photos").selection().filter(item => pattern.test(item.filename()));

// For every matching clip...
for (const clip of clips) {

  // Take the current filename
  const filename = clip.filename();

  // Extract the date and time from the above filename
  const [, year, month, day, hours, minutes, seconds] = filename.match(pattern);

  // Create a new Date object for the above (note months are 0-indexed in JavaScript)
  const newDate = new Date(year, month - 1, day, hours, minutes, seconds);

  // Log the change we are about to make for posterity
  console.log(`Setting ${filename} date to ${newDate} from ${clip.date()}`);

  // Update the date, persisting the change to iCloud Photos
  clip.date = newDate;
}

Using a Raspberry Pi for Time Machine

2019-11-12T00:00:00+00:00

Ever since Mac OS X Leopard, connecting an external storage device to your Mac will ask you if you want to use that storage to automatically back up all of your files with a feature called Time Machine. However, Time Machine can also be configured to back up to storage elsewhere on your network and not directly connected to your computer.

We’re going to use a Raspberry Pi (I used an old Model B) as a cheap alternative to a dedicated Network-attached Storage (NAS) device for use with Time Machine. I’m going to use an external hard drive connected via USB but the same principles should apply to any storage available to the Raspberry Pi.

Software dependencies
Readying our storage
Configuring Samba
Configuring Avahi
Accessing your backups in the Finder

Software dependencies

First, ensure you’re using a recent version of Raspbian so we can rely entirely on Debian packages already built for your Raspberry Pi and avoid compiling software ourselves (which can take quite some time on older models). If not, please see the official documentation for installing a Raspberry Pi operating system image on an SD card. At the time of writing, I used the September 2019 version of Raspbian Buster with a kernel version of 4.19.

With Raspbian up and running, let’s install the two packages we’re going to need: Samba and the Avahi daemon.

pi@raspberrypi:~ $ sudo apt install samba avahi-daemon

Samba is an open source implementation of the SMB file sharing protocol which is officially supported by Time Machine for backing up over a network. This is all we really need to back up to storage attached to our Raspberry Pi but we can make it slightly easier to set up on the macOS end by also using Avahi’s implementation of the mDNS/DNS-SD protocol (sometimes called “Bonjour”) to make our Time Machine share automatically discoverable on our network. This way we don’t need to manually connect to our Raspberry Pi from our Mac, it will automatically appear both in the Finder’s sidebar and in the Time Machine System Preferences.

It’s important to have a recent version of Samba so that we can benefit from the latest support for macOS-specific extensions. At the time of writing, the versions I used were Samba 4.9.5-Debian and avahi-daemon 0.7.

Note that we are not using Netatalk as we want to use SMB rather than AFP for file sharing as that seems to be the preferred file sharing protocol for network-attached storage in Apple’s current documentation.

Readying our storage

With our dependencies in place, let’s ready our storage. If you have this set up already, please feel free to skip ahead to Samba configuration.

I use a cheap USB enclosure with a 500 GB hard disk inside and I want to use the whole disk for Time Machine. Connecting it to my Raspberry Pi makes it instantly discoverable with blkid which will print attributes for all connected block devices:

pi@raspberrypi:~ $ sudo blkid
/dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="3725-1C05" TYPE="vfat" PARTUUID="d565a392-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="fd695ef5-f047-44bd-b159-2a78c53af20a" TYPE="ext4" PARTUUID="d565a392-02"
/dev/sda: UUID="e613b4f3-7fb8-463a-a65d-42a14148ea65" TYPE="ext4"
/dev/mmcblk0: PTUUID="d565a392" PTTYPE="dos"

The key entry here is /dev/sda: that’s my USB drive and its unique UUID I can use when configuring it.

As the TYPE="ext4" in the output reveals, I’ve already formatted it with the default Linux ext4 journaling file system but let’s do it again for good measure:

pi@raspberrypi:~ $ sudo mkfs -t ext4 /dev/sda

We’re making a trade-off here: using ext4 means we won’t be able to plug our drive directly into a Mac and read and write to it but we benefit from the reliability of a journaled file system. If we used Apple’s HFS+ instead, we would be able to plug our drive directly into a Mac but we would have to disable journaling thereby losing write safety. As the integrity of our backups is paramount and as long as we are happy doing both back ups and restores over our network, ext4 seems a good compromise between convenience and safety.

Note this isn’t an encrypted file system so anyone can read anything written to it. We could use Linux Unified Key Setup to avoid this but Time Machine will store any network backups in a sparse image which can itself be encrypted. To simplify setup, let’s rely on Time Machine’s encryption to keep our backups secure and leave our underlying storage unencrypted.

Now we want to mount this filesystem on our Raspberry Pi so that we can access it like any other directory. Let’s do this and ensure that the mount persists whenever the Raspberry Pi restarts by appending an entry to the fstab:

pi@raspberrypi:~ $ echo 'UUID=e613b4f3-7fb8-463a-a65d-42a14148ea65	/media	ext4	noexec,nodev,noatime,nodiratime	0	0' | sudo tee -a /etc/fstab

This instructs our Raspberry Pi to mount the drive identified by the given UUID (which we got earlier when calling blkid) at the directory /media as an ext4 filesystem with a variety of flags:

noexec: do not allow direct execution of any binaries on the mounted filesystem as it’s only going to be use for backup storage;
nodev: do not interpret character or block special devices on the filesystem;
noatime: do not store access times on this filesystem;
nodiratime: do not store directory access times on this filesystem.

(Update: I previously recommended using the sync flag but Dries Oeyen pointed out this typically has a big impact on performance and the ext4 filesystem doesn’t support it, instead preferring the use of commit=nrsec to specify a number of seconds after which data and metadata will be synchronised.)

The two numbers at the end of the line instruct your Raspberry Pi to ignore this filesystem when using dump and not to check it with fsck when rebooting.

I’m using /media because I have no other storage attached to my Raspberry Pi but you could use any directory of your choice.

Now we can actually mount the filesystem based on the definition we just added to our fstab:

pi@raspberrypi:~ $ sudo mount /media

This allows us to now read and write files on our freshly formatted USB drive.

Let’s create a separate user for storing backups to match the one on my Mac:

pi@raspberrypi:~ $ sudo adduser mudge

Let’s store all our backups in a new directory and assign ownership of the entire disk to this new user:

pi@raspberrypi:~ $ sudo mkdir /media/backups
pi@raspberrypi:~ $ sudo chown -R mudge: /media

As I’m using a spinning disk and Time Machine doesn’t run constantly, it’d be nice if we could put the disk in standby after 10 minutes of inactivity. To do this, we’ll need to install hdparm to set a standby (spindown) timeout in 5 second increments, again identifying our disk by its UUID:

pi@raspberrypi:~ $ sudo apt install hdparm
pi@raspberrypi:~ $ sudo hdparm -S 120 /dev/disk/by-uuid/e613b4f3-7fb8-463a-a65d-42a14148ea65

We can make this permanent by adding the following stanza to /etc/hdparm.conf:

/dev/disk/by-uuid/e613b4f3-7fb8-463a-a65d-42a14148ea65 {
	spindown_time = 120
}

Configuring Samba

Now that we have storage available to share with our Mac, let’s set it up with Samba so that it will work with Time Machine.

To do this, we need to add a new share definition for our directory at /media/backups owned by the user mudge to our Samba configuration in /etc/samba/smb.conf:

[backups]
    comment = Backups
    path = /media/backups
    valid users = mudge
    read only = no
    vfs objects = catia fruit streams_xattr
    fruit:time machine = yes

This creates a new share called backups which can only be accessed by using the mudge username and password.

The key configuration for Time Machine are the last two lines which enable Samba’s enhanced macOS interoperability module.

So that we can connect to the share as mudge, we need to explicitly add that user to Samba’s password file and set a password with smbpasswd:

pi@raspberrypi:~ $ sudo smbpasswd -a mudge

We can now use this username and password from macOS to connect to our backups share.

With these changes in place, we can now run testparm to check our configuration is free of errors and reload our Samba configuration:

pi@raspberrypi:~ $ sudo testparm -s
pi@raspberrypi:~ $ sudo service smbd reload

At this point, we could manually connect our Mac to the share by choosing Connect to Server... or pressing Command-K in the Finder, entering the IP address of our Raspberry Pi, e.g. smb://192.168.1.104/backups and authenticating with our user credentials. This would then become available in the Time Machine Preference Pane as a backup destination when you choose “Select Disk…”.

However, we can make this even easier by also setting up autodiscovery of our share with Avahi.

Configuring Avahi

Now that Samba is configured, let’s use Avahi to advertise it to Macs on our network. While Samba can do this itself if it is built with mDNS support, the version currently available for Raspbian does not have this functionality so we’ll have to do it ourselves based on the Samba source code.

We’re going to have to add a new Avahi service to /etc/avahi/services/samba.service which describes the two things described in Apple’s documentation for advertising Time Machine availability through Bonjour:

That we are running an SMB server on port 445;
That we have an SMB share point available for Time Machine.

As a bonus, we’re also going to advertise our Raspberry Pi as a now-defunct AirPort Time Capsule so that it appears as such in the Finder.

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">%h</name>
  <service>
    <type>_smb._tcp</type>
    <port>445</port>
  </service>
  <service>
    <type>_device-info._tcp</type>
    <port>9</port>
    <txt-record>model=TimeCapsule8,119</txt-record>
  </service>
  <service>
    <type>_adisk._tcp</type>
    <port>9</port>
    <txt-record>dk0=adVN=backups,adVF=0x82</txt-record>
    <txt-record>sys=adVF=0x100</txt-record>
  </service>
</service-group>

Let’s take a look at each service individually:

<service>
  <type>_smb._tcp</type>
  <port>445</port>
</service>

This advertises our SMB server on port 445.

<service>
  <type>_device-info._tcp</type>
  <port>9</port>
  <txt-record>model=TimeCapsule8,119</txt-record>
</service>

This advertises our Raspberry Pi as an AirPort Time Capsule but you can use any device model name found in /System/Library/CoreServices/CoreTypes.bundle/Contents/Info.plist, e.g. MacBook, Xserve, Windows, etc.

We use port 9 for both this and the next service as it is the port used by real Time Capsules and any data sent to it is discarded by definition.

<service>
  <type>_adisk._tcp</type>
  <port>9</port>
  <txt-record>dk0=adVN=backups,adVF=0x82</txt-record>
  <txt-record>sys=adVF=0x100</txt-record>
</service>

This is the crucial part that advertises our specific backups share as a Time Machine-compatible “AirDisk” and it consists of two parts:

<txt-record>dk0=adVN=backups,adVF=0x82</txt-record>

Advertise a share named backups (this must match the name of your Samba share) which has the following two AirDisk volume flags set:

0x0002: SMB is supported on this volume;
0x0080: Time Machine should allow this SMB volume as a backup destination.

Together, these make the single hex value string of 0x82.

The second part sets system-wide volume flags of 0x100 so that connecting to our share prompts for a username and password:

<txt-record>sys=adVF=0x100</txt-record>

With this service file in place, Avahi should automatically pick up the configuration change and your Raspberry Pi will now appear in the Finder and in the Time Machine Preference Pane when you choose “Select Disk…”.

Ensure that you check “Encrypt backups” when selecting the disk and wait for your first backup to complete!

Accessing your backups in the Finder

Note that choosing your Raspberry Pi in the Finder will initially connect as a guest (as indicated by Connected as: Guest in the Finder window) and your backups share will be visible there. However, attempting to open the share will fail with a puzzling error message stating that your share can’t be found. This is because you need to explicitly connect as the appropriate user (mudge, in my case) by choosing Connect As... in the Finder.

Once you do that, you should be able to see your backups stored as a sparse image with a .backupbundle extension.

If you have any trouble connecting (especially if you have changed your user credentials at all), it’s worth using the Keychain Access application in /Applications/Utilities to check for any cached network passwords and delete old entries. You may also need to forcibly relaunch the Finder by holding down the Option key and right-clicking on its icon in the Dock and choosing the Relaunch option at the bottom of the resulting menu.

Weeknotes 2

2019-11-10T00:00:00+00:00

While I was conflicted about publishing my first weeknotes last week (were they too self-indulgent? are they interesting to anyone else?), I received some nice words of encouragement and so on with this week’s!

A recurring lesson in the past few weeks has been the importance of the right tool for the job:
1. Our kitchen tap began very occasionally dripping so I decided to replace its cartridges. However, despite my best efforts with a series of adjustable spanners and a hastily-purchased set of box spanners, all I managed to do was chew up the metal of the existing cartridge. Frustrated and close to admitting defeat, I remembered that I bought a set of socket and ratchet spanners to replace a car brakelight many years before and—lo and behold—that worked straight away and I was able to swap out the cartridges in mere minutes!
2. While I was engaged in this exciting DIY adventure, E was having a customer service battle with our water company as we’d been sent a bill ten times our usual amount. Adamant we hadn’t suddenly taking to bathing around the clock, the company finally agreed to send out an engineer to investigate. Ever keen to see the workings of such things, I accompanied them to the street as they discovered they have been reading the wrong water meter for us for six years. In trying to identify our real meter, they accidentally broke the stop cock on it, leaving us without any running water. A second engineer promptly came to fix it using a simple-looking tool known as a stop cock key and complained to me that if only they sent all engineers out with one, there would be no need for him to have visited. I nodded politely.
3. As discussed last week, I’ve been baking some bread again and experimented with putting dough on baking parchment before lowering it into a Dutch oven. At least, I thought I was using baking parchment but, thanks to swearing my head off while trying to pick apart an unholy fusion of paper and dough, I have since learnt that “greaseproof paper” and “greaseproof and baking paper” are crucially different.
Reading Vincent Foley’s “Lesser Known Coding Fonts” sent me down a bit of a rabbithole trying to find the perfect monospaced typeface, particularly after watching the recent episode of “Abstract: The Art of Design” about typeface design with Jonathan Hoefler. In the end, I abandoned this quest for what it was: an act of pure time-wasting.
It has been five years since Casper Kelly’s “Too Many Cooks”. I continue to pay tribute to it in my own small way by having Rush Coil’s NES cover of it as my ringtone.
C having his 8-week vaccinations and talk of preventing fevers led me to Jenny Kleeman’s “Why parents are addicted to Calpol” so it was with an unearned sense of pride I marched into our local pharmacy and asked for whatever generic, liquid baby paracetamol suspension they had.
It has been quite the week of computer noodling for me:
- As I manage my family’s email accounts, I decided to spend some time implementing Domain-based Message Authentication, Reporting & Conformance (DMARC) for my domains. As a side-effect, I now get a daily report of the times people are sending emails pretending to be from mudge.name. These attempts should now go straight to spam but it’s still a little disconcerting.
- Following last week’s adventures with backups, I decided to set up a network-based Time Machine backup.
  
  The first step was to find a hard drive big enough to store backups and, in my search, I uncovered old videos of myself dating back to 1998. This led me to compress all these 21-year old videos using Automator and then import them into Photos. However, none of the videos had the correct date on them.
  
  Thankfully, I noticed the videos did contain their real date and time in their titles, e.g. CLIP-2003-08-02 21;40;28.MOV. As there were over 700 of these videos, updating them manually didn’t seem a smart option so I decided to look into scripting Photos via the Mac Automation Scripting Guide. While it seemed like I might have to learn a whole new programming language, I vaguely recalled that you can now script macOS applications with JavaScript.
  
  I wondered if I could use any modern JavaScript features and Matt MacLeod pointed out “ES6 Features in JXA” which explains that you can use any JavaScript features supported by the version of Safari bundled with your OS.
  
  I’ve since written up how I scripted Photos with JavaScript to update all the videos in one go.
- Videos of a much softer-faced and squeakier-voiced me safely stored in iCloud, it was time to get that Time Machine set up on my Raspberry Pi. This resulted in a new post about how to use a Raspberry Pi for Time Machine.
C is now ten weeks old and I’ve been thinking about something from Michael Please‘s BAFTA award-winning short film, “The Eagleman Stag”: yesterday was less than a 12,500th of my life but, for C, it was over 177 times more.
Tom Stuart said I might regret the semicolons in last week’s notes so I hope you enjoy this week’s luxurious full stops instead.

Weeknotes 1

2019-11-04T00:00:00+00:00

Following a trail already blazed by Nat Buckley, Alice Bartlett and Chris Zetter and at the very real risk of over-sharing, here are my first weeknotes:

I thoroughly enjoyed playing Lucas Pope’s multi-award-winning “Return of the Obra Dinn” which is now available for the Nintendo Switch, a game I finished while walking C back to sleep at 6 am. From the use of music and text alone, you can tell this is from the creator of Papers, Please;
I keep wanting to talk to people about how fantastic “Succession” is but so few people have heard of it, let alone seen it. I assume this is because it’s not on Netflix, Prime Video or the regular telly in the UK but I hope it gets the audience it deserves;
On that note, I was simultaneously amused and dismayed to read that Connor Roy’s “hyperdecanting” is a real thing coined by Nathan Myhrvold;
While I can’t really make time for London Computation Club any more, I have been thoroughly enjoying the #bread Slack channel, learning how to troubleshoot bad bakes and experiment with long autolyse times, baking from cold and scoring with scissors (in fact, as I write this, I have some dough in bulk fermentation);
Having C in my life has predictably altered my perspective so it’s been with some trepidation that I’ve been listening to Nick Cave and the Bad Seeds’ “Ghosteen” given its subject matter but it is rather wonderful;
On a similarly morbid note, I’m considering re-reading George Saunders’ “Lincoln in the Bardo” but I am having too much fun learning about the Shrike at the moment;
As a lot of care for C in the first few weeks involved staring directly at my phone, I went through a bit of a panic after reading Nir Eyal’s guide to avoiding digital distraction and followed quite a bit of advice on how to configure your iPhone to work for you, not against you especially around restricting notifications. I’m not entirely convinced it has helped though as I now compulsively check applications that may notify me and moving applications onto secondary screens and into folders hasn’t really increased friction significantly;
While I have very rarely opened my laptop in the past two months, I couldn’t resist the urge to format it and do a clean install of macOS Catalina (despite the growing criticism of Apple’s recent software quality). I decided to use it as a bit of a Disaster Recovery test to see how easily I could recover my files if my hard drive was lost or failed.

The most straightforward recovery were my photos and files stored in iCloud and passwords in our 1Password family account (though downloading over 15 years of photos took quite some time). Less successful was the restoration of my music library and many software projects from a full Backblaze backup as I hadn’t realised that Backblaze does not backup file permissions (including whether a file is executable or not). Michael Tsai wrote about Backblaze’s shortcomings in this area over five years ago and I ended up restoring those files from a Time Machine backup I’d done as a failsafe instead (I would have used David Nanian’s SuperDuper! but it doesn’t support Catalina yet).

Following advice from others, I’ll probably relegate Backblaze for total disasters and introduce a network-based Time Machine backup, most likely connected to my Raspberry Pi;
Thanks to Nat Buckley mentioning Claire Saffitz in their weeknotes, I binged quite a few episodes of Bon Appétit’s “Gourmet Makes” and ended up baking Dorie Greenspan’s “World Peace Cookies” mentioned in the Oreo episode and they were the best home-made cookies I’d ever made. Chris Zetter recommended Kate Davis’ “Brown Butter and Toffee Chocolate Chip Cookies” recipe so I might have to give that a go soon;
I’ve spent an unhealthy amount of time reading and talking about why the latest series of the Great British Bake Off went wrong but I was overjoyed to discover a recent episode of “Getting Curious with Jonathan Van Ness” about series 5 winner, Nancy Birtwhistle;
I have a very long, unfinished blog post about React Hooks and my attempts to implement debouncing with them that I really should finish and publish on my consultancy website (especially since Chris Patuzzo mentioned it in his latest blog post) but it’ll have to wait for another day;
Reader, you won’t believe me but I ran for the first time since I got married and, what’s more, it was my first time around the park we’ve lived next to for over six years;
I have now been a dad for nine weeks.

2018 Yearnotes

2019-01-02T15:36:00+00:00

Last year, inspired by Nat Buckley’s “2017 Yearnotes” and Julia Evans’ “2017: Year in review”, I wrote my own “2017 Yearnotes”. Despite the fact that year saw me getting engaged at the top of Goat Fell on the Isle of Arran and later married (not before completing the frankly colossal act of yak shaving of writing my own PHP web framework), I never published the piece. Reading it now, it’s somewhat strange to read my reflections on a lack of separation between my career and my identity and that my two hopes for 2018 were as follows:

Figure out a healthier work-life balance;
Start running again.

I flat-out failed at the latter and, as for the former, well, you could say I made some progress in that area but not in the way I could have predicted.

And so, some reflections on 2018.

Travelling

In February, I travelled to Tromsø, Norway for a week-long holiday exploring the breathtaking Lofoten islands.

It was the first time I’ve ever been within the Arctic Circle and the combination of natural beauty and outside temperatures of -13 °C made for a heady mix of awe and fearful respect. In retrospect, our desire to take it all in and reach Å within seven days meant we spent too much time on the road but I did get my first exposure to lane assist technology which was frankly terrifying on icy roads.

One of the intentions of the trip was to break our daily habit of being glued to our phones and to break our dependence on constant stimulation from the internet. As with the rest of the year, I admit I wasn’t wholly successful in this endeavour though the place itself was wonderful.

Not long after returning from Norway, we travelled to Cheshire for the first of several weddings this year (the rest would later take us to Liverpool, Shoreditch and Durham) which—in a change from last year—are all-the-more enjoyable when you’re not planning your own.

2018 saw two weekend breaks: the first to sunny Margate and the second, my very first weekend break abroad when we went to Porto, Portugal. The latter gave us 48 hours to spelunk in port caves and seek out pastéis de nata in very good company.

In further attempts to “get away from it all”, we journeyed up to Scotland several times, visiting family in Aberdeen and Glasgow and, in search of true respite, retreating to remote parts of Skye (where I ended up spending hours playing Hollow Knight on the Nintendo Switch).

You may have noticed a recurring theme in the aforementioned journeys: attempts to dramatically switch context, to shake off daily routine and to find some peace outside of daily life in London.

Open source

I began the year by open sourcing the aforementioned PHP web framework I wrote in order to get married: Engine.

$ composer create-project mudge/engine-skeleton:dev-master my-project
$ cd my-project
$ php -S localhost:8080 -t public

The core idea of Engine is that modelling a web application as a pure function that takes in a request and produces a response as an output (i.e. f(request) = response) no longer feels accurate. Perhaps it is better to think of a web application as something that takes both a request and a response as input and achieves its desired goal by using the response for side-effects (think of the response as a sort of open socket). This way, you can easily send your response in bits or stream responses indefinitely.

It was also a great excuse to take advantage of PHP 7.1’s type system and baffle people who haven’t seen any recent PHP. For example, here is some real source code from our wedding site:

<?php
declare(strict_types=1);

namespace Wedding;

final class TravelController extends PrivateController
{
    public function index(): void
    {
        $this->requireUser();

        $this->render(
            'travel.html',
            ['lodges' => $this->user->isStayingInLodges()]
        );
    }
}

In February, I followed this up with a failed experiment to implement a Readable module in Ruby that would allow users to create IO objects from arbitrary sources much like Enumerator allows users to create lazy Enumerable objects:

readable = Readable.new do |yielder|
  yielder << "\u001F\x8B\b\u0000[\u0017\x88Z\u0000"
  yielder << "\u0003\xF3H\xCD\xC9\xC9\xD7)\xCF/\xCAI"
  yielder << "\u0001\u0000)^\u0014\xFC\v\u0000\u0000\u0000"
end

gz_reader = Zlib::GzipReader.new(readable)
CSV(gz_reader).gets
#=> ['Hello', 'world']

Sadly, this didn’t fully work due to the sprawling surface area of the IO module (though, later in the year, Dmitry Kandalov’s “Coroutines explained” made me want to try this again with Fibers).

In July, thanks to the help of London Computation Club, a particularly thorny Graph theory problem at work led me to implement Dijkstra’s algorithm in Ruby.

While the initial implementation using arrays worked, it could take an hour to run and so I looked into implementing a priority queue data structure to speed things up. In a surprising twist, I discovered an ideal algorithm not online but in a printed textbook gathering dust on my shelf: Cormen, Leiserson, Rivest and Stein’s “Introduction to Algorithms, Third Edition”. Implementing their description of Fibonacci Heaps was difficult but ultimately rewarding when I published a new RubyGem imaginatively titled Fibonacci Heap.

The data structure allows you to insert nodes with a given priority and efficiently pop the highest priority node from the heap even if that priority has been modified after insertion:

heap = FibonacciHeap::Heap.new
foo = FibonacciHeap::Node.new(1, 'foo')
bar = FibonacciHeap::Node.new(0, 'bar')
baz = FibonacciHeap::Node.new(2, 'baz')
heap.insert(foo)
heap.insert(bar)
heap.insert(baz)
heap.pop
#=> #<FibonacciHeap::Node key=0 value="bar">
heap.decrease_key(baz, 0)
heap.pop
#=> #<FibonacciHeap::Node key=0 value="baz">

Replacing my array-based implementation of Dijkstra’s algorithm with a Fibonacci Heap-based one sped up our build times to mere seconds and was well worth it.

That would have been my open source highlight of the year until something else happened on Christmas Day: Ruby 2.6 was released containing a certain new feature.

That feature was the ability to compose Proc objects using << and >>. While I have dabbled in functional programming before, this feature is of particular interest to me as it is based on a series of patches I submitted in 2015. This is my first code contribution to Ruby to be accepted and, as such, has to be my open source highlight of the year.

Public speaking

I only did one talk this year: “HyperLogLog in 15 minutes” at the Drover Ruby Meetup in November. Huge thanks to those who listened to me rehearse this presentation (some people more than once), the feedback I received massively improved everything from the slides to my delivery.

I’m considering expanding this into a longer talk for the new year so that I can reduce the amount of hand-waving and dig into the mathematics a bit more but I’m still undecided.

Work

The elephant in this post is that in August of this year, after 5 years, I gave notice of my resignation as CTO of Altmetric.

The year leading up to that point had seen some great successes from the team: the rewrite of the company’s oldest data source, the addition of over 8 million patents, a huge user interface and technology overhaul to the flagship product, the integration of scholarly citations for the full database and some especially tricky performance work on the company’s Extract, transform, load processes.

Ultimately though, my struggle to find a sustainable place for myself within the company (as evidenced by my feelings in the previous year and by my attempts to take a break and even start meditating) was in vain. Considering how blissful I felt during my honeymoon at the end of 2017, it was shocking to me how much my work permeated my every waking thought and how quickly I fell into the same trap of my own doing.

At my lowest ebb, I booked a long Bank Holiday weekend off at the last minute and asked many people in my field for advice, many of them from London Computation Club. I am hugely thankful to all those who kindly took the time to speak with me and help me navigate a very difficult decision.

I worked the remaining four months of the year, handing over to my amazing colleagues, confident in their ability to learn from my mistakes and lead the team to even greater success.

I hope to write more about my time at Altmetric in the future as I learnt an incredible amount trying to build a software development team there over five and a half years and I would do it all over again in a heartbeat.

2019

I’m honestly not sure what is next. After working continuously for a decade in London, I plan to take a break from keyboards and screens for a short while before seeing how I can best be of help to others, be that as a contractor or as a full-time employee.

Despite having already published a new open source project this year, I hope to continue seeking a better balance between work and life and, as with last year, get back into running.

Cross-Compiling Rust for a Raspberry Pi on Travis CI

2019-01-02T00:00:00+00:00

I recently wrote a small program in Rust that I wanted to compile for an old Raspberry Pi Model B but running cargo build --release on the Pi itself took several hours to complete. Researching online, I found Jorge Aparicio’s guide to Rust cross compilation and their Travis CI and AppVeyor template extremely useful but wasn’t entirely successful in compiling a binary I could use directly on my device. After many experiments and many failed builds, I finally have a build pipeline that will compile a Rust binary for the Raspberry Pi, Linux and macOS in minutes using Travis CI.

Cross-compiling manually

The first thing we need to understand is how we can compile a Rust program on one architecture but target another and, most importantly, understand the current limitations of cross-compilation with Rust.

Unlike Go which only requires setting two environment variables to cross-compile to any architecture, cross-compiling Rust requires us to have a copy of the standard library for the target platform (e.g. in the case of the Raspberry Pi, the standard library needs to be compiled for arm-unknown-linux-gnueabihf) and for us to have an appropriate platform-specific linker for compilation. Thankfully, rustup allows us to easily add standard libraries for other platforms but we’ll need to source the appropriate linker ourselves.

Finding a platform-specific linker is what restricts us: if there is no toolchain for the target platform available on your build platform then we can’t cross-compile. For example, we can’t easily compile a binary for the Raspberry Pi from macOS because the Raspberry Pi toolchain doesn’t run on macOS.

Luckily for us, it does run on 64-bit Linux so we can use Travis CI to cross-compile as long as we download the Raspberry Pi toolchain and instruct Rust to use its compiler to do the linking. Let’s see how this would work if we ran it ourselves on the command-line.

Let’s assume we’re running this inside our Rust project on a 64-bit Linux machine and we have a recent rustup installed. We’ll start by first downloading the standard library for our target platform: Linux running on an ARMv6 CPU with hardware floating-point support.

$ rustup target add arm-unknown-linux-gnueabihf
info: downloading component 'rust-std' for 'arm-unknown-linux-gnueabihf'
 52.5 MiB /  52.5 MiB (100 %)  26.5 MiB/s ETA:   0 s
info: installing component 'rust-std' for 'arm-unknown-linux-gnueabihf'

Then let’s download the Raspberry Pi toolchain from GitHub (only fetching the latest commit) into /tmp and add the necessary binaries to our path:

$ git clone --depth=1 https://github.com/raspberrypi/tools.git /tmp/tools
$ export PATH=/tmp/tools/arm-bcm2708/arm-linux-gnueabihf/bin:$PATH

We now need to tell Rust where to find the linker for the arm-unknown-linux-gnueabihf target and we can do this in one of two ways:

Add configuration to ~/.cargo/config
Set an environment variable

For the first approach, we would add the following to ~/.cargo/config (creating it if it does not already exist):

[target.arm-unknown-linux-gnueabihf]
linker = "arm-linux-gnueabihf-gcc"

For the second, we need to export an environment variable for the specific target:

$ export CARGO_TARGET_ARM_UNKNOWN_LINUX_GNUEABIHF_LINKER=arm-linux-gnueabihf-gcc

With either of those in place, we can now attempt to compile our project for the target platform with cargo:

$ cargo build --target=arm-unknown-linux-gnueabihf --release

If you’re lucky, this will work and produce a binary in target/arm-unknown-linux-gnueabihf/release/ that you can run on your target platform. However, I ran into issues trying to compile parts of the openssl crate as it was attempting to find OpenSSL headers for the target platform on my system but they aren’t binary compatible.

Thankfully, we can solve this by taking advantage of the openssl crate’s vendored Cargo feature which will compile and statically link its own copy of OpenSSL meaning that it will be compiled for the appropriate platform. To do this, we need to add the following dependency to our Cargo.toml:

openssl = { version = "0.10", features = ["vendored"] }

Hopefully you will now have a cross-compiled binary you can run to your heart’s content.

Running on Travis CI

So how can we automate this to run on Travis CI? We can make use of Travis CI’s Build Matrix functionality to specify multiple target platforms and run a Bash script before deployment to prepare assets for deployment to GitHub Releases.

Let’s start with only one target platform to keep things simple.

First, you’ll need to generate an encrypted GitHub OAuth token to use for the deploy configuration. The easiest way to do this is to use the Travis CLI:

$ gem install travis
$ travis init
$ travis setup releases

With that set up, add the following to your .travis.yml (leaving your generated deploy API key and replacing mybinary with your desired name):

dist: trusty
language: rust
cache: cargo
env: TARGET=arm-unknown-linux-gnueabihf CARGO_TARGET_ARM_UNKNOWN_LINUX_GNUEABIHF_LINKER=arm-linux-gnueabihf-gcc
before_deploy: ci/before_deploy.sh
deploy:
  file: mybinary-$TRAVIS_TAG-$TARGET.tar.gz
  on:
    tags: true
  provider: releases
  skip_cleanup: true

We can now write our ci/before_deploy.sh script to perform the steps we did earlier (you’ll need to alter the last line to compress the appropriate binary with your desired name as specified in your .travis.yml above):

#!/bin/bash
set -euo pipefail

# Install the Rust stdlib for the current target
rustup target add $TARGET

# Download the Raspberry Pi cross-compilation toolchain if needed
if [ "$TARGET" = "arm-unknown-linux-gnueabihf" ]
then
  git clone --depth=1 https://github.com/raspberrypi/tools.git /tmp/tools
  export PATH=/tmp/tools/arm-bcm2708/arm-linux-gnueabihf/bin:$PATH
fi

# Compile the binary for the current target
cargo build --target=$TARGET --release

# Package up the release binary
tar -C target/$TARGET/release -czf mybinary-$TRAVIS_TAG-$TARGET.tar.gz mybinary

Make this file executable:

$ chmod a+x ci/before_deploy.sh

You should now be able to build this on Travis CI when pushing a tag and see the release archive pushed automatically to your project.

To add another target platform, e.g. 64-bit Linux, alter your .travis.yml by replacing the env key with the following:

matrix:
  include:
    - env: TARGET=arm-unknown-linux-gnueabihf CARGO_TARGET_ARM_UNKNOWN_LINUX_GNUEABIHF_LINKER=arm-linux-gnueabihf-gcc
    - env: TARGET=x86_64-unknown-linux-gnu

This will run two builds, one with the TARGET set to arm-unknown-linux-gnueabihf (and with the Cargo linker configuration for that platform) and one with the TARGET set to x86_64-unknown-linux-gnu (as that’s the native platform for our build, there’s no need to specify a different linker).

For a fully worked example building for the Raspberry Pi, Linux and macOS, see my lightweight DNS-over-HTTPS proxy, Homer.

What Is a “Successful” Software Development Team?

2016-03-26T00:00:00+00:00

Last Friday, I attended Scale Summit 2016, my fourth time at the conference formerly known as Scale Camp. I make no secret that it is one of my favourite conferences with its open space technology format, the high calibre of attendees and its use of Chatham House Rule.

I spent the morning participating in two sessions:

“Breaking down barriers between teams”
“Code sharing in large organisations”

Richard Dallaway wrote up his notes from the former and both Anna Shipman and Barrie Bremner have shared their thoughts on the latter.

As I’ve spent the past three years attempting to grow a software development company, there was a common theme between these two sessions that fascinated me:

How do you measure the success of a software development team and, in turn, improve it?

The first session began with a discussion of how to encourage collaboration between teams (e.g. between different teams of developers, operations, product owners, etc.), break down organisational silos and consolidate tooling across an organisation.

The second session focussed on promoting code re-use within a large organisation, with the aim of reducing the amount of duplicated software built by separate teams.

Software development is rich with ideas and processes aiming to optimise efficiency and deliver better results, but has a somewhat spotty history of applying these practices. In such an environment, there is an ever-present risk of cargo culting:

McConnell describes software development organizations that attempt to emulate more successful development houses, […] by slavishly following a software development process without understanding the reasoning behind it.

With this in mind, in both sessions my first question was similar:

Why should we break down the barriers between teams?
Why should we share code in a large organisation?

Without a clear motivation for using it, how can anyone be truly convinced of a practice? A team forced to collaborate without incentive to do so beyond “our managers told us to” seems unlikely to truly benefit. What’s more, without clear reasoning, it’s hard—if not impossible—to measure the success of any endeavour. Sure, the teams are talking more but why is that a good thing?

The push for increased collaboration between developers and operations staff lead to the DevOps movement, which aimed to spread knowledge of and responsibility for the deployment software and its on-going maintenance. The pattern of developers “throwing code over the wall” to operations staff, who were suddenly responsible for its performance and uptime, contributed to antagonism between teams and increased both the risk and cost of failure.

Breaking down the barriers between these teams had some key goals in mind:

Reduce the risk of failures when deploying software
Reduce the time to recovery when failures do occur

We could similarly evaluate improving communication between product owners and developers, e.g.

Reduce the amount of unnecessary software written due to misunderstanding
Reduce the cost of software and ship higher value features in a more timely fashion, by sharing the complexity and risk of the backlog

However, choosing different evaluation criteria can have an unpredictable—and even damaging—effect:

Some of the managers decided that it would be a good idea to track the progress of each individual engineer in terms of the amount of code that they wrote from week to week. They devised a form that each engineer was required to submit every Friday, which included a field for the number of lines of code that were written that week.

— Andy Hertzfeld, “-2000 Lines Of Code”

Incentivising developers to write more code can accidentally reward those who write verbose software (often at the expense of maintainability). There are even stories of how introducing a bounty for fixing bugs created a black market within a company.

Not all incentives have such obvious (even if only in hindsight) side-effects. Discussion in the first session about consolidating tooling (specifically, continuous integration solutions) brought up a more difficult scenario: management want everyone to use the same tool in order to save paying for (and maintaining) multiple solutions to the same problem. What is the cost of forcing everyone to use the same tool? Does this accidentally reduce the autonomy of a team, coupling them to others in a way that might impede their progress? What if the teams’ requirements differ in a way that you missed?

The benefits of code reuse should also be subject to the same scrutiny: as Anna Shipman writes (emphasis my own):

The problem we are trying to solve is to reduce duplication of work, rather than specifically duplication of code. More generally, we do not want teams to waste time reinventing the wheel. We do not necessarily want “the best tool for the job”, we want the most cost-effective tool, and that might be copying someone else’s code, or the team solving the same, or similar, problem in a different way.

Perhaps using services would be a better way to share common functionality than reusing code itself?

Ultimately, these different attempts to improve teams were best described as a struggle between focussing on outputs and means.

The two are often confused: no one encouraging collaboration between teams does so without an implicit expectation that their mutual output will improve. No one attempting to consolidate tooling does so thinking that it will have a deleterious effect on users.

Without attempting to understand both output and means, we are at risk of jeopardising, not improving, our chances of success.

Data Structures as Functions (or, Implementing `Set#to_proc` and `Hash#to_proc` in Ruby)

2014-11-26T00:00:00+00:00

At the March 2015 meeting of the London Ruby User Group, I delivered a presentation based on this blog post with some supplemental material. I’ve since tweaked some of the examples to closer match the ones in the final talk but the content should largely be the same.

Reading Henrik Nyh’s “Array#to_proc for hash access” made me think about a similar concept in Clojure: data structures being used as functions.

Functions

First of all, let’s be clear what we mean by “functions”. The definition of a function in mathematics is as follows:

A function is a relation between a set of inputs and a set of permissible outputs with the property that each input is related to exactly one output.

So a function f can be considered as a way of mapping from some input (its “domain”) to some output (its “codomain”), e.g.

f(1)
# => 2

Crucially, whenever the function is called with a specific input, it should always produce the same output. This means f in the following example would not be a function because it produces two different outputs given the same input:

f(1)
# => 2

f(1)
# => 3

Whole programming languages are built out of this concept (it’s called functional programming for a reason) but they also appear in languages that aren’t purely functional, such as Ruby.

A common place you might find them is when using methods from Enumerable such as all? and map:

(1..10).all? { |x| x > 4 }
# => false

(1..10).map { |x| x * 2 }
# => [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]

In both of the above cases, the blocks passed to all? and map are actually simple functions.

proc { |x| x > 4 }

No matter how many times you call this function with the number 0, it will always return false.

proc { |x| x * 2 }

Similarly, no matter how many times you call this function with the number 1, it will always return 2.

With this in mind, let’s take a look at functions in Clojure.

Using sets as functions

In Clojure (as in many languages, including Ruby), there are data structures for sets and maps (not to be confused with the function of the same name but something akin to a Hash in Ruby or a dictionary in Python) but, unlike other languages, these structures can be used in places where you would expect a function.

For example, Clojure’s filter takes two arguments: a function pred and a collection coll. As you might expect from its name, it goes through coll and only returns elements that return true when fed into pred. We can use this to filter even numbers from a range using even?:

(filter even? (range 1 30))
;; => (2 4 6 8 10 12 14 16 18 20 22 24 26 28)

Of course, even? can be used by itself too:

(even? 5)
;; => false

Nothing too surprising there but what if we used a set instead of an existing function?

Let’s define a set of numbers we like:

(def perfect-numbers #{6 28 496})

Now let’s filter our range again but using the set instead of even?:

(filter perfect-numbers (range 1 30))
;; => (6 28)

To understand what happened here, let’s just try using the set as a function as we did even?:

(perfect-numbers 1)
;; => nil

(perfect-numbers 6)
;; => 6

It turns out that sets can be used as functions of their members: when given some input, they will return that input if (and only if) it is a member of the set; if it isn’t, it returns nil. This behaviour makes for a nifty shortcut when testing whether a value is in some whitelist:

(def valid-attributes #{:name :age})

(defn foo [attribute]
  (when (valid-attributes attribute)
    ;; insert money-making business logic here
    (println "PROFIT")))

The way this actually works in Clojure is that sets implement the IFn interface meaning that they have an invoke method that can be used to call them as if they were ordinary functions. There’s a very similar __invoke magic method in PHP that allows objects to be called as functions:

<?php
class Monkey
{
    public function __invoke($name)
    {
        return "Ook, {$name}!";
    }
}

$monkey = new Monkey;
$monkey('Bob');
// => 'Ook, Bob!'

We can achieve similar behaviour in Ruby by implementing Set#to_proc (in the same way Henrik implemented to_proc on Array):

class Set
  def to_proc
    proc { |element| element if member?(element) }
  end
end

This means we can do the same sort of filtering using select:

require "set"

perfect_numbers = Set[6, 28, 496]

(1..30).select(&perfect_numbers)
# => [6, 28]

If you’re unfamiliar with the syntax above, let’s take a quick detour (if this is old news to you, feel free to skip ahead).

What’s with the ampersand?

In Ruby, there are methods (such as select above) that take blocks. A good example of this is map which evaluates a given block for each item in a collection and returns a new collection with the results:

(1..30).map { |x| x.to_s }
# => ["1", "2", "3", ..., "30"]

Wherever you can pass a block in Ruby, you can also pass a Proc using an ampersand like so:

stringify = proc { |x| x.to_s }

(1..30).map(&stringify)
# => ["1", "2", "3", ..., "30"]

Note that stringify is already a Proc here but if it wasn’t, Ruby would internally call to_proc on it in an attempt to coerce it to the right type.

As mapping over a collection and calling a method on each element is quite a common task and armed with the knowledge of Ruby’s internal use of to_proc, a technique emerged of using symbols as a shortcut:

(1..30).map(&:to_s)
# => ["1", "2", "3", ..., "30"]

This works by implementing Symbol#to_proc with something like so:

class Symbol
  def to_proc
    proc { |x| x.send(self) }
  end
end

(The real implementation actually features a cache to reduce some of the performance overhead of this approach.)

You can think of this as expanding like so:

(1..30).map(&:to_s)
(1..30).map(&:to_s.to_proc)
(1..30).map(&proc { |x| x.send(:to_s) })
(1..30).map { |x| x.send(:to_s) }
(1..30).map { |x| x.to_s }

Calling Procs

With to_proc implemented on sets, can we use them as we did in Clojure?

perfect_numbers.call(1)
# NoMethodError: undefined method `call' for #<Set: {6, 28, 496}>

Sadly not, as sets themselves are not Procs. Perhaps we could use the ampersand trick?

&perfect_numbers.call(1)
# SyntaxError: unexpected &, expecting end-of-input
# &perfect_numbers.call(1)
#  ^

No luck there either. In fact, we’d have to do the following:

perfect_numbers.to_proc.call(1)
# => nil

This is where our efforts diverge from Clojure: simply implementing to_proc doesn’t mean that sets are now functions. You could maybe make them more Proc-like with something like the following but there is no equivalent to Clojure’s IFn or PHP’s __invoke in Ruby:

class Object
  def call(*args)
    to_proc.call(*args)
  end
end

perfect_numbers.call(3)
# => 6

Using hashes as functions

A slightly more interesting case is that of maps in Clojure:

({:a 2 :b 3} :a)
;; => 2

({:a 2 :b 3} :c)
;; => nil

Like the behaviour of sets, maps are functions of their keys: given the input of the key, they return the corresponding value (or nil if there is no such entry). This fits quite nicely with our original mathematical functions: the keys being the domain and the values being the codomain.

As for how this might be useful, Jay Fields wrote up an interesting use case when comparing and filtering two maps. It also gives us an alternate way to access hash keys as in Henrik’s original post.

First, let’s implement Hash#to_proc to mirror Clojure’s behaviour:

class Hash
  def to_proc
    method(:[]).to_proc
  end
end

Here we simply take Hash’s existing [] method for accessing values and convert it to a Proc. Now we can use it like so:

%w(a b c d).map(&{"a" => 1, "b" => 2, "c" => 3})
# => [1, 2, 3, nil]

Or, to use a more realistic example:

person = { name: "Robert Paulson", age: 43 }

name, age = [:name, :age].map(&person)
# => ["Robert Paulson", 43]

In this second example, it’s almost the reverse of Henrik’s Array#to_proc behaviour where the hash is passed into a function created from the keys:

[{ name: "A" }, { name: "B" }].map(&[:name])
# => ["A", "B"]

In our case, it is the key that is passed into a function created from the hash:

[:name].map(&{ name: "A" })
# => "A"

Other data structures as functions

Maps and sets are not the only data structures to implement the IFn interface in Clojure.

There are vectors:

([1 2 3] 0)
;; => 1

Which would be equivalent to the following Array#to_proc:

class Array
  def to_proc
    method(:[]).to_proc
  end
end

(1..3).map(&["A", "B", "C", "D", "E"])
# => ["B", "C", "D"]

There are also keywords which are similar to Ruby’s symbols:

(:name {:name "Robert Paulson", :age 42})
;; => "Robert Paulson"

To implement this in Ruby, we’d have to override the now standard Symbol#to_proc like so:

class Symbol
  def to_proc
    proc { |coll| coll[self] }
  end
end

[{ name: "Robert Paulson", age: 42 }].map(&:name)
# => ["Robert Paulson"]

Of course, this would mean you can no longer use it for method access.

In conclusion, we shouldn’t be afraid to experiment with concepts from other languages. After all, as Alan Perlis said:

A language that doesn’t affect the way you think about programming is not worth knowing.

A Summit for Scaling

2014-03-22T00:00:00+00:00

Yesterday, I attended Scale Summit: “a performance and scalability unconference” using the Open Space Technology format. Scale Summit is the successor to Scale Camp, an event organised by Michæl Brunton-Spall and previously held at The Guardian offices. Having attended Scale Camp in late 2012 and considering it to be one of the most rewarding experiences of my career, it was without hesitation that I signed up for its follow-up.

The quality of Scale Camp and Scale Summit is almost entirely dependent on the attendees: the unconference format and Chatham House Rule mean that people are free to run open discussions and Fishbowls and speak a little more freely than they might elsewhere. For example, my highlight of 2012’s Scale Camp was discussing continuous delivery with people I would consider experts and even pioneers of that practice.

As Scale Summit is multi-track, I could not participate in all sessions but thankfully other attendees such as Philip Potter, Matthew Cottingham and Zac Stevens have also written up their notes.

Real-Time Monitoring

My first session of the day was an open discussion on real-time monitoring and the collection (and retention) of large amounts of metrics.

There was a general consensus that everyone is using Graphite though Grafana piqued interest as an alternative dashboard and Carbonate suggested for rebalancing clusters.

There seemed to be a lot of people using Sensu for monitoring (as well as the usual crowd of Zenoss, Ganglia and Nagios users) though some (including myself) have been experimenting with Riemann, particularly for its ability to alert on event expiry. We currently use this at my day job to detect when an integration with an external service has started misbehaving as we can reasonably expect new data in a certain time period (e.g. a tweet every 10 minutes). Upon discussing this, it occurred to me that I could use this expiry technique a lot more than I currently do as a very simple way of flagging anomalies.

While discussing the resolution of metrics (e.g. every second, every 5 seconds, etc.), an interesting question was asked:

When investigating an outage, has anyone ever had too much data?

It seems that collecting the right data is still difficult to predict so the sensible approach seems to be measuring everything. The cost of doing so (particularly those wanting high resolution or retention of data) can be high with one individual’s monitoring server having “more cores than my entire [server] farm”.

Aside from collectd, there was some discussion around using Diamond to collect metrics largely due to it being written in Python for companies already using the programming language for their applications.

Microservices in the Real World

During the day’s introductions, quite a few people mentioned Microservices which I had heard little about so I decided to take part in an extremely fast-paced Fishbowl discussion on the topic.

There was some disagreement on the exact definition of a microservice and some took issue with Martin Fowler’s insistence on HTTP as a transport or purely using lines of code as a defining factor. The loose definition seemed to be modelling applications as a collection of simple services responsible for a single piece of functionality (e.g. translating place names). In this regard, it seems remarkably similar to service-oriented architecture (SOA) but with an emphasis on smaller services.

Many benefits were cited in terms of reducing complexity, affording diversity of technologies and the ability for teams of one or two people to hold entire services in their head (something that becomes unfeasible over time for monolithic systems).

The importance of consistent practice was said to be paramount: consistent documentation, clear ownership and predictable interfaces were all discussed. One intriguing practice mentioned was that of having a standardised endpoint that revealed the owner and current health of a system (with some parallels being drawn to a humans.txt). Even if ownership and responsibility for a service is unclear, the size and simplicity of such systems might make them easier to maintain.

While the benefits seemed appealing, I was more than a little concerned about the effects this would have on teams and technology organisations as a whole. One problem I have seen repeatedly across companies is that of information siloes: where team members become segregated and shared knowledge of systems is largely absent. I fear this model of many small services would encourage siloes but the difficult question is whether this is a bad thing: perhaps it is better for small teams to move fast at the cost of shared understanding? If the benefits are to believed then services might be small and simple enough to alleviate the dreaded Bus factor.

As an extreme example of this culture, the practice of allowing competing services was touched upon: where teams can freely develop another version of a service without collaboration and let client applications decide which they prefer. The implications this environment would have on teams (where people could simply supplant your work and openly duplicate effort) makes me more than a little apprehensive.

I remain sceptical but it has encouraged me to consider pursuing SOA more aggressively particularly considering success stories such as Songkick’s.

Visibility of Infrastructure

This discussion began with the problem of increasing awareness of infrastructure within an organisation to help reduce the divide between developers and operations but meandered around general visibility of infrastructure for various stakeholders.

It was agreed that the problem with documentation is that it is almost always out of date and there is little incentive to keep it updated. After some suggestions of automatically deriving architectural diagrams from infrastructure (using firewall rules to infer traffic between services or the likes of Zipkin), it was clear that human intervention is required. It was suggested that diagramming the high level, logical flow of a system might last longer than any including exact implementation details (the ultimate, eternally correct diagram being a single circle with the word “System” in it).

After a slightly alarming diversion into the world of Promise Theory, there was a discussion about tools for describing the relationships between services beyond what is currently offered by the likes of Puppet and Chef with Ubuntu Juju suggested as a possible option.

When discussing the problem of service discovery, one architectural practice that caught my attention was that of running HAProxy on all servers and having applications talk to localhost with the load balancer configuration determining where services currently reside. A few people seemed to be using this pattern (and some cited Airbnb’s Synapse) though the difficulty of debugging network traffic and managing health checks coming from every server were mentioned as downsides.

Surviving a DDoS Attack

The discussion of this topic was obviously a little more sensitive than most but the general recommendations were to use a content delivery network such as Fastly to immediate drop all attacks below Layer 7 and use services such as Prolexic and intrusion detection systems such as Snort.

Ultimately, the most sophisticated attacks are extremely difficult to distinguish from normal user behaviour though these are fortunately rare with most attacks being easy to detect. That said, there seemed to be no fully automated way to deal with an attack and human intervention is currently required.

Linux Containers

The last and most popular session I attended was that on Linux containers with Docker being the impetus but LXC and Solaris Zones also being discussed.

There was a lot of interest in the promise of containers as a way of providing isolation with less of a performance penalty than fully virtual machines but startlingly few people were using them in production. A few people were using them during continuous integration and to build packages for multiple operating systems extremely quickly but most people were still experimenting.

The current best practice seemed to be using a container to run a single process (e.g. one container for nginx and one for php-fpm) though it wasn’t explored what benefits this gives.

One appealing use of containers seemed to be encouraging the Immutable Server pattern where persistent data is separated from application servers and configuration management is eschewed in favour of disposable instances: e.g. if you want to upgrade a package, simply build a new container with the change, spin it up and kill the old instances (this reminded me of Adrian Holovaty’s move to AWS from Heroku).

There was a brief discussion on service discovery in the face of many containers with some success using Hipache instead of using the HAProxy/Synapse approach mentioned in the earlier “Visibility of Infrastructure” session. For further production use, Mesos and Marathon seemed to be the technologies to watch.

In general, everyone seemed quite excited about the potential of containers but it is still very young with little tooling available. At the very least, it made me want to join the cabal of people toying with Docker in my spare time.

Conclusions

In general, there was a strong theme of small, composable services (perhaps in containers) and a particularly impressive demonstration of deploying an application server in minutes to something resembling a private Heroku by using Dokku.

As with the last Scale Camp, it did feel like absolutely everyone is using AWS though there were some, like myself, with a mix of hosting providers. Amongst those not entirely in the cloud, there was some fondness for OpenStack.

Besides the aforementioned renewed vigor to pursue SOA, I was convinced by a discussion between sessions to make better use of DNS as a means of service discovery: e.g. instead of specifying the location of your MySQL cluster by managing configuration files with Chef, simply hardcode the location as mysql.service and then use an internal DNS server to resolve that as needed (even allowing for test environments to resolve to a different location). This separation of service location from application configuration seems obvious but I have long coupled such things in the past and used configuration management to work around it.

If you are at all interested in any of the above or just want to sit in a room and hear tales of scaling woe, I heartily recommend attending the next Scale Summit.

Implementing if in Ruby

2012-07-09T00:00:00+00:00

It was with great interest that I watched J. Pablo Fernández’s presentation at the London Ruby User Group tonight entitled “What Ruby Can’t Do”. Concerned of being myopic about the language that happens to provide my income, I am keen to see what we can unabashedly steal from other languages (see Joseph Wilk’s excellent “Testing Outside of the Ruby World” for a similar sentiment). Pablo offered two alternatives to Ruby – Smalltalk and Lisp – and then demonstrated their relative power by implementing his own if condition in both.

While the Lisp example demonstrated homoiconicity and the power of macros, the Smalltalk example was (barring syntactic oddities) much closer to home for Rubyists.

To summarise: Smalltalk eschews keywords for control structures such as if in favour of message sends (to use the Smalltalk parlance for method calls). This means that conditional logic is just another method on an object (not entirely dissimilar to Ruby’s infamous 5.times { ... } example):

somePredicate ifTrue:  [ Transcript show: 'I am true'.  ]
              ifFalse: [ Transcript show: 'I am false'. ].

The implementation of these messages is simple: define a method that takes two arguments, one block to call when true (e.g. trueBlock) and one block to call when false (e.g. falseBlock). Objects that can be considered “truthy” evaluate the first block and those that are “falsey” evaluate the second:

ifTrue: trueBlock ifFalse: falseBlock
  "Implementation for truthy objects."
  ^ trueBlock value

ifTrue: trueBlock ifFalse: falseBlock
  "Implementation for falsey objects."
  ^ falseBlock value

Pablo bemoaned that such a construction cannot be done in Ruby easily without chaining Proc objects and Tom Stuart summarised the problem on Twitter: “Spoiler alert: Smalltalk lets you pass multiple blocks to a method naturally, Ruby doesn’t.” However, I tweeted that this might not be true from Ruby 1.9 onwards.

To mimic the Smalltalk example, the following syntax is actually valid Ruby:

some_predicate.if ->{ "It's true!" }, else: ->{ "It's false!" }

While it might at first seem odd, it’s actually equivalent to the following:

some_predicate.if(lambda { "It's true!" },
    { :else => lambda { "It's false!" } })

To implement this functionality, we can take advantage of Ruby’s notoriously open classes:

class BasicObject
  def if(if_true, options = {})
    if_true.call
  end
end

module Falsey
  def if(if_true, options = {})
    if_false = options.fetch(:else, ->{})

    if_false.call
  end
end

class NilClass
  include Falsey
end

class FalseClass
  include Falsey
end

Note that I default the if_false case to a no-op to avoid using if internally (and therefore sidestep some furore).

(Those of you wishing to draw ire from your colleagues are welcome to download and use the above code as a gem.)

Practically speaking, there is likely to be a performance penalty in avoiding Ruby’s intrinsic optimisations for the if keyword. However, it implements functionality in typical Ruby that was previously reserved as a “special” language feature and, at the same time, demonstrates achieving conditional logic purely via polymorphism.

Update: Yehuda Katz beat me to this by two years in his 2009 post “Emulating Smalltalk’s Conditionals in Ruby”. He uses separate, chainable if_true and if_false methods rather than multiple blocks to mirror Smalltalk’s own ifTrue and ifFalse messages giving you code like so:

some_predicate
  .if_true { puts "true" }
  .if_false { puts "false" }

Testing Vim Plugins on Travis CI with RSpec and Vimrunner

2012-04-18T00:00:00+00:00

I have been using Vim as my primary text editor for several years but it was only recently that I decided to try and write my own Vim plugin. My aim was simple: to create a function that would run the tests for whatever file I was working on. If the file was a test itself then simply run it but if it wasn’t, try to find the corresponding test and run that.

Having heard Vim script (sometimes referred to as VimL) is a rather esoteric language, I pored through the official Vim manual section and Steve Losh’s “Learn Vimscript the Hard Way” and got stuck in.

The code was simple at first but, as it progressed, I found myself programming and then firing up Vim to check whether everything worked as expected. At the recent London Ruby User Group “TDD Fishbowl” debate, this method of verifying code correctness was derisively referred to as “Refresh-Driven Development”. Mindful of this bad practice and having grown tired of manually inspecting functions in Vim, I decided to look into solutions for not only automating the testing of Vim script but also techniques for test driving development.

My initial searches resulted in a Stack Overflow post about Vim script unit test frameworks but none of the solutions appeared particularly popular or actively maintained. However, by adding “driving” to my search terms, I came across Andrew Radev’s “Driving Vim with Ruby and Cucumber”.

I recommend you read Andrew’s full post but the summary is that he wrote a gem called Vimrunner which uses Vim’s client-server functionality to drive Vim from Ruby. Using this gem, he was able to create an extension to Cucumber to write acceptance tests for his Vim plugins.

This was almost exactly what I was looking for except that I prefer to use RSpec for small libraries rather than Cucumber. Luckily, the core of cucumber-vimscript is very simple, so I decided to explore using only Vimrunner and RSpec to test-drive my plugin.

To begin with, I had the following directory layout with the standard files:

.
├── plugin
│   └── runspec.vim
├── autoload
│   └── runspec.vim
└── doc
    └── runspec.txt

In order to get up and running with these Ruby testing frameworks, I added a Gemfile with the following contents:

source 'https://rubygems.org'
gem 'vimrunner', '~> 0.3.1'
gem 'rspec',     '~> 3.1.0'

The actual process of testing the plugin is as follows:

Start a Vim server by invoking Vim with an explicit servername;
Load our plugin under test into the server (viz. put it in Vim’s runtime path and execute the main script);
Start sending commands to our Vim server using remote-expr and check the results for correctness;
Quit the Vim server.

If we’re writing a spec called spec/runspec.vim_spec.rb then steps 1, 2 and 4 are straightforward thanks to Vimrunner:

require 'vimrunner'

vim = Vimrunner.start
vim.add_plugin(File.expand_path('../..', __FILE__), 'plugin/runspec.vim')

RSpec.describe "runspec.vim" do
  after(:all) do
    vim.kill
  end
end

As you can see, we require the gem and then start a new Vim server with start. We add our plugin to the server using the add_plugin helper method and pass the directory (given relative to our current test file) and the relative path of the main plugin script. We also add an RSpec after hook to shut down the server once we are done.

With this in place, we can start writing some basic tests.

As the point of my plugin is to run tests by executing a shell command with :!, it could prove difficult to exhaustively test. However, instead of focussing on the actual execution, I can test the parts of the plugin leading up to that final stage. In particular, let’s look at the part of the plugin that tries to find the most appropriate spec for the currently open file.

Before I started writing tests, this function was private and accessible only to the plugin thereby making it difficult if not impossible to test. By switching to using Vim’s autoload functionality, I could now access (and test) the function from outside the plugin but without worrying too much about namespace clashes.

Here’s the first (admittedly simple) spec:

require 'vimrunner'

vim = Vimrunner.start
vim.add_plugin(File.expand_path('../..', __FILE__), 'plugin/runspec.vim')

RSpec.describe "runspec.vim" do
  after(:all) do
    vim.kill
  end

  describe "#SpecPath" do
    it "returns the current file if it ends in _spec.rb" do
      expect(vim.command('echo runspec#SpecPath("bar/foo_spec.rb")')).to
          eq("bar/foo_spec.rb")
    end
  end
end

Running the spec produced the following, rather thrilling output:

$ rspec spec/runspec.vim_spec.rb
.

Finished in 0.49066 seconds
1 example, 0 failures

As it ran, I saw an instance of Vim fire up and then be shut down as the test suite completed. More importantly, all my specified behaviour was checked without my having to interfere.

This was a great start but is only really good if you are testing functions that have no side-effects. However, one part of my plugin relies on checking the file system to determine its behaviour: namely, it reads a Gemfile.lock to determine whether to use rspec or not. How could we test this sort of functionality?

Well, we can simply write out a dummy Gemfile.lock during our test and see if the function works correctly:

File.open("Gemfile.lock", "w") do |f|
  f.puts(<<-EOF)
GEM
  remote: https://rubygems.org/
  specs:
    diff-lcs (1.1.3)
    rake (0.9.2.2)
    rspec (2.9.0)
      rspec-core (~> 2.9.0)
      rspec-expectations (~> 2.9.0)
      rspec-mocks (~> 2.9.0)
    rspec-core (2.9.0)
    rspec-expectations (2.9.1)
      diff-lcs (~> 1.1.3)
    rspec-mocks (2.9.0)

PLATFORMS
  ruby

DEPENDENCIES
  rake
  rspec
  EOF
end

However, this file won’t be removed at the end of your test runs and, what’s worse, it could potentially interfere with other tests.

To fix this, we can use Ruby’s Dir.mktmpdir and Dir.chdir in an RSpec around hook to create a temporary directory and cd into it for every single test. As we are re-using the same instance of Vim for each test, we’ll also have to cd the server into each new directory:

# At the top of your spec:
require "tmpdir"

# In your example group:
around do |example|
  Dir.mktmpdir do |dir|
    Dir.chdir(dir) do
      vim.command("cd #{dir}")
      example.call
    end
  end
end

By using the block form of both mktmpdir and chdir, Ruby will destroy the temporary directory and restore the current working directory after every run.

This made it possible for me to write specs like the following without worrying about the disk being littered with test files:

it "finds a test with the most similar name" do
  FileUtils.mkdir_p("test/unit")
  FileUtils.touch("test/unit/user_test.rb")

  expect(vim.command('echo runspec#SpecPath("app/models/user.rb")')).to
      eq("test/unit/user_test.rb")
end

At this point, there was quite a bit of set up in the spec file so I decided to move it out into a separate spec_helper.rb to keep the actual tests concise:

require 'tmpdir'
require 'vimrunner'

RSpec.configure do |config|
  config.around do |example|
    Dir.mktmpdir do |dir|
      Dir.chdir(dir) do
        VIM.command("cd #{dir}")
        example.call
      end
    end
  end

  config.before(:suite) do
    VIM = Vimrunner.start
    VIM.add_plugin(File.expand_path('../..', __FILE__), 'plugin/runspec.vim')
  end

  config.after(:suite) do
    VIM.kill
  end
end

Note the use of a VIM constant in before(:suite) instead of a simple local variable so that the server is available to all specs and the changing of after(:all) to after(:suite) so that the Vim server is shut down once the entire test suite is finished and not just those in the current example group.

So now we have a fully automated test suite that we can run locally; what next? How about continuously integrating that test suite to ensure that no commit breaks the build? How about using Travis CI for that purpose?

If you are unfamiliar with Travis, have a look at their Getting started guide which will lead you through creating an account, hooking it up to your GitHub repositories and running your first build.

In this case, we’re going to use their Ruby support to run our test suite. In order to do that, we need a default Rake task that will run the tests.

Firstly, we need to add Rake as a dependency to our existing Gemfile:

gem 'rake', '~> 10.3.2'

Then we need to create a Rakefile and a task to run the tests. Luckily, RSpec ships with such a task by default which we can use and define as the default like so:

require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new(:spec)

task :default => :spec

(Note that RSpec::Core::RakeTask.new will name the task :spec by default but I’m being explicit here for clarity.)

You should now be able to run your suite like so:

$ rake
..............

Finished in 4.85 seconds
14 examples, 0 failures

All that is left is to configure Travis to do that too. Simply create a file named .travis.yml with the following contents:

language: ruby
rvm:
  - 2.1.3
before_install: sudo apt-get install vim-gtk
before_script:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"

This will inform Travis to use the latest version of Ruby (as of writing) and to install a version of Vim with the necessary client-server functionality we need for testing. We also need to start a X virtual framebuffer so that we can create a Vim server successfully which is what the two before_script options are doing (see GUI & Headless browser testing for more information).

With that in place, the next time you push to GitHub, it should trigger a build on Travis: see Build #8 of runspec.vim for an example.

To see a fully worked example (and my finished plugin), feel free to browse the source code of runspec.vim and look at the build history on Travis CI.

Update: Updated for RSpec 3.

Naming Dynamically Created iframes

2012-01-29T00:00:00+00:00

For the past year, I have been working on a web application that relies heavily on inline frames (aka iframes). During the course of development, we have run into some interesting issues and I thought I would record some of these for posterity.

The W3C specification for the IFRAME element states the following document type definition:

<!ELEMENT IFRAME - - (%flow;)*         -- inline subwindow -->
<!ATTLIST IFRAME
  %coreattrs;                          -- id, class, style, title --
  longdesc    %URI;          #IMPLIED  -- link to long description
                                          (complements title) --
  name        CDATA          #IMPLIED  -- name of frame for targetting --
  src         %URI;          #IMPLIED  -- source of frame content --
  frameborder (1|0)          1         -- request frame borders? --
  marginwidth %Pixels;       #IMPLIED  -- margin widths in pixels --
  marginheight %Pixels;      #IMPLIED  -- margin height in pixels --
  scrolling   (yes|no|auto)  auto      -- scrollbar or none --
  align       %IAlign;       #IMPLIED  -- vertical or horizontal alignment --
  height      %Length;       #IMPLIED  -- frame height --
  width       %Length;       #IMPLIED  -- frame width --
  >

The key thing I’m interested in here is the name attribute. Our particular application allows iframes to fire events which are then caught by the parent window. In order to identify the source of these events, the iframes send along their name (accessed via window.name).

If you have an iframe like the following in your web page:

<iframe frameborder="0" name="bob" src="/some-page"></iframe>

And /some-page contains the following HTML:

<!DOCTYPE html>
<title>Test</title>
<script>
document.write('My name is ' + window.name);
</script>

Then the iframe should output the following:

My name is bob

Here’s that in action:

However, if you are dynamically creating iframes with JavaScript (perhaps to avoid BFCache bugs) then you may run into some issues where window.name is undefined in Internet Explorer 6 and 7.

Specifically, if you create an iframe and set its name like so:

<script>
(function () {
    var iframe = document.createElement('iframe');
    iframe.src = '/some-page';
    iframe.name = 'bob';
    document.body.appendChild(iframe);
}());
</script>

Then /some-page will report the following:

My name is

Again, a live demonstration (note that this will only be broken in Internet Explorer):

The same will occur if you use jQuery:

<script>
$(function () {
  $('<iframe/>', {
    src: '/some-page',
    name: 'bob'
  }).appendTo(document.body);
});
</script>

The issue arises because the name of an iframe cannot be changed in Internet Explorer, much like the type attribute on input (there is a caveat in the jQuery documentation about the type issue when creating elements).

This means that you must set the name attribute on creation of the element, not afterwards.

With jQuery, it’s a case of doing the following:

<script>
$(function () {
  $('<iframe name="bob"></iframe>').attr({
    src: '/some-page'
  }).appendTo(document.body);
});
</script>

With pure JavaScript it is slightly trickier as you can no longer use document.createElement. However, you can create a temporary container element (say, a div), and then modify that element’s innerHTML to create your iframe with its name from the beginning. You can then take your newly created iframe and insert that directly into the DOM (without ever inserting your temporary div) like so:

<script>
(function () {
  var temp = document.createElement('div');
  temp.innerHTML = '<iframe name="bob" src="/some-page"></iframe>';
  document.body.appendChild(temp.firstChild);
}());
</script>

(This is actually what jQuery does internally in the prior example.)

In this way, all browsers should now correctly report:

My name is bob

And a live demonstration:

Managing Web Application Servers with Puppet

2011-08-11T00:00:00+00:00

At the August 2011 meeting of the London Ruby User Group, I delivered a presentation titled “Managing Web Application Servers with Puppet”. You can watch a video of the whole presentation and the Q & A session on the Skills Matter web site but for those of you who prefer to read, here is an edited transcript:

Introduction (who I am, what I am going to talk about);
The Old Way (manualling setting up a sample Rails application and some problems);
Enter Configuration Management (introduction to configuration management and Puppet);
The Better Way (setting up the Rails application from earlier but entirely with Puppet);
Testing (a few words on testing configuration);
Wrap-Up (closing remarks).

Introduction

My name’s Paul Mucur and I’m going to be talking about managing web application servers with Puppet.

First of all, who am I? I’ve been a Ruby developer since late 2006 and I currently work for the Nature Publishing Group: they publish Nature which is a scientific journal that has been going for around 140 years. However, I don’t work on the journal side; I work for their Web Applications team building mainly Rails apps (with a couple of Sinatra bits and pieces) for the scientific community. It’s quite a large organisation and the actual system administrators are not only not in the office but they’re actually in a totally different time zone and I want to talk a bit about how we try to mitigate problems with regards to having that amount of separation.

Who is this talk for? Really, it’s for developers not unlike myself (unsurprisingly):

You’ve deployed an application (maybe a personal project that you’ve put on Linode or Slicehost);
You still have to deploy to servers (i.e. you’re not lucky enough to be on Heroku);
You’ve heard of Puppet and/or Chef and have maybe had a play around but want to know more;
You’re open to the idea that there is a better way to set up servers than just running a load of commands as root.

I’m going to be taking the example of a simple Rails application – just because it touches on quite a lot of things sysadmin-wise and hopefully you’re all familiar with the standard set up – and we’re going to do the following:

Run through how we would set up the application normally;
Some problems with doing it like that;
Offer an alternative using configuration management (and specifically Puppet) as a better way of doing things.

What I’m not going to talk about is:

The nitty-gritty of how to set up Puppet (you can install it from a package manager but we’re just going to take it as read that you have it set up);
Using Puppet as a replacement for your normal deployment workflow (we’re still going to be using Capistrano to get our application code on the servers).

The Old Way

Let’s introduce our Rails application. Because I’m a geek and because it will fit neatly on slides, it’s just going to be called “mcp” and it’s a normal, no-frills Rails application.

$ ls mcp
Capfile      config       log
Gemfile      config.ru    public
Gemfile.lock db           script
README       doc          spec
Rakefile     features     tmp
app          lib          vendor

It has a complete test suite, has been set up with Capistrano but has not been deployed yet. First, let’s make sure all the tests pass…

$ bundle exec rspec spec
................

Finished in 0.00816 seconds
Many, many examples, 0 failures

Everything seems to work so let’s get a server prepared for a first time deploy.

We’re going to make some assumptions:

You’ve already got a server (you don’t have to worry about buying it);
You can already log into the server via SSH;
You have permission to run commands as root as you are responsible for setting things up.

Let’s log into the server…

$ ssh mudge@server1
Last login: Mon Aug  1 21:08:06 2011 from 123-45-67-89.

We like to separate our applications by user so – as this is an application called mcp – we’ll use an mcp user and we will actually keep the application code inside the user’s home folder.

So the first thing we need to do is create a user but I can’t quite remember what flags we need to pass to adduser on this machine…

$ /usr/sbin/adduser -h
adduser [--home DIR] [--shell SHELL]
[--no-create-home] [--uid ID] [--firstuid ID]
[--lastuid ID] [--gecos GECOS]
[--ingroup GROUP | --gid ID]
[--disabled-password] [--disabled-login]
[--encrypt-home] USER
   Add a normal user

It looks like this machine’s adduser will create home directories for us by default (some operating systems require a -m flag to do that). Let’s drop into a root prompt (just to make the slides a bit easier) and add our new user.

$ sudo -i
# adduser mcp
Adding user `mcp' ...
Adding new group `mcp' (1002) ...
Adding new user `mcp' (1001) with group `mcp' ...
Creating home directory `/home/mcp' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
...
Is the information correct? [Y/n] 

After filling out a little information, we now have our user and, from the output, it looks like we got a group named after the user created as well.

As we’re going to be deploying our application with Capistrano and we don’t want to be passing around passwords, we now want to set up SSH keys. If you’re not familiar with the concept, you can use SSH keys to log into a user account without entering a password by putting your public key in a file called authorized_keys in the user’s home directory. There are a few tricky things to be aware of though particularly regarding permissions of the files involved in this process. The first thing we need to do is create a .ssh directory but do it in a way that means only the owner can access it…

# mkdir -m 700 ~mcp/.ssh

Then we’re going to add our key to the authorized_keys file and, as I am running as root and not mcp, we need to make sure that we chown everything to the right user so they can access it.

# vi ~mcp/.ssh/authorized_keys
# chmod 600 ~mcp/.ssh/authorized_keys
# chown -R mcp: ~mcp/.ssh

That’s now done and only the mcp user can access our list of public keys.

We want to have a particular directory structure for our deployed application and we’re going to copy the one used by Capistrano:

A releases directory that will contain timestamped versions of your application code;
A shared directory that persists across deploys making it ideal for storing configuration like your database.yml.

The final part (which will be managed by Capistrano itself) is a symbolic link called current which points to the currently active version of your code within the releases directory.

While Capistrano will create those directories for us, we want to take advantage of the shared directory now so that we can put our configuration in place before we deploy. Let’s create our directory structure within the mcp user’s home directory…

# mkdir -p ~mcp/apps/mcp/shared/config

As we’re doing everything as root, we just need to make sure everything is owned by the right user…

# chown -R mcp: ~mcp/apps

As a developer in a large organisation, ideally I’m not supposed to see live usernames and passwords for things like the database so let’s say, at this point, we hand over to a system administrator who does have authority to handle those details. Perhaps we could supply them with a database.yml with “FILL IN THE BLANKS HERE” for the username and password and then guide them to put the file in the right place. Once they have done that then we can make use of the file in our deploys without having to store it on development machines.

# cd ~mcp/apps/mcp/shared/config
# vi database.yml

Again, make sure that the user is correct.

# chown mcp: database.yml

To run our application, we’re going to need to install Ruby. As we might have several applications using different versions, let’s install RVM to manage Ruby for us [NB: this predates the release of rbenv and the ensuing controversy].

Firstly, we’ll need to install RVM’s dependencies; as we are using Debian, we’ll need to use apt-get to install the required packages…

# apt-get install curl git-core subversion
Reading package lists... Done
Building dependency tree
Reading state information... Done

You can install RVM using a one-liner but, as we are on a production server, we want to be a bit pickier about the version we are using. Luckily, RVM offers an alternative way to install by downloading a single script…

# cd /root
# curl -s \
> https://rvm.beginrescueend.com/install/rvm \
> -o rvm-installer

Making it executable…

# chmod +x rvm-installer

And then using that to install a known good version, let’s say 1.6.32…

# ./rvm-installer --version 1.6.32
Installation of RVM to /usr/local/rvm is
complete.

As we are running as root, everything will be installed to /usr/local.

In order to actually compile and install versions of Ruby, there are a load of other dependencies we need (the list of which you can get by running rvm notes on your machine)…

# apt-get install build-essential bison \
> openssl libreadline6 libreadline6-dev \
> zlib1g zlib1g-dev libssl-dev libyaml-dev \
> libsqlite3-0 libsqlite3-dev sqlite3 \
> libxml2-dev libxslt-dev autoconf \
> libc6-dev ncurses-dev libcurl4-openssl-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done

Now that we’ve got all those, we can finally install our chosen version of Ruby. We are lucky enough to be using the latest version of 1.9.2 so let’s go ahead and use RVM to install that…

# rvm install 1.9.2-p290
Installing Ruby from source to:
/usr/local/rvm/rubies/ruby-1.9.2-p290, this
may take a while depending on your cpu(s)...
Install of ruby-1.9.2-p290 - #complete 

The final piece of the puzzle is our web server: we like to use nginx and Passenger to serve our applications so let’s install and configure them. First we’ll need to source RVM in order to switch environments…

# source /usr/local/rvm/scripts/rvm

Now we can choose Ruby 1.9.2…

# rvm use 1.9.2-p290
Using /usr/local/rvm/gems/ruby-1.9.2-p290

And then we’re going to install a specific version of Passenger…

# gem install passenger -v3.0.8
Fetching: fastthread-1.0.7.gem (100%)
Building native extensions.  This could
take a while...
Fetching: daemon_controller-0.2.6.gem (100%)
Fetching: rack-1.3.2.gem (100%)
...
Successfully installed passenger-3.0.8

Then we can use the passenger-install-nginx-module command to actually compile an nginx for us.

# passenger-install-nginx-module
Welcome to the Phusion Passenger Nginx
module installer, v3.0.8.

Then after following the installation wizard, you will have a complete, compiled nginx in /opt/nginx. All we need to do now is configure it for our application.

By default, there is only one monolithic nginx configuration file but we’re going to break that down so we can manage an application’s configuration separately. In order to do that, we’re going to use a pattern that I originally saw in the Debian packaging of Apache by having two directories:

sites_available, with all possible application configuration;
sites_enabled, with only enabled applications.

The main nginx.conf is then instructed to include any configuration found in sites_enabled and that is merely a collection of symbolic links to configuration in sites_available.

# cd /opt/nginx/conf
# mkdir sites_available sites_enabled
# vi nginx.conf

Now we just need to put our application configuration in sites_available and link to it from sites_enabled (this way you can easily disable applications without having to delete all their configuration).

# cd sites_available
# vi mcp.conf
# ln -s \
> /opt/nginx/conf/sites_available/mcp.conf \
> /opt/nginx/conf/sites_enabled/mcp.conf

As we installed nginx via Passenger, we will also need to make sure that we can control the web server and start it up on boot by creating an init.d script. Let’s say we already have a stock one that we use and put that into place…

# vi /etc/init.d/nginx

Make it executable…

# chmod +x /etc/init.d/nginx

As we are using Debian, we need to use update-rc.d to start nginx up on boot…

# update-rc.d -f nginx defaults

And then finally we can start up nginx ourselves for the first time.

# /etc/init.d/nginx start
Starting nginx: nginx

Our server is now totally set up and ready to go.

It’s just a case of going back to our deployment tool of choice (say, Capistrano) and performing the usual first-time deploy procedure. So we do a deploy:setup to create the directories we need…

$ cap deploy:setup

Do a quick check to make sure everything is OK…

$ cap deploy:check

And then do our big first deploy (hoping that everything will work first time).

$ cap deploy:cold

You might not think that’s too bad if you’ve done it many times before and it’s become second nature to you but there are a few big problems with setting up servers in this way.

It’s not easily repeatable: what if I want to run through the same process on a second server? What if a colleague of mine wants to run through the same process? Perhaps you only have one server but something has gone wrong and you need to re-do the process from scratch, how can you easily do that? Without writing down this procedure in some way, you have no canonical resource that describes how the server was set up and there is little audit trail for you to follow.
It’s not consistent: I just ran a series of commands and was content with their output but, let’s say an hour has passed, how confident can I be that the system is still in the state that I left it? What if someone else has logged in and accidentally deleted that authorized_keys file or perhaps subtly changed permissions somewhere critical? There isn’t really a good way for me to be sure about things until they start breaking which is less than ideal.
It’s not portable: earlier, I had to check to see what flags I needed to pass to adduser as it differs from platform to platform. When I installed the dependencies for RVM, I used apt-get; when I wanted to start nginx up on boot, I used update-rc.d all because I knew I was using Debian. What if I wanted to run the same process on a RedHat or CentOS machine where I’d have to use yum and chkconfig instead? More importantly, why do I have to care what the exact flags to pass to adduser are?

The other big problem is that all of this was done assuming that I could both SSH into a live server and also run commands as root; in a large organisation, that is extremely unlikely. I definitely do not have such authority and neither should I have so the real way this work would have to be done is to be described laboriously in a JIRA ticket. Perhaps I would write a list of instructions in English with a little justification for each command.

That brings me to the most important weakness of all: people make mistakes. If I end up writing a script in English describing the commands to run, what happens if a step gets missed out? What if my explanation is ambiguous and there is a misunderstanding about the order in which commands need to be run? You can get into a sticky situation quite quickly and not realise until much further down the process and it might not be obvious or trivial to rectify.

Perhaps there is a better way to do this. I would argue that there is and it’s called configuration management.

Enter Configuration Management

Firstly, what does the term mean? It’s quite a dry title and obviously refers to the management of configuration but what is important to grasp is the oft-cited idea of “infrastructure as code”. Some people describe Cucumber as “executable documentation” in that it is both human-readable and executable by machine to verify your business requirements; I would argue that “infrastructure as code” is a similar idea. Imagine if you could describe your infrastructure not in an ambiguous language like English but a formal grammar with something like Chef or Puppet: it would both describe the state of your servers in an unambiguous way but could also be executed to ensure consistency.

How many times have you had to debug a problem with a system administrator by asking them to check the permissions or content of certain files? What if you could just state all these assumptions up front and in code?

There are two main solutions for configuration management in the Ruby community (obviously there are more but these are the two most popular at the moment): Opcode’s Chef and Puppet Labs’ Puppet.

I’m going to be talking about and sharing examples written with Puppet because that’s what I use in my day job. When I joined the company, there was already some existing Puppet infrastructure so it made sense to leverage that but I did actually start learning Chef before I found that out. I want to make it clear that it doesn’t matter which tool you use; give both a go and see which you prefer but the key thing is to embrace configuration management in some form. Don’t let the nuances of one put you off the concept as a whole. You will see Puppet examples from me but Gareth Rushgrove will be showing examples with Chef.

Let’s start with a little primer; if you’ve never seen Puppet before, let’s do some simple examples to show you how you need to start thinking about things.

Firstly, Puppet revolves around the idea of resources and this is one such resource…

user { 'henry':
  ensure     => present,
  uid        => '507',
  gid        => 'staff',
  shell      => '/bin/zsh',
  home       => '/home/henry',
  managehome => true,
}

This is written in the Puppet language which looks a little like Ruby but isn’t Ruby. (There is a Ruby DSL for Puppet but I’m only going to be using the original Puppet language.)

This resource is describing a user called Henry: it’s ensuring that he exists, that he’s got a certain user ID, that his primary group is “staff”, that his preferred shell is ZSH, that his home directory is in the standard place and that we will create it for him if it doesn’t already exist. Notice I’m not saying, “you must run adduser”; I’m simply saying “there’s a user with these properties”. By describing the user at this level, you can make one change, say, from present to absent, to remove his account from the system. By delegating the responsibility of how to actually achieve these things to Puppet, you are free to keep your configuration quite sparse.

How would you actually use this? Let’s take that definition, save it in a file called henry.pp and then run the following command…

$ sudo puppet apply henry.pp

What Puppet’s going to do is look at your system, check for a user named Henry and, as he doesn’t yet exist, create him.

notice:
/Stage[main]//User[henry]/ensure: created
notice: Finished catalog run in 0.25 seconds

We can verify that everything has run successfully by having a look in /etc/passwd…

$ grep henry /etc/passwd
henry:x:507:50::/home/henry:/bin/zsh

As we can see, the user has been created and with the properties we specified.

If someone now logs into the server and decides that Henry should be using Bash instead of ZSH…

$ sudo chsh henry
Changing shell for henry.
New shell [/bin/zsh]: /bin/bash
Shell changed.

Then the next time you run Puppet it’s actually going to look at the system and spot the discrepancy between our specification and the state of the system. Most importantly, it will attempt to resolve this issue itself…

$ sudo puppet apply henry.pp
notice:
/Stage[main]//User[henry]/shell:
shell changed '/bin/bash' to '/bin/zsh'

Puppet is clever enough to just to make the appropriate changes to bring the system in-line with our specification. Notice that we didn’t tell it to explicitly run chsh and that it didn’t need to delete and recreate the user. This is the idea of consistency that was missing from the manual approach and also shows how powerful Puppet can be in enforcing system state.

As well as specifying users, Puppet has many different types of resource. The official documentation refers to three in particular as the “Trifecta”:

package { 'openssh-server':
  ensure => installed,
}

A package resource is simply a software package that you might install via apt-get on Debian, yum on RedHat, etc.

file { '/etc/sudoers':
  ensure => present,
}

The file resource allows you to perform operations on the file system; this example simply ensures that a file at /etc/sudoers exists and will create one if not. This is actually a very powerful resource type and we will see more of it later.

service { 'sshd':
  ensure => running,
}

The service resource describes long running processes like those you would manage with an init.d script; in this example, it just makes sure the process labelled sshd is running.

I’ve shown examples using puppet apply but, in real life, you would probably use a different approach altogether: by having a separate Puppet Master server and having your nodes all run the Puppet Agent, you would actually store your configuration on the Master and watch it be applied every 30 minutes by default on all nodes. This is how you can be reasonably confident of your systems’ consistency but to keep things simple, we’re going to leave that out for now.

The Better Way

So let’s go back to our Rails application and go through its set up again but this time using Puppet.

Last time, I ran adduser and a group was created simultaneously but it’s good to be more explicit about it. So let’s state there is a group named mcp and then a user named mcp with that as its primary group and let’s manage their home directory as well…

group { 'mcp':
  ensure => present,
}

user { 'mcp':
  ensure     => present,
  gid        => 'mcp',
  home       => '/home/mcp',
  managehome => true,
}

One thing that is interesting to note here is that the user refers to the group we created above it. You might think that is due to Puppet executing things sequentially but that’s not true: Puppet does not use the order of definitions to derive dependencies. Instead, when you write resources, you refer to them with a lowercase resource type such as group:

group { 'mcp':
  ensure => present,
}

But if you then use title case such as Group:

Group['mcp']

You are actually creating a reference to your resource which can then be used in other definitions like so:

user { 'mcp':
  ensure  => present,
  require => Group['mcp'],
}

This explictly states that the user mcp will not be checked until the group mcp has been enforced. The reason we didn’t do that earlier is because Puppet will autorequire a lot of obvious things (you can check this in the documentation) but it never hurts to be explicit with dependencies.

If you accidentally set up resources that depend on each other like so:

user { 'mcp':
  ensure => present,
  require => Group['mcp'],
}

group { 'mcp':
  ensure => present,
  require => User['mcp'],
}

Puppet will actually spot the issue when you attempt to run it:

err: Could not apply complete catalog:
Found dependency cycles in the following
relationships:
User[mcp] => Group[mcp], Group[mcp] => User[mcp];
try using the '--graph' option and open the '.dot'
files in OmniGraffle or GraphViz

This is because Puppet builds a directed acyclic graph of your resources and can actually output the whole thing as a .dot file as you can see in the output above.

Now that we have user and group, we need to set up our SSH keys. Previously, we had to create the .ssh folder, add our key to authorized_keys and make sure that the ownership and permissions were set correctly. With Puppet, you can forget all that and simply use the built-in ssh_authorized_key resource type:

ssh_authorized_key { 'mcp-mudge':
  ensure => present,
  key    => 'AAAAB3NzaC1yc2EAAAAB...',
  type   => dsa,
  user   => 'mcp',
}

As for the directory structure: we created apps, apps/mcp, apps/mcp/shared and apps/mcp/shared/config so let’s do the same using the file resource:

file {
  '/home/mcp/apps':
    ensure => directory,
    owner  => 'mcp',
    group  => 'mcp';

  '/home/mcp/apps/mcp':
    ensure => directory,
    owner  => 'mcp',
    group  => 'mcp';

  '/home/mcp/apps/mcp/shared':
    ensure => directory,
    owner  => 'mcp',
    group  => 'mcp';

  '/home/mcp/apps/mcp/shared/config':
    ensure => directory,
    owner  => 'mcp',
    group  => 'mcp';
}

This makes sure that each path is a directory owned by mcp; you can make this more concise by listing multiple paths at once but I’ve kept each definition discrete here for clarity.

The next step is slightly more interesting: the management of sensitive configuration files. Last time we had someone trusted put the file in place for us; we can recreate that process again:

file { '/home/mcp/apps/mcp/shared/config/database.yml':
  ensure => present,
  owner  => 'mcp',
  group  => 'mcp',
  source => 'puppet:///modules/mcp/database.yml',
}

The first part is clear enough but the source parameter is the interesting one. Without going into too much detail, as well as having manifests written in the Puppet language, you can also include other artefacts such as whole configuration files that will be copied into place. However, this example is no good as it assumes that I have the live database.yml to hand.

However, the source option is really quite powerful and allows you specify not just one but many paths for a particular file:

file { '/some/config.yml':
  source => [
    'puppet:///confidential/config.yml',
    'puppet:///modules/mcp/config.yml'
  ],
}

When Puppet attempts to create this file, it will check each path in the source list until it finds one it can access. In this way, you can have a confidential file server that can only be accessed from the live servers: when you run this manifest there, it will simply fetch that version but when you run it anywhere else, it will fall through the second version.

However, this still isn’t quite right as it assumes that you only have two types of servers: live and not. What if you have tiers such as staging or test? What you can actually do is use variable interpolation in the source paths like so:

file { '/some/config.yml':
  source => [
    "puppet:///confidential/config.yml.$hostname",
    "puppet:///confidential/config.yml.$tier",
    'puppet:///modules/mcp/config.yml'
  ],
}

As $hostname will be the hostname of your server (e.g. webserver1) then you can have a file such as config.yml.webserver1 and that will be used on that machine; if no such file exists, it will fall through to config.yml.$tier and $tier can be set as follows:

$tier = 'test'
# => "puppet:///confidential/config.yml.test"

$tier = 'staging'
# => "puppet:///confidential/config.yml.staging"

$tier = 'live'
# => "puppet:///confidential/config.yml.live"

By simply setting a variable per tier, you can have several different configuration files but the same resource definition. This is the pattern that we are currently using at my job but there are still problems with it. The main being that you have redundant copies of configuration files when really the only thing that needs to be change between environments are usernames and passwords. When you are managing things like Java .properties files, it can get tiresome to make sure that any changes are rolled out to all versions of a particular file.

A better approach might be to extract the username and password out into variables…

$db_username = 'bob'
$db_password = 'letmein'

file { '/some/database.yml':
  content => template('mcp/database.yml.erb'),
}

If you can then keep those variable declarations in a file only on the Puppet Master then you can make use of them in a template. Luckily, as well as the source parameter, Puppet supports specifying the exact contents of a file as a string with content and, if you use the template function, you can actually process an entire ERB template just like you might be used to in Rails. In this case, our database.yml.erb might look like this:

production:
  adapter: mysql
  username: <%= db_username %>
  password: <%= db_password %>

This means that I can be free to change the adapter in one place without worrying about the other versions of this file.

There seems to be a lot of different recommendations for how to approach this management of sensitive configuration (extlookup, External Node Classifiers, etc.) and Chef has a built-in concept called Encrypted Data Bags for this purpose but I would be keen to see how other people are doing this so please feel free to comment below. [NB: It seems the consensus is to use Hiera along with an encrypted backend such as hiera-gpg. Hiera is actually being made part of Puppet itself but for now you can use it by installing the Hiera and Hiera-Puppet gems.]

Let’s install RVM and Ruby. First, we need all the dependencies and we can just use package to do so:

package { 'rvm-dependencies':
  ensure => installed,
  name   => [
    'curl',
    'git-core',
    'subversion',
    'build-essential',
    ...
  ],
}

The way that we installed RVM was very sequential: we ran a list of specific commands in a particular order. When writing manifests for Puppet, it’s best not to think in terms of sequence but in terms of state and, most importantly, in a way that is idempotent.

Firstly, we need the RVM installer; as it is just a single script and to save fetching it from the internet, let’s include it with our modules and use the source parameter we saw earlier to copy it into place with the right permissions and ownership…

file { '/root/rvm-installer':
  ensure => present,
  owner  => 'root',
  group  => 'root',
  mode   => '0700',
  source => 'puppet:///modules/mcp/rvm',
}

Now we need to run the installer and, for that, we can use the exec resource which allows you to run any arbitrary command. This is discouraged when an alternative is available but we have no other choice in this situation…

exec { 'install-rvm':
  command => '/root/rvm-installer --version 1.6.32',
  cwd     => '/root',
  unless  => 'grep 1.6.32 /usr/local/rvm/VERSION',
  path    => '/usr/bin:/usr/sbin:/bin:/sbin',
  require => Package['rvm-dependencies'],
}

You can see the command to run and the directory to run it from in cwd but don’t forget that I said that Puppet will keep running your resources every 30 minutes and, by default, this would continually re-install RVM. This is what I meant when I said that resources must be idempotent: you must be able to run them multiple times without side effect. In order to stop this from happening, we can use the unless parameter to define a command that, should it return successfully, means that this resource doesn’t need to be executed. The command that we are using is grep and we are checking to see that our particular version of RVM is installed; in this way, when RVM isn’t installed at all, the command will fail and therefore the installer will run and if the version is out of date, it will also run, thereby seamlessly upgrading your installation. Also note the require dependency which ensures that RVM is not installed before its dependencies are satisfied.

Now we can do rvm install ruby in much the same way:

exec { 'rvm install ruby-1.9.2-p290':
  creates => '/usr/local/rvm/rubies/ruby-1.9.2-p290',
  timeout => 1800,
  path    => '/usr/local/rvm/bin:/usr/bin:/usr/sbin:/bin:/sbin',
  require => File['install-rvm'],
}

Instead of using unless which runs any arbitrary command and checks its exit code, we can use creates which checks for the presence of a file. This resource will therefore only be installed if /usr/local/rvm/rubies/ruby-1.9.2-p290 doesn’t already exist. As compiling Ruby can take some time, we also bump up the default timeout so Puppet doesn’t assume something has gone wrong.

Next is the installation of nginx and Passenger for which we will also use the exec resource.

exec { 'install-passenger-3.0.8':
  command => 'rvm-shell ruby-1.9.2-p290 -c "gem install passenger -v3.0.8"',
  unless  => 'rvm-shell ruby-1.9.2-p290 -c "gem list passenger -v3.0.8 -i"',
  path    => '/usr/local/rvm/bin:/usr/bin:/usr/sbin:/bin:/sbin',
  timeout => 1800,
  require => Exec['rvm install ruby-1.9.2-p290'],
}

We need to always think about the two commands: one to run and one to check whether it has already been run. Here we are simply installing a gem but not if it is already installed (gem list -i just being something built into RubyGems).

Nginx is much the same:

exec { 'install-nginx':
  command => 'rvm-shell ruby-1.9.2-p290 -c "passenger-install-nginx-module..."',
  creates => '...agents/nginx/PassengerHelperAgent',
  timeout => 1800,
  path    => '/usr/local/rvm/bin:/usr/bin:/usr/sbin:/bin:/sbin',
  require => Exec['install-passenger-3.0.8'],
}

From our experience, we know that when Passenger and nginx compile successfully, it results in PassengerHelperAgent being present in a certain directory so we can use this to detect whether nginx has been installed or not. I’ve truncated some of the paths to fit on the slides but the top command simply has flags to perform a headless install of Passenger and looks like so:

$ rvm-shell ruby-1.9.2-p290 -c \
>  "passenger-install-nginx-module --auto \
>  --auto-download \
>  --prefix=/opt/nginx"

This is well-documented on the Passenger web site and just means that it will not prompt for user input when installing.

Finally, our nginx configuration uses the file resource:

file {
  '/opt/nginx/conf/sites_available/mcp.conf':
    ensure => present,
    owner  => 'root',
    group  => 'root',
    source => 'puppet:///modules/mcp/mcp.conf';

  '/opt/nginx/conf/sites_enabled/mcp.conf':
    ensure => link,
    owner  => 'root',
    group  => 'root',
    target => '/opt/nginx/conf/sites_available/mcp.conf',
}

This just demonstrates that you can also manage symbolic links with Puppet as well and we could easily disable the site by changing link to absent.

Then we’ve got our init.d script which we just copy:

file { '/etc/init.d/nginx':
  ensure => present,
  owner  => 'root',
  group  => 'root',
  mode   => '0755',
  source => 'puppet:///modules/mcp/nginx',
  require => Exec['install-nginx'];
}

Finally, we used update-rc.d to register our service previously but we can now use the service resource type to do that for us:

service { 'nginx':
  ensure     => running,
  enable     => true,
  hasrestart => true,
  hasstatus  => true,
  subscribe  => File['/opt/nginx/conf/nginx.conf'],
  require => [
    File['/opt/nginx/conf/nginx.conf'],
    File['/etc/init.d/nginx']
  ],
}

This is quite powerful as it describes the functionality of the service (whether it supports restarting natively, whether it can report its own status, etc.) and also sets up a special type of dependency in the subscribe parameter. By subscribing to the nginx configuration file, this service will automatically reload when any change to the configuration is made. This means that the web server can be tweaked and dependent services seamlessly reloaded without any need to run init.d scripts. You can also see a require dependency with more than one resource showing that you can have multiple dependencies at once.

Now that is complete, you might be thinking that it is a lot of work and maybe you only have one server, so what’s the point? I would simply ask if there was no part of that you have ever done twice; have you never had to create users or use the same directory structure more than once? If you have any sort of “best practices” for your server setup, would they not benefit from being documented in some way? What if that documentation took the form of a Puppet manifest or a Chef cookbook? If you have that then you have your one canonical source of information about your infrastructure, you can correct any mistakes in one place and watch it roll out across all your nodes and you can subject your infrastructure to normal code workflows: versioning, code review, etc. [NB: versioning is something I neglected to mention during my presentation but it is important to mention now.]

Testing

Puppet is obviously very powerful and, as with anything that runs as root, the possibility for damage is great. Therefore, it is important to be able to test your Puppet manifests before they are applied to live servers.

Puppet ships with a --noop flag you can use with puppet apply that will effectively do a dry-run of your changes. If you are using the file resource, for example, it will show you a diff of the changes it would have made without actually modifying the original files. While this can be useful as a sanity check, it falls down when doing anything slightly more involved as it is not really possible for Puppet to predict what state your system will be in after installing new software or running an arbitrary exec resource.

A much more thorough way to test your configuration is to make use of Vagrant. If you’re not familiar with it, Vagrant is a RubyGem for managing virtual machines using Oracle’s VirtualBox. Using this, you can effectively have one of your live servers set up as a virtual machine running on your development machine and then run your manifests against it. In this way you can see what effect your changes would have on a running system.

To make use of Puppet, you modify a file named Vagrantfile to point it at your modules and manifests:

config.vm.provision :puppet do |puppet|
  puppet.module_path    = "modules"
  puppet.manifests_path = "manifests"
  puppet.manifest_file  = "mcp.pp"
end

Then it is simply a case of bringing up your virtual machine:

$ bundle exec vagrant up
[default] Box natty was not found. Fetching box...
[default] Downloading with Vagrant::Downloaders::HTTP...
[default] Downloading box: http://mudge.name/downloads...
[default] Extracting box...
[default] Verifying box...
[default] Cleaning up downloaded box...
[default] Importing base box 'natty'...
[default] Matching MAC address for NAT networking...
[default] Clearing any previously set forwarded ports...
[default] Forwarding ports...
[default] -- ssh: 22 => 2222 (adapter 1)
[default] Creating shared folders metadata...
[default] Running any VM customizations...
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.

The first time you run it, Vagrant will inspect your Vagrantfile to download a base box (a blank slate virtual machine, e.g. a freshly installed Ubuntu server) and then provision it with Puppet. Then you can tweak your manifests and run the following command to do another Puppet run:

$ bundle exec vagrant provision
[default] Running provisioner: Vagrant::Provisioners::Puppet...
[default] Running Puppet with mcp.pp...

All of the examples in this presentation are actually available as a sample project with a Gemfile, Vagrantfile and all the modules and manifests required. Simply get it from GitHub, run bundle and then bundle exec vagrant up to download a base box I’m hosting on Dropbox and prepare a virtual machine for our mcp application including the user, SSH key (though it’s just a fake key so you won’t be able to log in with it), RVM, Ruby, nginx and Passenger as a demonstration of Puppet’s power.

While that is extremely useful, it is just another method of visual inspection; what might be better is to actual test-drive your Puppet manifests with a tool like Cucumber-Puppet. That allows you to write Cucumber features for your infrastructure:

Feature: General catalog policy
  In order to ensure a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario Outline: Compile and verify catalog
    Given a node specified by "features/yaml/eg.yml"
    When I compile its catalog
    Then compilation should succeed
    And all resource dependencies should resolve

    Examples:
      | hostname  |
      | localhost |

This is just a sample feature from their web site and I admit that I haven’t used this yet but it is definitely something I want to look into more closely. There are some exciting tools emerging for Chef regarding testing with minitest and my hope is that Puppet will also benefit from the enthusiasm in this area. [NB: shortly after my talk, there was an extensive post on testing Puppet on the Puppet Labs blog.]

Wrap-Up

You might be thinking that, as a developer, it is not really your job to worry about the maintenance of web servers but I would argue that being a developer doesn’t stop once the specs pass on your machine: your job is to deliver your software all the way to the customer and therefore successfully onto live servers. There is this gathering movement of “devops” trying to bridge the gap between operations and development and I think it is an obvious idea to collaborate more: no one ever really benefited from operating entirely in a silo. It’s a movement that has been picking up speed within our organisation and has definitely improved things from the dark days of throwing code and instructions over the wall and waiting for a ticket to come back with a minor misunderstanding or a simple missed step.

If you want to know more, Puppet Labs recently revamped all of their documentation so it is well worth reading “Learning Puppet”.

You can follow me on Twitter, @mudge: it’s mostly pictures of coffee and cakes but occasionally some technical stuff.

github.com/mudge is where you’ll find the source code to this presentation, it’s called managing_web_application_servers_with_puppet and it’s just a ShowOff presentation with the example directory I mentioned earlier.

Thanks for listening.

Combining and Compressing Disqus’ and Google Analytics’ JavaScript

2011-01-29T00:00:00+00:00

In setting up this Jekyll-powered blog, I had cause to dust off my Google Analytics account and finally take Disqus for a spin. Both services provide small snippets of JavaScript for inclusion in your web pages; Google Analytics’ looks much like this (with line breaks added for readability):

<script type="text/javascript">
  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXXX-X']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script');
    ga.type = 'text/javascript';
    ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') +
      '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0];
    s.parentNode.insertBefore(ga, s);
  })();
</script>

Disqus’ is as follows (with line breaks added and extraneous comments removed):

<script type="text/javascript">
  var disqus_shortname = 'example';
  // var disqus_identifier = 'unique_dynamic_id_1234';
  // var disqus_url = 'http://example.com/permalink-to-page.html';

  (function() {
    var dsq = document.createElement('script');
    dsq.type = 'text/javascript';
    dsq.async = true;
    dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
    (document.getElementsByTagName('head')[0] ||
      document.getElementsByTagName('body')[0]).appendChild(dsq);
  })();
</script>

Both seem terse enough ~~and are meant for inclusion at the bottom of your web page~~ (Update: enomar on Hacker News pointed out that Google’s asynchronous tracking code is ideally meant to be included at the end of the head element while Disqus’ JavaScript is meant for inclusion wherever you want your comments to appear but both can be placed at the bottom of your page, see “Getting Started with the Asynchronous Snippet” and the end of this post for more information).

A quick check with Amy Hoy and Thomas Fuchs’ “DOM Monster” shows that the number of script tags should ideally be kept to a minimum so let’s combine the two into one (and let’s specify those recommended Disqus variables while we’re at it):

<script type="text/javascript">
  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXXX-X']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script');
    ga.type = 'text/javascript';
    ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') +
      '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0];
    s.parentNode.insertBefore(ga, s);
  })();

  var disqus_shortname = 'example';
  var disqus_identifier = 'unique_dynamic_id_1234';
  var disqus_url = 'http://example.com/permalink-to-page.html';

  (function() {
    var dsq = document.createElement('script');
    dsq.type = 'text/javascript';
    dsq.async = true;
    dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
    (document.getElementsByTagName('head')[0] ||
      document.getElementsByTagName('body')[0]).appendChild(dsq);
  })();
</script>

We could stop at this point but I am something of a sucker for compressing web pages and JavaScript down to their bare minimum (I have been known to pore through the HTML5 specification’s “optional tags” section and butcher my markup appropriately).

We might decide to put the whole thing into something like Google’s Closure Compiler to optimise the code for us but let’s take it more slowly and start by looking at the script tag itself: it turns out that the type attribute is optional and is specified as having a default value of text/javascript so we can immediately lose that (Update: this is only true if you’re using HTML5). It turns out that the actual JavaScript itself is also creating script tags and specifying the type attribute so let’s remove those as well:

<script>
  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXXX-X']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script');
    ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') +
      '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0];
    s.parentNode.insertBefore(ga, s);
  })();

  var disqus_shortname = 'example';
  var disqus_identifier = 'unique_dynamic_id_1234';
  var disqus_url = 'http://example.com/permalink-to-page.html';

  (function() {
    var dsq = document.createElement('script');
    dsq.async = true;
    dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
    (document.getElementsByTagName('head')[0] ||
      document.getElementsByTagName('body')[0]).appendChild(dsq);
  })();
</script>

Another thing that has caught my eye is the odd set up of the _gaq variable: it checks to see if it has already been defined (which it has not in my case), assigns an empty array and then pushes two arrays onto it. This is due to the fact that _gaq can be modified after Google’s tracking code has loaded to use more advanced tracking features (see the documentation for _gaq.push for more information) but seeing as I am only doing basic analytics and taking a cue from Mark Pilgrim’s code from “dive into mark”, we can simplify this quite drastically:

<script>
  var _gaq = [['_setAccount', 'UA-XXXXX-X'], ['_trackPageview']];
  // ... omitted for brevity ...
</script>

While we are looking at variable declarations, another easy optimisation we can make is to take advantage of JavaScript’s syntax to declare multiple variables at once:

<script>
  var _gaq = [['_setAccount', 'UA-XXXXX-X'], ['_trackPageview']],
      disqus_shortname = 'example',
      disqus_identifier = 'unique_dynamic_id_1234',
      disqus_url = 'http://example.com/permalink-to-page.html';
  // ... omitted for brevity ...
</script>

With those low-hanging fruit out of the way, we need to consider what both snippets are actually doing. They are, in fact, very similar: they are both creating script tags set to load some external JavaScript on your page in the following way:

Create a new script element with document.createElement;
Inform the script that it is to load asynchronously by setting the async attribute;
Set the src of the element thereby identifying the location of the external script;
Insert the finished element into the web page.

With this knowledge, we can start to remove some repetition in the code. Firstly, we can streamline the creation of the two script tags by setting them up simultaneously (and moving them into the same closure while we’re at it):

<script>
  var _gaq = [['_setAccount', 'UA-XXXXX-X'], ['_trackPageview']],
      disqus_shortname = 'example',
      disqus_identifier = 'unique_dynamic_id_1234',
      disqus_url = 'http://example.com/permalink-to-page.html';

  (function() {
    var ga = document.createElement('script'),
        dsq = document.createElement('script');
    ga.async = dsq.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') +
      '.google-analytics.com/ga.js';
    dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
    var s = document.getElementsByTagName('script')[0];
    s.parentNode.insertBefore(ga, s);
    (document.getElementsByTagName('head')[0] ||
      document.getElementsByTagName('body')[0]).appendChild(dsq);
  })();
</script>

We’ve stated that both snippets insert their newly-created script elements into the DOM but they are currently doing it in different ways. Let’s change this and, in the spirit of following Google’s Performance Best Practices, let’s append the script elements to the body tag by using document.body and appendChild:

<script>
  var _gaq = [['_setAccount', 'UA-XXXXX-X'], ['_trackPageview']],
      disqus_shortname = 'example',
      disqus_identifier = 'unique_dynamic_id_1234',
      disqus_url = 'http://example.com/permalink-to-page.html';

  (function() {
    var ga = document.createElement('script'),
        dsq = document.createElement('script');
    ga.async = dsq.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') +
      '.google-analytics.com/ga.js';
    dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
    document.body.appendChild(ga);
    document.body.appendChild(dsq);
  })();
</script>

We’re now in much better shape than when we started and a lot closer to following the Don’t Repeat Yourself (DRY) principle.

If you want to take this even further then there are a few other changes that you might consider making:

If your site is only served via HTTP and not HTTPS then you can lose the 'https:' == document.location.protocol check;
You can remove the string concatenation that uses the disqus_shortname by explicitly setting the src to your own Disqus URL such as http://example.disqus.com/embed.js;
Instead of calling document.body twice, you can cache it once in a variable which you can then reuse.
A quick check with JSLint reveals that invoking a function defined inside parentheses (e.g. (function() { ... })()) should be done inside the parentheses instead of outside (e.g. (function() { ... }())).

If you decide to take these (admittedly more drastic) steps then you might end up with something like the following:

<script>
  var _gaq = [['_setAccount', 'UA-XXXXX-X'], ['_trackPageview']],
      disqus_shortname = 'example',
      disqus_identifier = 'unique_dynamic_id_1234',
      disqus_url = 'http://example.com/permalink-to-page.html';

  (function() {
    var ga = document.createElement('script'),
        dsq = document.createElement('script'),
        body = document.body;
    ga.async = dsq.async = true;
    ga.src = 'http://www.google-analytics.com/ga.js';
    dsq.src = 'http://example.disqus.com/embed.js';
    body.appendChild(ga);
    body.appendChild(dsq);
  }());
</script>

At this point you can now turn to more brutal compressors such as YUI Compressor or the aforementioned Closure Compiler which (after some tweaks to keep JSLint happy) will result in a single snippet like so (line breaks added for some semblance of readability):

<script>
var _gaq=[["_setAccount","UA-XXXXX-X"],["_trackPageview"]],
disqus_shortname="example",disqus_identifier="unique_dynamic_id_1234",
disqus_url="http://example.com/permalink-to-page.html";(function(){
var a=document.createElement("script"),b=document.createElement("script"),
c=document.body;a.async=b.async=true;a.src="http://www.google-analytics.com/ga.js";
b.src="http://example.disqus.com/embed.js";c.appendChild(a);c.appendChild(b);}());
</script>

Of course, there’s no reason for it to end there. I’ve got it down to 412 characters while still passing JSLint, feel free to post your own attempts in the comments:

<script>
var _gaq=[["_setAccount","UA-XXXXX-X"],["_trackPageview"]],
disqus_shortname="example",disqus_identifier="unique_dynamic_id_1234",
disqus_url="http://example.com/permalink-to-page.html";(function(){
var a=document,b=a.createElement("script"),c=a.body,d;
b.async=true;d=b.cloneNode(false);b.src="http://www.google-analytics.com/ga.js";
d.src="http://example.disqus.com/embed.js";c.appendChild(b);c.appendChild(d);}());
</script>

There are only two rules:

Your solution must not produce any errors from JSLint with “Assume a browser” ticked;
The only global variables that should be declared are _gaq, disqus_identifier, disqus_shortname ~~and disqus_url~~ (Update: Disqus software engineer Anton Kovalyov says in the comments that disqus_url is not required if disqus_identifier is set so feel free to leave that one out).

Update: Based on feedback from Steve Klabnik, Anton Kovalyov and Andrew Walker, here is a solution that consists of only 345 characters:

<script>
var _gaq=[["_setAccount","UA-XXXXX-X"],["_trackPageview"]],
disqus_shortname="example",disqus_identifier="unique_dynamic_id_1234";
(function(a,b){
var c=a.createElement("script"),d=a.body,e;
c.async=b;e=c.cloneNode(b);c.src="//www.google-analytics.com/ga.js";
e.src="//example.disqus.com/embed.js";d.appendChild(c);d.appendChild(e);
}(document,true));
</script>

If you want to follow Google’s recommendation of inserting their snippet at the bottom of the head element (and continue inserting Disqus’ snippet at the bottom of the body) then you will have to sacrifice the single, unified script for two separate ones:

<head>
  <!-- Usual HTML head elements here... -->
  <script>
    var _gaq=[["_setAccount","UA-XXXXX-X"],["_trackPageview"]],
    disqus_shortname="example",disqus_identifier="unique_dynamic_id_1234";
    function a(b){var c=document,d=c.createElement("script");
    d.async=true;d.src=b;
    c.documentElement.firstChild.appendChild(d);}
    a("//www.google-analytics.com/ga.js");
  </script>
</head>
<body>
  <!-- Page content here... -->
  <script>a("//example.disqus.com/embed.js");</script>
</body>

This version consists of a total of 325 characters but introduces a new global function named a which will create a new script element and append it to the head element of the page. The benefit of this approach is that it will immediately start loading the Google Analytics tracking code which might otherwise be delayed (particularly on large web pages) but the extra script and population of the head element won’t win you any favours with the DOM Monster.

Update #2: I have since taken to another approach that reduces the number of script elements created by synchronously requiring the Disqus JavaScript yourself at the bottom of the body:

<head>
  <!-- Usual HTML head elements here... -->
  <script>
    var _gaq=[["_setAccount","UA-XXXXX-X"],["_trackPageview"]],
    disqus_shortname="example",disqus_identifier="unique_dynamic_id_1234";
    (function(a){var b=a.createElement("script");b.async=true;
    b.src="//www.google-analytics.com/ga.js";
    a.documentElement.firstChild.appendChild(b);}(document));
  </script>
</head>
<body>
  <!-- Page content here... -->
  <script src=//example.disqus.com/embed.js async></script>
</body>

Passing Blocks in Ruby Without `&block`

2011-01-26T00:00:00+00:00

There are two main ways to receive blocks in a method in Ruby: the first is to use the yield keyword like so:

def speak
  puts yield
end

speak { "Hello" }
# Hello
#  => nil

The other is to prefix the last argument in a method signature with an ampersand which will then create a Proc object from any block passed in. This object can then be executed with the call method like so:

def speak(&block)
  puts block.call
end

speak { "Hello" }
# Hello
#  => nil

The problem with the second approach is that instantiating a new Proc object incurs a surprisingly heavy performance penalty as detailed by Aaron Patterson in his excellent RubyConf X presentation, “ZOMG WHY IS THIS CODE SO SLOW?” (beginning around the 30 minute mark or from slide 181).

This can easily be verified with the following benchmark, block_benchmark.rb:

require "benchmark"

def speak_with_block(&block)
  block.call
end

def speak_with_yield
  yield
end

n = 1_000_000
Benchmark.bmbm do |x|
  x.report("&block") do
    n.times { speak_with_block { "ook" } }
  end
  x.report("yield") do
    n.times { speak_with_yield { "ook" } }
  end
end

The results of this on my own machine are as follows (the numbers themselves aren’t as important as their difference):

$ ruby block_benchmark.rb 
Rehearsal ------------------------------------------
&block   1.410000   0.020000   1.430000 (  1.430050)
yield    0.290000   0.000000   0.290000 (  0.291750)
--------------------------------- total: 1.720000sec

             user     system      total        real
&block   1.420000   0.030000   1.450000 (  1.452686)
yield    0.290000   0.000000   0.290000 (  0.292179)

So it is clearly preferable to choose yield over &block but what if you need to pass a block to another method?

For example, here is a class that implements a method tell_ape which delegates to another, more generic method named tell. This sort of pattern is commonly done using method_missing but I’ll keep the methods explicit for simplicity:

class Monkey

  # Monkey.tell_ape { "ook!" }
  # ape: ook!
  #  => nil
  def self.tell_ape(&block)
    tell("ape", &block)
  end

  def self.tell(name, &block)
    puts "#{name}: #{block.call}"
  end
end

Such a thing is not possible with the yield keyword:

class Monkey

  # Monkey.tell_ape { "ook!" }
  # ArgumentError: wrong number of arguments (2 for 1)
  def self.tell_ape
    tell("ape", yield)
  end

  def self.tell(name)
    puts "#{name}: #{yield}"
  end
end

Neither does it work by using an ampersand:

class Monkey

  # Monkey.tell_ape { "ook!" }
  # TypeError: wrong argument type String (expected Proc)
  def self.tell_ape
    tell("ape", &yield)
  end

  def self.tell(name)
    puts "#{name}: #{yield}"
  end
end

However, there is a way to only create a Proc object when needed and that is to use the little known behaviour of Proc.new as explained in Aaron Patterson’s aforementioned presentation.

If Proc.new is called from inside a method without any arguments of its own, it will return a new Proc containing the block given to its surrounding method.

def speak
  puts Proc.new.call
end

speak { "Hello" }
# Hello
#  => nil

This means that it is now possible to pass a block between methods without using the &block parameter:

class Monkey

  # Monkey.tell_ape { "ook!" }
  # ape: ook!
  #  => nil
  def self.tell_ape
    tell("ape", &Proc.new)
  end

  def self.tell(name)
    puts "#{name}: #{yield}"
  end
end

Of course, if you do use Proc.new then you lose the performance benefit of using only yield (as Proc objects are being created as with &block) but it does avoid unnecessary creation of Proc objects when you don’t need them. This can be demonstrated with the following benchmark, proc_new_benchmark.rb:

require "benchmark"

def sometimes_block(flag, &block)
  if flag && block
    block.call
  end
end

def sometimes_proc_new(flag)
  if flag && block_given?
    Proc.new.call
  end
end

n = 1_000_000
Benchmark.bmbm do |x|
  x.report("&block") do
    n.times do
      sometimes_block(false) { "won't get used" }
    end
  end
  x.report("Proc.new") do
    n.times do
      sometimes_proc_new(false) { "won't get used" }
    end
  end
end

Which makes the following rather significant difference:

$ ruby code/proc_new_benchmark.rb 
Rehearsal --------------------------------------------
&block     1.080000   0.160000   1.240000 (  1.237644)
Proc.new   0.160000   0.000000   0.160000 (  0.156077)
----------------------------------- total: 1.400000sec

               user     system      total        real
&block     1.090000   0.080000   1.170000 (  1.178771)
Proc.new   0.160000   0.000000   0.160000 (  0.155053)

The key here is that using &block will always create a new Proc object, even if we don’t make use of it. By using Proc.new only when we actually need it, we can avoid the cost of this object instantiation entirely.

That said, there is a potential trade-off here between performance and readability: it is clear from the sometimes_block method signature that it takes a block and therefore will presumably do something with it; the same cannot be said for the more efficient sometimes_proc_new.

In the end, it comes down to your specific requirements but it is still a useful language feature to know.