In 2023, the Silicon Valley Bank (SVB) had a historic bank run, with investors withdrawing $42 billion in a single day. The Federal Reserve’s postmortem report essentially blamed two modern technologies — social media and online banking for the incident.

the combination of social media, a highly networked and concentrated depositor base, and technology may have fundamentally changed the speed of bank runs.

It was considered instantaneous in nature, well, compared to how such large-scale withdrawals happened in the past. And the correlation between social media and the bank run was so strong that an academic paper confirmed it with data:

During the SVB run period, banks with high pre-existing exposure to Twitter lost 4.3 percentage points more stock market value. ref: Social Media as a Bank Run Catalyst

But with the pervasive access to agentic AI, especially in the financial sector, which has been innovating with automated trading for years, people are rightly pointing out that the next financial disaster could happen in a fraction of the time it took for SVB to collapse.

Now imagine agents doing this

Here is an excerpt from the Washington Post’s AI & Tech brief today that captures this concern:

Agentic AI in finance “means you’re going to have bank runs in the middle of the night based on rumors on Reddit or Moltbook,” said Foster, referring to the social network built for AI agents.

For trading, now even the friction of humans socializing online and reacting to rumors (credibly or not) is removed.

Even AI agents can socialize now, which when I first heard about it, I thought was someone’s silly idea. But I kind of see how that can be useful somewhat, for personalized agentic workflows reacting to external signals. Who would not want to automatically buy tickets for a concert the moment the rumors of it being available start surfacing online?

But there can be serious consequences of such a system in the financial markets, especially when the agents are programmed by less experienced end users. Actually, it is less to do with the experience of the end users, but rather the scale such access opens up. Even if a small percentage of users have their agents set up to react to market signals, the sheer number of users and agents could lead to a cascade effect.

We recently had meme stocks like AMC and GameStop get hammered by users on online forums. I can’t imagine what would happen to companies if agents get into that game. I can imagine the authorities banning automated trading, but agentic AI can so very easily assume the identity of their users, launching regular browsers, navigating like humans, making it really difficult to police.

Crypto has no closing bell

And for assets that are traded 24/7, like cryptocurrencies, I wonder if there is even a way to put a circuit breaker in place. I mean crypto already has crazy volatility. With mass agentic AI trading, I can imagine it being a disaster.

We have already seen some of this happening recently.

On 11th October 2025, when a US tariff hike on China was announced, $19 billion worth of crypto was liquidated.

In the stock market, this would have triggered at least two trading halts and a restart auction. In the crypto world, it simply triggered panic among investors, who knew there was no “limit down” option to save them. They could only watch helplessly as crypto’s worst liquidation event unfolded while the code continued to perform its programmed task. ref: The Price of Freedom: Does Crypto Need Its Own Circuit Breakers?

And in February 2026, a vulnerability in AI trading agents on Solana DeFi led to a $45 million breach, with no opening bell to pause the damage. ref: AI Trading Agent $45M Crypto Breach

And they’re easy to fool

What makes it even worse is that these AI agents can very easily be fooled by misinformation inserted by prompt injections. A recent study from Google DeepMind identified six categories of “traps” that can exploit AI agents, and found that content injection attacks trapped AI agents 86% of the time in tested scenarios.

A fake financial report released at the right time could trigger synchronized sell orders among thousands of AI trading agents.

Cover photo by Jamie Street on Unsplash

I’d been hardcoding width and height attributes in my Hugo templates to prevent layout shift. It worked fine, but it was tedious — every time I changed an image, I had to look up the new dimensions and update the template by hand.

Today, I had to add a new image to the sidebar, and I felt lazy enough to ask copilot to find the dimensions for me and insert them into the template. It instead did something unexpected. It used an odd new Hugo function called imageConfig instead.

Curious, I looked it up. I haven’t kept myself up to date with Hugo’s latest features the last few years. The embarrassing part is that, as it turns out, this isn’t a new function. It was added in 2017 (!), but I hadn’t heard of it until now. I had no idea that Hugo could discover image dimensions at build time. Seems I really should read up more about what all Hugo can do.

Anyway, I found out that it is part of what Hugo calls “Hugo Pipes” — an asset processing pipeline that allows you to do a lot of things with your assets at build time.

It needs me to move my images from the static/ directory to the assets/ directory. And then I can use resources.Get (even imageConfig is considered to be legacy) at build time, discover the dimensions of the image, and get the URL via .RelPermalink.

So earlier, I had to do this in my template:

<img src="/images/kagi-small-web.png" width="202" height="64" 
     alt="Small Web seal">

Now, I can do this:

{{ with resources.Get "images/kagi-small-web.png" }}
  <img src="{{ .RelPermalink }}" width="{{ .Width }}" height="{{ .Height }}" 
       alt="Small Web seal">
{{ end }}

Today I discovered that there is a MCP server for Terraform which can provide all the provider metadata to AI editors, and the resultant code is just completely correct in one-shot. The best part is that AI editors can easily find the versions of providers you are really using, and can request documentation for those specific versions.

I made a note about how to add the MCP server to Cursor. Just a heads-up, you need to have Docker running on your machine to run the MCP server.

I have been using Helm charts, like everyone else, for the Kubernetes cluster in my homelab. Until a few months back, I never gave a thought to the reliability of the Helm chart repositories I was using. And then the Bitnami news dropped where they announced that they were going to stop supporting their Helm chart repositories.

Everyone has been scrambling to handle this situation, and most are settling on one of two options:

1. Vendor the sources of existing charts in the Git repositories.
2. Use a Helm chart repository, paid or free, to mirror them in a more scalable way.

I began with the first option, but it has left me feeling uneasy. So I looked into the second option, and found two alternatives: Chartmuseum and Harbor.

Harbor seemed full-featured and OCI-compliant, and it handled both Helm and Docker artifacts. But it seemed a bit overkill for my needs. So I went with Chartmuseum, which was pretty simple to set up with Docker compose on my Synology NAS and storage on my local Minio instance.

Another factor was that Chartmuseum supported multi-tenancy, which for some strange reason is missing from OCI registries like Harbor. I really don’t like the idea of having a flat namespace for charts from different organizations. Multi-tenancy allows me to create a separate mirror for each of the upstream repos, preventing even the slightest possibility of them sharing charts with the same name.

I documented my whole setup on my notes website, in case it helps someone else.

I normally install Ubuntu on my Raspberry Pi machines because I am comfortable with its ecosystem. Most of the time though, I have been using these boxes connected to my network via Ethernet.

Recently, I got a new Raspberry Pi 5, and as usual, I installed Ubuntu on it. This time I used the official HAT to install the OS on an NVMe drive. The Raspberry Pi imager tool does a great job of setting up the machine with Wi-Fi enabled. What I never paid attention to was how much the country setting in the options affects the Wi-Fi band.

After booting up the machine, I noticed that the Wi-Fi band was set to 2.4GHz. I found it odd. What followed was a lot of detail that, as usual, I wish I didn’t need to know, but now had to. :(

$ iwconfig wlan0
wlan0     IEEE 802.11  ESSID:"xxx"
          Mode:Managed  Frequency:2.462 GHz  Access Point: 60:22:32:xxx:xxx:xxx
          Bit Rate=72.2 Mb/s   Tx-Power=31 dBm
...

Turns out that the country setting affects what is called the regulatory domain (kernel.org) for the Wi-Fi configuration. This regulates the frequency bands that the wireless card can use.

Because I didn’t pay attention to the country setting, the first time it was mistakenly set to AE (United Arab Emirates). Because of this setting the kernel command line in Raspberry Pi bootloader was set to AE. I also saw that cloud-init configured netplan to set the AE regulatory domain on the wifi interface.

$ cat /mnt/cmdline.txt
console=serial0,115200 multipath=off dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc cfg80211.ieee80211_regdom=AE

$ cat /etc/netplan/50-cloud-init.yaml
network:
  version: 2
  wifis:
    wlan0:
      optional: true
      dhcp4: true
      regulatory-domain: "AE"

This was preventing the wifi card from using the 5GHz band. I tried to change the country setting by running iw reg set CA (Canada), and even changed the kernel command line, but it didn’t work.

root@rpi3:~# iw reg get
global
country CA: DFS-FCC
	(2402 - 2472 @ 40), (N/A, 30), (N/A)
	(5150 - 5250 @ 80), (N/A, 23), (N/A), NO-OUTDOOR, AUTO-BW
	(5250 - 5350 @ 80), (N/A, 24), (0 ms), DFS, AUTO-BW
	(5470 - 5600 @ 80), (N/A, 24), (0 ms), DFS
	(5650 - 5730 @ 80), (N/A, 24), (0 ms), DFS
	(5735 - 5835 @ 80), (N/A, 30), (N/A)
	(5925 - 7125 @ 320), (N/A, 12), (N/A), NO-OUTDOOR

phy#0
country 99: DFS-UNSET
	(2402 - 2482 @ 40), (6, 20), (N/A)
	(2474 - 2494 @ 20), (6, 20), (N/A)
	(5140 - 5360 @ 160), (6, 20), (N/A)
	(5460 - 5860 @ 160), (6, 20), (N/A)

While the global setting was set to Canada, it is overridden by the country setting for the phy#0 interface - the country 99: DFS-UNSET setting.

According to the Linux kernel documentation on wireless regulatory processing rules:

Upon the initialization of the wireless core (cfg80211) a world regulatory domain (highly restrictive) will be set as the central regulatory domain. If you get a 00 country code it means you are world roaming.

…

World roaming means we cannot initiate radiation on certain channels given that certain countries may prohibit initiating radiation on some channels or may require DFS master support prior to initiating any radiation.

This means that unless the phy#0 interface is set to a country other than the world roaming (country “00” is world roaming; “99” means the driver couldn’t determine a country code and therefore similar behavior), the wifi card will not initiate radiation on the 5GHz band.

And why is the phy#0 interface set to the world roaming when the global setting is set to Canada? I spent a few hours scouring the internet for answers, but didn’t find anything at first. Then I found a post about a person who switched back from Ubuntu to the Raspberry Pi OS and the wifi started working. Turns out that the Raspberry Pi OS uses an updated firmware version which honors the global setting.

I could confirm this by checking the firmware version on the Raspberry Pi OS and Ubuntu.

$ # On Raspberry Pi

$ cat /lib/firmware/brcm/brcmfmac43455-sdio.bin | strings| grep -Eo "Version: \S+"
Version: 7.45.265
$ lsb_release -d
Description:	Debian GNU/Linux 13 (trixie)

$ # On Ubuntu

$ zstdcat /lib/firmware/brcm/brcmfmac43455-sdio.bin.zst | strings| grep -Eo "Version: \S+"
Version: 7.45.234
$ lsb_release -d
Description:	Ubuntu 24.04.3 LTS

For now, I will stick with the Raspberry Pi OS, till the firmware on Ubuntu releases catches up.

What an absolute waste of time otherwise, but learning about the logic around kernel wifi regulatory domain handling was fascinating.

One annoyance I found a solution for, is getting the bash completion for Homebrew commands to work on Linux. The problem is that Ubuntu (and other Linux distributions) have their own bash completion scripts for system commands. But the way most bash completion scripts work is that they have a check to see if completions are already loaded.

So if I have a line in my bashrc to load homebrew completions, it won’t load, because it detects the completions from the system already loaded. Specifically, it looks for the environment variable BASH_COMPLETION_VERSINFO which is set by any bash completion script that is already loaded.

Solution

The only satisfactory solution I found is to disable the bash completion for system commands.

mv /etc/profile.d/bash_completion.sh /etc/profile.d/bash_completion.sh.disabled

My only concern was that I might lose completions for system commands. However, it seems the Homebrew completion loader also checks for system completion scripts.

You can verify that it works here:

$ # A homebrew command

$ which helm
/home/linuxbrew/.linuxbrew/bin/helm

$ complete -p helm
complete -o default -F __start_helm helm

$ # A system command

$ which apt
/usr/bin/apt

$ complete -p apt
complete -F _apt apt

Reddit used a testing technique called “tap compare” for read migrations. The concept is straightforward:

• A small percentage of traffic gets routed to the new Go microservice.
• The new service generates its response internally.
• Before returning anything, it calls the old Python endpoint to get that response too.
• The system compares both responses and logs any differences.
• The old endpoint’s response is what actually gets returned to users.

This approach meant that if the new service had bugs, users never saw them. The team got to validate their new code in production with real traffic while maintaining zero risk to user experience.

Other references that I have found on this topic:

• Tap Compare Testing with Diferencia and Java Microservices (InfoQ)

You can now install the latest Python 3.14 alpha with uv pic.twitter.com/yw24LKW7nq

— Charlie Marsh (@charliermarsh) January 7, 2025

I tried it out.

$ uv python install 3.14
Installed Python 3.14.0a3 in 1.22s
 + cpython-3.14.0a3-macos-aarch64-none

You can only run the installed Python via uv.

$ uv run --python 3.14 python
Python 3.14.0a3 (main, Jan  5 2025, 06:22:51) [Clang 19.1.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Interestingly, uv can detect Pythons installed via other mechanisms.

$ uv python list --only-installed
cpython-3.14.0a3-macos-aarch64-none    .local/share/uv/python/cpython-3.14.0a3-macos-aarch64-none/bin/python3.14
cpython-3.12.5-macos-aarch64-none      /opt/homebrew/opt/python@3.12/bin/python3.12 -> ../Frameworks/Python.framework/Versions/3.12/bin/python3.12
cpython-3.12.5-macos-aarch64-none      .asdf/installs/python/3.12.5/bin/python3.12
cpython-3.12.5-macos-aarch64-none      .asdf/installs/python/3.12.5/bin/python3 -> python3.12
cpython-3.12.5-macos-aarch64-none      .asdf/installs/python/3.12.5/bin/python -> python3.12
cpython-3.11.9-macos-aarch64-none      /opt/homebrew/opt/python@3.11/bin/python3.11 -> ../Frameworks/Python.framework/Versions/3.11/bin/python3.11
cpython-3.11.6-macos-aarch64-none      .asdf/installs/python/3.11.6/bin/python3.11
cpython-3.9.6-macos-aarch64-none       /Applications/Xcode.app/Contents/Developer/usr/bin/python3 -> ../../Library/Frameworks/Python3.framework/Versions/3.9/bin/python3

And you can run those Python installations via uv.

$ uv run --python 3.9.6 python --version
Python 3.9.6

$ uv run --python 3.11.9 python --version
Python 3.11.9

$ uv run --python 3.12.6 python --version
Python 3.12.6

However, that last one above was not in the list. uv downloaded it on-demand and ran it. This seems to be the default behavior which can be overridden by an environment variable.

Let me uninstall and try again with a command line option which is equivalent to the environment variable.

$ uv python uninstall 3.12.6
Searching for Python versions matching: Python 3.12.6
Uninstalled Python 3.12.6 in 300ms
 - cpython-3.12.6-macos-aarch64-none

$ uv run --no-python-downloads --python 3.12.6 python --version
error: No interpreter found for Python 3.12.6 in virtual environments, managed installations, or search path

Homebrew has two versions of bash-completion: bash-completion and bash-completion@2. The former is meant for bash 3.x and the latter is for bash 4+.

Normally it hasn’t mattered to me till now, because most formulas set up compatible completion scripts.

But it seems Taskfile’s completion script is only compatible with bash-completion@2.

Now normally I would just create a socks/http proxy to one of my home machines and set the proxy environment variable like HTTP_PROXY and most apps would just work. But Ansible doesn’t seem to respect that environment variable.

There is an environment keyword that lets you set http_proxy variables, but that is for tasks executing remotely. They can use that environment variable for commands they are executing which need to call over the Internet. But what we need is a way to reach the target host in the first place.

Jeff Geerling has a post about using ssh args in ansible.cfg which looked to be exactly what I need.

But then I discovered the -J jump host parameter, which for some reason I had never spotted before all these years.

-J destination

Connect to the target host by first making an ssh connection to the jump host
described by destination and then establishing a TCP forwarding to the
ultimate destination from there.  Multiple jump hops may be specified
separated by comma characters.  This is a shortcut to specify a ProxyJump
configuration directive.  Note that configuration directives supplied on the
command-line generally apply to the destination host and not any specified
jump hosts.  Use `~/.ssh/config` to specify configuration for jump hosts.

So to actually use this, I had to add the following to ansible.cfg:

# For tailscale
[ssh_connection]
ssh_args = -J guardian.rolling-doe.ts.net

Here guardian.rolling-doe.ts.net is the (fictional) Tailscale hostname of the host I am trying to use as a bastion host.

A new study released Thursday by research group Epoch AI projects that tech companies will exhaust the supply of publicly available training data for AI language models by roughly the turn of the decade—sometime between 2026 and 2032. …

In the longer term, there won’t be enough new blogs, news articles and social media commentary to sustain the current trajectory of AI development, putting pressure on companies to tap into sensitive data now considered private—such as emails or text messages—or relying on less-reliable “synthetic data” spit out by the chatbots themselves.

As more and more slop keeps getting churned out on the internet and floods our search results, there is little and little incentive anyway for humans to generate content in the first place. This data doom feels like a self-fulfilling prophecy.

One of the major reasons I switched to VS Code completely some years back is its excellent extension system, in particular the seamless remote SSH editing extension.

Remote SSH extension is really cool!

Using a local editor on a remote filesystem without fiddling with sshfs or the like, managing to use extensions like Python, Go and Copilot, all setup and configured locally but running on content on a remote system has been pretty cool. It helps me keep my office and personal content separate - I edit my personal codebase on my personal computers using vscode on my office laptop when I need to.

The annoyance about the code cli

But one of the most annoying thing while using the VS Code remote ssh extension was the VS Code cli, which I use a lot. I spend a lot of time on the terminal inside VS Code, and sometimes it is just easier to open a file from the command line using code FILE instead of reaching for the mouse to click on the file in the explorer.

Here is where it gets tricky.

If you have a VS Code installation on the remote computer, and you used it to install the CLI, that CLI executable (On Mac installed in /usr/local/bin/code ) will always open files in that local VS Code installation. It will not open the file in the remote SSH VS Code workspace that you have open.

TLDR: Use the script at the end of this post instead of the code cli in path if you want to open files in the remote ssh extension workspace.

This has something to do with how the CLI discovers the “server” instance - the VS Code instance that is running on machine and which hosts the editor. The communication happens via a socket, and the path to the socket is discovered using a combination of the CLI path and environment variables.

So if you use the code cli in the terminal of the remote SSH workspace, it will open the file in the local VS Code.

Incidentally, the remote SSH extension installs a VS Code installation on the remote server in the home directory, and it has its own cli executable. If you use this cli instead of the one in the path, it will discover the remote ssh workspace correctly and open the file in it.

So all we have to do is to discover this cli path correctly and use it instead of the one in the path. I found a suitable environment variable set by the core Git extension which can be used to find this. I used the bash function below in my bashrc to add a command (code_remote) that I now use instead for opening files in the terminal of remote SSH extension workspaces.

I had removed disqus comments from my blog a few years back when I noticed the horrible privacy issues with Disqus.

But I always felt that I was missing out on some feedback I had received from visitors in the past, especially when I would benefit from correcting my mistakes.

This article from Chris Wilson resonated with me.

I wanted:

1. Feedback
2. Didn’t want to spend money on a low traffic, zero commercialized blog.
3. Some friction for commenters by requiring identity to reduce comment spam.
4. Relatively low privacy impact.

Github fits those requirements. Most developers already use Github to some extent for either their own source control or interacting with other software hosted there. I am aware of the criticism of Github and I use Gitea at home myself (big fan). But what is important to me is to reduce 1000 cuts of cloud subscription for hosting it myself.

So, like Chris, I decided to install giscus for comments on this website. Feel free to use the comments for this post to test out how it works - I am ok with unrelated comments for this post as long as it is no spam/promotion of any kind.

One of the niftiest tools that I have been using a lot nowadays is the deceptively simple encryption/decryption command line tool called age by Filippo Valsorda.

Encrypting secrets using keypairs instead of a single passphrase is obviously a more secure option, as it separates the concerns for encryption and decryption. Therefore you can share around the public key without fear for others to encrypt data for you that only you can decrypt.

Traditionally, we have been using tools like gpg e.g. while using eyaml in Puppet, etc. However, gpg comes with its own overhead of key management - multiple keyrings, tool configuration, etc. which makes the use case of simple encryption to be fairly complicated.

age packs a lot of functionality into its dead simple CLI interface - a single file to “manage” the keypair. It requires zero configuration. More importantly, it supports not just its own X25519-based key pair format, but it can even use SSH key pairs for encryption. For simple use cases, it even supports symmetric passphrases.

I have been keeping this document in my PKM for a while, and I thought it best to share in public as a cookbook as well.

• Installation
• Usage
  • Key generation
  • Encryption
  • Decryption
  • Identity encryption
  • Using SSH keys for encryption
  • Using a simple password for encryption instead of a key pair
• Other uses

Installation

I mostly work on Mac nowadays, and prefer to use homebrew for installing age. Instructions for other platforms are in the official docs.

$ brew install age
...

Usage

Key generation

Before you can accept encrypted content, you need to generate your keypair. Key generation gives us a public key and a private key. The public key is also referred to as a recipient in the tool documentation.

$ age-keygen
# created: 2023-07-06T22:35:40-04:00
# public key: age1lm45u4v4nka4s3uv68dpfv2c33fdgak6mjvdn5sfsv8rlvss9vyss927mg
AGE-SECRET-KEY-167YP7YLNUE74Y7PA3X9SKXCS92JJ2DQCVN584WV20Y9WW5VL67ZSFFLDZ4

Specifying a file to save, using -o, will store the previously shown output to the file. But it hides the secret key from being displayed to the console. It still displays the public part.

$ age-keygen -o key.txt
Public key: age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t

$ cat key.txt
# created: 2023-07-06T22:39:59-04:00
# public key: age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t
AGE-SECRET-KEY-1FP43QXCQ3VM4EJANLG3DJ0VRHZD9FY0HW4RM5WVP9JNPZPVARD4SGAXD94

You can extract the public key from a key file using age-keygen -y, similar to how you might have used ssh-keygen -y for ssh keys.

$ age-keygen -y key.txt
age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t

However, unlike ssh-keygen, you can extract public keys from a bunch of age identity files in one shot.

$ cat key*.txt | age-keygen -y
age1sa3srvuxwx4g4sshgxvgc0cx8c5kwukv5kmvsmnp3d43levctehsla3m2f
age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t

Encryption

You encrypt using a recipient name. The output is by default in an efficient binary form. You can use the -a option to armor the output into a printable version.

$ cat /etc/hosts | age -r age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t -o hosts-1.age

$ file hosts-1.age
hosts-1.age: data

$ cat /etc/hosts | age -r age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t -a
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKdXpDQURkeVUzNmxKdk9X
ckgvS2FYZ2laRjlCZzUvV3IwYzMwcGNpakY0Cm1WeC9DakFFMUZjWGdqalYweHB3
MkppYlV3YXhNenpxSlg4MEhsV1R2ckEKLS0tIFQvcjMvVFZLM1UrRWlRYTVNenFl
WU93bFcwT011dFpLektWTnJzZkFoNVUKNd5jWDu9jzly/x9NPRFwTiTSPDrSn5A3
8UJ6sPtY4s2z6rlH6v2di3/PYympUNsJFlYIVchOor8YhivI80JWOZyMdge2h6Qu
49C+pEftVeyqYcaAhusgcQREWgXLp53GbpLbsWst4tiXFE5Hku4M5nLGIMrPjDw7
bPR+/9GFLVyBEAF27iwFGpFsR4ywKzgwgbLWJBvzZfv13/g6GDu8XAmMOUc3aGr9
7oUq6SNNmHWhpVImZc/h2a73LVPZqmXj515jx0vIYaBsaZMEMPFybiIETrNXTuQO
UEl6WrDniCtfxz9hvNMt4kkRoj2F8ebv/yYK+tc=
-----END AGE ENCRYPTED FILE-----

Repeated encryption of the same file for the same recipient will give different output. Good thing to keep in mind if you were planning on using file hashes to check if the encrypted files have been generated by the same key. You shouldn’t.

$ cat /etc/hosts | age -r age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t -o hosts-2.age
$ diff -u hosts-1.age hosts-2.age
Binary files hosts-1.age and hosts-2.age differ

You can specify multiple recipients by repeating the -r parameter, or put them all in a recipients file and specifying the file using -R.

$ age-keygen -o key-alt.txt
Public key: age1sa3srvuxwx4g4sshgxvgc0cx8c5kwukv5kmvsmnp3d43levctehsla3m2f

$ cat key*.txt | age-keygen -y > recipients.txt

$ cat recipients.txt
age1sa3srvuxwx4g4sshgxvgc0cx8c5kwukv5kmvsmnp3d43levctehsla3m2f
age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t

$ cat /etc/hosts | age -R recipients.txt -o hosts-3.age

The same encrypted file can be decrypted by both the recipients. That is, you don’t have to generate different encrypted versions for each recipient.

The recipients file format supports comments, so you can document the recipient keys.

$ cat recipients.txt
# Build key
age1sa3srvuxwx4g4sshgxvgc0cx8c5kwukv5kmvsmnp3d43levctehsla3m2f

# QA key
age1l36wdcfx825h727h5wjs7swh675f3jd4qde0zj47gyslmuja3agspge22t

You can mix SSH public keys and age public keys in the recipients file.

$ cat ~/.ssh/id_github.pub ~/.ssh/id_rsa.pub > r.txt

$ age-keygen -y ~/.age-chezmoi.txt >> r.txt

$ age -R r.txt -a < /etc/hosts
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHNzaC1yc2EgU0wyUlJBCk5OS2QzSzRl
ZCtVUmJxMXgwVVRBNjFJWU00emxwbmRFbmlsWDRCUlhwNjF2WEJoZUJtZysxSEVm
...

Decryption

Here the same file encrypted for both the recipients can be decrypted with their respective private keys.

$ age -d -i key.txt hosts-3.age
##
# Host Database
#
...

$ age -d -i key-alt.txt hosts-3.age
##
# Host Database
#
...

Identity encryption

Normally the encrypted file can only be decrypted by the recipient public key. Even you cannot decrypt it afterwards.

You can specify your own public key as recipient to work around it. Or you can just specify your key file (using -i), containing your private key, for an identity encryption.

Here the file is being encrypted for a different recipient, as well as for the person having a private key locally.

$ age -e -i key.txt -r $(age-keygen -y key-alt.txt) -a /etc/hosts | age -d -i key.txt
##
# Host Database
#

Using SSH keys for encryption

New versions of age can also use your existing SSH key pairs for encryption/decryption.

$ cat /etc/hosts | age -R ~/.ssh/id_rsa.pub -o hosts-4.age

$ age -d -i ~/.ssh/id_rsa hosts-4.age
##
# Host Database
#
...

While doing decryption, if your SSH private key was encrypted, you will be asked for the passphrase.

Using a simple password for encryption instead of a key pair

For even simpler use cases, age also supports simple passphrase based encryption.

$ cat /etc/hosts | age -p -o hosts-5.age
Enter passphrase (leave empty to autogenerate a secure one):
Confirm passphrase:

$ age -d hosts-5.age
Enter passphrase:
##
# Host Database
#

Other uses

age is a simple data encryption/decryption tool. It is not aware of file formats. So if you wanted to encrypt just a single key value in an YAML file, you will need to extract the value from the file, provide it to age for encryption, and then add the (probably armored) value back to the file.

Higher level tools like Mozilla SOPS understand file structure of formats like YAML, JSON etc. and can use age for encryption/decryption for individual values in these data structures.

For a while now, I have been wanting to see the options I have today to convert speech to text at the command line. Lots of commercial options exist today for doing that, but I also wanted to check out open source options.

In this experiment, I decided to compare:

• An opensource option: spchcat
• A commercial option - Google Cloud’s ML Speech API

Table of Contents

• spchcat
• Google Cloud’s ML Speech API
• Audio samples that I tried out
  • Sample one from Kaggle
  • Sample two from a speech by Gandhi
• Prepping the audio samples
• Trying out the first sample
• Trying out the second sample
• Punctuation in transcription
• Conclusion
• References

spchcat

spchcat describes itself as:

spchcat is a command-line tool that reads in audio from .WAV files, a microphone, or system audio inputs and converts any speech found into text.

It runs locally on your machine, with no web API calls or network activity, and is open source.

It is built on top of Coqui’s speech to text library, TensorFlow, KenLM, and data from Mozilla’s Common Voice project.

It seems to be written in C, and only has pre-built debian packages for amd64 and arm (for Raspberry pi).

Interestingly, it can stream convert audio from a microphone to text, which lets us do creative real time transcription applications.

Google Cloud’s ML Speech API

Google Cloud’s ML Speech API lets you use the gcloud CLI tool to transcribe audio from the command line. It has streaming abilities as well.

It is pretty useful for anybody to play around with speech-to-text without paying, as it has generous enough free limits for dabbling.

New customers get $300 in free credits to spend on Speech-to-Text. All customers get 60 minutes for transcribing and analyzing audio free per month, not charged against your credits.

Audio samples that I tried out

I used two different audio samples to compare the tools.

Sample one from Kaggle

The first one is a short (18s) audio sample from a Kaggle data set. They seem to be Harvard Sentences, used in speech quality measurements. [Src]

Actual transcript:

The stale smell of old beer lingers. It takes heat to bring out the odor. A cold dip restores health and zest. A salt pickle tastes fine with ham. Tacos Al Pastor are my favorite. A zestful food is the hot-cross bun.

Sample two from a speech by Gandhi

I also wanted to test how well these two software do with non-native English accents. For this, I picked up the audio from Mahatma Gandhi’s speech at Kingsley Hall in London, on October 1931. [wikimedia src]

Actual transcript of the first 60 seconds of the audio (reason explained later in the post):

There is an indefinable mysterious power that pervades everything. I feel it though I do not see it. It is this unseen power which makes itself felt and yet defies all proof, because it is so unlike all that I perceive through my senses. It transcends the senses. But it is possible to reason out the existence of God to a limited extent. Even in ordinary affairs we know that people do not know who rules, or why, and how He rules and yet they know that there is a power that certainly rules.

In my tour last year in Mysore I met many poor villagers and I found upon inquiry that they did not know who ruled Mysore. They [t58] simply said some [t60] God ruled it. …. (full text in the link below)

– Mahatma Gandhi’s famous speech at Kingsley Hall in 1931

Prepping the audio samples

Both the apps didn’t like the mp3 format audio that I had. WAV files are universally supported. This made me convert the audio format using ffmpeg.

$ ffmpeg -i harvard.mp3 harvard.wav

But gcloud also refused to process stereo audio.

$ gcloud ml speech recognize harvard.wav  --language-code=en-US
ERROR: (gcloud.ml.speech.recognize) INVALID_ARGUMENT: Must use single channel (mono) audio, but WAV header indicates 2 channels.

So I had to convert the audio to mono adding the -ac 1 parameter to ffmpeg

ffmpeg -i harvard.mp3 -ac 1 harvard.wav

For the Gandhi speech, gcloud also asked for extra configuration for audio longer than 60 seconds.

$ gcloud ml speech recognize gandhi.wav    --language-code=en-US
ERROR: (gcloud.ml.speech.recognize) INVALID_ARGUMENT: Sync input too long. For audio longer than 1 min use LongRunningRecognize with a 'uri' parameter.

So I truncated the audio to 60 seconds using ffmpeg.

$ ffmpeg -i gandhi.mp3 -t 60 -ac 1 gandhi_t60.wav

Weirdly, gcloud still complained of the length of the audio. So I finally truncated it to 58 seconds, and that worked.

Trying out the first sample

This was a short speech by a native English speaker, speaking kind of slowly and carefully, presumably for clarity.

spchcat handled most of it correctly. But completely truncated the last spoken sentence. I am not sure if this is a misconfiguration at my end or a problem with the software.

$ spchcat ./harvard.wav
TensorFlow: v2.3.0-14-g4bdd3955115
 Coqui STT: v1.1.0-0-gf3605e23
rate: rate clipped 13 samples; decrease volume?

the stale smell of old bear lingers it takes heat to bring out the odor a cold dip restores health and zest a salt pickled taste fine with ham tackles all pastor are my favorite existlessness

glcoud was completely on point.

$ gcloud ml speech recognize harvard-mono.wav   --language-code=en-US
{
  "requestId": "4451273191690268245",
  "results": [
    {
      "alternatives": [
        {
          "confidence": 0.9559944,
          "transcript": "the stale smell of old beer lingers it takes heat to bring out the odor a cold dip restores health and zest a salt pickle taste fine with ham tacos al Pastore are my favorite a zestful food is the hot cross bun"
        }
      ],
      "languageCode": "en-us",
      "resultEndTime": "18s"
    }
  ],
  "totalBilledTime": "18s"
}

Trying out the second sample

spchcat started well, and again completely botched it at the end. It makes no sense that this is a problem with its speech detection. If I have to guess, it probably has got to do with stream buffering sync issues.

$ spchcat ./gandhi_t60.wav | fold -s
TensorFlow: v2.3.0-14-g4bdd3955115
 Coqui STT: v1.1.0-0-gf3605e23
there is an indefinable mysterious power that pervades every thing i feel it
though i do not see it it is this unseen power which makes itself felt and yet
defies all proof because it is so unlike all that i perceive through my senses
it transcends the senses but it is possible to reason out the existence of god
to his relatedness know that people do not know who rose or hand how he does
and yet they know that there is a power let certainement very poor pieni found
upon inquiry that he did not know who ruled my sore the simple

gcloud did also started reasonably well. But completely fudged it towards the end, missing whole words (“Even in ordinary affairs”), and then completely mixing them up.

$ gcloud ml speech recognize gandhi_t58.wav --language-code=en-US \
  | fold -s
  "there is an indefinable mysterious power that pervades everything I feel it so
  I do not see it it is this unseen power which makes itself heard and yet
  devised all proof because it is so unlike all that I perceive through my senses
  it transcends the changes but it is possible to reason out the existence of God
  to a limited extent we know that people do not know who rules or why and how he
  rules and yet they know that there is a power that certainly who's in my tour
  last year in My Sword I made many poor villagers and I found upon inquiry that
  they did not know who ruled my soul"

I tried some additional parameters to see if they fix the problems with gcloud:

• Providing an encoding rate (e.g. --sample-rate 44100) didn’t make any difference.

• Using the latest_lang model however, made a big difference in the accuracy:

  $ gcloud ml speech recognize gandhi_t58.wav \
    --language-code=en-US --model latest_long \
    | fold -s
  "there is an indefinable mysterious power that pervades everything I feel it
  though I do not see it it is this unseen power which makes itself felt and yet
  defies all proof because it is so unlike all that I perceive through my senses
  it transcends the senses but it is possible to reason out the existence of God
  to a limited extent even in ordinary Affairs we know that people do not know
  who rules or why and how he rules and yet they know that there is a power that
  certainly rules in my tour last year in my soul I met many poor villagers and I
  found upon inquiry that they did not know who ruled my soul"
  

Punctuation in transcription

All transcription seems bereft of any punctuation - sentences run into each other.

I tried the automatic punctuation feature in gcloud, and it sorta worked in the second sample. Even in the smaller sample with clear native speech, it didn’t do so well.

$ gcloud ml speech recognize harvard-mono.wav  --language-code=en-US \
  --model latest_long --enable-automatic-punctuation \
  | jq '.results[].alternatives[].transcript'| fold -s
"the stale smell of old beer lingers"
" it takes heat to bring out the odor."
" A cold dip restores health and zest a salt pickle tastes fine with ham tacos
al pastor are my favorite a zestful food is the hot cross bun."


$ gcloud ml speech recognize gandhi_t58.wav --language-code=en-US \
  --model latest_long --enable-automatic-punctuation \
  | jq '.results[].alternatives[].transcript'| fold -s
"There is an indefinable mysterious power that pervades everything. I feel it
though. I do not see it it is this unseen power which makes itself felt and yet
defies all proof because it is so unlike all that. I perceive through my
senses. It transcends the senses but it is possible to reason out the existence
of God to a limited extent even in ordinary Affairs. We know that people do not
know who rules or why and how he rules and yet they know that there is a power
that certainly rules in my tour last year in my soul. I met many poor villagers
and I found upon inquiry that they did not know who ruled my soul."

Conclusion

In my two tests, gcloud is definitely more accurate than spchcat. But just two audio samples are ridiculously small data points to derive conclusions on these tools. Accuracy will definitely be improved with the accent, but also there are many other tunables to explore with both these tools, that I haven’t yet touched.

But given the cost factor of gcloud, playing around a lot for free with transcription is probably only feasible with spchcat, so we should definitely not rule it out.

This is just my initial foray in this domain. I might revisit these samples after I understood some of the underlying theories better.

References

• Blog post by the author of spchcat annoucing the tool
• Recognizing Speech with a Raspberry Pi: A demo of doing real time speech transcription on a Raspberry PI using spchcat

I wanted to quickly jot down some of the PDF tasks that the wonderful qpdf has been helping me do. This ranges from merging multiple PDF files to storing decrypted versions of annoying PDFs sent by some banks.

I used to use pdftk till some time back, but it had a lot of dependencies which were a pain to install. I exclusively use qpdf now instead.

Concatenating PDFs

Normally qpdf expects an input file for all operations. But while concatenating PDFs, you would prefer a blank slate to which you will be adding the files. The trick is to use --empty as input, and then add the pages.

--empty

This option may be given in place of infile. This causes qpdf to use a dummy input file that contains zero pages. This option is useful in conjunction with --pages.

This following simple syntax to add the pages from other PDF documents to the input file was added a few years back.

If I want to concatenate multiple PDFs into one, I use:

qpdf --empty --pages *.pdf -- out.pdf

Decrypting PDFs

I had mentioned this in a prior blog post a while back (14 years ago! :old_man:).

qpdf --decrypt --password=mypassword input.pdf output.pdf

I actually now use a script to wrap this, which lets me forget the specific syntax of the command 😄 and also gives me various invocation options as documented at the start of the script.

As part of my journey in learning Kubernetes this year, I ventured today in pushing my first tiny helm chart to Artifacthub.

I had been using custom made charts for a couple of months now, but mostly in the CI/CD environment of my company. Packaging a chart for public consumption needed me to learn a few more things.

• Getting my charts bundled up a compliant helm repo format. Turns out there is a Github action which helps in creating one using Github pages.

  This gives you a nice web page where you can put a README to help people with instructions on how to use your charts.

  Keeping the README synced on that webpage required a separate action workflow which I stole from the prometheus-community chart repo.

• The next step was discoverability. I created a repository on Artifacthub and fiddled around with both the annotations in Chart.yaml and the artifacthub-repo.yml file till I got most of it working. I still find the changelog annotations feature to be awful though.

  Copying the unique id for your repo from your artifacthub control panel to artifacthub-repo.yml gave me the fancy Verified Publisher badge on my chart, which was nice.

Now the hard part - continuing to read around for best practices to write good charts for public consumption.

This was primarily driven by a recent privacy test I did on the blog, and to my utter shock, almost a dozen different websites seem to be contacted with every page load. I generally expected Google (for the analytics) and Disqus (for the comments) to be the websites figuring out here, so I felt absolutely betrayed by the bunch of other websites they call under the hood.

For comments, it is not worth keeping open an avenue for my handful of visitors who may give me some feedback. I have a contact page and a Twitter profile where people have contacted me in the past. That should be good enough for me, I guess.

For stats, which I would still love to have, I am presently trying out Plausible, to see if it is worth paying $9 per month.

When using Ansible for my home lab, an initial problem was about how to keep sudo passwords for my various machines in an practical manner (I really don’t like the idea of password-less sudo even in my homelab).

The remote users used for my ansible logins to each of my machines are different, and can be managed via the inventory file. But the sudo passwords for them are not the same, and it is pretty annoying to enter them while running ansible on the command line.

I decided to find a way to make this a little less annoying.

Storing the encrypted sudo passwords

I first started keeping the individual sudo passwords for my hosts in an ansible-vault encrypted file outside my repo. Decrypted, they would look like this:

$ ansible-vault view ~/.ansible/sudo_passwords.yml
---
become_passwords:
  host1: host1password
  host2: host2password

Using the encrypted sudo passwords automatically

To use these passwords in my playbooks transparently, I did two things:

1. I put this in a group_vars/all/111_setpasswords.yaml file. This will now make Ansible look for sudo passwords from this variable that is kept inside the encrypted file:

   ---
   ansible_become_pass: '{{ become_passwords[inventory_hostname] }}'
   

2. Now to make Ansible read the encrypted file right at the beginning, I put a symlink to my encrypted vault file in group_vars/all directory as well.

   $ ln -sf ~/.ansible/sudo_passwords.yml group_vars/all/000passwords.yml
   

By this point, you should be able to use Ansible by providing a single password (for the encrypted file) and still use unique sudo passwords for each of your machines.

$ ansible --ask-vault-pass  ubuntu -a "free -m"
Vault password:
host1 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          32005        4677       22684          25        4643       27052
Swap:             0           0           0
host2 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          15884        1088       12706           1        2089       14619
Swap:             0           0           0

Avoiding entering vault password everytime

I work on my playbooks iteratively, so I run ansible or ansible-playbook frequently. I still was mildly annoyed having to enter the vault password every time.

Fortunately, Ansible provides a way to bypass this. If you set an environment variable ANSIBLE_VAULT_PASSWORD_FILE and point it to a shell script, Ansible will execute the script and use the output as the password.

So I created a pretty simple script which would do the same and put it in ~/.ansible/vault-env.

$ cat ~/.ansible/vault-env
#!/bin/bash
echo $ANSIBLE_VAULT_PASSWORD

Now I needed a way to set the environment variable without exposing it in bash history. For this, I created a support script which is checked into the repo.

$ cat support_scripts/set_vault_password_env.sh
#!/bin/bash
export ANSIBLE_VAULT_PASSWORD_FILE=~/.ansible/vault-env
export no_proxy='*'

read -sp "Vault password:" ANSIBLE_VAULT_PASSWORD
export ANSIBLE_VAULT_PASSWORD

Now whenever I have some Ansible work to do, I change directory to my playbook repo, source this script and I don’t have to enter passwords again for the rest of the session:

$ source support_scripts/set_vault_password_env.sh
Vault password:

$ ansible ... 

$ ansible-playbook ...

Making it even simpler on Mac

On Mac, this process can be made even simpler by storing the password in the keychain and avoiding entering it ever!

Add password to keychain

Add the password to keychain. You willl have to run this script only once per mac machine you work on.

$ cat support_scripts/mac/keychain_set_vault_password.sh
#!/bin/bash

read -sp "Vault password:" ANSIBLE_VAULT_PASSWORD
echo
security add-generic-password -U -a $USER -s ansible-vault -w "$ANSIBLE_VAULT_PASSWORD"

Run it directly (i.e. you don’t source it).

Point ANSIBLE_VAULT_PASSWORD_FILE to a mac specific script

$ cat ~/.ansible/vault-env-mac
#!/bin/bash
security find-generic-password -a $USER -s ansible-vault -w

And this is the file you source before your Ansible sessions.

$ cat support_scripts/mac/set_vault_password_env_from_keychain.sh
#!/bin/bash
export ANSIBLE_VAULT_PASSWORD_FILE=~/.ansible/vault-env-mac
export no_proxy='*'

Now you don’t have to remember the vault password. The keychain will keep it safe.

From time to time, I have felt the need to install a specific version of a Homebrew formula. Like the other day, I was investigating whether a particular problem I was facing with Podman was because of a version bump from 2.x to 3.x.

At some point in the past, a way around it was to find the formula file at a particular commit of the Homebrew tap, and directly install it via brew install $URL.

That is no longer supported, for good reason. But the tip provided is pretty terse.

I thought I will just jot down the steps I took to do this the recommended way.

Finding the right version of the formula

First, let us find the last 2.x version of the formula I am interested in -podman.

Step into the right repo for the Formula.

$ brew tap-info homebrew/core
homebrew/core: 2 commands, 5786 formulae
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core (6,417 files, 487.8MB) From:
https://github.com/Homebrew/homebrew-core

$ cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core

I am interested in the last 2.x version. I used the following command to find the first commit which switched the formula to 3.x.

$ git log -p -G url.*podman -- Formula/podman.rb|grep -e ^commit -e https://github.com/containers/podman/
commit 8070a9ba0b21a0285890544c88f0479a922de809
-  url "https://github.com/containers/podman/archive/v3.3.0.tar.gz"
+  url "https://github.com/containers/podman/archive/v3.3.1.tar.gz"
...
...
commit 623626510d622e64044d1d656e96386cda5cc5b9
-  url "https://github.com/containers/podman/archive/v2.2.1.tar.gz"
+  url "https://github.com/containers/podman/archive/v3.0.0.tar.gz"
^C

Ok. So 2.2.1 is the version of the formula I am interested in.

Installing the RIGHT way

1. Create a local tap for storing the formula.

   I have made it formula specific (local-podman). But if you tend to do this often for multiple formulas, you can probably name it more generically. To re-emphasize, it doesn’t need to be named after the formula.

   # Create a local tap for storing the formula
   $ brew tap-new $USER/local-podman
   

2. Actually create the copy of the formula at the specific version. brew does all the hard work for you! Look at how it automatically found out the last git commit of the formula when the formula version was what you wanted.

   $ brew extract --version=2.2.1 podman $USER/local-podman
   ==> Searching repository history
   ==> Writing formula for podman from revision e2c833d to:
   /usr/local/Homebrew/Library/Taps/sandipb/homebrew-local-podman/Formula/podman@2.2.1.rb
   

3. Now your formula will appear in your brew search to be installed the usual way.

   $ brew search /podman/
   ==> Formulae
   podman                                    sandipb/local-podman/podman@2.2.1
   
   If you meant "podman" specifically:
   It was migrated from homebrew/cask to homebrew/core.
   
   $ HOMEBREW_NO_AUTO_UPDATE=1 brew install sandipb/local-podman/podman@2.2.1
   

I had an unique case though. For some reason though, the checksum in that commit of the formula was broken.

$ HOMEBREW_NO_AUTO_UPDATE=1 brew install sandipb/local-podman/podman@2.2.1
...
Error: SHA256 mismatch
Expected: bd86b181251e2308cb52f18410fb52d89df7f130cecf0298bbf9a848fe7daf60
  Actual: 3212bad60d945c1169b27da03959f36d92d1d8964645c701a5a82a89118e96d1

I found out the actual checksum from the official repository, edited the formula to update it and then it installed properly. Perhaps the change in checksum was worth an investigation. It was too late in the night to do that. 😄

$ curl -Ls https://github.com/containers/podman/archive/refs/tags/v2.2.1.tar.gz | sha256sum
3212bad60d945c1169b27da03959f36d92d1d8964645c701a5a82a89118e96d1  -

$ brew edit podman@2.2.1
# ... changed checksum of archive

$ HOMEBREW_NO_AUTO_UPDATE=1 brew install podman@2.2.1

SUCCESS!

Installing the WRONG way

As mentioned before, I used the following command to find the first commit which switched the formula to 3.x.

$ git log -p -G url.*podman -- Formula/podman.rb|grep -e ^commit -e https://github.com/containers/podman/
commit 8070a9ba0b21a0285890544c88f0479a922de809
-  url "https://github.com/containers/podman/archive/v3.3.0.tar.gz"
+  url "https://github.com/containers/podman/archive/v3.3.1.tar.gz"
...
...
commit 623626510d622e64044d1d656e96386cda5cc5b9
-  url "https://github.com/containers/podman/archive/v2.2.1.tar.gz"
+  url "https://github.com/containers/podman/archive/v3.0.0.tar.gz"
^C

Find the commit previous to this which changed the formula.

$ git log -n 1 623626510d622e64044d1d656e96386cda5cc5b9^ -- Formula/podman.rb
commit e2c833d326c45d9aaf4e26af6dd8b2f31564dc04
Author: Rylan Polster <rslpolster@gmail.com>
Date:   Wed Feb 3 00:13:48 2021 -0500

    formulae: use new bottle syntax

So, I found out that e2c833d326c45d9aaf4e26af6dd8b2f31564dc04 is that last update to the formula which had the 2.x version. Now to install this.

$ FORMULA_URL="https://raw.githubusercontent.com/Homebrew/homebrew-core/e2c833d326c45d9aaf4e26af6dd8b2f31564dc04/Formula/podman.rb"

$ HOMEBREW_NO_AUTO_UPDATE=1  brew install $FORMULA_URL
Traceback (most recent call last):
        9: from /usr/local/Homebrew/Library/Homebrew/brew.rb:129:in `<main>'
        8: from /usr/local/Homebrew/Library/Homebrew/cmd/install.rb:131:in `install'
        7: from /usr/local/Homebrew/Library/Homebrew/cli/parser.rb:308:in `parse'
        6: from /usr/local/Homebrew/Library/Homebrew/cli/parser.rb:629:in `formulae'
        5: from /usr/local/Homebrew/Library/Homebrew/cli/parser.rb:629:in `map'
        4: from /usr/local/Homebrew/Library/Homebrew/cli/parser.rb:633:in `block in formulae'
        3: from /usr/local/Homebrew/Library/Homebrew/formulary.rb:414:in `factory'
        2: from /usr/local/Homebrew/Library/Homebrew/formulary.rb:180:in `get_formula'
        1: from /usr/local/Homebrew/Library/Homebrew/formulary.rb:185:in `klass'
/usr/local/Homebrew/Library/Homebrew/formulary.rb:277:in `load_file': Invalid usage: Installation of podman from a GitHub commit URL is unsupported! `brew extract podman` to a stable tap on GitHub instead. (UsageError)

Wtf! Frantic web search ensues.

So the tech world went a little crazy today, as usual ignoring more consequential problems happening in the world. Docker posted a notice saying that its Mac and Windows desktop clients will no longer be free for anybody other than individuals and small companies.

Here is The Register:

Docker will restrict use of the free version of its Docker Desktop utility to individuals or small businesses, and has introduced a new more expensive subscription, as it searches for a sustainable business model.

The company has renamed its Free plan to “Personal” and now requires that businesses with 250 or more employees, or higher than $10m in annual revenue, must use a paid subscription if they require Docker Desktop.

This gave me an excuse to try out something which I had been meaning to do for a while - use Podman as a replacement for Docker desktop on my Mac.

I found an article by Dave Meurer to replace Docker with Podman on the desktop. It uses Vagrant (with Virtualbox as the provider) to easily create a VM, and then install Podman on it, with a few scripts taking care of some niggles of operating Podman as a non-root user.

The steps were pretty detailed and complete, but there were couple of enhancements that I wanted to make:

1. I definitely wanted to convert the raw shell script provisioning to Ansible, my favorite provisioning tool. It just gives me more confidence in iteratively develop the right steps to provision the image.
2. I wanted to use private docker image repositories. Like most corporations, our Docker images are stored in a private (often Artifactory) image registry.

After a few hours of reading a bunch of man pages of containers.conf, registries.conf, etc. again and again, I finally found my sweet spot.

The code is here: https://github.com/sandipb/vagrant-setups/tree/main/podman

Just check it out and follow the instructions, and within a few minutes, you will be able to alias docker to podman and have a Docker Desktop-less alternative, in case you need it.

For most simple to moderately complex web servers in Golang, I have always preferred to use the standard net/http library, laboring through the parsing of RequestPaths and query parameters when necessary.

But when the boilerplate code for request parsing starts obscuring the business logic, I tend to look out at other libraries. I however have always hesitated using too heavy a framework to keep dependencies simple.

For HTTP routing, for example, I have found httprouter to satisfy most of the gaps in the standard library that I have frequently faced, and at the same time the library is totally dep free!

Here are some really quick notes about using the library, mostly for my own reference. 😄

CONTENTS:

• Overview
• Example Usage
• Additional features

Overview

• Github: https://github.com/julienschmidt/httprouter
• Godoc: https://godoc.org/github.com/julienschmidt/httprouter

This is a very minimalistic http router library which addresses the most frequently felt gaps in the standard library:

1. http method specific handlers.
2. Named parameters in the path.

The feature set is deliberately kept simple, and for users who want more, the author encourages them to use one of the many other libraries that have popped up wrapping this routing library.

Example Usage

Notes:

1. A route registration call specifies the HTTP method - router.GET().

2. The Index() and Hello handlers are not stdlib compatible and have an extra library specific parameter.

3. The Hello() handler uses a named parameter retrieved inside the handler using the new third parameter in the handler.

4. Calls to /hello/xxx/yyy will not be matched to /hello/:name - the routing is explicit.

5. You cannot add a static route overlapping the named parameter. e.g. You cannot now have a route for /hello/somewhere. That second path component is already taken by the route for /hello/:name.

6. As shown in the Welcome() handler, you can use a wildcard named parameter (e.g. *where here) at the end of a routing pattern using a * prefix to slurp up everything to the right of the URI. You can use a named parameter as many times as you want as well, but the wildcard one has to be at the end of the pattern - see (C).

   $ curl -s localhost:8888/welcome/sandip/to/wonderland/for/ever
   Welcome, sandip to to/wonderland/for/ever!
   

7. You can use a standard http handler too - see Salut(). To reach for the named parameters, you need to use httprouter.ParamsFromContext() helper function to retrieve the usual third parameter in the custom handlers, the Params from the request context - see (A).

   However, note that the routing function is now a non-standard router.HandlerFunc() which expects the HTTP method name as the first parameter - see (B).

8. You can provide an endpoint for static file serving using ServeFiles() - see (D).

   !!Note!! however that directory index display is enabled, so an empty file name can display directory contents.

   $ curl -s 127.0.0.1:8888/files/
   <pre>
   <a href="go.mod">go.mod</a>
   <a href="go.sum">go.sum</a>
   <a href="simple.go">simple.go</a>
   </pre>
   

Additional features

The router has some optional fields for provide default handling of specific exceptions:

1. router.Notfound can be set to handle requests to paths which are not matched by any routing patterns.
2. Setting router.HandleMethodNotAllowed to true allows you to set a handler in router.MethodNotAllowed to handle situations when a request has been sent to a router pattern which exists but with a different http verb. e.g. Only a GET route exists but a POST verb was used in the request.
3. You can recover from panics in your server handlers by setting router.PanicHandler to an appropriate handler which can send a suitable 5xx error.
4. You can set CORS headers by setting router.GlobalOPTIONS to a suitable handler. The Godoc provides a code sample.

Credit: “Blue Gopher” icon by Ashley McNamara

But if you are one of those people who migrated from a pre-5.6 MySQL version, you probably avoided enabling GTID to make the upgrade easier. In such a situation, typically, you would upgrade the slaves to the new MySQL version, and then failover the master to an upgraded slave. This would let you do an online upgrade of a MySQL cluster from a pre-5.6 to a 5.6+ installation, but you will not be able to do this with GTID enabled.

I recently had a requirement to move a bunch of such MySQL 5.7 clusters using the old binlog position based replication to a GTID based replication setup.

Now there is a pretty good official document about how to do this manually, which you should definitely read up before doing this. But it is a pain to do this manually on a bunch of servers.

I captured all the steps mentioned in the document into an Ansible playbook to automate the whole process. It also includes a procedure missing in the official document, to actually flip the ongoing replication to GTID protocol.

The playbook source is here with step by step documentation here.

But my biggest hurdle for doing that, has been the enormous pain in managing certificates in a way that makes everybody - the servers, the browsers, the local http clients happy. From my previous attempts, there was always the browsers which annoyed me to no end, and I ended up getting by using improperly made self-signed certificates and accepting all the invalid certificate warnings that my browser threw up.

So this Friday night, I spent my late night hours trying to get at the bottom of it all, and several frustrating hours later, finally made everybody happy.

Now, remember how I mentioned 2020 in my post title? The reason is that the world of PKI infrastructure is not static, and is changing way faster than people realize.

At this point, most SOHO people have probably stopped paying for certificates and are using the wonderful automated certificate system provided for free by Lets Encrypt. But while that utopia is astonishingly real, it mostly exists outside in the real Internet world. Behind the firewall, where your private services lie, the verification systems of Lets Encrypt can’t reach easily. Besides, you would probably not need to use a real-world domain inside your firewall, with entries most likely to be using private IP addresses, which Letsencrypt cannot really verify.

So the only option is to run your own certificate chain inside your walls. That means setting up a root CA, and an intermediate CA (optional but recommended) etc.

The best guide that I found

I read a lot of different guides over time, and even this weekend, but the one which seemed the most useful was OpenSSL Certificate Authority by Jamie Nguyen. The guide provides step by step instructions for creating the root CA certs, the intermediate CA certs, and finally the server certs that you needed in the first place. It even has a section on the revocation of certificates using CRL and OCSP.

While the generation of root and intermediate certificates themselves were pretty much as described, there were several problems with the final certificates which were generated - they were rejected by all three browsers I was testing on - Chrome, Firefox, and Safari. And this was after I had added the root CA certs to the Mac Keychain and Firefox certificate store.

And this is where the 2020 part of my post comes in. Jamie’s guide from 2015 has been succeeded by a bunch of changes in recent years.

The “Common Name” field in an SSL certificate is no longer supported

Chrome 58 and later specifically started enforcing this for the last couple of years. As this article from Hashedout blog explains:

Many people don’t know that the “Common Name” field of an SSL certificate, which contains the domain name the certificate is valid for, was actually phased-out via RFC nearly two decades ago. Instead, the SAN (Subject Alternative Name) field is the proper place to list the domain(s).

However, this has been ignored and for many years the Common Name field was exclusively used. Chrome is finally fed up with the field that refuses to die. In Chrome 58, the Common Name field is now ignored entirely.

This means certificates that were exclusively using that field to indicate the valid domain name are no longer supported. Publicly-trusted SSL certificates have been supporting both fields for years, ensuring maximum compatibility with all software – so you have nothing to worry about if your certificate came from a trusted CA.

This change will only affect private PKIs and other software that have not been following spec. If you notice any sites returning the error NET::ERR_CERT_COMMON_NAME_INVALID, it’s likely due to the certificate not using SANs properly.

This additional blog post mentioned in the article has some good historical background on the topic.

Starting September 1 2020, major browsers will not support SSL/TLS certificates with validity longer than 398 days

At first, Apple, and then Google Chrome team have announced that they would only support certificates with 1 year validity (+~30 days grace period). That is 75% of the browser market.

Chrome joins Apple in limiting public TLS certificates to 398 days starting Sept 1st.

— Dean Coclin (@chosensecurity) June 10, 2020

Just to clarify, it seems only leaf certificates are impacted here - I created intermediate certificates with 5 years validity (the root one was 20) and didn’t see a problem yet. I will find out more after September 1st, I guess.

Working around these new certificate constraints

So I had two things to fix - reduce my leaf certificate validity to 1 year, and add subjectAltName field to the certificates.

The first part was easy. Creating leaf certificates is a hassle when you have to create multiple ones. So I had, in any case, created a script to make it easier. I just had to reduce the validity period there.

But the second one was a puzzle. I tried a bunch of hacks all around the Internet to add the field in the Certificate Signing Request (CSR), but no matter what I do, it won’t appear in the final certificate. So I read around a bit more.

For one, this is a bad practice, as this Stackoverflow comment mentions. CAs will not copy all attributes to the final cert for pretty reasonable security reasons.

Secondly, it appears that openssl itself has two behaviors regarding copying this attributes. The default behavior is not to copy anything in the ca subcommand. If you do want this, adding the parameter -copy_extensions will copy everything.

So the consensus was to add the fields explicitly in the ca command, and, from the point of correctness, but not usefulness, add it to the CSR as well.

Adding the field itself to the commands is much of a chore - you have to add sections to the configs dynamically (in case you want to avoid creating a config file for every certificate you create), and all the hacks seemed tedious. I finally found what I felt was the most elegant option - using environmental variables. (SO comment)

So in the [server_cert] section in the openssl.cnf config file, I added this line:

subjectAltName=${ENV::SAN}

And in the [req] section, I added a line req_extensions = san_env and created a section at the bottom of the config with this:

[ san_env ]
subjectAltName=${ENV::SAN}

Now all I needed to do was to make sure in the script, I added this environment variable before invoking the openssl commands. And now, I just need a single invocation of my script per certificate and it will generate the certificate and the keys for me.

The final certificate creation script is here in this gist.

Adding trusted certificates to the browsers

The best way to add the trusted certificates to all my home devices was to just add the root CA certificate in all the right places. For the Mac, this meant adding it to the system keychain - both Safari and Chrome use this location. For Firefox, the way to add it seemed to be in its preferences. Notice, that I am not adding the intermediate certificate here. That is because I am sending it from the servers (next section).

Adding the certificate to nginx

There are plenty of documentation on how to do this. The important thing to remember here is that if you have only added the root CA certificate in the browser (which you should, and then hide the root CA private key somewhere safe), then you need to add the intermediate certificate to the server certificate in nginx.

As the nginx doc mentions:

Some browsers may complain about a certificate signed by a well-known certificate authority, while other browsers may accept the certificate without issues. This occurs because the issuing authority has signed the server certificate using an intermediate certificate that is not present in the certificate base of well-known trusted certificate authorities which is distributed with a particular browser. In this case the authority provides a bundle of chained certificates which should be concatenated to the signed server certificate. The server certificate must appear before the chained certificates in the combined file:

$ cat www.example.com.crt bundle.crt > www.example.com.chained.crt

The resulting file should be used in the ssl_certificate directive.

If you add this bit of code in the <head> section of an HTML 5 page, clicking on every link will open in a new tab.

<base target="_blank">

This tag is mostly used to set a URL to resolve all relative links on a page. But it offers an additional attribute target to set a default target policy for links.

You can, of course, override the target attribute on a link to link basis, if you are using this feature.

Privilege escalation permissions have to be general. Ansible does not always use a specific command to do something but runs modules (code) from a temporary file name which changes every time. If you have /sbin/service or /bin/chmod as the allowed commands this will fail with ansible as those paths won’t match with the temporary file that ansible creates to run the module.

Our company, for good reasons, has these restrictions. Root shells bypass audit logs and can let really bad things happen by mistake. Even if I am working on my personal hosts, I try to make it a habit of not working in a root shell. The convenience is just not worth the risk.

So, anyway, ansible is pretty useless for anything involving root user on our systems.

However, if we look at the actual problem in more detail, the issue is clearly the way sudo is invoked.

Without become ansible essentially runs the following on the remote host:

sh -c "/usr/bin/python /path/to/ansible/compiled/python/script"

With become however, ansible runs something like this:

sh -c "sudo -u root /bin/sh -c 'echo BECOME-SUCCESS-oqgdgdpwngxeakkmhtvcxzvhnwtsxzzm ; /usr/bin/python  /path/to/ansible/compiled/python/script"

It is that echo statement, which is possibly being used by ansible to distinguish between a privilege escalation failure and a task failure, which is bringing in that requirement to run a shell within sudo. Unfortunately, this is something the ansible team has decided to stick with. So to me, one way out was to stop relying on ansible to run sudo, and run sudo myself in an ordinary command and figure out a way to supply a password to the inevitable prompt.

I tried to see if I can use that old trick of using expect to supply the password. Ansible has an expect module. Unfortunately, it requires the python library pexpect, which performs similar functionality in pure python, to be pexpect >= 3.3 which is not available in RHEL7 hosts in our production environment.

$ sudo yum list pexpect
...
Available Packages
pexpect.noarch           2.3-11.el7         release-rhel-x86_64-workstation-7-r03

$ cat /etc/redhat-release
Red Hat Enterprise Linux Workstation release 7.3 (Maipo)

So I was left to use the regular expect command, available in most server environments for eons, to do my job. I had to read up a bit of TCL to find my way around. That and this stackoverflow answer helped me construct this proof-of-concept playbook which does the job.

- hosts: all
  gather_facts: no

  vars:
      sudo_cmd: id

  vars_prompt:
      - name: sudo_password
        msg: Enter sudo password
        private: yes

  tasks:

      # 
      - name: "run sudo"
        shell: |
            set timeout 10
            spawn sudo {{ sudo_cmd }}
            expect {
                "sudo*password*: " { send "{{sudo_password}}\n" }
            }
            expect eof
            lassign [wait] pid spawnid os_error_flag ret_code

            if {$os_error_flag != 0} {
                exit $os_error_flag
            }
            exit $ret_code
        args:
            executable: /usr/bin/expect
        no_log: True

This script will prompt for the sudo password once at the beginning.

You can then run your sudo command like this.

ansible-playbook -v  --limit somehost sudo.yml -e "sudo_cmd=ls /root"

Of course, one gotcha is that in many stock deployments, the expect package is not installed and you might have to install it first.

The usual security risks of having secrets passed into shell commands in ansible apply, of course. The no_log parameter is especially important, without which ansible will leak your password to syslog on the remote host.

It is a hack, I know. If there is a better way of doing it, I would be really interested to know. I couldn’t find anything on ansible forums and documentation for my particular needs.

I just found another significant part of the spec that I had previously glossed over, this one is also about untyped constants - numeric constants in Go live in an unified space with arbitrary precision and a fungible numeric type.

So, coming from a language like Python, this might not surprise us:

$ python2
Python 2.7.15 (default, Sep 18 2018, 20:16:18)

>>> i = 2000

>>> print i / 1000, type(i / 1000)
2 <type 'int'>

>>> print i / 1e3, type(i / 1e3)
2.0 <type 'float'>

This is because the exponential format (1e3) was a float, and forced the division to be a float division.

This therefore was a surprise for me when I found out that this works in Go.

i := 2000
j := 2000.0
fmt.Printf("i: %v (%T)\n", i/1e3, i/1e3)
fmt.Printf("j: %v (%T)\n", j/1e3, j/1e3)

// Prints:
//   i: 2 (int)
//   j: 2 (float64)

So, you can use the exponential notation for working with large numbers in Go, something I thought was a no-no earlier.

now := time.Now()
secs := now.Unix()
millis := now.UnixNano() / 1e6

fmt.Printf("Secs   since 1970: %v (%T)\n", secs, secs)
fmt.Printf("Millis since 1970: %v (%T)\n", millis, millis)

// Prints:
//    Secs   since 1970: 1538782570 (int64)
//    Millis since 1970: 1538782570453 (int64)

(Play link, note that you will find more useful values by running this on your own computer, because current time is a weird concept in the Go playground)

The Go blog post on constants has an excellent section demonstrating this:

The concept of untyped constants in Go means that all the numeric constants, whether integer, floating-point, complex, or even character values, live in a kind of unified space. It's when we bring them to the computational world of variables, assignments, and operations that the actual types matter. But as long as we stay in the world of numeric constants, we can mix and match values as we like.

All these constants have numeric value 1:

1
1.000
1e3-99.0*10-9
'\x01'
'\u0001'
'b' - 'a'
1.0+3i-3.0i

Therefore, although they have different implicit default types, written as untyped constants they can be assigned to a variable of any integer type.

My first thought was to increase the system ulimit for nofile in /etc/security/limits.conf. I changed this from 30k to about 60k. But strangely, the service still keep dying.

Since the service uses chpst, my first suspect was that chpst was trying to be overbearing and changing the system limits before starting up the process. The service I was running was Python based, and so I added this small snippet in the beginning of the program to see what limits was it seeing.

import resource
logger.info("rlimit for nofile: %s", resource.getrlimit(resource.RLIMIT_NOFILE))

The logs showed:

[INFO] rlimit for nofile: (1024, 4096)

That was shocking. None of those values are anywhere configured in the system. lsof of the process had earlier showed me that the process had about a thousand open files before it crashed, so that explained why it was crashing - it was breaching the soft limit.

I looked at the documentation for chpst and found a option -o which changes the open files limit. So I set that in the chpst invocation (exec chpst -o 60000 ...), and I got:

[INFO] rlimit for nofile: (4096, 4096)

It seems that the -o only affects the soft limit. I took the win and the service recovered, but after the crisis passed, I kept digging. I was curious where all these limits were coming from.

The chpst source didn’t reveal any limits being imposed. I couldn’t figure out any other call to setrlimit in the rest of the runit sources either.

On a hunch, I tried to print the limits before chpst is called using this run file.

#!/bin/bash
exec 2>&1

echo "Soft limits"
ulimit -S -a

echo "Hard limits"
ulimit -H -a

exec chpst ...

It got me:

• open files (-n) 1024 # soft limit
• open files (-n) 4096 # hard limit

The actual ulimit values for the root user were 60000 (soft),60000 (hard). That showed me that this limits modification was not happening because of chpst, but probably because of runsv or some other part of the runit system. I could be wrong though because, like I said before, I couldn’t find a call to setrlimit in the sources.

Curiously, printing the ulimit values in the run file showed me another odd change from the system limits - the max procs limit (ulimit -u). It seems that when the run file is executing, the soft limit for this setting is set to the hard limit.

On the RHEL6 machine I was running, the root user showed these limits in the root shell: (1024,514975). In the run file however, the equivalents were (514975,514975). This was so weird, I double checked.

Why is the runit service being conservative with resource limits with regard to the max open files limit, but more generous with the max procs limit?

I have no idea.

In the end, yes, you can workaround the default limits of runit by using ulimit in the shell script before invoking chpst, but when you don’t, I hope this helps in reminding ourselves the limits it places on the services that it manages.

Sometimes instead of creating a logger and then passing it around, it is convenient to just use a global logger.

The standard log library allows you to both create a custom logger using log.New() or directly use a standard logger instance by calling the package helper functions log.Printf() and the like.

zap provides such a functionality as well using zap.L() and zap.S(), however using them didn’t seem so straight forward to me.

You can access the global standard logger using zap.L(). The function returns the shared logger instance. A sugared version is accessible via zap.S().

From what I have seen in my brief usage of these loggers, this shared logger is not useful out of the box - if you just use them right away, they provide no output. Their sole purpose seems to be to provide a simple way to retrieve this instance anywhere in your code.

If you really want to use this standard logger usefully, you need to replace the core of the logger with that of a different logger using zap.ReplaceGlobals().

fmt.Printf("\n*** Using the global logger out of the box\n\n")

zap.S().Infow("An info message", "iteration", 1)

fmt.Printf("\n*** After replacing the global logger with a development logger\n\n")

logger, _ := zap.NewDevelopment()
zap.ReplaceGlobals(logger)
zap.S().Infow("An info message", "iteration", 1)

Output:

*** Using the global logger out of the box


*** After replacing the global logger with a development logger

2018-05-02T16:24:40.992-0700    INFO    globallogger/main.go:17 An info message {"iteration": 1}

There is also a way provided to undo a core replacement in the global loggers.

fmt.Printf("\n*** After replacing the global logger with a development logger\n\n")
logger, _ := zap.NewDevelopment()
zap.ReplaceGlobals(logger)
zap.S().Infow("An info message", "iteration", 1)

fmt.Printf("\n*** After replacing the global logger with a production logger\n\n")
logger, _ = zap.NewProduction()
undo := zap.ReplaceGlobals(logger)
zap.S().Infow("An info message", "iteration", 1)

fmt.Printf("\n*** After undoing the last replacement of the global logger\n\n")
undo()
zap.S().Infow("An info message", "iteration", 1)

Output:

*** After replacing the global logger with a development logger

2018-05-02T16:24:40.992-0700    INFO    globallogger/main.go:17 An info message {"iteration": 1}

*** After replacing the global logger with a production logger

{"level":"info","ts":1525303480.993161,"caller":"globallogger/main.go:22","msg":"An info message","iteration":1}

*** After undoing the last replacement of the global logger

2018-05-02T16:24:40.993-0700    INFO    globallogger/main.go:26 An info message {"iteration": 1}

The various implementations of field encoders provided in zap can sometimes feel inadequate. For example, you might want the logging output to be similar to that in syslog or other common log formats. You might want the timestamps in the log to ignore seconds, or the log level to be wrapped within square brackets.

To have your own custom formatters for the metadata fields you need to write custom encoders.

Contents:

• 1. Performance considerations
• 2. Customizing timestamp formatting
• 3. Customizing level formatting

1. Performance considerations

You can use custom encoders for formatting time, level, caller, etc. One caveat is that you need these encoders to be as efficient as possible so as to not negate the memory/speed advantages of zap itself. After all, these functions are called for every line of log to be emitted! So avoid creating temporary variables, or doing any expensive calculations.

That said, the examples below are just a demonstration. I am not claiming at all that they are efficient replacements of the default functionality. 😄

2. Customizing timestamp formatting

Here is an implementation which uses the syslog format often found in the wild.

func SyslogTimeEncoder(t time.Time, enc zapcore.PrimitiveArrayEncoder) {
	enc.AppendString(t.Format("Jan  2 15:04:05"))
}

...

cfg.EncoderConfig.EncodeTime = SyslogTimeEncoder

logger, _ = cfg.Build()
logger.Info("This should have a syslog style timestamp")

Output:

May  2 18:54:55 INFO    This should have a syslog style timestamp

If you noticed in the implementation that the encoder is supposed to append primitives to an array like object. zap uses this array to efficiently encode output with minimal memory allocations.

3. Customizing level formatting

func CustomLevelEncoder(level zapcore.Level, enc zapcore.PrimitiveArrayEncoder) {
	enc.AppendString("[" + level.CapitalString() + "]")
}
...
cfg.EncoderConfig.EncodeLevel = CustomLevelEncoder

logger, _ = cfg.Build()
logger.Info("This should have a bracketed level name")

Output:

May  2 18:54:55 [INFO]  This should have a bracketed level name

Similar customization can be done for other metadata fields as well. You can look at the zap source to find the general pattern of these implementations.

Using the logger presets in zap can be a huge time saver, but if you really need to tweak the logger, you need to explore ways to create custom loggers. zap provides an easy way to create custom loggers using a configuration struct. You can either create the logger configuration using a JSON object (possibly kept in a file next to your other app config files), or you can statically configure it using the native zap.Config struct, which we will explore here.

Contents:

• 1. Using the zap config struct to create a logger
• 2. Customizing the encoder
• 3. Metadata field encoder alternatives
• 4. Changing logger behavior on the fly

1. Using the zap config struct to create a logger

Loggers can be created using a configuration struct zap.Config. You are expected to fill in the struct with required values, and then call the .Build() method on the struct to get your logger.

// general pattern

cfg := zap.Config{...}
logger, err := cfg.Build()

There are no sane defaults for the struct. You have to, at the minimum, provide values for the three classes of settings that zap needs.

• encoder: Just adding a Encoding: "xxx" field is a minimum. Using json here as the value will create a default JSON encoder. The other alternative is using console.
  
  You can customize the encoder (which you almost certainly have to, because the defaults aren’t very useful), by adding a zapcore.EncoderConfig struct to the EncoderConfig field.

• level enabler: This is an interface type which allows zap to determine whether a message at a particular level should be displayed. In the zap config struct, you provide such a type using the AtomicLevel wrapper in the Level field.

• sink: This is the destination of the log messages. You can specify multiple output paths using the OutputPaths field which accepts a list of path names. Output will be sent to all of these files. Magic values like "stderr" and "stdout" can be used for the usual purposes.

2. Customizing the encoder

Just mentioning an encoder type in the struct is not enough. By default the JSON encoder only outputs fields specifically provided in the log messages.

Here are the least number of struct fields which will not throw an error when you call .Build():

logger, _ = zap.Config{
    Encoding:    "json",
    Level:       zap.NewAtomicLevelAt(zapcore.DebugLevel),
    OutputPaths: []string{"stdout"},
}.Build()

logger.Info("This is an INFO message with fields", zap.String("region", "us-west"), zap.Int("id", 2))

Output:

{"region":"us-west","id":2}

Even the message is not printed!

To add the message in the JSON encoder, you need to specify the JSON key which will have this value in the output.

logger, _ = zap.Config{
    Encoding:    "json",
    Level:       zap.NewAtomicLevelAt(zapcore.DebugLevel),
    OutputPaths: []string{"stdout"},
    EncoderConfig: zapcore.EncoderConfig{  
        MessageKey: "message",  // <--
    },
}.Build()

logger.Info("This is an INFO message with fields", zap.String("region", "us-west"), zap.Int("id", 2))

Output:

{"message":"This is an INFO message with fields","region":"us-west","id":2}

zap can add more metadata to the message like level name, timestamp, caller, stacktrace, etc. Unless you specifically mention the JSON key in the output corresponding to a metadata, it is not displayed.

Note that these metadata field names have to be paired with an encoder else zap just burns and dies (!!).

For example:

cfg := zap.Config{
    Encoding:         "json",
    Level:            zap.NewAtomicLevelAt(zapcore.DebugLevel),
    OutputPaths:      []string{"stderr"},
    ErrorOutputPaths: []string{"stderr"},
    EncoderConfig: zapcore.EncoderConfig{
        MessageKey: "message",

        LevelKey:    "level",
        EncodeLevel: zapcore.CapitalLevelEncoder,

        TimeKey:    "time",
        EncodeTime: zapcore.ISO8601TimeEncoder,

        CallerKey:    "caller",
        EncodeCaller: zapcore.ShortCallerEncoder,
    },
}
logger, _ = cfg.Build()
logger.Info("This is an INFO message with fields", zap.String("region", "us-west"), zap.Int("id", 2))

Will output:

{"level":"INFO","time":"2018-05-02T16:37:54.998-0700","caller":"customlogger/main.go:91","message":"This is an INFO message with fields","region":"us-west","id":2}

3. Metadata field encoder alternatives

Each of the encoder can be customized to fit your requirements, and some have different implementations provided by zap.

• timestamp can be output in either ISO 8601 format, or as an epoch timestamp in seconds, milliseconds and even nanoseconds.
  
• level can be capital or lowercase. Each of them even have colored options. Note that the colored options don’t make sense in the JSON output encoder because the terminal escape codes are not stripped in the current implementation.
  
• The caller can be shown in short and full formats.

4. Changing logger behavior on the fly

Loggers can be cloned from an existing logger with certain modification to their behavior. This can often be useful for example, when you want to reduce code duplication by fixing a standard set of fields the logger will always output.

• logger.AddCaller() adds caller annotation
• logger.AddStacktrace() adds stacktraces for messages at and above a given level
• logger.Fields() adds specified fields to all messages output by the new logger. Creating loggers this way, and not specifying additional fields during the actual log call can make your logging faster with less memory allocations.
• logger.WrapCore() allows you to modify or even completely replace the underlying core in the logger which combines the encoder, level and sink. Here is an example:

fmt.Printf("\n*** Using a JSON encoder, at debug level, sending output to stdout, all possible keys specified\n\n")

cfg := zap.Config{
    Encoding:         "json",
    Level:            zap.NewAtomicLevelAt(zapcore.DebugLevel),
    OutputPaths:      []string{"stderr"},
    ErrorOutputPaths: []string{"stderr"},
    EncoderConfig: zapcore.EncoderConfig{
        MessageKey: "message",

        LevelKey:    "level",
        EncodeLevel: zapcore.CapitalLevelEncoder,

        TimeKey:    "time",
        EncodeTime: zapcore.ISO8601TimeEncoder,

        CallerKey:    "caller",
        EncodeCaller: zapcore.ShortCallerEncoder,
    },
}
logger, _ = cfg.Build()

logger.Info("This is an INFO message")

fmt.Printf("\n*** Same logger with console logging enabled instead\n\n")

logger.WithOptions(
    zap.WrapCore(
        func(zapcore.Core) zapcore.Core {
            return zapcore.NewCore(zapcore.NewConsoleEncoder(cfg.EncoderConfig), zapcore.AddSync(os.Stderr), zapcore.DebugLevel)
        })).Info("This is an INFO message")

Output:

*** Using a JSON encoder, at debug level, sending output to stdout, all possible keys specified

{"level":"INFO","time":"2018-05-02T16:37:54.998-0700","caller":"customlogger/main.go:90","message":"This is an INFO message"}

*** Same logger with console logging enabled instead

2018-05-02T16:37:54.998-0700    INFO    customlogger/main.go:99 This is an INFO message

I was intrigued when Uber announced zap, a logging library for Go with claims of really high speed and memory efficiency. I had tried structured logging earlier using logrus, but while I did not experience it myself, I was worried by a lot of folks telling me about its performance issues at high log volumes. So when zap claimed performance exceeding the log package from standard library, I had to try it. Also, its flexible framework left the door open to a future plan of mine of sending logs filebeat style to ELK.

The documentation for the library was pretty standard, but I could not find a reasonable introduction to explore the various ways one can use the library. So I decided to document some of my experiments with the library.

I collected my code examples in Github, and decided to break it up into a series of posts.

Contents:

• Trying out the examples
• Using logger presets
  • Using the Example logger preset
  • Using the Development logger preset
  • Using the Production logger preset
  • Comparing the presets
• Using the Sugar logger

1. Trying out the examples

For working code examples, clone my github repository, setup the environment, install zap and then start running the examples:

$ git clone https://github.com/sandipb/zap-examples.git

$ cd zap-examples

$ source env.sh

$ go get -u go.uber.org/zap

$ go run src/simple1/main.go

2. Using logger presets

zap recommends using logger presets for the simplest of cases. These presets have pre-configured log levels and output formats.

There are three presets currently available:

• Example
• Development
• Production

2.1. Using the Example logger preset

logger := zap.NewExample()
logger.Debug("This is a DEBUG message")
logger.Info("This is an INFO message")
logger.Info("This is an INFO message with fields", zap.String("region", "us-west"), zap.Int("id", 2))
logger.Warn("This is a WARN message")
logger.Error("This is an ERROR message")
// logger.Fatal("This is a FATAL message")  // would exit if uncommented
logger.DPanic("This is a DPANIC message")
//logger.Panic("This is a PANIC message")   // would exit if uncommented

Output:

{"level":"debug","msg":"This is a DEBUG message"}
{"level":"info","msg":"This is an INFO message"}
{"level":"info","msg":"This is an INFO message with fields","region":"us-west","id":2}
{"level":"warn","msg":"This is a WARN message"}
{"level":"error","msg":"This is an ERROR message"}
{"level":"dpanic","msg":"This is a DPANIC message"}

2.2. Using the Development logger preset

logger, _ = zap.NewDevelopment()
logger.Debug("This is a DEBUG message")
logger.Info("This is an INFO message")
logger.Info("This is an INFO message with fields", zap.String("region", "us-west"), zap.Int("id", 2))
logger.Warn("This is a WARN message")
logger.Error("This is an ERROR message")
// logger.Fatal("This is a FATAL message")   // would exit if uncommented
// logger.DPanic("This is a DPANIC message") // would exit if uncommented
//logger.Panic("This is a PANIC message")    // would exit if uncommented

Output:

2018-05-02T13:52:44.332-0700    DEBUG   simple1/main.go:28      This is a DEBUG message
2018-05-02T13:52:44.332-0700    INFO    simple1/main.go:29      This is an INFO message
2018-05-02T13:52:44.332-0700    INFO    simple1/main.go:30      This is an INFO message with fields     {"region": "us-west", "id": 2}
2018-05-02T13:52:44.332-0700    WARN    simple1/main.go:31      This is a WARN messagemain.main
        /Users/snbhatta/dev/zap-examples/src/simple1/main.go:31
runtime.main
        /Users/snbhatta/.gradle/language/golang/1.9.2/go/src/runtime/proc.go:195
2018-05-02T13:52:44.332-0700    ERROR   simple1/main.go:32      This is an ERROR message
main.main
        /Users/snbhatta/dev/zap-examples/src/simple1/main.go:32
runtime.main
        /Users/snbhatta/.gradle/language/golang/1.9.2/go/src/runtime/proc.go:195

2.3. Using the Production logger preset

logger, _ = zap.NewProduction()
logger.Debug("This is a DEBUG message")
logger.Info("This is an INFO message")
logger.Info("This is an INFO message with fields", zap.String("region", "us-west"), zap.Int("id", 2))
logger.Warn("This is a WARN message")
logger.Error("This is an ERROR message")
// logger.Fatal("This is a FATAL message")   // would exit if uncommented
logger.DPanic("This is a DPANIC message")
// logger.Panic("This is a PANIC message")   // would exit if uncommented

Output:

{"level":"info","ts":1525294364.332839,"caller":"simple1/main.go:43","msg":"This is an INFO message"}
{"level":"info","ts":1525294364.332864,"caller":"simple1/main.go:44","msg":"This is an INFO message with fields","region":"us-west","id":2}
{"level":"warn","ts":1525294364.3328729,"caller":"simple1/main.go:45","msg":"This is a WARN message"}
{"level":"error","ts":1525294364.332882,"caller":"simple1/main.go:46","msg":"This is an ERROR message","stacktrace":"main.main\n\t/Users/snbhatta/dev/zap-examples/src/simple1/main.go:46\nruntime.main\n\
t/Users/snbhatta/.gradle/language/golang/1.9.2/go/src/runtime/proc.go:195"}
{"level":"dpanic","ts":1525294364.332895,"caller":"simple1/main.go:48","msg":"This is a DPANIC message","stacktrace":"main.main\n\t/Users/snbhatta/dev/zap-examples/src/simple1/main.go:48\nruntime.main\n
\t/Users/snbhatta/.gradle/language/golang/1.9.2/go/src/runtime/proc.go:195"}

2.4. Comparing the presets

By comparing the outputs you can make the following observations:

• Both Example and Production loggers use the JSON encoder. Development uses the Console encoder
• The logger.DPanic() function causes a panic in Development logger but not in Example or Production
• The Development logger:
  • Prints a stack trace from Warn level and up.
  • Always prints the package/file/line number (the caller)
  • Tacks any extra fields as a json string at the end of the line
  • Prints the level names in uppercase
  • Prints timestamp in ISO8601 format with milliseconds
• The Production logger:
  • Doesn’t log messages at debug level
  • Adds stack trace as a json field for Error, DPanic levels, but not for Warn
  • Always adds the caller as a json field
  • Prints timestamp in epoch format
  • Prints level names in lower case

3. Using the Sugar logger

The default zap loggers expect structured tags, i.e. for every tag, you need to use a function for the specific value type. It can sometimes feel too verbose.

logger.Info("This is an INFO message with fields", 
            zap.String("region", "us-west"), 
            zap.Int("id", 2))

This is the fastest option for an application where performance is important. (Actually it can get even faster if the logger has tags pre-configured, which will see in further posts in this series.)

However, for a just a small additional penalty (which is actually still slightly better than the standard library log package), you can use the sugar logger, which uses printf-style reflection based type detection to give you a simpler syntax to add tags of mixed types.

slogger := logger.Sugar()
slogger.Info("Info() uses sprint")
slogger.Infof("Infof() uses %s", "sprintf")
slogger.Infow("Infow() allows tags", "name", "Legolas", "type", 1)

Output:

2018-05-02T18:13:22.376-0700    INFO    simple1/main.go:56      Info() uses sprint
2018-05-02T18:13:22.376-0700    INFO    simple1/main.go:57      Infof() uses sprintf
2018-05-02T18:13:22.376-0700    INFO    simple1/main.go:58      Infow() allows tags     {"name": "Legolas", "type": 1}

If you need to, you can switch from a sugar logger to a standard logger any time using the .Desugar() method on the logger.

I admit I had not paid much attention to Netlify earlier. It sort of seemed like yet another web performance related startup.

But on reading Fatih’s article on hosting Hugo on Netlify, it piqued my interest. A CDN/hosting service which puts your content in caches all around the world, and triggers Hugo (and bunch of other common scripts) on Github commits? And all this for free? Sounds too good to be true, and memories of Posterous floated in my mind.

But again, the best part of using static blogging software like Hugo, is that there is so less to lose from trying out a new hosting option - no databases to setup, no old content to migrate.

And so i decided to try it out as well. And it turned out to be blindingly simple! Netlify turned out to be awesome!

Here are all the stuff I needed to do to move my Hugo hosting from my shared hosting account at Dreamhost to Netlify.

Steps:

1. Create account and link to Github
2. Setting up a custom domain
3. Setting up SSL
4. Setting up preview website
5. Tweaks for Hugo

Create account and link to Github

I first created an account of Netlify, and after logging in, I set up access to Github. If you use the “Login using Github” option while signing up, I suspect that both the account creation and Github linking will happen at the same time. I prefer using a distinct email based account in every site, for various reasons.

After linking to Github, there was an option presented to choose an existing repository to pick up changes from, and since I was already pushing the source of my Hugo site to Github, this wasn’t much trouble. I was asked the command to build (I used hugo), and the directory where the static content is to be generated (I used the default public/). And that was it!

Netlify picked up my changes and created a random host name under the netlify.com and published it there!

An interesting thing is that my Hugo source uses themes as Git submodules. Netlify checked out these theme submodules before running Hugo. Neat!

Setting up a custom domain

Netlify lets you point a custom domain to your hosted website for free! Crazy! All I had to do was to log in to my DNS provider, remove the A record for my earlier website, and add a CNAME instead to the random hostname on which Netlify was hosting my website.

$ dig  blog.sandipb.net

...
;; ANSWER SECTION:
blog.sandipb.net.   3582    IN  CNAME   awesome-blackwell-069f9b.netlify.com.
awesome-blackwell-069f9b.netlify.com. 2 IN A    35.199.180.1

Within a few minutes, I could already see my content on Netlify! I tried to use a curl from various servers to believe my eyes!

$ curl -sI https://blog.sandipb.net/
HTTP/1.1 200 OK
Cache-Control: public, max-age=0, must-revalidate
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 29 Jan 2018 06:02:15 GMT
Etag: "98934c8fe85d0e8c73379da28cedef89-ssl"
Strict-Transport-Security: max-age=31536000
Age: 1
Connection: keep-alive
Server: Netlify          <<<<-------

As an aside, you can change the hostname under netlify.com domain as well, but since the only characters allowed as alphanumeric+underscore, you can’t get too creative here. Interestingly, even though I changed my Netlify hostname, the DNS setup didn’t break. Netlify still keeps the original random hostname alive. Nice.

$ dig +short blog.sandipb.net
awesome-blackwell-069f9b.netlify.com.
104.198.106.181

$ dig +short awesome-blackwell-069f9b.netlify.com.
104.198.106.181

$ dig +short blog-sandipb-net.netlify.com.
104.198.106.181

All this was nice and good, but there were a few more details to sew up!

Setting up SSL

Just like my previous hosting provider, Dreamhost and many other hosting providers nowadays, Netlify has automated Lets Encrypt SSL sourcing and installation features. Within a few minutes, my site was running on SSL.

BTW, if you choose to only use your xxx.netlify.com hostname, you don’t need this, because Netlify uses a wild-carded SSL certificate for all their sub-domains.

There was also an option to force HTTPS for the website, which I of course turned on.

Chrome showed a niggling security warning about a resource being insecure, and I hunted down and found out that the badge I pull in from Feedburner had a non-https URL. Fixing that and committing to Github, caused Netlify to rebuild, and within a few minutes, the site was completely free of SSL warnings.

Setting up preview website

A fantastic feature that Netlify offers, again for free, was an automatic staging website to preview pull requests and branch changes.

If you have a pull request on your master branch, there would automatically be a build done on it, and made viewable on a unique hostname created from the commit id. You can view the final changes, and proceed to Github to merge it in.

You also have the option to set up one or more dev/staging branches, all of which will get automatic previews on every commit.

There were some niggling UI issue in configuring this though. The UI provides a weird combination of dropdown+text field to specify the branches. The dropdown is empty by default, you are actually expected to type in the branch name, even if it already exists on the repository.

In any case, previews will start happening from the next commit to the branch, and not automatically from the present content.

Tweaks for Hugo

As Fatih mentions in his post, there is an apparent conflict between Hugo configuration and how the preview system works.

You might be using a baseurl of / for links to work locally using relative URLs. But you might want the production baseurl to be that of the website address so that all links are absolute.

This doesn’t work very well with the default set up in Netlify because the domain name for previews are random, and the production baseurl is different.

One way Fatih suggests is to use Netlify configuration files. These files allow you to change environment variables based on the deployment context - production or deploy-preview.

Also, Netlify allows you to specify the Hugo version to use for building.

Since I mostly use Hugo via Homebrew which keeps the Hugo version pretty up to date, I specified the Hugo version in the configuration file as well.

So this is the final netlify.toml file that I currently use, which is pretty much the same that Fatih proposed.

[context.production.environment]
  HUGO_VERSION = "0.30"
  HUGO_BASEURL = "https://blog.sandipb.net/"


[context.deploy-preview.environment]
  HUGO_VERSION = "0.30"

Update: One thing I found out, was that while each pull request and each commit to a non-master branch got an unique hostname for the preview, Netlify also maintains a constant hostname for every branch. For example, for my dev branch on the website blog-sandipb-net.netlify.com, there is always a preview site at https://dev--blog-sandipb-net.netlify.com/ with a build off the top of this branch. So, I could actually make absolute links work in the deploy-preview environment as well by specifying the HUGO_BASEURL environment variable in netlify.toml.

I admit, I had shied away from trying out this editor for so long even though I had heard good things about it - partly because of its Microsoft heritage (I am an old LUG guy after all), its Visual Studio heritage (a really resource hungry IDE, from a single really brief encounter several years back) and its Electron heritage (embedded browsers haven’t really impressed me in performance). 😄

But VS Code has really really impressed me.

• Its Go plugin works really well out of the box.
• It has out of the box support for Git in the core
• Has a really refined global/workspace/folder settings hierarchy,
• an embedded terminal which is really usable
• A very responsive UI, probably the fastest JS (Actually TS, nevertheless) based editor that I have seen

And best of all … it … is … free!

Much as I love Pycharm, I love the Sublime Text model of pricing more - a single one time pricing for a major version, not an annual one.

And with Microsoft funding full time developers on the project, there is amazing support! Updates are frequent! I actually have an anecdote about the super support by the team.

I had tweeted sometime back about the inability to override GOPATH at the folder level, now that VS code supports workspaces with multiple folders in a workspace.

Ramya from the dev team noticed my tweet on a Sunday and I clarified the use case in a follow up tweet.

I then filed a Github issue, and within 4 hours on a weekend, she had a commit out!

I was sold. Even for a paid software this is amazing, and this is free!

I don’t like being split between editors. I already use Sublime Text 3 for my adhoc editing work (I am typing this in it right now), Pycharm for Python - both with personal licences. I can afford to have another editor for a specific language - I am not even considering the irritable matter of paying for two editors from the same company (no, the all inclusive package from them is way out of my budget). Now I am trying to see if I can move my Python work to this editor - so far it seems pretty encouraging, it has pretty good support for virtualenvs. I need to spend more time on it though - Pycharm has set a pretty high bar.

Here is an example. In various tutorials, pauses are made using time.Sleep().

The first time I saw an example like the following, it made me stop in my tracks.

package main

import (
	"time"
)

func main() {
	time.Sleep(100 * time.Millisecond)
}

Here is the signature of time.Sleep

func Sleep(d Duration)

And here is what Duration and time.Millisecond means.

type Duration int64

const (
        Nanosecond  Duration = 1
        Microsecond          = 1000 * Nanosecond
        Millisecond          = 1000 * Microsecond
        Second               = 1000 * Millisecond
        Minute               = 60 * Second
        Hour                 = 60 * Minute
)

Coming from my past experience with other languages, my question was, in a strongly typed language like Go:

1. How does 100 * time.Millisecond work? Aren’t they entirely two different types? And I know that Go doesn’t support operator overloading. So how does Go know how to use the operator * between a Duration and what looks like an int. My first guess was that Go was converting the duration time.Millisecond to an int to make the multiplication work, like in other language.
2. Even if somehow the multiplication worked, how is the result satisfying the type requirement for the parameter passed to Sleep() which requires a Duration.

Turns out my guess was wrong.

There are a couple of points in the spec that explains what is happening.

• Since Duration is of type int64, this confirms that a Duration and a int64 are different even though the latter is the underlying type of the former.

  A type definition creates a new, distinct type with the same underlying type and operations as the given type, and binds an identifier to it. The new type is called a defined type. It is different from any other type, including the type it is created from.

• However, type identity is the most crucial part which establishes that int64 and Duration as considered to be identical type whenever the question arises.

  A defined type is always different from any other type. Otherwise, two types are identical if their underlying type literals are structurally equivalent; that is, they have the same literal structure and corresponding components have identical types.

• However, this is the missing link I was looking for. This is how 100 * time.Millisecond becomes time.Duration(100) * time.Millisecond

  For other binary operators, the operand types must be identical unless the operation involves shifts or untyped constants. … Except for shift operations, if one operand is an untyped constant and the other operand is not, the constant is converted to the type of the other operand.

• And what happens after the multiplication? The result is considered to be of type time.Duration

  Arithmetic operators apply to numeric values and yield a result of the same type as the first operand.

So, if I have got it right:

1. Go will convert the untyped integer value 100 to the type Duration.
2. Since both have the underlying type of integer, it will follow the integer operator rules for *
3. The result will be converted to the type of time.Duration

Well, the good news doesn’t stop there!

• We now have Go template editing features
• Several nifty updates to the editor, like reminder/wizard for adding comments to public exported symbols, empty slice warning/wizard

And we finally have a release date and pricing.

GoLand 2017.3 is going to be released in early December. GoLand will join the ‘All Products’ pack and will be priced exactly the same as PyCharm, PhpStorm, RubyMine, DataGrip, CLion, and AppCode.

I have been using Hugo as a static website generator for a while. I love the speed, coming from its Go origins. I love a static website generator for the peace-of-mind it gives me (No did I forget to update my XXX blog software after that bug came out? ).

But of course, it is not all peachy.

First of all, it still requires me to keep my theme in sync with the theme author, just in case he has fixed some bugs that you saw. Theme management is best done in Hugo using git submodules which is its own bundle of fun. And if you made some changes to the theme of your own, keeping your changes merged well with the upstream theme author is just the cherry on the top.

Now I just need to figure out where this error is coming from :-/

  ERROR 2017/11/05 01:49:56 Page's Now is deprecated and will be removed in Hugo 0.31. Use now (the template func).

Update: Ok, my comment above was a bit uninformed. The right way to make modifications to a theme in Hugo is not to edit the theme code itself, but to override it. I just switched my theme to Blackburn and this time I took care in ensuring that the stuff I wanted modified are made in a way it is overridden.

So for CSS styles, like the blockquotes in this blog, were overriden using a custom local CSS file. The theme actually has a rarely provided config to specify the css file name used to override the theme styles. This made it pretty easy

In case a theme doesn’t provide that, I would have needed to copy the partial template corresponding to the html head to the local site file hierarchy, and edit that. The docs explain this pretty well.

The sun is exactly overhead twice a year in Lahaina, Hawaii, once in May and once in July. Poles don’t cast shadows, giving the urban landscape an eerie appearance. Hawaii is the only state in the US where the sun’s rays are perpendicular to the surface of Earth. It’s called a subsolar point.

Via Boingboing

The ‘Singleton’ DP is all about ensuring that just one instance of a certain class is ever created. It has a catchy name and is thus enormously popular, but it’s NOT a good idea – it displays different sorts of problems in different object-models. What we should really WANT, typically, is to let as many instances be created as necessary, BUT all with shared state. Who cares about identity – it’s state (and behavior) we care about!

By Alex Martelli at Singleton? We Don’t Need No Stinkin’ Singleton: The Borg Design Pattern.

I lost an hour of my life today trying to figure out why Sphinx would not document a class inside a module specified by :automodule:.

I learnt two things today.

Firstly, Sphinx will not document a class automatically as part of :automodule: if the class itself doesn’t have a docstring. You can force the class to be displayed by having an additional :autoclass:directive though.

If the class does have a doc string, it will be displayed including the documentation of its regular (ones without a leading _ in the name) methods. It will still not show the constructor though.

To display the constructor, I first thought I would add the :special-members: option to the :automodule: directive. But that didn’t work very well, as it showed various unnecessary private functions as well, e.g. _weakref.

I finally discovered that the best way to get the constructor documentation displayed was using the autoclass_content config in the sphinx config file.

autoclass_content = 'both'

As per the documentation,

This value selects what content will be inserted into the main body of an autoclass directive. The possible values are:

• “class”: Only the class’ docstring is inserted. This is the default. You “can still document init as a separate method using automethod or “the members option to autoclass.
• “both”: Both the class’ and the init method’s docstring are concatenated and inserted.
• “init”: Only the init method’s docstring is inserted.

This might be a very esoteric topic for most people, but since I could not find information about this anywhere, I decided to document this in a post.

Here is the problem. I use Jira at work, and today, I needed to close a bunch of tickets based on a search result. Now, searching or doing batch operations is simple enough from the browser, but a small detail made the exercise impossible via the web UI.

Our Jira project requires a field to be filled while closing the ticket - the time spent in the ticket. This breaks Jira in all sorts of ways - the batch operation doesn’t work, some of the email-to-jira interface at work breaks as well.

So I looked at doing this via the Jira Python library. But it didn’t work as expected.

>>> from jira import JIRA

>>> jira = JIRA("https://JIRA_URL", basic_auth=('USER_NAME', 'Password'))
>>>
>>> issue = jira.issue("ISSUE-123")
>>>
>>> [(t['id'], t['name']) for t in jira.transitions(issue)]  # What are the workflows available?
[(u'4', u'Start Progress'), (u'5', u'Resolve Issue'), (u'2', u'Close Issue'), (u'711', u'Planning'), (u'751', u'Blocked'), (u'801', u'To Monitor')
>>>
>>> jira.transition_issue(issue, '2')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
...
jira.exceptions.JIRAError: JiraError HTTP 400
    text: Time Spent is required
    url: ...
    response headers = {...}
    response text = {"errorMessages":["Time Spent is required"],"errors":{}}

The documentation on transitions mentioned that we could add fields in the call to jira.transitions(). That didn’t work as well.

>>> jira.transition_issue(issue, '2', timespent="1h")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
...
jira.exceptions.JIRAError: JiraError HTTP 400
    text: Field 'timespent' cannot be set. It is not on the appropriate screen, or unknown.
    url: ...
    response headers = ...
    response text = {"errorMessages":[],"errors":{"timespent":"Field 'timespent' cannot be set. It is not on the appropriate screen, or unknown."}}

So I scoured the Internet for a long time, till I found this post about how to do it via the REST API - the only official interface to Jira. Here is what needs to be sent as the body to the POST request.

{
    "transition": {
        "id": "2"
    },
    "update": {
        "worklog": [
            {
                "add": {
                    "timeSpent": "2m"
                }
            }
        ]
    }
}

I felt that to be odd, till I looked at both the api documentation for transition and the doc for transition using the python library and I found out why I have not been successful till now.

The REST api supports two ways to update the issue while doing a transition - you can set certain fields using the fields option, or you can use the update option to do more complex changes.

The comment in the Python code revealed that the update method has not yet been implemented.

def transition_issue(self, issue, transition, fields=None, comment=None, **fieldargs):

    # TODO: Support update verbs (same as issue.update())

That put me in a bind. I had only one way to hack around this problem now - using the REST api for the specific operation I wanted, and the Python library for the rest of the work - ugly, but works for now, till the Python library is complete.

So here was the final solution that did what I wanted.

from jira import JIRA
import requests

jira = JIRA("https://JIRA_URL", basic_auth=('USER_NAME', 'PASSWORD'))

d = {}
d["transition"]={"id": "2"}
d["update"]={"worklog": [{"add": {"timeSpent": "1h"}}]}

s = requests.Session()
s.auth = ("USER", "PASSWORD")
s.headers.update({"Content-Type": "application/json"})

j="https://JIRA_URL"

issue_list = jira.search_issues("assignee = currentUser() AND resolution = Unresolved  and status != Closed and updatedDate < '2015-10-01' and project='PROJECT' ORDER BY updatedDate DESC")

for i in issue_list:
    print i
    s.post(j+"/rest/api/2/issue/"+i.key+"/transitions", data=json.dumps(d))

Via TheNextWeb

In any case, writing or not, it is much better to move to a hosted solution, and I moved my domain and migrated my Jekyll website (painfully) to the wordpress.com hosted site.

It is impossible to ignore avro at work - it is the data serialization format of choice at work (and rightly so), whether it is to store data into Kafka or into our document database Espresso. Recently, I had the need to read avro data serialized by a Java application, and I looked into how I might use Python to read such data.

The avro python library uses schemas, and can store data in a compact binary format using both deflate and snappy compression. I wanted to test out how compact the serialization format is as compared to say, CSV.

I set up a Python virtual environment using the nifty virtualenv wrapper and installed the python avro library and the snappy library. On my Ubuntu system, the python snappy library is available from the package repository, but I used the Pypi one anyway. I had to install the c development package (libsnappy-dev) as a prerequisite though.

$ mkvirtualenv avro

$ workon avro

(avro) $ pip install avro

# Remember that it is 'python-snappy', and not just 'snappy', which is a
# completely different library
(avro) $ pip install python-snappy

And I was ready for my code. First I decided on a simple data set: a weather data set which was part of the pandas cookbook.

Based on this, I created this simple schema.

{
    "namespace": "net.sandipb.avro.example.weather",
    "type": "record",
    "name": "Reading",
    "doc": "Weather reading at a point in time",
    "fields": [
        {"name": "time", "type": "int", "doc": "Seconds since epoch"},
        {"name": "temp", "type": "float", "doc": "Temperature in Celsius"},
        {"name": "dew_point_temp", "type": "float", "doc": "Dew point temperature in Celsius"},
        {"name": "humidity", "type": "int", "doc": "Relative humidity %"},
        {"name": "wind_speed", "type": "float", "doc": "Wind speed in km/h"},
        {"name": "visibility", "type": "float", "doc": "Visibility in km"},
        {"name": "pressure", "type": "float", "doc": "Atmospheric pressure in kPa"},
        {"name": "weather", "type": "string", "doc": "Weather summary"}
    ]
}

Now to actually process the file, I created this Python script to read the CSV file and to write to three different avro files, each with a different compression format - null (no compression), deflate and snappy.

Here are some relevant parts of the code:

# Load the avro schema to validate and serialize data
schema = avro.schema.parse(open("weather.avsc").read())

# Open using a avro container class, providing the schema to use and the
# compression to use
writer_deflate = DataFileWriter(open("weather_data_deflate.avro", "wb"), DatumWriter(), schema, codec="deflate")

# Write a dict with the data
writer_deflate.append(row)

# close the container
writer_deflate.close()

The result of this was … unexpected. The deflate codec showed more compression than snappy.

$ ls -lSh *.avro *.csv
-rw-rw-r-- 1 sandipb sandipb 492K May 20 01:34 weather_2012.csv
-rw-rw-r-- 1 sandipb sandipb 317K May 20 02:29 weather_data_null.avro
-rw-rw-r-- 1 sandipb sandipb 178K May 20 02:29 weather_data_snappy.avro
-rw-rw-r-- 1 sandipb sandipb 121K May 20 02:29 weather_data_deflate.avro

It is possible that a larger dataset could show better compression for snappy. This is the first time I have used this codec, I need to read up a bit more about it. I will try this experiment with a larger dataset and update this post in the future.

In Python, dictionaries with default values is not part of the core language, and need to be imported from a standard library. For some reason I still don’t know, you cannot set a simple default value, you need supply a function object which is going to create a value for you. :/

So if I want to create a default dictionary in Python, you need to do something like this:

>>> from collections import defaultdict

>>> d = defaultdict(lambda: 5)

Now when you access any non-existent key in this hash, the hash gets magically populated with that key and the default value. See below.

>>> if d["one"]: print "not empty"
... 
not empty
>>>
>>> print dict(d)
{'one': 5}
>>>

Ruby hashes support default values in the core language. So you can do something like:

>> d = Hash.new(5)
=> {}

>> puts d["a"]
5
=> nil

>> d
=> {}

Wait! See the difference? Evaluating a non-existing hash position is Ruby returns the default value, but unlike Python, it doesn’t set the value!

After it drilled down to this quirk, I looked around and found some interesting articles.

This article from 2008 by David Black was particularly interesting. He points out in this article how this Hash behaviour breaks a very popular Ruby idiom.

Normally, the idiomatic Ruby way to initialize or return a variable goes like this:

>> d = Hash.new
=> {}

>> d["name"] ||= "Skye"
=> "Skye"

>> d
=> {"name"=>"Skye"}

However, if you use a Hash with a default value, it breaks this idiom because evaluating a non-existing key returns the default value. However, you would intuitively expect the ||= operator to at least set the missing key to the default value! But that doesn’t happen due to the peculiar treatment to that operator by Ruby. Ruby evaluates A ||= B NOT to A = A || B, but A || A=B. This never lets the value to be assigned to keys in default hashes.

>> d = Hash.new("May")
=> {}

>> d["name"] ||= "Skye"
=> "May"

>> d
=> {}

Some gotcha lurking there in an otherwise beautiful language.

Another nice ref: http://www.rubyinside.com/what-rubys-double-pipe-or-equals-really-does-5488.html

Ok, I don’t particularly like calling a bug fantastic, in this case, it is more of a fantastic troubleshooting of a bug. What I found interesting was the layers that were unpeeled one by one to reach the probable region of the root cause. (Yeah, the root cause is probably so esoteric and confined to a specific combination of version, that it is unlikely to be looked at by anybody).

Here is Pagerduty’s summary of the bug.

After more than a month of tireless research and testing, we have finally got to the bottom of our ZooKeeper mystery. Corruption during AES encryption in Xen v4.1 or v3.4 paravirtual guests running a Linux 3.0+ kernel, combined with the lack of TCP checksum validation in IPSec Transport mode, which leads to the admission of corrupted TCP data on a ZooKeeper node, resulting in an unhandled exception from which ZooKeeper is unable to recover. Jeez. Talk about a needle in a haystack… Even after all this, we are still unsure where precisely the bug lies. Despite that fact, we’re still pretty satisfied with the outcome of the investigation. Now all we need to do is work around it.