Way too often than I’d care to admit, I will quickly create a new branch:

❱ git switch feature/🔥
fatal: invalid reference: feature/🔥

Then pop the stash and/or do the fast and furious writing thing. Much later I would discover that I had forgotten the -c option to actually create a new branch, realizing I just messed up another unrelated branch

Obviously I need something to bring the error exit more to my attention, so I cooked up this little attention functionality

Now, when git fails, this banner pops up in the shell

Perfect Not too noisy and and not too big, but enough to catch my attention

Its a small function added to .zshrc. It captures the last command that was run and compare it to git (that part can of course be omitted or changed to any other program). In case of error exit from git, the banner is shown

print_error_box() {
    local msg=" $1 "
    local width=${#msg}
    local border

    border=$(printf '%*s' "$width" '')
    border=${border// /─}

    print -P "%F{red}┌${border}┐%f"
    print -P "%F{red}│%f%K{red}%F{white}${msg}%f%k%F{red}│%f"
    print -P "%F{red}└${border}┘%f"
}

typeset -g LAST_CMD=""

preexec() {
    LAST_CMD="$1"
}

precmd() {
    local exit_code=$?
    if (( exit_code != 0 )) && [[ "$LAST_CMD" == git* ]]; then
        print_error_box "GURU MEDITATION ERROR"
    fi
}

The post Error banner in shell appeared first on Monzool's Personal Publishing.

The problem

GitHub recommends pinning actions to full commit SHAs for security. We also apply this practice to our own workflows and action for reproducebility. In practice that means that workflow files are full of lines like this:

uses: Internal-actions/utilities/.github/actions/read-project-file@1028f1aedf5426726c149b3bdfca0b68bf001567 # v1.0.12

When a colleague bumps a versions, git simply cannot pick one by itself, thus every merge produces conflicts. Every. Single. Time. The resolution is always the same: always pick whichever side has the higher version.

After manually resolving the same type of conflict for the umpteenth time, I went looking for a better way.

Git merge drivers

It turns out git has a built-in mechanism for exactly this kind of thing: custom merge drivers. A merge driver is a program that git calls to resolve conflicts on specific files, instead of using its default three-way merge.

The concept is simple. Two pieces of configuration work together:

1. .gitattributes maps file patterns to a named driver
2. .gitconfig defines calling of that named driver

For example, telling git to use a driver called latest-uses-version for workflow files, first add a file matching rule in .gitattributes

.github/workflows/*.yml    merge=latest-uses-version

Then define the driver by name and how it should be called in .gitconfig

git config merge.latest-uses-version.name "Pick latest uses version"
git config merge.latest-uses-version.driver "python3 merge-latest-uses-version.py %O %A %B"

When git encounters a conflict in a matching file, it will call the driver with three temporary files: %O (common ancestor), %A (ours), and %B (theirs). The driver writes its result into %A and exits 0 for success or 1 if conflicts remain.

The driver

I wrote a Python script to handle the uses version conflicts. In our setup, every uses line is sha-pinned and always has a semver comment appended (ensured by another automation script), like @abc123... # v1.0.12. That comment is what makes it possible for the driver to compare the two sides of a conflict and pick the newer version.

#!/usr/bin/env python3

import re
import subprocess
import sys

ANCESTOR = sys.argv[1]
OURS = sys.argv[2]
THEIRS = sys.argv[3]

# Attempt standard merge first
result = subprocess.run(
    ["git", "merge-file", "-p", OURS, ANCESTOR, THEIRS],
    capture_output=True,
    text=True,
)

if result.returncode == 0:
    # Clean merge — no conflicts
    with open(OURS, "w") as f:
        f.write(result.stdout)
    sys.exit(0)

# Conflicts exist — write the conflicted output, then resolve what we can
content = result.stdout

# Pattern: uses: some/path@<exactly 40 hex chars> # vX.Y.Z
SHA_VERSION_RE = re.compile(
    r"uses:\s+\S+@[0-9a-fA-F]{40}\s+#\s*v(\d+\.\d+\.\d+)"
)

CONFLICT_RE = re.compile(
    r"<<<<<<< [^\n]*\n(.*?)\n=======\n(.*?)\n>>>>>>> [^\n]*",
    re.DOTALL,
)


def pick_latest(match):
    head_block = match.group(1)
    incoming_block = match.group(2)

    head_ver = SHA_VERSION_RE.search(head_block)
    incoming_ver = SHA_VERSION_RE.search(incoming_block)

    # Only resolve if BOTH sides are sha-pinned uses: lines with version comments
    if not head_ver or not incoming_ver:
        return match.group(0)  # Leave conflict markers for manual resolution

    h = tuple(int(x) for x in head_ver.group(1).split("."))
    i = tuple(int(x) for x in incoming_ver.group(1).split("."))

    return head_block if h >= i else incoming_block


resolved = CONFLICT_RE.sub(pick_latest, content)

with open(OURS, "w") as f:
    f.write(resolved)

sys.exit(1 if "<<<<<<< " in resolved else 0)

It first runs git merge-file to get the standard three-way merge. If conflicts remain, it scans each hunk for the sha-pinned version pattern. Only when both sides match does it pick the higher semver. Anything else is left with conflict markers for manual resolution.

Setting it up

Git merge drivers need two things configured: the driver definition and the file pattern mapping.

The driver definition goes in git config – either globally (~/.gitconfig) or per-repo (.git/config). As our pipeline is spread across multiple repositories I did not want to maintain the same setup in all repositories. By doing the global configuration the setup can be shared between all. Choose a per-repo when the driver is specific to one project.

The file pattern mapping goes in .gitattributes – either a global file (pointed to by core.attributesFile) or a per-repo .gitattributes. The global and per-repo attributes are additive; git merges them together, so they won’t conflict.

To automate this setup for coworkers, I made a script to do the git setup.

./setup-merge-driver.sh

setup-merge-driver.sh:

#!/usr/bin/bash

# Install the "latest-uses-version" custom merge driver globally.

set -euo pipefail

script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
driver_path="${script_dir}/tools/⚙/merge-latest-uses-version.py"

if [[ ! -f "${driver_path}" ]]; then
    echo "ERROR: merge driver not found at ${driver_path}" >&2
    exit 1
fi

# Register the merge driver in global git config (idempotent).
git config --global merge.latest-uses-version.name \
    "Auto-resolve sha-pinned action version conflicts by picking the latest"
git config --global merge.latest-uses-version.driver \
    "python3 '${driver_path}' %O %A %B"

echo "✔ Merge driver 'latest-uses-version' registered in global git config."

# Ensure a global attributes file is configured.
attr_file="$(git config --global core.attributesFile 2>/dev/null || true)"

if [[ -z "${attr_file}" ]]; then
    attr_file="${HOME}/.gitattributes"
    git config --global core.attributesFile "${attr_file}"
    echo "✔ Global core.attributesFile set to ${attr_file}"
fi

# Expand ~ in case the user configured it that way.
attr_file="${attr_file/#\~/${HOME}}"

# Add patterns (idempotent).
patterns=(
    ".github/workflows/*.yml merge=latest-uses-version"
    "common/*.yml merge=latest-uses-version"
)

touch "${attr_file}"

for pattern in "${patterns[@]}"; do
    if ! grep -qxF "${pattern}" "${attr_file}"; then
        echo "${pattern}" >> "${attr_file}"
        echo "✔ Added to ${attr_file}: ${pattern}"
    else
        echo "· Already present in ${attr_file}: ${pattern}"
    fi
done

echo ""
echo "Done. The merge driver is ready for all repositories."

What it does

• Registers the latest-uses-version driver in global ~/.gitconfig (this is idempotent and safe to re-run)
• Ensures core.attributesFile is set. It respects an existing one and creates ~/.gitattributes only if none is already configured
• Adds file patterns (.github/workflows/*.yml and common/*.yml) – skips if already present
• Uses absolute path to the driver script so it works from any repo

The result

With the driver installed, what used to be a manual chore now just works. Given a conflict like this during merge:

<<<<<<< HEAD
      uses: Internal-actions/utilities/.github/actions/read-project-file@1028f1aedf5426726c149b3bdfca0b68bf001567 # v1.0.12
=======
      uses: Internal-actions/utilities/.github/actions/read-project-file@386f499a1f68c20a15df7ad877b77fa139d7e932 # v1.0.15
>>>>>>> feature-branch

The driver picks v1.0.15 and the merge completes cleanly:

      uses: Internal-actions/utilities/.github/actions/read-project-file@386f499a1f68c20a15df7ad877b77fa139d7e932 # v1.0.15

No manual intervention needed. Git now just does the right thing

Multiple drivers in git config

As a note, then the uses version driver is just one driver. Git supports as many custom merge drivers as as desired

# Driver 1: sha-pinned action versions
git config merge.latest-uses-version.name "Pick latest sha-pinned action version"
git config merge.latest-uses-version.driver "python3 scripts/merge-latest-uses-version.py %O %A %B"

# Driver 2: hypothetical package-lock resolver
git config merge.package-lock.name "Regenerate package-lock on conflict"
git config merge.package-lock.driver "scripts/merge-package-lock.sh %O %A %B"

# Driver 3: always keep ours for generated files
git config merge.keep-ours.name "Always keep our version"
git config merge.keep-ours.driver "true"

.gitattributes routes files to drivers

.github/workflows/*.yml     merge=latest-uses-version
package-lock.json           merge=package-lock
generated/**                merge=keep-ours

Rules

• Each file matches at most one merge= attribute (last match wins, like .gitignore).
• If a file matches no merge= rule, git uses its built-in merge.
• If you need two strategies on the same file (e.g., resolve uses versions and something else in the same YAML), that must be handled within a single driver script. Unfortunately drivers can’t be chained.
• You can use any .gitattributes pattern: exact paths, globs, directory prefixes, etc.

Conclusion

I am super stoked about the result. The algorithm for this particular case was pretty straight forward, but it works just perfectly – and I got to learn another cool git feature.

The post Custom git merge driver appeared first on Monzool's Personal Publishing.

What happened

I am not really sure why this situation ended up happening. It might have been a brew update or a oh-my-zsh update?

Digging into zsh’s fpath, I discovered that Homebrew ships its own _git zsh completion wrapper alongside the git formula. It lives at:

/home/linuxbrew/.linuxbrew/share/zsh/site-functions/_git

This is a lightweight wrapper that narrows down known git commands, to a finite set of “common commands” – things like add, commit, push.

My Ubuntu’s own _git at /usr/share/zsh/functions/Completion/Unix/_git is far more comprehensive. It has descriptions for all commands and proper alias support.

Figuring out which completion is active

To check which _git zsh is actually loading, start a new shell and run:

❱ print -l $fpath
/home/linuxbrew/.linuxbrew/share/zsh/site-functions   # <-- brew
/home/monzool/.oh-my-zsh/plugins/git
...
/usr/share/zsh/functions/Completion/Unix              # <-- system

zsh resolves autoloaded completion functions by searching fpath in order and using the first match. Brew’s _git is found before the system one because brew shellenv prepends its directory to fpath:

fpath[1,0]="/home/linuxbrew/.linuxbrew/share/zsh/site-functions";

The fix

The brew site documents how to activate its shell completions. In this case I wanted to have the system completions to take precedence.

The fix is to move brew’s site-functions to the end of fpath, so system completions take priority while brew-only completions (like _brew, _rg, _jj) still work.

I added this to ~/.zshrc right after eval "$(brew shellenv)":

# Move brew's zsh completions to end of fpath so system completions take priority
fpath=(${fpath:#$HOMEBREW_PREFIX/share/zsh/site-functions} $HOMEBREW_PREFIX/share/zsh/site-functions)

Then cleared the completion cache

〉rm -f ~/.zcompdump*
〉compinit

Full git completion restored. Aliases and descriptions for all commands now works

The post Homebrew breaking git tab completion in zsh appeared first on Monzool's Personal Publishing.

tl;dr

Switching phones on Android sucks, it’s just a miserable experience. Okay, maybe a bit over-dramatic, but it genuinely is a lackluster experience. With a few exceptions all finely tuned settings are lost on both the Android system and native apps. Some native apps supports backup/restore, but its a proprietary and mainly manual solution that I expect exactly zero normies would ever mange to complete.

What Android is missing is an integrated backup/restore system and transparent cloud storing (or syncing) of app and system settings.

From 🢡 to

My Pixel 7a has been a good experience, but ultimately the catastrophic inconsistency in photos made we throw in the towel. I instead acquired a Samsung Galaxy S23. For transferring data from my Pixel phone to the Samsung phone I used the Samsung SmartSwitch app. Android can install all apps itself when switching to a new phone, but all the other files like notification sounds, emulator files, documents etc are left behind. Samsung SmartSwitch will copy all files over

Nova Launcher

So I don’t quite know where to throw blame her. After using the Samsung SmartSwitch functionality, Nova Launcher was installed. But after importing my backup not all settings was restored. This was due to Nova Launcher Premium was not installed in the transfer. After installing that from Google Play – and restarting the phone, then after another restore, all settings worked. The widgets was missing. Same goes for the wallpaper… so only a partial success.

☆☆☆

System setup

Language

I run my Android in English(Danish) language setting. For some apps it makes more sense to switch from default english to native danish. Some apps support doing this through the Android settings, which is a really nice feature. Unfortunately not all apps (particularly public transportation apps) support this and you have to poke around in the individual app settings to find where to flip the language

☆☆☆☆

Notification sounds

This one is a weird weird. For many apps I had to use their proprietary backup/restore solution. But for every single one, the notification sound setting was not restored and had reverted to default sounds. All notification and ring-tone files were replicated by Samsung SmartSwitch to be at the exact same place in the filesystem, but it did not work. This make me think that notification sounds are part of the Android OS settings

Perhaps this is a difference between Samsung Android and Pixel Android, but not all notifications are no longer to be found in the various apps. Thanks to the Samsung SmartSwitch app, the files are there in the same location as on the old phone. They are just not all presented. Seems a bit inconsistent between apps, so not really sure what is going on there…

☆☆☆☆

App notifications

I had a lot of app restricted on which notifications they were allowed to disturb with. None of these setting ported over, so a horde of attention seeking notifications jingled the phone the first few days until I got the throttle back on that

☆☆☆☆

Google apps

The Samsung SmartSwitch app filtered out quite a few of the installed Google apps. Gmail, Lens and a few others were transferred, but these were missing:

• Calendar
• Files
• Clock
• Contacts
• Phone
• GBoard

I suspect this is because Samsung has their own competing apps, and blocked the install. What a shit move, Samsung Weird thing was, that the icons from the apps were actually present. First time activating them, I would then be notified that: the app was not installed, but I could search for them in an app store. After install I realized that none of the settings was restored. Every app had to be set up again from scratch. I don’t know if this was because of the shitty move by Samsung SmartSwitch or just Google not caring…

☆☆☆☆

Google Chrome

Chrome have a feature (if sync is enabled), that when opening Chrome on the new phone it will ask if to open all the tabs that are synced. It asked to sync only 124 tabs. Where were all the rest! It turns out it does not sync inactive tabs Chrome has a feature where tabs that has not be active for a set time (default 21 days) the tab will become an inactive tab. I had to disable the inactive tab feature, then wait a bit, then wipe Chrome app data on the new phone. Now when opening, all 463 tabs opened in the new phone

☆☆

Google BGoard

As mentioned many of the Google apps was not transferred and had to be set up from scratch. I’ve made quite a few changes to GBoard on the old phone. I could only find a way to backup the personal dictionary though. Not that it mattered, because I could not find any way to restore the dictionary on the new phone. Why are these settings not just stored in the cloud?

☆☆☆☆

Google Weather

On the new phone the weather app was not there. It was not to be found. I then searched the Play store but could not find the Google Weather app (the app store is littered with weather apps – many of questionable intent). Then one day, at a seeming arbitrary point it popped up a notification about needing precise location permissions. So it was installed. I then looked at the apps list in settings. I probably should have looked there, but didn’t think of it as I couldn’t find it the app drawer. In the settings I found this setting: Show Weather on Apps screen. After enabling that, the app could suddenly be found and started manually. What an obnoxious setting. I want to see the weather to plan ahead – not when this brain-dead app wrongly thinks its relevant. A weird this is that on the new phone it thinks the time is one hour later that it actually is. I ended up deleting the app

☆☆☆☆

Other apps

Signal

Signal was nice. You basically just select transfer on the old and new phone, and the transfer happens automagically I do wonder how this process will be if you have lost your old phone

☆

Whatsapp

Except for the notification sound everything was restored automatically after login. It makes backup/restore to Google Drive. Works perfectly

Microsoft Authenticator

The Microsoft Authenticator has a built-in cloud solution to do automatic backup and manual restore to another phone. However, restoring was not working. It kept saying “Your backup is not stored with this email”

“Fortunately” this is a common bug, that was complained about before had a solution for (Microsoft incompetently do not provide linking to answers, but look for the answer by Arif IT-Pro). An authenticator app that cannot restore do not bring confidence…

☆☆

OK Benzin

This was a particular annoying experience, partly because I had forgotten my password. Their reset procedure was to send a reset link in an email. That link would open the app, but the app would go to normal login, not password reset. Had to use a computer and their website to get my password reset

☆☆☆

AMDroid

Moving an alarm app (AMdroid) requires three system setting changes

1. Add it to the never sleeping apps list
2. Set it to unrestricted battery usage
3. Allow it to appear on top

By bad luck I forgot to do the manual backup/restore for that app, so I overslept the next workday.

☆☆☆

F-Droid

F-Droid itself seemed to just work. Maybe that is from using the Samsung SmartSwitch, because from reading online that would have required a backup/restore session, but at first glance all apps where transferred Later I found that it was actually a mixed story. The apps that had a presence on both F-Droid and Play Store, were now all installed as the Play Store edition?!

☆☆☆

MacroDroid

Had to do backup/restore, but that worked fine enough. Then I manually had to sideload their permission helper. That required flipping several options to get permission to install the apk. I guess in the future this kind of system tweaking will only become even more complicated. I really hope they manage to keep MacroDroid and option, because its invaluable to un-fuck some of the crap in Android

☆☆☆

DGT GTD

I’ve been using this todo/reminder app for quite a long time, but are not transitioning to Tasks It automatically does a local zip backup of all tasks, and by using Samsung SmartSwitch the backups were available for restoring.

☆

Apps (non Google) with online accounts

Many apps like social media (Strava, Reddit, Bluesky, Lemmy etc) rely on online services and only require a login to get up and running again. Apart from throttling on their notifications, surprisingly few issues with any of them

☆

Wall of shame

Digital wellbeing

What an atrocious system app. This app would endlessly notify me with various well being advice. Seriously, just shut the uppppppp!!!!!!!!. The app didn’t even allow me to disable notifications The toggle button was disabled. I added it to my app kill list in MacroDroid

Waze

It was copied to the phone just fine and only required a login to work; but as with many other apps, the settings had reset. Initially the notifications from Waze angered me quite a bit. Every day it would notify me that there is 30 minutes until arriving at work. What am I suppose to use that information for? I’ve already been in public transportation for 20 minutes before that, and know exactly when I will arrive at work. A complete pointless notification. The feature might have a value if you drive to work in a congested area, and you need to be at work at a precise time, but I think that is a notification you would want to activate for that specific need. Some interweb search and I managed to disable that notification. I generally had to poke a lot of settings in Waze to make it less obnoxious. Waze also got added to my app kill list in MacroDroid

☆

Misc

My digital drivers license app did not transfer settings. To activate the drivers license I have to go through a validation process that includes scanning of my passport. I am not really too upset about that. It could be a fair security restriction.

Summary

An seemingly endless task of logins and permissions acceptances. The system settings app list counted around 350 apps installed, and most of them require some sort of login. I took the most important ones, that I could remember in the moment in one go, and then delayed the rest until I needed them… which is also why I held up the line in the supermarket as I had to login, using my national digital id (MitId), in the supermarket app to get the discounts

Most apps, including many of Google’s had their settings lost. I had to set up language, notifications and a lot of other things once again

Why do Google apps not store all their settings in the cloud? Why don’t all apps not store their settings in the cloud. Is there no api for storing settings/backup in Google Drive? There should be. That would make changing phones not so much of a shit show

System settings might be a bit more difficult to depending on vendor and version differences between phones, but its not an unsolvable problem. Like it migrated wifi and bluetooth connection registrations just fine, so why not also migrate hotspot settings, personal dictionaries, app language settings etc. Should be possible

Switching Android phones is a experience.

The post Moving to a new android phone sucks appeared first on Monzool's Personal Publishing.

When waking up at the morning, I would find my phone being stuck waiting for my SIM pin login. At the night before nothing extraordinary events had happened. I just put down the phone, plugged in the powercord and went to sleep. As this had happend quite a few times the last couple of months, I decided to investigate even though I an not an Android expert

I started by investigating information from adb logcat.

--------- beginning of kernel
05-25 01:30:15.540     0     0 I trusty  : boot args 0x*** 0x*** 0x6 0x0

<snip>

05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_tee_storage][gf_tee_open_object] exit. err=GF_ERROR_OPEN_SECURE_OBJECT_FAILED, errno=1035
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_secure_object][gf_so_load_persistent_object] so(finger_0_3.so) not exist
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_secure_object][gf_so_load_persistent_object] exit. err=GF_ERROR_OPEN_SECURE_OBJECT_FAILED, errno=1035
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_fpcore_common][fpcore_load_template_object] exit. err=GF_ERROR_OPEN_SECURE_OBJECT_FAILED, errno=1035
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][I][gf_tee_storage][trusty_file_open] 0 LHBM_finger_0_3_bak.so 1
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_tee_storage][trusty_file_open] Unable to open LHBM_finger_0_3_bak.so on port com.android.trusty.storage.client.td
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_tee_storage][trusty_file_open] Failed to open LHBM_finger_0_3_bak.so on com.android.trusty.storage.client.td. No fallback available.
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_tee_storage][gf_tee_open_object] exit. err=GF_ERROR_OPEN_SECURE_OBJECT_FAILED, errno=1035
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_secure_object][gf_so_load_persistent_object] so(finger_0_3_bak.so) not exist
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_secure_object][gf_so_load_persistent_object] exit. err=GF_ERROR_OPEN_SECURE_OBJECT_FAILED, errno=1035
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_fpcore_common][fpcore_load_template_object] exit. err=GF_ERROR_OPEN_SECURE_OBJECT_FAILED, errno=1035
05-25 01:30:46.436     0     0 I trusty  : [GF_TA][E][gf_fpcore_common][fpcore_get_finger_ids_from_so] exit. err=GF_ERROR_OPEN_SECURE_OBJECT_FAILED, errno=1035
05-25 07:56:03.561     0     0 I trusty  : [GF_TA][I][gf_tee_storage][trusty_file_open] 0 LHBM_auth_att.so 0
05-25 07:56:47.350     0     0 I trusty  : sac-ssmt: 111: TPU: Setup TZ SSMT 0
05-25 07:56:47.369     0     0 I trusty  : sac-ssmt: 111: TPU: Setup TZ SSMT 0
05-25 07:56:58.807     0     0 I trusty  : sac-ssmt: 111: TPU: Setup TZ SSMT 0
05-25 07:56:58.876     0     0 I trusty  : sac-ssmt: 111: TPU: Setup TZ SSMT 0
05-25 07:57:56.248     0     0 I trusty  : [GF_TA][I][gf_delmar_sensor][determine_expo_mode] operation: 2, retry: 0, need_dec_gain_due_to_als: 0, expo mode:0

I did a SIM pin unlock 07:56, but before that the phone had just being doing nothing since 01:30:46 where it had been failing since boot at 01:30:15.

When feeding the errors messages to ChatGPT. It concluded that this is from a failing Trusty TEE (Trusted Execution Environment). Trusty TEE is a security feature that runs isolated from the Android OS, and uses trust zones for handling keys, bio-metrics, pins and other sensitive data.

So if Trusty Tee was the last thing running – what made it then be needed? I decided to investigate the cause of the reboot. To get access restricted logs, I had to generate and download a bugreport from the phone

❱ adb bugreport reboot_at_night-2025-05-25.zip

Grepping a bit in kernel message log for failures…

❱ grep -a -iE 'panic|watchdog|fatal|reboot|crash|stack|Call trace' ./FS/data/misc/recovery/last_kmsg

revealed some interesting information

[474014.231799][  T547] init: Received sys.powerctl='reboot,unAttended,mainline_update' from pid: 1776 (system_server)
[474014.232080][    T1] init: Got shutdown_command 'reboot(unattended,mainline_update)'...
[474016.354876][    T1] init: Service vold has 'reboot_on_failure' option and failed, shutting down System.
[474020.389800][    T1] init: Reboot ending, jumping to kernel
...
Reboot Info:
  Reboot reason: 0xcafe - User Reboot(S/W Reboot)
  Reboot mode: 0x0 - Normal Boot

So the device has rebooted due to an unattended mainline update. Unattended updates is a new feature introduced in Android 14 that allows the Android system to automatically apply certain system updates (mainline updates) and then start the phone again, such that the user will not know that the update happened (apart from all background apps being closed).

I then grepped for updates from today, but the list was surprisingly long, so narrowed it down to mainline updates close to the reboot timestamp

❱ adb shell dumpsys package apex | grep -E 'Package \[|lastUpdateTime' | grep -A1  mainline |  grep -B1 2025-05-25
  Package [com.google.mainline.telemetry] (17fc754):
    lastUpdateTime=2025-05-25 01:30:43
  Package [com.google.mainline.adservices] (df25ac):
    lastUpdateTime=2025-05-25 01:30:43

Two mainline updates! That looks like the smoking gun.

Reading up a bit on the unattended updating mechanism, I believe the story is:

• A nightly automatic update of apps included a mainline update that required rebooting (com.google.mainline.adservices).
• The phone is issuing an unattended reboot (sys.powerctl='reboot,unattended,mainline_updated').
• When booting the system will use Resume on Reboot” to do Sim Pin Replay* to do SIM pin login on behalf of the user.
• Resume on reboot will use Trusty TEE to do the critical trust action of doing SIM PIN login.
• Trusty TEE fails to access the fingerprint storage (Unable to open LHBM_finger_0_3_bak.so), which I presume is required for validating access to SIM pin data and the sequence stops prematurely

This is a particular annoying issue as the phone will fail to reboot at any arbitrary night. It fully depends on when Google decides to update a mainline app.

Not sure how to fix this… I guess the error is related to the fingerprint storage, so perhaps either switching to another unlock method (which I am not interested in), or clearing current stored fingerprints and re-create. Regardless there could be long time for a verification as this depends on when Google pushes the next mainline update

The post Pixel 7a Trusty TEE failure appeared first on Monzool's Personal Publishing.

ESET RUNTIME ERROR because of locale settings. That is what I by coincident discovered one day, when looking at the Ubuntu syslog for other things. ESET is a malware protection software that is bestowed upon us by $WORK. It works on Windows, Mac and Linux and usually it doesn’t make much fuzz.

The syslog was filled with “endless” entries of runtime errors due to failure in locale initialization:

root@L5:~# tail -f /var/log/syslog
2025-04-22T06:46:16.404457+02:00 L5 egui[9000]: Locale not supported by C library.#012#011Using the fallback 'C' locale.
2025-04-22T06:46:16.901839+02:00 L5 eset.eea[9000]: terminate called after throwing an instance of 'std::runtime_error'
2025-04-22T06:46:16.902010+02:00 L5 eset.eea[9000]:   what():  locale::facet::_S_create_c_locale name not valid
2025-04-22T06:46:17.573264+02:00 L5 eset.eea[4139]: Aborted (core dumped)

First question might be: “Why are you poking around in the syslog when we modern linux folks have journald?”. It’s quite possibly habit, but I just find it easier to open the syslog file at browse around for stuff

Sidenote: journald

I have yet to find a good way to discover these kind of issues with journalctl. There is journalctl -p err..alert -x --boot but for this case none of the messages can be found with that filter! Also, annoyingly, the core dump errors are listed as owned by the systemd-coredump service, and not the eea.service (for reasons discovered later there is a reason for that in that case).

Then having already discovered a suspicious message, does not always make it straight forward to investigate further.

❱ journalctl --since "2 days ago" -g 'locale\:\:facet\:\:_S_create_c_locale'
Apr 22 06:41:59 L5 eset.eea[4204]:   what():  locale::facet::_S_create_c_locale name not valid

That entry told us that it originated from eset.eee, but as that is not part of the MESSAGE, grepping for it

❱ journalctl --since "2 days ago" -g 'eset.eea'

will yield no result. Instead we can attempt to filter on the script name

❱ journalctl --since "2 days ago" _COMM=eset.eea

which also yields no results. Then using the syslog identifier finally gives a result

❱ journalctl --since "2 days ago" SYSLOG_IDENTIFIER=eset.eea
Apr 22 06:41:59 L5 eset.eea[4204]: terminate called after throwing an instance of 'std::runtime_error'
Apr 22 06:41:59 L5 eset.eea[4204]:   what():  locale::facet::_S_create_c_locale name not valid
Apr 22 06:42:00 L5 eset.eea[4139]: Aborted (core dumped)

Ignoring the rotated syslog files, then there is a lot of entries of that error

root@L5:~# grep "locale::facet::_S_create_c_locale" /var/log/syslog | wc -l
2750

The current syslog file ranges 13 days of errors (using sed to print only first and last hits)

root@L5:~# grep "locale::facet::_S_create_c_locale" /var/log/syslog | sed -n '1p;$p'
2025-04-10T06:44:53.808725+02:00 L5 eset.eea[1227413]:   what():  locale::facet::_S_create_c_locale name not valid
2025-04-22T09:51:36.491258+02:00 L5 eset.eea[42077]:   what():  locale::facet::_S_create_c_locale name not valid

The error let me to check the core dump registry

root@L5:~# coredumpctl | head
TIME                             PID  UID  GID SIG     COREFILE EXE                                                                                                              SIZE
Mon 2025-03-17 07:58:38 CET  4065929 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui
Mon 2025-03-17 07:59:39 CET  4066715 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui
Mon 2025-03-17 08:00:40 CET  4067264 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui
Mon 2025-03-17 08:01:41 CET  4067862 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui
Mon 2025-03-17 08:02:42 CET  4068615 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui
Mon 2025-03-17 08:03:43 CET  4074040 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui
Mon 2025-03-17 08:04:45 CET  4077727 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui
Mon 2025-03-17 08:05:46 CET  4080214 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui
Mon 2025-03-17 08:06:47 CET  4088606 1000 1000 SIGABRT missing  /opt/eset/eea/lib/egui

Well. It sure crashes a lot

root@L5:~# ls -1 /var/lib/systemd/coredump/*egui* | wc -l
1728

root@L5:~# du -chs /var/lib/systemd/coredump/*egui*
3.7G    total

Looking more closely to the core dumps, confirms that the error appear to be some locale related stuff on the main thread of the program egui.

root@L5:~# coredumpctl info /opt/eset/eea/lib/egui
       Message: Process 4067264 (egui) of user 1000 dumped core.

                Module libstdc++.so.6 from deb gcc-14-14.2.0-4ubuntu2~24.04.amd64
                Module libsystemd.so.0 from deb systemd-255.4-1ubuntu8.4.amd64
                Module libzstd.so.1 from deb libzstd-1.5.5+dfsg2-2build1.1.amd64
                Module libgcc_s.so.1 from deb gcc-14-14.2.0-4ubuntu2~24.04.amd64
                Module libprotobuf.so.32 without build-id.
                Module libcommon.so without build-id.
                Module egui without build-id.
                Stack trace of thread 4067269:
                #0  0x00007cac3e89eb1c __pthread_kill_implementation (libc.so.6 + 0x9eb1c)
                #1  0x00007cac3e84526e __GI_raise (libc.so.6 + 0x4526e)
                #2  0x00007cac3e8288ff __GI_abort (libc.so.6 + 0x288ff)
                #3  0x00007cac40563623 n/a (libcommon.so + 0x363623)
                #4  0x00007cac40a5223a _ZN10__cxxabiv111__terminateEPFvvE (libcommon.so + 0x85223a)
                #5  0x00007cac40a522a5 _ZSt9terminatev (libcommon.so + 0x8522a5)
                #6  0x00007cac40a523f7 __cxa_throw (libcommon.so + 0x8523f7)
                #7  0x00007cac40564f16 _ZSt21__throw_runtime_errorPKc (libcommon.so + 0x364f16)
                #8  0x00007cac40b07ce8 _ZNSt6locale5facet18_S_create_c_localeERP15__locale_structPKcS2_ (libcommon.so + 0x907ce8)
                #9  0x00005f5d61c8eff8 _ZNSt6locale5_ImplC2EPKcm (egui + 0x8eff8)
                #10 0x00005f5d61c902a6 _ZNSt6localeC1EPKc (egui + 0x902a6)
                #11 0x00005f5d61c87ac8 _ZN9AgentView26ConvertISO8601ToLocaleTimeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE (egui + 0x87ac8)
                #12 0x00005f5d61c885a3 _ZN9AgentView6UpdateERK17ConfigurationData (egui + 0x885a3)
                #13 0x00005f5d61c82d35 _ZN13WindowManager26ProcessConfigurationChangeERK17ConfigurationData (egui + 0x82d35)
                #14 0x00005f5d61c53fc0 _ZN16GuiThreadManager11ProcessTaskEv (egui + 0x53fc0)
                #15 0x00005f5d61c7f9be _ZN19GtkGuiThreadManager8OnWakeUpEPv (egui + 0x7f9be)
                #16 0x00007cac3f59d48e n/a (libglib-2.0.so.0 + 0x5d48e)
                #17 0x00007cac3f5fc717 n/a (libglib-2.0.so.0 + 0xbc717)
                #18 0x00007cac3f59ca53 g_main_context_iteration (libglib-2.0.so.0 + 0x5ca53)
                #19 0x00007cac3f7d388d g_application_run (libgio-2.0.so.0 + 0xe788d)
                #20 0x00005f5d61c80835 _ZN19GtkGuiThreadManager10RunGuiLoopEv (egui + 0x80835)
                #21 0x00005f5d61c4a749 n/a (egui + 0x4a749)
                #22 0x00005f5d61c90793 n/a (egui + 0x90793)
                #23 0x00007cac3e89ca94 start_thread (libc.so.6 + 0x9ca94)
                #24 0x00007cac3e929c3c __clone3 (libc.so.6 + 0x129c3c

I checked my locale settings, but it seemed fine

root@L5:~# locale
LANG=C.UTF-8
LANGUAGE=en_GB
LC_CTYPE=C.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_TIME=en_DK.UTF-8
LC_COLLATE=C.UTF-8
LC_MONETARY=en_DK.UTF-8
LC_MESSAGES=C.UTF-8
LC_PAPER=C.UTF-8
LC_NAME=en_DK.UTF-8
LC_ADDRESS=en_DK.UTF-8
LC_TELEPHONE=en_DK.UTF-8
LC_MEASUREMENT=en_DK.UTF-8
LC_IDENTIFICATION=en_DK.UTF-8
LC_ALL=

I then thought to check the environment of the ESET docker service.

root@L5:~# systemctl show eea.service --property=Environment
Environment=

I’m guessing here that the ESET eea.service starts very early and doesn’t get any LANG or LC_ALL passed?

Lets try a workaround this using systemd and poke the service configuration to specify those locale settings specifically for ESET

root@L5:~# systemctl status eea.service
● eea.service - ESET Endpoint Antivirus
     Loaded: loaded (/usr/lib/systemd/system/eea.service; enabled; preset: enabled)
     Active: active (running) since Tue 2025-04-22 07:38:41 CEST; 1h 5min ago

Patching /usr/lib/systemd/system/eea.service

[Unit]
Description=ESET Endpoint Antivirus
After=network.target
Before=eea_upgrade.service
Requires=eea_upgrade.service
ConditionPathExists=!/var/opt/eset/eea/updated/app/eea.bin

[Service]
Type=notify
+Environment=LC_ALL=C
+Environment=LANG=C
ExecStartPre=/opt/eset/eea/lib/install_scripts/check_start.sh
ExecStart=/opt/eset/eea/sbin/startd
ExecStartPost=-/opt/eset/eea/lib/install_scripts/launch_gui_all_users.sh
ExecStopPost=-/usr/bin/killall /opt/eset/eea/lib/egui --quiet
KillMode=process
Restart=always
TimeoutStartSec=180
TimeoutStopSec=120
OOMScoreAdjust=-800

Reload

systemctl daemon-reload
systemctl restart eea.service

Verifying it is set

root@L5:~# systemctl show eea.service --property=Environment
Environment=LANG=C.UTF-8 LC_ALL=C.UTF-8

Still crashes

root@L5:~# tail -f /var/log/syslog
2025-04-22T09:02:01.799953+02:00 L5 cron[1719]: (*system*eset-eea-vapm) RELOAD (/etc/cron.d/eset-eea-vapm)
2025-04-22T09:02:15.027347+02:00 L5 egui[25254]: Locale not supported by C library.#012#011Using the fallback 'C' locale.
2025-04-22T09:02:15.526396+02:00 L5 eset.eea[25254]: terminate called after throwing an instance of 'std::runtime_error'
2025-04-22T09:02:15.526550+02:00 L5 eset.eea[25254]:   what():  locale::facet::_S_create_c_locale name not valid

Hmm. What starts what?

root@L5:~# ps -eo pid,ppid,comm,cmd | grep eset
   1295       2 nvidia-modeset/ [nvidia-modeset/kthread_q]
   1296       2 nvidia-modeset/ [nvidia-modeset/deferred_close_kthread_q]
   2202       1 ERAAgent        /opt/eset/RemoteAdministrator/Agent/ERAAgent --daemon --pidfile /var/run/eraagent.pid
   2276    2250 Xorg            /usr/lib/xorg/Xorg -nolisten tcp -background none -seat seat0 vt2 -auth /run/sddm/xauth_CRQCla -noreset -displayfd 16
   4511    3845 egui_autorestar /bin/sh /opt/eset/eea/lib/install_scripts/egui_autorestart.sh --gapplication-service
  22570    3845 egui_autorestar /bin/sh /opt/eset/eea/lib/install_scripts/egui_autorestart.sh --gapplication-service
  23898       1 startd          /opt/eset/eea/sbin/startd
  23900   23898 logd            /opt/eset/eea/lib/logd
  23901   23898 scand           /opt/eset/eea/lib/scand
  23902   23898 sysinfod        /opt/eset/eea/lib/sysinfod
  23903   23898 updated         /opt/eset/eea/lib/updated
  23904   23898 licensed        /opt/eset/eea/lib/licensed
  23905   23898 schedd          /opt/eset/eea/lib/schedd
  23906   23898 utild           /opt/eset/eea/lib/utild
  23908   23898 confd           /opt/eset/eea/lib/confd
  23909   23898 econnd          /opt/eset/eea/lib/econnd
  23944    3845 egui_autorestar /bin/sh /opt/eset/eea/lib/install_scripts/egui_autorestart.sh --gapplication-service
  23958   23898 oaeventd        /opt/eset/eea/lib/oaeventd
  24000   23898 wapd            /opt/eset/eea/lib/wapd
  24016   23898 execd           /opt/eset/eea/lib/execd
  24036   22570 egui            /opt/eset/eea/lib/egui --gapplication-service

There is a ESET daemon called startd but it doesn’t appear to be parent of the egui related script – that I assume is egui_autorestart.sh.

Oh wait, that script is started from the service from before

root@L5:~# cat /usr/lib/systemd/system/eea.service
[Service]
Environment=LANG=C.UTF-8
Environment=LC_ALL=C.UTF-8
Type=notify
ExecStartPre=/opt/eset/eea/lib/install_scripts/check_start.sh
ExecStart=/opt/eset/eea/sbin/startd
ExecStartPost=-/opt/eset/eea/lib/install_scripts/launch_gui_all_users.sh

The ExecStartPost script is starting the crashing egui

root@L5:~# grep egui /opt/eset/eea/lib/install_scripts/*.sh
/opt/eset/eea/lib/install_scripts/egui_autorestart.sh:    /opt/eset/eea/lib/egui "$@" &

Why is that not inherting the systemd Environment settings? The egui_autorestart.sh is called from a script launch_gui_all_users.sh, so it probably launches after login. Could it be that the uses su for switching users, but forgetting to pass the environment?

Nope, it just calls /opt/eset/eea/lib/egui "$@" &

Hmm, who is actually starting egui_autorestart.sh then? Taking a closer look at the ps output, reveal it parent as pid 3845

root@L5:~# ps -eo pid,comm,cmd | grep 3845
   3845 systemd         /usr/lib/systemd/systemd --user

Enlightening. So the gui is started by the systemd user instance. I would have expected the system user instance to inherited the environment variables set in the service file, but that might not be the case?

❱ systemctl --user show-environment | grep -E '(LC_|LANG)'
LANG=en_GB.UTF-8
LANGUAGE=en_GB
LC_ADDRESS=en_DK.UTF-8
LC_IDENTIFICATION=en_DK.UTF-8
LC_MEASUREMENT=en_DK.UTF-8
LC_MONETARY=en_DK.UTF-8
LC_NAME=en_DK.UTF-8
LC_PAPER=da_DK.UTF-8
LC_TELEPHONE=en_DK.UTF-8
LC_TIME=en_SE.UTF-8
LC_COLLATE=C.UTF-8
LC_CTYPE=C.UTF-8
LC_MESSAGES=C.UTF-8
LC_NUMERIC=en_US.UTF-8

Strange output. Some values look fine, but some are missing and others have different values. Like, why do the environment have an seemingly arbitrary swedish time setting (LC_TIME=en_SE.UTF-8)?

My system setting has danish time setting

❱ localectl | grep LC_TIME
LC_TIME=en_DK.utf8

Appears this is because the systemd --user session do not inherit all environment variables (ref section 2.1 in the arch documentation). A bit strange though, that only LC_TIME has an apparently arbitrary setting.

The same Arch page suggests that environment.d can be used for fixing up environment variables like e.g. LC_TIME. environment.d is a systemd feature that lets you define persistent environment variables in simple files.

It reads files from:

• /etc/environment.d/
• /usr/lib/environment.d/
• ~/.config/environment.d/

I didn’t like this for several reasons:

• It’s quite hidden that an environment variable is overridden here
• Its on system or user level, while the problem is in a specific program

I decided instead to do a KISS solution and just patch the ESET script

/opt/eset/eea/lib/install_scripts/egui_autorestart.sh

+   export LANG=C.UTF-8
+   export LC_ALL=C.UTF-8
    /opt/eset/eea/lib/egui "$@" &

Then restart

root@L5:~# systemctl daemon-reload
root@L5:~# systemctl restart eea.service

root@L5:~# tail -f /var/log/syslog

2025-04-22T09:51:42.836121+02:00 L5 systemd[1]: eea_upgrade.service - ESET Endpoint Antivirus upgrade was skipped because of an unmet condition check (ConditionPathExists=/var/opt/eset/eea/updated/app/eea.bin).
2025-04-22T09:51:42.849737+02:00 L5 kernel: ESET Web access protection module 1.0 <www.eset.com> loaded
2025-04-22T09:51:43.243928+02:00 L5 systemd[3845]: Reloading requested from client PID 42298 ('systemctl')...
2025-04-22T09:51:43.244022+02:00 L5 systemd[3845]: Reloading...
2025-04-22T09:51:43.248730+02:00 L5 kernel: eset_rtp: registered syscall handlers
2025-04-22T09:51:43.248740+02:00 L5 kernel: ESET Real-time file system protection module 3.0 <www.eset.com>
2025-04-22T09:51:43.248742+02:00 L5 kernel: eset_rtp: scanner enabled
2025-04-22T09:51:43.467181+02:00 L5 systemd[3845]: Reloading finished in 223 ms.
2025-04-22T09:51:43.506111+02:00 L5 systemd[3845]: Starting eea-user-agent.service - ESET Endpoint Antivirus user monitoring service...
2025-04-22T09:51:43.945661+02:00 L5 systemd[3845]: Finished eea-user-agent.service - ESET Endpoint Antivirus user monitoring service.
2025-04-22T09:52:00.934259+02:00 L5 egui[43212]: Locale not supported by C library.#012#011Using the fallback 'C' locale.
2025-04-22T09:52:01.954114+02:00 L5 cron[1719]: (*system*eset-eea-vapm) RELOAD (/etc/cron.d/eset-eea-vapm)

And then suddenly the ESET gui popped up

The post ESET crashing with locale error appeared first on Monzool's Personal Publishing.

With no further ado, this is my new list:

• 20. 28 Days Later (2002)
• 19. Equilibrium (2002)
• 18. The City of Lost Children (1998)
• 17. Pitch Black (2000)
• 16. Chappie (2015)
• 15. Brazil (1985)
• 14. Children of Men (2006)
• 13. V for Vendetta (2005)
• 12. Elysium (2013)
• 11. RoboCop (1987)
• 10. Logan (2017)
• 9. Akira (1988)
• 8. Gattaca (1997)
• 7. District 9 (2009)
• 6. The Creator (2023)
• 5. The Terminator (1984)
• 4. Dark City (1998)
• 3. Blade Runner (1982)
• 2. Dredd (2012)
• 1. Alien (1979)

The Matrix went out. I think I just saw that movie too many times Otherwise the majority of the top of the list is probably unsurprising for most people (including myself).

Movies like Alien and Blade Runner are just timeless classics that is as good today, as they were then. Same goes for classics like The Terminator and RoboCop – although they perhaps show there age a bit more.

I placed Dredd at second best. I just absolutely love that movie. Its brutal, claustrophobic, have an bad-ass anti-hero and is absolutely dystopian.

It is obvious that most of the movies are rather old by now. I made the original list in 2007, and only six movies created since the first list, made it to the list! But its a list of all good movies, and it makes me want to see them all again. It’s been a really really long time since I saw Dark City, Gattaca and Brazil. I think I’ll start there – they are perfect for a dark and rainy winter night

The post My Favorite Dystopian Movies as of 2024 appeared first on Monzool's Personal Publishing.

Abstract

Chapters: Actions, calculations and data

First two chapters are about learning some basic design principals. Three fundamental categories are identified:

• Actions
• Calculations
• Data

In perhaps more common functional terms, actions are impure functions and calculations are pure functions. The reader is introduced to these concept gradually, by a guided tour of refactoring a (obviously worst-case) piece of imperative software. While this is done in iterations, the pros and cons are discussed.

The lesson is to learn how to eliminate implicit inputs and outputs, and how to convert actions into calculations

Chapter: Improving designs

Some fundamentals about the problems of global variables are discussed, and the reader is then guided through how the example code can be changed to avoid these.

The example in the book is a shopping cart being added with items and a total cost is calculated. The exercise of this chapter is to identify which functions have to many concerns and operate on both the cart and items and even also business logic. The agenda is to separate concerns and then discuss the implications of these refactorings.

Also discussed are how functions that do general things, should not have domain specific namings. The concrete example is a function to append an item to the cart, which essentially is making a copy of the array and returning it with the new item added last. That is refactored like this:

-function add_item(cart, item) {
+function add_element_last(array, elem) {

The point of this chapter is to realize, that only dedicated functions should know the integrate details of the data structures used. These function should then expose general functions. The rest of the code should then only have an opaque knowledge of the data, and instead use the general functions to manipulate it. At the end of the chapter, functions that did actions, calculations and data changing, now only do only of the things.

Chapter: Immutable in a mutable language

Immutable data has already been introduced, but in this chapter it goes full fletched on immutable data. Functions are identified by their read and/or access to data. Next it is explained how copy-on-write can be used to convert write functions to read functions.

function delete_handler(name) {
-   remove_item_by_name(shopping_cart, name);
+   shopping_cart = remove_item_by_name(shopping_cart, name);

As more and more functions become immutable, it is shown how the code can be optimized to only make copies when necessary

The example code in the book is in javascript, and is thus a language with mutable data structures. In javascript Array.shift is both a read and write function. An example is made to show how one can create immutable alternatives to the shift functionality.

Chapter: Handle untrusted code

An immutable codebase that has to interact with an mutable codebase, comes with a lot of potential issues. This chapter shows how to use defensive copying as a method to defend the immutable strategy. The chapter also discusses shallow copying versus deep copying.

Chapters: Stratified design

Stratified design is a technique for building software in layers. The book list four design patterns for a stratified design.

This first pattern is “Straightforward implementations” which is about splitting code up for higher cohesion and placing the extracted parts in relevant layers. There is an interesting technique used, where the internals of a function is visualized as a call-graph:

By evaluating the levels of abstraction in the call-graph, it is possible to see what to split out, as straightforward implementations should call functions from similar layers of abstraction. With this principal rule, the reader is guided though how to identify how functions access layers, and how to mitigate if the rule is not followed.

The end result is more general functions that allows more code reuse, and a call-graph that reduces “spaghetti code”.

The second pattern is “Abstraction barriers”. This is about creating an api of sorts, where the upper functions do not depend on implementation details of the lower layer. Exposing only opaque data types and general functions help to accomplish this. The concepts is especially useful for delivering functionality to a second development team.

The third, and penultimate, pattern is “Minimal interface”. The agenda here is to provide an abstraction barrier that is sufficient flexible for the layer above the barrier to add additional features from the building block provided by the abstraction.

Last pattern is “Comfortable layers”. This is a reminder that we also need to be practical. In production code the business need will often outweigh the timely efforts of making the perfectly stratified solution. Also highlighted in the chapter is the fact that lower levels carry higher risk. There are some observation about how to three functions in different layers. From that, three nonfunctional requirements rules are identified: Maintainability, Testability and Reusability.

Chapters: First-class

Here is explained how enumerations and functions can be used as arguments to reduce code duplication, and improve extensibility of functionality without causing great impacts.

It is also shown how higher-order functions can compose new functionality with great flexibility. The example is how to add a logging system to existing code, without having to plaster logging calls all over the code base.

Chapters: Functional tools

Functional tools like for-each, map, filter and reduce are explained along with the benefits they can bring to code. They are very well explained from an imperative standpoint.

Concepts as chaining functions are then covered before continuing on to update. When handling nested data, then instead of manually using get + modify + set an approach of using an update function is proposed.

An update function is created that eventually lead into explaining recursion. Also this is explained really well with clear explanations and informative illustrations.

Chapters: Timelines

The example code in this book, is a web client that put items in a cart. As the client has to communicate with the server for price calculation, it shows a plausible scenario where the timing can go wrong and incorrect totals are presented to customers. For visualizing this, this chapter teaches the reader how to visualize timings with timeline diagrams

The chapter also have some good insights on what can go wrong in different models like async, single-threading sync/async and multi-threading. The browsers job queue is also discussed along with ajax requests. A lot of good details are touched in these chapters.

Chapter: Reactive vs onion design

This chapter starts out by showing how to implement reactive updating (like React etc.). It has a nice story about the pros and cons to such a design and what kind of changes to the system design the reactive strategy entails in regards to decoupling and timelines.

The other design pattern onion is then discussed. The onion design leans into functional programming and strive to separate side-effects from pure functionality.

The signature idea about the onion design align well with the actions/calculation concepts discussed throughout the book:

• Interaction to external entities, is done only in the interaction layer.
• Layers call in toward the center.
• Layers don’t know about layers outside of themselves.

There are some great points to how common MVC design patters are not good for functional desires to isolate impure functions. This is what onion design tries to do – keep a clean separation between calculations and actions

Evaluation

This book is perfect for any typical imperative programmer that have heard all the hype of functional programming merits, but never really grokked how to begin doing actual functional programming. The book gently transforms a typical (albeit worst-case) imperative application to incorporate more and more functional programming functions and design patterns. It does this with clear, and to the point examples. To improve understanding, a lot of really well made informative diagrams, drawings are supplied. Perhaps the best trait of this book, are the many users-stories (from a developer viewpoint) that all helps to understand why the proposed changes improves the overall design. The target group for this book is definitely beginner programmers, or people that have no theoretical or practical experience with functional programming concepts; but if this fits your bill, you will for sure learn a bunch from this book.

Rating: ☆

The post Grokking Simplicity appeared first on Monzool's Personal Publishing.

SHELL SCRIPTING IS something most programmers will encounter often – especially if doing dev-ops, automation and general Linux work. Unfortunately we are mostly stuck with bash. There surely must be better alternatives…

I have been in search for a bash replacement before. My conclusions was that I would try out: scsh, luash and mruby. I’m addressing that experience in Addendum Take 1,

Since my last endeavor into finding a bash alternative, I’ve seen my work-life distance myself more and more from the small embedded devices, and move into higher level domains of JavaScript, Python and C#. In the environments where these run, the restrictions that apply to small embedded devices are not really applicable, and the options for scripting is much more open. Now I find myself back in the embedded world, and thus I figured it was time for another round of looking at bash alternatives. A lot of interesting stuff has happened in the programming language world these last years. That also include the scripting languages, where there seem to be a consensus that we might deserve more modern alternatives.

This post is available in two places

• On the monzool.net blog, for a pretty viewing.

• On github where the source code is also available. This post is the Take2 part of this series (direct preview link here: preview)

The approach

In this post I will not sprinkle code examples to exemplify how each and individual language look, feel and have of specialties. Nor will I do a deep dive into each of them. I therefore encourage visiting each individual homepage of each language to make you own opinion.

What I will do, is to actually try out the language, instead of just do assumptions about them. For each language I will implement the same sample, that do the following:

1. Change current directory to that of the script. With bash scripts I sometimes change the current directory, to be the directory of the script. It depends, if this improves the user experience, or makes it worse. When the user have to provide paths as program arguments, is a bad idea. If the script requires to know the location of a lot of external files, is makes thing easier…
2. Get program arguments from command-line. I rarely do fancy stuff with options, but is a feature I use often. Basic support is fine
3. Find files and get a list of found files. Searching for files is paramount – but resorting to plain old find is perfectly fine, if the output is easy to grab.
4. Split path, file name and file extension. This requires string manipulation.
5. Group files by extension. This is best solved with dictionaries.
6. Get a random number. Not an essential feature (the operating system I use, have /dev/urandom)
7. Search a file for some text string. This is probably my bread on butter use-case. It would actually benefit if this could be a call to grep (or rg) as these tools work perfectly for what they do.
8. Pretty print in colors. I am a heavy user of color printing. Its an easy way to bring focus on the essential parts.

A little introduction to this exercise. I first made a small sample script in bash. It takes to optional arguments. One that points to a directory of files, and another argument to possible disable color output

This is how the output looks in color

And with color suppression

The exercise is then to implement the equivalent in the other scripting languages, while learning the language is I go. I will be writing my thoughts and findings during the exercise, including any doubts, misunderstandings and stupid mistakes. I’m sure many will laugh of my struggles, but I think it will be a good exercise to get some indication on the difficulties learning the language for the average programmer. The eventual struggles I had, would probably also be the struggles of my colleagues. The friction of introducing a new language should be as little as possible.

Disclaimer: I will critique things here that someone or someones put their heart and soul into. This surely entail a risk that I unintentionally offend your favorite language, or even worse, a language that you created. Please be assured that I am not criticizing you personally or your work directly. In fact I am most likely just another idiot on the internet, ranting about stuff that I do not understand . I for sure do not own the abilities to create such exiting stuff, so I salute you for your efforts and work

Languages

I am searching for a scripting language, that

• Have proper arrays and dictionaries
• Have versatile return values
• Do not have “everything is a string” syndrome
• Can call external programs and grab values and exit code from it
• Is low on cruft and boilerplate, but still readable
• Do not use dynamic scoping
• Is production ready
• Is not abandonware
• Is an actual interpreted language, where scripts can be edited on a remote target

Some of the languages also provide a shell. For the intended purpose, this is not important and bares no positive or negative impact, as long as the language can be used with another shell.

Documentation and examples weight in hard as a requirement, as otherwise guessing and confusion would entail and make the experience frustrating and unproductive – both for the writer and reviewer/reader.

Below is a list of languages in the pool for trying out. Each is marked with the following icons to indicate the level of application

– have an implementation of the sample code
– attempted sample implementation
– no sample consdered
– not considered

Shells

• bash
• elvish
• murex
• ion
• nushell

Shell scripting languages

• hush
• koi
• abs
• ysh

Clojures

• Babashka
• Planck
• Joker

Lisps

• Fennel
• Janet

Rejected contenders

There are some really interesting languages out there that unfortunately do not fit within all parameters, regarding being applicable for a small embedded system.

• elk

  This is a tremendously interesting project. It looks very complete, versatile and pretty much check-marking all the boxes . There is just that issue that its a .NET 8 application and requires several hundreds of megabyte in memory and storage. There are some efforts in progress getting NativeAOT to cross-compile to arm32, but for now I do not consider it a viable contender

• dart with dlic

  dlic looks very interesting, and dart has a rumour of being very approachable. There is unfortunately a relatively high size and memory use if to be used as non-compiled scripts. Hank G have some interesting measurements indicating up to 100 MB of memory usage for the runtime. If abandoning script interpretation and settle for complied programs, the resource usage can approach about a tenth of that.

• gleam or elixir on AtomVM

  I’ve been making a few small projects in gleam. Its a interesting language and a very well driven project overall. The BEAM ecosystem is probably a bit on the bigger side. There is the AtomVM alternative, but it is unclear if gleam or elixir is able to run on that. Both are compiled languages, so they don’t really fit the agenda anyway.
  (P.S. consider doing like me, and sponsor the amazing gleam project).

• arsh

  Appears to be bash with types? The documentation is sadly pretty much non-existing.

Future contenders

For sure I have not depleted the realm of bash alternatives, so I am going to survey additional languages, either in this post or another. The addendum Future candidates has more details.

bash

Sample

The sample task is obvious to solve with a dictionary, where the key is the extension, and the values are a list of files that have that extension. Bash do have associative arrays, but do not support arrays of arrays. This puts on some limitations on how to construct the solution.

For each key (file extension) I serialize the list of files associated to that extension.

file_list["${extension}"]+=$'\n'"${filename}"

When printing, I then deserialize to get the filenames back in entity form.

deserialize_array() {
    local serialized="${1}"
    IFS=$'\n' read -r -d '' -a file_category <<< "${serialized}" || :
}

The implemented solution actually have a limitation, in that it does not support filenames with newlines, as \n is used as serializing separator (I did attempt to use \0 as that is not a valid character in filenames, but I could not get that to work).

For getting the file list, I’ve resorted to just use find. Now, bash could glob the files

for file in "${dir}"/**; do
    if [[ ! -f "${file}" ]]; then
        continue
    fi
    file_list+=( "${file}" )
done

but for this to work, then you need to remember to poke the shell options shopt -p globstar – and possibly remember to restore them again…. it gets complicated.

The sharp observer will notice that the IFS is temporarily poked for find files.

while IFS= read -r -d '' file; do
    file_list_all+=("${file}")
done < <(find "${dir}" -type f -print0)

Its a common technique prevent word splitting and pathname expansion… IFS is complicated

Parsing program arguments

Bash have getopts builtin for parsing program argument. There is also getpopt for some additional features. I rarely use ’em. They both kinda suck, are both complicated and I can never remember how they work. A simple loop of case matching, is far sufficient for most of my situations

while [[ $# -gt 0 ]]; do
    case "$1" in
        --list-dir)
            option_list_dir="${2}"
            shift 2
            ;;

Passing arguments to functions

Passing arguments to bash functions are kind of old school. There are no parameter list, instead arguments are passed in to be referenced by a numeric index variable. First argument is ${1}, second argument is ${2} and so on. I’m used to it, but its kind of odd, come to think of it. There are various gotcha’s when having values with spaces, but otherwise it works as expected… Except for associative arrays – then it gets complicated. Therefor I resorted to pass them as references

present_results() {
    declare -n extension_count_ref="${2}"
    declare -n file_list_ref="${3}"

It might come as a surprise that bash supports this kind of indirection, but bash is fully dynamic and quite powerful in that domain; but yeah, it gets complicated pretty fast. The terse syntax also does not make it easier to read later on

Returning values from functions

Why even mention such a trivial matter as returning values from? Well in bash its… limited. We either return integer values or echo strings. Using echo to return strings, makes other printing complicated which, among other things, can make debugging hard. The basic string return, do let you set your own rules when in a script. It is for example possible to fake tuple like return values

get_tuple() {
    echo "Year|2024"
}

IFS='|' read -r message value <<< "$(get_tuple)"

But using such tricks, quite quickly make things get… complicated. Being bash there’s lots of hacks to return other ways, but most have some footgun or another… it gets complicated.

I do use the above mentioned trick sometimes, but over time I’ve mostly settled on a simpler principle: return zero or non-zero if to indicate success or fail, and if returning a value, put it in a global variable. This might sound chocking, but in practice it works quite fine.

return_value=
get_return_value() {
    return_value="magic"
}

It do have its limitation in some corner cases (its a global variable after all), but its not much worse that bash’s dynamic scoping anyway. What it certainly does is reducing complexity instead of loops and tricks and hacks to simulate proper return values. One key principal to limit the “pollution” of global variables, is then to never use the variable directly, except as arguments to other functions

use_return_value() {
    local value="${1}"
    echo "${value}"
}

use_return_value "${return_value}"

Theres a “How I Learned to Stop Worrying and Love the global return value” blog post material lingering there. I’ll probably never get there, so just keep calm and carry on

On an interesting note, then its possible to simulate named parameters in bash. However, I can testify its a submachine gun of a footgun, so I never do that anymore Speaking of footguns…

Footguns

What does this snippet print?

get() {
    return false
}
get && echo success || echo fail

return: false: numeric argument required

Below is how it is actually supposed to be:

get() {
    false
}
get && echo success || echo fail

That’s just one footgun that I got hit by in the sample script.

In an early incarnation of the sample script, this was how the line to increment the extension count looked like

(( extension_count["${extension}"]++ ))

At first glance this seems fine, but in fact it is not. It can be very puzzling, as the script will exit on that line. No error, just exit.

Inspecting the exit code, gives some indication

❱ echo $?
1

What happens is that lookup returns 0, then incrementing causes the overall expression to return non-zero, which then triggers errexit

Lets just bypass errexit and override the non-zero with a NOP

(( extension_count["${extension}"]++ )) || :

Similar issue is the reason why the deserializer looks like this

deserialize_array() {
    local serialized="${1}"
    IFS=$'\n' read -r -d '' -a deserialized <<< "${serialized}" || :
}

In this configuration, read returns non-zero on EOF – which we will inevitable reach

The unofficial bash “safe” mode is not without its issues

elvish

elvish

tl;dr

Elvish is an amazing effort that just continues to improve, commit after commit, year after year. Unfortunately I decided to cut the sample short before completing it. I would have loved to say the documentation was awesome, but I spent way too much time on every single thing that I wanted to do.

vscode plugin: https://marketplace.visualstudio.com/items?itemName=elves.elvish

Shell:
Approachable for a bash’er: ☆☆☆☆
Enjoyment: ☆☆☆☆

Returning values from functions

I browsed the entire learnsection with no hint of how to return something from a function. I had to visit the reference to find an example that used echo to return a string.

In fact, the first thing that looks like a function was this example:

var hosts = [[&name=a &cmd='apt update']
             [&name=b &cmd='pacman -Syu']]
peach {|h| ssh root@$h[name] $h[cmd] } $hosts

The explanation was clear enough, so rolled my own edition

say {|message| put $message} "Hello, world!"

But got this back

Exception: exec: "say": executable file not found in $PATH

Oh. Not a function definition, but instead a command. Lots of reading later I was able to put this together

fn say {|message| put $message }
say "Hello, world!"
▶ 'Hello, world!'

As mentioned, a frustrating lot of reading was then required to figure out, how to make a function return stuff. Eventually I tumbled upon a tiny example that used put to return a value. It might be that I had some preconceived ideas on things, but I didn’t expect elvish to rely on capturing printed stuff as return values.

Finding the put documentation was not easy, but eventually I found a mention here.

which then lead to the actual documentation

put – Takes arbitrary arguments and write them to the structured stdout.

Interesting with the structured stdout, because printf outputs to something else

printf – Prints values to the byte stream according to a template.

So from deduction, I could reason that elvish in fact use put for returning values from functions, but have different “channels” of returning them

After quite some searching, I found an example of how to capture the return values. It appears to work, and even for lists

fn get-url {
    put ["monzool" "." "net"]
}

var url = (get-url)
echo $url

▶ [monzool . net]

That naturally made me curious. What if you have more that one put or printf?

fn get-url {
    printf "Debug: calling get-url"
    put ["monzool" "." "net"]
}

var url = (get-url)

This gives a compile error:

Exception: arity mismatch: assignment right-hand-side must be 1 value, but is 2 values

Sweet

I then tested echo, printf and put. None print directly to stdout, but are all captured, with the compiler forcing a receiving variable for all.

sample.elv

fn get-values {
    echo "first"
    printf "second"
    put "third"
}

var first second third = (get-values)
printf "first = %s\n" $first
printf "second = %s\n" $second
printf "third = %s\n" $third

The above sample lead to much confusion. On first run I thought I had the order wrong.

❱ ./sample.elv
first = third
second = second
third = first

Running it a few more times showed that, that was not the issue. Its arbitrary which variable receives which value from the function

❱ ./sample.elv
first = first
second = second
third = third

❱ ./sample.elv
first = second
second = third
third = first

First thought was that tings were running in parallel, but it’s actually is a documented “feature”

If the chunk outputs both values and bytes, the values of output capture will contain both value outputs and lines. However, the ordering between value output and byte output might not agree with the order in which they happened

Sure enough, switching to same print function, make the return values predictable

fn get-values {
    put "first"
    put "second"
    put "third"
}

Interesting footgun

I sort of got fatigued here. At first glance, the documentation looks very elaborate, but I found it very very difficult to get started with just the basic things. I even (desperately) tried to find some inspiration in sample code or unit-tests in the elvish repo. The few things I found was primarily in the implementation language Go.

Error handling

The thing that turns me interest away from elvish, is its use of exceptions

[snip] non-zero exits from external commands are also turned into exceptions:

IMO exceptions should be for the exceptional case. An external command returning non-zero is not an exceptional case – it might not even be an error situation

grep ERROR run_with_no_error.log

I suspect I’ll be required to add a lot of try/catch in most scripts then. One might argue that this is much like when setting set -e in bash. That flag will cause script termination of first non-zero exit code. It also requires using a lot of tricks false || : to circumvent for a line – but that is also not what it is intended for. The flag is more a special operation mode, where only success is expected, and failure should stop all processing immediately.

A situation like “command not found” would probably be a situation I could live with, being an exception. Instead of general exception use, I would have preferred a result type, that I would then have to inspect if I cared about the exit code. Alternatively, do like murex and make it up to the user if an exit code or an exception is desired

murex

murex

tl;dr

I stopped the murex sample short. I ended up spending way to much time getting no where. The documentation do have some minimal usage examples for most components, but I found it difficult to figure out how to combine the individual pieces. The nushell which shares much spirit with murex, have better examples that made that language more approachable

Shell:
Approachable for a bash’er: ☆☆☆☆
Enjoyment: ☆☆☆☆

Special features

Murex has an pretty neat features backed in. One is events. This allows to do stuff on e.g. file system changes

event onFileSystemChange example=. {
    -> set event
    if { $event.Interrupt.Operation =~ "create" } then {
        git add $event.Interrupt.Path
    }
}

(source: https://murex.rocks/events/onfilesystemchange.html)

It also have baked-in unit-testing, pretty json printing, parsing of json, yaml and toml, and many other nice features Sadly it also have some complications build in. The description of runmode looks like an advanced topic. Hoping its for special corner cases

Error handling

Error handling in murex is dual paradigm. You either choose to get success/fail, or you opt in to get an exception on failure. This is a brilliant idea, and makes so much sense.

For most situations you really on care if a specific command succeeds or not

if { code-block } then {
    # true
} else {
    # false
}

(source: https://murex.rocks/commands/if.html)

If you care about the exit-code, murex have a similar concepts as bash, by providing an exitnum variable.

Other times, you just need to run a batch of commands, and if any of them fails, then just bail.

try {
    out "Hello, World!" -> grep: "non-existent string"
    out "This command will be ignored"
}
catch {
    out "An error was caught"
}

(source: https://murex.rocks/commands/catch.html)

Command names

Question. What does this do?

if { g /dev/null } then {
    out "true"
}

Apparently g is a globbing command. The above example check if the file exists. What could the f command do? That would be the file object filter function. Other examples are ja, ta pt and mtac… I guess when you know them, the brevity is nice. It also follow the old-hat unix tradition of short abbreviated names, but I am a little worried that too many abbreviation makes writable code, not readable code

Returning values from functions

Just like with the elvish documentation, it is frustratingly difficult to find any documentation on how to return values from a function. There is a tiny indirect example, in the return documentation, which empathizes on exitnum… What I deduced is that murex also uses stdout for returning data from functions. I then looked at the out documentation, which then suggests using tout, cast and format for data-type return vales. All examples shows json or yaml as return structures. So if wanting something other that text returns, one has to use json, yaml or toml? Some digging around, I later found a data-type reference. It contains a list of builtin datatypes like bool, int, string, the mentioned structured data and more. The murex repo, actually have some examples. Unfortunately none of the examples showed how to use a function… but at least they showed that murex files are suffixed with a ‘.mx’ type name . Eventually I found an example in the code block parsing section… although that did not show how to pick up the return value

Sample

Listing files

I found no immediate candidates for file listing tools like find. Instead murex provide a globbing tool to do filesystem wildcards. I found an example in the rosetta stone page.

❱ f +d | foreach $dir { out $i }

Error in `out` (1,14): variable 'i' does not exist
                     > Expression: out $i
                     >           :      ^
                     > Character : 5
Error in `out` (1,14): variable 'i' does not exist
                     > Expression: out $i
                     >           :      ^
                     > Character : 5
Error in `out` (1,14): variable 'i' does not exist
                     > Expression: out $i
                     >           :      ^
                     > Character : 5

Well, it sure is eager to tell you where you went wrong But the error message is actually quite informative, so it was easy to fix the official example

f +d | foreach $dir { out $dir }
.git
test

Where f is filtering, g is for doing the globbing, but it appears that there is no way to recursively list all sub-directories?

❱ g test/**
[
    "test/ untrimmed .doc",
    "test/A filename with spaces.doc",
    "test/monzool.net",
    "test/subdir",
    "test/ᛗᛟᚾᛉᛟᛟᛚ.net",
    "test/🄼🄾🄽🅉🄾🄾🄻.net"
]

❱ g test/**/*
[
    "test/subdir/1.dat",
    "test/subdir/2.dat",
    "test/subdir/3.dat"
]

I guess its possible to build a command from the building blocks provided, but using the external find command is quite fine

Function arguments

One would think calling a function with some arguments would be simple. None the less, then I can’t make it work. The function documentation has zero examples of how to do this

Objective: call this function with a path

function get_file_list (dir: path) {
    file_list = ${ find $dir -type f }
    out $file_list
}

Like bash?

get_file_list "test"

No

unexpected closing bracket '}'
Expression: }  get_file_list "test"
          :  ^
Character : 1
Error in `./sample.mx` (0,1): exit status 1

Use parentheses?

get_file_list ("test")

No (single quoting gave same result)

find: "test": No such file or directory
Error in `find` (./sample.mx 1,102):
      Command: find $dir -type f 
      Error: exit status 1

Placing the value in a variable before calling?

set list_dir = 'test'
get_file_list ($list_dir)

Yes

Calling with no space between function and argument? …

get_file_list('test')

Yes

Split a filename

I need to split the file extension and name from a file path. In great frustration I can cannot locate any documentation on how to split a string, do substrings, match on things or anything that sounds remotely like something I need.

Could not find a native way to do the splitting, so resorted to use sed

Data types

How do I create a a variable of a type? I first went to Data Types section, but that instead appears to be describing types that murex can use to interchange data between functions? Okay, found some enlightenment in the section of Operators and Tokens. That section lists these options:

• %[] Create Array: Quickly generate arrays
• %{} Create Map: Quickly generate objects and maps

(are there slow editions of these then? )

Maps

I cannot figure out how to add to a map. There are documented examples where a map magically has manifested and then you do formap on the data in it – but how do you add to an existing map?

Lets create a map m

❱ m = %{ "net": ["a", "b"]}

Use a key and value to add an entry

❱ m["txt"] = ["c"]

No

unexpected symbol '[' (91)
Expression: m1["txt"] = ["c"]
          :   ^
Character : 2

append is only for arrays, but lets try

❱ $m | append %{ "txt" : ["c"]}

No

[
    "{\"net\":[\"a\",\"b\"]}",
    "{\"txt\":[\"c\"]}"
]

Using map gives me some kind of mutant result

❱ map { $m1 } { %{ "txt": ["c"] } }

No

{
    "net: [\"a\",\"b\"]": "txt: [\"c\"]"
}

Oh well. I decided to throw in the towel here, and abandon the murex sample

Hush

hush

tl;dr

Using hush was an absolute pleasure. The language is simple, but I consider that a good thing. It does everything that is needed for scripting. There is perhaps a few lacking features in the standard library, but overall it gets the job done. A stellar feature of hush is command blocks, which allow to run shell commands with capturing output and errors

Documentation is great with complete and precise examples. I had no real issues or frustrations, implementing the sample in hush. Interestingly the bash and hush samples are almost identical, the Hush edition just had much less friction

vscode: https://marketplace.visualstudio.com/items?itemName=hush-vscode.hush

Approachable for a bash’er:
Enjoyment: ☆

About hush

Sadly hush appears to be borderline abandonware

❱ git log --date=format:'%Y' --pretty=format:'%ad' | sort | uniq -c | awk '{print $2 ": " $1}'
2020: 4
2021: 211
2022: 75
2023: 6
2024: 3

Installation

I installed from cargo cargo install hush, but when parsing arguments using regex I was blocked on this error

error: "invalid regex" ("regex parse error:\n    --(\\w+)=?(\\w+)?\n       ^^\nerror: Unicode-aware Perl class not found (make sure the unicode-perl feature is enabled)")

That matches an issue on the hush issue tracker, with a fix merged in May 2023. I then looked at the crate and it was last updated in May 14th of 2022. So I cloned the source and build and installed with cargo.

The regex now worked; however, now all my std.print would no longer end with printing a newline, making everything print on one long line. The documentation on the standard library did not mention std.println… but it worked. That feature was added May 24th of 2022. I got a bit curious here, on how much had changed in the last two years. In the issues closed section there is only a limited amount of stuff added. Primarily std.println and elseif. In the open issues section there was not a whole lot either. There was this thing though: hush [cannot do filesystem queries] yet. Reported in May 2022, is that hush lacks features like checking if a file exist, if a file is a directory etc.

From the issue responses its clear that the author do not have time to maintain the project – but its not abandonware. Just recently, a pull request was merged. Anyway, the missing file operations is not a deal breaker. Hush have command blocks where we can execute shell code.

Conditions

As far as I understand, hush only accepts single bool expressions as conditions. This means that this is not possible

if value == 1 or value == 42 then

This could a bit lacking if comparing range values, but for most cases this is an okay limitation.

Command blocks

I mentioned that hush do not have any file I/O built-in, but command blocks can be used as a fallback. The ability to just “switch” a section to shell scripting is incredible flexible. Got that magic bash one-liner that generates rainbows and unicorns ? Just put it in a command block and enjoy

function get_file_list(dir)
    let file_list = ${
        find "${dir}" -type f -print0
    }.stdout

    return std.split(file_list, "\0")
end

If did found one thing I could not get to work with command blocks, and that was to access program arguments. Hush will crash if trying to access any bash $ variables. I discovered this, when having failed to find a native way to get the script directory.

let script_dir = { dirname "$0" }
std.cd(script_dir)

This crashes hush with a rust error. I find it not to be an unreasonable limitation. Other convenient bash features like arithmetic expansion $(( expression )) is not supported either in command blocks.

Dictionaries

Dictionaries access is a bit inconsistent on the keys. Initialization do not accept string type as keys, so any key with space or dash is not possible

let dict = @[
    "key 1": nil,
    "key-2": nil
]

Both key/value pairs above give compile errors.

Error: (line 9, column 4) - unexpected '"key 1"', expected identifier
Error: (line 10, column 4) - unexpected '"key-2"', expected identifier

However assign and access is possible with both spaces and dashes

dict["key 1"] = "space"
dict["key-2"] = "dash"

Speaking of dictionaries, how do you remove an entry from it? Don’t think its possible. Setting as nil value is only option, as far as I could find.

Program arguments

Hush can give you a list of arguments with std.args, but otherwise it offers no argument parsing features. I then set in motion to do a generic argument parser with validation and value extraction. It was fun and hush was quite capable. Then it dawned on me, that this was a false approach. If hush have had a parsing library, I would have used that, but I didn’t implement an argument parser in bash, so I shouldn’t do that in any of the other languages. That would also make them look unfair larger that the bash edition. When I scrapped the parser, I found that handling only the specific arguments, made the hush code look very similar to the original bash code.

Printing

Color printing using ansi codes did not work. I could not get std.print to handle escape sequences for setting colors, so I had to fallback to command blocks to make this work. That lead to another issue. Mixing the native std.print and echo/printf in command blocks do not work well together, due to output stream flushing issues. That is perhaps not unexpected.

Error handling

Printing dictionaries gives a pretty good idea on what is in it, however when printing an error, it will give output like this

error: "command returned non-zero" (@[ "stderr": "", "stdout": "", "error": @[ "pos": "\u{1b}[38;5;2m./sample.hsh\u{1b}[39m (line 109, column 8)", "status": 1 ] ])

There are actually two hidden keys in the dictionary. The value “command returned non-zero” , which is a textual message of the error, is in a key description. The rest of the printed content is in a key context

if (std.has_error(search_result)) then
    std.println("Description: " ++ search_result.description)
    std.println("Status: " ++ std.to_string(search_result.context.error.status)))

It is documented, just not very obvious

Compiler

Compile errors was pretty good and its reasonable easy to find the culprits from its messages. A lot better that bash

if std.contains(arg, "--list-dir=") then

Panic in ./sample.hsh (line 28, column 23): value ("--list-dir=") has unexpected type, expected character

Sample

I didn’t make any notes during development of the Hush sample, as there wasn’t really any issues to write about

I only had these obstacles during the process 1) I could not find a way to change the current directory to that of the script 2) I could not find a easy way to do generate random numbers 3) Ansi escape codes do not work for setting color. Had to use tput 4) How to extract error context values is not obvious

Koi

koi

tl;dr

Writing Koi code was pretty much just like writing Hush. It was quite straight forward to write the sample as the language is simple and intuitive. The simple interop with external commands is excellent. There are sadly three significant pain points, that will make me not pick up Koi

1. Program options are not supported
2. On coding mistakes, the compiler generally prints a completely generic message
3. Appears abandonware

Documentation was fine, but for the most part implicit from examples

Approachable for a bash’er: ☆
Enjoyment: ☆☆

About Koi

Where most other script languages use $ for referencing variables ($var), in Koi variables are just referenced by there name (var). For string interpolation variables are referenced in {} ({var}).

Koi also takes a more object oriented approach where variables have methods.

print('koi'.upper().split('o'))

Sadly it appears to be abandonware

❱ git log --date=format:'%Y' --pretty=format:'%ad' | sort | uniq -c | awk '{print $2 ": " $1}'
2020: 134
2021: 181
2022: 11
2023: 1

Sample

No script arguments

It is not documented, and I found no way to do it… Koi scripts cannot receive program arguments

Instead of the script receiving arguments, the Koi interpreter itself would hijack the arguments

❱ ./sample.koi --list-dir=test
error: Found argument '--list-dir' which wasn't expected, or isn't valid in this context

USAGE:
    koi [FLAGS] [OPTIONS] [PATH]

For more information try --help

That is somewhat of a major bummer

No errors

Made a mistake, and referenced an non-existing variable

fn get_file_list(dir) {
    let files = $(
        find {dir} -type f -print0
    ).split('\0')
-   return file_list
+   return files
}

This gives no indication of error

Get list of files

Koi will not split on \0 so had to change the find command to separate by newlines

fn get_file_list(dir) {
    let files = $(
-       find {dir} -type f -print0
+       find {dir} -type f
-   ).strip().split('\0')
+   ).strip().split('\n')
    return files
}

Append to array

How to append values to an array is not documented. Figured out this would work

let a = [1]
a += [2]

#=> [1, 2]

Color printing

I could not get any color printing to work with Koi’s built in print function. Neither would printf or echo work with ansi codes, however just like Hush, tput would work

Error messages

As mentioned, Koi do not always detect mistakes, but when it does, the errors messages are lacking detail

thread 'main' panicked at src/parser/stmt.rs:163:13:
expected right brace
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

(adding RUST_BACKTRACE=1 only reveal details of Koi internals, and nothing of the script generating the error)

This is the extend of message from pretty much any mistake you can make in Koi (non-existing variable or function, syntax mistake etc). This makes mistakes pretty hard to locate

fn blue() { tput setaf 4 }

thread 'main' panicked at src/parser/stmt.rs:61:30:
only assignment, call and command expressions are allowed as statements
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This is not helpful – especially when you have added many lines since last run.

Btw the fix is this

fn blue() {
    tput setaf 4
}

abs

abs

tl;dr

Creating the sample was very easy. As I made the Koi sample before this, the abs edition was more or less a copy paste. Compared to Koi, the abs language have a few more features and makes for a more complete language. The printing was the weakest point of abs, but otherwise the experience was very good. It’s a shame it appears to be abandonware

vscode: https://marketplace.visualstudio.com/items?itemName=abs-lang.vscode-abs

Repl:
Approachable for a bash’er: ☆
Enjoyment: ☆

Returning values from functions

Let me praise abs here. It actually documents, in plain sight, how to return values from functions

Sample

Program arguments

abs have a nice cli library for making it easier to make cli tools that take arguments. Unfortunately it seems to be in the format script <command> [option] only. I need script [option]. For convenience, I decided to use it anyway as ./sample.abs run [option]

Listing files

Like most of the other languages (bash and hush being the exceptions), abs would not split on null terminations

-files = `find $dir -type f -print0`.split('\0')
+files = `find $dir -type f`.lines()

The lines was convenient. Then I also didn’t have to pop the last empty item from a split

Printing without newline

I cannot make echo not add a newline. It seems not to be a feature. This is especially annoying when having to print background colors… Discovered in https://github.com/abs-lang/abs/issues/374 that I could see that echo has more options, but the non-newline printing appears to be still on the wish list

Color printing

I cannot find any way to make color printing work properly

echo with ansi codes just print the uninterpreted text

echo("$prefix \e[34m $text \e[0m")

An all bash command do the same

`printf "$prefix \e[34m %s \e[0m" "$text"`

What sorta work is using tput, however as I could not find any way to print without newline, lines with background color will color the background until next line could reset the color. Mixing bash printf and abs echo do not (unsurprising) go well, so had to settle for fully colored lines or no colors at all

Error messages

I accidentally did .[ instead of [ and got a parsing fail stack back

if categorized_files.[extension] == null {

 parser errors:
        expected next token to be [, got IDENT instead
        [46:30]         if categorized_files.[extension] == null {
        no prefix parse function for ']' found
        [46:40]         if categorized_files.[extension] == null {
        no prefix parse function for '==' found
        [46:42]         if categorized_files.[extension] == null {
        expected next token to be ), got IDENT instead

I guess the error message could have been more precise, but it was perfectly easy to figure out what the mistake was. So served its purpose

However more than once I ended up with something like this, which could be somewhat hard to locate

bash: -c: line 1: unexpected EOF while looking for matching ``'

nushell

nushell

tl;dr

Diving into nushell was quite a thrilling and enjoyable experience. Nushell is quite different from bash, and I learned a lot from my small encounter.

Documentation is very good. There is an official book Thinking in Nushell, cookbook a great command overview and a language reference guide. What sets nushell apart from perhaps elvis and murex, is that nushell appears relatively broadly adopted, and it is possible to find a reasonable amount of blog posts and articles to get one started or find inspiration.

Nushell can be a little perplexing initially. When coming from bash, nushell can do so very much more. One thing that was especially mystifying to me, was how the data exchange between commands worked. Then found the commands section in the contributor book. Commands communicate using a data structure PipelineData. That really made it click for how nushell can operate as it do.

There is lots to learn when switching from bash to nushell, but I think nushell is tremendously powerful when it comes to data manipulation. I see a lot of of situations where my toolbox of grep, sed, awk and other old-hat tools could be replaced by nushell.

vscode language plugin: https://marketplace.visualstudio.com/items?itemName=TheNuProjectContributors.vscode-nushell-lang

Shell:
Approachable for a bash’er: ☆☆☆☆
Enjoyment: ☆

Function return values

“Funny” enough, just like for the other shell-first languages elvish, murex and ion, I found no documentation on how to return values from a nushell function. There is documented something about function [type signatures](https://www.nushell.sh/book/command_signature.html that also covers function return types. Actually functions in nushell are called commands and appear more to resemble, calling a subscript in bash. It seems that commands stream output (to stdout and/or stderr) and can then be picked up by the next command. Like murex (and perhaps also elvish?) this steam can be formatted to different types

I think what makes these “shell first, scripting second” projects a little difficult to consume from a scripting angle, is that most commands are documented like they would be used in a shell.

One example is the complete command. I stumbled upon complete when reading about capturing exit-codes.

> cat unknown.txt | complete
╭───────────┬─────────────────────────────────────────────╮
│ stdout    │                                             │
│ stderr    │ cat: unknown.txt: No such file or directory │
│ exit_code │ 1                                           │
╰───────────┴─────────────────────────────────────────────╯

It wasn’t immediate clear to me how that transitioned to be used in a script, but I was lucky to find an example on an external site, a blogpost by Jiri Pospisil:

let response = (curl --fail -s $url | complete)

match $response.exit_code {
  0 => {
    $response.stdout | jq ...
  },
  22 => {
    print -e $"Project \"($name)\" not found or server error!"
    exit 1
  },
  - => {
    print -e "It's all broken."
    exit 1
  }
}

This example shows how to:

• call an external program
• capture the response
• evaluate the exit code

Documentation for nushell is otherwise quite good. Nushell have an incredible amount of features, but it is very well presented. Nushell even support colors

The nushell introduction have this quote, which I think it true. Nushell is very different from bash

Thinking in Nushell: If you’re used to using mutable variables for different tasks, it will take some time to learn how to do each task in a more functional style.

Sample

Types

I wondered why I got a type error when attempting to filter file_list. Found the describe function for revealing the underlying type

let file_list = (ls "./test" | where type == file)
$file_list | describe

=> table<name: string, type: string, size: filesize, modified: date>

Oh. I was attempting to stream a table into something that expected a list. So how to convert the table to a list?

This is one way to do it

let file_paths = $file_list | get name
$file_paths | group-by { path parse | get extension }

Getting a list of files

I am not getting files from sub-directories from my ls ./test. Nushell find is something else, so it is not really clear how to find a three of files? It required a visit to stack-overflow to find out that globbing can be used. ls ./test/**/*. I kind of like it

Update: found the documentation in the bash comparison sheet

Iterating a record

I have a record of the file categories. I cannot figure out how to extract key value pairs from this

let file_categories = $file_list | group-by { path parse | get extension }
$file_categories | describe

#=> record<doc: list<string>, net: list<string>, dat: list<string>>

Lots of reading later: records should be transposed to tables, before iterating them.

Functions

Functions in nushell are called commands.

nushell is a typed language, so function parameters can optionally be annotated with types. For the printing functions I want to pass in another function to do color handling. From the parameter types list I guess closure is the best fit. I could however not find any documentation on that type directly.

I am a little confused on how commands work. This example do not act like I would expect

def print_category [extension: string, filenames: list] {
    echo "Category: ($extension)"
}
print_category "dat" ["1", "2"]

#=> Category: ($extension)

The parameter value is not expanding? I noticed the command examples all use string interpolation.

def greet [name] {
  $"hello ($name)"
}
greet nushell

#=> hello nushell

Using echo or print appears not to work in commands, while interpolation do?

def print_category [extension: string, filenames: list] {
    print "Category: ($extension)"
    $"Category: ($extension)"
}
print_category "dat" ["1", "2"]

#=> Category: ($extension)
    Category: dat

print do not return anything, but echo acts more like an identity function, so I would have thought echo would have worked?

Aha! This worked

```nushell
def print_category [extension: string, filenames: list] {
-   print "Category: ($extension)"
+   $extension | print
    $"Category: ($extension)"
}

Ohh. Maybe I understood nothing in regard to printing. This also works

print $"Category: ($extension)"

At this point, I am not entirely sure how things work. Its nothing like anything that I am familiar with. Learning nushell feels a lot like being educated

Filter to filename only

From the categorizing, I got files with full path and extension, but should on have filename for each extension

-let file_categories = $file_list | group-by { path parse | get extension }
+let file_categories = $file_list | path basename | group-by { path parse | get extension }

This makes data change from (e.g.) path/filename.ext to filename.ext Now how do I cut the extension from the data, but still be able to do grouping by the same cut data?

Opted to just split it in two operations. First group by extension, and then get stem when needing the filename only.

Passing in a printing function

Was okay easy to find out that do is required, just by browsing the command list

Random value

Nushell have native support for several random number generators. The integer random generator fit perfectly here. Only this is that the examples are all inclusive ranges

random int 1..10

First attempt was just to do random int 1..(10-1), but that gave syntax error. The command documentation of range is quite sparse. After some digging around, I found a chapter in the nushell book about ranges which offered some details

random int 1..<10

Thats actually amazing

I also figured out that my attempt of subtracting (random int 1..(10-1)) actually was possible if using proper syntax random int 1..(10 - 1). That was a me error, not nushell

Handling grep errors

It was relatively easy to find out how to handle errors from called external programs. Seems that I have two options.

Use exceptions

try { 
    grep $text $filename
    true
} catch { 
    false 
}

Or, better to my preference, check the exit code

let result = grep $text $filename | complete
return ($result.exit_code == 0)

That could then be rewritten to

(grep $text $filename | complete | get exit_code) == 0

Program arguments

Took me a while to figure out how to get arguments. It wasn’t until this bug report, that I realized you just make a main function. You then combine that with command flags.

Update: scripts with main is documented here

Really weird. I cannot get main to work. It is never called. I have put this at the top of my sample file, but it never prints

#!/usr/bin/env nu

def main [input: string] {
    $"Ere I am, J.H." | print
    print $input
}

The second weird thing is, that if I put the same code in another file, then it works fine!

Figured it out, when I put a print on the top level also

#!/usr/bin/env nu

def main [input: string] {
    $"Ere I am, J.H." | print
    print $input
}

"He's got away from us, Jack" | print

When run, it gives this output

❱ ./brazil.nu "And this is my receipt for your receipt"

He's got away from us, Jack
Ere I am, J.H.
And this is my receipt for your receipt

So it runs all top level code before running main. That explain why it looked like it never worked in my sample code. To test out main and not needing to comment out all code, I just put a return after main.

def main [] {
}
return

Hence, the first thing the script will do is to return (and then exit)

Builtin usage help

It is very clever how the command arguments can auto-generate a help message. For nushell I then did not have to do this myself – its baked in

./sample.nu -h
Usage:
  > main {flags} 

Flags:
  --list-dir <String> - Directory to list (default: 'test')
  --no-color - Disable color output. (default: false)
  -h, --help - Display the help message for this command

Though its a shame that it is listing the command name, and not the script name

Usage:
  > main {flags} 

This should have said sample.nu instead, as main make no sense outside the script. I found no way to change this

I also wished that the description texts were aligned… but thats a nitpick

Getting a list of files, take 2

After getting program arguments working, I wanted to pass in the user specified path to ls instead of the hardcoded value.

-let file_list = (ls ./test/**/* | where type == file) | get name
+let file_list = (ls ./$list_dir/**/* | where type == file) | get name

This do now work

let file_list = (ls ./$list_dir/**/* | where type == file) | get name
                     ─────────┬────────
                              ╰── Pattern, file or folder not found

I couldn’t figure out a way to format this input to ls in a way that would not generate that error, as ls do not support string input. I tried glob also, but that returned an empty list when using a variable argument

I was unable to find any combination of ls, glob path parse and would compile or give the expected answer. I’m sure the is one, but I was just not able to find it

After a lot of trail and error, I finally figured it out using glob.

let glob_list = (glob $"($list_dir)/**/*")

I quickly discovered that glob returns full path items, instead of relative paths

I eventually found a suggestion to just use the external find command: https://superuser.com/a/1721043. Sadly, falling back to use the external find command was also not without its own pains…

- let glob_list = (glob $"($list_dir)/**/*")
- let file_list = $glob_list | where ($it | path type) == file
+^find test -type f

Apparently the format returned from the find command, did not sit well with next functionality

Error:   × Main thread panicked.
  ├─▶ at /Users/brew/Library/Caches/Homebrew/cargo_cache/registry/src/index.crates.io-6f17d22bba15001f/rand-0.8.5/src/rng.rs:134:9
  ╰─▶ cannot sample empty range
  help: set the `RUST_BACKTRACE=1` environment variable to display a backtrace.

Inspecting the output from ^find, revealed that nushell seem to interpret all output as one single entry. How to split into individual file items then?

Tried all path and split incarnations I could imagine, but all gave me all files as a single entity

^find test -type f | path parse
^find test -type f -print0 | split list '\0'
^find test -type f | split list '\n'
^find test -type f | split row '\n'

I was approaching a solution (I think)

^find $list_dir -type f | complete | get stdout

But I went for the interweb, and thanx to the Arch wiki, I finally had the solution

^find test -type f | lines

Oh man. So easy – when you know how

Documentation

The reference page for each command, provide only the simplest examples. For more advanced/real usage, the explaining is done elsewhere. This make sense if reading as a book, but do not work very well as a reference. Some inter-linking or tags would have helped.

Error message

Error messages are really good, and point directly to where the error is, and what is wrong

 69 │     let l = (glob $list_dir | path parse)
    ·              ──┬─
    ·                ╰── value originates from here
 70 │     $l | path type | print
    ·          ────┬────
    ·              ╰── expected: string, row or list
 71 │     let file_list = []

… that is, unless you you assign the output, then you also mute errors

let files = $file_list | where (it | path type) == "File"

No errors. Lets remove the redirect

$ | where (it | path type) == "File"
·                   ─┬
·                    ╰── Command `it` not found
╰────
  help: Did you mean `bits`?

ysh

ysh

tl;dr

Is feels more like an evolution of an existing language that is very much bash, but not bash. It has less footguns, many more features, but is equally complicated – just not from quirks, but from feature complexity. I found it to be pretty confusing to differentiate between when it was bash, Osh or Ysh. Then there is the very different syntax/semantics of proc and func which seem unnecessary (see Hush) and just adds to the confusion.

It’s a shame that the oilshell site do not have section anchors, so that when you are on a section, you can copy the link (you have to find the entry in the top of the pages, and copy the TOC link). But its a minor/pedantic nitpick.

I am sure you could write a rocket launcher in Ysh, but I am less confident that it is appropriate for the rare shell script need. It is for the advanced scripting needs

Shell:
Approachable for a bash’er: ☆☆☆
Enjoyment: ☆☆☆☆

It’s complicated

I have a bit hesitation on ysh. To me ysh appears to cater shell experts to be the most flexible multi tool for any purpose. This can be great, but also makes it… complicated

There are many things in ysh that makes me think there is a relatively high learning curve, and a touch of a “writable language, not readable language”

How to make an array. Simple

var foods = ['ale', 'bean', 'corn']

How also to make an array, if that array is of strings

var foods = :| ale bean corn |

Functions

Ysh divide functions into two modes: command and expression.

Commands are units of shell-like functions which operates on exterior boundaries. Commands are defined by the proc keyword and should use kebab-casing.

Expressions are more similar to python functions, and operate only on internal values as pure functions. Expressions are defined by the func keyword and use camelCasing.

This is a fantastic separation, which enforces a clear distinction between external functionality, and native script logic (similar to the concept in Hush)

I think where it gets complicated, is how to call func’s and especially proc’s; that is also why there is a rather large documentation on how to call these: Guide to Procs and Funcs

Here is an example from the guide… We have a proc my-cd. Lets call my-cd with two arguments

my-cd /tmp {
  echo $PWD
  echo hi
}

When we look at the definition of my-cd, we notice that it has four parameters

proc my-cd (dest; ; ; block) {
  cd $dest (; ; block)
}

This is because proc’s have four types of parameters. As described in the guide:

proc p (
    w1, w2, ...rest_word;     # word params
    p1, p2, ...rest_pos;      # pos params
    n1, n2, ...rest_named;    # named params
    block                     # block param
) {
  eval (block)
}

So you can call the above proc like this

var pos_args = [3, 4]
var named_args = {foo: 'bar'}
p /bin /tmp (1, 2, ...pos_args; n1=43, ...named_args; { echo "hi" })

Btw, in the my-cd example, I am still not sure what the block part in the body ( cd $dest (; ; block)) does? The documentation says () is an expression. Ysh have cd as a builtin function (ysh-cd) which takes a block as second argument. Not sure how (; ; block) becomes a block…

It appears that ysh has twofold implementation of most features. One with command syntax, and another using expression syntax? This speaks very much into the split between the two modes, but also complicates things as everything now has… well, two syntax’es

I was initial very exited about these modes, that divided concepts. Reality is that its like to different languages mixed together, both in code and in my head Frankly, I hate it

Guess this last thing is just an observation. Ysh have three usage of ...: – ... as the spread operator. – ... as the rest parameters – ... as a prefix that allows to split long commands over multiple lines

shopt

It has some of the same flags (shopt) like bash to modify runtime behaviour. I never really liked that part of bash. It’s an obtuse feature, but at least Ysh allows to contain it to only just blocks of code

shopt --unset errexit {  # ignore errors
  cp ale /tmp
  cp bean /bin
}

(source: https://www.oilshell.org/release/latest/doc/ysh-tour.html)

Error handling

The developer of ysh has a remarkable insight in bash error handling and quirks related to bash $? error handling. ysh takes a firm stance on failing on every error. It then provide you with options to specifically handle errors

try {
  ls /bad
  grep fail /also_bad
}
case (_status) {
  (0)    { echo 'found' }
  (1)    { echo 'not found' }
  (else) { echo "grep returned status $_status" }
}

This will exit the try block on first error. The _status is the ysh equivalent of bash $? I have some thoughts about this – Looks like one can just ignore errors by not looking at _status – Is _status a global thing? I don’t think background processes can overwrite this, as there is a _process_sub_status – Which of the commands in the try block failed? Hush uses a context array to indicate which command failed.

Another example is a comment on how some programs signal usage errors with an exit code other than 0 (success) or 1 (failure), but the shell will just interpret non-zero exit codes as operation error

if grep 'class\(' *.py; then  # grep syntax error, status 2
  echo 'found class('
else
  echo 'not found is a lie'
fi

In practice I think this is fine. For the most part with shell scripting we just need to know if it succeeds or fails. If we care about the exit code, check the exit code explicitly.

In fact this is such a common situation that ysh have a special boolstatus operator for this

if boolstatus egrep '[0-9]+' myfile {  # may abort
  echo 'found'               # status 0 means found
} else {
  echo 'not found'           # status 1 means not found
}

Ysh returning values from functions

Ysh documentation have documented how to return values from functions. Guess this is also necessary, as there are three ways to return values.

Sample

Program arguments

Ysh has an argument parser. The short option is required, otherwise an error is thrown

flag --no-color ('bool', default=false, help='''
   ^~~~
./sample.ysh:18: fatal: proc 'flag' wasn't passed word param 'long'

I could not get string arguments to work on options, so that I could supply --list-dir=test. Looking at the ysh source, it appears only to take integer types and bool (unfortunately I cannot link it, as its generated source). Instead I resorted to making the source directory a general argument

-./sample.ysh --no-color --list-dir=test
+./sample.ysh --no-color test

Now the above workaround is not without is own problems. Apparently there is no way to set arguments as optional; nor to set a default value. This then have the negative consequence that… you can’t get help

❱ ./sample.ysh -h       
        error "Usage Error: Missing required argument $[arg.name]" (status=2)
        ^~~~~
stdlib/args.ysh:206: fatal: Usage Error: Missing required argument list-dir

Gave up on parseArgs.

Where is the documentation for case? Is this Osh or Ysh? Looks like bash, so figure Osh. Eventually I only found a couple of examples that still made me have lots of questions

Printing a dictionary

How do I print a dictionary? Only found this example

var person = {name: 'bob', age: 42}
json write (person)

Value comparison

How do I compare (boolean) values?

 if (options.help == true) {
                  ^~
./sample.ysh:48: Use === to be exact, or ~== to convert types

Oh, use === for exact matches.

if (options.help === true) {

Why?

Calling a func with no ()

I did a mistake of calling a function with wrong syntax, but was unaware of it as no complaint was given by the compiler.

func showHelp() {}

call showHelp    # Does nothing?
call showHelp()  # Calls `showHelp`

Its not impossible that both are valid syntax for two different things… but it surprised me negatively

Navigating the documentation

I have a real hard time locating stuff in the documentation. The titles are not descriptive enough to me, to be able to figure out where to go to read about things. I mostly landed on the all docs list of content

Update: found this: https://www.oilshell.org/release/latest/doc/ref/toc-ysh.html

Returning values from proc

I keep forgetting that there are two different syntaxes for getting return values. Proc is like bash (Osh?) var files = $(get-files options.list_dir)

Listing files

I did not find any builtin way to list files, so figured just to use find.

proc get-files (dir) {
    find $dir -type f
}

var files = $(get-files options.list_dir)

That just gives me one long string with newlines. Putting that into an array type just made an array with one (long) item

var file_list = [ files ]

I then attempted to use the readarray/mapfile function

var file_list = []
cat $files | mapfile file_list

That gave me a type if bashArray. Could not figure out what to do with that type. Nothing seemed to accept it?

I had previously found split, but was a bit hesitant, as it have a big fat TODO marker. But oh well, it worked

var file_list = split(files, b'\n')

I initially attempted to use find with -print0, but none of the mentions function was happy about that, so settled on newline separation for listed files

Print type

Why do printing of type not work, if not stored in a variable?

var d = { "a": 1, "b": 2 }

echo type(d) #=> error: Space required before (

var t = type(d)
echo $t  #=> Dict

Color printing

Could not find any documentation on color support. Some tickets on github mentions that colors use ansi escape codes. I could not get this to work with echo nor write?

I tried using a func to do printf with ansi colors (just like the bash edition). The colors then work, however I could not get redirection of stdout to a variable to work. This was just be me again confusing capabilities of the two. proc are for streams

Converted to proc (I don’t like that ; parameter separation for separating parameter categories). Now I got another problem:

I passed the color function to the actual printing function. As a func I could pass (colorPrint) as an argument just fine – but not use it at the receiving function. With proc (print-color) I cannot figure out how to pass it as a function pointer. Also tried the various unevaluated expressions, but could not make it work

print-category (^(color-print), options.color, extension, files)

The solution became to be not pass the color function as a function, but instead just call the color-print directly

proc print-category (; option_color, extension, files) {
    color-print (option_color, "Extension: ", blue, extension)

Random number

I couldn’t find any random number function, so first thought was that I would just drop in the bash edition. That was of course not the right approach

echo $((RANDOM % upper)) # [0; upper-1]
              ^~~~~~
./sample.ysh:118: POSIX shell arithmetic isn't allowed (parse_sh_arith)

The documentation even states this is not a supported feature

Shell arithmetic like $(( x + 1 )) and (( y = x )). Use YSH expressions.

Also RANDOM is a Bash feature.

Okay, resorting to same solution as used in the Hush solution: use the operating system.

proc get-random-number (; upper) {
    var rand = $( od -vAn -N2 -tu2 < /dev/urandom | tr -d '[:space:]' )
    echo (rand % upper)
}

That didn’t work either

./sample.ysh:115: 'echo' got unexpected typed args
      echo (rand % upper)
      ^~~~
./sample.ysh:115: errexit PID 6347: command.Simple failed with status 2

Putting it in a variable first made it work.

proc get-random-number (; upper) {
    var rand = $( od -vAn -N2 -tu2 < /dev/urandom | tr -d '[:space:]' )
-   echo (rand % upper)
+   var value = (rand % upper)
+   echo $value
}

Haven’t found the passage in the documentation that discusses this, but I have now resorted to this intermediate variable workaround (?) several times

Returning true/false

First I forgot that only func functions can use return, not proc functions.

    (else) { return false }
                    ^~~~~
./sample.ysh:138: fatal: 'return' expected a small integer, got 'false'

Must admit I am getting a little frustrated with the myriad of things that is different between the two concepts… Okay, switching to just true or false

  (else) { false }
           ^~~~~
./sample.ysh:138: errexit PID 17830: command.Simple failed with status 1

Ah yeah. Naturally This was a good opportunity to try out out parameters

proc search-text-in-file (; file, text, out) {
    (else) { call out->setValue(false) }
}

var has_search_match 
search-text-in-file (search_file, "monzool", &has_search_match)

That worked great

Ion

Ion

td;dr

Ion shell was not what I hoped for. I think this is definitely shell first, shell scripting second… third… forth. Documentation is a bit like elvis and mutex. At at glance they seem well documented, but in reality reveal only little detail.

The annoyance level was just too high. I stopped early. It is maybe my own fault, and not that of Ion, but it is what it is

Shell:
Approachable for a bash’er: ☆☆☆☆
Enjoyment: ☆☆☆☆☆

Sample

Program arguments

In the documentation I found no trace of a program argument parsing functionality. What then? A bit weird… there seems to be two matches features? One that resembles a switch/case, and one that searches for substrings. None of them appear to support pattern matching

Then there is the string function find

No result and no syntax error

I’m surely blind, but I did struggle a while to understand why replace would not give me the replaced result, instead of just returning input unmodified… Then I saw it

-list_dir = $replace($arg, "--list-dir=", "")
+list_dir = $replace($arg "--list-dir=" "")

Gave no error, but no expected operation either.

Modifying variables

This distilled piece of code is invalid

let list_dir = "."
for arg in @args
    if matches $arg "--list-dir"
        list_dir = $replace($arg, "--list-dir=", "")
    end
end

It gives this error

ion: pipeline execution error: command not found: list_dir

I wish this error was more precise on where the error is… And what is the compiler complaining about? Are variables immutable? I found no mention of this in the documentation

Then I tried this

-list_dir = $replace($arg, "--list-dir=", "")
+$list_dir = $replace($arg, "--list-dir=", "")

This was greeted by this error:

ion: pipeline execution error: command exec error: Permission denied (os error 13)

Clearly not the solution.

Actually I fail to find a single example in the documentation, where a variable is re-assigned to a value. Everything is just echo’ed.

Update: finally found an example. The while loop show how to update a variable

let value = 0
while test $value -lt 6
    echo $value
    let value += 1
end

Returning values from functions

I could not at all find any example, on how to return values from a function. I first tried return

fn sample arg
    let value = $arg
    return $value
end

let value = sample "ion"

This error’ed on something that perhaps pointed mostly to the call site?

ion: assignment error: extra values were supplied, and thus ignored. Previous assignment: 'value' = 'sample'
ion: expansion error: Variable "value" does not exist

I am left for guessing here.

-let value = sample "ion"
+let value = $(sample "ion")

This gave no error – but no result either?

Success!

fn sample arg
    let value = $arg
    echo $value
end

let value = $(sample "ion")
echo $value

In hindsight probably not unexpected; just not what I was hoping for

Hashmap errors

Hashmap appears to take only the same type, but then everything also seems to be a string

fn setup_options options args
    echo @options
end

let options_defaults:hmap[str] = [ list_dir="test" no_color=false ]
let options = setup_options @option_defaults @args
for key value in @options
    echo $key: $value
end

ion: assignment error: extra values were supplied, and thus ignored. Previous assignment: 'options' = 'setup_options'
ion: expansion error: Variable "options" does not exist

Error messages

Error messages are a quite sparse. There is no indication of which line failed, and given more complex code it becomes very difficult to figure out where the actual mistake is

ion: expansion error: Variable "arg," does not exist

Babashka

babashka

Babashka is super interesting. The sole purpose of babashka is to make Clojure a feasible option for scripting. Its a well made clojure implementation that features super fast start up times due to its use of GraalVM

Here are some relevant links:

• Good introduction: https://www.braveclojure.com/quests/babooka/
• Running shell commands: process
• Babashka as a bash alternative: how-to-do-things-with-babashka
• Easier packaging of single file executable using bbb
• File handling library: fs

Embedded system application

I’m a bit unclear on what requirement there actually are to get babashka cross-compiled to a system that would not fit Java nor GraalVM.

There does not seem to be much information on cross-compiling Java itself. I got only a few hits from Google on the matter. Openjdk had an article, but otherwise it seems not to be a common thing to do. Then again, I am not sure if that is even a requirement. Babashka uses [GraalVM] (https://www.graalvm.org/) to compile itself to native. Perhaps the challenge is to get GrallVM to cross-compile. It certainly can do cross-platform compilation, so the same program can be compiled to Linux, MacOS and Windows. The situation is a bit more muddy when it comes to cross architecture support. It looks like GraalVM do not have official arm32 support. There are some mentions of arm32 working anyway. Given the lack of official documentation, I suspect I would be in for a world of hurt, if attempting to target a tiny embedded system.

I think it’s a safe assumption, that if I go with babashka, I keep on bash’ing on my embedded platforms. I would also add, that I have the feeling that babashka is made more for scripting, than for shell scripting

Planck

Planck

Planck is actually a ClojureScript. It is implemented in C and utilizes the JavaScriptCore. Planck can consume jar files for extra features, but provide its own library also. Its standard library supports the shell commands of Clojure

Its front page has a perfect example

(require '[planck.core :refer [line-seq with-open]]
         '[planck.io :as io]
         '[planck.shell :as shell])

(with-open [rdr (io/reader "input.txt")]
  (doseq [line (line-seq rdr)]
    (println (count line))))

(shell/sh "say" "done")

The fact that it is written in C, possibly makes it very portable. However being a ClojureScript variant, it builds on top of what I expect is a rather large dependency – the JavaScriptCore library. I could not find any size requirements for JavaScriptCore, but Sean McPherson mentions that the Bun runtime (which also builds upon JavaScriptCore) weights in at about 98 MB. That is sadly way to much for a small embedded system

Joker

Joker

Joker is not as such meant for shell scripting, but it has a shell function sh (and variants) that make it possible to run external programs. Joker is written in Go and thus should be pretty portable. Being a Go project, I had hoped it would have small space requirements. Unpacking the prebuilt packages show a roughly 26 MB binary. That is sadly still to much for a tiny embedded system

Fennel

Fennel

It does not appear that Fennel have any additions that enables calling external processes. This makes sense as it sits on top of Lua, and Lua’s built in capabilities for running and managing external programs are pretty poor. Erik Sank have a nice summary of the downsides. Erik Sank also made a library localexec to enhance Lua’s handling of external programs. I can see the implementation uses temporary files in /tmp, ipc over shared memory files (/dev/shm) and other tricks. In essence… it is complicated to retrofit this feature to Lua in Lua

In my previous search for a bash alternative, I mentioned zserge’s luash . Btw, I recommend to read his post about luash, where he also links to Edgar Toernig description of why popen is complicated. Already then, I also mentioned that luash appeared to be abandonware. Last change was in 2015. But! I have discovered Johannes Blaschkes fork that at time of writing is 46 commits ahead of the original. Anyway, the luash implementation also messes with files in /tmp to make things work. It’s complicated when the underlying engine do not have the facilities…

I am not confident Lua is a fitting platform for doing heavy lifting in calling external programs

Janet

Janet

tl;dr

The fact that I am not familiar with lisp was certainly a downside, but in general I think it was easy to adapt to. A downside to the experience was, that the documentation is lacking. There are definitely some great and elaborative articles in the documentation, especially the most used data types are very well described. The situation get a bit more lacking when venturing into its large library of functions. Some features have been documented by the community (which is great), but most function just have a short summary. There are also a few function that do not even have a summary. I managed to get by with a mix of the Janet documentation, the Clojure documentation, the Janet ticket system and some trial and error.

Despite my struggles, I actually had great fun. To my surprise, the end result also have lot of similarity with the original bash sample. I realize this is because I probably don’t lisp like a lisper. Anyway the end result reads pretty easy from a bash mindset. Writing script in lisp is perhaps not as far fetched an approach as I original thought.

Janet have been a great experience!

vscode language plugin: https://marketplace.visualstudio.com/items?itemName=janet-lang.vscode-janet, https://marketplace.visualstudio.com/items?itemName=CalebFiggers.vscode-janet-plus-plus vscode formatter plugin: https://marketplace.visualstudio.com/items?itemName=dlyanb.janet-formatter

Repl:
Approachable for a bash’er: ☆☆
Enjoyment: ☆

About Janet

Janet is a lisp by Calvin Rose, and have this nice catchphrase:

“Janet is a programming language for system scripting, expressive automation, and extending programs written in C or C++ with user scripting capabilities.”

Coincidentally Calvin Rose is also the original creator of Fennel.

Janet have many advanced features, and is implemented in portable C. The website show an example of compiling to 32 Haiku, and on my blog have a small guide on how to cross-compile Janet to armv5. I would have stamped this as “perfekt ”, but it appears that since my cross-compile exercise, janet now requires a boot step to be run on the destined host. That kind of bootstrapping always complicates the cross-compiling machinery

Andrew Chambers has two projects to make Janet have better shell capabilities – janetsh – a systems shell in Janet – janet-sh – a library to make it easy to call external programs from Janet

janetsh appear abandonware, but its janet-sh that is of interest anyway.

A hesitation about janet is that these one-man lisp or scheme implementations often die off with time (probably for life and reasons). Janet however, is very alive and kicking

❱ git log --date=format:'%Y' --pretty=format:'%ad' | sort | uniq -c | awk '{print $2 ": " $1}'
2017: 178
2018: 550
2019: 978
2020: 928
2021: 640
2022: 357
2023: 518
2024: 110

The community appear to be a big part of janet. There is a community driven library spork hosted on the official webpage. Also many documentation examples are provided by the community. Spork can be installed in one big bulk with jpm install spork – or one can use jpm to install the packages individually.

For the janet sample, I chose to allow the use of the external libraries., something that I otherwise refrained from using. I think the Janet situation is a bit different from e.g bash

• It is supported with an official package manager (jpm)
• The documentation is part of the official documentation
• Packages can be included individually or as a spork library
• I am already using janet-sh to get shell capabilities

In fact, the Janet package system reminds pretty much of that of mruby..

Additionally for documentation, there is a free book Janet for Mortals for learning Janet. It has a great chapter on shell scripting with the sh module

Installation

Install from source is trivial. Following the install guide will also install the package tool jpm

Then to install the janet-sh extension:

jpm install https://github.com/andrewchambers/janet-sh.git

The entire output directory i 5.4 MiB including man-pages and Janet as a binary, static library and dynamic library. The Janet binary and sh library is 825 KiB

Calling external programs

Janet have several alternatives to make Janet call external programs In descending abstraction level

janet-sh

The contributor library janet-sh gives janet capabilities to operate very much like a shell scripting language. janet-sh adapts an api that bridges unmanaged shell commands with strict Janet types. The means that, for example, files referenced in a command must be given as real Janet file handles

(with [f (file/open "monzool.txt")]
  (sh/$ echo "monzool" > ,f))

Redirecting stdin and stdout streams are interfaced with e.g. the buffer type.

(def output @"")
(sh/run echo "Hello shell" > ,output)

os/shell

The janet os/shell function give a non-integrating experience that is more traditional from non-shell languages.

A string is given to os/shell and handed verbatim to the standard shell

(os/shell "ls > listing.txt")
(os/shell "top")

os/spawn

The os/spawn command utilizes no shell, but executes a process directly on the system. This is a low level function, and perhaps not normally relevant in shell scripting contexts – but can be useful in some situations

Smoke test

Just for good manners, and to smoke test the installation, I tested out the janet-sh frontpage example…

(import sh)

# raise an error on failure.
(sh/$ touch foo.txt)

# raise an error on failure, return command output.
(sh/$< echo "hello world!")
"hello world!\n"

# return true or false depending on process success.
(when (sh/$? true)
  (print "cool!"))

# pipelines
(sh/$ cat ,path | sort | uniq)

# pipeline matching
(match (sh/run yes | head -n5)
  [0 0] :ok)

cool!
./janet-sh_example.janet:15:1: compile error: unknown symbol path

Ah shoot. A runtime error on path. I have infinitesimal lisp/clojure experience, but I managed to make the example run with few modifications

-(sh/$< echo "hello world!")
-"hello world!\n"
+(def result (sh/$< echo "hello world!"))
+(print result)

+(def path "foo.txt")
(sh/$ cat ,path | sort | uniq)

This worked fine. I just needed to put some data in foo.txt to make the example actually do all it was meant to. Then it got strange. I added this

(sh/$ echo "Janet" > ,path)

and got a stack trace in the terminal

error: unsupported redirect :> :string
  in thunk [./janet-sh_example.janet] (tail call) on line 16, column 1

How do I make it accept anything with >?

I found an example from the janet-sh test suite

(sh/$ echo hello > ,out-buf)

Why do my addition not work, if this works? This made me a little suspicious, so I tried running the test suite

error: assert failure in (deep= @"cba" out-buf)
  in thunk [./test.janet] (tail call) on line 29, column 1

I commented out as much as possible to narrow it down, and the test ran without complaining. Ohhh, its using a buffer!

(def out-buf @"")
(sh/$ echo hello > ,out-buf)
(assert (deep= out-buf @"hello\n"))

I found a blog post on janet-sh by Andrew Chambers. That also had a redirect to buffer example. That made me reflect at bit. unsupported redirect :> :string – okay Einstein, the compiler is telling you that a string won’t work. A buffer works, so I figured a file would probably also work

(def f (file/open "foo.txt" :w))
(sh/$ echo "monzool" > ,f)
(file/close f)

Success Okay, I am now educated from the school of Janet

Sample

Printing

Print functions seem kind of oddly named. prin for print without newline and print for printing with newline… whats wrong with print and println?

Get list of files

There is os/dir, but that lists files and directories in one directory level only. No option to list sub-directories or to filter in only files.

Oh no, I get the impression that I manually have to roll out a solution for listing. It have the building blocks I need: os/dir, os/stat, os/cd and then probably apply each or loop in a recursive manner.

I found a package janet-filesystem, which do have a function filesystem/list-all-files. It doesn’t document how that data is presented. And then I have to bring in yet another package.

Okay, good that I have the janet-sh package. Classic find it is then.

Btw. Wonder if this is just the printing that goes wrong, but the utf-8 filenames do not pretty print when using the os functions. The shell edition prints fine

(os/dir "test")
@["\xF0\x9F\x84\xBC\xF0\x9F\x84\xBE\xF0\x9F\x84\xBD\xF0\x9F\x85\x89\xF0\x9F\x84\xBE\xF0\x9F\x84\xBE\xF0\x9F\x84\xBB.net"

(sh/$ find test -type f)
test/🄼🄾🄽🅉🄾🄾🄻.net

main

Like the other language samples, I had defined a main and tried to call it, but I could not figure out have to call it

compile error: <function main> expects at least 3 arguments, got 0

When reading https://janet.guide/all/#compilation-and-imagination I found out, that its a built in function. It is called automatically if defined

Program arguments

There is a community library argparse for parsing arguments, that works very well. It even generates the help option itself.

spork/path

The spork library have a path library for various path manipulations. I had intended to use path/ext. It has the description “Get the file extension for a path.”, but the result is wrong

(spork/path/ext "file.ext")
".ext"

I would argue that “ext” is the extension here, and “.” is the extension separator. I resorted to use string/split instead.

The function to get the path delimiter also returns unexpected values

(print spork/path/delim)
:

If operating on "${PATH}" this would be a correct answer, but this library works on file paths and should have given / in my case.

I ended up dropping spork/path.

Filename splitting

I spent a bit of time with the threading form to do splitting. Common knowledge I supposed, but fun enough to play around with

repl:1:> (string/split "." "file.ext")
@["file" "ext"]

repl:2:> (-> "." (string/split "file.ext"))
@["file" "ext"]

repl:3:> (->> "file.ext" (string/split "."))
@["file" "ext"]

Checking table for key

It took a bit to figure out how to check if a key is present in a table. Neither the table introduction nor the api explained this. I expect this would be general lisp knowledge, so I guess I cannot blame janet for this. Searching the index for the word “key”, let me to has-key?. The documentation index saved me many times

Random numbers

My random function always return 3?

(defn get-random-number [max]
  (->> (math/random)
       (* max)
       (math/floor)))

(let [random-number (get-random-number (length file-list))
#=> 3

I then tried math/rng-int

repl:113:> (math/rng-int (math/rng) 8)
2
repl:114:> (math/rng-int (math/rng) 8)
2
repl:115:> (math/rng-int (math/rng) 8)
2

Surely a seed is needed. There was no community example on how to use the random functions, so I found a unit-test to model from

(defn get-random-number [max]
  (let [seed (math/rng "monzool")
        engine (math/rng (:int seed))]
    (math/rng-int engine (- max 1))))

repl:162:> (get-random-number 8)
4
repl:163:> (get-random-number 8)
4
repl:164:> (get-random-number 8)
4

Okay, this is probably a global seed function then

(defn get-random-number [max]
  (let [seed (math/rng "monzool")
        engine (math/rng (:int seed))]
    (math/rng-int engine (- max 1))))

repl:169:> (get-random-number 8)
4
repl:170:> (get-random-number 8)
5
repl:171:> (get-random-number 8)
2

Yeah… but that won’t help me one bit, as I will only be running the script once, and then always get the same “random” number (reminds me of that Dilbert random number sketch).

Ended up using /dev/urandom, but surely there must be a way to get a random random number in Janet

janet-sh

I struggled a bit with getting some shell expressions to work with janet-sh

To get a random number from the operating system, I reused a command expression to read from /dev/urandom

repl:3:> (sh/run od -vAn -N2 -tu2 < /dev/urandom | tr -d '[:space:]')
repl:3:60: parse error: mismatched delimiter ),  opened at line 15, column 59

It was less that obvious to me, what the error was. I simplified the command, just to get different error

repl:4:> (sh/$ od -vAn -N2 -tu2 < /dev/urandom)
repl:4: error: :dup2 value must be a file, got /dev/urandom

That was easy solvable

repl:5:> (def f (file/open "/dev/random" :r))
repl:6:> (sh/$ od -vAn -N2 -tu2 < ,f)

That worked. By elimination, that would mean that the tr command was the issue. I studied the Janet for Mortals a bit deeper, and found this great hint

“Janet’s backtick -quoted strings are a really nice way to sidestep shell quoting problems.”

repl:14:> (sh/run od -vAn -N2 -tu2 < ,f | tr -d `[:space:]`)
33720

String to int

Captured stdout from the shell arrives in a buffer. This I attempted to convert to an integer. The functions I could find that looked like useful for the conversion was int/u64. Before this I had to convert the buffer to a string. I ended up with this

(def output @"")
(sh/run od -vAn -N2 -tu2 < ,f | tr -d `[:space:]` > ,output)
(let [random-number (->> output
                        (string/split "\0")
                        (first)
                        (int/u64))]

Then I found a cook-book PR which showed some better tools.

-(let [random-number (->> output
-                        (string/split "\0")
-                        (first)
-                        (int/u64))]
-  (% random-number max)
+(-> output
+  (scan-number)
+  (% max))))

Color printing

I found no dedicated color printing function, but the jpm program (the janet package manager) do have a helper module shutil that provides a color function. Sadly, as many other modules in janet, shutil is largely undocumented, and I could not get it to generate working color commands

repl:41:> (import jpm/shutil)

repl:42:> (shutil/color :blue "monzool")
"\e[0mmonzool\e[0m"
repl:43:> (shutil/color blue "monzool")
repl:43:1: compile error: unknown symbol blue
repl:44:> (shutil/color 34 "monzool")
"\e[0mmonzool\e[0m"
repl:45:> (shutil/color "\e[33m" "monzool")
"\e[0mmonzool\e[0m"

Epiloge

Please note that this is based on my own experience. You might have a completely different experience. My expectations might also have been off for some of the projects. Take this as a subjective evaluation that do not necessarily apply to others. It could very well be that I am just an idiot, and have understood nothing

I think the languages that was most like a mental Jean Claude Van Damme round house kick to the head, was nushell and janet. The bell got rung, and I learned a lot in the process

For the shells, the stdin/stdout is a fundamental way of transporting data between functions. This makes a lot of sense on the command line, especially as structured data that can be manipulated with great flexibility. From a strictly scripting perspective, I think this approach has an inherit downside: the stdout is occupied for data exchange and is not available for printing to the user. I think these next generation shells should have taken the opportunity to separate information to the user/developer from the data for the commands

Enjoyment

How enjoyable was the experience. This is a combination of how good the documentation is and the “ergonomics” of the language. Enjoyment does not correlate with how much the language is similar to Bash, and thus how familiar the experience was, but more if the experience was fun.

bash: ☆☆☆
elvis: ☆☆☆☆
murex: ☆☆☆☆
hush: ☆
koi: ☆☆
abs: ☆☆
nushell: ☆
ysh: ☆☆☆☆
ion: ☆☆☆☆☆
janet: ☆

Ease of adoption

If you know bash, how easy is it to pick up the language

bash: ☆☆ ^{(no one really knows bash)}
elvis: ☆☆☆☆
murex: ☆☆☆☆
hush:
koi: ☆
abs: ☆
nushell: ☆☆☆☆
ion: ☆☆☆☆
ysh: ☆☆☆
janet: ☆☆

Production ready

In the end, how much do you trust including the language in a commercial product? Is it field tested? How is the test coverage? How is the general quality? Is there a chance anyone will answer the questions you have, or fix that bug you found?

What if to be used in medical industry equipment, or other high reliability demanding domains – what would you trust then?

What is the longevity of the language? If the project just dies out and becomes abandonware, then no one will praise you for selecting that language. It might not be a real problem, but if a problem arise, then now you are on your own and have to decide if your client/boss/timeline allows you to maintain your product code and a scripting language.

Good documentation I consider a must for a language that is to be used in a team environment. Chances are that several people are going to maintain the scripts, and then learning material is essential, as not everybody has the desire invest the effort and time to learn some “arbitrary” chosen language.

As the word goes “It is tough to make predictions, especially about the future.” and from my limited knowledge of these projects, answering these questions would guesswork and gut feelings… But if venturing into that (guesswork and gut feelings), I would think that babashka and nushell are currently best bets on a languages to fulfil those requirements, with ysh approaching confidently. I do not know anything about elvish or murex, but for sure they both have a long pedigree of continued development

Conclusion

I am really happy about the decision to make this an exercise of actually trying out the different languages, instead of just theorising about the properties and capabilities of a language. It is also the only way to really get the authentic feeling of a language, and a proper evaluation of its documentation.

The contestants grouped into widely different domains. Some modernizes the shell concept (perhaps inspired by PowerShell), while others draw on the heritage of lisps with a more kitchen-sink mindset; and then some have pursued the philosophy of making a simple script language that provide easy access to external tools.

In hindsight I think I did not accept up front, that the next-gen shells (elvish, murex, nushell) are primarily shells that occupy stdin/stdout for streaming structured data, and kind of leaves the use-case for scripting as a secondary tier. I don’t know if this is a fair assessment, but that was how I perceived the experience. While nushell was an eye-opening experience, then perhaps one day the rosy retrospection will kick in and I will revisit elvish and murex, but I believe from the contenders tried in this round, a new shell is not what I am looking for.

I was perhaps most excited about Janet, but that cooled a bit when I discovered it requires to run on the host for cross-compiling. It is not an impossible task, but that kind of bootstrapping always complicates the cross-compiling machinery. Then there is the giant elephant in the room, if I can persuade team members to use a lisp

Where I think ysh have taken effort to handle every conceivable special case and scenario, it has also gotten complicated. With the opposite philosophy are hush, koi and abs, which was all very good, but also very simple. They provide the structural skeleton, control flow and data types with perhaps a few helper functionalities – and that’s about it. This might be good enough though, and make a sufficient compromise? Unfortunately it appears that both koi and abs is now abandonware, and hush is on the verge of becoming it Being abandonware doesn’t mean that they are not useful, but facts are that they do not feel 100% complete as is, and without active development that will never happen

In conclusion, I learned a lot – but I am not sure I actually came closer to finding a true bash alternative…

Perhaps it is time to accept that small embedded systems are stuck on bash, and future midsize to large embedded systems approach a size where e.g. python is a viable route.

Addendum

Future candidates

Rash

Rash

Rash is a shell that is build on top of Racket. Rash can also be used for scripting, and provides a nice terse language extension to Racket that do not require the famous parentheses of scheme and lisp. Rash have many interesting ideas on how to intermixing shell commands and racket source I have not yet looked into Racket for size requirements. I hear some people references Racket as the scheme equivalent of python. Cross-compiling to arm32 appears to be supported.

See this interesting presentation for features of Rash: https://www.youtube.com/watch?v=Acjqx1MPkw4

The main concern is that is appears to be gravitating towards becoming abandonware

Python

Maybe it is time to accept that python will assimilate us all, and resistance is futile . It used to be a bit complicated to put on an embedded system, but times have changed and as using systemd is not given much second thought, then lots of desktop systems now use python and that trend is sieving into the embedded marked also. Personally I have done a lot of python stuff on the desktop, but I do not find it particularly exciting. I also find python a horrible language for doing shell scripting work in, but there are some projects that lift the experience to something worth considering:

xonsh

xonsh

Xonsh is a shell first project that mixes bash like syntax with python. Xonsh has been in active development for a long time, yet at time of writing the current version is 0.17.0, with no indication of a version 1.0 in the horizon. An interesting side-note is that different language in the python ecosystem can tap into xonsh. Coconut has such xonsh integration.

Plumbum

plumbum

Plumbum does not attempt to be a shell alternative like xonsh, but instead focus on making the shell scripting experience better, by making external program work as seamless as possible with python code.

Take 1

I will just shortly address the results of my previous hunt for an alternative to bash scripting. The selections fell on: scsh, luash and mruby

scsh

I pitched this at work, but eventually every one agreed that this was too high a mountain to climb. None of us had scheme experience, and the deadlines was too pressured for us to learn something that was that much different from our C/C++ and bash experience. I mentioned already then, that scsh development seems to have stopped. That appears still to be true

❱ git log --date=format:'%Y' --pretty=format:'%ad' | sort | uniq -c | awk '{print $2 ": " $1}'
2009: 123
2010: 44
2011: 28
2012: 25
2013: 20
2014: 47
2015: 3
2016: 1
2022: 2
2024: 2

luash

luash appeared abandoned then, so we eventually decided not to proceed. The situation appears to be the same still

❱ git log --date=format:'%Y' --pretty=format:'%ad' | sort | uniq -c | awk '{print $2 ": " $1}'
2015: 23
2016: 1
2017: 3

mruby

I brought in mruby for a task that had a size and complexity where I would never use bash, but instead write a C program. The downside of using a compiled program, is that you have to compile, strip and copy to target on each change. This specific task would benefit greatly from being able to make tweaks directly on target, so a scripting language was a perfect fit. I skunk-worked a mruby solution.

What I remember as my takeaways from the experience in 2017 or 2018 of mruby was:

1. Calling external programs was not a first class feature. It was possible, but was not particularly great
2. The custom plugin system was stellar. It was very easy to write a plugin in C, and the use it from mruby. That was perfect for the task I had to solve
3. I do remember hitting some bugs. No deal breakers, but yet something that made friction
4. No one in the team (including me) really fell for the ruby syntax. Shame on us perhaps, but reality is reality

I mentioned some bugs and issues… it was a long time ago, and since then the mruby development has only gained more traction. I have nothing but confidence that mruby is working great today. A lot of focus and effort have been channelled into mruby development since then

❱ git log --date=format:'%Y' --pretty=format:'%ad' | sort | uniq -c | awk '{print $2 ": " $1}'
2012: 1933
2013: 2279
2014: 2405
2015: 870
2016: 607
2017: 1185
2018: 580
2019: 1411
2020: 1082
2021: 1225
2022: 1116
2023: 851
2024: 399

I haven’t kept up to date with mruby, but at a glance it appears that there have not been innovations in the area of calling external programs. Regular ruby have gotten a library tty-command that appears to do a good job of simplifying calling external programs. With some limitations it is possible to run ruby code from mruby. It might be worth a look…

The post A Second Search for Bash Scripting Alternatives appeared first on Monzool's Personal Publishing.

My immediate contestants are

Babashka – A clojure variant that emphasize on scripting.

DCli – A dart library for making cli’s.

Shellout – A library adding shell operations to Gleam.

I have next to nothing experience in these languages, but I guess this would be a fine opportunity to learn one of them. Actually I had a fourth contestant, but I dont know OCaml and its documentation was just to sparse for me to evaluate if this was any good or not

Shexp

I’m in doubt which language I should pick up. If I had time for it, I would try them all and compare, but unfortunately I do not Any suggestions on which would be the better choice?

The post Chose Babashka, DCli or Gleam/Shellout? appeared first on Monzool's Personal Publishing.