6.34942

Wrapping up the long tail

2011-09-22

I don’t listen to a wide range of music. I tend to stick to just a handful of artists, listening to the albums I love over and over.

This gives the distribution of what artists I listen to a well-defined long tail, and I think Last.fm’s histogram doesn’t quite do it justice. With the assistance of the Last.fm API, the R wordclould package, a code sample, and some inspiration, I’ve generated a new visualization:

I think this more clearly illustrates the sheer difference in scale between the few dozen artists that I listen to regularly and the thousand others I’ve had opportunity to evaluate.

Custom keyboard symbols for Unity

2011-09-03

Dot stickers make an excellent tool for customizing the Apple aluminum keyboard.

Adrenaline triphosphate

2011-07-12

ATP.

That’s what it stands for, right?

Automatic submission of Rockbox scrobbler logs

2011-07-07

Rockbox has the ability to generate Audioscrobbler logs for later submission to Last.fm. A custom udev rule can be combined with a command-line Last.fm scrobbler to automate the process of submitting tracks every time the digital audio player is connected to the computer.

The udev rule is designed to call a script whenever a Rockbox device is connected:

# /etc/udev/rules.d/80-custom.rules

ACTION=="add", SUBSYSTEM=="block", ATTR{partition}=="1", ATTRS{vendor}=="Rockbox ", ATTRS{model}=="Internal Storage", RUN+="/opt/rockbox/upload-scrobbler-log"

Last.fm credentials can be stored in GNOME Keyring and accessed using gkeyring:

gkeyring.py --set --name 'http://www.last.fm/' --password 'YOUR_PASSWORD_HERE' -p username_value=YOUR_USERNAME_HERE,origin_url=http://www.last.fm/

The script will wait for the device to be mounted, retrieve the Last.fm credentials from GNOME Keyring, display a graphical notification using notify-send, call Laspyt to submit the scrobbler log, and then display a report:

#!/bin/bash
# /opt/rockbox/upload-scrobbler-log
# WARNING: This script must finish quickly because udev is effectively paused while it is running

# Run all this in the background and detach from the parent process
{

    ubuntuuser="ak"
    logfile="/tmp/upload-scrobbler-log.log"
    backupscrobblerlog="/tmp/scrobbler.log.backup"
    gkeyring="/opt/rockbox/gkeyring.py"
    laspyt="/opt/rockbox/laspyt.py"

    echo "Script starting at $(date)" >> "$logfile"

    dapdescription="$(echo -e "$ID_FS_LABEL_ENC")"
    echo "Detected $dapdescription at $DEVNAME" >> "$logfile"

    # Wait for filesystem to be automounted
    echo "Waiting up to 20 seconds for automount..." >> "$logfile"
    counter=0
    while [ $counter -lt 20 ]; do
        grep -q "$DEVNAME" /proc/mounts && break
        sleep 1
        let counter++
    done

    # Abort if timed out
    if [ $counter -ge 20 ]; then
        echo "Unable to locate mountpoint; aborting" >> "$logfile"
        exit 1
    fi

    # Determine mountpoint
    mountpoint="$(echo -e "$(grep $DEVNAME /proc/mounts | awk '{print $2}')")"
    echo "Detected mountpoint at $mountpoint" >> "$logfile"

    # Identify log file
    scrobblerlog="$mountpoint/.scrobbler.log"
    echo "Using scrobbler log file $scrobblerlog" >> "$logfile"

    # Get credentials from keyring
    echo "Getting credentials from keyring..." >> "$logfile"
    credentials="$(sudo -H -u $ubuntuuser \
        env $(grep -v "^#" /home/$ubuntuuser/.dbus/session-bus/$(cat /var/lib/dbus/machine-id)-0) \
        $gkeyring -p origin_url=http://www.last.fm/ --output username_value,secret)"
    username="$(echo "$credentials" | awk '{print $1}')"
    password="$(echo "$credentials" | awk '{print $2}')"
    if [ -z "$username" -o -z "$password" ]; then
        echo "Failed to get credentials; aborting" >> "$logfile"
        exit 1
    else
        echo "Got credentials for $username" >> "$logfile"
    fi

    # Back up scrobbler log
    echo "Backing up scrobbler log to $backupscrobblerlog" >> "$logfile"
    cp "$scrobblerlog" "$backupscrobblerlog"

    # Upload scrobbler log
    timezone="$(date +%:::z)"
    echo "Uploading scrobbler log file with timezone offset $timezone..." >> "$logfile"
    sudo -H -u $ubuntuuser \
        env DISPLAY=$(w -s $ubuntuuser | grep -m 1 " :" | awk "{print \$3}") \
        notify-send --icon "/usr/share/icons/Humanity/devices/48/multimedia-player.svg" \
            "Uploading scrobbler log" \
            "Submitting $(grep -v "^#" "$scrobblerlog" | wc -l) tracks from $dapdescription to Last.fm as $username"
    sudo -H -u $ubuntuuser \
        $laspyt --file "$scrobblerlog" --timezone "$timezone" --clear \
            --user "$username" --password "$password" \
            2>&1 | sed 's/\x1B\[[0-9;]*[mK]//g' >> "$logfile"

    # Report
    echo "Done" >> "$logfile"
    sudo -H -u $ubuntuuser \
        env DISPLAY=$(w -s $ubuntuuser | grep -m 1 " :" | awk "{print \$3}") \
        zenity --text-info --title "Scrobbler log upload" --filename "$logfile" &

} & disown

exit 0

Details from a run:

and confirmation on Last.fm:

Learning to use R

2011-06-03

To begin familiarizing myself with R, I’ve been experimenting with binned kernel density estimates using data from Ask Ubuntu.

The resulting plots aren’t very meaningful, but they’re quite pretty.

Implementing a hybrid SSD/HDD home folder

2011-05-17

I’m currently experimenting with a low-effort technique for mimicking the benefits of a hybrid solid state/hard disk drive.

My approach is to use a union mount with multiple writable branches to combine folders from both an SSD and HDD into a single home folder in the root filesystem. The result is a unified home folder that contains a mixture of files from each drive in a way that is transparent to the operating system:

Newly created files are stored on the same device as their parent folder. Files created at the root of the home folder are typically configuration files and so default to the SSD.

As the screenshot reveals, I have also drafted a Nautilus extension that detects which storage back end each file uses and labels those which use the SSD.

Implementation

The branches of the union mount exist externally as folders within the root filesystem. I chose to place them within /home/.ssd and /home/.hdd. On my system / already uses the SSD so /home/.ssd can simply exist as a regular folder, while /home/.hdd is mounted separately. /etc/fstab contains:

# / containing /home/.ssd (Physical SSD)
UUID=... / ext4 noatime,discard,errors=remount-ro 0 1
# /home/.hdd (Physical HDD)
UUID=... /home/.hdd ext4 errors=remount-ro 0 2

The default file creation policy in aufs is to store new files within the same branch as their parent folder, with files created in the root directory of the union filesystem defaulting to the first listed branch. As long as the SSD branch is listed first, no other policy needs to be explicitly defined:

# /home/ak from SSD and HDD mounts
none /home/ak aufs noauto,br:/home/.ssd/ak=rw:/home/.hdd/ak=rw 0 0

It is critical that this entry in /etc/fstab be mounted only after the previous two lines so that the incorporated branches contain mounted filesystems and not empty mount points. Until I learn how to specify a particular mount order in /etc/fstab, I have disabled automatic mounting of the union mount (noauto above) and instead mount it during /etc/rc.local:

# Workaround to mount aufs only *after* its branches exist
mount /home/ak

Back end detection is accomplished with a custom emblem in ~/.local/share/icons/hicolor/24x24/emblems and a Nautilus-Python extension saved as ~/.nautilus/python-extensions/ssd-emblem.py that checks for parallel files in /home/.ssd:

import os.path
import nautilus

class SsdEmblem(nautilus.InfoProvider):
    def __init__(self):
        pass

    def update_file_info(self, file):
        filepath = os.path.realpath(file.get_location().get_path())
        if filepath.find('/home/ak/') == 0 and os.path.exists(filepath.replace('/home/ak/', '/home/.ssd/ak/',1)) == True:

Thoughts

After using this for a week I’m pleased with the results. There is a noticeable improvement in login times compared with my previous configuration that stored the entire home folder on a HDD.

I would like there to be an easy way to switch the storage back end of individual files and folders, ideally via a Nautilus context menu item. This looks possible via UDBA, but there are several potential quirks involved that I haven’t spent the time to thoroughly investigate.

Obviously, more work would need to be done for this to be scalable to multi-user environments or to be compatible with encrypted home folders. I have no plans to develop these areas.