I have a small DigitalOcean droplet (VM) that runs this blog, and a few other personal services; the main one is storage sync, which for years has been NextCloud. This was fine, but a but clunky (not helped by 3-container docker-compose setup), and sync was pretty unreliable in the incoming direction: for example, the "open up laptop and receive incoming changes" use-case was a bit flaky without a manual poke, and the even more complicated "laptop is already open, receive incoming changes recently made elsewhere" story was basically non-functional (without, again, manually clicking "sync now"). Recently though I'd been hearing good things about OpenCloud¹ and after a bit of testing decided to make the switch. I needed to update the OS as well anyway, so good time for changes and a rebuild.

The switch was essentially painless and has been running smoothly, but I did need to make some tweaks beyond the initial install, hence this post.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

export GOMEMLIMIT=700MiB
export GOGC=50

[Container]
ContainerName=opencloud
Image=docker.io/opencloudeu/opencloud
AutoUpdate=registry
PublishPort=9200:9200

Environment=OC_URL=https://opencloud.example.com
Environment=PROXY_TLS=false
Environment=OC_INSECURE=true

# For webdav:
Environment=PROXY_ENABLE_BASIC_AUTH=true

Environment=STORAGE_USERS_DRIVER=decomposeds3
Environment=STORAGE_USERS_DECOMPOSEDS3_ENDPOINT=https://syd1.digitaloceanspaces.com
Environment=STORAGE_USERS_DECOMPOSEDS3_REGION=syd1
Environment=STORAGE_USERS_DECOMPOSEDS3_ACCESS_KEY=XXXXXXSTORAGEACCESSKEY
Environment=STORAGE_USERS_DECOMPOSEDS3_SECRET_KEY=XXXXXXSTORAGEACCESSSECRET
Environment=STORAGE_USERS_DECOMPOSEDS3_BUCKET=bucketname

# Memory optimisation
Environment=GOMEMLIMIT=700MiB
Environment=GOGC=50
# Exclude unused services:
Environment=OC_EXCLUDE_RUN_SERVICES=search,antivirus,app-registry,app-provider,invitations,notifications

Volume=/local/data/dir:/var/lib/opencloud:z
Volume=/local/config/dir:/etc/opencloud:z

# Entrypoint=/bin/sh # not supported on podman 4.9.3
PodmanArgs=--entrypoint=/bin/sh
Exec=-c "opencloud init --insecure true; exec opencloud server"

# This is the internal uid/gid; see podman inspect docker.io/opencloudeu/opencloud --format '{{.Config.User}}'
User=1000:1000
# And this maps the internal ids to the local opencloud user (which is uid 1002 as of current testing)
UserNS=keep-id:uid=1000,gid=1000

[Service]
Restart=always

[Install]
WantedBy=default.target

Sometimes I'll make a note or two as I read (frequently, starting with why I'm picking this particular book, as I'm starting it), but sometimes I'll make a perfunctory comment at the end and that's it. The former feels much more productive.

In terms of content my recollection is that it was a mix of technical (largely explaining why I didn't get through as much this year, particularly reading technical content in bed at night) and "easy reading" when I felt like a break. Looking back over the list though there's a bit more variety than I might have suspected.

TL;DR

(defun project-find-subroot-for-eglot (dir)
  (when eglot-lsp-context
    (let ((root (locate-dominating-file dir ".eglot")))
      (when root
        (cons 'transient root)))))

(add-hook 'project-find-functions #'project-find-subroot-for-eglot)

My numbers were a little bit down on last year; 35 compared to 40. At times progress felt so slow — reading only at night and barely managing a few pages — that I'm surprised that discrepancy wasn't worse.

I read 40 books in total, with another 4 on the go at the end of the year (Aurelien Geron's Machine Learning with TensorFlow book, "Remember It!", a mini collection of Aurelius' "Meditations" from a second-hand shop years ago that I never actually opened, and Brendan Gregg's Systems Performance which is so good that I ordered a hard-copy. I'll read a few more chapters, but probably leave it as a reference for now).

The last few years I've been off most social media and I (mostly) don't pick up my phone for entertainment, but I still notice that my ability to focus on a long article (non-work, ie I'm reading for entertainment or self education) has diminished.

I blame the sheer amount of content! There's just so much to get through that there's always more in the queue, and anything that slows you down is discarded ("for later", perhaps). But as an early precursor, I also speculate that tabbed browsing had a big part to play here. My first introduction was the original iCab browser, and it was a revelation: pages loaded slowly in those days, so I could open a bunch in parallel and flick between them to consume those that loaded first. (Then of course you don't get around to finishing all of them so some are left for "later"… but I suspect that bad habit didn't fully develop until later)

I don't have any particular answers, but in addition to re-starting a mindfulness meditation practice I have been making a concerted effort to read more books again which I have been happy about.

I've recently started using Atuin, which is essentially a souped-up tool for managing your shell history. Every shell (bash, zsh, etc) will keep a record of your previous commands, but they'll all store them in their own file, using their own format, with their own tweaks on accessing it. Using atuin delegates all of that to a separate tool, which is now relatively shell-agnostic, and you get additional metadata such as the directory the command was executed in and the return status and duration, and you get more advanced search capabilities, and you can even sync history across machines (I don't use this feature, currently).

If you clicked the link above you'll see that atuin provides a search UI as well. I've also been using fzf lately, which is an absolute game-changer. I don't even make heavy use of its capabilities (although I'm starting to integrate it more, eg as a jq repl), but the real-time fuzzy filtering is amazing, and mirrors the experience I get in Emacs via helm or more recently selectrum et al.

Atuin has a few different options for searching (prefix, fuzzy, exact, excluding, etc), including mostly mimicking the experience I was used to with fzf and Emacs, but ultimately didn't quite reproduce it.¹

What completed the puzzle for me was a snippet from a Hacker News comment, that combines the fzf UI with atuin for search management. I have changed it slightly; by default it restricts the search to the current directory only, but a second control-r will search the entire history.

atuin-setup() {
    if ! which atuin &> /dev/null; then return 1; fi
    # bindkey '^E' _atuin_search_widget

    export ATUIN_NOBIND="true"
    eval "$(atuin init zsh)"
    fzf-atuin-history-widget() {
        local selected num
        setopt localoptions noglobsubst noposixbuiltins pipefail no_aliases 2>/dev/null

        # local atuin_opts="--cmd-only --limit ${ATUIN_LIMIT:-5000}"
        local atuin_opts="--cmd-only"
        local fzf_opts=(
            --height=${FZF_TMUX_HEIGHT:-40%}
            --tac
            "-n2..,.."
            --tiebreak=index
            "--query=${LBUFFER}"
            "+m"
            "--bind=ctrl-d:reload(atuin search $atuin_opts -c $PWD),ctrl-r:reload(atuin search $atuin_opts)"
        )

        selected=$(
            eval "atuin search ${atuin_opts} -c $PWD" |
                fzf "${fzf_opts[@]}"
                )
        local ret=$?
        if [ -n "$selected" ]; then
            # the += lets it insert at current pos instead of replacing
            LBUFFER+="${selected}"
        fi
        zle reset-prompt
        return $ret
    }
    zle -N fzf-atuin-history-widget
    bindkey '^R' fzf-atuin-history-widget
}
atuin-setup

This "toggle between directory and global history" feature is something I've been using in zsh via the per-directory-history plugin. Atuin does have a history-import feature, but not surprisingly it doesn't work with the per-directory-history which is in a different format. I wrote a little python snippet (below) to handle this. The gist is that per-directory-history keeps a separate history file for each directory, where the directory tree is mirrored under a separate directory in your home: for example if you run a command in /var/log/httpd, it would go into the file ~/.directory_history/var/log/httpd/history. Most of the rest of the metadata we have to fake or guess (or in the case of timestamp resolution, adjust). You may need to tweak to your situation, and I've elided a few non-essential bits, but hopefully it helps someone!

  import os
  import os.path
  import sqlite3
  import sys
  import uuid


  def load_history(fpath, session, hostname, hist_dir):
      history = []
      cwd = fpath.removeprefix(hist_dir).removesuffix('/history')
      print(f'Using cwd {cwd}')
      lastcmd = ''
      with open(fpath, 'r') as f:
          for line in iter(f):
              try:
                  while line.strip()[-1] == '\\':
                      line += next(f)
                  line = line.lstrip(': ')
                  tss, line = line.split(':', 1)
                  _,cmd = line.split(';', 1)
                  if cmd != lastcmd: # Ignore dups, since the timestamp doesn't always have enough resolution
                      history.append({
                          "id": uuid.uuid4().hex,        # id
                          "timestamp": int(tss) * 1e9,   # timestamp
                          "duration": 0,                 # duration
                          "exit": 0,                     # exit
                          "command": cmd,                # command
                          "cwd": cwd,
                          "session": session,
                          "hostname": hostname,
                      })
                      lastcmd = cmd
              except Exception as e:
                  print(f'Error: {e}')
      return history


  INSERT_SQL = """
  insert into history (id, timestamp, duration, exit, command, cwd, session, hostname)
  values (:id, :timestamp, :duration, :exit, :command, :cwd, :session, :hostname)
  """


  def main(basedir):
      basedir = os.path.expanduser(basedir)
      basedir = os.path.realpath(basedir)
      print(f'Using basedir: {basedir}')

      db = sqlite3.connect('history.db')
      cur = db.cursor()

      session = uuid.uuid4().hex
      hostname = 'hostname:username' # update as necessary
      for dirpath, _dirnames, filenames in os.walk(basedir):
          print(dirpath)
          if 'history' in filenames:
              history = load_history(os.path.join(dirpath, 'history'),
                                     session,
                                     hostname,
                                     basedir)
              print(f'Loaded {len(history)} commands')
              cur.executemany(INSERT_SQL, history)

      db.commit()
      db.close()

This is not copmletely transparent; in particular, hooks defined for the old python-mode don't automatically run for python-ts-mode as well.

The other day I found another omission, albeit with a fairly unobtrusive workaround. I use the yasnippet package for snippets¹, and mostly rely on the yasnippet-snippets predefined collection. These are grouped by modes, and… not surprisingly a snippet defined in python-mode does not automatically become available in python-ts-mode.

However — this examples assumes that you also load ~/.emacs.d/snippets/ — if you add a file ~/.emacs.d/snippets/python-ts-mode/.yas-parents, containing the single line "python-mode", then all snippets defined in python-mode will also be loaded and available in python-ts-mode.

(prompted by this toot)

I wanted to see if the native library was actually being loaded, and my first thought was "this is a job for strace -e trace=open"! (reference: the always-excellent Julia Evans. You should go read those even if you don't read any further here).

Something made me search first though, and what I actually discovered was the pmap command, which can tell you exactly what file(s) are mapped (the -p is for "path", not "pid"):

pmap -p <pid>

It turns out that this is also exactly the contents of the ls -l /proc/<pid>/map_files, or cat /proc/<pid>maps.

(Note that it is not the same as my original strace plan, which was to look for the library being opened; this shows files that are mmap-ed into the process)

To test it out, I grabbed the mmap example from the man page, added a getchar() call to keep the process running, and ran it:

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#define handle_error(msg)                                       \
        do { perror(msg); exit(EXIT_FAILURE); } while (0)

int main(int argc, char *argv[]) {
        char *addr;
        int fd;
        struct stat sb;
        off_t offset, pa_offset;
        size_t length;
        ssize_t s;

        if (argc < 3 || argc > 4) {
                fprintf(stderr, "%s file offset [length]\n", argv[0]);
                exit(EXIT_FAILURE);
        }

        fd = open(argv[1], O_RDONLY);
        if (fd == -1)
                handle_error("open");

        if (fstat(fd, &sb) == -1)           /* To obtain file size */
                handle_error("fstat");

        offset = atoi(argv[2]);
        pa_offset = offset & ~(sysconf(_SC_PAGE_SIZE) - 1);
        /* offset for mmap() must be page aligned */

        if (offset >= sb.st_size) {
                fprintf(stderr, "offset is past end of file\n");
                exit(EXIT_FAILURE);
        }

        if (argc == 4) {
                length = atoi(argv[3]);
                if (offset + length > sb.st_size)
                        length = sb.st_size - offset;
                /* Can't display bytes past end of file */

        } else {    /* No length arg ==> display to end of file */
                length = sb.st_size - offset;
        }

        addr = mmap(NULL, length + offset - pa_offset, PROT_READ,
                    MAP_PRIVATE, fd, pa_offset);
        if (addr == MAP_FAILED)
                handle_error("mmap");

        s = write(STDOUT_FILENO, addr + offset - pa_offset, length);
        if (s != length) {
                if (s == -1)
                        handle_error("write");

                fprintf(stderr, "partial write");
                exit(EXIT_FAILURE);
        }

        while(getchar()!='\n'); /* ADDED: Wait to exit */

        munmap(addr, length + offset - pa_offset);
        close(fd);

        exit(EXIT_SUCCESS); }

Running:

$ gcc maptest.c
$ ./a.out maptest.c 5

while this is running, in another terminal:

PID=$(pgrep a.out)
pmap -p $PID

and sure enough, there's our file mapped into the process' memory.

The following posts are a great place to start: overview, note in particular major-mode-remap-alist, details.

For another example of what can be unlocked, check out this minimalist expand-region replacement.

A quick post today, mostly my working notes.

I have a Friday habit of building Emacs from the master branch, and reading the additions to etc/NEWS¹. The last few years have been particularly active, with support for built-in LSP (via eglot), native compilation, improved long-lines performance, etc.

The biggest news recently has been the incorporation of treesitter support. This has already been available via dynamic modules, but is now available natively, with a bunch of language modes updated with support for it as well. I won't dwell on what treesitter is, but benefits include much improved font-lock and code-folding support and the potential for semantic navigation, etc.

I wanted to try this out, but because tree-sitter itself is an external library, and new enough not to be available in my distribution's package manager, it requires a bit more leg-work. Because I'm just experimenting (and don't use an immutable distribution²), I didn't fancy installing the library, so had a few more hoops to add and jump through. More detailed notes are available here.

1. git clone https://github.com/tree-sitter/tree-sitter
2. cd tree-sitter; make all
3. Now we need to configure Emacs --with-tree-sitter, but because it's not installed we need to tell it the library to link against (and where to find it), and where to find the dev headers. Ignoring other options³, for me this looks like TREE_SITTER_CFLAGS=-I~/Projects/tree-sitter/lib/include TREE_SITTER_LIBS="-L~/Projects/tree-sitter -ltree-sitter" ./configure --with-tree-sitter
4. Now, that's dynamically linked, so to run it we still need to modify the environment so it can find the library: LD_LIBRARY_PATH=~/Projects/tree-sitter ~/Projects/emacs/src/emacs Update: I now also needed LD_LIBRARY_PATH set to build as well, because building involves starting a bare-bones Emacs in order to dump. Not sure if I missed this earlier, or the build process has changed.

Now you have an Emacs with built-in support, and you can verify by evaluating (treesit-available-p), but it doesn't get you very far because the language-specific information is still missing. To see this, open a supported mode, say a javascript file, and run M-x js-ts-mode. You will probably see a warning like "Cannot activate tree-sitter, because language definition for javascript is unavailable (not-found): (libtree-sitter-javascript libtree-sitter-javascript.so) No such file or directory".

To resolve this we need the language-specific modules (the equivalent of https://github.com/emacs-tree-sitter/tree-sitter-langs for the dynamic-module project), which you can obtain from https://github.com/casouri/tree-sitter-module/. These can either be built, or just downloaded from the releases; you can extract them into /.emacs.d/tree-sitter (see variable treesit-extra-load-path for details or customisation). Now if you try again — no restart necessary — your new mode should load successfully!

So far I haven't observed any issues from having both the built-in and the dynamic-module support. You will see that the built-in support uses the prefix treesit- while the third-party package uses tree-sitter-. The updated modes contain -ts-, eg python-ts-mode, js-ts-mode, etc.

Previously all I've done is follow some instructions and type out a few commands. Actually understanding what I was doing beyond the rough concepts was a different thing. The commands, and there are plenty, have names like pvcreate and lvconvert, which were completely unfamiliar and after a short search which got bogged down in terminology, I moved on.

The third time I resolved to actually dig in just a bit more, and happily the terminology at least now makes more sense! The key for me was the high-level hierarchy, and secondly the similar structure of the commands (once the intuitive concepts made a bit more sense).

• Physical Volumes: These are the disks themselves¹. Commands all use the pv prefix.
• Volume Groups: These create a storage pool out of the physical volumes; every virtual disk is drawn from a single volume group. Commands all use the vg prefix.
• Logical Volumes: These are the "virtual disks" that you mount as a single block device. This is what gets mounted in the server's file system. Commands all use the lv prefix.

"Imposter syndrome" can feel like something designed to make you feel better as long as you don't look too closely under the covers: "smart people often feel like imposters" does not, unfortunately, imply that if you feel like an imposter then you must in fact be competent.

The other day I came across one of those articles that skewers directly to your soul, specifically targeting those deep insecurities! It was via a tweet from John Carmack, referencing an article on lesswrong.com:

Even so, these people of the Power Elite were visibly much smarter than average mortals. In conversation they spoke quickly, sensibly, and by and large intelligently. When talk turned to deep and difficult topics, they understood faster, made fewer mistakes, were readier to adopt others' suggestions.

No, even worse than that, much worse than that: these CEOs and CTOs and hedge-fund traders, these folk of the mid-level power elite, seemed happier and more alive.

This, I suspect, is one of those truths so horrible that you can't talk about it in public. This is something that reporters must not write about, when they visit gatherings of the power elite.

Because the last news your readers want to hear, is that this person who is wealthier than you, is also smarter, happier, and not a bad person morally. Your reader would much rather read about how these folks are overworked to the bone or suffering from existential ennui.

In some ways the last time I remember feeling so confronted, as if the author were focussing on me particularly across a crowded room and pointing out that thing you were feeling awkward about, was Neal Stephenson's Snow Crash:

Until a man is twenty-five, he still thinks, every so often, that under the right circumstances he could be the baddest motherfucker in the world. If I moved to a martial-arts monastery in China and studied real hard for ten years. if my family was wiped out by Colombian drug dealers and I swore myself to revenge. If I got a fatal disease, had one year to live, devoted it to wiping out street crime. If I just dropped out and devoted my life to being bad. Hiro used to feel that way, too, but then he ran into Raven. In a way, this is liberating. He no longer has to worry about trying to be the baddest motherfucker in the world. The position is taken.

I first read that when I was 23. I was a gym junkie and martial artist (albeit, starting to realise I might never be Bruce Lee), and with just enough budding awareness to realise that I was exactly the sort of person he was talking to, and that he was 100% right.

It is possible that the rest of your life is determined, in some ways at least, by what you do with that realisation. Do you pack it in once it is obvious you'll never be the best; do you keep switching paths searching for something you might be the best at (or at least, switching frequently enough to plausibly claim that perhaps you could have been if only you had kept going); or are you comfortable enough to be your best?

I think by and large I settled on the last option. If anything, I became a little too comfortable. This can manifest as abrogating responsibility: I followed the training program and did everything I was supposed to do, so if I'm not great it's just because I was never going to be and that's ok.

(To be clear, it is ok, particularly if you're ok with it, but there is still the nagging realisation that anything beyond satisfactory requires some amount — possibly lots — of perseverance through sticking points and plateaus, and perhaps just a little bit of stubborness and foolhardy desire will get you through. Stuhlberg and Magness cover just this contradiction in The Passion Paradox).

My whole life has basically been assuming that I could, to a point, outwork people or otherwise accomplish things I didn't seem talented at, just by being persistent ("showing up") and studying up on it. I have obsessive-compulsive tendencies and enjoy the routine so this is not a hair shirt I wear.

I was never particularly good at chess, relative to others who spent more sporadic time in the lunchtime chess room in high school, but I played board 1 in the state high-school championship because I spent a lot of time playing through games and learning opening lines. I was never interested in endurance sports at all, and instead spent 25 years in the gym trying to get as strong as I could, but recently became involved in ultra running purely because so much of it involves simply putting in the hours¹.

So back to Competent Elites: the author is correct, it's a confronting observation because we all know (and have probably seen) the Peter Principal, and we often hear how the pursuit of money or success is a Faustian bargain so contradictory evidence can make us feel like we've been lied to.

That's actually not quite what the author is saying, on closer examination. The two points I took away are that some types of job (such as CEO) are their own particular type of difficult, and that some people are more widely informed than others.

Delving in a little, it seems entirely possible that the people he's describing (CEOs, venture capitalists, and to simplify wildly, the sort of people who get invited to those gatherings) are by necessity smart and skilled in a particular way: of being widely read, and connecting the dots, often on their feet. You could hypothesise that a deep specialist such as Donald Knuth would not seem as impressive in this company, by the author's assumed definition.

Having worked through that, possibly the reason this article struck a nerve is simply the very personal realisation that it's easy to fall too far on the "good enough" side. It's a timely kick-in-the-pants that in my current career I might be better served studying more on systems-thinking, since these days I don't find the time to specialise much. It's also a reminder to assess how effective my current practices are. For example, my chess improvement was limited because I didn't actually play enough², probably because ego got in the way and it's easier to open another book. In general while I love diving in and absorbing new topics, I often tend to focus on that aspect instead of critically analysing or thinking of wider implications, or sometimes even of exercising the new knowledge.

All in all, I doubt I'll ever get invited to those parties, but I enjoyed the thought-process the article prompted!

or; why I still use isearch over ctrlf

I heard you like builds so I built inside my build

There were plenty of useful suggestions, but lacking for me was any discussion of exactly how use-package works, so this article attempts to explain the ones I found most useful.

ELISP> (macroexpand '(use-package foo))
(progn
  (defvar use-package--warning146
    #'(lambda
        (keyword err)
        (let
            ((msg
              (format "%s/%s: %s" 'foo keyword
                      (error-message-string err))))
          (display-warning 'use-package msg :error))))
  (condition-case-unless-debug err
      (if
          (not
           (require 'foo nil t))
          (display-warning 'use-package
                            (format "Cannot load %s" 'foo)
                            :error))
    (error
     (funcall use-package--warning146 :catch err))))

  (if
      (not
       (require 'foo nil t))
      (display-warning 'use-package
                       (format "Cannot load %s" 'foo)
                       :error))

ELISP> (macroexpand '(use-package foo :defer t))
(progn
  (defvar use-package--warning147
    #'(lambda
        (keyword err)
        (let
            ((msg
              (format "%s/%s: %s" 'foo keyword
                      (error-message-string err))))
          (display-warning 'use-package msg :error))))
  (condition-case-unless-debug err nil
    (error
     (funcall use-package--warning147 :catch err))))

ELISP> (macroexpand '(use-package foo :defer 1))
;...
      (run-with-idle-timer 1 nil #'require 'foo nil t)
;...

  (use-package foo
    :bind ("C-cd" . foo-bar))

  (progn
    (unless
        (fboundp 'foo-bar)
      (autoload #'foo-bar "foo" nil t))
    (bind-keys :package foo
               ("C-cd" . foo-bar)))

  (use-package foo
    :mode "\\.asdf\\'")

  (progn
    (unless
        (fboundp 'foo)
      (autoload #'foo "foo" nil t))
    (add-to-list 'auto-mode-alist
                 '("\\.asdf\\'" . foo)))

  (use-package foo
    :mode ("\\.asdf\\." . foo-mode))

  (use-package flycheck-mode
    :hook prog-mode)

  (progn
    (unless
        (fboundp 'flycheck-mode)
      (autoload #'flycheck-mode "flycheck-mode" nil t))
    (add-hook 'prog-mode-hook #'flycheck-mode))

  (use-package lorem-ipsum
    :commands (lorem-ipsum-insert-sentences
               lorem-ipsum-insert-paragraphs))

(use-package dired-quick-sort
  :after dired)

  (eval-after-load 'dired
    '(if
         (not
          (require 'dired-quick-sort nil t))
         (display-warning 'use-package
                          (format "Cannot load %s" 'dired-quick-sort)
                          :error)))

  (use-package dired-quick-sort
    :commands (dired-quick-sort)
    :config (dired-quick-sort-setup))

  (progn
    (unless
        (fboundp 'dired-quick-sort)
      (autoload #'dired-quick-sort "dired-quick-sort" nil t))
    (eval-after-load 'dired-quick-sort
      '(condition-case-unless-debug err
           (progn
             (dired-quick-sort-setup)
             t)
         (error
          (funcall use-package--warning206 :config err)))))

At work I usually have some mindless background noise via Soundcloud in a browser tab. That does the job against office noise, but it's slightly inconvenient having to switch to that tab just to pause.

Recently I learned that browsers should support media keys out of the box, just like a regular media player, without having to switch to the tab! I'm a hair-shirt wearing Linux user however, and occasionally you don't get the just-works integration with every new bit of kit that other OS users may enjoy, so I wasn't too surprised that it didn't work for me, and forgot about it. Until, that is, I was booted into Windows and suddenly realised that it did just work, and regained some motivation to investigate.

Anyway, the short version is: it more or less does just work™, but you need to decide how to wire it up. There are two parts to the puzzle; MPRIS and AVRCP.

#Play/pause
"playerctl play-pause"
    m:0x0 + c:209
    XF86AudioPause 

#Skip Forward
"playerctl next"
    m:0x0 + c:171
    XF86AudioNext 

#Skip Back
"playerctl previous"
    m:0x0 + c:173
    XF86AudioPrev

The thing is… I hardly ever use pinboard! Bookmarking just isn't on my radar these days; that's what search-engines are for.

What I've long wanted though is some sort of serendipity in the service. I don't have a problem locating something again if I remember what it was, but I have no idea what I've since forgotten about. As I discovered in my last post though, now I know that it's a fairly simple job to implement a "random links from your past" email.

In readiness, I've tried to be a lot more cavailier about bookmarking anything that seems remotely interesting. I have a few "interesting links" emails in my Inbox these days (Digg, Refind, etc) that try to use social cues or AI to find interesting stuff… now I have one more, but based on stuff I've personally and explicitly already marked as interesting!

Feel free to clone the repo and run your own. It should run out of the box with a few secrets configured.¹

Overview

name: Harvest Budget Summary Report
on:
  workflow_dispatch:            # Enable manual invocation for testing
  schedule:
    - cron: "0 22 * * THU"      # 8am Fri, AEST

      - run: python projects_status.py
        env:
          HARVEST_ID: ${{ secrets.HARVEST_ID }}
          HARVEST_TOKEN: ${{ secrets.HARVEST_TOKEN }}
          ACCOUNT_EMAIL: ${{ secrets.ACCOUNT_EMAIL }}

      - uses: dawidd6/action-send-mail@v3
        with:
          server_address: smtp.gmail.com
          server_port: 465
          username: ${{ secrets.SMTP_USERNAME }}
          password: ${{ secrets.SMTP_PASSWORD }}
          subject: "[Harvest] Project Budget Tracking"
          html_body: file://email.html
          from: ${{ secrets.ACCOUNT_EMAIL }}
          to: ${{ secrets.ACCOUNT_EMAIL }}

The 2FA setup makes automation difficult, or at least so I assumed, but the other day I finally managed to solve it.

I am very happy and paid-up user of BitWarden (my vault contains over 350 passwords), and my usage in this scenario would typically involve switching over to my browser, opening the extension then searching for and copying the password, back to my terminal to paste it in, back to the browser to search for and copy the TOTP code, and finally back to the terminal again to finalise login. It beats remembering and typing, but it’s still frustrating.

BitWarden has a number of clients, including for the command-line, but because the vast majority of my usage is browser-based I’d never bothered downloading it. It did seem like the perfect candidate to automate my password-entry papercuts though.

Bitwarden Usage

Installation of the CLI is straightfoward, either via npm or several package-management options.

Usage is also fairly simple:

• bw unlock will unlock your vault (and bw status reports several statistics included locked status).
• Unlocking outputs a line to paste into your shell that sets an environment variable for the current session.²
• bw get password <item-id> will print the plain-text password for an item. You can of course search for items to find this id.

Output is in JSON, if you need to script anything in conjunction with jq for example.

As a bonus, you can enable shell completion via the tool itself:

eval "$(bw completion --shell zsh); compdef _bw bw;"

First attempt

When I was noodling this idea around my first thought was some variant of expect.

However: expect is designed around automating the entire session, whereas all I want is to be dropped into a shell with less typing. So, that wasn’t going to work.

Second, successful, attempt

SSH goes to some lengths to avoid this type of automation, but it is feasible. There is a tool that allows several methods of password entry (file, CLI, environment variable) and passes it to SSH: https://sourceforge.net/projects/sshpass/

Disclaimer: Possibly like you, I saw “sourceforge” and also assumed it must be abandoned. However it does appear that the maintainer, while busy, is interested in considering modern needs for the tool (just not adding patches that solve one specific case but not the general requirements).

As of now it doesn’t support TOTP codes though, but there are numerous forks with varying approaches; I settled on https://github.com/dora38/sshpass

This variant supports TOTP codes by specifying an executable file which generates a code. Ours just needs to be a script containing bw get totp <id>.

I didn’t want to have a file containing my password, even temporarily, so my first instinct was to use bw to pipe the password in; eg

bw get password $passwordid | sshpass....

This doesn’t work because ssh then realises that stdin is redirected and doesn’t open a pseudo-tty.

So, my ultimate solution was to use an environment variable, just for that invocation (-e to indicate it should look for a variable, and you probably also want to configure the password and TOTP prompts):

SSHPASS=$(bw get password $passwordid) sshpass -e -c generate-totp ssh user@bastion.host

Success!

Follow-ups

Right now this is what I do, and it’s only a control-r away (even easier to search for since I’ve started using fzf). I did look at turning it into a command for the purposes of this post, but there were a few issues:

• an alias is problematic because of the amount of quoting, and not wanting the get-password sub-shell to be prematurely evaluated.
• A shell function would be ideal, but… doesn’t give me an interactive prompt at the end.

I’m sure it’s solvable, but for a quick post I didn’t invest enough effort — sorry!

This blog is written in Hugo. The source is hosted on Gitlab, and deployed automatically by a Gitlab Pipeline when I push a new post. This is a fairly standard setup, involving running Hugo and then rsync-ing the build to my server.

There was still one aspect that I wanted to improve, and couldn’t find much assistance on: best-practice on locking down the account used by rsync, in case the ssh key is ever compromised. It should obviously be a dedicated account, with no unecessary privileges and only used for that purpose, but that still leaves our hypothetical scenario of escalation following a regular ssh entry.

I have previously used OpenSSH’s internal-sftp to set up an SFTP server that can’t be used for general shell access, so I was asking: what is the equivalent for rsync?

Rush

The answer seems to be: GNU Rush! (For “Restricted user shell”, which gives you an idea of its purpose)

I initially found the examples a bit hard to follow, and furthermore the syntax in the online manual is a bit different from the version included with the Ubuntu 20.04 LTS system I was using it on.

So, two useful points to get started:

• The manual you want (on similar systems) is this one; and
• The first line in your config should probably be debug 3, because any errors are otherwise hard to spot. Output will be in /var/log/auth.log by default.

How it works

Firstly, set the CI user’s shell to /usr/sbin/rush.

The configuration file (/etc/rush.rc) contains a number of rules, which consist of conditions defining the commands to match on, and transformations to undertake.

Conditions can include regexp matches over the whole command line, or individual parts. By default, only one rule matches unless you specify fall-through. The default configuration uses a fall-through rule to set some ulimits and a restricted environment-variable profile.

Transformations can include forcing a particular working directory (possibly involving chroot), and manipulating paths.

Rsync is a common use-case, and the examples provided should get you started.

Warning: the default configuration probably will not work out of the box! This is because it refers to paths such as /srv/rush that you probably don’t have on your system. Increasing the debug verbosity as recommended above will help you track this down, but it is still a lot to wade through.

A simple example

We can see this in action with a very simple local example. We will create a user with the Rush shell, and only allow them to run the “ls” command, and furthermore only in the /tmp directory.

apt install rush
adduser --shell /usr/sbin/rush --disabled-password rushdemo

Simplify /etc/rush.rc to the following:

debug 3  # change back to 1 when you're happy

rule ls-demo
  command ^ls
  set /bin/ls
  chdir /tmp

Now we can try it out:

$ sudo -u rushdemo -i ls
... /tmp listing elided...

Success! Now try another directory:

$ sudo -u rushdemo -i ls /etc
... /tmp listing elided...

Also success! Or is that really what we wanted? We can do better; it is more likely that you want to block attempts to list any other directory. We’ll do this with 2 rules; the first to match legal commands, and a second to reject anything else.

rule ls-demo
  command ^ls$
  set /bin/ls
  chdir /tmp

rule ls-trap
  command ^ls
  argc > 1
  exit fatal: arguments not allowed

Note that we’ve changed the first rule to match only on “ls”. The second will match anything else that starts with ls, and has more than one component (the argc) line. If this rule matches we exit with an error:

$ sudo -u rushdemo -i ls
... /tmp listing elided...

$ sudo -u rushdemo -i ls /etc
fatal: arguments not allowed

Now we’re in business!

TL;DR Docker + disk-space issues meant I had to be a little bit creative, or at least careful.

This blog is hosted on a small DigitalOcean droplet, which I also use for a personal Nextcloud instance.¹

There’s some redundancy already built into this; my files are replicated across a couple of devices aside from the server, but it’s not ideal either. A theoretical attack that deleted my files would get propagated across those same devices, I suppose.

So, backups. DigitalOcean offer “Spaces”, their implementation of object storage with an S3-compatible API, which seems ideal. My threat model at this stage does not include their data centre being razed.

The S3 API means I can use something like s3cmd to upload snapshots. In theory, I could just tar up the data directory and upload it via s3cmd.

In practice, there are a couple of issues that motivated this post in case anyone else has similar constraints:

• Nextcloud is running via Docker, meaning the data directory is slightly abstracted
• I don’t have a large server, so Nextcloud occupies > 50% of disk space and there’s no room for that tar file! I’ll have to figure out streaming as well.

Solution

Preparation

Minor note, but we want to put Nextcloud into maintenance-mode for the duration. The following command will do that:

php occ maintenance:mode --on

I also install a trap-EXIT in the shell script to disable maintenance mode at the end.

Docker

Well… docker-compose, which is another minor layer of abstraction to thread through. The command above would actually be invoked by

docker-compose exec -T --user www-data app php occ maintenance:mode --on

(“app” is the name of the docker-compose container running nextcloud; there’s also one for the database, and the server)

Accessing the data was something I tried a few things around, and it’s possibly not the best solution. I feel like you should be able to back up the volume in a portable way, but for now I rely on invoking tar directly from within the container, streaming to stdout.

That final point — “stdout” — is also a bit tricky; the secret is the -T argument to exec, so we wind up with:

docker-compose exec -T --user www-data app tar czf - data

(and something similar for the database). The -T disables the pseudo-TTY that otherwise gets allocated.

Uploading, and gotchas

Now we have a stream of data to upload. From memory the documentation wasn’t great, but s3cmd can operate in this way, as long as you specify the object name, so you end up with something like (note the “-” in both commands):

tar cvzf - ... |
s3cmd put - s3://mystoragebucket/backup_file

And we’re almost done. But we don’t want to back up everything for ever more! So you need to configure a lifecyle for your backups, so they will expire (and also, so I don’t tip over the included storage quota).

As it turns out, I did tip over the edge (only by $4), because there’s a trick I wasn’t aware of. Your lifecycle can configure a prefix, which determines the files your expiration policy applies to. I have everything under /backups, so I tried:

<Prefix>/backups</Prefix>

This did not work! Sparing the colourful language that ensued, it turns out that you don’t want to include the first /. I needed

<Prefix>backups</Prefix>

(In fact, that bucket is only used for backups so now I just use <Prefix />, but note that <Prefix>/</Prefix> has the same problem). I’m assuming the API refers to object names, and I’ve been thinking in terms of directory structures.

Putting it all together

To wrap up, this is the shell script (as an Ansible template) that I install, and then gets invoked by cron:

#!/bin/bash

set -eufo pipefail

DB_BACKUP=nextcloud-db-$(date +%Y%m%d-%s).dmp
DATA_BACKUP=nextcloud-data-$(date +%Y%m%d-%s).tgz

# maintenance mode off again, even if we error out:
finish() {
    docker-compose exec -T --user www-data app php occ maintenance:mode --off
}

trap finish EXIT

cd /srv/www/nextcloud

# maintenance mode on
docker-compose exec -T --user www-data app php occ maintenance:mode --on

# docker-compose to tar to output

docker-compose exec -T --user www-data app tar czf - data |
s3cmd --access_key={{ storage_apikey }} \
--secret_key={{ storage_secret_key }} \
--region={{ storage_region }} \
--host=digitaloceanspaces.com \
--host-bucket="%(bucket)s.{{ storage_region }}.digitaloceanspaces.com" \
put - s3://{{ storage_bucket_name }}/backups/$DATA_BACKUP

# db backup
docker-compose exec -T --user postgres db pg_dump -Fc nextcloud |
s3cmd --access_key={{ storage_apikey }} \
--secret_key={{ storage_secret_key }} \
--region={{ storage_region }} \
--host=digitaloceanspaces.com \
--host-bucket="%(bucket)s.{{ storage_region }}.digitaloceanspaces.com" \
put - s3://{{ storage_bucket_name }}/backups/$DB_BACKUP

I could certainly do with the practice in writing. Mostly though, it’s an incentive to start exploring again: when I first got into computers, I grew up on Linux systems and really appreciated the simplicity and documentation available. I felt that I could run ps, and see 20-ish processes and know what each one of them did.

These days, I’m still most at-home on a minimalist Debian derivative, but even there I don’t recognise most of the output I see in a ps dump. I maintain a bunch of Linux servers at work, yet I’m increasingly aware of the topics and sub-systems that I never learned in the same way as when I hoovered up that early knowledge.

One of my nerd-heroes is Julia Evans. Her willingness to learn out in the open and enthusiastically document it all, from deep topics to “TIL” tidbits, is incredibly refreshing. I’ve bought a bunch of her zines, and even though some are topics I know in principle I’ve gained something of value from each and every one.

In the same vein I’ve been collecting a range of topics I want to understand, or understand better, or just topics where it may be beneficial to have one more problem-solution document out there. Some are quickies for when I have time, while others (possibly most) could turn into research deep-dives. There’s no pre-defined schedule, or even a plan to publish either incrementally or post-hoc, just — at least for now — an enthusiasm to get started again!

To date, I have gone through:

• Posterous (it’s dead now, but it was a simple service that you emailed posts to)
• Jekyll-bootstrap; also more or less dead or at least abandoned, but offered some structure around the Jekyll static-sites engine.

An advantage of Jekyll at the time was its support by Github-pages, which was where I moved after migrating from Posterous. I’ve found it’s a bit clunky though, I don’t want to invest the time in understanding the jekyll-bootstrap templates enough to do any hacking on them, and the community has moved on.

In addition, hosting has migrated from Posterous, to Github-pages, to my own server. Arguably I have spent more time hacking on the tech than actually posting. Hugo templates may also take some learning, but it’s nice and fast and I’m happy with the current template.

Process

I’m not going into details this time around, but I’ll share some notes in case anyone wants to follow up.

There was an initial import script, both built-in and a few python scripts I experimented with, but they mostly tweaked the front-matter a bit. I used a few sed scripts to convert some syntax from jekyll to Hugo, as well as just fixing some manually.

Deployment was a chance to practice some stuff. I’ve switched to Gitlab for repository-hosting, so I set up a pipeline there to build the site, added a section to my Ansible playbook to download that and install it (as a bootstrap measure), then additionally added a stage to the Gitlab pipeline to push after build for ongoing posting without involving Ansible as well. There were a few interesting steps there around locking it down that I may write quick posts about.

To Do

It’s not perfect. The main thing is the galleries are currently dead, which is unfortunate as the original purpose of setting up a blog was to share travel pictures. They were hosted on Picasa — also dead — though, so until I figure out how to embed Google-photos galleries most posts will remain devoid of colour.

Some search-engine links have broken, and I’ve only just realised that there’s a mix of /yyyy-mm-dd-title/ urls, and hierarchical /yyyy/mm/dd/title/, but I think I can live with that now that it’s out there.

• Idiomatic XML processing in Clojure is possible and even easy, but under-documented;
• Namespace-XML support seems to be particularly hidden;
• It works naturally by mapping to namespaced keywords.

The Details

I still occasionally find myself writing XML processing code in Clojure, either for production or one-off data munging tasks. XML support is quite good, and quite idiomatic, but I’ve found that getting started is often confusing. Is data.xml all you need? Where does data.zip fit in?

The short explanation is that the first function you are likely to find returns a Clojure structure with maps representing elements, but then there’s a similar function that is lazy, and there’s also a zipper library with some awesome processing functions that is probably what you want, at least on the input side. The best overview I’ve found is this blog post.

My code usually starts off something like this:¹

(:require [clojure.data.xml :as xml]
            [clojure.data.zip.xml :as zx]
            [clojure.java.io :as io]
            [clojure.zip :as zip])

(-> filename io/resource slurp xml/parse-str zip/xml-zip)

which you can then process using the data.zip.xml utilities, generally starting with either xml-> or xml1-> (the latter just produces a single result), and using keywords to traverse down tag names:

(zx/xml-> zipped-xml :RootElement :SomeChild custom-processing-fn)

(where custom-processing-fn is probably returns a structure out of the <SomeChild> element)

The problem is: what if your document is namespaced, such as <myns:SomeChild>? Fortunately, it turns out that you can map these exactly to Clojure namespaced-keywords. The trick is the alias-uri function:

(xml/alias-uri 'sld "http://www.opengis.net/sld")

and then you can proceed exactly as before, but with namespaces:

(zx/xml-> zipped-xml ::sld/StyledLayerDescriptor ....)

Leiningen is an awesome part of the Clojurescript landscape, doing live reloading of code and CSS. It even offers a static-file server, if you happen to be developing a client-only application.

Less appreciated is that you can plug in your own Ring handler. A particularly convenient use-case that I’ve recently discovered is to plug in a proxy server, which has uses beyond client-only applications.

There’s three main use-cases justifying it for me:

• We frequently use a Django API backend so it would normally make sense to use the Django development server, but on Windows this is unbearably slow (with the number of files generated by Clojurescript in dev-mode). Our previous solution involved nginx, which works but is a more complicated set-up.
• If you are using a third-party service that hasn’t enabled CORS yet, you can proxy to it from your local server instead.
• If you are still doing client-only development, but need to assume an AJAX service will be available on the eventual host, you can still refer to it using server-less URL paths and simply proxy to the test service.

How-To:

There is a sample project in the leiningen repo, but essentially all you need to do is define your handler, and specify it in the :ring-handler entry of your :figwheel configuration in project.clj .

You have a few options for Ring proxy handlers, but the one I went with is this one: tailrecursion/ring-proxy.

To use it, grab the leiningen sample and extend it for your needs:

(ns com.etbwc.dev-handlers
  (:require [compojure.core :refer [defroutes GET]]
            [compojure.route :as route]
            [ring.middleware.defaults :refer [wrap-defaults site-defaults]]
            [ring.util.response :as response]
            [tailrecursion.ring-proxy :refer [wrap-proxy]]))

(defroutes app-routes
  ;; NOTE: this will deliver all of your assets from the public directory
  ;; of resources i.e. resources/public
  (route/resources "/" {:root "public"})
  ;; NOTE: this will deliver your index.html
  #(GET "/" [] (-> (response/resource-response "index.html" {:root "public"})
                  (response/content-type "text/html")))
  (route/not-found "Not Found"))


(def app (-> app-routes
             (wrap-defaults site-defaults)
             ;; Use a django API on a different localhost port:
             (wrap-proxy "/api" "http://localhost:8000/api")
             ;; Use the django static-files if you need for eg the admin site:
             (wrap-proxy "/static" "http://localhost:8000/static")
             ;; Proxy requests to a third-party API without CORS enabled:
             (wrap-proxy "/pq" "http://third.party.com/pq")))

Now to use the handler simply mention it in project.clj:

(defproject "etbwc-example" "0.1.0-SNAPSHOT"
  ;;...

  :figwheel {:http-server-root "public"
             :server-port      3451
             :nrepl-port       7892
             ;; Boom:
             :ring-handler     com.etbwc.dev-handlers/app
             :css-dirs         ["resources/public/css"]}

  ;;...
  )

That works fine, but it does mean you have to manually invoke that function. You could run the function on load, but then you have to remember to start git-bash first.

It’s not an ideal situation, and the other day I finally wondered if Emacs had any file-watching library… of course it does.

(defvar ssh-file-watcher nil)

(when (require 'filenotify nil t)
  ;; We're loading up, slurp SSH envs in case in they're already
  ;; correct:
  (mh/parse-sshagent-env)
  ;; Clear any existing watcher:
  (if ssh-file-watcher
      (file-notify-rm-watch ssh-file-watcher))
  ;; Create the new watcher:
  (setq ssh-file-watcher
        (file-notify-add-watch (expand-file-name "~/.ssh/agent.env")
                               '(change attribute-change)
                               'mh/parse-sshagent-env)))

In other words, another 5 minutes of hacking and now we simply load the variables just in case, clear any existing watcher, and start watching the file for changes — when it does, we simply slurp again. If you follow this verbatim, the only addition you’ll need to make is to ensure that your callback function accepts the arguments filenotify will give it; I just ignored them: (defun mh/parse-sshagent-env (&rest ignored)...

Much more painless!

Creating circumstances, typically by omission of information, through which another person is more likely to fail.

I’m perhaps overly dramatising since I was first introduced to it when I said “Meet me at Cafe X” when there were in fact two of them; none the less, I believe it deserves to be in common usage.

Unfortunately, I was witness to a much more serious example only yesterday. I’ve been doing some work with Microsoft’s SQL Server, working on a local copy of a database (importing data, writing procedures, and so on). It came time to reconcile some changes, and to be cautious we decided to do it manually, using the backup-and-restore feature to copy my database to the destination server.

The Management Studio has a nice interface to backing up a database, pretty much as you’d expect a GUI environment to provide: right-click, select “backup” and a save location, and you have a single portable file.

Restoring from this file is a little less intuitive, but the 2012 update made it much, much worse.

By default, when you restore a file it will restore the entire database, to a database of the same name (granted, probably the main use-case). In our case though we were going to restore to a temporary database on a different server, and manually copy selected changes across to the original that way.

The process is basically to create a new database, right-click and select “restore”… and now the fail environment emerges. You have to switch to a different tab to find your backup file, and when you select it the destination database silently changes, on the first hidden tab, to the one with the same name as the original database, NOT the temporary one you just selected. My colleague doing this is very competent and experienced and actually knew this complication, but while we were checking something else he forgot…¹ and overwrote the last three days of his work with mine.

I call that a fail environment.

Naturally, there is a markdown-mode for Emacs, and it’s quite good.

I did run into one issue though, due to interactions with Liquid syntax. By way of example, this is the syntax for a link in markdown:

[anchor text](http://www.google.com)

(Which renders as “anchor text”). Jekyll also provides a useful Liquid tag to link to previous posts, for example:

[older post]({% post_url 2014-05-20-parsing-ssh-agent-variables-in-emacs %})

This renders as “older post”.

The issue occurs in conjunction with paragraph filling. Both markdown and liquid require the examples above to be unbroken on a single line. Markdown-mode does the right thing and ensures the anchor text won’t be broken, but also correctly assumes that the URL won’t have spaces and thus will be safe. In conjunction with liquid though, it’s possible to have a situation like this, which will break Jekyll:

[older post]({% post_url
2014-05-20-parsing-ssh-agent-variables-in-emacs %})

Creating a fix requires some understanding of how filling commands work.² Roughly, when a command like fill-paragraph is invoked Emacs starts at the beginning of a line, jumps forward to the fill-column, and then looks for white-space at which it can insert a newline. The wrinkle is it first checks if it is safe to break there, using a customisable variable called fill-nobreak-predicate. To get an example, we can look at how markdown-mode ensures it won’t break inside the square brackets of a link definition:

(defun markdown-nobreak-p ()
  "Return nil if it is acceptable to break the current line at the point."
  ;; inside in square brackets (e.g., link anchor text)
  (looking-back "\\[[^]]*"))

As you see, it just looks back to see if there’s an opening bracket (that isn’t followed by a closing one). We can use the same strategy to check if we’re in a liquid tag:

(defun mh/liquid-nobreak-p ()
  (looking-back "({{ "{%" }}[^%]*"))

Notice that we’re not worried about the anchor-text case here! That’s because fill-nobreak-predicate is a hook variable; that is, a list of functions, and in this case invoked until one of them returns non-nil, indicating that we shouldn’t break at this point.³ In other words, we want to add to this list, not replace or advise anything.

So now to incorporate our addition we just use the mode hook, and remember to also use add-hook with our new predicate:

(add-hook 'markdown-mode-hook
          (lambda ()
            (add-hook 'fill-nobreak-predicate 'mh/liquid-nobreak-p)))

And now we’re done, and can get back to blogging.

A variation of this ambiguity still exists; date-parsing functions such as strptime have to deal with it all the time. The standards² pick a pivot value—69—and everything on or after that is interpreted as 19XX, everything before as 20XX.

For example:

#include <stdio.h>
#include <time.h>

int main(void)
{
    struct tm result;
    char buf[255];

    if (strptime("68", "%y", &result)) {
        strftime(buf, sizeof(buf), "%Y", &result);
        puts(buf);              /* 2068 */
    }

    if (strptime("69", "%y", &result)) {
        strftime(buf, sizeof(buf), "%Y", &result);
        puts(buf);              /* 1969 */
    }

    return 0;
}

By my reckoning, this means we are heading for another crisis in about 45 years.³ I suggest you start stocking your bomb shelter with non-perishables now.

Overview

Essentially, we need to do three things:

1. Find the git SHA of the working copy;
2. Embed this somewhere useful;
3. Make sure it does this automatically before you deploy.

^ The last step is the easiest: you just run it as a pre-build hook.

How to display it

The closest description I could find of what we envisaged was this article.² He outlines how to update the deployment label³ in the .azurepubXml profile, which was exactly what we were looking for. We eventually abandoned that aspect though because when you deploy directly from Visual Studio the profile is read before you hit “publish”, or in other words before your pre-build script has run and done anything useful.

Instead, we just write it to an HTML template that can be sourced anywhere: in a comment in every page, or on the administrators-only dashboard, or in the footer… it’s up to you.

The catch of course is you will now have a modified-file showing up in your git status every time you build. The solution to this is two-fold:

• Add the file to your .gitignore, and
• Change your .csproj file to only include the file if it exists.

For the latter, you will need to edit it by hand:

<Content Include="Views\Shared\_GitRev.cshtml"
         Condition=" Exists('Views\Shared\_GitRev.cshtml') " />

As a bonus third step, you might want to think about how to gracefully fail if that template doesn’t exist for some reason. Assuming razor templates, an extension method is a useful approach:

public static MvcHtmlString PartialIfExists(this HtmlHelper html, string viewname)
{
    var controllerContext = html.ViewContext.Controller.ControllerContext;
    var result = ViewEngines.Engines.FindView(controllerContext, viewname, null);

    if (result.View != null)
    {
        return html.Partial(viewname);
    }
    else
        return MvcHtmlString.Empty;
}

Then your templates might contain something like

...
<!-- @Html.PartialIfExists("_GitRev") -->
...

Attempt 1: git.exe and powershell

The first version used the git executable to find the revision:

git rev-parse HEAD

A simple powershell script ties it together, and we’re in business:

if (Get-Command "git.exe" -ErrorAction SilentlyContinue)
{
    $revision = git rev-parse HEAD

    # make sure we don't include old versions if for some reason we
    # have one, but now can't find git.exe:
    $template = "$PSScriptRoot\..\Views\Shared\_GitRev.cshtml"
    if (Test-Path $template) {
        Remove-Item $template
    }

    Set-Content $template $revision
}

For completeness, it’s also worth pointing out here that it may be necessary to enable script execution for powershell permissions. I also include this line in my build hook:

powershell -Command "Set-ExecutionPolicy -Scope CurrentUser Unrestricted"

Attempt 2: Powershell-Native

That worked, but it introduces complications such as the need for git to be in your path, and generally feels a little fragile.

Luckily, I remembered libgit2, a portable standalone implementation of core git functionality. There are .NET bindings and nuget handles installation so this was much more promising:

try {
    # Note use of wildcard to ignore the version.
    Add-Type -Path $PSScriptRoot\..\..\packages\LibGit2Sharp*\lib\net35\LibGit2Sharp.dll

    $repo = New-Object -TypeName LibGit2Sharp.Repository -ArgumentList "$PSScriptRoot\..\..\..\.git"

    $template = "$PSScriptRoot\..\Views\Shared\_GitRev.cshtml"
    if (Test-Path $template) {
        Remove-Item $template
    }

    $revision = $repo.Head.Tip.Sha
    $dirty = ( $repo.Index.RetrieveStatus() | ? { $_.State -eq "Modified" } ).Count -gt 0

    if ($dirty) {
        $revision = "$revision*"
    }

    Set-Content $template $revision
    Write-Host "Labelled with $revision"
}
catch
{
    Write-Host "Exception during git-rev labelling; ignoring"
}

We use Add-Type to load the DLL (the hard-coded paths here feel slightly dirty, I admit, but we can use a wild-card to mitigate against version changes at least), then instantiate the Repository object. The SHA can then be accessed with $repo.Head.Tip.Sha. The next line also checks if any files are modified, a tweak which wasn’t present in the previous version.

Aftermath

This is the script now running as part of our builds. It works like a charm, but looking back it feels like more work than it should have been. I’m still relatively new to the Microsoft environment and tool-chain, so if there’s an easier way of doing this that I’m overlooking please let me know!

The only problem was I still couldn’t fetch or push using magit, because it didn’t have the ssh-agent environment variables. This was because I wasn’t launching Emacs from a shell, but fortunately I remembered that the bit of shell code I’d cribbed from github to set up ssh-agent there also handily left them in a file.

The file looks like this:

SSH_AUTH_SOCK=/tmp/ssh-WOAJeX1728/agent.1728; export SSH_AUTH_SOCK;
SSH_AGENT_PID=4880; export SSH_AGENT_PID;
echo Agent pid 4880;

After realising this it was just a few minutes of rolling up the sleeves and writing some elisp to parse this and set the variables locally:

;;; Set variables for ssh interaction, assuming they exist in a file ~/.ssh/agent.env.
;;; See https://help.github.com/articles/working-with-ssh-key-passphrases
(defun mh/parse-sshagent-env ()
  (interactive)
  (with-temp-buffer
    (insert-file-contents (expand-file-name "~/.ssh/agent.env"))
    (dolist (varname '("SSH_AUTH_SOCK" "SSH_AGENT_PID"))
      (goto-char 0)
      (re-search-forward (concat varname "=\\([^;]+\\)"))
      (setenv varname (match-string 1)))))

It’s very quicky and dirty, but it works for me:

• slurp the file into a temporary buffer;
• for each of the variables we’re interested in, look for its value (the regexp simply grabs everything in between the equals and the following semi-colon);
• finally, use setenv to make it available inside Emacs.

Of course, I was able to develop this without leaving Emacs once; searching the documentation to remember how to access captured groups, testing the regexp, incrementally developing and finally “installing” the function and getting back to work. I’m sure this particular hack has been done many times before, but it was just as much fun and didn’t take much longer to do it myself—this is why I love working in Emacs!

The other day I came across the old subversion repository for my thesis¹, and thought it would be a good idea to convert it to git and back it up somewhere. Immediate problem: I couldn’t even read it! This post is a short description of what the problem was, and how it was resolved.

The Problem

The repository uses the old (deprecated) BerkelyDB file-system type. Let’s try and browse it:

$ svn ls file:///home/mark/_SVNREPOS/thesis
svn: E180001: Unable to connect to a repository at URL 'file:///home/mark/_SVNREPOS/thesis'
svn: E180001: Unable to open an ra_local session to URL
svn: E180001: Unable to open repository 'file:///home/mark/_SVNREPOS/thesis'
svn: E160029: Berkeley DB error for filesystem '/home/mark/_SVNREPOS/thesis/db' while opening environment:

svn: E160029: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
svn: E160029: bdb: BDB2530 Ignoring log file: /home/mark/_SVNREPOS/thesis/db/log.0000000018: magic number 88090400, not 40988
svn: E160029: bdb: BDB2527 Invalid log file: log.0000000018: Invalid argument
svn: E160029: bdb: BDB0061 PANIC: Invalid argument
svn: E160029: bdb: BDB1546 unable to join the environment

Oops. The key is in the fragment “magic number 88090400, not 40988”, which looks suspiciously like an endianness issue (88-09-04-00 / 00-04-09-88). It was created on a PPC Mac and I’m now trying to access it from an Intel box, so that seems like a reasonable explanation.

At this point though I ran into something of a black-hole. It was hard to even find confirmation that it might be possible; “bdb is platform-independent” is the prevailing opinion. Some evidence did exist but was otherwise thin on the ground, which is one reason I’m writing this. One helpful post mentioned that while the main database is platform-independent the log files are not, and hence my problem.

Solution

I no longer had access to a PPC machine, but the solution came when a friend pointed out that QEMU can virtualise big-endian architectures, even on a little-endian host. After that, it was just a matter of implementation details.

Create a PPC guest

With this post as a guide, grab a pre-configured Debian image, and apply the openbios fix as well. Install subversion on it (apt-get install subversion).

Copy the repository onto the guest

QEMU doesn’t provide out-of-the-box shared folders (well, apparently it has a built-in SMB server, but I didn’t investigate), but it does make the host addressable on 10.0.2.2 which is all we need. Tar up the repository and run python -m http.server² from the same directory on the host, then from the guest download it with wget http://10.0.2.2:8000/repo.tar.

Migrate the repository

The reason we’re here; we’ll convert it to the FSFS format. This is actually slightly fiddly; essentially you create a new repository in the FSFS format, dump from your old repository and import into the new one. Assuming it has been extracted to oldrepo:

1. svnadmin create ./fsfsrepo --fs-type fsfs
2. svnadmin dump ./oldrepo -q | svnadmin load ./fsfsrepo

Copy the recovered repository back to the host

Once again there are probably smoother ways, but I used a quick-and-dirty approach (good old netcat!):

1. (from the host) nc -l -p 8000 > fsfsrepo.tar
2. (on the guest) tar cvf - fsfsrepo | nc -p 8000 10.0.2.2

Import your recovered repository into git

This step is entirely optional of course, but I’m mentioning it for the sake of completeness since that was my goal in this exercise. I used svn2git, which is a wrapper around git-svn.

Conclusion

I hope this helps someone else in the future, and at the very least adds Google-weight to the problem. Thanks to the fine folk on Google+ for their suggestions, and Paul for the QEMU tip!

I flew back to Ontario airport (I’d learned my lesson about LAX), Kevin picked me up again and we met up with Dan for one final meal together before Dan and I hit the road for a late assault on Las Vegas.

Vegas had never been on my itinerary. Based purely on impressions, it seemed like a place that might be fun with the right crew, but flying solo it held little appeal for me. I was still curious though, so after we checked in around midnight I didn’t require much urging to go for a wander up the strip. It is absolutely quite a sight, to say nothing of the human spectacle with girls flicking cards in your face to attract attention and nightclub touts every 15 metres (“Gentlemen! No cover charge tonight… we got drunk chicks inside”).

That was to be the extent of my Vegas experience though! All of about 8 hours, and most of them asleep. The next morning we were on the road again, bound for Zion National Park. We arrived at the town of Springdale in late afternoon, so we kept going for a short drive into the park. This was also the first time I really “understood” this type of scenery. I am used to admiring lush or snow-covered mountains, rain forest scenery, and so on—but when everyone I talked to kept raving about all the “red rock” parks I didn’t really get it. Sure, the rock formations looked cool, but how much of it did you really need to see? Well, standing there in the middle of it, with enormous red sandstone cliffs and formations eroded into being over thousands of years, in colours more vivid than any photograph can ever convey, I did understand why. I was definitely looking forward to exploring properly the next day.

The light was going down though so we headed back to Springdale… only to discover that the hotel we’d booked was actually located on the other side of the park! So, back through that road for a third time, with the people at the gate giving us strange looks.

Over dinner that night I learned that not only does (Mormon) Utah actually have at least one brewery, but they have a sense of humour about it: I washed my food down with a Polygamy Porter.

The following morning we again drove that road through the park for a fourth time, saying good morning to our friends at the gate, and parked at the visitor centre to investigate hikes before catching a shuttle in. We had planned to explore Hidden Canyon, but when that was closed off not far from the entrance we took another path winding up to Observation Point, with glorious views back over the main canyon and river.

My penultimate full day had us take a fifth and final trip through the park, and head along the Arizona border towards Lake Powell. We didn’t have time to see Rainbow Arch, but the boat tour through the lake’s canyons was still striking. Sunset over Horseshoe Bend was even more spectacular (and put Grand Canyon firmly on my list for my next trip, when you realise this enormous bend was a fraction the size of those in Grand Canyon!)

Hitting the road for the final time together, we drove through Monument Valley—an expanse of mesas and buttes that looks just like every cowboy movie you’ve seen, because that is indeed where they were all filmed—towards the tiny town of Durango in Colorado. Dan’s friends there showered us with hospitality, and in the morning even took me on a short hike in the hills. I was really sorry not to have more time in Colorado; mountains still draw me, and there’s a lot to explore there.

Unfortunately, it wasn’t going to happen this trip, for that was the end. Dan drove me to the airport, and I began my 32-hour, 6-flight (of which I had to sprint for 3!), return voyage.

Èl Fin, Un Comienzo

I always pictured writing some final paragraph with my favourite places and memories, but in reality I don’t think I could separate much. I will pick out Alberta as my favourite outdoors experience, but Yosemite and Zion were both very different and equally spectacular. New York was great, Portland was too for very different reasons… but again, there were very few places I wouldn’t gladly revisit and a long list of places I still intend to but couldn’t this time.

As always of course the people made the most difference—so to everyone I met, old and new, and invited back to my home-town… I was serious, come visit!

This post effectively marks the end of this blog as a place for my travel pictures.¹

As I mentioned at the beginning, I quit my job before this trip, and together with a friend I’m now well into something new. That means a lot fewer travel-related posts for now, but on the other hand I do have a long list of posts in technical and business areas I plan to write. Stay tuned! (Or update your subscriptions accordingly)

So I returned Kevin’s truck, spent a few more days in Venice exploring with friends (celebrated my birthday in Silverlake, went to an anti-Monsanto rally, and followed up with chicken-and-waffles breakfast at Roscoe’s—I think I’ve covered LA!), and caught a flight to Portland.

Portland

I’d booked close to a week in Portland, again to decompress and geek out a little, and it really didn’t disappoint. By far the most consistently excellent espresso of any place I’d visited to date, and just a thoroughly comfortable town. I’m from a small town myself, a similar distance from the equator, so while the beautiful sunny weather of the last two months had been great it was still quite nice to feel a nip in the shade, and even to see the town mostly closed as I explored on a Sunday.

I really hadn’t done a lot of research², and all I knew about Powell’s Bookshop was that it seemed to be one of the things to do in Portland. It was only a block from where I was staying, and I like a good bookshop so on my first morning I wandered over.

I would not emerge until well after lunchtime. Powell’s, I discovered, is famous because it’s the largest independent bookstore in the world. It literally covers an entire block with 4 floors, and if you wanted the technical books section… well, that’s across the road in another building, the size of most normal bookshops. They couldn’t fit it in the main store. Obscure topics (say, “interior decoration for attics with sloped ceilings”) that would barely rate a mention on Amazon, let alone your radar, in Powell’s have multiple shelves dedicated to them.

I caught a bus up to the Rose Gardens, only to overshoot the bus stop and find myself speeding past progressively more distant stops. Eventually I got off, and on a whim started walking down a path off the road-side. It not only went exactly where I was intending, but soon became a very pleasant educational trail—winning. The gardens themselves were picturesque enough and provided a fairly nice view back out over the city, with Mount Hood just visible behind.

Portland is justifiably famous for its food trucks, and many are excellent (many others, of course, are not). On my final morning I made one final trip to an earlier favourite (the peoples’ pig, for excellent porchetta sandwiches), and my afternoon bus to Seattle came up in conversation. They immediately told me about a place for cured meats called Salumi, literally right next to where I would get off the bus—score! It even turned out that the place is run by the father of celebrity chef Mario Batali.

Seattle

So I rolled into Seattle and made a bee-line for Salumi… in something of a harbinger of things to come, it was closed (only open to the public a small number of days, and limited hours). Undeterred I checked into my hostel and after a quick chat with room-mates found a pumping sports bar to grab a counter meal and watch the football game. The local team won, spirits were high, and it turned out the people around me were friends with the owner and we wound up doing quite a few shots. The night was rosy but not out of control… until I cheerily rolled back to the hostel, and agreed to go straight back out again with a room-mate who had itchy feet. I fear I may not have fully appreciated the excellent whisky we wound up drinking.

The next morning my good mood from the night before had deserted me and so, it was feeling like, had my persistent good luck on this trip. Slightly underwhelmed by the hostel breakfast I went out, straight into a thick damp fog making it apparent I didn’t have enough warm clothes on, and I immediately jarred my knee stepping down off the curb, with my opposite hip already complaining about lugging my backpack around yesterday. Feeling slightly sorry for myself I figured any coffee would help, and ducked into the first half-way decent looking cafe I came across. The vile espresso I forced down had me cursing that decision too, and I was all but ready to either go back to bed or pass through Seattle entirely.

Like a true soldier though I pushed on, and after exploring the Pike Place Markets for a while I emerged into sunshine, with my headache and other pains gone, and ready to conquer the town.

This time I didn’t fail at Salumi, and enjoyed a tasty lunch in a secluded Japanese waterfall-garden I stumbled upon nearby.

Late in the day I figured I should check out the standard Seattle cliche, the Space Needle. I wasn’t that excited about it, but for once in my experience of “queuing to go up tall things”, the queues weren’t too bad. Not only that, but by the time I was up the top the skies were clear and the light was great… and only then did I realise that if I just stuck around for another half hour or so I would also get some great sunset and night-time shots!

Bellingham

Bellingham was somewhat of a random choice. An early tip from a friend (Dan) was to check out the San Juan Islands, and Bellingham was one jumping-off point for exploring them. Also, and completely randomly, it’s where The Postal Service originate from!

An early bus from Seattle unfortunately made such good time that we were dropped at the terminal an hour early—which was already scheduled to be about an hour before the first bus into town. A long cold wait, two buses and a long walk later and I was at the motel I’d booked, still hours before check-in time but a very friendly owner let me grab breakfast and found a vacant room for me (as well as telling me that the reason they didn’t have microwaves was because drug dealers used cheap hotel rooms as disposable cooking-labs). I made another long, grey walk into town, only to find (it was a Sunday again) almost everything shut. I did find some passable coffee, and stumbled upon a water-side path that took me all the way to adjoining Fairhaven (also shut). On my return home I took a wrong turn and wound up passing through the university campus—also scenic, and rather close to my motel but with no way to cut through so by the time I got home it had been quite a long day, and I had already seen most of the town! I grabbed take-away and started researching ways to occupy the next couple of days.

This time though, my luck really had run out. Hiring a sea kayak? Sorry, the season finished last month. Perhaps whale watching again? Nope, same story. A visit to the San Juans? Actually, it turns out that you can’t go directly from Bellingham any more, and my best option was via Anacortes (the other town I’d been considering), which would be a 3 hour journey each way. Oh well, it looked like this was going to be another chill-out stop! I spent the next day walking the town, trawling record stores, and hacking in cafes. My final day I hired a bike in Fairhaven and hit a trail that eventually cruised through pine forest along the water’s edge, before a steep climb finished with a short lake circuit hike.

I had one more night back in Seattle before flying back to LA, and used the time to find the fantastic bridge troll (also, Top Pot Donuts).

So, rather than heading inland a bit we made a beeline back West to the coast. I dropped my Irish hiking buddy off in San Francisco, and headed south to the old capital of Monterey.

Monterey

Monterey was a curious mix; a very touristy small town centre, and otherwise a very relaxed beach-town feel. One of the big attractions is its world-famous aquarium and associated research institute, on my radar from my previous life in another marine research centre. To be honest, I found the aquarium amazing and incredibly well done, but definitely aimed at children.

Big Sur

Just South of Monterey is Big Sur, a coastal area with steep mountains of redwood forest, rising straight up from the ocean. I was planning on moving there next, but having left it to the last minute (a recurring theme) and with no camping gear I decided to stay on in Monterey and just drive down to Big Sur for day trips.

The first day was actually somewhat of a let-down. I began by taking the much-hyped 17-mile drive; basically a gated-commune of golf courses and beaches just outside Monterey. You pay to drive around, and occasionally take photos through the bus-tour crowds while Kenny Rogers plays on some oblivious gentleman’s iPad. Avoid; if you drive just a little further you get much better views, for free. Then it was lunch-time, deep in Big Sur at the also over-hyped and even more over-priced Nepenthe (I was at least prepared for the cost). I did at least manage a short walk, up through redwoods to a pleasant but unremarkable view. The second day started much more promisingly at the iconic McWay Falls, the only coastal waterfall in California. After that I went on a steep hike up the mountains behind it, with a rewarding view at the summit dropping right down to the water.

As an aside¹: I’ve been a fan of the author Jack Kerouac, and it seemed appropriate to read more while road-tripping. Fittingly, there’s even one called Big Sur, so that was night-time reading for this part of the journey. It is, however… rather dark, I warn you! Gone is the carefree and foot-loose innocence, and all that’s left is a descent into alcoholism and depression as he struggles with fame and worries he’s losing his mind. It put a bit more of a sombre mood on this leg than I was planning on.

Highway 1

Then it was time to leave Monterey for good, one last time through Big Sur and keep going down that amazing coastline to LA again. It is a single-lane highway but obviously very popular, and steering the enormous truck I was driving around blind corners knowing how many cycle tourers were around got quite nerve-wracking.

Yet again though I realised how charmed I’d been with the weather: a heavy fog completely blanketed the ocean all morning, making for a great change in atmosphere as I drove down, but probably less appealing for anyone on their first trip. Even the McWay Falls were completely obscured as I drove past, thanking my luck for the perfect weather I’d had the previous 2 days as I did so!

Once the views have eased off a little, your next stop is likely to be Hearst Castle. This place is hard to describe: from the reception building you are escorted up to the castle itself by coach. It’s a bizarre mix of old and new; built within the last century, but with most of the interior coming from the antiques collection of (the late) William Hearst. He even had a private zoo at the time; I saw a herd of zebras, still roaming around, as I was leaving!

I stopped for a couple of nights in San Luis Obispo. It was a small town, pleasant to wander without making undue demands, and in my case a nice communal hostel to stay in.

I made another stop in Santa Barbara. Many people talk about how great it is, and I suspect it would be a fantastic place to live. As it was, I drove in through the opulent strip, stayed in a motel on the beach which turned out to be fairly unexciting, and left the next morning rather underwhelmed. The one saving grace was Dan linking me up with a good friend of his, who was kind enough to meet up and show me a few of the watering holes, giving me some inkling of life in the area. Perhaps next time!

On to Los Angeles

One of the great pleasures of travelling… is meeting other travellers, who can then help you out on your next journey! And so it was; the year before in Patagonia I had met 3 Los Angeles residents: Kevin, understated but with immense resolve (hiking Patagonia while needing both knee and shoulder surgery) and one of the most thoroughly decent men you would ever meet; his friend and counterpoint Dan, ebullient and a brilliant (and constant) story teller; and a beautiful Czech lady with fascinating tales mixing old communist Czechoslovakia and modern Hollywood, Zdenka. A year later I was determined to catch up with all of them again, so despite the city itself having no other appeal to me at the time, it was my obvious next stop.

Dan was out of the country again, but it was great meeting Kevin again and spending more time with him, sharing another hike in the mountains behind LA. Then from suburbia outside LA (an hour and a half of freeway traffic outside, which just appears to be a standard commute), I spent a couple of days in the trendy-hippy beach towns of Venice/Santa Monica with Zdenka, seeing a completely different side of the area.

Yosemite

In so far as I had any plan for this trip, it was to spend roughly 2 months going East across Canada then back West across the US, seeing mostly cities, before touring as much of the amazing national parks on offer in America during my final month. To date I was on schedule, and in yet another display of generosity Kevin loaned me his truck. Swallowing my nerves about driving on the other side of the road—I’d overcome some fear in Alberta, but Californian freeways were another matter entirely—I made a last-minute hostel booking and drove 7 hours up to Yosemite.

The next day, the US Government Shutdown closed the national parks.

Ok, that was a slight exaggeration for dramatic effect. The reality is the next day I had a full day of hiking up to the (bone-dry) Yosemite Falls, where I had lunch overlooking the incredible valley view and watching crows joy-riding the thermal off the cliff edge.

The next day they really did close the parks. No-one had any idea what that was going to mean in practice however, so together with an Irish dorm-mate (are Irish ever not great travellers?) we drove in to try our luck. The ranger at the gate also seemed unclear what the shutdown entailed,¹ but said we could go in “today”. From the sounds of it we just got lucky; others were supposedly turned away, while others were allowed in but later evicted by different rangers. This was a Tuesday, and at that time people expected things to be resolved by Thursday—no doubt the parties of the 6 weddings apparently planned inside the park that weekend were particularly hopeful, as well as the climbers with months-long camping plans who were being told to leave.

Still, we counted our blessings and enjoyed a much quieter hike to Vernal and Nevada Falls. There was no way I wasn’t taking advantage of any break—only the day before I’d optimistically paid for a year-long national-parks pass, and the chances of breaking even on my investment by visiting 4 parks were now looking rather slim.

So, it was time for a holiday from my holiday—time to chill out without too much pressure to explore or plan the next few day’s travel, and time to hack if and when I felt like it. San Francisco was the perfect spot for this: I’d already visited in the past so wouldn’t feel like I was missing out if I didn’t spend all day exploring, and of course it has a thriving tech community. Add to that a couple of friends in town and the deal was sealed.

It was glorious: I rented a nice room via AirBnB, perfectly located in a great area near Alamo Square and on a couple of bus routes; I went to meetups and pitch events; I had a lot of great coffee and food; I hung out with friends, and wrote code in cafes.¹ Nerd-wise it was exactly what I was looking for, although no-less intimidating (“Hi, what do you do?” “Well, I wrote WireShark, but now I’m…” to pick a random example). I can definitely imagine working in SF for a few years to turbo-charge your development, but by the same token it is also easy to see why people feel the need to work such crazy hours just to keep up, which doesn’t seem sustainable or healthy.

I did of course still do plenty of walking—it’s a very pretty and walkable city. My charmed run of weather continued, barring a single morning downpour while I was doing laundry, in flagrant violation of the apocryphal quote “The coldest winter I ever spent was a summer in San Francisco”.² The lack of touristy activities unfortunately meant that I took very few photos; the car show pictured was a random find on a walk through the Castro when for some reason I did happen to have my camera with me.

Millennium Park and Willis Tower

The weather was already looking glorious, and I headed into town to hunt down, first and foremost, the “bean”! (Its actual title is the cloud gate). This surprisingly-funky sculpture is just one of many in Millennium Park, and on a sunny Saturday I was not the only one enjoying it.

The bean was definitely a highlight, and I couldn’t help giggling when I realised you could also stand underneath it. Afro beats at a near-by sound stage provided a backdrop for the rest of my exploration.

That afternoon I yet-again braved the crowds end endless queuing for Willis Tower, still the second-tallest building in the United States. It was a pretty familiar story for me by this stage however: great views, but by the time we got up there I was more or less done.

To unwind I wandered the streets a little, although on a Saturday evening in the business area the streets were pretty quiet. There is a collection of sculptures by famous artists, all roughly on the same intersection, and I hunted those down. My favourite was undoubtedly a horse by Picasso, a 3-dimensional instantiation of his minimalist line-drawing style, but up close rather massive and physical. I even watched some cyclists using it as a ramp at one point! Apparently this is a bit of a mascot of the city and they even dress it up in team colours for sports events at times. Close by was a Miro, and around the corner one by Dubuffet, known colloquially and for obvious reasons as “Snoopy in a blender”.

Rain and Architecture

The following day the weather gods had deserted me, and I had to run for the train downtown. After grabbing a coffee I found a river architecture cruise—for those unaware, Chicago is renowned for its architecture (and in particular as a pioneer of the skyscraper age). The old city was razed almost completely by a fire in 1871, followed by architects flocking from all around lured by the blank slate offered and the promise of modern building materials (steel and concrete instead of wood).

The tour was impeccably run by a volunteer guide (a river docent), although I was less impressed having to buy a poncho!

The rain only got heavier, and I splashed out on a spectacular lunch at a gastro-pub. The rain eased slightly so I had a bit of an explore around the hipster area of Logan Square.

Then we all sat back and waited for the Boston bus. Soon enough a bus pulled up… and ignored us, while we watched the driver eat fruit salad in her seat. About 10 minutes later she did briefly emerge to say that she wasn’t going to Boston, before locking the door again. Confusion reigned, as both our boarding then departure times came and went (observed silently by fruit salad lady. I never did figure out what route she was driving). Eventually, a bus that had pulled up behind her, and also denied he was going to Boston, did relent and agree to take us—he did not look happy about it, and I suspect we were eating into his home-time. Missing: one NY-Boston Bolt Bus, possibly still out there. We did not have wifi for the journey either. First-world problems abounded.

Owing to my lack of advance planning (spontaneity has its downsides) my hostel was actually in Everett, a suburb about half an hour by train out from the city centre, and then a bus ride from the train station (they did have a free shuttle once you’d checked in). The hostel actually felt more like a half-way house than a typical backpackers, but on the positive side it was located in a Brazillian district and all-you-can-eat Brazillian BBQ joints abounded. I considered that a net win, really.

Boston and the Freedom Trail

I headed into the city the next day with a friend from the dorm, on a brutally hot day (I think it pushed 40°C). The main thing to do, apparently, is to walk the Freedom Trail; a walk along sites of historical significance associated with the American Revolution. Most of these sites seemed to involve Paul Revere, often in rather tenuous connections—his first-grade teacher’s first house? I’m probably making that particular example up, but coupled with the heat and an overloading of history, side-notes to an event to which we felt no personal connection anyway, it started to feel that way. But apparently I am not the only one to be slightly ignorant of the history!

After lunch we eventually managed to extract ourselves from the air-conditioned bliss of a pub, where after a walk through the financial district we parted ways for a bit and I hugged the shade on a wander through a leafy upper-class neighbour hill, around the river, and back along the main shopping and entertainment district.

Nerding out in Cambridge

One of the things my dorm friend and I had bonded over was that we were both, well, nerds (“engineers”, in polite speak). So, coupled with some cooler weather we were both a lot more excited to head in to Cambridge to check out Harvard and MIT.

We explored MIT first, found the hilariously jumbled Stata centre, then headed for the museum for the best part of a couple of hours. Highlight here: an original LISP machine¹!

We took the subway another couple of stops to Harvard, but having missed out on the tours (and not aided by the near-complete lack of signage) the highlight was probably free liquid-nitrogen ice-cream samples.

I spent some time in the nerd-heaven that is the MIT bookstore, before going to a Clojure meetup at the Akamai headquarters.

Several people had mentioned that Harvard was the prettier place to see, with a beautiful historic campus, while MIT was just a mish-mash of buildings. Visually that may have been true, but accurately or not the impression we both got was rather different: Harvard felt elitest, old-money, look-but-don’t-touch, marvel at the number of US presidents among our alumni, etc. MIT, on the other hand, felt much more meritocratic: an absolute buzzing hive of activity, where if you were smart enough and prepared to work hard enough there could be a place for you. Who knows if that’s accurate in either case, but as inspiration it certainly hit home.

So, from Montreal it was time to head south across the border. It wasn’t an auspicious beginning, as the train was delayed and we all queued up the middle of the station while all around us everything else (the non-Amtrak trains) ran on time. Meanwhile, a gentleman behind me muttered how typical that was, how poor these trains were unlike in the golden days, and so on. No-one had much idea what was going on either; we all had our luggage, but periodically an Amtrak employee would walk up and down the line, mumbling something about reserved seating while looking away from us. My gentleman friend said absolutely no way was he going to try and find his luggage in central station, he wouldn’t trust them as far as he could throw them and he was holding onto it thank you very much.

Eventually I found a seat. “May I sit with you?” said a young lady behind me. Of course I replied; “Great! Could you put my bag up in the rack then please?”. I think I was used! Conversation after that was quite pleasant, coupled with nice views of autumnal colour through the countryside, and luckily since the journey was around 10 hours! She was a manager from a software development shop in Finland, which instantly gave us some common ground. At one point she asked about my chromebook and I was half-way through mentioning that I’d turned it into a passable development environment, before reflexively halting… even today, “Linux” isn’t that well-known. I needn’t have worried: Linus is a national hero in Finland!

Finally we disembarked, bleary-eyed and weary, into the cacophony of central station. My new Finnish friend was returning to New York, and kindly guided me to the subway I needed. A saxophone and bass busking pair were playing a driving rhythm at ear-splitting volume. It was almost hypnotic, but we had to keep moving. Even with guidance I still somehow jumped on the subway in the wrong direction, before realising several stops down and changing direction. Eventually I found the hostel (complete with a fellow Tasmanian in my room!), had a late-night taco from a truck nearby, and crashed.

New York attractions on the cheap

New York has a deserved reputation for being expensive, but while all the museums, galleries, buildings etc aren’t individually expensive they quickly add up just because there’s so many of them! If you are travelling on a tight budget it is possible to save significantly here though:

• Statue of Liberty? The regular Staten Island ferry goes right past it, at a fraction of the cost of a tour.
• Several museums are actually donation-only; the signs make it look like it’s $20, but you can drop $1 and waltz past the dirty looks.
• Those with an actual admission fee usually have a free afternoon.
• There is apparently a superb view from the cable car.

I did none of this (I never even made it to the cable car at all, which I am disappointed about). Instead, I bought a city pass book of tickets, which saves you $80 over 6 attractions, if you go to them all. Why would I do this? Because it also means you can jump a whole lot of queues (not all of them; there’ll still be a wait for the security check, etc). I was already well and truly over queuing, and the prospect of an entire week more of it easily made me open my wallet! Later walking past MoMA while the queue for free admission stretched around the block was enough to validate my decision.

Seeing New York

I had roughly two types of days in the city. Some days I just cruised different neighbourhoods, wandering around soaking up the place, checking out restaurants and cafes, and doing a lot of walking. Then there were the days I spent at various capital-A Attractions. New York has a lot of these, but as I hinted above, I found the queuing, the security checks, and so on all rather exhausting and border-line dehumanising at times. I’m definitely glad I made the effort (perhaps my recollection will be better than the experience!), but at the time I enjoyed the on-foot exploration much more.

It is certainly not news, but I’ll repeat it here: New York is a fantastic city to explore on foot. It’s always my favourite way to explore and experience a place, but some are much more suited to it than others. The subway goes everywhere, and often (random stoppages aside), the footpaths are nice and wide. Perfect.

Day 1 was all about exploration. Subway to Union Square to pick up a US SIM card, then walking past all the brownstone to a spectacular lunch at Momofuku. Off to a good start! That was followed by an espresso and (because I couldn’t pass it up) a nitro cold brew—this is exactly what it sounds like; a cold-brewed coffee, served from a nitrous keg so it looked like Guinness, and tasted exceptionally creamy to boot.

After a walk through Central Park I wound up at The Met, where I only had a short amount of time which was probably perfect for not over-doing my non-art-appreciating brain.

Highline Park

Don’t ask me where I’d heard about it, but I was really looking forward to the Highline Park. At the time I had the impression it was a bit of undiscovered secret, but by the time I arrived it had most definitely been discovered! The highline is a stretch of elevated railway, previously abandoned and now converted into a park and walk-way, with great views along the way to boot. The beautiful weather and popularity at least ensured there were plenty of lunch vendors available.

That afternoon I hit up the MoMA. I really enjoyed this; for some reason I do seem to have more of an affinity with (some) modern art. The lower floors also contained a lot of applied and industrial design, milestone computer artefacts (including an annotated print-out of the first Pacman code!), sound art, and so on. The upper two floors were more of a strain: this is where the big names reside, the Picassos, the Van Gogh, the Warhols… and apparently nearly all the visitors, screaming and running from piece to piece to take a picture with their iphone or ipad and running on to the next. I did not last as long on the top floors.

With the sun down the next stop was the Rockerfeller Centre. It’s not as tall or possibly as famous as the Empire State Building, but everyone says the views are better and I won’t dispute that.

Brooklyn, 9/11 Memorial, Empire State Building

There was a simple plan for the day: walk the famous Brooklyn Bridge, have a quick look at the 9/11 Memorial and make my way north on foot.

The first step was a success, and the bridge is a great walk. Unfortunately, I naively completely misunderstood what was going to be involved in a “quick look” at the memorial, and this consumed a lot more time than I realised. Even finding it is not simple, with many paths blocked off due to construction and no signage anywhere, and lots of coalesced groups of people that looked confusingly like queues to somewhere. Proving that capitalism was not defeated in the attacks, however, were swarms of people hawking commemorative booklets, tea-towels, and so on. Once it had been found… of course it involved more queuing. Lots and lots of queuing, and lots and lots of security checks, and at least two points at which your donation was strongly encouraged. The memorial itself is in fact a great and somber tribute to those who died, but after all that I did not stay long.

Naturally, I had to make it up north for more queuing! This time, it was for the Empire State building. Great views, but by this time I was done and after a late lunch and drink to unwind I turned in early.

Wall St, Central Park

If yesterday was all about crowds and queuing, the following day was going to be all on foot! Starting by walking from the hostel through hipster Williamsburg, then catching the subway down south to Wall Street in order to walk north. Up through Chinatown, Little Italy, NoLita, a late lunch at Mission Chinese. I did fit in an hour or so at the Natural History Museum, which was probably ideal to fit in the dinosaurs and meteor exhibits. It has been a while since I’ve thought more than passingly about dinosaurs—I felt like a kid again, staring in amazement! Being able to touch a massive lump of iron metal that fell from the sky, characteristic crystallisation and all, was also very cool.

Dinner was at the famous Katz’s Delicatessen. This was a recommendation from a friend, but of course it was also “where Harry met Sally”. I was completely unaware of this, until I realised that there was a large arrow sign at the table diagonally across from me, pointing down and telling everyone that’s where it all happened (amusingly, it had a solitary girl looking rather bored). It is a chaotic place and the ordering system takes a little while to figure out. For a start (this is not obvious!) there are multiple queues, and if you are there for the famous pastrami-on-rye (as you should be), you have about 4 to choose from. You give your order (and pay on the way out, based on the ticket they’ll write on), the cutter will get to work, first giving you a taste while he works, you tip him a couple of dollars, and off you go with a veritable mountain of unctuous meat! It wasn’t long since an equally large lunch so I was rather nervous about the quantity, but that’s what I train for!

Walking in the wrong direction for a while afterwards gave me another chance to explore a rather swanky area.

River Cruise

There was one attraction left in my City Pass book; a river cruise. I had mixed expectations about this, and was rather underwhelmed (by way of example, we stopped by the Statue of Liberty for a while, while the guide rhapsodised at length about “freedom”—a concept people from outside NY are not expected to understand apparently—without once mentioning France. We passed the One World Trade Centre and he complained that it used be called the Freedom Tower, but it’s all about making money after all and not everyone appreciates freedom so we hadda change it. And so on).

From here it was back on foot, through the Chelsea Markets, and down to Greenwich Village. This is an incredibly funky area still, and as a music fan it was amazing to stare at the who’s-who list of artists who have frequented Cafe Wha?.

Another meal at Momofuku rounded out my final day in the big apple!

This trip was easily the longest I had ever taken on my own, and I wasn’t sure exactly how I was going to go. Until now it hadn’t even been an issue as I caught up with friends in Vancouver and stayed with others in Alberta, and only had a few days to myself in Toronto. By pure fortune I was again blessed with company in Montreal; a good friend from Vancouver had just moved to start attending MacGill University with a few days free before term started, and a friend from home put me in touch with a friend of hers now in Montreal, who then spent 3 days showing us both around!

The company and the good times made Montreal one of my favourite cities of the trip, but I’d like to think that it would have been fun regardless. Toronto had been nice to explore, but perhaps I had had misplaced expectations or didn’t stumble over the right parts as it didn’t feel like anything more than that. On landing in Montreal at night though I checked in and headed out to find dinner, and there seemed to be a real crackle, and electricity in the air. I’m sure the brutal winters are a different story, but the summer certainly seemed like fun!

Graffiti Galore

New friend Liv picked me up from the hostel and introduced me to the Bixi Bike bike rental system. I later saw the same bikes under different names in later cities, but this was my first encounter, and what a brilliant system it is, even if the weather restricts their use to only 6 months in Montreal. Montreal also proved to be quite bike-friendly, and we followed a number of bike paths to brunch.

Well-fed, we wandered the old quarter, then headed north checking out funky shops and landmarks. There had recently been a graffiti festival and there were great works everywhere.

That night all three of us went to dinner in the gay village nearby… apparently it was fetish weekend, and the street fashion parade was nothing if not educational (no photos, sorry).

Tam Tams

After a much-needed sleep-in I met up with Vancouver friend Alissa and we walked up to the Tam Tams on Mount Royale. The mountain park looks over the entire city and we had quite a nice view from the top. The Tam Tams themselves is a summer weekends gathering, including several epic drum circles, lots of circus arts, DJs, some ancient battle reenactments, and generally a vibe somewhere between a hippie festival and the beach.

For all the colourful sites, the pick of the lot might have been watching the drum circle when I noticed an older shirtless guy stick a joint in one nostril, cover the other with a finger and inhale with such effort his head was on his knees by the time he’d finished his drag, then repeated it with his other nostril. I don’t think I’ve ever seen eyes roll that far back into someone’s head.

Joined again by Liv later that evening we hit up a whisky bar for a while, then a brew pub, then finally at 2 or 3am—my very first poutine. Surely there can’t have been a more appropriate time or frame of mind for it! (There was quite a queue for it too)

Ile St Helen

Feeling rather subdued the next morning, I met Liv again for the famous Schwartz’s Smoked Meats. This is one of those institutions that has a lengthy queue outside and ultra-efficient staff inside. Fortunately with local help I knew how to order (they ask how fatty you want it, for example). When it arrives it’s essentially a few tiny bits of bread, utterly dwarfed by the mound of meat inside, and of course nigh-impossible to eat without it all falling apart.

Even with that now lining our stomachs the residual hangovers, the grey weather and a public holiday to boot all meant today was not going to be energetic. Picking up Alissa we caught the subway over to Ile St Helene, now probably most notable as the somewhat-decaying former site of Expo 67. We listlessly explored the biodome, before jumping on the subway again to check out Atwater where we explored the market and found dinner.

Atwater

On my own for my final day, I managed to find a hair cut then set off to explore the Italian district. This didn’t excite me as much as I’d hoped, and I worked my way slowly back down, exploring on the way, then cycled back to Atwater for more of an explore.

I checked out a restaurant tip for my final dinner, where for some reason I was the only customer (apparently it is normally pumping) and the bored waitress was quite chatty and gave me plenty of tips for my next destination… New York!

I’m not sure what to make of it, but I suspect it means I’ve been more productive recently.

I was to stay with a friend and fellow programmer I knew from my time in Vancouver but who worked remotely out of Canmore, a little town not too far from the famous Banff national park. The view of Calgary from the plane was overwhelmingly flat, and I even fell asleep not long after catching the bus. When I woke up… suddenly I was in mountains! Waiting for him to pick me up after the bus dropped me off, gazing at the scenery all around, it wasn’t hard to see why he chose to work remotely. When even the bus stop has vistas like that your town has something going for it, and the views only got better that week.

The other thing that stood out as the traffic dawdled by, and apparently the first thing I said when he arrived… “Does everyone here drive a truck??”. Yes, yes apparently they do. We’re in Alberta! Trucks, or if you’re a tourist, a big motor-home. That was pretty much all I saw on the roads for a week.

Not surprisingly, the view from their balcony was even better.

I was staying with them for a week, and we sketched out 4 days of hiking.

Ha Ling Peak and Spray Trail

I didn’t stray too far from home base on my first day. Firstly, in Canmore you simply don’t need to go far to find something amazing, and secondly… driving. They very kindly let me borrow their car (truck), but I have only ever driven on the left side of the road and even though I was carrying an international driver’s license¹ I wanted some time to get my confidence up. With that in mind, we mapped out directions to Ha Ling Peak, visible from the balcony as a sheer rock face but more straightforwardly accessible from the rear.

Getting there involved a short trip through the town, which apart from turning my windscreen wipers on multiple times I survived without incident, then heading up through a saddle to the trailhead behind. The climb itself was nice; I overestimated my hiking fitness and blew out of the gate, through some nice forest, before having to concede defeat and slow down significantly as the route gave way to exposed rock, and my cardiovascular system humbled me.

The view from the top: out of this world.

After descending I kept driving for an hour or so along the Spray Trail, periodically exclaiming in amazement, until I realised it was useless to stop for photos every time.

By the time I turned back for home I was a little fatigued from the driving, and luckily was tailing another car as it suddenly slowed and pulled way out around a bend. I followed cautiously—only to have a moose on the road block out the full opposite side of the vehicle! Unfortunately I’d already put the camera away and by the time I’d wrestled the case from my backpack, and the camera from the case, I only got the one shot of its backside retreating back into the forest. Just the strangest combination of massive size (even for a small one), ungainly looking proportions, and majestic grace in its movement.

Lake Louise, and the Plain of 6 Glaciers

The next day, and time for my first entry into Banff itself. Lake Louise is a massive tourist draw, which this time does rather detract from the experience; both the huge crowds clustered around the shore, and the enormous ugly hotel right behind you. It is definitely pretty, just a little hard to appreciate in that atmosphere.

Fortunately as soon as you start walking even a little way around the shore you leave the vast majority behind, and while I never felt isolated as I headed up the valley it was much more pleasant.

First stop was the plain-of-6-glaciers tea-house where I had a snack and watched the extremely well-fed and cheeky ground-squirrels climb up onto the tables. I pushed up closer to the glacier (and having climbed off-track up a scree-slope, unfortunately missed witnessing an avalanche), then returned back down the same way, before looping around via Lake Agnes and Mirror Lake.

The other big tourist draw in the area was Moraine Lake, but unfortunately I didn’t have enough time left and had to head back.

Stanley Glacier

Stanley Glacier it turns out, is actually in British Columbia again. It was a fairly long drive to get there, followed by a moderate climb through a burned-out area (although not unpleasant views).

Suddenly, you emerge into a large basin, with Stanley Glacier in the corner and massive walls all around you—looking back it was actually quite hard to see where the trail even entered! Naturally I pulled out the camera to start snapping again… and found a completely dead battery. Never skip a recharge opportunity. The latter photos you see are all from my phone, unfortunately.

Moraine Lake. And Bears.

One final hiking opportunity left! Another big draw in the area is the ice fields, but we decided this was a bit much driving to be worth it and instead settled on a “compromise” of Helen Lake, including a pass on the edge of the fields. After the requisit stops for photos on the way there (no, apparently I’m not yet habituated to the scenery!) I finally found the trailhead… and a sign saying it was shut due to a bear + cubs in the area. Bugger.

The fallback plan I came up with was Moraine Lake, since I’d missed it a few days earlier, and I headed back up the highway and miraculously managed to find a park (unlike Lake Louise, it is rather light-on in that department). It is still a large tourist draw, but this time it detracted far less from the stunning view.

Once I’d taken enough snaps I headed around the lake to a short hike I was keen on… and another sign warning of bears in the area! This time the advice was not to hike in groups of less than 4, but after waiting a while no further candidates appeared and I gave up. Not an ideal way to wind up my visit, but I can hardly complain given the rest of the week!

I had some great times on this trip, but without detracting from the rest at all I don’t think anything quite topped Alberta.

Day 1: Bearings

My favourite way to explore a city is on foot, and there was perfect weather for it. My hostel, which was student accommodation most of the year, was quite close the Ryerson university campus and Dundas square. Lots of buskers around; it turned out there was a festival on. After a bit of an explore around this area I took off up Yonge Street, the (formerly) longest street in the world depending on your accounting method. One unexpected highlight before I left Yonge was a book store dedicated to cook books!

Turning left I wound back via the fashion district (I think?), the rather striking museum facade, the college area (looking very Ivy-league, and apparently portrays Harvard in movies more frequently than Harvard itself does), Kensington markets, and the financial district.

I closed the day up on the roof of the hostel when I realised it was accessible!

Day 2: Niagara Falls

I had booked a tour to Niagara falls through the hostel, and went down at the annointed time to wait… and wait and wait, introducing myself at some point to the gentleman also clearly waiting. After some time the hostel rang on our behalf, to discover they had left without us! This was one of the best pieces of luck to hit yet: they refunded our money, booked us onto a tour leaving at lunch time for free, and in the mean time gave us a free bus tour of the city, which added context to parts I’d wandered the day before and introduced me to others I hadn’t.

The tour itself I think is fairly standard; we stop for an ice-wine tasting along the way, pass through the creepily Stepford-wives-like town of Niagara on the Lake (so fastidious about preserving its appearance that tourist buses have to park on the outskirts and no franchise is allowed in the centre¹), and finally the falls themselves.

The volume of water is definitely breathtaking (as is the Disney-like shops and attractions surrounding it); the mist rises up at least as high again as the falls themselves, and if you take the boat out will drench you even further out than you’d expect.

There was an interesting side-note to the boat ride: the operators have a long-term lease that is now an incredible bargain. Recently there was a stand-off when the park authorities tried to raise their rent, the boat operators refused and threatened to dismantle the dock they had built themselves if evicted, and in the end were allowed to continue as-is. The lease expires very soon however, and not surprisingly they were not awarded the next contract.

Day 3: Whiskey District and People

Let us just say that I may have miscalculated my schedule! Again in glorious weather I wandered south this time, to the whiskey district (another one of those areas you’d recognise, apparently the same the world over; a long-time industrial complex of some description, now rennovated into lots of trendy cafes and art galleries) and Lawrence Market.

After lunch I headed over to the iconic CN Tower, with no inkling this might have been a bad idea. My Niagara companion had assured me he had visited and didn’t have to queue too long at all, and at no point had I considered that perfect weather and this being Saturday would alter anything (in truth, I had already lost track of what day it was anyway). Result: I had to queue to buy a ticket, then kill the next hour somehow, then spend another hour and a half in the queue getting up there, getting jostled by noisy kids and steadily going insane.

The view: yes, definitely impressive, but I also did not wait too long in getting out of there.

After this assault on the senses I wandered back across town to the ferry terminal, to check out the islands offshore. Unfortunately, the experience was remarkably similar: perfect weekend weather meant I was far from alone, and by the time I got over there I took a few skyline photos back at the city, and got the hell out of there too.

The busker festival was in full swing by the time I got back, so I did wander up Yonge for a while taking in some performances and grazing on street food for dinner, but getting thoroughly peopled-out and heading back to the hostel.

In August of 2013 I quit my safe and secure, well-paying, flexible job that gave me a lot of time to travel for… the incredible uncertainty of starting a company with a close friend but no clients lined up.

There is unlikely to be much international travel in my near future.

So, there was only one thing for it: one final excursion in between comfort and the unpredictable future. The only question remaining was where. A number of options were suggested or otherwise came to mind: back to South America perhaps, or to see friends in the Tyrol (and I still haven’t seen much of Europe!). In the end though, I decided to go to another great continent I’ve barely touched: North America.

I booked flights into Vancouver and out of LA, with a vague plan to work my way across Canada then back across the US. In amongst that I thought I could mix some of the classic cities and areas on offer with the majestic national parks, and with any luck catch up with a few friends scattered around the area. Any attempt at a more detailed plan only made it clear that 3 months was not going to be sufficient, so I just took off to figure it out along the way and take what opportunities presented themselves.

Apart from the general across-and-back opportunity afforded by this schedule, Vancouver was the first stop because I’ve spent a fair bit of time in Vancouver over the last few years, and I still love it and still have friends there (although a few are starting to migrate out, sadly). My starting point was thus always going to be more a social visit than anything touristy, although I did mount a decent investigation of the thriving espresso culture in the city (more on this aspect of my travels later!)

Victoria

The main exception was a quick visit to Victoria on Vancouver Island, the capital city of British Columbia. I splashed out on the sea plane across as well, well worth it for the views of the stunning coastline and islands.

Victoria itself was really pretty, plenty of good cafes if you’re in to that, a rather touristy but nice historical area, and an attractive harbour.

My biggest win by far here was whale-watching, which wasn’t even on my radar before visiting. When I realised it was a possibility I spent a bit of time at the hostel researching the options in their brochures, set off down to the harbor to ask more detailed questions… and promptly went with the first operator I met, who offered a discount and immediate departure (it turns out there are a lot of whale-watching operators, lining the harbour on both sides). Never have I made a better impulse decision! The guide explained that there are a couple of resident pods and also a transient pod of Orcas in the area, and that we would check in on the latest siting of the transients before moving on to hang out with the residents. As things transpired though, we quickly found the transient group and in a very playful mood, with only a couple of other boats for company so we spent all our time watching them. They repeatedly came quite close to the boat (one passed right under), and I even saw one do a 3/4 breach (not even close to being photographed though). Photographically it is another story, as anyone who has been whale-watching will probably tell you: most pictures you grab are the result of a quick swivel just in time to catch the animal on its way down again—the famous black humps.

In an effort to get the blog back up to date, this post will cover the entire 6 weeks in one hit, and be mostly photographs. Unfortunately this trip was also when I finally got frustrated with my photography skills and camera; I have since upgraded the camera, although not the ability. I apologise for the sub-standard quality of most of the pictures.

TL;DR: I loved South America. It is quite large though, and we attempted to cover way too much of it in 6 short weeks, but I wouldn’t change much (just so long as I get to go back).

Argentina

We flew into Buenos Aires, and owing to that flight winding up while we were away, were to fly out of Chile. The first step is paying the delightfully honestly named “reciprocity fee” (ie, “your country charges a fee to our citizens, so we’re going to make some money off you too”).

Struggling with jetlag we set off exploring from the hostel, and unfortunately managed to find the financial district — initial impressions of the city weren’t that glamorous. The morning after we met a returningtraveller at a hostel Spanish class, and she showed us around. Via a fasionable district (whose name I forget), and quite a lot of wine, we spent the afternoon in bohemian-hip Palermo then had some of the best steak I’ve ever eaten at La Cabrera. Secondary impressions were much different: Buenos Aires is a sexxxxxy city, full of extraordinarily attractive people and very European-styled architecture.

We also learned that wine carafs are shaped like penguins (“Pinguinos”). I don’t know why either, but we saw quite a few of them.

The next day travel exhaustion hit like a hammer, and we were woken about midday by the hostel staff evicting us. Blearily we wandered out with our guide friend again and explored La Boca, including the famous La Bombonera stadium where Boca Juniors (Maradona’s club) play.

On a whim, based on nothing more than a conversation at the hostel the day before, we found the bus station that evening and managed to figure out how to buy over-night tickets to Mendoza — wine country! Buses are a real treat (if you pay enough, which is still fairly cheap); much roomier than aeroplanes, plus they feed you, play you a movie, and even bingo! (Good for practicing our still-limited Spanish).

Mendoza was fun, and the main activity is hiring a bike and touring the myriad vineyards. The local drivers seem well acquainted with slightly unsteady tourists late in the day. Not many photos from that experience however.

Patagonia

From Mendoza it was time to head south for the Torres del Paine… and on no planning ahead that meant struggling with online booking forms to eventually find flights, then fly back to Buenos Aires and spend the night in a bar at the airport before an extremely early flight south to El Calafate. The plan was to catch a bus over the border into Chilean Patagonia, to the town of Puerto Natales. Unfortunately—it’s almost as if this was planned—there is a single bus a day, which leaves just before the only flight in arrives. So, there was nothing for it but to find accommodation, and book a tour to the Perito Moreno glacier (the only attraction in El Calafate!). The glacier is amazing, but they also leave you there to try and see ice breaking off, for 7 hours. There is a single cafe above the viewing area: again, it could almost have been planned! We instead stuck it out, and as luck would have it got chatting to 3 others (one couple and a solo) who we wound up walking with.

The next morning we caught the bus to Puerto Natales, found accommodation and visited the regular trekker’s information session held at a local gear shop. The rest of the day was shopping, hiring equipment, and packing.

The trek itself was magnificent, and naturally the photos don’t capture it at all. We were quite lucky with the weather too. While waiting for a ferry across the lake to the trailhead we went on a short walk to a nearby waterfall… just as the heavens opened and we got completely soaked through. I will admit to a few misgivings at this point, but after that we really had amazing weather (although “amazing” also includes wind gusts that made it impossible to stand).

As we were leaving the park we suddenly realised that the Navimag ferry was an option—we had heard about it before leaving Australia, but our then-“plan” meant we would miss it. Back in Puerto Natales though we managed to get tickets, then had to book an extra night (meaning two extra days since the ferry left the evening of the following day). The scenery through the fjiordlands is unquestionably magnificent, but it is similarly magnificent for 3 full days; that plus the extra days in Puerto Natales wrote off close to a week, which in a 6 week holiday we could ill-afford. I would recommend it if you have the time, but it is the only thing I would change about my own experience, really.

The boat arrives in Puerto Montt, for which the Lonely Planet’s inspiring advice is “catch a bus to Puerto Varas”. So we did that, and despite our early arrival meaning nothing was open we did find a local hostel that agreed to take us on a tour. Puerto Varas has quite a lot of Germanic influences, but our tour was exploring the volcanic mountains and the nearby lake. My personal highlight was noticing—and only just avoiding stepping on—a rather rare Darwin’s tree frog, which as you can see in the photo has almost lost its green colour in order to again blend in with the now-dead foliage.

At the end of the day we caught another plane up to Santiago.

Santiago de Chile

We arrived late in the evening and had no accommodation booked, apart from a “tip” from the hostel guy in Puerto Varas. So we caught a taxi to this tip, who was rather surprised to see us and certainly in no position to offer lodging. He did take us on a walk, rather bemused at this point, a few blocks away where a few people all stuck their heads out of second-story windows, and eventually a door opened and we had a bed for the night. It turned out that a few people on the street offered spare rooms for rent, mostly to university exchange students, but at the time it was quite a bizarre experience.

Initial impressions of Santiago were… rather underwhelming. It was only later that we realised it was Easter weekend and most people were away (and those remaining were obviously not in a mood to open their businesses). We experienced a much more lively Santiago when we stayed our final night there before flying home!

Peru (Macchu Picchu)

From Santiago we flew up to Cusco in Peru, the old capital and the launching place for trekking to Machu Picchu. This was the only thing we had booked in, and because the main Inca Trail books out months in advance (and we were not that organised) and has visitor limits, we had opted for the less well-known Salkantay route. I don’t regret this for a second; it was less populus, hits a higher altitude, and generally definitely felt like an adventure (albeit, unlike in Patagonia an adventure where most of our luggage was carried for us and all meals prepared for us!). It is also a porter route, which means lots of horses go on the path, which meant it was incredibly muddy.

We had a few slight detours around fresh land slips, none more so than at the end where we had to catch a jeep to the spot where most of the road had simply fallen down the steep incline, get out and nervously walk the 1 metre of remaining road to the other side and catch another jeep.

Eventually we arrived in Aguas Calientes for the morning visit to Machu Picchu itself. I honestly did not expect to be that impressed; granted the photos are spectacular, but you typically only ever see the same slightly-elevated view, and I cynically expected it to have more a plasticy theme-park feel to it. Boy was I wrong: not only is the setting so much more stunning in person, but the scale and detail up-close is just breath-taking. We also climbed Huena Picchu for an areal view.

Bolivia

We had booked flights into Bolivia before our trek, and had arrived back in civilisation (internet) to find that they had been cancelled, so we had to find a bus instead.

We spent a day or so exploring La Paz, which is a rather beautiful city at quite a high elevation, and with markets that go even higher with no end in site! Following that we caught a bus to Uyuni for the standard 3-day 2-nights “salt flats of Uyuni tour”. Our jeep had 3 separate travel-pairs, and we all got on like a house on fire. The scenery is just remarkable too, from the train graveyard (so much more fun than you’d expect!), to the bizarre lack of visual perspective on the salt flats themselves (watching people posing photographs is almost as much fun as posing them yourself), to the surreal colours and rock formations in the deserts. Bolivia is beautiful, and definitely on my list of places to visit more thoroughly.

We finished by crossing the border back into Chile in San Pedro de Atacama.

Valparaiso

For our final stop of the trip we had splurged on nice accommodation in Valparaiso. Unfortunately we were unable to splurge on transport there, where our usual last-minute strategy let us down and we had to watch our Canadian travel buddies have a leisurely dinner and fly there, while we sucked it up on a 24 hour bus ride to Santiago, then walk to a different bus station and catch another bus. Then we couldn’t find a taxi at that time of night, but some local guy loitering around was more than happy to pretend to be one.

Valparaiso is a beautiful town and amazing to wander. Set on some very steep hills, it was a once-booming port town that fell into disrepair as the port business dried up, only to be reinvigorated as a tourist town. It still has that slightly edgy, slightly gritty feel to much of it, while the other distinguishing feature is the astonishing amount of graffiti murals covering the steep hills.

Once upon a time (2010), I fulfilled a life-long dream and went trekking in Nepal. It was, happily since “life-long dreams” are frequently let-downs, amazing. It was so good that I knew I had to go back, and it was only going to be a matter of time… in fact, less than a year later I started planning with another travel-junkie friend.

This time, we did the Annapurna circuit. You will find this loop listed in many “World’s top treks” lists, but the reality is that it is rapidly losing its lustre as they build a road in parallel to the path. While this benefits many villages along the route, for whom tourism is positive but not their main source of income, its appeal to tourists is declining. Our guide’s tip as the “next big trekking circuit” was the Manaslu circuit.

Yes, our guide. In 2010 the three of us arrived with very little plan and assuming we’d find our own way. We were talked into hiring a guide by our hotel, which turned out in a stroke of luck to be a fine decision as we wound up with a fantastic guide, Shyam. So, as we were planning this trip I contacted him again, got a quote, and we paid our deposit.

Not long after, my new travel companion had friends telling her we’d been ripped off, that you can get a guide for half that in Kathmandhu, and so on. I felt bad for not even questioning it, but we were committed at this point. We surreptitiously asked fellow trekkers along the way and indeed most were paying significantly less.

However! There are definitely times when you get what you pay for. Without going into details (the stories aren’t mine to tell, for the most part), at the end of the trek every single other group was asking for Shyam’s business card. His experience and knowledge, English language ability, easy-going manner and generosity with his knowledge even for those who weren’t paying him was, shall we say, in contrast to the other guides we met along the way¹. I would recommend him to anyone.

I won’t go into much detail on the trek itself; check out some photos below. We didn’t have the best weather, probably because we were too early and on the tail of the monsoon season, and unfortunately had a complete white-out at the top of the pass.

It is a completely different experience to trekking around Everest. The Khumbu region is stark, and most tea houses are solely for tourism and close in the off seasons. You rapidly gain altitude, and stay there for most of your trip. In contrast, the Annapurna circuit is a constant gradual climb, starting much lower (and hence hotter). You stay in actual villages, which is a different experience (and you certainly eat better—not many vegetables grow in the Khumbu!).

It was a great trip, despite the weather, and won’t be my last.

You also may or may not be aware of the calc facility embedded in Emacs. It seems to be one of the lesser-known features, and even among those who do know about it, many are unaware how sophisticated it is. You can invoke it with M-x calc, or C-x * * and at first glance it seems quite primitive… until you read the manual, at which point it can instead seem impenetrable!

Hidden underneath the spartan, stack-based interface is matrix and vector arithmetic, date and time calculations, unit-aware calculations and conversion, symbolic manipulation, and on it goes.

Nearly a year ago Andrew Hyatt posted a series of tutorials that explored a lot of the corners, but one of the weaknesses of Google+ is that content can be hard to aggregate or find again.

So, when I remembered the other day I went back and dug up all the ones I could find, in order to provide a pointer back to them! Just as I was completing the list I discovered that there is already one archive on the EmacsWiki, but one more is certainly not going to hurt discoverability.

The Tutorials

• Hex-decimal conversion (and general base conversion)
• How old is David Hasselhoff? Basic date calculations. Also, toggle digit grouping (number formatting)
• More date-time manipulation In particular, Unix epoch conversion.
• Random numbers
• Unit conversion, including defining custom units.
• Pi and digit precision
• String to binary conversion
• Exact fractional arithmetic
• Algebreic manipulation
• More algebra
• Finance
• Calculus (Integrals and derivatives)
• Bit manipulation

What’s most interesting, at least for me, was that I did it en-masse via Emacs. This blog is a set of static files generated by Jekyll Bootstrap, as I have written about previously. A post is simply a file with a bit of YAML meta-data up the top, optionally including tags applying to that post:

---
layout: post
title: "Sample yaml front-matter"
tags: [SomeTag,AnotherTag]
---
Body of the post...

To simplify the repetitive work involved I wrote some elisp to take a bunch of files, query for the relevant tags, and add them to the front matter:

(defun mh/tag-posts (tags)
  "Apply tags (specified interactively) to the files marked in
dired, merging with existing tags.  Assumes the use of
jekyll-bootstrap, ie with the `---' delimited yaml front-matter
at the beginning."
  (interactive (list (read-from-minibuffer "Tag(s), comma-separated: ")))
  (let ((tags (split-string tags "[, ]+")))
    (labels
        ((do-tags (f)
                  (with-current-buffer (find-file-noselect f)
                    (save-excursion
                      (goto-char 0)
                      (if (re-search-forward "^tags: \\[\\([[:alnum:], ]*\\)\\]" nil t)
                          (let* ((existing-tags (split-string (match-string 1) "[, ]+"))
                                 (combined (union existing-tags tags))
                                 (new-tags (mapconcat 'identity combined ",")))
                            ;; replace-match wasn't working for some reason:
                            (goto-char (match-beginning 1))
                            (delete-region (match-beginning 1) (match-end 1))
                            (insert new-tags))
                        (progn
                          (goto-char 0)
                          (forward-line 1)
                          (re-search-forward "^---" nil t) ; find second occurence, assuming first is on line 1
                          (goto-char (match-beginning 0))
                          (insert (format "tags: [%s]\n" (mapconcat 'identity tags ","))))))
                    (save-buffer))))
      (mapcar #'do-tags (dired-get-marked-files)))))

The guts are in the last line: I mark files using dired in Emacs, then invoke mh/tag-posts, which applies a function to every marked file. This function then just has to take care of a few corner cases such as the tag already being included (by using union from the cl common lisp emulation library to avoid repetition), and when there’s no tags: [] entry at all, in which case it must add one.

This is a quick hack, but I’m throwing it out there in case anyone knows of a better solution (or better yet, if I’m missing something obvious in the standard distribution).

The Hibernate ORM for Java includes a query-by-example facility: you create a template object, setting the relevant fields to the values you’re interested in, then it retrieves all records that match that template (fields left un-set are ignored in the query).

To the best of my knowledge, the Django ORM doesn’t support this style of querying. I found myself wanting it though,¹ and this post is what I came up with.

Implementation

The implementation is fairly simple: we use the meta attribute to get the list of fields, and create a filter query from all non-None fields. As simplification, I’m currently ignoring non-direct fields and many-many fields (in part because the object I was working with hadn’t been saved yet). The id is also skipped, because of course that should be unique!

class QBEManager(Manager):
    """A custom manager that adds query-by-example functionality (inspired
    by Hibernate)"""
    def similar_to(self, template):
        """Find all objects that have fields matching those in the template
        object.  Fields that are set to None are ignored."""
        vals = {}
        for f in template._meta.get_all_field_names():
            _,_,direct,m2m = template._meta.get_field_by_name(f)
            if not direct or m2m or f == 'id':
                continue
            val = getattr(template, f, None)
            if val is not None:
                vals[f] = val

        return super(QBEManager, self).get_query_set().filter(**vals)

Then it can be used just like a regular manager:

class Config(Model):
    # ...
    objects = QBEManager()

Using it anger is up to you:

def some_view(request):
    # ...
    c = Config()
    c.start_time = datetime.date(2012, 3, 27)
    c.length = 5
    # get all configs that start on 27/3/2013 and have length 5:
    matching_configs = Config.objects.similar_to(c)

Improvements

This solved my problem, but it’s probably not fit for prime-time just yet. In particular, handling many-many fields could be added for objects that are already saved (and hence are able to have m2m fields established).

It is probably also better off written as a mixin; in particular, my case was actually based off a GeoManager, and a mixin would enable both base managers to be used. Something like:

class QBEManagerMixin(object):
    # ...

class QBEManager(QBEManagerMixin, Manager): # ...
class QBEGeoManager(QBEManagerMixin, GeoManager): # ...

But bear in mind that features that look irrelevant in isolation may be relevant in combination. For example, if the class is an XOR of k input features, each of them by itself carries no information about the class. (If you want to annoy machine learners, bring up XOR).

A fun paper, distilling the “folk wisdom” aspects of machine learning that take up the majority of time into pithy and amusing quotes. Others have already commented on the value it.

Context ctx = ContextFactory.getContext(Context.nullContext(|))

then hitting ; straight away takes you to getContext(...);| (mercifully, it has been a while).

I was missing this in python, where it was taking me a whole extra keystroke or two, every time!, to jump to the end of the line to insert a colon. In the python mode I was using semicolon is already electric, so this time I had to advise it:

(defadvice python-indent-electric-colon (around python-electric-colon-autoplace (arg)
                                                activate)
  (if (and (looking-at ")")
           (not arg))
    (progn
      ad-do-it
      (transpose-chars 1))
    ad-do-it))

I.e. assuming we’re not inserting multiple colons, insert then transpose to leave the colon and cursor outside… but in fact, just writing this I realise I didn’t really go far enough at the time! So I’ve just updated to this version:

(defadvice python-indent-electric-colon (around python-electric-colon-autoplace (arg)
                                                activate)
  (if (and (looking-at ")")
           (not arg))
    (progn
      (move-end-of-line nil)
      ad-do-it
      (newline-and-indent))
    ad-do-it))

Insert correctly, indent, then ready to keep going on a new line: much more efficient!

The awkward nature of pecking at glass, with no tactile feedback, also means that there is a certain necessity for invention in order to recapture the usability of a regular keyboard, and some nice ideas come out of it. Some of these make use of the different nature of the surface itself, such as “swipe” typing, while others might provide a greater efficiency (in the average case), such as automatically inserting punctuation, aggressive history-based auto-completion, etc.

I swear by SwiftKey keyboard, and it has been worth every one of the five dollars I think I spent on it — in fact of all the examples I just provided they actually implement all of them (now that they are developing swipe typing as well).

The other day at work, at a regular keyboard, I realised I was actually missing one feature in particular, but luckily since I was in Emacs it only took 10 minutes to implement! That particular feature was the way punctuation is correctly inserted in SwiftKey; if you choose an auto-completion it will also insert a space, for example leaving you with “word |” (where | is the cursor; note the trailing space). However, if you then type a comma it is inserted in the correct place: “word, |”.

Basically, you never have to think about punctuation or spacing too much, as it just works. Here was the result of my procrastination:¹

(defun mh/electric-punctuation ()
  "Tidy up whitespace around punctuation: delete any preceding
  whitespace and insert one space afterwards.  Idea stolen from
  the SwiftKey android keyboard."
  (interactive)
  (when (looking-back "\s+" nil t)
    (delete-region (match-beginning 0) (match-end 0)))
  (call-interactively 'self-insert-command)
  (just-one-space))

In other words (assuming this function is triggered by punctuation), if there is some preceding space then delete it, insert the original character, and add a space afterwards.

Then to trigger this whenever I type a comma, semicolon, or period within regular text, I add it to the base text-mode hook:

(dolist (punc '(?, ?\; ?.))
  (define-key text-mode-map `[,punc] 'mh/electric-punctuation))

I will admit that it is taking me a little getting used to (it turns out I have ingrained habits of typing punctuation+space, resulting in lots of double-spacing!), and there’s room for improvement. In particular, I suspect it needs a special case to handle numbers (you don’t want “3.14” to end up as “3. 14”). I really appreciate SwiftKey for the way they continually sand down corners, and there are no doubt lots of corner-cases remaining here. Still, it was the sort of easy-to-implement idea that I love about Emacs!

The other day I randomly listened to an episode of the Joe Rogan¹ podcast, featuring Chuck Liddell and Enson Inoue. In truth I chose it because of Chuck, but he only played a bit-part and I wound up gripped by the intensity of Enson.

Inoue was a pioneering mixed martial artist, with a reputation in Japan for exceptional bravery — so much so, that they gave him the nickname yamato damashii.

This is a peculiarly Japanese term, and the wikipedia page describes a lot more connotations than I was aware of during my own time in Japan (the only time it was applied to me was a joking gee-up when I hesitated at the top of a hairy ski run!). It is typically translated as “Japanese Spirit”, with strong Samurai and martial overtones, and it was quite unusual for it to be applied to a foreigner. Much of the intensity from the podcast was his description of how he felt obliged to live up to this honour, and the mental preparation he went through: literally prepared to die in the ring.

One discussion in particular stood out for me though, as he was discussing the subtlies in meaning of the term, and his own interpretation (paraphrased):

People compare it to bravery, but it’s different from that. A brave man will feel no fear… a man with yamato damashii spirit feels fear, but accepts it and moves forward anyway. A strong man is one who doesn’t feel pain, but one with yamato damashii feels pain but persists in what he knows he must do.

A lot of the discussion in general was around mental attitude and how that affects your choices and actions. I think that one will stay with me for a while.

Update: Right on cue, Ross Enamait just published a post on the same themes of confronting doubts and fears by first accepting them, using quotes from MMA and boxing (Mike Tyson’s trainer). Less of Enson’s acceptance of death itself, meaning he has nothing to lose by not stopping in the ring, but the same message I personally took away!

I recently read Pragmatic Thinking and Learning: Refactor your Wetware, after it had sat on my virtual bookshelf for a while. I took a lot of notes of things to try, and generally got a lot out of it (interestingly, Amazon commenters have turned on it a bit).

This post isn’t about the book itself though, but one idea in it: out-sourcing your memory and organisation to something like a wiki or notebook that you can use as an “exocortex” (their terminology).

As an Emacs user and finding that org-mode is increasingly taking over my day-to-day organisation anyway, it seemed the obvious choice. One requirement of your exocortex was that it be always accessible, and fortunately there is a mobile client. Unfortunately it might not work the way you immediately expect it to, but this can be rectified.

Org-mode

If you’ve used Emacs for any length of time, there’s a fair chance you’ve encountered org-mode. Org-mode is a fantastic piece of software, relatively new in the Emacs scheme of things, still growing at quite a clip and already with an amazing breadth of capability.

This can be a problem though: it is massive. Try any tutorial (and there are a few) and there’s a fair risk you’ll get overwhelmed: hierarchical notes — ok; notes can have tags — fine; what are drawers and why?; tables seem cool; wait, you mean you can do spreadsheet calculations in them too?; notes can have dates associated too; you can clock in and out; column mode; time-tracking;… on it goes! Many of them also introduce it in the context of GTD, which can be a further overload if you’re not already familiar with it.

So if you have previously been overwhelmed, or you haven’t tried it yet, I would offer two pieces of advice:

1. Just start using it

Skim the most basic tutorial you find, then just start using it for notes; for heavens sake don’t try and master everything in one sitting. It was originally designed for scientific note taking, and it does this brilliantly. Get used to the hierarchy and commands for manipulating it, then maybe start adding functionality as you need it: perhaps you go back over notes and start classifying sections by adding tags, for example. Its table editing facilities are also quite intuitive, but just get used to those first if you find yourself needing a tabular display; don’t worry about formulae.

2. Trial and error

One reason the interface is so intuitive is you don’t need to remember a tonne of commands in order to be productive straight away. Org achieves this by over-loading most key combinations, with the result that you can typically achieve what you want by trying one of them, undoing if that didn’t work, and trying the next most obvious candidate.

Basically, try: M-<return>, M-<arrow key>, or <shift>-<arrow key>, or even M-<shift>-<arrow key>, and if it doesn’t do what you want: C-/ to undo! C-c C-c is always worth trying too. Org is quite context-sensitive, so if you’re in a heading you will typically operate on a heading, or if you’re on a date you can manipulate the date, and so on.

Depending on the exact context, those commands are likely to shuffle headings around or insert new ones, promote or demote them, add tags, changes dates, or change the priority. Again, if something happened that you didn’t expect, don’t worry: undo, and move on.

Mobile-org

Emacs isn’t too mobile without some effort, but fortunately there are org-mode clients for the iphone and for android. I have no experience with the iphone app, but apparently they operate similarly.

However these do not offer, out of the box, the seamless experience you might expect: changes made on the phone do not silently propagate back to your org files. This is partly because the apps have been designed to include the agenda functionality (I have alluded to dates before; the basic idea is you can create your project notes in the hierarchy that suits, add deadlines and so on, then org-mode will generate an agenda from this), and partly because you can pick from a number of different server backends.

There is confusion here too, because a common back-end is Dropbox — but this is not the same as storing your org-files themselves on dropbox so they can be accessed from several computers!

So first of all, there are a few things to understand:

• You (initially; we will fix this) need to explicitly push any changes you make to the server, and pull any changes from your phone back again.
• This process also involves generating addtional files for the mobile client.

A common request is to have a directory of org files on Dropbox so they are accessible from multiple computers (say, home and work), and also to use dropbox as the server for the mobile client. This is possible, but it does mean keeping two copies of the files in dropbox (for Emacs and for mobile-org). I use a sub-directory of my org directory to keep the mobile files:

;; My org files:
(setq org-directory (expand-file-name (file-name-as-directory "~/Dropbox/org")))
;; Files for mobile-org:
(setq org-mobile-directory (expand-file-name "MobileOrg" org-directory))

Next, you will need to tell org-mode what files it should treat as mobile. By default this consists of your agenda files (again, not covered here), but you can set it to anything. I do use the agenda list, plus one file of general notes:

(setq org-mobile-files (cons (expand-file-name "notes.org" org-directory)
                              org-agenda-files))

Having changed the location of the org directory, you will find you also need to change one last variable (don’t worry about what it does if you don’t mind, it’s mostly transparent):

(setq org-mobile-inbox-for-pull (expand-file-name "from-mobile.org" org-directory))

You are now mostly there; go ahead and set up mobile-org for dropbox and read the rest of the documentation.

Automation

As mentioned, if you are used to something like Catch where changes on the phone get transparently synced with a server, you might find the mobile-org operation confusing. Fortunately, there is some elisp code in the FAQ for mobile-org-android to regularly push and pull (yes, in our case this does essentially mean copying from one dropbox folder into a subdirectory, but this is not always the case!). In case it ever moves, or just if you’re lazy, here are the snippets you need to include;

Combine this with a frequent update period in the mobile client, and you will have a pretty smooth experience!

[alias]
    dlp = diff ORIG_HEAD..FETCH_HEAD
    llp = log ORIG_HEAD..FETCH_HEAD

These stand for “diff last pull” and “log last pull” respectively. When dealing with third-party submodules in particular, such as my my emacs config where most extra code is contained in submodules, I’ll often just run git pull to update them. After that, either out of curiosity or occasionally afer the fact if I’m tracking down some changed behaviour, I want to know exactly what changed.

So this is what those two aliases accomplish. When you fetch, git creates an extra couple of refs referring to the HEAD before and after the fetch. You can then just use these to examine the new commits via diff and log.

I’m sure they’ve been proposed numerous times in the past, and may even be built-in for all I know, but they’ve been quite useful for me already so you’re welcome to them too!

Writing patch: a couple of hours. Getting permission from employers to submit patch: a couple of months. Contributing to Emacs: priceless.

Introduction

Matlab is not my favourite language. In fact, it’s one I’ve largely managed to ignore, which is remarkably lucky since it is also quite prevalent where I work.

One day though, I was morosely looking over a… rather organic mass of Matlab, and got so depressed I started procrastinating. I started watching the latest episode of Emacs Rocks, which was demonstrating a new minor mode to successively expand a selection around greater semantic units (for example, just on a page of text it would first select the word under the cursor, then the sentence, then paragraph, and so on).

This was much more interesting! I started hacking on it to make it work with Matlab. Admirably, it also came with a test suite and a request that all contributions include tests. These I finished on the weekend, fixing up a couple of corner cases they found along the way.

Flushed with a sense of achievement, I pushed to github and submitted a pull request.

Trials and Tribulations

All stories are fundamentally about conflict or struggle in some sense, and this is where mine started. The maintainer thanked me, and more-over informed me that his package had already been accepted into Emacs itself, and therefore so would my small contribution! I have had a long love affair with Emacs and virtually live in it, so I was ecstatic.

Well, ecstatic, and also rather nervous. The Free Software Foundation requires that all contributors must assign copyright over to them. Under local law, I do not in fact own the copyright, my employer does. Now, I work in a research organisation, and as it is their life-blood they are understandably twitchy about signing over IP. Clearly my patch was not going to leak valuable secrets, but it did put me in the position of needing someone in a large bureaucracy to accept responsibility for something that goes against the organisation’s grain, with absolutely no upside for them.

Merely getting to the stage of discovering what level of people might have the authority took a decent amount of time, and after the first apologetic rejection I did lose hope and let the trail go cold for a while. Eventually though I found someone who agreed, with experience releasing projects as open source and an “ask forgiveness not permission” attitude, to whom I will remain inordinately grateful. Actually getting the physical signature then took a week or so longer, but I was finally done.

Aftermath

All was not finally done, as it turned out. I gleefully informed the (patient, long-suffering) maintainer that all was finally good to go. He in turn pulled the patch in, and informed me that my tests were all failing! As it turned out, I had written my code against Emacs version 23 and he was using version 24, and the Matlab mode had received a major face-lift in between with many breaking changes. This was mainly due to me simply using what shipped with Ubuntu, but I did derive some hollow amusement that it felt like the process had taken so long my changes were all out of date! At any rate, I tweaked the patch to work against both version, and now I’m finally in a package, that at some point in the future will be in Emacs. In a very small way.

Colophon: Emacs Rhapsody

Browse any geek news site for long enough and eventually someone will mention Emacs, and someone else will reply “oh yeah, it’s a great operating system, but needs a good text editor, har har”. Very droll. The joke, in my opinion, is on them though because it is an operating system!

Some time, read up on the history of the Lisp Machines and watch a movie or two of one in action. For various reasons, not all technical, this vision of computing did not take off and we are poorer for it. Every aspect of the system, down to the metal, was not only written in Lisp but could be examined and changed by the user. Emacs makes sense in light of this: it’s essentially a poor-man’s Lisp Machine.

Emacs Lisp is not the nicest Lisp, but hacking on Emacs is fun both because what you write immediately improves how you work, and because you have that instant-turn-around where (nearly) every bit of functionality can be examined and updated, all without restarting.

Finally, if you’ve never really looked into Emacs, or only used it under sufferance as part of a Unix course or something, have a look at the Emacs Rocks movies some time. The guy is single-handedly doing an amazing publicity job, showing just what can be accomplished and surprising even veterans. And if I may say so, he also makes a very gracious and incredibly patient project maintainer :)

In the last, err, over a year since last posting I have, at least:

• Been back to Nepal (trekking around the Annapurna Circuit this time)
• Also spent 6 weeks in South America.

Unfortunately the weather wasn’t great for photos in Nepal this time, so there’s probably only one post there. I don’t have many great pictures from South America either, but I can probably manage to eke out a few more posts!

In the mean time though, I migrated this blog. It spent the first couple of years of its life on posterous, which was a great platform at the time but more recently seemed a bit neglected while the team cast around for ways to keep it alive before accepting a pity-acquisition from Twitter. I decided to move it to a new platform — the rest of this gets a bit nerdy, sorry!

Jekyll-bootstrap, github-pages, and the migration process

I decided to move to a static-site option, giving me a bit more flexibility as well as control over my data: in other words, all of the site is generated once per post, then served as simple files rather than from a database. I also wanted to be able to keep it in git, although that’s almost a given with most site-generator options.

There’s plenty of options for site generation, but one of the most popular alternatives is jekyll. By itself this only offers a flexible but bare site generation framework; jekyll-bootstrap adds a framework tailored for blogging on top, along with themes based on the now-ubiquitous Twitter-bootstrap. I also looked pretty hard at Octopress, again based on jekyll, but made a slightly-arbitrary decision to go with jekyll-bootstrap (Hyde deserves mention too, including because it’s built on technologies I’m more familiar with). I think this decision was at least in part due to the greater integration with github — see below — although in reality this is almost a non-issue.

For those of you who don’t know, github offers a static site hosting service, including custom domain names if you wish. All that’s needed is a specially-named repository, and they will serve it as a website for you. As a bonus, they also have jekyll integration and can generate your site for you (of course, otherwise you just generate it yourself before pushing to git-hub, which is what I ended up doing. If at any time github becomes less suitable I can easily host the files myself, or use Amazon S3, or any other alternative.

Firstly though, I wanted to take all my previous content with me. This wasn’t too challenging, but did involve a couple of fiddly steps — read on for the details!

Migration steps

1. Download previous posts from Posterous! Posterous does have an API, and Jekyll includes a posterous importer. The problem with this is I wanted my images as well, and the existing importer didn’t handle that. I hacked it a bit to download the images as well, into a separate directory for each post and as a stop-gap measure just source them in a HTML list. (I submitted a pull request for the importer patch, but haven’t heard back. It’s pretty hackish, but you’re welcome to it.)

2. This got ugly pretty quickly, so I wrote another hack to upload each image directory to picasa.

3. Unfortunately, Picasa’s “embedding” functionality seems to be limited to effectively a link to the album on their site, or a shitty flash slideshow. So, I took the smallest lightbox implementation I could find, and wrapped it with this plugin for pulling the image links from picasa’s RSS feed (it saved me writing my own at this point!), and tied them together with a bit of jquery. Instant lightbox, hosted by picasa.

4. Well, not quite. I wrote a simple little jekyll plugin to wrap some boilerplate HTML around the picasa album-ID, and used an emacs macro to help replace the old image links with that tag. Then I pushed to github, and immediately realised that of course for security reasons they don’t run extra plugins… so I used sed to swap the tags for the boilerplate, and moved on (fortunately it wasn’t a complicated plugin).

5. Fixing up backlinks. In some posts I have referred to earlier posts, and posterous uses a different link format to jekyll (/title-slug vs /yyyy/mm/dd/often-a-different-slug). I was testing out regular expressions ready to write a script to find all links, trawl the _posts directory looking for which it was likely to refer to and update it… when I realised that there were only 7 such backlinks. So I just fixed them up by hand! (Geek-cred lost, but time gained).

6. Almost there… I switched to one of the themes provided with jekyll-bootstrap, originally due to Mark Reid. I’d like to do my own at some point, but in the mean time it’s nice and clean. I like it.

And now we’re done! In theory this means I can go back to posting more frequently; in practice it at least means I have less excuses. I’d like to post more technical content as well, but I make no promises. If you’re interested in what I’m reading, I tend to share technical links on Google+ at the moment.