def stripped(f):
    for l in f:
        line = l.rstrip()
        if line and not line.startswith("#"):
            yield line

with(open "foo.txt", "r") as f:
    for line in stripped(f):
        print line

Lion ships with an app called Wi-Fi diagnostics. It allows you to monitor Wi-Fi networks your computer is connected to and collect various kinds of information. This utility could be very useful for finding problems or doing research on your Wi-Fi networks.

Enabling OS X Screen Sharing via the CLI

If you’re already at your Mac’s desktop, you can simply turn on Screen Sharing. But if you (1) didn’t plan ahead or (2) worry about security, here’s how to turn it on.

Converting from a string to boolean in Python

>>> import ast
>>> ast.literal_eval("True")

John Carmack Slashdot Archive

All of Carmack’s comments on Slashdot, from 1999 to 2008. See also his .plan file updates from 1996-2007 here.

Deploying node.js on Amazon EC2

Creating an AWS “micro” instance running Ubuntu 10.04LTS and setting up a Node.js server with deployment managed by Capistrano and process management by Bluepill.

If I Launched a Startup

Collection of linked advice from Ryan Roberts, aka @StartupLawyer.

Clever Algorithms

“This book is concerned with ‘clever algorithms’, which are algorithms drawn from many sub-fields of artificial intelligence not limited to the scruffy fields of biologically inspired computation, computational intelligence and metaheuristics. The term ‘clever algorithms’ is intended to unify a collection of interesting and useful computational tools under a consistent and accessible banner. ”

Deft

Emacs mode for quickly browsing, filtering, and editing directories of plain text notes, inspired by Notational Velocity. It was designed for increased productivity when writing and taking notes by making it fast and simple to find the right file at the right time and by automating many of the usual tasks such as creating new files and saving files.

Warehouse-Scale Computing: Entering the Teenage Decade

ACM keynote from FCRC 2011 by Luis Barroso Alvarez. Covers both Google-specific stuff and general datacenter industry trends.

Keymaster

A simple micro-library for defining and dispatching keyboard shortcuts.

Dropbox recently made news by updating their Terms of Service to explicitly state that (if asked by law enforcement) they can and will decrypt your files and hand them over.

As set forth in our privacy policy, and in compliance with United States law, Dropbox cooperates with United States law enforcement when it receives valid legal process, which may require Dropbox to provide the contents of your private Dropbox. In these cases, Dropbox will remove Dropbox’s encryption from the files before providing them to law enforcement.

This seemingly contradicts some language in their marketing materials that would suggest that user files are stored in an encrypted format on Dropbox’s backend and are completely inaccessible even to Dropbox employees.  This is similar to what competing tools like Jungle Disk do explicitly:  the files are encrypted before they ever leave your computer and the decryption key is only ever stored locally.

Unfortunately, it now seems possible that Dropbox is not encrypting files on the backend, given the comments from their CTO at the end of the TUAW article linked above.  Given this, it seems prudent to look for a solution that allows you to keep using Dropbox, but encrypt sensitive data before it’s ever uploaded to the cloud.

Enter EncFS

EncFS is an encrypted pass-through filesystem that’s implemented in userspace.  What’s that mean?  Pass-through means (basically) that the encryption is done on a file-by-file basis, as opposed to an encrypted block device, such as TrueCrypt or Apple’s DMG image format, where you have one monolithic encrypted “bucket” into which you put your files.

In a nutshell, an EncFS volume looks (mostly) like an ordinary folder on your hard drive. Anything you save into that folder gets transparently encrypted and stored in a separate folder elsewhere on your hard drive (the “backing store”). As far as any applications know, they’re working with plain, ordinary, unencrypted files but these files are never actually read or written on disk. Only the encrypted versions are actually accessed behind the scenes.

It’s this backing store functionality that makes EncFS a perfect companion to Dropbox. You read and write your files on your hard drive as normal (inside the virtual folder), and EncFS transparently stores the encrypted versions in your Dropbox folder, where they are synced in real-time to the cloud. Since each file is encrypted individually, Dropbox can sync changes incrementally instead of having to sync the whole disk image. To be fair, they do differential syncing, so only the actual changed bits of the image get transferred over the wire, but to my mind syncing the individual files is still better. File-based encryption also makes it much smoother to access your files from multiple machines at the same time, since Dropbox can resolve conflicts at the individual file level.

“Implemented in userspace” means that EncFS runs as an ordinary process (a daemon), like a web or FTP server, as opposed to a kernel module.  This is done through use of the FUSE (Filesystem in Userspace) project, which is a kernel module and provides the hooks into the file handling subsystem that something like EncFS, sshfs, or GMailFS requires to work their magic.

MacFUSE

FUSE was originally (and still is) a Linux project, using a loadable module for the Linux kernel to provide the necessary hooks to userspace filesystems like EncFS.  Fortunately, Amit Singh from Google (author of the terrific book “MacOS X Internals“) developed a MacOS X port, MacFUSE, which provides the same userspace API, but hooks into the Darwin kernel instead.  Thus, we can use EncFS, sshfs, and most if not all of the other FUSE filesystems on the Mac.

Installation

• download MacFUSE from Google Code. Don’t worry about the latest version being from 2008. It still works fine with (32-bit) OS X 10.6. Run the installer and click through, accepting the defaults.
• download EncFS pre-built for OS X from Google Code.  You want the file named something like “EncFS-10.5-2.0.0.zip” even though it says it’s for Leopard.  Run the installer and again accept all of the defaults.  This will install the EncFS FUSE module and the related CLI tools (in /usr/local/bin)

Creating Your First Secure Volume

On my machines, I keep the backing store (the actual files on disk) in my Dropbox, which is ~/Dropbox/Secure. The virtual folder where I can access these files is in ~/Documents/Secure.

$ encfs ~/Dropbox/Secure ~/Documents/Secure
Creating new encrypted volume.
Please choose from one of the following options:
 enter "x" for expert configuration mode,
 enter "p" for pre-configured paranoia mode,
 anything else, or an empty line will select standard mode.
?> p

Paranoia configuration selected.

Configuration finished.  The filesystem to be created has
the following properties:
Filesystem cipher: "ssl/aes", version 2:1:1
Filename encoding: "nameio/block", version 3:0:1
Key Size: 256 bits
Block Size: 512 bytes, including 8 byte MAC header
Each file contains 8 byte header with unique IV data.
Filenames encoded using IV chaining mode.
File data IV is chained to filename IV.

-------------------------- WARNING --------------------------
The external initialization-vector chaining option has been
enabled.  This option disables the use of hard links on the
filesystem. Without hard links, some programs may not work.
The programs 'mutt' and 'procmail' are known to fail.  For
more information, please see the encfs mailing list.
If you would like to choose another configuration setting,
please press CTRL-C now to abort and start over.

Now you will need to enter a password for your filesystem.
You will need to remember this password, as there is absolutely
no recovery mechanism.  However, the password can be changed
later using encfsctl.

New Encfs Password: dontuseme
Verify Encfs Password: dontuseme

The Results

Creating and manipulating files inside the virtual directory is just like any other directory on disk. Both UNIX and MacOS applications see nothing different about the file.

$ echo Hello World > ~/Documents/Secure/hello.txt

$ ls ~/Documents/Secure
hello.txt

$ cat ~/Documents/Secure/hello.txt
Hello World

However, if we examine the actual files stored on disk inside the Dropbox folder, we will see that both the name of the file and its contents are encrypted securely.

$ ls ~/Dropbox/Secure
CUUNdUhk0bp-k-eswFVtxG6D

$ cat ~/Dropbox/Secure/CUUNdUhk0bp-k-eswFVtxG6D
ѹ?7????3?V{1}=????|??D?x

A screenshot from my Dropbox folder

Caveats and Notes

An EncFS volume does not mount on boot, so you will need to either manually re-run the encfs command above, or create a startup script of some kind (possibly with AppleScript) to do it for you automatically.  Personally, I like this behavior — especially on my laptop. If my Macbook Pro is ever stolen, the thief would need to reboot to get past the screensaver. All of my sensitive data is securely encrypted inside the EncFS backing store and would be completely inaccessible unless the virtual filesystem is mounted (which would require my passphrase).

Probably the biggest downside to the EncFS solution (or any client-side encryption scheme) is that it breaks access to your files from the Dropbox web client, from mobile devices, or from any computer that doesn’t have EncFS installed. You’ll want to carefully consider the risk and perhaps only use your encrypted folder for especially-sensitive files that you’re not likely to need to access from a non-EncFS-enabled computer.

Amazingly, there actually does appear to be initial EncFS support for Windows. See here for details. It appears to at least work on Windows XP, but there seem to be issues with Vista and Windows 7.  Better than nothing, and quite an impressive hack given the major differences between the Windows OS kernel and Linux. At least MacOS X is based on Mach, so there’s a UNIX-like kernel underneath!

Below is a basic script that will add a newly-built ESXi host to vCenter and apply the common settings for my lab.  Still requires manual configuration of the management interface IP info (although that could be automated through a scripted install).

# Connect to vCenter
Connect-VIServer vc1.packetslave.local

# Add host to vCenter and setup NTP
$host = "tc2.packetslave.local"
Add-VMHost $host -Force -Location (Get-Cluster HA) -User root -Password mypass
Add-VMHostNtpServer -VMHost $host -NtpServer 'time.apple.com'

# Enable vMotion
$vk = Get-VMHostNetworkAdapter -VMhost $host -VMKernel | where {$_.DeviceName -eq "vmk0"}
$vk | Set-VMHostNetworkAdapter -VMotionEnabled $true

# Connect to my NFS datastore
New-DataStore -VMhost $host -Name mynfs -Nfs -NfsHost mynas -Path /vm

# Rename the default port group
$vs = Get-VirtualSwitch -VMhost $host -Name vSwitch0
$pg = Get-VirtualPortGroup -VirtualSwitch $vs -Name "VM Network"
Set-VirtualPortGroup -VirtualPortGroup $pg -Name "Production Network"

# Add my additional port groups
New-VirtualPortGroup -VirtualSwitch $vs -Name "Test Network"
New-VirtualPortGroup -VirtualSwitch $vs -Name "Storage Admin"

# Tell DRS to re-balance VMs including the new host
Get-DrsRecommendation -Cluster (Get-Cluster HA) -Refresh

#

Out of the box, ESXi 4.1 U1 will not detect this NIC, nor will it be detected with the 2.103 driver.

• any URL matching /foo/ will have HTTP authentication applied by the backend web servers.  We want to use the logged-in user as the sticky criteria for user sessions.
• All other URLs should use an ACE-inserted cookie called Backend

Solution

Use the “Authorization” HTTP header for sticky, since (with Basic authentication) this will contain the Base64-encoded username and password of the authenticated user.

Note: this assumes that basic HTTP load balancing is already configured.  See my previous post for an example.

sticky http-header Authorization USER_STICKY
  timeout 60
  replicate sticky
  serverfarm HTTP_FARM
sticky http-cookie Backend COOKIE_STICKY
  cookie insert browser-expire
  replicate sticky
  serverfarm HTTP_FARM

policy-map type loadbalance http first-match HTTP_LB
  match FOO http url /foo/.+
    sticky-serverfarm USER_STICKY
    action urlrewrite
    insert-http X-Forwarded-For header-value "%is"
  class class-default
    sticky-serverfarm COOKIE_STICKY
    action urlrewrite
    insert-http X-Forwarded-For header-value "%is"

After the most recent incident of this at $DAYJOB, I wrote a plugin for our Opsview server (which runs on top of Nagios) to check the “last changed” and “last saved” times of a device using SNMP.

It’s not perfect:  notably because IOS updates the “last changed” time every time you enter/exit config mode, whether you actually made any changes or not.  This is a recipe for false positives.  Unfortunately, there’s no easy way around this without the plugin actually downloading the device configs and comparing them.  Given the multitude of authentication and other challenges this would present, I’m happy to let tools like Rancid and Solarwinds NCM solve them instead of making the plugin much more complex.

Available on GitHub here.

After praying to the Google deity for a while, we hit upon the following KB article: vMotion fails on ESX/ESXi 3.5 and 4.0 with some versions of nx_nic and unm_nic drivers.  The issue only seems to crop up if you have VLAN tagging enabled on the vSwitch to which the NIC is connected, and are using TCP segmentation offload (which is enabled by default).

The fix is to either create a new vmKernel interface for vMotion with TSO disabled (and without using VLAN tagging), or to upgrade the NIC driver in ESX/ESXi itself.  In our case, since this was a new environment, we decided to fix it for good and do the upgrade.  A quick download and a little vMA magic, and vMotion is now working flawlessly over 10gb.

You can find it on my Hacks page: here